1、吳育德陽明大學放射醫學科學研究所台北榮總整合性腦功能研究室Introduction To Linear Discriminant Analysis Linear Discriminant Analysis For a given training sample set,determine a set of optimal projection axes such that the set of projective feature vectors of the training samples has the maximum between-class scatter and minimum
2、within-class scatter simultaneously.Linear Discriminant Analysis Linear Discriminant Analysis seeks a projection that best separate the data.wSwwSwwJwtbt)(Sb:between-class scatter matrixSw:within-class scatter matrix)along of projection the is y(i.e.to subject aaxxaynTn1121)21222-(max2 1,k 2 1,class
3、 :,GiveniiuxuxEiixEiuiTiiii)(1,2 ,iuaxaEyEiTiTi2,1)()()()(222 ,iakaauxuxEauaxauaxaEuaxaEyEiTiTTiTiTTiTTiiTTiiiSol:LDA Fisher discriminant analysis2def2122121)21 )()()()(-()(222bbbbIbbbbVSVbbVSVbaSaaSaakkauuaaJTTTwTTbTTwTbTTTwhere ,=k1+k2and let TbuuuuS)(2121wSbVa,.,:problem eigenvalue generalize sat
4、isfies i.e.,000000 ,let wewhere111niiibwinbTwTvvVvvSSVSVIVSVLDA Fisher discriminant analysis11120011 1 since ,with max bbbbTb)(3)(2),(1)from.(3)(1 )1()2(1a since.(2)(let ,(1).)(1 )(or satisfies 001,.,211211221121222121211112121111111uuSuuSauuSkkauuauuuuSaaauuuuSaaSSavvvVawwwTTwTwbwnLDA Fisher discri
5、minant analysisLet M be a real symmetric matrix with largest eigenvalue thenand the maximum occurs when,i.e.the unit eigenvector associated with .11max1uuMuTu ,1u11,.ni ,iiiMniR in basis orthonomal complete a forms )0 (assume nTTTTTTsrnIMMM,.)()(000000111.11Proof:LDA Generalized eigenvalue problem.T
6、heorem 2nnnuuu Ru,.,11 1,1,212niiuuuu IF since max1111111max2112max21111111,TTTniiniiiniiniiiniiiTniiininiiiiiMuuMuuuuuuuMuuuMuMuMNowLDA Generalized eigenvalue problem.proof of Theorem 2If M is a real symmetric matrix with largest eigenvalue .And the maximum is achieved whenever ,where is the unit e
7、igenvector associated with .)0(max21xxxMxT,Then11kx 1)(,Rk1Cor:LDA Generalized eigenvalue problem.proof of Theorem 2LDA Generalized eigenvalue problem.Theorem 1Let Sw and Sb be n*n real symmetric matrices.If Sw is positive definite,then there exists an n*n matrix V which achieves nbwTVSVIVSV000000 a
8、nd 1TThe real numbers 1.n satisfy the generalized eiegenvalue equation:generalized eigenvector :generalized eigenvalueiwiibvSvSiviGeneralized eigenvalue problem.proof of Theorem 1Let and be the unit eigenvectors and eigenvalues of Sw,i.einiri,.,1,nnnwiiiwrrSrS.11.1,000000 or Now define then nU.1Mrrr
9、rUUUSUnnTwT00000000000011 where IUUTSince ri 0(Sw is positive definite),exist 21211000000nrrZ)(whitening IMZZUZSUZTwTTLDA LDA Generalized eigenvalue problem.proof of Theorem 1 Aof seigenvalue the are 1nnTAWW,.,000.00012111111)()(WWWWWIWWWWWWAAAAWWAWAWwwATTTTTTTTiii or )symmetric is fact InIWWAUZSUZU
10、ZSUZUZSUZUZSUZTbTbTbTTbT i.e.matrix W,unitary a exists matrix A symmetric real a exists symmetric is that Note)()()()()()()()(LDA#or or VSVSvvSvvSvSvSUZWVwbnnwnbiwiib000000.2.1111We need to claim:IWWIWWWUZSUZWUZWSUZWTTwTTTwT)()()(.1(applying a unitary matrix to a whitening process doesnt affect it!)
11、()()()(.21VSVSVSVSVVVSVIVSVVSVWUZSUZWAWWIVSVbwbwTTbTwTbTbtTTwT(VT)-1 exists since det(VTSwV)=det(I)det(VT)det(Sw)det(V)=det(I)Because det(VT)=det(V)det(VT)2 det(Sw)=1 0 det(VT)0Generalized eigenvalue problem.proof of Theorem 1Procedure for diagonalizing Sw(real symmetric and positive definite)and Sb(real symmetric)simultaneously is as follows:1.Find i by solving And then find normalized ,i=1,2.,n2.normalized 0)det(1ISSbwiiibwivvSSv1 niIvSvvkviwiiii,.,2,1)(,LDA Generalized eigenvalue problem.proof of Theorem 1
