1、ChapterChapter 6 6 Classify Neutral network model By their synaptic connection topologies and by how learning modifies their connection topologies pathwaysfeedbackorloopssynapticclosediffeedbackloopssynapticclosedNoifdfeedforwar.2.11.:2.:Supervisedlearning useclass membershipinformationoftrainingsam
2、plingsUnsupervisedlearning useunlabelledtrainingsamplingssynaptic connection topologieshow learning modifies their connection topologiesNeural network:synapses,neuronsthree dynamical systems synapses dynamical systems neurons dynamical systems joint synapses-neurons dynamical systemsHistorically,Neu
3、ral engineers study the first or second neural network independently.They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks.RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.M MX X),(M M
4、X XEquilibrium is steady state(for fixed-point attractors)Convergence is synaptic equilibrium.Stability is neuronal equilibrium.We denote steady state in the neuronal field Another forms with noise Stability-Equilibrium dilemma:Neuron fluctuate faster than synapses fluctuate.Convergence undermines s
5、tability6.10 0M M 6.20 0X X xF36.0 0F Fx xX Xn n M MN NWe shall prove that:Competitive AVQ synaptic vector converge exponentially to pattern-class centroid.They vibrate about the centroid in a Brownian motionjm mCompetitive learning adaptively quantizes the input pattern space characterizes the cont
6、inuous distributions of pattern.nR)(xpcentroidXXAVQ7.6,6.6.321jiifjDiDKDDDDnRnRPatternThe Random Indicator function Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorithms dont require this pattern-class information.Centriod KDDDDIIII,.,3218601)(j
7、jDDxifDxifxIj96)()(jDdxxpjDdxxxpjxCompetitive AVQ Stochastic Differential EquationsThe Stochastic unsupervised competitive learning law:106)(jjjjjnmxySm116)(xISjDjWe assume The equilibrium and convergence depend on approximation (6-11),so 6-10 reduces:126)(jjDjnmxxImjCompetitive AVQ Algorithmsmiixmi
8、,.1,)()0(1.Initialize synaptic vectors:2.For random sample ,find the closet(“winning”)synaptic vector)(tx)(tmj2212.136)()(min)()(miijxxxwheretxtmtxtm3.Update the wining synaptic vectors by the UCL,SCL,or DCL learning algorithm.)(tmjUnsupervised Competitive Learning(UCL)156)()1(146)()()()1(jiiftmtmtm
9、txctmtmiijijjicdefines a slowly decreasing sequence of learning coefficient)(samples10,000for000,1011.0,instanceFortxtciSupervised Competitive Learning(SCL)176)()()()()()(166)()()()()1(DjxiftmtxctmDjxiftmtxctmtmtxtxrctmtmjijjijjjijjDifferential Competitive Learning(DCL)196)()1(186)()()()()1(jiiftmtm
10、tmtxtySctmtmiijjjtjj)1(tySjjdenotes the time change of the jth neurons competitive signal.In practice we only use the sign of(6-20)206)()1()1(tyStyStySjjjjjjStochastic Equilibrium and ConvergenceCompetitive synaptic vector converge to decision-class centroids.May converge to locally maxima.AVQ centr
11、oid theorem:if a competitive AVQ system converges,it converge to the centroid of the sampled decision class.2161)(PrmequilibriuatxmobjjProof.Suppose the jth neuron in Fy wins the activity competition.Suppose the jth synaptic vector codes for decision class jmjD2260jmSuppose the synaptic vector has r
12、eached equilibriummean-zero is singalnoise236jjnmxImofbecausejDj jjjDDjDjDDjjRjDjmExconcludestheoremcentroidAVQthexdxxpdxxxpmdxxpmdxxxpdxxpmxnEdxxpmxxImEonExpectatioTakejjjjjnj:)()(246)()()()()()(:Arguments:The spatial and temporal integrals are approximate equal.The AVQ centriod theorem assumes tha
13、t convergence occurs.The AVQ centroid convergence theorem ensure:exponential convergenceAVQ Convergence Theorem:Competitive synaptic vectors converge exponentially quickly to pattern-class centroids.Proof.Consider the random quadratic form L25621002nimjiji)m(xLThe pattern vectors x do not change in
14、time.(still valid if the pattern vector x change slowly relative to synaptic changes.)626627()628()612)2()()()629jjiijijiijiijijijiijijijijDiijijDiijiijijLLLxmxmLmmxmmbecauseof mIxmnIxxmxmnijij The average EL as Lyapunov function for the stochastic competitive dynamical system.Assume:Noise process i
15、s zero-mean and independence of the noise process with“signal”processijm-x 316)()(3062jDjijidxxpmxLELEgivesSo,on average by the learning law 6-12,0)(LE iff any synaptic vector move along its trajectory.So,the competitive AVQ system is asymptotically stable and in general converges exponentially quic
16、kly to a locally equilibrium.Suppose 0)(LEIf 0jmThen every synaptic vector hasReached equilibrium and is constant.Since p(x)is a nonnegative weight function.The weighted integral of the learning differencemust equal zero:ijimx 326)()(odxxpmxDjjiSo equilibrium synaptic vector equal centroids.Q.E.DArg
17、ument Total mean-squared error of vector quantization for the partition So the AVQ convergence theorem implies that the class centroid,and asymptotically,competitive synaptic vector-total mean-squared error.kDD,.1126)(jjDjnmxxImjByThe Synaptic vectors perform stochastic gradient descent on the mean-
18、squared-error surface in pattern-plus-error 1nRIn the sense:competitive learning reduces to stochastic gradient descentGlobal stability is jointly neuronal-synaptics steady state.Global stability theorems are powerful but limited.Their power:their dimension independence nonlinear generality their ex
19、ponentially fast convergence to fixed points.Their limitation:do not tell us where the equilibria occur in the state space.Stability-Convergence DilemmaStability-Convergence Dilemma arise from the asymmetry in neuronal and synaptic fluctuation rates.Neurons change faster than synapses change.Neurons
20、 fluctuate at the millisecond level.Synapses fluctuate at the second or even minute level.The fast-changing neurons must balance the slow-changing synapses.Stability-Convergence Dilemma1.Asymmetry:Neurons in and fluctuate faster than the synapses in M.2.stability:(pattern formation).3.Learning:4.Und
21、oing:the ABAM theorem offers a general solution to stability-convergence dilemma.00yxFandF.000MFandFyx.000yxFandFMxFyFThe ABAM Theorem(证明的关键是找到一个合适的证明的关键是找到一个合适的Lyapunov函数函数）The Hebbian ABAM and competitive ABAM models are globally stable.356346)i()()(336)()()(11jiijijniijijjjjjpjijjjiiiiiSSmmmxSyby
22、aymySxbxaxHebbian ABAM model:Competitive ABAM model,replacing 6-35 with 6-36366ijijijmSSmIf the positivity assumptions 0000jijiSSaaThen,the models are asymptotically stable,and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values:0,0,0222ijjimyxPr
23、oof.the proof uses the bounded lyapunov function L37621)()()()(200ijijjjjyjjjiiixiiiijijjimdbSdbSmSSLji386)(:dtdxdxdFtxFdtdgivesationdifferentiofrulechaintheiiiijjiijiijijjjjijijjiiiijijjiijjiijijjjijijjiiijijijjjjiiiijijjiiijijjjjijjiiiSSmmSbbSmSbaSthroughbymSSmmSbySmSbxSmmybSxbSmSSmSySmSxSLiji416)
24、()()(366346406)()()(396222Make the difference to 6-37:0000,ijijbecause of aaSS.,0,estrajectorisystemalongLSoTo prove global stability for the competitive learning law 6-36 ijijjiijijiijijjjjijijjiiimSSmSSmSbbSmSbaSL426)()()(22.01)(00)()(2estrajectorialongLSjifmSSjifmSSmSSmSSmijiijjiijijijjiijWe prov
25、e the stronger asymptotic stable of the ABAM modelswith the positivity assumptions.0000jijiSSaa4360406222ijijjjjjiiiimybSxaSLAlong trajectories for any nonzero change in any neuronal activation or any synapse.Trajectories end in equilibrium points.Indeed 6-43 implies:456044600222ijjiijjimyxiffmyxiff
26、LThe squared velocities decease exponentially quickly because of the strict negativity of(6-43)and,to rule out pathologies.Q.E.D because of the second-order assumption of nondegenerate Hessian matrix.Is unsupervised learning structural stability?Structural stability is insensitivity to small perturb
27、ationsStructural stability ignores many small perturbations.Such perturbations preserve qualitative properties.Basins of attractions maintain their basic shape.Random Adaptive Bidirectional Associative Memories RABAMBrownian diffusions perturb RABAM model.(也就是加进一种噪声）The differential equations in 6-3
28、3 through 6-35 now become stochastic differential equations,with random processes as solutions.ijijyjximsynapsetheinprocessmotionbrowianBFinneuroniththeinprocessmotionbrowianBFinneuroniththeinprocessmotionbrowianB:.:The diffusion signal hebbian law RABAM model:486()(476)i()()(466)()()()11ijjiijijnii
29、jijjjjjpjijjjiiiiidByjSxiSmdmdbjdtmxSybyadydBidtmySxbxadxWith the stochastic competitive law:496)()(ijijiijjijdBdtmxSySdmjSWith noise(independent zero-mean Gaussian white-noise process).the signal hebbian noise RABAM model:546)(,)(,)(5360)()()(526)()(516)i()()(506)()()(22211ijijjjiiijjiijjjiiijijjni
30、ijijjjjjipjijjjiiiiinVnVnVnEnEnEnySxSmmnmxSybyaynmySxbxaxRABAM Theorem.The RABAM model(6-46)-(6-48)or(6-50)-(6-54),is global stable.if signal functions are strictly increasing and amplification functions and are strictly positive,the RABAM model is asymptotically stable.iajbProof.The ABAM lyapunov f
31、unction L in(6-37)now defines a random process.At each time t,L(t)is a random variable.The expected ABAM lyapunov function E(L)is a lyapunov function for the RABAM.556),(.)(dxdydMMyxpLLELRABAM586576566)()(222jiijjiijijjiijijjjijjjiiiijjiijijjiijijjjijjjiiiijjiijjiijijjjijijjiiiijjiijijjiijijjijijjii
32、SSmnEmSbSnEmiSbSnELESSmnEmSbSnEmiSbSnESSmmSbaSmSbaSESSmmmSbySmSbxSELELEABAMQ.E.N000)(0)(,606)(596)()()(modelABAMABAMABAMABAMLorLasaccordingestrajectorialongLEorLESoLEnoisemeanzeroSSmEnEmSbSEnEmiSbSEnELERABAMtheintermnoiseadditiveandsignaltheofindepenceijjiijijjiijijjjijjjiii【Reference】1 “Neural Networks and Fuzzy Systems-Chapter 6”P.221-261 Bart kosko University of Southern California.
