1、 .HF 1D3A D.42 .D .232 1 MorseHiveOozieDr.elephantKylinPhoenixDruidK8SMapReduceCPUTezSparkFlinkMPITensorFlowPyTorchYARN Federation With Node Label (Docker On Yarn)MEMORY GPUAlluxioHBaseKafkaHDFS Router Based Federation 2 2. iACD 3gGiceahSdHGiFLhTSd 3nn&Wp EIND 27nWYhIRGGLFCSFNMQg6yd0( LLRTN ANPFP t5
2、13 1CQC9NEF 9 9NEF8CMCHFP OCP2TFDRQNP,4 r&+ 4511n 511 c 513u LLRTN khoxli 7NDCLQU fva&-() ke&-)(EINDp&-+ 627&)&112lS 2. AoxRS 12- 2. o Na Y ac B WeT112r .3pz 2. iR 12- kuNode ABlock 1Block 2Block 3Block 4Block 5Node BBlock 2Block 6Node CSpark ExecutorWhich NodeLabel AlluxioBlock 7NodeNodeNodeYARN No
3、deManagerAlluxio WorkerYARN NodeManagerAlluxio WorkerYARN NodeManagerHDFS DataNodeHDFS DataNodeHDFS DataNodeNode DBlock 1Block 2Node EBlock 2Block 6Node FBlock 3NodeNodeNodeYARN NodeManagerAlluxio WorkerYARN NodeManagerAlluxio WorkerYARN NodeManagerBlock 4Block 7Block 5HDFS DataNodeHDFS DataNodeHDFS
4、 DataNode 0 o sM01- n 00 MRlxWMMBaes uC scBPMkBhpgiCv vC C |ArS | 1 DH r1Node ABlock 1Block 2Block 3Block 4Block 5Node BBlock 2Block 6Node CB 1 aMrx Bln2liB 1kocAE|+- 2eo Node ABlock 1Block 2Node BBlock 2Block 6Node CBlock 3Block 4Block 7Block 5 1 mp guhJBC:ACI:EBB:CBmpnSfKPMLSPmplrNSSO XPy XdSNchKo
5、hRba TOucU 6e22 :E:ECHksvitd:E:ECfb 2 :J6BB:CC3-6B6:E:62:H:EC-6B6:Ead 1-Vebui- ) 0(23 PPRxMrTnYrTplurTnYNodeCPUMEMAlluxioc - oSoSbbYARNResourceManagerOkuc1- - ) 2 )- A g 1- uN u| )22 szL uHvRequest CPU, MEM, Alluxio ResourceNodeNodeMEMNodeMEMNodeNodeMEMCPUMEMAlluxioCPUAlluxioCPUAlluxioCPUMEMAlluxioC
6、PUAlluxioYARN NodeManagerSpark ExecutorAlluxio WorkerYARN NodeManagerSpark ExecutorAlluxio WorkerYARN NodeManagerSpark ExecutorAlluxio WorkerYARN NodeManagerSpark DriverYARN NodeManagerRun With Alluxio ResourceTezApplicationMasterHDFS DataNodeHDFS DataNodeHDFS DataNodeHDFS DataNodeHDFS DataNodepu %O
7、f10A0D DE 1l-AIyn DE 3 0D tLM DE V 0D thdekiW -)- ( 1zS 0D rR 0D MafHg cT 20 g D y DE m W DE go % xMap TaskReduce TaskMapReduceSplit Memory To DiskMerge On DiskMixture Of In-memory And On-Disk DataMap TaskMapAlluxioReduce TaskReduceSplit Memory To Disk Merge To AlluxioShuffle File StorageMixture Of
8、In-memory Or Network 0D DE% 0D DE xS + li2p 1 3k53 opaA - Sp+3 2.hl axp xEPLcoep xPudoeI ( l Dhh ) % DHit Rate 60%Hit Rate 30%Hit Rate 10%Hot Data(Latest 3 Day)Normal Data(Latest 3 Month)Load AlluxioHDFS RAID hpTR hU 30. | 7 n C 1 p % tEi 3 -5 d 2- 317 % 2 0 D 5) 5 5 5 5H()H(H(H 9 ?F 0 32371865p p N
9、M 60. 3 (0plkop leSoCB ( gcI ( 65 -() Db 66 EIuGcFON B H (1Loading Fsimage Saving CheckPointSafe Mode34%1%16%49%oEBStartLoading EditsServing+)uhrIB) Nkruhd 5meou J ) gH B F FF5Cl )5F5B - me me cMP )iuh kuh s ab , tl 0( 1 uh , abSHnab R 0DBFB uhfVhQhp Ioo +5D5F Q B CBDFCopy FsimageTestNodeFsimageName
10、Node(Active)ParseBlock ReportsNameNode(Stanby)Mock DataNode(MultiThreading)B)gCRFc RkmSRM mSl pdH NameNodesBlockMapsNameNodesBlockMapsFcBlock ReportOnce All BlockBlock ReportMulti timesPiece of BlocksLDmSNp DmSeG li 01 0 B oNa DataNode 1DataNode 1LogicStorageHDD 1HDD 2HDD 3HDD 1HDD 2HDD 3HDD 4HDD 5H
11、DD 4HDD 5% 0 4 0 ?FN 5) 5 5 5 5()( 9 M 0 32371865 S H DHp . 03 cp, , iEJPQrulDHk smp4ANL,JFJt, s - oeaJ gM s 976 C SdJp 6B G - hH haClient aorDFCRead Or Write At SameTime S- ynilet PCluster ANS1Cluster ANS1 s M PNameNode(Active)NameNode(Active)SyncNameNode(Stanby)NameNode(Stanby) Mcm DataNodeDataNod
12、eDataNodeDataNodeDataNodeDataNode 9FN 3) 3 3 3 3()( 8 M 21260735 H DH S IdopCnDLt-e C t r33146 t d dp NMtSRaFAi RaSEHUC ClientClientWhich One Should I Use?Get From Ns2Multi-ClustersMulti-ClustersMigrationNameService1NameService2NameService3NameService1NameService2(Old)NameService2(New)- P|PtN- EIHFm
13、Mac - :ABF lePmsr ZR/ (1 - State StoreLocalFileZooKeeperHDFSDBMSUClient EIHFdO :EFAH E :H: E:BAHK IAHBE EIHF tNfTh L ) nauthorizeadd auditlogRouterRouterRouterRouterXMemberShipMountTableRouterStateStoreServiceRouterRpcServiceAdminServiceNameNodeHeartbeatServiceRouterHeartbeatServiceRouterMetricsServ
14、iceadd client ipNS2 C HttpServerRouterSafeModeServiceRouterQuotaUpdateService D:CFAHH FEIHFHH u voNS1NSnNameNode(Active)NameNode(Active)NameNode(Active) yiTg C F: S|NameNode(Stanby)NameNode(Stanby)NameNode(Stanby)DataNodeDataNodeDataNodeDataNode-pO p wnaService DiscoverZookeeperCLientC - yService RegisterRead Data tD KHBase|SBFrontendCacheFrontendCacheGet IndexRowKeycf:a AD|sD ACLSidpath, offset, len, checksumData Transfer Aote A K F LP oBHDFSFileFileFileFile iRdH r ZiRR oo.uil x x.FSA .3x 2x 4H oo.x11Dx.
