1、20112011年年1111月月Monitoring and Evaluating ODAThis presentation will cover lWhy monitor?lWhy evaluate?lMonitoring in the World Bank,other multi-lateral development banks,bilateral donorslEvaluation in the World Bank,other multi-laterals,bilateral donorsWhat is monitoring?Definition of monitoring:a co
2、ntinuous process that uses systematic collection of data about progress in an on-going development project,for use by managers and other stake-holdersWhy monitor?To improve implementation of on-going projects;must provide practical informationTo assess whether development objectives likely to be met
3、To identify systematic issues and risks across projects,by country,by sector,and overallTo provide input for ex post evaluationWhat is Evaluation?Definition of(ex post)evaluation:assessment of a completed project,program,or policy,its design,implementation,and results.Why evaluate and who cares?Why?
4、Two reasons:To learn from experienceTo provide accountabilityWho uses(or should use)evaluations:Boards of Directors(MDBs)and Ministers and Directors General for oversight;ODA managers for policies and strategies;ODA staff for program and project designs,ways of doing business;Client countries,extern
5、al partners,and the public for policies,aligning programs,researchM&E in contextlWorld Bank commits/disburses$20-23 billion/year;lApproves 280 loans and credits/year;active loan portfolio:1,400 in 120 countrieslBank also provides grants and other non-lending services(diagnostic work)lBank has total
6、administrative budget of$2 billionMost other MDBs commit$1 6 billion/year Bilateral ODA:$200 million(Greece,Luxembourg,New Zealand,Portugal)-$25 billion/year(USA)Bilateral agencies have administrative budgets of$11 million(Luxembourg)to$700 million(Japan)Monitoring in the World BanklEvery on-going l
7、ending operation is monitored by visits from Bank staff about twice a year;lImplementation status and results(ISR)report issued twice each year:lProgress on implementationlProgress in meeting development objectiveslProblem projects identifiedlPortfolio at risk reports separate methodology updated mo
8、nthlylOther quality control activities include quality at entry reviews;quality of supervisionEvaluation:the topicslSelf evaluationlIndependent evaluation:why it matterslTools:projects;sectors;countries;etc.lCoverage:how many evaluationslResources/budgetlMain challenge for evaluationlDisclosurelOthe
9、r Evaluation IssuesSelf-Evaluation in the BankSelf evaluation in the Bank:lEvery project has an implementation completion report,done by staff of responsible unit(about 280/year)lEvery new country assistance strategy now has completion report covering previous country assistance strategy,done by sta
10、ff of responsible unit(about 20/year)Self-evaluation in other donorslMulti-lateral institutions:almost all MDBs have requirement for self-evaluation,although in some MDBs(Asian DB,EBRD,EIB),the evaluation department is involvedlBilateral donors:some require for projects over given value,others dont;
11、SDC(Swiss):external evaluators do self-evaluation;GTZ(German)has innovative eVAL(electronic)system.Independent evaluationlWhat is independence?The extent to which the evaluation unit:Reports directly to a Board or other governing unit;Has staff,budget,and functions independent from management;not in
12、volved with management of ODABehaves independently by issuing objective,impartial,and balanced reportsIndependent evaluationIn the World Bank,independent evaluation,since 1973,has looked like this:Board of DirectorsInstitutional IntegrityPresidentPaul WolfowitzIndependent Evaluation GroupInternal Au
13、ditingSVP&Chief EconMD OperationsCFOSVP&Gen CounselInspection PanelMD OperationsIndependent evaluationIn bilateral donors,independent evaluation has evolved to look like this:Ministry of Foreign Affairs Evaluation UnitDirector General Aid agencyOperational unitOperational unitOperational unitIndepen
14、dent evaluationlWhy independence matters:lAllows for objective,unbiased evaluationslEvaluation findings are not tailored to managements wisheslIndependence lends credibility to findingslIndependence difficult to achieve and maintainEvaluation Tools:World BanklProject evaluationslImplementation Compl
15、etion Report(ICR)reviewslProject performance assessment reviews(PPARs)lCountry evaluationslCAS Completion Report(CASCR)ReviewslCountry assistance evaluations(CAEs)lSector,thematic,global evaluationslProcess evaluationslCorporate evaluations(ARDE,AROE)Evaluation Tools:interactionsSynergies among diff
16、erent evaluation tools can provide valuable input Global(Global programs)Evaluation tools in the World Bank Corporate(ARDE)Sector(finance,education)Thematic(gender,decentralization)Process (PRSP)Country Assistance Evaluation CASCR Review CASCR ReviewICR Review(validation of self-evaluation)Project P
17、erformance Assessment ReviewCountry Assistance EvaluationICR Review(validation of self-evaluation)Project Performance Assessment ReviewProject levelCountry levelSector,thematic,policy levelGlobal and corporate levelEvaluation Coverage:World BankICR reviewsProject PPARCASCR reviewsCAEsSector reviews#
18、/year(%total)280(100%)70 (25%)20 (100%)7 (30%)6 -Site visitElapsed time90 days6-12 months3-5 months1 year1-2 yearsEvaluation coveragelIn most MDBs,full range of evaluations:project,country,sector,thematic evaluationslIn most bilaterals,few project evaluations;some country level;most focus on themes
19、or programslMost bilaterals produces 3-10 evaluations/yearlDFID produces 20-25 reports/yr;lPotential weakness of country and thematic evaluations without underpinnings of project evaluations to use as inputResources for evaluation:MDBslResources vary,but as percent of total administrative budget,fai
20、rly similar:#Evaluation staff Evaluation budget as%of administrative budgetAfDB111.6AsDB252.1-3EBRD151.8EIB101.0IADB171.1IFC220.9World Bank881.5Resources for evaluation:bilateralslBilateral donor agencies are more likely to hire external consultants for virtually all evaluations(UK,Norway,Denmark,Sw
21、itzerland,Australia)lBased on numbers from a few bilaterals(DFID,NORAD,SDC),evaluation budget is 2%of total administrative budget;figures of 0.5 2%have been cited by others.lSmaller donors(Greece,New Zealand)are still setting up their evaluation unitsMain challenge for evaluationlHow to make evaluat
22、ions usefullMost managers,staff focused on future,not on pastlEvaluation must/should be:lTimelylOperationally relevant:avoid generalizations,platitudeslEasily accessible:avoid long reportslConstructive:good practice should be rewardedlWidely disseminated and disclosedDisclosure of evaluation reports
23、lWorld Bank:public disclosure of all:lPPARslCAEslSector,thematic,global evaluations lProcess evaluationslCorporate evaluations lEBRD,ADB:selective public disclosure:evaluations involve private sectorlBilaterals:majority disclose,but varies:UK,Norway full disclosure;Germany selective disclosureDisclo
24、surelFor World Bank:reports and information on evaluation tools,approaches,on website:lhttp:/worldbank.org and scroll down to right-had side of home page,or:lhttp:/www.worldbank.org/ieglFor other donors:check their websitesOther issues in evaluationlEvaluation approachlUse of performance indicatorsl
25、Sources of informationlRatingslProcesseslPartnershipslOther challengesEvaluation ApproachWithin context of results-based management,Bank uses objectives-based approach,focus on outcomes and impactBank loans;grants;diagnostic work;dialogue Physical outputs;privatization;new laws;agency created;traini
26、ng completed;sub-loans disbursed.More efficient operations;better fiscal management;employment creation;better access to services;InputsOutputsExogenous factorsExogenous factorsOutcomeImpactLonger-term objective,like poverty reduction,better educational,health outcomesEvaluation ApproachBank assesse
27、s:Outcome(relevance,efficacy,efficiency)lUse of performance indicators for outputs,outcomes,impactlRisk to DevelopmentlBank and Borrower PerformanceSources of information for evaluationBaseline data in project appraisal documents(sometimes)Monitoring reports should have information on performance in
28、dicatorsData bases for project area;sector;country at beginning of project,end of projectOther country-level indeces(IMF financial statistics,Doing Business surveys;World Bank Indicators of corruption surveys;UNDP human development indicators)SurveysInterviews,focus groupsUse of RatingslWorld Bank u
29、ses ratings on project outcomes and outcomes of country assistance.No ratings on sector,thematic,other evaluations lOther MDBs use ratings for project outcomes;not for country assistancelBilaterals:some use ratings,only for projects;some do not use any ratingsEvaluation Approach:ratingslProject Rati
30、ngs in World Bank:lOutcomes:6 point scale(HS-HU)lRisks to Development:4 point scale(H-N)lBank and Borrower performance:4 point scale(HS to HU)lPros and cons of ratinglPro:provides data and accountabilitylCon:takes focus away from lesson learning and richness of evaluationProcessesIndependence does n
31、ot mean isolationMDBs and bilateral donors incorporate comments from operational managers and staff and from stakeholders in client countries into evaluation reports:lBanks initiating papers for country,thematic,global evaluations sent to management for comment and to Board of Directors for informat
32、ion;lIMFs initiating paper for all evaluations posted on IMF website and public comment invited;wide consultations throughout process.Partnerships in evaluationlCountry and sector evaluations sometimes involve partnerships with other MDBs and/or bilateral donors(joint or parallel evaluations)lOpport
33、unistic,not systematic,partnershipslPros and cons of evaluation partnershipslPros:wider coverage,other viewpointslCons:more costly and more time required;may not agree on outcomes;may affect timeliness of final reportOther challengeslData difficult to obtain,or different sources have conflicting datalOften difficult or impossible to have a control group(especially in sector and country evaluations)lBeneficiary surveys can be costly,low response ratelProblem of attribution(cause-effect)difficult or impossible to establish
