Laevaluacióndeimpacto:uninstrumentoparaelaprendizajeylaformulacióndeestrategiasypolí:casdedesarrollobasadasenevidencias

UniversidadLoyolaAndalucía,11Diciembre2015

Metodologíasdeevaluacióndeimpacto:ventajas,inconvenientesyusosdemetodologías

cuan8ta8vasycualita8vasMarioBiggeri*^andAndreaFerrannini^

*DepartmentofEconomicsandManagement,UniversityofFlorence^ARCOLab(AcHonResearchforCO-development),PINS.c.r.l.,UniversityofFlorence

www.arcolab.org

•  ARCO is not only academic research but Action-Research à Policy-oriented research that it is conducted with local stakeholders and it is

easily translated into action

#Impactful #Multidisciplinary #Participatory •  Our theoretical background lies in Amartya Sen’s Capability Approach and the

Sustainable Human Development paradigm à “Development lies in people's freedom to determine their own future”

ARCO supports public authorities, private organizations and civil society organizations in promoting local development and community empowerment ARCO contributes to the global academic debate on sustainable human development and the capability approach

OURMISSION

PARTNERS(amongmany…)

STRATEGICUNITS

Investigation, experimentation and production of scientific evidence on local development processes, in order to promote the design and implementation of tailored place-based strategies for sustainable human development at local level

Examples:

-  ARCO conducted a research on international development cooperation at the local level funded by UNDP, aimed at comparing traditional aid policies with new international cooperation initiatives that aim to empower local development systems to adopt a Sustainable Human Development perspective.

-  ARCO supported the association Prospettiva Casentino (composed by local entrepreneurs of Casentino Valley – Tuscany, Italy) on designing tailored and innovative project proposals for the development of the tourism sector in the valley with the final aim of increasing local well-being especially for young people

-  ARCO assessed the resilience of bergamot farmers in Reggio Calabria Province (Calabria, Italy) , in order to evaluate the ability of the socio-economic system to maintain its functions when shocks and various disruptions occur constitute a very relevant asset.

-  ARCO is supporting Oxfam in elaborating its Position Paper on “Local governance to face multidimensional poverty and inequality”.

Performing policy relevant and rigorous evaluations, relying on both quantitative and qualitative methods and using multidimensional models of impact analysis

Examples:

-  Impact evaluation of value chain development project by the Overseas Agronomic Institute in Bale, Ethopia – 2013-15

-  Evaluation of international cooperation project on disability by NGO Aifo in Mongolia - 2015 -  Impact evaluation of microcredit scheme in Sardinia Region – 2014-15 -  Qualitative impact evaluation of the health services provided by Zaporouka Foundation and

Soleterre, Ukraine – 2013 -  Impact Evaluation of health programme by WINFOCUS in Minas Gerais Region, Brazil – 2013-14 -  Intermediate evaluation of the UNDP/ART Global Initiative – 2012 -  Impact evaluation of Community–Based Rehabilitation programs for people with disabilities in

Mandya District, Karnataka, India – 2009/2011

-  Impact evaluation of Community–Based Rehabilitation programs in West-Nile, Uganda – 2011 -  Impact evaluation of rural cooperative enhancement project by Oxfam Italy in Neyba,

Dominican Republic – 2010

•  Our action-research approach is based on multi-disciplinary analysis and cross-fertilizing theories, with an integrated (top-down and bottom-up) and place-based perspective

•  Our research topics have to be policy-relevant and contribute to the well-being of the society

•  We believe in our own continuous upgrading and innovation in the methodological approach, based on international cutting-edge research, as well of that of our partners

•  We use a participatory approach in diagnostic studies and policy design, basing results and processes on local knowledge and ownership of development strategies

•  Our studies and researches are independent and transparent

OURPRINCIPLES

1.  IntroducHon2.  SoundtheoreHcalframework3.  Integratedmixed-methodsprocedure4.  InnovaHve qualitaHve tools for comparaHve

analysis5.  Finalremarks

Structure

1.Introduc8onTheinternaHonaldevelopmentarenaisnowadayscharacterizedbyprogramsthatcannolongerbeeasilyexemplifiedbythestandardlinearrela8oninputs-ac8ons-outputs-outcomes

à Complexity(mulHdimensionalfocus,soZacHviHes,processesandlong-termimpacts,evolvingcontext,mulH-levelgovernance)

à Context-dependency(shapedbythevariedcharacterisHcsofthegeographical,economic,social,insHtuHonalandculturalcontextofacHon)

+

à Eachprocessaimedat linking theanalysisof reality (e.g. impactevaluaHon)todecisionmakingcannotbeneutral(Harriss,2007)

i.e.Evalua8onshouldnotignoreitspoli8caldimension

The Standard Result Chain and theory of change

Gertler et. (2011)

Challengesforevalua8on-  CausalityisoZenlessclearlyordered-  Feedbackloopscangenerateviciousorvirtuouscirclesand

synergieswithindynamicprocessesofmul8-causality-  Processes and procedural aspects characterising these

programs appear as important as the outcomes atindividualandcommunitylevel

àItiscomplicatedto:-  isolate and disentangle a_ribuHon of (net) impacts to a

singleactoramongmany-  analyseintangibleaspects-  build-upcounterfactuals-  assessmulHdimensionaldevelopmentprocesses

Inordertocapturethecomplexityofcommunityand/orterritorialdevelopmentprogrammesandtheirdynamicandevoluHonaryelements,asoundtheore8calapproachandmixedmethodsarebothneeded.

(BarahonaandLevy,2007;Sternetal.,2012)

àTheirconsistentcombinaHonwithmul8-facedinforma8onalspacescanrepresentacrucialvalue-addedforcomprehensiveevalua8ons.

Therefore…

2.Soundtheore8calframework(1)Althoughtheimpactevalua8onliteraturehasbeenreallyboominginthe last tenyears, liUlehasbeendone fromahumandevelopmentandcapabilityapproachperspec8veà TheCA–beinganagency-orientedandopportunity-basedtheory–

can enrich the informa8onal base for mulHdimensional socio-economic assessments, moving beyond mainstream “projectapproach”informaHonalspace,designandtools

à TheCAshiZstheprimarya_enHonofevaluaHonstowhatpeopleare able to do and to be and have reason to value, includingimmaterialaspectsoftheirlife

à The CA gives salience to processes and to the individual’s andcommuni8es’experiences,valuesandpar8cipa8on

Soundtheore8calframework(2)

Most relevant consequences for the evaluaHon of complexdevelopmentprograms:1.  a people-centred analysis focused on beneficiaries’/

stakeholders’ outcomes in terms of expansion ofcapabili8es

2.  themul8dimensionalityofwell-being tobeembracedandcapturedbytheevaluaHonanalysis

3.  stakeholders’ wide par8cipa8on to increase theinforma8onal space for theevaluaHonand to fosterpublicdeliberaHon,voicingpoweranddemocraHcassessments;

4.  the integra8on among micro-meso-macro levels to takeinto consideraHon the mulHlevel arHculaHon ofdevelopmentiniHaHvesinfluencingtheevaluaHondomains.

Collec8vedynamics

Localdynamics

Source:BiggeriandFerrannini(2014,p.50)

Individualdynamics

TheSTEHDFrameworkSustainableTerritorialEvolu9onforHumanDevelopment

ExtraLocalLevel

3.Useofmixed-methods(1)àEvaluaHonsshouldbestructuredontwomainparalleltracks:1.  tofindmethodstosynthesisetheimpactofpolicymeasure2.  tounderstandtheprocesses(whyandhow)thatleada

programtoachievedcertainresults (RaoandWoolcock,2004;Sternetal.,2012).

•  Theuseofmixedmethodsaswidelydebatedintheliterature(Bardhan,1989;Bourguignon,2003;Chambers,2003;Kanbur,2003;BardhanandRay,2006;Cosgel,2006)

•  AremarkablepotenHalityofmixedmethodsisthecapacityofexplainingbothoutcomesandprocessesbycatchingtheircomplexity,especiallyregardingdevelopmentprograms

•  Thispotentcanbefullyexploitedonlyifthereisnoprejudice(e.g.againstQUAL)

Useofmixed-methods(2)ThereisnotasingleorrightwaytointegratequanHtaHveandqualitaHvemethodologies:e.g.parallel,sequenHalanditeraHveintegraHon.

RaoandWoolcock(2004)

•  “Confirming/Reinforcing/Refu8ng” integraHon when qualitaHvemethodsareusetotesteconometricanalysisfindings

•  “Enriching” integraHon when certain aspects of a phenomenoncannotbehandledthroughquanHtaHveinstruments

•  “Examining” integraHon to test qualitaHve methodologies-basedknowledgethroughquanHtaHvemethods.

•  “Explaining”integraHontoexplainunexpectedquanHtaHveresultsinaqualitaHveway

•  “Merging” when the same data are analyzed both throughquanHtaHveandqualitaHvemethods White(2008)

Integratedmixed-methodsprocedure

Anexampleofaprocedurein8steps:1)Analysisofexis8ngdataanddocuments- Deskreviewontheissueandtheprogram- Firstfieldmissions2)Theore8calframework-TheoreHcalissuesandtheoryofchange3)Qualita8vemethodstoexplorethepoten8aloutcomevariablesandcauses-In-depthinterviews-FGDstopreparebothstudiesàincludingdimensionsandoutcomevariables- Workinggroupofexperts

Integratedmixed-methodsprocedure(…conHnued)

4)Toolsdesign-QuesHonnaireandstructuredFGDs,usermanuals5)SamplingDesign(withcomparisongroup)-SamplingdesignforbothqualitaHveandquanHtaHve6)TrainingandToolstes8ng-Trainingofpersons,manualchecking,toolspilot7)Dataproduc8on-QuanHtaHvesurvey- QualitaHveanalysisàthroughstructuredFGDs- Dataentryandmanagement8)DataAnalysisandtriangula8on(includingfurtherinvesHgaHonifnecessary)

…followsreport/paperwriHngs

4.Innova8vequalita8vetoolsforcompara8veanalysisandevalua8ons1.StructuredFocusGroupDiscussion(SFGD)withaMatrixScore

BiggeriandFerrannini(2014b,JHDC)

•  ItaimsatcollecHngreliableinformaHonforprojectevaluaHons,contribuHngalsotothecommuni8es’empowermentandstakeholdersagencybecauseofitsparHcipatorynature

•  Thistoolisdefined“structured”since,oncecompleted,thestepsareclearlydefined(sequenceofacHonsandsequenceofquesHons)andthusreplicableforcomparability

•  Thetoolisquiteflexibleandadaptabletothetype,characterisHcsandcomplexityofdevelopmentprogramsandeventuallytothedifferentstakeholdersinvolved

à SFGDscancomplementquanHtaHvedata,orinitsabsencecanconsHtuteavaluablequalitaHvemethodbyitself

Preliminaryphase1.  In-depthanalysisontheachievedfuncHonings

withinthecommuniHesofinterest2.  AppropriateidenHficaHonofthedimensionsand

sub-dimensionsofanalysis(throughaccuratedeskreview,surveys,indepthinterviewsand/orFGDswithprivilegedobservers)

àLesnglocalactorsidenHfyingthemostrelevantdimensionsofanalysis,reducingprescripHvelistsorvaluejudgementsbyexternalresearchers

StructuredFocusGroupDiscussionswithMatrixscore

•  Amatrixofdatatocollecttheanswersofagroupques8onnaireisconstructed,composedbyanumberofquesHons(rowsxcolumns)withascoreforeachanswer

•  RowsrepresentthedimensionsofanalysisidenHfiedinthefirstphase,whilecolumnsrefertoeither:

A)differentbeneficiaries(e.g.varyingstatusesintermsofconversionfactors)àThescoreineachcellofthematrixreferstothelevelofopportunityxheldinthestatusy

OR

B)differentprojectstakeholdersà ThescoreineachcellofthematrixreferstothecontribuHonby

stakeholderytotheactuallevelofopportunityx

StructuredFocusGroupDiscussionswithMatrixscore

•  Thescoringmethodusuallyreliesona0-10scale:0indicatescompletelackofopportunity/capability,10correspondstothehighestlevelofopportunity

•  TheidenHficaHonofacertainscoreandassessmentforeachpointisthusbasedoncollec8vediscussionandgroupanswers,ratherthanonindividualanswers

•  ThemethodologicalprocedureisstructuredtobeconductedexactlyinthesamemannerineverygroupacHvitywithdifferentstakeholders,reducingtheambiguiHesandbiasa_ributedtostandardFGD

A)Withbaselineandcomparisongroupsi)  FamiliarizetheparHcipantwiththedimensionsii)  Familiarizewithmarkingiii) Validatethedimensionsusingabenchmark(column)iv) ParHalrankingofthedimensions(threemostrelevant)v)  AssessdifferentopportuniHesfordifferentcharactersvi) Significance/a_ribuHontotheprogrambydimensionsandby

charactersvii) ValidatethewholetheexercisebycommenHngitAlleastthreepersonsarenecessary:Oneleadfacilitator,Oneassistant,OnenotetakersTime:from2to4hoursdependingonthenumberofdimensions

B)Withoutbaselineandwithoutcomparisongroup

i)  FamiliarizetheparHcipantwiththedimensionsii)  Familiarizewithmarkingiii) Validatethedimensionsusinganidealbenchmark(column)iv) ParHalrankingofthedimensions(threemostrelevant)v)  AssessdifferentfuncHoninglevelforthecommunity/personsnow

andbeforetheprogramstarted(retrospecHveanalysis)vi) Significance/a_ribuHontotheprogramchangetodifferentactors

bydimensions(threequesHonseach)vii) IEexercise(thefuncHoninglevelwithout)viii) ValidatethewholetheexercisebycommenHngitAlleastthreepersonsarenecessary:Oneleadfacilitator,Oneassistant,OnenotetakersTime:from2to4hoursdependingonthenumberofdimensions

Example:UNDPARTinColombia

2.O-GapWalk

ObjecHvetoassesstheopportunitygapswithincommuni8esamongpeoplewithdifferentpersonalorgroupconversionfactors

•  Itisstructuredasacontemporaneousmentalandphysicalexercise,providinganintuiHveportrayalofdifferencesincapabiliHesfreedomanddiscriminaHoninacommunitycontext

•  ItproposesanadvancementofthePowerWalkmethodelaboratedbyUNICEF,linkingittothecapabilityapproach,wideningitsscopeofapplicaHonandprovidingpossiblequanHtaHveanalysesofthefindings

4.Innova8vequalita8vetoolsforcompara8veanalysisandevalua8ons

O-GapWalk(1)

1. Secretandrandomassigna8ontoeachpar8cipantofarolecard,eachrepresenHngvarioustypicalcharacterswithinacertaincommunity,whosecapabiliHesfreedomisassessedthroughouttheexercise

2. ParHcipantsareinvitedtojoinatanopenspace,takingposiHonstandinginarowbelowaline

3. ParHcipantsareaskedtoreacttoone-by-oneopportunitystatementsbyindicaHngthecapabilityfreedomenjoyedbytheindividualcharactertheyrepresent,i.e.numberofsteps0≤z≤3,where0correspondsto“impossibility”and3to“fullopportunity”

O-GapWalk(2)

4. Attheendoftheexercise,whenthereacHontoeachstatementhasbeensummedupandeachindividualhasreachedhis/herownfinalposi8on,anintui8veassessmentoftheopportunitygapandameasureoftheinequalityofcapabili8esisobtained

5. Toconclude,parHcipantsarefirstinvitedtorevealtheirsecretcharacter/roleandthentocollec8velyevaluateboththeopportunitygapandthelessondrawnfromtheacHvityitself

Methodologicaladvantages

•  Provisionofvaluableinforma8ontobeimmediatelyandeasilytranslatedintoquanHtaHvetermsintermsofpar8cipatorysta8s8cs

•  Widerangeofapplica8onanditsadaptabilitytothecontextsandobjectsofanalysis

•  Bothcross-sec8onalandlongitudinalanalysescanbeconducted,eithercomparingtheactualsituaHonindifferentcommuniHes(e.g.treatmentorcontrolgroups)ormonitoringandevaluaHngtheimpactofacertainprogrambyrepeaHngtheexerciseinthesamecommunitybefore,duringandaZertheimplementaHon+RetrospecHveanalyses

Poten8alissuesforthesetools

1.   Subjec8vepercep8onsandadap8vepreferencesà ToreducethepersonalisaHonofopinionsthroughreflecHve

reasoning,helpingparHcipantstoparHallytodetachtheirpreferencesfromtheirlifeexperiencesinordertobecomea“quasi-imparHalspectator”(Sen,2006;BiggeriandLibanora,2011)

2.   The“poten8alcapabilityset”mightbeneglectedbypartof

theconsideredcommunity,e.g.ifaminoritygroupistotallydeprivedinacertaindimension

à Toconsulta“representaHve”sampleofthepopulaHonandtoincludebasiccapabiliHesinthepotenHalcapabilityset

5.FinalremarksToevaluatecomplexdevelopmentprograms…

•  Itiscrucialtorelyonasoundtheore8calframeworktoexaminemulH-causalityandoncomplexmixedmethodsevaluaHonstrategiestocomprehensivelyassessdevelopmentprocesses

•  WeproposeaprocedurebasedonaCA-enhancedconceptualframework

•  Increasingstakeholders’voicingpowerandplacingcentralemphasisonlocalstakeholders’knowledgeofthecontextualdynamicsbylesngthemtoparHcipateinimpactassessment

•  CreaHngthegroundforasystema8cintegra8onbetweenquanHtaHveandqualitaHvemethodologies

•  Valorisingthepoli8caldimensionoftheevalua8onitself,contribuHngtotheexpansionofparHcipaHon,empowermentandagencywithinlocalsocieHes

BibliograjaBarahonaCandLevyS(2003)HowtogeneratestaHsHcsandinfluencepolicyusingparHcipatorymethodsinresearch:reflecHonsonworkinMalawi1999-2002.InsHtuteofDevelopmentStudiesWorkingPaper,n°212,UniversityofSussex.Bardhan,P.(Ed.)(1989),ConversaHonsbetweeneconomistsandanthropologists:MethodologicalissuesinmeasuringeconomicchangeinruralIndia,OxfordUniversityPress,Delhi.BardhanPandRayI(2006)Methodologicalapproachesineconomicsandanthropology.Q-SquaredWorkingPaper,n°17,CentreforInternaHonalStudies,UniversityofToronto.Biggeri,M.andFerrannini,A.(2014a),SustainableHumanDevelopment:ANewTerritorialandPeople-centredPerspecHve,PalgraveMacmillan,NewYorkandBasingstoke.Biggeri,M. and Ferrannini, A. (2014b), “OpportunityGapAnalysis: Procedures andmethods for applying the capability approach in developmentiniHaHves”,JournalofHumanDevelopmentandCapabiliHes,15(1):60–78.Biggeri,M.andLibanora,R. (2011), “Fromvaluing toevaluaHng:Toolsandprocedures tooperaHonalize thecapabilityapproach”, inM.Biggeri, J.BalletandF.Comim(Eds.),ChildrenandtheCapabilityApproach,PalgraveMacmillan,NewYork.Bourguignon,F.(2003),“QualitaHveandquanHtaHveapproachestopovertyanalysis:TwopicturesoftheSameMountain?”, inR.Kanbur(Ed.),Q-Squared:CombiningQualitaHveandQuanHtaHveMethodsinPovertyAppraisal,PermanentBlack,Delhi.Chambers,R.(2003),“QualitaHveapproaches:Self-criHcismandwhatcanbegainedfromquanHtaHveapproaches”, inR.Kanbur(Ed.),Q-Squared:CombiningQualitaHveandQuanHtaHveMethodsinPovertyAppraisal,PermanentBlack,Delhi.Cosgel,M. (2006), “ConversaHons between anthropologists and economists”,Q-SquaredWorking Paper, no 18, Centre for InternaHonal Studies,UniversityofToronto.GertlerPJ,MarHnezS,PremandP,RawlingsLBandVermeerschCMJ(2011)ImpactEvaluaHoninPracHce.Washington,DC:WorldBank.Harriss,J.(2007),“BringingpoliHcsbackintopovertyanalysis:WhyunderstandingofsocialrelaHonsma_ersmoreforpolicyonchronicpovertythanmeasurement”,Q-SquaredWorkingPaper,no34,CentreforInternaHonalStudies,UniversityofToronto.Kanbur,R.(Ed.)(2003),Q-squared:CombiningqualitaHveandquanHtaHvemethodsinpovertyappraisal,PermanentBlack,Delhi.Rao,V.andWoolcock,M.(2004),“IntegraHngqualitaHveandquanHtaHveapproachesinprogramevaluaHon”,inF.BourguignonandL.A.PereiradaSilva(Eds.),TheimpactofeconomicpoliciesonpovertyandincomedistribuHon:EvaluaHontechniquesandtools,WorldBankandOxfordUniversityPress,NewYork.Sen,A.K.(1999),Developmentasfreedom,OxfordUniversityPress,Oxford.Sen,A.K.(2006),“WhatdowewantfromatheoryofjusHce?”,TheJournalofPhilosophy,CIII(5):215–238.Stern,E.,Stame,N.,Mayne, J.,Forss,K.,Davies,R.andBefani,B. (2012),“Broadeningtherangeofdesignsandmethodsfor impactevelutaions”,ReportofastudycommissionedbytheDepartmentforInternaHonalDevelopment,DFIDWorkingPaper,no38.White,H.(2008),“OfprobitsandparHcipaHon:TheuseofmixedmethodsinquanHtaHveimpactevaluaHon”,NONIEWorkingPaper,no7,CentreforInternaHonalStudies,UniversityofToronto.

Muchasgraciasporsuatención!Comentariosysugerenciasmasque

bienvenida!

Contacts:

andrea.ferrannini@arcolab.org

mario.biggeri@unifi.it