What is included in an impact evaluation study?

Impact- if not measured or evaluated correctly, can be very difficult to understand and quantify. In this blog, we will see what an impact evaluation study is, what are the steps involved in completing an evaluation study, and how to communicate the study results to the stakeholders involved. 

An ideal program or intervention functions with a goal and certain objectives in mind. Impact evaluations are necessary to gauge to what extent have the intended impacts of the intervention been achieved. They provide information about not just the intended objectives but also the unintended impacts. Impact evaluations give an understanding of how well a program has worked, and what are the areas that can be strengthened in it.  Let’s look at the steps for an effective impact evaluation study. 

Step 1: Framing research questions

Research questions are questions that shape data collection, analysis, and reporting.

Process evaluation	Impact evaluation
Frame questions about the implementation of the project. Inputs, outputs, program activities, and stakeholder views will be included in this.	Frame questions about the changes created by the project. These can be short-term changes like knowledge, skills, and attitudes; medium-term changes like behavior and actions; and long-term changes like conditions and status.

Evaluate your research question. Is it feasible? Is it interesting? Is it novel? Is it ethical? Is it relevant?

Step 2: Indicator development

An indicator is a variable that measures the key elements of a program. An indicator has 4 parts.

Population	Who is changing?
Target	How many do we expect will succeed?
Timeline	By when does this need to happen?
Threshold	How much is good enough?

Make sure that the indicators you choose are easy to understand and will provide essential metrics to determine impact. They need to be cost-effective and fit within the program budget. Use a combination of quantitative and qualitative indicators. Only include indicators that are aligned with the program components and goals. Experts can be involved to get their insights on it. 

Step 3: Sampling

Sampling is the process of selecting a set of observations from a population to estimate a program’s impact, for that population. The chosen sample should be a fair representation of the population. The types of sampling are:

	Probability Sampling		Non-Probability Sampling
Simple random	Every single member of a population is chosen randomly	Purposive	Sample is selected purely considering the purpose of study
Systematic	Members of a sample are chosen at regular intervals of a population	Convenience	Used in studies which need to reach subjects which are difficult to trace
Stratified	The population is divided into smaller groups, that don’t overlap	Snowball	Dependent on the ease of access to subjects
Cluster	The entire population is divided into sections or clusters	Quota	Selection of members happens on basis of a pre-set standard
Proportion to percentage	Random sampling techniques to select a sample from subpopulations	Voluntary response	Made up of participants who have voluntarily chosen to participate

Step 4: Research tools

The research design and tools need to be decided prior to the start of data collection. These tools should focus on garnering data that will provide rich insights into the external factors which enable or deter the achievement of impact and program objectives. Make sure each question adds value and drives survey responses that relate directly to the research goals and evaluation questions. Keep wording focused on the respondent and their opinions, rather than introducing a point of view of your own. Undertake a pilot study to understand possible response biases. 

Step 5: Field research

A work plan needs to be made which sets out the various stages of data collection, analysis, and deadlines for the various deliverables. A qualified team needs to be chosen and trained to conduct the field research. There are several tools like Open Data Kit, SurveyCTO, KOBO, Google forms, Magpie, Track you Metrics, etc to collect data. Supervisors need to ensure the quality of data on the field along with the enumerators and monitor the data regularly. 

Step 6: Reporting

The report of an impact study should cover the objectives of the evaluation and should be easy to understand. It should reflect the purpose and answer the key questions of the research. The components of a report are:

• Cover Page 

• Table of Contents 

• List of Figures and Tables 

• List of Abbreviations 

• Executive Summary 

• Introduction 

• Methodology 

• Results and Findings 

• Summary of Results according to OECD DAC 

• Recommendations 

• List of Annex 
  

Step 7: Sharing findings with stakeholders and using the results for strategy

Just providing a comprehensive report to the stakeholders is not enough, communicating it in the best way possible is important too. The findings of the reports can be shared through presentations, summarised versions of the report, recorded videos, etc. Use existing online tools and software to create engaging and interesting content. Guide the stakeholders to use the findings of the report to bring about the necessary changes to the program. 

• Give real-time results to stakeholders to help them start working on it from the beginning. 

• Recommendations must be prioritized within a time frame to the extent possible so that the next measures can easily be taken. 

• The learnings must be incorporated into the overall programming cycle through an effective feedback system.