Recently, I attended a conference on Impact Evaluation hosted by the Community of Evaluators (COE)/ South Asia in Thimphu, Bhutan. The conference was all sorts of GREAT! Four days were divided into two days of pre-conference workshops, while the two-day conference saw several interesting panel discussions.
This being one of my first international conferences on Evaluation, I was not sure what to expect. However, sitting and reflecting on the conference, I would say it gave me a ton of interesting literature (which is keeping me busy), a bunch of new friends, profound conversations, tremendous realisations and learning, an understanding of evaluation from a global perspective with insights from different countries, and an opportunity to network and hear from a good mix of professionals: entrepreneurs, social sector leaders, researchers, Government and private sector personnel, etc.
To my absolute delight, Professor Robert Chambers hosted a full-day workshop on ‘Frontiers in Participatory Evaluation’. Considering he has been one of the key influences on how we approach development research, it was quite a privilege to be two feet away from him, discussing my work!
The interactive participatory workshop comprised many interesting games, exercises, and knowledge-focused presentations. He began the day by comparing evaluation to ‘cooking’; where he mentioned that a good evaluator (or chef) knows and applies a range of techniques and tools! His main message was that it is important to consider the Who/Whose questions, in terms of Whose voice? Whose reality, the importance to be constantly in touch with people we are working ‘with’ NOT ‘for’ and to understand that there is a constant change in conditions, awareness, aspirations, priorities of communities. He particularly was impressed with the gaining popularity of Participatory Statistics, which are defined as “a set of methods that enable local people to generate statistics for local-level planning, learning, and reflection, but which can also be aggregated at wider levels and feed into national level policy processes.”
Sharon Weir (Co-Founder, 4th Wheel) with Robert Chambers
Eventually, Robert Chambers concluded that “Innovation is in the gene of good evaluation” and the two most important aspects of good evaluation are good facilitation and harnessing the power of creativity. As an example, I shared the ‘hypothetical budget’ tool, which Payal and I conceptualised several years ago (to get ‘participatory statistics’ for a Social Return Investment Study), and have been using it ever since. The tool according to him signifies both – creativity and good facilitation. (So proud!)
Here is a short description of the tool:
Method
Community respondents are divided into groups of 4-5 people. Men, women, and youth form separate groups. Each group is given a chart paper. The method can be used in two ways – One where participants are given a set of pre-determined options by the research team, for example, health, education, infrastructure, sanitation, water, etc., and the other where the group should decide their responses. Once this list is prepared, the group is given a hypothetical budget of INR.10,000 to spend on the list. Respondents can prioritize any way they want with no minimum or maximum spending specified. Scores (budgets in rupees) are then averaged for each indicator/option to understand community priorities.
Application
This method can be applied in baseline research or an evaluation study. It helps to understand community perception of priorities and needs, can be used to compare donor/programme (percent) resource allocations with community needs (resource allocations), and serves as an inclusive triangulation tool among different stakeholders and heterogeneous communities. It also helps to build consensus on programme priorities and outcome indicators among respondents.
Examples – This method has been used several times by the 4th Wheel Social Impact (4WSI) research team. Some recent studies include:
~Needs assessment study: A construction company commissioned 4WSI to conceptualize and design their Corporate Social Responsibility (CSR) initiatives for construction workers at their site. Workers came from different states of India, having their own small regional groups and specific residence locations at the labour colony. All residents of the labour colony were invited to the Focus Group Discussion where CSR plans of the company were shared. They were briefed about the exercise and told the rules of the game. In this particular setting, the groups were asked to list the top five priorities in the labour colony first and then allocate budgets.
~Social Return on Investment (SROI) study: A CSR foundation had invested in improving the education status of 30 villages in Gujarat. We used this method to garner responses from diverse stakeholders- parents, teachers, local leaders, etc. We listed twelve education initiatives, beforehand and asked respondents to do two things – One, allocate budgets to the different initiatives, and two, put a price on the perceived benefits of the different initiatives. This helped to compare the foundation’s budget allocations to the (average) budget allocations of the different stakeholders. It also helped to gain a sense of the impact perceived by the community in monetary terms (this is slightly challenging).
Considerations
~ This method can build high expectations among a group. It is important to brief the groups on the aims and use of the results of this exercise.
~There is a possibility of some people dictating to the group. It is important to tell respondents to take everyone’s view and consensus, if needed, scores of the smaller group can also be averaged to arrive at one figure.
This blog is written by Sharon Weir, Co-Founder of 4th Wheel Social Impact.
If you need more information or have stories to share on using the tool, please write to us at – hello@the4thwheel.com
