In a previous blog post, we explored the crucial role evaluation plays in telling a compelling program or project story. Here’s a simple step-by-step process for setting up an evaluation that works for your organization – and helps you tell the story of why you do what you do and how it’s going.
Getting started
The first step is to determine what kind of evaluation will be most useful to you. Regardless of whether you need to improve your processes, measure your outcomes, or understand your impact, you should begin by asking:
- Why do you do your work?
- What are you doing to get there?
- Who’s involved?
- What are you hoping to gain from the evaluation? What is the object of the evaluation?
Answering these questions will help you determine the kind of evaluation you need. It will help you identify who should participate in the evaluation and what resources you need to complete the evaluation. It will also help you identify any gaps between what you’re doing and where you’re trying to go. Together they will put you on the path to a productive evaluation plan.
Choosing the right indicators
As Einstein once famously said, “not all things that can be counted matter and not everything that matters can be counted.”
How can you tell if what you’re measuring matters? One good way is if the measures (indicators) are informed by what program participants and beneficiaries, not just funders, researchers or third-party evaluators, think are meaningful. While not always feasible, a sound evaluation practice is to include a diversity of folks with knowledge of the project and/or the issue it addresses, in the creation and implementation of an evaluation.
Through such a participatory process a mix of relevant objective and/or subjective indicators will likely emerge. Objective indicators are impartial, and typically measure quantifiable outputs and outcomes like number of materials created, workshop participants etc.. These can be valuable for helping you assess whether a project is being implemented as planned. Such measures cannot tell you if the activities undertaken were valuable to beneficiaries, produced surprising outcomes or learning, or were better than the alternatives. Such judgements are inherently subjective and as such require subjective measures that glean opinions, feelings and experiences of those asked.
Other criteria for good indicators are that they are easy to understand relevant to the purpose of the evaluation, reliable and feasible to measure, which means they are informed by stable and accessible data sources. Ensuring there are sufficient resources to measure the chosen indicators is another pragmatic consideration.
Completing the evaluation
There are as many evaluation approaches and methods as there are evaluation purposes. Approaches can be theory or objectives based, focused on economic or societal impact, or participatory or experimental in design. Methods can be quantitative, qualitative or mixed. As a rule, the methods chosen should be appropriate, accessible, and comprehensible. They should protect the identities and interests of those involved, and data should be securely stored. Participants should understand what’s being asked of them and have opportunities to ask clarifying questions. We recommend that the tools should be easy for evaluators and participants alike to use. Finally, evaluation participants should be fairly compensated for their time and effort. Evaluators should strive to engage a robust and representative sample of participants.
What to do with the results
After reviewing and interpreting the data collected, a story should emerge that illuminates lessons learned, areas for improvement or further investigation, innovation potential, and hopefully, successes. By sharing stories of learning and change, there is an opportunity to deepen your connection with program staff and beneficiaries, funders, and the public at large. As a result, you might attract broader bases of support and new opportunities to grow the impact of your work.
Whether you’re brand new to evaluation or looking tweak your existing evaluation practices, we’re here to help. Please contact Kim [at] whistlercentre.ca to learn more about our evaluation services and how we can tailor them to meet your needs.