Identifying evaluation questions

Content type
Practice guide
Published

November 2019

1 November 2019

 

This short article is for people new to evaluation who are planning to conduct or commission an evaluation.


What are evaluation questions?

Evaluation questions, sometimes referred to as key evaluation questions or KEQs, are high-level questions that are used to guide an evaluation. Good evaluation questions will get to the heart of what it is you want to know about your program, policy or service. Evaluation questions are different from the questions that you might ask in a survey, questionnaire or interview but you need clear evaluation questions to determine what questions to ask in a survey or interview and what methods to use. You often need to ask several questions in a survey to address a single evaluation question. And depending on the scale of your project and your evaluation, you may require different sources of data, or data collected using different methods to answer a single evaluation question.

Why evaluation questions are useful

Identifying evaluation questions can seem like yet another step in an evaluation planning phase but taking the time to do this before you start an evaluation means that the evaluation is more likely to tell you what you want to know. When evaluation questions are not worked out at the start of the process, or are not thought through, it is common to get to the end of an evaluation and realise that the collected data do not address the questions that you really needed answered. Identifying evaluation questions at the start will also guide your decisions about what data collection methods are most appropriate.

How to develop evaluation questions

It works best to develop evaluation questions collaboratively – so bring together evaluators and evaluation users and get brainstorming, face-to-face if possible. Your evaluation users are the people who will use your evaluation; they are most likely program staff and managers. You could also include anyone you are working in partnership with; community members or other organisations, for example. Using your logic model is a really good place to start. Keep it simple; try asking: ‘What do we want to know about the program?’

How to prioritise evaluation questions

The challenge is often not in identifying evaluation questions, but in selecting which ones to focus the evaluation on. Depending on the scale of your evaluation you should aim for a maximum of five to seven questions. For smaller evaluations, two or three questions is plenty. It can be useful to group similar questions together and then prioritise those that are most important. BetterEvaluation recommends that you imagine scenarios where the answer to your question is being used, so that you can assess the relevance and usefulness of each potential question.

There are several factors to consider when prioritising questions:

  • Think about what you REALLY want to know. Cut out any jargon and unnecessary words to ensure your questions are clear and direct.
  • Consider the requirements of your funders. For example, if you need your program to meet the 50% requirement for Communities for Children providers, your evaluation will need to meet particular criteria.
  • Review what the questions are measuring. Process questions are designed to measure the implementation of your program, while outcome questions [opens in Youtube] are designed to measure the effect of the program on participants and communities. Both are useful. You should think through your questions to ensure you will get the information that you need.

This checklist from the Centers for Disease Control and Prevention in the US can help you to assess and prioritise your evaluation questions.

Sample evaluation questions

The questions below have been adapted from the BetterEvaluation website and the Ontario Centre of Excellence for Child and Youth Mental Health. 

Process or implementation questions

Were activities implemented as planned? (How often, when, where, duration, content
To what extent was the program implemented in line with the model? (Fidelity)

Were participants from the target group reached by the program? 
How satisfied were program participants?

What were the challenges and enablers to implementation? 
Were the required resources in place and sufficient?

Outcome questions

To what extent did the program achieve its short, medium, and long-term outcomes?  
(You can specify specific outcomes, and you don’t need to measure them all). 

For who are outcomes being achieved, in what ways and in what circumstances?

What unintended outcomes (positive and negative) were produced? 
To what extent can changes be attributed to the program?  
What was the influence of other factors?

Further reading

References

  • BetterEvaluation. (2016). Specify the key evaluation questions (KEQs)Retrieved from www.betterevaluation.org/en/rainbow_framework/frame/specify_key_evaluation_questions
  • Ontario Centre of Excellence for Child and Youth Mental Health. (no date). Sample key evaluation questions. Retrieved from www.excellenceforchildandyouth.ca/sites/default/files/docs/tools/t_imp_mo_38_sample_evaluation_questions.pdf

Featured image: © GettyImages/alvarez

Share