Theory-based approaches to evaluation, such as realist evaluation,² make explicit "for whom" and "in which circumstances" programs work and "how" they work to achieve outcomes.

Realist evaluation has the potential to strengthen the basis for understanding how programs work, particularly for Aboriginal urban populations where the needs are complex, especially given the history of colonisation, institutional racism, mobility and kinship structures and obligations which often do not fit with mainstream programming requirements.

Characteristics of a realist evaluation

A realist evaluation approach starts with the premise that a program works by enabling participants to make choices.

Choices are enabled by providing participants with new resources through the program (i.e., information, a safe place to yarn) that influence their reasoning (i.e., perceived competencies).  The combination of resources and reasoning is the mechanism that explains "how" a program leads to expected program outcomes.

Mechanisms, however, are activated in different contexts. The context pertains to "for whom" and "in which circumstances". Contexts are broad-ranging and can include characteristics of the community, programming environment (e.g., resourcing) and participant (e.g., living situation).  From a realist evaluation perspective, the same program can work in different ways for different participants, depending on the context.

A program can be represented by one or more theories and displayed in a logic model or as hypotheses. Whether visual or written, the theory is expressed as a series of context-mechanism-outcome (CMO) configurations that depict how program strategies will lead to outcomes for participants. Realist evaluation utilises mixed methods to test CMO configurations.

Example of a realist evaluation

Realist evaluation was used to evaluate the Aboriginal Parental Engagement Program (APEP), funded by the former federal government Department of Education and Employment Relations, and implemented by the Young Women’s Christian Association of Adelaide. The APEP was a two-stage group-based program aimed at enhancing the school readiness of 0-5 year old Aboriginal children by strengthening parents’ capacity to engage in education services, as well as their skills to parent. The program was designed and facilitated by a local Aboriginal woman, and engaged 31 parents and carers of Aboriginal children. The APEP logic model was developed over a six-week period and was based on extensive community consultations conducted by the facilitator. The consultations and negotiations with the community ensured that the stories, needs and wants of the Aboriginal community were understood and that the program was culturally appropriate and supported. To meet the needs of the funding and sponsoring organisations, a pipeline logic model was developed to illustrate how program resources and strategies led to shorter and longer-term impacts and outcomes for parents and children. A more culturally appropriate visual model for participants and the facilitator was developed. It depicted the facilitator, group, mum, child, and the mum and child’s interface with education, community, and health services along with hypothesised shorter, intermediate and longer-term outcomes (see the feature image). The model enabled the facilitator to understand what she needed to do to set in motion the positive changes in parents to enhance the school readiness of their children. The logic model along with the program story was shared with parents at the first session. Parents immediately understood how they would benefit from the program. For the evaluator, the logic model provided a blueprint for developing survey and interview questions to test the causal mechanisms in the outcomes chain, and in relation to different parent characteristics and circumstances.

The example CMO Configuration below shows that the outcome of parental educational engagement was expressed differently depending on parents’ and children’s circumstances (the "context"). It also shows that multiple mechanisms contributed to parental engagement. In particular, parents who lacked confidence or self-esteem talked about also needing to feel less intimidated in regards to meeting with the school to discuss their children’s concerns. Note that some mechanisms were generated inductively from the analysis and reflect a refinement of the initial theory.

* This additional mechanism was activated for parents with children in their care who also had low levels of confidence, self-esteem, or both.  

Further reading

Kazi, M. A. F. (Ed.). (2003). Realist Evaluation in Practice. London: SAGE Publications Ltd. doi: http://dx.doi.org/10.4135/9781849209762

Priest, N. (2007). ‘Motor Magic’: Evaluation of a community capacity-building approach to supporting the development of preschool children. Australian Occupational Therapy Journal, 53, 220-232.