"Realist evaluation" in action: A worked example of the Aboriginal Parental Engagement Program

"Realist evaluation" in action: A worked example of the Aboriginal Parental Engagement Program

5 December 2013
"Realist evaluation" in action: A worked example of the Aboriginal Parental Engagement Program

Margaret Cargo and Lisa Warner discuss the "realist" approach used to evaluate the Aboriginal Parental Engagement Program (APEP).

Theory-based approaches to evaluation, such as realist evaluation,2 make explicit "for whom" and "in which circumstances" programs work and "how" they work to achieve outcomes.

Realist evaluation has the potential to strengthen the basis for understanding how programs work, particularly for Aboriginal urban populations where the needs are complex, especially given the history of colonisation, institutional racism, mobility and kinship structures and obligations which often do not fit with mainstream programming requirements.

Characteristics of a realist evaluation

A realist evaluation approach starts with the premise that a program works by enabling participants to make choices.

Choices are enabled by providing participants with new resources through the program (i.e., information, a safe place to yarn) that influence their reasoning (i.e., perceived competencies).  The combination of resources and reasoning is the mechanism that explains "how" a program leads to expected program outcomes.

Mechanisms, however, are activated in different contexts. The context pertains to "for whom" and "in which circumstances". Contexts are broad-ranging and can include characteristics of the community, programming environment (e.g., resourcing) and participant (e.g., living situation).  From a realist evaluation perspective, the same program can work in different ways for different participants, depending on the context.

A program can be represented by one or more theories and displayed in a logic model or as hypotheses. Whether visual or written, the theory is expressed as a series of context-mechanism-outcome (CMO) configurations that depict how program strategies will lead to outcomes for participants. Realist evaluation utilises mixed methods to test CMO configurations.

Example of a realist evaluation

Realist evaluation was used to evaluate the Aboriginal Parental Engagement Program (APEP), funded by the former federal government Department of Education and Employment Relations, and implemented by the Young Women’s Christian Association of Adelaide. The APEP was a two-stage group-based program aimed at enhancing the school readiness of 0-5 year old Aboriginal children by strengthening parents’ capacity to engage in education services, as well as their skills to parent. The program was designed and facilitated by a local Aboriginal woman, and engaged 31 parents and carers of Aboriginal children. The APEP logic model was developed over a six-week period and was based on extensive community consultations conducted by the facilitator. The consultations and negotiations with the community ensured that the stories, needs and wants of the Aboriginal community were understood and that the program was culturally appropriate and supported. To meet the needs of the funding and sponsoring organisations, a pipeline logic model was developed to illustrate how program resources and strategies led to shorter and longer-term impacts and outcomes for parents and children. A more culturally appropriate visual model for participants and the facilitator was developed. It depicted the facilitator, group, mum, child, and the mum and child’s interface with education, community, and health services along with hypothesised shorter, intermediate and longer-term outcomes (see the feature image). The model enabled the facilitator to understand what she needed to do to set in motion the positive changes in parents to enhance the school readiness of their children. The logic model along with the program story was shared with parents at the first session. Parents immediately understood how they would benefit from the program. For the evaluator, the logic model provided a blueprint for developing survey and interview questions to test the causal mechanisms in the outcomes chain, and in relation to different parent characteristics and circumstances.

The example CMO Configuration below shows that the outcome of parental educational engagement was expressed differently depending on parents’ and children’s circumstances (the "context"). It also shows that multiple mechanisms contributed to parental engagement. In particular, parents who lacked confidence or self-esteem talked about also needing to feel less intimidated in regards to meeting with the school to discuss their children’s concerns. Note that some mechanisms were generated inductively from the analysis and reflect a refinement of the initial theory.

Example of a CMO Configuration
Context Parents with a history of social disadvantage and social marginalisation. As the program gained credibility, it became a referral point for parents with Aboriginal children in state care. The following educational engagement outcomes were observed for parents with children in their care (and not in state care and not parents who were pregnant).  The * depicts that this additional mechanism was activated for parents with children in their care who also had low levels of confidence, self esteem or both.
Program Strategies
  • Yarning in a group
  • Facilitators educating parents on their obligations and responsibilities in their child’s education
  • Educating parents on how the education system operates and what resources and supports are available for Aboriginal children
  • Educating on the elements of culturally safe schools and quality learning environments
  • Role playing interpersonal communication supported by story telling
  • Ongoing feedback for parents taking steps to engage with the education system
  • Mentoring
Mechanisms
  • Feel competent to ask questions and communicate concerns
  • Understand the education system
  • Understand parental role in child’s education
  • Feel less intimidated*
Educational Engagement Outcomes
  • Parents enter the kindy and talk with the teacher
  • Parents set meetings with teachers or school staff to address child’s educational needs
  • Parents visit new kindies
  • Parents switch kindies based on assessment of educational quality

* This additional mechanism was activated for parents with children in their care who also had low levels of confidence, self-esteem, or both.  

Further reading

Kazi, M. A. F. (Ed.). (2003). Realist Evaluation in Practice. London: SAGE Publications Ltd. doi: http://dx.doi.org/10.4135/9781849209762

Priest, N. (2007). ‘Motor Magic’: Evaluation of a community capacity-building approach to supporting the development of preschool children. Australian Occupational Therapy Journal, 53, 220-232.

2. For more information see Pawson, R. (1997). Realist Evaluation. London: Sage Publication Ltd.

Add a comment

Authors

Margaret Cargo

School of Population Health, University of South Australia.

Lisa Warner

School of Population Health, University of South Australia.

Need some help?

CFCA offers a free research and information helpdesk for child, family and community welfare practitioners, service providers, researchers and policy makers through the CFCA News.

Subscribe

CFCA News

Sign up to our email alert service for the latest news and updates

Subscribe