Companion toolkit: Evaluating a playgroup

What are playgroups?

Playgroups are community-based groups that meet regularly to bring together young children (0-5 years) and their parents or caregivers. Playgroups aim to provide opportunities for parents and carers to build social networks and for children to increase engagement in developmentally appropriate play activities (Commerford & Robinson, 2017).

There are 2 main types of playgroups: 'community' playgroups and 'supported' playgroups.

Community playgroups are organised by local communities and are usually self-managed by parents and carers who attend the playgroup.

Supported playgroups are run by a paid facilitator and often aim to engage families who are experiencing vulnerabilities and may face barriers to attending community playgroups. Supported playgroups have a further aim of improving the development and wellbeing of both parents/carers and their children.

What are the challenges of evaluating playgroups?

Due to the diversity of playgroup offerings and the variety of intended outcomes for playgroups across the levels of community, family, parents/carers and children, there is no single evaluation tool or methodology that is most effective for evaluating playgroup outcomes.

It is therefore important to plan for an evaluation design that has the flexibility and elasticity to cater to the variability of playgroups (Dadich & Spooner, 2008) and that reflects the differences in style, operation and context within which they are delivered. In addition to the many possible intended outcomes of a playgroup, some of the main challenges in evaluating playgroup outcomes include:

• engagement of families in the evaluation process: It can be a challenge for evaluators to engage families that are experiencing vulnerabilities or disadvantage. This may be particularly true for supported playgroups, which often focus on supporting families experiencing vulnerability.
• burden on volunteers: Evaluation has the potential to place a considerable burden on playgroup conveners. It may be difficult for them to allocate time and resources to do this work consistently in addition to their regular roles (Dadich & Spooner, 2008). This may be a particular issue for playgroups that operate with the support of volunteers.
• high turnover of participants: Playgroups can face the issue of high turnover of participants. For many playgroups, the membership can change from week to week, which can make it difficult to maintain a consistent cohort of evaluation participants or to assess changes in outcomes over time (Dadich & Spooner, 2008).
• not always clear start and end dates: Participants will often start or stop attending a playgroup at different times. This can make it difficult to collect data consistently or to collect data from a whole cohort of participants.

A common challenge in evaluating all community-based programs - including playgroups - is that, for all the reasons outlined above, it is often not feasible to undertake an evaluation using experimental or quasi-experimental methods (such as a randomised control trial). Without these methods, it can be difficult to obtain the kind of causal evidence that would allow you to clearly attribute any observed changes to participation in the playgroup (Dadich & Spooner, 2008). This means it is important to consider how to design the most rigorous evaluation you can to manage the challenges and to get information you can use.

How do I manage the challenges of evaluating a playgroup?

There are steps you can take to manage the challenges involved in an evaluation of a community or supported playgroup. Some of these are summarised below.

• Carefully consider your evaluation design and what research questions you can realistically answer based on the data you can reasonably access or collect (Dadich & Spooner, 2008). It is better to undertake a well-considered evaluation that focuses on collecting good quality data in relation to a small number of questions than to try and measure everything.
• Reduce the burden on families, staff and volunteers by using either existing data (i.e. data already being collected by playgroup facilitators) or secondary data sources (Dadich & Spooner, 2008). Other sources that may be useful include the SEIFA Index, the Australian Early Development Census and the Longitudinal Study of Australian Children (LSAC), which includes an infant survey that identifies whether children are engaged in playgroups. Be aware that secondary data are unlikely to address all the outcomes you are looking to measure and will need to accompany other data collection methods. However, it may provide a baseline from which to compare the outcomes of children attending a playgroup (Dadich & Spooner, 2008).
• Choose your measurement tools carefully. To ensure that you have quality data, it is important to choose data collection tools that are valid, reliable and will collect the information you need. However, it is also important to think about what is feasible in your context. In particular, you should consider how any measurement tools will be administered so that you can avoid damaging the relationship the facilitator is trying to build with the families, preventing families from returning to the playgroup or negatively affecting the role of the playgroup as a soft-entry point to other services.

In later sections of this toolkit we suggest some helpful instruments and methods to use when individuals or communities might need more support to engage in an evaluation.

See Table 2 for a list of outcome measurement tools that you could use in an evaluation of a playgroup.

• Undertake pre- and post-testing using an outcomes measurement tool even if your program has a high turnover of participants or doesn't have clear start or end dates. Instead of trying to collect data from families after they leave the playgroup (which can be difficult), you can consider using a measurement tool to collecting pre-program data from participants when they commence with the playgroup and then at regular time points (e.g. you could survey everyone in attendance at a nominated session every 3-6 months). This approach can help to measure outcomes for a greater number of participants.

Key tips for evaluating a playgroup

There are some steps to consider when planning an evaluation, including determining the evaluation purpose, design, data collection methods, audience and available resources. These key steps are shown at Figure 1. The following sections offer some more specific considerations in relation to playgroups.

Figure 1: Key steps in an evaluation

For more detailed information on the different steps involved in designing your evaluation see AIFS resources:

• Evaluation design
• Planning an evaluation

Identify outcomes, measures and indicators relevant for your playgroup

Program logic

Your program logic model is a good place to start when identifying outcomes for your evaluation. It provides an overview of the anticipated changes the playgroup intends to influence over the short, medium and long term. Once you have identified your intended outcomes, you should be able to draw out your indicators. 

We have developed examples of Supported and Community Playgroup Program Logics (link) that are intended as a starting point from which you can consider your playgroup's particular needs and anticipated outcomes.

For more information on how to develop a program logic see AIFS resources:

• How to develop a program logic for planning and evaluation
• A guided tour through: Program logic models
• Getting the most out of program logic models

Selecting indicators

An indicator is the thing that you need to measure or observe to see if you have achieved your outcomes. Selecting your indicators will help you know what data to collect.

Table 1 provides a list of possible indicators for measuring outcomes for playgroups. This list is a starting point and there may be other outcomes and indicators relevant to your playgroup.

Selecting measures

Table 2 describes some valid and reliable tools that you can use to measure the outcomes of your playgroup. You can also refer to the Communities for Children Outcomes Measurement Matrix for a more comprehensive list of measurement tools. The matrix is not an exhaustive list of available tools but provides suggestions for some commonly used measurement tools. You will still need to thoroughly assess any particular tool before you use it. Explanatory details about the instruments are provided in the matrix and it is recommended that you contact the developer of the tool to discuss your requirements.

Table 2: Sample measurement tools for use in playgroups

For more information on how to choose an outcomes measurement tool see:

• How to choose an outcomes measurement tool
• Communities for Children Outcomes Measurement Matrix

Approaches to consider where communities need more support to engage with an evaluation

Collecting data from parents/carers in communities experiencing disadvantage can be challenging for several reasons, including because they may:

• be reluctant to provide information because they have concerns about confidentiality and anonymity
• have had previous negative experiences with other services and/or researchers that could mean they are reluctant to provide information for your evaluation (particularly if it is not being conducted in a trauma-informed way)
• have low English language or literacy skills and may have difficulty participating in surveys, interviews and other data collection processes
• be having difficulties managing everyday stress, making it hard for them to prioritise a task such as completing a survey or taking part in an interview or focus group.

Some measurement tools may be too intrusive for use in playgroups. Care must be taken to review the suitability of standardised measurement tools for use with people who have experienced disadvantage to ensure that using that tool will not damage the relationship the facilitator is trying to build with the families or prevent families from returning to the playgroup.

The following approaches are examples of the use of an observational tool and a qualitative tool that may be appropriate for playgroups - particularly where you want to avoid using tools that may be perceived as intrusive. These example tools are not strictly outcome measurement tools but are commonly used in process evaluation and may help to develop a more complete picture about your playgroup participants. For more ideas of tools that can be used in playgroups see Table 2.

Leuven scale

The Leuven scale uses a three-part process: (1) direct assessment of children's levels of wellbeing and involvement through a scanning of the group of children; (2) analysis of the observations; and (3) reflective discussion and action planning for improvements (Marbina et al., 2015).

The Leuven scale is concerned with the child's wellbeing (how the child feels about the playgroup) and involvement (how deeply the child engages in the activities offered at the playgroup) (Marbina et al., 2015).

By focusing on the child's wellbeing and involvement through structured observations, the Leuven Scale provides a less invasive, more accessible and potentially more accurate method of evaluation in vulnerable communities (when compared to less structured observations). This approach can help ensure that the data collected are both meaningful and respectful of the participants' circumstances and needs.

Photovoice

In photovoice methods, photographs are used as a prompt for reflection and discussion. This method of data collection aims to give voice to people whose voices may otherwise not be heard and/or who may not be comfortable or able to express themselves in other data collection methods (such as interviews or surveys). It is often used with people who may be experiencing disadvantage such as children and youth in difficult circumstances, homeless adults and families and people with disabilities or mental health issues (Community Tool Box, 2016). It is also commonly used for promoting community capacity-building and evaluating community programs (Boddy & Cartmel, 2011).

Photovoice enables children and parents/carers to document their views about their playgroup and community through photos (Boddy & Cartmel, 2011). Beginning with training users in photo taking and establishing the playgroup as the focus of the photo taking, the photos taken by participants are then compiled and used as a basis for discussion and reflection on the participants' strengths and any concerns about the playgroup (Boddy & Cartmel, 2011). This process aims to promote dialogue with the participants and a shared understanding of the most important issues (Boddy & Cartmel, 2011). The photographs, and any artworks, are then presented to an audience of key stakeholders (Boddy & Cartmel, 2011).

Evaluating playgroups with Aboriginal and Torres Strait Islander communities

There are additional considerations when conducting research with Aboriginal and Torres Strait Islander people and communities. Gibson (2018) has suggested the use of mixed methods evaluation with Aboriginal and Torres Strait Islander communities to measure a range of outcomes and to ensure that Aboriginal and Torres Strait Islander perspectives are heard.

For more information on evaluation with Aboriginal and Torres Strait Islander communities see the AIFS resource:

• Resources to support evaluation with Aboriginal and Torres Strait Islander people and their communities

If you decide to use standardised outcome measurement tools when working with Aboriginal and Torres Strait Islander communities, it is useful to check that the participants who are likely to be attending your program understand the items. Cultural and language differences can lead to ambiguity and confusion. Testing the items with people from relevant cultural groups can help ensure that the measure is likely to collect the right data (in part by reducing the chance people will not answer confusing or unclear items) and reduce the risk of collecting inaccurate or misleading data (because the questions are misunderstood).

The following case study provides an example of how standardised outcome measurement tools can be used when working with Aboriginal and Torres Strait Islander communities.

Conclusion

Evaluating community and supported playgroups can be challenging for a variety of reasons related to the service delivery model and the context in which playgroups are delivered. But there are steps you can take to manage these challenges.

This companion toolkit provides tips to address some of the common challenges that can arise when evaluating playgroups and highlights approaches that may be useful when working with communities needing more support to engage in evaluation. This guide is intended to be read alongside our more general evaluation resources.

A list of other resources and further reading to complement this toolkit can be found below.

Examples of playgroup evaluations

The following published evaluation reports offer some examples of how playgroups have been evaluated in Australia. Note these evaluations have not been undertaken by AIFS and their inclusion here does not imply any assessment or endorsement of quality.

• Evaluation of the KindiLink Pilot Initiative in Western Australia - Overview and key findings
• PlayConnect Evaluation - Playgroup Australia
• 'Sharing the Happiness': The role of supported playgroups in achieving settlement outcomes for newly arrived families from refugee backgrounds

Further reading

AIFS resources

• A guided tour through: Measuring outcomes
  This video guides child and family practitioners through measuring outcomes, explaining key terminology, practical steps, and standardised measures in an accessible way for those with limited evaluation experience.
• A guided tour through: Program logic models
  This video walks child and family practitioners through creating program logic models, explaining key terminology and practical steps with clear examples.
• Evaluation design 2021
  This resource gives a quick overview of some of the main evaluation designs used for outcomes evaluations or impact evaluations.
• Getting the most out of program logic models 2021
  This resource describes how two Families and Children (FaC) Activity funded programs used program logic to facilitate program planning and evaluation.
• How to choose an outcomes measurement tool 2016
  This article outlines how to choose an outcomes measurement tool, and provides links to established examples for use with children and families.
• How to develop a program logic for planning and evaluation 2016
  This resource has been designed for Communities for Children service providers, but may be used by anyone who is interested in developing a program logic.
• How to do trauma-informed research and evaluation 2024
  The practice guide covers what trauma-informed research is, the value of considering trauma throughout the research process and high-level principles of trauma-informed research and evaluation to apply within projects.
• How to write a survey questionnaire for evaluation: A guide for beginners 2023
  The resource provides basic information and practical tips to help you design and implement simple survey questionnaires for your program evaluation activities.
• Identifying evaluation questions 2019
  This short article is for people new to evaluation who are planning to conduct or commission an evaluation.
• Planning an evaluation
• Principles for doing trauma-informed research and program evaluation 2020
  This resource is for evaluation beginners needing help with developing an evaluation plan for a program, project or service for children and families.
• Resources to support evaluation with Aboriginal and Torres Strait Islander people and their communities 2024
  This resource sheet provides a list of resources for non-Indigenous evaluators and/or child and family services who need to plan, undertake, or commission an evaluation for programs that work with Aboriginal and/or Torres Strait Islander children and their families.
• Using qualitative methods in program evaluation 2016
  This article outlines some key considerations for using qualitative methods in program evaluation.
• Using a survey to collect data for evaluation: A guide for beginners 2023
  This practice guide is for people working in child and family support services who have limited experience or training in using surveys.

Other resources

• Australian Evaluation Society - First Nations Cultural Safety Framework
• Better Evaluation
• Community Tool Box
• Evaluation: What is it and why do it?
• Families and Children Activity SCORE Translation Matrix
• Implementing Photovoice in Your Community
• Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide Program Evaluation Home - CDC
• SNAICC Aboriginal and Torres Strait Islander Child and Family Services: Evaluation Readiness Toolkit 2024
• Tools to Support Culturally Safe Evaluation - Lowitja Institute
• World Health Organization Evaluation Practice Handbook

References

Australian Research Alliance for Children Youth (ARACY). (2020). PlayConnect Evaluation: Final report - March 2020. 
www.playgroupaustralia.org.au/resource/playconnect-evaluation

Boddy, J., & Cartmel, J. (2011). National early childhood care and development programs desk top study: Final report. Prepared for Save the Children. Brisbane: Griffith University.

Commerford, J., & Robinson, E. (2017). Supported playgroups for parents and children: The evidence for their benefits. Melbourne: Child Family Community Australia, Australian Institute of Family Studies. aifs.gov.au/sites/default/files/fm99-commerford-robinson_0.pdf

Community Tool Box. (2016). Chapter 3: Assessing community needs and resources. In Tools to change the world. Lawrence, KS: Work Group for Community Health and Development, University of Kansas. ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/photovoice/main

Dadich, A., & Spooner, C. (2008). Evaluating playgroups: An examination of issues and options. The Australian Community Psychologist, 20(1), 95-104.

Gibson, M. (2018). Rapid evidence review of supported playgroups that support Aboriginal communities. Melbourne: The Centre for Family Research and Evaluation.

Marbina, L., Mashford-Scott, A., Church, A., & Tayler, C. (2015). Assessment of wellbeing in early childhood education and care: Literature review. Melbourne: Victorian Curriculum and Assessment Authority.

Muir, S., & Dean, A. (2017). Evaluating the outcomes of programs for Indigenous families and communities (CFCA Practice Resource). Melbourne: Australian Institute of Family Studies. aifs.gov.au/cfca/sites/default/files/publication-documents/evaluating-outcomes-porgrams-indigenous.pdf