Resources to support evaluation with Aboriginal and Torres Strait Islander people and their communities
February 2024
Overview
This resource sheet provides a list of resources for non-Indigenous evaluators and/or child and family services who need to plan, undertake, or commission an evaluation for programs that work with Aboriginal and/or Torres Strait Islander children and their families.
Introduction
Evaluation can be a powerful and transformative tool. With programs that are designed for or used by Aboriginal and Torres Strait Islander people, evaluation can highlight individual and community strengths, support self-determination and contribute to positive outcomes. To achieve these aims, best practice is for the evaluation to be designed and led by Aboriginal and Torres Strait Islander people. However, evaluations are often commissioned by non-Indigenous organisations and/or involve non-Indigenous evaluators. Although non-Indigenous evaluators or researchers may be competent within their own skill set and culture, they may not always apply these skills in a way that is viewed as culturally safe by Aboriginal and Torres Strait Islander people (Gollan & Stacey, 2021).
For non-Indigenous organisations or evaluators to plan, commission or take part in appropriate and culturally safe evaluations, they need to ensure that the evaluation process incorporates key values and approaches. These include valuing Aboriginal and Torres Strait Islander culture and knowledge, building positive relationships with Aboriginal and Torres Strait Islander people and/or organisations, placing Aboriginal and Torres Strait Islander people at the centre of decision making and using reflective practice (Gollan & Stacey, 2021; National Indigenous Australians Agency, 2020).
Building positive relationships with Aboriginal and Torres Strait Islander people requires understanding First Nations people’s deep connection to culture, family, kinship, community, and country. Approaching evaluation with an understanding and knowledge of Aboriginal children and their family’s ways of knowing, being and doing can help create a foundation for collaboration built on mutual respect, credibility, and trust (Department of the Prime Minister and Cabinet, 2021; Gollan & Stacey, 2021; Kelaher et al., 2018; National Indigenous Australians Agency, 2023).
The resources listed in this resource sheet provide guidance on the evaluation skills, approaches and values needed to evaluate programs designed for or used by Aboriginal and/or Torres Strait Islander people. Please note that this resource is an introduction to other key resources on this topic and is not a guide to undertaking evaluation with Aboriginal and/or Torres Strait Islander people or communities.
General resources are organised into the following topics:
- The role of partnerships
- Culturally safe evaluation
- Evaluation approaches, toolkits, and frameworks
- Evaluation examples
- Ethical evaluation
- Data sovereignty.
For comprehensive resources on how to plan, conduct or implement an evaluation, the Evidence and Evaluation support team have a suite of evaluation resources you can access.
If you have any other resources or examples that you would like included in this resource sheet, please email details to [email protected].
The role of partnerships
If you work in a mainstream organisation or are a non-Indigenous evaluator, consulting with and/or forming partnerships with Aboriginal and/or Torres Strait Islander people will be key to the success of your evaluation project.
- To learn more about working with Aboriginal and/or Torres Strait Islander people, listen to Advancing partnerships with Aboriginal and Torres Strait Islander organisations (webinar).
- The Commissioner for Children and Young People in Western Australia has also developed a toolkit for engaging with Aboriginal Children and Young People.
- For guidance on partnering with an Aboriginal and/or Torres Strait Islander organisation in and evaluation SNAICC's partnerships resources can help
Culturally safe evaluation
Cultural safety as described by SNAICC in a report for the Department of the Prime Minster and Cabinet (2021) is:
The positive recognition and celebration of cultures. It is more than just the absence of racism or discrimination and more than 'cultural awareness' and 'cultural sensitivity'. It empowers people and enables them to contribute and feel safe to be themselves. (page 7)
In evaluation, establishing and practising cultural safety is a way of addressing power imbalances that exist between non-Indigenous and Aboriginal and Torres Strait Islander peoples and of supporting self-determination. Aboriginal and Torres Strait Islander peoples have a rich and diverse cultural heritage with a strong connection to culture, land, family, and language (National Indigenous Australian Agency, 2023). In practising cultural safety and recognising the unique voices and perspectives of Aboriginal and Torres Strait Islander people, we can make better decisions about how to support those communities to thrive (Gollan & Stacey, 2021).
The resources in Table 1 can help you to understand and set up culturally safe practices when evaluating programs and services that involve Aboriginal and/or Torres Strait Islander people.
| Resource | Authors | Description |
|---|---|---|
| Australian Evaluation Society First Nations Cultural Safety Framework | Sharon Gollan and Kathleen Stacey | Outlines the principles underpinning culturally safe evaluation and gives practical guidance on establishing cultural safety throughout the evaluation process. |
| Evaluation literature informing the AES First Nations Cultural Safety Framework | Sharon Gollan and Kathleen Stacey | A companion document to the Australian Evaluation Society First Nations Cultural Safety Framework that summarises evaluation literature on culturally safe, culturally competent, and culturally appropriate approaches to evaluation practice. |
| Tools for Culturally Safe Evaluation | The Lowitja Institute | A suite of tools that evaluators and people commissioning evaluations can buy to support culturally safe evaluation. The tools consist of:
|
Evaluation approaches, toolkits, and frameworks
The resources in Table 2 can help you to understand your level of readiness to begin an evaluation and make decisions about how to design and implement your evaluation.
| Resource | Authors | Description |
|---|---|---|
| Evaluation readiness toolkit | SNAICC | Supports child and family services working with Aboriginal and/or Torres Strait Islander people to identify suitable program outcomes and to develop a monitoring and evaluation plan. |
| Indigenous Evaluation Strategy | The Productivity Commission | A whole-of-government framework that is designed for Australian Government employees involved in the evaluation of policies and programs affecting Aboriginal and/or Torres Strait Islander people. |
| A Guide to Evaluation under the Indigenous Evaluation Strategy | The Productivity Commission | This guide was developed as a companion document to the Indigenous Evaluation Strategy. It is designed for program managers and for anyone who is thinking of commissioning or using evaluations. The document has links to evaluation sources and references. |
| Indigenous Advancement Strategy Evaluation Framework | National Indigenous Australians Agency | A guide for conducting quality and ethical evaluations of programs under the Indigenous Advancement Strategy |
| Evaluating the outcomes of programs for Indigenous families and communities | Stewart Muir and Adam Dean, AIFS | Outlines some key considerations for community sector organisations and service providers who are thinking about evaluating the outcomes or impact of a program for Aboriginal and/or Torres Strait Islander families or communities. |
| An Evaluation Framework to Improve Aboriginal and Torres Strait Islander Health | Kelaher et al. (2018), University of Melbourne and the Lowitja Institute | A user guide that outlines principles underpinning Aboriginal and Torres Strait Islander health and wellbeing programs |
| Ngaa-bi-nya | Megan Williams, University of Technology Sydney | A practical evaluation framework for health and social programs. The framework encompasses 4 domains (landscape, resources, ways of working and learning) and provides practical tools for reflection. |
| Yanha-yalinya: Cultural responsiveness checking tool for services and programs | University of Technology Sydney and Yulang Indigenous Evaluation | A toolkit for services to assess their cultural responsiveness when designing, delivering and evaluating services and programs for Aboriginal and Torres Strait Islander people. |
Evaluation examples
Table 3 includes a brief selection of published evaluation reports commissioned by various organisations including government, community services and Aboriginal and/or Torres Strait Islander-led organisations. It is not intended to be a comprehensive resource but rather to show the evaluation objectives, characteristics of participants and main data collection methods that can be used to inform evaluation findings. You may find these examples helpful for planning your next evaluation.
| Report title | Program description | Evaluation objectives | Target group/s | Data collection methods |
|---|---|---|---|---|
| Culture and Kinship Evaluation Victorian Aboriginal Community Controlled Health Organisation | Explores the role of Community, Country, and Culture in improving the health and wellbeing of Aboriginal and Torres Strait Islander people in Victoria | To identify community-determined program outcomes experienced during the pilot phase and develop a theory of change | Four Victorian Aboriginal Community Controlled Organisations (ACCOs) and their program participants, which included Aboriginal and Torres Strait Islander youth, young adults, families, and Elders | The report is based on a developmental evaluation that primarily used yarning to understand the program’s impact on Aboriginal and Torres Strait Islander communities. These findings were integrated with a Social Return on Investment methodology to demonstrate the value of the ACCOs’ programs. |
| Aboriginal Family Violence Prevention and Legal Service Victoria's Early Intervention and Prevention Program Evaluation Aboriginal Family Violence Prevention & Legal Service Victoria | The program’s Early Intervention and Prevention (EIP) activities aim to build capacity to prevent family violence, address the impacts of family violence, and reduce family violence re-victimisation. | The evaluation explored the outcomes expected from the EIP activities, and the availability and usefulness of existing data (qualitative and quantitative) to understand program outcomes. | Aboriginal and Torres Strait Islander women in Victoria. | A mixed methods evaluation that used qualitative and quantitative data to understand the short, medium- and long-term impacts of EIP activities. |
| Evaluation of the Parental and Community Engagement report (PaCE) Benevolent Society | Assists families and communities to engage in their children’s education and help improve educational outcomes. | The evaluation aimed to assess the effectiveness of the PaCE program for children and parents/caregivers. | Aboriginal and Torres Strait Islander families and children aged 0–19 years of age. | The evaluation used a collaborative mixed methods design that included yarning and Most Significant Change methods. The evaluation also sourced data from program activity templates and a client management system. |
| Evaluation of Marram-Ngala Ganbu Children’s Court of Victoria | A Koori-centred approach that aims to accommodate the needs of Aboriginal and Torres Strait Islander families involved in court processes. | The evaluation assessed the performance of Marram-Ngala Ganbu against its stated aims. | Families who appeared in court through Marram-Ngala Ganbu. | The evaluation was informed by stakeholder interviews, court data and literature. |
| Think Martu. Think Differently. Act Differently. SVA consulting | Kanyirninpa Jukurrpa (KJ) supports Martu to care for culture and country and maintain strong and sustainable communities. | The evaluation sought to understand the extent to which KJ contributed to Martu outcomes over a 10-year period. | Martu communities. | The evaluation was largely qualitative, using workshops and community consultations as the main data collection methods. Evaluators also drew from annual reports and recordings of past interviews and consultation sessions. |
| Rante-rante ampe Marle and Urreye: "Safe, Respected and Free from Violence" projects evaluation ANROWS | These are 2 projects that aim to foster gender change narratives. ‘Girls Can Boys Can’ creates early childhood messaging to promote equitable relationships with both boys and girls. ‘Old Ways are Strong’ challenges narratives on colonial gendered attitudes of violence against Aboriginal women. | A core objective of this evaluation was to assess the communities’ attitudes and beliefs about gender and violence. Other objectives included identifying to what extent the partnership project had an impact on gender attitudes, violence, and Aboriginal culture; and if the resources developed by the project partnerships had communicated gender equitable messaging to a broad audience. | First Nations people from within Central Australia. | The evaluation used a participatory mixed method and Indigenist methodological approach. Project staff and community members alike were engaged in all aspects of evaluation design. Quantitative data collection included undertaking surveys in an accessible format and qualitative data were collected from key project informants at baseline and endline time points. |
| Evaluation of Ampe-kenhe Ahelhe: Children's Ground in Central Australia Children’s Ground | The Children’s Ground (CG) Approach is a 25-year longitudinal community-led strategy that aims to foster generational social, cultural, and economic change for First Nations children. | This evaluation examined the implementation progress and early changes of the first 3 years of the CG Approach. | Children and families who live in Central Australia | The evaluation used a participatory approach in which First Nations leaders and staff were involved in designing, implementing and communicating the evaluation. Both qualitative and quantitative data collection methods were used including surveys, administrative data, interviews and focus groups. |
| Uti Kulintjaku Watiku Project NPY Women’s Council | The Uti Kulintjaku Watiku (Men’s) Project aims to build the capacity of community, foster resilience, heal, and prevent family violence. | The objectives of this evaluation were to understand key project features, key developments and achievements, and the contributions made to preventing family violence. | Anangu men living in Central Australia. | This evaluation undertook a developmental evaluation approach. This involved holding workshops in which storytelling, visuals and drawings were used to convey key messages. Data collection also included participant observation, reflections, interviews, and a literature review. |
Ethical evaluation
While it is important to consider the principles of ethical evaluation in all research and evaluation activities, there are additional considerations when undertaking research and evaluation with Aboriginal and/or Torres Strait Islander people. The resources in Table 4 offer information and guidance on conducting ethical research and evaluation with Aboriginal and Torres Strait Islander people.
| Resource | Authors | Description |
|---|---|---|
| AIATSIS Code of Ethics for Aboriginal and Torres Strait Islander Research | Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) | Explains the principles underpinning ethical and responsible Aboriginal and Torres Strait Islander research and related responsibilities. |
| Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities | National Health and Medical Research Council (NHMRC) | Guidelines to support safe, respectful, responsible, high-quality, and beneficial research. This document is relevant to researchers, evaluators, and service providers. |
| Keeping research on track II | National Health and Medical Research Council (NHMRC) | A companion document to the NHMRC Ethical conduct guidelines that is aimed at Aboriginal and Torres Strait Islander research participants and communities. |
Data sovereignty
Data sovereignty refers to ‘the right of Indigenous peoples to exercise ownership over Indigenous Data.’ (Maiam nayri Wingara Indigenous Data Sovereignty, 2018). Given that data are central to evaluation, it is essential to have discussions about data sovereignty before beginning your evaluation project.
The Lowitja Institute offers a data sovereignty readiness assessment and evaluation toolkit that organisations can download. Kowa also have a tool for grassroots First Nations organisations to support the implementation of data sovereignty principles: OCCAAARS Framework for First Nations Data Sovereignty (FNDSov).
To learn more about data sovereignty and how it can be used to support the self-determination of Indigenous people around the world, read this free-to-access book, Indigenous Data Sovereignty, published by the Australian National University.
Bibliography
Department of the Prime Minister and Cabinet. (2021). Keeping Our Kids Safe: Cultural Safety and the National Principles for Child Safe Organisations. Canberra: Department of the Prime Minister, and Cabinet. Retrieved from www.snaicc.org.au/wp-content/uploads/2021/06/SNAICC-VACCA-OCS-ChildSafeReport-LR-with-alt-tags-May2021.pdf
Gollan, S., & Stacey, K. (2021). Aboriginal and Torres Strait Islander cultural safety framework. Melbourne: Australian Evaluation Society. Retrieved from AES_FirstNations_Cultural_Framework_finalWEB_final.pdf
Kelaher, M., Luke, J., Ferdinand, A., Chamravi., D., Ewen, S., & Paradies, Y. (2018). An Evaluation Framework to Improve Aboriginal and Torres Strait Islander Health. Melbourne: Lowitja Institute.
Kukutai, T., & Taylor, J. (2016). Indigenous data sovereignty. Canberra: ANU Press. Retrieved from doi.org/10.22459/CAEPR38.11.2016
Lowitja Institute. (2022). Tools for cultural safe evaluation. Melbourne: Lowitja Institute. Retrieved from www.lowitja.org.au/page/services/tools/evaluation-toolkit
Lowitja Institute. (2022). Indigenous data sovereignty readiness assessment and evaluation toolkit. Melbourne: Lowitja Institute. Maiam nayri Wingara Indigenous Data Sovereignty. (2018) Defining indigenous data sovereignty and indigenous data governance. Maiam nayri Wingara Indigenous Data Sovereignty. Retrieved from www.maiamnayriwingara.org/definitions.
National Health and Medical Research Council. (2018). Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders. Canberra: National Health and Medical Research Council. Retrieved from www.nhmrc.gov.au/about-us/resources/ethical-conduct-research-aboriginal-and-torres-strait-islander-peoples-and-communities
National Indigenous Australians Agency. (2020). Evaluations and evidence. Canberra: National Indigenous Australians Agency. Retrieved from www.niaa.gov.au/indigenous-affairs/evaluations-and-evidence
National Indigenous Australians Agency. (2023). Indigenous Advancement Strategy (IAS) evaluation work plan. Canberra: National Indigenous Australians Agency. Retrieved from www.niaa.gov.au/indigenous-affairs/evaluations-and-evidence/evaluation-work-plan
Acknowledgements
This resource was compiled by Deb Osborne and Kathryn Goldsworthy. We wish to thank Tony Kiessler of Akaltye Pty Ltd and members of the Child and Family Evidence Critical Friends Network for reviewing this resource sheet.
Related publications
Advancing partnerships with Aboriginal and Torres Strait…
This webinar explored the value of Aboriginal and Torres Strait Islander and non-Indigenous services working together…
Read more
Evaluating the outcomes of programs for Indigenous families…
Key considerations for those thinking about evaluating the outcomes or impact of a program for Indigenous families and…
Read more
Measuring outcomes in programs for Aboriginal and/or Torres…
This webinar discussed ways to measure the outcomes of programs for Aboriginal and/or Torres Strait Islander families…
Read more