Slide outline: Measuring outcomes in programs for Aboriginal and/or Torres Strait Islander families and communities

Measuring outcomes in programs for Aboriginal and/or Torres Strait Islander families and communities

1. Measuring outcomes in programs for Aboriginal and/or Torres Strait Islander families and communities<

    • Sharon Barnes
    • Kylie Brosnan
    • The views expressed in this presentation are those of the presenter and may not necessarily reflect those of the Australian Institute of Family Studies or the Australian Governmen

    2. Introducing Sharon Barnes

      • Home of the Ngunawal People

      3. Introducing Kylie Brosnan

        • Living on Yugara (Yuggera) Country

        4. Overview

          • Challenges we have had to overcome
          • Start with the basics
            • What is an outcome?
            • What is data?
            • What is evidence?
          • Learning the language of outcomes
          • Collecting the data
          • Turning data into evidence
          • Telling the story

          5. Challenges we have had to overcome

            • Lack of baseline data
            • Quick funding is up
            • Too soon to tell
            • Moving goal posts
            • Value free evidence
            • Reflexivity in evaluating the facts

            6. Evidence

              • [Venn diagram]
              • Circle 1 Values
              • Circle 2 Refection
              • Circle 2 empirical
              • Dialogue is when the circles overlap

              7. Outcomes


                • Clarity and collaboration on what behavior needs to change
                  • Behaviour identified
                  • Outcome
                  • Causal links identified
                • Theory on what causes change
                • Little appreciation what the underlying drivers of behavior
                • Little evidence to hypothesis about behaviours from Indigenous perspective

                8. Data

                  • (graphic)

                  9. Data

                    • A data point is not a data record - A data record is not an outcome
                      • Who (profile), did it with them< /li>
                      • What they did, did it with them, their behaviour was after
                      • Why they did it, things changed or not
                      • When they did it
                      • How often they did it, they felt before, they felt during, they felt after
                    • But importantly a data record is one that is all linked

                    10. Data

                      • Diagram showing the data cycle

                      11. Learning the language of outcomes

                        • Context
                        • Behaviour
                        • Reasoning

                        12. Collecting the data

                          • Photos of communities

                          13. Self-determination in the evaluation process

                            • Photos of communities

                            14. Tell the story

                              • [Graphic of a word cloud]
                                • responsibility
                                • accomplishment
                                • investigated
                                • define
                                • defining success
                                • true story
                                • advocate
                                • assessed
                                • ownership
                                • community
                                • voices
                                • empirical
                                • strong
                                • evaluation
                                • Aboriginal
                                • Telling the story
                                • Standards
                                • Researcher
                                • Flexibility
                                • Values
                                • Meaningful
                                • Stories
                                • Torres Strait Islander
                                • Perspective
                                • Balances
                                • Aspirations
                                • Outcomes
                                • Working
                                • Celebrate
                                • Measures
                                • Work
                                • patterns

                              15. Summing up

                                • Venn diagram’of intersecting circles of labelled Empirical, Values, Reflection
                                • Surrounded by a larger circle with the words Power and opportunity

                                16. The Ipsos Vision

                                  • Through our research and evaluation work and what we can contribute as an organisation Ipsos will make a positive difference to the lives and expectations of Aboriginal and Torres Strait Islander researchers and communities wanting to tell their story.”
                                  • Thank you for listening

                                  17. References

                                    • Altman, J., & Russell, S. (2012). Too Much'Dreaming': Evaluations of the Northern Territory National Emergency Response Intervention 2007-2012.
                                    • Biddle, N. (2016) Indigenous Insights for Indigenous Policy from the Applied Behavioural Sciences. Asia and the Pacific Policy Studies. John Wiley & Sons Australia, Ltd and Crawford School of Public Policy at The Australian National University.
                                    • Bromell, D. (2012). Evidence, values and public policy. ANZSOG Occasional Paper, ANZSOG, Canberra.
                                    • Cairney, P. (2016). Centre for Constitutional Change at the Economic and Social Research Centre in the UK in his post on Power to Persuade,Retrieved at: (
                                    • Patton, M. Q. (2008). Utilization-focused evaluation. Sage publications. 452
                                    • SenGupta, S., Hopson, R., & Thompson‐Robinson, M. (2004). Cultural competence in evaluation: An overview. New Directions for Evaluation, 2004(102), 5-19.
                                    • Symonds, J. E., & Gorard, S. (2010). Death of mixed methods? Or the rebirth of research as a craft. Evaluation & Research in Education, 23(2), 121-136.
                                    • Scott, D. (2007). Resolving the quantitative–qualitative dilemma: a critical realist approach. International Journal of Research & Method in Education, 30(1), 3-17.
                                    • Walter, M. and Andersen, C. (2013). Indigenous Statistics : A Quantitative Research Methodology, Left Coast Press, Los Angeles, California, pp. 158. ISBN 9781611322934 [Authored Research Book]
                                    • Woolgar, S. (1982). Laboratory studies: A comment on the state of the art. Social studies of science, 12(4), 481-498.

                                    18. Recommended reading

                                      • Good examples of realist evaluation methods for finding the “reasoning” and “behaviour” and “context”.
                                        • Jagosh, J., Macaulay, A. C., Pluye, P., Salsberg, J., Bush, P. L., Henderson, J., ... & Seifer, S. D. (2012). Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Quarterly, 90(2), 311-346.
                                        • Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review–a new method of systematic review designed for complex policy interventions. Journal of health services research & policy, 10(suppl 1), 21-34.
                                        • Cartwright, N., & Hardie, J. (2012). Evidence-based policy: a practical guide to doing it better. Oxford University Press.
                                        • Cargo M & Warner L 2013. “Realist evaluation” in action: a worked example of the Aboriginal Parental Engagement Program. Melbourne: Australian Institute of Family Studies. Viewed 14 March 2014.
                                        • Robinson, G., & Tyler, W. (2005, December). Ngaripirliga’ajirri-cross-cultural issues in evaluating an Indigenous early intervention program. In TASA Conference Proceedings 2005.
                                      • Good examples of where Aboriginal and/or Torres Strait Islander communities set the standards and values to define outcomes:
                                        • Nguyen, O. K., & Cairney, S. (2013). Literature review of the interplay between education, employment, health and wellbeing for Aboriginal and Torres Strait Islander people in remote areas: working towards an Aboriginal and Torres Strait Islander wellbeing framework. Alice
                                        • Springs: Ninti One. Taylor, J., Doran, B., Parriman, M., & Yu, E. (2014). Statistics for community governance: the Yawuru Indigenous population survey, Western Australia. International Indigenous Policy Journal, 5(2).
                                      • Ethics, privacy and confidentiality
                                          • National Health and Medical Research Council (Australia). (2003). Values and ethics: Guidelines for ethical conduct in Aboriginal and Torres Strait Islander health research. The Council.
                                          • Grove, N., Brough, M., Canuto, C., & Dobson, A. (2003). Aboriginal and Torres Strait Islander health research and the conduct of longitudinal studies: issues for debate. Australian and New Zealand journal of public health, 27(6), 637-641.
                                          • AIATSIS Guidelines for Ethical Research in Australian Indigenous Studies 2012

                                        20. Questions?


                                        Need some help?

                                        CFCA offers a free research and information helpdesk for child, family and community welfare practitioners, service providers, researchers and policy makers through the CFCA News.