Evaluation methodology: Family support services

The latest material added to the Australian Institute of Family Studies library database is displayed, up to a maximum of 30 items. Where available online, a link to the document is provided. Many items can be borrowed from the Institute's library via the Interlibrary loan system.

See more resources on Evaluation methodology: Family support services in the AIFS library catalogue

How to develop a program evaluation plan

Child Family Community Australia
Melbourne, Vic. : Child Family Community Australia, 2017.
This resource has been designed specifically for Communities for Children service providers to assist them in meeting the 50% evidence-based program requirement, but may be used by anyone who wants to evaluate the outcomes of a program. It provides advice on how to develop an evaluation plan to measure what impact a program has had on participants. Sections include: are there different types of evaluation?; developing an evaluation plan for an outcome evaluation; identifying evaluation purpose and audience; deciding on evaluation design; selecting outcomes; evaluation questions and indicators; types of data; data collection methods; data collection quality; sampling; ethics; data analysis; and writing up the evaluation and disseminating the findings. Templates and links to additional resources are included.

Measuring outcomes in programs for Aboriginal and/or Torres Strait Islander families and communities.

Barnes S and Brosnan K
15 March 2017
This webinar will discuss ways to measure the outcomes of programs for Aboriginal and/or Torres Strait Islander families and communities. Drawing on past evaluations, practice wisdom and lessons learnt in the field, this webinar will encourage professionals who are thinking about evaluating the outcomes of a program for Aboriginal and/or Torres Strait Islander families or communities to consider evidence in a different light. The authors have also published a practitioner resource on this topic: 'Evaluating the outcomes of programs for Indigenous families and communities.'

How to develop a program logic for planning and evaluation

Child Family Community Australia
Melbourne, Vic. : Child Family Community Australia, 2016.
Program logic models show how a program works and draw out the relationships between resources, activities and outcomes - it is both a tool for program planning and also for evaluation. This resource document summarises what program logic is, its use, and how a program logic model can be developed. It features a checklist for development, links to additional resources, and templates and examples. This resource has been designed for service providers under the Communities for Children program, but may be used by anyone who is interested in developing a program logic.

Successful NGO evaluation cultures: literature scan

Bailey R, McKegg K, Wehipeihana N and Moss M
Wellington N.Z. : Superu, 2016.
The 'Using Evidence for Impact' project aims to inspire all those working in the New Zealand social sector to use evidence in decision-making. This report reviews the international and New Zealand evidence on what enables non-government organisations (NGOs) to build their evaluation capacity successfully. It investigates the key infrastructure elements that support a successful evaluation culture within an NGO setting, reviews evaluation capacity assessment and measurement tools, and considers capacity development for Maori and Pacific NGO providers.

Building your organisation's ability to do evaluations

SuPERU (New Zealand)
Wellington N.Z. : Superu, 2016.
This resource aims to help non-government organisations to build their capacity for evaluation. Their are 4 sections. Part 1 is an introduction, Parts 2 and 3 are tools for Evaluation Capacity Assessment and Evaluation Capacity Planning, and Part 4 provides some advice on implementing a plan. The website also features documents on how these tools were developed and the supporting evidence.

Building a better outcomes framework for families : a story from the Mallee.

Harley F and Myer J
19 Jul 2016
This webinar will tell the story of the relationship between Mallee Family Care (MFC) and Social Ventures Australia (SVA), and how they worked in partnership with key agency and community stakeholders to develop outcome measures. MFC is a Facilitating Partner for the Communities for Children (CfC) activity and provider for the Children and Parenting Support (CaPS) activity. After recognising that their approach to measuring outcomes for families required a radical change, MFC developed an innovative and collaborative approach to engaging both CfC Community Partners and CaPS staff to establish good practice in evaluating the success of their service delivery. This webinar will take participants on MFC's journey with SVA in challenging the thinking of staff from the 'doing' to the 'measuring'. It will outline the actions taken to build rapport and trust with stakeholders, ensuring they had a clear understanding of how and why the focus shifted from measuring outputs to measuring outcomes. This partnership was facilitated by the Families and Children Activity Industry List, and this webinar is part of that project's dissemination activities.

Participatory action research

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2015.
Participatory action research (PAR) is an approach to research that aims to situate power within the research process with those who are most affected by a program - the intention is that the participant is an equal partner with the researcher. This resource provides an introduction to participatory action research, and is one of a series on evaluation written for practitioners. Sections include: What is participatory action research?; The principles of PAR; Ownership, commitment and responsibility through participation; When should PAR be used?; Benefits, limitations, and appropriateness of PAR; The PAR process; The role of the evaluator; How is PAR done?; and Further information.

Empowerment evaluation

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2015.
Empowerment evaluation aims to create a sense of ownership to ensure program outcomes will actually be beneficial for participants, and is more a set of principles that guide the evaluation at every stage rather than a research method. This resource provides an introduction to empowerment evaluation, and is one of a series on evaluation written for practitioners. Sections include: Principles underlying empowerment evaluation; Similarities with other participatory evaluation methods; When should empowerment evaluation be used?; Strengths and criticisms; The empowerment evaluation process; The role of the evaluator; Impact of empowerment evaluation on stakeholders; and Further resources.

Approaches to evaluation in Australian child and family welfare organizations.

McCoy A, Rose D and Connolly M
Evaluation and Program Planning v. 44 Jun 2014: 68-74
The role of evaluation in child and family welfare organizations around the world is being strengthened by the rising demand for accountability and accountability in the nonprofit sector. This article provides an insight into some of the issues involved. It describes the Australian context for child and family welfare organizations and reviews the benefits and challenges of different evaluation approaches available to this sector. Sections include: Child and family welfare organizations: the Australian context; A brief history of Australian child and family welfare organizations engagement with evaluation; Approaches to evaluation in child and family welfare organizations; Evaluation use and organizational approaches to evaluation for child and family welfare organizations; and Considerations for organizations when choosing an evaluation approach.

Considerations in applying benefit-cost analysis to preventive interventions for children, youth, and families: workshop summary

Olson S and Bogard K
Washington, DC : The National Academies Press, 2014.
"Benefit-cost analyses hold great promise for influencing policies related to children, youth, and families. By comparing the costs of preventive interventions with the long-term benefits of those interventions, benefit-cost analysis provides a tool for determining what kinds of investments have the greatest potential to reduce the physical, mental, and behavioral health problems of young people ... [This book] is the summary of a workshop convened by the Board on Children, Youth, and Families of the Institute of Medicine and the National Research Council in November 2013 as the first phase of a possible two-part effort directed toward guiding future benefit-cost studies and enhancing the relevance of benefit-cost analysis to governments and other organizations wanting to make sound prevention decisions. The workshop brought together leading practitioners in the field, researchers who study the methodological and analytic dimensions of benefit-cost analysis, and representatives of organizations that use the results of benefit-cost analyses to shape and implement public policies. This report discusses a wide range of issues about benefit-cost analysis, including the level of research rigor that should be met before results from an evaluation are used to estimate or predict outcomes in a cost-benefit analysis; best practices and methodologies for costing prevention interventions; prevention outcomes that currently lend themselves to monetization; processes and methodologies that should be used when linking prevention outcomes to avoided costs or increased revenues; and best methods for handling risk and uncertainty in estimates."

Creative clinical data mining : glimpsing the engagement of families with an integrated service system.

Earles W
Evaluation Journal of Australasia v. 13 no. 1 2012: 28-35
This article illustrates the potential of clinical data mining (CDM) for exploring families' engagement with integrated early intervention and prevention (EIP) service systems. New approaches to EIP under the Communities for Children program seek to achieve systemic change towards more integrated service provision in an area. To date, program monitoring has focused on contacts by families with individual services, not necessarily engagement across an emerging system. As part of an evaluative capacity-building (ECB) process, practitioners asked the question: What can we learn about users' engagement with the service system from existing records of use? The approach taken drew from CDM methodology. EIP does not usually involve significant user database systems as the work entails community development methodologies and users can be variously the community, groups, families and individuals. A partial dataset, from routinely recorded contacts, was available for one area. The dataset relied on a family number system created as a membership club for consciousness-raising on child-friendly communities. The de-identified dataset was mined for 'connection' with, and 'movement' within, the service system. There was evidence beyond the intuitive knowing of practitioners that the contractual obligation to engage families and children with services across a system was achieved. CDM provided a means to demonstrate this within a broader evaluative capacity-building process.

Dissemination of evaluation findings

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2013.
This paper explores a valuable aspect of program evaluation - disseminating your findings to the broader sector. Regardless of whether an evaluation revealed that a program had a positive, negative, or little measurable impact on participants, the findings should ideally be communicated to the broader sector so that the evidence base is expanded and future program participants can benefit from what was found. The paper provides advice on avenues of dissemination, reporting findings, writing reports, publishing articles, and presentations. Companion resource sheets are also available on other aspects of program evaluation, including: the basic terms and principles of evaluation; innovation; and getting an evaluation plan under way and keeping it on track.

Planning for evaluation. II, Getting into detail

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2013.
The first paper introduced the basic principles of planning the evaluation of a program: this second paper explores the simple and minor issues and tasks that can quickly derail an evaluation if not addressed early in the process. By thinking through the why, who, what, when, where and how of conducting an evaluation, many of the possible sources of derailment can be avoided. Resource sheets are also available on other areas of program evaluation, including: broader issues relating to evidence and ethics in evaluation; the basic terms and principles of evaluation; getting an evaluation plan under way and keeping it on track; and what happens when the evaluation is complete.

Planning for evaluation. I, Basic principles

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2013.
This paper introduces the basic principles of planning the evaluation of a program. It broadly describes: writing program objectives; program logic models; evaluation designs; and methods of data collection. A companion paper will look into these aspects in more detail. Resource sheets are also available on other areas of program evaluation, including: broader issues relating to evidence and ethics in evaluation; the basic terms and principles of evaluation; getting an evaluation plan under way and keeping it on track; and what happens when the evaluation is complete.

Evidence-based practice and service-based evaluation

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2013.
The call for services to ensure that their programs and practices are informed by, if not grounded in, well-conducted, relevant, scientific research evidence has grown louder in recent years. Written for social service practitioners, this paper introduces evidence-based practice and service-based evaluation. Part 1 explains types of evidence and different types of evaluation studies, while part 2 looks at service-based program evaluation, including in-house and external evaluations. Part 3 discusses developing a culture of evaluation, then part 4 reviews issues of ethical conduct, consent, and the protection of privacy and confidentiality. Companion resource sheets are also available on other aspects of program evaluation, including: the basic terms and principles of evaluation; innovation; getting an evaluation plan under way and keeping it on track; and what happens when the evaluation is complete.

Evaluation and innovation in family support services

Child Family Community Australia, Australian Institute of Family Studies
Melbourne, Vic. : Australian Institute of Family Studies, 2013.
This paper provides a brief overview of evaluation and innovation in the context of family support services. Companion resource sheets are also available on other aspects of program evaluation, including: broader issues relating to evidence and ethics in evaluation; the basic terms and principles of evaluation; getting an evaluation plan under way and keeping it on track; and what happens when the evaluation is complete.

Evaluation of the ESF/DWP families with multiple problems/troubled families initiative: a feasibility study

Morris S
Sheffield U.K. : Dept. for Work and Pensions, 2012.
"This short feasibility study considers approaches to evaluating the Department for Work and Pensions (DWP) Families with Multiple Problems/Troubled Families programme. The current government is delivering a wide range of services and support to families in need. The DWP Families with Multiple Problems/Troubled Families programme is one of a number of such schemes available to families facing multiple problems, and is a key component of the Government's wider Troubled Families strategy. It is a complex programme, funded through the European Social Fund (ESF) and delivered on a regional basis through private/not-for-profit organisations on a prime provider/subcontractor basis."--End page.

Assessing practice in a child and family centre in Australia.

McNamara P
Maluccio, Anthony N, ed. Improving outcomes for children and families : finding and using international evidence. London : Jessica Kingsley Publishers, 2011. 9781849058193: 192-203
This book explores the challenges of designing and undertaking outcome-based evaluation of child and family services. This chapter describes the methodology and findings of a study into the 'sensitive indicators' of long-term outcomes in community-based outcomes - in other terms, "what is the difference that makes the difference?" The study was conducted at Berry Street, a non-government agency providing child and family services in Melbourne, Victoria.

Investing in the early years : issues for today's economic times.

Gray A
Wellington N.Z. : Families Commission, 2011.
This paper reviews the effectiveness of early intervention programs and services for young children. Drawing on the New Zealand and international literature, it examines what works in early intervention, factors for success, characteristics of high-quality programs, identifying and engaging vulnerable families, and evaluation and cost-effectiveness considerations.

Collecting data from parents and children for the purpose of evaluation: issues for child and family services in disadvantaged communities

McDonald M and Rosier K
Melbourne, Vic. : Australian Institute of Family Studies, 2011.
Many family services will be required to collect data from children or parents for the purposes of evaluation. This practice sheet outlines both the challenges involved in data collection and also the benefits for services and families. The practice sheet also describes methodological considerations in data collection and provides examples on engaging children and parents in the evaluation process.

Evaluation in family support services

Parker R
Melbourne, Vic. : Australian Institute of Family Studies, 2010.
This resource aims to encourage the use of evaluation among family support service providers and practitioners. It features five separate sections on different aspects of evaluation, including: evaluation and innovation; evidence-based practice, service-based evaluation, and ethics; basic principles in preparing for evaluation; getting started in evaluation; and dissemination and beyond. This resource provides introductory guidance and support, and addresses issues arising in the family support sector.

Stronger Families in Australia study : the impact of Communities for Children : Stronger Families and Communities Strategy 2004-2009

Edwards B, Wise S, Gray M, Hayes A, Katz I, Misson S, Patulny R and Muir K
Canberra : Dept. of Families, Housing, Community Services and Indigenous Affairs, 2009.
This report examines the impact of the Communities for Children (CfC) program on child, family, and community outcomes. This study was conducted as part of the 2004-2008 evaluation of the Federal Government's 'Stronger Families and Communities Strategy 2004-2009'. The CfC program funded non-government organisations serving 45 disadvantaged communities in Australia, to develop and implement a whole-of-community approach to enhancing early childhood development. Four priority outcomes were measured: healthy young families; supporting families and parents; early learning and care; and child-friendly communities. The analysis included a longitudinal study of 2,202 families. Positive impacts include fewer children living in a jobless household, parents reporting less hostile or harsh parenting practices, and parents feeling more effective. Positive impacts were also found for hard-to-reach groups. Some negative findings were reported for the health outcomes of hard-to-reach, low education and low income groups. These results suggest that the CfC model makes an important contribution to the wellbeing of disadvantaged children.

Knowing what you do works: measuring your own effectiveness with families, parents and children : a short guide

Rhodes H
London : Family and Parenting Institute, 2009.
"'How effective are we as practitioners?' is a question to which we all want an answer. Understanding what, when and how to measure the impact of individual pieces of work with families, parents and children is of significant benefit. This short guide is a practical resource that includes the following topics: pre- and post-measuring; reliability and validity; finding the right tools for the job; ethical and practical issues; information technology and measurement tools; helping families to answer the questions; what not to do. It is designed to help individual workers and whole services understand the terms and issues within the act of measuring and, better still, how to go about it. It is a companion guide to 'Supervising family and parenting workers: a short guide' and 'How to help families in trouble: a short guide'."

Positive family functioning: scoping study

Access Economics (Firm), Australia. Dept. of Families, Housing, Community Services and Indigenous Affairs
Canberra, A.C.T. : Dept. of Families, Housing, Community Services and Indigenous Affairs, 2009
This study was commissioned to assess the feasibility of, and the methodology for, undertaking a research project on the economic benefits of early intervention. The project would aim to quantify the value of goods and services provided by positive family functioning and the returns to government and society for investments made in supporting family functioning. This report presents the findings of the scoping study, and discusses methods for measuring positive family functioning, methods for measuring the efficiency of interventions, and the use of outcomes contained Australian longitudinal data sets.

Evaluation of the Stronger Families and Communities Strategy 2000-2004: evidence-based policy and practice : issues paper

Rogers P, Williams B and Stevens K
Melbourne, Vic. : RMIT University CIRCLE, 2008.
Key ideas in evidence-based policy and practice are set out and evaluated in this paper, and different approaches are discussed. The six key activities are: retrieving or generating evidence; validating evidence; synthesising evidence; communicating evidence; using evidence to guide policy and practice; and, further contributing to the evidence base. Particular approaches to evidence-based policy and practice undertake these activities in different ways. Four approaches are described: statistical analysis of experimental and quasi-experimental research studies; synthesis of diverse evidence; literature review of demonstrably effective projects or policies; and, corporate and community memory. The report argues that effective evidence-based policy and practice draws on an appropriate mix of these different approaches. Where there is agreement on outcomes and certainty about how these can be achieved, a prescriptive approach is appropriate; where there is not, a more adaptive and responsive approach is more appropriate.

A framework for the evaluation of the family law reform package

Australian Institute of Family Studies, Australia. Attorney-General's Dept., Australia. Dept. of Families, Community Services and Indigenous Affairs
Canberra, A.C.T. : Attorney-General's Dept., 2007.
The Commonwealth government's family law reform package largely implements the recommendations made in the 2003 Parliamentary report 'Every picture tells a story', with legislative changes to promote shared parental responsibility after separation, the addition of new services such as the Family Relationship Centres, and an expansion of existing early intervention services. The Attorney-General's Department and the Department of Families, Community Services and Indigenous Affairs have joint responsibility for the family law reform package. This evaluation framework was developed at their request, by the Australian Institute of Family Studies, to assess how well the new and expanded services meet the policy and legislative objectives, and the impact of the reforms on Australian families. It discusses the framework for evaluating the entire reforms as well as individual components.

Evaluation plan: Early Intervention Program.

Fisher K, Everingham S, Katz I, Sawrikar P, Parker S, Van Gool K, Haas M and Johnston C
Sydney, NSW : Social Policy Research Centre, University of New South Wales, 2006
The NSW Department of Community Services commissioned this plan for the evaluation of its Early Intervention Program (EIP). The EIP provides support services for low to medium risk families with problems that affect their ability to care for their children. The evaluation plan sets out background information about the EIP, the evaluation framework and the proposed evaluation management.

Family support as reflective practice

Dolan P, Canavan J and Pinkerton J
London : Jessica Kingsley Publishers, 2006.

Incorporating Indigenist methodology into the evaluation of DOCS' Early Intervention Program.

Parker S
SPRC Newsletter no. 94 Nov 2006 7-8,20
The NSW Department of Community Services' Early Intervention Program, for low to medium risk families, is expected to be of benefit to a number of Indigenous families. Two Indigenous communities will take part in a four year evaluation of the program. This article discusses Indigenous access to mainstream programs, and culturally appropriate design of evaluation research for Indigenous communities.

Family Relationship Services Program: developing and implementing a performance framework: performance framework 2005-2008.

Australia. Department of Families, Community Services and Indigenous Affairs
Canberra, ACT : Department of Families, Community Services and Indigenous Affairs, 2005
The Family Relationship Services Program (FRSP) was reviewed in October 2003. A performance framework discussion paper was subsequently circulated, a revised version of which is presented in this paper. This paper outlines the purpose and key elements of the performance framework, which are research questions, target groups, principles, outcomes, outputs, processes and inputs. Progress made in relation to other elements, which will be further developed over the next year, is also outlined.
Subscribe to