My one word is "Data". The collection and application of data will have a massive effect on service delivery in social services and indeed in most other areas in government …

The Honorable Christian Porter, MP, Minister for Social Services. Speech to the Family and Relationship Services Australia Senior Executive Forum, 24 February 2016, Parliament House, Canberra

There is an increasing policy and practice focus on the need to better understand the effects of social services on families. Governments across Australia are increasingly making data collection and evaluation part of their conditions for funding policies, programs and individual services. This has led to a new national conversation in the child and family welfare sector about the role of evidence-based practices and programs in the delivery of services to families. As the Social Services Minister stated:

In the past the determination of what expenditure and effort in this area has achieved has been guided essentially by instinct. The reality has been that our efforts to measure our performance have been limited to two substantive criteria: reach and satisfaction with service … Such measures are not completely unhelpful but in reality they fail to go to the core question that everyone wants to know: what was the outcome?

(The Honourable Christian Porter, MP, Minister for Social Services. Speech to the Family and Relationship Services Australia Senior Executive Forum, 24 February 2016, Parliament House, Canberra)

Two recent government initiatives exemplify this trend. Within the family services area, the Australian Government's Department of Social Services (DSS) requires 30% of all funding for Communities for Children programs be "evidence-based", with this requirement increasing to 50% in mid 2017. While this might seem a modest target, there are relatively few programs operating within the government and community sectors that have strong evidence of their effectiveness. Those services that do have good evidence, from rigorous research or evaluations, are mostly focused on single issues such as parenting skills rather than complex social services.

Governments are also increasingly concerned that social policy spending delivers benefits both in the immediate and longer term. This concern is reflected in several new policy initiatives. The Australian Investment Approach, for example, is seeking to use actuarial valuations to identify risk factors driving long-term welfare dependency and then to quantify the benefits of "investments" in interventions with specific cohorts. This actuarial modelling approach introduces the need to measure the immediate benefits of a program and recipients' particular needs as well as the longer-term benefits (and hence potential lower future costs to government) of the intervention.

The contest over "evidence"

There are increasing calls within the research and policy literature in both Australia and overseas for better understanding of "what works" in social policies and programs - including child and family policies and programs (Flatau, Zaretsky, Adams, Horton, & Smith, 2015; Fox et al., 2015; New Zealand Productivity Commission, 2015; Organisation for Economic Cooperation and Development [OECD], 2015).

The calls for better evidence about program outcomes occur, in part, within the context of a perceived scarcity of resources and a subsequent desire for social spending to provide better value for money. Desires for a better understanding of "what works" also resonate with current debates about the need for major service system reforms and with a growing literature about the need for service systems to change if they are to better support vulnerable families (e.g., see Fox et al., 2015 and OECD, 2015). Current service systems have been described as fragmented and poorly coordinated within and across service systems, program-focused rather than client-focused, and as giving more attention to symptoms (e.g. crisis support) rather than anticipating and preventing problems (Fox et al., 2015). Social services have also sometimes been described as having a limited capacity for supporting families with complex problems because they are too often focused on single issues and because there is little incentive to innovate (New Zealand Productivity Commission, 2015; Shonkoff & Fisher, 2013).

Calls for change in the social service system are also sometimes coupled with the observation that there is relatively little evidence about what services, programs or polices would actually improve the lives of families (Fox et al., 2015; New Zealand Productivity Commission, 2015; OECD, 2015). There appears to be a particularly limited evidence base for many of the services and programs already being implemented. This perceived lack of evidence has led to calls for "shared outcome frameworks" that services can use to collect more rigorous and more consistent evidence of their effectiveness.

However, what counts as "evidence", and how evidence can best be collected, is a matter of ongoing debate and these debates can take on political, economic and epistemological dimensions (Meagher, 2002; Featherstone, White, & Morris, 2014). The fundamental questions that underpin this ongoing debate among key stakeholders - including researchers, policy-makers, service providers and communities - is what should be measured, by whom and in what way? As Featherstone and colleagues (2014) argued in regard to research around the effectiveness of child welfare approaches, "our ideas about it, and how it should be properly conducted, constitute sites of contestation and struggle with very high stakes. Research and what counts as valid knowledge are political matters" (p. 54). For example, randomised control trials (RCTs) are often framed as the answer to the evidence debate, as they are viewed as the most "rigorous" methodological approach (Fox et al., 2015). However, as this edition of Family Matters outlines, it may not be the case that RCTs are always necessary to understand if programs are working, nor are they always feasible or ethical.

Keeping this in mind, it should be remembered that discussions about performance measures, evaluations and other forms of research are not necessarily neutral and objective, but rather "embody and convey political and institutional values and constraints, and reflect the power inequalities that underpin the formation of government policy" (Cortis, 2007, p. 400). Therefore, to fully understand the evidence presented about what is working and not working, it is important to be clear about the context in which evidence was produced - for example, by whom, at whose request, and how (Cortis, 2007; Featherstone et al., 2014; Meagher, 2002).

What does the family services sector say?

Since the AIFS and FRSA conferences last year, the discussion on evidence with government and the child and family welfare sector has continued. In February this year, the Institute held the first of a series of roundtable conversations to be conducted across Australia that explore both the opportunities and challenges facing the sector around evidence and evaluation.

The roundtable was attended by managers and executive leaders of organisations funded through the Families and Children Activity as well as DSS policy, the department's Data Exchange (DEX) and NSW state office staff. The roundtable encouraged a dialogue to explore the ways that both service providers and policy-makers are looking to demonstrate the effectiveness of services to families, how far they have come and the challenges they have faced.

Participants in the roundtable spoke about a strong desire to demonstrate that the services they are providing or commissioning are effective. There was a strong investment in improving data collection systems and tools. However, there were also concerns about the need to balance the collection of high quality data with not over-surveying clients who may attend multiple services across the sector and be invited to complete client satisfaction or outcome surveys multiple times.

Further development of a shared understanding of the purpose of collecting data and of how it would be used was seen as an important next stage both within services and between services and funding bodies. Building capacity to ensure that the data that was collected was both meaningful and useful was also seen as crucial within the current agenda. This includes developing workforce capacity across the sector to both collect and interpret data. Additionally, there was agreement among participants that a key question to answer in to the future was how best to use data to make a difference.

There was an uncertainty about what was "good enough" evidence and about what types of data were appropriate in what context. Reflecting the current debates within the literature, there was also a lack of clarity about the context in which different levels of evidence were required and the best ways to approach evaluating programs. This was particularly important in the context of few resources and little time - and in a sector where different services are at very different places along their "evidence journey". The resourcing and capacity available for evaluating programs and practices varies greatly across services, and deciding where to concentrate efforts to understanding what works - and even where to start - was a daunting proposition for some providers.

Where do we go from here?

It is clear that there is strong agreement across services, government and researchers that evidence is important. Both services and government agencies are actively working towards building capacity to better understand what is working for families and what further supports they may need. This is an important and valuable development and an exciting project to be part of.

However, the nuances in the debates about what evidence is, and the political context of how evidence is used, should not be lost. Taking a critical view of what the evidence is, how it is produced and what it means for families is the key to ensuring that policy and practice in Australia continues to meet the needs of families. The reality is that effectively measuring service systems and the individual services requires reflecting on, and taking account of, the complex problems that families can experience and the complex systems that policies and services are implemented within.

References

• Allen, G. (2011). Early intervention: Smart investment, massive savings. The Second Independent Report to Her Majesty's Government. London: UK Government.
• Cortis, N. (2007). What do service users think of evaluation? Evidence from family support. Child and Family Social Work, 12, 399-408.
• Featherstone, B., White, S., & Morris, K. (2014). Re-imagining child protection: Towards humane social work with families. Bristol: Policy Press.
• Flatau P., Zaretzky, K., Adams, S., Horton, A., & Smith, J. (2015). Measuring outcomes for impact in the community sector in Western Australia (Bankwest Foundation Social Impact Series No. 1) Western Australia: Bankwest Foundation.
• Fox, S., Southwell, A., Stafford, N., Goodhue, R., Jackson, D., & Smith, C. (2015). Better systems, better chances: A review of research and practice for prevention and early intervention. Canberra: Australian Research Alliance for Children and Youth (ARACY).
• Meagher, G. (2002). The politics of knowledge in social service evaluation (Discussion Paper No. 1). North Parramatta: Uniting Care Burnside,.
• New Zealand Productivity Commission. (2015). More effective social services: Draft report. Wellington: New Zealand Productivity Commission.
• Organisation for Economic Cooperation and Development (OECD). (2015). Integrating social services for vulnerable groups: Bridging sectors for better service delivery. Paris: OECD Publishing. Retrieved from <dx.doi.org/10.1787/9789264233775-en>.
• Shonkoff, J. & Fisher, P. (2013). Rethinking evidence-based practice and two-generation programs to create the future of early childhood policy. Development and Psychopathology, 25(4 pt 2), 1635-1653.

Kelly Hand is a Senior Research Fellow and Acting Deputy Director - Research, at the Australian Institute of Family Studies