Karen Field, Chief Executive Officer

Drummond Street Services

If granted a miracle, Drummond Street Services CEO Karen Field would like to know more about the lives of her organisation's vulnerable clients.

"The biggest gap in our understanding is the changing nature of the contexts of people's lives," she says.

Drummond Street Services deliver community-based services for children and families, including counselling and programs for vulnerable families, refugees, and LQBTI and youth communities.

"We also don't always understand how much of an 'intervention dose' is enough for people with 'clusters' of risk factors or how many services people can fit into their lives."

Field says this despite her organisation having an already advanced client information system allowing caseworkers to identify, track and target the changing needs of disadvantaged families and children.

"We use population data of local communities to help inform our programs and service planning, to really understand who is in our community and what their specific needs are from our programs and services. In addition, the collection of family-level data is essential in delivering Drummond Street's evidence informed interventions to individual clients and families."

Established in 2008, Drummond Street's client record system involves a centralised intake and assessment process allowing workers to prioritise needy clients and allocate services based on needs. It contrasts to its older way of working where staff would match individuals with single-focus clinicians based only on the client's presenting issue.

"We now think of it as a public health opportunity to screen for multiple health risks, then target them with goal oriented responses - we have the tools to select the right interventions," she says.

Drummond Street also conducts public health risk assessments across multiple health and wellbeing domains to ensure it can tailor evidence-informed interventions to meet the need of individual families.

"Screening maximises the opportunity to identify and ameliorate problems early and to ensure tertiary responses are timely and effective for the whole family."

The 130-year-old organisation has always had the dual purpose of delivering services and researching ways to improve the lives of disadvantaged people. Drummond Street Services currently has a significant research team consisting of a senior researcher, research assistants and consultants.

"The researchers aim to continue to improve services to be as responsive to client needs as possible," Field says. "The days of families just coming with one issue are long gone. We see families with multiple risk issues and research allows us to target those issues but also identify issues early."

And while Field considers the social services sector's move to evidence-informed programs as encouraging, she believes that services should be guided by community and client needs.

"Multiple risk assessments require multiple interventions and often a single-issue, evidence-based program does not fully meet the needs of an individual family or client. Evidence-informed therapeutic assessment and goal-oriented interventions ensure the right programs are matched and 'wrap around a family'," Field says. "We are committed to using the best evidence to ensure the right mix of interventions to respond to dynamic risk - that is the changing needs of clients."

Drummond Street's evidence-informed approaches assist its clinicians to respond to dynamic risk and include case management, outreach, home visiting and matching the right intervention "dose" to families' needs.

"The problem with stand-alone, off the shelf programs can be adapting them while still making sound judgments based on clinical assessment tools and consultation with practitioners and families.

In real life, when client demand for services is resulting in long waiting lists, using evidence-informed processes to make decisions about adapting the intensity of interventions is equally as important as deciding which programs should be used for each client."

Field is proud of Drummond Street's clinicians' openness to this approach.

"Workers see the value in these programs, but they know they have the flexibility to respond to risk as it arises. Given our strong assessment and intervention matching processes at the client intake stage, staff make good decisions about when families are in the best position to make the most out of evidence-based programs."

Despite this, Field still sees challenges in evaluating programs within community services organisations.

"Getting good data is tricky because clients do not always follow a set plan. For example, clients find it difficult to juggle services in their busy lives, particularly as their lives begin to feel better through interventions.

Clients see improvement in their lives as enough 'right now' and it can be a challenge for planned completion and follow-up for post-outcome surveys. There are issues with program evaluations within community services organisations just because of the nature of our client's lives," she says.

The way Drummond Street has tried to resolve this is by making data collection easier for staff through online pre- and post-surveys that can be accessed anywhere with clients.

While Field welcomes the government's aim of funding evidence-informed programs, she also thinks this requires a planned capacity-building approach with each organisation, assessing their capacity and readiness to adopt evidence-informed practice.

"It's a mistake to jump straight to evidence-based programs without strong evidence-informed processes to assess and match each client to the programs they need. This is where the scale-up of evidence-based programs often fails. We need to look at each organisation's prior readiness to change to this approach - some are only starting," she says. "It's like we have just jumped to the other end of the spectrum. People need to get used to collecting and using data to identify client needs and to inform programs. It's going to take organisations time to know how to do that well."

Instead, Field believes the focus should be on using data and engagement to understand communities, before being expected to implement and evaluate evidence-based programs matched to community needs.

"There a lot of evidence-based programs out there and it's confusing - our starting point should be to better understand the communities and families we work with and getting staff inspired about using evidence to address what we know about our clients. We also need to better acknowledge the increasing complexity of vulnerable families with multiple health risks, and therefore the need for matching multiple risk interventions."

As always, a key driver for Field is allowing services to make the best evidence-informed decisions for each client.

"The biggest learning we have had is to have evidence-gathering processes to allow good decisions, and in integrating the tools, practices and programs to make those decisions," she says. "This has meant having a sophisticated model of practice as a starting point rather than solely being driven by funding, which often takes a single-issue approach to responding to family needs."

Wendy Field, Head of Policy and Programs

The Smith Family

Manages:

• All The Smith Family programs and volunteering initiatives
• Learning for Life Program
• Communities for Children Facilitating Partner

Even though the Smith Family's Wendy Field manages a successful program monitoring team, she is still searching for the "sweet spot" between implementing evidence-based programs and adapting them to the complex needs of disadvantaged families.

"The challenge is the context - the ability to drill down to the core elements of an evidence-based program and then working through what are the bits we can tweak," she says. "There is a constant push and pull around how programs get delivered. It's important to stay open to what our teams are telling us are lessons happening on the ground."

As Head of Policy and Programs at the Smith Family (based in its Sydney headquarters), Field oversees the organisation's policy development and program design and implementation. To Field, good evidence is central to providing quality programs and ensuring clarity.

"Evidence underpins the process you use to develop, implement and continuously improve programs. When we are considering how to address an issue, we will look to what the literature says, then develop a theory of change based on literature, our own practice and input from on the ground staff. Then we trial that approach."

The Smith Family has a small dedicated Research and Advocacy team, led by Anne Hampshire, as well as program design and implementation support teams. One feature of their work is that a policy analyst works with each program to assist with implementation and evaluation.

"Each analyst works with local teams and provides program guidelines, planning support and tools. They support teams on the ground, but are also a driver of quality assurance in implementation."

Despite this, there is tension in keeping "faithful" to evidence-based programs, both in delivering and evaluating them. Field highlights the challenge of maintaining an effective "feedback loop" between practitioners and researchers.

"It requires being really clear with on the ground staff about why particular components of the program are important and getting their buy in. Communication is vital and it's a constant challenge in busy environments. What can happen is all the data that workers collect goes into a black hole and by the time it comes out workers are already onto the next tranche of the program. We are trying to speed up the feedback to staff and to give more targeted information about the impact of programs in their communities."

The Smith Family is currently working on the development of an organisation-wide 'Enterprise Information Program', which will enable more real-time data feedback to practitioners.

While she believes evidence-based programs need to be balanced with on the ground needs, Field is adamant about the value of proven programs. One way the Smith Family retains 'fidelity' to evidence-based programs is through a 'program quality framework' that stipulates what program elements are sacrosanct and what elements can be adapted by practitioners to local circumstances.

"We are duty bound to ensure that interventions with vulnerable families are based on what we know can make a difference for them. However, there are different understandings of what makes a difference in practice. In that sense, I don't think that we can say that the evidence base is complete - it's a work in progress, and we need to stay open to different ways of working that update the existing evidence."

Field uses the example of the Smith Family adapting the evidence based Let's Count numeracy program to suit local needs. Run by the Smith Family since 2010, the program is traditionally conducted by specifically trained educators in early childcare centres with disadvantaged communities. But based on recent feedback from practitioners on the ground, new implementation models will be trialled by the Smith Family in other locations (such as libraries).

Field also warns the policy preference for evidence-based programs needs to be weighed against the need to have flexible and localised responses to clients with multiple issues.

"There is a strong tension between place-based and evidence-based approaches. The risk is evidence-based programs can be siloed, and developing a whole community approach is really key. The issues facing families aren't nicely cut - they are so complex and interconnected."

Despite these challenges, Field remains focused on clients.

"We need to get the balance right across localised responsiveness and the integration of programs," she says. But ultimately we keep our work focused on the families and the kids we are trying to support - that is the central guiding thing around everything that we do."

Zoe Upson, Communities for Children Program Manager

Amity Health, Great Southern Region

When asked whether Amity Health has a dedicated research team, Program Manager Zoe Upson is brief.

"We are a small organisation and don't have that dedicated in-house evaluation support - so research has to be part of everyone's job. Staff are a jack of all trades," she says proudly.

And despite Amity Health not having specialist researchers, the organisation still successfully delivers and evaluates its programs by supporting staff to do the job.

"Our staff come from a variety of backgrounds and we bring different areas of expertise. In the area of evaluation, we like to upskill staff where we can."

Based in Albany, Amity Health is a not-for-profit organisation delivering various allied health and community services for children and adults. It has delivered services in the Great Southern region of Western Australia (located within two hours of Albany) for more than 20 years. Amity Health is also the Communities for Children Facilitating Partner for the region.

"Amity Health may not have the same resources as other larger community service organisations, but its commitment to delivering programs based on evidence is strong," Upson says. "Evidence is very important and we definitely use evidence-based practice. We have a lot of government-funded programs and we like to know what we are implementing is achieving the outcomes of the program and for the community."

This commitment is reflected in the way the organisation understands their local communities and how they select programs to address their needs.

"We look at the needs of the community first by checking demographic datasets and then at the outcomes of evidence-based programs to see if they match up to people's needs," she says. "We also talk to local providers to source what is needed in that community. Often local champions know what is needed, but you also need the evidence to back it up."

Amity Health also has reporting systems to monitor programs every 3-6 months to see if changes are needed. Upson is proud of staff adapting to the rigorous task of data collection and reporting.

However, as a small organisation, there are times where outside support is needed. Upson recalls Amity Health's support from Drummond Street Services to assist with the final evaluations of its programs.

"The majority of the time we have enough in-house expertise to satisfy our funder, but where we are lacking we will source help from the sector," she says. "It's great to have another organisation's critical eye - sometimes when you are so embedded in your practice it's hard to take that step back."

The push for more evidence-based programs within community service organisations is something Amity Health is committed to as well.

"We must be accountable for a large amount of money," Upson says. "But you also need the flexibility to start new programs and innovate practices."

Upson believes this ability to remain flexible in service delivery is important because of the nature of the diverse regional communities they serve.

"It is tricky in a rural and regional area, maybe only a few evidence-based programs are suitable for our areas. You need to look at client needs and whether they match the outcomes of the programs," she says. "We try to pick programs with a universal approach that will support the needs of more people, particularly in small communities."

So what does the future hold for Amity Health given the Communities for Children program's requirement for all community service organisations to use half of their funding on high quality programs?

For Upson, the immediate aim is to get more of their smaller programs "assessed" as meeting the minimum standard for good quality programs.

"We like to support local programs, and the next step is to get these programs approved as high quality," she says. "We know what we have been doing has been working, and we want to get our programs supported in the future."

Regardless of the outcome, Amity Health will continue to work on improving its services with its own skilled staff and with the help of others.

"In our sector there are lot of people to draw upon - community organisations, the Australian Institute of Family Studies and Drummond Street," Upson says. "My key learning is you can't do it alone, you always need someone to review what you're doing, and help keep you on the right track."

Cathie Valentine, Program Manager

Community Services Southern Region, Anglicare Victoria

Manages:

• Communities for Children (CfC), Frankston Northsite
• Communities for Children (CfC), Cardinia Shiresite
• Parentzone - Regional Parenting Service

After more than 20 years as a social service practitioner working with vulnerable families, Anglicare Victoria's Cathie Valentine still finds it "tricky" balancing the different perspectives of practitioners and researchers trying to establish evidence-based programs.

"It is like two opposite ends of the scale for researchers and practitioners to work through" she says. "The practitioner needs to be flexible and adapt to families, compared to researchers needing to prove that a program that is working is delivered exactly as it was written and that what caused the outcome was not from anything else. But it's not a disconnect - researchers and practitioners are simply working from two very different premises."

As Program Manager Community Services at Anglicare Victoria's Southern Region (located in Melbourne's southern suburbs), Valentine sees the tension between practice and research daily. She manages several programs including two Communities for Children sites (Frankston North and Cardinia) as well as other parenting services.

While Valentine works closely with on the ground services, she is definite about the value of research to improving services.

"Evidence tells us about the purpose of our programs, what are the outcomes and what have we achieved. We gather evidence for a range of reasons - predominantly for quality improvement for clients, but also planning and reporting to funding bodies, and making sure we are using the most current theories of practice. Where we can, we try to contribute back to the evidence base."

Since 1997, Anglicare Victoria has invested significantly in its research capabilities. It has an in-house research team - headed by Dr Tatiana Corrales - that plays an active role in evaluating Anglicare programs, translating emerging research into practice, advocating for policy change, as well as drafting funding and government submissions.

The organisation's research covers everything from literature reviews, analysis of existing datasets, improving program designs, quantitative and qualitative data collection and analysis. It has contributed to national and local program evaluations and research projects.

"Given the nature of our clients, much of our research is mixed methods and quasi-experiential, working with various individuals and organisations. We try to look at areas where people are most vulnerable and bring that to the attention of government," Valentine says.

One of the challenges Valentine sees in making research "part" of everyday practice is for time-poor practitioners to see the value of regular data recording for research.

"The big challenge is keeping staff consistent in data reporting. The practitioners' focus is their direct work with clients, and research is often seen as a secondary component of their role. We have to show evidence is relevant to their daily work, that what they do has made a difference and improves work into the future - when we do that we have a high degree of staff completing records. If we get good data, we can highlight good outcomes. If we can show it helps clients, it keeps the fire in the belly of practitioners," Valentine says.

To improve staff support, Valentine believes program managers need to set aside funds and time for gathering evidence.

"Most managers seem to be concerned with having high client targets, but the targets aren't the only issue, we need to be building the evidence base. Then they can go back to increasing targets once they know what they've done is effective."

However, Valentine still sees tension in implementing and assessing evidence-based programs in real world settings. She recalls a time at Anglicare when an evidence-based program was being strictly run alongside normal services that were more effective because they were adapting to client needs.

"With evidence-based programs, it's a different way of working and a different relationship with clients. But I don't think the approaches are totally incongruent."

So, what does she think will resolve this tension between research and practice?

"It's around partnerships, ensuring that one discipline is not elevated above the other, and each can see the importance of the other. But at times it doesn't appear to be that way."

Valentine also believes researchers can learn from practitioners.

"Practitioners innovate on a daily basis to find the appropriate strategy to help a family. By the time the evidence base is built, workers may have already advanced practice two or three steps ahead. Research bodies can be far too slow to keep up with that."

Moving to an "outcomes" rather than an "evidence" focus is also what Valentine sees as preferable for delivering future services.

"Evidence-based programs are often a specific intervention for a specific issue. We need to move from single-issue responses to more complex responses to issues - rarely do single interventions cover the complexity families are experiencing. We've got to make the short time we have with families count, we don't get multiple goes at it".

Valentine believes collaboration between researchers and practitioners is required to help overcome the increase in what she calls "wicked" problems affecting families.

"We have to change practice and research to deal with these problems. It would be marvellous to have practitioners work closely with researchers. If we did that we would get some shifts and changes, simply by different questions being raised by both sectors," she says.

"We need to walk in each other's shoes - it's got to go both ways."