Audio transcript (edited)

ROBINSON

Good afternoon everyone and welcome to today's webinar, Building a better outcomes framework for families: A story from the Mallee. My name is Elly Robinson and I'm the Executive Manager of Practice, Evidence and Engagement here at the Australian Institute of Family Studies.  I would like to acknowledge the traditional custodians of the land on which we meet today.  In Melbourne the traditional custodians are the Wurundjeri people of the Kulin nation. I pay my respects to their elders past and present and to the elders from the other communities who may be present.

Today we will hear the story of the partnership between Mallee Family Care and Social Ventures Australia and how they worked with key agency and community stakeholders to develop an outcomes measure framework.  This webinar marks an important milestone in the delivery of the Expert Panel Project, which is managed by AIFS and funded by the Department of Social Services. The webinar is the first example of dissemination of key findings from Expert Panel and associated Industry List projects for the benefit of the wider child and family welfare sectors. The Industry List, which is facilitated by AIFS, is available to providers to purchase assistance from experts to plan implement and evaluate programs and services.  I would like to invite participants in the webinar today to submit questions via the chat box at any time during the webinar. There will be some time to respond to your questions at the end of the presentation. We'd also like you to continue the conversation we begin here today.

To facilitate this we've set up a forum on our website where you can discuss the ideas and issues raised and submit additional questions. We will send you a link to the forum at the conclusion of today's presentation. Please remember that this webinar is being recorded and the audio, transcript and presentation slides will be made available on our website and YouTube channel in due course. Accessible versions will also be available. It's now my pleasure to introduce today's presenters. Fiona Harley has worked in the community services sector for over 30 years. She is currently Deputy Executive Director of Mallee Family Care and director of the agency's education, training and research unit. The unit operates a number of programs that positively engage with families and children in learning and education as a means to good life outcomes. In 2014 Fiona was awarded the Order of Australia medal in recognition of her services to welfare and community.

Jon Myer is management consultant with a passion for social impact and sustainability. He joined Social Ventures Australia consulting in 2014 and his work there has focussed on helping clients define and deliver on their strategies and effectively measure and manage their outcomes. Jon recently founded and led a start up social enterprise operating in India and Africa after winning the regional Holt prize in 2013.  Please join me in giving our presenters a very warm virtual welcome.

MYER

Thanks, Elly.

HARLEY

Thank you, Elly. It's a great pleasure for Jon and I to be here and today we're going to share the work that we have done towards developing outcome managements approach for the children of the Mallee. We hope to help you understand why we did it and what the results were and we're also going to highlight key challenges and lessons learnt and we'll answer questions at the end.

So perhaps a little bit about the programs that we've been involved in doing this piece of work and that's the Communities for Children (CfC) and the Children and Parenting Support program (CaPS).  These are the two programs that we've focussed on with this piece of work.  Our Communities for Children program works across the Swan Hill local government area. So that's predominantly the townships of Swan Hill and Robinvale; two very, very different communities but doing different projects and work across that site for Communities for Children.

Mallee Family Care is the facilitating partner for that but also provides some services as part of the project.  And our Children and Parenting Support was part of the new initiatives that were - came out of the discretionary grants through the Department of Social Services.  And we are providing programs in the Shire of Buloke, which is quite a challenging site to work in, because it's quite a large site with very few services and quite a low population base.  We also provide services in Mildura, Merbein and Red Cliffs. So really, right across the Mallee we're working with families in our communities around parenting.

MYER

Thanks Fiona.  So, as Fiona said, we were engaged by Mallee Family Care with, you know, the central challenge of how do we measure and manage the outcomes for children of the Mallee across the C for C and CAPS programs.  And so fundamentally we looked at four key areas in this project and the first was really how we're doing things currently, what is the current approach that Mallee and their partners are taking and we assessed that.  We then moved on to actually co-designing a better outcomes framework, not just with staff from Mallee Family Care itself but with staff across their partner organisations.

And then thirdly, we really wanted to outline a guide for how can we embed the framework that we were going to design together, so that it was really used and practical across the organisations involved.  And then finally sort of trying to outline some of the longer term recommendations for Mallee and its partners towards being clear on the outcomes they're seeking and how those can be measured.  So those are the four steps we aim to take, stepping forward.

HARLEY

Thank you.  So Mallee Family Care had three key drivers in exploring how the outcomes management could prove and improve client outcomes.  I guess primarily we were - one of the big drivers was the changes with the Department of Social Service introducing the score rating system for C for C and CaPS programs and across their other programs as well.  And we saw this as a positive thing.  We had long been wanting to be able to really meaningfully measure the outcomes of the work that we were doing.  So we saw this as a great thing and we embraced it.  We had some concerns though I guess about the rigour of the SCORE approach and we're looking for how we could use it, but how we could use it in a meaningful way and so we looked to how we could do that.

Mallee Family Care has a strong focus and particularly the work that we've been doing, a strong focus on being able to measure outcomes and know that what we are doing is making a difference for our community.  It's what we aim to do and it's what we work for.  We're part of our community and we want to make a difference for it.  So we then had a number of community partners through Communities for Children who we then had to work with about that as well to ensure that they too were on the same track as us. Once we had the conversation, yes, all of us were wanting the same things.  We were wanting to know that the work we were doing was making a difference for the families we were working for and ultimately the communities that they live in.

So we then, I guess, were looking for the system that would look for us and how we could actually go about doing that.  And we wanted to know that our clients were actually, I suppose, being empowered by the work that we were doing and that it was making a difference in their lives.  So that was what drove us to do it and so we came about how do we do this.  How do we actually move down this pathway of being able to have a really good, relevant, user-friendly outcomes measurement system across Communities for Children and Children and Parenting programs and I guess - I think it was just a fortuitous conversation with someone from AIFS.

I think it might have been Sharnee that led me to sort of think, well maybe the Industry List might be the pathway to go down. I hadn't really thought about how I was going to do this.  So I contacted Sharnee and we worked it through and it was a really simple process, a really simple process and Sharnee came up with I think three possible providers. They put in tenders and I - my staff and myself could go through those and worked out the best fit for us and for the work that we were wanting to do and Social Ventures was the one that we felt would work for us and so our journey began.

So the Better Outcomes Framework that we developed, we needed to align it with the Department of Social Services’ SCORE, because we knew that that was what was - well one of the main drivers for it.  We also knew that we only wanted to collect the data once and we wanted the data to be able to be used in lots of ways.  But one of those was for it to be able to be input into the SCORE system. In our initial work we came across and investigated further ARACY's The Nest Framework. And we really felt the synergies with that and the work that we were doing was really significant. So we looked at that and we used that and certainly the outcomes that they had overall – young people are loved and safe, have material basics, are healthy, are learning and participating and have a positive sense of identity – was really what we were wanting for our communities and for our families and children in our communities.

So I guess that linked really well with the work that we were wanting to do. We wanted the Framework to be easily understood by the staff, the frontline staff that were going to be doing the work and collecting the data. We wanted it to have meaning for them because we knew if it had meaning for them, then the data would have integrity that they were collecting and that was really, really important. So it needed to be simple. We needed buy-in along the way from them and so they were certainly part of the conversation right from the beginning, so that they were on the journey with us and so we developed the Framework.

MYER

So in terms of some of the elements that we often focus on in developing frameworks, we're pretty conscious that it's not just about the systems. In fact, the systems come later, that first and fundamentally measuring and managing outcomes is about people. That's why we all do this work and work in this sector and the same goes for developing these frameworks.  So for us we really, as Fiona said, wanted to engage all of the different members early in the process and also to be conscious of you know where their skills are currently and perhaps where there might be opportunity to build those skills.

Also try and embed a bit of a culture of measuring outcomes across Mallee and their partners and then once those things are in place and, I guess, as we go through the process actually starting to develop the systems, whether that's IT systems or the actual measurement tools that we're using.  So fundamentally those were sort of three of the elements that we were looking at developing throughout this process.

Stepping forward, I guess, into more of the what we thought today it's nice to sort of put forward for the audience some of the outputs of this journey and, I guess, one of the things that we often do as a fundamental step is to try and be clear on what are the changes and the outcomes that we actually want here and as Fiona said, we were thinking collectively across the C for C and the CaPS investments.  But really it was all about the same set of goals, which is just good outcomes for children of the Mallee.  So a tool that we often use at SVA to do this is called a logic model.  I'm sure it's familiar to many of you but for us you know it really starts with identifying what are those issues in the community that we're seeing to address.  Who's going to take part in a set of activities that we deliver and, of course, being clear on what are the inputs to make that happen. And then saying, “Okay, well look, these are the outputs but beyond that what are the outcomes we want to achieve in order to have an impact on that issue?”

So once the outcomes are defined, of course, we can move on to looking at indicators, which are measures of how those outcomes happen. So the logic model was a starting point for us and we can talk in a moment about how we developed this shared logic model across the partners. But we also identified quite early that it's quite well aligned to the ecological model, which again may be familiar to some of you. It's used by people such as the AEDC and it's just an identification I guess that as well as outcomes for children themselves we need to be thinking about their relationships, particularly parents and families and that Fiona from the outset was quite clear that they really wanted to measure and manage community outcomes as well.

So we sort of adopted conceptually the ecological model as we went about this project. So let's take a look at what became known as the child outcomes logic model. It's quite a lot to take in on one page but we're going to step through it with you and really it was just our way of articulating what are the 15 or so outcomes that were are trying to achieve with these two investments. And you'll see there starting from the left you know we had three fundamental issues. There were the child participants themselves and we were looking at a demographic of 0 to 12 and also their parents, carers and families and then some of the service providers who we were working with as well. We had mapped out what were the changes we wanted for them.

The impact statements on the right in many ways were drawn from Mallee's own vision statements and mission statements but were very well aligned to those of some of the funding partners. So we also wanted to keep it connected to the policy environment as well. So the black dots you can see there as well were where we had those outcomes aligned to the Communities for Children with strategic plans.

So this was developed across a series of workshops. It sort of looks simple to some extent on one page but it was actually quite a bit to get to this point and yeah, we had workshops across several different sites. Many iterations of this but the important part was to get everybody on the same page that actually we're all working towards this and we should all see in this model our own work and why we're striving and how we're striving for positive impact.

HARLEY

And I think Jon, it was certainly involvement of the frontline staff in the developing of this model that, you're right, it did go through lots of iterations and for some it took them a lot longer to actually get it. Others got it really quickly but we worked with them until at the end everyone was happy with that and everyone understood it. So I think it was a really valuable exercise, even though it did take up quite a bit of time and energy.

MYER

Yeah and I guess to add to that you know we did have some resistance didn't we initially, Fiona, in terms of people being wary around, well, what are you trying to do here.  Are you trying to audit my work? Is this going to be an additional burden to what I have to do every day anyway? So one thing we found was by opening it up initially and saying, look, fundamentally this is about the great work that you're doing but being able to prove and improve that work. So that was, I think, an important learning for us to engage early the process.

HARLEY

I think it was great to see some people go from that, well we know what we're doing is working and doing a really good job to actually then really wanting to be able to measure that and wanting to be able to see those outcomes through data.  And I think, you know, there were some who were there at the beginning and some who took a while through the process to actually get to that point.  But by the time we finalised that outcomes logic model everyone was at that point so – 

MYER

M'mm.

HARLEY

Which I think bodes well for the future.

MYER

M'mm. So, in terms of some of the individual slices of this model, we'll highlight those in a moment but we did want to align this model with best practice in the sector. So we looked to ARACY's Nest, as Fiona said. So each of these outcomes we could see connected to one of the five domains within ARACY Nest and this was just a helpful reference for us to also look at other work that had been done on specific outcomes and indicators.  And then if we sort of highlight, you know, particular stakeholder groups this was the set of outcomes we identified for children, a critical medium term outcome for the group was early childhood development and school readiness, which we can unpack a bit more in a moment.  But then later term we were looking at elements of emotional wellbeing, participation and achievement in school and those sorts of things were critical.

In terms of the parents and carers of the children we were working with, one of the most critical outcomes that was identified by the group here was actually parental confidence and capacity, which of course would lead to a more supportive family environment. So that was one of the most important outcomes for this participant group. Then, at the bottom of this model, we still looked quite closely at who are the organisations that are participating in the Mallee and this is perhaps a less researched area to some extent. So, we wanted to find a way to actually measure the strength of the partnerships. There are a lot of school groups that were involved but also local council and then the funded organisations themselves. So these were a smaller set of outcomes but still equally important that would be measured more at the population level.

HARLEY

And certainly important for Mallee Family Care as the facilitating partner for Communities for Children to be able to have that understanding of the strength of the partnerships and the work that we're doing.

MYER

So that should give you a sense of where we got to with the logic model. In terms of some of the key takeaways, Fiona, I might pass to you to highlight a few of the things that we found.

HARLEY

Yes, so I guess what came out of the development of the outcomes logic model. The three things were, access to services as a springboard to a range of outcomes. And this critical early outcome was, I guess, seen as a springboard to a range of other outcomes across the healthy, loved and safe learning in systems environment domains. And in particular it linked with service availability, awareness, access, and referrals. And we know that in rural communities all services aren't available.  And if I use Buloke as an example, Buloke is a really interesting Shire in that, that it has services coming in from a range of different larger rural areas, you know, towns – so some might be from Bendigo, some might be from Horsham, some might be from Swan Hill. So for people to navigate that and understand where they can get their services from can be very, very different and difficult, and that can be quite disempowering. So that was one of the things that we really wanted to – it came out as a significant area to focus on.

The other was – a second one, I guess – was around that families consist in a system and they need to be viewed within the system they operate in, and that we need to understand all the variables and how they connect. So the piece of work that we're doing is only one piece of work that we're doing that's happening with this family. They're also interacting with schools, with churches or with other service providers in those communities. So we need to understand the work that we're doing and how it fits within that.  

And the other, I guess, the two main cornerstones of our logic model – and these were really critical for us because it underpins significantly the work that we do – was around the improved early childhood development and school readiness. And for really all of our programs that operate out of the Education Training and Research Unit, that the cornerstone is the engagement with early education, the means to being school ready and then a means to good life outcomes. So that was critical for us as part of this model. And all of the activities that are undertaken across Communities for Children and the Children Parenting Programs really are around that and around that preparation for, or some of them are happening within, schools.  But it's about dealing with the issues that are actually inhibiting children's ability to really engage with their education.

And the other was around increased parental confidence and capacity. And we know that if that happens for parents that they're more empowered to be in the parenting role, to make decisions, to access the services they need for their families, and that relationships in the families will improve.  So they were really the most significant things that came out of it.

So then we wanted to work out how we're going to measure these identified outcomes. A strong set of indicators was developed through, once again, a very consultative process.  So we spent a bit of more time then with the groups going back about, “So what are the indicators?” As Jon's pointed out, we'd already identified the issues, who the participants were, what the activities were, what the inputs were going to be, I guess, the outputs, and then the outcomes both the short and long term, and then what we hoped to be the overall impact.  We needed to measure these indicators and work out, well what are the indicators and how are we going to measure them so that we get the outcomes that we actually want.   So, initially we focused on measuring ten priority indicators.  There were so many, so many we could've, but we really felt we needed to bring it back to something that was really, really reasonable. So one of the one's, as I've indicated, that's important to us was improved early childhood development, school readiness. So the indicators that we chose for that were around number or per cent of children developmentally vulnerable below the cut-off line in ASQ3, which was the Ages and Stages Questionnaire, which was the evidence based measurement tool that we'd chosen to use for this particular area. The number and percentage of children participating in degree qualified early childhood education before starting school, it's under the three to five year period. So that's knowing that they're getting that quality, that quality early education, and the hours per week of attendance at an early childhood education setting for the three to five year olds.  

So they were the three indicators that we were going to measure, the areas that we're going to measure so that we could then see whether or not we were making a difference in respect of that outcome that we want to achieve around improved early childhood development and school readiness.  

Jon will talk about now the other priority indicators and how we were going to measure all those?  

MYER

Yes, so picking up on what Fiona said, I think what was important for us here was to figure out ways to make this process manageable for front line workers who we knew were ultimately going to be entering the data. So with, you know, something like early childhood development, there was a range of tools that we looked at and assessed, but the ASQ3 itself was already used by Mallee, so while there were others that we thought could also do the job to make sure we knew that that change was being achieved, once we did that scan we thought, look it's probably going to be easiest for staff members to continue to use the tool, but find a way for them to input the data into DSS SCORE. So what's visualised here is really just a snapshot of the other priority outcomes that we wanted to begin measuring, and then just a short set of five tools that we thought would do the job. There's a range of different organisations that we looked at here. We also wanted to keep, where possible, the tools consistent with the translation that AIFS had done into DSS SCORE again, which would make the job one step easier for staff members and for Mallee rather than having to figure out their own translations, so that things like SDQ.  And then in other cases we actually needed to devise our own measurement tools, and in this case I'm thinking of the intake and exit forms. We needed to tailor those to meet the needs of the logic model that we'd development together. 

A piece of work that VicHealth had done we found quite useful around assessing the quality of the partnerships. That's sort of more the community changes that we wanted to measure. So that was relevant for two of our outcomes as well. But again, just emphasising the point that we wanted to keep the set of tools limited, so that this going to be achievable in the short term.  

So look, overall we'd developed the Better Outcomes Framework at this point and we were recognising that we wanted it to collect data at these three different levels. You know, starting from the bottom, we needed it to be practical and useful for the community partners. And there were – what was it Fiona?  About ten or fifteen partners in total?

HARLEY

No, not that many.

MYER

Not so many?

HARLEY

No, no. I think there's five or six.

MYER

Okay.

HARLEY

Five or six community partners.

MYER

 So yes, first and foremost we wanted it to be valuable for them, so that when they went back into their communities and with their beneficiaries, that they had a clear sense of where they achieving their own goals. So that was sort of the first set.  And then at a higher level, you know Mallee Family Care was considering both the C for C and the CaPS investments, and we wanted to give Mallee a sense of how it was achieving its goals broadly, and then of course we wanted to be feeding that information up into DSS SCORE in order for funders to have a perspective on how the Mallee region was tracking.

So that was broadly the three different levels that we wanted this framework to be reporting on.

HARLEY

So the others things that we wanted to know I guess, and for our purposes, but about particularly our work in the Mallee, was around the geography.  So, you know, the Mallee is a very large area. I mean the Mallee Family Care catchment's about 100,000 square kilometers and about 100,000 in population. And it differs greatly in its makeup. So in Robinvale for example, it's a very multicultural community and a high Indigenous population. So to be able to look at the work that we're doing there and be measuring that, but then putting that against some of the work that we do in say, Merbein. It's got a totally different makeup. Or in the Buloke Shire.  

So that was important for us to do –  be able to be measuring the same thing, but in different places and then to be able to compare that.  And I guess some of the learnings we hoped would come from that would be, so this program's working really well with this community but not so well in this community – so why is that?  And what can we do to address that?  So that's part of the learning I guess from the geography in particular for us. So we really see that as being vitally important and so there's certainly a number of indicators that are the same across a lot of the programs so that we can do that and do that well.

We wanted also to look at particular client groups.  So for example, Indigenous versus non-Indigenous, pre-school aged children 0-5 and post-school aged children. So really looking at those different groups, we felt that that was really important.  

We then wanted to be able to use the data we've got to look at it and overlay it with, for example, ABS data, ADEI data, some of the results from what ARACY is getting with their work with The Nest, and certainly other things that are happening. So we're wanting to look at that so we can see where we sit by that at State level but also at national or even international level in respect of the work that we are doing, and in particular those key areas around engagement with early learning and real school readiness, and increased confidence around parenting. So they were the big areas for us. So we wanted to be able to have robust data of integrity that we could be comparing with other areas.

So, I guess, this is just a map for you to see the area that we cover and the shires that we're covering and the local government areas that we're working in. So to make this work, we actually – and we've just finished the process of going back into the community to embed the framework with the community partners in particular.  So for each of those community partners, we worked with them to develop their own program logic for the particular activity that they were undertaking, then looking at the indicators and certainly ensuring that there was some common indicators across all the activities, and then looking at what tools to measure it and how to go about doing that.  And so we've just completed that task. We now, I guess to a stage, probably the really exciting bit, which is about being able to start really gathering that data and being able to look at it.  We've already done a little bit of that, we've certainly seen the significant, I guess shift in the mind set in the culture of the community partners around the importance of collecting that data and a bit of an excitement about being able to do that.  So that's been a great outcome to see and really it is very much early days for that.

So I guess, you know, the workshops that we held with the community partners, really we just wanted to answer three simple questions. How do we select the outcomes?  So what are the outcomes we want to achieve?  And I guess the end of that, what is the impact we want to have within that community, within that particular cohort?

How do we prioritise these outcomes?  Because there's a whole lot of outcomes, but we can't really look at them all.  So which ones are most important and which ones should we focus on measuring?  And that's not to say that down the track we might look to measure other outcomes as well, or we might decide to, for a period of time, do a snapshot around certain outcomes because that's what we want to know or we're interested in that area.  

And then, how will we measure these outcomes?  So what indicators will we use and what measurement tools will we use. Making sure I guess, that those measurement tools are ones that have been translated and can then be put into SCORE, so that there is rigour and integrity in the data that we're actually using.

So, I guess our journey step one was to develop it and that happened from November last year to February of this year.  Step two was about embedding it and we're pretty much at the end of that phase and we've worked within our own organisation, but also with all the community partners about the embedding of that. The systems and the staff, the skills that they're going to need for it, and looking at I guess them adopting this and taking it on. One of the things that we also found that we needed to do – and we hadn't really thought about this at the beginning – was we needed to have some sort of software that they could them easily input the data into.  So we then needed to go on a search to be able to find that particular software that would be easy, that could be on a tablet or even a smart phone, so that when they're actually out doing the work they can just enter that data, and then that can be translated and then uploaded into DEX and SCORE. So we're finalising that at the moment, and I think that's going to be a really exciting piece to sit alongside this. And once again it's about making sure that the frontline staff can easily do this piece of work and gather that important data.  

So we're now into, I guess, what we're calling the launch phase, which is where we're actually starting the data collection and we're hopeful that we will have some outcomes from this next six months of data, so for our reporting period ending at the end of December, that there will be quality data around the outcomes.

And then ongoing, we see this just as the beginning and that we will be embedding this in further with the community partners and making sure that it just becomes the way we do our work.  We're also looking at how we can do this in other programs, particularly in our own unit, and maybe down the track do an SROI, so looking at actually, really at that community level, what has been the social return on the investment for those communities and the changes it's made.  So I think certainly the thinking has changed in respect of that.

So, lessons learned: I don't think any of us realised what we were getting ourselves in for, quite frankly, I have to say.  And that's okay, I mean you know, it's been a journey and sometimes I think we've taken the scenic route, but we've got where we wanted to in the end. I think, the process that we went through and the work that we did and our philosophy around ensuring that the frontline workers, who are going to be gathering our data were part of it, that understood it, had buy in, and too became true believers as we were, who started this. That was really, really vital, because we knew that, at the end of the day, I mean, I knew that I could say this is what we're going to be doing, but if there wasn't that buy in, then you weren't going to get the integrity in the data. It'd be, "Oh well, we'll just put this here and that there".  What we wanted was good robust data that we knew reflected the work that was being done. So that was really really important.

I guess the other thing that's important to note is that, developing a shared outcomes framework requires significant investment and of both time and money. This hasn't been a cheap exercise for our organisation because we needed to engage Social Ventures Australia to do that piece of work for us, and we've now needed to engage a provider to develop our system for framework for collecting data electronically. So those things have been reasonably high cost items for us. We certainly are totally committed to it, but I think that and believe that, the government's direction around measuring outcomes is excellent and we strongly support that. But there needs to be an understanding that there's a cost associated with this and a costing around resources, and that needs to be considered. There's no doubt about that. We were lucky that with both of those programs there was flexibility that some of the dollars could be used for that, but it's also taken a lot of time to do that.

Long-term perspective is really important, so not rushing things, knowing that, "Okay, we want to measure these indicators in the first twelve months, but there might be others that we want to measure in the next twelve months".  And there'll be times where we think we'll just pick one out and measure it. So connecting a flexible approach to it and keeping it really really simple – simple and relevant. Relevant to, I guess, the families we're working with, the staff who are investing their time and energies in doing that work, and also to the communities, so that we actually know we are getting the outcomes that we want to get.

Thank you.

WEBINAR CONCLUDED

IMPORTANT INFORMATION - PLEASE READ

The transcript is provided for information purposes only and is provided on the basis that all persons accessing the transcript undertake responsibility for assessing the relevance and accuracy of its content. Before using the material contained in the transcript, the permission of the relevant presenter should be obtained.

The Commonwealth of Australia, represented by the Australian Institute of Family Studies (AIFS), is not responsible for, and makes no representations in relation to, the accuracy of this transcript. AIFS does not accept any liability to any person for the content (or the use of such content) included in the transcript. The transcript may include or summarise views, standards or recommendations of third parties. The inclusion of such material is not an endorsement by AIFS of that material; nor does it indicate a commitment by AIFS to any particular course of action.

Slide outline

1. Building a better outcomes framework for families: A story from the Mallee

• Seeking solutions, shaping success. Towards stronger more caring communities.
• Mallee Family Care
• Social Ventures Australia
• The views expressed in this presentation are those of the presenters, and may not reflect those of AIFS or the Australian Government.

2. Agenda

• Overview of and our work together to develop an outcomes management approach for the children of the Mallee
• Discuss why and how we undertook this project and what the results were
• Highlight key challenges and lessons learnt and answer questions

3. We wanted to build an effective outcomes management approach across the Communities for Children (CfC) and Children and Parenting Support (CAPS)

• Our work together focussed on four key areas:
  1. A review of MFC’s current approach to outcomes measurement in CfC and CAPS
  2. A new co-designed Better Outcomes Framework for CfC and CAPS
  3. A guide to embedding the Better Outcomes Framework with specific Programs and Community Partners
  4. Overall recommendations for developing outcomes management across CfC and CAPS

4. Mallee Family Care (MFC) had three key drivers in exploring how outcomes management can prove and improve client outcomes

1. Funders: DSS Score
   • On 1 July 2015, DSS introduced the SCORE rating system and for CfC has a requirement that 30% of funded activities are Evidence-Based Programs (this requirement went up to 50% in July 2016).
2. Management: Better decision making
   • MFC and Community Partners were keen to understand how outcomes measurement can be used to prove and improve your work
3. Clients: Better outcomes
   • MFC and Community Partners were committed to collecting and evaluating client outcomes to ensure the best possible outcomes for clients
4. MFC has now developed an outcomes framework for the Communities for Children and Children and Parenting Support (CAPS) funding streams.

5. The Better Outcomes Framework was developed to be aligned with both Department of Social Services’ SCORE and ARACY’s the Nest Framework

• Rationale:
  • To ensure that the outcomes we measure can more easily be input into the DSS SCORE platform
  • To ensure that MFC's outcomes framework is developed in-line with best-practice thinking in the area of child wellbeing
  • To ensure that MFC's outcomes framework is consistent with other providers

6. Effective outcomes management requires the right systems, staff, skills, structure and culture

• Systems
  • The processes and procedures of the company.
  • They reveal business’ daily activities and how decisions are made.
  • They determined how business is done - they should be the main focus for managers during a change process.
• Culture
  • Reflects management style of company’s leaders.
• Structure
  • Represents the way organisational divisions and units are organised.
  • Includes the information of who is accountable to whom.
• Staff
  • Type and number required to function well.
  • Approach to recruitment, training, motivation and retention.
• Skills
  • The things an organisation’s employees do well.
  • Includes capabilities and competences.
  • Skills and competencies need to be fully aligned with any new strategies, structures, or service offerings.
• Adapted from McKinsey’s 7S Framework of organisational effectiveness, where Systems includes IT and other key processes, and Culture refers to Shared Values and Style combined.

7. Mallee Family Care’s Better Outcomes Framework is underpinned by a logic model that links activities to outcomes and ultimately impact

8. The Framework draws on the ecological model of child development – also used by the AEDC – to measure outcomes for children, families and the community

• Participants identified in Mallee Family Care workshops
  • Societal
  • Community: Service providers working with the children and families above
  • Relationship: Parents, carers and families of the children above
  • Individual: Children in the Swan Hill, Mildura and Buloke communities aged 0-12 (especially those from vulnerable backgrounds*)
• ARACY report http://www.atsdr.cdc.gov/communityengagement/pce_models.html

9. We co-designed an over-arching ‘Child Outcomes Logic Model’ that highlighted the data that Mallee Family Care and it Partners need to collect

• *closely linked to MFC's CfC Priority Areas

10. Each of the 15 outcomes was coded to the domains of ARACY’s the Nest: Healthy; Learning; Loved and safe; Participating; Systems & environment

11. One set of outcomes is primarily focussed on children...

12. …another is primarily focussed on parents, carers and families…

13. …and a final set of outcomes is primarily focussed on service providers

14. Three key insights can be drawn from MFC’s Better Outcomes logic model

• Access to services is the springboard to a range of outcomes
  • This critical early outcome was seen as springboard to a range of other outcomes across the “Healthy”, “Loved and Safe”, “Learning and “Systems and Environment” domains
  • Service availability, awareness, access and referrals need to be captured
  • Increased service availability, awareness, access and referrals
• Health/education (other domains) are inter-related and self-reinforcing
  • All of the five outcome domains are inter-related, suggesting that child wellbeing requires activities across a range of areas
  • For MFC’s CfC and CAPS programs health and learning were particular self-reinforcing
  • Healthy, Learning
• Two main outcomes are the cornerstones of MFC overall logic model
  • The most critical linkages in the logic model lead to and stem from just two main outcomes
  • Improved early childhood development and school readiness
  • Increased parental confidence and capacity

15. To measure the 15 identified outcomes a strong set of indicators was developed through a consultative process

16. Initially we are focussed on measuring just 10 priority indicators

• TAMARIKI OUTCOME
  • Improved early childhood development and school readiness
• INDICATORS
  • Number / % of children developmentally vulnerable (below the cut off in ASQ3)
  • Number and % of of children participating in degree-qualified ECE before starting school (3-5 years)
  • Hours per week of attendance in ECE (3-5 years)

17. Overall our 10 priority indicators can be measured with a set of five key tools

18. MFC’s Better Outcomes Framework will collect outcomes data at three distinct levels

• Outcomes data:
  1. Limited set of outcomes data entered into DSS SCORE (beginning with CAPS and growing over time and in-line with funding requirements)
  2. Complete set of outcomes data based on the MFC’s Better Outcomes Logic Model (beginning with priority indicators and growing overtime)
  3. Complete set of outcomes data based on the Program Logic of each CP or relevant Mallee program

19. MFC will also want to evaluate and report on outcomes achieved by considering geography, demography and outcomes domains

• Geography
  • Under the Better Outcomes Framework, MFC and Community Partners may seek to compare and contrast outcomes achieved across different sites (eg Playgroups in Mildura vs Red Cliffs or PuP in Merbein vs Robinvale)
• Demography
  • Mallee Family Care may also want to evaluate and report on outcomes achieved for particular client groups (eg indigenous vs non-indigenous, pre-school age children 0-5, and post-school age children).
• Outcomes domains
  • To provide an overall picture of the outcomes being achieved for children of the Mallee region, MFC may seek to report on aggregate ‘Learning’ outcomes (more example total numbers of positive shifts achieved across the four key Learning outcomes)

20. We then went back into the community to embed the Better Outcomes Framework within Programs & Community Partners

21. We held workshops together to help our Community Partners to answer the following three simple questions

1. How do we select outcomes?
2. How do we prioritise these outcomes?
   • i.e. which ones are most important and which ones should we focus on measuring.
3. How will we measure these outcomes
   • What indicators will we use?
   • What measurement tools will we use?

22. We have developed an overarching Better Outcomes Framework and we continue to embed it across the CfC and CAPS programs in the Mallee

• Step 1 - Develop
  • Nov 2015 - Feb 2016
  • The Better Outcomes Framework has been developed (as outlined in Section 2 above and in the Outcomes Spreadsheet).
• Step 2 - Embed
  • March – June 2016
  • Mallee now needs to embed the Framework by taking action in three main areas:
    1. Systems*
    2. Staff & Skills*
    3. Structure & Culture*
• Step 3 - Launch
  • July 2016 – June 2017
  • In this Phase Mallee will launch the data collection approach highlighted in the Outcomes Framework.
• Step 4 - Evolve & Evaluate
  • Ongoing
  • In this Phase Mallee will evolve the Better Outcomes Framework in response to feedback from relevant stakeholders and undertake program specific evaluations (eg SROI).
• * Adapted from McKinsey’s 7S Framework of organisational effectiveness, where Systems includes IT and other key processes, and Other captures Structure, as well as Shared Values and Style (i.e. culture).

23. There are three main lessons we learned throughout this project

• Lessons learned
  • Getting buy-in from front-line workforce is essential – the systems need to work for them
  • Developing a shared outcomes framework requires significant investment (time and money)
  • Taking a long-term perspective and keeping it simple can contribute to the uptake of outcomes frameworks
• Learn more from the SVA Quarterly http://www.socialventures.com.au/sva-quarterly/topic/measurement-evaluation/

24. Questions?

• Continue the conversation started here today and access a range of related resources on the CFCA website: www.aifs.gov.au/cfca/news-discussion