Audio transcript: Strengthening what works for children
Return to CFCA webinar - 24 May 2017
Audio transcript (edited)
Good afternoon everyone and welcome to today's webinar, "Strengthening what works for children: Aligning target group, theory of change and program components to outcomes". My name is Elly Robinson and I'm Executive Manager, Practice Evidence and Engagement here at the Institute of Family Studies.
I would like to begin by acknowledging the traditional custodians of the land on which we meet. In Melbourne, the Wurundjeri people of the Kulin nation. I pay my respects to their elders, past and present, and to the elders from other communities who may be participating today.
Today's webinar will outline the Australian Centre for Child Protection's theory of change and evidence matching process. This process is an initial, rapid and cost effective step to give funders, managers and practitioners the confidence that their programs target group activities, and outcomes are in alignment and match to evidence. It will also provide an opportunity to hear from service providers who have used the process and the insights they experienced as a result.
This webinar today is a dissemination activity as part of the Expert Panel project, which is funded by Department of Social Services and administered by AIFS. The Expert Panel project offers support to Families and Children Activity funded providers to plan and implement programs, evaluate outcomes and share the results with others in order to improve outcomes for families and children.
Just before we begin, I'll just briefly mention some housekeeping details. You're able to submit questions via the chat box at any time during the webinar. We will respond to as many questions as we can at the end of the presentation. There will also be a post-webinar forum available on the CFCA website where you can discuss the webinar and access related resources. We will send you a link to the forum at the end of today's presentation, and you'll notice that as you leave today's webinar a short survey will open in a new window. We would appreciate your feedback. Please remember that this webinar is being recorded and the audio, transcript and slides will be made available on our website and the YouTube channel soon.
It's now my pleasure to introduce today's presenters. Professor Leah Bromfield is Deputy Director of the Australian Centre for Child Protection and one of Australia's foremost child protection researchers. She works closely with state, national and international governments on establishing and implementing child welfare reforms. Leah is also Professorial Fellow to the Royal Commission into Institutional Responses to Child Sexual Abuse and leads its extensive research program.
Dr Mary Salveron is a post doctoral Research Fellow at the Australian Centre for Child Protection. Mary has over ten years’ experience in the fields of child protection and early childhood and has undertaken research into factors that help and hinder the spread of innovations in child protection, as well as research and evaluations with diverse populations.
Jonathon Main is the Executive General Manager of Policy and Programs at Relationships Australia, South Australia. Jonathon is a qualified social worker with over 30 years experience in the provision of services to children and families with complex needs in both clinical and management roles in the health and welfare and non-government sectors.
Please join me in giving Leah, Mary and Jonathon a very warm virtual welcome.
Thanks so much for that introduction, Elly. This is Leah Bromfield here. I also wanted to acknowledge country. I acknowledge that we the speakers are here today on Kaurna land in South Australia in Adelaide. I also extend that acknowledgement to the traditional custodians of the lands of our audience and pay my respects to elders past, present and future.
I wanted to acknowledge Dr Mary Salveron who assisted us in developing this presentation today. She won't be speaking but will be, at the end, answering questions for everybody. I wanted to acknowledge the members of the ACCP’s Expert Panel who have shared their insights towards this presentation; and many programs which our centres have the benefit of working with across Australia. We certainly appreciate the way in which you have allowed us into your services and programs and we've learnt a lot from it.
Today I'm speaking about essentially a theory of change in the evidence matching process, and how that can be used to help improve service provision. But I just wanted to say a little bit of context first. That is just to start with an acknowledgement on just how important the work that we do with vulnerable children and their families is. We know that the cumulative impacts of the adverse childhood experiences affect all aspects of children's development.
We've learnt from South Australian data that 1 in 4 children are being notified to Child Protection by the age of ten years. We learned from the South Australian Royal Commission that of those notifications, the vast majority are actually legitimate allegations of child abuse and neglect that meet the statutory threshold. So what that means is, we're not talking about a needle in a haystack. We're talking about a serious and highly prevalent public health issue, one that couldn't be more urgent.
We know that there is an incredibly high burden of disease from child maltreatment and adverse childhood experiences, and in fact research has put the burden of disease for child maltreatment as equivalent to breast and prostate cancer.
We also know - all of us know who work within this sector – that we work within a limited funding envelope, and therefore what we do needs to be cost effective, it needs to be accountable. Our kids and our families need it to work, and at the very least we need to be absolutely confident that we are doing no harm.
So I don't think there would be any person in the audience today who would disagree that our most vulnerable children and families both need and deserve the absolutely best support and services that we can provide them. But we also know that best practice is hard to attain. There's many, many challenges associated with actually achieving the goal of best practice. Funding buckets, the cost associated with some evidence based interventions, the challenges associated with attracting, attaining and up skilling our workforce in order to be able to provide evidence-based and best practice interventions. So I guess the provocative question I'd really like you to all think about as I'm speaking today is, "Is something better than nothing?"
I wanted to talk about a piece of research that's been incredibly influential for us within the Australian Centre for Child Protection. This was a piece of research that looked at rigorous evaluations of home visiting programs which were designed to reduce child abuse and neglect. And what that research found was that programs which had clear aims and objectives, program components or activities, a clearly articulated mechanism of change or a theory of change, and outcomes, and all of which were consistent with their target population were effective 100 per cent of the time. We never get to say 100 per cent of the time as researchers. It's quite a phenomenal finding. However, when they looked at this – within this research they actually looked at 52 home visiting program evaluations – and the vast majority actually had a partial match, so they had some of those elements; and you can see from the slide here, the findings there were quite mixed. The other kind of astounding thing from a research perspective, was that those programs which had a zero match – those elements didn't match at all – none of them were effective.
We looked at this and I suppose we thought, "This is not that dissimilar to what we see within the child and family welfare sector in Australia, and in fact, when we start working with agencies some of our initial attempts at a program logic can look something like this – where you have an input of a .5 full time equivalent, we'll call her Carol. Carol's target group is everyone. She does home visits, some counselling, she has conversations, and meetings with other professionals, and then the magic happens and Carol achieves world peace.
What we found is that – and I suppose the feedback we've had from agencies – is that walking alongside programs to develop their program logics has been quite helpful and led to service improvements and insights. Optimally, we develop logic models as part of program planning and development. As researchers we use logic models as the first step in designing and conducting evaluations. At the Centre what we started noticing was, when we were conducting evaluations, we'd get through that first step and realise that a program was not what we were calling "evaluation ready". But really, what we were saying was that, from the logic model, we could see that misalignment and we could see from the outset this program wasn't poised for success.
We’re now seeing a growing movement to apply this process of developing up a theory of change and doing some evidence matching retrospectively as part of quality assurance and quality improvement processes. That was certainly part of the capacity building that was offered within DSS's Expert Panel.
Within ACCP we formalised our approach, and we call it our "Target Grouped Outcomes Approach", and as Elly said, it provides an initial rapid cost effective process to increase our confidence that a program's target group, activities and outcomes, are in alignment and matched to evidence.
I'm just going to run through some of the elements of the target grouped outcomes process. So the first step obviously is to really understand with agencies, other programs and service providers, what are the characteristics of their target population. And in unpacking this we start talking about both the ‘who’ and the ‘why’. A lot of our initial starting point with agencies is essentially that we see everyone. When we start to unpack this, what we find is that agencies are offering, in their promotional material, will be very broad because they're trying to provide a non-stigmatising entry point. But when you actually ask why they're doing it you start to understand how it is that they're recruiting their clients. It may be that their promotional material is very broad but they actually do have a particular target population in mind and a very particular purpose as to why.
We also try and unpack with agencies what other characteristics, or the components, that they think make their program work. So what is it they're actually doing? Again, at the outset this tends to be fairly high level with a lot of the programs that we've worked with. So people will talk about things like assessment, counselling, case work, parenting education, referral. In order to go to that last step of evidence matching, we really have to start to articulate exactly how those processes work. What are you doing in parenting education? What sort of model is it? What are the steps? How many weeks? So the real nuts and bolts of what we're doing. I suppose I've always felt that that's of great benefit to programs as well. Especially to new practitioners who are coming into programs because it saves from making some of that trial and error process they can learn from the colleagues.
We also unpack, what are the aims and objectives? What we're really trying to figure out here is, what are the short, medium and long term outcomes that the program is trying to achieve. We then step this out, put the pieces together – the target group, the activities and the outcomes – to form a theory of change. And this is really – sounds fancy pants, but it's not – it's just a plausible and sensible model of how we think something is intended to work. So we're just kind of stepping out. So you do those activities for this group, and the kind of the pathway that you think that it's going to take to get to the outcome as it is, work through that with the programs.
The final step is evidence matching, and I want to really clarify this. This is not the same as doing an evaluation. It doesn't prove that the program's effective. What it primarily does is, it's a flag for us. Is there anything within the existing evidence to suggest, for this program logic, that there's a misalignment or that this will be harmful if we were to undertake it?
What I'm going to do now is just run through – because I know this is sounding probably a bit academic – some of the common mismatches that we've seen in undertaking this process with different agencies, because I think it starts to put some flesh on the bones of this and demonstrate a way in which it can be quite helpful for programs, just in service improvement. So some of the common mismatches that we see is a mismatch between target group and activities, we seen program drift. We sometimes see logic models with multiple contingent outcomes, so that's where you've got a large number of kind of, “If we do this, then that will happen.” Then we think, “…and then that, and then that.” And what that means is that you've got a large number of potential failure points, weakening the likelihood of success. So as I say, I'm just going to work through some of these examples.
In this first one that we've got up on the screen here, we have a – our target group is parents who've had their children removed by Child Protection, and what's been offered here is an evidence based parenting program. The theory is that, if we can improve the parenting knowledge and skills of the parents, that their parenting will improve and the risks to children will decrease. The ultimate outcome is reunification for children.
But, when we actually do the evidence matching step, what we see is the evidence based parenting program that's selected, is based on social learning theory. It's one of the key components that makes that program effective. Now social learning theory, what that basically means, is that the program says, we're going to teach the parents a skill and then the parents are going to have a chance to practice that skill with their kids. And then we're going to give them feedback so that they can kind of build on that.
Where you've got non-resident parents – so in this case the children have been removed – the parents aren't having the opportunity to practice that skill. In the next example, this is one of program drift. This is another one where a service has selected an evidence based program. This is a program for parents with substance abuse issues, and was designed for children aged six to 12 years. The program in terms of its initial design, tried to target simultaneously the substance addiction, some of the underlying problems, and the parenting. And so it's trying to improve parent-child interaction and reduce substance use.
What we see in this example is that, in terms of the actual activities that have been undertaken, some of the activities have kind of dropped off, they've morphed a little bit over time, and critically that the target group changed over time. They ended up providing it to infants rather than the target population of six to 12 year old children. And without adapting the parenting strategies and children of different developmental stages are going to need different parenting strategies.
The last example I want to give you is a program which is actually contra indicated by evidence. This is one, this comes from an evaluation that's recently been published by Associate Professor Sally Brinkman in the Lancet. So a really high quality evaluation. And the target population for this is teenagers and – I'll run some of these, I'll get these boxes up for you. This essentially, this is the simulator dolls if people are aware of them. So the theory here is that, if we give teenagers simulator dolls to take home and they have to truly experience the challenges associated with parenting, it will put them off getting pregnant and having babies as teenagers. So the outcome is, in terms of the theory of change, is to prevent teenage pregnancy.
It sounds like a really, quite a sensible theory of change. But the evaluation of this one and I suppose this is why you need both plausible theory of change and the evidence match. The evaluation actually showed that teenage pregnancy increased and actually in fact it almost doubled. So those teenagers who had the simulator dolls had teenage pregnancy rates at almost double the rate of the teenagers who didn't have the simulator dolls.
This is an example where we'd be saying stop it now. The evidence is saying that this is, in fact, having the opposite to the intended effect. In our work with agencies, there have been some occasions where that has come up. And I have to say, it's a really, really difficult message to hear for agencies who are doing things that sound sensible, with the best intentions. And I have to pay respect to those agencies who heard that message and said, we do need to stop it now.
Interestingly, so this example, the evaluator has suggested that the problem with this is probably that in fact, the teenagers like it, right? So they get a lot of positive attention when they've got the simulator dolls. They often paired with a, as a couple and that's kind of this whole, very positive experience. So rather than it being the aversion therapy they were wanting. They see it as a great thing.
I just wanted to also think about this from the perspective of in the limited way that I can, to bring in the perspective of the client. And this is a quote from a mum whose children were removed: "We had to do the parenting program and see a counsellor to sort our issues. He did anger management, he went to a psychiatrist. The issue was alcohol. I thought they were there to pull the family together, that was the upsetting thing."
And so I suppose, what I want you to take from this is, it's one thing for me as an academic to talk about mismatches and lack of alignment between target group and outcomes of activities. But this really means something to our families. They’re noticing it when we've got it wrong, when there's a mismatch between their problems and what we're offering them. And that it matters to them, this is a family who lost their children.
So coming back to that provocative question that I asked, “Is something better than nothing?” I think that we have an ethical duty of care to be confident that we are doing no harm. So, if evidence indicates an activity or intervention is associated worsening outcomes or it's likely to be harmful, then doing nothing is better than doing that thing. I don't ever want to lose sight of a goal towards best practice with vulnerable children and families, but I think that we also need to put within our sights a minimum standard for our most vulnerable children and families.
And I propose that that minimum standard is programs that have clear aims and objectives, clear mechanisms of change, program components or activities and outcomes which are consistent with the target population. And programs that are confident that what they’re doing is not making the outcomes worse. Now, there are more and more resources being developed to support programs to actually achieve this standard and to undertake this process for themselves. So I would really recommend to audience members, if you don't have that theory of change, you haven't done some of that evidence checking within your program or service, to access some of these resources. So I would recommend the CFCA’s evaluation and Expert Panel resources. And also to let you know that there's some, there's a website called Theory of Change <http://theoryofchange.org> in which they have the free software app where you just kind of, it prompts you and you can plug in the information, and it will develop a theory of change for you.
I'm going to hand now to Jonathon who's just going to give you a service providers perspective on things. Thank you.
Thanks Leah. I just want to talk about our experience with the Expert Panel through the Australian Centre for Child Protection, which went for us over a number of sessions from late 2015 through to about October of last year, July of last year. We had a group program that we had developed in another context called ‘the Garden’. And that program we were looking to also implement to another separate target group and through a new program under the Children of Parenting Support funded program from DSS.
The group program was called ‘the Garden’, really because it was drawing on the analogy of gardens. It's a structured therapeutic program aimed at strengthening and repairing parent/infant/toddler relationships, and to do this through sensory exploration and shared positive experiences. And so the garden theme was around gardens are easily accessible, they're cost free, they're natural environments that families would already have experience with. And so there's an opportunity to offer an ever-changing, if you like, abundance of sensory delights that easily allow for play for some quality shared experiences to occur.
So that was the rationale for this program. We had actually developed it in the context of a homelessness funded service that we run called, Together4Kids. And that service is specifically for women and children who are homeless as a result of domestic and family violence. And so we wanted to have a program that helped support some repair, if you like, in their relationship following their experience of becoming homeless.
So the program runs over eight weeks, one and a half hour sessions. The intent is to have about, around five mums and their infants/toddlers with them in the session. And a focus around 0–4 in age was really how we'd constructed it. We had some other programs we developed for older age group kids within this cohort, but the Garden was specifically for children in the 0-4 age range.
So the aim of the program really is to strengthen the relationship between parents and their kids, and develop effective parenting and parenting capacity. And our content was really drawn from current research and evidence base around attachment and trauma, arousal and affect regulation, sensory integration and play therapy. So that was kind of the precursors that made up how we designed the Garden, and this happened about seven years ago, thereabouts.
So we’ve been running this program mostly through our homelessness sector work but to some related agencies. Given the success of it we thought it had application in other settings. So we wanted to look at other community based settings where there might be sole parents, or isolated parents, or disadvantaged parents that could benefit from a group experience like this, which would enhance and develop their effective parenting and parenting skills and the child's development and their own outcomes.
So that had us thinking about how the Garden might be adapted to our Children and Parenting Support funded program, which, within the western region of Adelaide where it's located, we also had a Children's Contact Service (CCS). And so for those who aren't aware, Children's Contact Services are particularly targeted for high conflict, separated couples where we provide a supervised handover or supervised contact service. And so given that we had begun to broaden into other community based settings, we thought, well this might be a particular focus that we could think of the Garden being offered in. And we used the Expert Panel to really help us nut through how we thought we would do this, and what would work and what wouldn't work.
So from the outset, what we got from the Expert Panel was a real clarity around the three areas of focus. And as Leah has already spoken about, the first one being about the alignment of the target group, the theory of change, the activities and the outcomes. The measures for intervention, particularly in terms of, for example, a parent-child relationship, and the implementation and practicalities, if I might call it that of, if you like, translating the Garden program into a CCS environment.
So going back to the target group, as Leah was talking about before, in terms of the who and the why of the target group, we had chosen fathers accessing Child Contact Services support. Obviously, because of the high conflict nature of their relationships with their ex partners, we thought that obviously they're a cohort who often don't get the sort of support that they always need. Men aren't good at help seeking behaviour as we well know, and our particular physical environment of the Children's Contact Service was potentially ideal for offering an environment for men who might not otherwise engage in programs.
So that was our kind of our who and our why. Again, the aim was still the same, which was to strengthen dads’ relationships with their infants or toddlers through this eight session structured program that we had. Our early conversations really with the Panel in many ways affirmed that our theory of change was strong and the important thing was about nailing the detail of that in terms of the activities and the measures that we assigned. But also what they would actually look like in terms of the program structure.
So early on, for example, we realised that we had some goals – perhaps too grand or too broad goals such as promoting child development – and we recognised that we really had to work back from what we believed were the longer term, or realistic if you like, goals of the program, which was primarily around reducing the risk of long term negative developmental outcomes for children, and promoting parenting skills, knowledge and confidence. And so from that, to map out what the conditions or the short and medium term goals that should be in place for this to be achieved.
So this then had us thinking about those short and medium term goals, those pre-conditions if you like, and what sort of activities parents could be participating in that would be developing those capacities. And by that I'm referring to their observational skills, their reflective functioning, their awareness of their own arousal affect regulation and their sense of empathy with their children. As well as obviously strengthening the relationship with their child.
So, all these activities were really aimed, in the context of the delivery of the Garden program, at providing if you like moments for ‘sensitive care giving’ – which is how we've constructed it – around parents and their children. And so at the opening of the program it was much broader, in terms of what that focus was, became more closely engaged and interactive between the parents and the children over the eight sessions. So being able to gauge progress towards the achievement of the goals through these activities also really led to a clearer focus of how we could better measure or evaluate the program outcomes.
So to give one example, one of the early sessions is under the theme of making nature art together. So it's one of the early activities that parents and children participate in where they're invited to explore, to discover, to collect things in their own natural environment or garden that can be used to create a piece of art work. So parents are encouraged to provide commentary on what they're noticing, what they're sensing, seeing, feeling and so forth, and to share this with their child. They're also supported to notice what their children are drawn to, what the central aspects of the activity are that they're responding to. What they like and don't like. So we're beginning to engage them around their, if you like, reflexivity and their effectiveness as a parent.
Through the activities, parents develop their observation skills, and are informed by an understanding about children's different needs at different ages. Obviously infants are different to say a toddler or a three year old. So that then took us with the Expert pPanel, around considering the sort of measures that we're using. When we constructed the Garden in the first instance, we were using the Karitane Parenting Confidence Scale for infants, and for the older, for the toddlers we were using the BITSEA (Brief Infant Toddler Social and Emotional Assessment) . And we'd chosen those because we thought that they would produce the sort of results that we thought were valid, were reliable and would be comparable. So, we were happy with that.
What we discovered in our conversations with the Panel over the meetings we had was that we needed to be able to look at other measures as well, particularly in relation to this new target group. That was the reason for us wanting to really make sure that we weren't just reapplying a proforma program to a whole new target group without being clear about whether that was actually matched.
So we talked about other potential measures, such as the PEEM, the parenting empowerment and efficacy measure. We talked about the importance of how the delivery of The Garden group was also meant to be in the context of providing, if you like, a soft entry for fathers using our Children’s Contact Service into a program, such as our Children and Parenting Support program. It wasn't just a stand alone, “Here, do the Garden and then go away.” So what we learnt from these discussions is that – and what we ended up settling on – is using another tool which is the Ages and Stages Questionnaire; [a] social and emotional (ASQ-SE). Because we wanted to assess children for social and emotional difficulties, given that one of the longer term aims or goals of the Garden was to reduce children's vulnerability.
The measure was also chosen because it engages parents, it's simple to complete, you know, using observations from the parents, it had decent psychometric properties and of course it's also easy to score. So we're conscious about maintaining a balance of needing to have some good, effective measures that align to what our activities were and our theory of change in our target group. But not to deluge people with a battery of evaluation measures or instruments. So the Panel discussions that we were having, the advice we got from the panel was really useful for framing that.
We still had some unknowns and some of the unknowns were about a key part of the program around the use of a journaling by parents – both in the session but also outside of it – to draw on their reflections about those moments that are captured within the session; what they'd noticed about their children, about themselves and what they'd learned. And this became a really, I think, critical element, given we were working with non-residential fathers coming for once a week or once a fortnight handover. We really were looking at how this program then fitted for this target group, because they weren't then in the environment of interacting with their children between the sessions.
We talked about some ideas with the Panel about whether we could do some variation on journaling for example and maintain some contact with the dads outside of that, the times when the sessions were being delivered. We found ourselves talking about alternative programs to implement instead of the Garden. And what that had us coming back to was, probably in a way, a central realisation that given the sorts of modifications we'd need to make to the program, it was almost like we were trying to potentially fit the program to the wrong target group.
So that was a really useful outcome for us. It was a bit unexpected because obviously we thought you know, this might be a suitable fit. But having said that we had already started it in one context around working with children who are homeless and their mums as a result of domestic and family violence. We had seen that it had some good uptake in some other community base settings. But clearly within this case, the issue for us around applying it to people who don't have a continuing engagement with their children between sessions actually would run the risk ultimately of turning the group into something which would be a lot less than what it could've been. At worst it would, or – no, sorry – at best it might turn it into a playgroup or a series of psychoeducation sessions, and that wasn't the intent of the program at all. It was really actually to achieve some more deeper, if you like, changes in parenting reflectiveness and interactions with their children.
So what we decided to do instead is, going on the advice to use the ASQ:SE, continue to use the Garden as a program in community base settings where we didn't have those difficulties. And ultimately I think that difficulty was, as Leah was talking about before, a mismatch between the target group and the activities and the outcomes. Clearly, we weren't having that within the other settings because those issues didn't arise for us. And so we were able to then, through some different community based settings around Adelaide and through one of the Communities for Children programs, offer a number of the programs which resulted in us having data from 30 parents and 41 infants who participated in the Garden between a period of December 2015 and June of last year.
And that data revealed an improvement in ASQ:SE scores over the eight week period. So that, for us meant that children's risk of social and emotional difficulties was reduced, and that this reduction was both statistically and clinically significant. So that suggested to us that children's participation in the program did directly contribute to the achievement of the chosen medium term outcomes, which included things like families being better connected, parents becoming more responsive, parents capacity for reflection being enhanced and so forth.
And of course, that this change was noticeable – the change in children was noticeable. So for us that was a very productive outcome even though it didn't lead to us delivering the program within the setting that we first had thought. And certainly from our point of view we're hoping that, you know, this kind of process would encourage other people to both use an Expert Panel process, and to consider how, in a sense, an idea which seems like a good idea to implement a program in a certain place, may not be a good thing if that results in a mismatch as we found. And that ultimately offering nothing can be better than something if the program is not well matched to the target group. So I think it's been a really constructive process for us in understanding that.
IMPORTANT INFORMATION - PLEASE READ
The transcript is provided for information purposes only and is provided on the basis that all persons accessing the transcript undertake responsibility for assessing the relevance and accuracy of its content. Before using the material contained in the transcript, the permission of the relevant presenter should be obtained.
The Commonwealth of Australia, represented by the Australian Institute of Family Studies (AIFS), is not responsible for, and makes no representations in relation to, the accuracy of this transcript. AIFS does not accept any liability to any person for the content (or the use of such content) included in the transcript. The transcript may include or summarise views, standards or recommendations of third parties. The inclusion of such material is not an endorsement by AIFS of that material; nor does it indicate a commitment by AIFS to any particular course of action.