Finding the common ground: Developing a generalisable theory of change

STEPHANIE FISHER: Welcome, everyone to today's webinar, ‘Finding the Common Ground: Developing a Generalizable Theory of Change’. My name is Stephanie Fisher and I'm a Research Fellow working in the Child and Family Evidence and Evaluation team here at AIFS. I would like to start by acknowledging the Wurundjeri and Woiwurrung and Bunurong people of the Kulin nations, who are the traditional owners of the lands in Melbourne where I'm speaking to you from. I also pay my respects to traditional owners of country throughout Australia, recognizing their continuing connection to lands and waters. We pay our respects to Aboriginal and Torres Strait Islander cultures and to elders, past and present. Today's webinar is a part of a series designed to share examples of evaluative practice and ideas for collecting and using evidence in the context of Australian child and family services. The format is a bit different to our usual webinar program. It includes a slightly shorter conversation between facilitator and guests that aim to provide an overview or introduction to a particular topic as a foundation for further thinking and learning. Today, we're going to be discussing how AIFS and Relationships Australia collaborated to navigate complexity, diversity, and differing perspectives to co-design a generalizable theory of change for men's behaviour change programs. My guests today are Genevieve Herd and Glenn Althor from Relationships Australia. Before we dive into our discussion, I have a little bit of housekeeping to cover. This webinar has been prerecorded, which means it does not include a live Q&A. For those of you who require captions, please watch the webinar via the AIFS website. There is a link in the chat, and a full transcript will be available. There will be some related readings and resources, including a copy of the slides we share, as well as a copy of the Theory of Change we'll discuss and some additional resources on theory based evaluation. These are available in the handout section of the GoTo Webinar Control panel. A short feedback survey will open at the end of the webinar. Finally, this content discusses men's behaviour change programs that aim to support men to end their use of controlling and abusive behaviours in their relationships. Please take care while listening and if you would like to seek support on domestic and family violence, including coercive control, there are some numbers on the screen that you can call. Gen and Glenn, did you want to say hello and tell us a little bit about yourselves and your roles in the context of this project.

DR GLENN ALTHOR: Hi, Steph. Thank you for having us. I'm Glenn, I'm the Head of Research and Impact at Relationships Australia, New South Wales. I have a research background in human geography and particularly distributive justice, but I've also spent several years in the sector, working in an evaluative and research capacity. The team that I'm heading up is responsible for evaluating the impact of our programs that we have quite a few of and we do this doing it using a theory-based approach. We designed this specifically to feed into a learning approach that we take in our organization to use the evidence that we gather to better improve our services. Over to you, Gen.

DR GENEVIEVE HEARD: Thank you Steph. Thank you. Glenn. I'm Gen, Manager of Research at Relationships Australia, Victoria, where I've worked for nearly ten years now. My team does project work, which examines client needs and outcomes of our services, in more depth than is possible using our routine evaluation methods. So, it may be clear already that Relationships Australia is actually a federation of many organisations, one from each state and territory. And Glenn and I represent two different state organisations. Most are in most states and territories, deliver men's behaviour change programs that work with men to change their violent behaviour and support the safety of women and children. However, each state and territory RA operates in a different context, with significant diversity in these programs, in levels of funding, in legislative contexts, and in modes of service delivery. And, quite differently in terms of approaches to engaging and working with the men, and with the women and children and other affected family members, through the service. So, the challenge for this project was finding the common ground between these programs that could underpin a generalizable theory of change.

STEPHANIE FISHER: Thank you Gen. Glenn, can you tell us about theory-based evaluation and can we get the first slide up, please.

DR GLENN ALTHOR: Sure thing. So, the way that we use theory-based evaluation is we consider an intervention, which most of our services are as a piece of theory, which is a story for ‘well, we've got some idea that we're going to do some things, and we're going to make a change in the world based on that’. So, the theory-based evaluation centres that story and says, we need to understand what we're going to do and how that's going to work. We can put this into some sort of a just a program design document, or more formally articulate that into something like a theory of change or program logic, which many folks are familiar with. But for those who aren't, a theory of change or a program logic are both types of imperfect is a really important part, but imperfect causal models that, really articulate how and why we think a program will generate specific changes for the intended recipients. The model itself can be based on like an evidence base built from the literature, peer reviewed literature, but it can also be built. And this is something that we really enjoy doing with working with the stakeholder groups, particularly co-designing with the people or practitioners who are expected to deliver that service on, ‘why do you think this is going to make a difference? What are you going to do and how is that going to change people's lives?’ And also really important, you could do that with the intended recipients of any given program or intervention to understand better how to do that. So, a theory of change can be really simple. When I run workshops on this, I tell people, you've maybe never even heard of this, but I promise you, you've got one in your head. You know how and why your program's supposed to work. You can write that down in a paragraph. Or it could be incredibly complex. Uh, like the ones we're going to talk about here today and articulate a really deep expectation of what's going to happen as a result of the intervention. That tends to depend on the nature of how complex intervention is and how resource intensive it is to actually do the work of developing a theory of change. I really like this because it from the start says, what are we trying to achieve before we go and measure stuff. I would like us to really go. This is why we think change is going to happen, and we treat that as an assumption. I think that I treat a program and a theory of change as a set of assumptions, which the evaluation actually then goes away and tests. So that can become the basis for learning more about how your service is working, which we use then to feed into continuous program design. It can also be used just to judge the effectiveness of your evaluation. If anyone's interested in reading and learning more about this stuff, the most influential text for me on this was a book called Realist Evaluation by Pawson and Tilley. Really famous text. It's not specifically just about all this theory stuff. It's more about realist evaluation. But I think they cover that really well in that and also I have some awesome content on this as well. Thanks, Steph.

STEPHANIE FISHER: Thank you Glenn. On the slide, we've included an example of a theory of change from one of our resources. It may be a little bit hard to read on the screen. So we've also included this as one of our handouts if you're interested to read more. Can we get the slide down please? Thank you.

DR GLENN ALTHOR: So Steph, I need to ask you a question because I came to ask for support with our project, which was developing a theory of change. So can you tell us a little bit about that process and how it went for you?

STEPHANIE FISHER: Yeah. Thank you Glenn. So the process we took to develop the theory of change was deliberately collaborative and drew on research evidence, balanced with practitioner expertise to build an understanding of how RA s men's behaviour change programs operate in real world contexts. We began with a national audit of the programs delivered across the federation of Relationships Australia. This aimed to capture detailed information on the differing program models. All of the different core features and the very different operating contexts in which these programs exist. Where gaps emerged, we followed up with targeted requests to deepen our understanding and make sure we were capturing all the really relevant information. In parallel, we conducted a desktop review of the literature to examine what the evidence has to say about how and why men's behaviour change programs are expected to work. This approach ensured the theory of change was grounded in evidence and practice expertise. A key element was engaging the Relationships Australia National Research Network, which assisted to navigate the realities of the federated model, where each jurisdiction operates as a distinct entity with its own structures, constraints and legislative requirements. Our aim was to develop something genuinely generalizable that highlights the similarities and acknowledges the differences. We brought together the evidence and practice insights that we gathered through a series of workshops. These workshops were critical for teasing out assumptions and testing whether the emerging theory made sense to the people on the ground, and also surfaced further insights from those practice experts. This iterative co-design process helped to refine core components, language and assumptions, allowing us to identify the common ground across very different programs. Ultimately, this led to shared ownership of the final generalizable theory of change. One of the challenges we all experienced included engaging with the right stakeholders. Gen, can you tell us a little bit about getting the right people in the room for the workshops and how we approach this? Can we get slide two up, please?

DR GENEVIEVE HEARD: Yeah. Thanks, Steph. Well, engaging the right people was certainly, essential and certainly challenging. So in the broader project, I worked very closely to identify the right representatives, from the states and territories, ensuring representation at different stages from national leadership program managers and frontline practitioners. But the workshops were deliberately practice focused in order to draw on the deep expertise held by our practitioners about how the programs actually operate in the real world, what change looks like for program participants, and the points at which change is likely to occur. So, these design workshops involve two practice representatives for each state from the RA agencies, which deliver MQP. These staff were mostly facilitators of the group programs, and some program managers who could provide deep insights about how these programs work. So, we're talking about staff who have qualifications in social work, psychology or counselling. They have deep expertise in family, family violence dynamics in trauma informed practice, in group facilitation and in risk assessment. And of course, they're very familiar with the MBCP curricula that they deliver, which combines both therapeutic and educational approaches. Steph, can you talk us through the final theory of change that was developed out of this process and why it's so important for these programs? We might need the next slide, please.

STEPHANIE FISHER: Thank you Gen. What this theory of change does is bring together a very diverse set of programs into a shared and generalizable depiction of how change is understood to occur across the men's behaviour change programs delivered by the RA federation. As I said, it emphasizes the similarities and acknowledges some of the differences, and very specifically is grounded in that practice expertise, balanced with research evidence. I'll step through the visual briefly, but it may be a little bit difficult to see the detail on the screen. So, I've also included a version as a handout if you wish to look more closely. Underpinning the whole diagram is the overarching purpose of these programs, specifically supporting men to stop using abusive, violent or controlling behaviour while also supporting the safety of women and children. This framing was deliberate, informed by the expert insights we heard from practitioners about the importance of being explicit about both the violence and the centring of women and children. Importantly, the workshops with practitioners were essential in providing us permission to be this explicit. Above this is the key factors that impact that change. Specifying the assumptions, system factors, practice approaches, principles, and elements that support quality. These draw on insights from the audit workshops and what the evidence has to say. In the centre, here is the Theory of Change, which provides a comprehensive overview of the men's behaviour, the men's behaviour change programs even, delivered by Relationships Australia and outline the features of the programs and alignment with the current evidence base. It specifies the key program inputs, the key program features and aims, the intended program outcomes, the key activities and the partner or ex-partner. Contact and safety planning and supports for women, children and affected family members. Importantly, this theory of change also positions these programs as one part of a wider system and series of interventions that respond to domestic and family violence. This depiction aims to balance a linear depiction of change for ease of understanding with the uneven and nonlinear, complex and complicated nature of change. This we have attempted to reinforce and emphasize, by the way the outcomes are shown with curly arrows that highlight the changes complicated, messy and gradual, and that some outcomes emerge at differing times for differing people. Overall, the real power of this theory of change is that it creates a shared language and framework. It emphasizes the common ground that exists between these diverse program offerings delivered across the country, and it distils this complexity into something easily consumable, while simultaneously drawing together practitioners day-to-day work into a larger picture of how this all contributes to change at a national scale. How will this theory of change be used from here? What are the next steps? Gen.

DR GENEVIEVE HEARD: Thank you Steph. And if we could have slide four, please. I guess that early on in the development of this project, we identified a critical gap in the evidence base for MBC (men’s behaviour change) in Australia. And it's our hope that the theory of change and the resulting evaluation framework will be taken up across the sector so that we can build a more cohesive approach to producing evidence for MBC. But for RA specifically, this theory of change will form the basis for a national evaluation of MBCPs that will assess the impact and outcomes of these programmes. And a key objective is to provide RA with a better understanding of the extent to which these programmes operate, to reduce men's violent behaviour and to increase the safety of women, children and affected family members. The theory of change really provides a template against which we will assess the effectiveness of our MBCP interventions. It means that we can examine whether we're actually succeeding in doing what we hope and what we think we are doing in these programs. Given the differences in service models across jurisdictions that we touched on, this evaluation will also support improvements to our service delivery. The evaluation will explore whether these differences have any impact on outcomes, which is important to the ongoing improvement of our MBCPs and to the underpinning evidence base.

STEPHANIE FISHER: Amazing. Thank you, Gen and Glenn, for joining me today to discuss the collaborative approach we have taken to developing this generalizable theory of change. I think this work is really exciting, and I look forward to the next steps of this project. I'd like to thank my colleague Sharnee, Liz and Kat who were instrumental in delivery of this project. I'd like to also extend my thanks to the AIFS Communication team and the Child and Family Evidence and Evaluation team for your support in delivering today's webinar. And finally, I'd like to extend my thanks to our audience today. I hope you found this useful and interesting. If you haven't already, you can subscribe to the Evidence and Evaluation Support newsletter via the website. We'd really appreciate it if you could fill in the feedback survey that will come through to your email after the webinar. It will take about one minute and the feedback helps us to improve these webinars. You can also continue your learning journey by signing up to attend the conference later this year. The conference is always a personal highlight for me and brings together government, practitioners and researchers to explore how policies, programs and systems need to evolve to support families in all their diversity in a rapidly changing world. We look forward to joining you at our next webinar. Take care and we will see you again soon.