Steph Beckwith: Hi. Welcome everyone to today's webinar. My name is Steph Beckwith, and I'm a Senior Research Officer at the Australian Institute of Family Studies working in the Evidence and Evaluation Support team. I'd like to start today by acknowledging the Wurundjeri Woiwurrung people of the Kulin nations who are the traditional owners of the lands in Melbourne, where I'm speaking to you from, a particular part of Melbourne that I'm in today belonged to the Marin-balluk Clan, who are part of the Woiwurrung. I also pay my respects to the traditional owners of country throughout Australia, and recognise their continuing connection to lands and waters. I pay my respects to Aboriginal and Torres Strait Islander cultures and to Elders past and present. Today's webinar is part of a series designed to share examples of evaluative practice occurring within Australian child and family service contexts, and some ideas for collecting and using evidence. The format is a little bit different to our usual webinar program, in that it will be a brief conversation between myself and two guests.

So today my guests are Jo Smith, who is the local service manager at UnitingCare. Hi, Jo. And Dr Chez Leggatt-Cook, who is the research and knowledge translation manager, also from UnitingCare in Queensland. And today we're going to be discussing the evaluation of the Men's Sustaining Change program. So before we dive into our discussion, I do have just a little bit of housekeeping to cover. Chez, I think I'm getting a little bit of feedback. I might get you to just pop your thing on mute for a second. Um, so first of all, the webinar is being pre-recorded. So because it's not live, there won't be a Q&A aspect. For those of you who require captions, please watch the webinar via the AIFS’ website. There's a link for this in the chat, and a full transcript of the webinar will also be available. There are some related readings and resources associated with this webinar, and you'll be able to access that in the handout tab in in the webinar. And of course this is a really big topic and we can't cover everything today in 30 minutes. So if you do have unanswered questions, those handouts are probably a really great place for you to start. There's also going to be a short feedback survey that will open at the end of the webinar, and we would really appreciate you just taking a minute to complete that survey. This gives us some really good information about how we can improve our webinar program. So let's dive into our topic for today. So today we're talking about the evaluation of a men's behaviour change program that UnitingCare runs. Our audience have probably all heard of men's behaviour change programs, but when I came across your Men's Sustaining Change program, I thought it seemed a bit different to the models that you usually see with men's behaviour change program. It's not something that I had really seen before. So I'm really excited to have you both here today to hear more about the Men's Sustaining Change program and also the evaluation that you've been able to do of the program. So welcome to you both.

And thanks so much for joining me. Chez, maybe to start us off, would you like to share with our audience a little bit more about UnitingCare and the organization's involvement with men's behaviour change programs?

Dr Chez Leggatt-Cook: Well, thanks Steph, and thanks for your welcome. We're really pleased to be here. I also wanted to thank you for your acknowledgement and just recognize that I'm connecting in from Turrbal country north of Brisbane. So just a little bit about UnitingCare. We are the largest NGO in Queensland, established over 100 years ago, and we support the community through a range of health, aged care and community services. And both Jo and myself sit within our family and disability services area, which has about 80 different programs in itself, including men's behaviour change. So we've been working in the men's behaviour change space for around 18 years now. So we do have quite a lot of experience in this space. So we have our core programme, Men Choosing Change and we deliver that in five regions across Queensland. And then of course Men's Sustaining Change, which is our voluntary follow-on program after men have completed men choosing change. Currently that one is just in two regions, but we hope to expand that soon. We also provide court support for men, and we have a specialist counselling service for children impacted by domestic and family violence called Safe Spaces. Have I missed anything there, Jo? I think I've, I think I've got them all. Just to give you an idea of our scope. So every quarter we support over 400 men, over 300 women and over 300 children in in these programs.

Steph Beckwith: Wow. Yeah, that's a lot. And it's a really long history that Uniting has in in this space with men's behaviour change programs. Thanks for that overview of your services, Chez. I might throw. Now over to you, Jo, to explain a little bit more about what the Men's Sustaining Change program looks like and how it differs from kind of regular men's behaviour change program.

Jo Smith: Sure. Thanks. And I'm coming from Gubbi Gubbi country today. It's where I'm sitting in the beautiful Sunshine coast. So the men choosing change programs are a set behaviour change program, as you said, and that's delivered over the 16 weeks. And men are required to attend weekly for the 16 weeks, and some are mandated and others voluntary. And in that group, you know, we're using discussions and activities and psychoeducation to support the participants to develop an understanding of the impact that their domestically violent behaviours are having on the people experiencing it, like their partners and former partners. And we have a woman's advocate attached to men choosing change and that's where we're connecting to partners and former partners, offering information, advice and referral. We're assessing risks for safety planning, sharing information happens really frequently between our team. And children are always a priority as well. So we're constantly risk assessing. So that's sort of the choosing change program and the Men's Sustaining Change program. This group is for men who have completed MCC (Men Choosing Change), as you heard Chez say. So Men Sustaining Change works only with the men who have really explicitly expressed their commitment to deepen their journey towards nonviolence, and they've really accepted accountability for their past behaviour. So it's really aiming to support men with the positive changes and strengthen and deepen and sort of adds to their tools and strategies that they've gained through Men Choosing Change program. It is held weekly, but men don't have to attend weekly. And while we have similar approaches, it's not a set program. Instead, we glean our topics from the men we explore what their goals and hopes are, and we deliver our groups around that focus. And another key difference is that we deliver social connection events like bowling, and that assists to strengthen positive peer supports between them. We also connect men through introductions with other services, and we do that by hosting guest speakers at groups. So we've had financial advisors, housing advisors, counselling professionals. But that the identification of that speaker, it really comes directly from the men. It's their request and their identified need. And we also support social interaction through food. But if I could, I think it'd be good to explain it through the eyes of a client. So if you let me introduce you to Carl, it's not his real name, obviously. So he's a father and a partner to Jessie. And they're parents of two children. And domestic violence has been a real feature of their relationship. And during his pre-group assessment for suitability to men choosing change, he was supported to address his substance misuse. Prior to attending. He went to rehab, he completed it, he came back and he did complete men choosing change. But in group of men choosing change, he'd sit there and he'd have his arms folded and he'd had his hat over his eyes and he rarely spoke. And then at his sixth session in that group, a new man came, Dave, and he talked about his domestic violence order and his situation and how this largely being other people's fault. And he was system blaming. Dave was questioning whether he should even be in group. But it was Carl who spoke up. Carl said when he started group, he was blaming everyone else for what happened, blaming his former partner, the child protection system. But he now recognised that he needs to be responsible and what his part was and why. And he actually said, I built a good foundation now. And this was because of group. It works man. So he'd really taken on that learning from group. And he's on this change journey and we could even see physical changes. He didn't wear his hat. He had open body language. His arms were unfolded. You could see his head up. He'd even smile at times. And he as I said, he did complete the group. But at that ending of the group, we have a transitioning out of men choosing change. And it was there where he said he felt he'd changed. He was feeling more confident about himself, his behaviours, but he was really worried about maintaining this progress without support. He knew he didn't want to access that 1 to 1 support. He wanted group work. Even if he could just drop into Men Choosing Change now and then, which we couldn't support. He shared he didn't have a good peer basis either. He wasn't trusting of services. He was scared he'd return to drugs. So we have this man. He's really on this journey. But he didn't have the support to access that he felt comfortable or safe with. So Men Sustaining Change is really developed for clients like Carl. Men who have really expressed a want and a need for more support over that longer period of time where he could feel connected to other men, where he could become more confident to access services that he needs. So really, Men Sustaining Change is really for men like him. It's this more flexible group for men who are really now self-motivated and want to maintain changes, and we know how valuable that is for the safety of women and children. So yeah, men like Carl really benefit from Men Sustaining Change.

Steph Beckwith: Yeah. Thank you so much Jo. That really helps to kind of give a sense of what the program looks like and what, you know, participants journey might look like. And I mean, it's a lot to expect significant behaviour change from people within a 16-week period, isn't it? So I can really see the value in kind of being able to provide that extended period of support for men, and particularly those men who are really committing to, to doing further programs. Yeah, thank you. That's a really great overview of the origins of the program. So I'm curious to hear a bit more, Jo, about how the program came about and what led UnitingCare to go down this path and kind of develop an extended version of Men's Behaviour Change program.

Jo Smith: Sure. So a few years ago we actually evaluated Men Choosing Change, and our evaluation results really backed up what clients like Carl were actually saying. We found that most men were externally motivated to attend. Typically they're mandated by family court or by probation parole conditions. And it does take time to internalize motivation and accept responsibility for their abusive behaviour. We know that Men Choosing Change can act as a catalyst for change, but for many men 16 weeks wasn't long enough to embed the new ways of thinking and acting. So this is what the evaluation was coming out with. And we found that many were doing well, at follow-up. So when we spoke to them 6 to 20 months post program, they'd typically accessed additional support, for example, mental health and substance abuse. The evaluation also found, that there's a gap in the service system around supporting women for longer term recovery, for children's longer term recovery. Our findings were really backed up by studies, other studies that have shown that changing violent behaviour takes time, and it was backed up by years of anecdotal evidence from our clients and practitioners that men and women really needed more support in this space. I think it's really important to understand that, given the seriousness of the behaviour and its impact, these programs, they're fairly brief interventions, and we really need to be realistic about what they can do. In 2019, we put together a working group of really passionate, experienced staff and managers like Chez and myself, along with research staff to co-design the program for Sustaining Change. And this went for several months. And once we had this clear model, we were really fortunate to be able to pitch it, but we got knocked back essentially. And we also submitted a couple of grant applications. Unfortunately, they were unsuccessful. And the message from both of these sources was that they really weren't prepared to fund a follow up program until we had evidence that we were producing results. And we were evaluating at the time, but we didn't have the figures yet, the findings. So eventually we were approved to fund a pilot program, a pilot of MSC and evaluation through our internal innovation scheme. And in 2021, we proudly started our two-year pilot at our Maroochydore site. And in 2023, so two years ago, our existing Men Choosing Change had funding received and we got it approved that we could also run our Men's Sustaining Change program, and not only in Maroochydore. We've also been able to extend it to a second location in Caboolture. And excitingly, it's looking like it'll be a third location in Ipswich as well. 

Steph Beckwith: Well thanks Jo. That's really great to hear and I'm glad you were able to kind of continue on and manage to get that funding as well. That's really excellent news. Yeah, that's wonderful. Chez, I wonder if you would like to tell us a little bit more about that evaluation and kind of how you went about doing the evaluation? And yeah, I mean, you mentioned in there, Jo, that you used a co-design process as well. I'd be really keen to hear a bit more about that too, and kind of how you went about doing a co-design process for this evaluation.

Dr Chez Leggatt-Cook: Sure. Thanks, Steph. So yeah, really happy to share what we did with the evaluation, but I just wanted to point out that I am not the evaluator for the program. My role was to work with our external evaluator, Dr Sue Carswell, from the Queensland Centre of Domestic and Family Violence at Central Queensland University. So she can't be here today, unfortunately. So I'll be talking more from the perspective of an evaluation partner. But I was quite heavily involved. I think if it's okay, I would like to give a bit of the background about our first evaluation of Men Choosing Change, because I think we learned quite a lot about how to work in this space. And it really informed the approach that we later took with the Men Sustaining Change evaluation. So just to kind of go back quite a long time, 8 or 9 years ago, we were doing quite a lot of practice development in Men Choosing Change, and we were really confronted at that time by how difficult it was to identify some solid evidence to inform what we were doing. And ANROWS at that time had published their State of knowledge paper, that showed that there was very limited Australian research, and the evaluations that had been done were mostly small scale and not generalizable. So the state of evidence overall was quite poor. And evidence of effectiveness was very mixed. And I think this is a challenge that we still have in this space, although I feel like things are moving a little. So we saw an opportunity to not only do some evaluation to strengthen our practice, but also to be able to share what we had found and hopefully make some sort of contribution to, to what we know about these programs. So, fortunately, we were, our leaders were willing to put some money behind this. You know, not a lot of money, but enough to bring in somebody with expertise in this area. A really good track record. So, our partners at Queensland Centre for Domestic and Family Violence Research, and the first evaluation was led by Professor Annabel Taylor. Dr Sue Carswell was also in there at that time. And she actually did the long-term follow-up component of the evaluation. And we did it over a four-year period, which was really long. It was interrupted a bit by COVID that pushed out our timeframes a bit. But it was also a very steep learning curve, and it was much more challenging than we had expected. And we were very ambitious, you know, we had a process evaluation in there to look at service delivery. We had our outcomes evaluation, looking at initial outcomes at, program exit for men, women and children, as well as a follow-up component to look at how things were going up to 18 months post program. So, all of that learning was very much taken into the next evaluation. And, again, we're very lucky that UnitingCare decided that it was worth not only funding the pilot, but funding the evaluation as well. I think we really believed in the work our findings from Men Choosing Change, which really backed up that need for a longer term support option. And I think we really believed in it and wanted to know if it was going to work. So in terms of co-design. So it was a pilot, so we had to do an evaluation, sorry, a developmental approach. And while there were some targeted co-design activities, I feel like co-design is a general approach was woven right through everything that we did. And one of the reasons why we chose to work with Sue and the Queensland Centre for Domestic and Family Violence was because there was an excellent alignment with our ways of working, and it really felt I think, you know, Jo and I have talked about this at length. It really felt like Sue was walking alongside us as the service delivery team for the whole time. So we had monthly meetings with Jo and with her manager, Jo Bierman, and with Jen Williams, who's our wonderful practice improvement advisor in this space. So Sue was able to share her findings as they were emerging. We were able to really unpack implementation challenges with her, and she brought a wonderful spirit of curiosity to everything and had lots of creative ideas about what we could try when things were getting really tough with implementation. There was a lot of trust in that relationship. And Jo and I, you know, have come to think of her in that space for us as a critical friend. She was able to ask those curious questions, and we were able to benefit from that and constantly adjust both the implementation and the evaluation. So alongside that, we also did a Theory of Change co-design workshop right at the start of the evaluation. That was online because of COVID. It involved external stakeholders, you know, the partners that we thought we would be referring men to, and women too, so some really key stakeholders, as well as all of our staff involved with this pilot. Out of that, the evaluation methods were co-designed with Jo and her staff. And then we sought ethics approval. And 12 months into the evaluation, we had a second Theory of Change workshop with the same group of stakeholders and staff. And this time it was looking at what the emerging evidence was showing, and we used that to tweak the outcome statements. So yeah. So there's specific stuff, but also a general co-design approach I think.

Steph Beckwith: Yeah, great. And it's great that you were kind of able to do that, along the process as well. Not kind of just at the start, but you continued that engagement with people. Yeah. Great. And I'm also kind of wondering what sort of methods you used, like with the evaluation itself. Yeah. Like what methods did you end up deciding to use for that? I mean, I think Jo did mention earlier that you did interviews with people. Would you like to speak a little bit more to the methods you used?

Dr Chez Leggatt-Cook: Sure. So it was a mixed methods approach, with a range of quantitative and qualitative tools. So we had brief feedback surveys that clients were invited to complete every eight weeks. There was a separate survey for men and one for women, and qualitative interviews with both men and women. Fourteen men agreed to be interviewed through the evaluation, but some had two or even three interviews, which helped to get a sense of their experiences over time. The inclusion of women's voices I think is really critical for any evaluation in this space, because these programs exist to increase the safety of women and children. So I like to think of their voices as our reality check. We had eight women who agreed to have at least one interview, and some of them had several interviews. So they were able to speak about, you know, their perspective about were men really changing and provide feedback about the support they were getting from the advocate. We weren't able to interview children, but both men and women were asked about their children's feelings of safety. So we also had a high-level aggregate program data about demographics and outputs and things, and routine outcomes measurement. Any client who consented to an interview, we also sought consent for their de-identified case file to be provided to Sue. So that was useful for providing a fuller picture of their story. There were staff interviews. All staff were interviewed, I think at least three times, Jo through the evaluation. And we also had external stakeholders, and many of them had been guest speakers. So they were they were able to give the perspective from outside UnitingCare about how they thought the program was going.

Steph Beckwith: Well. Yeah. It's really a range of methods that you used and were able to get really quite a broad range of data. Yeah. It's great. And so good that you were able to do some in-depth interviews as well, I think with not only men, but also women in this space that is really important. And I imagine being able to kind of do those qualitative interviews would have helped you to get some really strong, in-depth insights into the kind of impact that the program would be having. I'm curious to hear how you went with recruiting participants to those interviews, because I think it can sometimes be a bit tricky recruiting for interviews. It's I mean, it's quite a commitment for people to commit their time to that and also feeling comfortable, to do that. How did you go with recruitment for interviews?

Dr Chez Leggatt-Cook: It is a really good question. I think especially in, in this space or any space where it's difficult to engage people to hear their lived experience. So I would never say that it was easy, but I think it was easier than recruitment in our previous evaluation. And I think that had a lot to do with the fact that the men are in a different space in their journey and they're more reflective. And I think that the interviews or the offer of the interviews was like another opportunity to reflect on how far they had come and where they wanted to go. So overall, our sample is still quite small, but I think it's important to note that this was a pilot implemented at one site, and we were only accepting referrals from our core program on the Sunshine Coast. So there were only 68 men who were referred during the two-year pilot, and of those, only 51 were assessed and accepted as suitable. Then only 39 of those actually attended one or more group sessions, and had the opportunity to be recruited. So when you consider that we had 14 out of 39 men, that's actually about one third of our program participants who gave at least one interview. And that's actually pretty good when you look at other evaluations in this space. Yeah. So we were, yeah shall I keep going?

Steph Beckwith: Yeah, yeah. Go. Sorry.

Dr Chez Leggatt-Cook: Sure. So we were reliant on staff very much to support recruitment. So they would provide information to clients, ask any initial questions they might have, and then seek consent to pass their contact details to Sue. So Sue then handled the informed consent and it worked pretty well. But I think I'm more and more conscious of the burden on staff when they're asked to play this sort of role because they're, you know, their working life is just so busy. And on reflection, I think there's some things that we could have done to reduce that burden. Overall, I would say that women were harder to recruit, and that is the same across both evaluations. We relied on the advocate to introduce the research, the evaluation to the women. But we were very, very careful to ensure that the advocates knew that we trusted them to use their professional judgment about whether it was appropriate to bring into the conversation. So we didn't have any targets but we did have some regular check-ins to discuss the challenges and try to problem solve, but also to keep it front of mind for staff.

Steph Beckwith: Yeah. Great. And it's yeah, great that you were sort of still able to include some women in those interviews, even though it's, yeah, a trickier space. I'd really love to hear as well a bit more about the findings from the evaluation and, and what you found.

Dr Chez Leggatt-Cook: Okay. So I thought that I would talk about some findings from men and then findings around women and children, and then briefly about some of the program design elements. So, hopefully I don't go too long, but I will try and keep it short. So the majority of the men that were involved reported that the program was helping them to continue changing their behaviours and develop as a person. So they often talked about, aligning and realigning their core values, and increasing their awareness of themselves as a person. They continue to learn more about the behaviours and patterns that constitute domestic and family violence. And that was a learning process that was continuous, as was understanding the impact of those behaviours on others. They were able to really explore their perceptions of masculinity and relationships, and they were developing compassion and empathy. They were more conscious of how their behaviours were affecting their children. And a lot of them had engaged with parenting programs to get some more support in that area. And they were also reporting some positive changes with their communication, and that was with their partner and children or ex-partner and children, but also with others such as their work colleagues and family. And they all seemed quite aware that this sort of change does take a long time and it's not easy. And they're very conscious that they were still on their journey. For women and children, the findings were more mixed, which again is very common in this space. And some of the women still felt unsafe or thought that their children still felt unsafe. Our sample of women was small, but it did seem to us that where women had stayed in the relationship or had returned to the relationship, they were the ones who could more clearly see the impact of the program. Whereas where there had been a breakup, there often wasn't a lot of contact, so there wasn't any information to kind of inform a change in women's feelings of safety. But even when women thought that the program was working well and they all thought that it was beneficial, they still expressed a lot of caution about the depth of men's changes. And was it really going to be sustainable. So I just think that that is just, really reflective of the journey that women have been on and the time it takes also for them, to really trust what's happening. They rated the support from the advocate really highly, which was great. In terms of program design, it was really interesting for us to find that overall, what we had designed and intended to deliver actually worked pretty well, and it was quite close to what we had intended. The program really helped me to build on what they had learned at Men Choosing Change and to go a lot deeper, to unpack they had time to practice strategies outside group and then bring those reflections back to group. The connection with referrals to support services was also a really strong finding. And the guest speakers were especially successful for enabling that. And we also found that the program provides a way to get a much longer term monitoring of safety and risk for women and children. So it helps to keep the whole family in view for a much, much longer period. And enables many more opportunities to connect everyone and to support. And it was good to hear that our external stakeholders were also very positive about the program and highlighted that, it really built capacity into the service system because so many of our services are focused on that initial crisis after DV, and there's not a lot available beyond the 3-to-6-month period.

Steph Beckwith: Great. Thanks, Chez. That's yeah, a really broad range of findings that you've got out of that data. It's yeah really great to hear and again, good that you were kind of able to hear from men and women and get those outcomes. And as well looking at process and thinking about how the program ran and getting those evaluation findings on the program itself. We are getting really close to time. But, I'm also keen to just really quickly ask you one last question. If you've got plans for how you're going to use those evaluation findings from here?

Dr Chez Leggatt-Cook: Yes we do. In fact, just on Monday we had a full-day workshop with all of our men's behaviour change staff. So that includes the staff that are already delivering Men Sustaining Change, as well as the ones who we hope will be delivering it in the future. So we were able to look at the evaluation findings. Sue was able to be there and she presented on that. And the purpose of the day was to understand how we could link those findings into continuous improvement. So we ended the day with a commitment to create three quality improvement projects, or quips, as we prefer to call them. So we're going to develop a facilitator handbook to articulate the practice framework a bit more clearly, which I think will really help when it comes to expansion to other sites. Our advocates have identified that they would like more access to resources that they can share with the women, because that longer term period of engagement is really helping to identify many more areas where women would like some information. And we also want to do something that's a little bit unclear at the moment, but we want to do something that's a bit more intentional around the goal setting for men when they first come into the progress, first come into the program to help track their progress and enable us to make a better judgment about when it might be time for men to start thinking about transitioning out.

Steph Beckwith: Oh, great. Yeah, it's really great. You've got some really practical ideas about how you're going to use the evaluation, for continuous improvement. We really love to kind of hear and share examples of evaluation that's really being used to help improve practice. So yeah, it's great to hear, I could continue asking questions about this, but I think our 30 minutes is nearly up, or maybe up. So I think we'll have to leave it here. So I just want to say thank you so much Chez and Jo for joining us and sharing about the evaluation of the Men's Sustaining Change program. It's been really interesting for me to hear about it all. So thank you so much for taking the time to share your experience with me. I'd also like to thank the audience for coming along today and being engaged with this new webinar format that we're running. Hopefully it's been an enjoyable experience for you. And I'd also really love to thank our amazing communications team, who are doing everything in the background to make this webinar happen. These webinars really couldn't happen without them. So a huge thank you to everyone involved. Please go and subscribe to our Evidence and Evaluation Support Team newsletter to hear more about what's going on with our team. And please also fill out the feedback survey when this webinar closes. Being evaluators, we're really keen to get your feedback so that we can improve our program as much as possible. So I'll leave it there. We really look forward to you joining us at our next webinar. Take care and we'll see you soon.