STEPH BECKWITH: Hi everyone, and welcome to today's webinar. My name is Steph Beckwith, and I'm a Senior Research officer at the Australian Institute of Family Studies working in the Evidence and Evaluation Support team. I'd like to start today by acknowledging the Wurundjeri, Woiwurrung and Bunurong people of the Kulin nations who are the traditional owners of the lands in Melbourne, where I'm speaking to you from. The particular part of Melbourne that I'm in belonged to the Balluk clan, who are part of the Woiwurrung. I also pay my respects to the traditional owners of country throughout Australia, and recognise their continuing connection to lands and waters. I pay my respects to Aboriginal and Torres Strait Islander cultures and to elders past and present. So, today's webinar is part of a series designed to share examples of evaluative practice occurring within Australian child and family service contexts, and ideas for collecting and using evidence. The format is a little bit different to our usual webinar program, in that it will be a brief conversation between myself and a guest.

Today my guest is Lena Etuk, who is the Director of Research and Evaluation at Culturally Inclusive Research Centre Australia, or CIRCA. And today we're going to be discussing culturally inclusive evaluation. Before we dive into our discussion, I do have a little bit of housekeeping to cover. So, first of all, this webinar is being pre-recorded. So because it's not live there won't be a Q&A aspect. For those of you who require captions, please watch the webinar via the AIFS website. You'll find a link to that in the chat, and a full transcript of the webinar is also going to be available. Of course, this is a big topic that we're covering today and we can't cover everything. So, if you have some unanswered questions at the end of this, yeah. The handouts on our website are probably a good place to start. There's also going to be a short feedback survey that will open at the end of the webinar, and we'd really appreciate you just taking a minute to complete that survey. It gives us some really good information about how we can improve our webinar program.

So we might jump into our topic today. Today our webinar is going to discuss cultural inclusive evaluation and CIRCA’s approach to culturally grounded in-culture and in-language evaluation. We at AIFS often hear from practitioners that they really want to be doing work that's culturally inclusive, and they see the importance of it, but they don't always know where to start. And I heard you speak Lena at the Australian Evaluation Society conference last year about CIRCA’s approach to culturally inclusive evaluation. And I thought you had some really practical ways that people can try and apply these practices to their work. And so we thought it would be really great to invite you along to one of our webinars to chat further about the work that you do. So to start us off, Lena, maybe you could tell me a little bit more about yourself and your evaluation experience.

LENA ETUK: Yeah. Excellent. Thanks so much, Steph, and thanks so much for the opportunity to speak with you today and talk to AIFS folks, I really appreciate it. I'm also dialling in from the land of the Gadigal people of the Eora nation. I'd like to pay my respects to their elders, past and present, and recognize this land was never ceded. Therefore, it always was and always will be Aboriginal land. Yeah, as you could probably tell from my accent, I'm not from Australia originally. I'm from the US originally. And so my evaluation experience has spanned Australia as well as the US context, which is a great opportunity for me. I've been in evaluation for 19 years. I started off as a sociologist. I'm trained as a sociologist and I started working at Oregon State University. And as many evaluators do, we kind of just fall into evaluation. And that's what I did. Yeah, I was doing applied sociology and had an opportunity to start doing some evaluation of a rural leadership development program in the US, particularly in Oregon. And that's how I fell into it. And it was a multi-year evaluation. And then along that journey, I did other evaluations and then moved to Australia almost nine years ago. And in Australia have been doing, outcomes measurement, evaluation, working for a nonprofit, doing evaluation of government programs and policies and things like that. So really kind of been all over the place in terms of evaluation.

STEPH BECKWITH: Awesome. Yeah, it sounds like some great experiences that you've had. And I understand that now you're working at CIRCA, and you've been really focusing on doing culturally inclusive evaluation at CIRCA. Would you be able to kind of share a little bit more about the work that you do at CIRCA?

LENA ETUK: Yeah, absolutely, absolutely. So, yeah as you said, of course, CIRCA stands for the Culturally Inclusive Research Centre  Australia. Um, we used to be known as the cultural um and Indigenous Research Centre Australia, but we changed our name last year to really recognise the inclusivity that our approach really can, can offer to many cultural groups. Um, so But because of our name and who we are, culture is like a really, really important part of what we do. And so all of our research and evaluation is culturally grounded. So that means we do all of our data collection in culture and in language, really to make sure that we're inclusive of some of the people who have the most to gain from social programs and initiatives. And the way we deploy that cultural grounding is really thinking about culture at every phase and every stage of the evaluation, effort or endeavour. So when we're thinking about design and evaluation, we're thinking about culture and the cultural considerations of people.

When we're thinking about how we're going to recruit participants, we also bring cultural considerations to bear. And then, of course, in how we collect the data, the questions that we're asking, the way that we're asking the questions, the context in which we're asking those questions. We think about culture and the needs of participants from diverse backgrounds. And then finally, in analysis and reporting, we're also thinking about how are we presenting the results in a way that's empowering to cultural communities, not playing into stereotypes and really being open and frank about structural conditions that different marginalized communities face all the time. And it's just been such an honour to be able to work at CIRCA, where I'm able to bring my sociological expertise, but then also bring myself to this because of my own cultural background. Yeah, it's really really amazing.

STEPH BECKWITH: Yeah. Oh, wonderful. Yeah, it sounds like some really great work that you're doing there. And it's great that you're able to kind of have that really clear focus on, on culturally inclusive practice. Thank you for sharing. I wonder if maybe you'd be able to kind of share with us, an example of some work you've done to kind of help us. To kind of understand a little bit more about how you really apply some of the practices, the principles that you work on at CIRCA.

LENA ETUK: Absolutely, absolutely. Yeah. So we've done many evaluations, over so I've been at CIRCA for just over five and a half years, and I've worked on 19 evaluations at CIRCA in that timeframe. What that's meant is that I've been able to get this really, really rich, sort of experience and sort of how we how we use our methodology and what that looks like. And so just one example is we did an evaluation of a service program implemented by New South Wales Department of Communities and Justice, or DC, and it was really focused on, child and family well-being. So it was called Brighter Futures Voices and Choices. That was the name of the program. And so what we did, and it was, the program itself was a voluntary targeted intervention to families where concerns of risk of significant harm had been raised. And so the voices and choices element of the Brighter Futures program, it was a trial to really try to see, how the Brighter Futures program could be adapted to make interaction with families a little bit more proactive, to really work with families, to plan decision making and really sit alongside them and work alongside them to improve, you know, the family.

So it really was trying to be proactive with families. And so we worked with the Centre for Evidence and Implementation on this evaluation of this trial program. And so what we did is we did an implementation evaluation as well as an outcome evaluation. You know the implementation is we want to see how the trial is rolling out. And then with the outcome evaluation, we'll understand the impact of the program on parents as well as the children. And so CIRCA’s role was really focused on the clients experiences, the parents experiences. They were the ones getting the direct support from the service provider. And so we wanted to look at how satisfied they were with the support they received. The relationships with caseworkers, whether the services they got offered, were offered in a culturally appropriate way. And that's another thing that we always make sure that we're very cognizant of, because we bring that cultural focus. If we're not asking the question of is the program culturally appropriate, then we're already missing the mark. We're already behind. Behind the eight ball in terms of that cultural consideration. So yeah, we did 36 interviews with parents, who were recipients of this program in metro regional and remote New South Wales. And so we did interviews with Aboriginal and Torres Strait Islander parents, parents from culturally and linguistically diverse backgrounds, and then also parents from more mainstream backgrounds as well.

STEPH BECKWITH: Yeah. Oh, wonderful. Sounds like a really interesting evaluation that you did. I'm just, I guess I'm wondering if you could kind of share a little bit more detail around how it is that you really made that evaluation work for the participants that were from Aboriginal or Torres Strait Islander backgrounds, or from culturally and linguistically diverse backgrounds. What are some of those practices that you use within that evaluation to really make it work for them?

LENA ETUK: Yeah, absolutely. Absolutely. Well, we at CIRCA, we've got kind of a bit of a checklist sort of process that we can go through. And, you know, of course, as I said before, you know, we're thinking about evaluation, all the phases of evaluation, those big aspects of evaluation we're bringing culture to bear. And so what we kind of do is we have some considerations that we ask at each of those phases. So it's a bit of a checklist. So we've got like about eight items on this checklist. And it's very much still a work in progress. But those, those eight items cover things like the first thing is of course, recognising that people from diverse backgrounds are going to be included or that they should be included in the evaluation and that they actually need to be accommodated. The second item is around considering and minimising risk to individuals. The third is around considering and minimizing risk to cultural communities. The fourth is around using culturally appropriate data collection methods and making sure that they are actually appropriate. The fifth is considering and working around cultural timings, events, calendars of sort of activities, as we're headed into Ramadan. Right? This is a period of time where we need to be very conscious of folks from Muslim backgrounds, participating in research and evaluation. We also think about the question style that we're using, making sure that it's culturally appropriate, that we're not reinforcing stereotypes, and that we're using culturally appropriate terminology. And so for this particular evaluation, you know, at the start, we thought through those check checklist items, you know, we think about the recognition piece. So fortunately for this particular evaluation, the request for proposal that came through, the client had already done some homework, and recognized that Aboriginal and Torres Strait Islander kids and families made up about 30% of their clients. So already they knew, right? Which was great. We're not always in that position, but it's always great when the client recognizes that.

And they, even in the RFP, again, recognize that folks from culturally and linguistically diverse backgrounds were part of the population, but they didn't give any estimates in terms of how many and things like that. So in the RFP process, or in the development of our proposal, we had to think about that recognition. So we looked at the data. We also recognized that folks from CALD background, the CALD community or CALD communities are very big and diverse community. You've got folks who are second generation migrants, third generation migrants who still identify with a language background, but their English proficiency is probably pretty high. Then you've got first generation migrants whose English may be not so great or from different backgrounds. And so we recognize that English proficiency is a really important factor for how folks are navigating a service or a program. And so not that low English proficiency was something that we really keyed into at that point to say, you know what? It's not just good enough to say our evaluation will speak to people from cultural and linguistically diverse backgrounds. Let's be specific. Let's speak with folks with low English proficiency and other folks, you know, from other diverse backgrounds as well. So that recognition piece was that's like the first checklist, right? Because if you don't recognize you're gonna you're gonna miss. Right?

STEPH BECKWITH: Yeah. It's a really crucial starting point. Yeah.

LENA ETUK: It is, it is. And for this evaluation we also thought about like the risk to individuals and communities. So we did go through an ethics process for this particular evaluation and many of our evaluations, we do, but we consider the risks to children. And for this particular evaluation, we didn't do interviews with kids, for a couple of reasons. One, kids weren't the ones that are the direct beneficiaries. The parents were the ones directly receiving the service from the you know, they were the ones having the conversations with the service providers. So we felt, okay, that makes sense. But then also, you know, there is a risk to involving children in research. And we always want to make sure that we're not involving people unnecessarily. So, we considered that risk, and also considered the community risk because we are the, the data that we're collecting was going to be representing different cultural communities. And so how is it that we actually have to navigate that? Well, for Aboriginal and Torres Strait Islander evaluations and research, it's really important to get Aboriginal community consent to participate. It's not just about individual consent, it's about the community's consent. So we had to navigate that and we navigated that for other evaluations as well. And you have to think about doing that.

STEPH BECKWITH: Sorry. Just to jump in. How do you get community consent rather than like. Yeah, because I guess most of us do kind of think about individual consent, rather than it being on that community level.

LENA ETUK: Yeah, absolutely. Yeah. It's a really important question and there isn't a hard and fast rule around this. Every ethics committee, every Aboriginal ethics committee, depending on the shape of your program and the ethics committee is going to have a slightly different sort of requirement. So it is really important to understand the context in which you are working, or the state and territory in which you're working and the ethics committee's requirements. But for this particular project for New South Wales, that community consent can be demonstrated a couple of different ways. Actually for this one, we would have had an Aboriginal advisory group, who actually reviewed our evaluation and provided input into it. Yeah. Other, other times you might do an evaluation where the actual service providers are of the program you're evaluating are an Aboriginal community controlled organization. And so if you get their consent, that is community consent. So it really depends on the structure of the of the program and the requirements.

STEPH BECKWITH: Yeah. Thank you.

LENA ETUK: Yeah. No worries. Yeah. No I mean it's a fascinating and also I should also say that ethics process it's always changing. There's no like, I mean I think most ethics committees like there's evolution, right? We recognize that research that was done ten years ago might not be up to scratch for how we would ethically do it these days. And the same goes for Aboriginal ethics committees as well. So it's really important to sort of keep your fingers on the pulse of how those ethics committees, those requirements change over time.

STEPH BECKWITH: Yeah, a really important point.

LENA ETUK: Yeah, absolutely. And also for this evaluation, our data collection methods were also really, really important and that was key to how we were, how we went about this evaluation. So for this particular one, we did, as I said, interviews with the parents, and we did interviews in language and in culture. So the interviewer was from the same cultural background or linguistic background as the participants, unless a participant indicated they actually didn't want that type of pairing. There are situations, particularly in this one, you can probably imagine, right? We're dealing with vulnerable families who've been flagged, for risk of harm to children. And so there are small communities, small ethnic communities that, even though of course, we guarantee confidentiality, there's still a little bit of concern around if I bring this up with someone from my same cultural background, there's still a little bit of a worry that, you know, that might get out, even though, of course we're confidential and nothing is disclosed. But there's still a little bit of that risk, that participants perceive. And so to mitigate that, we gave participants the choice, when we screened them. And when we sort of started recruiting them, would you like to be interviewed by someone from your cultural background, or would you prefer not to be? And then they were able to nominate.

STEPH BECKWITH : That’s a really important point to kind of give that choice to people, rather than just kind of assume that they will definitely want someone who speaks their language.

LENA ETUK: Yeah, exactly. And that choice piece is also really, really important to us at CIRCA. We really try to meet folks where they are. And what I mean by that is like physical like if, if they want to be ii an interview, if they want to do an interview in person, we'll meet them wherever they want to meet. A cafe, a library, a community centre, a restaurant, or what have you. Or if they prefer, I mean, folks with kids, oftentimes it's actually easier to do, like a zoom interview because the kids in the background playing over here, and they can just be, you know, talking to the interviewer and still keep an eye on what's happening. But really giving folks that choice is really important. And flexibility. So as we did our data collection, you know, we as I mentioned before, we made sure to ask questions about are the services that are being delivered. Were they done so in a culturally appropriate and safe manner and really unpacked that with folks to say, okay, to what extent, you know, did your service provider help you connect with your culture? Did they did they prioritize your cultural ties just as much as, you know, achieving like a parenting type of skill or something like that. And so, that was a really critical part of the way that we asked the questions, the actual questions we ask and our methodology as well. So, yeah, those are just a couple of ways that we kind of apply that checklist to this particular evaluation.

STEPH BECKWITH: Yeah. Thank you. Thanks for sharing that. It's really helpful to get a clearer sense of how you would apply those checklist points. And I really love the idea of a checklist actually. And being able to, kind of every time you're working on a new program or throughout working on a program, being able to have those points. So kind of, you know, go back to and reflect on how are we going with each of these different steps that kind of provides that point for reflection? It's yeah, it's really handy, I think.

LENA ETUK: Yeah, totally.

STEPH BECKWITH: You were talking about that you kind of provide interviews in language and you know, where giving that choice to people, but where you can, or where people prefer to really doing interviews in language, which seems like a really important part to me, kind of communication and being able to speak to people in a language that they, they feel most comfortable speaking in is a really important part of supporting engagement and people to being able to participate. But I'm curious to know how you kind of managed to do that? I guess, like, you know, there's a lot of languages spoken in Australia and, yeah, I mean, you could have people from all different backgrounds working in a project. So, yeah, I'm curious how you manage that?

LENA ETUK: Yeah, absolutely. Carefully is the short answer. No, I mean, so the way that we manage it is very unique to our organisation and our research firm. But there's also some lessons that can be sort of applied to folks that don't necessarily work, you know, here or with us. But the way that we go about it is we actually have a network of research consultants across the country. And that network can be tapped at any moment to be able to deliver data collection or deliver research in that language. And so we have worked over 25 years to cultivate this network. And our network right now is about 250 research consultants across the country who are native speakers of over 50 languages and who are Aboriginal and Torres Strait Islander. And so with that network, you know, we're constantly growing that network as well. So if anyone who's listening to this as anyone, with a sort of research interest and background, who has language, who's a native speaker of non-English languages, get a hold of us.

But yeah. So that's how we that's how we do it. Because we couldn't maintain a full staff that represent all the languages. There's also emergent language groups and communities. So you have to be responsive. And so with that network we're building it and we cultivate it and make sure that they've got, you know, all the research skills and capacities that they need to be able to do to work on projects. But yeah, we're able to mobilize it in that way.

STEPH BECKWITH: Yeah. Oh that's great. I mean, I guess it's yeah, it's really about those relationships, isn't it? And it does take time. I suppose you're at a point now where the organization's been going for a period of time, where you've been able to develop those relationships and have those connections to be able to do that work? Yeah. It's wonderful.

LENA ETUK: Yes, yes. Relationships are key to everything. Yeah. But it's also like. I think it's key to make sure that our, our research firm, succeeds. But it's also relationships are also just it's like such a core element of doing culturally appropriate research. It's recognizing that relationships and relational relation building through the evaluation research process is just as important as the research outcome. And that has been like for me, that's been a bit of a shift, coming into this work from more, you know, sort of traditional standard mainstream kind of evaluation and research. It's really recognizing relationships need to get built through this. It's just as important as everything else.

STEPH BECKWITH: Absolutely. And that does take time, though, doesn't it? The relationship building and I think often about trying to be able to build that time into projects that you're doing as well.

LENA ETUK: Yep.

STEPH BECKWITH: We've been talking kind of quite a bit about the process of when you're doing an evaluation at the time. I'm wondering, like, I'm just wondering a little bit more about when you're at those earlier stages of setting up an evaluation project and developing your monitoring and evaluation plan and things like that? For this project, did you know, were there cultural considerations that you thought about at that phase as well?

LENA ETUK: Yeah. For this particular project, which was actually kind of unique, we didn't have to develop an M.A. framework or a monitoring and evaluation framework for this one. The DC at New South Wales, New South Wales Department of Community Justice had already created one. So our evaluation consortium could kind of just slot in and just implement the data collection, or, you know, implement the evaluation. But typically, as I said, this one was unique. Typically we are developing, a monitoring and evaluation framework. And yes, absolutely we think about the cultural considerations as we're developing the logic model, as we're developing the theory of change or the reporting plan. And we're also thinking about it in the stakeholder engagement sort of aspect of, of an AA framework. Who are the stakeholders we're going to engage and how? Recognizing that, you know, different cultural communities are going to have a different, different stake, and recognizing that they need to be engaged in an appropriate way.

So, absolutely, yes, we do think about those things. I mean, one of the things that, I often think about when we're evaluating the big programs that are sort of whole of population programs, very rarely or I guess I should, I should critique, I should just say what my, what I like to do is when we're thinking about, developing a logic model is, identifying equity, outcomes in that logic model. Because, yes, we say, you know, that this program might be delivered to, you know, a pretty large, let's say, all let's all New South, all children in New South Wales, right? That is a very diverse population. So in your logic model, as you're thinking about how you're going to be measuring things, measuring that equity, which kids are succeeding, which kids are benefiting the most, is it by demographic characteristics? Is it by socioeconomic status? Things like that. Bearing those kinds of questions in mind is again part of that recognition piece. Because if we don't recognize and we don't call out the cultural diversity of the population, you're going to miss it because you're not you're not going to be paying attention to it.

STEPH BECKWITH: Yeah. Yeah. Absolutely. Yeah. It's really important to be kind of thinking about these things right from the outset, as you're kind of planning and developing an evaluation to think.

LENA ETUK: Exactly.

STEPH BECKWITH: How you can do it.

LENA ETUK: Yeah, yeah, yeah. And that is, that is like the thing that I, I think that's the pitfall I think that folks fall into most easily. They don't realize that there's cultural diversity until they're already like down the track and doing data collection. They get to like, oh wait, oh, I'm. Suddenly, you know, potentially going to do an interview tomorrow with someone who speaks Mandarin. Oh, shoot. Like I don't speak Mandarin. What am I gonna do here.

STEPH BECKWITH: Yeah. Yeah. Absolutely. Yeah, I think it's. Yeah. A really important point to be thinking about this. really from the outset and thinking about who your community is and who are the people you're going to be trying to speak to. Yeah, absolutely. Yeah. Well thank you Lena. I think we're kind of starting to run out of time getting close to our 30 minutes. But it's been so interesting to hear about your work, and I think it's a really important conversation to be having about culturally inclusive practice and, one that I think should be ongoing. So I'm really grateful that we've had the chance to have you here to have a chat with us today. Thank you so much and for kind of giving me some things to reflect on as well in my work. So, yeah. Thank you so much for taking the time.

LENA ETUK: My pleasure, my pleasure. Absolutely. Thanks for the chat, Steph.

STEPH BECKWITH: And I'd also just like to thank the audience for coming along today and being engaged with this new webinar format that we're running. Hopefully it's been an enjoyable experience for you, and you've had some good takeaways from today as well. And I'd also really love to thank our amazing communications team, who are doing everything in the background to make this webinar happen. It really couldn't happen without them. So a huge thank you to everyone who's involved. Please go and subscribe to the Evidence and Evaluation Support Team newsletter to hear more about what's going on with our team. And please also fill in the feedback survey when this webinar closes. Being evaluators, we're really keen to get your feedback so that we can really try and improve our programs as much as possible. So I will leave it there for today. We really look forward to you joining us at our next webinar. Take care and we'll see you soon.