Setting the scene for implementing Monitoring, Evaluation and Learning (MEL)

KAT GOLDSWORTHY: Welcome, everyone to today's webinar setting the Scene for MEL implementation. My name is Kat Goldsworthy and I'm a research fellow here at the Evidence and Evaluation Support Team at AIFS. I'd like to start by acknowledging the Wurundjeri Woiwurrung and the people of the Kulin nations, who are the traditional owners of the lands in Melbourne where I'm speaking to you from. I also pay respects to traditional owners of country throughout Australia and recognise their continuing connection to lands and waters. We pay our respects to Aboriginal and Torres Strait Islander cultures and to elders, past and present. Today's webinar is part of a series designed to share examples of evaluative practice and ideas for collecting and using evidence in the context of Australian child and family services. The format is a little bit different to our usual webinar program. We're doing a brief 30-minute conversation style between us, between myself and a guest. Just to provide a bit of an overview around an evaluation topic. So, my guest today I'm happy to introduce is Emily McDonald, who is a senior evaluation officer here at AIFS and a colleague of mine. Our discussion is going to focus on how to implement monitoring, evaluation and learning approaches so that we get the evidence that we need.

And for anyone new to monitoring, evaluation and learning or MEL, as we will be referring to it throughout this discussion, we're essentially talking about a coordinated approach to collecting data, making judgments about what worked, who it worked for and why, and learning from the evidence that you're generating. So, MEL is a massive topic. We absolutely cannot cover it in any great detail in this 30-minute webinar. And we will not be getting into specifics around how to actually create and use MEL. That would be a couple of days session, I think. But if that is something that you're interested in, please look at our step-by-step interactive MEL guide. We've linked to that in your handouts. That's a guide that we've recently published. It was commissioned by the Department of Social Services. And it's primarily aimed at child and family services. But it'll be relevant to many others. So, before we dive into our discussion, I do have a little bit of housekeeping to cover. This webinar has been prerecorded, which means there is no live Q&A element. And for those of you who require captions, please watch the webinar via the AIFS website.

There is a link in the chat and a full transcript will be available. We will have some related readings and resources available in the handout section of the GoTo Webinar Control Panel, so please access those during or after the webinar. We'll link to any further resources we think are relevant. And because we're evaluators, there will be a short feedback survey at the end of the webinar. We'd be grateful for you to take some time and respond to that survey. Okay. All housekeeping out of the way. Let's begin. Emily, welcome. Thank you. It's great to have you here. I'm interested to be picking your brain around MEL implementation today, and I want to start the conversation by asking you from your perspective, what makes a MEL approach different from the routine data collection and reporting that most of us do day to day?

EMILY MCDONALD: Sure. Thanks, Kat. As you were saying, MEL is a continuous process. It has three components. The first is to monitor and measure the implementation of your program, gaining insights into what's working and what's not. And that also includes the unintended insights that you might gather. The second is about evaluating your data and your insights, making those judgments, as you said before. And finally, it's about learning. This relates to the kinds of changes and adaptions that are important to make from looking at your outcomes. I always think about MEL as doing MEL and using MEL as a continuous process, but it's different to just collecting and reporting data because it's specifically tailored to your own program and your community. The way to tailor your MEL plans and design your plan is to start with a theory of change or a program logic process, because that helps you to identify your end of program goals and what you're trying to achieve. Then it's working backwards about all of the activities and practices that you're going to need to implement in order to achieve those outcomes.
Once you've got that theory of change, you're able to work out a measurement and strategy plan. That's why it's quite different to just collecting and reporting data.

KAT GOLDSWORTHY: That makes sense. It's like as organizations we get together, we have that theory of change and we decide what's important and what needs to be prioritized and then develop a plan based on that.

EMILY MCDONALD: Yes.

KAT GOLDSWORTHY: I've already said that I'm not going to be talking about how to create MEL because it's quite complicated and it will be different for every organization, but I am interested in how we set ourselves up to do it well. What I mean by that is the systems, practices and processes that enable us to move a MEL approach from theory into action. That can be tricky, and I know you're interested in this. You've got some experience in doing it, Emily. Tell us why you think this element is important.

EMILY MCDONALD: Sure. If you're thinking about the purpose of MEL as having visibility on how your program is performing, getting a performance story for yourself in a continuous way, it's important to be confident that you have a story that represents what's really happening. So, it's about the quality of the story and the quality of your MEL data. What we know about research into implementation is that the successful uptake of new practices in organizations needs more than just training. It requires a set of different processes and some structure to help you know that you're on track. These are the kinds of processes and structures that help you answer questions like, am I getting the data that I need? Is data being collected? It also helps you know whether your practitioners have integrated the MEL practice into their own practice with service users. And it gives practitioners an opportunity to tell you or program managers whether the process that you've put in place for your MEL is feasible for your consumers and your practitioners.

It's a great process to track. The other thing I like about having a more structured approach to implementation is that it allows the implementation to be shared across a number of people in an organization, not just sitting with one person, for example, your internal evaluator. It starts the process with a group of people implementing MEL from the start, which is going to lead to more success.

KAT GOLDSWORTHY: That makes sense. I want to pick up on a couple of points. It's refreshing because when you think about creating monitoring and evaluation plans, there often is someone leading the charge, building that plan, putting the information together and thinking about how it works. So, it's refreshing to know that doing the work maybe doesn't lie with one person, and it is supposed to be part of a broader, collaborative effort because it can be a huge amount of work.

EMILY MCDONALD: It's huge. It needs lots of different levels to it, the implementation. It needs oversight and it needs to be gathering data along those levels.

KAT GOLDSWORTHY: Some of the things I heard you say are around having structure around how we implement MEL and turn that plan into action. So, it's intentional, it has purpose and there is structure. We're potentially talking about developing new practices that people might have to get their head around in order to implement the MEL well and get the data they need to learn from. I know you're going to be talking more about that in a minute. In the context of all of this, where should organizations start? We're talking about implementation processes here, and it is quite separate to creating the MEL. As I've said, the structures, processes and systems. Where should organizations start to build these things?

EMILY MCDONALD: Good question. First of all, that goes to the design of MEL and who is involved in that design. I'll talk more about that later, but I do think it's a good idea to have a broad range of people involved in the design of the MEL, and those people usually end up being the people who help with the implementation and form an implementation team. So, it's probably a good idea to have implementation strategies in mind, and particularly who is going to be involved, while you're designing. It usually works out that it will be the same crew of people in the organization who are involved in both.

KAT GOLDSWORTHY: Would you do these things side by side? If you're in the process of developing a MEL, given there's a bit of a crossover between the teams that might be involved in both, are these activities that you do at the same time, or does one follow the other?

EMILY MCDONALD: I think you've got to design your MEL first because there's a lot to think about, and then come up with a good implementation plan that aligns to the design of your MEL. The reason I'm talking about the importance of keeping implementation in mind is around who is involved: the roles, the people and the stakeholders. Think broadly about who we need to help us design this so it reflects the program. Sometimes that's also going to involve service users in terms of co-designing. But definitely think about those roles, and who in the future is going to be implementing.

KAT GOLDSWORTHY: That makes sense, always having it there in the background that you're going to be doing this next. You were talking about implementation teams around where organizations should start. Is that the place you start?

EMILY MCDONALD: It is. You start with an implementation team, gathering the people who are going to have the most visibility over what needs to be implemented. These are a group of people who are going to be working together over a period of time on the implementation. It does take some resources. When you're gathering your implementation team, it's important to think about the different roles of the team and the governance areas. How are decisions going to be made with the implementation team? Who is going to play what role? The next stage is to develop your plan together as a shared group. That's a plan that has strategies and activities that will help you know that your MEL is being implemented. After that, it's useful to work backwards because you need to have some indicators for yourself around what good implementation looks like in your organization. Implementation is tracked in the same way as we track MEL.

EMILY MCDONALD: We are receiving feedback about implementation and tracking information about barriers and things that are going well with the implementation of the MEL. We're adapting as we go. So, it is a similar process to MEL itself. You need to have a conversation among the group first around what kinds of indicators would tell us that we are doing a good job at implementing our MEL. What would tell us? Sometimes that's things like our practitioners and team leaders are telling us that the MEL design is a good fit and feasible for their work, it's easy to integrate into their practice, and we're getting good feedback from our service users. That might be an indicator that your implementation is going well. Other organizations might say, we'd like to know in the first section of implementation that 60% of our families are using the tools, and we're getting 60% of our families involved in our MEL. There are different indicators that tell implementation teams how implementation is going, but you've got to decide what those indicators are.

To get to that stage, you need to consider readiness. You're checking in with the structures and processes you've got in place in order to implement this particular practice change. Have you got the resources to train staff around the implementation of the MEL? Have you got local champions who are going to be able to provide feedback to the implementation team? Have you got IT systems that are fit for purpose and align to the way that you're collecting data, and are they all in place for people to use? Those are some of the readiness pieces: thinking about the structures and processes you need.

KAT GOLDSWORTHY: And I guess having the implementation team there, the group that is responsible for assessing that, then potentially building that readiness and finding opportunities to build readiness where they are.

EMILY MCDONALD: Absolutely. It's not just training. It's very much tracking the implementation, getting feedback about how it's going and changing it as you go.

KAT GOLDSWORTHY: It's interesting. I can see where you're getting at, where you're saying looking at the implementation of your MEL is similar to a MEL process. What I heard you say was you need some kind of plan: what do we want to get out of the implementation? What do we want to monitor? What do we want our MEL to do? And how will we know that it's doing what we want it to do? You need a plan to have visibility and know what to prioritize. You could look at so many different aspects of implementation. It's about finding the important things that we need to build structures and systems around and monitor. Then it's about having people feed into the process and getting data from staff internally around whether this is working, what isn't working and what's going on, so you can respond in real time as you're implementing the MEL and address any issues that come up. That sort of agility, I suppose.

EMILY MCDONALD: Exactly. Because it can feel overwhelming implementing something like a MEL or a new program, the important thing is to choose the strategies and activities that will have the most leverage in your organization and are not too taxing or burdensome for people. Here you're looking for strategies that are strategically useful for practitioners to take up quickly. Most organizations are already gaining a lot of information through their review processes and supervision process. You're just bouncing off those types of processes that you've already got in place. The key is to make it as simple as possible so that you can have the most impact.

KAT GOLDSWORTHY: I was wondering about that because you're right. Even in my own line of work, we have routine ways that we come together and talk to each other. We obviously do our own monitoring and evaluation work here at AIFS and in our evidence and evaluation team. We have a way of adding something extra to something that we're already doing to collect a certain kind of feedback, to make sure that the surveys are going out in the way that we want, people are responding to those surveys, there is an avenue to talk about those things, and someone is responsible to action that. Is there going to be follow up and then continued monitoring of how that's going? I can see how organizations would have existing structures already that you could build on. Are there particular strategies that you have seen work well in organizations around implementation?

EMILY MCDONALD: There are common ones, and like I said, most organizations are already doing them. So why not use them to add a different element in, incorporating your implementation strategies? Examples include local champions: people who work on teams who can be the link between the team and the implementation team, feeding barriers and good stories back about the implementation. Coaching techniques are important, as are communities of practice that organizations may already be doing on a monthly basis, where they can add an item around feedback and collecting qualitative and quantitative data around how the implementation is going. Sometimes people embed processes into supervision, having a routine check-in. Again, it's just one item to add into supervision where you can ask practitioners, how are we going with the implementation of the tools? Is it fit for purpose? Do you think they're working? What feedback have you got for us? That can be something people can do.

People sometimes have a review process. Lots of organizations have six-weekly review processes around, for example, the goals consumers are working towards. Again, that six-weekly review process is another great place to pop in a conversation around how the implementation is going to feed data back to the implementation team. Some bigger organizations might have a clinical governance team of supervisors that gets together to talk about strategy and implementation. That can also be a good place to leverage. Finally, some organizations use a data-informed continuous improvement process. They might already be using tools where they can turn practice into data and track that. But you don't have to do that. You can use simpler forms to gather data and information about how your implementation is going.

KAT GOLDSWORTHY: I love that. The simpler you keep it, the easier it is for everyone.

EMILY MCDONALD: Totally.

KAT GOLDSWORTHY: It goes back to that point before of maybe not feeling so overwhelming to do this implementation work. If you start with, what are we already doing and how do we leverage that? I was thinking, as you were talking, how important that readiness piece is. In order to have these kinds of strategies working well, like using supervision and coaching techniques or using local champions, you need templates for those people to collect the information you actually need. If you're in supervision, for example, something needs to guide that conversation around what questions are being asked and how we record what's happening in the implementation. From the outset, we need to have some visibility of what we're going to be looking for, and how we're going to build that into those conversations so you can collect data that's useful for the implementation team. I was also thinking about how important it would be for whoever is in that implementation team to have good visibility over what existing things are in place to make it as easy as possible, particularly on practitioners who are often doing the frontline work and are responsible for doing all the testing, thinking about what's going on, looking at everything and feeding it back.

KAT GOLDSWORTHY: Right. That readiness bit is so important.

EMILY MCDONALD: Totally. Readiness and engaging practitioners into the whole process. Ideally, by the time you get to implementing your MEL, your practitioners understand the rationale, value and merit of having MEL in the first place. They've got an attachment to the process of MEL and why it's important and what it means. They've got an understanding of how you can adapt practice and programs to get better outcomes for service users. Practitioners are more likely to support the implementation of MEL in a more effective way if they've understood the value and been part of the process to develop the MEL. That's a key strategy. Ideally, you will have developed a theory of change with your team, because what you want is their experiential knowledge integrated into your MEL. That's gold information for the design of your theory of change. If you've got them on board throughout the process of your theory of change, your MEL design and your implementation, you're really set, even if you've run your program for a long time.

EMILY MCDONALD: It's time to have a theory of change exercise to really home in on what people are doing. It can be a great way to start the implementation of your MEL, having a shared agenda around what you're trying to achieve in the first place.

KAT GOLDSWORTHY: Those workshops are so powerful. I've been involved in doing lots of theory of change work, and they are so powerful. People come on board. You hear about all of this stuff from different perspectives that you never would if you didn't involve everyone in that process.

EMILY MCDONALD: Incredibly rich.

KAT GOLDSWORTHY: I love that sentiment of getting the key people attached to programs involved in the design of monitoring and evaluation processes, planning and implementation, because you can't have one without the other. One doesn't work without the other. You need those key perspectives. It's reminding me we did a webinar, probably last year or the year before now, about a routine outcomes measurement tool. Basically, the evaluators, I think they were from Centacare, were working in tandem with the program managers and practitioners to develop this tool, implement it and refine the process. It was brilliant. People should go and listen to that webinar if they want a real-life example of what this looks like, because it led to such rich insights around what was working and what wasn't working. The thing on the page doesn't always work in practice, and you need visibility over that. It was such a beautiful example of how that worked together.

KAT GOLDSWORTHY: And those teams became really close and developed a shared understanding of each area of work, which is an unintended benefit of the whole project.

EMILY MCDONALD: Yeah, I agree.

KAT GOLDSWORTHY: I think we're probably about out of time. Emily, this conversation has gone. If you've got any last comment, anything you want to add that we didn't get to, to comment on, please go ahead.

EMILY MCDONALD: The only thing I'd add is there's a whole bunch of great implementation frameworks out there. They are really good project plans. They run like project plans. So, you do need a little bit of architecture to roll your plan out. Have a look at all of the implementation frameworks out there. Finally, I would reiterate that last point: finding the smallest thing you can find in terms of your strategies in order to have the biggest visibility over your implementation. That's how I'd emphasize it.

KAT GOLDSWORTHY: Thank you. Thanks, Emily. Lots of rich stuff that you shared. I really appreciate it. I'm sure our audience has benefited from the practical tips you've shared. We also have the MEL guide for anyone interested in learning more about monitoring, evaluation and learning. And that is a step-by-step interactive guide to help people get their heads around this work and set up MEL for their own organizations. Thank you. Thanks, Emily.

EMILY MCDONALD: Thanks, Kat. Great.

KAT GOLDSWORTHY: We'll wrap it there. Thank you very much to everyone who came along and listened to this webinar. Thanks very much to the communications team, especially Aaron, for making all of this work so beautifully. And of course, the child and family evidence team. If you haven't already, please subscribe to the evidence and evaluation support newsletter via the website. We'd really appreciate it if you would fill in the feedback survey via email. It will take about one minute, and the feedback helps us improve this webinar program. We look forward to joining you at our next webinar. Take care and we'll see you soon.