Audio transcript (edited)

SHARNEE MOORE: Good afternoon and welcome everyone to today's information session focusing on the family's and children’s activity Program Logic requirements, and how you need to complete a Program Logic using their DSS template. So just a reminder that these particular requirements do not apply to Communities for Children providers. So if that's you, you're welcome to keep listening but you do have different funding requirements. So please keep that in mind as we go through the presentation today. So my name is Sharnee Moore, I'm the manager of the evidence and evaluation support team formerly known as the expert panel project, here at the Australian Institute of Family Studies, and I'll be presenting this session today with my colleague Kat Goldsworthy.

So Kat and I are speaking to you today from Melbourne, where AIFS is located on the lands of the Birrarung and Woiwurrung Wurundjeri peoples of the Kulin Nation. We would to acknowledge the Aboriginal owners of this land and pay our respects to their elders past, present and emerging, and extend that respect to any elders and First Nations people who are attending the session today.

KAT GOLDSWORTHY: Hi, everyone, welcome, and thank you very much for joining us today. Before we start just a bit of background about who we are and what we do. The Australian Institute of Family Studies is the Australian government's key research body in the area of family well-being. AIFS has a range of activities to improve our knowledge about Australian families and what works for them. This includes conducting research evidence synthesis through platforms the CFCA information exchange, and workforce training and support in evidence-based practice and evaluation.

SHARNEE MOORE: And that last point is where we fit in. So the evidence and evaluation support team, we exist specifically to support families and children's activities service providers in building knowledge and skills in program planning, implementation and evaluation, and you were funded by DSS and we've been doing this work since around 2014/2015. Now before we begin today's presentation, I do just need to quickly run through some housekeeping. So we've dedicated a good proportion of our time today to answering some questions at the end, but we are also going to spend about half our time obviously talking through the Program Logic template. If we don't answer your question today, we will follow up with you in the next week and we'll help you directly. So please do send all your questions and comments through, no matter how specific or not they are.

KAT GOLDSWORTHY: So in today's session, we're going to discuss how to complete the DSS Program Logic template. The template which we will get into, in a minute, is something that we help DSS to develop, and it's something that you can use to meet your grant agreements and your grant requirements. So I will say that you don't have to use the template, you can submit an existing Program Logic or use an alternative template to meet the requirement if that's your preference. If you're hoping for a more general how-to-do Program Logic today, we won't be doing that, but we do have a range of resources on our website that you can access. Just put in Program Logic and AIFS into an internet search engine or look in the handouts that we’ve provided today.

Right, so let's get into the focus of our presentation. Many of you will be familiar with Program Logic and Program Logic models by now. But for those who aren't, can you just tell us a little bit about what that is, Sharnee?

SHARNEE MOORE: Yeah, I'd love to. I do want to note though, that we are going to use terms like 'Program Logic', 'Program Logic model' and 'Logic model' interchangeably throughout these this presentation today. So a Program Logic model is a visual representation of how a program is intended to work. It's a living document that sets out the resources that you have available, the activities that you deliver as part of the program and the specific changes that you'd expect to see as a result of the change of the program that you're delivering. So on your screen, there's an example of a Program Logic, and this is a pretty typical Program Logic format.

KAT GOLDSWORTHY: Yeah, and ideally, your Logic model would represent an if-then relationship of these components. So by this, I mean that you'd be organising information about your program in a way that shows, if you have these inputs, that would lead to these activities. If you have these activities, they would lead to these outcomes and so on. So this is really where the term logic comes from, and there are ways of checking that you have demonstrated these connections, but we'll get to that a little bit later on.

SHARNEE MOORE: And there's many different types of Logic models, isn’t there, Kat?

KAT GOLDSWORTHY: Yeah, there's a whole range. And if you did a Google search, you'd find many, many, many different diagrams, I'm sure. And as I already said, you don't have to use this template to meet the requirement. But I will say that if you're going to use something else, just check that it still addresses the assessment criteria. And the assessment criteria is something that Nicole mentioned in her presentation last week, I don't think it's been published or communicated to anyone just yet, but it will soon be made available, and I think DSS will be publishing it on their website so keep an eye out for that one. And if you are concerned or have any concerns about using a different template or diagram, I would be contacting your funding arrangement manager and just having a bit of a discussion with them there.

SHARNEE MOORE: Now I know you're all here, because you're required to develop the Program Logic, but there are some compelling reasons to develop a Logic model, aren't there, Kat?

KAT GOLDSWORTHY: Well, yes, I mean I'm partial to one, so I think that there definitely are. For one, having a good Program Logic model is a great way of communicating what you do in the program and why you're doing it. And this is, especially if you're talking to your funders, program partners, and program participants, and I think there's also really a lot of value in going through the process of actually creating the Program Logic, because it can help clarify what happens in the program, your expectations for how outcomes will be achieved and the investments that you actually need to make a program work. It can also help to establish a shared understanding between stakeholders, about how the program is meant to work.

SHARNEE MOORE: And I also the way that Program Logic models reveal the assumptions that we make about how a program will work, and the DSS template really lends itself to testing assumptions a bit more than some other templates, because you're required to also include a theory of change statement with that template.

KAT GOLDSWORTHY: Yeah, I mean that's a really great feature of this particular template that you'll have to use.

SHARNEE MOORE: Yeah, and speaking of testing assumptions, Program Logics can be a really useful way to monitor and evaluate your program. And having one can really help you plan what data you will collect, and when you will collect that. We tend to focus on the outcomes columns when we're talking about evaluation, but other parts of our Program Logic are just as useful. So take the outputs column, for example, and we've got one on the screen. Outputs usually include key information about how a program is delivered. The outputs in the parenting program on your screen, for example, they're focused on the program's target population, attendance rates, and program dosage.

And the actual process of listing outputs can help you to identify any gaps in your data collection processes to make decisions about how regularly you collect and reflect on your output data, and it can also help to inform the questions that you ask if you do opt to do a formal evaluation of the program. And using these outputs, an evaluation question might be, to what extent the program reaches the intended population?

KAT GOLDSWORTHY: Yeah, thanks, Sharnee. It's nice to see outputs actually get a mention because they really do in our line of work. So we're now going to look at the program template, the Program Logic template, and several other resources that we've developed to help you meet your requirements. These resources that I'm going to run through in a minute have all been tested with the families and children sector or selected providers from that sector, and they've been refined based on their feedback and hopefully some of you are here with us today. So the documents I'm referring to include a blank template that you can download and write directly into. We also have a template that's filled with guidance on how to complete each of these sections in the template, and we've got two Program Logic examples.

So we have a single service example, which is based on a parenting program. This will be most relevant to organisations that are funded by DSS to deliver just one service, possibly two, but usually just one. The other example that we have is one for multiple services, and we base this on a FARS example, so Family And Relationships Services. And this will be most relevant to organisations that are funded to deliver more than one service, and I imagine that’s most of the people joining us today, and this distinction between the single and the multiple service examples will become a little bit more clear as we go on. You will find a link to all of these resources in the handout section of the screen of the Go To Webinar platform, and we will also share them in our project newsletter or you can download them directly from the DSS website.

And I understand, as I said before, that the assessment criteria will also be published soon on the DSS website. So that should pretty much cover it, in terms of if you're going to take the approach to completing the template.

SHARNEE MOORE: Thanks Kat. What do you reckon? Should we take a closer look at the template?

KAT GOLDSWORTHY: Yeah, I think we should.

SHARNEE MOORE: So we're going to talk you through what's required in each section but first, there's a couple of things to be aware of. So you'll see that this template has the usual suspects like objectives, inputs, outputs, and outcomes. But there's also a couple of unusual things in here too, like the theory of change statement that we mentioned earlier, and you might also notice that this template doesn't have an assumptions or an external factors box.

KAT GOLDSWORTHY: Yeah, and I think you can add those things in if you really want. We made a decision, when we were developing this template or helping to develop it, to leave those elements out because with the addition of the theory of change statement, it was already a little bit crowded so we've excluded them for the time being.

SHARNEE MOORE: And the other thing to note is that you will need to provide references for any statements that you make in the needs statement or the theory of change boxes, but we will say more on that later.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: So now to get into the detail about how to approach each section of the Program Logic template, we're going to use the multiple service Program Logic example to explain each section. Kat, let's start with the objectives.

KAT GOLDSWORTHY: Okay, objectives. So this one is very straightforward. It's really just writing one or two lines about what you want to achieve through service delivery, your information probably already exists in other program documentation that you have. So in our multiple service example the objective is family and relationships services and to strengthen family relationships, prevent breakdown and ensure the health and well-being of children through broad-based counselling and education.

SHARNEE MOORE: So next is your needs statement, which is also sometimes referred to as a problem statement or a needs analysis.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: So in this section, you can describe the problem or the need that your program will address. So in doing this, you need to consider the nature and the extent of the problem or the need within your community, why that situation needs a response, what will happen if it's not addressed, and who is primarily affected by the situation or who it is that needs support?

KAT GOLDSWORTHY: Yeah. So Sharnee, I think that's quite a lot of questions that you've put out there when you're trying to develop the needs statement, but the box that we actually have on the Program Logic template is pretty small. So can you give any advice about how to shrink that down?

SHARNEE MOORE: Yeah, it is a good point, Kat. So try to make it as short as possible. A few sentences that summarise your population’s needs will do, and you can leave the detail for something like a full needs assessment or for your funding application. So the needs statement, from our example, is, ‘In this community, many families are living in difficult and complex social environments, including in situations where family domestic violence is occurring or has occurred, and there are high levels of interpersonal conflict. Without assistance, families in these situations are at risk of negative outcomes related to parenting, efficacy, relationships and health and well-being. And by providing prevention and early intervention programs to families in this community, we can support children's development, strengthen parenting relationships and enhance family well-being.

So you can see that we've used references to support what we're saying here, and this is something that you will also need to do. So populating the inputs column is also another fairly straightforward task, and this is about identifying the material and non-material resources that you have available for service delivery. Kat, what would you put in here?

KAT GOLDSWORTHY: Okay, so I'm getting all the straightforward ones, so thanks for that. So I would be thinking about the people that are actually required for delivery, things like staff training, funding, research, research and evidence, and the actual physical program space and things like IT systems. So there are many, many more, so you can really put whatever you want in that column and make that list as long as you want it. So the inputs that we have, in our FARS example, include some of the things I just mentioned. So funding, they've got staff, partner organisations, transport, office infrastructure and staff training and development.

SHARNEE MOORE: And that's basically it.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: It really is one of the easy columns.

KAT GOLDSWORTHY: It is.

SHARNEE MOORE: Moving on to the activities column though, this can be slightly more tricky, can’t it, Kat?

KAT GOLDSWORTHY: Yeah, it can but hopefully, it won't be for this process. The main thing that you really need to be aware of is the information that you list here will really depend on whether you're doing that multiple service Program Logic or you're doing a single service Program Logic. So for the single service Logic models, what we want to see here is a list of the core components, and what I mean by that is the sixth part of the service and the things that are needed to achieve the desired program outcomes. They usually relate to the service content or the mode of delivery.

SHARNEE MOORE: Right, so in our single-service parent example, the activities are focused on parent education topics, approaches to learning and building social networks. And ideally, these activities would be consistent with evidence about what works to achieve program outcomes.

KAT GOLDSWORTHY: Yeah, that's right. And so if you deliver multiple activities, the activities you list here will be slightly different, they'll just be the name of the actual services. So if we look at the multiple service example, we have things that are, you can take an assessment, counselling, family dispute resolution, mediation, relationship, education, parenting programs and referrals. And you can see that, I've used some quite high-level descriptions here, and I’ve some more specific interventions in brackets just to further explain what it is that the whole FARS program does. And you can do either, it's really up to you.

SHARNEE MOORE: So it's quite different in terms of what you would list if you had a single service program, compared to if you have a multiple service program. Is that right?

KAT GOLDSWORTHY: Yeah, that's right. And it’s because it's not very practical for organisations, that are delivering multiple services, to show every single aspect of what that service does. You’d end up with a Program Logic that is about 50 pages long and would be very hard to read. So it's really about having a balance between having enough detail to communicate what you're doing in your suite of funded services, and how those things work together to achieve outcomes, while also being simple enough for someone else to read and digest that information. So for organisations funded to deliver one or possibly two services, you could provide a little bit more detail about the individual service without actually overwhelming your audience.

SHARNEE MOORE: Yeah, that makes sense.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: We’ll move on to outputs.

KAT GOLDSWORTHY: Yes, please.

SHARNEE MOORE: So as we mentioned earlier, outputs are the measures of what you do in the service and who receives the service. So it's a really practical column, and it very much appeals to the evaluator in needs. What would you put in here, Kat?

KAT GOLDSWORTHY: Yeah. No, I really liked the outputs column too. I think that I gave that one away a little bit earlier, Sharnee. So I think I would probably put in here information about the number of activities or sessions that need to be delivered as part of your service or services, the number of clients who attend the service, the client demographic, the relevant intake, assessment and referral numbers, and it just so happens that that's exactly what we've put in our example.

SHARNEE MOORE: And so do you need to quantify your outputs?

KAT GOLDSWORTHY: Again, it's up to you, you can. We haven't, in our examples, and really that's something you tend to do in evaluation, which is why I haven't actually put them in examples, but you absolutely can if you want to.

SHARNEE MOORE: Thanks Kat. Let's move on to outcomes.

KAT GOLDSWORTHY: Okay.

SHARNEE MOORE: We could talk for hours about outcomes, couldn't we?

KAT GOLDSWORTHY: We really, really could and in fact we have. If you've ever attended one of our workshops that we ran in 2018, we've spoken at length about them. We've published videos about outcomes, we've written a whole range of resources about outcomes. But today, we're just going to really focus on what you need to do in terms of meeting the requirement. So Sharnee, can you just define for us what we mean by outcome here?

SHARNEE MOORE: Yeah. So for our purposes, an outcome is the change that occurs as a result of the service or program. So a Program Logic template typically has three outcomes columns, and you'll see that this particular template also has three. So there's short, medium and long-term. These columns should describe the changes that you would expect to see in your target group after attending the service, and then how those changes will build and develop over time.

KAT GOLDSWORTHY: Yeah, and short term outcomes, they usually include changes and things like skills, knowledge, awareness. Medium-term outcomes are usually what would happen to participants when they apply those skills and knowledge that they acquire in the short term. And long-term outcomes share similarities with your objective, and they offer a solution to your needs situation. They usually take quite a long time to be seen and can often be fairly aspirational as well.

SHARNEE MOORE: So here on the screen, you can see the outcomes from our multiple service example. So in the short term, we've got outcomes around improvements in knowledge and skills for the target group, and some awareness and access to different kinds of support. For this from here, we expect that the participants will be able to use their knowledge and those skills that they've acquired to make behavioural changes and start to develop confidence in their abilities in the medium term. And then the longer term, we hope that the program will start to lead to sustained positive changes in the participants and for their families.

KAT GOLDSWORTHY: Yeah, that's right. But having said all this, we know that the program outcomes, the different people will vary and it will depend entirely on the type of services that you actually deliver. So it might not look exactly like this, and that's perfectly fine. So around about this time, we usually start getting questions about how to identify appropriate outcomes and achievable outcomes as well. How would you approach this, Sharnee?

SHARNEE MOORE: Well, I would start by consulting the literature and other forms of evidence to see what's known about what works in terms of addressing the problem, or my need in my community and supporting the target groups and the types of services that I'm running. So the outcomes, in our example, are based primarily on published research, and this is journal articles and great literature about what's known about effective interventions and about behaviour change. They're also based on assumptions about how the program will be delivered, and how participants will interact with the program. In reality, though, we know that not everyone will progress in this very linear and simplistic way. Program Logics, it's important to remember that they really represent an intention. They don't necessarily represent exactly what will happen in every situation.

KAT GOLDSWORTHY: Yeah, thanks, Sharnee. The other thing to be mindful of, when you're identifying outcomes, is that they should directly relate to the activities that you've described in your Logic model. So again, if we look at our multiple services example, you can see that the outcomes focus on communication, emotion regulation, relationships and support networks, which makes sense in the context of activities that include things like a parenting program, counselling and relationship education. If there were outcomes in here about things like school readiness or crime reduction, I'd be really questioning how do they actually relate to these activities that we've listed here, and to the needs as well that are described in the need statement? Because at face value, I mean I know you don't have much time to read through this but, at face value in this Logic model, there's just no apparent connection between those things.

SHARNEE MOORE: That's a good point, plus having appropriate outcomes will really increase your chance of collecting data that can best illustrate the impact of your program. I would also recommend wording your outcomes in a way that's measurable. So you would say here in our example that we've used words like improved and reduced to describe the direction of change that we would want to see in our participants.

KAT GOLDSWORTHY: Yeah, thanks. And so we haven't yet touched on the families and children outcomes framework. How does that fit into all of this?

SHARNEE MOORE: Well, it's most relevant when it comes to describing your long-term outcomes, because you should align them with the framework. So this could mean using the wording that's in the outcomes framework, or a closely related term, so we've opted for a combination of this in our example.

KAT GOLDSWORTHY: Yeah, and I put that slide up a little bit late, but that was the outcomes framework that we're talking about. And I think it will soon - it's one of those things that will be published fairly soon, as well. So before we move on completely from outcomes, I just wanted to take a quick note on setting timeframes for your outcomes. Some Program Logic models have assigned timeframes. So your short, medium and long-term, it might be zero to 12 weeks for short term, and then four-plus years for long-term or something like that. We have not included any in our examples but, if that's useful to you, go ahead and do that. You can add them in or not, it's completely up to you. And we have quite a lot of guidance on our website about how you think about outcomes, frame them, word them, there's videos and various written resources.

So I've put a link to our program planning and evaluation guide in the handouts document. So if you want to have a look at that, just click on that link and read through.

SHARNEE MOORE: Right, that’s outcomes done.

KAT GOLDSWORTHY: Wow, that was quick.

SHARNEE MOORE: So now for the theory of change statement, so this is the last section in our Program Logic template. Now if you don't know what a theory of change is, that's okay because the template actually requires something that's a lot more simplistic than the formal process of doing a theory of change. So your theory of change statement will need to explain how and why you believe your program activities will lead to the types of outcomes that you've described, and you'll need to use evidence to support your explanation and reference where that evidence came from.

KAT GOLDSWORTHY: Yeah, and by evidence, we mean things like practice, framework, theories, published research in the form of journal articles, evidence summaries, evaluation reports and government reports. The guidance document that I referred to a little bit earlier on has some information and some guidance about how to search for evidence if you're getting a little bit stuck.

SHARNEE MOORE: And the statement should provide a brief description of each activity that you've listed in your Program Logic, along with an explanation about how those activities reflects the evidence about what works and how those activities will contribute to your intended outcomes. So let's go back and have a look at our example activities and our outcomes. So this family relationships service is delivering counselling, FDR, relationship, education and parenting programs and, as a result of delivering these programs, it is expected that we'll achieve outcomes around improved communication, emotional regulation, relationships and support networks, and so here's our theory of change statement for this program. Now I won't read it because it is quite long, but you can see that we describe what's involved in delivering the main intervention.

So we've talked about the counselling, the relationship, parenting, education and FDR, and we explain why we think that they will lead to the intended program outcomes, which are around improved conflict management, communication strategies, improved relationships, parenting capabilities and mental health, and we also discuss the importance of key organisational and administrative practices.

KAT GOLDSWORTHY: So when it comes to writing a statement like this, Sharnee, it would probably help if you've already consulted the research and the evidence and done a little bit of that work earlier on, right?

SHARNEE MOORE: Yeah, it should. If you've already examined the research underpinning your services and you've investigated ways of achieving the program outcomes, then completing this section should really just be a matter of translating those five findings into this shorter format.

KAT GOLDSWORTHY: But what if you can't find any evidence about the effectiveness of particular activities and approaches? Or some target groups just aren't well researched?

KAT GOLDSWORTHY: Yeah, this is true. So while we would want people to refer to research and other forms of evidence wherever you can, we also know that that's not always possible. So in these cases, you should acknowledge the absence of evidence in your theory of change statement.

KAT GOLDSWORTHY: And I guess doing that, it's probably a good way of identifying opportunities for further research and evaluation. So you could potentially make a plan to collect data about your intervention, and start actually building some sort of an evidence base. We should also just clarify that the theory of change doesn't need to be squeezed into the bottom of the Program Logic. It can go over another page or two, if you need. You can also submit it as its own document, as an attachment along with the Program Logic, if it's getting a little bit long, I've actually cropped the theory of change example, and a lot of the slides, these are the slides that I've been showing, just because ours is actually a lot longer. When you go into the DSS website and you download them, you'll see that our theory of change statement is actually a lot longer than what it actually appears on the screen.

So we're not in any way saying that it needs to be this short little thing because, especially if you're doing a multiple service-type Program Logic, the theory of change, like this one, is probably going to be a little bit longer and you're going to need a little bit of room for it. So does that cover it, Sharnee?

SHARNEE MOORE: I think so, just about. But just before you finalise your Program Logic model, it is a good idea to check that there's a connection between the different parts of your Program Logic and that it makes sense as a whole, so there's some specific things that you can look out for to help with this. So for one, you can check that your long-term outcomes connect with your program activities and your needs statement. So if we look at our example, you'll see themes across the Program Logic model about health and well-being, relationships and family functioning. So these themes are reflected in the objective, the needs statement and in our outcomes. There can also be a bit of a tendency to go really big and aspirational with long-term outcomes. There's definitely scope to do this, but I would be looking to see if those outcomes align with the other parts of your Logic model.

In our FARS program, having outcomes around crime prevention or drug and alcohol reduction, as we said, just wouldn't be appropriate. They're beyond the scope of our program.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: So you can also check that short, medium and long-term outcomes are actually logical and interconnected, so the easiest way to do this is to trace your outcomes backwards. So for instance, one of our long-term outcomes in this example is positive parenting practices. So if we then look at the medium term outcomes, you can see outcomes around parenting confidence and responding to children's needs. And in the short term, the focus is on increasing knowledge around communicating and responding to children in developmentally appropriate ways. So the outcomes develop over time, but they remain connected. And if you can't find these connections, you might need to go back and make some changes to your outcomes.

SHARNEE MOORE: Yeah, so while reviewing your Logic model, it's also a good idea to look at the resources. So the things you've actually listed in your inputs column, and consider if they are adequate for the activities that are being proposed, and you might find that you need to invest more resources into the program or scale things back a little bit. And my next point is mostly relevant for single-service programs, and it's to try to avoid including activities that aren't actually key to the program, or that are unlikely to contribute to your outcomes. So your organisation might do a lot to attract people into the program, or provide other forms of assistance alongside your particular program, but we don't necessarily want to say those things reflected in these single service Program Logics.

It can be confusing to the person reading them, and it can complicate things when it comes to evaluation and data collection efforts and even the results, so avoid it if you can.

SHARNEE MOORE: The last thing I just to add is that it's really good to do a check that you've identified and included all of your key target group, and no one else. In our FARS example, parents and children are the primary target group, and we refer to them across the Logic model. They’re in our needs statement, in our outputs, our outcomes and our theory of change boxes. It can be really tempting to want to add outcomes for closely-related groups like schools or communities, but it's best to avoid that unless your program actually specifically targets those areas. If this all feels like a bit much, you can always ask someone external to the program to read the Logic model as well and see if it makes sense to them. That's usually a good test.

KAT GOLDSWORTHY: Yeah, it is. And while we're talking about involving other people, if you're just at the beginning, if you're just beginning the process of developing the Logic model, I really recommend involving other people in that process. So that might include people of the program delivery team or leadership team, evaluation and data people in your organisation, program partners and even families themselves. Usually, you'll build a much better Program Logic if you involve multiple people who actually have a stake in the program.

SHARNEE MOORE: It can sometimes be a bit hard to get people to engage in developing a Program Logic model though, can't it?

KAT GOLDSWORTHY: Yep.

SHARNEE MOORE: But we'd really recommend taking whatever approach to your organisational culture, and how you like to work together and build on that. So most of the time, it's really about having someone who's driving the process and talking to people about why it's important, and then giving people time to do the work. Workshops are often a really good way to go about it. We can help you with this process. So our team offer one-on-one support to all fact providers free of charge, we can guide you through the process of developing a Program Logic or we can review documents and we can provide feedback. So if that's assistance that you would like to tap into with us, email us. Someone will get in touch, probably me or Kat.

KAT GOLDSWORTHY: Yep.

SHARNEE MOORE: And our email address is on the next slide, so don't hesitate to get in touch.

KAT GOLDSWORTHY: Yeah, please do. So that's really all that we're going to cover today in this presentation. Just a quick summary before we get on to questions. So Program Logic models are useful tools for planning, delivering and evaluating programs and services. So we’ve spent most of today talking about how to complete the DSS Program Logic template. However, you do not have to use that. You can meet your grant requirements by using another Program Logic template, or an existing Program Logic you have, just as long as it meets the assessment criteria, which DSS will share with you shortly. If you are submitting an existing Program Logic or an alternative template, you will need to still develop a theory of change statement and you'll need to still use evidence to inform that Logic model.

So if you haven't articulated how you’d use that evidence in existing Program Logic models, you'll just have to add some of those bits into your existing Logic models, if you haven't already done that. And finally, my colleagues and I in the evidence and evaluation support team here at AIFS, are available to help you to develop Program Logics or to find them or just provide some sort of advice or testing things out. So you can contact us on the email address on your screen to find out more, and I think we're just about ready to take some questions, Sharnee.

SHARNEE MOORE: We've got a few here.

KAT GOLDSWORTHY: Do we?

SHARNEE MOORE: Are you ready, Kat?

KAT GOLDSWORTHY: Okay, I'm going to get my pen and paper out just in case I need to take notes. Hi, everyone, Sharnee and I have been here on standby, also ready to take questions. Not pre-recorded questions, but live questions here. So Sharnee, there's a couple of questions here already, I might throw over to you and then we can see if we can answer them together. So one of the first ones is, we've been given a few different templates over the years, can you share the best or latest Program Logic template for the CAPS program?

SHARNEE MOORE: Well, I would say, in the first instance, to use the DSS template that they've prepared for this process. So if you're developing your Program Logic for the purpose of submitting it for the DSS requirement, and you don't have another template that you've already used to develop a Program Logic, I would say stick to the safest and the cleanest version, which is to use the DSS template. It will make that process of getting it assessed and, through that requirement, much more straightforward for you. And we also think it's a pretty good template, it's a pretty straightforward template to use. So in the first instance, we'd say use that one.

KAT GOLDSWORTHY: Yeah, thanks, Sharnee. And someone else has asked, is it correct to assume that we would use a single service Program Logic for each activity work plan that we report on?

SHARNEE MOORE: So my understanding is that DSS’s advice is that you should do one Program Logic, and this is for the most part. There are exceptions to this but, for the most part, you should be looking to do a one Program Logic for each of your program types. So if you deliver a CAPS program, for example, and you deliver a few different types of interventions within your CAPS umbrella, your Program Logic should focus on that CAPS program as a whole. Say you just deliver one type of program and it's a supported play group, for example, you would just obviously do the Program Logic around the supported playgroup. There are some exceptions to that general rule. If you've got programs or interventions that are really different, and you're finding it very difficult to pull them up into a whole of CAPS Program Logic or a whole of FARS Program Logic, have a chat to your fam because there is room to do a couple of different Program Logics for the interventions, if it just doesn't make sense to combine them together.

But for the most part, you should be doing it at that activity level, so one for the CAPS program to deliver. If you also happen to deliver, say some FARS programs as well, you would do it for the FARS programs. But that's a rule of thumb, but talk to your fam if you're not sure.

KAT GOLDSWORTHY: And just to add to that, if you are doing one Program Logic for your program area, then what's in your activity work plan, whether you use the single service Logic model, or the multiple service really depends on how many activities or services you run within that program. So if you're running a CAPS program, just delivering one thing like a playgroup, you would do the single service Logic model. If you are running multiple things, like a lot of FARS programs, they’d be running multiple different services and activities and so you'd be doing a multiple-service Program Logic for that. But if it's really not clear, just make a time with us and we can talk you through that one-on-one. We've got another question here, Sharnee. I mean I can probably just quickly answer this. It’s about DEX.

So someone's asking, ‘Have you got any advice about using DEX data and whether it's useful to link it to the Program Logic?’ I was just going to say that this is something we get asked a lot. We're going to be running a workshop on this topic at the upcoming FRSA conference in May, but I don't know that we would necessarily want to address that in this webinar. Do we, Sharnee?

SHARNEE MOORE: Well, except to say - I mean and perhaps if the person who asked the question, they're welcome to jump in and clarify but I'm reading that to say whether - I'll step back. So your Program Logic should certainly inform the type of data that you're collecting and reporting through Score, so there should be alignment there between the types of outcomes that you're describing in your Program Logic, and then the types of domains that you're reporting through Score, because obviously, you're talking about the same program so they should reflect one another. You wouldn't include data, for example, in the Program Logic though, so it's a separate kind of reporting process. So they should definitely reflect each other, they should be consistent.

So you want to see that alignment between the types of outcomes that you're describing in the Program Logic, and the types of things that you would then be - that then guides the types of outcomes or domains that you'd be reporting through Score. But it doesn't work the other way, you don't include Score data, for example, in the Program Logic itself. Has that answered? I think that question came from Louise, can we get a thumbs up or no if at some, we didn't answer that?

KAT GOLDSWORTHY: I don’t know that people do that on there, Sharnee.

SHARNEE MOORE: They can’t? They can’t do that? Okay.

KAT GOLDSWORTHY: Or the people - yeah, I don’t know. But yeah, and we are, as I said, we're running a workshop in May that is going to explore that relationship a little bit further and work out how you really can use Program Logics to guide your data collection in order to produce meaningful data for you, but also to be used for DEX reporting and Score reporting, so keep an eye out for that if you’re interested.

SHARNEE MOORE: And just Kat, Louise has just clarified that part of the issue here is that it's hard to reflect the Program Logic in DEX data.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: And so yes, that can sometimes be a challenge. We can help you work through that a little bit so, if it's something that you want to talk to us about out of this webinar, get in touch. We can take a look at your Program Logic, and we can talk to you about how you might align that with Score and think about it in terms of the reporting frameworks that you're working to.

KAT GOLDSWORTHY: Yeah, and what we'd usually do with the workshop that we're going to run at the FRSA, we will figure out some kind of way to disseminate that information more broadly. And we're still working on the one from last year at FRSA, but I'm sure we'll do something that will make that will make that workshop content available in some format. So presumably, you're signed up to our newsletter, and that's how you'll get information about how to access that. All right, thank you. So there's another question in here about how much detail is required for evidence of needs. Statistics, client feedback, general research, local data can be hard to obtain. Do you want to answer that, or do you want me to, Sharnee?

SHARNEE MOORE: Go for it.

KAT GOLDSWORTHY: I would try to keep it pretty high-level. I mean we've been looking at a lot of the Program Logic as people are getting their feedback from DSS. People have been coming to us to help work through that feedback and address it, and I've definitely seen needs statements that go for a full page, and people like to include a lot of detail, a lot of statistics and data. Obviously, I think that's really up to an individual's preference, but I would be keeping it quite high-level. If you look at some of the examples that were drafted or that we made for this requirement, they're published on the DSS website. And we'll also link, I think Anne’s going to send through some of the resources that we talked about. So the examples will be in those resources, and you can get a sense, from the multiple service and the single service, the expected level of detail, so not much.

I think the important thing is, if you're making a claim about a need, you need to be able to justify that claim and say where you've got the information underpinning that claim. I agree that local data can be really hard to obtain. I think we published a resource last year about sources for needs, data sources for needs. I can’t remember exactly what it's called, but it has also some ideas about where you can source that information. And sometimes it might be that your need is observed or it’s through community consultations or collaborations with other standards providers, and you can certainly reference that too. Sharnee, did you have anything to add to that?

SHARNEE MOORE: Not to that answer, but just to say the follow-up question was that to clarify whether Queensland stats are okay, as compared to needing to find that local government area stats? And I would say probably yes, what you'd be wanting to do is, if you're going to use those Queensland stats, you're probably wanting to be making some kind of explanation about why it is that you think that your community is reflective of those stats.

KAT GOLDSWORTHY: Yeah.

SHARNEE MOORE: So it's a bit of a judgment call, I suppose, as to the extent to which it makes the case for you and what else you might need to do to bolster that. So you might, if you can't get a hold of some of the local government statistics, and you're wanting to use some more State-based stuff, that's potentially okay but I think you'd be wanting to also just say a bit of an explanation about, ‘We know from the work in it in our community, or we know from our consultations, that these kinds of trends seem to be applicable in our community as well, and we're seeing data through our work and through the consultations that indicates that this stuff's reflective, and that the problem that this data points to is a problem in our community as well.’

KAT GOLDSWORTHY: Yeah. Thanks, Sharnee. Another question here asking, ‘In our feedback from DSS, we were told we didn't address what will happen if the situation is not addressed? How can we do this most effectively without going too big? And also acknowledging that other programs might support families instead? How can you address what will happen if the situation is not addressed?’ I think again, you can just be very general, I think. Sharnee, what did I say when I used an example that I made, but basically just saying, if we don't support parents in this particular way, obviously, I think there's probably an assumption that you're not going to address that need on your own, and there usually is a collaborative effort to address these quite complex needs.

But you can, I think. You can say, in general terms, that there might be not issues, but there might be some problems down the track for children, or relationships, or whatever it might be. And there's evidence out there that links those particular, usually, it’s evidence that links those particular things, but I think you can be pretty general about it. Sharnee, did you want to add anything to that?

SHARNEE MOORE: Yeah, so I mean I think what you're thinking about here is it's that kind of, if we don't do this, X will happen, so these are the consequences of not intervening. And so one of the things, those really point to - your long-term outcomes are solving those consequences, I guess. So what you want to do is be referencing the evidence which says, if we've got - it's really hard to think of examples. If you've got a bunch of kids who have really poor school attendance, for example, it's thinking through, well, what are the consequences if poor school attendance is allowed to continue? So what are the kinds of the social, emotional well-being and life outcomes that are on that trajectory, if poor school attendance continues throughout the school journey? And then the intervention is about solving that problem.

So by intervening, we are addressing the problem of poor school attendance, we're supporting kids to get to school, we're supporting better outcomes for them. And the flip side of the consequences is the problem-solving, which is what you see in those long-term outcomes. So you're addressing the potential consequences around its development around their life trajectories, employment outcomes, educational outcomes, pro-social behaviours, all of that kind of thing. But in the needs statement, you probably don't need to go into that in enormous detail. It's really just a sentence or two, where it just says, ‘This is the problem. It's a problem because of this, and it's a problem for these people.’ And if we don't do something about it, this is what will happen, is the level that the needs statement sits at.

KAT GOLDSWORTHY: Thanks, Sharnee. There’s only a couple other questions here. One is, ‘Which workshop is that at FRSA?’ It'll be a workshop on a Monday that we're going to be holding, and do you remember what the title of it is called, Sharnee? It’ll just be the one that's done by AIFS?

SHARNEE MOORE: Yeah, there's only one. So that's one of the pre-conference workshops, which will be on the Monday in this program. I don't think FRSA have put the program out yet, and I couldn't tell you off the top of my head when that was coming out, but I assume fairly soon. So have a look. There's only one pre-conference workshop by AIFS this year, but it's got a title that's around collecting and using more meaningful data. And you'll see that Kat and I are listed as the presenters, so that'll be the one.

KAT GOLDSWORTHY: Yeah, thanks. And the last question, unless anyone else has anything they want to add, is, yeah, ‘Is a support free for CAPS programs and how do we arrange this?’ I mean yeah, it definitely is free. If you're funded by the Department of Social Services within the families and children activity, it's free. You can basically just email that email address that we linked to earlier, and say that you want some support, and we'll be in touch to set up a time with you to have a chat.

SHARNEE MOORE: Absolutely. I've just located the name of that workshop, Kat.

KAT GOLDSWORTHY: Thank you.

SHARNEE MOORE: So I can - it is, so we gave it the title of Balancing Reporting Requirements and Meaningful Data Collection, How To Make Data Work For You and Your Funders, so look out for that.

KAT GOLDSWORTHY: Beautiful, thank you. There's another question that's just come in so, ‘Due to the change in method of delivering the webinar, I missed the first part of the webinar, will it be available for me to access?’ Yes, I mean it was a pre-recorded webinar so we can find a link to that pre-recording, and you can watch it anytime as well because they will do that, the resources they’ve shared.

SHARNEE MOORE: Yeah, so it's listed as one of our AIFS resources. So I think it was in the documentation, but this was a rerun of the video part of a series of two webinars that we did late in 2021 alongside DSS, to talk about this requirement. So we've rerun that component now along with this Q&A, because we note there's still a number of providers that are working to finalise these Program Logics, but you can access both of those videos. So the DSS-presented video about the requirement itself, and then this video in full about the template on the website, and we can definitely include links to that.

KAT GOLDSWORTHY: Okay, I don't think we've got any more questions coming in. We’ve only got a couple minutes left anyway, and I've just shared the name of the workshop in the general chat for those of you who are interested. So we might leave it there, I think we've just about run out of time anyway. So thank you. Thanks again, once again, for all of your patience and coming along to this webinar today, hope it was helpful while you are finalising your program logic models. As we said, you’re very welcome to reach out to us if you want support in addressing some of the feedback from DSS. And as you're finalising them, we're absolutely here to help so just send us an email. But yeah, thanks very much for coming along.

SHARNEE MOORE: Thanks, everyone.