Surveys in the real world: What works and what doesn’t

SHARNEE MOORE: Welcome, everyone to today's webinar ‘What makes a good survey and how to administer it in practice?’ My name is Sharnee Moore. I'm a Senior Research Fellow at the Australian Institute of Family Studies. I would like to start by acknowledging the Wurundjeri, Woiwurrung and Bunurong people of the Kulin nations, who are the traditional owners of the lands in Melbourne, where I'm speaking to you from. I also pay my respects to the traditional owners of country throughout Australia and recognise their continuing connection to lands and waters. We pay our respects to Aboriginal and Torres Strait Islander cultures and to elders past and present. Today's webinar is part of a three-part series designed to share examples of evaluative practice and ideas for collecting and using evidence in the context of Australian Child and Family services. The format we're using today is a little bit different to our usual webinar program. I'll be having a conversation with our guest, and our aim is to provide an overview or an introduction to a particular topic as a foundation for further thinking and learning. My guest today is Dr Michelle Harrison, who's a Senior Research Specialist at the Parenting Research Centre. Today, Michelle and I are going to be discussing surveys for the purpose of data collection for monitoring or evaluation data. When do they work well? What are the key challenges? And to think about some tips for how to administer surveys in practice. Before we dive into our discussion, I do have some quick housekeeping to cover. This webinar has been pre-recorded, which means it does not include a live Q&A. For those of you who require captions, please watch the webinar via the AIFS’ website. There is a link in the chat and a full transcript will be available. There will be some related readings and resources, including a copy of any slides that we share, and these will be available in the handout section of the GoTo webinar control panel. And today is really an introductory conversation. So those handouts also include related readings and links to further resources for you to explore after today's webinar. And because we love both feedback and surveys, there will be a short feedback survey at the end of this webinar, and we're very grateful for your responses to that survey. So, Michelle, I am keen to get into the meat of this topic, but before we do that, I just want to make sure we're all on the same page. So what are we talking about when we talk about surveys in this context?

DR MICHELLE HARRISON: Thanks, Sharnee. So basically, a survey is just one of the things that researchers or evaluators have in their toolkit. There are various ways to gather information, but generally a survey is a structured tool where there's a, you know, a range of questions being asked and there tends to be standardized responses. So, there might be things like, you know, yes or no, or it could be strongly agree down to, you know, strongly disagree. Or it could be a frequency, you know, how often do you do this or 1 to 5 sort of thing, or how do you rate this or how was your experience? Also, I suppose, uh, you know, the, the information gathered in that respect is generally quantitative. You know, it's a number, a factor that we can analyze data, but you can also get data through qualitative means too. So open ended questions where you might ask, first of all, did you, you know, how do you rate the seminar? And then afterwards, you know, how could we improve on the seminar? And there might be an open ended question where, you know, people can pop in some information. Yeah. Yeah, I think that probably covers it. I don't know. You've been around long enough to know.

DR MICHELLE HARRISON: Such as? Me.

SHARNEE MOORE: No, I think you've explained that really nicely. I mean surveys are often a bit of a go to. Sorry.

DR MICHELLE HARRISON: Oh, I was just. I was also going to just say, you know, in terms of surveys, you know, there's different ways you can administer them as well. You know, you can obviously you can send them out, on paper form, you can do them face to face with people. You can do them over the phone and, you know, online is, you know, a quite a common way to do things. Now people text them out, you know? Lately we've been using them in group sessions where we use QR codes and we do it as a group. So, you're not actually, you know, identifying one person saying, hey, you, it's your turn. It's, hey guys, you know, we're going to do this together. So, it becomes a little bit more comfortable. So, I think, you know, working out what, what's best for your group or individual is important as well. And I suppose it depends what you get, right. If you're looking for, feedback on how you as a practitioner is going. You probably won't be asking the questions. It would be, you know, here's a survey. Take your time. 

SHARNEE MOORE: I was at the train station the other day and a student from a local school presented me with a QR code to do a survey for an assignment. And it took me through to the survey. And it was, I was reflecting on how, it was very easy, but it also meant that she could probably quite efficiently collect survey responses, because she didn't need to stay. She didn't need to kind of stay with me and she could, you know, use that as a mechanism to, to recruit lots of people in a really anonymous and quite, quite efficient way.

DR MICHELLE HARRISON: That’s the thing, isn't it? It's. Yeah. How many people are you looking at? And, you know, what's the best way? And it depends on the, on the group that you're targeting. 

SHARNEE MOORE: Absolutely. Know your audience. It's the important lesson, isn't it? I mean, surveys are often, you know, as I was saying, a real go-to for collecting data. But in your experience, when do surveys tend to work better? And do you have any tips that you can share for practitioners or organizations, how they might make decisions about when a survey is the best choice?

DR MICHELLE HARRISON: Yeah, I think, you know, we're carrying on from what we just said, you know, knowing your audience is, I think, really important. I think they're excellent in when you've got a large group of people and you don't have much funding. So you need to get responses, you need to get a bit of a, I always think of it a bit of a feel, a pulse for what's happening. And then you might have the opportunity to follow up with some, you know, some interviews or focus groups. So, what you get, you know, you might get a snapshot, but not everything. And then you might follow it up if you have the opportunity, with other mechanisms of, of getting that information. Yeah, I think a lot of people go, oh, let's do a survey. But if you've only got a small amount of people, you know, you can get some really good information by, as I said, other means, you know, sitting down and having those conversations with the group. I also think it's really important to consider, who the audience is and whether they'll understand the questions in the first place. So, if you're, you know, if you've got multicultural or, you know, people with low literacy, it's really important of how you set up the survey. You don't want it to be too long. You don't want it to be complicated, you don't want, and I've learned this over time, you don't want double barrelled questions where they've got to think, you know, one thing and then go to another. I think those sorts of things really complicate. I tend to start. I mean, I've learnt over time, you know, it used to be really long surveys. I think it depends on where you are. Are you doing research or are you actually just in a program and wanting to know if it makes a difference? If you. And look, I've evaluated recently a program, a national parenting program, and we had to use a lot of validated tools with a lot of questions. But we took that on board because we wanted to get that deep information so we can benchmark it across other parenting programs and write up in the literature. But if it's a program that you're just delivering and you want to know, do they like it? Is it making a change to the families? Then smaller is sweeter. You know, it's easier. You're going to get the answers you want and you're more likely to get responses. And, you know, we're evaluating some surveys we're doing in a statewide program at the moment. And the feedback, you know, we're getting like 10 to 20%, you know, response rate and you've got to go, something's wrong here. And it is because it's just for, for all the reasons I've mentioned, it's too long. It's, you know, not the target audience don't understand. Obviously, if you've got the capacity and you've got people that can sit with people and go through the surveys, then great, you'll probably get the responses. And definitely if you've got multicultural, you know, that speak the language or you can translate them. But a lot of people don't have the time, the funds, the money, you know, the, the expertise to do those things. So simple is best. And obviously piloting is really important. If you can is seeing and you know, I know with some of our surveys, we pilot just with parents in our organization before we go out more broadly, you know, if you had the opportunity with some, you know, past clients, then that's a great opportunity as well to see how it how it sits. And I also, you know, if I'm doing it more broadly, I will test with the facilitators, the people that are running the program to say, you know, how does this sound? Do you reckon your parents could do this survey? So there's lots of different ways that you can do it. 

SHARNEE MOORE: Yeah. And I think that really speaks to, you know, this is not a webinar about, kind of Evaluation or monitoring planning more generally, but the data collection methods that you choose, whether it be surveys or something else, really needs to be informed by that earlier planning work. So you need to, you know, know your purpose and then use that purpose to define the types of questions that you're wanting to ask and understand who your audience is in terms of who you're collecting that data from. So know your community, know your, your participants know how they're going to be able to engage. And then, but also it's about knowing your audience later on. So who are you going to communicate this information to? And, you know, and then kind of, as you say, really pilot testing this, if you're, if you're not sure, or even if you are sure, it's good practice to, to really test that approach out with the people that you're wanting to do those surveys with and doing that, that, that planning groundwork and really understanding, you know, the purpose, what is the information is that you're wanting to know and what are the needs of the people that you're wanting to engage with this community and how can you best, kind of promote that engagement and support that engagement? That's, it's, you know, you're never going to kind of lose out by investing that time. Are you sort of up front?

DR MICHELLE HARRISON: No, no.

SHARNEE MOORE: It's, I mean, we've sort of touched on where things can go wrong a little bit. And there are lots of lots of challenges, and risks that we need to think through when we're planning data collection activities, whether that's for a survey or for some other kind of data collection mechanism. I'm sure we all have experiences where things have not quite gone to plan. With surveys, do you have any practical guidance you can offer practitioners, to kind of think through some of those challenges and overcome them?

DR MICHELLE HARRISON: Can I first say that challenges happen even when you're experienced, right? Like I had one just recently, you know, and I've been doing evaluation for too long and research. Yeah, there's lots and lots of different challenges. You know, the first challenge is, you know, coming up with the questions I think is the first thing. And you know, like in the past, I would probably sit down and try and write some questions. You know, just from scratch, I think, you know, I'm thinking back to my early days. But as time went on and I think that you, you really need to think about and, you know, there's so many resources these days online to find questions, you know, don't I suppose the thing to say is that using a question that has been trialled before and has shown to provide the answer you want is, is a good start. So, you know, we call those sort of like validated tools. And I suppose one good example, is the personal well-being index, which, you know, has seven questions from, and touches on lots of domains. You know, it talks about satisfaction in terms of their general life, in terms of finances, in terms of their health and, connection to community, and, safety now and, and they're, they're simple questions to ask. Now you might look at them all and go, well, I don't need all of those. And look, we've asked them in surveys and people have gone, well, why are you asking that question? And we've, you know, we've gone, oh yeah, you're right. We probably don't need that one and we've dropped it. Or if it doesn't come into it easy for people to talk through that question or we've gone, okay, so we only really need to know about the safety. So let's use just that one question. And with the parenting, sorry, the personal well-being index, it is a validated, every question is validated as well. So, and what's really great about using something like that is you can say, oh, well, my sample of people said there are 5 or 6. I can then go back and look at what the data has shown. You know, there is a norm, what we'd expect them to be. What ranges, and you can say, okay, so they're a bit low. We still need to keep pushing them along. So it gives you guidance of, you know, when you get a value, it gives you an idea of what that value means in scale and reference. So I think that's really good. I suppose the other thing.

SHARNEE MOORE: I was going to say validated tools are really, you know, they're a really powerful mechanism because they allow us to tap into that greater body of research. And I mean, you've got a slide up on the screen at the moment about ethics and surveys. And I think you sort of touched on this idea that you should only ask the questions that you need. And that really points to that kind of ethical dilemma of making sure that you're only collecting the data that you actually need and that you're going to use. Are there other ethical considerations that you wanted to draw out?

DR MICHELLE HARRISON: Well, I suppose there's always the. Yeah, it's always, you know, important that the data that you do collect is, as you said, what's required, but also giving, I suppose a heads up to the clients and to the facilitators, if you're not all the person who, the practitioner who's, doing the survey or administering it is that they understand that they provide that background of why we're doing the survey, what the information is being collected, how it will be used, and who will have access to it. Because I think a lot of people are concerned, especially, you know, very vulnerable groups and they might be in a parenting program they've been told to come to and all of a sudden you're asking some questions. It might feel a bit personal and might be it might be, oh, well, this might go to child safety. I'm not really comfortable with that. Or I will just say, yes, everything's fine. So you're not really going to the heart of, you know, how things are traveling. And I always find, you know, that's sort of true when people come into programs anyway, they always say, yeah, it's all good. And then you might repeat that survey down the track a bit and you find there's a bit of drop in results. And I think you need to keep that in mind that maybe now you've built that relationship, that you're going to get some more honesty. And I think that's coming back to who's going to administer, who's going to be, you know, providing that survey that you try and use someone that's trusted in that community, you know, so it's really hard when someone like me comes in and they go, well, who's she? Where's she from? It's much better to be at a programmatic level to get someone that's trusted in that community to help, you know, support the survey and get the best results.

DR MICHELLE HARRISON: The other thing. Sorry. Go on.

SHARNEE MOORE: It's a real ethical dilemma, isn't it, in the sense that you really wanting to kind of go through this ethical process of making sure that people have, you know, you know, you've, you've built the right processes that people feel really comfortable with that survey, but there's a pragmatic element as well in that you will get better data if you put in the work to to think not only about what questions you're going to answer, but the ordering of those questions. And what kind of setup have you got in terms of kind of how you're actually going to go about administering that survey? So as you say, who's asking the questions? Are you doing it in a context where the person feels safe and free to provide honest responses? Have you given them enough information in that process to so that they feel confident and comfortable that you're going to treat that data with confidentiality and respect, and that you're going to use that data for a purpose that they feel comfortable with. And so, you know, there's, there's, there is the ethical lens to that. But it's, it has, it has practical considerations as well, because you will, if you do that well, you will get better engagement and better data through the survey in the end.
 

DR MICHELLE HARRISON: Yeah. And I think that's a real, it's a learning experience. It's a curve, you know, we're going through. I go through it every day. You know, I look back and go, God, why did we do that? You know, don't hammer yourself on it. Just move on. You learn. And that's a great thing when you're learning. I mean, you know, yeah. And also, I suppose when getting back to the, you know, the questions, you know, it really comes back also to that pre-planning that you're talking about and making sure that you've tested it, you know, because you can read through something ten times and think it's all good, but you will miss something every time. I've missed, you know, I've put surveys online and then realized that I'd left out. Instead of saying strongly agreed. I put strongly disagree because I've copied and cut from another page sort of thing. So you've really got to test it. You don't want it to be out there in the field and then find that you're getting the wrong answers because you've set it up incorrectly. And I've seen surveys where they go, you know, and this feedback has come back recently is it'd be great to have all the responses the same. So you know how some questions might be, you know, 1 to 5 and then you've got strongly disagree and then you've got sometimes always and they say, oh, that's a bit confusing. Can you make it all the same? And I reviewed a survey that someone developed recently that had all this strongly agreed all the way down, but it didn't make sense when it got to what gender are you? And it was asking to strongly agree or disagree. So I think you've just got to be careful and read everything. And as I said, test it and test it with someone in your room is really important as well. Can you make it all the same? And I reviewed a survey that someone developed recently that had all this strongly agreed all the way down, but it didn't make sense when it got to what gender are you? And it was asking to strongly agree or disagree. So I think you've just got to be careful and read everything. And as I said, test it and test it with someone in your room, is really important as well. Yeah.

SHARNEE MOORE: It's a balance, isn't it? Because I mean, writing survey questions is it's hard. And even when you, you have lots of experience in doing it, it's, it's, it's not something that you can kind of go in and nail. First time, they can often take a lot of refinement and a lot of work. And, and as you say, thinking through not just the questions, but what the response frames are for those questions. And, you know, do they make sense in relation to the question that you're asking? And but also kind of that balance of trying to maintain some level of consistency as you move through the survey, because that really reduces the, the kind of the mental burden that people need to engage with to, to complete the survey. They have to kind of think through lots of different types of responses all the time. It's, it's harder to move through it. And so it is, it can be a really tricky balance. And it can take, you know, a bit of work to, to get it right.

DR MICHELLE HARRISON: Yeah. And that bit of work could come from training. You know, training up people is really important. And I know I work with one organization where they were using a number of tools. But they wanted to develop something that across a number of programs, whether it be in, you know, domestic and family violence or it might be a parenting program or it might be, financial capacity. And when we were talking, they were all using different tools, but the organization wanted really one that they could see across all the programs that they could monitor what was happening. And the PWA, the personal well-being sort of sat in, in there, but a lot of, a lot of them went, nah, can't use that. Not, not. They were very much against it. But you know, after three months of not continuously me talking to them, but after investigating initially it was let's make up our own. You know, that sounds great. Let's no, we need to use this tool. No, someone wants that one, you know, but there's specific, you know, some of are just for Aboriginal and Torres Strait Islander, which, you know, is not validated in a different, in a different, cohort of, of, of people that you're trying to get the information from. But basically after we did a lot of training and, and fact finding, we landed on the personal well-being and after they understood how they could use that tool. I think a lot of people and look, I spoke to someone yesterday who's on a phone service and we have a very short. And that's the other thing. Short is best questions. I said, oh, so do you when you're talking about whatever it might be. You might bring in the coping question. You know, you don't have to have them all together. You you're immersing it in the conversation, so it doesn't feel like that they're actually answering. And she said, oh, no, it's easier for me just at the end to say, hey, we've got this survey. I'm going to ask you these five questions. And I go, well, easier for you, but not necessarily possibly easier for the client. So maybe you could try trial it that it might actually, some of these questions might naturally fit into a conversation like we're having at the moment. So we've been talking about your health, Sharnee, but if I was to ask you on a scale of 0 to 10, how you feel about your health at the moment. Are you satisfied with it? What score would be? Where would you be? And that way they could say, oh gosh, I don't know, maybe I'll be a five or a six. And you say ‘oh that’s pretty normal’, because you might know six I sjust under the norm, and it’s not quite there. Or they might say, you know ‘I’m an eight’. And you could come back and say, ‘you know I’ve been listening to what you said today, and you’ve got some really traumatic things going on, and I was thinking, you know, if you’re thinking about how you’re feeling you might be more like a three or a four. And they say, ‘yeah, you’re probably right.’ So it’s having those conversations and sense checking, because if you’ve had a conversation you know where they’re sitting. So you know, sometimes people say to me, ‘disregard that data because I know what they’re like. They’re not really an eight or a nine.’ So you’ve got to make sure. That’s part of getting the data if you’re having that one on one. Obviously you can’t have that in a group. But you can bring up, with the group and say. Try and be honest because we’re going to repeat this at the end of the program. And this program is all about improving, blah, blah, blah. And maybe it's your, health or your connection to community. You know, so if we all sit up at a ten, that means you probably shouldn't be here. You know, you're doing really well. You know, this program is for people that are really struggling at the moment. You know what I'm saying? It's having that conversation, setting it up so people feel comfortable to say, hey, I'm not doing well at the moment.

SHARNEE MOORE: And balancing that, I think that that conversation where you kind of you're building this sense of trust and validating the experiences that people are going through without kind of leading them through those responses, because you do want that survey to be given honestly and from their own perspective. And so it's, it can be a real art form and it really speaks to, I think, how critical that process of deciding who's going to administer the survey is, and then making sure that whoever you decide, whether it's going to be kind of a person external to the program or it's going to be the individual worker that's already got that relationship with the program participant, whoever that might be, makes sure that that person has the training and the support that they need to administer the survey as well, and to feel confident and comfortable with that role that they have because they're critical to the success of the survey.

DR MICHELLE HARRISON: Oh, absolutely. Absolutely. And it's really hard. You know, you come into a program, you don't know these people, they don't know you. And if you've got to get the survey done straight away into a program, you know, think about it, you know, maybe the first session mightn't be the way, or you might have already developed a relationship by, you know, making calls and what have you prior to them coming in. You know, try and, you know, measure the temperature in the room and know when the best time to ask the questions are. And the other thing is how you administer it. You know, if you do it, send it off to them home, you know, it's quite common. Well, I, I sent it, but they, you know, it's almost like practitioners will say, well, it's their fault because they didn't respond. But you know, if you've got parents, you know, they're busy. So are they going to be able to do that on their own? Are they going to remember to bring it back the next time they come in? Those sorts of things. You know, if you've got them there and it's the right time, you know, we've gone to looking at doing group check-ins, as I said, just so we can get that information there. And the other thing to do, you know, with surveys is simple is best, as we've already said. Don't go for a 20-question, half-an-hour survey if you don't need it. You know, I always start with the least and work up. A lot of people, as I look at surveys, start with the most and then they cull. Yeah. And, you know, no one ever gets. Well, I suppose some people do, but you never feel like you get it right. You know, if you look at a validated tool, you know, within a validated tool, sometimes there's like subscales. You know, so look at those. You might be able to go, oh, well I'm actually looking at confidence. So look at the confidence questions. You know, for example, Parenting Research Centre have the Me as a Parent, which looks at parent self-efficacy and confidence. Now that's 16 questions. And, you know, how many people use the 16 questions? Not often because it's a lot of questions. They've made a short form. So short forms, I always look for surveys of short forms; short forms are great. The short form for that one is four, and even that I have used it in a program where we ended up cutting it down to the one question. And the reason we did that was for two reasons. One was that we got feedback from the facilitators telling us that all the practitioners telling us that they, they couldn't understand why we were asking these questions because they're around parenting now. The program was a peer support. Now hopefully at the end they're going to be improving their confidence and it will impact on their parenting. But in the room, what it was doing was providing a sense of connection community. So we ended up reducing it to one question around the confidence and, and added in questions around their sense of belonging, you know, feeling like part of that group that they could speak up questions that made sense to them, why they were in the program, questions more distal, if you think about, you know, oh, that was the other thing. I was going to bring up more distal long-term outcomes. You can see what it's going to do. But the people in the room are there. They're there that day. What changing for them as part of the program that they can see is that their knowledge, you know, their, you know, their, where to go for information, you know, they might have turned up and not known where to go to for information around disabilities or whatever it might be.
 

DR MICHELLE HARRISON: So I always, if possible, start with a theory of change, a program logic that sets out, if I do this, then this will happen. And if that happens, then this further down will happen. And if that happens, then we eventually will get better well-being. But what? And you've got little, I suppose, little steps along the way, a, b, c, d. And I was asking a lot we or we in some of our surveys were asking a lot of the question D's, but they couldn't see that longer term. So we pushed it back and we're now getting some really good data. And if we know that those things are happening, then we expect those next things to happen. So if they feel like they're getting the information and they feel like they belong, we know then that can lead to them having more optimism, hope for the future, more confident they could actually do something and that will have an impact on their family. So they're the longer term things. So I think, you know, you've got to think about what in that space or in that program, what you expect if it's ongoing? Yeah, you can measure those other things. But the initial things, what we expect as part of that program targeting there, and that way it makes sense to them as well, because what's changing for them is not they don't see as the well-being initially. They see it as their knowledge and having connection with other people in the room.

SHARNEE MOORE: Absolutely. I mean, the, I mean, there's so many different ways that are, that are kind of a survey and administrative survey can, can have some challenges in it. And I think that, you know, things can fall apart in terms of your planning. If you haven't kind of thought through those challenges at the beginning and thought through how you might mitigate some of them. It can fall apart if you've identified the wrong questions, or the wrong way to kind of go about that survey. It can fall apart if you haven't done enough thinking about how people are going to actually complete the survey. And you know what that what that process is going to look like for people and it can fall apart a little bit if you're at your analysis point as well. But I think that, you know, the unifying thread of all of those places where there can be, potentially some, some difficult parts is, is in that planning. And like, as you know, I'm a huge fan of program logic as a, as a basis of planning. And I think, you know, a good program logic model can, can really help you to identify what it is that you're wanting to focus on. And as you say, you know, often, particularly if it's a smaller program or a smaller organization, it's, it's impossible to try and measure some of those long term outcomes. And it's really kind of setting yourself up to fail a little bit in your data collection. If you, if you kind of try and aim too far away. Whereas if, if you've got a really tight program logic model and you make a decision to do simple things well, and in a really concerted way and focus on the things that are really important in that kind of short to medium term, you can collect much, much richer data and much more usable data in terms of trying to unpack some of those questions. Yeah.

DR MICHELLE HARRISON: I think also the program logic helps everyone, be on the same page. And when I was working more within the last organization I was working for, we co-design it. So everyone agrees that this is what's happening as part of the program, especially if you come in and over the top and go, okay, we're going to develop this program logic for you. I mean, you can look at documents, but you've got to hear the voices. You've got to hear the voices of those on the ground doing the work. So everyone needs to contribute. And if possible, contribute to the questions, as I said, by, you know, you might come up with a bit of a starting point and, and, and, and, and touching back and forth with the, with those that are in, in the field to see if they're going to sit well with the community and they're asking the right questions, but having that program logic is almost like the base that you can always go back to and say, well, this is why we're asking the question, because, you know, if we know that, you know, whatever is happening we expect to happen is happening, then we know that by theory and the theory is coming from the evidence. Usually program logics are based on what the evidence says. You know, that there is going to be this change. You know, there is an association between knowledge and confidence. So if we improve the knowledge or the skills that's going to lead to increasing confidence. And then, you know, that confidence might change in terms of the child's behaviour, you know, or, you know, or reflection, you know, being able to reflect on the parenting, on their skills. And we know if they can do that, then they can understand why their child is acting out. And it's not a matter of this child's naughty, it's okay, they're tired or they don't understand. And going through those stages will lead you down the path to the outcomes that you want.

SHARNEE MOORE: Indeed.

DR MICHELLE HARRISON: And I was going to say, and then there's the opposite. If you're not getting the results or people aren't turning up and you're getting that data, then you go, well, that's where it's falling down. We need to focus in here and change. And don't be afraid to do that to, to change and, and also to change the survey questions as well. You know, sometimes we've been focusing in one area for a long time. And we know, for example, in one program that we're working, we know that, uh, confidence, you know, it has improved the confidence. Do we keep asking that or do we add more questions in another area? Or do we take away the confidence questions and put the new questions in? So we're not making the survey longer, so to speak, that we're keeping it short and sharp, but we just want to focus in on a different area now.

SHARNEE MOORE: Yeah, indeed. And I think that really comes back to, you know, knowing your purpose and your audience and what it is that you're wanting to, discover through this data collection. I mean, we know that organizations want to collect data for lots of different reasons. And, you know, it might be because they want to satisfy funding or contract requirements. They might, as you say, want to understand particular questions around their program or their community or to showcase impact that they're having in that community or maybe some other reason altogether. I guess I just to kind of to finish our conversation today, I just want to spend a little bit of time reflecting on how organizations can move from using a survey to collect data to doing that as part of a process that supports learning and reflection. So I'm interested in how organizations transform kind of evidence into insights and action. What would be your kind of quick key tips for practitioners who want to ensure that their surveys are supporting kind of a learning system?

DR MICHELLE HARRISON: So learning for the practitioners?

SHARNEE MOORE: Learning for kind of the process of continuous improvement or service?

DR MICHELLE HARRISON: Oh, okay. Yeah, yeah, yeah, yeah. Don't keep doing what doesn't work. Don't feel that you have to keep doing something. Be true to yourself in the program, and work out what you need before working out what others need. Others being funders: you might be pressed to gather information, but sometimes you can actually work out what you need and then find a way of providing that information to the funders, but it's still providing new information. It's really important obviously to know if the program's working. And I suppose it's not just about, at the end of the day, the outcomes. You know, some people go to the end and go, okay, well, oh gee, we didn't see any change in the parents. They were a really hard group. Oh yeah. They're special. You know, this program's probably not for them. Maybe the program's not for them because of the way we're doing things, and we might need to reflect back there and look at further, as I said, what data are you collecting? You know, are people turning up? Are they getting enough dosage? You know, what are you delivering? Is it actually resulting in engagement? You know, are we seeing change? Are they, you know, really coming? Are they turning up and just sitting there and not engaging in the program? And having said that, I shouldn't say that should be a measure because I know with a program that we did recently, there's a number of parents that come in and, you know, not everyone is outgoing and are going to be putting their hand up every second. And then you go, oh, well, that didn't work for them. But when you get feedback from their organization saying they loved it and it really has changed. So yeah. But I think, yeah, gathering information on an ongoing basis. It's not just gathering information to send out. It's also to send in as well and to say, how are we going as an organization in delivering out on this program? So having some mechanisms to be able to measure that as well, and reflecting on changes. And we often, we've got a program, for example, that we run nationally, called MyTime. And, you know, we used to do evaluations every so often. They were, you know, a regular sort of thing. And we'd get less than 100 people. Now we've got thousands in this program. And then we changed it. As I was saying before, we've changed it because it was too hard, like the practitioners were out there trying to gather this information. Oh, God. Now she a new person? Is it her turn? Is it six months? And so it was all, you know, set out. It wasn't like one week we're doing everyone. It was one week someone. And then. Oh, she didn't turn up. I'll have to remember next time, you know, or I'll send it out. And they don't reply. You know, we said we tried everything. Links, you name it. But we know at the end of the day, everyone's busy and it's not their priority to fill out this information, but it's really important information. So what we did was we changed it and we reframed it as a check in. So it wasn't a survey for you. It was a check in on the group. How's the group going? And we did it by everyone had the same QR code and they don't have to put their names. And basically, oh no, they do have sorry, they did have to put their names because we had to match the data. That's right. Yeah, but we removed, so we explained what it was for. And now over the last, we've been doing it for two years now and the numbers have gone up over 700 each six months. We're getting this data. We also then that data. And that was changing from parenting to more. How are you going. Do you feel like you're connecting to community, that belonging, that question they could relate to as well, which is great. And we tweaked on those too. As I said, there was some more on the parenting. So we dropped it down and  made it more condensed as much as we can. And we're going to do that again, because the belonging question is actually quite large. Well, I think it's got ten questions, but I want to drop that down again so we can add in some more questions about some other things that we might be interested in touching in on, because we know these things are happening and we can see it lines up with our, our theory of change. If we do this, this is going to happen. But we also do reflections with the practitioners. So we ask them again, we're learning. So we started by after each after each session, they would go into and reflect on what, what the clients were doing and how they were coping at the practitioner was coping, but it was just too many questions. So we ended up changing. Because we couldn't, even though they were having some issues, we didn't really know how to help them. So we're now changing that into a once a term check in for the, for the practitioners to say, hey, is this happening? You know, what helped you? Because we don't want to just know about the challenges. We want to know about the successes, what's working really well, so we can spread the love to everybody else and say, hey, this is how it works in this group. Maybe try it. So I think that's really important as well is learning from both ends. And the other important thing is getting the data back. How often do people say, I never see anything. It gets sucked into this system and it's gone. So we have in my time, for example, we've now developed every time we do these check ins, six months, we send out a newsletter to facilitate. It's about what's happening, not individual, and it's not group level. It's overall. Here's where everyone's sort of sitting and what changes and that they can then speed that back. We also develop dashboards where the information live data is there and they can cut it down for group. And, and it's a way of not saying pointing a finger and saying, look, your group's not getting numbers or you're not getting changes. It's overall, here's what everyone else is doing. And here's you. Well, you know, what's happening for you in your region is quite common. Don't worry, you're doing actually not too bad. But where can we where do you think we can, you know, improve on. So it's a practice improvement learning thing. It's not about trying to, you know, catch people out. And I think that's it's the way you frame it. And it's the culture of the organization in terms of this is about you. We want to support you. We've heard for years that this is not working. Let's see if we can make a change here and or make this program that's been developed at a, in a different community, work in our community. And how can we tweak it? So it does work and that's what the data can do.

SHARNEE MOORE: And I think that's a beautiful place for us to finish up. Michelle, I think it's, you know, I think hopefully what people have gleaned from this webinar is that planning is really important. Surveys are a really powerful tool in a, in a kind of, in a learning system and can help you to really unpack some great information about how your program is tracking and how you might be able to make tweaks, but that it can be really challenging. And, you know, doing that planning work is really important. Keeping it simple and then not being afraid to reflect on the process itself and to make changes to that process if you need to, if things don't seem to be, you know, the data is not panning out the way you think it should be, or that you know that your process of administering that survey is not working as well as you think it should be. Don't be afraid to sort of pause, take stock and, and, and make some tweaks to that process and to just keep communicating with, with people testing, talking to your stakeholders, and then sharing back what you find out about this.

DR MICHELLE HARRISON: And don't be too afraid. Just don't feel it's like your fault that it's not working. You know, like I have even now, you know, yesterday got data back on something and asking about, you know, home coaching. And they said, yeah, I use in-home coaching. When I asked them how they use it. And I'm going, yeah, of course you're using it. I wanted the information about how are you using it? You know, it's how you frame it and putting prompts in to give people, you know, the context, what, what level that you want that information from. And don't be afraid to reach out, like even to me if you needed to, because I'm happy to go, yeah, use this tool. We use it all the time, you know, and that's how I, you know, after I got my confidence, I suppose, is that how I work that I was reaching out to people who work in the parenting space and say, hey, I'm looking for a tool on this or I'm, you know, I'm struggling to find the answers. What, what do you use? And you'll find that most of the time they will respond. Maybe not straight away. 

SHARNEE MOORE: And we've got, we've got the list of handouts and different resources that people can access. As you said, reach out through your networks, reach out through your, your colleagues, ask questions. AIFS can provide support. Michelle is ready and willing to also provide some support. So reach out and ask questions. I really want to thank you, Michelle, for this conversation today. And I want to thank you to our audience, and to the communications team and the Child and Family Evidence team at AIFS who've helped do all of the background work to pulling this together. If you haven't already, audiences can subscribe to the evidence evaluation support newsletter by the apes website. You can stay up to date with different things that are happening in the evaluation and monitoring space through that newsletter and webinars such as this. You can also continue your learning journey by attending the AIFS conference later this year. Hopefully I will see Michelle there. The conference is always a personal highlight for me. And it really brings together government practitioners and researchers to explore how policies, programs and systems need to evolve to support families in all their diversity in a rapidly changing world.

SHARNEE MOORE: As I kind of flagged at the beginning of this webinar, we'd really appreciate it if you could fill in the feedback survey. It will take about one minute. We promise, and the feedback will really help us to improve our webinars. But that's it from us. We thank you for attending today. Thank you again to Michelle, and we really look forward to joining you at our next webinar. Take care and we'll see you again soon.