Audio transcript (edited)

KAT GOLDSWORTHY: Welcome everyone to today’s webinar: Using Surveys to Understand your Service, Research and Practice Perspectives. My name is Kat Goldsworthy, and I’m a research fellow in the Evidence and Evaluation Support team at the Australian Institute of Family Studies.

I’d like to start by acknowledging the Wurundjeri, Woiwurrung and the Boonwurrung people of the Kulin Nations, who are the traditional owners of the lands in Melbourne where I am speaking from today. I would like to pay my respect to Elders past, present and emerging, and extend that respect to Elders and First Nations people joining us online today, and on the panel today.

Before I introduce my guests and we get into our discussions, I do have a little bit of housekeeping. So, at the end of the discussion there will be a live Q&A, and if you’ve got a question, you can submit that question in the question section of the GoTo Webinar dashboard. This webinar is being recorded, and if you’d like access to it, it will be available in about two weeks’ time. So, you can either get the recording straight on our website, or we’ll also promote it through AIFS newsletters. And if you’d like to subscribe to that, you can go to the AIFS website and find the subscribe button beneath the AIFS banner.

There will be some related resources that we mention during the webinar, and you will be able to find those in the handouts section of the GoTo Webinar dashboard. So, they’ll be things that we either mention, or they might be resources that you can access to do some further reading about the subject. And finally, there will be a short feedback survey that will open at the end of the webinar, and we’d really appreciate it if you’d just take a very short amount of time to complete that.

So, today we are going to talk about a topic that comes up really frequently in the conversations that I have with service providers, and that’s how to integrate surveys into your service. And in particular, I’m interested in integrating surveys that are designed to collect evidence of client outcomes. And I know that surveys can be used for all sorts of reasons, and no doubt that those things will come up in today’s discussion as well, but I am quite interested in the client outcomes aspect.

We are running this webinar to coincide with two practice slides that we released this week on using surveys to collect data, and writing surveys for beginners. And those resources again, they will be in the handout section of the GoTo Webinar, if you would like to read them after this.

So, we’ve pooled together a really exciting panel of speakers today who are going to share their perspectives with us from their different roles that they’ve got. Unfortunately, Lorna Samuels, who was going to be joining us today, she’s unable to, but she does send her apologies. We are very lucky though to have her colleague Celleste Regan who is here, and she’ll be able to share her practice experience and perspective from the work that she does at VACCA.

So, the panellist bios are available in the handout section of the GoTo Webinar dashboard, so I won’t go over them now. But I will ask each person to introduce themselves, and just give us a bit of an outline of how they use surveys to collect data in their organization. So, Rhonda, I would really like to start with you.

RHONDA SMITH: Sure, thanks, Kat. So, hi everyone, I’m Rhonda. I work at Mallee Family Care in Mildura. And I work in a range of programs that work in the early years space, but specifically I guess relevant to this webinar, is around our supported playgroups that we deliver in the Mildura and Buloke areas, so quite kind of rural communities. And we use those surveys to collect our data around our outcomes around our playgroups, under the DSS funding. So, we’re actually using those surveys with parents and children, to see what kind of impact our playgroups are having.

I would also, before I move – sorry, I would just like to acknowledge the people of the Millewa-Mallee, which is the lands that I live on, and pay my respects to Elders past, present and emerging as well.

KAT GOLDSWORTHY: Thanks, Rhonda. I’m going to throw to Celleste next.

CELLESTE REGAN: Thanks, Kat. So, my name is Celleste Regan, I’m the family services training lead at Victorian Aboriginal Child Care Agency. We use outcomes measurement tools or surveys with the families that we work with at VACCA. So, it’s based on our organisational framework. It’s got some aspects of whether our service is meeting the needs for the children and families. We also use them as a motivational tool and for goal setting, and to ensure that culture and connection to culture is at the forefront of our work with the children and families accessing family services at VACCA.

KAT GOLDSWORTHY: Thanks, Celleste, great to have you. And Kate, if you could introduce yourself, please?

KATE FREIBERG: Yes, thanks, Kate. Hi everyone. I’d like to acknowledge that my feet are on the lands of the Yaggera people.

And for many years, I was a member of an applied research program at Griffith University here in Brisbane. It was focussed on creating positive pathways to child wellbeing, and the project was action oriented, and community based, and it operated in partnership between the university, local schools, and community organisations. And part of my role was to work with our community partners to make sure we had reliable ways to gather information about how the families and the children in the community were going, because we needed consistent processes for capturing the voices of children and families, so we could understand whether our work was making a real difference in their lives.

And so, as the project unfolded, we developed some specific measures ourselves to focus everyone’s attention on children’s social and emotional wellbeing, and on parents’ sense of empowerment and confidence. And we spent a lot of time developing, validating and using those measures in our pathways project. So, we didn’t want all that learning to just end when the project ended. So, the measures were certainly, they came out of research, but they were designed for real people to use in their own real-world settings, and so we wanted to share them beyond our project. And that’s why we set up an initiative called RealWell. It embeds the measures into online user management systems to help practitioners anywhere, use them independently.

KAT GOLDSWORTHY: Thank you Kate, and no doubt I’m sure a lot of people joining us today will be very familiar with those tools that you created and worked on. So, they certainly come up a lot in conversations that I have, but we might touch on some of those a bit later.

Okay, well thanks everyone, let’s get started. The first question I wanted to throw to you all is around something that I’m seeing a lot when I’m working with service providers in my role, and that is that there’s this real tension between collecting data for compliance purposes, and what I mean by that is collecting data that’s basically for reporting requirements, it just gets sent to funders to meet their requirements and needs. So, the tension between doing that and then collecting data that’s actually meaningful for your programs and your services. And so, collecting data to answer the question that you might actually have about the service, or about clients and communities. And I know it can be really difficult to try to use surveys and implement strategies that cover off on both things, and it can be really frustrating for people and difficult to navigate.

So, I just wanted to get your sense and your advice on how do you balance these things in your organisation, in your experience in managing these particular things. So, Celleste, have you got some thoughts on that?

CELLESTE REGAN: Definitely. So, the outcomes tools that we use at VACCA, one we do use which is largely part of our reporting requirements, and we need to use it to meet those requirements. But it’s not necessarily a culturally safe tool, and there are fairly – so, some practitioners and some staff that don’t favour this tool so much. So, a big strategy that we implemented to use this tool, was to develop some culturally safe approaches and resources, and really embed it within our training and practice development and capacity development with our family services practice lead as well. And a lot to do with that training, was about identifying the ways in which we could use that tool as a strength in terms of advocating for families, and using that data in a meaningful way.

So, a good implementation strategy, good training and consistent and continuous practice development with the use of those tools has been a real key to make it relatable for practitioners in their practice, and to support their work, and align it with the other pieces of their work that it would support.

KAT GOLDSWORTHY: So, it sounds like those people then had an opportunity to kind of air their grievances around that particular tool, and then come up with some other ways, or try to understand how that kind of fit into a bigger picture, and then kind of add onto that as well so it was meaningful to them. Is that, am I right in that interpretation?

CELLESTE REGAN: Yeah, most definitely. So, the training that we facilitated for the use of that tool and our own tool, it was a very interactive training. And then we did some regional visits to each office and did some capacity building with all the practitioners, where they had the opportunity to voice their concerns, or any barriers or issues they faced with using the tool.

But then obviously, a big part of that is saying well, if you look at all aspects of practice and how much this tool can inform those other aspects and making it relatable to those pieces of work, so that it’s not seen as an add on, or extra work that’s not necessary. And really trying to highlight that when we’re doing those sessions with them, with practitioners. 

KAT GOLDSWORTHY: Yeah, that’s really interesting. I guess it just shows that there are opportunities within things, that might at first glance not look great, but then there may always be opportunities to adapt and evolve them. Thanks, Celleste.

Kate, do you have any thoughts on managing these two tensions? And you obviously come from a research background, where I imagine you’re quite interested in collecting the meaningful program data, and you’ve worked at organisations, so I imagine you’ve come up against this tension in the work that you’ve been doing, and how you’ve sort of navigated it?

KATE FREIBERG: Yes, indeed. And I think it is concerning when people see requests from funding bodies as the main reason for collecting data, and I worry that that might suggest that they don’t feel supported. I certainly think that is true. And I know, and as Celleste’s just been describing, a lot of services and organisations do take a lot of care to prepare everyone for data, and I believe that’s the key to finding that balance between other people’s needs and your own needs. So, it’s turning the challenge into an opportunity for capacity building, as Celleste has been talking about, really. Because good measures obviously lend itself to use in program evaluation, but it can be also a valuable aid to practice and program planning.

And I’d like to think that surveys, I kind of think of them as listening tools, or conversation starters, because the items in a survey can offer a nice structure for framing discussions with your clients, and they can highlight areas where people feel strong, as well as where they might feel, or that they might welcome some timely support. So, the data they provide can be a great guide for tailoring the way you work with your clients.

But maybe I think what we do need to do is get better at acknowledging what’s involved in that whole process, and people need time, training, organisational support et cetera, to make sure the use of the data, its quality data, and it becomes part of the DNA of the continuous improvement process at the service. When a service offers an evidence-based program, there’s no doubt that they will provide practitioners with training before they implement the program. And I think the same principles need to apply to the way people are prepared to use surveys and other data collection methods, it’s not just surveys.

So, I hope down-to-earth training and mentoring in how to make use of surveys, and the data they provide will become the norm, and help people take control of the data and use it for their own purposes.

KAT GOLDSWORTHY: Yeah, I think they’re such important points to raise, because I think that a lot of people are given these surveys, particularly if there’s a compliance element to it, and just told to go away and do it, and there’s not often either conversations that happen around that. So, it’s really good to hear that in order to potentially get really good-quality data and get good use out of surveys, there is actually all of these other elements that usually need to be involved. Thank you, Kate.

I think I’m going to go to my next question, and I’ll come to you in a minute Rhonda. Because some of the things you are talking about, both Celleste and Kate, it sounds like you’re using surveys in this way to build relationships with people, to have a conversation, which presumably gives you real-time data, enables you to inform your decision making on the spot. But it sounds like it could be quite a time-consuming process, if you are doing those surveys with each individual on a one-on-one setting. I guess a lot of the questions that we get, is there such a thing as this magical short survey that we can just give out to people and deliver in an efficient way, and will give us all the kind of data we need, or tick off all our boxes, does that exist? Rhonda, I might bring you in at this point.

RHONDA SMITH: Short answer is no, there isn't actually a quick fix. And I know this, because at one stage in the last 12 months, we ourselves thought, “Surely there’s an easier and a quicker way of doing this?” And we did a whole heap of research to see what else is out there, and what we actually discovered was that what we are doing is useful and is meaningful and does meet all of our purpose that we want to have.

So, I think really once again following on from what Kate and Celleste have already said, it’s around the way we use the surveys, and how we’re using the information that we get. It needs to be purposeful. If you are just doing the surveys to tick some boxes and to file them away and enter them into your data portal, and you never look at them again, and it’s not guiding what you are doing, I just wonder how useful that is. So, then it’s balance, when we talk about that tension, around how is it being purposeful and meaningful for families as well as the business that we’re doing, and what we’re trying to achieve.

KAT GOLDSWORTHY: Yeah, absolutely. And touching on what Kate said earlier, going back to that purpose and trying to get something out of it, and not just doing it for a tick the box process. I know that that is really challenging, and it’s going to look different in different organisations, because some organisations are very focussed on compliance, and don’t necessarily have the resources and capacity to delve a little bit deeper. But I think it’s a good thing to strive for. Was that a conversation, have you always operated in that way?

RHONDA SMITH: Yes. So, I would say it takes, we’ve had a great – I think our values as an organisation are around that evaluation is meaningful, and that data is meaningful. And that we don’t just do things for the sake of them, for the sake of doing it because that’s what a funder says.

So, what we try, the philosophy we try and take across the whole organisation is, yes there are some things we have to do that our funders ask us to, but how can we make that meaningful. And how can we make that part of our practice, into our everyday practice, so that we’re getting to know clients, and it’s actually a tool for us, not just a tool to meet compliance. So, we’ve had that value. And then part of my role has been helping staff with that and thinking about, “Well, how do I make this useful for them? How do they make this useful for families, so that we’re not just running through a process?” 

KAT GOLDSWORTHY: Yes. And I guess similar to what Celleste was saying earlier. I’m curious Celleste, is short surveys something that you guys at VACCA are concentrated on, or you’re more interested – do you have a short survey that you use, is that something that actually works in practice that can give you all the things that you need?

CELLESTE REGAN: So, with our VACCA community outcome survey, we did develop a VACCA brief community outcome survey, but that’s not what we use within our practice, that’s to support our mainstream colleagues that are doing work with Aboriginal families. So, yes there’s a brief survey that exists, but in practice whether or not it’s brief I’m not sure. We encourage our pracs to have more of a conversational style approach with families, to get that really meaningful information or data. But not just for that purpose, because we want it to really be centred around what the family wants to get out of the service, what their goals are, and support them in identifying areas that they need, or want more connection in or to.

But in short, I suppose I don’t think that you can go and do a survey with a family or a client and get any sort of meaningful response that’s going to support them and the organisation and the practitioner, within a one-off visit in a 20-minute, half an hour timeframe. We say life happens, and for example if someone’s the parent of five children or even two children, however many children, and they’ve got other priorities on their mind at the time, then that data is not going to generally be quality or meaningful for any party involved. So, we encourage deep listening, and practitioners to really take the time that’s needed to get those responses with families.

KAT GOLDSWORTHY: And you find that that works well in practice, when you do, do that?

CELLESTE REGAN: Yeah, most definitely. I think it refers back to what we were talking about before, relating the tool to other pieces of work, and also explaining to the family the use of that tool, and using that as a motivational tool as well. Or as a conversation starter, as I think Rhonda was saying before.

KAT GOLDSWORTHY: Yeah, it’s interesting. Because I guess what I’m hearing is, the reason I asked that question, is people do ask us that a lot about, “Where do we get this short survey? How can it deliver on everything?” And I think the reason that people want that, is a) because it saves time, when they’ve got very limited time obviously, or you need to save time. But also, it doesn’t potentially burden the client as much, potentially. But what I’m hearing you guys say, is actually investing in the time upfront, potentially saves time later down the track. Or not time, but it actually is a worthwhile thing, because it might mean that you create, establish then a rapport and trust, and connection with the person that you are working with, and you’re actually able to get that engagement.

Because I imagine that in the work that you’re doing, it’s not just one-off surveys that you are doing, it’s potentially using these tools at different timepoints as well, across the relationship with the client, is that right?

CELLESTE REGAN: Yeah. So, we, within our program, family services programs, majority of them will take three points of collection. But yeah, in terms of engagement, I think that specifically our community outcomes survey, because it focusses on domains such as heal, protect and connect, and is really culturally centred, that information really underpins where the family can start to heal. And potentially, the issues that have impacted them that have brought them into the situation where they’re needing to access a service at that point in time.

If we look at issues that have impacted Aboriginal communities and people, then definitely you can relate that. The intent and the meaningful response that you could get from that client or family when using that survey, can really underpin a strong source of not only source of engagement, but work to build trust, and start some really meaningful work that’s going to inform long-term healing and change for families and children.

KAT GOLDSWORTHY: Thank you. Kate, I’m wondering what role research and evidence plays into developing surveys, and how important that is. You’ve worked on these tools that you’ve developed, the PEEM tool and Rumble’s Quest, which we can talk more about those, as we’ve already had people asking about those already. But I’m just curious as to how important do you think it is to have standardised and validator-type measures when we’re using surveys around client outcomes, what’s your take and perspective on that?

KATE FREIBERG: I think if you need to collect information about something that a valid and standardised measured has already been developed to address, I’d suggest you use that, because there’s not necessarily any reason to reinvent the wheel. Take advantage of the hard work that others have already done. You were talking about magic wands before, and I’d like to believe in magic, but people who develop reliable measures don’t have magic wants. What they have probably is years of training and deep knowledge of the content area that their survey is designed to address. And that training will have also provided them with an understanding of both the art and the science of survey design.

And they’ve probably also spent a lot of time as we did, in developing the PEEM and Rumble’s Quest, conducting ethically approved research studies to validate the measure, to test its reliability and to explore its performance with a very large representative group to develop some norms as a general reference guide for scorning.

And honestly, maybe not always, but I think in most surveys and measures that are suitable for use in family support practice, they’ll have been developed, these surveys, even though a researcher might have developed, that they’ll have developed it in collaboration with the kind of people who it’s designed to be used with.

So, I think as Celleste has been saying, really the magic is not the survey itself, it’s more the way you use it, and I’d focus on that.

KAT GOLDSWORTHY: Yes, absolutely. I think that those messages are definitely coming through loud and clear. And I think that often forgotten aspects, when we talked about surveys, very focussed on the actual tool itself, and again not things that sit around it.

I think you’ve probably already covered this already, but I did want to get your perspectives. And Rhonda, it would be interesting to hear from you on this, around how do we get people to – how do we boost our response rates effectively when we’re doing surveys. You work in playgroup environments, and obviously there’s probably a lot of people dropping in and dropping out, and I’m hearing loud and clear that relationships are a very clear part of this. But are there other things that we need to consider when we’re trying to get people to complete the survey, people want to give the data, they want the data for themselves to make sense of it, but they want to give those numbers as well to funding bodies. Have you got any advice or experience in that?

RHONDA SMITH: Yeah, absolutely. So, I think once again what’s most helpful to get engagement for families in this process, is to actually make it meaningful for them, and make it sure that they understand the purpose of why we’re collecting the information. So, often families can feel judged around why we’re asking someone these questions, and then, and particularly if we’re not using that to actually build that relationship or engage them, they kind of feel like they’re being done to, they’re a bit of an experiment. So, I think the main way is to be upfront and clear around why we’re collecting that information and what we’re doing with it.

And I think that the other one is to build it into that day-to-day practice as we said, beyond the tool, to actually look at it. So, for example, we do with our playgroups, because you do often have that coming in and out and things like that, we’ve built that process into our intake. So, when a family first starts, we send them a bit of a welcome letter to playgroup saying, “Hey, welcome to playgroup. We’re going to catch up with you in a couple of weeks, and we’d love to get to know you better.”

So, then the practitioners will meet with the family, and they will have a chat and they’ll just talk about, “Hey, we just want to get to know you and your child, what kind of things you want to get out of playgroup? Is there anything that you want to share with us that will help us to meet your needs?” So, it’s that kind of conversational type thing. And then we actually deliver the survey as that conversation style, as like what Kate was saying, using the PEEM as a bit of a framework for the conversation. Some of the times the questions can throw them a little bit, so it’s part of that just having that as a dialogue, and helping them to unpack some of the questions, and to maybe think about what that means for them.

And then something that’s been really useful that we’ve started to do, is to actually have some questions at the end, after we’ve asked those survey questions, to kind of say something, so it becomes meaningful again for the family. So, it’ll be something like say, “Out of all of these things that we’ve kind of talked about today, are there any of these that you actually want to think about, or that you might need some support with?” So, then they can identify areas that they might want a bit more information, or something that they’re struggling with, and then we can help connect them to the supports that they need. And I think that helps them to see the value of that conversation, rather than just I’ve answered some questions and it’s put away in a pile.

We also use that opportunity to highlight their strengths, because a lot of the families we’re working with, the parents and carers we’re working with, they can feel quite judged. So, we take the opportunity, that perhaps if they’ve identified something that they’re doing really well through that survey, that we have a chat around that too, and really help to bring that out. So, that can be a really useful way, and I think it’s just then they’re not as threatened by that experience. And then when we go to do the post-survey later, they are kind of not seeing that as much of a threatening experience.

KAT GOLDSWORTHY: You raise a really good point there about it being a two-way relationship, and it’s not just trying to think about it in that way of not just taking information for our own purposes, but what can you give back. I think that’s a really beautiful message, and something that we probably could all benefit from thinking about a little bit more. Celleste, I’m interested to hear your thoughts on that, just because Aboriginal and Torres Strait Islanders have traditionally been over-researched, and data is just being taken, and not potentially used for the benefit of those communities. So, what kind of training do you give to your staff, what are the conversations that you’re having around trying to frame surveys, and how the data is going to be used and those things, with families. Do you have any insight here.

CELLESTE REGAN: Yeah, I definitely agree that that’s an issue that Aboriginal children and families would face a lot. So, it’s about being really, really careful not to assume that they have to do the survey so get it done. But you also have to be very transparent at the same time and say, “We do collect this data.” I would probably not even use the word data with them.

KAT GOLDSWORTHY: What would you use; what would you say?

CELLESTE REGAN: Gather some information to support our work with you, and help us get more resources in the future and shape our programs, to build a stronger community. Using sort of that more scientific language doesn’t generally sit well and might make people feel like an experiment.

But yeah, I think it’s about being really transparent, that yes, we will collect the data, but at the same time that it’s safe. It’ll be safe with us. We’re there to advocate, and use it in a really meaningful and sensitive and purposeful way. So, it’s about being transparent, and using it as a motivation.

So, as Rhonda was saying, and it might not always be the case that it’s a motivation, but you use your follow-up collection point either as a motivational tool and say, “Look, how much better are you feeling about your connection to country or community now, as opposed to when we first caught up with this survey?” And maybe if they’re not feeling as confident or less confident in that area, then it’s a good conversation starter to say, “Well, what do we need to change? How can we further support you to feel more confident and more comfortable in that space?”

KAT GOLDSWORTHY: Yes, that’s really valuable. I think having some of those script starters can help people as well. People often want to know, “How do I start these conversations? What do I actually say here?” So, thanks, I really appreciate that.

I’m conscious we’ve only got about 10 minutes left of this discussion, so I’m going to move onto the next thing, which is I’m curious – I think you might have chatted on some of this a little bit before, Rhonda. And we’ve talked a lot about surveys as a tool of relationship building, but how do we create that buy-in from practitioners and actual people administering the survey?

That’s a really important part of this too, and I’m running some focus groups at the moment, and some of the feedback that we keep getting through those focus groups, is that we just need, how practitioners and our people collecting the data, they don’t understand necessarily always the importance of collecting the data, and that can be a real barrier. Or they’re not involved in the design of these things and the development or the decision making around it. So, I just wanted to hear from all of you about how do you create that buy-in basically, from people actually doing the survey work? Rhonda, you’re smiling at me, I’ll throw to you first.

RHONDA SMITH: So, just as we are saying it needs to be meaningful for the families that we’re working alongside, it also needs to be meaningful for the people who are delivering the surveys. If they’re just the people that are like, “Please fill in this survey so that we can file away and put in the data,” and then they don’t get to understand the purpose of that, then it’s going to feel like an add-on and extra. So, a lot of what we do is, a couple of different things. But one is actually talking to them about what – so, a lot of people are anxious about doing it too, because they can think it’s intrusive and that they might raise some things for families, and then they worry about how they’re going to support families.

What we’ve found though is, having the opportunity, a little bit of support to maybe do a bit of roleplay, or to watch a bit of modelling of how you can do that as a conversation style, can really help to take those fears away. And also, talking about what are they finding difficult, what is it that they find challenging about that, and often it’s around that awkwardness from their own point of view.

But then what happens is, as they start to do them, they actually do feel a bit more confident. And what’s coming out is, they see the value and the impact that conversation style has had on that family. And so, a lot of the practitioners will actually reflect later to say, “I actually know so much more about that person now than what I could possibly have known just around conversations in playgroup.” So, I think once again they see the purpose for them, and how meaningful that is in the work that they do. 

The other thing is, we try and share the data back with them. So, when we actually collect the data and we are looking at our results to see how has that gone, we try and share that back with the staff, and kind of help them and say, “Oh, what do you think that’s telling, what’s the story behind this? What do we think is happening here?” and get them to contribute to that discussion. And then they may actually identify some area of work that they want to focus on in the next 12 months, because maybe they noticed a theme, and that they’re noticing lots of our parents are feeling like this, and it can actually guide their practice.

So, I think once again, it needs to be purposeful for the staff who are implementing those surveys.

KAT GOLDSWORTHY: You’ve obviously taken a lot of time to have that sort of inclusive sharing approach, which sounds really wonderful.

Kate, you’ve done a bit of work in helping around the implementation of tools. Did you have anything you wanted to add on this particular question, of creating that buy-in and building the confidence?

KATE FREIBERG: Yes. So, as we’ve all been saying, surveys don’t always feel like the most instinctive way to help clients tell their stories. So, the million-dollar question that we’re all, how to help people move from that awkward sense where it just doesn’t feel natural, to use a set of surveys to guide a conversation to the point where they do feel comfortable about it. I actually don’t have the million-dollar answer, but I do know that spending time with your teammates to do some training, share ideas, roleplay like Rhonda’s been saying, and practise together, that’s a really good start. And I think when you practise together as a team, remodelling is a really good idea, because when you see something, it can be quite powerful. Talk about it forever, but when you see it, it really can come home, I think.

And something we’ve recently done to support the way people use the PEEM, is to produce a couple of videos that model different ways of introducing the PEEM in practice. One of them gives just a nice example of how to use it in a standard, but respectful way. But the other one models the process of threading the questions through a natural conversation. And that second video really tries to focus on the purposeful use of PEEM in strengths-based practice. So, it also models the way conversations around the items provide practitioners with opportunities to offer reassurance to their clients, acknowledge the challenges that they may be feeling, but also to recognise their strengths in the responses that they are giving.

And also, because it’s always very uplifting, as Rhonda was saying, to end a conversation with a positive step, that video also shows the way the discussion can prompt on the spot opportunities to set-out some takeaway plans, and steps for achieving them.

We’re going to try and add those new videos sometime soon to the Real Well Resource Guide that people who’ve registered licence to use PEEM can access via their online management dashboards. But we haven’t had the opportunity to do it yet, but it should be there soon.

KAT GOLDSWORTHY: Great, thank you. And just for everyone, people that aren’t familiar with the PEEM tool, Kate, what does that stand for?

KATE FREIBERG: Parent Empowerment and Efficacy Measure.

KAT GOLDSWORTHY: Thank you. And again, we will share those things in the related resources as well, if you want to know more. It’s interesting isn’t it, we’re talking a lot here about trying to build confidence and do all this roleplaying, but I’m personally not aware of where do people get that training? It’s great Kate, that you guys have done that work and if people who are licenced and using those tools, they’ll potentially get access to that help. But do you know if that’s something that anyone offers in terms of getting training around doing, administering these tools and having confident, being confident in conversations with clients around collecting data? Are you guys aware of anything that exists like that?

KATE FREIBERG: I think it comes down to the organisation. I will jump in. Obviously, Rhonda and Celleste will know better from their perspective. But I think it probably just comes down to the organisation to decide, “Yes, we are going to do this.”

I do know of a couple of organisations who do, and they’ve asked me to give workshops, for instance. But I’m not sure what else happens.

RHONDA SMITH: I was just going to say yeah, I think you are right, Kate. And we obviously haven’t had external people come in, but we have some great strengths within our team, and I actually think the practitioners who are delivering the surveys anyway, do have those skills, they just need to see that they know how to build relationships with people. So, they actually have some great skills already, and it’s just more the context of the survey is what probably throws them. And once you remind them that the skills that they actually have already, in building those beautiful relationships, are what you need, then that can be really helpful.

So, yes, I think you can get in some expert help if there isn't anyone in the team, but really, a lot of people who work in this field know how to build positive relationships with people. 

Dr Kate Freiberg 
I think it’s better to do it in-house if you can, that’s way better.

KAT GOLDSWORTHY: Yeah, absolutely. What a great message, Rhonda. Celleste, you’re in charge of doing all the training and that sort of thing at VACCA. What do you guys do?

CELLESTE REGAN: So, I think we’re pretty lucky here at VACCA at this point in time. So, we’ve got myself as a training lead, then we have our practice lead, and we have a central reform and implementation team.

So, just adding to the points that Kate and Rhonda had made, training is really important, but at the same time what’s going to stand the test of time and have that continuous improvement kind of effect, is embedding the culture of that. It’s not just facilitating or developing a training around it and facilitating it, it’s supporting staff, supporting team leaders to feel confident in running sessions with their team in, whether it be a roleplay or whatever kind of methods that they want to use. It supports them in their learning and their feeling comfortable. It also supports consistency in administering the surveys and the data collection side of things.

So, it’s more about embedding a culture within the workplace as well, of strong leadership, and people feeling confident to continuously have these sessions with their staff, and support them when they are feeling not so confident with the tools. 

RHONDA SMITH: The other thing I would add, is just also I think that that have a go, that’s how we learn everything isn’t it. If you have a go at doing something, you might not get it 100% the way you liked it, but then you learn from that experience, and then it helps you to hone your practice.

So, I think this whole, sometimes we can go in thinking we need to be perfect when we’re doing this, but actually just using it as a chance to learn ourselves around what works and what doesn’t. And we also ask our families after the surveys around, “Was there anything around this process that you have feedback around, so that we can start to learn what works for you and what doesn’t?”

CELLESTE REGAN: And I think it as well, just adding onto that, is having a feedback loop. We obviously at VACCA, want Aboriginal people in community informing the development of these tools. But we need to have an ongoing feedback loop, so that when our team that develops these tools goes to refresh that tool, or have a look at it and see if there’s any improvements that need to be made, that we have all that opportunity for collection of what’s really, really maybe not appropriate or working within practice. So, that it then again feeds back to practitioners that they’re heard and supported, and that continuous improvement is managed really well.

KAT GOLDSWORTHY: Thank you. Those are really beautiful notes to end this discussion on, and we’re only at the time where we’re going to be referring to questions unfortunately, because about the topic about using surveys, it feels like it could have been very technical, it could have been a very straightforward conversation, but actually there’s just that skillset around this, and so many aspects that can really elevate our survey use, which you’ve all really touched on. And I love that idea of just having a go, because yeah, we don’t have to get it right. I think just being open and transparent and upfront about what we are doing, and having a really crystal, kind of clear purpose. And hopefully having some good culture around us as well, to support the survey is the way to go. So, thank you all so much.

I’m going to jump into the questions. Not surprising, there’s been a lot of questions coming through. But the one that I’m going to start us off with is something that again I hear a lot in other conversations that I have with service providers, and it relates to creating and adapting surveys for different groups of people, and in this case, we’re talking children. I guess that’s not very surprising, because there is a huge interest in listening to children, and collecting their feedback and getting their input, and hopefully responding to that feedback and those voices as well.

But I know from my own research, that surveys aren’t always that suitable for using with children, particularly very young children. I wanted to get your perspective on that, have you used surveys with children before, what did that look like, how did it go? Or has it not ever been on the cards for you? Kate, I might start with you, just because I know you have experience in working with kids and hunting treasures.

KATE FREIBERG: Yeah, well it’s a big question. I obviously do feel very strongly about the importance of using client voice to guide goal setting and service provision, and my personal philosophy is that to support someone, we need to understand them. And to understand them, we need to listen to them. But to listen to them, we need to provide them with a good way to tell their own story and share information with us, and that is especially true for children.

So, the onus is really on us as practitioners or whoever, researchers, to find a good way to let kids talk, let kids speak. And it really isn't easy to develop something that’s genuinely age-appropriate for young children. We actually spent, I can’t even tell you how many years, a long time, developing a measure of social and emotional wellbeing for primary school aged children. And it focusses on things that are key to their wellbeing, like attachment to school, self-concept, the way they manage their emotions and behaviours, whether they get along with their peers, how they feel supported by the grown-ups in their life and so on. There is really so many things to consider.

But our solution was to embed the validated set of questions into a computer game. The process really offers that same opportunity to have a guided conversation like we were talking about before, but this time it’s done through a game, which is called Rumble’s Quest. And in the game the child goes on an adventure with this mythical creature, and they help rumble to discover who he is and where he belongs. And as he’s learning about himself, he asks the kids about their world and their feelings. So, the context of the story takes the questions out of the abstract, so the children can understand them and respond in a way that’s genuinely reflects their thoughts and feelings.

As I said, that took 10 years. That’s not something that most people are able to do, especially if they need to do something next week with kids. So, I don’t know what a good quick answer is. But with little kids, normally surveys are not a very effective means of talking to them.

KAT GOLDSWORTHY: Yeah, and I think a lot of the other things I’ve read that you’ve touched on is, when you are working with kids, it is about trying to make it fun and engaging, and trying different things, that Rumble’s Quest sounds like it would be a really fun thing to do.

KATE FREIBERG: It’s fun and engaging, but it’s also meaningful, and I think that’s the key. Comprehension at that age is really, it’s really important that they know what you’re asking them, so that they can give you the right, not the right answers, but the answer that they genuinely feel. In using Rumble’s Quest, I’ve been in situation where I’ve been called over by a little child who said, “Oh, I didn’t mean to say that. Can you tell Rumble I didn’t want to say.” They really want to tell what they really feel, and if they feel like they can’t go back and change their answer, they are upset.

So, understanding is key. It’s not just a fun thing, it’s buy-in, and helping them understand.

KAT GOLDSWORTHY: So, maybe testing out some questions, if people are developing surveys for an older cohort, is really making sure that comprehension is there, and that you are taking the time to make sure that kids are able to come back to those questions and given those meaningful answers. Yes, thanks.

Celleste, is this something that you guys do in your work at all, surveying children?

CELLESTE REGAN: So, we haven’t at this stage got a specific community outcome survey for children. However, we do have practice guides and templates that are centred around the child’s voice. So, we try and really embed that within practice. But then I suppose, there’s other considerations when you are thinking about surveying children, and how it could be sometimes difficult or influenced. Ethically, obviously you need the parents around, but then their responses might be influenced with others around as well. So, it’s trying to really balance that approach when you are working with children, to encourage them to feel safe in communicating what their thoughts and feelings and needs are, and doing it in a creative way that can really create that safe space for them.

KAT GOLDSWORTHY: Yes, very sage advice.

All right, we’ve got a few other questions, I’m going to try and get through a couple of them. I’m wondering if you have any advice for using surveys with clients that really don’t want – the answer is probably don’t do it. But I know, I guess something in the context of, and we talked a little bit about this earlier, but in the context of people being very over-surveyed. And also, with clients often attending multiple different services, and they’re all asking questions, and they’re all being asked for data. I imagine again, about relationship building, but have you had experiences with this, and how have you navigated that?

Rhonda, I might start with you, if that’s okay.

RHONDA SMITH: Yes, and that is a big question often the practitioners will ask themselves. So, first of all yes, it goes back to the relationship building, and that if they feel safe to have those conversations, they are more likely to engage in that process. But at the same time, we do have families sometimes where we fear we could do more damage to them putting them through that process, than what is needed. So, we will make a decision around is it appropriate. Our philosophy is, if it’s not going to be meaningful and useful, and potentially cause damage, we’re not doing it. So, we need to keep that ethical framework there as well.

So, I think usually if you put those strategies in place about making it meaningful for the family, making it purposeful, they nine times out of 10 actually do reflect that they quite enjoyed that conversation, or that they got something out of it. But occasionally there are families that are experiencing such complexity and so many things, that that would just be too overwhelming. And we, as practitioners, have an ethical obligation to make that judgement around if that’s safe to do or not.

KAT GOLDSWORTHY: Yes, absolutely. We’ve got about four minutes, I’m just going to move onto the next one which is, this is a really interesting question. Someone’s asked if we’ve got any advice in managing integrity in data collection, when client bias might come into play. And I’m not really sure exactly what this question is asking around client bias, but it could be around clients wanting to please the person, particularly if the service provider, the person offering the service is also doing the collecting the data, that can kind of influence potentially the responses that people are giving. But there might be other forms of bias as well. I guess I’m just curious.

Kate, I might throw this one to you initially, just because again coming from that research role. Is this something that we need to be thinking about when we’re administering surveys?

KATE FREIBERG: I think aspects like, make sure that in terms of a survey, the way you ask questions, if new players to the game of putting a survey together, might inadvertently ask it in a way that does lead towards a particular answer. So, that’s one thing. But I don’t think that’s what this question – I’m not sure what this question is asking.

But I think Rhonda’s already answered this question, I believe, because it’s talking about purpose. When practitioners are clear about their own purpose, or whoever is asking the questions. It’s probably easier for families to see and understand that the information they provide is really valued and it will be used to guide the way that their service provider hopes to work with them in partnership, to set goals and develop plans to achieve them. And I think when that happens, if we’re able to achieve that and people are understanding why these questions are being asked, then they’re more happy to share their genuine thoughts. But I think that really comes back to what Rhonda’s already been talking about

KAT GOLDSWORTHY: Celleste or Rhonda, do you have anything to add on that question?

RHONDA SMITH: I think that it is really like what Kate was saying. But also, I think somebody is around that judgment. And also, we know that a lot of families we have, will initially score themselves quite high. And I think first of all, I share with the practitioners that that’s actually a fairly common thing that can occur, and so not to be too worried about it. But it’s actually around then the skill of the practitioner to kind of unpack that, and to say, “Oh, that’s really interesting. I see you’ve scored yourself really high for that, tell me more about that?” And a lot of the time it’s actually either fear of being judged by the practitioner, so that wanting to please, which can bring in that bias, I guess.

And then sometimes it’s around, you actually don’t know what you don’t know. So, often what we find is, when we go to do the second survey, that’s when they will revisit and you will say, “Oh, look at the difference between these two, this is really interesting. Do you want to have a chat around why you’ve scored differently?” And they often then may share actually that if they retrospectively, they were like, “Oh, I actually didn’t really – I thought I knew what was happening there, but I’d probably rate myself differently now, because I’ve learnt so much.” And that’s actually really valuable once again.

So, sometimes we can get caught up in, “Oh, it’s not going to meet, the funders are going to see that we haven’t shown improvement,” or whatever it is. But I actually find if you can get the story behind that and share that, that’s actually much more purposeful and meaningful, and you can help the family to have that. And then we also share that back with our funders if we can around, “Oh, this is actually the story behind that, and this was really useful for us.”

KAT GOLDSWORTHY: It’s that whole thing of bringing different thought forms of that, or sources of data together really isn't it, to tell a story, not maybe relying on just one form. I could continue this conversation, but it’s one minute to 2.00pm, so I’ve probably got to wrap up.

I just wanted to flag quickly as well, that one of the resources that we’ve published today is around writing surveys, and that can help reduce some bias as well potentially, in the way that you frame questions as Kate mentioned before.

So, we’re going to end there, unfortunately. But I’d just really like to have to say a genuine thank you to our panellists: Celleste, Kate and Rhonda. It’s been really wonderful having you here to discuss all things using surveys. It’s been incredibly valuable, and I have learnt a huge amount, as I’m sure everyone else who attended did. Thank you very much to our audience for joining us and asking some really wonderful questions. And to the AIFS communications team who are doing the production side of this webinar, and to the broader child and family evidence team as well.

So, please subscribe to our newsletters if you’d like a recording of this webinar, and also a reminder that there will be a little feedback survey that pops up at the end, and please take the time to do it if you can. So, thanks once again. We really look forward to joining you at the next AIFS webinar, which will be in two weeks’ time, on behaviour support for children with disability working alongside parents.

Take care, and we will see you soon. Thanks Kate, Rhonda, and Celleste.

KAT GOLDSWORTHY: Welcome back to the extended Q&A of this webinar. Our panellists have very kindly put aside some extra time to answer some additional questions that we’ve had come through.

One of the additional questions that we had is probably best directed at you, Celleste, and it’s some people asking a lot about culturally safe and culturally appropriate survey tools. What are some of the things that you consider when looking for a culturally safe tool? You mentioned earlier that one of the ones that you guys have been given early on was not deemed maybe appropriate or safe, so yeah.

CELLESTE REGAN: Within the context for VACCA and Aboriginal clients, families and workers as well, there’s probably the more obvious things like ensuring that language is culturally safe. We also have our tools with some nice artwork on them, so they don’t look too formal or clinical. But I think the main factor is looking at the tool and considering what the value system is within it. So, there might be questions that are completely irrelevant to Aboriginal children and families, and that are going to score badly on a certain tool, but it’s actually not part of their way of living or culture, or what’s normal or safe for them. Or for us I should say – I’m an Aboriginal woman.

So, I think that that’s probably the most important thing is looking at the values that its aligning with, and what its actually seeking to inform, really. And the approach as well is really important. We encourage practitioners and case managers generally not to sit there on their laptop filling in the survey, because it mimics a lot of other services that have inflicted a lot of pain on the children and families that we work with. And also, it’s not transparent, because they’re not going to feel comfortable sitting there with the screen facing the worker, and the worker speaking for them, really. We need to be transparent and do it as a collaborative approach.

KAT GOLDSWORTHY: Yes, that makes sense. And when you were talking before about using language that’s culturally safe, are you referring to things, just trying to use words that wouldn’t sort of be harmful, or triggering or traumatising in anyway, or are using more safe space language.

CELLESTE REGAN: I think it’s more about traumatising or re-traumatising, or using language that’s associated with certain systems or policies or events that would be specific to Aboriginal people, children and families that would be triggering. And making the language relatable to what’s important to Aboriginal communities. So, using country, connection to culture, family and kinship, that sort of language resonates with the people that we are working with.

KAT GOLDSWORTHY: And it probably goes without saying, but if you’re trying to develop a survey that you’re going to be using with Aboriginal children and families, that you probably need – it needs to be a collaborative effort, and it needs to be really led by Aboriginal people, to make sure that it is culturally safe, right?

CELLESTE REGAN: Yeah, most definitely. So, tools and resources that we have at VACCA, always have working groups with Aboriginal staff on them, and where the opportunity arises, Aboriginal community, Aboriginal families also inform the work and the resources that we develop. So, there’s always that consultation process to make sure that it’s relevant really, and appropriate, and going to meet the needs of the community.

KAT GOLDSWORTHY: Thank you. Thanks, Celleste, that’s really valuable, and definitely answers a lot of questions that we’ve had come through on that topic.

The next question that I want to ask is around when to administer a pre- and post-program survey, and the timing around that. Because we hear a lot from people around it can be very difficult to do it in that very first session when you’re trying to establish rapport and those things, and playgroups are probably a good example of that one. And maybe not necessarily doing it in the very last session when people might not actually show up. Is there wiggle room in there? I might actually just start with Kate first, and then I’ll come back to you Rhonda, if that’s okay. Are there guidelines around that Kate, do you know?

KATE FREIBERG: Yeah, I think it is important, as you said. If you’ve got the room and space to do it, to wait until you have developed some rapport with the families before you start the process of obtaining information from them. It’s best not to include a survey, like for instance in a service intake form, because it’s not an administrative set of information, it’s something that you will use in your practice to support the family. So, if it’s not a short-term one-off program that you’re only going to see people for three weeks, and you have to get it now in this week and that week, then yes, I’d say just be guided by the time that you have available, and make sure you’re not rushing people.

KAT GOLDSWORTHY: And it probably goes back to that point Rhonda was saying before as well, about have a go, see what works, try some things out. Not everything has to be perfect in terms of doing it, and these kind of standards that we talk about.

Did you have anything you wanted to add on that Rhonda, given that you’re coming from a playgroup context?

RHONDA SMITH: Yes, definitely. And it can be really challenging, particularly to do that first survey. So, that’s why I think, as you say that having a go at something and then reflecting on how that worked. Initially we were doing them in the first few weeks at the start of the year of playgroup and trying that. And you’d have quite a number to do, and it would be quite difficult. Which is still kind of the case. But we look at is there opportunity through that, like what we said that welcome letter. So, we aim for about from Week 3 to 4 onwards. So, once they’ve attended three or four sessions, where we’ve started to build a relationship, they’re getting to know what that playgroup looks like and meeting the staff, and then we would do that little meeting with them.

We’ve also learnt at the start of the year when we have an influx is, we’ll actually make some time to do that with them before the playgroup sessions get up and going. So, the families may be that came last year and then they’ve come back, or we know they’re going to be attending, we’ll spend some time with them before the playgroup sessions start. So, once again, it’s within the context of building that relationship.

And then we actually spend a bit of time balancing across, because we can have a hundred and something – I think we survey 150 families across all of our playgroups. So, you do need to manage time around how you’re doing that, but it needs to be within that context of building a relationship.

Same at the end, we’re in the process of collecting our surveys now. We have a due date of those by the end of the year, but we would have started that process – we actually monitor and look well, when did they do their last survey? What’s a reasonable amount of time to make that information valid and to show impact? And then can we slowly start tracking towards getting those done, so that they’re all completed by the time that we need, but we’re not trying to ram them into two or three weeks to have them completed.

So, it actually is having some thought around the timing of doing those, and recognising when a family is ready, so that we’re not putting undue pressure on them.

KAT GOLDSWORTHY: Yeah, that’s really, really great advice. Try and really tailor and tune-in to the people that we are working with, and provide a response that works for them as well.

I’m just going to get final last thoughts on this question, because it’s interesting and it’s kind of related, but thoughts on can we put a time limit on how long you spend with someone doing a survey, is that ever necessary, do you need to do that? Thoughts, Celleste?

CELLESTE REGAN: Well, I mean like in a circumstance maybe where the kids are in a hurry to go to swimming lessons or whatever it might be, that might very well be the family’s priority at that time. But in terms of putting the time limit on the family, I definitely think that that’s a big no-no, and that the data is not going to be relevant anyway, because then they’re not going to care about doing the survey whatsoever. I don’t know if that is helpful or not.

KAT GOLDSWORTHY: Would you agree Kate and Rhonda?

KATE FREIBERG: Yes. And I think if you have limited time available.

RHONDA SMITH: Yeah, and it is a big challenge, because – sorry.

KATE FREIBERG: Sorry, will I keep going?

KAT GOLDSWORTHY: Yeah, keep going, Kate.

KATE FREIBERG: So, just picking up on what Celleste said. If you know that there’s going to be limited time available today, then maybe you might ask the first, a couple of questions, and say, “Well, we might pick up on some of these next time we chat.” And so split it, if you can, if it’s something that you can do that without disrupting the flow of what you’re talking about. Then again, I think it’s what you’ve just been saying, sort of tailor it to what’s going on in people’s lives and work with it.

KAT GOLDSWORTHY: Yes, absolutely. Rhonda, you were trying to say something before.

RHONDA SMITH: Yes, totally agree. Sometimes it can be hard. We have playgroup practitioners who are doing surveys, and then running to another, to actually deliver a group. So, I do have to add in, there is a reality around you can’t take two or three hours. So, there are some time constraints. But it’s being respectful in that conversation, and as Kate’s just alluded to, if you can’t finish that conversation, don’t do all of it and then make it not purposeful or meaningful. It’s, find that kind of natural segue that concludes that conversation, that you can then pick up the rest of the survey questions at another time, so that you’re really being respectful to the family’s time, if you absolutely can’t continue it in that timeframe.

KAT GOLDSWORTHY: Yeah, thank you. I think that well and truly answers that question, and you’ve given some really practical advice for anyone that’s kind of struggling to do that anyway.

I think we’ll leave it there. I’ve definitely taken up enough of your very valuable time. I really just want to thank you again Rhonda, Kate and Celleste for joining us, it’s been such a privilege to be here, and really appreciate your very considered comments and thoughts, and I’m sure our audience did too. So, thank you as well to all of our audience for joining us, and sticking with us this far. We’ll see you at the next AIFS webinar. Bye.