This short article discusses how you can assess the changes you have made to programs and services in response to the coronavirus pandemic.

This short article discusses how you can assess the changes you have made to programs and services in response to the coronavirus pandemic. This article is for staff in community service organisations. It will help you to plan, collect and assess data to understand how clients and staff are responding to any changes in service delivery.

Why collect data to assess service delivery?

Periods of rapid transformation can result in changed client needs or changed conditions of service delivery. This can create both opportunities and challenges for services. Collecting data to assess how clients and staff are responding to changes in your service will give you information that you can use to make adjustments to service delivery.

For example, you might learn through assessment that a particular group are no longer accessing your service, so you might want to do some outreach to this group. You might use data you collect to identify areas for staff training. Or you might be assessing the changes you have made to see if some phone or video service delivery could become permanent.

Planning what you want to find out

Consider what you want to find out through assessment.

For example, you might want to know:

• Are clients’ needs being met through remote service delivery?
• Are there any unintended consequences of the changes?
• How has the change to service delivery made your service more or less accessible for different population groups?
• What are the barriers and enablers for staff delivering services remotely?

What resources you might invest in assessment

If this is a temporary change and you intend to return to your regular program or service as soon as possible, then it is likely you would only invest limited resources in assessment. However, if you are considering making some of these changes permanent, then it would be sensible to do a more thorough assessment of the changed service.

Data collection

Data collection needs to place a minimal burden on staff and clients while being as systematic and rigorous as possible. Below we list three different sources of data: administrative data, client feedback, and staff feedback. If possible, we recommend collecting data from each source. This will give you a broader range of perspectives and strengthen your findings.

Before you plan your data collection, document the changes that you have made – so you know exactly what is different about your service and what changes you want to assess. See this short article about documentation for more information.

Administrative data

Administrative data are data that are collected as part of your regular service provision; data that you collect for DEX¹ is one example. It is easier to use existing data because there is no additional work needed to collect the data, and usually little or no additional burden on clients to answer questions. In most cases, to assess change, you will need to compare current administrative data to data you collected prior to the coronavirus pandemic. Administrative data can be used to explore who is using your service and how. You might then use this knowledge to make service improvements. For example, to increase outreach to a population group that is currently under-represented.

Some types of administrative data are discussed below.

Attendance and demographic data

These data will need to be compared to data collected before the changes to your service delivery. Some questions you might ask are:

• How many clients are using the service? Which population groups (e.g. age, cultural background, location, etc.) are they from?
• What is the frequency of attendance/duration of attendance?
• What is the client dropout rate? Are there differences between the people staying in the program and the people dropping out?

Expenditure data

• What are the costs of service delivery? How does this compare to the costs prior to the coronavirus-related changes?

Client feedback

Collecting data from clients can help you understand how they are experiencing changes to programs or services. If possible, it can also be valuable to make contact with clients who are not participating in your service to understand the barriers to their participation

Options for collecting client feedback:

• Asking a few set questions at the end of a client session
• Having a focus group at the end of a group session
• Sending out a short (five questions or less) survey to clients (this could also be useful for needs assessment)

Staff feedback

You are probably already getting informal feedback from staff about the effects of service changes. With some planning and structure, you could more systematically collect staff feedback to see how programs and services are operating in the current environment.

Data collected from staff could address:

• Staff training or support needs
• Implementation (e.g. Are the service changes being delivered as intended? What is working? What is challenging? Has time spent on different tasks changed?)
• Staff perceptions of client need and engagement with the service and/or with others, if in a group program
• Unintended consequences

Options for collecting staff feedback could include:

• A short focus group in a team meeting
• Short questionnaires (could collect qualitative data, quantitative data, or both)
• Short interviews

Analysis

Making sense of the data you collected does not need to be complicated. The type of analysis you will do will depend on the amount and type of data you have collected. For example, if you have done a focus group with staff or collected data from clients using a three question mini-interview, you could look for common themes and ideas. You could then write this into a single-page summary, including any findings from administrative data, and present this to your team and managers for discussion.

Next steps

As social distancing restrictions are lifted, some services may return to face-to-face delivery. However, this might take longer for some services, particularly those that work with vulnerable clients, and some may prefer to continue using phone or video conferencing. Some services may choose to offer a range of options to clients. The data you have collected can help you make decisions about how you deliver your service now and in the future.

If you decide to make some of the changes permanent, you may need to consider how this could influence the outcomes your program or service aims to achieve. You might need to review your program logic. For example:

• Have the outcomes changed?
• Are the time frames still appropriate (e.g. it could take longer to achieve the same outcomes)?
• Is there evidence that the new activities will lead to the intended outcomes?

Additional resources

For more detailed resources on evaluation methods and strategies there are a range of resources in the AIFS Evidence and Evaluation Support Program planning and evaluation guide.

Families and Children Activity service providers can also contact AIFS Evidence and Evaluation Support for free advice and support.

Further reading

• Evaluating and improving e-mental health services: A guiding framework for evaluating e-mental health services
  Ontario Centre of Excellence for Child and Youth Mental Health and Child Mental Health Ontario
• Responding to COVID-19 - Principles of rapid innovation and evaluation
  Australian Research Alliance for Children and Youth (ARACY).

From AIFS Evidence and Evaluation Support:

• Program logic: Step by step
  • A guided tour through program logic
  • How to develop a program evaluation plan
  • Downloadable program logic checklist  - [PDF, 195 KB] 
  • Getting the most out of program logic: case studies of how program logic can be used