Building confidence in family and relationship support services across Australia

Building confidence in family and relationship support services across Australia

7 November 2013

The importance of introducing measures of fidelity into the ongoing evaluation of family and relationship support services

Written by T.R. Heard1, 4, K. Hamall1, 2, 3, F. Kay-Lambkin2, 3, K. Inder1, 2, and K. McGill2

Family and relationship services across Australia have progressively introduced the collection and reporting of standard evaluation measures into routine practice. This includes the collection of occasions of service count data; service user demographic information; pre and post process and knowledge impact measures; and, to a lesser degree, the collection of pre and post psychological impact measures (e.g. Kessler Psychological Distress Scale K-10 and/or measures of family functioning)

The introduction of standard evaluation measures provides a sound basis for the ongoing evaluation of family and relationship services. An important limitation of this approach is demonstrating with confidence that changes in process or impact outcomes for a service user are attributable to the family and relationship intervention itself, and not the result of other factors. Other factors within the context of evaluating a family and relationship intervention could include:

  • variability in the delivery of the prescribed intervention between service staff; and/or
  • factors external to the intervention, such as participation in a parallel government funded program such as the better access to mental health program.

One possible approach to address this limitation is to introduce measures of program fidelity into the routine evaluation process for family and relationship services. Program fidelity is broadly defined as the degree to which an intervention is delivered as prescribed. Measures of program fidelity directly measure a program's level of integrity and the level of confidence that can be drawn from the program outcomes, i.e. the degree to which the intervention is driving any reported changes in the process and/or impact outcomes associated with the program's evaluation. The following diagram demonstrates the practical role of fidelity in the ongoing process of evaluating family and relationship services in Australia.  

The practical role of fidelity in the ongoing process of evaluating family and relationship services in Australia.

 The degree to which a program is delivered as prescribed, is traditionally measured across five domains:5  

Domain Description Example
1. Program adherence To what degree was the program delivered as described in the intervention guidelines? Make a record of components of the intervention that were delivered and those that were not and why.
2. Completeness and dosage Was the program delivered over the prescribed timeframe and to the required level of intensity? The number of sessions that the intervention was conducted over and if changes were made, why.
3. Level of quality Was the program delivered to a satisfactory level of quality? Seek feedback from participants about the quality of the intervention.
4. Participant responsiveness To what degree were participants engaged in the program? Would participants recommend the program to others?
5. Program differentiation To what degree were the critical elements of the program present during the implementation of the program? Complete an intervention checklist to monitor the completion of key intervention components.

In order to build confidence in the integrity of an intervention program, family and relationship services could consider incorporating at least one measure of program fidelity across each of the five domains.  

Further reading

For more information about how to incorporate measures of program fidelity into standard evaluation measures, please refer to the following online information:

Presentation: Partners in Depression: Ensuring the fidelity and quality of a program during its national dissemination (PDF).

Published report: Supporting effective program delivery for those who care: Partners in Depression program delivery analysis (PDF).

1. University of Newcastle.

2. Hunter Institute of Mental Health.

3. University of New South Wales.

4. Hunter New England Local Health District.

5. Dusenbury, L., Brannigan, R., Falco, M., & Hansend, W.B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research: Theory and Practice. 18(2), 237-256.

Add a comment

Related topics

Need some help?

CFCA offers a free research and information helpdesk for child, family and community welfare practitioners, service providers, researchers and policy makers through the CFCA News.

Subscribe

CFCA News

Sign up to our email alert service for the latest news and updates

Subscribe