The Australian Government Department of Social Services (DSS) commissioned the Australian Institute of Family Studies (AIFS) in 2014 to assist with the establishment of a panel of experts to help agencies funded under the department's Families and Children Activity. The Families and Children Activity provides early intervention and prevention programs and services to improve the wellbeing of families and children.

The panel of experts (the Panel) will help service providers deliver evidence-based programs and practices and to continue to build this evidence base through evaluation, with a focus on prevention and early intervention approaches. The Expert Panel project is funded from 2014-19.

This article outlines the different methods by which the Panel will provide advice and support to the sector, activities undertaken in the first year of the project and future initiatives.

Background

In a series of consultations conducted by DSS with the family support sector in 2013, there was broad-based consensus for increasing the use of evidence-based programs and practices, encouraging innovation and changing the focus of data collection from just measuring outputs to true outcomes measurement. There was also recognition of the need for additional assistance to achieve these goals across the sector.

At the Family & Relationship Services Australia (FRSA) Senior Executive Forum in March 2014, former Minister for Social Services Kevin Andrews announced his intention to establish a panel of experts that would assist services with these needs. The Panel's overarching objective is to increase the use of evidence-based programs and practices in organisations funded under the Families and Children Activity and to contribute to the evidence base by trialling new or adapted approaches and sharing the knowledge. This approach is ultimately designed to improve outcomes for children and families.

It was proposed that the Panel would comprise research, practice and evaluation experts from a range of service delivery, research, training, academic and service support backgrounds. The work of the Panel will supplement, and not replace or circumvent, research and evaluation activities that are already being undertaken by the sector.

Project development and delivery

AIFS were funded to help establish the Panel through its Child Family Community Australia (CFCA) information exchange, which will also operate as the dissemination point for panel-related projects. This will facilitate the sharing of knowledge on what works and what doesn't work to improve outcomes for children and families.

A number of legal and procurement issues in the project's early stages resulted in DSS and AIFS opting for parallel processes for selecting and contracting organisations. DSS would establish and manage the Panel and AIFS would establish and manage an "Industry List". This distinction and the related processes are explained in more detail below.

The main tasks involved in the initial stages of the project were to:

• establish a panel of experts to build sector capacity in the areas of program planning, program implementation and program evaluation/outcomes measurement;
• develop a system to match the needs of the sector to the most appropriate and available source of support from organisations on the Industry List; and
• provide support to the Communities for Children Facilitating Partner (CfC FP) subactivity in regards to evidence-based program requirements within their funding agreements.

Each of these is described in more detail below.

Establish a panel of experts to support service providers to improve outcomes for families

AIFS and DSS worked together to establish the Panel through an open tender selection process in late 2014. Eighty-one submissions were received and assessed against a set of criteria that focused on the applicants' expertise in program planning, program implementation and/or program evaluation/outcomes measurement. Interested parties were also required to have experience working with service providers on these issues or to indicate that they had the capacity to do so. Applicants were also asked to indicate if they had experience working with Indigenous organisations.

Forty-two organisations or sole traders were appointed to the Panel as a result of the tender process. DSS entered into Deeds of Standing Offer with the 42 successful applicants.

It is expected that a further selection process will be conducted to refresh the panel membership approximately three years after the initial establishment.

The Steering Committee (discussed below) and DSS will establish priorities to inform the Panel's activities. Broadly, Panel members may be engaged to undertake one or more of the following tasks:

1. provide implementation support and training for organisations in the use of evidence-based programs and practices;
2. support the development of outcome measures that organisations can use to evaluate the extent to which they have helped their clients;
3. provide training and support in the evaluation of outcome measures;
4. support organisations to trial and evaluate new approaches, particularly in prevention and early intervention; and
5. conduct other research, quality assurance and evaluation activities.

DSS will directly contract Panel members to deliver identified services to the sector in support of the government's strategic priorities for the Families and Children Activity. DSS will issue a Request for Quotation to select Panel members for each project it intends to fund. Successful Panel members will be contracted to deliver services through an Official Order raised by DSS.

Initial projects

DSS has been working with a number of Panel members to implement the Panel's first two projects. These projects are designed to support service providers and enhance the great work they already do with children and families.

The first project involves three Panel members assisting up to 41 Families and Children service providers to improve the way they measure outcomes for families. Panel members will assist service providers to work proficiently within the department's Families and Children Performance Framework and Program Logic. The service providers who would benefit from receiving support were identified through an online survey sent to more than 100 organisations in March 2015.

For the second project, four Panel members are assisting up to 34 service providers who are delivering the new Children and Parenting Support Component to improve their program planning and program implementation capacity.

Develop a system for service providers to access support from the Industry List

During the tender process, organisations and sole traders who were appointed to the Panel were also given the opportunity to be included on an Industry List. The Industry List is available to service providers for accessing support for program planning, implementation and evaluation, using their existing funding. All Panel members asked to be included on the Industry List.

AIFS has developed a database that allows the needs of a service provider to be matched with Industry List members who are able to offer the support they require. In the first instance, service providers who have identified a possible need can contact the CFCA information exchange to discuss their ideas. CFCA staff members work with the provider to formulate a proposal that can be distributed to Industry List members who have the expertise in the required area and can deliver services in the provider's geographical location. The Industry List members are invited to quote for the work, using hourly rates that were set out in their tender response documents. CFCA will forward any quotes to the service provider, who can then choose and directly contract the most appropriate Industry List member.

To help to track projects delivered with Industry List assistance, Industry List members are asked to submit a project plan once it has been developed in consultation with the service provider. The project plan needs to clearly outline the project objectives, outcomes and timeline. At the completion of the project, CFCA will work with the service provider and Industry List member to disseminate any significant findings or other activities of interest through the information exchange.

Provide support to the Communities for Children Facilitating Partners

Communities for Children Facilitating Partners (CfC FP) are a sub-activity under the Families and Children Activity that aims to deliver positive and sustainable outcomes for children and families in disadvantaged communities throughout Australia. CfC FPs are place-based and offer a whole of community approach to support and enhance early childhood development and wellbeing for children from birth to 12 years.

CfC FPs are committed to evidence-based practice and actively support the provision of services that will improve outcomes for children and families. Specifically, new grant agreements for CfC FPs require that from 1 July 2015, at least 30% of funding used for direct service delivery should be used to deliver high quality evidence-based programs.

For the purpose of this particular requirement, what constitutes an "evidence-based program" has been approached in two ways: CfC FPs can choose from a list of evidence-based programs or ask CFCA information exchange researchers to assess an alternative program. These approaches are described in more detail below.

List of evidence-based programs

A set of criteria¹ was developed by CFCA that were specifically tailored to the purposes of this project. The criteria were used to determine a list of evidence-based programs that would address the CfC FP program objectives. Twenty-six programs met the criteria and information on each of these programs was published in online profiles² in March 2015. A further four programs were added to end 2015. The preference is for CfC FPs to consider first whether they are able to implement one of the approved programs.

Assessment of alternative programs

There is also provision for CfC FPs to submit an existing or other program for inclusion in the 30% requirement for assessment by CFCA against a set of "rigorous enough" criteria.³ The rationale behind the inclusion of this method of assessment was as follows:

• The evidence-based programs presented in the online profiles are limited in relation to CfC program objectives. For example, many programs address parenting skills, but few programs specifically have strengthening communities as their focus.
• The evidence-based programs were chosen from existing databases and clearinghouses; for example, Blueprints for Healthy Youth Development. However, it is recognised that good practice and programs exist outside of these sources. Providers were encouraged to submit programs that they thought would also meet these criteria.
• The assessment of other programs allows CFCA to consider the strength and quality of the evidence base of existing programs, their program logic, staff qualifications and program activities, and provide feedback on how the Industry List or other sources can help to strengthen these programs. Ultimately, it is hoped that this will increase the range of programs available for delivery in order to meet the 50% requirement currently flagged for 1 July 2017.
• The approach offers a more gentle, bottom-up, guided approach to the delivery of evidence-based programs, rather than a top-down directive.

Several discussions with CfC FPs, DSS and Grant Agreement Managers occurred in order to create a system that works as well as possible for all involved.

The web form for submission of "other" programs opened in March 2015. To the end of December 2015, 68 programs have been submitted for assessment, with 13 being fully approved and 43 being provisionally approved (see below; the remaining programs are yet to be assessed or were not approved). Programs that are assessed as fully approved are also assessed against the criteria for the evidence-based program profiles, and are included on the list if the criteria are met.

Provisional approval

Early in the process of program assessment, there was an acknowledgement that CfC FPs were delivering many valuable programs that could meet the criteria for full assessment with some additional time and assistance. Accordingly, a new rating of provisional approval was introduced into the assessment procedures. In order to receive a provisional approval from CFCA, the program's research and/or theoretical background must have been articulated and some form of evaluation conducted in the past (which may not meet the rigorous evaluation criteria at this stage).

In addition, CfC FP providers were required to submit a plan for meeting the five criteria for this program, in order to gain full assessment by 30 June 2016. In regards to the evaluation criteria, there was a recognition that a more rigorous evaluation methodology may take longer to implement, in which case the evaluation must be in the field by 30 June 2016 and due for completion before 30 June 2017.

A few common issues have arisen in relation to provisional, as opposed to full, approval. While programs may technically meet the criteria, there is often a lack of clarity about how the theoretical/research base relates to the program activities. In turn, it may not be clear how and why these program activities are the best way to meet outcomes for the target group. This is particularly problematic when an evidence base appears to be retrospectively fitted to a program or the program has been in existence for some time and the evidence base is dated.

We have encouraged providers to consider first and foremost the programs that meet the criteria for the evidence-based program profiles. Many programs currently being offered have been created to meet the needs of a target group but don't have a strong evidence base or good-quality evidence that the program is effective. The program profiles showcase programs that have both. Service providers are encouraged to consider how better outcomes may be achieved for families and children if these programs are implemented effectively.

The process of provisional approval has required a very close working relationship between CFCA researchers and service providers. Detailed written feedback has been provided to CfC FPs that helps them to understand what is needed to obtain full approval by 30 June 2016. A key aspect of provisional approval was also to encourage service providers to purchase assistance from the Industry List in order to meet the additional criteria.

Steering Committee

A high-level steering committee was established early in the project to provide guidance and advice on the work of the Panel, including:

1. identification of effective programs;
2. opportunities for innovation in the Families and Children Activity; and
3. identification of emerging issues or trends in service delivery and evidence-based programs.

The Steering Committee comprises members who bring extensive experience and expertise in the area of children and families research.

The department and the former minister were consulted in the selection of Steering Committee members and the former AIFS director, Professor Alan Hayes, invited members to sit on the committee. Steering Committee members are not remunerated; however, travel costs for meeting attendance are covered by project funds.

The Steering Committee will convene two to three times per year, including face-to-face meetings and teleconferences, as needed. Relevant AIFS and DSS staff will attend Steering Committee meetings as required.

Expert Panel project evaluation

The evaluation plan for the Panel project is focused on assessing the implementation and impacts of the 5-year project. The evaluation will use administrative data collected by AIFS and DSS over the lifespan of the project, in addition to collecting data from Panel members and Families and Children Activity service providers.

A central strategy in the evaluation will be to combine qualitative and quantitative information from these different sources and informants. This allows findings from each of these sources to be drawn together to produce robust and reliable evaluation findings.

The evaluation has a broad focus on both the implementation (process evaluation) and outcomes resulting from the project (outcomes evaluation) across the 5-year life of the project. The core evaluation question is:

To what extent does building the program planning, implementation and evaluation capacity of service providers lead to an increase in the use of evidence-based programs designed to improve outcomes for children and families?

A process evaluation will consider:

• the extent to which the program logic model and activities are based on a well-founded theory;
• how the project identifies the needs of the target service sector and the extent to which these needs are met;
• how the project works and how it was implemented; and
• the extent to which the project has facilitated a sustainable model of change.

The outcomes evaluation will be linked to the short-, medium- and long-term outcomes articulated in the program logic for the project.

Future

Knowledge and lessons from Panel activities and partnerships with sector organisations will be disseminated to the sector through CFCA. It is envisaged that dissemination will occur using existing communications strategies adopted by CFCA, including publications and resources, webinars, podcasts and short articles.

Information regarding the Panel and associated activities will be disseminated through existing AIFS and DSS channels. In particular, CFCA News,⁴ the fortnightly information exchange e-newsletter provided to over 6,300 subscribers, will be a key dissemination point for information on the Panel.

The Panel reflects the government's commitment to Australian families and responds to calls from the sector for government support in strengthening service design and delivery. For further information, please contact Elly Robinson ([email protected]).

Endnotes

1 See <tinyurl.com/q2e6tdb>.

2 See <apps.aifs.gov.au/cfca/guidebook/programs>.

3 See <tinyurl.com/pwwombj>.

4 Subscribe at: <aifs.gov.au/cfca/subscribe>.

Elly Robinson is the Executive Manager - Practice Evidence and Engagement at AIFS. Marian Esler is the Director, Research and Data, in the Family Safety Taskforce, DSS.