Evaluation and Value for Money
Evaluation and Value for Money
Program evaluations assist organisations to plan, develop and improve their programs, with the aim of improving outcomes for clients (Alston & Bowles, 2003).
This snapshot of evaluation and Value for Money is one of a series of CFCA resources on evaluation. For further information about other evaluation approaches and evaluation in general, see CFCA's research and evaluation resources.
What is Value for Money?
In the context of program evaluations, Value for Money (VfM) is a term used to describe a systematic process of understanding whether an investment (of money, time or other resources) in an intervention represents good value. In crude terms, VfM is a tool to assess whether an intervention offers the most "bang for the bucks" (Levin & McEwan, 2001). There are three elements that are usually considered in a VfM assessment:
- Economy: this is the cost of the program inputs (e.g., people or resources). In order for those costs to be considered "economic", the inputs should have been acquired at the least cost for the required level of quality.
- Efficiency: this relates to the value of program outputs in relation to the total cost of program inputs (at the required level of quality).
- Effectiveness: this is achieving program outcomes in relation to the total cost of program inputs (at the required level of quality)
Some organisations also add a fourth dimension of Equity to VfM assessments, which refers to ensuring that program benefits are distributed fairly to participants, stakeholders and/or the community (Fleming, 2013).
Figure 1 illustrates the relationship between these four dimensions.
Figure 1: The four elements of Value for Money
Source: Besley, 2015
It is important to understand that VfM assessments are not audits of the financial arrangements of programs or organisations, or intended to gauge compliance with expenditure requirements; rather VfM is about "weighing the costs of interventions against their outcomes, then choosing the best intervention" (Levin & McEwan, 2001).
Why include a Value for Money assessment in an evaluation?
It is not essential to conduct a VfM assessment as part of undertaking a process or impact evaluation. However, including an assessment of costs alongside an assessment of outcomes can allow for more comprehensive conclusions to be drawn about the value of an intervention (Levin & McEwan, 2001).
Figure 2 illustrates the relationship between a VfM assessment and an evaluation focused on program processes and/or outcomes.
Figure 2: A model showing the integration of cost analysis with program evaluation
Source: Calculating the Costs of Child Welfare Services Workgroup, 2013
Value for Money assessments are often also used as part of procurement decisions, and it is important to be clear about which aspect of the program cycle the assessment relates to (Fleming, 2013).
Advantages and disadvantages of a Value for Money assessment
There is some debate in the evaluation community about how appropriate VfM assessments are in the context of social policy.
The arguments in favour of undertaking a VfM assessment as part of an evaluation are that such assessments can help to:
- demonstrate that program spending and resources produce a return on investment;
- support public and stakeholder confidence in the program;
- justify both ongoing investment in the program and efforts for continuous improvement; and
- demonstrate and validate models that encourage the most effective resource allocation.
The disadvantages of undertaking a VfM assessment are that such assessments:
- reflect the concerns of funding bodies, rather than program beneficiaries;
- can lead to an emphasis on activities that are easy to measure or to implement, rather than activities that may be innovative but are more difficult to measure; and
- emphasise quantitative outcomes, rather than more complex social change (Fleming, 2013).
The Value for Money process
An important first step in most VfM assessments is to undertake a process of identifying and analysing program costs. The notion of costs in this context includes all of the resources that are required for the program, not just the money spent (Calculating the Costs of Child Welfare Services Workgroup, 2013). In some circumstances this also includes what is given up by using those resources in one way rather than another way (known as an opportunity cost). Human time and energy, and building costs and materials all have associated costs that need to be valued as part of a VfM assessment (Levin & McEwan, 2001).
When identifying total program costs, it is important to be aware that the information included in budgets and expenditure statements usually does not reflect all of the costs that need to be considered in a VfM assessment. Budgets often do not include resources such as volunteer time, donations and other unpaid or in-kind inputs. The usual approach for estimating costs is known as the ingredients method. This involves identifying all of the "ingredients" required to create or replicate the program and calculating their value. Program ingredients might include the cost of personnel (including volunteers), facilities, equipment and materials, other program inputs and inputs required from clients (Levin & McEwan, 2001).
Box 1: Key concepts and principles in VfM
- Costs are resources used, not money spent.
- The perspective of the analysis affects the costs considered, that is, the costs to whom?
- Costs must be adjusted to account for the passage of time.
- Costs may be variable or fixed.
It is important to distinguish between the average cost of providing a service and the marginal cost of providing one more unit of service. (Calculating the Costs of Child Welfare Services Workgroup, 2013)
Identifying a suitable method
There are a number of methods that can be used when undertaking a VfM assessment. Some of the most common methods are described below. A decision about which method to use is based on what the evaluator is seeking to understand. Regardless of the method chosen to undertake a VfM assessment, there are some common questions.
Box 2: Common questions for all VfM methods
- How will value be measured? Which elements will be included (economy, efficiency, and effectiveness)? Will the assessment also include equity?
- Who will decide value? Will stakeholders participate?
- Is the evaluation assessing one project or comparing a number of projects?
- Will the evaluation be based on monetary terms and/or will it rely on proxy measures of value?
- How will the evaluation process make sure costs and benefits are agreed and transparent?
- Will these methods be used in ways that promote/enable participation and accountability to communities and partners? (Fleming, 2013)
There are three variations of a traditional cost analysis. These are described below.
Cost effectiveness analysis
When to use: To compare alternative programs that have the same aims.
Source: Davidson, 2009.
Cost effectiveness analysis is a method of comparing the costs and effects of two or more alternatives with similar objectives. This method relies on determining a measure of effectiveness that reflects as closely as possible the main objective of the alternatives. It is important that the measure of effectiveness is reliable (it produces the same results on repeated occasions with the same individuals) and valid (it measures the concept it is intended to measure). For this reason, the objective used to measure effectiveness is usually an intermediate objective, rather than a longer-term objective.
There are three common methods used to establish effectiveness: experiments, quasi-experiments and correlational evaluations. Once estimates of the costs and effectiveness are obtained, they are combined to establish the cost-effectiveness ratio, which is the cost of the alternative divided by the effectiveness. This ratio illustrates the cost of each unit of effectiveness. Some cost effectiveness analyses also calculate the effectiveness-cost ratio (effectiveness divided by cost), which illustrates how many units of effectiveness are obtained by a single unit of cost (Levin & McEwan, 2001).
Cost utility analysis
When to use: There is a need to consider individual preferences; there is a need to consider multiple outcomes.
Source: Davidson, 2009.
A cost utility analysis is a method of comparing cost-effectiveness that also takes into consideration the relative utility of the alternatives to identify the program that provides the given amount of utility at the lowest cost. Utility in this context refers to a process of identifying and weighing multiple outcomes, which is usually done in consultation with program stakeholders.
There are several techniques used to undertake an estimate of the utility of each alternative, sometimes known as a decision analysis. The most basic approach is called multi-attribute utility theory, which provides a set of techniques to a) quantify the utility of different program attributes, and b) combine the utility of each attribute to generate an overall measure of program utility. Once this estimate has been obtained for each alternative, the analysis proceeds much like a cost-effectiveness analysis; that is, a cost-utility ratio is calculated in order to rank alternatives and determine which program has the most utility for the lowest cost (Levin & McEwan, 2001).
Cost benefit analysis
When to use: To compare alternative programs that have different aims; to compare alternative programs in different sectors; to understand whether program benefits outweigh costs.
A cost benefit analysis can be used to compare monetary estimates of benefits and costs; that is, to assess whether the benefits of an alternative outweigh the costs. This approach allows comparisons of several alternatives to understand which one provides the greatest amount of benefits relative to the costs.
The concept of benefits, and expressing those benefits in monetary terms, is central to a cost benefit analysis. Some benefits are relatively simple to value, whereas others are harder to put in monetary terms. The notion of "willingness to pay" is commonly used to place values on outcomes, which is the maximum amount an affected individual would be willing to pay to receive desirable outcomes (note: this does not mean that they will be required to pay this amount).
There are several methods that can be used to value benefits - commonly, experiments, quasi-experiments and correlational evaluations. Once the outcomes have been given a monetary value, there are three main methods that are used to compare those benefits with the costs: cost benefit ratio, net benefits and internal rate of return (Levin & McEwan, 2001).
Social return on investment (SROI)
When to use: There is a need to consider social value.
Social return on investment (SROI) is a method for understanding a broader concept of the value of an intervention by incorporating social, environmental and economic costs and benefits. Organisations can use this method to tell a story about the change generated by the intervention, based on information drawn from a variety of sources (e.g., case studies, quantitative and qualitative data, financial information). To use this method successfully, it is critical that good and detailed data are available. The method also relies on high levels of engagement with stakeholders and a participatory approach to evaluation.
SROI is based on seven principles: involve stakeholders, understand what changes, value what matters, include only things that are material, do not over-claim, be transparent, and verify the results (Nicholls, Lawlor, Neitzert, & Goodspeed, 2012).
Evaluation Methods for Assessing Value for Money <www.betterevaluation.org/en/resource/assessing-value-for-money>
A Guide to Social Return on Investment <socialvalueuk.org/what-is-sroi/the-sroi-guide>
Cost Benefit Analysis </www.bca.com.au/publications/policy-essentials-cost-benefit-analysis>
Implementing a Value for Money assessment
It is often possible to include a VfM assessment as a complement to a standard evaluation plan (Calculating the Costs of Child Welfare Services Workgroup, 2013). However, it is important that an evaluator who is experienced in these methodologies conduct the assessment. All VfM methods have standards that should be applied to assure the quality of the analysis. Some methods, such as social return on investment, have a formal assurance and practitioner accreditation process (Nicholls et al., 2012).
- Alston, M., & Bowles, W. (2003). Research for social workers (2nd ed.). Crows Nest, NSW: Allen & Unwin.
- Besley, M. (2015). A practical approach to evaluating the value for money of development programs. AES Conference Presentation, 16 September 2015. Retrieved from <www.aes.asn.au/images/AES_Presentation_15.09.15._AES_distibution.pdf> .
- Calculating the Costs of Child Welfare Services Workgroup. (2013). Cost analysis in program evaluation: A guide for child welfare researchers and service providers. Washington DC: Children's Bureau, Administration for Children and Families, US Department of Health and Human Services.
- Davidson, E. J. (2009). Value for Money: What does it mean for evaluators and commissioners? Presentation to the ANZEA Wellington Branch. Retrieved from <realevaluation.com/pres/valueformoney-anzea09.pdf>.
- Fleming, F. (2013). Evaluation methods for assessing Value for Money. Melbourne: Australasian Evaluation Society.
- Levin, H., & McEwan, P. (2001). Cost effectiveness analysis: Methods and applications, (2nd ed.). Thousand Oaks, CA: Sage.
- Nicholls, J., Lawlor, E., Neitzert, E., & Goodspeed, T. (2012). A guide to social return on investment. London, UK: SRoI Network.
CFCA has developed a range of resources on evaluation, the full list of resources is available here: aifs.gov.au/cfca/practice-guides#evaluation
Authors and Acknowledgements
This resource sheet was developed and written by Sharnee Moore, Research Fellow, with Elly Robinson, CFCA information exchange.
The feature image is by istockphoto/kitzcorner, CC BY 2.0(link is external).
This sheet will provide program practitioners and providers with a range of resources relating to conducting program or practice evaluation.
Key considerations for those thinking about evaluating the outcomes or impact of a program for Indigenous families and communities
This resource presents nine principles to apply to your evaluation to maximise its use.
This NCPC Issues paper examines evidence for the impact of media-based social marketing campaigns.