Building a positive evaluation culture
Key considerations for managers in the families and children services sector
Download Practice guide
By using evaluation to assess what we do we can make better decisions and improve services for our clients. Evaluation works best when the whole organisation embraces it, the organisation acts on evaluation findings, and there is shared responsibility for evaluation. This can be called having an 'evaluation culture', and managers are essential to building a positive evaluation culture.
In this resource we share some tips and key considerations for managers who want to help build an organisational culture where evidence and evaluation are valued and can inform and improve practice. Although staff at any level of an organisation can be involved in evaluation, and be evaluation champions, managers often have a special role in supporting and facilitating a positive evaluation culture.
This resource is predominately for managers but may also be useful for others in an organisation who wish to promote an evaluation culture.
We have drawn on a recent research project to show what works for managers to effectively engage in building such an organisational culture.
Before we start - some key concepts
Some key concepts used in this resource can be defined in different ways, here's a description of how we use them here.
What is an evaluation culture?
An evaluation culture refers to a commitment within an organisation to deliberately seek out, act on and value evidence to better manage and deliver programs and services.
What is organisational culture?
Organisational culture is the shared beliefs, values, assumptions and ways of doing things within an organisation. This culture often forms as groups work together and encounter challenges but is it also shaped by leadership and executive guidance.
What is an authorising environment?
This refers to the structures, rules, processes and people in an organisation that have the 'authority' to influence what is seen as valuable or important. This authorising environment can act to approve or support new initiatives. An authorising environment can be set by formal policies, funding requirements and budgets. Less formal structures of authority include the support and interest of specific influential individuals within an organisation.
Who is an evaluation champion?
Evaluation champions (also sometimes referred to as 'advocates' or 'influencers') are individuals who promote and advocate for evaluation in their organisation. Champions can be at all levels of an organisation and can include practitioners, evaluation officers, team leaders, program managers and executive staff.
The information presented in this resource draws from a recent research project undertaken at the Australian Institute of Family Studies (AIFS). That project investigated the ways in which managers and other types of evaluation champions can encourage the use of evaluation findings (both at an executive and service level), and the successful approaches and processes that build a positive evaluation culture.
Why managers and the organisational environment are important
Organisations that successfully use evaluation create a culture where evidence and evaluation are seen as important and staff at all levels are comfortable using evidence to improve their practice.
The reason we come to work today is to make people's lives better. We want to be confident we're doing that. We want to be confident we're doing that to the best we know today, and that if we're using evidence, we can keep improving not only what we're doing today but tomorrow. So I think having really strong communication about why we're doing evaluation is really, really important. (Program Manager)
Building this kind of organisational culture usually can't be achieved without some behaviour change and professional development. This often means senior leadership needs to set the tone and help to create the right environment. Less senior evaluation champions, such as evaluation officers and frontline practitioners, can also help to create a positive evaluation culture but these staff are most effective when they have managerial support.
Many of the managers and evaluation champions we spoke to referred to the importance of the 'authorising environment' in building an organisational culture where research evidence and evaluation were seen as valuable and a priority for the organisation. Without the right authorising environment, and particularly without the support of key decision makers in an organisation, it can be hard to achieve change. However, the right authorising environment can help build a positive evaluation culture by making available the necessary resources (e.g. time, funding and evaluation infrastructure) and encouraging staff to see their role in creating and sharing knowledge.
As an agency we're supporting staff to implement the things they learn about evaluation. So it's about us having structures that encourage, and also making sure that staff see the outcomes of it and not just that it goes off into the distance and people think it's been great, but there's no way of genuinely knowing whether it's a better or worse way of doing things. (Executive Manager)
In-house evaluation support
Organisations with research and evaluation teams often have an advantage because they are able to carry out evaluation and research activities themselves. Having in-house expertise can greatly assist in building wider staff capacity to understand and use evidence and evaluation data. However, organisations without evaluation teams or that are not equipped to carry out internal evaluations can still have an authorising environment in which evaluation and evidence are valued and seen as important for improving practice.
What works in leading evaluation in your organisation
We talked to managers in the families and children services sector to find out more about what they have found most helpful or effective in building positive evaluation culture within their organisations. They helped us to understand 'what works' when leading evaluation in an organisation; that is, how to encourage the use of evidence and evaluation data, and how managers can support staff undertaking evaluation activities.
- high level endorsement
- a clear strategy for evaluation
- investment in building workforce capacity.
High level endorsement
Senior managers' high level endorsement of evaluation can help create an environment within an organisation where staff feel encouraged and capable of undertaking evaluation activities and acting on the data. This sets the tone for how staff perceive evaluation - both in terms of their attitude towards conducting evaluation work and their understanding of how evaluation can be used. It can also assist in embedding evaluation in your organisational processes.
So we're really lucky at our organisation to have a CEO who is very data positive and who demands high levels of evaluation activity, throughout the tender process to reporting, to everyday management of the program. So that's coming from the top, which is really helpful and enabling when you're trying to do activities at that kind of more team or individual level. (Evaluation Officer)
Effectively endorse evaluation by demonstrating the value, purpose and usefulness of evaluation to the organisation as a whole, and not just as a way of meeting funding requirements.
For example, by communicating the usefulness of evaluation, you can demonstrate to staff how evaluation findings contribute to:
- the development of program resources
- enhanced understandings of client needs and experiences
- advocacy and providing clients with a 'voice'
- building the knowledge base.
Foster a culture of continual learning and improvement that empowers staff to undertake routine evaluation activities themselves and to see the value of this to their work and personal development.
It is important to identify your staff's strengths and provide positive reinforcement. Finding simple but effective ways to show your appreciation to staff can build a positive culture by helping them see the results of their work and how it matters. One example from our research came from an organisation where the evaluation team put the results of an annual survey of a particular program into a small book for staff as a means of recognising and valuing their work.
Assess your organisation: levels of endorsement
Ask questions of yourself and your workplace to reflect on the levels of endorsement you provide to staff, such as:
- Do I demonstrate and communicate an understanding of the value, need and purpose of evaluation outside of its use in providing accountability to funding bodies?
- Are those actions and communication strategies perceived as consistent by organisational staff?
- Do I recognise and reward other staff members who undertake and/or use evaluation in their practice?
- Do I encourage and reward the sharing of evaluation skills and research findings with different teams across the organisation?
- Do I regularly engage with evaluation champions to discuss the organisational capacity to undertake and use evaluation?
A clear strategy
Having a clear strategy for how your organisation will use evidence and carry out evaluations can:
- contribute to more effective and streamlined evaluations
- promote more evidence-based decision making in staff
- ensure that any evaluation and research is aligned with organisational values.
A clear strategic approach may include developing a whole-of-organisation strategy that embeds evaluation and research activities in organisational policies, frameworks and processes.
A clear strategic approach also helps set the tone for how staff perceive evaluation and research. It can influence the attitudes and motivation of staff because they can see how the collected data will be used to improve practice (beyond just being an administrative requirement). Embedding evaluation in organisational processes can also assist in evaluation becoming normal practice.
Of course everyone always knows you have to report back to funders but it's like a cultural shift now. It's not just about reporting back to funders, it's about informing us at the organisation, evaluating what we're doing, improving, sharing our knowledge. We want to move to that. (Program Manager)
Embedding evaluation into work systems, processes and policies to make it part of regular practice is an important leadership strategy.
This can include:
- embedding program logics into broader organisational planning
- embedding evaluation into whole-of-organisation frameworks and policies
- streamlining monitoring and information management systems to support evaluation activities.
Provide clear oversight of all scheduled evaluations within your organisation (regardless of funding source) to help with managing organisational resources and maintaining accountability to funding bodies.
A useful strategy is committing to an overarching evaluation and research framework for the entire organisation. You can develop this through internal processes within your organisation and it doesn't necessarily require a lot of resources. It generally includes who has oversight and responsibility for research and evaluation activities, how the research evidence will be used, and a schedule of all evaluation activities within the organisation. This helps to keep everyone on track with individual program evaluations and ensures that the staff involved in an evaluation have the capacity to undertake all of the activities it involves.
By sticking to the whole-of-organisation strategy, you can more effectively champion evaluation and have better control over all evaluations within an organisation. This can help in managing evaluation budgets and ensures that evaluations of different programs or services contribute to organisational goals and learning. With this approach your organisation can also keep track of individual evaluations of programs tied to discrete funding sources, and you can also share the learnings from program-based evaluation across your organisation.
Promote the use of evaluation for continuous monitoring, learning and improvement - not just for evaluating the impact of service.
You can adjust client intake and assessment tools so that they work as pre-service measures or adjust clients' final sessions to include post-measures. For example, develop short questionnaires for clients to fill out before and after receiving the program or service. You could include aligning intake and exit tools with the Data Exchange System (DEX) categories in order to streamline evaluation processes for Commonwealth Government funded activities.
One of the things that I think that everyone is missing and what inspires staff culture is that services need to see themselves as evidence builders. So our staff are really kind of inspired, they see data, they use data, they are part of making data and evidence. And so they see how that evidence and every level of that evidence is used, say in policy submissions, to advocate around the need for specific client groups. (Chief Executive Officer)
Increase buy-in and understanding of evaluation among staff by involving them in co-planning processes.
For example, have staff contribute to the design of program logic models and data collection processes, and include them in discussions on setting strategic goals and whole-of-organisation strategies.
Assess your organisation: strategic approach to evaluation
Ask questions of yourself and your workplace to develop and/or support a clear and strategic approach in your organisation, such as:
- How would my organisation benefit from having an overall theory of change or program logic that clearly sets out the need and purpose of evaluation?
- How would my organisation benefit from having a whole-of-organisation strategy as a framework for all research and evaluation activities?
- What kind of adjustments can be made to program processes in order to align existing data collection and monitoring tools with evaluation measures?
- Can I integrate data collection into normal practice and increase the use of evidence for continuous learning?
- Can individual evaluations for specific programs and services be aligned with the whole-of-organisation strategy?
Investment in building workforce capacity
Committing to an investment in building workforce capacity is one of the most important ways an organisation can create a positive evaluation culture and make evaluation genuinely useful to an organisation. One way to do this is by providing staff with access to technical support and expertise as well as giving staff opportunities for hands-on learning. Another is to make sure your organisation internally shares information, resources and research in order to develop evaluation capacity. Building workforce capacity may also involve investing in appropriate monitoring and information management systems.
I think you have to kind of understand different ways that people can engage with data. So not everyone's going to have like a thrill of joy when they hear the words 'statistically significant'. Like it doesn't mean anything. (Evaluation Officer)
Recognise the different levels of skill and evaluation capacity within an organisation and adjust expectations to appropriately support the learning needs of staff.
Use different strategies for staff at different levels of the organisation. Organisations with internal research and evaluation teams can run training at a basic level for staff with little or no previous experiences of evaluation, and more advanced training for staff with more experience. A good launchpad for staff with no prior knowledge of evaluation includes training in the 'why' and 'how' of developing program documentation (e.g. program logic models) and monitoring activities (e.g. routine collection of measures). Introducing staff to evaluation in empowering ways will help to demystify evaluation and research activities and make the purpose of those activities clear.
[Evaluation] is not just a one-step, two-step process - we can't remove the practitioners from it and we can't remove the participants from it. It's all about them at the end of the day and the families that we work with. It's about managers taking the staff on the learning journey and explaining why the research and their contribution is important and how this information is valuable. (Initiatives Manager)
Staff development may require both internal and external supports.
Internal actions to build workforce capacity can include ensuring that staff have access to appropriate resources such as time, funding, training or technical support as well as encouragement to undertake continual learning. External supports can also be important and include the types of training and learning opportunities that might not be available within an organisation. This may include tailored and general support from external consultants or other services such as the AIFS team.
As a way of developing support for staff and investing in their evaluation capacity, managers can engage in relationship building with external stakeholders (funding bodies and others). This includes building partnerships with universities and evaluation networks and employing contractors when required.
Assess your organisation: investing in your workforce
Senior level managers can work out a strategic plan for how to best support staff to develop their evaluation capacity. Think about organisational capacity and where best to invest resources to develop your workforce. Some questions to reflect on, that can help you get started, include:
- What would be the benefits of creating a dedicated research and evaluation team within your organisation?
- Has there ever been an evaluation 'inventory' to assess the skills and needs of staff at your organisation? What would this look like and what could some benefits be?
- What kind of external support does your organisation draw on to assist in evaluation and research activities? Are there opportunities to share and learn with other organisations or existing evaluation communities in your local area?
- Who are the evaluation champions (at all levels) within your own organisation?
- Does your organisation consider recommendations from all levels of staff about the type of resources needed for the organisation to develop this capacity?
Resources and further reading
Hodge, L. M., & Turner, K. M. (2016). Sustained implementation of evidence-based programs in disadvantaged communities: A conceptual framework of supporting factors. American Journal of Community Psychology, 58, 192-210.
Jones, A. D. (2018). Understanding executive leaders' thinking about evaluative inquiry: An exploratory study. (Masters Research Thesis). Melbourne: Melbourne School of Graduate Education, University of Melbourne.
Lopez, A. A. (2018). Leadership in evaluation capacity building: Influencing organizational change and learning about evaluation. (Doctoral Dissertation Thesis). Claremont, CA: The Claremont Graduate University.
Norton, S., Milat, A., Edwards, B., & Giffin. M. (2016). Narrative review of strategies by organizations for building evaluation capacity. Evaluation and Program Planning, 58, 1-19.
Preskill, H., & Boyle, S. (2008). Insights into evaluation capacity building: Motivations, strategies, outcomes, and lessons learned. The Canadian Journal of Program Evaluation, 23, 147.
Stewart, J. (2014). Developing a culture of evaluation and research (CFCA Paper No. 28). Melbourne: Child Family Community Australia (CFCA).
Weiner, B. J. (2009). A theory of organizational readiness for change. Implementation Science, 4, 67.
This resource was authored by Alissar El-Murr, Senior Research Officer at the Australian Institute of Family Studies.
This document has been produced as part of AIFS Evidence and Evaluation Support funded by the Australian Government through the Department of Social Services.
Featured image: © GettyImages/SolStock