How to review the evidence
This short resource provides guidance and links to additional information to step you through a basic literature review. It is particularly for people working in the community services sector who want to use a literature review to inform the design, delivery or evaluation of a program, service or approach to practice. The resource will also help those unsure of how to go about a literature review. Specifically, this resource outlines the process for conducting a narrative-style literature review.
Reasons for conducting a literature review
A good quality literature review is the starting point for many exciting initiatives: a new program, project or service, improvements to practice or policy, a program logic or evaluation. It can be really rewarding to take time out from practice to review and think about evidence and theory in the area you work in or are passionate about progressing in. The knowledge you gain can enhance your practice.
A literature review forms the foundation of an evidence-informed approach. An evidence-informed approach to practice brings together research evidence, practitioner knowledge and experience, and client values and experiences. A good quality literature review contributes to evidence-informed programs or services at different stages in programs' development and evaluation. For example, a literature review can help to:
- understand an area of practice: A literature review can help you understand a topic or update your understanding of your area of practice. For example, a literature review could identify risk and protective factors contributing to a particular social issue, explore the underlying causes or consequences of an issue or describe how it is experienced by individuals and communities.
- identify effective interventions and strategies: It may be that there are many published studies describing evaluations of relevant interventions, approaches or strategies. Doing a literature review before you design a program or service can save you from reinventing the wheel and ensure your program design is aligned with good practice. It can also alert you to new developments or interventions in your field.
- support evaluation: A literature review can help you identify the most relevant and likely outcomes for a program or service, and provide evidence to support a program logic model.
The different types of literature reviews
There are many different types of literature reviews, and they can be used for different purposes. This article by Grant and Booth (and listed in 'Further reading') discusses different types of literature reviews and has a useful summary table outlining the differences between each type. This current resource describes the process for conducting a 'narrative literature review'. A narrative review summarises and critically reviews a body of literature about a topic and can be a good choice if you want to build your knowledge about an area of practice. It has been selected as the approach for this resource because it is relatively simple and can be used in a range of different contexts.
Other types of literature review that may also be used in the Child and Family sector include:
- scoping review: This aims to get a broad picture of 'what is out there' on a particular topic by mapping the size and nature of existing literature.
- systematic review: This is a rigorous and comprehensive type of literature review that includes a detailed methodology and clear criteria for including or excluding articles and publications. The idea is that a systematic review could be reproduced by another author. They are often very time-consuming to complete.
- rapid evidence assessment: Sometimes called a rapid review, this type of review provides an overview and critical assessment of the literature on a particular topic using a structured and rigorous search strategy similar to that of a systematic review.
- meta-analysis: A form of systematic review that uses statistical techniques to combine the results of different quantitative studies on a particular topic and then provide an overall estimate of results or effectiveness.
Think you need some help? If you are a Families and Children Activity service provider, the Expert Panel Project team at AIFS can provide some free advice. Or, if your organisation has a research and evaluation designated staff member or team, they should be able to offer some support.
Key steps in conducting a narrative literature review
Step 1: Define the research question
Before you start a literature review, you need to define exactly what you want to know. This is often written as a research question. A good research question will guide your literature search, focus your review and help ensure your review is manageable and useful. It is worth investing the time upfront to get your research question right.
The answer to your research question - the information you get from your review - should be directly translatable to the design, delivery or evaluation of your program or service. It should be relatively simple but not so simple that it can be answered with a 'yes' or 'no'. You should avoid jargon, and be careful not to have too many big or vague concepts in your research question that you will then need to define. For example, terms such as 'complex needs' or concepts such as 'positive parenting practices' will likely need further definition. It is okay to include these where they are necessary but you should try to avoid having too many of these types of complex concepts in your research question.
You also need to find the balance between a question that is too narrow and one that is too broad. If your question is too broad you are likely to get lost in the literature and produce a superficial overview of the topic without providing enough depth of information; that is, you won't get useful, targeted information that you can use to design or deliver a program or service. On the other hand, if your question is too narrow, you might not be able to find enough literature to answer the question, or your review might not have sufficient detail to inform your program or service.
Here are some example research questions for child and family services:
- What works to improve parenting practices for families experiencing relationship breakdown?
- What are the development and wellbeing outcomes for children aged 8-12 who regularly attend an after-school arts-based program?
- How can family dispute resolution practitioners ensure that children's preferences are heard and included in parenting agreements?
Step 2: Decide what to include in the search
Once you have refined your research question, you can work out the scope of the search and what search terms to use. The search terms will cover the main concepts in your research question and could include the target group, issues, interventions or approaches and outcomes. For example, for question (1) above, you might want to look for literature on the intervention (parenting interventions) and the target group (families experiencing relationship breakdown).
You would then come up with a list of search terms. A good research question will make it easier to identify your search terms. One way to do this is to first identify the key concepts within your question, and then consider alternative terms to use for your search strategy. This library guide from Monash University outlines a process to identify search terms, and Table 1 outlines some possible search terms for question one above.
Scoping the search
It is also useful to think about what limits (restrictions) you might want to place on your search. These are sometimes called inclusion and exclusion criteria. Some limits might be:
- Publication date: For literature reviews looking at interventions, you might only want studies from the last five years - practice can get outdated quickly. If you are interested in understanding risk and protective factors or outcomes you could look at a longer time period.
- Location: You might only want studies conducted in Australia. However, this could restrict your search too much. Researchers in Australia often include literature from other English-speaking countries with similar economic, political and cultural profiles; for example, the United States, the United Kingdom, Ireland and New Zealand. If your focus is on Indigenous people, you might, for example, want to look at literature from Canada and New Zealand as well as Australia.
- Type of resource: You might only want literature published from peer-reviewed journals, or you might be interested in publications from other sources - books, evaluation reports from community organisations or government reports. Either way, you should specify what type of literature you will include in the search.
Step 3: Decide where to search
In the first instance, if you do not have access to a research or university library and its database, you could begin looking for information on a public website, such as the CFCA Information Exchange, which houses summaries of evidence written specifically for practitioners and policy makers. CFCA has a large back catalogue on a broad range of topics related to families and children. Other websites that produce evidence summaries and reviews for the Child and Family sector are:
- Campbell Collaboration
- Child Welfare Information Gateway
- Cochrane Library
- Emerging Minds
- National Society for the Prevention of Cruelty to Children (NSPCC) library collection
- Parenting Research Centre.
Searching academic databases will allow you to systematically search for everything that has been published on a topic, but these searches often require more advanced search techniques. You can use Google Scholar to get an idea of what is out there; but although there are increasing numbers of open access publications, most databases and journals will require a subscription for you to be able to access them. To access journals, some university libraries provide limited free access to their databases and the National Library of Australia also provides access to some academic journals. Practitioners who are also studying at university will have access to university databases, and university alumni often have some form of library access. Colleagues who have access to academic databases can also be a good resource.
You will need to refine your search strategy based on how many results you get. If you don't find much, you will probably need to go back and think of additional search terms (look at what terms are being used in the articles you already have) or broaden your search. Alternatively, if you have too many results you might need to try a more refined search. Using the search terms in Table 1 in an academic database returned more than 6,000 results. Limiting the search to articles that were: published in English, peer-reviewed, published in the last 15 years and located in Australia returned just under 100 results. Reviewing the titles of these 100 articles would give you a good indication of if you are on the right track. If the results aren't what you were looking for, you would need to reconsider your search terms. Reading some of the relevant literature can help alert you to other relevant search terms. You may need to adjust your search strategy as you go.
Step 4: Assess the evidence
You need to critically read and assess what you have gathered from your search. Step 5 covers reading and note-taking, but before you begin your detailed reading you should assess the quality and relevance of each source, as well as the completeness of your collected resources. For sources where primary research has been conducted, you should consider what type of study it was, how many people participated in the research, when and where it was conducted, and other possible limitations. The CRAAP test (currency, relevance, authority, accuracy, purpose) can be applied to evaluate information, and if you are unfamiliar with critical reading, you can use this checklist as a guide.
Finding an already completed literature review can save you time and effort, but you might need to do some additional work so that it answers your research question - perhaps by looking at more recent publications or looking at another topic area to complement it. For example, if you were doing a literature review to look at any evidence about playgroups for parents who were experiencing mental health problems, you might begin with this paper from CFCA that reviews the evidence for supported playgroups. However, you would probably also need to look at literature about effective parenting and social support interventions for parents with mental health problems.
In subject areas where there are different professional approaches (e.g. counselling) you may need to describe some of these different theories or practices and their different strengths and weaknesses or their different applications. Reviewing and describing different approaches and theories gives your review credibility, because it demonstrates that you have critically engaged with the literature in your topic area rather than reviewing only the literature that aligns with your chosen approach. This will be particularly important if you are using the literature review as a basis for a new program or to review or update an existing program. It can also be important if you are developing a program logic and need to ensure it is grounded in evidence.
Step 5: Reading, note-taking and referencing
Once you have gathered and assessed the quality of your literature, your next step is to read and review your articles and resources. When reading, it is useful to take notes as you go. There are a few options for how to do this, and it is very much down to personal preference. Many people use a spreadsheet to keep track of sources and provide an overview of what they have read. Some people also take more detailed notes in a Word document that they can use later to provide the basis of their literature review.
This guide provides some examples of spreadsheets but you should adapt the columns to suit your purpose - you may like to rate each article based on its relevance to your review, or add a 'notes' column where you can include a sentence or two reflecting on each article. For taking more detailed notes in a Word document, you could note down anything particularly interesting, relevant quotes and your reflections on what you are reading. This can make it much easier when you get to the writing stage, because your notes can be reordered into a very rough draft of the literature review.
Referencing your sources shows that you have engaged with the literature, and that you are recognising and acknowledging the work of others. If you need to brush up on your referencing skills, most university libraries or academic skills units have guides about how to do it. This resource will not cover referencing in detail, except to say that you should reference your sources as you go. It is almost impossible to go back once you have finished writing and add in references - not only is this more time consuming than doing it as you go, but it is really difficult to remember where things came from so you might end up plagiarising someone else's work.
Programs such as EndNote can be invaluable if you have lots of references to organise and are well worth the time to learn if you have multiple reviews to do. You also need to choose a referencing style. At AIFS we use the APA style (see our cheat sheet here), which is widely used in the social sciences.
Step 6: Summarise the literature and write up your insights
Rather than presenting your review study by study, you should synthesise and organise your findings by grouping them into themes or topic areas. You can usually identify the main themes by reviewing your notes after you have read all the articles and studies that come up in your search. It is a good idea to draft an outline for your review at this stage. An outline should identify the main themes and consider how the review should be sequenced and what the findings might be. This resource from James Cook University outlines a simple structure and has prompting questions that you can use to develop your outline.
Then, get writing. You should write to the outline you have drafted but it is also quite likely that you will need to make changes as you go. Just be careful not to get too far off track. You should have a discussion section that includes the gaps and limitations from the literature and unanswered questions that you still have that would be important to better understanding the topic. Your conclusion should summarise your main points and outline the relevance of the literature to your work in your organisation with your target group. This paper published by CFCA provides one possible example of how a simple literature review can be structured.
- Aveyard, Helen. (2014). Doing a literature review in health and social care: A practical guide. Berkshire, England: McGraw-Hill
- Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91-108.
- Most universities have guides on literature reviews and literature searching. This is one of the more comprehensive ones: libguides.jcu.edu.au/litreview
This resource is for practitioners and policy makers who want to learn more about the needs assessment process or how to conduct a needs assessment
Provides practical information on the structures, practices and actions that support a change toward a strong culture of evaluation and research
A snapshot of participatory action research, an approach that views the participant as an equal partner with the researcher
This resource has been designed for Communities for Children service providers, but may be used by anyone who is interested in developing a program
Authors and acknowledgements
This resource was written by Jessica Smart, Senior Research Officer at the Australian Institute of Family Studies.
The author would like to acknowledge reviewers for their valuable contributions to this resource.
Featured image © GettyImages/fgtrade