Risk assessment instruments in child protection

 

You are in an archived section of the AIFS website 

 

Content type
Resource sheet
Published

June 2016

The purpose of this Resource Sheet is to outline the different approaches used to assess whether children are at risk of maltreatment, as well as to explore some of the issues and criticisms surrounding the use of standardised risk assessment instruments in child protection.

Risk assessment in child protection occurs during the processes of notification and investigation. It aims to determine whether a report of concern about a child needs further investigation and which service, or level of intervention, is required to respond.

Consensus-based and actuarial risk assessment instruments

Child protection practice, both in Australia and internationally, has seen a marked shift from largely unstructured clinical decision-making to the widespread use of standardised risk assessment instruments. This shift has been accompanied by considerable debate as to the most effective methods of assessing risk; debate that has at times been so intense that some have dubbed it the "risk assessment wars" (White & Walsh, 2006). Disagreement has often centred on the relative advantages or disadvantages of using either consensus-based risk assessment tools (i.e., those that were developed from the child maltreatment literature and/or the opinions of expert practitioners) or actuarial risk assessment tools (i.e., those that were developed by statistically identifying the factors associated with maltreatment). (See Table 1 below for a more detailed description of these instruments. See Case Studies 1 and 2 for detailed examples of specific consensus-based and actuarial assessment instruments.)

 How are items derived?How are decisions reached?
Actuarial assessment toolsItems are derived empirically, using statistical analysis to identify and weigh the factors that predict child maltreatment.Practitioners score each item (e.g., "Do either of the parents have a history of incarceration" N=0, Y=1). The scores of individual items are added, and families are assigned to a risk category according to the their overall score. Some tools grant practitioners a degree of latitude to override an assessment rating (e.g., in the Structured Decision Making tool practitioners can increase the risk category by one level).
Consensus-based assessment toolsItems are derived from the child maltreatment literature and/or the opinions of expert practitioners. These tools attempt to bridge the gap between unstructured clinical and actuarial decision-making.

Consensus based tools utilise one of two decision-making strategies:

  1. Individual items guide practitioners to consider risk factors. However, the final decision as to the overall family risk category is left to the practitioner's (guided) discretion. This is the most common form of consensus-based tool.
  2. As with actuarial tools (see above), the scores of individual items are added, and families are assigned to a risk category according to the their overall score. However, consensus based tools are more likely to grant practitioners greater latitude in overriding assessment ratings.

Source: Austin et al. (2005); Robinson & Moloney (2010); White & Walsh (2006).

Strengths and weaknesses of different types of risk assessment instruments

Both actuarial and consensus-based risk assessment instruments have their strengths and weaknesses, which are summarised in Table 2. As this table illustrates, there are few simple answers in the debate between different types of assessment instrument, and it is certainly not clear (as some authors have claimed) that one type is always more effective than the other. Neither approach can guarantee consistently accurate decisions; there will always be a proportion of "false positives" (i.e., screening and finding grounds for an intervention when an intervention is not actually required) and "false negatives" (i.e., investigating and finding no grounds for intervention when an intervention is in fact needed).

 StrengthsWeaknesses
Actuarial assessment tools
  • Tend to use fewer factors than consensus-based tools - helping practitioners to focus on the most important and influential factors.
  • Provide precise, probabilistic estimates of further maltreatment.
  • Often the empirical analysis is done in the area or state in which the tool will be applied, which helps ensure its accuracy and relevance.
  • Use separate variables to predict the likelihood of different forms of child maltreatment.
  • Show stronger evidence of inter-rater reliability and predictive validity.
  • Place less emphasis on unique, unusual, or context specific factors that may be identified by the more flexible consensus-based tools.
  • Tend not to incorporate or facilitate the practice knowledge of practitioners.
  • May be ineffective if applied in situations that are very different from the one in which the tool was developed (e.g., a tool developed with a sample of the general population used with a minority cultural group or in a remote Indigenous community).
  • May be rejected by some practitioners due to a perceived lack of supporting theory. Conversely, can be vulnerable to perceptions that they will always make an accurate prediction.
Consensus-based assessment tools
  • Allow greater flexibility of assessment than actuarial tools.
  • Often do not impose restrictions on the weighting or combining of different risk factors.
  • Emphasise a comprehensive assessment of risk.
  • Incorporate the clinical judgement and practice knowledge of skilled practitioners.
  • Show some evidence of reliability and validity.
  • Inter-rater reliability and predictive validity has been reported as poor compared to actuarial measures.
  • Can be poorly conceptualised, with loosely defined and ambiguous risk indicators.
  • May be overly subjective and too reliant on professional discretion.
  • Often use the same variables to predict all forms of child maltreatment, even though separate forms of maltreatment can have different indicators.

Source: Austin et al. (2005); Robinson & Moloney (2010); White & Walsh (2006).

In principle, if the goal of assessment is to identify those children whose situation warrants further investigation, then actuarial assessment tools will likely produce a more accurate prediction than consensus-based tools. This is because actuarial tools are usually developed by statistically modelling the factors that increase the risk of re-referral to child protection services. They enable practitioners to make evidence-based judgements about which children are at the highest risk of re-referral to child protection services (and therefore warrant greater scrutiny), and direct practitioners to prioritise their finite resources on the cases of these children. It is worth noting that the statistical modelling for actuarial tools is not based on whether the child was abused or not, or on child death data. However, it is an assumption of these instruments that those children at greatest risk of re-referral to child protection will be the same population as those at greatest risk from abuse and neglect. In brief, while actuarial tools are good for decisions about how to prioritise finite resources, they are not a case planning tool.

If the goal of assessment is to gain a comprehensive understanding of an individual child or family in order to ascertain their service needs, then a consensus-based tool may be most effective as they tend to be more flexible, incorporate more items, and provide more information (Austin et al., 2005). The outcomes of assessments using consensus-based tools are therefore more readily linked to an intervention plan for working with children and their families to reduce risks.

Assessing risk in a "risk society"

Although the use of assessment tools (both consensus-based and actuarial) has become widespread in the child protection practices of a number of countries, the use of these instruments - and indeed the very notion of structuring child protection systems around the prediction of future risk - has been criticised by a number of authors. These authors have argued that risk assessment, as a practice tool in child protection, has been strongly influenced by some very distinct social trends. According to sociologists (e.g., Beck, 1992; Giddens, 1990), individuals and institutions in modern Western societies are increasingly preoccupied with the future and with the systematic prediction of, and protection from, potential risks. This has become such a marked social trend that these theorists refer to contemporary Western societies as "risk societies".

Within the popular discourses of "risk societies", risk to children is considered to be measurable and manageable. The implication of this is the widespread belief that harm to children can always be effectively predicted and prevented - and that if it is not, then someone is to blame (Gillingham, 2006; Gillingham & Bromfield, 2008). One can see this belief in action when child protection services become the subject of negative media attention for making the "wrong" decision or having the "wrong" procedures in place, particularly in the case of child deaths (Connolly & Doolan, 2007). In response to such attention, the process of assessments, as well as general child protection practices, have become increasingly risk-averse. Some contend that the ready acceptance and application of risk assessment instruments has little to do with protecting children more effectively and much to do with organisations protecting themselves from blame when mistakes are made (Gillingham, 2006; Goddard, Saunders, Stanley, & Tucci, 1999).

Criticisms of standardised risk assessment instruments

A number of authors have made more specific criticisms of structured risk assessment tools (both consensus-based and actuarial), arguing that these tools are deficient in that they:

  • are often unable to identify or accommodate the idiosyncratic risk factors of individual cases (Goddard et al., 1999);
  • are ill-suited for Indigenous communities and other minority and/or marginalised groups (Maiter, 2009; Strega, 2009);
  • undermine the importance of clinical experience and intuition (Goddard et al., 1999);
  • neglect social and environmental influences on behaviour and therefore mask social problems and structural inequalities by attributing sole responsibility for problems to parents/caregivers (Strega, 2009);
  • neglect any strengths, resources, and competencies that families may possess (Turnell & Edwards, 1999);
  • tend to exclude important voices and perspectives from the assessment (e.g., the father or male guardian, siblings and, perhaps most importantly, the victim of abuse themselves) (Goddard et al., 1999);
  • focus too much on future harm and not enough on the processes and consequences of cumulative harm (Bromfield, Gillingham, & Higgins, 2007;Goddard et al., 1999); and
  • often present the relationship between risk factors and abusive or neglectful behaviours as a causal one, when in fact it is at best one of association or correlation (Strega, 2009).

Additionally, as Lennings (2002) argued, although many experts call for the use of structured risk assessment tools, "decision making in the care and protection area is yet to develop the research base and statistical activity necessary to develop such approaches" (p. 10).

Risk assessment when working with Aboriginal and Torres Strait Islander families

In responding to child protection reports concerning Aboriginal or Torres Strait Islander children, it is widely recognised that responses and interventions should be culturally appropriate. This is based on the recognition of the particular importance that connection to family, community and culture has for the wellbeing of Aboriginal and Torres Strait Islander children, as reflected in the Aboriginal and Torres Strait Islander Child Placement Principle (for more information, see the Secretariat of National Aboriginal and Islander Child Care website).

In some jurisdictions, protocols have been established to ensure that risk assessments are conducted in ways that are sensitive to the needs of Aboriginal and Torres Strait Islander families and communities. In Victoria, for example, the Department of Health and Human Services and the Victorian Aboriginal Child Care Agency (VACCA) have established protocols that, in part, seek an Indigenous perspective in the risk assessment and case planning process.

Alternative approaches

There have been attempts to incorporate the insights of the above criticisms into the structure of risk assessment programs. For example, Turnell and Edwards' (1999) "Signs of Safety" approach to assessment in child protection evaluates not only risk factors, but also family competencies, strengths, and resources. According to Turnell and Edwards (1999), focusing solely on risk factors is "rather like mapping only the darkest valleys and gloomiest hollows of a particular territory" (p. 49), and therefore their approach attempts to "consider danger and safety simultaneously and to achieve a balanced, comprehensive assessment" (p. 100). Additionally, practitioners using the Signs of Safety assessment framework seek to identify and understand the values, beliefs, and meanings held by all members of the family with which they are working, as well as to determine the willingness and capacity of the family to carry out any suggested plans. In recent years the Signs of Safety approach has enjoyed considerable attention and influence both in Australia and internationally.At the time of writing, no formal evaluation of this approach has been published in Australia. However, a research project is underway to evaluate its effectiveness within Western Australia (Australian Centre for Child Protection). Other research conducted by the National Society for the Prevention of Cruelty to Children in the UK shows positive signs, but only provides limited evidence that it improves outcomes for children (Bunn, 2013).

Other authors have emphasised the need for child protection practice to become more culturally sensitive (e.g., Connolly, Crichton-Hill, & Ward, 2006; Maiter, 2009; Strega, 2009). It was noted earlier (see Table 2) that unless actuarial instruments are developed using a sample of a particular minority cultural group (e.g., Indigenous Australians), it is unlikely that the risk factors of these instruments will be entirely appropriate for that minority group. However, some authors add that even many consensus-based instruments incorporate theoretical models of child development and family functioning that have been developed in white, middle-class contexts, and therefore may implicitly marginalise the perspectives and practices of different cultures (e.g., Connolly et al., 2006; Strega, 2009). Strega (2009) proposed that child protection practitioners working with individuals from diverse cultural backgrounds must constantly reflect upon their own practice, and identify ways in which they may be imposing dominant cultural standards onto minority cultures. In a similar way to the Signs of Safety approach, Strega emphasised that "strengths are as important as problems and challenges, and it is essential to develop a picture of strengths and challenges with clients rather than about them" (p. 154, emphasis added). Strega (2009) also noted that, especially when working with marginalised cultural groups, it is very important that any risk assessment instrument or procedure has scope to account for the wider, structural forces that may be contributing to parental or familial problems. Again, these approaches have not been evaluated.

Conclusion

There is no "magic bullet" when it comes to risk assessment in child protection. The two main types of assessment instrument each have advantages and disadvantages, and may be more or less useful in different contexts or in different stages of the intervention process. Moreover, assessment instruments often need to be augmented by other practices and approaches; for example, when working with diverse cultural groups, explicitly anti-racist and anti-discriminatory procedures, practices, and attitudes may enhance the efficacy of an intervention. White and Walsh (2006) noted that the so-called "risk assessment wars" may be over, and that what has emerged in the literature is the more sophisticated view that there is no one "ultimate tool" that will solve the difficulties of assessment in child protection. Instead, there is an acknowledgment that while some tools may indeed be more effective than others at classifying risk, this does not rule out the need for alternative approaches and for the continued utilisation of clinical judgement and practice knowledge.

Case Study 1: Example of a popular consensus-based assessment instrument

The Strengths and Stressors Tracking Device

Background

The Strength and Stressors Tracking Device (Strengths and Stressors) is a consensus-based assessment instrument that was developed by modifying the North Carolina Family Assessment Scale (NCFAS). According to those who developed Strengths and Stressors, it:

guides new and inexperienced caseworkers to the critical indicators of family well-being, provides the ability to assess a family's strengths as well as their stressors, incorporates an ecological array of conditions and skills into a contextual assessment, and can be completed at multiple points in time, providing a quick assessment of how well a family is increasing its strengths and reducing its stressors. (Berry, Cash, & Mathiesen, 2003).

As the name of the instrument suggests, it is designed to go beyond simply predicting the immediate danger to the child and the likelihood of the child experiencing maltreatment in the future by also assessing family wellbeing and psychosocial development.

Description

Strengths and Stressors has 55 items, which are divided into four domains:

  • environment (17 items);
  • social support (7 items);
  • family/caregiver (14 items); and
  • child wellbeing (17 items).

The child protection practitioner assesses whether each item on the form is affecting the family as a strength or as a stressor (the scale ranges from -2 [serious stressor] to +2 [serious strength]).

The explication by Strengths and Stressors of the different strengths and stressors of a family is designed to assist child protection practitioners in project planning and evaluation. Strengths and Stressors does not, however, offer structured advice as to which interventions should be implemented if specific scores on the instrument are obtained.

Evidence of effectiveness

Only one small-scale study has tested the efficacy of Strengths and Stressors. In this study, which was conducted by the team that developed the instrument, Strengths and Stressors demonstrated high internal consistency, was able to distinguish between different forms of child maltreatment, and appeared to accurately detect changes made by families during the assessment period (Berry et al., 2003). However, this study did not adequately assess the validity of the instrument. As such, more research is needed before Strengths and Stressors can be said to have an adequate evidence base.

Case Study 2: Example of a popular actuarial assessment instrument

Structured Decision Making (SDM)

Background

The Structured Decision Making (SDM) model of child protection comprises a series of actuarial tools developed by the Children Research Center in Wisconsin, USA. As there are distinct issues to be addressed at each stage of the child protection process, different tools or scales are necessary for each decision point. Therefore, SDM is based around eight areas of assessment. In illustrating the use of SDM, this summary will concentrate on those SDM tools directly relevant to risk/strength and needs assessment: the SDM Family Risk Evaluation (Version 3.1), the SDM Child Strengths and Needs Assessment (Version 3.1), and the SDM Family Strengths and Needs Assessment (Version 3.0).

Description

The SDM Family Risk Evaluation comprises two subscales:

  • a neglect scale (12 items: e.g., "Primary Parent has criminal history as adult or juvenile - Y/N"); and
  • an abuse scale (11 items: e.g., "Two or more incidents of domestic violence in the household in the past year - Y/N").

Each item is scored with a 0, 1, 2 or 3 (in the example items above the "No" responses scored a 0 and the "Yes" responses scored a 1, 2, or 3 depending on the severity of the risk factor). Based on the subscale with the highest score, families are classified into a low, moderate, high or very high-risk category. Child protection workers can override the risk classification and increase the risk category by one level.

The SDM Family Strengths and Needs Assessment consists of 12 items (e.g., Alcohol and Drug Use, Mental/Emotional Health, Household Resources and Basic Care). According to the scores of these scales, families are classified into one of four "strengths" categories: exceptional strength, good/adequate functioning, some need, or significant need. This measure also helps to identify specific needs to be included in the family case plan (e.g., a low score on the Household Resources and Basic Care item would indicate that assistance in this area should be included in the case plan).

The SDM Child Strengths and Needs Assessment is very similar in structure to the SDM Family Strengths and Needs Assessment. It consists of 12 items (e.g., Alcohol and Drug Use, Emotional Stability, Behaviour).

Evidence of effectiveness

The SDM Family Risk Evaluation has performed well in tests of predictive validity (i.e., it was shown to be able to quite accurately predict which families should be classified into which risk category). This instrument has also shown moderate levels of inter-rater reliability. No studies were found that looked specifically at the efficacy of the SDM Family Strengths and Needs Assessment or the SDM Child Strengths and Needs Assessment. Finally, compared to demographically matched jurisdictions that used other types of risk assessment instruments, those countries that used SDM instruments had lower substantiation rates, re-referral rates and levels of injuries (For a detailed review of this evidence, see Austin et al., 2005). It is worth noting that almost all of the research into the SDM model and tools has been conducted by the Children's Research Center, the organisation that developed SDM.

References

  • Austin, M., D'andrade, A., Lemon, K., Benton, A., Chow, B., & Reyes, C. (2005) (PDF 263 KB). Risk and safety assessment in child welfare: Instrument comparisons. Berkeley: University of California. Retrieved from <cssr.berkeley.edu/bassc/public/risk_summ.pdf>
  • Australian Centre for Child Protection. (2016). Research and evaluation. Adelaide: University of South Australia. Retrieved from <www.unisa.edu.au/Research/Australian-Centre-for-Child-Protection/​Research/​​#Signs%20of%20Safety%20Program%20of%20​Research>
  • Beck, U. (1992). Risk society: Towards a new modernity. London: Sage.
  • Berry, M., Cash, S., & Mathieson, S. (2003). Validation of the strengths and stressors tracking device with a child welfare population. Child Welfare, 82(3), 293-318.
  • Bromfield, L. M., Gillingham, P., & Higgins, D. J., (2007). Cumulative harm and chronic child maltreatment. Developing Practice: The Child, Youth and Family Work Journal, 19, 34-42.
  • Bunn, A. (2013). Signs of safety in England. London, UK: National Society for the Prevention of Cruelty to Children. Retrieved from <www.nspcc.org.uk/globalassets/documents/​research-reports/signs-safety-england.pdf>
  • Connolly, M., Crichton-Hill, Y., & Ward, T. (2006). Culture and child protection. Reflexive responses. London: Jessica Kingsley Publishers.
  • Connolly, M., & Doolan, M. (2007). Child deaths and statutory services: Issues for child care and protection. Communities, Families and Children Australia, 2(1), 26-38.
  • Giddens, A. (1990). Consequences of modernity. Cambridge: Polity Press.
  • Gillingham, P. (2006). Risk Assessment in child protection: Problem rather than solution? Australian Social Work, 59(1), 86-98.
  • Gillingham, P., & Bromfield, L. M. (2008). Child protection, risk assessment and blame ideology. Children Australia, 33(1), 18-24.
  • Goddard, C. R., Saunders, B. J., Stanley, J. R., & Tucci, J. (1999). Structured risk assessment procedures: Instruments of abuse? Child Abuse Review, 8, 251-263.
  • Lennings, C. (2002). Decision making in care and protection: The expert assessment. Australian e-Journal for the Advancement of Mental Health (AeJAMH), 1(2), 1-13.
  • Maiter, S. (2009). Using an anti-racist framework for assessment and intervention in clinical practice with families from diverse ethno-racial backgrounds. Clinical Social Work Journal, 37, 267-276.
  • Robinson, E., & Moloney, L. (2010). Family violence: Towards a holistic approach to screening and risk assessment in family support services (AFRC Briefing No. 17). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
  • Strega, S. (2009). Anti-oppressive approaches to assessment, risk assessment and file recording. In S. Strega & S. Aski Esquao (Eds.), Walking this path together. Racist and anti-oppressive child welfare practice. Halifax: Fernwood Publishing.
  • Turnell, A., & Edwards, S. (1999). Signs of safety: A solution and safety oriented approach to child protection. New York: Norton & Company.
  • White, A., & Walsh, P. (2006) (PDF 317 KB). Risk assessment in child welfare. An issues paper. Sydney: NSW Department of Community Services. Retrieved from <www.community.nsw.gov.au/​data/assets/​pdf_file/0005/321647/research_​riskassessment.pdf>
Acknowledgements

This paper was updated by Adam Dean, Project Officer with the Child Family Community Australia (CFCA) information exchange at the Australian Institute of Family Studies.

Previous editions have been compiled by Rhys Price-Robertson and Leah Bromfield.

The feature image is by Donnie Ray Jones, CC BY 2.0 

Share