Collecting data from parents and children for the purpose of evaluation
Issues for child and family services in disadvantaged communities
Myfanwy McDonald, Kate Rosier
This Practice Sheet outlines the challenges child and family services may face when they seek to collect data directly from parents and children for the purposes of evaluation. It contains some examples of how child and family services can engage children and families in the evaluation process in order to overcome some of those challenges.
- Many child and family services will be required to collect data from children and/or parents for the purposes of evaluation.
- Collecting data from children and parents for the purposes of evaluation can be challenging for service providers. For example, parents may be wary of sharing information with service providers. Collecting data from parents who speak languages other than English can be challenging if bilingual staff and/or interpreters are not available. Furthermore, collecting data directly from young children (i.e., 6 years and under) can be difficult because traditional methods of data collection (e.g., surveys and interviews) are usually not appropriate or effective.
- Despite the challenges associated with collecting data from parents and children, the process provides parents and children with an opportunity to be heard. Being heard and listened to as respected citizens in their own right enhances children's feelings of importance in their community.
- Four key methods that child and family services can use to assist them in collecting data from children and parents for the purposes of evaluation are:
- become familiar with culturally competent evaluation;
- be aware of issues relating to consent, privacy and confidentiality when collecting data from children;
- become familiar with techniques for collecting data from children; and
- involve children and parents in the evaluation process.
Why is collecting data from parents and children for the purpose of evaluation important for child and family services?
Child and family services are well aware of the importance of evaluation - the "systematic process of determining whether (and how) [a] program achieves its objectives" (Parker, 2010a, p. 3). In addition to demonstrating the effectiveness of a program, evaluation findings can be used for a range of purposes, including supporting applications for new or continued funding and contributing to the broader evidence-base about what does and does not work for clients (Parker, 2010f).
There is a range of resources available for Australian-based child and family service providers to develop their knowledge of and capacity to undertake evaluation (e.g., Parker, 2010a-f). This Practice Sheet does not seek to replicate these pre-existing resources, rather, it focuses on one particular aspect of evaluation that can prove challenging for service providers: the collection of data directly from parents and children. Although in some cases child and family services will not be involved in the collection of data (e.g., when the evaluation of a program is conducted by an external evaluator), for most, this will be required at some stage.
The benefits of child and family services collecting data directly from parents and children, rather than relying upon external evaluators, are multiple and include the potential for evaluation to become more deeply embedded in organisational culture as well as cost-savings (Spooner, Flaxman, & Murray, 2008). However, collecting data from parents and children can be challenging. Parents may, for example, be wary of sharing information with service providers. Collecting data from parents who speak languages other than English can be difficult if bilingual staff and/or interpreters are not available/appropriate. There are multiple challenges associated with collecting data from children (Christian, Pearce, Roberson, & Rothwell, 2010). For example, collecting data directly from young children (i.e., 6 years and under) that accurately and authentically reflect their experience can be difficult because traditional methods of data collection (e.g., surveys and interviews) are usually not appropriate or effective.
In the context of a busy service delivery environment these challenges can be overwhelming for program staff. Without the tools to address these issues an organisation's ability to demonstrate to funding bodies the positive impact of the work they undertake may be compromised. In an era when program evaluation is often a requirement of funding, this can generate a significant amount of stress and frustration for organisations.
This Practice Sheet outlines the nature of the challenges associated with the collection of data for evaluation. It then provides some examples of how child and family services can engage children and families in the evaluation process in order to overcome some of those challenges.
This Practice Sheet will be useful for any organisation that delivers services to children and families and is actively involved or interested in the process of evaluation. However, it is especially targeted towards those services working within disadvantaged communities. The resource is designed for organisations that are conducting impact or outcome evaluations, but it may also be useful for process evaluations.
What are the main challenges for child and family services in collecting data from parents and children?
For the purposes of evaluation, it is important to collect data from as many participants as possible (Parker, 2010c). However, collecting data from participants, regardless of their socio-economic circumstances, can be challenging. For example, participants may be too busy to contribute or may believe that they have nothing meaningful to add.
When attempting to evaluate programs, some of the challenges of collecting data from parents that services may encounter include:
- Parents may be reluctant to provide information because they have concerns about confidentiality and anonymity.1 For example, the evaluation of a suite of programs for children and families in a disadvantaged community in South Australia reported that:
Families accessing the [program] for the first time are reluctant to provide personal information. We have had difficulty ensuring that all parents sign the "sign in" book which simply records their attendance ... on a particular day. A number of parents have refused to complete a registration form, which simply records name and address information and name and ages of children. Others have completed it reluctantly and only after reassurance about how the information will be used. (The Salvation Army, 2009, p. 9)
- Parents who have poor English language and/or literacy skills may have difficulty participating in surveys, interviews and other data collection processes (Larkey & Staten, 2007; Parker, 2007) and parents who have low levels of literacy may have difficulty responding to questionnaires and surveys (Donnelly, 2010);2 and
- Parents may be having difficulties managing everyday stress that makes it difficult for them to prioritise a task such as completing a survey or taking part in an interview or focus group (Sullins, 2003).
Some of the challenges services may encounter when seeking to collect data directly from children (rather than relying upon parent reports) include:
- context - children will respond differently depending upon the environment in which they are interviewed; a "natural" environment (such as a playground) is preferable to a formal interview room (Lewis & Lindsay, 2000);
- method of data gathering - the method that is used needs to be considered in relation to the age of the child and their skills and capabilities (Coyle, Russell, Shields, & Tanaka, 2007);
- ethical issues - care should be taken to factor in vulnerability and potential harm for children from their involvement in data collection processes, recalling incidents of conflict for example may cause distress to a child (Hill, 2005); and
- privacy and confidentiality - assuring children that any data collected will be kept confidential is especially important because of the power imbalance between children and adults, however in some cases there may be instances when the information a child discloses needs to be reported (e.g., reports of child abuse) (Thomas & O'Kane, 1998).
Despite the challenges of collecting data from children and parents, including their perspectives in the evaluation of services can be beneficial to them and to service providers. Being heard is a key building block of social inclusion initiatives therefore by asking parents and children for their perspectives child and family services are enabling them to participate and be connected to their community (McDonald, 2011). For children especially, participating in activities such as evaluation and research helps them gain new skills and build a sense of their own agency (Graham & Fitzgerald, 2010). When children have opportunities to be heard and listened to as respected citizens in their own right (rather than being viewed as "citizens of the future"), their feelings of importance to their community is enhanced (Tranter & Pawson, 2001). Children can also offer a unique perspective that may help service providers develop new perspectives on service provision.
These practice considerations are general principles based upon lessons from research and practice. They are designed to provide additional guidance to service providers about specific issues related to collecting data to demonstrate program outcomes.
What methods can child and family services use to collect data for the purposes of evaluation?
This Practice Sheet outlines four key methods that child and family services can use to assist the process of data collection. They are:
- become familiar with culturally competent evaluation;
- be aware of issues relating to consent, privacy and confidentiality when collecting data from children;
- become familiar with techniques for collecting data from children; and
- involve parents and children in the evaluation process.
Overall, the most important issue to note is the importance of building and maintaining relationships of trust with families. Parents and children are unlikely to share information with people they do not know or trust. Issues of trust may be especially relevant in disadvantaged communities.
Become familiar with culturally competent evaluation
- A common obstacle for researchers seeking to recruit research participants is recruiting people from culturally and linguistically diverse (CALD) backgrounds (Larkey & Staten, 2007).
- Some of the reasons why it can be difficult to engage people from CALD backgrounds in research include: limited English literacy levels, distrust and suspicion of researchers, limited exposure to the benefits of research and cultural reticence for offering negative opinions (Larkey & Staten, 2007; Parker, 2007).
- Cultural competence in evaluation will ensure that the processes of data collection are ethical and respectful of participants' needs and experiences. Cultural competence in evaluation is also necessary to ensure data validity and accurate interpretation of evaluation findings (Botcheva, Shih, & Huffman, 2009).
- Botcheva et al. (2009) identified three principles for culturally competent evaluation:
- Collaboration with stakeholders who are knowledgeable about the cultural characteristics of participants will help to ensure that the evaluation is not dominated by a narrow worldview. Collaboration can inform evaluation goals, methods and measures to ensure they are relevant to the cultural context of participants (Botcheva et al., 2009).
- Reflective adaptation is the process of acknowledging one's own biases, actively listening to other's worldviews and integrating these views and interests into the evaluation design and implementation process (see text box below for an example of reflective adaption) (Botcheva et al., 2009).
- Contextual analyses requires the evaluator to situate cultural views and interests within the surrounding societal context. In other words, participants in an evaluation will not be shaped by one single "culture", rather they will be shaped by a range of characteristics and circumstances (e.g., gender, age, class). The impact of a program on a participant, and the way the participant experiences a program, will be informed not only by their cultural background but also by these other intersecting characteristics and circumstances (Botcheva et al, 2009).
- When accessing services, confidentiality may be an issue for families from CALD backgrounds (Sawrikar & Katz, 2008) and this may also be a concern for them when they are asked to participate in evaluations. In the same way that reassurance regarding confidentiality is critical for families from a CALD background at the time they are accessing services (Sawrikar & Katz, 2008) so too should it be critical when they are approached to contribute to evaluations.
- One of the ways of overcoming language barriers in evaluation is to employ bilingual members of the community to collect data from participants. When employing bilingual members of the community to collect data it is important to ensure that they understand and follow the rules of systematic data collection (Parker, 2010c). For example, if survey questions are not asked in the same way every time the data will not be reliable (Parker, 2010d).
For more information about undertaking evaluations within Indigenous contexts see Parker (2010d) and Spooner et al. (2008). For more information about undertaking evaluations within CALD contexts see Parker (2010d) and Parker (2007).
Questions for consideration
- Is it viable to include respected community leaders from CALD communities in evaluation processes in order to build the trust and understanding of participants from CALD backgrounds in the evaluation process?
- Have members of each of the relevant communities been consulted in relation to the tools being used to collect data? Are questions in surveys and interviews suited to the perspectives of participants?
- Are the participants' potential concerns regarding confidentiality being taken into account?
Case study: Designing a culturally appropriate survey
Botcheva et al. (2009) undertook an evaluation of a program that sought to reduce the spread of HIV/AIDS in Zimbabwe. The program trained adult soccer players to use an HIV/AIDS life skills curriculum to educate at-risk pre-teen children. The service providers used a survey to evaluate the outcomes of the program (e.g., increased understanding of HIV/AIDS, increased availability of services) but noted that the survey had yielded contradictory results.
Upon consulting further with service providers, the evaluation team identified the following characteristics and values of Zimbabwean culture and the target group (i.e., pre-teen children):
- a narrative culture;
- a collectivist (rather than an individualist) perspective;
- moral choice issues (rather than individualist choice that is typical of Western cultures); and
- the importance of religion.
Based upon these factors, the pre- survey was adapted in the following ways:
- The survey questions were rewritten as simple scenarios followed by true/false answer options, as students preferred this format to knowledge-oriented multiple choice questions.
- The survey questions were rewritten to align with the values of moral judgement (i.e., right vs wrong) as the concept of choice (e.g., making a "healthy choice") does not align with the reality of Zimbabwean culture, where few choices are available to the majority of the population.
- As students came from a narrative culture vignettes featuring imaginary characters were used to evaluate students' attitudes.*
* The vignettes were piloted with students after they were written. Based upon the responses from students to the vignettes, a number of answer options were developed to make the data easier to code.
Be aware of issues relating to consent, privacy and confidentiality when collecting data from children
- Children and/or their carers must be able to provide informed consent, and depending on the context of the child's engagement with the service, care should be taken to factor in vulnerability and potential harm from data collection (Hill, 2005). For example, if the child is engaged with the service as a result of some family dysfunction, being asked to specifically recall incidents of conflict or neglect may cause distress to the child.
- As with any ethical data collection process, children have the right to refuse participation in evaluation projects, even if their parent has provided consent.
- Issues of privacy and confidentiality are especially significant when collecting data from children because of the power imbalance between children and adults (Thomas & O'Kane, 1998). Although it is important to assure children that the information they provide is confidential, there may be some instances where the information a child discloses needs to be reported (e.g., disclosure of child abuse) (Thomas & O'Kane, 1998).3 While in most circumstances, this type of disclosure is unlikely, it is important for service providers who plan on collecting data from children to prepare for this possibility.
- Although children's vulnerability and potential for harm from participating in evaluation and research is a critical issue, this needs to be balanced with the understanding that they have a right to have a say and be heard (Graham & Fitzgerald, 2010).
- When children are accessing services without their parents' knowledge (e.g., a helpline or counselling service) this can pose specific problems regarding consent, that is, asking for parental consent can breach the child's right to confidentiality.
Questions for consideration
- Has the child's parent/guardian given their consent?
- Has the child given their consent? Is the child aware that they are allowed to refuse participation regardless of whether their parent/carer has provided consent?
- Is there a plan in place to deal with instances where children disclose information that needs to be reported?
Become familiar with techniques for collecting data from children
- It can be difficult to collect data directly from children that accurately and authentically reflect their needs and experiences. This is especially the case with young children (i.e., aged 6 years and under) (Pascal & Bertram, 2009).
- Research suggests that most children aged 6 years and older have the skills to participate in interviews and focus groups (Coyle et al., 2007). However, when utilising these methods it is important to note that:
- children may not be comfortable being interviewed by someone they don't know (Coyne, 1998);
- young children (i.e., those aged under 10) may need frequent breaks during interviews and focus groups as their attention span is shorter than older children (Coyne, 1998); and
- children who are shy may not feel comfortable participating in a focus group (Hill, 2006).
- Research suggests that most children aged 9 years and older are capable of completing a written survey (Riley et al., 2001). However, Coyle et al (2007) have identified a number of factors that should also be taken into account including:
- children may see written surveys as an onerous task, similar to schoolwork. Using characters or pictures to break up written surveys can make them more appealing to children; and
- the quality of the data collected will depend upon the child's age - the older the child the more skilled they will be with language and the more able they will be to maintain interest in the task.
- Generally, when collecting data directly from children, interviews and focus groups are viewed as more appropriate and effective than surveys. Focus groups for children are likely to be more effective if they incorporate a range of enjoyable and challenging activities, along with time for relaxation, food and drinks (Thomas & O'Kane, 1998).
For more information on the advantages and drawbacks of utilising surveys, interviews and focus groups with children aged 9-13 years of age see Coyle et al. (2007).
- Although traditional methods of data collection (e.g., interviews, focus groups and surveys) may be appropriate for children of school age, other methods, such as observation, will be more appropriate for young children (i.e., under the age of 6 years).
- Children often display high levels of social desirability and will "aim to please" with their responses. They will also often find questions to be leading even though this may not be obvious to the adult interviewer (Lewis & Lindsay, 2000). Children may also think their answers to questions asked during evaluations will be "correct" or "incorrect" as a result of their experiences at school. Therefore, it may be necessary to reassure them that there are no "right" or "wrong" answers (Solberg in Thomas & O'Kane).
- For children of any age, non-traditional methods of data collection may make it easier for evaluators to collect data and may make the process more enjoyable for children. Having a range of tools that children can use to contribute to evaluations (e.g., verbal and non-verbal tools) can make the process more enjoyable for children and enables them to draw upon their own preferences and strengths (Thomas & O'Kane, 1998).
- The format of data collection can be integral to the relevance and robustness of the data. Having a "natural" versus an artificial setting for collecting the data may also impact on the quality of the data. For example, talking with children in an everyday environment including home, classrooms, playgrounds or other surrounds may produce a higher quality of data than a more controlled or contrived setting such as a formal interview room. Findings gained from more natural settings can often be more representative of the general population than those from an artificial setting (Lewis & Lindsay, 2000).
- If asking children to participate in interviews, be aware that when a carer or parent is present, children's responses may be different to when they are alone. In some cases, children will be more willing to share information if their carer/parent is not present (Mauthner, cited in Thomas & O'Kane, 1998). In many cases, the most appropriate approach is to give children the choice to undertake an interview with a carer/parent or alone.
- One alternative method that research demonstrates is appropriate and effective for collecting data from children is Photovoice. This is a qualitative method that involves children taking photos of their everyday lives and environments and then discussing their photos in a focus group (Eichhorn & Nagel, 2009).
- Photovoice has been used by researchers amongst primary school aged children and was found to provide important additional information to quantitative findings (Eichhorn & Nagel, 2009). The Photovice method has been used by Australian child and family services to understand children's perspectives on their neighbourhoods (Broadmeadows UnitingCare, 2010).
Questions for consideration
- Are developmentally appropriate data collection tools being used with children? Are children enjoying the process of data collection? If not, how can the data collection tool be amended to meet the needs of children? What alternative methods could be used (e.g., Photovoice)? Can a suite of tools be used that meet the preferences and strengths of individual children?
- Has consideration been given to the environment in which the data are being collected? How might this setting impact upon the quality of the data?
- Have children been informed that there are no "right" or "wrong" answers to the questions they will be asked?
Involve parents and children in the evaluation process
- Involving parents and children in the evaluation process extends beyond simply collecting data from them to involving them in the design and implementation of an evaluation and in the analysis and dissemination of evaluation findings.
- Involving parents and children in the evaluation process may make the process of collecting data easier for organisations. The reasons for this are multiple and include:
- If parents and children have a better understanding of the purpose of evaluation, they may be more likely to contribute to it via data collection processes (e.g., surveys, focus groups, interviews).
- By becoming involved in the evaluation process, parents and children may feel they have a greater investment in it and thereby be more willing to contribute.
- Those parents and children who are involved in the evaluation process may demystify the process for other parents and children and thereby increase overall participation.
- Involving parents and children in the evaluation process (i.e., design, implementation, analysis, dissemination) draws upon the ideals of collaborative evaluation (also referred to as "participatory evaluation").4 Collaborative evaluation approaches encourage the involvement of stakeholders (e.g., program staff and program users) and are a direct contrast to evaluation approaches that rely solely upon "external experts".5
- In order to ensure scientific rigour it is important for collaborative evaluation approaches to include expert evaluators who are able to provide technical advice. However, expert evaluators need to be open to the principles of collaborative evaluation.
- The involvement of parents and children in evaluation can take many forms and can take place during any phase of the evaluation. Table 1 is adapted from Checkoway and Richards-Schuster (2003) who envisaged a continuum for collaborative evaluation. Although Checkoway and Richards-Schuster's (2003) work referred to the involvement of young people it is equally relevant to the involvement of any participant group, including parents and children. The table also provides some examples of how children and parents can be involved in each step of the evaluation process.
- Parents can become involved in the evaluation in a formal way (e.g., by participating in regular evaluation meetings and committees with staff and/or the local partnership group) or in an informal way (e.g., sharing ideas and opinions as they participate in program activities, during lunch or coffee breaks etc.) (Sullins, 2003). For younger children, informal participation will be most appropriate. For older children, formal participation (e.g., participating in regular evaluation meetings) may be suitable. Formal involvement will be inappropriate in some circumstances, as Sullins (2003) pointed out:
When people have no stable housing, income, family, or physical or mental health, Evaluation Team meetings are rather low on their list of priorities (pp. 389-390).
- As Sullins (2003) suggested, for families experiencing significant challenges in their everyday life, formal involvement may not be feasible. This issue is especially relevant to agencies delivering services to families in disadvantaged communities.
|Parents/children as evaluation subjects||Parents/children as evaluation consultants||Parents/children as evaluation partners||Examples of the "partner" approach|
|Defining the questions||Evaluators define questions||Evaluators define questions||Evaluators define questions with parents'/children's input||Parents and children given the opportunity to define questions that reflect what they think are the most important aspects of a service|
|Collecting data||Evaluators collect data||Evaluators collect data||Parents/children assist evaluators with collection of data||Children take photos of their local community and use these to discuss their everyday lives and environment (Eichhorn & Nagel, 2009)|
|Analysing information||Evaluators analyse data||Evaluators analyse data||Evaluators take the lead in analysis, parents/children assist||Create a space where parents or children can collectively reinterpret the material collected from participants (Thomas & O'Kane, 1998)|
|Disseminating findings||Evaluators disseminate findings mostly to professional audience||Evaluators disseminate findings mostly to professional audiences with or without parental/children's input||Evaluators take lead in dissemination, parents/children assist||Children make an audiotape of comments about the evaluation then select and edit those comments based upon what they agree should be taken from the research (Thomas & O'Kane, 1998)|
|Role of parents||Parents/children are the subject of the evaluation||Parents/children play a limited role as consultants||Parents/children assist evaluators in roles such as data collection and dissemination|
|Role of evaluators||Evaluators take lead in all stages of the process||Evaluators play most of the key roles||Evaluators initiate and implement but enlist parents/children to assist them|
Adapted from Checkoway & Richards-Schuster (2003).
* Evaluators could be an external evaluation consultant and/or the organisation implementing the program.
- One of the risks of collaborative evaluation identified by Spooner et al. (2008) is, "naive assumptions about participation leading to empowerment" (p. 30). It is important to recognise the potential negative aspects for parents and children when they become involved in evaluation processes, such as the time commitment involved. Where parents or children are making significant contributions of time to evaluation tasks, it may be appropriate to pay them for their involvement (Thomas & O'Kane, 1998). For parents, another potential negative aspect of involvement in evaluation processes is the risk of having one's own personal beliefs and values made public (Mathison, 2001). For example, Mathison (2001) described the way in which her personal beliefs about discipline were highlighted when she became involved, as a parent, in the evaluation of her child's after-school program. Parents who become involved in evaluation processes can feel vulnerable if their views and opinions are not respected and conflict within the group is not managed appropriately (Mathison, 2001).
Questions for consideration
- Do parents have the capacity to become involved in the evaluation in a formal way (e.g., a member of the evaluation committee)? If not, how can parents become involved in evaluation processes in an informal way?
- Have parents who are interested in becoming involved in evaluation in a formal way been informed of the potential drawbacks of their involvement (e.g., time commitment)?
- Are the views and opinions of all stakeholders - especially parents - being respected in the evaluation process?
- Botcheva, L., Shih, J., & Huffman, L. C. (2009). Emphasizing cultural competence in evaluation: A process-oriented approach. American Journal of Evaluation, 30, 176-188.
- Broadmeadows UnitingCare. (2010). This is what's important for me! Capturing children's voice through photography: A consultation with children in the City of Hume. Melbourne: Broadmeadows UnitingCare.
- Checkoway, B., & Richards-Schuster, K. (2003). Youth participation in community evaluation research. American Journal of Evaluation, 24, 21-33.
- Christian, B. J., Pearce, P. F., Roberson, A. J., & Rothwell, E. (2010). It's a small, small world: Data collection strategies for research with children and adolescents. Journal of Pediatric Nursing, 25, 202-214.
- Coyle, K. K., Russell, L. A., Shields, J. P., & Tanaka, B. (2007). Summary report: Collecting data from Children Aged 9-13. ETR Associates & The Preteen Alliance.
- Coyne, I. T. (1998). Researching children: Some methodological and ethical considerations. Journal of Clinical Nursing, 7(5), 409-416.
- Donnelly, J. (2010). Maximising participation in international community-level project evaluation: A strength-based approach. Evaluation Journal of Australasia, 10(1), 43-50.
- Eichhorn, C., & Nagel, E. (2009). Photovoice. A participatory instrument for analysing and evaluating settings. Präv Gesundheitsf, 4, 207-216.
- Graham, A., & Fitzgerald, R. (2010). Children's participation in research: Some possibilities and constraints in the current Australian research environment. Journal of Sociology, 46, 133-147.
- Higgins, D., Bromfield, L., Richardson, N., Holzer, P., & Berlyn, C. (2010). Mandatory reporting of child abuse (NCPC Resource Sheet). Melbourne: Australian Institute of Family Studies.
- Hill, M. (2005). Ethical considerations in researching children's experiences. In Greene, S. & Hogan, D. (Eds.), Researching children's experience: Approaches and methods (pp. 61-83). London: Sage Publications.
- Hill, M. (2006). Children's voices on ways of having a voice: Children's and young people's perspectives on methods used in research and consultation. Childhood: A Global Journal of Child Research, 13(1), 69-89.
- Larkey, L. K., & Staten, L. K. (2007). The Lay Advocates' Communication Assessment Tool (LACAT). Journal of Cultural Diversity, 14(3), 135-140.
- Lewis, A., & Lindsay, G. (2000). Researching children's perspectives. Buckingham: Open University Press.
- Mathison, S. (2001). What's it like when the evaluator is a "genuine" stakeholder? American Journal of Evaluation, 22(1), 29-35.
- McDonald, M. (2011). What role can child and family services play in enhancing opportunities for parents and families:Exploring the concepts of social exclusion and social inclusion (CAFCA Practice Sheet). Melbourne: Australian Institute of Family Studies.
- Parker, R. (2007). Helping parents with complex needs: Integration of the Strength to Strength and Resources for Adolescents and Parents programs. Family Relationships Quarterly, 14, 18-20.
- Parker, R. (2010a). Evaluation in family support services. Part A: Evaluation and innovation (AFRC Issues No. 6). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
- Parker, R. (2010b). Evaluation in family support services. Part B: Broader issues. Evidence-based practice, service-based evaluation, and ethics in evaluation (AFRC Issues No. 6). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
- Parker, R. (2010c). Evaluation in family support services. Part C: Basic principles in preparing for evaluation (AFRC Issues No. 6). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
- Parker, R. (2010d). Evaluation in family support services. Part D: Getting started in evaluation. The devil is in the detail (AFRC Issues No. 6). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
- Parker, R. (2010e). Evaluation in family support services. Part E: Evaluation. Dissemination and beyond (AFRC Issues No. 6). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
- Parker, R. (2010f). Evaluating programs (CAFCA Resource Sheet). Melbourne: Communities and Families Clearinghouse Australia, Australian Institute of Family Studies.
- Pascal, C., & Bertram, T. (2009). Listening to young citizens: The struggle to make real a participatory paradigm in research with young children. European Early Childhood Education Research Journal, 17(2), 249-262.
- Riley, A., Rebok, G., Forrest, C., Robertson, J., Green, B., & Starfield, B. (2001). Young children's reports of their health: A cognitive testing study. Paper presented at the Seventh Conference on Health Survey Research Methods.
- The Salvation Army. (2009). Salisbury Communities for Children 2004-2009 report. Adelaide: The Salvation Army.
- Sawrikar, P., & Katz, I. (2008). Enhancing family and relationship service accessibility and delivery to culturally and linguistically diverse families in Australia. (AFRC Issues No. 3). Melbourne: Australian Family Relationships Clearinghouse, Australian Institute of Family Studies.
- Spooner, C., Flaxman, S., & Murray, C. (2008). Participatory research in challenging circumstances. Evaluation Journal of Australasia, 8(2), 28-34.
- Sullins, C. D. (2003). Adapting the Empowerment Evaluation Model: A mental health drop-in center case example. American Journal of Evaluation, 24, 387-398.
- Thomas, N., & O'Kane, C. (1998). The ethics of participatory research with children. Children and Society, 12, 336-348.
- Tranter, P., & Pawson, E. (2001). Children's access to local environments: A case-study of Christchurch, New Zealand. Local Environment, 6(1), 27-48.
- Westcott, H. L., & Littleton, K. S., (2005). Exploring meaning in interviews with children. In Greene, S. & Hogan, D. (Eds.), Researching children's experience: Approaches and methods (pp. 140-157). London: Sage Publications.
1. This is likely to be more of an issue with impact or outcome evaluations, rather than process evaluations. Impact or outcome evaluations that rely upon parent-report data usually require parents to express their opinions, values, attitudes or beliefs which may be more threatening than simply sharing demographic information. However, even collecting process evaluation data can be difficult, as the quote from The Salvation Army (2009) suggests.
2. Anecdotal evidence also suggests that there may be challenges in collecting meaningful data from participants who come from a cultural background where expressing dissatisfaction is considered rude or disrespectful (even in those cases where “constructive criticism” is requested) (Parker, 2009).
3. All states and territories in Australia have some form of mandatory reporting requirements (i.e., the legal requirement to report suspected cases of child abuse and neglect) (Higgins, Bromfield, Richardson, Holzer, & Berlyn, 2010). For more information on mandatory reporting requirements see Mandatory Reporting of Child Abuse <www.aifs.gov.au/nch/pubs/sheets/rs3/rs3.html>
4. In some contexts participatory evaluation is viewed as a sub-category of collaborative evaluation. In other contexts, participatory evaluation is used as a broad “umbrella” term and the equivalent of the term collaborative evaluation (Spooner et al., 2008).
5. Collaborative evaluation approaches have much in common with the strengths-based approach used by professionals who work with families. A strengths-based approach emphasises the ways in which parents are experts on their own children and that service providers can learn from parents (and vice versa) within relationships of mutual trust and respect. This approach also has much in common with what is often termed the “new sociology of childhood” which emphasises that children are “articulate and insightful commentators on their own lives and their perspectives” (Graham & Fitzgerald, 2010, p. 135).
Myfanwy McDonald is the Coordinator of the Communities and Families Clearinghouse Australia. Kate Rosier is a Research Officer in the Communities and Families Clearinghouse Australia.
Thanks to Briony Horsfall and Robyn Parker for providing valuable comments and feedback on this Practice Sheet.