Commonwealth Place-Based Service Delivery Initiatives

Commonwealth Place-Based Service Delivery Initiatives

Key Learnings project

Sez Wilks, Julie Lahausse, Ben Edwards

Research Report – April 2015
Commonwealth Place-Based Service Delivery Initiatives

Read the publication

Executive summary

This report sets out the findings of the Commonwealth Place-Based Service Delivery Initiatives: Key Learnings project. The project was commissioned by the Australian Government's Department of the Prime Minister and Cabinet (PM&C) and undertaken by the Australian Institute of Family Studies (AIFS), between May 2013 and March 2014. It aimed to identify key factors associated with successful outcomes of Commonwealth place-based initiatives (PBIs), in order to inform the future design, implementation and continued delivery of such programs.

The project identified evidence of "common elements" from international and Commonwealth contexts. Within the literature on international place-based programs, original copies of final evaluation reports, or reports that integrated or synthesised multiple stand-alone evaluations of PBIs, were prioritised alongside systematic reviews of PBIs. Information about Commonwealth PBIs was obtained through a formal consultative process with Australian Government departments. The Commonwealth materials included both publicly available and classified program guidelines, program logic models, evaluation frameworks and interim and final evaluation reports. Analysis of both Australian and overseas examples allowed AIFS researchers to: situate the Commonwealth PBIs within a broader international setting; explore how Commonwealth approaches compare or contrast with approaches overseas; and understand the transferability of international findings to the Australian context. One of the key findings was that evaluations of Commonwealth PBIs were lacking sufficient evidence to establish the causal effects of initiatives, their cost-effectiveness and an understanding of how these initiatives work to achieve their goals. This was more pronounced among the evaluation of Commonwealth initiatives than in international evaluations.

Defining place-based initiatives

PBIs are programs designed and delivered with the intention of targeting a specific geographical location(s) and particular population group(s) in order to respond to complex social problems. The most recent initiatives focus simultaneously on place(s) and people. PBIs typically focus on areas and communities with entrenched disadvantage or deprivation. Policy areas targeted by PBIs include: health, education, child development, family wellbeing, community strengthening, housing, urban regeneration, liveability, crime, employment and participation, economic development, immigrant communities, Indigenous communities, social inclusion and social exclusion. These themes were commonly identified in both the international and Commonwealth PBI literature. A number of national and international PBIs were also multifaceted - they addressed a range of types of disadvantage in combination.

Classifying place-based initiatives

A useful typology of place-based initiatives was found in Griggs, Whitworth, Walker, McLennan, & Noble (2008), which classifies initiatives into five types according to their policy objectives and targeting relating to place and person:

  • Type 1: Major focus on place in order to impact place;
  • Type 2: Major focus on place in order to impact person;
  • Type 3: Major focus on person in order to impact place;
  • Type 4: Major focus on person in order to impact person; and
  • Type 5: Simultaneous major focus on place and person in order to impact both.

Applying this classification to international and Commonwealth PBIs enabled identification of the kind of initiatives most relevant for inclusion for review. The review of national and international PBIs concentrated on highlighting key success factors of initiatives that targeted both place and person (i.e., Types 2, 3 and 5).

Identifying common elements of place-based initiatives

Literature from the United States (US), European Union (EU) and United Kingdom (UK) was helpful in suggesting "what works" in the design, delivery and evaluation of place-based initiatives. Several common elements were repeatedly articulated by systematic reviews and syntheses from these regions. From this literature, a list of common elements was devised.

Design and delivery: Common elements

The following principles were identified as being common to PBI design and delivery:

  • flexible delivery - demonstrating a flexible approach to service delivery according to community needs, as well as a flexible approach to expenditure of funding;
  • local autonomy - involving the local community (organisations and individuals) through consultation and active involvement in decisions;
  • joined-up working - integrating or coordinating and developing partnerships between organisations within local areas across the government, private and community sectors; and
  • governance - Commonwealth agencies having good checks and balances in place regarding community discretion in the allocation of government resources, taking care also to engage state governments.

Program implementation: Common elements

There are number of common elements that also need to be considered in terms of implementation:

  • capacity development - sharing training and mentoring opportunities to build the core skills required to deliver services in different ways or to change service delivery methods;
  • lead times - ensuring time is made available to set up programs, build relationships within communities, build capacity within service delivery organisations and ensure evaluations are in place; and
  • long-term focus - recognising that disadvantaged areas have significant problems that take time to remedy and require sustained investment and patience in realising results.

Evaluation: Common elements

Evaluation of PBIs is critically important to enhance policy design, overcome implementation issues, and demonstrate effectiveness and savings to government. We identified the following common elements:

  • causality - establishing that PBIs are working by using international best practice such as matched comparison areas, longitudinal data (survey and/or administrative) and sophisticated statistical analyses to rule out other confounding factors in establishing the effectiveness of PBIs (using randomised trials is difficult);
  • attribution - considering the presence of other initiatives when trying to establish whether a particular PBI "works", as it is possible to have several PBIs operating in the one area;
  • a theory of change - having a well-articulated "program logic" or mechanism by which the PBI effects on the key outcomes of interest can be measured, especially in the short term, to enable policies to be refined, applied to other contexts, "scaled up", and adjusted to address elements of the PBIs that are not working;
  • residential mobility - accounting for population flows into and out of the area in the context of assessing whether a PBI is effective; and
  • cost-effectiveness - routinely analysing the costs associated with the delivery of a program and being clear about the long-term benefits in order to establish its cost-effectiveness.

Key lessons

Many Commonwealth PBIs reviewed as part of this report have features that accord with the internationally identified common elements listed above. They target particular areas and social groups, engage in flexible service delivery and funding models, attempt to provide communities with a say in the services provided (local autonomy), and attempt to join up services.

However, evaluation of these common elements has rarely been done by either international or Commonwealth PBIs. Moreover, the pursuit of the principles of "local autonomy" and "flexibility" makes it very difficult to ascertain why "what works" actually works. Common practice in place-based initiatives means that there needs to be a flexible approach and "locally grounded" design and implementation, but there is a lack of national and international research comparing the effectiveness and benefits of local autonomy and flexibility for a place-based initiative relative to an initiative that did not have local autonomy and flexibility as an underlying principle. Ultimately, there is an absence of empirical studies that set out to rigorously test this policy assumption and whether the "good things" of greater local autonomy and community involvement are realised in practice (Burton, Goodlad, & Croft, 2006; see also Burton et al., 2004).

High-quality evaluations of international PBIs do show that they can be effective and can realise significant cost savings to government. However, among Commonwealth PBIs, the causal effects and cost-effectiveness of programs have rarely been evaluated. And an understanding of the precise mechanisms of "what works" was lacking not only in the Commonwealth PBIs, but also in the international PBIs we reviewed. Significant learnings could be generated to improve policy design and cost savings to the community if there were investment in such evaluations. There is an opportunity to enhance the evidence base, and the evidence generated can enhance the efficiency and effectiveness of service delivery to some of the most disadvantaged communities in Australia.

1. Introduction

This is the final report for a project entitled Commonwealth Place-based Service Delivery Initiatives: Key Learnings, which was commissioned by the Australian Government's Department of the Prime Minister and Cabinet (PM&C) and conducted by the Australian Institute of Family Studies (AIFS). The first step in this project was the completion of a literature review that surveyed international literature relating to the evaluation of place-based service delivery responses to locational disadvantage. One of the key objectives of this international literature review was to identify common elements in the practice and evaluation of place-based initiatives (PBIs), which would then be used as a framework for a systematic meta-review of materials evaluating Commonwealth place-based initiatives in operation between 1 January 2008 and 31 December 2013. The approach to the international literature review is detailed in Appendix A.

Chapter 2 and Chapter 3 re-present findings and learnings from the international literature review originally presented in Evaluating Best Practice in Place-Based Service Delivery: A Literature Review. This includes findings from a close review of 12 international PBIs selected for their dual place-and-person focus, which reflects the type of PBIs typical to the Australian policy context. While much of the content will be similar to the initial report, it has been updated in accordance with feedback received from PM&C.

The methodology employed for the collection of the PBI evaluation documentation from Commonwealth departments is detailed in Chapter 4, along with a discussion of the extent to which the initiatives have incorporated the "common elements", both individually and as a whole.

Finally, Chapter 5 discusses the major implications of the findings of this meta-review and provide recommendations regarding how place-based initiatives and their evaluations can produce the best possible outcomes. This includes comparing the findings for both the international and Commonwealth reviews of place-based service delivery initiatives.

2. Classifying place-based initiatives

There is consensus within the international literature that PBIs have wide-ranging foci, rationale, funding, geographic scale and boundaries, target populations and target locations. Certainly, the international initiatives identified in the literature review represented a multitude of priorities, echoing Campbell & Meadows' (2001) description of PBIs as a "jigsaw of policies" (p. 3). Considering the themes of disadvantage alone, the international initiatives spanned health, education, child development, family wellbeing, community strengthening, housing, urban regeneration, sustainability, liveability, crime, employment, participation, economic development, immigrant communities, social inclusion and social exclusion. A number of initiatives were multifaceted in nature and aimed to address a range of themes of disadvantage in combination, echoing what Griggs, Whitworth, Walker, McLennan, & Noble (2008) described in the UK and what Parkinson (1998) described across Europe:

Some ABIs [area-based initiatives] focus on specific elements of disadvantage … or on specific subgroups, … while others are more holistic and encompass different dimensions. The New Deal for Communities (NDC), for example, tackles five elements (or themes) of disadvantage - worklessness, high levels of crime, educational under-achievement, poor health and problems with housing and the physical environment. (Griggs et al., 2008, p. 11)

Parkinson (1998) stated that:

policy instruments vary enormously across countries. In addition, the target groups for the policy vary. They include the unemployed, those lacking skills and qualifications, public housing tenants, lone parents, ethnic minorities, the mentally ill as well as drug and alcohol abusers. (p. 3)

Moving beyond a "consensus of variety" are typologies of PBIs that strive to classify initiatives according to their policy objectives and targets relating to place and person. For instance, Katz (2004) grouped PBIs in the United States to form three types:

  1. Improving the neighbourhood, which is strictly "place-based" and "focuses on making urban communities quality places to live", sparking revitalisation by improving physical stock and the commercial quality of the local area while also giving neighbourhood institutions a central, "community development" style role in planning and development (Katz, 2004, p. 13).
  2. Expanding opportunity, which is "people-based" and "focuses on giving residents of distressed neighbourhoods improved access to quality jobs and good schools in the broader metropolis" (Katz, 2004, p. 13). In the US, this strategy includes housing voucher programs like the Moving To Opportunity (MTO) initiative (in which people may move to areas of less poverty), as well as programs that improve links to jobs, or allow low-income families to make educational choices beyond the local school.
  3. Transforming the neighbourhood, which contains both people- and place-based components and "focuses on fundamentally altering the socio-economic mix of distressed neighbourhoods and creating communities that are economically integrated and attractive to a broad range of households" (Katz, 2004, p. 13). The US place-based initiative Hope VI is given as an example of this third type of strategy: a housing transformation effort that improved physical public housing stock while simultaneously funding support services to assist returning residents in their transition to work.

Griggs et al. (2008) further broke down this three-type classification, distinguishing policies from the United Kingdom according to whether the objectives were more focused on targeting the person or the place. There were five types of PBIs (see also Figure A1, Appendix A):

  • Type 1: Major focus on place in order to impact place - These often have a regional development and sustainable infrastructure focus; for example, the UK's City Growth Strategy.
  • Type 2: Major focus on place in order to impact person - These improve local infrastructure explicitly to enhance the lives of current and future residents; for example, the UK's Sure Start, Excellence in Cities (EiC), and Employment Zones (EZ) policies.
  • Type 3: Major focus on person in order to impact place - These enforce improvements in individuals' behaviours for the benefit of the neighbourhood; for example, Anti-Social Behaviour Orders.
  • Type 4: Major focus on person in order to impact person - These provide universal delivery of services, irrespective of location; for example, social security benefits, in which "the same benefits are payable throughout the country and have traditionally been delivered in a uniform fashion by national organisations" (Griggs et al., 2008, p. 3).
  • Type 5: Simultaneous major focus on place and person in order to impact both - These exploit synergies between the twin goals of place and person, and recognise that the separation of place and person "does not reflect a reality in which poverty and disadvantage are mediated by place, and places are affected by the poverty or otherwise of their inhabitants" (Griggs et al., 2008, p. xii); for example, the UK's New Deal for Communities (NDC), and the Working Neighbourhoods Pilot.

Both typologies point to an evolution of place-based policy in which the most recent initiatives simultaneously focus on people and places (Dyson, Jones, & Kerr, 2009; Griggs et al., 2008; Katz, 2004; Meadows, 2008).

Given the trend towards approaches that combine the dual concerns of place and people, and this project's emphasis on PBIs in operation over the past five years, we have focused more attention on the initiatives included for review that feature the "twin goals" of place and people.

3. Common elements of international place-based initiatives

Systematic reviews and syntheses identified in the international literature review described place-based practices from a number of different perspectives. Some sources openly critiqued national place-based policy initiatives and questioned their overall effectiveness, or the efficacy of certain elements (e.g., Chatterton & Bradley, 2000; Modarres, 2002; Power, Rees, & Taylor, 2005; Warin, 2007). Many sources implied a "what works" approach by addressing its negative - "what doesn't work" - and defined future needs and possibilities in relation to the limitations of the PBIs and outcomes that were "mixed" (O'Dwyer, Baum, Kavanagh, & Macdougall, 2007, p. 329; see also Cytron, 2010), "uncertain" (Thomson, 2008, p. 933) or "patchy" (Power et al., 2005, p. 106). A study by Parkinson (1998) illustrates this tendency, where lessons from the author's analysis of place-based social exclusion programs in Europe were presented under the banner of "progress, problems and prospects", and summarised as "challenges" rather than as "critical success factors". Indeed, the authors of the UK's Working Neighbourhoods Pilot evaluation were considered "brave" for proposing lessons for policy-makers and practitioners in developing education-focused PBIs (Griggs et al., 2008). The reluctance of researchers to speak positively about "what works" probably reflects the varied nature of PBIs (Meadows, 2008; Parkinson, 1998), and also the capacity of evaluations to define and provide a detailed understanding of the important elements of exactly what does work (e.g., Burton et al., 2004).

Of most relevance to this review were evaluation reports for the 12 international PBIs selected for close review (as designated in the final column of Table B1 in Appendix B), as well as syntheses and systematic reviews of PBIs in the US, UK and EU. These sources analysed place-based practice through various lenses. A number of initiatives considered "what works" for particular themes of disadvantage:

  • employment (Austin & Lemon, 2005; Campbell & Meadows, 2001; Griggs et al., 2008; Meadows, 2008);
  • education (Dyson et al., 2009; Griggs et al., 2008);
  • urban regeneration and neighbourhood revitalisation (Cytron, 2010; Katz, 2004; Rae, 2011; UK House of Commons, 2003);
  • housing (Hulse, Jacobs, Arthurson, & Spinney, 2011; Katz, 2004; Ware, Gronda, & Vitis, 2010); and
  • health inequities (O'Dwyer et al., 2007).

Others examined in isolation "what works" for particular elements of PBI design and delivery:

  • integration and "joined-up working" (McGregor, Glass, Higgins, Macdougall, & Sutherland, 2003);
  • community involvement (Burton et al., 2004); and
  • participatory governance and "ownership" (Sullivan & Stewart, 2006).

Irrespective of which lens has been applied to examine good practice in PBIs, several common elements are voiced by the literature, and these are presented below. However, it is crucial to recognise that these common elements are based on a literature whose strength of evidence is very limited. Assertions are made in the evaluation reports, but there is limited evidence to substantiate the claims made.

3.1 Common focus on place and person in place-based initiatives

Spatial targeting

Spatial targeting - having an appropriate focus on geographical areas - is seen as a common element of PBIs (Dyson et al., 2009; Rae, 2011; UK House of Commons, 2003). A theme in the literature is that place-based initiatives should be designed with sensitivity to boundary issues and the shape of the "natural community" (UK House of Commons, 2003), including "proper analysis of how local factors interact with each other and with more macro-level factors operating beyond the designated area" (Dyson et al., 2009, p. 26; see also Dyson, Kerr, & Raffo, 2012). Much advice focuses on taking into account how broader metropolitan patterns shape local opportunities and cultures, thereby ensuring that an initiative is not "focusing so closely on a neighbourhood that its wider regional context is ignored" (Cytron, 2010, p.5; see also Hulse et al., 2011; Katz, 2004; Rae, 2011; UK House of Commons, 2003). In systematically reviewing evaluations of place-based health initiatives, O'Dwyer et al. (2007) argued that initiatives work best when the size of the target area is appropriate to the particular inequality (p. 331), echoing an earlier statement by the UK Department for Communities and Local Government that "the services that are delivered best at neighbourhood level are those that interact at that level with service users" (as cited in Rae, 2011, p. 343). However, it is important to note that according to O'Dwyer et al. (2007), the scale of a geographic area and whether this was associated with the outcomes of interest had been neglected in the 24 place-based initiatives they examined. This was also the case for the 12 international PBIs closely reviewed for this project.

Of the international PBIs reviewed, all but one, Harlem Children's Zone, demonstrated spatial targeting.* Spatial targeting was demonstrated by the initiatives' implementation in multiple geographic areas, though the size of these areas widely differed between initiatives. For example, UK Sure Start Local Programs (SSLPs) targeted small areas, typically under 13,000 people (Barnes, 2007; Malin & Morrow, 2008; see also Lawless & Pearson, 2012), whereas the US Moving To Opportunity program targeted large cities (Goering et al., 1999; see also Hutchings et al., 2012). Occasionally, very specific spatial targeting recommendations were imposed: the UK Neighbourhood Nurseries Initiative (NNI), for example, advised local authorities to locate neighbourhood nurseries near major roads, with the intention of attracting higher income parents on a "travel to work basis", for reasons of social mix and sustainability (see "Long-term focus") (NNI Research Team et al., 2007). The reason for initiatives targeting areas of particular sizes was not often explicit. Where it was articulated, the size of targeted areas tended to be associated with the nature of the initiative and/or its theory of change. For example, in the UK, in practice, Employment Zones reflected existing Jobcentre service catchment areas (Hales, Taylor, Mandy, & Miller, 2003), and education-themed programs - with the exception of City Challenge (CC) - tended to target schools located within broader local authority boundaries (Ghate, Asmussen, Tian, Hauari, & Policy Research Bureau, 2008; Kendall et al., 2005). In the case of City Challenge, the theory underpinning the program was that issues of poor school performance and low education attainment functioned across the boundaries of local authorities and were, as such, city-level problems, needing broader geographical area targeting (Hutchings et al., 2012; Rudd et al., 2011). Finally, while most initiatives reviewed operated within one country, On Track operated throughout both England and Wales, and the URBAN I Community Initiative (URBAN I CI) operated across the then 15 European Union member states.

International PBIs have tended to undertake spatial targeting on the basis of the prevalence of a specific type, or theme, of deprivation or disadvantage in a location of a given size. Despite a common process involving areas bidding for selection to participate in an initiative, the rationale for choosing between potential areas was inconsistently articulated by the international PBIs. The UK National Strategy for Neighbourhood Renewal (NSNR) and the Neighbourhood Nurseries Initiative exemplify initiatives that clearly articulated an indicator of deprivation. Spatial targeting under these initiatives applied the Index of Multiple Deprivation in order to identify England's most deprived neighbourhoods, or "pockets" of deprivation existing within more affluent areas (Amion Consulting, 2010; NNI Research Team, 2007). By contrast, other initiatives cited the selection of areas based on ensuring funding equality across the country, or a mix of urban, semi-rural and rural communities (New Zealand [NZ] Ministry of Social Development [MSD], 2004).

Evaluating spatial targeting

Only a few international PBIs evaluated spatial targeting. Where this occurred, evaluations typically measured the extent to which the initiative had reached the disadvantaged areas in which it intended to operate. For example, the Neighbourhood Nurseries Initiative evaluation found that by 2005, almost 60% of neighbourhood nurseries were located in the 20% most disadvantaged neighbourhoods in the country (and almost 75% if the definition of disadvantage was widened to be the 30% most disadvantaged neighbourhoods) (NNI Research Team, 2007). Similarly, the 24 areas targeted by On Track were found to demonstrate substantial deprivation, as evidenced by a median district deprivation ranking for all areas of 5 (where 1 is the score of the most deprived and 500 is the score of the least deprived local authority in England [Ghate et al., 2008]). Evaluators of the National Strategy for Neighbourhood Renewal provided insights into factors affecting the achievement of deprived area targeting. The evaluation found that about half the authorities did not spatially target within their borough, due to the frequent perception that the prevalence of deprivation throughout the entire borough would make spatial targeting discriminatory and inequitable. The evaluation also reported that there was a tendency for greater geographical targeting of interventions where neighbourhood management structures had been established (Amion Consulting, 2010).

Social targeting

Social targeting - having an appropriate focus on populations - is likewise viewed as a common element of PBIs. Of the 12 international PBIs closely reviewed, all demonstrated social targeting. Population groups targeted included:

  • low-income families (Dobbie & Fryer, 2010; Goering et al., 1999; Sanbonmatsu et al., 2011);
  • residents of deprived areas (Amion Consulting, 2010; Lawless & Pearson, 2012), such as children in families living below the official poverty line (National Evaluation of Sure Start [NESS], 2012), or the long-term unemployed (Hales et al., 2003);
  • primary and secondary school students and "disadvantaged pupils" (Hutchings et al., 2012; Kendall et al., 2005);
  • children of a specific age range and their parents, such as 4-12 year olds and their families (Ghate et al., 2008) and 0-4 year old children and their unemployed parents, especially lone parents (NNI Research Team, 2007); and
  • Indigenous communities, such as members of 14 whānau, hapū, iwi and Māori communities (NZ MSD, 2004).

Initiatives targeting socio-economically deprived groups often emphasised one or more socio-demographic characteristic of the population, such as their living within public housing (Sanbonmatsu et al., 2011); disorder, child health and developmental outcomes (Barnes, 2007; Malin & Morrow, 2008); or socio-spatial factors, such as high unemployment, poor housing conditions and a lack of social amenities (GHK, 2003).

Austin and Lemon (2005) noted that the intentional use of social targeting to reach "a segment of the workforce that is not generally 'captured' by social services" (p. 67) is a success factor among place-based employment initiatives. The authors gave the example of the Neighbourhood Jobs Initiative, implemented in high-poverty US neighbourhoods as a place-based program, whose success was enabled by such social targeting. Griggs et al. (2008) identified similar social targeting as a success factor in the UK's New Deal for Lone Parents and Pathways to Work initiatives. Both initiatives targeted population groups that had been excluded from previous employment schemes. According to the authors' analysis of evaluation materials, the initiatives had "the strongest impact on entries into employment" as a result of tapping "a pool of people eager to return to work who had previously effectively been prevented from doing so" (p. xiv).

Evaluating social targeting

Of the international PBIs closely reviewed, a small number evaluated social targeting. Under the City Challenge initiative, attainment outcomes were analysed for both students who did and did not exhibit a particular indicator of disadvantage - the receipt of Free School Meals - in all three Challenge areas, and compared with national figures (Hutchings et al., 2012). Using data from the UK Census 2001, evaluators of the On Track initiative calculated an approximate figure for project "reach" as the proportion of children aged 5 to 14 in each intervention area recorded as users of an On Track project. On average, 18% of children were service users, ranging from 3% to 94% across the 23 areas (Ghate et al., 2008).

3.2 Common elements in design and delivery

Flexible delivery

Flexible delivery is the ability to demonstrate flexibility in both service delivery according to community need, and expenditure of funding. The ability to adjust the rules in order to adapt government place-based policies and programs to meet the needs of local communities is considered vital to the success of PBIs. Despite considerable variation in initiative design, flexibility is a common characteristic of successful place-based employment initiatives (Griggs et al., 2008; Meadows, 2008), education initiatives (Griggs et al., 2008) and initiatives targeting urban regeneration and neighbourhood revitalisation (Katz, 2004; UK House of Commons, 2003).

Flexible delivery was demonstrated by most international PBIs and was built into their design in a range of ways. These included considerable administrative freedom in the case of the Harlem Children's Zone initiative (Dobbie & Fryer, 2010), or managerial freedom, as demonstrated by England's City Challenge initiative, in which each area was permitted slightly different arrangements for managing the program (Hutchings et al., 2012). Flexible delivery was also demonstrated regarding the type and mix of services offered as part of the initiative. The Excellence in Cities initiative enabled diversity of service provision so that the needs of all pupils were met (Kendall et al., 2005), and the Harlem Children's Zone sought feedback from the community about the programs and services they would like to see delivered (Whitehurst & Croft, 2010). Some PBIs featured customisable support as a single intervention strand that operated alongside a standardised core of activities (Hutchings et al., 2012). The On Track initiative's use of a "Specialist" intervention category, which enabled flexibility in service provision to families with more specific individual needs, is an example of this type of flexible delivery (Ghate et al., 2008). These examples reflect the assessment by Griggs et al. (2008) that flexibility is a key factor in "what works" for education-oriented PBIs, as it allows programs to meet the "disparate and changing needs of learners" (p. 49).

Additionally, flexible delivery was demonstrated through discretionary funding mechanisms (Hales et al., 2003; NZ MSD, 2004). For example, under England's Employment Zones initiative, Personal Advisors could spend funds according to activities most pertinent to job-seeking participants (Hales et al., 2003). Similarly, Dewson and colleagues (2007) described the UK's Working Neighbourhoods Pilot as having the flexibility "to tailor provision in response to an individual client's needs" as an essential part of effective place-based employment initiatives (as cited in Griggs et al., 2008, p. 26). Lessons from the UK and US are similar in their advice that national governments should remove rules that impede cooperation and limit the creative use of funding to enhance flexibility (Katz, 2004, p. 35). However, it should be noted that this advice is in contrast to other research showing the effectiveness of manualised programs that focus on program fidelity (NESS, 2012).

Evaluating flexible delivery

Evaluation of flexible delivery by international PBIs was mixed. The majority of international PBIs either did not measure the effects of flexible delivery, or captured it only to a limited degree. An exception was found in the evaluation of the On Track initiative, which measured the extent of services provided through its "Specialist" intervention strand, which collectively accounted for 45% of all services offered by On Track in Phase Two (Ghate et al., 2008). On Track was also found to demonstrate flexible delivery by allowing sites the freedom to choose between service delivery models and environments. The evaluation also reported the frequency with which different delivery methods and settings were adopted.

Local autonomy

Strongly associated with flexible delivery is the principle of local autonomy. Local autonomy is the involvement of the local community in a program through consultation and active involvement in decision-making. According to the literature, successful PBIs are "locally grounded" and pursue meaningful community involvement and engagement via the identification and pursuit of local context and priorities (Campbell & Meadows, 2001; Griggs et al., 2008; Meadows, 2008). The importance of involving the local community - including local organisations, authorities, practitioners, young people and parents - throughout the life of an initiative is underscored in desktop reviews of literature investigating lessons learned from area-based regeneration policies (Campbell & Meadows, 2001; Hulse et al., 2011; Griggs et al., 2008; Meadows, 2008). Successful PBIs are said to involve local people in the process of defining their area's problems, needs and solutions, and are grounded in local economic and social circumstances. For example, Griggs et al. (2008) identified that building local networks was a success factor in the UK's Working Neighbourhoods Pilot initiative, citing Dewson et al.'s (2007) evaluation of the Working Neigbourhoods Pilot, which showed that unemployed clients are better able to address their particular barriers to work "when given a full range of locally determined support measures" (as cited in Griggs et al., 2008, p. 27). Oral evidence from key stakeholders collected by a UK House of Commons (2003) inquiry also indicates that community-led regeneration was a feature of successful New Deal for Communities programs.

The application of community processes feeds into other critical success factors, such as ensuring community "ownership" of an initiative (Meadows, 2008; UK House of Commons, 2003), increasing community cohesion (Austin & Lemon, 2005; UK House of Commons, 2003) and building public will (Cytron, 2010). Literature reviews of "what works" in place-based employment initiatives, for example, suggest that a positive sense of ownership is likely to encourage others to take part in an initiative and to generate greater enthusiasm and commitment among participants (Campbell & Meadows, 2001; Meadows, 2008). Austin and Lemon's (2005) analysis of "promising practices" in US place-based employment initiatives proposed that successful programs encourage community support for work and increase the quality and quantity of residents' social networks to facilitate information sharing. Finally, the UK House of Commons (2003) inquiry posited that effective initiatives may go so far as to invite residents from neighbouring wards to participate in the development of projects beneficial to both communities, aspiring to build bridges not only within, but also between, divided communities. While the reviews to date are supportive of this as a key element, caution should be exercised. As Burton, Goodland, and Croft (2006) noted:

While community involvement is assumed to be a good thing, there have been few empirical studies that set out to test rigorously whether the "good things" of greater involvement are realised in practice … The practical expression of the principle can and should be described empirically in order to be able to judge the extent of its realisation. (pp. 307-308)

Just over half of international PBIs we reviewed demonstrated local autonomy. Some initiatives, like the Harlem Children's Zone and Sure Start Local Programs, sought to involve all of those concerned with children in the local community. Harlem Children's Zone pursued a holistic, neighbourhood-based approach to students' academic achievement (Whitehurst & Croft, 2010) and emphasised parents' involvement in the educational process (Dobbie & Fryer, 2010). Sure Start Local Programs aimed to bring together parents with the health, social and education services, and the private and voluntary sectors, under the banner of community participation (Belsky et al., 2006). The NZ Whānau Development Project (WDP) also prompted a coming together of community, where most communities that had not previously received any focused government funding held hui (a gathering or assembly) to discuss the best way to respond to community needs (NZ MSD, 2004). The initiative aimed to allow communities as much control over the process and content of local initiatives as possible. Communities were encouraged to define whānau development for themselves and decide how they wanted to proceed, while the NZ Ministry of Social Development (2004) took a "hands off" approach.

Community involvement approaches, often activated through local partnerships, were another common expression of local autonomy. A key principle of the National Strategy for Neighbourhood Renewal was for community involvement in planning for and delivering the improvement of their areas, which was encouraged by Local Strategic Partnerships (Amion Consulting, 2010). Similarly, the New Deal for Communities initiative sought to place communities "at the heart" of the regeneration process, and the involvement of local people in the planning, design, delivery and review of local programs was achieved through partnerships (Batty et al., 2010).

Evaluating local autonomy

Evaluation of local autonomy by international PBIs was generally poor. As with flexible delivery, the majority of international PBIs either did not measure the effects of local autonomy, or only partially evaluated it. The National Strategy for Neighbourhood Renewal was an exception: its Local Research Project evaluation strand researched ways in which residents engaged in neighbourhood renewal, and identified benefits derived from their involvement (ECOTEC Research and Consulting Ltd, 2010). The evaluation indicated the extent of different types of engagement within case study areas and found that much community involvement was of a consultative and formal representational variety, as opposed to more interactive and empowering methods involving residents in design, delivery and decision-making aspects of neighbourhood projects. Evaluation of local autonomy by the whānau Development Project found that having local people, rather than central government, facilitate community participation in an initiative was highly effective, and reported greater community involvement and ownership of projects where community consultation and needs assessment occurred. The practice of local autonomy, in terms of maintaining positive and open links, was found to be especially important in small communities (NZ MSD, 2004).

Joined-up working

Joined-up working incorporates integrating or coordinating and developing partnerships between organisations within local areas across the government, private and community sectors. Much of the literature analysing good practice in place-based initiatives proposes joined-up working as a common element (Campbell & Meadows, 2001; Hulse et al., 2011; Katz, 2004; Meadows, 2008; O'Dwyer et al., 2007; Rae, 2011; UK House of Commons, 2003). Underpinning joined-up working is recognition that "the multiplicity of problems in one location requires a coordinated policy response across government departments" and that taking strategic, coordinated action requires a networked governance approach (Rae, 2011, p. 344). Similarly, Campbell and Meadows (2001) agreed that coordination between governments is crucial as more ministries and departments become involved due to widening policy foci and the increased complexity and scale of national PBIs.

Partnership working is considered a key aspect of joined-up working and is seen as being beneficial to delivering the wide range of support characteristic of multi-faceted place-based initiatives. As Meadows (2008) asserted: "No one organisation can provide solutions to all issues, so effective partnership working is essential" (p. 8). Katz (2004) agreed that "the intervention has to happen horizontally" (p. 35), proposing partnerships between local organisations with complementary strengths and a mix of skills and experience in order to deliver holistic place- and people-focused interventions. Griggs et al. (2008) referred to the Working Neighbourhoods Pilot evaluation report in evidencing the importance of partnerships to the success of UK place-based employment initiatives. According to Dewson et al. (2007), partnership working contributed to the initiative's success by enhancing the understanding of local barriers to employment and creating additional support for providers (as cited in Griggs et al., 2008, p. 26). Reviewing evaluations of place-based health initiatives, O'Dwyer et al. (2007) described successful interventions as including an increased number of strategic partnerships and an increased degree of partnership between government and community organisations. Hulse et al. (2011) also pointed to local partnerships as being key to the success of place-based social housing initiatives.

Joined-up working was demonstrated to some extent by all the international PBIs reviewed. Almost all the initiatives incorporated joined-up working as a fundamental component of their design and objectives. Those that did not reported post hoc a high degree of coordination for particular relationships - such as between government and service providers - and, accordingly, were deemed to have partially demonstrated the element.

Overwhelmingly, joined-up working was demonstrated by partnership working and by emphasising collaboration over competition (Rudd et al., 2011). Several international PBIs pursued joined-up working at multiple, interconnected levels throughout a community (see Ghate et al., 2008, for example). This way of working is based on an ecological model of human development, which recognises that risk and protective factors exist and interact bi-directionally at individual, family, peer-group, school and community levels. Harlem Children's Zone, New Deal for Communities and the National Strategy for Neighbourhood Renewal demonstrated particularly comprehensive scope in partnership working, including partnership working between schools and further education colleges, local people (such as students and their families), local businesses, voluntary and community sector organisations, local authorities and public agencies (Amion Consulting, 2010; Dobbie & Fryer, 2010; Lawless, 2004).

Among other initiatives, cooperation and joint working were variously demonstrated as operating between:

  • different schools, as exemplified by City Challenge's "school-to-school" and "cluster working" approaches (Hutchings et al., 2012);
  • schools and Local Education Authorities (Hutchings et al., 2012; Kendall et al., 2005; Rudd et al., 2011);
  • practitioners with different experiences, ideas and levels of expertise (Rudd et al., 2011);
  • government and Indigenous communities (NZ MSD, 2004);
  • public housing authorities and non-profit organisations (Goering et al., 1999);
  • different levels of government, such as through frameworks and tools supporting processes of public sector vertical and horizontal integration (Amion Consulting, 2010), or by linking in local representatives and committees as part of governance management structures (GHK, 2003); and
  • different service providers via multi-agency partnerships, which sought to offer the maximum amount of different services individuals needed in a "multimodal" service offering that facilitated a "no-wrong-door" approach (Ghate et al., 2008).
Evaluating joined-up working

The evaluations by international PBIs of the effectiveness of joined-up working were mixed. One-third of those reviewed were found to have formally evaluated joined-up working, and their measurement approaches varied. The evaluators of the Excellence in Cities program sought to demonstrate an overall "partnership dividend" - additional benefits that accrue to schools and pupils from working together and from sharing resources - based on the finding that pupils' attainment at the end of Key Stage 3 was greater in those areas where there was evidence that schools were demonstrating a high level of engagement with the EiC partnership (Kendall et al., 2005). The Excellence in Cities evaluation also exemplified good practice in seeking representation of the perspectives of all entities engaged in partnership working, such as its interviews with government-based Partnership Coordinators, school managerial staff and teachers. In contrast, On Track evaluators measured partnership working as the extent to which service users received multiple services. They found that over half of all users of targeted services were offered a multi-modal service and 46% of all users of targeted services received multiple services (Ghate et al., 2008). The evaluators noted that the On Track data did not permit analysis of whether those who used multiple services had better outcomes than those who only used one service, although the evaluation's qualitative component suggested that multi-modal working with families was an important factor in meeting their needs.

Governance

As a recent report, Governance Models for Location Based Initiatives (Australian Social Inclusion Board, 2011), addresses governance issues and makes recommendations as to key governance elements of PBIs, we do not seek to replicate this discussion in any detail in this report and refer the interested reader to that report. Many of the issues that were identified as common practice with respect to governance are also mentioned here separately, such as the emphasis on joined-up working, the devolution of the task of identifying issues and solutions to enhance local autonomy, the need for capacity development, and enhanced measurement and cost-effectiveness. One of the tensions with governments enabling flexibility in service delivery and enhancing local autonomy - which seems to be at the heart of place-based initiatives that focus on people and places - is that departments may need to cede some accountability for government funds to local communities (e.g., UK House of Commons, 2003). Developing appropriate accountability mechanisms is important in this regard. Accordingly, in reviewing international PBIs for demonstrations of governance, we focused most on governance mechanisms that supported devolution in decision-making and, in some way, negotiated the tension between too much and too little government accountability.

Just over half of the international PBIs reviewed demonstrated governance mechanisms that devolved decision-making. These mechanisms negotiated the government accountability tension in diverse ways:

  • under Excellence in Cities, decision-making regarding funding expenditure was devolved to local area educational partnerships on the condition of receiving an approved delivery plan (Kendall et al., 2005);
  • under the whānau Development Project, decision-making was devolved via a culturally relevant facilitating organisation, which provided communities with project management and business, developmental and mentoring support (NZ MSD, 2004);
  • under Moving to Opportunity, public housing authorities were given some degree of autonomy, but were required to follow a set of standardised procedures outlined in the initiatives' Programs Operations Manual (Goering et al., 1999);
  • with New Deal for Communities, boards comprising 50% local resident membership were given a degree of freedom in proposing 10-year strategies and in allocating funds, though operational restraints such as government approval still applied (Lawless & Pearson, 2012); and
  • under the National Strategy for Neighbourhood Renewal, a series of accountability structures resulted in the definition of distinct responsibilities, expectations and approaches at different levels, but also obligated a degree of vertical integration between the levels (such as regional and local, and local and neighbourhood). For example, NSNR's regional networks acted as a support for Local Strategic Partnerships, which were responsible for administrating a number of funds, and were in turn expected to ensure effective consultation of all community sectors and encourage neighbourhood management approaches (Amion Consulting, 2010).
Evaluating governance

Few international PBIs formally evaluated governance, but there were a few exceptions. The On Track initiative qualitatively examined the evolution of governance over the course of the initiative (Ghate et al., 2008), while the whānau Development Project qualitatively examined the effective management of relationships (e.g., advantages and disadvantages of relationship dynamics, aspects of lesser or greater authority and control) between communities, service providers, the Ministry of Social Development and the facilitating organisation (NZ MSD, 2004). The National Strategy for Neighbourhood Renewal investigated the effectiveness of governance arrangements, such as Local Strategic Partnerships and a pilot approach to neighbourhood management, the Neighbourhood Management Pathfinder (Amion Consulting, 2010). The NSNR evaluation found that Local Strategic Partnerships enhanced local governance by developing a collective vision, establishing strategic priorities, better representing a wider variety of local interests, improving perceptions of local governance processes and creating a more coherent local voice with which to exert regional and national influence. The initiative's Neighbourhood Management Pathfinder model was found to be most influential to substantive local involvement in decision-making, and resident satisfaction in areas using this model rose faster than in comparator areas (Amion Consulting, 2010).

3.3 Common elements in program implementation

Capacity development

The international literature suggests that capacity development, at both the local level and in government, is essential for effective PBIs. Capacity development includes technical assistance in the form of resources and training pertinent to the implementation, operation and evaluation of PBIs. Much is also written about the necessary calibre of staff involved in place-based service delivery, including an appropriate level of practitioner skill and experience, as well as an awareness of the potential for community "burnout" and "fatigue" arising from over-reliance on a limited number of skilled community representatives (UK House of Commons, 2003). Staff with the personal qualities of enthusiasm and commitment, and who have skills in communication, management and partnership working, as well as access to professional support, are considered to be key to the success of PBIs (Dewson et al., 2007, as cited in Griggs et al., 2008; Meadows, 2008).

The range of skills required to design, implement and maintain PBIs is often overlooked, and mechanisms for sharing best-practice learnings are not well established (Campbell & Meadows, 2001; UK House of Commons, 2003). Government skills and knowledge exchange programs, community capacity audits, and the incorporation of skills and capacity targets are all proposed as practice mechanisms to establish and contribute to the level of capacity investment required for PBIs (UK House of Commons, 2003). It is notable that in the United States, the Promise Neighborhoods initiative set up a Promise Neighborhoods Network (a "community of practice" that involved sharing tools and resources, attending training and webinars, and supporting each other's work) in 2011, as Promise Neighborhood grantees moved through the planning and implementation stages (Promise Neighborhoods Institute, 2011).

Finally, and particularly for place-based employment initiatives, the capacity to undertake outreach work was considered a critical element of success. Austin and Lemon (2005) identified that facilitating good service access by "offering services close to residents home [sic]; providing services during non-business hours; and conducting home visits" (p. 70) are all examples of promising practice driven by outreach work. Meadows (2008) described outreach work as "an important way of overcoming some of the reluctance to engage with organisations and interventions that might be able to help" (p. 7). This reluctance can relate to: poor information or social networks; disability or chronic illness; difficulties relating to income, transport or language difficulties; low self-esteem; ex-offender status; or negative attitudes towards participation among "hard-to-reach" population groups. In this way, outreach work also links to the element of flexible service delivery by helping to reach clients who require specialised support as a result of their particular experiences of disadvantage (Dewson et al., 2007, as cited in Griggs et al., 2008).

Capacity development was an element demonstrated by half of the international PBIs reviewed. Some PBIs demonstrated capacity development at an overall, or macro, level. For example, the Harlem Children's Zone reported substantial community investment, with over 20 programs implemented across various participant groups with the aim of improving both communities and schools so as to positively affect student achievement and ensure children are surrounded by an enriching environment of college-oriented peers and supportive adults (Dobbie & Fryer, 2010). A similarly "top-level" approach to capacity development was demonstrated by the European Union's URBAN I Community Initiative, which aimed to undertake "networking activities", such as mutual cooperation and exchange of experience and best practice among URBAN programs (GHK, 2003). Other initiatives demonstrated capacity development through more specific objectives, including: knowledge collection and sharing, particularly across management levels/areas of expertise (Rudd et al., 2011); training and workshops (Hutchings et al., 2012; NESS, 2011); learning support resources (Kendall et al., 2005); capital resourcing (Hutchings et al., 2012; Kendall et al., 2005); and provision of expert guidance and advice. Examples of the latter include City Challenge's highly skilled and experienced Challenge Advisors (Hutchings et al., 2012), as well as the whānau Development Project's contracting of a consultancy focusing on capacity-building that had experience in managing Māori and Pacific provider development, which assisted the Ministry for Social Development with project management, and provided business, developmental and mentoring support to communities as needed (NZ MSD, 2004).

Evaluating capacity development

Examples of evaluating capacity development were found in just under half of the initiatives reviewed. For example, the Excellence in Cities initiative evaluated improvements to teaching and learning (Kendall et al., 2007); the establishment of a culture of professional development; and the role and effects of City Learning Centres in supporting and disseminating the use of information and communications technology (Ghate et al., 2008). Benefits of the initiative most frequently cited by teachers included: being able to practice a wider range of teaching activities; having more opportunities to try new teaching and learning methods; having more opportunities for exchanging ideas with colleagues; having access to additional and/or more appropriate resources; and using information and communications technology more in their teaching. Where its specific strands had a capacity-development component, the City Challenge initiative similarly assessed the effectiveness of these elements. For example, head teachers of schools participating in the Good to Outstanding (later Good to Great) intervention strand were surveyed on the effectiveness of nine capacity-development elements, such as "attending a conference, seminar or master class", "leadership training", or "working with the head of an outstanding school". Feedback on these aspects of the program was overwhelmingly positive (Hutchings et al., 2012; Rudd et al., 2011).

Lead times

Lead times refer to set-up periods prior to program implementation, which are needed to build relationships within communities, build capacity within service delivery organisations and ensure evaluations are in place. There is consensus in the literature that PBIs of all types take time to establish in order to be implemented effectively. This includes time to:

  • develop strategies and delivery plans (Campbell & Meadows, 2001);
  • build relationships and mutual knowledge bases (Meadows, 2008);
  • have a "year zero" in which local partnerships can be established and planning take place (Hulse et al., 2011);
  • build the capacity of communities and other partners involved in partnership delivery (Campbell & Meadows, 2001; see also Dewson et al., 2007, as cited in Griggs et al., 2008);
  • ensure evaluation capacity is in place before interventions are put in place (O'Dwyer et al., 2007); and
  • allow initiatives "sufficient time to bed in" (Dewson et al., 2007, as cited in Griggs et al., 2008, p. 27).

Understanding the context in which the initiative is operating is also seen to be important. Cytron (2010) advised that funders make an effort to understand a community's institutional assets and systems ahead of time, to help avoid "reinforcing [socially] exclusive or otherwise broken systems that might derail the overall aims of a community initiative" (p. 5).

Most international PBIs did not report on lead times - of those reviewed, only three built lead times into their design. Piloting the initiative in "prototype" areas, where participation was voluntary, was a feature of England's Employment Zones. The pilot Employment Zones were formally evaluated, with this evaluation leading to policy modifications for the "fully fledged" Employment Zones initiative, for which participation was mandatory for long-term claimants of the Job Seeker Allowance (Hales et al., 2003). Similarly, Harlem Children's Zone was developed in 1970 from an amalgam of after-school programs, truancy prevention services and anti-violence training for teenagers, and was applied to a 1990s pilot of birth-to-college services (Dobbie & Fryer, 2010). In 1997, Harlem Children's Zone was implemented across a 24-block area and in 2007 was further expanded to almost 100 blocks (Dobbie & Fryer, 2010; Harlem Children's Zone, 2014). Sure Start Local Programs took into account lead times in a different way, by gradually increasing the number of targeted areas over time. The number of targeted areas rose progressively from 60 small areas at the initiative's commencement in 1999, to 524 areas in 2004 (Belsky et al., 2006).

Evaluating lead times

Evaluation of the effects of having lead times was not an explicit feature of any international PBI reviewed. While several evaluation reports briefly acknowledged the negative effects associated with an initiative's lack of lead time, these seemed to have been observed in a post-hoc manner, rather than in response to a formal research question about lead times. For example, the City Challenge evaluation observed that while common objectives were met across all three City Challenge areas, this was most so in London where the initiative had built on the work of a previous intervention called London Challenge (Rudd et al., 2011).

Long-term focus

Long-term focus represents the understanding that disadvantaged areas have complex problems that take time to remedy and require sustained investment and patience in realising results. Long-term funding was seen as being important for PBIs (Hulse et al., 2011; Rae, 2011; UK House of Commons, 2003). The UK House of Commons inquiry recommended the use of community-based organisations, such as neighbourhood trusts, as one mechanism to provide an ongoing asset revenue stream for initiatives and to ensure sustainability beyond the end of an initiative. Tied to this is a need to change popular perception of what constitutes "long-term" for the implementation and evaluation of PBIs. Rae (2011) reflected on the evaluation of the UK New Deal for Communities initiative that "even a decade is 'short term' when compared to the persistent problems that regeneration seeks to address". Rae called for an intergenerational commitment to "view 25 rather than five years" (p. 344) as an appropriate time frame in which to effect change.

Stable, dependable and predictable policy is seen to be imperative in the successful delivery of PBIs. Reflecting on the US and UK neighbourhood policy contexts, Katz (2004) described the inconsistent operation of PBIs as "characterised by numerous starts and stops, lurches and reversals" (p. 36), which he noted as being due to both changes in political administration and the natural evolution of policy. Katz argued that continuous policy application is needed both to keep a program's long-term objectives on track and to prevent burnout among practitioners involved in their design and implementation. O'Dwyer et al. (2007) supported this opinion, drawing on findings from a meta-evaluation of UK place-based health initiatives to suggest that initiatives work optimally when there is political commitment, adequate funding and unchanging program objectives.

Half of the international PBIs reviewed demonstrated a long-term focus, such as through long-term funding, and operational and policy commitments. For example, Harlem Children's Zone had a long-term commitment to its Promise Academy charter schools, with a new kindergarten and sixth-grade cohort to be introduced each year until they were full K-12 schools (Dobie & Fryer, 2010). Several initiatives were found to run for close to a decade or longer: On Track was operational for approximately nine years; New Deal for Communities ran from 1998 through to 2011; and Sure Start Local Programs have been in operation for up to 14 years (at minimum, 9 years) (Belsky et al., 2006; Ghate et al., 2008). Accordingly, these initiatives have involved sizeable long-term investments; for example, program-wide funding for New Deal for Communities was about £2 billion, or approximately £50 million to each of the 39 areas from 1998-99 through to 2010-11 (Lawless & Pearson, 2012).

Long-term focus was also demonstrated by the initiatives' aims to have successful service components taken forward by mainstream providers (Amion Consulting, 2010; Ghate et al., 2008; NNI Research Team, 2007). Strategies to taper funding over the life of an initiative were built into program designs on the assumption that successful services would become embedded into the core work of voluntary and statutory agencies over time (Ghate et al., 2008; NNI Research Team, 2007). The Neighbourhood Nurseries Initiative provided a sliding scale of support for nursery running costs in the first three years, and sought to attract higher income parents to bolster the program by taking up non-NNI-funded places in these nurseries. Forming effective partnerships was also a vehicle for mainstreaming in both On Track and the Neighbourhood Nurseries Initiative, with the latter encouraging partnerships with other providers and funders, such as from Sure Start Local Programs, the New Deal for Communities initiative and the National Health Service (NNI Research Team, 2007).

Evaluating long-term focus

The few initiatives that formally evaluated long-term focus did so by measuring the extent to which mainstreaming had been achieved (Amion Consulting, 2010; Ghate et al., 2008; NNI Research Team, 2007). Findings from exit interviews indicated at least half of the On Track projects had successfully mainstreamed at least one of their services, and in some instances had mainstreamed most, if not all, services (Ghate et al., 2008). The National Strategy for Neighbourhood Renewal evaluation similarly found that, for 123 projects assessed, 68% of interventions initially reliant on its National Renewal Fund were subsequently fully funded by mainstream providers. Certain types of services were more likely to be adopted for delivery by mainstream providers, such as those affecting crime and community safety (Amion Consulting, 2010), those operating effectively in schools, and those perceived as a priority by the local authority (Ghate et al., 2008).

3.4 Common elements in evaluation

Common elements of PBI evaluation also reflect common practice in evaluation more generally. Hence, while the lessons presented here relate specifically to the evaluation of PBIs, they are also applicable to other, non-place-based government service delivery initiatives. Though not a focus of this report, the reverse may also hold true.

Causality

Earlier reviews of the place-based literature have been particularly critical of the absence of evaluations that enable causal inferences to be made (Burton et al., 2004; Parkinson, 1998). The lack of an adequate control group or counterfactual was highlighted by some as being a key limitation of some of the evaluation studies to date. More recently several large-scale national UK evaluations of PBIs have employed matched control groups. For instance, both the National Evaluation of Sure Start and the national evaluation of New Deal for Communities attempted to match areas that did not receive the PBIs with those that did, based on the characteristics of the residents in the area, and collected longitudinal information from residents in the area. The evaluation of the Employment Zones also used a matched sample of areas (Hales et al., 2003). These evaluations were a significant advance on earlier evaluations. Indeed, just over 80% of the international PBIs reviewed demonstrated a robust and sophisticated methodology that was able to come close to establishing causality.

However, there are several issues with the "matched area" approach:

  • Areas are matched on the observed characteristics of the area, but other characteristics that are not observed are unable to be controlled. As noted in the Sure Start evaluation (NESS, 2012), the optimal scientific method for establishing causation in a PBI is to randomly allocate some areas to the initiative and to not allocate other areas. As a consequence, any measured differences that are observed can be said to be attributable to the initiative. It is acknowledged that randomised allocation is the "gold standard" in assessing causal inference in program evaluation, because observed and unobserved characteristics are accounted for.
  • Identifying comparable areas to "match" may not be possible, particularly if the PBI was rolled out or expanded nationally before an evidence base could be developed to establish its effectiveness (e.g., on Employment Zones, see Hales et al., 2003; on Sure Start, see NESS, 2012). In Sure Start, the evaluators were explicit in stating that the early roll out and rapid expansion of the program meant that identifying a control group was difficult and randomisation was not possible.

Other PBIs such as the Harlem Children's Zone have taken advantage of natural experiments to estimate causality. In this instance, Dobbie and Fryer (2010) took advantage of the fact that children living in a particular geographic zone were eligible to apply to be involved in the program, while those who were just outside the boundary were ineligible. They assumed that differences between those children living "in" and immediately "outside" the zone were similar in other respects. For local evaluations, this approach is possible to implement, but for national evaluations it is very difficult to identify natural experiments that apply across the nation.

Because PBIs are usually spatially and socially targeted and the areas are decided on policy lines, it is exceedingly difficult to implement randomisation in a PBI. Therefore, the evaluations that did an excellent job of trying to ascertain causality usually had four features, namely:

  • matched comparison groups or counterfactuals that enabled ascertainment of what would have happened in the absence of the initiative;
  • longitudinal data from surveys or administrative data;
  • statistical analyses that controlled for measured confounders and took advantage of the longitudinal data; and
  • more than one source of evidence.

First, the areas that received the initiative were "matched" to other similar areas to form a comparison group or counterfactual. In many instances, a sophisticated technique referred to as propensity score matching was used to ensure that, based on a range of known characteristics (usually demographic characteristics from the Census), the comparison areas were truly comparable. Second, the evaluations used high-quality longitudinal data that preceded the implementation of the initiative that were drawn from administrative records or longitudinal surveys, or both. Third, the evaluations used sophisticated statistical techniques that optimised the use of the longitudinal data by examining changes in the outcomes of interest over time to come closer to making causal claims, as well as taking into account other confounding demographic characteristics (e.g., difference-in-difference (DID) estimation in combination with matched areas). Fourth, conclusions about the effectiveness of the PBI were not reliant on one piece of evidence or one statistical estimation technique, as findings can occur by chance and statistical models have different assumptions underlying them.

It is also worthwhile noting that the quality of the methodology for many of the evaluations from the United Kingdom was very high and probably reflects not only a period of significant investment in PBIs, but development of the capacity to evaluate these sorts of initiatives.

Attribution

Given that areas characterised by disadvantage are the targets of PBIs, it is possible for several of these interventions to be operating in the same area (Coote, Allen, & Woodhead, 2004). This was particularly so in the United Kingdom in the late 1990s and 2000s (Beatty, Foden, Lawless, & Wilson, 2010), and as a consequence, it is very difficult to attribute the positive benefits observed to investment in a particular PBI (Adamson, 2010; Griggs et al., 2008; Parkinson, 1998). For Australia too, having three levels of government (local, state and Commonwealth) means that overlapping initiatives are more likely to occur than in other countries, so establishing whether other PBIs are operating in the area is also important.

Problems of attribution even occur within the one PBI if there is significant flexibility and local autonomy for the community to choose the mix of service types that best meets the needs of the community. In these instances, if there is not sufficient rigour in documenting the services that are delivered as part of the PBI, then it is very difficult to ascertain what the "active constituents" are that make the initiative effective. A failure to adequately document the nature of and the quality of what is implemented is particularly problematic for further policy development and service improvement, as it is difficult to learn what distinguishes successful sites from less successful sites.

Of those reviewed, few international PBIs were able to demonstrate that the outcomes measured by their evaluation were strictly a result ofthat initiative. Most evaluation reports showed an awareness of the problem of attribution. Some, such as the On Track initiative, went so far as to map the extent of overlap and prevalence of co-located initiatives within target areas. The evaluators found the co-location of PBIs to be common throughout On Track areas, with 15 or more other initiatives operating simultaneously in several On Track sites (Ghate et al., 2008). This analysis revealed that Sure Start Local Programs were operating simultaneously within the same local authority boundaries as all On Track areas, with the exception of two Welsh projects.

An example of a PBI that tackled attribution issues well is the evaluation of the New Deal for Communities initiative. As part of the evaluation, a measure was devised that allowed the 39 targeted areas to be compared with each other on a like-for-like basis, in order to examine the patterns and drivers of change. The Composite Index of Relative Change (CIRC) standardised and combined data for 36 core indicators (six for each of the initiative's six outcomes, as indicated in its theory of change) and reflected changes that might plausibly be achieved during a six-year period (2002-08). A benchmarking exercise conducted in 2001-02 allowed the extent to which change in any area targeted by New Deal for Communities was less than, or exceeded, that occurring in other similar, deprived areas. Hence, the CIRC enabled a "net" effect of the NDC program to be calculated, over and above changes occurring as a consequence of national, regional or local authority trends and, ultimately, helped to determine the extent to which the initiative had caused the change (Lawless, 2012).

A theory of change

Related to the problem of attribution is the need for a well-articulated theory of change or program logic that provides clear and explicit expectations about what would be the short-, medium- and longer term outcomes to be anticipated from the place-based initiative, prior to the intervention being implemented (Campbell & Meadows, 2001). Griggs and colleagues (2008) were critical that often this was missing, and "evaluation findings are often interpreted after the event and frequently without a detailed theory of change" (p. 6). Thomson (2008) recommended that a pre-specified theory through which change could be expected should be developed at the planning stages of a place-based initiative. Having a well-articulated theoretical model also enables rival hypotheses to be formulated that can be disconfirmed through the collection of evaluation data. If two place-based initiatives are operating in the area with two different mechanisms of action, then the pattern of results collected in the evaluation provides some guidance around the attribution of outcomes.

Almost all international PBIs articulated their theory of change. An exception was the On Track initiative - although the evaluators presented a logic model, they emphasised its construction post hoc due to the original documentation being unclear as to the initiative's theory of change (Ghate et al., 2008). Some initiatives' theories of change contained multiple objectives, or sub-components, within one or all of their causal "links". This tended to add difficulty to the evaluation's ability to successfully measure the causal pathway and thus demonstrate the theory of change in action. For example, the National Strategy for Neighbourhood Renewal sought to provide a strategic and joined-up approach to the complex problems posed by neighbourhood renewal - including enhanced and focused mainstream service delivery, real community involvement in planning for and delivering the improvement of areas, and better local, regional and national coordination - in order to reduce worklessness and crime, and improve health, education and skills, housing and the physical environment, and tackle deprivation at neighbourhood level (Amion Consulting, 2010).

The evaluations of several international PBIs successfully measured individual components of the theory of change separately; however, they commonly failed to measure, and thus demonstrate, the causal pathway between components. Such initiatives were ultimately unable to provide evidence about the extent to which specific components had an effect on, or were positively associated with, later components - or "links" - in the theory of change "chain". Accordingly, these initiatives were given a "partially met" rating.

Positively, some best practice examples of measuring a theory of change were found among the international PBIs. In each instance, the outcomes of the intervention were demonstrated to be a result of the course of action, or process(es), underpinning the theory of change. For example, the theory of change underpinning the Moving To Opportunity initiative was: to offer low-income families the chance to move to private-market housing in lower poverty neighbourhoods, in order to improve families' employment, income, education and social wellbeing outcomes (Sanbonmatsu et al., 2011). The study design for the evaluation of Moving To Opportunity followed a clear rationale that measured the theory of change as follows:

  • measuring the extent to which the experimental group (who were provided with assistance to move to less disadvantaged neighbourhoods) improved their housing/neighbourhood conditions in comparison to the control group; and, in turn
  • measuring the extent to which the experimental group experienced better outcomes in comparison to the control group in regard to health, educational, risky/criminal behaviour and other outcomes, as a result of living in less disadvantaged conditions.

Similarly, the Excellence in Cities evaluation enabled outcomes for each "link" in its theory of change sequence to be measured both in isolation and in association. The initiative's theory of change was to provide academic support and resources through a framework of cooperation and partnerships, in order to raise standards in urban schools (Kendall et al., 2005). The Excellence in Cities evaluation measured this by (a) measuring the extent to which academic partnership and collaboration was achieved; (b) measuring the change in educational attainment for pupils; and then (c) measuring the relationship between the partnership working and student outcomes (i.e., the effect of academic support partnerships on educational attainment). Partnerships were categorised into five types - ranging from being at an early stage of the partnership working to having a very well-embedded shared, collaborative approach - based on the evaluation team's subjective interpretation of the Partnerships Coordinator's reports as to the "condition" of the partnership as at mid-2003. The evaluation's single-cohort statistical model was then extended to include both level of partnership engagement and phase of Excellence in Cities in the analysis of pupils' academic attainment (Kendall et al., 2005). In this way, the evaluation was able to demonstrate a "partnership dividend" (see "Joined-up working").

Residential mobility

The issue of residential mobility has largely been overlooked in the evaluation of place-based initiatives, although there has been a strong tradition of trialling residential mobility programs to address concentrated and entrenched disadvantage in the United States (Moving to Opportunity). The major concern from an evaluation perspective is that people who could benefit most from place-based initiatives may leave the area, therefore making it difficult to fully evaluate the effectiveness of the programs (Galster, Temkin, Walker, & Sawyer, 2004). As the UK House of Commons (2003) report noted, areas of entrenched disadvantage may also have higher rates of residential mobility, which indicates that evaluations need to have the capacity not only to look at individuals within local areas, but also to follow them if they move.

It is important to note that one recent evaluation of the UK New Deal for Communities initiative did address residential mobility in their assessment of educational outcomes for students (Wilkinson & McLennan, 2010) and found that there was little evidence to suggest program effects varied by the extent of residential mobility. However, this approach has been lacking from other evaluations of PBIs.

Understanding the nature or the drivers of residential mobility has been found to be important to inform residential mobility programs aimed at addressing concentrated disadvantage. For example, the mixed findings from the Moving To Opportunity program (which randomly allocated families to receive housing vouchers that enabled them to move out of poor neighbourhoods) has largely been attributed to the fact that after a few years, many families returned to the neighbourhoods they had been living in previously (Sampson, 2012a). Other research in the US also suggests that there is an intergenerational transmission of growing up in disadvantaged neighbourhoods, with individuals whose parents grow up in disadvantaged neighbourhoods being more likely to grow up in similarly disadvantaged areas, even when their parents and their own demographic characteristics are taken into account (Sharkey, 2013).

While the majority of international PBIs reviewed did not evaluate residential mobility, there were a few exceptions. Consideration of residential mobility was essential to the Moving To Opportunity evaluation, given a primary aim of the initiative was to assist families living in public housing to move to lower poverty neighbourhoods (Hutchings et al., 2012). The National Strategy for Neighbourhood Renewal evaluation formulated a typology of deprived neighbourhoods based on the in- and out-flows of households (Amion Consulting, 2010). Residential mobility was also investigated as part of the New Deal for Communities evaluation, which found that residents in NDC intervention areas were more frequent movers than in comparison areas (12% versus 9%). Younger people (aged 16-34 years), private rented sector households, recent movers, large and single-person households, residents with higher qualifications, males and white residents were found to be the most likely to move out of intervention areas. Further, intervention areas with higher levels of mobility experienced less positive change than the NDC average in regard to housing and physical environment indicators (Beatty, Lawless, Pearson, & Wilson, 2009).

Cost-effectiveness

Evaluations of the costs and cost-effectiveness of PBIs have also been largely absent from the international PBIs (Griggs et al., 2008; O'Dwyer et al., 2007). Although it is assumed that PBIs would be more expensive, the limited available evidence does not suggest that PBIs are expensive compared to mainstream services. In one of the few instances of comparisons to other types of service offerings, it was found that the New Deal for Communities expenditure between 1999-2000 and 2007-08 was about 10% of the total cost of mainstream services in the same areas (Batty et al., 2010). Another limitation of these studies is that few have collected specific costs for different elements of the initiative (Griggs et al., 2008). These limitations mean that the costs of elements of initiatives cannot be compared to one another, and because there has not been good documentation of the precise elements of initiatives in areas, the types of returns to be expected are also not known (O'Dwyer et al., 2007). A further limitation of the lack of cost data is that different types of initiatives cannot be compared.

Studies of the cost-effectiveness of PBIs need to take a long-term view, as often the benefits to society take years and even decades to be established. For example, a recent cost-effectiveness evaluation of Sure Start (NESS, 2011) was limited to only recent benefits due to parental engagement in employment and, as Meadows (2006) noted, benefits for other positive outcomes of Sure Start will take time to be understood.

The majority of international PBIs reviewed did not evaluate cost-effectiveness (i.e., did not calculate the monetary value of benefits), but rather presented a partial or overall project account, or a per unit cost, such as "cost per child" in the case of Harlem Children's Zone (2014) or, for Moving To Opportunity, the cost to counsel each family entering the intervention's treatment group (Goering et al., 1999). Approaches among those international PBIs that did evaluate cost-effectiveness included cost-benefit analyses (Kendall et al., 2005; NNI Research Team, 2007); and multi-method approaches, such as that of the National Strategy for Neighbourhood Renewal, in which financial information from program evaluations was reviewed and econometric modelling applied (Amion Consulting, 2010; Wilkinson, Whitworth, & McLennan, 2010). Some evaluations made conclusions as to the initiative's overall value for money or financial success (Amion Consulting, 2010; Batty et al., 2010; NNI Research Team, 2007). For example, the National Strategy for Neighbourhood Renewal's evaluation estimated the value of the net reduction in worklessness in deprived areas associated with the initiative as £1.6 billion, which represented over five times the direct cost (£312 million) of National Renewal Fund interventions specifically focused on worklessness, and about two-thirds of the total cost of the National Renewal Fund (£2.4 billion) (Amion Consulting, 2010).

Other evaluations found economic benefits at a particular age, or stage, of child development. Analysis of the cost-effectiveness of Sure Start Local Programs found that by the time children reached the age of five, SSLPs had delivered economic benefits of between £279 and £557 per eligible child (NESS, 2011). The Excellence in Cities initiative was found to be potentially cost-effective (in terms of the long-term wage return to individuals) when pupils were aged 14 (in Key Stage 3). In the early years of the initiative, the benefit of the policy was zero, and hence the rate of return was zero, and pupil costs outweighed pupil benefits. However, after two years in Phase 1 and Phase 3 areas, and after three years in Phase 2 areas, the policy was generating a positive return (Kendall et al., 2005). The Sure Start Local Program thus highlights that benefits of early childhood interventions typically do not emerge until at least 15 years after the intervention begins (NESS, 2011). Such cases emphasise the slow emergence of economic benefits and posit that future economic benefits may be applicable.

3.5 Summary: Common elements of place-based initiatives

In this section we presented examples from the international PBI literature of common elements in the design, delivery, implementation and evaluation of PBIs. We also presented examples from PBIs whose evaluations sought to evidence the achievement of objectives relating to these common elements. An additional four case studies of international PBIs are provided in Appendix C to demonstrate the close review process in greater detail. A description of the program, its evaluation design and program outcomes are also included in the case studies. The case studies feature four of the more prominent and well-evaluated PBIs that were included in our review. In choosing international initiatives to use as case studies, we selected three Type 5 initiatives and one Type 2 initiative, whose themes of disadvantage were pertinent to the Australian policy context, but were widespread internationally. These themes of disadvantage included: child development, family wellbeing, health and education (Sure Start in England, and the Harlem Children's Zones and Promise Neighborhoods in the US); employment (Employment Zones in the UK); and urban regeneration and sustainability (URBAN I Community Initiative in the EU). Please note that a supplementary list of references indicated using superscript letters in the program descriptions and tables is included in section 7.2.

We found that the 12 international PBIs closely reviewed, including the four case studies, displayed inconsistency in their evaluation of common practice elements. This is shown clearly in Table 1, in which a summary of the percentage of the 12 international PBIs that met each of the common elements is displayed. The "Demonstrates element" and "Evaluates element" columns both represent the proportion of the 12 initiatives that met the common elements. The proportion of international PBIs that met in full (i.e., scored "Yes") against the evaluation of any common element was low.

That PBIs do not tend to evaluate the elements theoretically underpinning them suggests that the common elements may be tacit - perhaps part of a place-based "ideology". The inconsistencies in evaluation also undermine the ability to link these common elements to program outcomes. As such, it is difficult to say exactly whether, and how, these common elements are associated with the program outcomes of interest.

Table 1: Summary of the extent to which the 12 investigated international place-based initiatives met the common elements
Common elements Demonstrates element (%) Evaluates element (%)
Note: * One initiative, the Harlem Children's Zone, was deemed to be not applicable to "Governance", as it was only implemented in one location. Hence, this element includes ratings from 11, rather than 12, place-based initiatives.
For the PBI Yes Partially met Yes Partially met
Spatial targeting 83 8 25 8
Social targeting 100 0 25 17
Flexible delivery 67 25 8 50
Local autonomy 58 25 17 33
Joined-up working 83 17 33 42
Governance * 64 36 25 25
Capacity development 50 17 42 8
Lead times 25 25 0 42
Long-term focus 50 25 25 33
For the evaluation Yes Partially met

Not applicable

Causality 67 17
Attribution 17 42
Theory of change - Articulated 92 8
Theory of change - Measured 17 50
Residential mobility 25 0
Cost-effectiveness 42 0

* Harlem Children's Zone was not deemed to be spatially targeted as it was limited to Harlem, a single neighbourhood of New York City. However, the initiative was included for review due to both its reported influence on education reform around the world and its subsequent replication as part of the Promise Neighbourhoods initiative, a program currently being implemented in more than 20 areas across the US (Whitehurst & Croft, 2010).

One exception was the Moving to Opportunity program, a housing mobility program that randomly assigned families to receive a voucher to live in a non-poor area (Sanbonmatsu et al., 2011); but see Sampson (2012b) for a discussion of the limitations of randomisation in this context.

4. Commonwealth place-based initiatives review

4.1 Process for obtaining and reviewing relevant literature

Figure 1 illustrates steps undertaken in the collection and review of Commonwealth PBI materials. The process involved: sending out and receiving Requests for Information (RFIs) and evaluation documentation from Commonwealth departments; further consultations with departments; receipt and classification of PBIs, including the basis for including/not including PBIs in the review; and applying the methodology for reviewing the selected PBIs.

Figure 1: Methodology employed for the collection and review of Commonwealth PBI materials

 Figure 1: Methodology employed for the collection and review of Commonwealth PBI materials

As can be seen in Figure 1, the process for determining which Commonwealth PBIs would be included in the review involved classifying the PBIs according to Griggs et al.'s (2008) typology, based on their policy objectives and whom they were targeting (i.e., "place" and/or "person"). This was the same process that was applied for the international PBIs, as specified in Appendix A. As per the international PBI review, it was deemed that in order to be included in the Commonwealth review, PBIs were required to demonstrate some consideration of both place and person. All of the Commonwealth PBIs included in this chapter were classified as Type 5 policies; that is, those demonstrating a simultaneous focus on place and person in order to impact both.

4.2 Illustrative case studies: Commonwealth place-based initiatives

This section includes a detailed description of five of the nine PBIs that were included in our review, which will serve as case studies for highlighting the extent to which the common elements were met, both in their delivery and for their evaluation (as outlined in Chapter 3). As per the international case studies (see Appendix A), a description of the program, and its evaluation design and outcomes, are also included. Please note that the supplementary reference list for the references indicated in the following program descriptions and tables is included in section 7.3. For details of the PBIs not discussed in this chapter, see Appendix E.

Communities for Children (CfC)

Department: Social Services (DSS)

(See Table 2 and Table 3)

Program description: CfC is an initiative that provides targeted services to approximately 50 disadvantaged communities across Australia. At each site, a non-government organisation (NGO) Facilitating Partner delivers the services, which are intended to benefit children and their parents, as well as families and communities as a whole. The broad objective of the CfC initiative was to implement three new service delivery innovations for young children and their families living in areas of disadvantage, including: (a) a greater number of services; (b) better coordination of services; and (c) a focus on improving community "child-friendliness".1 *

Evaluation design: The CfC evaluation was conducted within two phases of the Stronger Families in Australia (SFIA) study. The first phase ran from 2006 to 2008 (three waves of face-to-face interviews), with the second phase running in 2011-12 (two waves - Waves 4 and 5 - of telephone interviews). The overall objective of the CfC evaluation was to measure if, and the extent to which, CfC benefited preschool and school-aged children and their families living in CfC sites, in comparison to those living in socio-economically and geographically similar contrast sites, which did not offer CfC.1 The CfC evaluation not only involved the SFIA study, but also an outcome indicators framework, community profiles, service coordination and partnership model studies, a cost-benefit analysis and a progress report review.2

Program outcomes: SFIA Phase 1 found that CfC had small but positive effects on a number of outcomes for families, children and communities, in regard to parenting practices, parental employment, community involvement and children's early learning, although no effects were found for family health.2 In SFIA Phase 2, however, the positive, significant findings detected in Phase 1 were no longer present, with the children and families in the contrast sites appearing to "catch up" to those in the CfC sites after the study children started school.1

Table 2: Did the CfC initiative demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Demonstrated Evaluated
  • The Facilitating Partner model made flexible service delivery possible, including community involvement.2
  • Service delivery and discretionary funding flexibility was reportedly effective.2
  • There was tension between flexibility/capacity to evolve and establishing clear governance frameworks; relationship building between the government, lead agencies and local stakeholders was critical.2
Local autonomy Demonstrated Evaluated
  • Whole-of-community approach to early childhood development was adopted, in consultation with local stakeholders, who actively participated in committees.2
  • CfC committees were generally effective, but some disengagement did occur, and in a few sites, operated as advisory groups with little influence on decisions.2
  • Consultation time periods were key; in some complex and/or remote communities they were not adequate, which resulted in a higher project management focus.2
Capacity development Demonstrated Evaluated
  • Strategic objective to not only provide new/more services, but to improve coordination between them.2
  • The Community Partners who were responsible for delivering the services in each site were critical; they also gave smaller NGOs the opportunity to build their capacity and develop skills.2
  • Shared training and mentoring/support provided by the Facilitating Partners were also effective.2
Lead times Not demonstrated Not evaluated
  • Lead times were not substantial,3 and while not formally evaluated, the program would have likely benefited from longer consultation/partnership-building periods prior to implementation.
Long-term focus Demonstrated Partially evaluated
  • Most of the sites were established in 2006, with more established in 2009; all are still in operation.1
  • Longitudinal design of SFIA partly allowed the benefits of this long-term investment to be evaluated.
Spatial targeting Demonstrated Not evaluated
  • The CfC sites (approx. 50) targeted disadvantaged areas and were located across all states and territories, in both urban and rural areas, with disadvantage determined through the analysis of Australian Bureau of Statistics data, particularly the Socio-Economic Indexes for Areas (SEIFA).1,2
  • The government-defined CfC boundaries led to some issues in coordinating agencies/delivering services.2
Social targeting Demonstrated Evaluated
  • A primary aim of CfC was to improve outcomes for disadvantaged families in disadvantaged areas.1
  • The outcome indicator framework showed all CfC sites were disadvantaged on the relevant indicators.2
  • Hard-to-reach and the most socio-economically disadvantaged families/children were just as, or even more, likely to benefit from CfC as others residing in the community.2
Joined-up working Partially demonstrated Not evaluated
  • Some focus was placed on integration/coordination between the government, NGOs and local service providers through the Community Partner model,2 but this was not formally evaluated.
Governance Demonstrated Not evaluated
  • Governance arrangements were fairly clear: state/ territory officers from the then Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) were the primary contact for the Facilitating Partners, acting as "contract managers".2
  • Some anecdotal evidence suggests these relationships were generally positive,2 but governance as a whole was not formally evaluated.
Table 3: Did the CfC evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Demonstrated
  • Various statistical techniques were used in SFIA Phase 1 and 2 to explore patterns of change/stability across outcome measures in CfC and contrast sites from Waves 1-5, including cross-sectional regression analyses and difference-in-difference (DID) modeling.1
  • This approach enabled detection of effects over time for those residing in CfC sites, compared to contrast sites.1
Attribution Demonstrated
  • DID modelling enabled differences between CfC and contrast sites to be attributed to CfC, by controlling for other factors.2
  • The evaluation could not link outcomes with exposure to CfC services; the CfC model was based on service accessibility, rather than direct usage.2
Theory of change - Articulated Demonstrated
  • The clearly articulated theory of change was to "provide child- and family-focused services in disadvantaged communities throughout Australia, in order to enhance family functioning, child development and the community context in which children grow up".1
Theory of change - Measured Partially demonstrated
  • Although the SFIA study did investigate the extent to which child, family and community outcomes changed for those residing in CfC areas, it did not directly measure the extent to which CfC service usage was associated with these outcomes.1
Residential mobility Demonstrated
  • In general, patterns of residential mobility were similar for families in the CfC and contrast sites and reflective of the general population.1
Cost-effectiveness Demonstrated
  • As at 2010, total financial benefits associated with improvements in outcomes attributable to CfC were 4.77 times greater than the cost of funding the initiative (i.e., a benefit-cost ratio of 4.77, or a 377% return on investment for the CfC program.4
  • A similar cost-benefit analysis was not conducted, however, for SFIA Phase 2.

Cape York Welfare Reform (CYWR) trial

Departments: FaHCSIA and PM&C

(See Table 4 and Table 5)

Program description: The CYWR trial commenced in July 2008 under a tripartite agreement between the Australian and Queensland Governments and the Cape York Institute for Policy and Leadership (CYI). The primary aim of the trial was to restore positive social norms, while other objectives included re-establishing a local Indigenous authority, enabling children to achieve their full potential, supporting economic engagement, and moving individuals and their families from welfare housing to home ownership. The first key component of the CYWR trial was the Family Responsibilities Commission (FRC), a statutory body established under the Family Responsibilities Commission Act 2008, which expired on 1 January 2012. The FRC attached behavioural obligations to the receipt of welfare payments, and provided a range of community services and educational, economic development, employment and housing initiatives to FRC clients (e.g., wellbeing centres, school attendance case managers, parenting programs, and anti-violence, drug and alcohol services).5

Evaluation design: The evaluation framework and program theory was published by Courage Partners in 2009,6 and included the following:

  • a set of principles that could be used to guide the conduct of the evaluation;
  • four key strategic evaluation questions - Was the trial implemented as agreed by the three parties? Are social norms and behaviours changing? Has service provision changed in a way that supports the change of social norms and behaviours? Have governance arrangements supported changes in service provision and social norms and behaviours?; and
  • five key evaluation strategy components encompassing the priority needs identified by the trial partners - intelligence gathering to improve the implementation processes, an implementation review of the FRC, a progress evaluation, an outcome evaluation, and special/case studies that would provide more depth in areas where more detailed research was required to respond to the evaluation questions.6

Hence, qualitative and quantitative approaches both figured prominently in the final evaluation report.7 The trial was conducted in four communities (Aurukun, Coen, Hope Vale and Mossman Gorge), with the evaluation period beginning in 2008 and concluding at the end of 2011.6

Program outcomes: The key outcomes for the trial were assessed under the four streams of: Social Responsibility, Economic Opportunity, Education and Housing. While there were some variations between the four trial communities, overall, findings suggested that the wellbeing of residents improved over the course of the trial. Crime rates were lower, infrastructure and services improved and school attendance had either increased or remained at a high level. Nevertheless, progress over the short few years of the trial was tentative, and considerable challenges remained in all of these communities. Furthermore, while significant progress in implementation was made in relation to the Social Responsibility and Education streams, progress was slower for the implementation of projects under the Housing and Economic Opportunity streams (e.g., it took considerable time to address barriers to home ownership, and planned business precincts did not open until the end of the trial in two of the sites).8

Table 4: Did the CYWR trial initiative demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Not demonstrated Not evaluated
  • Much of the trial was mandated around its welfare reform objectives, and each community was required to deliver the 15 programs included in the trial (i.e., the program was designed and delivered as a "package").9
  • Differences/varying issues across each community meant that, in reality, implementation was not uniform, and some adaptation was required during the trial.9
  • Hence, while the trial was inevitably delivered differently in each community, this flexibility did not appear to be "built in" to the program design as such.9
Local autonomy Partially demonstrated Partially evaluated
  • Community involvement, including consultations with residents and local representatives was important, but needed to be balanced with limiting intrusion to local community members.6
  • Some issues were identified, with the communities feeling they were not kept fully informed about the trial, and there was also tension with the extent to which the communities had authority. While they had little control over the services/programs that were delivered, it was acknowledged that "Indigenous authority implies that the community would ultimately control what happens in their jurisdiction" (p. 117).9
Capacity development Not demonstrated Not evaluated
  • In the evaluation framework report, a study of service effectiveness was proposed, investigating the key trial objective of empowering individuals and communities to make the shift from a service dependence model.6
  • This study was not cited in the evaluation report, however, and it is unknown if it was undertaken.
  • The importance of training was identified in the FRC implementation review, with activities proposed, such as co-training between the FRC and service providers, and providing ongoing training to Local Commissioners.10 Whether such training was delivered is unclear; it was not referred to in the evaluation report.
Lead times Demonstrated Evaluated
  • There was a 15-18 month engagement process that began in 2006 and accompanied the design phase.11
  • During the process of implementation, it was also proposed that the evaluation strategy would provide "evaluative feedback", with information collected in the early stages of the trial about what could be improved or changed.9 These were subsequently reported on in the FRC implementation review.10
Long-term focus Demonstrated Not evaluated
  • The trial has run since 2008; the evaluation period concluded in 2011, with funding extended until the end of 2013, and then subsequently further extended.12
Spatial targeting Partially demonstrated Not evaluated
  • The trial has been conducted in four Indigenous communities in the Cape York region of far north Queensland. The geographical scope of the trial was relatively large, but the overall target population was small, with a total of just under 3,000 people residing across the four sites.13
  • There was wide variation in the size of the sites' populations, with Aurukun having the largest, at 1,449, and Mossman Gorge having the smallest (103).13
Social targeting Partially demonstrated Not evaluated
  • The four participating communities were chosen for the trial due to expressing early interest in welfare reform; the starting point for the reform agenda was the concept that social problems in Cape York were "caused largely by a social norms deficit" (p. 67).11
Joined-up working Not demonstrated Not evaluated
  • The evaluation framework stipulated that "whole of government work is partnership work", with the intent being to "bring together the relevant agencies so that resources can be arranged creatively and cooperatively to address the problems at hand" (pp. 33-34).6 This "partnership approach" was proposed to be a key factor influencing trial outcomes and it was argued that it should be evaluated as such.6
  • It is unknown if this approach was delivered, as it was not mentioned in the evaluation report.
  • In regards to the extent that "joined-up working" was achieved, the integration and cooperation between the Australian and Queensland Governments and the Cape York Research Institute was not always optimal;9 this element was not formally evaluated, however.
Governance Partially demonstrated Partially evaluated
  • Governance arrangements were complex, involving the Australian and Queensland Governments, in addition to a number of government and non-government agencies, including the Cape York Research Institute.6
  • A key question posed in the evaluation framework was how the trial's governance arrangements would contribute to outcomes; this included examining key policy, administrative and structural barriers that could influence the trial's success.6
  • While this question was not specifically addressed in the evaluation report, it was acknowledged that the "unique" governance arrangements inevitably created tensions and challenges for the partners, which caused some delays in the implementation of the trial.9
  • Furthermore, many of the governance arrangements set out in the initial agreement and the Welfare Reform Action Plan failed to materialise or were abandoned, although this was concluded as "likely to be a result of over planning rather than under delivery" (p. 94).9
Table 5: Did the CYWR trial evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Partially demonstrated
  • The evaluation framework acknowledged that empirical causal attribution would be difficult to establish, given the trial's complexity and the number of individual projects (15) it involved.6
  • Further, measuring the ultimate aim of the trial - rebuilding positive social norms - would most likely not occur within the evaluation time frame and, even if so, could not yet be shown to be sustainable.6
  • Interdependencies and possible causal pathways between the strategies/potential effects of the major elements of the trial were considered at a strategic level,6 although the evaluation mainly focused on investigating the combined effect of the trial as a whole.8
  • Despite these restrictions, the evaluation report did attempt to examine causality in two cases, when investigating trends in alcohol-related assault and school attendance rates;14 hence, while causality was not measured at an overall PBI level, attempts were made to measure it in these instances.
Attribution Partially demonstrated
  • The evaluation framework proposed that it was impractical to use comparison groups in evaluating the counterfactual.6 Comparison Indigenous communities were chosen for the evaluation, however, and compared to the four sites across outcome measures.14
  • Administrative data relating to school attendance, crime, child safety, housing and employment were used, with statistical analyses investigating the average change in key outcome variables in the trial and comparison communities over time; this enabled a "best estimate" of whether changes in the trial sites were part of an overall trend or specific to these communities.14
  • A limitation of this approach was that it did not enable attribution to any specific factor as driving changes in the affected population.14
Theory of change - Articulated Demonstrated
  • The theory of change was articulated in the evaluation framework and reiterated in the evaluation report, and proposed "a continuum [for social change] from putting in place foundations and enablers, to bringing about short- to medium-term behaviour change, and finally achieving sustainable improvements in the communities in the longer term" (p. 2).15
  • While this was the broad theory of change, there were others that applied to various features of the trial, which was attributed to the complexity of the initiative. The theory of change was also broken down to a strategic level, with "project-level logic maps" also devised.11
Theory of change - Measured Partially demonstrated
  • The project-level logic maps for each of the 15 individual programs that were part of the CYWR trial specified measurable indicators, which enabled program outcomes to be assessed.11
  • The extent to which the theory of change could be measured as a whole was limited, however, due to its complexity and challenges associated with devising tangible "whole-of-strategy" indicators.
Residential mobility Partially demonstrated
  • Not a focus of the evaluation, but the STEP Mobility Project, which was implemented by Mission Australia and ceased at the end of 2009, aimed to support people from the trial communities who were offered job opportunities in other regions; 50 participants completed training in the project and 33 gained employment outside of Cape York.16
Cost-effectiveness Not demonstrated
  • Economic evaluation of the CYWR trial was recommended in the evaluation framework report,9 which referenced the From Hand Out to Hand Up report. 17
  • Cost-effectiveness was not investigated, however, in the overall evaluation.15

Support for Day to Day Living in the Community (D2DL)

Departments: Health and Ageing (DoHA), then Health (DoH)

(See Table 6 and Table 7)

Program description: The D2DL program was designed to increase access to recovery-oriented services for people with a severe and persistent mental illness, who were also experiencing isolation and limited prospects of joining the labour force. It was anticipated that the D2DL program would deliver approximately 7,000 additional "places" (with a place being defined as 100 hours of service delivery) in new and existing structured day activity programs over 2007-09 (which was in addition to the 15,000 places that were already available nationally). The aim was to increase the participants' ability to participate in social, recreational and educational activities, improve their independence, and increase their community participation. There were three service delivery components of the program, including:

  • drop-in places/low-level support - offering a safe place for those recovering from a mental illness to visit, access support and be referred to other relevant services, with various social activities and discussion/support groups also provided;
  • medium level support - assisting participants in skill development and in accomplishing goals that improved their functioning in the community, which involved developing an individual management plan, using various assessment tools, and providing more structured support and skills-based training; and
  • flexible/discretionary funding - supporting clients' participation in community activities, education and other activities consistent with achieving their rehabilitation goals.18

Evaluation design: The evaluation of the D2DL program was conducted by Healthcare Planning and Evaluation (HPE),18 over two years from July 2008 through to June 2010. The program logic schema that was developed by HPE in consultation with DoHA representatives was used to prepare an evaluation framework for the D2DL program, which identified the evaluation areas to be addressed, the key evaluation questions to be answered in each area, the key sources of qualitative and quantitative data required to address each of the evaluation questions, the data collection processes, and the analyses to be undertaken. A sample of 25 D2DL sites were visited in late 2008 and late 2009, with activity data extrapolated from progress reports from these sites. Over 200 program participants were also interviewed, and an Internet-based survey was also promoted for participants in late 2009, with stakeholders, including state and territory mental health services, consulted. The final step in the evaluation methodology was a series of stakeholder workshops conducted in June 2010.18

Program outcomes: The D2DL evaluation found that the program had largely achieved its objectives, although the extent to which it affected labour market participation among participants was not reported in detail. Activity data from the 25 sampled sites indicated that over 7,000 places had been created each year, with service provider/consumer discussions and a review of progress reports indicating that the intended target group had been reached. It was also found that the use of discretionary funding facilitated the engagement of mainstream organisations and services to provide support for D2DL clients (although this was reduced half-way through the evaluation period), which reportedly resulted in communities becoming more receptive towards people with severe and persistent mental illnesses. While the development of structured assessment, planning and review processes within D2DL services had reportedly progressed, it was believed that the emphasis on the hours of activity reduced its capacity to provide tailored services to participating individuals. Furthermore, it appeared that people from culturally and linguistically diverse backgrounds and Aboriginal and Torres Strait Islander people were under-represented at most sites, indicating that increased efforts to promote the services to these individuals may be required.18

Table 6: Did the D2DL program demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Demonstrated Partially evaluated
  • There was, at least initially, flexibility in the program delivery, particularly in regard to how resources could be accessed to support the development and operation of services (discretionary funding was halved one year in to the two-year evaluation period).18
  • The use of discretionary funds reportedly increased the options available to service users, although it was also indicated that some organisations found it difficult to accept this level of discretion.18
  • There was reportedly some flexibility in the program guidelines for organisations to tailor services to their community's needs, although organisations felt that this narrowed as the program progressed, perhaps due to the program maturing.18
Local autonomy Not demonstrated Not evaluated
  • Community connection and support were specified as being important factors in the successful establishment of the D2DL program,18 but these concepts were not further expanded on or addressed.
  • DoHA personnel were reportedly readily available to D2DL organisations and personnel in the initial site visits.18
Capacity development Partially demonstrated Partially evaluated
  • The service delivery objectives were targeted at improving the capacity of NGOs to effectively assess and meet client needs, and to engage other organisations and service providers in the community.18
  • The program was reportedly relatively effective in achieving these objectives, but the need to increase training for NGOs and provide more opportunities for knowledge exchange was identified.18
  • The evaluation noted that participation in D2DL had a significant effect on smaller organisations, expanding their resource base and, at some sites, contributing to major organisational change, including new personnel, management and governance systems.
  • The evaluation found there was no provision of, or support to develop, information systems for facilitating data collection, collation and presentation; service providers were required to develop their own.18
Lead times Partially demonstrated Not evaluated
  • The lead times were relatively short for the D2DL program, although these were not specified.18
  • Although DoHA reportedly provided sufficient time for the participating organisations to plan and prepare for implementation and confirm key aspects of the program, there were two sites where no developmental period was available.18
Long-term focus Demonstrated Not evaluated
  • The D2DL program began in 2007 and is still in operation (as of late 2012); the evaluation period, however, was only two years, from July 2008 to June 2010, and therefore did not align with the program's longer term nature.18
Spatial targeting Partially demonstrated Not evaluated
  • A total of 60 sites nationally received funding under the program, with urban/regional (not remote) areas the focus; 25 sites were selected for the evaluation.18
  • Sites were selected in consultation with state/territory governments and the Council of Australian Governments (COAG) State and Territory Mental Health Working Groups.18
  • The evaluation report did not specifically outline the selection criteria for the D2DL sites, however, or why these particular areas were chosen.
Social targeting Partially demonstrated Partially evaluated
  • The target of the program was to establish 7,000 additional "places" (i.e., 100 contact hours) to assist people who were experiencing severe and persistent mental illness in various locations around Australia.18
  • Based on discussions with service providers and program participants, the evaluation report indicated that the intended target group had been reached;23 the mechanism/criteria for reaching this conclusion was not made explicit.
Joined-up working Demonstrated Evaluated
  • Strong links were developed with public mental health service providers, with around 30% of D2DL referrals made by public mental health services.18
  • There was wide variation across the sites, however, and a number of recommendations were made about how stronger links could be formed, including gaining support and leadership from senior personnel in both services to work collaboratively.18
  • A lack of streamlined processes across the sites was identified, with information exchange not optimal; this occurred to a greater extent, however, between service delivery personnel within the sites.18
  • As indicated in the "Governance" element, DoHA worked effectively with the state and territory governments.18
Governance Partially demonstrated Not evaluated
  • The D2DL program was funded by DoHA under the COAG National Action Plan on Mental Health.18
  • Collaboration with state/territory governments was a critical component in the success of the program, and the jurisdictions reported liaising with DoHA on a regular basis during the planning and implementation phases of D2DL.18
  • No in-depth information or evaluation was provided, however, regarding governance mechanisms.
Table 7: Did the D2DL program evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Not demonstrated
  • Measuring the effects of the program was problematic, given that there was not a defined assessment tool for D2DL, and while difficult to calculate, it appeared that only 30% of clients underwent a functional assessment.18
  • Much of the data collected relating to the potential benefits of D2DL services, including participant engagement, was qualitative.18
  • Rather than establishing causality, and investigating factors such as labour force participation, the focus of the evaluation was more on the amount/types of services offered, participation in these services, and clients'/key stakeholders' anecdotal views of these services and their effectiveness.18
Attribution Not demonstrated
  • Attribution was difficult, given that there was a range of other programs at the national and state levels that also targeted the needs of people with long-term mental illness (e.g., the Personal Helpers and Mentors Service; PHaMS). These other programs were beneficial, however, from the perspective of increasing access to important services, and were somewhat complementary in their approaches.18
Theory of change - Articulated Demonstrated
  • The objective of the program was to increase access to recovery-oriented services for people experiencing severe and persistent mental illness and isolation, in order to assist these individuals in developing new or relearning old skills, developing social networks, participating in community activities, accomplishing personal goals, developing confidence and, ultimately, improving their quality of life.18
Theory of change - Measured Not demonstrated
  • The capacity of the evaluation to objectively measure this theory of change and most of its intended outcomes was limited.18
Residential mobility Not demonstrated
  • This was not a focus on the D2DL program evaluation. More so, the emphasis was on helping participants to become active citizens within the community of their current residence.18
Cost-effectiveness Not demonstrated
  • Not investigated.

Centrelink Place Based Services (PBS) initiative

Department: Human Services (DHS)

(See Table 8 and Table 9)

Program description: The PBS initiative was designed by Centrelink to improve outcomes for participants through the use of collaborative local approaches that addressed issues pertinent to both "place" and marginalised groups. Another objective of the program was for Centrelink to use innovative, locally based approaches to improve the capability of the service delivery system, with the program consisting of six linked, but local, "action research" projects.19 The initiative aimed to improve outcomes for its participants by applying a new approach to developing and implementing policy and programs that focused on collaboration and placing the customer at the centre of the service delivery system.20

Evaluation design: The PBS evaluation involved three phases: (a) a desk-based assessment of Planning Office and Initiative documentation, including action research reports, monthly reports, case studies and metric data; (b) field visits to each of the six action research project sites and to the Centrelink central office, which involved approximately 160 interviews with local and central project personnel, senior executive and customers; and (c) a review of the program as a whole, based on the data collected in the earlier phases of the program.21 The evaluation did not attempt to make definitive judgements about the extent to which the program's innovative practices were effective in achieving positive outcomes for Centrelink customers, but rather, focused on the testing of assumptions and processes and on capturing organisational learning.21

Program outcomes: The evaluation of the PBS initiative found that the programs generally emphasised individual casework more strongly than community-level intervention, which was also partly attributable to the difficultly in measuring community development outcomes. Therefore, while the PBS projects all demonstrated some level of effectiveness in connecting or reconnecting very disadvantaged customers to the services they needed, and, in some cases, directly addressed specific issues such as life skills, literacy and feelings of social isolation or hopelessness, the initiative was limited in its capacity to address the effects of more systemic or structural disadvantage,21 which was largely attributable to being a small pilot program that was undertaken by a single government agency (Centrelink).19

Table 8: Did the PBS initiative demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Demonstrated Not evaluated
  • The six projects that comprised the PBS initiative were linked by a common planning and evaluation framework, but were designed to operate as discrete, self-managing local initiatives,20 with local managers and customer service staff and/or social workers given relatively free rein to work with other local stakeholders in tailoring and delivering project-specific services.21
  • Hence, each project was encouraged to adopt an innovative approach towards "local level problem definition and response" that was specific to the marginalised group it was targeting, by using the knowledge, understanding and experience of specialised local service delivery practitioners.19
  • It was reported that by adopting more flexible approaches that could better take into consideration the circumstances and needs of particular groups, the projects were able to achieve some success in assisting customers to engage more effectively with the service system.21
  • While flexibility was "built in" to the program design, the extent to which it was formally evaluated and could be linked to outcomes was limited.
Local autonomy Demonstrated Not evaluated
  • Service delivery partners in each of the project communities were directly involved in shaping the program logic and action research framework.19
  • The program logic and action research framework also provided the basis for local-level innovation through collaboration, with the intention of using the knowledge, understanding and experience of local service delivery practitioners to refine their problem-solving approaches.19
Capacity development Demonstrated Not evaluated
  • Although a detailed "Guide to Planning and Evaluation for Place Based Service Initiatives" was developed by the Planning Office and used to train local teams, the evaluation found that team members frequently engaged in more complex forms of customer support and intervention than they were formally trained for, and which were included in their conventional duty statements; hence, this raised issues concerning training and support and also professional liability.21
  • None of the programs sought to recruit new staff with specialist skills, however, and only one sought external training for team members, with half (3) providing varying amounts of inhouse training.21
  • While the project staff reportedly built-up considerable knowledge and experience of working with specific target groups in their area, improved practices through the action research process, and, in some cases, transferred knowledge and skills to other Centrelink staff/partner organisations, it was apparent that training should have been given a greater emphasis in the program design.19
Lead times Demonstrated Not evaluated
  • Lead times were not specified, although they appear to have been minimal or non-existent for some projects, which may be partly attributable to the fact that they were essentially pilots.19
  • The longest planning period appears to have been for the first project to be launched, while the five other projects either had no planning/implementation period, or limited planning within, for example, Centrelink's Multicultural Services Branch.19
Long-term focus Demonstrated Not evaluated
  • The first project to be launched as part of the PBS initiative had a planned duration period of three years,19 whereas the durations of the other projects were not specified.
  • While the long-term nature of the projects was unclear, this was unlikely to have been a priority, given that they were operating as pilots.
Spatial targeting Demonstrated Not evaluated
  • The six projects that received funding as part of the PBS initiative were located in specific areas throughout Australia, including Peachey Belt (SA), Logan City Council (Qld), Darwin (NT), Broadmeadows (Vic.), and Fairfield and Cooma (NSW).20
  • These local projects were chosen via a process where Centrelink Network Managers were invited to express their interest in sponsoring program initiatives.20 Fifty draft proposals were received, with the six projects being chosen after the development of "business cases" and "program logic" (no further information was provided regarding this process).19
  • Given that the projects were focused on specific groups of people in specific areas, the number of participants for each was relatively small, ranging from a maximum of 126, to a minimum of 22.21
Social targeting Demonstrated Partially evaluated
  • The six projects focused on different marginalised groups, including youth, single parents, unemployed men, people experiencing domestic/family violence, young people leaving state care and/or people with unmet mental health needs, disengaged Indigenous urban homeless people, young refugee job seekers and young carers.20
  • The premise of the program was to develop customer-centric service responses that more effectively targeted particular forms of disadvantage or exclusion in specific communities.19
Joined-up working Demonstrated Evaluated
  • The Social Inclusion Statement, upon which the PBS initiative was based, emphasised the requirement for strong partnerships to be formed between all levels of government, business and community organisations/ stakeholders.20
  • It was believed that fostering the development of strategic partnerships would assist in improving outcomes for marginalised groups within the community,19 with collaboration and relationship building with local service providers a key element of the initiative.19
  • A specific example of where this element was effectively applied was the capacity for Customer Service Advisors to develop face-to-face relationships with service providers in other agencies and contact them directly to discuss particular cases or make referrals, which was a key departure from conventional Centrelink practice.21
  • There were some issues that arose, however, in regard to building relationships with other agencies. Thorough consultation and planning with local service delivery networks was lacking, which resulted in responses ranging from misunderstanding the project's aims, through to suspicion and, in one case, a refusal to cooperate.21
  • At the project team level, it was found that "regular de-briefing meetings were central to team building, skills sharing and the teams' ability to discuss, learn from and respond to issues arising in its work" (p. 48).19
Governance Demonstrated Partially evaluated
  • Centrelink's National Planning Office had overall responsibility for the initiative, with advice from a Steering Committee.19
  • The sponsor for each initiative (who had business responsibility and accountability for the project objectives) was also the area manager for most projects, which enabled higher level support and resources to be provided.19
  • The initiative managers were responsible for the projects at an operational level and, in some cases, were the team leaders. When this was not the case, reporting and collaboration were reportedly more cumbersome.19
  • The reference groups, which were designed to assist the project teams in acquiring resources, support and intelligence, and the action research groups, which were responsible for the design, delivery and review of the action research plan and evaluation logic, had some success. It was generally found that multiple levels of hierarchy hindered the responsiveness and flexibility of these groups, but small teams assisted in carrying out "genuine action research".19
Table 9: Did the PBS evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Not demonstrated
  • The planning and evaluation guide did not specify outcome measures or data collection techniques, with local teams being largely left to develop their own data collection and management tools.21
  • Therefore, there were inconsistencies in data quality between the projects, standardised regular reports were often described as lacking richness and explanatory detail, and the capacity to draw overall findings regarding the outcomes of the PBS initiative was limited.21
  • The potential for deriving measures and indicators from Centrelink's standard database was reportedly discussed, but was not put in place.21
Attribution Not demonstrated
  • For similar reasons identified above in "Establishing causality", attribution was problematic for the PBS evaluation, which was further confounded by the absence of baseline data.21
  • It must also be noted that the evaluation took place as little as 5 months (or a maximum of 12 months) following the implementation of the projects,19 and hence its capacity to measure outcomes within this space of time, let alone establish causality, or specifically attribute them to the initiative was very limited.
Theory of change - Articulated Not demonstrated
  • The theory of change for the PBS initiative was to determine the extent to which local approaches, which identified issues pertinent to "place" and marginalised groups, could improve outcomes for participants. A further objective was to test Centrelink's ability to use collaborative/locally based approaches to improve the capability of the service delivery system.21
Theory of change - Measured Not demonstrated
  • Due to the lack of systematic data collection processes across the PBS projects, the extent to which local approaches improved outcomes for groups of Centrelink customers could not be investigated.
Residential mobility Not demonstrated
  • Not investigated.
Cost-effectiveness Partially demonstrated
  • The Planning Office and Steering Committee were reportedly "concerned that innovative practices should be able to demonstrate and measure outcomes in a way that allowed comparison with existing approaches and an assessment of "value for money" (p. 12).21
  • The lack of consistent, quality data collected, however, did not enable cost-effectiveness to be calculated, with the "intangible and non-economic character of the benefits to socially excluded customers" also identified as a barrier (p. 3).19

Building Australia's Future Workforce (BAFW) package

Department: Education, Employment and Workplace Relations (DEEWR), then Employment (DoEm)

(See Table 10 and Table 11)

Program description: The Building Australia's Future Workforce package was included in the 2011-12 Budget, and was designed to "build an educated and skilled workforce and provide opportunities for all Australians to experience the benefits of work" (p. 4).22 The BAFW package included measures from the former DEEWR (now separated into the Departments of Education and of Employment), the DHS, and the former FaHCSIA (now DSS), with a focus on promoting long-term economic participation in ten of Australia's most disadvantaged communities.22,23

At a strategic level, the BAFW package was conceptualised as having three inter-related streams - "skills", "workforce participation" and "place-based" streams - each with its own overarching objective. In total, the package contained 39 measures, with the place-based stream (which is also known as Better Futures, Local Solutions, or BFLS) containing nine measures. The overall objective of BFLS was to provide new services, opportunities and responsibilities to boost the educational attainment, job readiness, child wellbeing and functioning of families with young children in highly disadvantaged areas, by:

  • providing opportunities and responsibilities to assist teenage parents and jobless families in increasing employment and education outcomes, and enhance family functioning, parenting skills and child wellbeing;
  • encouraging community engagement in developing tailored solutions for improving workforce participation, as well as supporting school readiness and better outcomes for children;
  • ensuring children in disadvantaged locations are ready for preschool and school, and that children's best interests are considered in the expenditure of welfare payments;
  • maximising investment in employment, training and family services; and
  • strengthening education, jobs and skills pathways in regional Australia (p. 6).22

Evaluation design: The BAFW package was proposed to be evaluated at three separate, but related, levels, including evaluation of the overarching package, evaluation of the three streams and evaluation of each individual measure. The evaluation strategy was developed by the BAFW Evaluation Management Group, with advice and input sought from the National Place-Based Advisory Group, the BAFW Skills Working Group, the BAFW Place-Based Officials' Working Group, the BAFW Implementation and Evaluation Steering Committee, and all individual departments that were responsible for individual measures. An external evaluation expert also provided independent advice on the place-based stream.22

The interim BAFW evaluation chapter focusing on people in areas of entrenched disadvantage (as per the "place-based"/BFLS stream referred to above) included six specifically targeted measures that were expected to affect clients of selected government services. It was proposed that these measures would work together to address chronic and concentrated disadvantage, using a client focus. These measures were:

  • compulsory participation plans and supports for teenage parents (Helping Young Parents, HYP);
  • compulsory participation requirements for jobless families (Supporting Jobless Families, SJF);
  • income management (IM);
  • Job Services Australia demonstration pilots (JSA pilots);
  • Local Connections to Work (LCTW); and
  • School Enrolment and Attendance Measure (SEAM).24

BFLS also included a further three measures, including Community Innovation through Collaboration, the Priority Employment Area (PEA) initiative and Local Connections to Work, that were designed to coordinate and deliver social and support services.24 At the BFLS level, the evaluation proposed to focus on the collective effects of the suite of six measures, and whether its outcomes, with regard to addressing locational disadvantage, were achieved.22

The BAFW package evaluation strategy indicated that several existing and new data sources would be used, including administrative data, surveys, focus groups, forums, in-depth case studies, information from individual case management plans and program-level monitoring/evaluation reports. Data collections from some of the individual measures, including the JSA pilots, were also to be used, with quantitative data to be supplemented by qualitative information.22

In addition, one of BAFW's major data sources was the "My Journey" longitudinal survey; a national survey designed to capture person-level information about the effects of the package, including education and workplace participation, as well as incremental progress (e.g., attitudinal change, work readiness and social inclusion). "My Journey" commenced in March 2012 and involved four six-monthly waves of computer-assisted telephone interviews (CATIs) with approximately 6,000 respondents over a two-year period. The interviews were conducted by the Wallis Consulting Group. There were three cohorts, with single parents being in the first cohort, SJF clients included in the second cohort (beginning at Wave 2), and the final cohort consisting of apprentices and trainees.25

While there were other studies conducted involving the BFLS sites, including the Place Connections qualitative study (contracted to Urbis Pty Ltd), the Place-Based Linkages study and the BFLS Linkages survey,25 these were not completed and/or reported on at the interim evaluation stage.24

The evaluation strategy also proposed that, where possible, comparison groups/locations would be used to identify the specific impacts of BAFW on the target population.22 Five comparison sites were chosen from other disadvantaged local government areas (LGAs) in the remaining PEAs, on the basis of their similarities to the existing PEAs, although this only occurred for some measures, including HYP, SJF and IM.24

Program outcomes: It was reported that as of 30 June 2013, more than 55,200 people living in the 10 selected LGAs had been affected by BAFW, with 27% (15,100) affected by measures that were specifically targeted towards these communities. Parental responses to the BAFW place-based measures tended to be divided depending on whether parents felt that work or study was not important to them at that point in their lives, and by a smaller proportion who were already engaged or seeking to engage with education and employment. Benefits noted for this smaller group included: increased preschool and formal child care attendance; an increase in their ability and willingness to participate in job readiness activities, education and employment (due primarily to receiving financial assistance with child care costs); participation in education increasing by 15 percentage points to 39% over the interim evaluation period; and a 10 percentage point decrease in financial stress. Job seekers in the 10 LGAs also reported benefits associated with the "wrap-around servicing" that was provided as part of LCTW, with it helping clients to build stronger relationships with Centrelink, which was perceived as better at placing them to find work.24

Table 10: Did the BAFW package demonstrate and/or evaluate the common elements?
Common elements Demonstrated? Evaluated?
Flexible delivery Partially demonstrated Not evaluated
  • There was a degree of flexibility demonstrated in the planning and delivery of BFLS. For example, the Flexible Funding Pool was a key element, and reportedly enabled services to tailor funding according to customers' needs at the local level.24
  • Some BFLS measures, including the JSA pilots, also employed what was considered a "bottom-up" approach, where providers were able to implement innovative ideas at the local level, rather than adopting a more "top-down" approach, where solutions were identified at the department level and communicated down to providers. The bottom-up approach meant that providers felt they were able to learn and test out new ways of working with clients.24
  • Flexibility was not formally evaluated, however, in the interim evaluation report.24
Local autonomy Demonstrated Partially evaluated
  • An overall objective of the BFLS stream was to adopt a "community-led approach" to overcome the complex factors associated with the cycle of ongoing disadvantage.26
  • As part of this strategy, BFLS intended to incorporate the views of relevant (including local) stakeholders, with community organisations playing a role in its development and rollout.24 An example of how this was applied was in Rockhampton, Queensland, where Community Action Leaders supported a Community Innovation Forum, which aimed to improve workforce participation in the area through increasing community engagement and awareness.24
  • In earlier stages of BFLS, there were concerns from stakeholders that "there was not sufficient local control to be considered truly place-based"(p. 39).24 Views changed to some extent, however, with a number of stakeholders having the capacity to influence decisions at a local level, which, in turn, facilitated implementation.23 There was also a lack of clarity in some cases regarding the role of Local Advisory Groups, which may have negated their influence to some extent.24
Capacity development Partially demonstrated Partially evaluated
  • BAFW-specific funding was to be supplemented with inhouse data and evaluation capabilities across the participating departments, particularly within the then DEEWR (later DoEd/DoEm). The intention was for DEEWR to provide analysis support for the delivery of the core evaluation strategy, but on some occasions, there were insufficient IT support systems available for administrative data collection.22
  • Capacity development in regard to strengthening collaborative relationships at the local level was the most frequently identified benefit associated with BFLS participation, with about half of all stakeholders citing examples of increased collaboration (e.g., interagency collaboration), which, in turn, was perceived to enhance service delivery.24
Lead times Partially demonstrated Not evaluated
  • The capacity for meaningful baseline data to be collected was compromised to some extent by several of the programs beginning prior to the start of the evaluation and the lead times required to collect this new data. Hence, lead times were reportedly built into the evaluation design, but needed to be balanced within the scope of the relatively short evaluation time frame (see "Long-term focus" below).22
Long-term focus Partially demonstrated Not evaluated
  • The BAFW evaluation was funded over four financial years, commencing in July 2011 and was scheduled to conclude in October 2014.22 The BFLS interim evaluation report was completed in 2013.24
  • Individual BAFW measures differed in their time scope, with some proposed as being ongoing, and others shorter term. However, the four-year evaluation period restricted the evaluation to exploring outcomes that were achievable within this time frame.22
Spatial targeting Demonstrated Partially evaluated
  • Ten LGAs were selected for inclusion in BAFW, with three of these being in NSW, two each in Victoria and Queensland, and one each in WA, SA and Tasmania. There were no BFLS sites in either the ACT or the NT.24
  • While the 10 selected LGAs were the focus of the BAFW evaluation, many of the BFLS measures operated in geographical areas that were larger than the scope of these LGAs (e.g., HYP had 1,669 participants, 1,102 of which were from a BFLS LGA). Further, a number of participants moved out of the BFLS LGAs during the evaluation period, but still continued their participation.24
Social targeting Demonstrated Partially evaluated
  • The 10 LGAs were selected for BAFW due to their being identified as areas of entrenched disadvantage.23 All but one of these sites was the most disadvantaged LGA within a much larger PEA.22
  • Disadvantage was assessed using 70 indicators of social and economic vulnerability, including high unemployment rates, populations with high level of income support and low educational attainment. Five comparison sites were selected using these criteria.24,26
  • Due to several of the BFLS measures targeting parents on income support, women were over-represented, with HYP and SJF participants being almost exclusively female (although the majority of JSA pilot and LCTW participants were male).24
Joined-up working Demonstrated Evaluated
  • A wide range of stakeholders (which were divided into eight groups) were identified in the BAFW evaluation strategy as being critical to the program's success: clients; employers and industry groups; education and training providers; JSA providers and other case managers, including Centrelink staff; community stakeholders; community coordinators; federal and state government officers, BAFW governance groups and the National Place-Based Advisory Group; and non-BAFW-specific support services (e.g., housing, child care).22
  • The strengthening of collaborative relationships was viewed as a key benefit of BFLS, particularly with regard to agencies working in a cooperative manner to deliver services to key cohorts. It was reported that LCTW in particular increased collaboration among BFLS stakeholders and allowed Centrelink to build stronger relationships with both community organisations and service providers.24
  • The perceived effectiveness of local collaborative networks was reasonably high, with 66% of survey BFLS stakeholders reporting their local network to be effective in achieving its objectives, and 57% indicating it was effective in benefiting their local community.23 BFLS stakeholders also perceived, however, a lack of integration between employment service providers and the local community.24
Governance Demonstrated Not evaluated
  • The evaluation was managed through the BAFW Evaluation Management Group and included representatives from the then DEEWR, FaHCSIA, DHS, PM&C, the Department of Finance and Deregulation, and Treasury.22
  • The Evaluation Management Group reported to the BAFW Implementation and Evaluation Steering Committee whose membership consisted of deputy secretaries from the six agencies.22
  • It was proposed that there were three tiers of "BFLS participants": clients, stakeholders and governance bodies (including the Commonwealth and state governments, the National Place Based Advisory Group and the Place-Based Officials Working Group). Governance was not a focus, however, of the interim BFLS evaluation.24
Table 11: Did the BAFW package demonstrate the common elements?
Common elements Demonstrated?
Establishing causality Not demonstrated
  • Baseline data for the place-based stream was to be collected in the first 9 months of the evaluation, with ongoing data collection activities to occur at 6-monthly intervals.22
  • Data presented in Chapter 2.6, "People in areas of entrenched disadvantage", of the BAFW interim evaluation report, primarily focused on program participation rates across the key outcome areas, and participant and stakeholder views regarding their participation and its perceived effects, with comparisons made from baseline data (as collected by the "My Journey" longitudinal survey).24
  • At the interim evaluation stage,24 comparison sites were not included in the analyses (or at least reported on), and hence, direct causal links between BAFW participation and specific outcomes could not be established.
Attribution Not demonstrated
  • The evaluation strategy report22 acknowledged that while using a wide range of both qualitative and quantitative data sources would provide a basis for attributing results to individual measures, the complexity of the package and the long lead time associated with many measures would create challenges in identifying the key change agents.22
  • While there was some flexibility proposed in the evaluation framework to consider external factors (e.g., changes in the economy or employment opportunities), it was also recognised that it would be difficult to specifically recognise/identify their effects, and the evaluation would be largely restricted to providing contextual information.22
  • This was largely the case in Chapter 2.6 of the BAFW interim evaluation report, with a number of positive trends (e.g., an increase in parental employment) associated with, but unable to be specifically attributed to, BAFW measures.24
Theory of change - Articulated Demonstrated
  • The place-based (BFLS) stream of the BAFW package intended to positively affect people in areas of entrenched disadvantage in four distinct areas: supporting children's wellbeing (via parenting support, supporting early childhood learning and supporting children's education); improving social inclusion (through supporting social connectedness, improving access to support services and decreasing financial stress); supporting participation (through child care support, workshops, interviews and participation plans); and providing tailored support (via the co-location of support services and enhanced delivery of employment services support).24
Theory of change - Measured Partially demonstrated
  • In the interim evaluation report,24 results were presented for each of these four theory of change elements, which were in the form of changes that had taken place across the evaluation period. As indicated in "Establishing causality" and "Attribution", however, these trends could not be specifically attributed to the BFLS measures at the time of the interim evaluation.
Residential mobility Not demonstrated
  • Residential mobility was not a specific focus of the BFLS interim evaluation, although as indicated in "Spatial targeting", participants were tracked in regard to whether they remained or moved out of BFLS LGAs during the evaluation period.24
Cost-effectiveness Not demonstrated
  • In the evaluation strategy report,22 it was indicated that "to the greatest extent possible, within the time frame and resources for the evaluation, an assessment of the value-for-money of the BAFW package will be undertaken" (p. 10).22
  • However, cost-effectiveness was not addressed in the interim report.

4.3 Overall findings: Commonwealth place-based initiatives review

Based on these five case studies and the review of the other four Commonwealth place-based initiatives for which we were provided with evaluation materials, conclusions can be made regarding the extent to which the initiative and its evaluation demonstrated the common elements. A summary of the percentage of the nine Commonwealth place-based initiatives that demonstrated and/or evaluated each of the common elements is displayed in Table 12.

Table 12: Summary of the extent to which the nine investigated Commonwealth place-based initiatives demonstrated or evaluated the common elements
Common elements Demonstrated element (%) Evaluated element (%)
For the PBI Yes Partially met Yes Partially met
Spatial targeting 22 78 0 11
Social targeting 56 44 11 33
Flexible delivery 44 22 11 44
Local autonomy 44 33 22 33
Joined-up working 56 22 33 33
Governance 67 33 33 22
Capacity development 11 67 11 56
Lead times 33 44 11 0
Long-term focus 56 33 0 22
For the evaluation Yes Partially met

Not applicable

Causality 11 11
Attribution 11 11
Theory of change - Articulated 100 0
Theory of change - Measured 0 33
Residential mobility 11 22
Cost-effectiveness 11 0

Table 12 shows that there was wide variation in the extent to which the common elements were met by the reviewed Commonwealth PBIs. A description of how Commonwealth PBIs demonstrated and evaluated the common elements follows.

Spatial targeting

Spatial targeting was evident to some degree for all of the reviewed PBIs, with the majority of programs being implemented across various sites that covered expansive geographical areas. The target populations in these areas were not always large, however, which was primarily due to their remoteness. Financial considerations also limited the number of locations where the initiative could be implemented in some instances.

There were issues for a number of initiatives regarding the defined geographical boundaries, which did not always operate effectively in practice. For example, the Petrol Sniffing Strategy (see Appendix E) targeted Indigenous communities in remote areas (called "PSS Zones"), but these became less relevant over time when agencies moved outside of the zones to address emerging needs elsewhere.28 The flexibility to service those who are located outside set geographical areas could, however, also be viewed as an attribute of a program. Although spatial targeting was rarely specifically evaluated, it was evaluated in part for some initiatives within the context of social targeting (i.e., "place" was partly evaluated through assessing the extent to which the initiative reached its target group within the selected sites), which will be discussed next.

Social targeting

Social targeting was also at least partly demonstrated across all of the reviewed PBIs, with a rationale typically provided as to why a particular group or population was the focus. In some instances, however, the process for selecting the program sites was not described. Characteristics that illustrated this element included using specific criteria to identify the areas most in need of intervention (e.g., in the case of CfC, SEIFA data1,2), collecting detailed information regarding the disadvantage or challenges faced by the target population(s) (e.g., for the 10 LGAs selected for the BFLS stream of BAFW24,26), and in the case of the Centrelink PBS initiative, targeting particular forms of disadvantage or exclusion in specific communities.19

Just over half (56%) of the initiatives did not evaluate social targeting. The few PBIs that at least partially evaluated this element showed some evidence of assessing the extent to which the target (i.e., disadvantaged) population was reached by the program and/or benefited from the services provided. For example, the CfC evaluation investigated the extent to which the "hard-to-reach" and the most socio-economically disadvantaged families/children within the CfC sites were likely to have benefited from its services.2

Flexible delivery

Flexible delivery was at least partially demonstrated, as well as evaluated, for two-thirds (66%) of the initiatives. For the 44% of Commonwealth initiatives that were deemed to have achieved flexibility, this element was clearly "built in" to its program design, with features such as flexible service delivery and discretionary funding mechanisms apparent. For example, the CfC Facilitating Partner Model, which involved a non-government organisation at each site acting in an intermediary position between the government and the community sector, enhanced service provider collaboration and, ultimately, provided leverage with what and how services were delivered within the CfC communities.2 Flexibility demonstrated through the use of discretionary funds in the D2DL18 and National Partnership Agreement on Remote Indigenous Housing (NPARIH; see Appendix E)27 initiatives also enabled services to be tailored to the needs of their target individuals/communities.

While flexible delivery was demonstrated across most of the reviewed initiatives, there were some instances where flexibility was more reactive and occurred in response to issues that arose during implementation or practice, rather than being "built in" to the initiative's design. Furthermore, this element did not tend to be evaluated in a comprehensive manner, with only 11% of the initiatives deemed to have adequately evaluated this, and a further 44% partially evaluating it. The CfC evaluation showed that although the service delivery and discretionary funding flexibility that was inherent in the program design was effective, there were some tensions in achieving a balance between flexibility and establishing and adhering to clear governance frameworks.2 Other limited evaluations of flexible delivery indicated that this element enabled people to more effectively engage with the service delivery system, but the outcomes or benefits associated with enhanced engagement were not assessed.

Local autonomy

Local autonomy was at least partially demonstrated by approximately three-quarters (77%) of the Commonwealth PBIs reviewed. Those that were identified as fully achieving this element had community involvement and input as an integral part of the program logic, with local stakeholders having direct input into aspects of the initiative's design, implementation and practice. For example, CfC's whole-of-community approach was developed in consultation with local stakeholders who also actively participated in CfC committees,2 while the Centrelink PBS initiative drew upon the knowledge and experience of local service delivery providers in order to shape its design and implementation.19 Further, the BFLS stream of BAFW adopted a community-led approach, which enabled stakeholders to have some influence on decisions at the local level.23,26 It must also be noted that some initiatives drew upon aspects of local autonomy, but did not demonstrate or only partially demonstrated the element due to the focus being more on community consultation, rather than direct input.

The evaluation of local autonomy was mixed, with 55% of initiatives at least partially evaluating this element. When local autonomy was evaluated, it was found that local community engagement from the early stages of the initiative helped to enhance ownership and participation. This has been viewed as being particularly important for initiatives implemented in Indigenous communities, with their direct involvement being critical to success. The evaluation of this element has also indicated, however, that for some initiatives, such as the Cape York Welfare Reforms, a balance was required between achieving local involvement and limiting intrusion to community members.9

Joined-up working

Joined-up working and governance were two common elements that were generally well demonstrated and evaluated across the reviewed Commonwealth PBIs. Joined-up working was assessed as being met by 56% of the initiatives, and partially met by a further 22%. Initiatives that best demonstrated this element established strong links at both the government and community/service provider levels, thereby enhancing collaboration and information sharing across agencies and sectors. Another important aspect of joined-up working was developing clear and transparent roles, responsibilities and funding arrangements in the early planning stages of the initiative, which assisted in streamlining processes and maximising the effectiveness of these working relationships. There were cases, however, where integrated or whole-of-government approaches were proposed, but were not actually implemented, or the level of coordination decreased over time.

Joined-up working was at least partially evaluated by two-thirds (66%) of the initiatives, with these evaluations finding some evidence of the element being effectively applied, as well as other areas in which its practice could have been improved. For example, in D2DL, strong links were developed with public mental health service providers and between service delivery personnel within the sites, but there was a lack of streamlined process between the sites, which meant that information exchange was not optimal.18 Meanwhile, the Groote Eylandt and Bickerton Island Regional Partnership Agreement (RPA; see Appendix E) progress evaluation found that the RPA was working well as a government, community and industry partnership, with these collaborative relationships enabling progress to be made on complex and entrenched problems, such as Machado Joseph Disease.29 Furthermore, a key strength of the BFLS stream of BAFW was the strong collaborative relationships that were formed between agencies, key stakeholders and local organisations, which were generally found to enhance service delivery and outcomes in the target LGAs.23,24 Some of the methodology used in the evaluation was particularly promising. For example, the BFLS Linkages Survey surveyed key stakeholders to collect information on cooperation between stakeholders so that the degree of joined-up services could be assessed. While only one survey was conducted, the planned subsequent survey should enable an assessment of the evolution of cooperation between stakeholders in BFLS sites. Overall, these evaluation findings suggest that joined-up working was important, given that without effective collaboration and integration, resources could not be used as effectively to address the issues at hand.

Governance

Governance was at least partly demonstrated by each of the Commonwealth PBIs, with several sharing governance responsibilities across a number of agencies. Complex governance structures and mechanisms were also demonstrated, with other partners, such as state governments, local industry and local councils involved for some initiatives, and working groups/steering committees often assisting in their implementation and operation. For instance, the Groote Eylandt and Bickerton Island RPA involved five partner agencies (the Anindilyakwa Land Council, the Australian Government, the Northern Territory Government, the East Arnhem Shire Council and the Groote Eylandt Mining Company), in addition to other parties such as the Infrastructure and Planning Working Group, the Anindilyakwa Training and Education Board, and the Youth Steering Committee.29

Just over half (55%) of the initiatives at least partially evaluated governance, and generally indicated that, as a whole, leadership and governance structures operated effectively. On some occasions, there were conflicts between national and/or state "upstream" policies and systems, which meant that processes were not as well streamlined as they could have been, but in other cases, governance operated more seamlessly. To this effect, the evaluations for initiatives that were implemented in multiple states, including D2DL18 and NPARIH,27 indicated that cross-jurisdictional governance mechanisms and jurisdictional collaboration were a priority, and were important in improving the programs' outcomes.

Capacity development

Capacity development was demonstrated and evaluated at least to some extent by most Commonwealth initiatives, although CfC was the only initiative assessed as fully meeting this element in its practice and evaluation. In the case of CfC, collaborative efforts between the service delivery agencies contributed to skill and capacity enhancement, while shared training and mentoring opportunities were also provided.2 Other PBIs demonstrated some aspects of capacity development - for example, at the government level - but did not provide sufficient training opportunities to service delivery organisation staff, or had inadequate information and data collection systems. When capacity development was evaluated, it was reported as being important in the effective operation of programs. In other cases where it was not adequately demonstrated, it was identified as an area that required greater investment (e.g., the Groote Eylandt and Bickerton Island RPA29).

Lead times

Sufficient lead times were demonstrated at least to some extent for 77% of the initiatives, although this element was rarely evaluated (the Cape York Welfare Reform was the only initiative to evaluate it). In cases where lead times were well demonstrated, practices such as a baseline data study (Groote Eylandt and Bickerton Island RPA29), an extensive engagement process (Cape York Welfare Reforms11) and consultation workshops (Pornography Awareness Program;30 see Appendix E) were employed. For several initiatives, however, planning periods varied between program sites, meaning that while they were adequate in some locations, they were minimal or non-existent in others. This meant that some community stakeholders/members were not sufficiently informed or consulted about the initiative.

Long-term focus

A long-term focus was evident within most of the Commonwealth PBIs that were reviewed, although, once again, not commonly evaluated. The vast majority (89%) of the initiatives showed some indication of a long-term focus, particularly from a financial perspective, with several involving initial funding periods that were subsequently extended (e.g., the Petrol Sniffing Strategy,28 Cape York Welfare Reform Trial12 and the Pornography Awareness Program30). As highlighted in the Groote and Bickerton Eylandt evaluation,29 the definition of "long-term focus" can vary, depending on an initiative's theory of change, with those attempting to affect social change likely to require a substantial period of time to have a significant effect.

The relatively small proportion (22%) of initiatives that partially evaluated long-term focus, assessed the effects of their long-term investment to some degree, but also recognised the need for ongoing support in order to maximise their outcomes. For example, NPARIH, which is being funded over a 10-year period, is expected to result in significant gains for Indigenous housing arrangements, but ongoing collaborative policy effort will be required, particularly in areas experiencing intergenerational overcrowding.27

Causality

Causality was rarely demonstrated in the evaluation of the reviewed Commonwealth PBIs, with only the CfC evaluation deemed to have fully met this element. This evaluation used various statistical techniques to longitudinally explore patterns of change and stability across the outcome measures, thereby enabling effects over time to be detected for those residing in the CfC sites, in comparison to the contrast sites.1

For the PBIs that did not demonstrate causality, there were a number of reasons as to why this did not occur. Firstly, some evaluations were more of a strategic review than an evaluation per se, and the focus was primarily on aspects such as service accessibility and delivery, rather than on the measurement and reporting of any individual or community-level effects of the initiative (e.g., PSS,28 D2DL 18 and Centrelink PBS21). The complexity of some initiatives (e.g., CYWR trial, which involved 15 individual projects6) was cited as a barrier for demonstrating causality, with the outcome objectives for some programs unable to be realistically achieved within the evaluation time frame (e.g., relating to social change9 or systemic issues such as overcrowding27). Furthermore, for a number of evaluations, the data collected were predominantly qualitative and/or were collected anecdotally, and there were inconsistent data collection methods and variable data quality between program sites for some initiatives; hence, making a quantitative assessment of the program's effects was not possible (e.g., D2DL,18 Groote Eylandt and Bickerton Island RPA,29 NPARIH, 27 and Centrelink PBS21) Finally, in the case of the BAFW package, the evaluation was at the interim stage (i.e., two years into the evaluation period) and further follow-up data could potentially be used to test the effectiveness of the program.24

Attribution

Attribution was also not commonly demonstrated in the evaluations of Commonwealth PBIs, with only 22% either demonstrating or partially demonstrating this element. As indicated above, statistical techniques applied to the CfC evaluation enabled causality to be established, with the use of modelling also allowing differences between the CfC and contrast sites to be attributed to program exposure, by controlling for other factors.2 The evaluation of the CYWR trial partially met the attribution element by using comparison groups, which were compared to the four trial sites across key outcomes measures. Similarly to the CfC evaluation, statistical techniques were used to investigate the average change in the outcome variables over time across the trial and comparison communities. While this enabled changes in the trial sites to be attributed to the initiative, this approach did not enable the specific "drivers" of change to be identified.14

One key difficulty that evaluations faced in demonstrating attribution was being able to separate the effects of the program in question from other, related initiatives that were concurrently in operation in the same areas. For example, in the case of D2DL, there were other programs at the national and state levels that were also targeting the needs of those experiencing long-term mental illness, and its evaluation was unable to individualise its specific impact.18

Theory of change

There was a clear contrast between the evaluations regarding the extent to which the theory of change was articulated versus measured. While all of the evaluations of Commonwealth PBIs were able to articulate their theory of change - that is, clearly stipulate the rationale and objectives of the program - only one-third (33%) of the evaluations were deemed to partially measure the extent to which they were achieved. As an example, for the CYWR trial, the project-level "logic maps" that were developed for each of its 15 programs included measurable indicators, which enabled their outcomes to be assessed. What was lacking, however, was the development and application of whole-of-strategy indicators that could link these individual program outcomes together to address the overall outcomes stipulated in the theory of change.11

A predominant reason why theory of change was rarely measured was that demonstrating causal effects and being able to specifically attribute them to the initiative was generally not achieved in the first instance (as indicated above). Another issue associated with the evaluation of this element was that in the few cases where the overall objective of the initiative was measured, it did not measure the sub-elements that were proposed in the theory of change to contribute towards this outcome (i.e., the theory of change indicated that the program would affect sub-elements, which would subsequently contribute to achieving the proposed outcomes). Moreover, many initiatives had multiple features that made collection of information and attribution to the initiative very difficult. Overall, the extent to which this element was evaluated highlights a key discrepancy in the Commonwealth PBIs between articulating a theory of change (all PBIs were able to do this), and measuring if the program had actually followed the predicted theory of change, which was beyond the capacity of most PBIs to achieve.

Residential mobility

Residential mobility was evaluated by a third (33%) of the Commonwealth PBIs, although it is important to note that this element was not particularly relevant for some initiatives (e.g., the PSS28 and Pornography Awareness Program30), which is distinct from evaluations that did not investigate this element when it may have been relevant. This element was addressed in the CfC evaluation, with residential mobility compared over time between families in the CfC and contrast sites, where it was found that they possessed similar patterns that were comparable to the general population.1 The evaluations for other initiatives, including the CYWR trial and NPARIH, did not specifically focus on residential mobility, but did look at the extent to which the target population took up training or job opportunities in other areas, due to incentives (e.g., affordable accommodation) offered as part of the program.16,27 The uptake and effects of these incentives, however, appeared to have been minimal.

Cost-effectiveness

Cost-effectiveness was beyond the scope of most Commonwealth PBIs' evaluations to investigate, with the only detected cost-benefit analysis being presented in the first phase evaluation of CfC. In this case, financial benefits were attributed to the outcome measures that CfC was deemed to have positively affected, which were compared to the costs associated with funding the initiative.4

Although cost-effectiveness was often recommended or identified as being important in the early stages of initiatives, it was not included in evaluations due to issues such as a lack of quality outcome data, or potentially being too early in the program for its full benefits to be evident (e.g., the CYWR trial,6 the Groote Eylandt and Bickerton Island RPA29 and the BAFW package22). Essentially, however, this indicates that limited information has been provided by Commonwealth PBI evaluations regarding the cost-effectiveness of the financial investment in programs.

* The superscript numbers in this chapter refer to the sources in the supplementary reference list for Commonwealth place-based initiatives in section 7.3.

5. Discussion

In this section, the key lessons from the previous sections are distilled. We discuss how best to focus PBIs, key elements of successful PBI service design, and considerations for the implementation of such initiatives. Key lessons in evaluating PBIs are also examined, specifically for ascertaining what works and what savings can be realised, as well as how to generate sufficient evidence to inform translating a PBI to other settings and groups.

5.1 Where to focus: Social and spatial targeting

Focusing on the right population (social targeting) and areas (spatial targeting) is an important component of best practice in PBIs (Dyson et al., 2009; Rae, 2011; UK House of Commons, 2003). All of the Commonwealth and international PBIs engaged in targeting to some degree, but the extent to which it was done effectively was rarely ascertained by any of the PBIs reviewed. For Commonwealth PBIs, the geographical boundaries that were defined for the program did not always work well in practice. In some instances, these boundaries were artificially created, did not make sense to residents and were imposed on the community (e.g., CfC). In other instances, the original areas that were the focus of the initiative were expanded once it was ascertained that the needs serviced were required to be more flexibly deployed in adjacent areas. Almost all of the PBIs did not evaluate social targeting. Best practice was evident in one international PBI that used Census data on the population of interest and service engagement data to calculate the proportion of in-scope clients that had been addressed by the program(Ghate et al., 2008). This type of service reach methodology was not implemented in the Commonwealth PBIs.

5.2 Key features of place-based initiative service design

Unlike other service delivery models, place-based initiatives afford capacity to provide flexibility, a degree of local autonomy to the local community, and opportunity for services to be joined up or work together in an area, but due to the relative degree of freedom this affords, good governance arrangements are required to be in place. These are the key features of best practice PBI service delivery and design.

The Commonwealth place-based initiatives that were reviewed largely provided good examples of a flexible approach to service delivery and some discretion in how funds were spent and for what purpose. These features were also evident in the international literature. Like the international literature, however, there was little examination of how these flexible arrangements were implemented and no evidence as to whether there was a demonstrable effect on the key outcomes of interest.

It was quite common for communities to be given autonomy at the local level, but how this was instantiated varied in Commonwealth PBIs and also internationally. At one end of the spectrum, communities were consulted about what was happening, while at the other end, they were active decision-makers in offering the service. Community members were far less likely to be active agents in deciding on the service offering and, to some degree, there has to be caution exercised about when this is appropriate, as in some instances it was found that too much engagement was considered burdensome. Policy-makers need to use judgement about the nature of the services to be delivered and the expertise of the members of the community in making meaningful suggestions about the appropriate set of services for their needs. While there was little evaluation of the implementation and outcomes of local autonomy, some of the Australian evidence from a Commonwealth PBI suggests that a consultation process enhanced ownership and participation within the community.

Joined-up working constitutes an approach that focuses on integrating or coordinating, and developing, partnerships between organisations within local areas, across the government, private and community sectors. It is sometimes referred to as "no wrong door", because clients will be referred to the services that they need no matter at which point they enter the service delivery system. Most Commonwealth-funded PBIs showed some evidence of establishing links between different organisations to enhance collaboration, referral and information sharing. An important feature of doing this well was to develop clear and transparent roles and responsibilities and funding arrangements in the early stages of the initiative.

Commonwealth agencies need to have good checks and balances in place in the context of funding that enables communities' discretion over the allocation of government resources. However, there was no systematic approach taken to test the optimal governance structures that need to be put in place for such initiatives.

The sharing of governance responsibilities between government and the community was generally in place for Commonwealth PBIs. In some instances, conflicts between national and state policies or systems were an issue. Careful consideration of cross-sectoral involvement and engagement across government and the private and community sectors needs to be undertaken to avoid such conflict.

5.3 Program implementation: Capacity development, lead times and long-term focus

Often in PBIs, the methods of delivering services are quite different to how things have operated in the past. Having a flexible and joined-up service delivery system and active engagement in the community requires the development and acquisition of different skills. All Commonwealth PBIs demonstrated this to some degree, and examples of good practice included shared training and mentoring opportunities as service practitioners built core skills but also strengthening relationships between NGOs (Muir et al., 2008). Based on the limited evidence from the Commonwealth PBIs, capacity development was important in the effective operation of programs, but was sometimes not adequately resourced.

Having adequate lead times to set up programs, build relationships within communities, build capacity within service delivery organisations and ensure evaluations are set up were key elements of best practice. Three-quarters of Commonwealth PBIs demonstrated this to some degree. Time for consultation with the community is particularly important, and in some instances a lack of engagement with the community may have led to problems in the longer term.

Many of the communities in which PBIs are operating are some of the most disadvantaged in Australia and most Commonwealth PBIs were prepared to make a long-term investment of funds. There was also recognition that PBIs take a fairly substantial period of time to demonstrate significant outcomes. While there were a few instances of testing whether the practices developed and some of the international PBIs were implemented into mainstream practice internationally - including the NNI (NNI Research Team, 2007), NSNR (Amion Consulting, 2010) and On Track (Ghate et al., 2008) - in Commonwealth PBIs that evaluated this element, there was a recognition that ongoing investment was required to maximise the effects (see NPARIH; Department of Families, Housing, Community Services and Indigenous Affairs, 2013).

5.4 Does it work and are there savings?

While many of the international PBIs that were selected for review were instances of best practice, one of the most striking differences between the international PBIs and Commonwealth PBIs was the quality of the evaluations. Very few Commonwealth PBIs had evaluations that enabled causality to be examined, whereas two-thirds of the international PBIs were able to show that the evaluated programs affected the key outcomes. Some of the key features of the international PBIs that enabled them to make causal claims were having matched comparison areas, having large-scale longitudinal data from surveys or from administrative records, and undertaking sophisticated statistical analyses that controlled for confounding variables. In some evaluation reports, Commonwealth-funded PBIs argued that the complexity of the program meant that it was not possible to test the effectiveness of the program. However, many evaluations from the United Kingdom that were of a particularly high quality and tested for program effectiveness were evaluating complex programs. It is certainly the case that the programs in the United Kingdom reflected a significant investment by government in large-scale evaluations and the development of evaluative capacity that is associated with such an investment.

It is possible that there may be several PBIs operating in the one area, particularly when policy-makers target areas based on indicators of social disadvantage at the area level (as the United Kingdom did in the late 1990s and 2000s). Therefore it is important to consider the presence of other initiatives when trying to establish if PBIs "work", so that attribution of any positive (or negative) effects can be correctly rendered.

Another related issue in establishing whether a PBI works is taking account of residential mobility. People may leave the area, therefore making it difficult to evaluate the benefits, and higher rates of mobility may be a particular feature of disadvantaged areas. Only one Commonwealth PBI specifically examined whether residential mobility affected the outcomes of the program (CfC; Edwards et al., 2014), while the evaluation of the New Deal for Communities in the United Kingdom (Wilkinson & McLennan, 2010) was one of the few international PBIs to do so. In CfC, program outcomes did not seem to be affected by mobility; however, New Deal for Communities areas with higher rates of mobility experienced less positive change in relation to housing and the physical environment, underscoring the importance of taking account of the experiences of "movers" and "stayers" in communities.

Coupled to considerations about whether PBIs work, is whether they are cost-effective. While many Commonwealth PBIs recommended or identified in the early stages that cost-effectiveness was important, the fact that very few were able to test for such effectiveness precluded an examination of whether there were any savings to government. Other reasons included poor-quality outcome data and insufficient time for the program to be fully implemented. In contrast to the evaluations of Commonwealth PBIs, two in five international programs that were the focus of the review provided information on the cost-effectiveness of PBIs, with many reporting significant savings to government. These included Excellence in Cities (Kendall et al., 2005), NDC (Batty et al., 2010), NSNR (Amion Consulting, 2010; Wilkinson, Whitworth & McLennan, 2010) and SSLP (NESS, 2011).

5.5 So it works, then what? How can findings be translated to other settings and scaled up?

One of the key elements of best practice that we examined in this report is a well-articulated theory of change that provides explicit expectations about the short-, medium- and long-term outcomes and the mechanisms by which the program works. Often evaluations are focused on whether the program works, but do not provide sufficient consideration to how key outcomes are affected. A good understanding of what key mechanisms were important to shift outcomes in successful programs is critical to understanding how these initiatives can be amended or adapted to different contexts. In instances where programs were not successful in shifting key outcomes, understanding which key mechanisms were not affected provides policy-makers and service providers with the opportunity to identify barriers to successful implementation. In the absence of such data, PBIs with significant start-up costs that may be successful could be abandoned. In instances where it is necessary to scale back PBIs, understanding the successful constituents enables a reduction in costs without a commensurate reduction in benefits to the community. While most Commonwealth PBIs proposed a theory of change, very few collected sufficient information to test whether the program was working as expected. Having a good understanding of the precise mechanisms by which change is affected is an area of improvement in the evaluation of PBIs that is needed not just in Australia, but internationally (Ludwig, Kling, & Mullainathan, 2011). It is also an area that has been under-researched in broader policy evaluations.

6. Conclusion

Many Commonwealth PBIs reviewed as part of this report have features that accord with other PBIs internationally. They target particular areas and social groups and engage in flexible service delivery and funding models, attempt to provide communities with a say in the services provided (local autonomy) and attempt to join up services. However, evaluations of these common elements were rarely done by either international or Commonwealth PBIs. Moreover, by following practices based on the elements of local autonomy and flexibility in many of the PBIs, it makes it very difficult to ascertain why "what works" works, due to the variability in implementation. Common practice in the delivery of place-based initiatives suggests that there needs to be a flexible approach and "locally grounded" design and implementation, but research into the effectiveness and benefits of local autonomy and flexibility for a PBI relative to an initiative that did not have local autonomy and flexibility as an underlying principle is lacking in the international literature and nationally. Ultimately, there is an absence of empirical studies that set out to rigorously test this policy assumption and whether the "good things" of greater local autonomy and community involvement are realised in practice (Burton et al., 2006; see also Burton et al., 2004).

The high quality evaluations of international PBIs do show that they can be effective and can realise significant cost savings to government. However, among Commonwealth PBIs, the causality and cost-effectiveness of programs were rarely evaluated. An understanding of the precise mechanisms of "what works" was lacking in both the Commonwealth and international PBIs reviewed. Significant learnings could be generated to improve policy design if there was investment in such evaluations. There is an opportunity to enhance the evidence base, and the evidence generated can enhance the efficiency and effectiveness of service delivery to some of the most disadvantaged communities in Australia.

7. References

7.1 General references

  • Adamson, D. (2010). The impact of devolution: Area-based regeneration policies in the UK. York: Joseph Rowntree Foundation.
  • Amion Consulting. (2010). Evaluation of the National Strategy for Neighbourhood Renewal: Final report. London: Department for Communities and Local Government.
  • Austin, M. J., & Lemon, K. (2005). Promising programs to serve low-income families in poverty neighbourhoods. Journal of Health and Social Policy, 21, 65-94.
  • Australian Social Inclusion Board. (2011). Governance models for location based initiatives. Canberra: Department of the Prime Minister and Cabinet.
  • Barnes, J. (2007). Targeting deprived areas: The nature of the Sure Start Local Programme neighbourhoods. In J. Belsky, J. Barnes, & E. Melhuish (Eds.), The National Evaluation of Sure Start: Does area-based early intervention work? (pp. 25-44).Bristol: The Policy Press.
  • Batty, E., Beatty, C., Foden, M., Lawless, P., Pearson, S., & Wilson, I. (2010). The New Deal for Communities experience: A final assessment (The New Deal for Communities Evaluation: Final Report, Vol. 7). London: Department for Communities and Local Government.
  • Beatty, C., Foden, M., Lawless, P., & Wilson, I. (2010). Exploring and explaining change in regeneration schemes: Evidence from the New Deal for Communities Programme (The New Deal for Communities National Evaluation: Final report, Vol. 5). London: Department for Communities and Local Government.
  • Beatty, C., Lawless, P., Pearson, S., & Wilson, I. (2009). Residential mobility and outcome change in deprived areas: Evidence from the New Deal for Communities Programme. London: Department for Communities and Local Government.
  • Belsky, J., Melhuish, E., Barnes, J., Leyland, A. H., Romaniuk, H., & the National Evaluation of Sure Start Research Team. (2006). Effects of Sure Start local programmes on children and families: Early findings from a quasi-experimental, cross sectional study. BMJ, 332, 1476-1578. doi: 10.1136/bmj.38853.451748.2F.
  • Burton, P., Croft, J., Hastings, A., Slater, T., Goodlad, R., Abbott, J., & Macdonald, G. (2004). What works in community involvement in area-based initiatives? A systematic review of the literature (Online Report, 53/04). London: Research Development and Statistics Directorate.
  • Burton, P., Goodland, R., & Croft, J. (2006). How would we know what works? Context and complexity in the evaluation of community involvement. Evaluation, 12, 294-312.
  • Campbell, M., & Meadows, P. (2001). What works locally? Key lessons on local employment policies. York: York Publishing Services.
  • Chatterton, P., & Bradley, D. (2000). Bringing Britain together: The limitations of area-based regeneration policies in addressing deprivation. Local Economy, 15, 98-111.
  • Coote, A., Allen, J., & Woodhead, D. (2004). Finding out what works: Building knowledge about complex community-based initiatives. London: Kings Fund.
  • Cytron, N. (2010). Improving the outcomes of place-based initiatives. Community Investments, 22, 21-25.
  • Department of Families, Housing, Community Services and Indigenous Affairs. (2013). National Partnership Agreement on Remote Indigenous Housing (NPARIH): Review of progress (2008-2013). Canberra: FaHCSIA. Retrieved from<tinyurl.com/lhty9zn>.
  • Dobbie, W., & Fryer. R. G. (2010). Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem Children's Zone. American Economic Journal: Applied Economics, 3, 158-187.
  • Dyson, A., Jones, L., & Kerr, K. (2009, 2-4 February). Inclusion and social disadvantage in the English education system: The role of area-based initiatives. Paper presented at the International Research Forum, A Comparative Analysis of Equity in Inclusive Education. Stanford University, California.
  • Dyson, A., Kerr, K., & Raffo, C. (2012). Area-based initiatives in England: Do they have a future? Revue française de pédagogie, 178, 27-38.
  • ECOTEC Research and Consulting Ltd. (2010). Evaluation of the National Strategy for Neighbourhood Renewal: Local research project. London: Department for Communities and Local Government.
  • Edwards, B., Mullan, K., Katz, I., & Higgins, D. (2014). The Stronger Families in Australia (SFIA) Study: Stage 2 (Research Report No. 29). Melbourne: Australian Institute of Family Studies.
  • Galster, G., Temkin, K., Walker, C., & Sawyer, N. (2004). Measuring the impacts of community development initiatives: A new application of the adjusted interrupted time-series method. Evaluation Review, 28, 501-538.
  • Ghate, D., Asmussen, K., Tian, Y., Hauari, H., & Policy Research Bureau. (2008). "On Track" Phase Two national evaluation: Reducing risk and increasing resilience. How did "On Track" work? (Research Report No. DCSF-RR035). London: Department for Children, Schools and Families.
  • GHK (2003). Ex-post evaluation: URBAN Community Initiative (1994-1999). London & Belgium: GHK. Retrieved from <ec.europa.eu/regional_policy/sources/docgener/evaluation/urban/urban_expost_evaluation_9499_en.pdf>.
  • Goering, J., Kraft, J., Feins, J. D., McInnis, D., Holin, M. J., & Elhassan, H. (1999). Moving To Opportunity for Fair Housing Demonstration Program: Current status and initial findings. Washington, DC: US Department of Housing and Urban Development.
  • Griggs, J., Whitworth, A., Walker, R., McLennan, D., & Noble, M. (2008). Person- or place-based policies to tackle disadvantage? Not knowing what works. York: Joseph Rowntree Foundation.
  • Hales, J., Taylor, R., Mandy, W., & Miller, M. (2003). Evaluation of employment zones: Report on a cohort survey of long-term unemployed people in the zones and a matched set of comparison areas. London: National Centre for Social Research.
  • Harlem Children's Zone. (2014). History. New York: Harlem Children's Zone. Retrieved from <www.hcz.org/about-us/history>.
  • Hulse, K., Jacobs, K., Arthurson, K., & Spinney, A. (2011). At home and in place? The role of housing in social inclusion (AHURI Final Report No. 177). Melbourne: Australian Housing and Urban Research Institute.
  • Hutchings, M., Greenwood, C., Hollingworth, S., Mansaray, A., Rose, A., Minty, S., & Glass, K. (2012). Evaluation of the City Challenge programm. (Research Report No. DFE-RR215). London: Department for Education.
  • Katz, B. (2004). Neighbourhoods of choice and connection: The evolution of American neighbourhood policy and what it means for the United Kingdom. York: Joseph Rowntree Foundation.
  • Kendall, L., O'Donnell, L., Golden, S., Ridley, K., Machin S., Rutt, S., et al. (2005). Excellence in cities: The national evaluation of a policy to raise standards in urban schools 2000-2003 (Research Report No. RR675a). London: Department for Education and Skills.
  • Lawless, P. (2004). Locating and explaining area-based urban initiatives: New Deal for Communities in England. Environment and Planning C: Government and Policy, 22(3), 383-399.
  • Lawless, P. (2012). Can area-based regeneration programmes ever work? Evidence from England's New Deal for Communities Programme. Policy Studies, 33, 313-328.
  • Lawless, P., & Pearson, S. (2012). Outcomes from community engagement in urban regeneration: Evidence from England's New Deal for Communities Programme. Planning Theory & Practice, 13, 509-527.
  • Ludwig, J., Kling, J. R., & Mullainathan, S. (2011). Mechanism experiments and policy evaluations. Journal of Economic Perspectives, 25(3), 17-38.
  • Malin, N., & Morrow, G. (2008). Evaluating Sure Start: Interprofessionalism and parental involvement in local programmes. London: Whiting & Birch Ltd.
  • McGregor, A., Glass, A., Higgins, K., Macdougall, L., & Sutherland, V. (2003). Developing people - regenerating place: Achieving greater integration for local area regeneration. Bristol: The Policy Press.
  • Meadows, P. (2006). Cost effectiveness of implementing SSLPS: An interim report (Research Report No. NESS/2006/FR/015). London: Birkbeck, University of London. Retrieved from <www.ness.bbk.ac.uk/cost-effectiveness/documents/1240.pdf>.
  • Meadows, P. (2008). Local initiatives to help workless people find and keep paid work. York: Joseph Rowntree Foundation.
  • Modarres, A. (2002). Persistent poverty and the failure of area-based initiatives in the US. Local Economy, 17, 289-302.
  • Muir, K., Katz, I., Purcal, C., Patulny, R., Flaxman, S., Abello, D., et al. (2009). National evaluation (2004-2008) of the Stronger Families and Communities Strategy 2004-2009 (Occasional Paper No. 24). Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • National Evaluation of Sure Start Team. (2011). National evaluation of Sure Start local programmes: An economic perspective. (Research Report No. DFE-RR073). London: Department for Education. Retrieved from <www.ness.bbk.ac.uk/cost-effectiveness/documents/DFE-RR073_full.pdf>.
  • National Evaluation of Sure Start Team. (2012). The impact of Sure Start Local Programmes of seven year olds and their families (Research Report No. DFE-RR220). London: Birkbeck, University of London. Retrieved from <www.ness.bbk.ac.uk/impact/documents/DFE-RR220.pdf>.
  • New Zealand Ministry of Social Development. (2004). Whānau Development Project: Final evaluation report. Wellington: NZ Ministry of Social Development.
  • NNI Research Team. (2007). National evaluation of the Neighbourhood Nurseries Initiative: Integrated report (Research Report No. SSU/2007/FR/024). Nottingham: Department for Education and Skills.
  • O'Dwyer, L. A., Baum, F., Kavanagh, A., & Macdougall, C. (2007). Do area-based interventions to reduce health inequalities work? A systematic review of evidence. Critical Public Health, 17, 317-335.
  • Parkinson, M., (1998). Combating social exclusion: Lessons from area-based programmes in Europe. Bristol: The Policy Press.
  • Power, S., Rees, G., & Taylor, C. (2005). New labour and educational disadvantage: The limits of area-based initiatives. London Review of Education, 3, 101-116.
  • Promise Neighborhoods Institute. (2011). Promise Neighborhoods Institute Network sites. Oakland, CA: PolicyLink. Retrieved from <tinyurl.com/ljf353k>.
  • Rae, A. (2011). Learning from the past? A review of approaches to spatial targeting in urban policy. Planning Theory and Practice, 12, 331-348.
  • Rudd, P., Poet, H., Featherstone, G., Lamont, E., Durbin, B., Bergeron C., et al. (2011). Evaluation of City Challenge Leadership Strategies: Overview report. Slough: National Foundation for Educational Research.
  • Sampson, R. J. (2012a). Great American City: Chicago and the enduring neighborhood effect. Chicago: University of Chicago Press.
  • Sampson, R. J. (2012b). Moving and the neighborhood glass ceiling. Science, 337, 1464-1465.
  • Sanbonmatsu, L., Ludwig, J., Katz, L, Gennetian, L., Duncan, G., Kessler R., et al. (2011). Moving to Opportunity for Fair Housing Demonstration Program: Final impacts evaluation. Washington, DC: US Department of Housing and Urban Development Office of Policy Development and Research.
  • Sharkey, P. (2013). Stuck in place: Urban neighborhoods and the end of progress toward racial equality. Chicago: University of Chicago Press.
  • Sullivan, H., & Stewart, M. (2006). Who owns the theory of change? Evaluation: The International Journal of Theory, Research and Practice, 12, 179-199.
  • Taylor, M., & Edwards, B. (2012). The early implementation of the Helping Young Parents measure: Young parents and their children's development. Melbourne: Australian Institute of Family Studies. (NOT PUBLICLY AVAILABLE.)
  • Thomson, H. (2008). A dose of realism for healthy urban policy: Lessons from area-based initiatives. Journal of Epidemiology and Community Health, 62, 932-936.
  • UK House of Commons. (2003). Housing, planning, local government and the regions: Seventh report, the effectiveness of government regeneration initiatives. London: HMSO.
  • Ware, V., Gronda H., & Vitis, L. (2010). Addressing locational disadvantage effectively. Melbourne: Australian Housing and Urban Research Institute.
  • Warin, J. (2007). Joined-up services for young children and their families: Papering over the cracks or re-constructing the foundations? Children & Society, 21, 87-97.
  • Whitehurst, G. J., & Croft, M. (2010). The Harlem Children's Zone, Promise Neighbourhoods, and the broader, bolder approach to education. Washington DC: Brookings Institution, Brown Center on Educational Policy. Retrieved from <tinyurl.com/k3odv2p>.
  • Wilkinson, K., & McLennan, D. (2010). Narrowing the gap: Analysing the impact of the New Deal for Communities Programme on educational attainment. London: Communities and Local Government.
  • Wilkinson, K., Whitworth, A., & McLennan, D. (2010). Evaluation of the National Strategy for Neighbourhood Renewal: Improving educational attainment in deprived areas. London: Department for Communities and Local Government.

7.2 Supplementary reference list for international place-based initiatives

  • a National Evaluation of Sure Start Team. (2012). The impact of Sure Start Local Programmes on seven year olds and their families (Research Report No. DFE-RR220). London: Birkbeck, University of London. Retrieved from <www.ness.bbk.ac.uk/impact/documents/DFE-RR220.pdf>.
  • b Belsky, J., Melhuish, E., Barnes, J., Leyland, A. H., Romaniuk, H., & the National Evaluation of Sure Start Team (2006). Effects of Sure Start local programmes on children and families: Early findings from a quasi-experimental, cross sectional study. BMJ, 332, 1476-1578.
  • c National Evaluation of Sure Start Team. (2001). National Evaluation of Sure Start: Methodology report. Executive summary. Retrieved from <www.ness.bbk.ac.uk/documents/Methodology.pdf>.
  • d National Evaluation of Sure Start Team. (2008). The impact of Sure Start Local Programmes of three year olds and their families (Research Report No. NESS/2008/FR/027). London: Birkbeck, University of London. Retrieved from <tinyurl.com/pbabtu2/>.
  • e National Evaluation of Sure Start Team. (2010). The impact of Sure Start Local Programmes of five year olds and their families (Research Report No. DFE-RR067). London: Birkbeck, University of London. Retrieved from <www.ness.bbk.ac.uk/impact/documents/RR067.pdf>.
  • f Eisenstadt, N. (2011). Providing a Sure Start: How government discovered early childhood. Bristol: The Policy Press.
  • g Lomas, H., & Hannon, P. (2005). Community involvement. In J. Weinberger, C. Pickstone, & P. Hannon (Eds.), Learning from Sure Start: Working with young children and their families (pp. 193-204). Berkshire: Open University Press.
  • h National Evaluation of Sure Start Team. (2011). National evaluation of Sure Start local programmes: An economic perspective (Research Report No. DFE-RR073). London: Birkbeck, University of London. Retrieved from <tinyurl.com/l993er3>.
  • i Barnes, J. (2007). Targeting deprived areas: The nature of the Sure Start Local Programme neighbourhoods. In J. Belsky, J. Barnes, & E. Melhuish (Eds.), The National Evaluation of Sure Start: Does area-based early intervention work? (pp. 25-44).Bristol: The Policy Press.
  • j Malin, N., & Morrow, G. (2008). Evaluating Sure Start: Interprofessionalism and parental involvement in local programmes. London: Whiting & Birch Ltd.
  • k Ellison, S., Hicks, L., & Latham, P. (n. d.). Cost effectiveness in Sure Start Local Programmes: A synthesis of local evaluation findings. Retrieved from <www.ness.bbk.ac.uk/support/documents/1287.pdf>.
  • l Hales, J., Taylor, R., Mandy, W., & Miller, M. (2003). Evaluation of Employment Zones: Report on a cohort survey of long-term unemployed people in the zones and a matched set of comparison areas. London: National Centre for Social Research.
  • m The Policy Research Centre. (2006). Phase 2 evaluation of Multiple Provider Employment Zones: Qualitative study (Research Report No. 399). Leeds: Department for Work and Pensions.
  • n Griggs, J., Whitworth, A., Walker, R., McLennan, D., & Noble, M. (2008). Person- or place-based policies to tackle disadvantage? Not knowing what works. York: Joseph Rowntree Foundation.
  • o Carpenter, J. (2006). Addressing Europe's urban challenges: Lessons from the EU URBAN Community Initiative. Urban Studies, 43, 2145-2162.
  • p GHK. (2003). Ex-post evaluation: URBAN Community Initiative (1994-1999). London & Belgium: GHK. Retrieved from <tinyurl.com/lrrdbxj>.
  • q Batty, E., Beatty, C., Foden, M., Lawless, P., Pearson, S., & Wilson, I. (2010). The New Deal for Communities experience: A final assessment (The New Deal for Communities Evaluation: Final report, Vol. 7). London: Department for Communities and Local Government.
  • r Sheffield Hallam University. (2005). New Deal for Communities 2001-2005: An interim evaluation (Research Report No. 17). London: Office of the Deputy Prime Minister.
  • s Lawless, P. (2012). Can area-based regeneration programmes ever work? Evidence from England's New Deal for Communities Programme. Policy Studies, 33, 313-328.
  • t Lawless, P., & Pearson, S. (2012). Outcomes from community engagement in urban regeneration: Evidence from England's New Deal for Communities Programme. Planning Theory & Practice, 13, 509-527.
  • u Defilippis, J., Fisher, R., & Shragge, E. (2006). Neither romance nor regulation: Re-evaluating community. International Journal of Urban and Regional Research, 30, 673-689.
  • v Beatty, C., Lawless, P., & Wilson, I. (2011). The Composite Index of Relative Change (CIRC): Identifying change in NDC areas. NDC National Evaluation Phase 2. South Yorkshire: Centre for Regional Economic and Social Research.
  • w Beatty, C., Lawless, P., Pearson, S., & Wilson, I. (2009). Residential mobility and outcome change in deprived areas: Evidence from the New Deal for Communities Programme. London: Department for Communities and Local Government.
  • x Beatty, C., Brennan, A., Foden, M., Lawless, P., Tyler, P., Warnock, C., & Wilson, I. (2010). The New Deal for Communities Programme: Assessing impact and value for money (The New Deal for Communities National Evaluation: Final report, Vol. 6). London: Department for Communities and Local Government.

7.3 Supplementary reference list for Commonwealth place-based initiatives

  • 1 Edwards, B., Mullan, K., Katz, I., & Higgins, D. (2014). The Stronger Families in Australia (SFIA) Study: Stage 2. (Research Report No. 29). Melbourne: Australian Institute of Family Studies.
  • 2 Muir, K., Katz, I., Purcal, C., Patulny, R., Flaxman, S., Abello, D., et al. (2009). National evaluation (2004-2008) of the Stronger Families and Communities Strategy 2004-2009 (Occasional Paper No. 24). Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 3 Stronger Families Learning Exchange. (2005). Communities for Children initiative. Stronger Families Learning Exchange Bulletin, 7, 3-5.
  • 4 Access Economics. (2010). Positive family functioning. Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 5 Queensland Government. (2011). Cape York Welfare Reform trial extension: Consultation report. Brisbane: Department of Aboriginal and Torres Strait Islander and Multicultural Affairs.
  • 6 Courage Partners. (2009). Evaluation framework and program theory for the Cape York Welfare Reform trial. Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 7 Department of Families, Housing, Community Services and Indigenous Affairs. (2012a). Cape York Welfare Reform: Evaluation. Canberra: FaHCSIA. Retrieved from <tinyurl.com/kc327rf>.
  • 8 Colmar Brunton Social Research. (2012). Social change survey. In Cape York Welfare Reform: Evaluation (Chapter 4). Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 9 Social Policy Research Centre. (2012). Implementation. In Cape York Welfare Reform: Evaluation (Chapter 3). Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 10 KPMG. (2010). Implementation review of the Family Responsibilities Commission: Final report. Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 11 Department of Families, Housing, Community Services and Indigenous Affairs. (2012b). Introduction. In Cape York Welfare Reform: Evaluation (Chapter 2). Canberra: FaHCSIA.
  • 12Newman reverses decision to cut Cape York welfare funds. (2013, 29 March). The Age. Retrieved from <tinyurl.com/pn2dfog>.
  • 13 Putt, J. (2012). Service delivery, results from the survey of service providers: Report for the evaluation of the Cape York Welfare Reform trial. Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 14 Social Policy Research Centre, & Department of Families, Housing, Community Services and Indigenous Affairs. (2012). Outcomes. In Cape York Welfare Reform: Evaluation (Chapter 8). Canberra: FaHCSIA.
  • 15 Limerick, M. (2012). Evaluation overview. In Cape York Welfare Reform: Evaluation (Chapter 1). Canberra: Department of Families, Housing, Community Services and Indigenous Affairs.
  • 16 Department of Families, Housing, Community Services and Indigenous Affairs. (2012c). Project performance summary - July 2008 to December 2011. In Cape York Welfare Reform: Evaluation (Appendix B). Canberra: FaHCSIA.
  • 17 Cape York Institute for Policy and Leadership. (2007). From hand out to hand up: Volume 2. Cape York Welfare Reform project. Cairns, Qld: Cape York Institute for Policy and Leadership. Retrieved from <cyi.org.au/wp-content/uploads/2012/03/From-Hand-Out-to-Hand-Up-Volume-2.pdf>.
  • 18 Healthcare Planning and Evaluation Pty Ltd. (2010). Evaluation of Support for Day to Day Living in the Community: A structured activity program. Final report. Canberra: Department of Health and Ageing.
  • 19 Darcy, M., Gwyther, G., Perry, J., Wood, J., & Richardson, R. (2009). Centrelink Place Based Services Program evaluation. Penrith South: University of Western Sydney. (NOT PUBLICLY AVAILABLE.)
  • 20 Australian National Audit Office. (2009). 5. Centrelink: Place Based Services Initiative. Canberra: ANAO. Retrieved from <www.anao.gov.au/bpg-innovation/case-5.html>.
  • 21 Darcy, M., & Gwyther, G. (2010, 5-7 July). Centrelink's "Place Based Services": Can a national service delivery agency address local needs and conditions? Proceedings of the Social Policy Association Conference, University of Lincoln, UK.
  • 22 Department of Education, Employment and Workplace Relations. (2011). Building Australia's Future Workforce package: Overarching evaluation strategy. Retrieved from <docs.employment.gov.au/system/files/doc/other/final_bafw_evaluation_strategy_.pdf>.
  • 23 Department of Social Services. (2013). Building Australia's Future Workforce (BAFW) package: Evaluating place-based income management. Retrieved from <tinyurl.com/ll8eodh>.
  • 24 Department of Employment. (2013). Building Australia's Future Workforce: Interim evaluation report. 2.6 People in areas of entrenched disadvantage. Canberra: Department of Employment. (NOT PUBLICLY AVAILABLE.)
  • 25 Department of Employment. (2013). Building Australia's Future Workforce evaluation: Data sources. Canberra: Department of Employment. (NOT PUBLICLY AVAILABLE.)
  • 26 Department of Employment. (2013). The Building Australia's Future Workforce package: 1. Overview. Canberra: Department of Employment. (NOT PUBLICLY AVAILABLE.)
  • 27 Department of Families, Housing, Community Services and Indigenous Affairs. (2013). National Partnership Agreement on Remote Indigenous Housing (NPARIH): Review of progress (2008-2013). Canberra: FaHCSIA. Retrieved from<tinyurl.com/lhty9zn>.
  • 28 Origin Consulting. (2013). Whole of strategy evaluation of the Petrol Sniffing Strategy: Future directions for the PSS. Final report. Retrieved from <tinyurl.com/ojhygoh>.
  • 29 Tempo Strategies. (2012). Groote Eylandt and Bickerton Island Regional Partnership Agreement: Progress evaluation. Retrieved from <www.dss.gov.au/sites/default/files/documents/09_2012/evaluation_of_the_rpa_0.pdf>.
  • 30 Bryant, C., & Willis, M. (2009). Pornography awareness: A process of engagement with Northern Territory Indigenous communities (AIC Reports Technical and Background Paper No. 34). Canberra: Australian Institute of Criminology.
Appendices

Appendix A: Approach to the international literature review

The review's focus on international studies situates the project within a broader context, exploring how Australian approaches compare or contrast with best practice approaches overseas, and understanding the transferability of international best practice findings to the Australian context. In developing an understanding of "what works" for international PBIs, the literature review considers the international initiatives' themes of disadvantage, program rationales/theories of change, target populations, geographic scales of implementation, governance, outcomes and evaluation. As a recent report addresses governance issues and makes recommendations as to key governance elements of PBIs (Australian Social Inclusion Board, 2011), this area will not be as much of a focus as the others.

Although conducted using formal literature search techniques, the review better represents a conceptual policy review, focusing on international PBIs. For currency, and to reduce the vast quantity of literature produced, literature released from the year 2000 was prioritised. The research aimed to build upon existing reviews, such as that by Ware et al. (2010), in which the authors identified the need for locational disadvantage research, policy and practice examples "from a broader range of Anglophone countries" (p. 53). Nevertheless, the most relevant literature was derived from the United Kingdom, the United States and the European Union, with far less literature available from countries such as Canada and New Zealand.

Reflecting the socio-spatial nature of complex neighbourhood and community-level problems, the literature review is informed by a mix of "place-focused" and "people-focused" disciplines. Studies from urban policy, urban planning and neighbourhood renewal/regeneration fields were sought alongside literature from social science, social policy, health and community development domains. The types of publications sourced for review included academic journals, government policy documents, research and evaluation reports and briefs, conference papers and books. Given that not all evaluations are formally published, peer-reviewed texts were not prioritised, and Internet searches were also run in an attempt to capture informally published evaluations and other "grey literature" falling outside conventional channels.

The search and selection process followed a three-stage system.

Stage 1: Searching by key search terms

In Stage 1, an initial search was undertaken using combinations of key search terms in all databases available through AIFS access to the EBSCO subscription service (Academic Search Premier, Aust./NZ Reference Centre, Business Source Premier, Psychology and Behavioural Sciences Collection, SocIndex, Women's Studies International, E journals, PsychInfo and Regional Business News), as well as the AIFS library catalogue and the Informit Australian Family & Society Abstracts database. The key search terms used were: place-based, area-based, location based, community services, human services, social services, family service, best practice, program evaluation, policy, social policy, policy analysis, practice, what works, lessons, efficacy, effective, gold standard, disadvantage, community-level, community-based, locational, initiative, intervention, approach, community development, neighbourhood/neighborhood, concentrated disadvantage, social exclusion and social inclusion. These terms reflect the variety of ways in which these types of initiatives are referred to in the literature; for example, place-based initiatives are called "area-based initiatives" or "ABIs" in the UK, and more commonly described as "neighborhood" initiatives in the US.

This search generated approximately 110 Australian and international references, of which Australian sources comprised around 25% of the literature. The found literature included research and policy briefs, reports and journal articles by academics and social policy research organisations (including the UK Joseph Rowntree Foundation and the US Brookings Institution), government policy documents, and evaluation reports and toolkits, as well as systematic reviews and syntheses of "what works".

Within this literature some 32 international PBIs were identified for further consideration. These PBIs - five from the US; nine from EU countries; 16 from the UK (including 12 specific to England, two specific to Scotland and two whose distribution spanned more than one country within the UK); one from New Zealand and one from Canada - had the characteristic of being repeatedly mentioned across several review articles. Appendix B, Table B1 (column 2), shows the initial list of international PBIs considered for review.

In order to discount material that was not relevant, the initial search was supplemented by sorting. As the review was particularly concerned with studies of the effectiveness of national-scale PBIs, material was excluded if it only:

  • described the case for PBIs;
  • described the evolution of place-based policy over time; or
  • evaluated a smaller aspect of a national PBI, such as a single local program.

In addition, national evaluation reports for several Australian PBIs, which represented primary materials for the second phase of the project, were filed for later use and excluded from the literature review.

Stage 2: Searching by specific program names

In Stage 2, a second search was undertaken using the specific place-based program names generated by Stage 1 (e.g., Sure Start Local Programs), with a focus on sourcing final evaluation reports for each of the international PBIs. Other reviews of place-based literature have also found the need for this refinement of search terms:

The expression "area-based initiative" was not on its own especially helpful and it was necessary to rely more on the specific or proper names given to particular ABIs, such as City Challenge or New Deal for Communities. It was therefore decided to focus upon the results which were gained from the searches conducted on the proper nouns of specific ABIs. (Burton et al., 2004, p. 5)

The second stage search process generated around 40 additional references, with many originating from the UK. This vast literature was highly segregated, reflecting a large number of standalone evaluation reports for each PBI. National evaluations tend to be multifaceted and phased in over time, with multiple qualitative and quantitative aspects targeting an initiative's different stakeholders, target audiences and implementation strands. Frequently, each of these aspects is evaluated and reported separately, resulting in the numerous evaluations identified. Furthermore, the search also produced evaluations of single, local programs within a larger national initiative.

Another issue was the duplication of information by academic journal articles and research briefs, which tended to provide summaries of findings from national evaluation reports from a particular research angle or perspective. In order to retain control of the workload and reduce potential "double-handling" of information, a second-level sort was undertaken. In this sort:

  • original copies of final national evaluation reports were prioritised over summaries;
  • evaluation reports that integrated and synthesised standalone evaluations were favoured; and
  • evaluation reports focusing on a smaller aspect of a national PBI were again excluded.

During the second stage sort, PBIs from Belgium, Germany, France, Sweden and the Netherlands were excluded from review due to their having no publicly available, final evaluation reports written in English. This reduced the list of international initiatives for review from 32 to 25 (refer to Appendix D for a tabular summary of these 25 initiatives). Further reduction and some substitution occurred upon closer examination of the evaluation literature sourced. While summaries of the implementation and highlights of evaluation findings were sourced for Canada's New Deal for Cities and Communities initiative, Scotland's Scottish New Community Schools initiative and England's Action Teams For Jobs initiative, no in-depth, final evaluation reports were located. For this reason these initiatives were also excluded from review. No evaluation report was sourced for the Promise Neighborhoods program in the United States; however, a 2010 evaluation report for the initiative's predecessor, the Harlem Children's Zone, was located and included for review. Denmark's Kvarterloft initiative and the US Partnership for Sustainable Communities initiative were both excluded from review as the extent of evaluation materials available was lacking - English-language summaries of program achievements were the only materials sourced.

Further, a small number of national evaluation reports were no longer publicly available due to the dissolution of government departments and the archiving of their websites and content. For example, the New Zealand Government's Centre for Social Research and Evaluation, responsible for the evaluation of its Community Links initiative, no longer exists. Its overseer, the Ministry of Social Development, has not since made the initiative's evaluation report publicly available on its website, and so the Community Links initiative was excluded from review. During the process of searching for the Community Links evaluation report, however, evaluation materials for a different New Zealand place-based initiative, the Whānau Development Project, were located. This initiative was substituted in place of Community Links as a representative of New Zealand policy. This final process of sorting and searching retained 20 of the original list of 32 international PBIs identified in Stage 1 (see Table B1 of Appendix B).

Stage 3: Classifying and refining the international literature

Stage 3 represented the final phase of finding and refining literature. This phase sought to select from the list of international PBIs several case studies most relevant to the Australian context. It occurred alongside a formal request for information from selected Australian Government departments, a process through which Australian PBIs were identified as being potentially relevant to the broader study. A recurring question during communications with government departments was, "How can we classify place-based initiatives?", and this question was front of mind in deciding how to select case studies for review. Ultimately, the chosen international case studies would maximise the relevance and potential transferability of place-based best practice findings to the Australian context.

At this point, classifications and definitions suggested by the literature were taken into account, and a quadrant-based model of classification by Griggs et al. (2008, pp. 2-3) was applied to each of the 20 remaining international PBIs. Griggs et al.'s model classifies PBIs according to their policy objective and targeting relating to place and person (see Figure A1). Five criteria were used to determine the initiatives' classification into the Griggs et al. model and a blind test was undertaken to ensure objectivity of classification.

Figure A1: Quadrant-based model for classifying place-based initiatives

Figure A1: Quadrant-based model for classifying place-based initiatives

Source: Griggs et al. (2008, p. 3)

The five criteria were:

  • the initiative's focus, or "theme of disadvantage";
  • the initiative's program rationale, or "theory of change";
  • the types of services the initiative delivers;
  • the initiative's target population; and
  • the initiative's target location or area.

A challenge of applying Griggs et al.'s (2008) classification arose in relation to the initiative's perceived effects. A few Type 2 initiatives (those having a "major focus on place in order to impact person") may have indeed had unintentional outcomes on place, simply by nature of their focus on place. Ultimately, our classification depended on the articulation of the initiative's intended effects on place and/or person, as explicitly stated in its program rationale.

Of the 20 remaining international PBIs:

  • 0 were classified as policy Type 1 (major focus on place to impact place);
  • 4 were classified as policy Type 2 (major focus on place to impact person);
  • 0 were classified as policy Type 3 (major focus on person to impact place);
  • 1 was classified as policy Type 4 (major focus on person to impact person); and
  • 15 were classified as policy Type 5 (simultaneous focus on twin goals of place and person in order to impact both).

Given that the Commonwealth Place-based Service Delivery Initiatives: Key Learnings project aimed to consider place-based service delivery initiatives (the inclusion of "service" implying an emphasis on social policy initiatives), we applied the classification by Griggs et al. (2008) to identify the kinds of PBIs that might be most relevant to the project. The project's dual focus on place and person suggests that initiatives classified by Griggs et al. (2008) as Types 2, 3 or 5 would be most relevant to the project, as they include consideration of both place and person. Type 1 initiatives were likely to be less relevant, as they focus strictly on place and have no intended social targeting or effect. Likewise, Type 4 initiatives demonstrate no focus or intended effect on place, so were less relevant to the project. Further, as indicated above, in the 20 initiatives, we did not find any Type 3 initiatives, which focus on person to impact place. Of the remaining 16 initiatives, 12 initiatives were selected for close review (as designated in the final column of Table B1, Appendix B), in order to further demonstrate instances of common practice among Type 5 and Type 2 initiatives.

Appendix B: International place-based initiatives identified by the literature review

Table B1: International place-based initiatives identified by the literature review
Country Programs identified in Stage 1 Policy classification Retained?
Stage 2 sort Stage 3 sort
Notes: † Types 1-5, based on Griggs et al. (2008). * Whānau Development Project evaluation report sourced as an alternative. ** Harlem Children's Zone evaluation report sourced as alternative.
Belgium Sociaal Impulsfond - X X
Canada New Deal for Cities and Communities - X X
Denmark Kvarterloft - X X
England Sure Start Type 5
Aimhigher Type 2 X
Excellence in Cities (EiC) Type 5
Children's Fund Type 5 X
On Track Type 5
Early Excellence Centre Pilot Programme Type 5 X
Neighbourhood Nurseries Initiative (NNI) Type 2
Education Action Zones Type 5 X
City Challenge Type 5
Health Action Zones Type 5 X
National Strategy for Neighbourhood Renewal (NSNR) Type 5
New Deal for Communities (NDC) Type 5
England, Scotland, Wales Employment Zones (EZs) Type 2
Action Team for Jobs - X X
European Union EU URBAN Community Initiative (URBAN I CI) Type 5
France Politique de la Ville - X X
Zones Urbaines Sensible ("Sensitive Urban Areas") - X X
Germany Die Soziale Stadt ("The Social City") - X X
New Zealand Community Links Type 5 ✓ *
Scotland Scottish New Community Schools - X X
Social Inclusion Partnerships - formerly Priority Partnership Areas and Regeneration Programme Areas Type 5 X
Sweden Metropolitan Development Initiative/Local Development Agreement - X X
Stortstadssattsningen - X X
The Netherlands Grote-Stedenbeleid ("Big Cities") Programme - X X
United States Moving to Opportunity (MTO) Fair Housing Demonstration Type 2
Promise Neighborhoods - federal expansion of Harlem Children's Zone (HCZ) Type 5 ✓ **
Partnership for Sustainable Communities - X X
Hope VI Type 5 X
Early Head Start Program Type 4 X

Appendix C: Case studies of four international place-based initiatives

Sure Start Local Programs (SSLPs)

Program description: The SSLPs initiative was launched in the UK in 1999, with its ultimate goal being to enhance the life opportunities for young children growing up in disadvantaged neighbourhoods who were at risk of performing poorly at school, experiencing trouble with peers and authority figures, and ultimately facing compromised life chances.a * This initiative was a place-based program, with all children under five years of age and their families living in a prescribed area being the "targets" of the intervention. All SSLPs were expected to provide the core services of: family support; outreach or home visiting; support for quality play, learning and child care experiences; primary and community health care; advice about family and child health and development; and support to those with special needs.b

Evaluation design: The National Evaluation of Sure Start was based upon the premise that the initiative represented an effort to change existing services, by reshaping, enhancing, adding value and increasing coordination, rather than providing a specific service. Three core questions have guided the overall evaluation of Sure Start:

  • Did the existing services change?
  • Did the services that were delivered improve?
  • Did children, families and communities benefit?

The NESS addressed these three questions through five components:

  • implementation evaluation;
  • local community context analysis;
  • cost-benefit analysis;
  • support for local evaluations; and
  • impact evaluation.c

The NESS impact study followed over 5,000 children and their families in 150 SSLP areas, who were initially recruited when the children were 9 months old and followed up at 3, 5 and 7 years of age. A comparison group of children and families were also selected. This study concluded in 2012, with the publication of a report on the effects of SSLPs on children, families and communities.a

Program outcomes: The 2012 NESS impact evaluationa found that SSLPs had some beneficial effects on family functioning and wellbeing that persisted until the children were 7 years of age, including mothers engaging in less harsh discipline and providing a more stimulating home learning environment for their children. However, the modest benefits for SSLP children that were previously present at 3 and 5 years of age (e.g., higher levels of self-regulation and better physical health) were no longer present at 7 years of age, which was proposed to be at least partly attributable to the introduction of free part-time preschool in England.a,d,e

Table C1: Did the SSLPs initiative demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Demonstrated Not evaluated
  • When first implemented in 1999, SSLPs had a high level of flexibility in service delivery. This was reduced in 2005-06, when SSLPs came under Local Authority Control and were required to implement a children's centre model. While the guidelines became more specific about the services to be offered, they were not fully prescribed and SSLPs still had some autonomy.a
  • The extent to which these varying levels of flexibility affected service delivery was not formally evaluated.
Local autonomy Demonstrated Not evaluated
  • It was proposed that community participation was central to SSLPs, through local partnerships that brought together people who were concerned with children in the local community, including health, social and educational services, the private and voluntary sectors, and parents.b
  • It has been reported that the original design of Sure Start allowed very high levels of community control and therefore diversity of delivery. This was on the basis that "better outcomes for children will result from building social capital in communities" (p. 149).f
  • Eisenstadt suggested that a more measured approach would have been likely to result in a better quality program, but would have reached significantly fewer children, due to the profile generated for Sure Start through the "community control" approach.f
  • Local autonomy does not appear to have been evaluated as part of the NESS at an overall level, although it may have been investigated at an individual SSLP level through the Local Evaluation module.
Capacity development Demonstrated Evaluated
  • Capacity development was a focus at the community level, with a specific objective of Sure Start being to "involve families in building the community's capacity to sustain the program and thereby create pathways out of poverty" (p. 193).g
  • A specific development and opportunities path was constructed for Sure Start where parents could become increasingly involved and access support and training opportunities, thereby increasing community involvement and input.g
  • In regard to the impact evaluation, local support was provided to SSLPs through expert guidance and advice. Six regional units were created to offer this support, with a NESS website, a central office and regional workshops also integrated into the support program.h
Lead times Demonstrated Not evaluated
  • In January 1999, the first 60 areas to develop a Sure Start program were announced, which served as pilots prior to the program's broader implementation.f
  • From pilot establishment, the program gradually worked towards establishing 250 SSLPs by 2002.f
Long-term focus Demonstrated Not evaluated
  • SSLPs have been in operation for up to 15 years, and 9 years as a minimum (as of early 2014), and have hence involved a sizeable, long-term investment.b
Spatial targeting Demonstrated Not evaluated
  • The SSLPs were predominantly small areas of deprivation with populations under 13,000 people. They were intended to be very local (referred to as "pram-pushing" distance), covering around 800 children under the age of 4 years.f,i
  • When the number of SSLPs reached its peak of 500 in 2004, the program was reaching an estimated 400,000 children.i
Social targeting Demonstrated Not evaluated
  • The initial intent of the program design was to focus on the 20% most deprived areas within the UK, which included approximately 50% of all children in families living below the official poverty line.a
  • With the increase in the number of SSLPs (refer to "Spatial targeting"), the focus went beyond these areas, but they were still defined as disadvantaged according to socio-demographic features, as well as disorder, child health and developmental indicators.i,j
Joined-up working Partially demonstrated Not evaluated
  • Sure Start had a focus on integration and developing partnerships, particularly at the local level within the SSLPs. One example is parents and professionals, who reportedly worked together in an effective consultative manner as part of the Parents' Committee.j
  • Overall, the governance of Sure Start was problematic, given the complexity of cross-government involvement, and tensions between the Treasury and spending departments. The Sure Start Unit (SSU), however, assisted in defining ministerial and departmental roles and responsibilities.f
Governance Demonstrated Not evaluated
  • As indicated in "Joined-up working", the governance arrangements for Sure Start were complex, due to the cross-government nature of the program.f
  • The SSLPs coming under Local Authority Control in 2005-06 also added a layer of complexity, given that there were stricter regulations under which the areas were required to operate. Hence, a balance needed to be achieved between still enabling some level of local autonomy while government accountability increased.a
Table C2: Did the SSLPs evaluation demonstrate the common elements?
Common elements Demonstrated?
Establishing causality Demonstrated
  • The NESS impact module concluded in 2012, with the publication of a report based on the effects of SSLPs on a randomly selected subset of the then 7-year-old children and their families (refer to "Evaluation design" for more information).a
  • A comparison group of non-SSLP children and their families were chosen from the Millennium Cohort Study cohort, who resided in areas with similar characteristics to SSLP areas.a
  • An "intention to treat" design was adopted, under the premise that SSLPs were community-based, and thus, all community members could potentially be beneficiaries of the program.a
  • A methodological limitation of the NESS was that the SSLP areas were not randomly allocated, which made establishing causality more difficult; further, the "matched" comparison areas only took account of observed differences, and the data for SSLP and comparison groups were collected two years apart.a
Attribution Partially demonstrated
  • Although statistical techniques were applied with the intent of attributing specific effects to program exposure, this was problematic, given that differences between the SSLP and comparison areas could have partly reflected changes in the communities (or society in general) over the two-year measurement gap.a
  • Other non-SSLP services that were offered in the SSLP and comparison areas could have also confounded the effects of Sure Start, with the introduction of free part-time preschool across England potentially masking or cancelling out program benefits for school-aged children.a
Theory of change - Articulated Demonstrated
  • The initiative aimed to provide early and sustained support to children aged under 5 living in disadvantaged areas, including improving access to services that support parents, in order to improve children's health, ability to learn, and social and emotional development, and ultimately improve their life chances.a
Theory of change - Measured Partially demonstrated
  • The theory of change was measured by the impact evaluation to some extent, with outcomes in regard to the above aspects (e.g., health and social/emotional development) measured longitudinally.
  • However, the ultimate aim of the initiative - transforming children's life chances - as well as other aims, such as strengthening families and communities,a were beyond the scope of the impact evaluation.
Residential mobility Not demonstrated
  • Residential mobility was not a focus of the impact evaluation, with the emphasis being primarily on increasing the life chances for children growing up in disadvantaged neighbourhoods and decreasing the poverty gap, rather than families moving to more advantaged areas.a
Cost-effectiveness Demonstrated
  • A cost-effectiveness evaluation was one of the five components of the NESS, and aimed to consider the costs of Sure Start and the benefits achieved, both in the intermediate and longer term. A human capital framework was used for determining the range of benefits over time.c
  • A cost-effectiveness synthesis report, however, found that assessing cost-effectiveness was generally regarded by SSLPs and evaluators as the most difficult and labour-intensive element of the local evaluation to undertake, and was affected by concerns about how higher unit costs would be judged, if the associated increased benefits were not immediately apparent.k
  • A 2011 economic evaluation by the NESS Team, indicated that SSLPs cost around £4,680 per eligible child over the period from birth to the age of 4 at (2009-10 prices), and delivered economic benefits of between £279 and £557 per eligible child by 5 years of age. It was noted, however, that the economic benefits of early childhood interventions do not typically emerge until at least 15 years after they begin.h

Employment Zones (EZs)

Program description: EZs were first introduced in 2000, and targeted 15 British areas that had experienced persistent long-term unemployment and social and economic deprivation. EZs represented an "active labour market policy", in which the government directly intervened in order to promote employment.l This initiative involved the mandatory referral of long-term claimants of Jobseeker's Allowance, aged 25 and over, to private sector providers (selected by competitive tender) that provided support and guidance in obtaining employment. Other groups, including some younger jobseekers, lone parents claiming income support, and those with a physical or mental disability that affected their reading or writing, were also eligible to participate.m

Evaluation design: The research design for evaluating the effects of EZs involved a two-stage survey of EZ participants and a comparison sample, with the main purpose being to provide an estimate of any additional employment provided by the program. Analyses of administrative data were also conducted, and it was proposed that these studies would be complemented by a descriptive account of the labour markets in which the EZs were operating.m The evaluation specifically examined the experiences of a cohort of unemployed people who were eligible to participate in the EZ program between November 2000 and January 2001, when the EZs had been in operation for just under one year; this time frame was chosen to capture the program after its initial "setting-up" period.l

Program outcomes: The two-stage survey sample consisted of just over 1,500 people from the EZ and comparison areas. Analyses indicated that over the survey period, participating in EZs resulted in a higher likelihood of being employed in a job for 16 hours per week or more (32% in the EZ program and 24% in comparison areas). However, there were no differences between the EZ and comparison groups at Stage 2 when considering paid work across all levels of hours per week (both at 54%).l It was concluded that the first finding may have been at least partly attributable to EZs funding regime, which emphasised obtaining employment and sustaining it for at least 13 weeks,l although there was some concern that many jobs did not last much longer than this.n

Table C3: Did the Employment Zones initiative demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Demonstrated Not evaluated
  • There were several elements of flexibility in the design and practice of EZs, including participants having some say in how their Personal Job Accounts were spent, meaning they could choose activities that were of direct relevance to them, rather than being allocated to activities based on spaces available.l
  • EZ managers and advisers were also offered flexibility in their roles, which allowed them to apply individual approaches to each client they worked with. They had discretion to decide how to help each client and how much money to spend on them, provided they were able to demonstrate a sufficient level of job outcomes.l
  • The extent to which this flexibility led to the effective operation of EZs or resulted in favourable outcomes for participants was not specifically evaluated.
Local autonomy Not demonstrated Not evaluated
  • Local autonomy did not appear to be a focus of EZs in regard to community engagement and the EZs having direct input into the design and implementation.
  • The flexibility offered to the EZ advisors and clients (see "Flexibility"), however, enabled local people to directly contribute to and potentially enhance their sense of ownership of the program.l
Capacity development Not demonstrated Not evaluated
  • The evaluation report did not indicate if any training or technical assistance was provided to the private sector companies running the EZs, including managers and advisers.
Lead times Demonstrated Partially evaluated
  • Five prototype EZs were established in 1998 in Glasgow, Liverpool/Sefton, North West Wales, Plymouth and South Teesside. These trial programs were voluntary and focused on moving people from supported employment/ training to sustainable employment, as well as providing opportunities for gaining qualifications to improve employability and moving from welfare to self-employment.l
  • The prototypes then informed the design of the 15 "fully-fledged" EZs that were announced in 2000, although the specific outcomes and recommendations of the trial programs were not stipulated.l
Long-term focus Partially demonstrated Not evaluated
  • The 15 EZs started in April 2000, and in 2004 six of the largest zones were converted into Multiple Provider Employment Zones, where up to three (rather than one) contractors could deliver EZ provision.m
Spatial targeting Demonstrated Not evaluated
  • The 15 EZs were localities across the UK, which were initially defined as sets of electoral wards; however, in practice, they tended to reflect Jobcentre catchments.l
  • Funding for the first phase of EZs was anticipated to cover 48,000 participants across the 15 zones over a 15-month period.l
Social targeting Demonstrated Not evaluated
  • The EZs were identified as areas with a high degree of social and economic disadvantage and severe long-term unemployment, which were among the most deprived in Britain.l
  • EZs targeted the long-term unemployed, as well as those vulnerable to unemployment. More specifically, long-term claimants of Jobseeker Allowance aged 25 or over, and those aged 18-24 who were claiming Jobseeker Allowance and would have otherwise returned to New Deal for Young People were mandatorily referred, as were single parents claiming income support, and groups deemed as appropriate for "early entry" into EZs (e.g., people with a physical or mental disability that affected their reading or writing; those whose first language was not English, Welsh or Gaelic; and those with a criminal record).l
Joined-up working Partially demonstrated Partially evaluated
  • Generally, high levels of coordination were reported in the functioning of the EZs between the private sector providers, advisors and EZ participants, although little detail was provided.l
  • The EZs evaluation indicated that the advisors were more supportive and perceived by the clients to have influenced the outcome when a job had been obtained, in comparison to New Deal 25 Plus (another employment program). However, there was wide variability within and across the EZs in regard to the extent that, from clients' perspective, the advisors provided an effective and coordinated service.l
Governance Partially demonstrated Not evaluated
  • The private sector providers in each EZ had devolved decision-making power, with managers and advisers having the capacity to develop individual approaches, according to clients' needs.
  • Little detail was provided in the evaluation report regarding the government's role in the EZs program, aside from being the "policy driver" and funding source. Therefore, the extent to which the government was directly involved in the program and the accountability mechanisms that were in place, are unknown.
Table C4: Did the Employment Zones evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Partially demonstrated
  • The methodology for the EZs evaluation involved a two-stage survey of a cohort of EZ participants and a comparison sample, with the main purpose being to estimate any additional employment provided by the EZ program in comparison to New Deal 25 Plus, using regression analysis. Administrative data were also analysed to estimate the extent of substitution and displacement attributable to the program.l
  • Choosing the comparison areas was somewhat problematic, given that the EZs were among the most deprived in Britain and there were few with comparable levels of labour market and social conditions.l
  • The short-term nature of the study (i.e., the two survey periods were only 10 months apart) meant that while participating in EZs resulted in a higher likelihood of being employed at least 16 hours a week, there was no further follow-up of the study cohort, and hence, no way of verifying the proportion who were still employed in the longer term.n
  • Important aspects such as participants who failed to obtain work or those who started jobs but then lost/left them, and the underlying mechanisms that may have led to these different outcomes (or what else could have been done to assist them) were not addressed.l
Attribution Partially demonstrated
  • As indicated in the EZs evaluation report, few, if any, "employment treatments" are available solely through a specific program, as there tends to be a number of activities that are simultaneously taking place in the same area. Hence, the specific effects of an individual program such as EZs can be difficult to ascertain.l
  • It is also possible that a substantial proportion of outcomes that have been attributed to a program would have occurred in its absence. Therefore, in the case of EZs, participants who gained employment may have done so without it (although it may have taken longer), and the ever-changing nature of labour market conditions is also difficult to account for (e.g., the opening or closing of a large workplace can have a significant effect on employment in a local area).l
Theory of change - Articulated Demonstrated
  • The theory of change was to offer customised job search assistance, including personalised support, training advice and follow-up, to the long-term unemployed living in areas of high deprivation, in order to assist these individuals in gaining sustained employment and increase employment rates in these areas.l
Theory of change - Measured Partially demonstrated
  • The EZs evaluation primarily focused on whether being in an EZ area increased an unemployed person's chances of a getting a job, in comparison to unemployed people living in similarly deprived areas.l The extent to which overall employment in the EZs was affected was not investigated, however, and the employment of EZ clients was only tracked up until 13 weeks.l
Residential mobility Not demonstrated
  • Not investigated.
Cost-effectiveness Not demonstrated
  • Not investigated.

The URBAN I Community Initiative (URBAN I CI)

Program description: The URBAN I CI was launched in 1994 as a response to challenges facing towns and cities throughout the EU relating to economic, social and environmental disadvantage. More specifically, the initiative constituted a neigbourhood-focused urban regeneration program that aimed to address issues such as high unemployment, the risk of social exclusion and a neglected physical environment.o There were four types of strategies adopted for the implementation of the URBAN I CI in the program areas, which included (in order of priority):

  • a broad integrated approach (i.e., a balanced set of economic development, social integration and environmental measures);
  • an integrated approach with a particular focus, such as economic, social or environmental;
  • a community-focused strategy, which prioritised community involvement in the program; and
  • a "flagship" strategy, which used a limited number of visible projects to generate interest in the program.p

Evaluation design: A report by GHKp evaluated the effects of the URBAN I CI following its completion in 1999, according to three specific areas of achievement:

  • outcome objectives, involving the delivery of tangible benefits to the target area and population;
  • internal processes and approaches to urban regeneration, including the use of an integrated approach to deliver a wide range of measures, the combination of funding from both EU and non-EU sources, and the mobilisation of public and private funds in a joint action; and
  • the legacy of the URBAN I CI program, with sustainable and lasting outcomes, and the continuation of the projects and approaches beyond the end of the EU funding period.

A combination of different methods were used to evaluate the extent to which the initiative achieved its objectives (primarily at the overall EU level), including literature reviews, desk research, structured interviews with stakeholders, case studies, "street surveys" and the informed judgements of evaluation team members.p

Program outcomes: In regard to the first level of achievement - outcome objectives - the most visible success indicators of the URBAN I CI were those achieved against set physical and socio-economic targets, such as jobs created, persons trained, community centres, sports facilities and business space. For the second level of achievement, internal processes and approaches to urban regeneration, it was found that the initiative introduced significant changes, including the concept of cross-sectoral working, bringing together key stakeholders and involving the recipient community in the change process. The legacy of the URBAN I CI was also reflected in the number of programs and management approaches/structures that were adopted for the URBAN II CI (the follow-up program that was launched in 2001) and other regeneration activities. Despite these positive outcomes, however, it was acknowledged that the URBAN I CI programs were limited in scope, with interventions in areas such as housing, transport services and infrastructure, health and crime prevention, being almost non-existent. The initiative also lacked systematic monitoring and evaluation structures to measure the outputs, results and effects of its programs, with only about 20% of the programs being subject to an independent final evaluation.p

Table C5: Did the URBAN I CI demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Partially demonstrated Partially evaluated
  • "Community chest funds" were established for the URBAN I CI to encourage the local community to set up projects and get involved in the regeneration of their area. This flexible funding mechanism was found to be effective in encouraging grassroots participation.p
  • The need for greater flexibility in the planning of area-based interventions was recognised in the URBAN I CI evaluation, which included allowing some flexibility in time frames for programs to achieve their objectives.p
Local autonomy Partially demonstrated Partially evaluated
  • The level of community involvement and autonomy varied between the URBAN I CI programs, depending on the general type of management structure that was employed (refer to "Governance" for further information). For example, with municipality-led management and implementation, local residents and associations were consulted or informed about the program at various stages, but not heavily involved; whereas with the community partnership approach, there was very clear involvement of local, non-government individuals in project selection and implementation, without overall leadership of the municipal or town government.p
  • The URBAN I CI evaluation reported that the most important factor in the successful implementation and management of the program was the participation of the local community. The extent to which this actually occurred varied, however, depending on factors such as the existing administrative structures in the area.p
Capacity development Demonstrated Evaluated
  • The URBAN I CI evaluation reported that the program built capacity both at the level of the municipality and within the local communities, with social capital also being enhanced at the local level.p
  • It was also reported that capacity and experience were key factors affecting the success of program implementation and management, with establishing the "capacity-building process from the start of the program" viewed as critical and worth the time and resources that it required (p. 38).p
  • The full extent to which this occurred across the programs is unclear. Some areas, such as Bristol, developed a specific capacity building plan, but others lacked the capacity to embrace the new structures and systems accompanying the program.p
  • Other aspects of capacity development, such as information systems, were not demonstrated to an adequate extent. It was estimated that less than 6% of the program costs were spent on management and technical assistance aspects.p
Lead times Not demonstrated Not evaluated
  • Lead times were not specifically reported on in the URBAN I CI evaluation report, but they do not appear to have been substantial, given that the European Commission notice to member states indicating the program eligibility criteria was released in June 1994 and the program was launched later in 1994.p
  • As was indicated in the evaluation report, a specific period was not allocated for building capacity and achieving consensus over priorities, which was believed to negatively affect implementation.p
Long-term focus Partially demonstrated Partially evaluated
  • The URBAN I CI was in operation from 1994 to 1999, with the follow-up URBAN II CI also running for 5 years, from 2001 to 2006.p
  • While 5 or even 10 years is a relatively short period of time for an urban regeneration strategy, the evaluation report stipulated that the "legacy of the URBAN I CI program objective" aimed to put in place sustainable outcomes and "fund innovative projects that, where possible, formed part of a long-term urban integration strategy" (p. 26).p For this reason, several of the management approaches and structures from the URBAN I CI were adopted in URBAN II CI programs.p
Spatial targeting Demonstrated Not evaluated
  • The URBAN I CI involved 118 program areas across what were then 15 member states of the EU, which included almost 3 million inhabitants.p
  • The initiative predominantly targeted peripheral urban or inner city areas, with a minority being historic city centres and areas with "mixed" characteristics.p
  • There was high variability in the populations of the program areas, ranging from 1,500 to 130,000.p
Social targeting Demonstrated Not evaluated
  • The URBAN I CI program areas were selected via an Economic Community notice to member states, with the eligibility criteria being: (a) the area had to be located in a city of more than 100,000 people; (b) the target area had to be defined and geographically identifiable; and (c) the area demonstrated some level of deprivation in regard to high unemployment, a "decayed urban fabric", poor housing conditions and a lack of social amenities.p
Joined-up working Demonstrated Evaluated
  • Due to the complexity involved in the implementation and management of the URBAN I CI programs, a high level of integration was required between all parties involved, including local and national governments, non-government organisations, local residents and private sector representatives.p
  • Showing "an integrated and straightforward approach to program management and implementation", and demonstrating "strong partnerships and cooperation" were identified as key factors affecting the success of the program's implementation and management (p. 54).p
  • In the evaluation report it was concluded that the URBAN I CI took an "integrated approach to urban regeneration, involving a variety of actors in decision-making" (p. x).p
  • While the outcomes of this approach were generally viewed as positive due to the bringing together of all parties involved, thereby increasing collaboration and innovation, it also led to some difficulties in regard to clashing cultures and interests, a lack of clear communication and high administrative burden.p
Governance Demonstrated Partially evaluated
  • Local and national governments were the main authorities involved in the planning process for the URBAN I CI.p
  • Three main types of management structure were employed, including: (a) municipality-led management and implementation; (b) a committee arrangement with partnerships between regional and municipal government, institutions and local community organisations; and (c) a body set up as a separate legal entity to manage and/or implement the program.p
  • While the programs' governance arrangements were generally reported to be effective, the administrative complexity of the URBAN I CI and the high level of bureaucracy and paperwork that arose from this, led to some difficulties in planning and implementation.p
  • A lack of systematic monitoring and evaluation systems was also a problem.p
Table C6: Did the URBAN I CI evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Not demonstrated
  • The methods used in the evaluation did not enable causality to be established. "Street surveys', for example, asked residents and non-residents about their perceptions of change in the area over the life of the initiative, but did not scientifically measure change or causal effects.p
  • Also, given that only around 20% of the programs were subject to an independent final evaluation, the overall effects of the URBAN I CI were impossible to fully ascertain, regardless of the methodology.p
Attribution Not demonstrated
  • Given the lack of scientific impact management, few effects could be specifically attributed to the URBAN I CI.
  • A further factor making attribution problematic was the number of program areas that already had existing regeneration/urban policies and initiatives in place (e.g., 5 of the 15 member states already had a national urban policy when the URBAN I CI was implemented).p
Theory of change - Articulated Demonstrated
  • The URBAN I CI aimed to promote the spatial concentration of resources and take an area-based approach to urban regeneration, in order to address the economic, social and environmental disadvantage faced by communities throughout the European Union.
Theory of change - Measured Not demonstrated
  • The URBAN I CI aimed to promote the spatial concentration of resources and take an area-based approach to urban regeneration, in order to address the economic, social and environmental disadvantage faced by communities throughout the European Union.
  • Due to its non-scientific methodology and the lack of programs that were actually formally evaluated, the URBAN I CI was very limited in its capacity to measure these overall objectives.
Residential mobility Not demonstrated
  • Not investigated.
Cost-effectiveness Not demonstrated
  • Not investigated.

New Deal for Communities (NDC)

Program description: The NDC program, which was launched in 1998, was an intensive area-based initiative that aimed to transform 39 deprived neighbourhoods in England over 10 years. The 39 NDC partnerships implemented local regeneration schemes and aimed to achieve six key objectives, to:

  • transform these areas by achieving holistic change in relation to three place-related outcomes (crime, community and housing and the physical environment), and three people-related outcomes (education, health and worklessness);
  • "close the gap" between these areas and the rest of the country;
  • achieve "value for money" in the transformation of the areas;
  • work with other delivery agencies such as the police, Primary Care Trusts, schools, Jobcentre Plus, and their parent local authority;
  • place the community "at the heart" of the initiative; and
  • sustain a local effect after the NDC program funding ceased.q

Evaluation design: The national evaluation of the NDC program was commissioned in 2001 in order to: undertake a "summative" evaluation to identify the impact and cost-effectiveness of the program; support the partnerships through informed feedback and NDC-specific data (i.e., the "formative" element to the evaluation); and enhancing the evidence base in regard to "what works and why" in neighourhood renewal. The NDC national evaluation team employed a range of data collection tasks and analytical techniques, including case study work in a small number of areas, focus groups, and business and project beneficiary surveys. Essential data sources also included household surveys, secondary and administrative data, and annual partnership reports.r The final NDC evaluation comprised a number of volumes, culminating in the a final assessment report in 2010.q

Program outcomes: The NDC partnerships resulted in a total of approximately 6,900 projects or interventions that were designed together with the partner agencies to encompass the three place- and three people-related outcomes (refer to "Program description" above). Between 2002 and 2008, NDC areas improved in 32 out of the 36 core indicators that spanned the place- and people-related outcomes, with these changes being statistically significant for the majority. The largest improvements were for indicators of people's feelings about their neighbourhoods, with residents recognising changes brought about by the program and being more satisfied with their neighbourhoods as places to live as a result. In general, NDC areas "narrowed the gaps" with the rest of the country, when benchmarked against national equivalents, parent local authorities and similarly deprived comparator areas.q

Table C7: Did the NDC initiative demonstrate and/or evaluate the common elements?
Common element Demonstrated? Evaluated?
Flexible delivery Demonstrated Partially evaluated
  • Flexibility was offered to the 39 NDCs, particularly in regard to the allocation of funds. This was viewed as being important, given that it could be used as leverage to receive larger sums of money from other agencies.q
  • The implementation of NDC was somewhat mandated, with central government producing over 40 "Program Notes" to guide, and in some case impose procedures on, the Partnership Boards established for each NDC.s
  • Ultimately, however, the Partnership Boards had considerable influence in regard to overall strategy and project funding.q
  • While viewed as favourable, the funding flexibility offered through NDC did cause some tensions with agencies such as Jobcentre Plus, which operated within a more "output-driven" culture (p. 31).q
Local autonomy Demonstrated Partially evaluated
  • NDC aimed to place the community "at the heart" of the initiative, with a focus on engaging communities and developing interventions locally.q
  • The Partnership Boards consisted of at least 50% of local residents for the majority of NDCs,s which was beneficial both to the individuals involved and the NDC. For example, the final evaluation report noted their ability to "validate the additionality of proposed projects" (p. 7).q
  • Only a relatively small number of local residents were directly involved in the formal processes of decision-making and resource allocation.q
  • Other NDC initiatives designed to engage with local residents included: further community involvement in outcome sub-committees and appraisal panels; actively seeking to engage the community through forums and other initiatives; a range of communications media to keep local people informed; and dedicated community engagement or involvement teams.q
  • The full effects of local autonomy were not evaluated, however, and the final evaluation report recommended that, in the future, regeneration schemes needed to "establish what the community dimension actually means and to set objectives accordingly: does it mean consultation, involvement, engagement, empowerment or delivery?" (p.9).q
  • The report also stated that resident involvement in NDC had "generated questions around appropriate governance arrangements", including "the degree to which resident expectations and priorities should be the main driving force behind the allocation of resources, and the extent to which residents' views should be challenged by professionals" (p. 32).q
Capacity development Partially demonstrated Not evaluated
  • The final evaluation report indicated that the 39 NDC partnerships had made immense efforts to not only engage residents, but also enhance the capacity of the local community.q
  • Initiatives that were designed to enhance community capacity included training interventions for community representatives, constructing new/improving existing community facilities, and community-based small grant and loan schemes.q
  • The extent to which capacity development occurred, however, both at the community and government levels, was not thoroughly investigated or evaluated.
Lead times Partially demonstrated Not evaluated
  • The first 17 "Pathfinder" NDC funding recipients were announced in 1998, with the remaining 22 NDC areas announced in the following year.r
  • While no other details were provided regarding lead times, it is possible that learnings from the "Pathfinder" NDCs helped to inform the roll-out of the remainder.
Long-term focus Demonstrated Not evaluated
  • The NDC program was rolled out over 1998-99, with funding for the 39 programs ceasing in 2010-11. This represented a sizeable investment that aimed to transform disadvantaged areas.q
  • A key objective of NDC was to sustain a local effect after program funding ceased. Succession strategies were developed by the partnerships, and included creating successor bodies, developing income streams and influencing delivery agencies to secure continued financial support after NDC program funding ended.q
  • The evaluation was conducted over 10 years, from 2001 to 2010.q
Spatial targeting Demonstrated Not evaluated
  • The NDC was implemented in 39 neighbourhoods across England, with each accommodating around 9,800 people on average, but ranging from around 5,000 to 21,000 people (hence, the NDC was delivered to a total of over 380,000 people).r
  • Local authorities typically decided where NDCs were to be sited.t
  • It is important to note that while NDC was one of the more intensive area-based initiatives to be launched in England, it only represented a fraction of deprived neighbourhoods.q
Social targeting Demonstrated Evaluated
  • The 39 NDC areas were generally deprived, with about a quarter falling within the most deprived 1,000 of the 32,000 lower level Super Output Areas, as derived from the 2001 Census.q
  • The proportion of the non-white population across the program population was about 26%, with 55% of households living in social rented accommodation.q
  • The NDC final evaluation indicated, however, that the program made some progress in "closing the gap" with the rest of the country, with greater positive change for place, rather than people-related outcomes.q
  • It was also argued that while targeting regeneration investment in the "right places" is critical, identifying these can be complex, due to requiring a balance between selecting more deprived areas on the one hand, while prioritising areas with opportunities for, or fewer barriers towards, change on the other.q
Joined-up working Demonstrated Evaluated
  • A specific objective of NDC was for the program to be "fundamentally rooted in partnership working", in developing relationships with service delivery agencies such as the police, schools and Jobcentre Plus (p. 5).q
  • It was reported that, on the basis of the 2008 Partnership Survey, relationships between agencies and parent local authorities (e.g., the police) generally improved and intensified over the course of NDC, with the main challenge being for some delivery agencies, due to their partnership with NDC, requiring reorganisation and needing to develop the capacity to work to national targets.q
  • The final evaluation report also stated that achieving successful partnership working was a "gradual process with an emphasis placed on delivery: i.e. 'how can the NDC help you in what you do?' " (p. 19).q
  • It was anticipated that the investment in "partnership working" with agencies would help to sustain activity in the areas after program funding ceased.q
Governance Demonstrated Not evaluated
  • NDC adopted an "innovative" form of governance, with residents and agencies having responsibility for the delivery of local strategies through its residents and agencies.t
  • As indicated in "Flexibility", the Partnership Boards had a degree of freedom in proposing overall 10-year strategies and in deciding the allocation of funds across the six outcomes, but operational restraints were imposed by central government through its nine regional offices.t
  • The governance mechanisms for NDC were not formally evaluated, although aspects regarding the Partnership Boards and community engagement were to some extent (also refer to "Local autonomy").
  • It was argued that communities' role in governance can be a "double-edged sword" for initiatives such as NDC, as while it is important for communities to be active agents in their own social change, by nature of their relative disadvantage, they can be "constrained in their capacity to host such efforts" (p. 674).u
Table C8: Did the NDC evaluation demonstrate the common elements?
Common element Demonstrated?
Establishing causality Demonstrated
  • One of the primary objectives of the NDC evaluation was to assess the direct effect of the initiative on the total of six place- and people-related outcomes, with a variety of methods used for this purpose.s
  • The biennial household survey was the primary measure of change administered across all 39 NDC areas, with baseline in 2002, and three follow-up survey waves in 2004, 2006 and 2008 (a top-up sample was also used).s
  • The creation of a counterfactual was critical to establishing causality, with the survey also carried out in comparator areas (i.e., similarly deprived neighbourhoods in the same local authorities as the NDCs, but in non-adjacent wards to avoid potential "spillover" effects).s
  • The Composite Index of Relative Change (CIRC), which standardised and combined "change data" for 36 core indicators during the 2002-08 survey period, was then used to identify and understand the patterns of change across the 39 NDC areas.v
Attribution Demonstrated
  • As outlined above, the CIRC measure was developed to determine the longitudinal effects that could be attributed to NDC; that is, this tool allowed the extent to which change in the NDC areas was less or more than that in the comparison areas.v
  • Hence, the CIRC enabled a "net" NDC effect over-and-above change occurring as a consequence of national, regional or local authority trends to be calculated, and ultimately, determine the extent to which the regeneration investment associated with NDC produced change.v
  • With that said, however, the NDC final evaluation did not detect a great deal of relative change when assessed against the comparator areas. It was acknowledged that areas can change for many reasons that may not be within the control/influence of an area regeneration program such as NDC, and hence, attribution of effects (or a lack thereof) can still be problematic.q
Theory of change - Articulated Demonstrated
  • The theory of change for the NDC program was to transform 39 deprived neighbourhoods in England over 10 years by achieving holistic change in relation to three place-related outcomes (e.g., crime) and three people-related outcomes (e.g., education), in order to "close the gaps" between these areas and the rest of the country. Other key objectives included achieving a "value for money" transformation, establishing closer partnerships with other delivery agencies (e.g., the police), placing the community "at the heart" of the initiative, and to sustain a local effect after NDC program funding ceased.q
Theory of change - Measured Partially demonstrated
  • Most of the key objectives were measured in the NDC evaluation, with the place- and people-related outcomes, the extent to which the NDC areas had "caught up" and cost-effectiveness investigated.
  • The other aspects, including "partnership working", community involvement and sustenance of local effects, could not be measured scientifically and/or objectively for the final evaluation.
Residential mobility Demonstrated
  • Residential mobility was investigated as part of the final evaluation, and it was found that NDC residents were more frequent "movers" than in the comparison areas.. Younger people (15-34 years old), private rented sector households, residents with higher qualifications, males and white residents were more likely to move out of NDC areas.w
  • NDC areas with higher levels of mobility were found to experience less positive change than the NDC average in regard to housing and physical environment indicators.w
Cost-effectiveness Demonstrated
  • One of the key objectives of NDC was to achieve a "value for money" transformation of the 39 neighbourhoods.q For this reason, a volume of the final national evaluation report was dedicated to assessing the impact and cost-effectiveness of NDC.x
  • From the cost-benefit calculations it was found that NDC represented good value for money, with the monetary benefits arising from the program substantially exceeding costs.x

Appendix D: Characteristics of international place-based initiatives identified by the literature review

For full details, see Table D1: The characteristics of 25 international initiatives identified for further review.

Additional references

The following references relate specifically to Table D1 and are not mentioned elsewhere in the report.

  • Bauld, L., Judge, K., Barnes, M., Benzeval, M., Mackenzie, M., & Sullivan, H. (2005). Promoting social change: The experience of health action zones in England. Journal of Social Policy, 34(3), 427-445.
  • Bertram, T., Pascal, C., Bokhari, S., Gasper, M., Holtermann, S., John, K., et al. (2004). Early Excellence Centre Pilot Programme: Third annual evaluation report 2001-2002. Nottingham: Department for Education and Skills.
  • Casebourne, J., & Davis, S. (2006). Review of Action Teams for Jobs (Research Report No. DWPRR 328). London: Department for Work and Pensions.
  • Edwards, A., Barnes, M., Plewis, I., Morris, K., et al. (2006). Working to prevent the social exclusion of children and young people: Final lessons from the National Evaluation of the Children's Fund (Research Report No. RR74). Nottingham: Department for Education and Skills.
  • Halpern, C. (2005). Institutional change through innovation: The URBAN Community Initiative in Berlin, 1994-99. Environment & Planning C: Government & Policy, 23(5), 697-713.
  • Hughes, N., & Fielding, A. (2006). Targeting preventative services for children: Experiences from the Children's Fund (Research Report No. RR777). Nottingham: Department for Education and Skills.
  • Jensen, E. H., & Munk, A. (2007). Kvarterloft: 10 years of urban regeneration. Copenhagen: Ministry of Refugees, Immigration and Integration Affairs.
  • McCaig, C., Stevens, A., & Bowers-Brown, T. (2006). Does Aimhigher work? Evidence from the national evaluation. Sheffield: Sheffield Hallam University Research. Retrieved from <shura.shu.ac.uk/2382/>.
  • ODS Consulting. (2006). An overview of the social inclusion partnership (SIP) programme (Research Report No. 74). Edinburgh: Communities Scotland.
  • Office for Standards in Education. (2003). Excellence in Cities and Education Action Zones: Management and impact (Research Report No. HMI 399). Manchester: Ofsted Publications Centre.
  • Popkin, S., Eiseman, M., & Cove, E. (2004). How are HOPE VI families faring? Children: A roof over their heads. Changes and challenges for public housing residents (Research Brief No. 6). Washington, DC: The Urban Institute.
  • Reid, I., & Brain, K. (2003). Education Action Zones: Mission impossible? International Studies in Sociology of Education, 13(2), 195-214.

Appendix E: Characteristics of Commonwealth place-based initiatives selected for review

For full details, see Table E1: The characteristics of the eight Commonwealth initiatives selected for further review.

* The superscript letters in this appendix refer to the sources in the supplementary reference list for international place-based initiatives in section 7.2.

Acronyms
Acronym Description
AGD Attorney-General's Department
AIFS Australian Institute of Family Studies
BAFW Building Australia's Future Workforce
BFLS Better Futures, Local Solutions
CATI Computer-assisted telephone interview
CC City Challenge
CfC Communities for Children
CIRC Composite Index of Relative Change
COAG Council of Australian Governments
CYI Cape York Institute for Policy and Leadership
CYWR Cape York Welfare Reform trial
D2DL Support for Day to Day Living in the Community
DEEWR Department of Education, Employment and Workplace Relations (now DoEd and DoEm)
DHS Department of Human Services
DIAC Department of Immigration and Citizenship (now DIBP)
DIBP Department of Immigration and Border Protection
DID Difference-in-difference
DoEd Department of Education
DoEm Department of Employment
DoH Department of Health
DoHA Department of Health and Ageing (now DoH)
DSS Department of Social Services
DRALGAS Department of Regional Australia, Local Government, Arts and Sport (now DoH)
EiC Excellence in Cities
EU European Union
EZ Employment Zones
FaHCSIA Department of Families, Housing, Community Services and Indigenous Affairs (now DSS)
FRC Family Responsibilities Commission
HCZ Harlem Children's Zone
HPE Healthcare Planning and Evaluation
HYP Helping Young Parents
IM Income Management
JSA Job Services Australia
LCTW Local Connections to Work
LGA Local government area
MTO Moving to Opportunity
NDC New Deal for Communities
NESS National Evaluation of Sure Start
NGO Non-government organisation
NNI Neighbourhood Nurseries Initiative
NPARIH National Partnership Agreement on Remote Indigenous Housing
NSNR National Strategy for Neighbourhood Renewal
NZ MSD New Zealand Ministry of Social Development
PBI Place-based initiative
PM&C Department of the Prime Minister and Cabinet
PBS Centrelink Place Based Services
PEA Priority Employment Area
PHaMS Personal Helpers and Mentors Service
PSS Petrol Sniffing Strategy
RFI Request for information
RPA Regional Partnership Agreement
SEAM School Enrolment and Attendance Measure
SEIFA Socio-Economic Indexes for Areas
SFIA Stronger Families in Australia
SJF Supporting Jobless Families
SSLP Sure Start Local Program
UK United Kingdom
URBAN I CI URBAN I Community Initiative
US United States
WDP Whānau Development Project
Lists of tables and figures

List of tables

Table 1: Summary of the extent to which the 12 investigated international place-based initiatives met the common elements

Table 2: Did the CfC initiative demonstrate and/or evaluate the common elements?

Table 3: Did the CfC evaluation demonstrate the common elements?

Table 4: Did the CYWR trial initiative demonstrate and/or evaluate the common elements?

Table 5: Did the CYWR trial evaluation demonstrate the common elements?

Table 6: Did the D2DL program demonstrate and/or evaluate the common elements?

Table 7: Did the D2DL program evaluation demonstrate the common elements?

Table 8: Did the PBS initiative demonstrate and/or evaluate the common elements?

Table 9: Did the PBS evaluation demonstrate the common elements?

Table 10: Did the BAFW package demonstrate and/or evaluate the common elements?

Table 11: Did the BAFW package demonstrate the common elements?

Table 12: Summary of the extent to which the nine investigated Commonwealth place-based initiatives demonstrated or evaluated the common elements

Table B1: International place-based initiatives identified by the literature review

Table C1: Did the SSLPs initiative demonstrate and/or evaluate the common elements?

Table C2: Did the SSLPs evaluation demonstrate the common elements?

Table C3: Did the Employment Zones initiative demonstrate and/or evaluate the common elements?

Table C4: Did the Employment Zones evaluation demonstrate the common elements?

Table C5: Did the URBAN I CI demonstrate and/or evaluate the common elements?

Table C6: Did the URBAN I CI evaluation demonstrate the common elements?

Table C7: Did the NDC initiative demonstrate and/or evaluate the common elements?

Table C8: Did the NDC evaluation demonstrate the common elements?

Table D1: The characteristics of 25 international initiatives identified for further review

Table E1: The characteristics of the eight Commonwealth initiatives selected for further review

List of figures

Figure 1: Methodology employed for the collection and review of Commonwealth PBI materials

Figure A1: Quadrant-based model for classifying place-based initiatives

Acknowledgements

Sez Wilks is a Research Officer, Dr Julie Lahausse is a Research Fellow and Dr Ben Edwards is Executive Manager, Longitudinal Studies, all at the Australian Institute of Family Studies.


The findings are those of the authors and should not be attributed to the Australian Institute of Family Studies or the Department of the Prime Minister and Cabinet. The authors are grateful to Professor Alan Hayes, Dr Daryl Higgins and Kelly Hand for their assistance with scoping and conducting the review, and for their valuable comments on earlier versions of this report.

It should also be noted that Dr Ben Edwards was the lead author of the evaluation of the Communities for Children initiative (Edwards, Mullan, Katz, & Higgins, 2014). Moreover, Dr Edwards was also an author of a report to the Department of Employment that provided information on a survey of parents who were affected by the Helping Young Parents measure that fed into the evaluation of the Building Australia’s Future Workforce evaluation (Taylor & Edwards, 2012).

Publication details

Research Report
Published by the Australian Institute of Family Studies, April 2015
98 pp.
ISBN:
978-1-76016-017-3
Suggested citation:

Wilks, S., Lahausse, J., & Edwards, B. (2015). Commonwealth Place-Based Service Delivery Initiatives: Key Learnings project (Research Report No. 32). Melbourne: Australian Institute of Family Studies.