WIREDGOV

Key findings

NIESR’s analysis, published yesterday, forms part of the wider evaluation of the Troubled Families Programme, commissioned by the Department of Communities and Local Government from a consortium led by Ecorys. The key finding is that across a wide range of outcomes, covering the key objectives of the Troubled Families Programme - employment, benefit receipt, school attendance, safeguarding and child welfare - we were unable to find consistent evidence that the programme had any significant or systematic impact. The vast majority of impact estimates were statistically insignificant, with a very small number of positive or negative results. These results are consistent with those found by the separate and independent impact analysis using survey data, also published yesterday, which also found no significant or systemic impact on outcomes related to employment, job seeking, school attendance, or anti-social behaviour.  

Methodology

The purpose of the analysis was to estimate the impact of participation in the initial phase of the Troubled Families programme on a range of outcomes encompassing benefit receipt, employment, educational participation, child welfare and offending. The data used were compiled from information provided by local authorities and from national administrative datasets covering tax and benefits receipt, offending, educational attainment, schooling and child social care.  

Two methods were used to estimate the causal impact of the Troubled Families programme. The main focus in the report is on the analysis which uses a technique known as propensity score matching (PSM.  PSM matches those subject to the programme to a comparison group of families that have a similar propensity to be treated, based on their observed characteristics, but who do not in fact receive assistance. Provided the two groups are matched on all characteristics which determine both whether the family participates in the programme and which impact on the outcomes that they experience as a result of participation, the impact estimate should provide an accurate and unbiased estimate of the true impact of the intervention. A supplementary analysis exploited the fact that families started on the programme at different points in time. This was known as the waiting list analysis. Those who started on the programme at a later date were used as a comparison group for families that started earlier, with outcomes for the comparison group observed in the period before they started to receive support. 

Data issues and caveats

Fifty-six local authorities provided the data used in this study between October and November 2014.    These data were then matched to national level administrative data sets. Data was obtained on approximately 25 percent of the 120 000 families that participated in the programme, representing a large sample, and enabling us to undertake detailed analysis. However, the data supplied was of variable quality.  As a result, some important data was missing, and it was necessary to make certain assumptions in assigning individuals to treatment and control groups. In addition, a significant number of individuals were not matched to certain of the administrative data sets, and this necessitated further assumptions – for example, we assume that individuals not matched to employment records were not employed. It is possible that in some cases these assumptions did not reflect the true circumstances of particular families and as a result, the findings may be subject to measurement error. However, as long as the prevalence of missing or incomplete data is random and/or does not differ systematically between the treated and control groups, the conclusions will remain unbiased. None of the available evidence suggests that such systematic differences exist, and the possibility of significant bias therefore appears unlikely. The fact that separate analysis, using survey data, produces results consistent with our analysis is further evidence that any bias that would impact our results to a significant degree is unlikely. However, given the data issues this possibility cannot be entirely excluded. It is not possible to say with certainty how this might affect the reported results.

Notes for editors:

1. The analysis covers the first phase of the Troubled Families Programme, which was launched in April 2012 with the objective of “turning around” the lives of 120,000 of the most “troubled families” in England, with a budget of £448 million.
2. The impact of the initial stage of the Troubled Families programme was evaluated by an independent research consortium, led by Ecorys and including NIESR. The evaluation consists of a number of complementary strands which as a whole are designed to provide a detailed understanding of the implementation and operation of the programme and its impact across the range of outcomes that it seeks to affect. The full set of reports from the evaluation can be found here.
3. The NIESR report was authored by Helen Bewley (lead researcher), Anitha George, Cinzia Rienzo and Jonathan Portes. To discuss the findings of the report, contact Jonathan Portes (j.portes@niesr.ac.uk  or 07766 441148).

NIESR aims to promote, through quantitative and qualitative research, a deeper understanding of the interaction of economic and social forces that affect people's lives, and the ways in which policies can improve them.

Further details of NIESR’s activities can be seen on http://www.niesr.ac.uk or by contacting enquiries@niesr.ac.uk Switchboard Telephone Number: +44 (0) 207 222 7665