NIESR: No evidence Troubled Families Programme had any significant impact on key objectives
NIESR’s analysis, published yesterday, forms part of the wider evaluation of the Troubled Families Programme, commissioned by the Department of Communities and Local Government from a consortium led by Ecorys. The key finding is that across a wide range of outcomes, covering the key objectives of the Troubled Families Programme - employment, benefit receipt, school attendance, safeguarding and child welfare - we were unable to find consistent evidence that the programme had any significant or systematic impact. The vast majority of impact estimates were statistically insignificant, with a very small number of positive or negative results. These results are consistent with those found by the separate and independent impact analysis using survey data, also published yesterday, which also found no significant or systemic impact on outcomes related to employment, job seeking, school attendance, or anti-social behaviour.
The purpose of the analysis was to estimate the impact of participation in the initial phase of the Troubled Families programme on a range of outcomes encompassing benefit receipt, employment, educational participation, child welfare and offending. The data used were compiled from information provided by local authorities and from national administrative datasets covering tax and benefits receipt, offending, educational attainment, schooling and child social care.
Two methods were used to estimate the causal impact of the Troubled Families programme. The main focus in the report is on the analysis which uses a technique known as propensity score matching (PSM. PSM matches those subject to the programme to a comparison group of families that have a similar propensity to be treated, based on their observed characteristics, but who do not in fact receive assistance. Provided the two groups are matched on all characteristics which determine both whether the family participates in the programme and which impact on the outcomes that they experience as a result of participation, the impact estimate should provide an accurate and unbiased estimate of the true impact of the intervention. A supplementary analysis exploited the fact that families started on the programme at different points in time. This was known as the waiting list analysis. Those who started on the programme at a later date were used as a comparison group for families that started earlier, with outcomes for the comparison group observed in the period before they started to receive support.
Data issues and caveats
Fifty-six local authorities provided the data used in this study between October and November 2014. These data were then matched to national level administrative data sets. Data was obtained on approximately 25 percent of the 120 000 families that participated in the programme, representing a large sample, and enabling us to undertake detailed analysis. However, the data supplied was of variable quality. As a result, some important data was missing, and it was necessary to make certain assumptions in assigning individuals to treatment and control groups. In addition, a significant number of individuals were not matched to certain of the administrative data sets, and this necessitated further assumptions – for example, we assume that individuals not matched to employment records were not employed. It is possible that in some cases these assumptions did not reflect the true circumstances of particular families and as a result, the findings may be subject to measurement error. However, as long as the prevalence of missing or incomplete data is random and/or does not differ systematically between the treated and control groups, the conclusions will remain unbiased. None of the available evidence suggests that such systematic differences exist, and the possibility of significant bias therefore appears unlikely. The fact that separate analysis, using survey data, produces results consistent with our analysis is further evidence that any bias that would impact our results to a significant degree is unlikely. However, given the data issues this possibility cannot be entirely excluded. It is not possible to say with certainty how this might affect the reported results.
Notes for editors:
- The analysis covers the first phase of the Troubled Families Programme, which was launched in April 2012 with the objective of “turning around” the lives of 120,000 of the most “troubled families” in England, with a budget of £448 million.
- The impact of the initial stage of the Troubled Families programme was evaluated by an independent research consortium, led by Ecorys and including NIESR. The evaluation consists of a number of complementary strands which as a whole are designed to provide a detailed understanding of the implementation and operation of the programme and its impact across the range of outcomes that it seeks to affect. The full set of reports from the evaluation can be found here.
- The NIESR report was authored by Helen Bewley (lead researcher), Anitha George, Cinzia Rienzo and Jonathan Portes. To discuss the findings of the report, contact Jonathan Portes (email@example.com or 07766 441148).
NIESR aims to promote, through quantitative and qualitative research, a deeper understanding of the interaction of economic and social forces that affect people's lives, and the ways in which policies can improve them.
Latest News from
Budget fails acid test to level up: IPPR North responds to the Budget and Spending Review27/10/2021 16:35:00
Jonathan Webb, a senior research fellow at IPPR North responded to today’s Autumn Budget
Institute of Economic Affairs: Autumn Budget 2021 response27/10/2021 16:15:00
Mark Littlewood, Director General at free market think tank the Institute of Economic Affairs, commented on the Chancellor’s Autumn Budget 2021
Adam Smith Inst - Spendy Sunak: The ASI responds to Budget 202127/10/2021 15:15:00
The Adam Smith Institute’s Head of Programmes Daniel Pryor responded to Chancellor Rishi Sunak’s 2021 Budget
Public sector pensions cost £57bn per year more than is declared, finds new IEA research26/10/2021 13:35:00
Public sector pensions cost £57bn per year more than is declared, finds new research
Adam Smith Inst - Global minimum tax could cost Britain billions26/10/2021 12:35:00
Minimum global corporate tax will undermine sovereignty and key Government policies
There should be greater caution in setting minimum wages, says IEA expert26/10/2021 11:35:00
Professor Len Shackleton, Editorial and Research Fellow at free market think tank the Institute of Economic Affairs, commented on the Chancellor’s plans to raise the minimum wage to £9.50 an hour
IFS - School spending per pupil highest in Scotland, lowest in Northern Ireland26/10/2021 10:35:00
In the current year (2021–22), core school spending per pupil is expected to be highest in Scotland (over £7,500), similar levels in England (£6,700) and Wales (£6,600), and lowest in Northern Ireland (£6,400).
IEA - UK Climate Change Committee in urgent need of reform, says new research26/10/2021 09:35:00
A new paper, published by the Institute of Economic Affairs (IEA), puts the UK’s Climate Change Committee – and its activities – under the spotlight. Its author, IEA Head of Regulatory Affairs Victoria Hewson, concludes that the CCC has expanded well beyond its statutory obligations and acts in an overtly political way. It has failed to communicate and, at times, purposefully suppressed, the full cost of its proposed climate policies.
JRF - Large-scale study reveals scale of debt crisis among low-income households21/10/2021 13:35:00
A large-scale study of households on low incomes has revealed the extent of the debt crisis hanging over the UK’s poorest families as the country braces to weather a cost-of-living crisis.