Universal Credit stats – you can prove what you want…
An official report just issued seems to show that 5% of claimants come off Job Seeker’s Allowance (JSA) faster if they are being processed as a Universal Credit ‘Pathfinder’ case.
Looking at the cluster of blue cherries alone, we
have conclusive proof that the whole tree is blue…
The DWP report on initial Universal Credit outcomes is entitled an “Exploratory analysis”, and comes with all sorts of statistical outcomes.
The headline is there is a statistically significant improvement in people ‘exiting’ from JSA (i.e. coming off the JSA benefit) whether you take the stats at 60, 90 and 120 days.
Dangers of Cherry Picking the Data to Show What you Want
Let’s cut the data another way, and prove that for the under 24 group there is no consistent improvement:
Nicholas Cage and Swimming Pool Drownings
The problem is that you can take any sets of numbers, and if you subdivide them into enough separate groupings, there is a chance that a ‘significant’ relationship will emerge. This is called ‘cherry picking’ the data.
Let’s have a look at an absurd example:
Statistically Significant (R=66%)
OK – so what do the total Universal Credit Statistics say?
I have made some simple calculations and put all the ‘cherries’ back into one basket:
An 5% improvement in the statistics overall?
But the official report says that “given the limitations of small samples the analysis can only identify differences…to around 5 percentage points.”
Oh well, I suppose we will have to wait for the “more detailed work as outlined in the methodology paper” (See ‘Evaluating the impact of Universal Credit on the labour market‘, 22nd July 2014)
Other random observations:
1. To be a Universal Credit claimant you have to pass an onerous set of requirements. One of these is that you must have a bank account. JSA claimants do not have to have a bank account. Is it possible that claimants with bank accounts might be the sort of person with a stable salaried employment background that might find it 5% easier to get back into work?
2. It is based on only 2,760 Universal Credit Claimants. And it compares them with JSA Claimants is different but ‘similar’ areas.
3. These statistics count the people who are going back into continuous, full time work. They do not reflect the numbers of people that start irregular or part-time work. Universal Credit was meant to provide a strong incentive to claimants to take up any sort of work, secure in the knowledge that they would receive a taper payment. What insight does statistics like this taht concentrate exclusively on full-time exit from JSA provide on how effective the UC taper payment really is for encouraging part-time work?
4. Universal Credit does not encourage people into full-time stable work any more or less that JSA does at the moment. Is it possible that DWP are conflating the impact of the UC Claimant Commitment (an A4 piece of paper ‘contract’ to promise to look for work) with the effects of the UC Payment process?
© Brian Wernham 2014 CC BY-NC-ND