Skip to content

Universal Credit stats – you can prove what you want…

October 24, 2014

An official report just issued seems to show that 5% of claimants come off Job Seeker’s Allowance (JSA) faster if they are being processed as a Universal Credit ‘Pathfinder’ case.

cherrypicking

Looking at the cluster of blue cherries alone, we
have conclusive proof that the whole tree is blue…

tweetbutton shareonfacebook1

The DWP report on initial Universal Credit outcomes is entitled an “Exploratory analysis”, and comes with all sorts of statistical outcomes.

The headline is there is a statistically significant improvement in people ‘exiting’ from JSA (i.e. coming off the JSA benefit) whether you take the stats at 60, 90 and 120 days.

table - just totalsAfter 60 days claimants are 7% more likely to be
back in work (49% minus 42%)!

Dangers of Cherry Picking the Data to Show What you Want

Let’s cut the data another way, and prove that for the under 24 group there is no consistent improvement:

table - truncated

Nicholas Cage and Swimming Pool Drownings 

The problem is that you can take any sets of numbers, and if you subdivide them into enough separate groupings, there is a chance that a ‘significant’ relationship will emerge. This is called ‘cherry picking’ the data.

Let’s have a look at an absurd example:

Nicolas Cage v2

Statistically  Significant (R=66%) 

OK – so what do the total Universal Credit Statistics say?

I have made some simple calculations and put all the ‘cherries’ back into one basket:

Total stats

An 5% improvement in the statistics overall?

But the official report says that “given the limitations of small samples the analysis can only identify differences…to around 5 percentage points.”

Oh well, I suppose we will have to wait for the “more detailed work as outlined in the methodology paper” (See ‘Evaluating the impact of Universal Credit on the labour market‘, 22nd July 2014)

Other random observations:

1. To be a Universal Credit claimant you have to pass an onerous set of requirements. One of these is that you must have a bank account.  JSA claimants do not have to have a bank account. Is it possible that claimants with bank accounts might be the sort of person with a stable salaried employment background that might find it 5% easier to get back into work?

2. It is based on only 2,760 Universal Credit Claimants. And it compares them with JSA Claimants is different but ‘similar’ areas.

3. These statistics count the people who are going back into continuous, full time work. They do not reflect the numbers of people that start irregular or part-time work. Universal Credit was meant to provide a strong incentive to claimants to take up any sort of work, secure in the knowledge that they would receive a taper payment. What insight does statistics like this taht concentrate exclusively on full-time exit from JSA provide on how effective the UC taper payment really is for encouraging part-time work?

4. Universal Credit does not encourage people into full-time stable work any more or less that JSA does at the moment. Is it possible that DWP are conflating the impact of the UC Claimant Commitment (an A4 piece of paper ‘contract’ to promise to look for work) with the effects of the UC Payment process?

© Brian Wernham 2014 CC BY-NC-ND


 

Advertisements

From → Agile Governance

One Comment
  1. Reblogged this on sdbast.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: