Statistics for Dummies (Brendan O’Neill edition)

It seems that one of the Telegraph’s resident professional contrarians (aka trolls) – Brendan O’Neill – is just a little bit ticked off with claims that one in four people do/suffer/experience X…

The one-in-four fallacy is 100 per cent annoying

To mark Anti-Bullying Week, the Anti-Bullying Alliance has issued a press release claiming that one in four schoolchildren are bullied. I would be more inclined to believe this if it wasn’t for the fact that the magical “one-in-four” figure seems to pop up in the press releases of every single advocacy group known to man. So domestic violence campaigners claim one in four women experience domestic violence. Mental health charities say one in four people suffer from mental illness. The NSPCC claims one in four babies are at “high risk” because they are born into homes with high levels of mental illness and drink and drug dependency. That’s probably because, according to researchers, you guessed it – one in four of us drinks too much booze.

And on it goes. There is even grisly competition between charities for the “One In Four” moniker. The sexual violence charity One In Four is in competition with One In Four magazine, which is published by a mental-health charity. Can it really be true that a quarter of Brits are bullied or beaten up at home or are mentally ill, or is this simply a case of social campaigners exaggerating how bad life is in order that they can continue to make headlines, make an impact, and get funding? I reckon it’s the latter. Next time you see the “one in four” figure, be very sceptical – it’s probably Dickensian-style doom-mongering disguised as social research, where the aim is to convince us, against the evidence of our own eyes and ears, that loads of the people we encounter everyday are basket cases in need of rescue.

O’Neill is, of course, a journalist knocking out an opinion column and therefore rightly considers himself to be above engaging in menial tasks, such as running down the sources for these statistics and checking whether they are both accurate and backed up by robust evidence.

Evidence? Pah! Why make work for yourself when you simply point out that the same statistic crops up in a few different places and declare that this all looks a bit fishy.

Thankfully, running down the evidence is one of the things that bloggers do best, so let’s look this properly.

First things first, as a general point, just about any halfway decent mathematician, statistician or economist will happily tell you that there is nothing necessarily unusual about finding similar statistics or ratios cropping up in seemingly disparate and unrelated population datasets. One of the better known examples of this phenomenon is the 80/20 rule, or to give it its formal name, the Pareto Principle, a commonly used  rule-of-thumb in business circles which posits that, for many events, around 80% of the effects stem from 20% of the causes. Mathematically-speaking, there is nothing particular special about this 80/20 ratio but its nevertheless the case that it crops up often enough to make it a useful rule of thumb and a fair illustration of what mathematicians call a ‘power law’ relationship of which there are numerous examples, e.g. Newton’s inverses square law of gravitational force, Kepler’s third law of planetary motion, Zipf’s Law, fractals, etc.

In short, simply pointing out that the same one in four ratio has cropped up is several difference places and declaring this to be a bit silly is, well, all just a bit silly and a hell of a long way short of an argument. If you are going to criticise any of these figures then I’m afraid that you’ll just have to get your hands dirty and do a bit of digging to see exactly where they come from and how they been generated.

So, taking it from the top, our first one-in-four statistic is the Anti Bullying Alliance’s claim that ‘one in four schoolchildren are bullied’ – or is it?

Referring to the  actual press release what we find, after stripping away the PR, is that the actual statistic we’re dealing with her looks like this:

In a survey of 875 11-16 year olds, 26% said that they’ve experienced verbal bullying in the last year, most of which (79%) occurred in school.

As yet I’ve not been able to track down the full survey results or a copy of the original questionnaire so although we can safely say that 26% is well with the bounds of a one in four, allowing for rounding, margin of error, etc., there are still quite a few unknowns that need to be taken into account in our assessment of the reliability of this statistic.

How representative is this sample?

We don’t know, although as the survey has been conducted by a reputable market research company (TNS) we can reasonably assume that weightings for standard demographic characteristics, i.e. gender, ethnicity, social class have been applied to the data. But is this enough to give an accurate picture? possibly not – if I was trying to model patterns of bullying as a serious, academic, exercise then there are a range of environmental factors, e.g. type and location of school etc., that I’d also want to factor into the model and these tend to fall outside the kind of adjustments that market research companies undertake as a matter of routine.

And how, exactly, are we defining verbal bullying here? Is this data from a purely subjective question which leave the respondent to decide what verbal bullying is, or did the survey question give examples of the kind of behaviour that respondents should consider to amount to bullying? I suspect the latter, but without seeing the survey questionnaire its impossible to take a view on just how narrowly, or broadly, verbal bullying was defined in the survey.

So this is really a piece of PR-reviewed research, albeit one that sits at the tolerable end of this practice inasmuch as its being used to raise awareness of an important issue and not just to score a bit of cheap commercial publicity, and not a statistic that’s robust enough to use as a basis for formulating public policy.

Moving on to domestic violence, the primary source for the 1 in 4 figure appears to be this passage from an explanatory memorandum that was put to the Council of Europe in 2002 in the course of drafting a resolution which committed European states to taking action to protect women from domestic and other forms of violence.

The drafters were informed of six prevalence studies on domestic violence, and collected data on four more. There was astonishing consistency between the results, as all studies concluded that 25% of women suffered domestic violence and between 6% and 10% of women suffered violence in a given year.

Although the number of complaints of violence to law enforcement authorities varied, it would appear the rates were more consistent than for rape, and that the results of the prevalence surveys agreed sufficiently to suggest that at least one in four women in Europe suffers some form of violence at the hands of a male partner or ex-partner.

Unfortunately, the memorandum does not contain citations for these 10 prevalence studies, so there’s no way to check on data quality but the other primary source that is frequently cited by DV organisations and campaign groups is a 2004 Home Office research study based on data from the British Crime Survey and this does help to put the 1 in 4 figure onto a reasonably solid and rigorous footing:

The BCS estimates that one in four (26%) women and 17 per cent of men aged 16 to 59 have experienced at least one incident of non-sexual domestic abuse, threat or force since they were 16. If financial and emotional abuse are excluded, limiting the violence to the use of non-sexual threats or force, then 21 per cent of women and 10 per cent of men had experienced domestic violence since age 16.

What about mental health?

Luckily enough, Neuroskeptic has already been digging around after the origins of this 1 in 4 statistic, albeit with no real success when it comes to tracking down an original source in the medical literature.

Based on Neuroskeptic’s research and my own examination of media coverage, it seems that the claim that 1 in 4 people suffer from mental health problems emerged during the early 90s in the US a grew rapidly in popularity to peak, in terms of press citations, around 1995/6. Inasmuch as any of the article from this period refer to any kind of source, this figure is most often used in conjunction with a reference to prescribing data for common anti-depressants and I think its no coincidence that the 1 in 4 boom occurred at pretty much the same time as the Prozac boom and the publication of Elizabeth Wurtzel’s memoir, ‘Prozac Nation’.

Interestingly, in its original form, the 1 in 4 figure was commonly cited as a measure of annual prevalence, i.e. 1 in 4 people with mental health problems in a given year, and it appears that its was only in the early 200o’s that this mutated into what is, today, its most commonly cited form as an alleged measure of lifetime prevalence, largely on the back of estimates from the World Health Organisation, although the annual prevelance claim is still kicking around today and does, in fact, appear to have some evidentiary foundations.

Just under a quarter of adults (23.0%) met the criteria (or screened positive) for at least one of the conditions under study. Of those with at least one condition: 68.7% met the criteria for only one condition, 19.1% met the criteria for two conditions and 12.2% met the criteria for three or more conditions. Numbers of identified conditions were not significantly different for men and women.

Adult Psychiatric Morbidity in England, 2007 – Results of a Household Survey (NHS)

That’s not to say that there are a few methodological issues here – this a self-report study after all – and the 2007 edition includes several disorders that were not included in the edition cited by the Mental Health Foundation, one of which (ADHD) is rather mired in controversy over questions of possible overdiagnosis and the undue medicalisation of personality traits that are arguably within the parameters of normal human behaviour.

The overall picture here is complex because it is mired in complex problems of definition and diagnosis. On a strict application of the standard diagnostic criteria, both the annual and lifetime prevalence rates for mental health problems are likely to much higher than most people realise and higher, even, than the commonly cited 1 in 4 figure. But – and its major but – that rather ignore the fact that for many people their personal experience of mental health problems is likely to amount to nothing more than transient and relatively short-lived brush with one of the more common mental health disorders, i.e. depression, anxiety, which they will successfully overcome without the need for treatment and without this episode having a significant adverse impact on their life.

Now we get to the NSPCC’s claim that one in four babies are at ‘high risk’ of abuse and for that we have a very fresh source and some published research, which offers up this summary of its findings:

Around 26% of babies (198,000) under the age of one in the UK have parents affected by either one or a combination of: domestic violence, substance misuse, mental health problems.

14% of babies (109,000) in the UK are living with a parent who is a substance misuser.

19% of babies (144,000) in the UK have a parent who has a common mental health problem

5% of babies (39,000) in the UK have a parent who has experienced domestic violence, in the past year.

And an important caveat:

Please note: percentages are rounded to the nearest whole number. Numbers are rounded to the nearest 1,000.  These are estimates based on analysis of a survey: they are not exact numbers.

The good news is that the survey in question is National Psychiatric Morbidity Survey (n=7403), the bad news is that the number of survey respondents with a baby under 1 year old was only 186, so its a very small subgroup we’re dealing with here and of these 126 were female.

The two biggest components in the estimate are those for substance misuse, which includes alcohol and drug use, and common mental health disorders and its here we run to a problem because the NSPCC have used the broadest possible definitions of both when compiling their estimates. For example, this is how the study reports its findings for alcohol misuse:

129/186 (69.4%) parents reported that they drink alcohol, a total of 131 babies live with these parents therefore 67.5% (131/194) of babies live with a parent who drinks alcohol. Twenty-three parents met criteria for (hazardous or harmful) problem drinking. Since a total of 24 infants were reported to be living with these 23 parents, the proportion of infants living with a problem drinker was 12.37% (24/194). Since the total number of infants (under 1) during the year of the survey was 641,000 in England alone and 756,000 in the UK, the estimated number of infants living with a parent who is a problem drinker is 79,291 (CI=49,594–108,990) in England and 93,517 (CI=58,491–128,543) in the UK. Seventeen parents were hazardous drinkers, and lived with a total of 18/194 (9.28%) babies, 6 parents met criteria for harmful drinking and lived with a total of 6/194 (3.09%) babies and 7 parents met criteria for at least mild dependence and lived with 8/194 (4.12%) babies. For estimates on the total number of children affected in England and the UK see Table 1.

The statistically minded will note, straight away, that there are some pretty wide confidence intervals in the there. The report gives an estimate for the number of babies living with a parent who is ‘problem drinker’ of 79,291 but the actual figure could fall anywhere from a low of 49,594 to a high of 108,990 which raises some significant questions about the reliability of the estimate.

Then there’s this question of what is a ‘problem drinker’? A problem drinker is some who meets the criteria for hazardous or harmful drinking on a standard scale, which is called the AUDIT (Alcohol Use Disorders Identification Test) scale.

So how do you get to be classed as a hazardous drinker? Well, you need to achieve a score on the AUDIT scale of 8 or more and you can easily achieve this with the following answers:

1. How often do you have a drink containing alcohol? Monthly or Less.

2. How many drinks containing alcohol do you have on a typical day when you are drinking? 5 or 6.

3. How often do you have six or more drinks on one occasion? Less than Monthly.

So – and please excuse the stereotyping, its purely for dramatic effect – a night out with the girls every couple of months consisting of four Bacardi Breezers and a couple of shorts to round off the night is enough to get to through to the second part of the test with four points, where you can easily rack up the other four your need with these answers:

5. How often during the last year have you failed to do what was normally expected from you because of drinking? Less than monthly. (Yes, there was that one time when you had a bit of hangover and took a sickie.)

7. How often during the last year have you had a feeling of guilt or remorse after drinking? Less than monthly? (Well, flashing my tits at the barman was a bit out of order.)

9. Have you or someone else been injured as a result of your drinking? Yes, but no during the last year? (Well, I wouldn’t have fallen off my 6 inch heels and turned my ankle if I was sober.)

Congratulations, you’re now a problem drinker and a danger to your kids in the eyes of the NSPCC. Wahey!!!

That said, of the 27 parental substance misusers in this subgroup, 20 were male and for a bloke you can hit the magic figure of 8 on the audit scale if have 3 or 4 pints with the lads every couple of weeks, with a 6 pint session once in while, as long as you’ve took a sickie in the last 12 months and have got into a fight in the pub at any point in your drinking life.

None of this is to diminish the very real risks associated with alcohol and substance misuse, rather its to point out that you really do need to approach with issue with a degree of caution when making inferences from survey data if all you have to work with is the statistical data and not the full survey responses. Many, if not most of the people classified as problem drinkers here may genuinely have a problem, but you still need to mindful of the limitations of these kind of score-based assessment systems – they’re not a 100% reliable, which is why, in clincial practice, they’re used a diagnostic guide but not as the final arbiter of a diagnosis.

As for the data for common mental health disorders:

37/186 parents (19.9%) met the assessment criteria for the presence of a common mental health disorder (a score of >12 on CSI-R*) and these reported having a total of 37 infants, thus (19.07%) of babies under 1 lived with a parent with a common mental health disorder.

*This is a typo and should refer to the CIS-R scale. The CSI-R scale, if it exists at all, is probably a measure of the implausibility of plotlines in a popular American TV crime series set in Las Vegas.

Well, we’ve already looked at some of the issues in looking at the 1 in 4 statistic in the context of mental health, but this is not the only issue here, as illustrated by this explanation of the relevance of the CIS-R score:

CMDs include different types of depression and anxiety. They cause appreciable emotional distress and interfere with daily function, but do not usually affect insight or cognition.

In the APMS survey series, CMDs were assessed in the phase one interview using the revised Clinical Interview Schedule (CIS-R), which covers non-psychotic symptoms in the past week. Responses were used to generate an overall score and to diagnose six types of CMD. A score of less than 12 indicated the presence of no clinically significant neurotic symptoms in the week prior to interview.

Most adults (84.9%) scored less than 12 on the CIS-R. Among the 15.1%of adults scoring 12 or more, half (7.5%) were in the range 12-17 indicating a level of neurotic symptoms that was significant, but unlikely to warrant treatment. The other half had symptoms of a level of severity likely to require treatment.

So, in the full dataset, half the people with a score above the cutoff of 12 for a common mental disorder such as anxiety or depression (or both) had symptoms which were significant but not sufficient to warrant treatment – but what kind of symptoms?

Well this is where it gets interesting because, in general terms, the range of symptoms covered in CIS-R fall under the following list of general headings/categories:

  • Anxiety
  • Appetite (i.e. loss of/weight loss)
  • Concentration and forgetfulness
  • Depression
  • Depressive Ideas
  • Fatigue
  • Irritability
  • Panic
  • Phobias

Now if you’ve got kids then I’m willing to bet that you’ve just read that list and – thinking back to that first year with your little bundle of joy – you’ve been saying to yourself, “Yep… Yeah… Had that… That too… Mmm… Fatigue and Irritability – No shit Sherlock…”

Against, there is a very largely ‘handle with care’ warning when it comes to drawing inference from this data as it relates to people who are known to be subject to a very common cause of (at the very least) anxiety, fatigue, sleep loss and irritability – a baby.

There are, I think, too many methodological issues here to consider the NSPCC’s estimates of the number of babies at a ‘high risk’ of abuse due their living in a household with a parent (or parents) to be in any sense robust, even allowing for the fact that substance misuse, parental mental health problems and domestic violence are known risk factors, because the estimates that the NSPCC are pushing are derived from a very small dataset and on the basis of assumptions, in regards to the prevalence of common mental disorders in particular, which may not be anything like as robust as the NSPCC think.

That leaves us, finally, with the 1 in 4 drink too much which, from the link provided by O’Neill, is based on officially sanctioned recommendations, i.e. 21 units a week for men, 14 for women.

Contrary to opinions held in some quarters, these recommendations do have some pretty decent scientific unpinnings, even if one of the key reports is unfortunately out of print. The main problem here is one of presentation and communication, the recommendations for sensible drinking limits have, over the years, become almost entirely detached from their scientific underpinning and are too often put in a rather nannying fashion and without explanation of where they come from and how they relate to people’s health and well-being.

The message is too often expressed in terms of ‘doctors say’ and not in terms of ‘evidence shows’ and where empirical data is given it is,. more often than not, data relating to hospital admissions and the putative cost to the NHS which a lot of people have difficulty relating to because, even if they do drink a bit than the evidence suggests is good for them, they don’t tend to drink themselves into the kind of stupor that will get them admitted to hospital with alcohol poisoning, or drunk in evironments where they’re likely to get caught up drink-fuelled violence and they certainly don’t see themselves as drinking enough on regularly basis to put them at a significant of developing chronic liver disease, so all the talk of the bad things that are associated with alcohol misuse tend to pass them by as something which affects other people and not themselves.

So what have we actually learned here?

Well, for starters, it would appear that the most robust of our 1 in 4 claims relates to the prevalence of domestic violence, while the 1 in 4 claim for verbal bullying has to be considered the weakest for the simply reason that its derived from a survey conducted my a market research organisation.

So far as mental health’s claim to the 1 in 4 statistic what the evidence suggests is needed in a full-on systematic review – there’s no great shortage of prevalence estimates for specific conditions but nothing, other than the 2001 WHO report, that really tries to synthesise that data into a clear estimate of the prevalence of mental health problems.

The NSPCC’s claims about 1 in 4 babies being at ‘high risk’ of abuse don’t really stack up at the moment – the analysis from which this estimate is derived just isn’t rigorous enough to pass muster and as for the figures for alcohol consumption the problem is not so much whether 1 in 4 people do drink too much but whether many of the people understand why the medical profession believes that they think too much. There, we need to stop treating people like children and actually explain the risks properly in terms of the actual clinical evidence which supports the recommended ‘limits’ instead of banging on incessantly about how much drink-related issues cost the NHS.

So yes, there is a hint of Dickensian doom-mongering about the NSPCC’s contribution to the 1 in 4 meme but otherwise we can believe at around a quarter of British women will be threatened, violently assaulted and/or sexually abused in their own home, by their partner, at some point in their life and that its entirely possible that 1 in 4 people, and perhaps considerably more, will experience symptoms consistent with the onset of a psychiatric disorder, even if this turns out only to be a short-term transient episode in their life.

As for Brendan O’Neill, the one thing we can all believe is that he spend most of his life talking entirely out of his arse.

  • Pingback: Unity()

  • Pingback: Dakota Erickson()

  • This is a very thorough answer. I didn’t have the patience for data-mining, so I responded to O’Neill’s article by Going straight to calling him a numpty.

  • Paul Newman

    I found the process fascinating ,its a marvellous post .The problem I have is that none of the findings look remotely like real life, not mine anyway. The article works because it knows  a majority will have the same reaction.
    Why ? 
    My guess is that there are selective facts between charity and survey left out and  the impact comes form these , arresting statistics grouped together . When they are it is clear they are not true .
    So I go straight for , the victim industry are lying and work backwards. 

  • Pingback: Angie Gordon()

  • Neuro Skeptic

    Very interesting.

    I don’t know much about 1 in 4 except in regards to mental health, so thanks for that.

    There is a grain of truth in O’Neill’s skepticism of 1 in 4 which is that I think it’s become a kind of nice round number which is used as a kind of shorthand for “Surprisingly Many People”.

    As I speculated in my original 1 in 4 posts, I think 1 in 4 is popular because 1 in 3 is “too high” (creates incredulity) and 1 in 5 is “too low” (not striking enough). The Goldilocks principle if you will.

    Clearly it’s unlikely that all of these things affect exactly 25% of people, no more no less. It must be serving as a rounded off number to an extent.

    But while 1 in 4 certainly is a meme as O’Neill suggests that doesn’t mean it’s wrong in any given case. You need to look at the evidence. Not just point and sneer.

  • Pingback: Sharon Howard()

  • Pingback: Dakota Erickson()

  • DanB

    In comparing you treatment of the above studies and statistics, and your treatment of studies dealing with abortion and physical/psychological sequelae, your analysis and conculsions seem disparate, probably because of your already-held views on those topics, with a particular bias when it comes to abortion.