The Charlatan and the DWP

A couple of months ago, myself and Alex Harrowell did a big of digging into the background of the company whose ‘voice risk analysis’ technology is being introduced by the DWP as a means of screening benefits claimants for the possibility that they may be committing fraud. (see here and here)

As I recall, I think we more or less came to the conclusion that there was something altogether a bit dodgy in the whole exercise and that while there may be some moderately interesting, if limited, science behind the idea that you can identify when an individual is feeling stressed, the claims being made for the system that the DWP were bring in were unlikely to stand up to scrutiny.

In short, apart from scaring a few people into behaving a little more honestly, we reckoned that the system itself wouldn’t actually do what its developers, and Israeli company called Nemesysco, claimed it would do.

Today, while catching up on some of the commentary on Ben Goldacre’s spat with LBC/Jeni Barnett, I ran across some rather intriguing additional information about Nemesysco and its proprietor, Amir Liberman, in the form of a rather unusual academic paper by two Swedish researchers,  Francisco Lacerda and Andrew Eriksson, published in 2007 in the International Journal of Speech Language and the Law, a magazine for voice experts working for the police and security services.

The paper is unusual, for one thing, because it appears to so enraged Liberman that he’s written to the researchers’ publishers and indicated that Lacerda and Eriksson could be sued for libel if they publish any more papers on the same subject.

The paper, which reviews 50 years of research into the use of lie detectors, is entitled ‘Charlatanry in forensic speech science: A problem to be taken seriously’, which should give a clue as to precisely why Liberman is rather upset about it, but there’s more to this than just a title, as the paper goes to pay particular attention to Liberman/Nemesysco’s LVA (“Layered Voice Analysis”) system, which the DWP is using via a third party supplier.

On the subject of how the system, ostensibly, works, the paper explains that:

Contrary to the claims of sophistication – ‘The LVA sofware claims to be based on 8,000 mathematical algorithms applied to 129 voice frequencies’ (Damphousse et al. 2007: 15) – the LVA is a very simple program written in Visual Basic. The entire program code, published in the patent documents
(Liberman 2003) comprises no more than 500 lines of code. It has to be said, though, that in order for it not to be possible to copy and run the program as is, some technical details like variable declarations are omitted, but the complete program is unlikely to comprise more than 800 or so lines. With respect to its alleged mathematical sophistication, there is really nothing in the program that requires any mathematical insights beyond very basic secondary school mathematics. To be sure, recursive flters and neural networks are also based on elementary mathematical operations but the crucial diference is that these operations are used in theoretically coherent systems, in contrast to the seemingly ad hoc implementation of LVA.

Oh dear, 800 lines of code sounds about the size of the project I had to turn in when I did my ‘A’ level in Compter Science more than 20 years ago (using good old BBC BASIC) and the sophisticated mathematical algorithms turn out to be nothing than you might expect a GCSE student to cope with?

The researchers, working with a Swedish journalist, also managed to track down Liberman and, in what is an extreme unusual section to find in a journal paper, entitled ‘Who is Mr Liberman?’, note that:

We might as well have asked: Who is Nemesysco, the company behind the LVA products, because Mr Liberman and Nemesysco seem to be one and the same.  Damphousse et al. (2007: 14) report as follows: ‘The LVA was developed in Israel by Amir Lieberman [sic] who applied mathematic algorithm science to voice frequencies’, giving the impression that the program is based on some advanced mathematical theory. As we have pointed out, this is far from the truth.

And the truth is?

Afer some research he [ Arne Lapidius, a Swedish journalist working in  Israel for the Swedish daily ‘Expressen’] managed to locate Mr Liberman, a 32 year old (in 2004) businessman in a small office in the town of Natania. The business appeared to be a one-man operation. Mr Lapidus interviewed  Mr Liberman about his academic background and was told that he basically had none. He has no degree (never had time to get one, he explains) but has taken some courses in marketing at an Israeli open university. As we have explained above, the LVA is a simple program written in rather amateurishly used Visual Basic. Given what we now know about Mr Liberman, that is about what one would expect rather than ‘8,000 mathematical algorithms applied to 129 voice frequencies’ (Damphousse et al. 2007: 15). What still remains for us to understand is how insurance companies, security agencies, police depart-ments can be willing to invest hundreds of thousands of dollars, pounds, and euros in equipment without ever asking who are behind the products, what are their qualifcations, what are the scientifc principles upon which the products are based.

[UPDATE]

So as to avoid any confusion regarding the passage above…

In 2004, Professor’s Eriksson and Lacerda appear to have become embroiled in a ‘controvesy’ with Nemesysco’s Scandinavian representatives when, after making a careful study of the claims made for the system, they challenged its scientific validity. In simple terms, their examination of the ‘scientific’ basis upon which the compnay claims its products operate led them to conclude that the system, itself, is a wholly ‘bogus product’. Elements of this dispute appear to have spilled over into the Swedish press, attratching the interest of Arne Lapidus, who conducted his own investigation into Mr Liberman’s background, reporting his own findings in the Expressen.

This information was then incorporated into their 2007 paper as relevant background to their own scientific examination of the system.

….

The paper also notes that Liberman’s LVA system sells, in the US, at $25,000 for a laptop (estimated cost $2,000),  his ‘amateurishly’ written software and the training necessary to use the system.

As in numerous other recent cases, when resorts to the law rather than empirical research to defend a product that supposedlt operates within the realms of science then that’s usually a sign that something is seriously amiss.

In this case, Eriksson and Lacerda put their view of Liberman’s LVA system in no uncertain terms:

The program code is part of the patent documents and may be downloaded from patents on-line. Any qualifed speech scientist with some computer background can see at a glance, by consulting the documents, that the methods on which the program is based have no scientific validity.

And yet the DWP have decided to buy into this system, as have several local authorities.

Normally I’d leave it there, but in this case I fully intend to raise this both with James Purnell and forward Eriksson and Lacerda’s paper to the National Audit Office and the Public Accounts Committee because, as far as these researchers are being concerned, the public purse is being bilked to the tune of many hundreds of thousands of pounds for a system that looks, to them, like someone’s A level homework.

  • Pingback: Links and stuff for February 6th - Chicken Yoghurt()

  • Even taking it as read that his program can detect “stress” in a person’s voice, it’s a big leap from there to detecting fraud. If you’re legitimately trying to get the DWP to give you benefits, chances are you’ve lost your job/acquired a disabling condition/are experiencing marital breakup/facing unplanned parenthood/etc and have no money – wouldn’t that sort of situation make most people a bit stressed? Particularly when making a telephone call to the organisation that decides whether you and your dependants can have some money to survive on?

  • Pingback: AlisonW()

  • Pingback: Arkady Rose()

  • Pingback: Liam Green-Hughes()

  • Pingback: Voice analysis software doesn’t work « Amused Cynicism()

  • Pingback: Snake Oil « Alternate Seat of TYR()

  • See here for my take on this.

  • [800 lines of code may or may not be powerful and important. That’s not a line to go with in any seriousness. When I learned we used to cut cards for programs and data. And less was more back then. I did it for wages for a while. Oh my god. Less was more.]

    And, presumably, this charlatan solution to the non-critical-problem of finding housing benefit cheats would compare the voice when Jacqui Smith or Chris Grayling or Charles Kennedy was explaining why Kanye West was not on their iPod with their scarcely credible explanation of their ramped up second home allowance … and then find the system guilty of producing contrivance, accountancy, and jeopardy.

  • As someone who has actually done some work around biometric systems, I can tell you that they just don’t work. Voice is particularly prone to false results and I wouldn’t trust it for anything.

  • Robert

    Of course they do not work, but in America they use to be used by the FBI for new entrants, but so many failed it was removed, then the military removed them, so a new market had to be found, funny how Tony Blair brought them here the same Tony Blair that had a payment of $500,000 payment for his greatness in Israel two weeks ago.

    The fact is voice stress analyzers do not work because anyone who phones up the DWP, would end up failing because your blood boils at the stupidity of this group. I phoned up with a broken back a spinal lesion and both legs in plaster after my accident, to be told are you sure you cannot work.

  • Liking Laski, But Only The Early Stuff

    MoT,

    My friend, Sam Tarry, is running to be chair of Young Labour. You may find him interesting, as he’s worked for Unlock Democracy, is an Anti-BNP activist whose lived almost all his life in Barking, and has had favorable reviews from Liberal Conspiracy.

    Anyway, we we’re wondering if you’d like to talk to him as we’re trying to get some Blog coverage. If you speak to him and think he’s appalling, then by all means go ahead and trash, that’s the risk we take. But I think your politics have quite a bit in common and it’d be a good conversation.

    thanks for considering

    LLBOTES

  • Interesting post. Seems strange that the mainstream media haven’t picked up on this. Perhaps they have, but they like the idea of those on benefits being given a tougher time I guess.

  • Well, count us as mainstream media who have caught up on this – see http://www.guardian.co.uk/technology/2009/mar/12/voice-analysis-system-vra and followup blog post on Tony McNulty results (http://www.guardian.co.uk/technology/blog/2009/mar/12/voice-risk-analysis-lie-detection-benefits-government-results).

    Let us know if the NAO takes an interest…

  • Oopps:
    “Lie detectors are unlikely to be used in job centres after genuine benefits claimants were wrongly identified as “risky” during a pilot project.”
    http://news.bbc.co.uk/1/hi/england/nottinghamshire/7938447.stm