deodands

Times drops signal, shows noise

A Times story about Waitrose being worst at selling alcohol to minors.

Findings from an alcohol enforcement campaign suggested that Waitrose had the worst record for selling alcohol to under-18s, followed by Somerfield and the Co-op.

Waitrose sold alcohol in 22 per cent of cases where trading standards officers used under-18s in secret test purchases. The other big chains similarly tested were Somerfield (20 per cent) the Co-op (17 per cent), Tesco and Sainsbury (both 16 per cent) and Morrisons (15 per cent).

To be fair on Waitrose staff, they were probably so shocked to see anyone who looked under 18 walking around the most expensive supermarket chain in the country that they forgot to ask.

A rather better way to be fair to Waitrose staff might be to observe the likely uselessness of those numbers to support this conclusion. Think about opinion polls - carefully sampled, typically several thousand respondents, and they still report a sampling error of +/- 3%. Journalists are not in general good at pausing to consider how much numerical difference is likely to be attributable to noise and the Times ones are apparently no exception. In fact the only time one routinely sees uncertainty estimates and sample sizes is in those opinion poll reports, presumably because people subsequently remember when newspapers call elections wrongly.

A little digging on the Home Office site reveals firstly that the journalists have factually misreported - there was, contrary to their claim, another supermarket involved, Asda down at 7%, much more of an outlier than Waitrose.
Secondly it hints at a sample size by discovering that there were nearly 600 charges as a result (a rather large sample: obviously the police are finally discovering the true meaning of power). Across seven supermarkets, one of which is small and regional (until Waitrose Wilmslow some of us Northern affluent, young, urban consumers have been guiltily supplied by Ocado from, yes , Stevenage...) its unlikely that more than 50 were from Waitrose, so about 250 trials maybe. Estimating .22% from 250 trials gives
you what the opinion pollsters would call a range of error of about +/- 5% (ie the 95% confidence interval for the true population parameter in 250 binomial trials with 55 successes is 17.02-27.65%).

A third observation is that all of these differences are dwarfed by the massive difference observed between the trials
done last summer (nearly half of all underage purchases unchallenged) and last month. Since the Home Office press release explicitly mentioned this, the Times must have consciously chosen to neglect this, probably meaningful difference, in favour of a probably meaningless one dissing Waitrose. They probably know their readership goes to Sainsburys (scroll to the bottom, pausing to marvel at Asda.)