A study commissioned by the UK Food Standards Agency concluded that:

there is little, if any, nutritional difference between organic and conventionally produced food and that there is no evidence of additional health benefits from eating organic food.[1]

Yes and no. What the study actually shows is that organic food typically does have higher levels of important nutrients but the high degree of variability in the measured levels means that we cannot be 95% sure that these higher levels are not the outcome of chance. The Food Standards Agency and the report’s authors have misled people interested in this topic and should revise the summaries of their work.

***

The FSA study abuses statistics. Let’s take one example. Flavonoids are part of a plant’s defence mechanism against pests. So it is plausible that organically grown fruit and vegetables might contain higher levels because they might have had to protect themselves against insects. Conventionally grown products have the advantage of pesticides and so don’t need to produce flavonoids to the same extent. There’s a reasonable scientific hypothesis that organic foods should contain more flavonoids.

In reasonable doses, flavonoids probably have benefits to human health. When eaten they are excreted quickly but may prompt the presence of higher levels of uric acid in the bloodstream which may help ‘cleanse’ the body of toxic products.

The FSA report looked at 158 experiments that measured the flavonoid content of organic foodstuffs, including strawberries, wine, apples, and tomatoes. On average, these foods contained 38.4% more flavonoids per unit of weight than their conventional equivalents. Whatever you might have read in the press about this study, this means that organic food may well have more flavonoids than conventionally farmed equivalents.

The study denies this for two reasons. First, it removes most of the data from consideration because it doesn’t meet the best standards of scientific research. When the slightly dodgy studies have been taken out, we’re left with only 48 data points. In these cases, the average flavonoid content was only 32.9% higher than the conventional equivalent, down from 38.4% in the wider sample.

The second reason is more important. The high degree of variability in the results means that we can have less than perfect statistical confidence that the organic results really are better. In fact, rather than being 99% statistically certain (the figure for all 158 studies) our confidence falls to 78%. This is principally because some of the surveys of organic fruit or vegetables show a smallish reduction in flavonoid content. Most are much higher, but some are lower. So the careful statistician says that we shouldn’t assume that the average result of a 32.9% increase is truly valid.

I will, if I may, use an analogy to explain this a bit more. Imagine we take a group of 100 20-year-old males and ask them to run one hundred metres. We measure the time taken. Then we ask 100 20-year-old females. On average, the males will be faster. But some females will be faster than some males. So someone looking at the data cannot be absolutely certain that males are, on average, faster than females. But as we increase the number of runners, we are increasingly sure that the average male is genuinely faster than the average female and we are more confident about quantifying the underlying difference. We can do this both because we have a larger number of times but also because we can better measure the underlying variability between males and between females.

In the food study, cutting out the dodgy data cut the number of data points by 70% (so, as it were, we had fewer males and fewer females). And the apparent reliability of the data fell. But, nevertheless, the organic food (the males) was on average significantly better than the conventional food (the females) even though some results (5 out of 48 trials) suggested that conventional food had measurably more flavonoids than organic equivalents.

The result for flavonoids is replicated with the nutrient beta carotene. Beta carotene levels were over 50% higher in the average study, falling to 21% once the slightly dubious studies were extracted. In fact, organic food contains – on average – a higher percentage in 18 out of 23 specific nutrients. It is simply untrue to say, as the FSA does, that organic food contains ‘no difference’ in nutritional content. Organic foodstuffs studied in this work actually contained measurably more nutrients. But the statistical techniques used showed a relatively high probability that this was simply a matter of chance. Few nutrients showed the required 95% confidence level.

If I may, I want to repeat the comment from the FSA press release that I carried in the first few lines:

there is little, if any, nutritional difference between organic and conventionally produced food and that there is no evidence of additional health benefits from eating organic food.

What the FSA should have said is as follows: ‘there appear to be substantial differences in some of the nutritional compounds in organic and conventional foods. But, in most cases, these differences are not great enough to meet the standard statistical requirement of 95% confidence that these differences are not due to chance. The higher apparent availability of some micro-nutrients in organic foods may be very important for human health. Nobody yet knows, certainly not us. We will continue working on this issue rather than publishing conclusions which are not based properly on science or on statistics.’

We can never be truly certain about anything. Science moves ahead by noticing patterns in data and trying to find plausible explanations. And not by baldly stating that because differences are not large or consistent enough, that there can’t be an underlying pattern.

Why is a piece about organic food and nutrition carried on a blog about energy and climate change? Because if we carried the FSA conclusions through to global warming issues, we would be taking no action on climate change. Virtually nothing we think that we know about climate is understood with a confidence exceeding 95%. As denialists are ever fonder of pointing out, 1998 was the hottest year in recorded history. If the FSA was in charge, this would surely mean that we would now be claiming we did not have a sufficiently high level of certainty to want to bother to reduce global emissions.

The fact that the immediate past chair of the FSA, Lord Krebs, now runs part of the UK Climate Change Committee’s activities should therefore make us very nervous. Footnote [1] Food Standards Agency, ‘Organic review published’, Wednesday 29 July 2009 http://www.food.gov.uk/news/newsarchive/2009/jul/organic [accessed Wednesday 12 August 2009].

This article was also published on the Guardian Environment Network on Thursday 13 August 2009.