subscribe: Posts | Comments      Facebook      Email Steve

The Critic vs. the Computer: A case of perceptual discrepancy



Did you know that I prefer organic wines to non-organic wines? I didn’t, either. But then I read this new paper from the American Association of Wine Economists, entitled “Does Organic Wine taste better? An Analysis of Experts’ Ratings,” and I found out that, yup, I do.

Well, kinda sorta. See, the paper’s authors decided to study “data from the three influential wine expert publications: Wine Advocate, Wine Enthusiast, and Wine Spectator,” and as it turned out, “During our period of study [74,148 wines produced in California between 1998 and 2009], the main tasters for California wines for Wine Advocate, Wine Enthusiast and Wine Spectator were Robert Parker, Steve Heimoff, and James Laube, respectively.”

The big P-H-L! They took our scores, crunched them in that esoteric way only economists can, and lo and behold, “Our results indicate that the adoption of wine eco-certification has a statistically significant and positive effect on wine ratings.”

How much? Not a lot: Being eco-certified,” the authors found, “increases the score of the wine by 0.46 point on average.”

Well, one hardly knows where to begin. Right off the bat, I have a problem when the lesson that people will take away is that P-H-L (and by extension major critics) prefer organic wines to non-organic ones. Less than half a point difference? I suppose if they fed 74,148 scores into a computer and found a 0.46 point difference, then who am I to argue with HAL? But a 0.46 point difference doesn’t seem like very much to me. It’s not even round-uppable to the higher score (87.46 rounds down to 87).

But wait, there’s more. The following factors also had an impact on the scores of organically-certified wines, according to the paper:

  • ” a 1% increase in the number of cases will decrease score by 0.003


  • ” An increase in the number of years of certification experience by one [winery] decreases score by 0.09 point.”

Confused? I am. So the more cases wine the winery produces, the lower the score is; but the longer the winery has been certified organic, the lower the score also is!

How about the winemaker’s hair color? Did they include that?

The authors also counted the number of words in each review and found this: “Next, we examine the impact that eco-certification has on the number of words used in wine notes. As shown in regression (1) of Table 6, wine notes of eco-certified wines are not significantly longer than those of conventional wines. However, as shown in regressions (2) and (3), eco-certification increases the average number of positive words by 0.4 but has no statistically significant impact on the number of negative words.”

My interpretation of this is that it’s gibberish. The authors compiled a list of words [Table 7] but I don’t understand how they infer whether their use is positive or negative. Is “jammy” positive or negative? Do Parker, Laube and I even use it in the same way? How about “offbeat”? Is that good or bad? And “peat”: if I tasted that in an Islay Scotch it would be good, but in a Chardonnay?

The authors also state something that I don’t think is objectively true, or, even if it is, is irrelevant. “Second, as a related point, wine experts have a better knowledge about wine eco-certification and are able to differentiate between different types of eco-labels, namely organic wine and wine made with organically grown grapes, which represent different wine production processes with different impacts on quality.”

I’m not going to sit here and tell you I know the difference between different types of eco-labels. There are so damn many (different certifying agencies, “natural,” biodynamic, etc.), I get confused—and, while I’ll let Parker and Laube speak for themselves, I bet they get confused, too. Besides, if “All the publications claim blind review,” as the paper’s authors write, then we critics don’t even see the labels when we’re tasting and reviewing (much less would we have a tech sheet in front of us).

But finally, this statistic seems to be to be the last nail in the coffin of the study: “On average, 1.1% of the wines in the sample are eco-certified.” By my calculations, that’s a little over 800 wines—out of 74,148. I fail to see how you can extrapolate any useful information from such a small sample, compared to the huge number of wines in the study. Apples and oranges.

I’m no economist, it goes without saying. If I were, I guess I’d spend my days crunching numbers and coming up with interesting factoids. But I have to say, I don’t see the point of this particular study—not if it’s going to be used to make a claim that I don’t regard as true. For the record, let me say that I do not think organic wine is better. And you know what? I don’t care what the numbers say.

  1. It sounds like the authors of this paper are confusing correlation and causality.

  2. Too lazy to read all that but it seems quite reasonable that eco-certified wines are actually better than non eco-certified wines. Maybe for real reasons, but almost certainly as a *signal* for a broader commitment to quality (BrianM”s correlation vs. causality above).

    On the whole, organic (et al) growers simply do a better job than non-organic growers. They pay more attention to the vineyard (they have to) and that translates into better canopy management, more fruit thinning, better timing of irrigation, etc. This isn’t about the advantages of organic viticulture, but simply the associated mindset that improves all decisions. It’d be pretty embarrassing to be planting cow horns at midnight only to be growing grapes that taste like asparagus.

    Same concepts apply to wineries.

  3. I agree with the other commenters, that organic certification may simply be a measure of commitment to quality rather than inherent superiority of organic techniques.

    The other thing I note is that, although the paper reports the findings re organics as “statistically significant,” that is true only at the p<0.10 level, which is not exactly a powerful finding. (It's also a little aggravating that I had to go to the tables in the back to find that out — my understanding is that it's generally considered best practice to report the p-values in the text so people don't have to go digging.)

    Anyway, one interesting comment in the paper is that there seemed to be an abnormally low number of 89-point wines.

  4. I could not disagree more strongly with Mr. Brill’s comments which in my view, are sweeping and unsubstantiated by anything other than his own assumptions. I know dozens of champagne makers (and I believe the same is true in other wine regions) who have decided not to chase organic certification yet who are every bit as meticulous in their work in the vineyard and in the winery as their organically certified colleagues and it is insulting and incorrect, in my opinion, to question their commitment to producing the best possible wines they can.

    One of the reasons many chose not to be organic is the reliance that entails on using copper sulfate in the vineyard which they regard as highly damaging to the environment.

    So who has more regard for the environment and a more rigorous approach to their work, those wine makers who jump on the currently trendy bandwagon that advocates organic certification to please a minority of consumers who, in reality, understand almost nothing about what being organic really means and media who are happy to have some new cause to write about, or those winemakers who, despite the increasingly noisy and sometimes ill-informed frenzy surrounding everything organic, decide to stick to their non-organic guns because they are convinced it is better for the quality of their wine and for the environment?

    There’s a legitimate debate to be had on the subject but please let it be based on something more than generalisations and personal assumptions.

  5. Bob Henry says:

    Adding to the discourse . . .

    From Decanter magazine
    (January 26, 2016, Page Unknown):

    “French winemaker drops organic status for ‘better treatments’”

  6. Tried to post a much longer response but it fails every time, so:

    * I didn’t say that organic viticulture produced better fruit or was better for the environment – the latter is almost never true.
    * Yes, there are many, many producers who decide to go with conventional growing and are producing great wines. But “on the whole,” eco certification is a signal that the winegrower is committed to quality and that if you look at wines within a class, on the whole you’ll find the higher-end (points and/or prices) producers with more eco certification than the lower-end (points and/or prices) producers.
    * This is not causality. I could have said the same thing about $2 corks or heavy bottles. Those are signals. Of course there are great wines made with composite corks. And there are crap wines made with $2 corks. But, on the whole, wines with $2 corks are better than wines with composite corks. It’s just a signal.

  7. Scott Wiegand says:

    Sorry Jiles, Michael is right. A generalization can be made of the average organic producer vs the average traditional (non-organic) producers where the average organic producer spends more time and energy focusing on high quality growing and production processes. Of course there are outliers but you can not define a group based on the outliers. When looking at a large group such as non-organic producers, there are bound to be extremely high quality and extremely low quality versions (i.e. 2 buck chuck). The same is not true of organic producers. There just aren’t organic 2 Buck Chucks.

    You can think of it this way: just because there are some fantastic burgers out there made by dedicated and serious burger makers doesn’t mean that the average hamburger sold is not McDonalds junk food. The same is true of beer. With 90+% of US market share owned by swill like Bud Light, the average beer sold in the US is garbage but we all know there are phenomenal examples out there to be had.

  8. Bob Henry says:

    Let’s define our terms . . .

    “Organic Wine Definitions – Behind The Label”

Leave a Reply


Recent Comments

Recent Posts