subscribe: Posts | Comments      Facebook      Email Steve

The straight dope on wine reviewer inconsistency: A long post on an important topic


I alluded yesterday to reports of judge inconsistencies at the California State Fair’s annual wine competition. Today I want to go more deeply into this report, because I have a feeling it’s going to have a certain impact, both in the press and, by extension, in people’s minds. You can read about it here at the website of the American Association of Wine Economists, whose reports I have frequently cited in this blog, and whose efforts I respect. Or check out this synopsis.

Briefly, the AAWE study found that over a three-year period, 90% of the judges — some of whom are my friends — basically blew it in blind tastings when presented with the same wine. Blew it so badly that, in the AAWE’s estimate (and these guys are statisticians), the results are “amazing, and show[s] us that the Wine Competition Results are seriously flawed.”

This is the secret of wine reviewing, the fact that scores/reviews are not replicable by the same tasters over time. Now, instead of asking why this is, the proper question ought to be, why should anyone think they should be? Replicable, that is. I realize I’m going out on a limb here, because I’m a wine critic who uses scores, and it’s a little weird for me to be sawing off the limb I’m sitting on. But the truth is the truth, and the only way to deal with inconsistency is to address it head on.

I’ve said it a million times: WINE REVIEWING IS NOT ARITHMETIC. If I say 2+2=4 and you say 2+2=5, YOU’RE WRONG. You may not like it, but you are objectively incorrect (unless you live in a parallel universe where 2+2 may well equal 5). However, if I say this wine deserves 97 points and you say it’s undrinkable, then neither one of us is right or wrong. You’re entitled to your opinion and I’m entitled to mine.

That should be easy enough for everyone to agree to, right? The problem is that there are powerful forces out there that want wine reviewing to be like arithmetic. One of these forces is our natural human inclination that desires for our “authorities” to be correct in their assessments. In a confusing, incomprehensible world, we long for gurus, priests, shamans, masters — call them what you will — to explain things, so that we don’t have to try and understand them ourselves. As I said, nothing wrong with that; it’s only human. Trouble is, in this day and age when authorities routinely get everything wrong (Iraq, the economy), we should be dubious of pronouncements from on high.

Does this mean that wine critics’ findings are completely irrelevant? No, but… there’s always a but attached. (No puns, please.) The reason why wine critics still play an important role in our understanding of wine is because those of us who have been at it for a while are better at evaluating wine than most people. We’re not perfect, but when you’ve tasted, and thought about, as many wines as I have over my career, you do learn a thing or two about quality. You learn to appreciate things like balance and harmony, and to detect flaws like TCA, and to understand when a wine has too much residual sugar, etc. etc. Okay, I can hear the skeptics going, “Fine. But what about those S.F. Wine Competition inconsistencies? How do they square with what you just said about experience?”

Tough question. I’m not going to pretend to rationalize it away. I’ve conceded many times that if you give me the same wine twice, I might give it different scores. Which leads to the question of why this should be. Either it’s because my palate, or brain, varies from day to day (which we know is true) or because there may be bottle variation (which we also know is true) or because of some combination of both. That’s just the way it is.

So how should the consumer interpret this? I expect some will say all critics are full of it and ought not to be trusted. Fine. Others of them (if they’re even aware of this, and other, studies) will choose to disregard the inconsistencies and stick with their favorite critics regardless. That’s what I would advise. But then, I have a dog in this race. Ultimately, it’s what you, the consumer, decide. You’re the kings and queens of the wine industry; everything revolves around you. You may not be feeling empowered in these perilous times, but believe me, you are.

So there it is. Wine critics are not infallible. Far from it. Parker and Laube might not tell you that. I just did. Salud!


  1. I think also a point scale, especially a 100-point scale contributes to an illusion of mathematical accuracy (Differentiating between a 94 and a 95, say, would indicate a high level of precision, higher perhaps than truly exists).

    I don’t mean to be point-scale bashing. I don’t use them personally; I’m also far from a professional.

    I do think that the illusion created by a highly precise numerical scale, however, contributes to the reaction.

  2. Thom Calabrese says:

    I always find it interesting how much importance people put on reviews and reviewers. I think it’s silly! Reviews and reviewers are in the business of keeping what they believe and say relevant! Otherwise they wouldn’t have a job.
    People generally want to be led. There are very few that dance to their own drummer, so to speak.
    I was in the wine business for over 10 years and I was always amazed at how many customers cared more about Parker or Wine spectator then their own palate.
    So when they find out that wine isn’t arithmetic they are shocked. If they are angry, it’s at the wrong people. I never fault “the experts” at selling their advice, opinions. Everyone is free to choose to buy or not. It’s capitalism at it’s finest!
    I use reviews as information from very fallible people, who give mostly subjective information and take it as such.
    More then anything else, I rely on the my taste and what I precieve as good/bad. Because when it’s all said and done, I’m the most important opinion in my wine drinking world and I’m the one paying the frieght!

  3. Steve, as usual, you deserve credit for taking on a topic that other critics won’t touch with a 100-point pole. And the volume and quality of the comments your posts like this generate speak to the fact that the topic is still of great interest and importance. THe two comments above are nail-on-head true. I would add:

    1) If a critic cannot replicate a score with statistical reliability (and, as you say, “I’ve conceded many times that if you give me the same wine twice, I might give it different scores”), then numbers should not be used to begin with. It is inherently misrepresentative, if not outright deceptive.

    2) The real reason scores (far more than “medals”) perpetuate is because marketers and sellers are lazy. This is not easily changed because (as noted by Thom Calabrese), people want to be led. Scores are built on the American love for shortcuts. There is nothing inherently wrong with this, except it is clear that there is a dark side in how the ratings/medals/awards are used/misused.

    3) Eventually, the *truth* gets through to wine lovers. The truth — as evidenced both by the Hodgson study and your own impressions as a critic changing with the same wine @ different times — that tasters are human. Let’s take this as a humbling reminder that wine *is* is a shapeshifter. It’s meant to vary by context, food, time and even mood. People who drink a lot of wine over time come to understand this and embrace it. When they do, ratings (and tasting notes) then become closer to your apt definition of them: (paraphrasing) a single individual’s assessment of a wine when it was tasted. And there’s nothing wrong with that. Over time, a critics’ notes form a body of work; tasting inclinations are apparent and people can follow critics whose notes/numbers concur with their own.

    4) To bring it back on topic, it will be interesting to see whether the Hodgson study spills over into the mainstream media. (LA Times ran a piece, which will probably get picked up widely.) Human fallibility is not news, but people who think they are infallible are. It reminds me a little of the Wine Spectator to-do last summer… sacred cows get served humble pie.

  4. As someone who is very, very new to the wine world (I’ve done a lot of reading, not a lot of drinking), the whole reviewer thing seems very hackish to me. If you serve a reviewer the exact same wine from the same bottle and they claim two are undrinkable and one double-gold worthy, there is some thing wrong. I severely doubt what some people taste in wines in some of their reviewing notes. Maybe it’s because my palate isn’t nearly as close to refined as their is, but honestly…slate? Flint? Ink? How are these flavors determined? Does the reviewer taste rocks and break open a pen and dab it on his tongue? Some notes I can agree with. Others, like pencil shavings and flint make me chuckle.

    Point being, I don’t really care what reviewers say. I’ll drink what I want. Is it interesting and, dare I say, a bit fun to look into what others say about the same wine? Sure. But just because someone says a wine is swill I won’t believe it until I taste it.

  5. Glenn, you have the right attitude, it seems to me! Regarding “slate” or “flint” etc., I agree these are difficult terms to understand, although I’ve used them. The best way I can explain is that certain wines have a minerally tang to them, underneath the overt fruit and oak (if any). Imagine touching a piece of cold steel to your tongue. It’s not so much a flavor as it is a feeling in the mouth, a firmness. I’ve also touched the tip of my car ignition key to my tongue and found something metallic, and also mixed with diesel or petrol. Our vocabulary for mouth impressions is severely limited, compared with our vocabulary for sights and sounds.

  6. Tish, good comment. I suspect the study will be picked up — it will be interesting to see. Anyhow, I’m reading a book now on the making of the atom bomb (a little light reading…) and just finished a chapter on Heisenberg, who as you know developed the famous uncertainty principle. I sense some analog between that and tasting wine, although I haven’t quite worked it all out intellectually. If I do, I’ll blog on it.

  7. Context and environment are everything. Competition judges will typically taste in flights of the same variety sometimes of the same age and price point. I do not know if reviewers do the same? A luxury priced Cabernet will present itself quite differently when pre and ante-ceded by nine others of the same ilk. Change the environment and it will likewise (however slightly) morph. Reviewers are practiced opinion peddlers with nothing more to sell than their credibility. As such, they have the freedom to use Mom’s apple pie as the criterion/standard by which all apple pies they taste will be judged and reported. Wine competition judges should not presume to grant themselves that same authority. Of course, they do. By bringing their own prejudices and provincial style preferences to what should be a comparison contest, the true spirit of a structured competition is subverted. Jerry Mead said it best each time he would address us as judges at the Orange County Fair Wine Competition, ‘If it’s not flawed and it is the best of its type on the table that day, it’s a Gold’ . If you doubt the ego and agenda taint in wine competition results just ask yourself how a class can achieve but a silver medal as its highest award. Most “judges” actually want to be reviewers tasting against some phantom standard ensconced within their personal and ephemeral memory (Mom’s apple pie). To be more consistent, competition tasters should stick with reporting and leave the editorializing to the wine scribes. A Jekyll and Hyde trick for those of you whom purport to be both.

  8. Boy, the way people are reacting to this you’d think it really matters. The truth is wine journalists like you and Parker and Laube and Tanzer make a living, such as it is, by expressing your opinion, numerically or otherwise. And Wine Competitions, and I’ve judged in many, make money because the results they produce are marketable for the wineries that submit wines and win Gold Medals. It wouldn’t surprise me if I were one of the judges who had been unable to hate a wine three times in a row. So what? The quality of any competition hinges on its weakest judge. But if a competition judges 4700 wines, as was the case with the recent SF Chronicle Competition, it is likely to make as many qualitative mistakes as you might judging those same 4700 wines over a longer period of time. Again, so what? Consumers aren’t stupid, though we often seem to talk of them as though they’re cattle and don’t understand how wines are judged and numbered, if they buy a bunch of Gold Medal winning wines and then hate them, they’ll stop buying on that basis and the competitions will eventually die off.

    I don’t really find any of this, the study done at the State Fair or the resulting “controversy” about it, particularly notable. Wine Competitions don’t advertise that they’re omnipotent and infallible. They are simply a way of promoting wine in general to a public that desires results. On that level they succeed as well, or better than, many individual critics. Ask wineries.

    And, honestly, judging at those competitions is incredibly fun and incredibly hard work at the same time. That judges are often incompetent and the results questionable is far from news. We the judges know it and would be the first to admit it. But the many County Fairs that continue to exist because of the profit they make from the volunteer judges (we are rarely paid) and the volunteers who make the competition possible, the ones who wash the glasses and pour the wines, are grateful that we do what we do. Again, and again, so what?

  9. Steve,

    Very good post. In the past I have noticed that you have reviewed a wine twice in the WE and have given it different scores. Here is one that stood out to me because it was a wine that I really liked:

    Shenandoah “ReZerve Barbera” 2003.

    I believe that you scored it 90 in your 1st review and then 87 in your 2nd. Proof that wines change over time and that our palates interpret them differently over time.

    BTW I noticed that the WE has changed its wine search page and is now allowing the public to add its own reviews…Good move. Snooth and VinCellar both allow this and it is a very cool feature.


  10. Derek, the public review addition should prove to be very popular. Thanks for your comment.

  11. Case in point. Ramey 2001 Jericho Canyon Cabernet Sauvignon. When David Ramey was using this friut, he would split it with the owners, 400 cases for him, 400 cases for Jerichco Canyon Winery…exact same wine. Parker scored Ramey’s wine 96 points Parker scored the Jericho Canyon Winery wine 91 points….a 5 point difference, does not sound like much, but to me that’s quite a spread from an experienced critic for the same wine.

  12. Frank, you mean that the wine was all made by Ramey? It was the exact same wine? That’s pretty interesting!

  13. Morton Leslie says:

    I welcome studies like this. There should be many more. I wish there were a way to similarly test critics for tasting ability and consistency. (Not that winemakers who criticize critics should get a pass either.)

    Wine tasting might be described as an art, but accuracy and validity in wine tasting comes from science. Having taken classes in Sensory Analysis in the Food Science Department at UCD, and later under Maynard Amerine in Enology, I learned formal tasting methodology. I gained an awareness of the many causes of error and bias and the conditions that lead to valid results. Through testing, tasting, statistical analysis, I became aware of my own strengths and weakness in sensory capability.

    I know when my senses are fatigued. I know where I am liable to make a contrast error. I know when I am likely to pass over a wine because it is delicate or when I might be attracted to a wine because I am fatigued and it is bigger than others. I try not to let my personal biases close my mind to a wine. I know when a tasting environment is compatible to accuracy and when the lighting or distractions are problematic. I learned the importance of methodical tasting and good notes. I know oak, TCA, sulfide, mercaptan, SO2, smoke, 4-ethyphenol, acescence, and diacetyl when I smell it. Examining the laboratory analysis of wines I taste and comparing it to my own sensory preferences has been enlightening. Yet, despite knowing all these things, I make errors every time I taste wine.

    Tasting wines blind with other wine people I have also seen how variable preference is from one expert to another, usually because of completely different standards of what is, or is not, wine quality. A few nights ago I tasted a Burgundy brought to dinner by a prominent winery owner. The wine was actually made with 200% new oak (that probably gives away the shipper.) I have never tasted Brett as strong in a wine as in that wine. Yet, here a prominent winery owner loved the wine, looked at it as pure “Burgundy” and hadn’t a clue that the powerful “incense” aroma was oak modified by Brett. (I am a believer in “good Brett”, but never when it is ALL Brett.)

    I know this might sound egotistical and self serving, I mean it as factual, but it is interesting that education and experience similar to mine is rare in judges in the many competitions. I have sat with spouses of winery owners, restaurateurs, chefs, wholesalers, retailers and journalists – none of whom had much of an idea of what they were doing. Most would finish a flight of a dozen wines in less than five minutes. By the end of the day, anxious to finish, they would be finishing up a flight after just a couple minutes and making no notes. I can’t blame them, 150 wines in one day is a pain. But qualifying judges would be a big step toward improving these competitions.

    I’m glad someone has finally done something to shine a little light on this. Good on you, Robert Hodgson…and you too, Steve.

  14. I’m sure the owners of Jericho Canyon had the same feeling 🙂 What do you think that cost them in terms of launching a brand?

  15. Thanks Morton, but I doubt if we’ll ever reach the stage where we have judges who are like the mentats
    in Frank Herbert’s novel, Dune. From Wikipedia: “Mentats are able to sift large volumes of data and devise concise analyses in a process that goes far beyond logical deduction: Mentats cultivate ‘the naïve mind’, the mind without preconception or prejudice, similar to the contemporary practice of Zen, that can extract the essential patterns or logic of data, and deliver useful conclusions with varying degrees of certainty.” We’ll either have to learn to live with imperfect judges, or we’ll do away with them altogether. Or something in the muddled middle!

  16. John Corcoran says:

    The assigning of a number to the evaluation of a wine would tend to imply to the reader that the reviewers view of quality has been engraved in concrete. That quality is static and not dynamic. Robert Prisig identified in his books ‘Zen & the Art of Motorcycle Maintenance’ and ‘Lila’, that quality was an immediate experience, as reconfirmed by Malcom Gladwell in ‘Blink…you know it when you see it (taste it). But the world is not static, and to view it as such obviates the reality of a dynamic world.
    The idea of assigning scores to wine goes back to my earliest days in the wine business, with tasting @ the Vintners Club in San Francisco when the Cal-Davis 20 point system was used to evaluate wine, and to prioritize ones choices, or in that moment of time, one’s view of quality. Somehow, this has been commercialized in the world of wine reviewing. Not in itself an evil objective. This scale was morphed into a more familiar form, one that we grew to love and hate in High School and College. The 100 point scale. Somehow, we, as a broad group of wine consumers lemmings accepted the scores as dogma. Shame on us. Perhaps it’s time for the seemingly disingenuous practice of subjective wine scoring presented as objective fact be rejected, and that we learn to trust our own judgements on the nature of preference and quality.

  17. John, agreed. But what are you going to do with the consumers’ demand for quality assessments? The folks out there want guidance, and they want it in the form of an instantly-telegraphed icon, be it a score or some other sort of visual symbol.

  18. Nothing beats a story on wine ratings to get some discussion going on their validity. I have well over 40 years in the business all from the wholesale end or as a broker representing wineries to wholesalers. There is no question ratings from a marketing standpoint work. There was always an under current that to get a good rating from the publication you are part of, a full page ad was a requirement. The Spectator did it with one of Gallo’s wines, gave it a 58 and then Gallo ran a couple of full page ads and 3 months later the same product got an 80. Hmm!
    Makes one wonder. personally I would like ratings to be given a word score like Excellent down to a bad wine calling it Scary or even a star rating system, 10 stars down to 1 star. I will never forget when the 98 Cab’s were rated. The so called critics (possibly you) panned the vintage. Boy were they wrong. Some of the most enjoyable Napa or Sonoma Cab’s I have had were the 98 vintage. That includes Simi, Opus One, Mondavi, Raymond, Chat. St. Jean, Krug, etc.
    They are still good even today.

  19. Over at Open Wine Consortium, we’ve been having a related discussion: can quality be isolated from personal preference. Even more interesting, though, may be the implication(s) of your statement, Steve: “The reason why wine critics still play an important role in our understanding of wine is because those of us who have been at it for a while are better at evaluating wine than most people. We’re not perfect, but when you’ve tasted, and thought about, as many wines as I have over my career, you do learn a thing or two about quality.”

    But what is the relevance of “quality” and “understanding” when it comes to informing consumers who lack the miles on the odometer about purchasing decisions. The experts represent far less than 1% of the wine drinking public. If they are able to detect levels of perfection, so what if the wine buying public can’t. Are the experts just talking to other experts?

    Consumers are usually looking for other qualities that equate to degrees of “deliciousness” to them: roundness, softness, fruitness etc. They don’t know nothin’ about “complexity” and “balance” and, for sure, ~terroir~. But they too don’t really count in trying to arrive at conclusions about the ~market~ merit of wine. They pick up a bottle on the basis of how much they like the label, comments they may have heard from others. Or they blindly follow shelf talkers. Or they acquiesce to the recommendation of the guy/gal in the wine store.

    It’s the so called “Core Consumers” who drink wine with their meals, those who will pay a bit more than a single digit for their vino, who deserve our focus. They fall to the right of the bell curve of wine consumption, though not over at the tail with the wine critics. It is they who read reviews and wine articles and want some guidance on what’s worth plopping down bucks for.

    The best solution: a People’s Choice evaluation with multiple scores given over some passage of time and of course in different contexts. ~CellarTracker~ IMO comes closest to this methodology, even though it is top heavy with geeks and collectors. Nonetheless, if you have 20 mini reviews of a wine, the emerging average score counts for something, particularly since the individual “judges” usually add some copy to buttress their own assessment.

    Power to the People!

  20. Frankly, I am more than sick of the whole “taste is subjective” argument. There are standards and obvious things to look for in wine. For example, they could get all the critics together, give them a wine and force them to score it and defend any variations from the mean. Continue to do so repeatedly to “harmonize” the scoring process and do so periodically to calibrate different tastes. Sure this runs the risk of creating a standard and inhibiting creativity, if done wrong, but I am tired of picking up highly rated wines and being disappointed again and again.

    @tom merle- As for regular people, people thought sweet crap rose was the greatest thing on earth for years. There is a difference between people who know and people who don’t – much as our democratic instincts want everyone to be an instant expert. No thanks, I’ll take MacNeil’s suggestions over cellartracker every day (cellartracker has the worst sort of phenomenon where people grade either on an absolute scale (this $4 wine was a 83) or a relative scale (this $4 was better than the $15 one, so it is a 99), plus there is the additional fear of being seen as having scored out of the range of previous reviews so there is a regression to the mean.

  21. Yes, this study has caused a tempest in the teacup this week.
    I blogged about it on MyDailyWine this week as well.
    Bottom line is that consumers want to drink wine and do not have time to study up on which wine to purchase. They need and want guidance.
    But the difference is that they are increasingly interested in what their peers have to say and not just relying on professional reviewers for information.

  22. Amy, you’re so right about consumers turning to their peers for information. Since I began this blog, this has become a real eye opener for me. I wonder what the role of social media will be down the road.

  23. Steve – I wanted to join the others here in giving you mad props for highlighting a topic with which you’re so intimate – nice show!

    I wrote about this same topic the day I heard about it ( ). For me, what this means is that palate fatigue is real.

    I look at it this way – who would want to judge *anything* 150 times in one day and hope to be consistent?

  24. Hey Joe, nice to hear from you. I read your post. I completely agree that tasting a zillion wines in one day is insane, which is why I have never been a part of these fairs and never will. I limit myself to about 12 wines a day, although a couple times a year I make a trip to someplace in California and will do maybe 40-50. But I disagree with you that palate fatigue is the only reason for the inconsistency. I think the inconsistency is inherent in the act of tasting. Granted, palate fatigue accentuates inconsistency, but the inconsistency would be there regardless. It’s because we’re only human and not tasting machines. I’m fully aware that there’s an M.W. philosophy that holds that reviewing can be replicable if the tasters go through rigorous training. It’s malarkey.

  25. Andrea Fulton-Higgins says:

    Steve – Very good of you to discuss this rationally and the discussion is lively. As a longtime Sommelier who has written wine reviews & judged in several competitions, here are my thoughts…A good wine review will accurately describe the wine and leave it up to the reader to discern whether the wine style appeals to them. And there should be more attention paid to food affinities. Flawed wines should not be reviewed at all, with Brett being an oddity that begs exception. (what imparts character to some is considered a flaw to others). Not everyone has the stamina to stay focused through 4 to 6 flights of 45 to 75 wines per day for 2 days in a row. Judges aren’t paid a lot of $$, and, unfortunately, over inflated egos tend to get in the way. In any given competition “some of the prettiest girsl stay home” (Robert Mondavi). Finally, the reason number ratings are so popular is that consumers are either lazy, or insecure. As we go through the coming period of time, where the bottle of wine you can afford supercedes anyones opinion of the wine, our prioritites will, hopefully, change and we will be more grateful for what we have, & less consumed with perfection.

  26. Hey Steve – I wouldn’t say PF is the *only* reason; but at 150+ wines, it’s a big, BIG reason :-).

    Do agree with you that being human and not a tasting machine is another big contributing factor – and thankfully so. If our perception of something did not change to somehow align with our surroundings, the situation we’re enjoying at the time, other things that we are imbibing / eating, etc… what a dull, dull, sterile world that would be.

    I almost think we should be *celebrating* the fact that tasters don’t have machine-like consistency – it means they might actually be living & enjoying themselves!

  27. Dude: Sure. I’m glad I’m not a machine. But I do worry when I inadvertently give the same wine 2 different scores on different occasions. I expect myself to be consistent, even though I understand that it’s physically impossible. That’s the paradox of reviewing. This also points out the importance of being diligent in the “back end” of tasting: When I formally review a wine, I try to make sure I haven’t previously reviewed it. That’s not always easy, since I’ve reviewed something like 30,000 wines. When I do inadvertently re-review something, I’m glad if it lies with a few points, either way. Sometimes it’s more than a few points. I was interested to read Frank’s comment [below] on Parker being 6 points off on the same wine. (I don’t know that for a fact, but will take Frank’s word for it.) Parker has nothing to be ashamed of. It could happen to anyone — even you!

  28. Steve,

    One aspect of scores that I have seen, beyond its use to lure the consumer is to display your wine as better than the other guy’s wine. This is very impactful on the trade. The trade is absolutely focused on scores: Distributors, Restaurants and Retailer all alike. they all read religiously scoring agencies (aka WS and somewhat WE) and they firmly believe that a good scores sells more wine. So distributor A goes to Retailer B and what they talk about is the great score the wine gets and they decide to stack it to the ceiling and put a 90 point tag on the stack and guess what? It sells like hot cakes but really, is it because of its position in the store? or because of the score? This has made scoring agencies very important to the industry. But , are we fooling ourselves and at the end of the day? we are misleading the consumer with the impression of a all powerfull scoring/judging system (I agree that consumers are actually very smart just like me), we make and sell monolithic wines (the one that the top 50 reviewer in America like) and we do not have an opportunity to invent or test new wines because no one would take a bet unless there is a score. All that makes for a stale industry in my mind. Free us from reviewer and let the market decide what wine is great and what wine is not.

  29. @akatsuki: I wasn’t referring to “regular people” but to that much more limited subset that enjoys wine with dinner. References to the unenlilghtened past and the lowest common denominator only obfuscates the discussion.

    I’ll take the compostie recommendation of a cross section of wine drinkers in a large enough sample over any individual even if it is Karen McNeil (this moves recommendations to one’s peers that Steve mentions).

    If enough CT members evaluate a given wine, the difference in the two methodologies you mention is cancelled out. Also, I doubt that “fear of being seen as having scored out of the range of previous reviews so there is a regression to the mean.” That’s your educated hunch, but without data, my contrary view is just as likely to be more valid.

  30. Nicholas, in an ideal world your argument and suggestion would be accepted by everyone. But this is not an ideal world! (As we have all learned, painfully.) I think it will take some time for the current system of ratings to be overturned. I even see bloggers beginning to rate wines, suggesting that the next generation of reviewers will be the same as the last generation. But I don’t agree that the current system prevents new wines from rising up “because no one would take a bet.” There are new wineries coming online everyday that are “discovered” by critics (including me). In fact, one of the joys of my job is discovering new, little wineries and promoting them.

  31. Ron- Great rant! Thanks for saving me the trouble, the saliva, and you did it better.
    Morton- Great post.
    Lots of opinions here, but I would like to get back to the original study. The point was that wine competitions are PERCEIVED as being an accurate means of measuring wine quality, when in fact a study of the results show them to be neither accurate, nor reproducable. This is, to some, A BIG DEAL. From there, Steve takes the leap of applying that finding to the scores awarded by winewriters. HUH? Wine writers are writers. No one in their right mind expects them to supply anything other than their own subjective opinion. That is sometimes useful, and often entertaining, but not seriously presented as any kind of technical sensory analysis, even though numbers are used. The real question here is: “Do we need to improve wine competition judging; if so, how?” Personally, I think judging training, procedures and standards could stand improvement, but these competitions persist, so they must have some merit. If you want statistically valid quantitative descriptive analysis of wine, go to U.C.Davis, not the State Fair.

  32. Mark, I agree. But sensory analysis of wine — the way enologists do it at UC Davis — is different from wine reviewing & criticism,
    which is what I do. And in my judgment, the consumer benefits more from an individual reviewer than from a panel, as in the SF
    Wine Competition. That’s because in panel reviews, the highs and lows tend to be eliminated, so everything comes out in the muddled
    middle. I think consumers want to know when a critic absolutely loves a wine, or hates it, or is indifferent.

  33. Hmm, Steve, I wonder why you think an individual reviewer is more beneficial than a wine competition…let me think…

    And what about reviewers, like Parker, who claim they taste more than 100 wines a day and can judge them accurately? You may not be guilty of that, but the Tanzers and Parkers of the world certainly are. Their judgments are more valuable too? They never tell you which wine, the 2nd or the 101st, the wine they are reviewing was that day. I’d like to know.

    I also submit that any individual wine reviewer forced, by trickery or not, to judge the same wine blind three times in a day would be unable to give it the same score each time. Unless he gave every wine an 89 which is what most do anyway.

    I still say, so what? Whether or not you buy a wine on the basis of a Gold Medal or a 92 from Steve Heimoff, you’re going to judge the wine on your own after you taste it. When you agree with the recommendation, your source is impeccable. If you disagree, the source is an idiot. Many very competent individual reviewers, Frank Prial, Anthony Dias Blue, Patrick Comiskey to name a few, work as wine judges. Does this mean they are in some way misguided as to palate fatigue, competence, tasting skills, etc.? I’d love to hear that discussion.

    It’s just wine. It’s very personal. I don’t think Miss America is the prettiest woman in the country, but, even if she’s not, the panels of judges that got her there didn’t do too badly. Maybe someone should enter triplets and see if they can tell the difference.

  34. Mark and Steve: With all due respect, if you want a statistically valid quantitative descriptive analysis of ( what’s wrong with a wine) go to U.C. Davis.

  35. Hosemaster, I hear your rant. All I’ll say now is that the reason I think an indie reviewer is better than a panel is because people can learn about that reviewer’s biases, and judge accordingly. For example, I’m fairly well known for hating residual sugar in table wines. Not all critics do.

  36. Next time I run a questionnaire to our club members and direct consumers, I promise to post their opinion on ratings and reviews. Until then, we have removed ourselves from consideration of any judge other than the one that pays for a bottle (hopefully over and over again).

    Wine writing I believe in. Wine judging is hooey.

  37. Wine scores, especially those from Laube and Parker, have become much more about PR and marketing than wine quality. Neither critic tastes their highly rated wines blindly. Worse, if they know you, it’s possible to arrange private meetings to sit with them in an atmoshere you can control, have them taste your wine, and hear your “unique” story. To say that has no influence on the final rating would be naive. I’ve seen “special” blends representing a barrel sample developed for Parker and Laube to taste and rate, as well as claims of wines being unfined or unfiltered when they actually were just because certain critics like to hear the words unfined and unfiltered.

    The inconsistency of human tasters or the insider politics of the most powerful critics shouldn’t be surprising. The power and influence some critics have is amazing. Someday, when I’m no longer involved in the wine industry, I would like to carefully decant a “100 point” wine into a really cheap bottle with a cheap looking label, send it to Laube and Parker, and watch them give their 100 point wine an 80 point rating.

  38. Jim Caudill says:

    Since I both judge at wine competitions and enter wine into those competitions, nothing about that study surprised me. But one thing I do know: when any individual wine consistently scores a top medal across a long list of competitions (I enter wines into more than two dozen around the country) then you have a pretty solid indication that the wine has some attractive features and appeal to it. I never sweat the results from any one competition, but I am interested to see how things do across the spectrum.

    Another issue, I think, is that each competition has its own set of judging rules, categories and even a certain set of standards that are understood by the judges. It might actually help consumers if the wine competition judging rules and approaches were a little more consistent, and reflective of a certain agreed upon standard.

    Meanwhile, I try and emphasize that it’s the tasting notes that are critical, not the scores or medals. What do people find attractive in the wine, and does it fit your own personal sense of what you want in a wine. For that reason, tasting notes that “speak to the people” are the thing for me.

  39. Jim, it occurs to me that wine competitions are like movie awards. We have the oscars, gloden globes, people’s choice, director’s guild, etc. etc. Each has different standards and rules. Sometimes one movie is liked by all (Slumdog Millionaire) which is a pretty good indication that the average movie-goer will like it.

  40. James, your proposed scam on Parker/Laube has been talked about for a long time in the blogosphere. I’m surprised no one has actually done it.

  41. Steve- You like dry wine? Cool. I understand the distinction you plead, one judgement as opposed to a committee “average”. As you say, it’s a different thing, criticism/writing is, from judging.
    Ray- Yeah, the UCD attitude does kinda ask what’s wrong with a wine, rather than what’s right, but, based on their purpose, that’s a good thing. There are a few hangovers from the bad old days, like the 20 point methodology, that over- rewards color (just like most of us) and lack of ascesence

  42. David Creighton says:

    a couple of small and large points. i too am a judge and wine writer.
    wine competitions are artifical environments where like it or not wines are judged against each other. i’d love to know if the same wines were scored radically differently among the same set of wines or in different sets.

    this problem of consistency ONLY occurs when wines are judged in competitions. i am positive that any judge drinking the same wine with appropriate and similar foods(something you would normally do at home) will judge the same wine similarly over time. for this reason, i try never to recommend a wine in print that i have not had a chance to have alone and with food.

    i also wish the statisticians would help us all out by telling us – if they know – how the inconsistencies occur. do wines tend to be judged better earlier in the day and less well later; or vice versa; or is it random?

    one minor point: judges who say that wines are worth 97pts or are undrinkable probably don’t last long as judges – at least i hope not. the discussion at least initially should be about the attributes of the wine – why it is worth high marks or why it is a bad wine. if one person thinks it undrinkable because of high levels of brett and the others either aren’t sensitive(mostly the case) or don’t mind it, then we know the issue and can discuss it. ‘undrinkable’ doesn’t help.

  43. An interesting footnote from a small winery: the people from the Dallas Morning News Competition won’t stop calling and emailing trying to get us to submit our wines. Seems like revenues must down there, too and I can’t say I am disappointed. Think and drink for yourselves consumers!

  44. Methinks the competitions are in dire straits these days, like everyone else.

  45. Very well said, Steve, and a great admission to the human condition- there are so many things that can play into the perception of how a wine tastes or smells. As a winemaker, not only my own brand, but also for a custom crush facility, I have to weigh on a regular basis how things are progressing. With that said, there are days that I decide not to taste or to put it off because my mood isn’t right, I just had a piece of gum, or it is late in the day, allergies, whatever-and I know I will not judge them with the proper perspective. Wine judges do not have that latitude- wines are there to be tasted regardless of what you have just tasted or something in the lineup that can make the next wine either better or not quite right. In my view if all things could be controlled- black glasses, taste the same wine at the same time of the day, everyone gets a sudafed, use robots, and more controls that I could put on the page. But that wouldn’t be any fun, would it?
    I agree that sticking with a reviewer that you seem to match tastes with is a good way to go if you need suggestions. Wines in competitions should be looked at as cumulative- meaning if it has the same ‘rating’ or seems to at least be recognized, you probably have something worthwhile.
    The use of particular words or descriptors is an interesting debate with respect to wine. Sweet for one person may have a good connotation, bad for another. The word itself can be perceived wrong-does it really mean sugar sweet or is it just ‘fruity’? I had a descriptor of hazelnut and I had a consumer come back asking if the wine had nuts in it……
    Not sure that there can ever be a perfect way to score or evaluate-we are human.
    So much more that an be said on this topic-but I think I have put in more than my 2 cents,

    The Mustang Winemaker

Leave a Reply


Recent Comments

Recent Posts