subscribe: Posts | Comments      Facebook      Email Steve

Massive wine competitions are O.K., but there are problems


I have many friends who are among the 55 judges at the San Francisco Chronicle Wine Competition, whose results were published over the weekend, and among those I do not personally know are people whose professional reputations I admire. So I am not being disrespectful of these men and women when I say that the nature of a competition so heavily larded with judges is one reason to question its results. But there are others, as well.

Making decisions by committee may sound democratic, may appeal to our national spirit of teamwork, and may even present the illusion of balance. After all, it eliminates the extremes of “love” and “hate” in favor of a broad consensus right down the middle. But this median-izing of the wine review process is exactly what weakens it. Too often, the wines that make it through this gauntlet are those that are clean, properly varietal, and don’t turn anyone off. Wines with a little eccentricity, that reach beyond the ordinary in order to achieve extraordinary distinction, need not apply.

How could it be otherwise when so many people are being asked to agree about something? Anyone who’s ever participated in a group tasting with other professionals (critics, winemakers, sommeliers) knows that dissension is bound to occur. If critic “X” is dazzled by a wine but critics “Y” and “Z” aren’t, then that wine suffers in the results, robbing consumers of a potentially exciting experience. And the more people who are asked to make a decision, the more mediocre — in the literal sense — their choices will be.

That’s one reason to doubt the outcome of such a large group tasting. Another is the fact, widely reported last month, that critical inconsistency is endemic to these sorts of things, and such inconsistency can seriously undermine the validity of the results. As I wrote here on Jan. 30, a study “found that over a three-year period, 90% of the [California State Fair’s] judges…basically blew it in blind tastings when presented with the same wine.” Clearly, if that could happen at the California State Fair, it can, and is likely, to happen at the San Francisco Chronicle Wine Competition.

There’s another reason to wonder about the results. Nearly everything wins something, so everyone has something to brag about. For example, 81 Pinot Noirs, by my count, won awards. Since these were divided into 4 tiers by price, this enables 8 of them to claim they won a coveted “Best of Class” award. Is there a difference between “Best of Class” under $14.99 and “Best of Class” above $35? Probably. But the consumer wouldn’t have any way of knowing that, since most wineries tout only their “Best of Class” medal. Ditto for the 139 Cabernet Sauvignons that won. It’s a little like the children of Lake Woebegon: they’re all above average.

And don’t even get me started on the palate exhaustion problem that plagues these marathon tastings. Believe me, fatigue sets in and makes it nearly impossible to render a fair opinion.

Look, wine competitions have become big business for struggling print periodicals. They have their place. Awards make wineries happy, because they can use them in their P.R. The Chronicle’s Wine Competition is no worse than any of them. But it would be interesting to take the results and match them against the reviews that some of the individual judges gave the same wines in other venues. (Some of the judges review and score wines for their employers, be they wine stores or newspapers.)

I just happen to be among those who believe that, when it comes to wine criticism, it’s better to go with a single, trusted taster, whose palate you understand, than with a huge, largely anonymous panel, about which you know little or nothing, except that they’re in the wine trade.

  1. Precisely why the late Jerry Mead and the Orange County Wine Society devised the OCFWC as a non-consensus competition. Judging panels do not (officially) caucus nor grant final medal awards. The judges score the wines based upon their own various experience and biases and may only recommend a medal level. The medal recommendations and scores(100 point system) are extrapolated as well as interpreted by an objective OCWS committee working from the judges’ numbered scoring sheets. It’s a lot more work than the systems other competitions use and, I suppose, still a consensus per se. It does eliminate the carping, lobbying and stonewalling by certain factions and personalities. Ever tried arguing with John Parducci?

    Wine “median-izing” has as her bed fellow, judge “homogenization”. How tedious to see so many competitions, desperate in their quest for credibility, conscripting the usual suspects core of “experts”. Absent a true single trusted master/taster such as the Great Corti, the infusion of fresh faces and palates (bloggers?) will keep wine reviews and competitions alive and, perhaps, useful.

  2. Sounds like the wine equivalent of trying to write a mission statement in a business meeting. 🙂

  3. …or Einstein coming up with E=Mc2 in a rush hour Manhattan subway

  4. Ray: totally agree. Good observations.

  5. I couldn’t agree more. In the industry, wine competitions are generally regarded as being the equivalent of spinning a roulette wheel. Take the very different subjective likes and dislikes of a zillion judges, and throw in a huge measure of palate fatigue (which for many of the bigger wines sets in about a tenth of the way through the judging), and you have a recipe for total arbitrariness, which is what you get. It’s good someone is pointing this out.

  6. Steve, your critique of wine competitions is interesting and I find myself largely in agreement. I will say, however — as a wine PR guy — that I and many of my colleagues are not big fans of the competitions. They can eat up a PR budget quickly and honestly, my sense is that out in the market, unless you’ve got a Best of Class, nobody really cares. And even BofC doesn’t do much for you except give you a warm feeling. Yet many of us feel pressure from our winery owners and winemakers, and the writers who often run the competitions, to be involved. Many good people work very hard to put on good and fair wine competitions, but I personally wouldn’t mind if all but a few of them disappeared. Instead, we seem to be getting more and more every year!

    Meanwhile, since you brought up methodology, I would be grateful if you could share your own tasting methodology as an Enthusiast editor. The magazine is comically vague in what it says. “Tastings are conducted individually or in a group setting…” Oh. And: “(And are) performed blind or in accordance with accepted industry practices.” Uh-huh.

    So: Do you taste alone? Do you taste blind? Do you score blind?

    Whatever information and insights you might wish to provide are greatly appreciated.


  7. Morton Leslie says:

    Steve – You are sooo right on the three or four points you made. One that I have observed is that the best wines are never entered into competitions. A wine that is critically recognized and sells out quickly has nothing to gain from a competition. In fact, for a successful luxury category winery there is a stigma attached to entering your wines. The medals would be junk you would just throw away and you might get beat out by a wine half the price by a group of tired judges who overlook your beautifully structured wine and go for the monster.

    In fact, there is a strong impetus for wineries to enter their worst wines…wines that have nothing going for them. Parker doesn’t like them, Laube pans them, Heimhoff points out their shortcomings, so you throw them in the competitions and hope that probability and palate fatigue brings you something to crow about. You got nothing to lose.

    I last judged about a year ago in a competition that had over 2,000 international entries. I remember that I , along with the other three members of the panel, became so angry at the crap that was being thrown at us that we complained to the organizers… who themselves were upset with us because we weren’t giving out medals.

    It was downright disrespectful of the competition for wineries to enter such wines. On flight of twelve in an over $50 a bottle category, every single wine had a serious defect obvious to every judge. Defects that would for most wineries disqualify a wine from being bottled.

    We just stared at one another with an incredulous expression. I’ll never again waste my time in such a farce.

  8. Pete, sometimes I taste blind and sometimes I don’t and that’s the goldarned truth…same as it is with most if not all of the wine writers with whom I’m familiar, including you-know-who and what’s-his-name. I do taste alone 99% of the time. The exceptions are at a trade event or with a winemaker, but I almost never review & rate wines unless I’m doing it alone, with my regular routine. Occasionally I’ll have someone set me up with a big tasting when I’m on the road, but even then, I’m tasting alone.

  9. Paul in Boca Raton says:

    Much ado about nothing. I have been selling wine at both the wholesale and supplier level since 1983, and “this medal” or “that trophy” or “this score”, unless you are the WS or Parker, doesn’t mean squat. The challenge we have, as professionals selling in this business, is to wean buyers and consumers off the WS/P ratings, and teach them to trust their local independent wine shop experts. IMO, that is our most significant challenge.

  10. Morton Leslie says:

    People who review wines particularly those who can affect the success or lack thereof of a winery, owe it to all of us to taste blind. I know how hard that is, but the power of the human brain and the power of suggestion is just too much for any individual to counteract. My guess is the knowing the label throws 30 or 40% error into any sensory evaluation…stronger than the placebo effect in medicine.

    I personally don’t care if the big names get the recognition. I have had my share. But the struggling small winery in a unappreciated region or the corporate winery with ineffective marketing and public relations, but who have a talented young winemaker who is doing great things need to be tasted in the same context as everyone else.

    A persons success as a critic may have little to do with accuracy or truth, but may be more to do with posturing, public relations, or a bogus scoring scheme that implies an ordered regimen. But the industry deserves a higher standard of profession conduct among people who are not part of it, but have a strong affect upon it.

    Any judgment of quality must be done without prejudice. Any judge who does not take strong measures to avoid prejudice should really take a look at themselves in the mirror. I mean, what is it all about?

  11. Thanks for the reply, Steve. Since asking the questions of you I found an earlier post in which you talked extensively about your tasting methodology, so I appreciate you putting up with my tiresome questions.

  12. I agree on most of your points. But wine competitions can help sift through the huge number of wines out there. I may not get the best wine buying BOC, but its probably better than the average.

Leave a Reply


Recent Comments

Recent Posts