Big wine competitions have lots of problems
If consumers didn’t value wine ratings and competitions, there wouldn’t be any. The market doesn’t reward things for which there is no demand, which is why nobody works hard at stuff that doesn’t pay. (Except for wine bloggers, and one of these days I’ll explore their peculiar psychology.)
There are as more ways to rate wine than you can shake a stick at (an archaic reference whose origin is in some doubt). The two extremes are individual rating and group ratings. I do the former; competitions such as the San Francisco International Wine Competition, run by my old buddy Andy Blue, are examples of the latter.
I used to participate as a judge in some of these competitions and have declined joining others for years. Another old buddy, Dan Berger, has invited me to judge in his Riverside International Wine Competition, but I’ve never done it–and not simply because Dan can’t afford to pay me what I think my services are worth. No, it’s because, based on my prior experiences, I’ve come to the conclusion that these big competitions aren’t a very good way to evaluate wines, in a way that benefits the public.
For an instructive peek behind the scenes of how these marathon events work, read this article, from a Washington State publication that was published on Monday. A staff writer, Rick, went to a big competition called the North Central Wine Awards. He’s not a wine expert, but was invited to hang out with the judges as they did their thing and observe them, kind of like an anthropologist in the wild. Go ahead, read the article. It’s a valid portrayal of what a big tasting is like. Then come back and pick up here.
The main thing I took home from Rick’s article was: “The judges…appraised more than 220 wines from 40 wineries that day, and then several a second time for best of show honors. How they do it, I don’t know.” Well, I don’t know, either. I’m in pretty good physical condition, but even 60 wines is a chore for me. After that, your mouth just gets tired, unable to sense nuances, so you tend to gravitate to the strongest wines.
Another weakness of a gigantic tasting is that the results have to be voted on. Reviewing wines is not like electing a President. A President is elected by a majority of the voters (judges), according to a prescribed set of rules. That doesn’t mean that every elected President is a good one. We’ve had some very mediocre Presidents in our history (Zachary Taylor, Millard Fillmore and the handsome but hapless Franklin Pierce). In an election, some good candidates are eliminated, usually on some pretty nonsensical basis. In a big tasting, some good wines are thrown under the bus, often because in the discussion (which can be an argument) that follows the tasting, the more persuasive (louder, stronger) of the judges may prevail over more timid, less assured ones (just as can happen in a jury deliberation).
You don’t run into these issues with an individual taster working his or her way through a moderate number of wines. No palate fatigue, no getting drunk. Subtle wines have as much of a chance to star as strong ones, especially in a variety like Pinot Noir, but also in many white wines we tend to take for granted, like Sauvignon Blanc and Chardonnay. I’ve recently reviewed Chardonnays from 2010 that were lean, modest in alcohol and lacking fruit, but I loved them for their minerality and acidity. I suspect that, in a mass tasting of 250 Chardonnays, such wines would find themselves out of the running, because most of the judges, already half in the bag, would prefer the big, bold, buttery, oaky Chards.
Wine reviewing should be an act of reverence, as steadfast as meditating or prayer. You shouldn’t do it wham-bang. It’s just plain wrong. A wine needs to be tasted several times as it changes in the glass. The more subtle the wine is, the closer you have to listen to what it has to say. This requires silence and contemplation, which are not the hallmarks of the atmosphere of a giant wine competition. I’m not saying these big events don’t have their value. I always read their results. But another difficulty with them is that the “important” wineries–the smaller, more prestigious ones–almost never enter them. As a result, the entrants are mainly what I call supermarket wines. That’s a huge weakness of the big competitions. They don’t rate “the good stuff.”
Trackbacks/Pingbacks
- How important is winning medals? « The Horse's Mouth - [...] Steve Heimoff – Big wine competitions have lots of problems Share this:TwitterFacebookMoreEmailPrintStumbleUponDiggPinterestTumblrRedditLinkedInLike this:LikeBe the first to like this. [...]
- A Year in Wine: Big Problems, Or Big Opportunities? - [...] Earlier today, fellow wine blogger Steve Heimoff posted a provocative commentary under the headline "Big wine competitions have lots…
Steve, I would not hold up the North Central Wine Awards as the norm. Far from it. The judges tasted 220 wines in one day??? That’s ridiculously over the top.
You lost me at “meditation” and “prayer” and wines saying things.
Having judged in one of the competitions you mention, I agree they are flawed: only those wines submitted are evaluated (who sends in wines and why is another issue- after all, it’s not like the Olympics, where entrants/competitors have been somehow pre-qualified….), the wines get only one of 3 medals or ‘no award’.
This creates a falsehood exploited by PR and headline writers: “the best in the state”…..
I also think wines that are distinctive in any way will tend to fare poorly in the big competitions. Even if they appeal strongly to some judges, they are also more likely to clash with the sensibiities of others and be downgraded. What emerges from these competitions are pretty middle of the road wines without strong distinguishing features. I would rather have an individual reviewer evaluate and describe those distinguishing features, for better or worse, rather than have them just fall off the radar screen when a bunch of judges’ evaluations get averaged.
so that explains why I can get a “double Gold” one week and get skunked the next week for the same wine. No consistence is the norm
Big problems? Really? a blanket statement for all wine shows? How naive Steve. When is the last time you actually judged at a wine show? Years? Decades? Are you trying to justify the one person score is better than the crowd sourced results? I think I’d prefer to think that “most of the judges” are not half in the bag but rather trying their collective best to come to a grounded conclusion. You know I feel you are one of the most thoughtful wine jounos in the biz but until you actually come to Riverside or SF (and I would gladly give you my stipend so you can get paid what you are worth) your second hand information is deeply flawed. Just look at some of the top winners at either of the 2 you mention…perhaps you’ll discover that sometimes the winners are the “good stuff”
I believe the best wine is the one you like best. Everyone is different. I can’t tell you how many times I’ve tasted a so-called winner of the xyz wine and thought to myself “this is horrible.” Just go out and do your own wine tasting and go with the one that tastes best to you. Cheers!
While most of what you say is absolutely true, I don’t think it’s the whole picture of the benefits and deficiencies of the different methods of wine critique.
Not every competition has a judge taste as many wines as the North Central Wine Awards. And while palate fatigue is certainly an issue, at least the wines are judged on a level playing field. That is, most of the wines are being judged again the backdrop of a fatigued palate. I can think of worse ways to judge a wine.
With a single critic, we don’t know anything about the method. Was a wine tasted in neutral environment, in front of the winemaker / owner, or on a toilet seat on a Tuesday morning?
Competitions tend to praise clean, simple, and inoffensive wines. While this may eschew some worthy candidates, I will say that the results are largely applicable to the wine buying public. However, very high ratings from individual critics tend to be, almost always, an exercise in individual taste with poor relevance, even for connoisseurs.
I can’t tell you how many times I’ve been burned with a buying decision based on a very high numerical wine rating. But that generally doesn’t happen with a “gold medal” winning wine, even if doesn’t wow me.
Steve,
With all respect due, I believe you misinterpreted how this competition was conducted. In fact, I ran the competition, so I have an inside view of this.
We had two panels of four judges (standard at many competitions), who tasted about 110 wines each. While this sounds like a lot, most wine judging veterans will agree this is an average day. We work hard to make sure the judges have sufficient breaks between flights and have the correct food and water to refresh their palates. We switch between varieties to make sure palate fatigue is minimized. An example of this might be as such:
— A flight of 10 Cabernet Sauvignons judged over a 30- to 45-minute period.
— A five-minute break.
— A flight of 10 Cabernet Sauvignons judged over a 30- to 45-minute period.
— A five-minute break.
— A flight of eight Sauvignon Blancs.
— Etc.
We are careful to use veteran judges who can handle 80-120 wines per day without issue. I keep a close eye on the judges to make sure they are doing OK, and we take additional breaks as needed.
For this particular judging, we started promptly at 9 a.m., took a 90-minute lunch break, and finished by midafternoon. None of the eight judges complained about fatigue, none weaved to the parking lot when they were done. They judged the last flight of wines with the same fervor as the first flight.
This was a small judging for an upcoming wine region, but it was run with the same level of professionalism as Riverside, L.A. International, etc.
All of this does not take away from your premise on the usefulness of wine judging, but I wanted to make clear just how this and many other competitions are conducted.
Respectfully,
Andy Perdue, editor
Wine Press Northwest
These big wine competitions might have some meaning….
If the judges were of similar background and experience.
If the judges were professionals and had training in proper techniques in sensory analysis.
If the judges had to qualify, were tested, and had to prove that their assessments were accurate and reproducible.
If the judges were given specific criteria by which the wines were to be judged.
If the judges were given specific instructions and commonly agreed upon what a given award or medal should represent.
If the group results had to meet statistical significance.
If the number of wines were controlled to eliminate fatigue as a factor.
If the judging was limited to the best wines available in the market.
If wineries were not allowed to throw in all their “dogs’ hoping for some miracle and a credential to help them market what they knew was a problem wine. (Or if these wines were weeded out before the real judging.)
If all of the details that pertain to the above, all the judges tasting notes, judges scores, everything were published or available online to the public and the wineries involved.
Otherwise, these big tastings have little value.
What Morton said.
You need to read George Taber’s good book “A Toast to Brgain Wines”, Chapter Four. He cites Robt. Hodgson’s conclusions re wine judging, which he Hodgson submitted in a paper to the “Journal of Wine Econics”, which can be summed up with the sentence: “the likehood of a gold medal can be sstatisically explained by chance alone”. No, I am not an exhibitor.
200 wines? Absurd. I run the Central Coast Wine Competition. This year the judges did 90 wines in about 5 hours. Only the next day did we do the sweepstakes round. If you take a shot at all competitions, how about talking about the norm not some way out there format?
I went to a tasting of award winners at a competition last year and a friend and I played “stump the winemaker”, taking turns bringing each other wines blind. Many of the red Best of Class, Double Gold and Gold Medal winners were so over the top in ripeness that we couldn’t name the grape variety or AVA. Silver and Bronze Medal winners – we could nail the grape varieties every time. When wines are judged to be the best example of a grape variety when they don’t have the classic characteristics of the grape you’ve got to question the methodology.
Steve –
Some of the most wonderfully educational wine writing I’ve come across has been from Dan Berger. That doesn’t change the fact that I agree with you, and Dan (who I deeply respect) has fired some arrows my way after NYCR published this piece two years ago:
http://newyorkcorkreport.com/we-wont-participate-as-judges-in-wine-competitions-heres-why/
You’ll note that among the 172 comments, Howard Goldberg (whom I also deeply respect) attacks us for essentially being nobodies in the world of wine; he does not attack the substance. Bob Foster, who has posted in this thread, was particularly aggressive (and he certainly has the right to be; he works hard in his own competition, no doubt).
That piece was the single most divisive thing we’ve put out, and to this day we’re still occasionally lambasted in a private email — or we’re celebrated by a “recovering judge”, as we heard more than once.
This is a long way of saying: Yes, you’re right. The issues raised in our piece two years ago dovetail with some of what you say today, and there still is not a good answer for most of it. Having said that, some of the very sharpest people in wine are at the top of those competitions. They know more than I could hope to know in a lifetime. I wish they would take on the specific critiques. And I wish consumers had a better idea regarding what medals mean; just yesterday a friend brought over a “silver medal wine.” I asked what the silver medal meant. He replied, “I assume this wine took second place in a competition.”
Evan, if you indeed have “some of the sharpest people” in the judging, then I’d rather hear their individual, personal reports, rather than something hashed into a stew.
Steve,
I would like to add a few notes to your posting about big wine competitions. But first, I need to clarify a few points about you posting on November 18, 2009 regarding the article in the Wall Street Journal focused on my publications on wine competitions. In that blog, you extended my views about wine competitions to wine critics and wine criticism. I have never done that. When presenting my work, I always make the point that it covers wine judges in a competition setting. So, really, I believe we hold similar views: wine competitions do not provide reliable information to consumers, the difference being that your view is simply that, a viewpoint. My view is supported by extensive analysis of data supported by the California State Fair. By the way, I never promised not to publish the results, as you stated in you blog.
My current research, presented at the annual meeting of wine economists at Princeton University, extends my earlier work looking at the results of numerous U.S. wine competitions from 2003 to 2008. The principal result is that winning a Gold is predominantly a matter of chance. Companies large enough to enter multiple competitions can almost guarantee getting a Gold, with a success rate increasing as a wine is entered in more competitions. This is based on theory and substantiated by fact. An additional finding is that when a wine is entered in more than 4 competitions, a majority of those wines achieving Gold status fail to make the Bronze cut-off at another venue. I anticipate this work being published in the next issue of the Journal of Wine Economics.
Your audience might find the recent Taste of Princeton interesting. Organized by George Taber (Judgement of Paris) this event paralleled the 1976 Paris tasting substituting New Jersey wines for those from California. Readers can Google AAWE, Taste of Princeton for complete results with extensive analysis by Richard Quandt. Some judges did not mind that their individual scores be made public. Those opposed were wine critics.
Sorry, a few typos. Second sentence replace you with your.
last sentence in first paragraph. replace you with your.
I find that a ‘competition’ label on a bottle is quite reassuring when I’m trying something new. But I agree with Sergio about the ‘best’ wine being the wine that you enjoy.
Quite shocked at how many wine panel judges taste side by side – surely in such a marathon it’s really hard to pick out all but the most ‘bold’…even if you are an expert!
Another article I came across which might be of interest (linked me to here) is http://excelsiorwineblog.wordpress.com/