The straight dope on wine reviewer inconsistency: A long post on an important topic
I alluded yesterday to reports of judge inconsistencies at the California State Fair’s annual wine competition. Today I want to go more deeply into this report, because I have a feeling it’s going to have a certain impact, both in the press and, by extension, in people’s minds. You can read about it here at the website of the American Association of Wine Economists, whose reports I have frequently cited in this blog, and whose efforts I respect. Or check out this synopsis.
Briefly, the AAWE study found that over a three-year period, 90% of the judges — some of whom are my friends — basically blew it in blind tastings when presented with the same wine. Blew it so badly that, in the AAWE’s estimate (and these guys are statisticians), the results are “amazing, and show[s] us that the Wine Competition Results are seriously flawed.”
This is the secret of wine reviewing, the fact that scores/reviews are not replicable by the same tasters over time. Now, instead of asking why this is, the proper question ought to be, why should anyone think they should be? Replicable, that is. I realize I’m going out on a limb here, because I’m a wine critic who uses scores, and it’s a little weird for me to be sawing off the limb I’m sitting on. But the truth is the truth, and the only way to deal with inconsistency is to address it head on.
I’ve said it a million times: WINE REVIEWING IS NOT ARITHMETIC. If I say 2+2=4 and you say 2+2=5, YOU’RE WRONG. You may not like it, but you are objectively incorrect (unless you live in a parallel universe where 2+2 may well equal 5). However, if I say this wine deserves 97 points and you say it’s undrinkable, then neither one of us is right or wrong. You’re entitled to your opinion and I’m entitled to mine.
That should be easy enough for everyone to agree to, right? The problem is that there are powerful forces out there that want wine reviewing to be like arithmetic. One of these forces is our natural human inclination that desires for our “authorities” to be correct in their assessments. In a confusing, incomprehensible world, we long for gurus, priests, shamans, masters — call them what you will — to explain things, so that we don’t have to try and understand them ourselves. As I said, nothing wrong with that; it’s only human. Trouble is, in this day and age when authorities routinely get everything wrong (Iraq, the economy), we should be dubious of pronouncements from on high.
Does this mean that wine critics’ findings are completely irrelevant? No, but… there’s always a but attached. (No puns, please.) The reason why wine critics still play an important role in our understanding of wine is because those of us who have been at it for a while are better at evaluating wine than most people. We’re not perfect, but when you’ve tasted, and thought about, as many wines as I have over my career, you do learn a thing or two about quality. You learn to appreciate things like balance and harmony, and to detect flaws like TCA, and to understand when a wine has too much residual sugar, etc. etc. Okay, I can hear the skeptics going, “Fine. But what about those S.F. Wine Competition inconsistencies? How do they square with what you just said about experience?”
Tough question. I’m not going to pretend to rationalize it away. I’ve conceded many times that if you give me the same wine twice, I might give it different scores. Which leads to the question of why this should be. Either it’s because my palate, or brain, varies from day to day (which we know is true) or because there may be bottle variation (which we also know is true) or because of some combination of both. That’s just the way it is.
So how should the consumer interpret this? I expect some will say all critics are full of it and ought not to be trusted. Fine. Others of them (if they’re even aware of this, and other, studies) will choose to disregard the inconsistencies and stick with their favorite critics regardless. That’s what I would advise. But then, I have a dog in this race. Ultimately, it’s what you, the consumer, decide. You’re the kings and queens of the wine industry; everything revolves around you. You may not be feeling empowered in these perilous times, but believe me, you are.
So there it is. Wine critics are not infallible. Far from it. Parker and Laube might not tell you that. I just did. Salud!

Bay Vieux Briefs: On AVAs, Australia and inconsistent reviewers
Paso expansion goes through
Last Nov. I blogged on a petition to expand the Paso Robles AVA by 2,635 acres — about 4% of the current total — in a cooler region that’s a little closer to the Pacific Ocean. This was during a period of confusion at the Tax and Trade Bureau, the arm of the Treasury Department that approves AVAs. Well, effective Feb. 20, the TTB approved the expansion, according to their press release, based on the usual parameters of climate, geology and soils. I don’t really care one way or another. Its just one more AVA expansion; there have been many before, there will be many to come. The key sentence in the TTB’s statement is “After careful review of the petition and comments [7] received, TTB finds that the evidence submitted supports the expansion…”.
Now, anyone who’s ever worked in a government office (and I used to) knows how they work. This is from the same Federal govenment that “reviewed” Bernie Madoff’s outfit and found nothing out of order! I can imagine how the discussion went in the TTB’s AVA branch:
Boss: Jim, I want you to carefully review this Paso petition.
Jim: But boss, I’m swamped! I’ve got Leona, Calistoga, Snipes Mountain and Tulocay on my plate — and you just fired my assistant.
Boss: Well, times are tough. Have your decision to me by the end of January.
[Later that night]
Jim [to wife]: Honey, he wants me to do another expansion. This *&%$# is killing me. How am I supposed to get my work done when I don’t have any help?
Wife: Did any of the commenters object?
Jim: Out of 7 comments, only one.
Wife: Was it an important person?
Jim: No, just somebody little.
Wife: Well, screw it then. Approve it, and say you were really careful to examine all the evidence.
Jim: Gee, I guess you’re right. Hey, what’s for supper?
Australia worries
At Wine Enthusiast’s recent Wine Star Awards, which I reported on yesterday, one item making the conversational rounds was the dismal state of affairs in Australia’s wine industry. “Too many grapes” seemed to be the conventional wisdom. It’s the old story of supply and demand. Poor Australia.
Wine judges “inconsistent”? Say it isn’t so!
The recent issue of Wines & Vines reports on a new survey suggesting that wine judges are inconsistent when it comes to judging big competitions like the California State Fair. For example, the judges on one panel were given the same wine three times, without knowing it. They rejected it the first two times, then loved it the third time. It went on to receive a double-gold medal. How embarrassing!
Yet how true. It’s not only judging panels that can be inconsistent. So, too, can individual judges, a truth I’ve pointed out here many times. There’s no loss of face if you rate a wine different ways at different times. Anybody who tells you a judge should give the same rating to a wine over multiple exposures is lying, or seriously misled. That’s why wine judging should be taken for what it is: A considered opinion at a particular time and place. It’s just like a movie review, in which the reviewer can change his mind at a second showing. Does that mean wine reviews are irrelevant? No. They’re have some value — and an individual wine review is better than a panel, which is why I’ve never participated in any of these big fairs, and never will.
Live! From New York! It’s Steve
I’m in Wine Enthusiast’s world HQ just north of NYC on a cold day with a big snowstorm blowing in. We’re about to begin a tasting with the other editors, so I only have a few minutes, but wanted to report on our Wine Star Awards last night, held in the ballroom of the New York Public Library on 42nd St.
For a brief, shining moment, it was the center of the wine world, an assemblage of some of the wine industry’s most illustrious people. Among them (in no particular order) were Gina Gallo, Margrit Mondavi, Piero Antinori, Jean-Charles Boisset, Robert Sands (Constellation), Margo Van Staaveren (Chateau St. Jean), Mel Dick and Wayne Chaplin (Southern Wines & Spirits), Ted Baseler (Ste. Michelle), Ray Chadwick (Diageo), Vernon Underwood (Young’s Market), Xavier Barlier (Maison Marques & Domaines), Chris Indelicato (Delicato), Constantine Boutari and others too numereous to mention — and if I’ve left anyone out, I’m sorry, and apologize in abject humiliation.
I sat with the Constellation people, and man, what a nice corporate culture they have! I might have thought (before I knew) that the world’s biggest wine company was hidebound and starchy, but these folks were wickedly funny, personable and smart. My neighbor to the left was Rebecca Hopkins, a VP of public relations and irreverent Aussie whose sense of humor pretty much matched my own. To my left was Nora Feeley, the young director of corporate communications. To dine in the company of companionable people is a delight under any circumstances, but especially when they’re so beautiful, savvy and amusing (and if that’s sexist, sue me!).
Incidentally, I pressed Rebecca on her opinion of blogs and social media. She does feel they’re going to be increasingly important for producers to get their message out. The bad news is she doesn’t see a revenue stream , any more clearly than the rest of us. Maybe that will change.
I’m sure many of the 340 people in the ornate 1911 Beaux Arts-style ballroom were apprehensive about the economy. But for this one magical evening, people put their cares aside to eat, drink, laugh, celebrate, gossip and just have fun. As Robert Sands put it in accepting his Man of the Year award, we may have looked the same as a gathering of people from the computer chip industry, “but we have a lot more fun!” And, I would add, a lot better wine!
I’m in the Big Apple…
for Wine Enthusiast’s big Wine Star Awards dinner at the 42nd Street Library, the wonderful Beaux Arts building near Times Square where I used to go to study when I was in high school. We have a great dinner and drink great wines. Then, it’s up to Wine Enthusiast World Headquarters, for a couple days of meetings.
Back late Wednesday. Have a great week, everybody.

42nd St. Public Library
Meritage Schmeritage
The Meritage Association is doing just fine, MSNBC reports, citing figures showing that the 21-year old trade association now has more than 220 members, with the recent additions of Costco, Robert Mondavi Private Selection and Sterling Vineyards. Those are some heavyweight players to have on your side.
I remember when the M.A. was born. They didn’t have a name for the red and white wines blended in the Bordeaux style back in 1988, and so the organizers — Agustin Huneeus, Mitch Cosentino and Julie Garvey [Flora Springs] launched a contest. They asked the public to come up with suggestions, and the winner would receive a case of wine from every member winery, every year, for the rest of his or her life. Or so I recall…
Anyhow, I entered, and came up with some silly pseudo-French name that I no longer recall. They eventually chose “Meritage” (rhymes with “heritage”). A year or so later, I ran into the guy who coined it. He was a bagboy at a big supermarket in the East Bay, just a young kid who lived in a studio apartment. He told me he had cases of wine piled up to the ceiling! I sometimes wonder about him, and if he’s still getting free wine. Two hundred and twenty members x 12 bottles each = 2,400 bottles of wine a year. That’s a lot of wine, although it’s somewhat less than the estimated 4,000 I review annually.
I always liked the idea of Meritage. A word was needed because wine store owners didn’t know how to display the wines, since they’re not clearly varietal. Restaurateurs also weren’t sure where to put them on the list. These blended wines do seem to be in a different category, at least terminologically, from varietal wines. On their website, the Meritage Association, quoting Huneeus, says that the term “Bordeaux blend is a misnomer.” I’m not sure that’s true; I use “Bordeaux blend” frequently in my articles and reviews, and to me, there’s no difference between it and “Meritage.” As far as I can tell, most California Bordeaux blends don’t use the word Meritage, although I’m not sure why that is. The word never became the hit that the originators thought it would. But it’s still useful.

Regardless of whether you call it Meritage, Bordeaux blend or whatever, these wines, mainly the red ones, are some of the best in California: Harlan, Rubicon, Colgin, Phelps Insignia, Opus One. (Blended Bordeaux whites have some room for improvement, to put it mildly.) Yet it’s also important for consumers to realize that just because a bottle of wine has the word “Meritage” on the label doesn’t mean it’s a great wine! Once again, it’s not about words, or AVAs, or anything like that, it’s about what’s in the bottle.
And this (bad news) just in…
The Santa Rosa Press Democrat is reporting, in its online edition, that Food & Wine’s talented wine writer, Lettie Teague, is being laid off, and will now work for the magazine as a freelancer (as I do at Wine Enthusiast). The article also reports that “Gourmet is rumored to be close to calling it quits in all but web form.”
Bette Davis: Fasten your seatbelts. It’s going to be a bumpy ride.
Bob Dylan: Could this really be the end?
And finally, as long as I’m in an evil mood…
I got this email press release from an El Lay p.r. firm I won’t identify, except their letterhead calls them “Specialists in Launching New Concepts & Bringing Existing Ideas to the Mainstream”
Steven:
A decade ago LA native Scott Palazzo was nominated for a Grammy for his career as a music video producer. Today he’s a vintner whose wine, in only four short vintages, is being called “the next Harlan.” Palazzo’s entry into this status has been a more than charmed rise, after his first vintage landed into the legendary French Laundry. Then the famed founder, chef Thomas Keller, requested he create an exclusive Cab Franc. Furthermore, Parker is giving him high-90 scores. Interested in featuring this former Grammy winner turned cult wine sensation in Wine Enthusiast?
Readers: please rate this pitch on a 100-point scale, and don’t forget to include a brief review!

