Blind tasting and Parker: the issue that won’t go away
After last week’s brouhaha over Jay Miller I decided to double check what Robert Parker says about blind tasting. From “The Wine Advocate Rating System” page on his site:
“When possible all of my tastings are done in peer-group, single-blind conditions.”
You can see that the loophole here is “when possible,” but how big a loophole is it? So small you can barely squeeze a pinky? Or big enough to drive an 18-wheeler through? Well, here’s Parker on his own “exceptions to this policy”, followed by my comments:
(1) all barrel tastings
That’s cool. I’m down with that.
(2) all specific appellation tastings where at least 25 of the best estates will not submit samples for group tastings
I had to read this a couple times to understand it. I would guess this means, for example, Napa Valley Cabernet. I, personally, am never sent the wines of more than 25 Napa Cabs (Colgin, Araujo, Staglin, Screaming Eagle, etc.), and I would guess Parker isn’t, either. Perhaps he buys them, but my educated guess is that Parker actually travels to the wineries, or to local third party venues, to taste (of course, from now on it will be Galloni), and that these tastings are open. If you roll in other “specific appellations” (Bordeaux, the Northern Rhône, Burgundy), and if you assume that lots of the wineries there “will not submit samples” (do the First Growths or DRC?), then you have to also assume that Parker’s Rule #2 gives him ample leeway to taste openly, pretty much whenever he wants to.
Sodden thought: Who determines what are “the best estates”? And if you know you’re tasting one of “the best estates” wouldn’t that bias your perception of that wine?
(3) for all wines under $25
This is a pretty weird “exception to this policy.” Why should wines under $25 be held to a different standard than wines over $25? Parker doesn’t make this clear. It’s especially difficult to understand, given this statement, from his “Wine Advocate Writer Standards” page:
“In a tasting, a $10 bottle of petite chateau Pauillac should have as much of a chance as a $200 bottle of Lafite Rothschild or Latour.”
Truer words never were spoken! But how can that $10 petite chateau wine have “as much of a chance” if it’s tasted openly? Why not sneak it into a blind tasting against Second, Third and Fourth Growths? That would be giving it “as much of a chance” to earn a high score. If Parker (or anybody else) is staring at the label (and, even worse, at the tech sheet)–and particularly, if he’s sitting down to an open tasting of petite chateau wines–isn’t it possible, and even likely, that his mind is being influenced by knowledge of what he’s tasting? I should think so. “These are just petite chateaux, so they can’t possibly be very good. Everybody knows that,” is how the mentation would go.
Okay, back to blind tasting. Here’s Parker’s guideline (on the Writer Standards page) for “The Other Wine Advocate / eRobertParker.com Wine Critics”:
“All tastings…are done under both blind and non-blind conditions…”.
It’s curious that on this very long page, which is practically an essay, this is the only mention of blind tasting. You would think Parker would focus much more deeply and candidly on this topic, since it’s really at the heart of everything he (and we) do. But no–just this slapdash little reference. And even it is unsatisfactory. “Both blind and non-blind”….How? When? Why? Under what circumstances? So this looks to me like another loophole, as big as the “when possible” loophole mentioned above.
Loopholes are funny things. Everybody uses them. Most of the time, it doesn’t really matter. We give ourselves just enough wriggle room so that, if we have to break a promise, we can say, “Well, I didn’t swear on a stack of Bibles, did I?” But sometimes it does matter. I should think in the case of a wine writer it would be obligatory to have a little note beside every review indicating how the wine was reviewed, and where. In a blind, big regional tasting? Individually and openly, at the winery, with the proprietor? These things matter. People have the right to know. We’ll never do away with loopholes, but we can make them so tiny that only a pinky can fit through.
“a $200 bottle of Lafite Rothschild or Latour.”
When was this written??
When I first got the wine bug Parker was the “Be-all, end-all” in guiding or validating my purchases, but declaring 3 vintages in one decade (2000, 2005, 2009) as “Vintages of the Century” dulled my enthusiasm and trust in WA’s reviews and integrity.
I’ve learned so much more through blogs large and small (Yours included, naturally). I appreciate what Wine Advocate has done in promoting wine culture in the United States, but I am so friggin’ tired of a merchant telling me how many points a wine got from Parker.
Tell me what you think Mr/Ms. wine seller. Tell me of the aromas, mouthfeel, length, possible pairings, etc.
Parker is not in the room, you are.
Sediment Blog: It was written a long, long time ago.
— “Loopholes are funny things. Everybody uses them.”
With all due respect, Loopholes are loopholes.
I see no reason why there need to be loopholes for most of what Parker mentions. Go back and look at those loopholes. They constitute the bulk of his highest ratings and all of his so-called bargain priced wines.
I see no reason why wine critics should taste wine for evaluative review in a variety of settings. Taste the wine in your tasting setting and in the same numbers. Routine may be boring but it is also the path to consistency.
I see no reason why wine critics should ever taste wine for review with the labels showing and at the wineries. It is not enough to be transparent about this practice, about which most writers are not including Parker. This practice is tantamount to payola for new wines. Knowing that one is tasting Screaming Eagle at Screaming Eagle or DRC at DRC is the pathway to exaggerated ratings. When I asked a winery owner if he thought that this practice was intellectually honest, he replied, “I get better scores that way”.
There is a simple standard. It applies to Parker and his minions. It applies to me and it applies to you. It is the caesar’s wife rule. Do we follow practices that leave our ratings above suspicion. If we do not, then our ratings are suspect.
Sadly for all writers, when someone like Parker or the folks in New Zealand who charge for reviews do things that raise big suspicions, all of us who try to write unbiased, independent reviews come under suspicion.
I have long felt that if you asked Mr. Parker to review 20 wines blind and give a public score, and only two from the group would be his pet favorites (say Harlan or Screaming Eagle), and the other 18 would be very highly rated Cabs, he wouldn’t do it. What would happen if by chance he scored four $30 Cab’s as high or higher than the cults?
Same goes for SQN or other wines he likes. Providing these loop holes allows him to protect the people he wants to protect. That is why he tastes in person at those places. That’s where the consumer should be cautious. In theory, a 97 point wine should stand out in any group, be it $10 or $225.
The problem with tasting blind is that you cannot tell biodynamic from the scorched earth farming, the unfiltered from the filtered, the French consultant from the Davis grad, the corporate ownership from the privately wealthy and elite (to whom you wish to socially climb.) And if you can’t do these things then it becomes a difficult story to write. If a $20 Sonoma Cab scores better than a $1000 Growth, your ability to discern what is fine and what isn’t comes into play and well as who you get to hob nob with.
Much more convenient to write the story and then choose wines with back-stories that fit your narrative. Instead of facing a difficult Newsletter to write, you can easily write it right! And in a world of ethical relativism, you are doing the ignorant wine consumer a favor. You’re telling him what he wants to believe.
OR, we could all just stop scoring wine and write tasting notes. You know, let the consumer, (remember those that we are trying to help out here) decide what sounds good or superior to them. Just a thought….
Ms. Dugan, that makes far too much sense.
This is an old issue. It was first raised by me (and others) with Parker way back in the days when he was the resident wine expert on Prodigy (Expert 42b). He replied that he had “trained” himself so he could taste as accurately blind as unblind. When challenged for proof he would always refer to the number of subscribers (never understood the logic of this point.) When I told him he was self-delusional he got huffy.
Thank you Samantha. I could not have said it better.
Mr Heimoff, estates and regions which were not included in Mr Miller’s trips arranged by Mr Campo (like it happened in DO Madrid) had little chance to be tasted and thus reviewed by the Wine Advocate, and this meant big money, or at least that’s what the bodegas and regions were led to believe by Mr Campo. This is a big bias. This is unfair. This is something Mr Budd is right to denounce.
And this is by no means the way Mr Parker says he wants the job done.
Samantha, while I understand the position that we should just stop scoring wine, I would point out it’s not going to happen anytime soon. Not trying to be snarky or anything, but if you talk to enough distributors, sales managers and people on that end, they’ll tell you they need scores to sell wine!
Steve,
Oh trust me, I know it won’t but I think this is a good step forward. As a retailer I have to tell you, as many times as I’ve heard a salesman say they need scores to sell wine, I’ve never heard a customer tell me that they need a score to buy wine. So the question is, who should we be helping out?
Samantha, my dear–
The reason that people review wine is that there are too many wines for the ordinary punter to possibly come to grips with on his or her own.
The question that you have failed to answer is what takes the place of the critical reviews you would like to see eliminated.
Please do not answer “knowledgeable wine merchants”. A wine merchant is also a critic if that wine merchant makes choices about which wines to carry.
Substituting one critic for another is not an answer to the “let the consumer decide”. In either case, the consumer is still relying on outside inputs, and while I have unending respect for very good wine merchants, they are still just one part of the puzzle and not the only possible answer.
Thus, it all comes back to this. How does a consumer understand all the choices without some outside force intervening.
Charlie My Dear,
I never said do away with reviews or reviewers. I happen to think critics are a essential resource for the consumer. What I said is we could stop scoring wines and just write, well write reviews, therein not placing value on one palate over another. If, “rich with coco, cooked berries and toasted coconut” appeals to one person they can buy from that review….just as someone more into, “tart red fruit, clay, mushrooms and soy sauce” can buy from that review. See what I’m saying? Why is it every time anyone suggests that we might not need a rating system for wine that critics see that as the masses shaking their stick at them? Not everyone can review or write about wine the way that you, Steve, Jon or Eric do….and yes, I mentioned those four names to further make my point, 4 critics/reviewers with very different palates and guess what, you are all right for someone.
Samantha, I have heard many tales of customers asking for reviews. I’ve been told a good Heimoff review will empty a Costco shelf. Having said that, the wines wouldn’t even be on the shelves if not for the salesmen, so the old question is: which came first, the salesman or the customer?
Sam–
One quick and simple point. Rating systems are not the problem in the way that are meant to be used–as an adjunct to the very words that you want consumers to comsume.
The problem is that wine mags contain hundreds and hundreds of reviews, and no one except a glutton for punishment is going to wade through all that verbiage for each of those hundreds and hundreds of wines–all the while trying to make sure that he or she undertsand what the reviewer was getting at in terms of overall desirability. That is why reviewrs in all kinds of fields us rating systems as adjuncts to their words.
And, Sam, not every wine buyer has the luxury of shopping at a hand-sell wime merchant like the one where you rule the roost.
Charlie My Love,
I rule nothing, just a passionate match maker for people and wines that I know, (thorough learning their palate, likes and dislikes) that they will fall madly in love with. There’s no luxury in shopping at The Wine Country, we offer and champion wines for daily drinking and for every budget but…well stores like ours could in fact be a luxury some day if people keep clearing the shelves at Costco. I’ve said it before and I will say it again, you don’t get your car repaired at the florist or get a pap smear at the doughnut shop, why the hell would you buy your wine anyplace other than a store where people know their products, (ever asked for assistance at Costco or Vons in the wine department?) and have devoted themselves to the thing it is you are seeking?
I didn’t mean to turn this into a retailer vs critics thing. In fact I was merely saying that we should just write our impressions about wine and let the consumer decide the value in that. Anything else is a beauty contest and ego fest and I just can’t see how either of those is going to bring more wine drinkers to the table…
Sam–
I understand your point, but the Olkens do buy bread at Costco rather when we are there and we have bought a fancy watch there rather than at a jewellers.
If a consumer knows what he or she wants, the Costco wine department is quite adequate. It is not a wine merchant in the way that your store is, and I certainly hope that we never lose those kinds of shops anymore than I hope we continue to have small, local banks, independent car repair places and mom and pop restaurants.
But, I think you have skirted the essential issue here. Other than relying on someone’s expertise, yours or mine or Steve’s, how does the ordinary punter expand his or her horizons? And if a rating accompanies the many, many words that I or Steve write, then how does that rating disqualify our work? Think about having to read through hundreds and hundreds of wine reviews without any rating or shorthand guidance as to meaning. That would be an ungainly task. No one would do it. That is why ratings exist in all kinds of fields of comprehensive critical evaluation.
Yes, this is not a wine merchant versus critic thing. We are talking about how people receive and perceive comparative information. If the consumers wanted no ratings, then wine publications, restaurant reviews, movie reviews, Zagat, Trip Advisor and Consumer Reports would all be published with no ratings.
I’m a wine consumer and a scientist, but I have some but minimal concern for how wine is reviewed. My primary concern is whether I can use the reviews to find wine that I enjoy.
I’ve worked with many local wine merchants and they are not a reliable source for me. On average (and, being a data nerd, I do keep track), I pay more to get a wine that I usually like less than if I pick using Parker’s scale. I find the same problem with most wine reviewers — they are less consistently good at producing recommendations that allow me to go to the store, buy what they suggest, and consume it with satisfaction.
Right now, I am systematically going through Paul Gregutt’s recommendations and buying one of everything he recommended over a several month period to see what my hit rate is for his selections. If I find a better hit rate than with Parker, I will begin to use his ratings. Otherwise, I’ll cease reading him and move on to the next.
As a scientist, I think the discussions about ethics and process are interesting. As a consumer and a scientist, I wonder if wine reviewers have a handle on the “effect size” of better ethics and process. In science, a statistically significant result doesn’t imply a meaningful effect on real outcomes. While I may find many aspects of wine reviewing and the wine industry distasteful, at some level I’m interested in finding wines that I enjoy and Parker’s rating system does help me economize my time and money in the real marketplace. There is just no denying it.
Two things…
First – I agree that tasting blind it’s what really puts every bottle in a fighting chance to earn a top mark. That being said, i think scoring wine is much like being a teacher. Your always going to find things that makes the scoring uneven. Your mood, the setting of the tasting, your health (somedays you just don’t feel 100%). To me far more important that tasting in a certain manner is being consistent. And to me Parker (an his staff) tend to be very consistent. I can usually taste a wine before the Parker review and just now it’s going to be a 95+…
Second – About the need for reviews and scores… People need good information to make good decisions, specially when those decisions involve spending a lot of money (even if you only buy wines in the $15-$25 range). A score helps you get summarized information on the opinion of a taster about any particular wine.
I always prefer to have information than to not have.
Charlie asks, “how does the ordinary punter expand his or her horizons?” For this punter the best formula seems to be a writer who’s first focus is that of an educator. Introduce a type or class of wines. Present facts that are important to understanding the wine and different styles or permutations. Give examples of excellent wines available on a national basis at different price points which illustrate the the type or class of wine and its styles. Forget about points.
We miss you Alex.
Morton, I wonder if you are the average punter?
That aside, what you advocate is what I see in comprehensive wine reviews. An introductory article followed by carefully crafted descriptions of each wine reviewed. Adding a symbolic notation, whether stars or puffs or numbers, does not change that equation.
The body of work that I present in Connoisseurs’ Guide or Steve presents in the Enthusiast adds up to all that you have suggested is needed.