Blind tasting at Wine Spectator again an issue
Once again the charges are flying concerning if advertising influences scores at Wine Spectator. I have no reason whatsoever to think it does. If I’m in a position to claim that Wine Enthusiast doesn’t let advertising influence scores — and I am in that position — then it would be churlish of me to think the opposite when Tom Matthews asserts, as he does, that there’s a firewall between the advertising and editorial departments at his magazine. It’s true at Wine Enthusiast, just as Tom asserts it is at Wine Spectator, that “our advertisers frequently complain about their ratings,” although I don’t think in our case it’s “frequently” so much as occasionally. (I may be shielded from some of their complaining.) I do know Wine Enthusiast’s advertising team is sometimes disappointed with my scores. But they’ll be the first ones to tell you, “Hey, we can’t tell Steve what to do” and thankfully they don’t try. They know that once a magazine loses credibility it’s done for.
After I Ieft Wine Spectator, around 1993, I got a job as an editor at the first Wine Business Monthly. Then-editor Lewis Perdue gave me an assignment to interview a guy in Sonoma County who claimed he’d done a computerized statistical analysis of Wine Spectator box ads and scores and found a positive correlation between the size of the ad and the rating. Lewis asked if I would also get a response from Marvin Shanken.
Marvin took my call. I explained the background, then asked if I could ask him a few questions. He exploded and hung up. A day or so later I got a letter from Wine Spectator’s lawyer threatening legal action against me if I reported the study. So did Lewis. Needless to say, being in no position to defend ourselves against Marvin’s attorneys, we decided in favor of discretion and killed the story.
In Tom’s reply to the Wine Economist story, he wrote: “…every review of a newly-released wine is the result of a blind tasting, where neither producer nor price is known by the taster.” That’s a pretty unequivocal statement, and it made me wonder about it for 4 reasons.
1. I’ve been with Jim Laube when he tasted wines openly and took notes.
2. I’ve been reliably told that at some regional tastings Jim tastes openly.
3. Certain winemakers have told me that when Jim visits with them he tastes openly.
4. On the rare occasions when Wine Spectator let me sit in (and weigh in) on formal tastings, the panel knew the general tier of the wines, e.g. new premier cru white Burgundy. So they may not have known the specific bottle prices but they had a pretty good idea.
So I emailed Tom Matthews to ask about these anomalies and he replied, in part: “…any editor can taste any wine in any setting, blind or non blind, for informational purposes…But such ‘reviews’…are not published in our Buying Guide.” To which I replied: “If Jim did in fact taste and take notes with these producers at their wineries, does that mean he did so strictly for informational purposes, and then later reviewed the wines formally under blind conditions?” To which Tom replied: “…yes, if he was tasting wines at wineries, or in other public settings, he was taking notes for informational purposes only, and the wines were subsequently officially reviewed in our normal blind tastings.”
Tom said he’d have Jim get in touch with me, but so far he hasn’t. If and when he does, I’d ask Jim who sets up his tastings (if anyone) and what he knows about the wines (vintage, variety, region, general price bracket, etc.). It’s hard — no, make that impossible to think he (and the other Spectator critics) know nothing about the wines they review except their color. Still, the issue (of blind or not) is irrelevant to that of advertising affecting scores. I am absolutely convinced it doesn’t at Wine Spectator. That potboiler ought to be permanently put to rest.
Readers can make up their minds how they think Wine Spectator critics taste. I can tell you how I taste. Sometimes blind. Sometimes not. Sometimes semi-blind, in that (having no staff) I’ll set up my own tasting so I know in advance what wines are included, but at the time of tasting I don’t know which is which. But I’ll tell you something that may surprise you (and irritate some people). I don’t think it matters how a good taster tastes. We have fetishisized blind tasting into a religious cult, just as we’ve demonized open tasting into a sin. There are no simple answers, just simple questions.
Trackbacks/Pingbacks
- The Pendock Paradox | Pendock Uncorked - [...] swine flu, and infected the perfume rating industry. Meanwhile US wino Steve Heimoff opens up a new old front…
Steve,
Not seeking to argue points, but to bring up a couple:
Someone once asked on another blog if anyone has ever seen Kendall Jackson wines rated in WS–they do not advertise there.
The overriding issue may not be connected completely to a relationship between advertising and ratings, but one way for magazines to kill the suspicion completely would be to stop reviewing only wines that are submitted by wineries and start reviewing wines that are on retail shelves instead.
As for your comment about fetishizing blind tastings into a religious cult: in order for you to understand why some of us worship at that altar you might start with a re-read of Emile Peynaud and then do some reading concerning the power of suggestion and pre-conditional knowledge as it relates to perception. That’s a much better approach then fetishizing an inability to accept scientific understanding into a religious cult…
Thomas, I always have my copy of Prof. Peynaud’s book open on my table and having read it many times I’m even more confirmed in my beliefs.
From my own conversations with Jim and the observations of several individuals in a position to know, I am confident that what Tom Matthews has told you is true. Moreover, I know for a fact that advertising does not affect the scores directly. That doesn’t mean that the scores can’t be influenced. Wines in Jim’s office are tasted in flights and a decision is made as to what is included in the individual flights and what you want is to be included in the flight that has Harlan, Silver Oak, Insignia, Shafer, etc. You have to have a pretty bad wine not to get a score above 90. This is far more effective than advertising if you want to improve your scores.
Most of the time in enjoying and appreciating wine we taste and drink openly. Only when we are doing critical analysis like learning how our wine stands up to the competition or learning and improving our winemaking do we need to taste blind. We want to know what the wine tastes like, not what we think the wine tastes like.
Critics can enjoy and appreciate wine openly. Only when they are making judgments, comparing and rating wines do they need to taste blind. They owe this to the wine industry and all the effort that goes into making a wine. If they are going to sniff, sip and write down a score they should at least make a small effort to be professional about it.
Blind tasting is not so important if you are one of the producers or winemakers with the big reputations…you are happy with the status quo and know that as long as you keep your image up, your score will remain up. But if you are the new guy on the block or don’t have the McMansion or impressive looking wine temple or are trying to recover from previous poor ratings, you want someone who will look at your wine objectively. This can only be done blind.
Steve,
Thank you for your affirmation that Wine Spectator is not biased in favor of its advertisers.
If you still have questions about Wine Spectator’s tasting methodology, I refer you (and your readers) to our Web site, where they are fully described: http://www.winespectator.com/display/show/id/about-our-tastings
But let me repeat: all official reviews are the result of blind tastings organized by Wine Spectator employees, where the taster does not know the producer or the price of the wine. Beyond the details, our goal is to ensure that all wines are tasted in comparable, neutral conditions, so that every wine has an equal opportunity to show its true character.
As for your statement “I don’t think it matters how a good taster tastes,” I simply can’t agree. I do agree with Thomas Pellechia, Emile Peynaud and the entire scientific community, that avoiding observer bias, through some form of “blinded” methodology, is crucial to the validity of experimental results, including wine reviews.
Thomas Matthews
Executive editor
Wine Spectator
Steve,
What am I missing?
“I don’t think it matters how a good taster tastes. We have fetishisized blind tasting into a religious cult, just as we’ve demonized open tasting into a sin.”
It isn’t to “fetishize” what is based in scientific study and it isn’t to make perception a sin. It is to question the notion that non-blind tasting (or half-assed blind) offer enough unbiased results for its claim to be taken as seriously as truly blind tastings. Nothing more or less.
It’s one thing to believe in someone else’s talent, but it’s quite another to test that talent–and I’m not saying that I would pass the test either. In fact, every time I take the blind tasting test I gain more humility, which is sorely lacking in the wine evaluation arena.
But to the WS point: I believe that it and other magazines could use a little reformatting of their modus operandi.
From the standpoint of sensory science, I don’t provide any information regarding producer or price to my judges performing analytical sensory evaluation so as not to bias them.
Why not just taste blind and avoid the debate over methodology? I presume that you do most of the time anyhow.
Of course, even that won”t satisfy everyone. There will always be those who want you to taste randomized samples of everything from black glasses and who will tell you that if you cannot do that and tell Alsatian Pinot Blanc from Australian Verdelho that way, you have no business being in the business. Those folks can be ignored.
But tasting blind is, to my way of thinking, the way that folks like you and me are able to guarantee every wine, and thus every one of our readers, a fair shake.
Let me ask this. In what I believe is your def. of a blind tasting, you will know the variety and region before you taste the wine. If you taste, for example, a Cabernet Sauvignon from Virginia (which is where I happen to work) and you know that you have never given a Virginia Cabernet Sauvignon a score higher than, say 88 points, can you really say that you don’t have any preconceived notions about that wine?
“From the standpoint of sensory science, I don’t provide any information regarding producer or price to my judges performing analytical sensory evaluation so as not to bias them.”
Precisely, Sue, and if they know the variety or region, that would also produce a bias or preconception.
Do you subject tasters to random multiple samples of one or more wines to establish consistency? That’s where it gets interesting.
After all these years of tasting and evaluating wine, I am beginning to believe that only the extremely rare individual can do justice to wines beyond a handful at one sitting.
Jason: I agree. I don’t think it’s possible to review a wine without some notion or context to it.
Steve,
I don’t think that’s what Jason meant, but whether or not he did, if there’s one major message in Peynaud’s book, it is that context produces bias.
I’d like to ask a question of Thomas Matthews, if he’s still with us here.
You say that WS reviewers don’t know producer or price when they taste. I assume that they know the varietal or region in advance, which are clues in themselves, but do they also know in advance what’s in the line up and only don’t know which individual wine is being served in each glass?
Thomas, I think when Peynaud tasted wines he knew more or less what they were. He wasn’t tasting everything from Zinfandel, Petite Sirah and Grenache rosé to unoaked and oaked Chardonnay, Riesling, Sauvignon Blanc and Vognier. He was tasting the major appellations of France and I bet when he was tasting Gamay it was not a Beaujolais snuck into a tasting of red Burgundies with a couple bottles of Chateauneuf du Pape as ringers. In other words, Peynaud had context. What you seem to be talking about is double blind tasting and nobody I ever heard of tastes that way, except maybe WMs when they’re performing their parlor tricks.
Steve, I don’t quite remember the legal threat thing the same way. I never hesitated to go up against Marvin’s blustering. But 1993 is a long time ago and the precision of details tends to blur.
Whatever the case, I SO agree with your statement, that “We have fetishisized blind tasting into a religious cult, just as we’ve demonized open tasting into a sin.”
Hi Lewis, maybe the threat didn’t freak you out as much as it did me. I never forgot it.
Steve and Lewis,
Forgive me if I say that this “We have fetishisized blind tasting into a religious cult, just as we’ve demonized open tasting into a sin.” is total B.S.
It’s simply an attack on a concept you refuse to accept. I could very well be just as vicious and say that you have fetishized personal subjectivity as some sort of standard in wine enjoyment, but I wouldn’t say such a thing in a discussion because I know it would be aimed at ending the discussion, unless that was my aim.
Open tasting for so-called authoritative review is a sin. Let’s be honest here. It is one thing to know that you are tasting a bunch of West Coast Chardonnays. It is quite another to know which wines you are tasting.
It is one thing to know that you are tasting Left Bank Bordeaux reds. It is a sin, in my view, to be told that you are tasting First Growths because there is a built-in bias that is as unavoidable as if you were told that you were tasting DRCs in a Burgundy tasting.
There is no fetish here. It is a plain and simple fact. Knowing only the variety or type and the general region (West Coast, Left Bank, Tuscany) is enough for any competent taster to make reasonable judgments. I see no reason to have to also guess whether the wine is from Priorat or Happy Canyon or the Barossa or Argentina or Spain or France in order to taste Cabernet-based wines. Sure, if someone said, these are all Cabs from Fresno, that would be an indication of something.
But, if one tastes a bunch of CA Cabs all with the labels showing, I would argue that one would be much more hardpressed to make independent judgments–especially if there were a range of wines from $12 Castle Rock to $300 cult bottlings and everything in between.
So, it comes down to this. A certain amount of context is useful to provide focus. But, intimate knowledge aforehand is a guarantee that some judgments are going to be wrong.
Thomas, there is no end to this discussion, rest assured.
Steve,
Then don’t reduce it to name calling.
You say that you read and understand Peynaud, but you seem unable to grasp his basic message. Bringing up points like his knowledge and immersion in certain wines does not negate his message–in fact, he admits that it supports his message of bias why throughout his book he points oput the experiments done by others as well as him.
You can miss those salient facts only if you are insistent on not seeing them, as they are right in the book and in front of you. I can give you the page numbers, but I won’t sink that low–yet 😉
Charlie,
We are almost in agreement. Break out the expensive Dom alongside the inexpensive Cava and let’s celebrate the incremental step!
“Parlor tricks”
If someone is so good that they are better than a vast majority of others at sensory evaluation of a wine, it seems like sour grapes to use that term.
Arthur: Nobody’s that good. Never has been, never will be. You betcha!
Thomas, Peynaud for all his literary ability (and I love his book) is just a guy. Repeat: just a guy. Entitled to write whatever he wants. Don’t put him up on a pedestal as some sort of god among wine tasters. He represents a certain point of view among a certain class of wine industry people who flourished in a past age.
Again, I have to respectfully ask, Steve. Is it that your personal experiences and especially successes with blind tasting are coloring your view on this?
Those that are able to do it, are systematic and methodical and have an algorithm they follow.
Granted, a lot of wines are made from overripe fruit and typicity is blurred there. Some wines are so beat up in winemaking that they taste 15 years older than they are. This gets tricky and it may be difficult to peg the wine. But then this all this begs the question whether these are quality wines (if they bear the hallmarks of technical flaws – be they growing or elevage).
Tom P.–
Maybe we would agree on bubbles, but I disagree with you and agree with Steve for the most part about so-called double blind tasting.
It may make sense, although I really do not think so, for some people to be able to taste any wine any time any place and identify it. But, that is not what sensory analysis is about, and it is not what wine criticism is about.
I don’t know how many MWs you know, but I know a few, and they are no better tasters than anybody else. And in fact, their tasting training is geared to what Steve calls “parlor tricks”. They are not learning how to evaluate greatness or even goodness in terms of enjoyment. They are learning how to recognize wines from a certain place. To them, if they can say that Lynch-Bages is from the Left Bank, that is good, and if they can peg it as a Pauillac that is great. But, they are not being trained to evaluate the difference between Latour and Lynch-Bages and to then tell the consumer about those differences on a hedonistic enjoyment scale.
I don’t find that bad, but it is not the way that the millions of folks who read wine mags want their information. It does not matter whether we call it a parlour trick or an esoteric irrelevancy; it has very little to do with pleasure, mealtime use, grandeur.
Arthur: Read Charlie’s latest.
My Neurology professor had a saying: “If you don’t know what you’re looking for, you’ll never find it”. If you don’t know what you’re looking for, don’t know if and when you find it, then how in the world do you know what it is that you have? And how can you reliably make an authoritative judgment and recommendation to another person?
Charlie,
I’m not sure if guys like you and Steve either don’t understand or don’t want to understand, but I do not espouse the ridiculous notion that you seem to continually claim that I do as in your latest post.
First, I’ve met a lot of MWs, and haven’t been impressed by many of them.
Second, you don’t know what technical sensory training is, or you wouldn’t make the claim that having that training and knowledge is a parlor trick.
Third, I know exactly what wine criticism is, which is why I harbor a healthy suspicion of it.
Finally, those who think there is nothing left to learn flirt with being used up.
Steve,
For your edification, I checked: Einstein was just a guy with a theory. I agree: why all the fuss about just guys?
Thomas, well if you’re equating Emile Peynaud with Albert Einstein, I guess you’re entitled to your opinion!
Steve,
As you must know, I answered your non-response with a non-response that showed how empty your response was.
But if Einstein doesn’t do it for you and you still need the distraction, how about that guy Adam Smith?
You are engaging in the distasteful method of belittling what you refuse to understand, and I really don’t think there’s a reason for us to go on with this. Do you?
Tom–
We are getting off track here. I have not ever said that sensory training was a bad thing, I have not even said that MW training was a bad thing. I was simply commenting that MW training is not especially helpful as regards being able to make good hedonistic judgments and being able to transmit those judgments in words to an audience.
No one who ever earns an MW is a wine dunce. But neither are they necessarily good analysts of fine wine in the way that good writers need to be. As to whether we are or are not, that is answered in large measure by the marketplace.
Does it matter that I agree with Steve or Parker or Laube all the time on their sensory analyses? No, it does not, because they and I practice a different form of criticism than the type that I understand you to want.
And, Tom, please tell us what you want again if you think that Steve and I misunderstand you. Do not do it with reference to Peynaud or Einstein. Try specific types of judgments that you think critics need to be able to make in order to pass ourselves off as knowledgable.
Tom, I do wish to understand your point on this. Clearly, you have not articulated it in a way that I get it at this point. Please do try again.
Charlie, you really hit the nail on the head with what makes a wine critic relevant and their work important as opposed to what an MW does. BUT, in terms of what most readers use WE, WA, WS for (a split-second acknowledgment of numerical score), IS context really that important? My gut tells me no.
Context is really just another word for “acquired taste” and certainly not static. Furthermore, we all know that hedonistic enjoyment is suggested to and accepted by even experienced connoisseurs. Context might be relevant on a micro level (comparing different vintages of the same producer), but should have little influence on a macro level (hedonistic enjoyment reduced to a numerical score).
Charlie,
The best way that I can think of to try again is this:
Either you, Steve, or anyone so inclined please respond by explaining, without the use of a personal pronoun, how context avoids producing a biased result.
Maybe we can restart at that point.
Jason–
By context in wine criticism, I refer only to the amount of knowledge one has about a group of wines being tasted.
I see useful purpose in doing double blind tastings of sample that vary by place, grape and vintage. Basically, I am arguing for single blind tastings in which one knows enough basic info to bring a standard of expectations to bear on the tasting.
For example, tasting Red Burgs simply has to be judged by a different standard than tasting West Coast Pinot Noir. I dont want to know who is in the tasting. I don’t want to know where across the West Coast or Burgundy or New Zealand the wines came from.
I expect CA bubblies to be very good but not to taste precisely like Champagne. I do expect them to share similar structural elements in general.
That is what I mean by context. And I would argue that both wine criticism and wine enjoyment have those kinds of elements in their makeups.
So, while these seem lilke a lot of quibbles, I do think that we agree in very large measure.
Charlie
OK, Charlie, although you didn’t address my request specifically, and you did use personal pronouns 😉 let me pick up on your response anyway.
Assuming that we agree, that certain context is necessary in order to work within a standard, are the standards within which you work codified anywhere, written down so that everyone at the evaluation (or across all evaluations) works from the same set of standards?
If so, where are they, who developed these standards, and how?
If not, of what value are your newsletter’s standards in relation to the standards at WE, WS or even WA? And why should there be a separate set of evaluation standards at each separate point of evaluation?
What if without context, a Russian River wine is evaluated as if it were a Burgundy, or the other way around? With context, can there ever be that kind of result? Without context, does that kind of result rear its head?
Finally, how does context avoid producing a biased result?
Oops, should have written that I see no useful purpose in double blind tastings.
Sorry to return to this topic long after it has gone cold, but I wanted to answer some questions directed specifically at Wine Spectator, and respond to a few general points.
It is an interesting debate to determine how much context is appropriate when tasting wines.
Wine Spectator rejects double-blind tastings, since part of a wine’s quality and character should reflect its vintage and origin. On the other hand, we reject non-blind tastings, since so much evidence shows how inescapable various kinds of external bias are (price, label prestige, etc.). Therefore, the question becomes: how blind?
I think the amount of information will vary depending on the goal of the tasting. For new-release wines being officially reviewed, the most important thing is to blind the producer and the price, since many studies have determined that they almost inevitably bias judgment. On the other hand, as I’ve said, since wine is supposed to reflect growing season and vineyard origin, vintage and appellation seem necessary for appropriate assessment of quality and character.
Here’s what our tasters know when they sit down to a flight of wines: vintage, appellation, and varietal if it’s a formal part of the wine name.
Our tastings are set up by tasting coordinators; tasters do not know what specific wines are in any flight. The flights aim for some level of coherence without becoming predictable; that is, classified growths are mixed with non-classified chateaux, or $15 wines are mixed with $150 wines. It’s an art form, not a formula.
Our goal is to give every wine a fair and equal chance to show its best. Our methodology may not be perfect, but we think we have arrived at a procedure that avoids bias while giving enough context for informed assessment of quality, character and typicity. We hope readers will find our reviews credible and reliable.
Thomas Matthews
Executive editor
Wine Spectator