Is it fair to review a wine at the winery, with the winemaker, staring at the label?
I’ve asked this question before, so please forgive me if you think it’s a little tired as a topic. But it has arisen in a new, somewhat virulent form lately, for a variety of reasons I’m not prepared to explain now, but the details don’t matter. The big question is: Can a wine reviewer properly review a wine at the estate, with the winemaker, knowing what the wine is?
Consider, please, from the point of view of the reviewer. He or she has been cordially welcomed at the property. Probably there’s some kind of tour: of the vineyard, which of course is presented as the greatest viticulture on earth, and of the winery, with fabulous new equipment. All this is interspersed by probing conversations in which each side is trying to get to the essential personhood and humanity of the reviewer’s host.
Sometimes, it’s impossible to truly know your host. They haven’t invited you for that. But all the reviewer can do is try.
Now, we at Wine Enthusiast have a firm policy that wines cannot be reviewed except under strict blind circumstances. That is a severe policy, but it does make sense. The theory of our tasting department is that, if you know what the wine is, you cannot possibly be objective. Even if you think you know yourself pretty well–and who among us doesn’t think that?–you cannot help but be influenced by the facts. If you know the wine is an $8 California appellated wine, you’ll be unable to give it a very high score. On the other hand–I’m just reaching here for an illustration to make my point–if you’re at Screaming Eagle, you “know” you’re tasting a great Napa Valley Cabernet, and will be predisposed to give it a high score.
Think about your own integrity. Do you believe you’re incapable of “objectively” reviewing Screaming Eagle if you know what it is? Well, if you’re just some “average consumer” whose review won’t make it into print, you don’t really have to worry about that. But then, the “average consumer” will never be invited to Screaming Eagle to taste with the winemaker.
I don’t mean to pick on Screaming Eagle. I could substitute dozens of other wineries, mainly Napa cults but also including Pinot Noirs, Chardonnays and Rhônes from around the state, who refuse to allow critics to taste their wines, except at the estate, with the winemaker, after the necessary indoctrination. Here’s my question: What are they afraid of? I’m sure there are immature, insecure wine reviewers who are so flattered by being invited to the winery that they inflate the score. I don’t want to get into any lawsuits here, but to the best of my knowledge–and I tend to get good information–Robert Parker and his new California replacement, Antonio Galloni, taste openly, at the winery. Do you think that’s a fair, objective way to taste? I’m not saying it isn’t. I think I can know what I’m tasting and be uninfluenced by that knowledge. But I can’t prove it–and you, the consumer–deservedly need to know to what extent a review is “biased” and to what extent it’s unfettered.
To me, the more interested question ins, Why would a winery welcome a critic like me to review their wine, but only on the property, with the winemaker, openly looking at the bottle? I cannot answer this question: it’s something only the proprietor can. But I’ve been around long enough to venture a guess. When a critic tastes at the winery, he or she tends to overestimate the wine. If you’re dealing with the hundred-point system, this can amount to 3, 4, 5 points in the wine’s favor. Maybe more; who knows? Nobody’s ever done a scientific study, and no one ever will. That’s why wineries want us to taste with them: they know that, no matter how good their wines are, they’re often no better than a competition that costs less. They have something to hide, and to protect. If I have any legacy to leave behind me, it’s that a budding wine critic should view an invitation to taste only at the winery with high suspicion.
Yes. Any wine critic who dismisses out of hand any wine priced $8 or thereabouts needs to find another line of work. He or she can get in the same unemployment line with the (former) critic who gave high points to a Screaming Eagle because he/she knew it was Screaming Eagle. As to your concluding point, what’s to keep a budding critic from retasting a wine he/she first sampled at the winery with the vintner? And don’t give me any crap about the price of a Screaming Eagle or a Harlan; how many consumers do you know who actually buy and drink those winess? But God bless them; the resale and resale of those wines on the auction circuit underwrite all sorts of worthwhile medical and other supportive endeavors.
Steve, I love your consideration of incentives here – why would winemakers and proprietors require you to taste with them, openly, if they didn’t believe it to be in their favor?
Mike Dunne hit on eaxctly what I was thinking. If a critic can’t drink an $8 red and give it a 94 if it so deserves, and Screaming Eagle an 89 if it so deserves, then they are not doing their job. That’s like saying a food critic can’t rave about the taco truck down the street from Chez Panisse because it isn’t, well Chez Panisse.
However, I think Critics should regularly meet with winemakers, if anything to get to know the people they are judging.
I will always champion good honest people over vanity projects. I am no professional, so I can do as I please, but I wish there was more emphasis on the culture and people of the winery than just 92 vs. 94. I still don’t want the 94 if it is made be jerks. Give me honest, real people, and honest real wines. You can only learn that by shaking hands.
Wayne, you make some good points. But tell me, how do you propose I let readers know if I think a winemaker is “a jerk”? Should I write, “This wine is great, I give it 95 points, but the winemaker is a complete a-hole”? I. Don’t. Think. So!!!
Steve: Having met a few (fortunately very few) true a-hole winemakers I think it would be hilarious to have them called out. Terribly unprofessional, but hilarious.
As to reviewing at the winery, of course it biases the reviewer. At my client wineries there is nothing we like better than getting a buyer (and staff) up to our winery for a tour, tasting and lunch. We all have a good time, and it helps sell wine. If a winery will only allow a review at the winery, perhaps the answer is to provide tasting notes, but no score.
Critics, who happen to be journalists and are plying a professional trade, need to be out in the field and to meet with winery personel when they can and as often as they can.
But, critic evaluation is another topic entirely. Any critic who tastes open label wines at the winery, and this is not confined to Parker/Galloni, has become part cheerleader. And that is the way the wineries want it to be.
I have been told by several wineries that I am welcome to come and to taste their wines with them but that they will not allow those wines to go into my blind tastings. When I asked one of those esteemed individuals if he thought that was a fair way to review wine, he replied, “I get better scores that way”.
Consumers should not put up with this very bad practice. And maybe one of these days, some of us will start mentioning names in public. Not that Screaming Eagle or Littorai will really care because they tend to sell all or most of their wine either to their mailing lists or to high-priced restaurants.
Steve, you are an intelligent man who has a lot of adjectives at your disposal. You would find a clever way of “telling” a deeper story whether it be about an areshole or not! And that’s the point Wayne makes, I believe, that there’s so much more to a wine than simply what is in the bottle; it’s the people and place that are integral to its being, and that can only be experienced by being there. I don’t believe a winemaker is afraid of anything, but tired of impersonal reviews churned out that take the wine so far from its original context.
I think it is important for wine writers to visit the properties and get to know the players and the rest of the context. I think it is OK to taste wines with the winemaker and proprietor, and to write about that experience.
I think it is absolutely unethical to give those wines a numerical score and put the tasting notes in the publication alongside the wines you have tasted professionally, i.e. blind. That would disadvantage everyone else who has sent wines in for blind review, and shame on the proprietors who think they deserve special treatment.
Steve, I don’t understand why this continues to come up. We keep discussing transparency and authenticity, but in my opinion even giving your consideration to the idea that it might be OK to do a professional scored evaluation on a wine, tasted openly at the winery with the winemaker, casts a shadow – however small – on the credibility of your blind reviews. Why would you want to taint your stock-in-trade by treating this as a serious issue?
If a proprietor insists you taste openly at the winery, it seems to me that you should ask your editor if that winery’s story is worth at least 500 words, and if there is room in an upcoming issue. If the answer is no, then by all means accept the winery’s offer to come taste for context, but don’t do any evaluation. There is no other ethically acceptable answer.
On the other hand, if get such an invitation you could assume a disguise and taste in the tasting room – like a restaurant critic. But make sure your readers know that’s what you did. And again, such reviews have no place among the ratings for wines you have evaluated under your standard blind protocol.
This is a valid discussion. There are points on both sides of the ledger and I think it is more complex than simple.
Without question if treated well, one’s impression can clearly be enhanced by tasting at the winery. If somehow the person is not treated they way they think they should be, it can work in reverse. I know of situations both ways (neither has had anything to do with any of my wines).
If a writer is referring to a visit and/or tasting where they clearly know the people and the what the wines are, then they should reference that the notes are from a tasting non-blind and from a visit (if that is the case). If one is invited to taste a new project in Italy with Piero Antinori, then doing such and reporting on it would be of interest to the readers. I think initial tasting comments and descriptions are expected, at least on the highlights. I think in a case like this, if the circumstances are noted it would be fine in context.
Blind tastings are not fool proof either, I have been judging numerous competitions for over 25 years myself and there are some flaws in this process which can lead to wildly different ratings and awards across various competitions and/or publications.
All too often I see several issues as causing problems in comparisons. One is tasting wines out of order when it comes to residual sugar. Chardonnays for many years have suffered in this kind of tasting. The companies that make sweeter chardonnays for years would never declare the actual RS. They would often say “dry” or list the RS as 0 (zero). Therefore moving them to be tasted before the true dry wines that actually listed a number of 0.01 or 0.1, thus messing up the order of tasting where the sweeter wines should always follow the dry wines. One large winery used to do this all of the time to put the odds in their favor and it worked for them very well.
Another issue is blends. Often I have seen other publications (NOT the Wine Enthusiast) taste Meritage blends that may have less than 50% Cabernet Sauvignon reviewed with and against Cabernet Sauvignons. This can be unfair to either or both.
I fear that there may not be one perfect way of judging and even more difficult in larger numbers.
This should not discourage reviews or competitions, because we all still like to read the comments and results.
Having read all of the above….., my opinion is that the only objective way to taste wine is blind tasting and I fail to see where the personality of the winemaker or proprietor has anything to do with the final evaluation of the wine.
Steve: “But tell me, how do you propose I let readers know if I think a winemaker is “a jerk”?”
That’s the problem, you can’t and shouldn’t do that. I often think of restaurant critics, they can’t separate the product from the experience.
Stephanie said it better than me, ” there’s so much more to a wine than simply what is in the bottle; it’s the people and place that are integral to its being, and that can only be experienced by being there.”
We expect Critics to have opinions, that’s why you exist. Should you be fair in the way you rate wines? Yes. Should you steer readers towards producers you think are a cut above as people or organizations? I think so. There are “creative” ways to do that while still being fair and balanced.
Steve, you say you as a result of working for “Wine Enthusiast have a firm policy that wines cannot be reviewed except under strict blind circumstances.” Harlan Estate and Screaming Eagle are notorious for not submitting wines for review and only let critics taste at the winery. Am I mistaken?
In a previous post (4/21/11), you describe your visit to Harlan and in the comments you stated that do not review the wines formal and publish no scores from these visits. Yet, you do publish scores alongside reviews of both of these wineries (I assume not from that visit, but previous visits). Are you being a hypocrite, lying or do Harlan and Screagle actually send you review samples, which you then put in a lineup of other high-end Napa cabs in brown bags at you house?
Basically I agree with the post, but there are critics who claim to taste blind “whenever they can” in their newletters,and whom we know almost never taste blind. The results are always predictable.
And there are different degrees of tasting blind. There are critics who separate their “blind tastings” into flights that they are aware of the important details. I know, some will say separating the five digit priced cults from the hoi polloi or, vineyard designated from blended, or Napa Cabs from Sonoma Cabs is necessary for some reason, but it gives results which are tainted with that knowledge and are predictable. If you taste a bunch of inexpensive Cabs and one has a lot of oak it is usually seen as a defect. But when you taste a line up of cults, they all have a lot of oak, and it is not seen as a defect. So why not give that $8 Cab a shot at the cults?
I give you more digs than you deserve, Steve. But I seem to remember a post where you described a visit to Harlan where you tasted several wines “blind.” (I know what you meant.) But I will never forget my disappointment with Dorothy and John, when they bought a line up of first growth clarets and, not only did they not put them up blind against New World equivalents, but they had them with dinner with labels showing, and no surprise, they gushed in their praise.
All in all the tragedy of all this is that the most interesting and fun thing about winemaking is unpredictablity. We should encourage surprises.
Persosonlly, I don’t have a problem with people tasting at estates. I think it’s important for critics to know and associate with the people amd properties they review. However, I would really like more transparency in the matter. As simple as someone writing “tasted at Screaming Eagle in March 2011” at the end of the note. I want to know if the review was done blind or not, because as you say, it certainly inflates the score
I have a somewhat unique perspective on this subject. As someone who spent nearly two decades as a wine buyer for specialty retailers, I didn’t taste blind because of the impracticality of sampling a variety of wines on any given day and retain that regimen even though I started publishing my reviews last Fall.
Although I respect the choice of some writers to taste blind, I don’t see any measureable influence on a review I give a wine tasted non-blind, be it in my office or at a winery. You posit that tasting a wine non-blind with a winemaker can result in possibly more than 5 points in a wine’s favor. Seriously? I don’t know of any established professional critics I would put in that group. For any of your readers who want to see an example of that, I recommend visiting the Blackbird Vineyard website since it contains reviews from several sources including WE, tasted blind and others tasted non blind. To my point, I seem to recall you are higher on at least one wine from the most recent reviews posted there.
Finally, I don’t hear back from every winery I contact, but if they do make it a condition to taste their wines at their property, many offer me the option to taste privately which is what I prefer.
This whole thing reminds me of something that had different names back in the late 80’s/ early 90’s. One name I remember was “the Dominus effect”. When Dominus came onto the marketplace, it was thought, by many retailers and restaurateurs, that the wine, while good, was getting higher reviews than many thought it deserved. The initial vintages were more tannic. Ownership actually released the riper ’84 vintage first and the leaner ’83 (original vintage) afterwards a year later.
Anyway many industry people felt the wines were getting higher than deserved ratings due to the fact that it was owned by the Moueix family who owned Petrus. The feeling was that one didn’t want to ruin a chance to miss out on a tasting of Petrus. Fair or not, that was a perception at the time.
Interestingly, the early Dominus wines were more Bordeaux like than they later became and quite different in style than many of the Napa Cabs or for that matter the Merlot based Petrus.
SIDE NOTE: I actually bought Cabernet Sauvignon from them in 1983 for use in my second Napa Valley Cabernet and that wine is still alive.
When some of you speak of visiting the winery to add “context” to the review I agree that many consumers value this but at that point it must be distinguished as a feature article and no longer as a critical review.
By definition, I don’t think a critical review can be made “at the winery” with winemaker present. How many of us have enjoyed “great” wines in small French or Italian villages only to bring them home and be disappointed???
Steve…Hey…
I recall from my halcyon days
that the early negociants in Burgundy had a saying…
“Buy on apples, sell on cheese.”
Personally, I look at it as follows.
If you review a wine blind and give it a 94 point rating and consumers try the wine and don’t feel like it is a 94 point wine, you lose face.
If you review a wine non-blind and give it a 94 point rating and consumers try the wine and don’t feel like it is a 94 point wine, you lose face.
Bottom line, if your reviews aren’t reflective of what’s in the bottle – regardless of the setting – your probably won’t be at it for too long.
I have noted that The Wine Advocate tastes some wines blind and some wines non-blind. Personally, I believe that if you’re going to strive to provide blind reviews, then it has to be all or nothing. I understand the logistical issues behind this, but tasting some wines blind and some non-blind – usually for TWA the top producers – means that there is not a level playing field for everyone. The top producers appear to be given special treatment. Again, I understand the reason for this and not being funded by advertisements puts time and money at a premium. However, even if there were no difference between blind and non-blind tastings, the appearance of a difference detracts from the review. Additionally, I would like to see the format (blind/non-blind) more clearly called out in TWA reviews. If you are doing some blind and some non-blind but not specifically calling it out, for me they are all non-blind.
As an independent writer, I taste wines in a variety of settings, including visits to wineries. If I could taste all wines in the same, blind format, I would. I cannot, so I don’t. What I do do is list in my tasting note database the setting (private, meaning I’m tasting at home, or public, meaning I’m tasting at a winery). I also list whether the bottle was received as a sample or not. My hope is that by providing transparency, people can make up their own minds.
If a winery does not submit wines for blind tastings, that winery must be afraid of something.
Steve,
When you first posted this entry yesterday, I rolled my eyes and said to myself: “This again, after such a great, thought-provoking entry about the future of ethics the day before?” And BTW, I do reccommend people follow the link you had to Lindstrom’s article, maybe someone will even start a “WineLeak” website where unethical stuff can be exposed.
Now I think I realize that the real story is in your first paragraph: “forgive me if you think it’s a little tired as a topic. But it has arisen in a new, somewhat virulent form lately” and – of course, I want to know what that story is.
In the title of the post, you ask about fairness. I’d like to raise a question in response: “What has fairness got to do with this?” As Vic Motto says in his post, it is all about selling. We treat business as we treat war, and in war – all is fair. Don’t get me wrong, I think you and many other reviewers are super honest and ethical players, but the field in NOT level and nothing about the rules is fair.
To be fair to the CONSUMER, you should only rate wines that you buy with your own (hard earned) cash, I am not aware of anyone still doing it.
Does anyone else find it odd that Steve has been almost silent in the comments of this post? Does he really feel that no comments deserve a response?
Dear Colorado, which comments do you wish I’d replied to, and what do you think I should have said?
Well, I personally want to know about the Harlan scores/samples. I do think Doug, Oded and Mark amongst others have comments that deserve comments.
Mostly, I was just surprised at how quite you have been. In lots of other posts you respond to almost every comment.
And just for the record, I have no problem with tasting at the winery as long as you (collective, not you specifically) are 100% honest with your methods. TWA for one is not. I still believe blind tasting is the best method, but as long as the critic is transparent consumers can choose to whom they listen.
Colorado, I never respond to “almost every comment.” If you think that, you don’t read my blog. I reply when my instinct is to say something. I deliberately do not respond to every comment in order to build up certain rankings in the online world, which sadly seems to be the practice of other wine blogs. But you see, if someone says something that requires a response (as you have), I will rise to the occasion. This is in no way an insult to my commenters, whom I respect and bless. It’s simply to acknowledge that not every comment needs a reply.
I do read your blog almost everyday. I don’t always agree with what you write, but you usually write things that deserve a response.
One of the things I admire is that you do respond quite often. No, that does not mean to every comment, but you are very active in the comment section. Today you were not. I figured calling you either a hypocrite or a liar was sure to get a response from you 😉
Oh, and is Susan K still working at WE? I’ve sent her a few emails, but have not had any sort of response.
Are you purposefully not answering my question about your Harlan and Screagle scores? I truly am not trying to be rude, but just am confused with an apparent discrepancy.
Steve –
You frame the discussion very well. I wonder if it will always remain a dialogue with no conclusion. Not because I wouldn’t desire a determined answer, but because I wonder if one is possible (and you intimate this in your tone). Much like the discussion of blind tasting in general, the answer of its value is often “it depends”.
I will only add to the lengthy discussion and your comments a wondering of whether an argument could be made that scores tasted at a winery, or at least ones that are not blind, might actually be more reflective of consumer experience? Again, just posing the question. What I mean is, who tastes blind at home while dining? If a wine scores a few extra points because of a relationship or the history of the label, is that different than someone opening a Saxum and enjoying it more because they know if has made the cover of Spectator? How do we integrate, in a credible and ethical way, the fact that this kind of psychology plays a large role in how everyone tastes?
Since very few people ever enjoy wine at home in a un-objective manner, might not that kind of critical evaluation be more reflective of consumer experience? I realize this is a very slippery slope…but one wonders.
Tyler, you raise all of the most pertinent questions, to which there is “no conclusion,” in your words. All that the consumer can do is be armed with the most information possible, and then make informed decisions.
Steve, above, said, “All that the consumer can do is be armed with the most information possible, and then make informed decisions.”
Good point. That’s precisely why I pointedly REFUSE to buy wine based on anyone’s criticism or scores. Scores and criticism are mostly misinformation of one type or another.
I read, and learn, and buy according to my budget. I make mistakes but invariably find gems.
Steve,
I think you condemn the wineries to quickly. You assume their motives are solely for an unethical competitive edge. (There certainly are those.) A winery might simply want to know, for certain, the circumstances of the tasting to believe they could receive a fair shake. Besides rampant poor tasting techniques, there’s more than just a hint of corruption to the industry. It’s not just non-disclosure of visits that can be viewed as corrupted scores. I often hear stories of winemakers present at “blind” tastings, scores that magically went up when the winery hired the friend of reviewer x, rewards for reviews, blind tastings that aren’t actually blind, and pure for profit score inflation. I appreciate your attempts to combat this with transparency and discussions such as this one. Winery cynics, though, might consider it more “fair” if they controlled the circumstances of the tasting.
Scott Mahon, if you’re asking for “a fair shake,” then the fairest way is to taste all wines under the same circumstances, i.e. blind, in flights. I obviously can’t speak to “rampant poor tasting techniques” in the industry, but only for myself. I taste professionally and objectively. But I can state that a winery that demands that I taste openly, at the winery, with the winemaker, under special circumstances, is afraid. I’m not saying I don’t have the ability to be objective, even at the winery. This speaks more to the mindset of the wineries than to my abilities.
You still haven’t answered my Harlan question…
Steve,
Thanks for the feedback. As a winery owner, I tend to get a little defensive about the winery point of view. I wished to bring up a perspective that was missing from the conversation. That it isn’t just the consumer whose trust is compromised by reviewers who lack transparency and objectivity. And, that this was just the tip of that iceberg.
However, in my haste to defend the view of the winery, I glossed over the situational irony and hypocrisy of those wineries “demand”s. You are right. A winery that demands that a reviewer only taste their wines at their place is afraid their wine won’t hold up. If they don’t trust your process, why should they want you to review their wines? Unless, they think you are corruptible. I would never ask for my wines to be reviewed by someone who I could not trust in their own methodology. (Excepting post hoc discoveries.)
While those wineries may be hypocritical and a bit unethical, I still sympathize with the wineries’ lack of faith in the promise of a blind review (though not their method for dealing with it.) I’m happy to hear that you routinely dismiss this “demand.” I look forward to a time when I can feel as confident about wine reviewers’ objectivity.