subscribe: Posts | Comments      Facebook      Email Steve

Minimizing the subjectivity of wine reviewing

34 comments

It comes as no news to me that “lighting can influence both how wine tastes and how much consumers are willing to pay for it.”

Everything
influences how wine tastes: temperature, setting, time of day or night, what you previously ate, how you feel, if you got enough sleep, the glasses you taste from, the flight in which the wine is included, what you see outside the window (if there is a window), whom you are with — I could go on.

Under these circumstances, the curious reader will wonder, “Well, then what’s the value of a wine review?” This is a fair question, and one that can’t be analyzed enough.

I know a fellow — Rod Smith, whom many of you also know — a fine writer. We once were at a tasting that Andy Beckstoffer held in his Rutherford offices of Cabernet Sauvignons from his portion of the To Kalon Vineyard (Robert Mondavi’s portion is spelled “Tokalon”). There was a small group of us scribes sitting around a table, tasting and scribbling. Rod had been fairly silent, so I asked him what he thought of the wines.

I remember Rod giving me a less than charitable glance and then saying, in fairly withering terms, “I don’t review wines. I write about them.” Well, sure; I took his point. Rod had reached the conclusion (I’m doing a lot of inferring here, but I think it’s true) that wine criticism is so inherently subjective, there’s no point in doing it. His approach is to write beautifully and elegantly and factually on all aspects of wine’s history and production.

I do that, too, both in my articles for Wine Enthusiast and in my books for University of California Press. (In my Russian River book, there are only one or two critical remarks made about specific wines, and none at all in my Conversations book.) But I also am paid to be a critic, and so a critic I must be. That means I have got occasionally to defend our practice, in spite of the many instabilities that afflict it.

Along these lines, 1WineDude wrote yesterday of his experience at the pre-Premier Napa Valley tasting, where our hosts had graciously set up big flights of Cabernet and Chardonnay. The Dude described his aversion to tasting his way through such massive events (and gently prodded Vinography for doing so). I didn’t make it into print in that posting, but I was there at the Culinary Institute of America, and ran into Dude at one point. When he asked me what wines I liked, I had to tell him, “None,” because the fact is that I wasn’t there to drink or taste. It makes no sense at all to me to try and review wines seriously under the circumstances of a mob scene, in a fairly alien environment of fuss and confused commotion. Instead, I took advantage of the scene to study it, rather like an anthropologist in the field (Margaret Mead among the Samoans?), witnessing the sometimes odd, sometimes amusing, sometimes baffling behavior of the populace. You can learn a lot from just watching people, especially when so many of them are bloggers.

When I taste wine formally, it has to be under precise circumstances in my home. Same time of day, same glasses, same table, same computer, same pattern of opening bottles in the kitchen and bagging them, same corkscrew (a standard somm’s), same view outside my window of a terraceful of geraniums and cacti, same lighting, even with the same TV turned on (with the sound off), which comforts me. Only then can I be assured that all the influences I described above can be minimized in their impact.

Does that make my winetasting less subject to distortion? Yes. Does it make it perfect? No. People who are deadset against individual wine reviewing will always find plenty of reasons to criticize it, and their reasons have some validity. All I can do is do my job, as carefully as I know how, and hope it has some value.

  1. Stand up for your standards, Steve!

  2. Don’t forget to close the window–the smell of those geraniums outside your window might make you think of sorbate having met with malo-lactic fermentation–or maybe it won’t.

  3. When I wrote & edited news stories for CBS News 10 years ago, objectivity was paramount.

    In my personal life, I own every book Ayn Rand wrote & consider myself an Objectivist.

    But when I write about wine I have ZERO interest in objectivity. NONE. For me, taste is entirely subjective & I don’t try to pretend otherwise.

  4. Steve, interesting post. I taste wines every day, and like you, I have precise circumstances that I prefer to do my formal tastings in. At my desk in my office at the winery, at the same time in the late morning, with the same lining up of the samples, same tasting sheet, same 0.9 mechanical pencil for writing, etc, etc. As an aside, I use the 100 point scale when tasting formally. I find it a valuable tool in assessing my wines through the various phases of production, and it gives me a metric of gauging the development of a wine that I may taste as many as 24 times over the course of a two year aging program. It helps me too, when I review through pages and pages of production tasting notes leading up to blending trials. Like you, I’ve found that this particular set of circumstances, while maybe not perfect, creates my best results.

  5. As I sit here working with the TV on and the sound off, I have to say “bravo, Steve” for making the effort to minimize the variation in the conditions of your critical tastings. I hope others out there who choose to engage in criticism are making the same effort – I do know some who are, but I fear many are not. Like Rod Smith I no longer taste wines critically (except for my own, and only in two venues – the winery or my office, about the same time of day, after a double espresso, same glasses). When at dinner or with friends I am frequently asked what I think of a wine, though, and I try to keep it general: “Hmmm, I like it.” If I am pestered to break it down I preface my comments with “At this moment, under these conditions, it seems to me…” Anything else would presume a certainty I do not feel and an authority I do not possess.

  6. As someone who is involved on the receiving end of the reviewers pen I can very much appreciate your words. The value of a particular review only has validity if the commentators palate is in sync with ones own. At large “tasting” events I am always puzzled at what attendees gravitate towards and how they can distinguish one wine from another. Palate fatigue of professionals is just as possible as amateurs so controlling your tasting environment just as you described is critical for consistency. When you review it is not about “liking” or “disliking” a particular wine but rather accurately recording your experience.
    Being able to distinguish a wines quality from personal preferences is what makes a professional good at his/her job.

  7. Steve,

    Thanks for your insight. Just as Jimmy, I too, am on the receiving end of wine reviewing and I am a bit surprised by the recent backlash against wine reviewers. Your reviews, while I may not always agree, are well done, tasteful, and never angry or mean. And while there is almost always complaint against Robert Parker, he is presenting his view of the wine – it is not gospel – though many may believe it is and live and die by his words… The same with James Laube or any other reviewer. Are Parker and Laube as subjective as Steve Heimoff? I would like to think they may be – they just have different preferences.

    I have had one of my wines ranked from an “88” to a “95” and the same wine won a “gold medal” for what that was worth. Why the wide variation? Personal taste? and is it possible there isn’t enough subjectivity? Was the “88” because the reviewer had a fight with his wife that day and was getting over a cold? or was the “95” because that reviewer just got a bonus check and had a nice dinner at Bottega and was loving life? Don’t know the answer to that, but your comments, at least, give us some perspective on how you evaluate wines and try to make it more subjective and less subject to outside influences.

    Richard.

  8. Carlos Toledo says:

    The blog writers in Brazil where i live now (sigh) are just interested in getting free samples (i.e., to get drunk for free) and making several bucks by ”hinting” that importer has good stuff or that store is a good pick for the week.

    If everyone were as honest and plain (no offense here) as you we’d be living on a slightly better planet. Too bad you can’t write about more corners of our small world…… just California….

  9. Carlos, that’s sad to hear about the Brazilian bloggers. The wineries should try to find out who’s honest and who isn’t, and just send samples to the honest ones.

  10. Morton Leslie says:

    As much as I hate to say it I every now and then make a buying decision on the basis of what a critic has to say about a wine. But it is never based on a numerical score or some large assemblage of tasting results. Even when tasted blind there are too many ways that such criticism and scoring can go wrong from taster fatique, contrast errors, lack of concentration, or other variables. But it seems that the volume of scores rather than the quality of scores is what really pays the bill, so I understand.

    Usually the critic I take recommendations from is a retailer who knows my tastes and knows his wines. But a few years ago there was writer for a now defunct East Coast newspaper who used to forward me a copy of his weekly column on the day it was published. It was a short column about a wide range of wine subjects. Each week there were usually two wines that he recommended, though he was tasting a great many every week. There was usually a paragraph or two about the wine and why he thought it was special. He does have an excellent palate, but I always sensed he had swallowed a fair amount of the wines he recommended, maybe over a meal. I started to buy some of his recommendations from the East Coast retailers and have them shipped here. I think over the years he batted a thousand.

    I think one thing really missing today in wine criticism is there is too much sniffing, spitting and scoring and not enough swallowing. What I’m looking for in a critic is wine appreciation.

  11. Well said, Morton, but if critics swallowed we’d be drunk all the time.

  12. We have been sending every wine we make to you since our 2005 vintage. We started doing this because WE is transparent in their process and we can have you evaluate everything we do. We do not always get the results we clearly deserve :), but we have been able to build a baseline that we can improve upon. I thank you for your consistancy and allowing us to learn on your experience.

  13. Steve, thank you back.

  14. Some years ago, Rod Smith and I alternated writing the weekly wine column in the Los Angeles Times. Russ Parsons was the Editor of the Food Section and he asked me to write a wine tasting column every second week. In the alternate weeks, Rod wrote wine stories. We rarely clashed on topics or coverage because we had very different briefs.

    On the rare occasion when we would show up at the same event, we would compare notes and decide whether there was something for me or something for him. Mostly him, of course, because, aside from the occasional tasting of older wines typically from one winery, I only wrote about wines tasted blind at my own tastings.

    He deserves a great deal of respect for sticking to his last. Not all of have to be wine critics–thank goodness.

  15. I think there’s an elephant in the virtual room here… which is: Is the reason that we even have to talk about minimizing subjectivity that some people have established money-making enterprises by stating that wine can be reduced to totally objective components and then, therefore, reviewed totally objectively?

  16. Dude, you’re being too mental. I don’t know anybody who ever said “wine can be reduced to totally objective components and then, therefore, reviewed totally objectively” except for some of the commenters on my blog who say that. I don’t say it. I get paid to give my opinion. Opinion: definition: “a belief not based on absolute certainty or positive knowledge but on what seems true, valid, or probable to one’s mind.”

  17. His Dudeness may be onto something here. He may have realized that some wine reviewers’ opinions are so rigorously acquired as to become objective instead of subjective. And he may also have finally accepted that the 100-point scale is totally and unchallengingly objective, finite, infallible and holy.

    I always said that boy would go far in life. That is why I adopted him. In twenty years or so, when I am sitting around the old-age home sucking back the pablum, he will be rich and famous and will send me postcards.

    Or maybe, he is just pricking a balloon or two, of thee and me and a few ohter folks who think our opinions are about as close to being objective as subjective can get. I plead guilty to that sin because of the way we taste at Connoisseurs’ Guide.

    Perhaps the best thing to do is ask him what he meant. Hey, Joe Dude, to whose and which elephants are you referring?

  18. The Dudester has properly characterized the holy grail being pursued by Arthur Prezebinda of Redwinebuzz and to a related extent by Clark Smith, Appellation America, Ken Payton and other objectivists.

    From Arthur’s ~About Us~ page: “Arthur drew on his medical specialty in diagnostic imaging [he is an M.D.] to create a way of describing wine that is more concrete, consistent and objective. He compares it to the way diamonds are rated on the basis of color, clarity, cut and weight – separately. This approach allows him to look at each wine objectively and not through the subjective lens colored by preference. By separating himself from he description of the wine, he offers readers the information necessary to select wines according to their own tastes and preferences and make more informed purchase decisions.”

  19. Thanks, pops! I’ll stock up on the postcards now! :-)

    What I meant was not that WE or Steve claim to be totally objective. What I meant is that TWA and WS have nice explanations on how their tastings are rigorous, but not to take too much stake in the scores.

    Then, PR, retailers, and consumers go ahead and take way too much stock in the scores and treat them as objective truth.

    So what do TWA and WS and others do to counter that misguided notion?

    Nothing. Because it probably boosts their cred and sales.

    I just wish that more reviewers would stand up and stand behind what they’re doing and how they’re doing it, and speak out when it gets out of hand. EXACTLY as Steve is doing here.

  20. And now for the Merlean way to deal with subjectivity….

    If I were able to be a poor man’s Michel Rolland I would advise winemakers like Chris Corley, John Kelly, Steve Christian, Steve Mirassou and others who surf in here every so often to create a tasting panel of Real People (wine club members?), people who have to buy their wine and don’t try to evaluate more than say eight wines in one sitting to serve as advisors. Yes, a kind of focus group–my beloved “Crowd”. Or even “consulting winemakers”. Oh the hit to the ego…but, potentially, not to the bank account.

    As the winemakers are tweaking their wines from fermentation through all the steps to final bottling, but mostly toward the end of the process as blending percentages change, the panel would provide feedback with an eye toward what the market will reward i.e. what they would like to buy. Examples of various styles from others could also be inserted as well.

    This alternative runs counter to placing wines in huge competitions where we know there is only a 10% replicability factor (talk about subjectivity) or relying on a single palate of someone whose persistence and journalism skills got them a wine writing gig or on the winemaker’s own palate which may not be in sync with what consumers are seeking.

    TOM

    PS. Rod Smith is one of the master wine writers. He manages to capture the back, front and side stories–the human dimension of the wondrous world of wine–in just the right number of words.

  21. I can’t come down too hard on TWA and WS or anyone else who gives scores/reviews. I might be a little more transparent than they are concerning the inherent flaws in the system. But every system has flaws. There’s no perfect way to evaluate wine, or recommend it, or describe it. None. Zilch. Nada. I don’t care what method somebody dreams up, it’s always going to have flaws. In my judgment, a single reviewer, with a known palate, is better than the chaos of a big competition, and the 100 point system is a well-understood form of shorthand describing the reviewer’s reaction to the wine.

  22. Tom, well, anyone can make any claim they want. I’ve seen lots of claims in the last few months about “We’re the most objective rating panel there is!” or “We take the subjectivity out of wine reviewing!” Bollocks, as my old friend Johnny Rotten used to yell. Just because somebody says something doesn’t make it so (with all due respect to Arthur). Ultimately, every wine reviewer is only human, with a human’s limitations and variability. The only objective analysis of wine occurs in a laboratory, using equipment, not people.

  23. “I don’t care what method somebody dreams up, it’s always going to have flaws. …The only objective analysis of wine occurs in a laboratory, using equipment, not people.”

    EXACTLY.

  24. Whoops – for got to add this thought (which I also left in a similar thread on Jamie Goode’s blog at http://www.wineanorak.com/wineblog/business-of-wine/why-we-should-be-grateful-for-robert-parker ):

    It could be argued that Parkers of the world now play into (or at least ignore) the condition that the rating systems hath wrought BUT…

    Blaming TWA or WS for how the system is misused I suppose is a bit like blaming George Lucas for every effects-laden bad movie made since Star Wars was filmed.

  25. I sense there is an irreversible trend, not only in the wine world but everywhere, towards endowing consumers to form their own opinion, construe their own palate, and take their own decisions, through information, facts…
    There is a ceaseless pursuit of data-information-knowledge going on and it is not an exclusive “wine nerd” facet anymore. It is a new paradigm; information became free (e.g. WE ratings), real-time and easy to get.
    Steve, you forgot the white tablecloth and to fine-tune room temperature.

  26. I think people are always going to turn to “experts” when it comes to wine, movies, consumer products, cars, etc. There are just too many options for most people to take the time to study things in detail. I don’t know who the wine experts will be in 5-10 years, or where they’ll be writing (print? online?), but I do think they’ll be around, and consumers will still be listening to them.

  27. Update: in the dialogue we’ve been having over the past months between the guidance offered by the single professional/expert vs. groups of amateurs/enthusiasts, readers will want to know that Eric LeVine of CellarTracker fame has just launched his successor site http://GrapeStories.com. The paradigm shift, to use Peter’s word, just got more pronounced.

  28. Tom Merle, do you have a business relationship of any sort with Cellar Tracker. There are reasons to like CT, and that is fine, but I get this nagging feeling that your constant promotion of CT has a very cozy ring to it.

    Yes or no. Transparency one way or the other, please. And nothing derogatory meant in any event. On the subject of self-promotion, I throw no stones.

  29. Nope none, Charlie. Never even met Eric LeVine. My infatuation has solely to do with its capacity to capture the opinions of “core wine drinkers”. I would support any vehicle that does a good job at crowd sourcing. If you click on my name you’ll see my very modest effort at such an activity.

    I wouldn’t keep injecting it into our discussions, but so many of Steve’s blog posts touch on the theme (meme) of wine evaluation. Maybe this thread is a bit of a stretch since it has to do with how individual tasters, be they winemakers, critics or wine lovers, deal with subjectivity. But there are always new posters showing up so I jump back on my hobby horse. I’ll try to restrain myself more because for regulars like you it’s become quite tiresome. I just couldn’t resist responding this way to Peter O’Connor’s comment (whose name I don’t recall seeing here)

  30. Tom:

    The whole idea of a “focus group” is odious to me (I don’t think you meant to ascribe the “make-everything-to-the-great-middle” connotation by which that term has come to be defined).

    I started making wine with the idea that I can’t really “know” the palate of the folks who may buy my wine in the future so my obligation is to hew as closely to my winemaking vision as possible, making the best wine I can, and provide those who are interested the opportunity to taste it.

    So far this approach has worked well for Steven Kent and La Rochelle. The fact of the matter is, though, that everyday we have a “focus group” visit us from Noon to 4:30, and it renders its verdict with its wallet.

  31. While I appreciate your attempt to control tasting variables, Steve, I take the exact opposite tact: I attempt to recreate the consumer’s experience of wine as closely as possible.

    As a reviewer, I am essentially a stand-in for informed wine consumers – who are far more likely to enjoy wine in variable company at restaurants, tasting rooms, and dinner parties than alone at a desk. As a consumer representative, I need to know: can the wine can rise above the din of competing claims on consumers’ attention, and leave positive impressions of a specific flavor profile and a winemaker’s particular talents, plus overall satisfaction with quality/price and food/event pairing possibilities?

    These are not secondary considerations for wine consumers, and if we eliminate them from our purview as reviewers, we will soon make ourselves obsolete.

    That said, I do agree the mayhem of mass tastings like SFChron or Passport days can be exhausting, and I know I can be a tougher critic as a result. But to stay in line with consumer expectations and experience, I participate in mass tastings regularly – and often find pleasant surprises. Wines that quietly transcend setbacks of stumbling crowds and harried pourers are standouts indeed.

    Collegially,
    Alison Bing

  32. Alison, thanks for the comment. I guess each of has has a different job to do, and different ways of doing it. My way works for me, and your way works for you. Cheers!

  33. Quite right, Steve: if I were routinely required to provide detailed notes on wines that don’t make the grade, I’d need to be left alone with the wine too … and I’d probably need a stiff drink afterward. Those critical reviews are always subject to the closest scrutiny, and that takes rigor, humor, and a thick skin. Glad to have you on the job.
    Cheers back at you,
    Alison

  34. Steve,

    I beg to differ with your comment that “The only objective analysis of wine occurs in a laboratory, using equipment, not people.” The field of sensory science exists because human beings CAN be trained to be objective analytical measuring devices. The wine industry has lagged behind the food industry, health care products and others in utilizing these methods, mainly because wine is perceived as an art form and therefore subjective. There is no piece of wine laboratory equipment that can rate the intensity of “fruit” aroma. A GC-mass spec can measure peaks for various esters, but it will not integrate that information into a general “fruit” aroma like a human is able to do when properly trained and calibrated. Since human beings are variable, the data must be analyzed using analysis of variance techniques to show that results obtained were not due to chance alone. Trained descriptive analysis panelists are not asked their preferences nor are they asked to rate or score the quality of the wine (or other product). One doesn’t ask a gas chromatograph if it liked the wine it analyzed.

Leave a Reply

*

Recent Comments

Recent Posts

Categories

Archives