subscribe: Posts | Comments      Facebook      Email Steve

The last word on single-blind, double-blind and open tasting

18 comments

 

I don’t know that I’ve ever fully laid out, in this blog, my views on the three most common forms of wine tasting: single-blind, double-blind and open. So let me do so.

I’ve long argued that no one way of tasting is “right.” Each has its pluses and minuses. If there were one “correct” way we all would have accepted it by now, so the fact that we haven’t suggests there is no one correct way.

Single-blind tasting, in which you know something about the wines (maybe their region, variety and vintage) is useful, in that it’s blind enough to prevent bias, but gives you some context in which to make evaluations. This question of “context” has been woefully underreported by the wine and academic media, in my judgment. Why context matters is difficult to prove to those who think it doesn’t; it’s an almost political point of view. But suffice it to say that context gives the taster at least some parameters within which to base his conclusions. The following metaphor is a little stretched, I’ll admit, but gets the point across. Let’s say a juvenile is on trial in a courtroom for some youthful infraction. The D.A. wants to throw the book at the kid, but the judge permits his parents and teachers to testify. They offer a view of the kid that’s far different from the one the D.A. presented. The judge, after taking all these views into account, adjusts his sentence accordingly. That is “context.”

Double-blind tasting by contrast allows for no context (except, obviously, knowing the color and stillness or bubbliness of the wine, which may be sweet or dry). To extend the above metaphor, the judge allows no “character witnesses.” He simply bases his conclusions on the law/s the defendant is alleged to have broken, and makes his sentence based on prescribed punishments. This is “unfair” in that it fails to take anything into account other than the actual act of lawbreaking; but one can argue (and extreme law-and-order societies do so argue) that it is after all the fairest, most objective and consistent and least emotional approach to jurisprudence.

Open tasting must break with the metaphor and use a different rationale. In open tasting, you know exactly what you’re tasting. The argument in favor of it is that it’s only fair to know all the facts before making a judgment. This is why so many proprietors insist on critics visiting the winery and tasting with the winemaker; they will not send wine to the critic to taste anonymously in some lineup. This approach, too, makes sense if you view it through the lens of another metaphor. Let’s say your child wants to marry someone. You, the parent, feel you should have some say in the matter: your child’s protestations of love for the fiancé are not enough to convince you that this is a marriage that will succeed. So you insist on meeting the lover and knowing more about him or her: About the parents—the financial situation—the prospects for making a living—the person’s moral fiber—his or her commitment to your child. Surely this is not an unreasonable approach for a parent: He wants to experience the lover “openly.”

How a person tastes wine depends on his reason for doing so. A negociant or a winemaker assembling a final blend may decide to taste double-blind: the sole purpose is to produce the best wine possible. A wine critic may choose to taste single-blind, as I did at Wine Enthusiast (and as I believe many other major critics do). But a wine critic also may choose to taste openly, as I know for a fact some of the world’s most famous English-speaking critics do. They will argue (and who am I to disagree?) that they are perfectly capable of disregarding their knowledge of the wine so that they arrive at an objective conclusion. One can even suggest that open tasting is simply an extreme case of single-blind tasting: After all, if you already know something about what the wine is, then how much “worse” can it be if you know still more?

In the end, we can conclude one thing with certainty: The critic who tastes openly will be a lot more consistent in her reviews that the critic who tastes double-blind. If you value consistency in critics (and the winemakers I know say it’s the single most important thing they look for), then you probably want your critic to taste openly. Finally, I’ll just say that this discussion involves a lot of inside-baseball stuff: All this may be controversial within the critical/winemaking community, but the general public doesn’t give a hoot how their critics taste. They can’t be bothered with the details: All they want to know about is the review and score.

  1. Dan Fishman says:

    Hi Steve,

    Nice summary of the ways a critic could taste. I’m not sure I agree that tasting openly leads to consistency-at least, not to the kind of consistency people want from a critic. I totally agree open tasting would lead to the critic being much more likely to give a wine the same or a very similar score to the one they gave that wine before, or to previous vintages of that wine. However, I think what most of us mean by consistency is that when a critic tastes a wine multiple times OR when they taste wines that are very similar in taste and quality, they respond to them the same way. To me it seems like the only way to evaluate that is to taste blind (single or double would work about as well, although double would be more “pure”), and then see how consistent a critic’s scores are for the same wine.

  2. “In open tasting, you know exactly what you’re tasting. The argument in favor of it is that it’s only fair to know all the facts before making a judgment….
    But a wine critic also may choose to taste openly, as I know for a fact some of the world’s most famous English-speaking critics do. They will argue (and who am I to disagree?) that they are perfectly capable of disregarding their knowledge of the wine so that they arrive at an objective conclusion. ”

    This seems like a self-defeating argument. “I taste openly so I have more information about the wine, but then I disregard that information to reach an objective conclusion.” So then what’s the point? At most it suggests that open tasting is no worse than blind.

  3. pawineguy says:

    Good explanation Steve. I believe a critic, as in most other arenas, should have some context with which to judge a wine. I do cringe, however, every time I read these articles (from anyone in the wine business) about blind tasting. Our industry completely ignores the true definition of blind, as accepted in every other industry or area of academia.

    Perhaps we need our own terms for the way wine is tasted and abandon single and double blind. (especially double blind, which is egregiously mis-used)

  4. Let’s not kid ourselves, most critics like to agree with the fraternity and love the idea of the 1885 classification that keeps all the stars neatly in a place where they would like them to be.

    When we observe athletes performing there is a much greater variation in placings than one sees in wine scores and classifications of wineries, particularly the latter.

  5. Double-blind tasting can be a little too clinical: Such tasting can only leave out all contextual remarks, such as “the best/worst of all winemakers producing from xyz vineyard this year” or “a new take on the usually fruit-forward style of xyz wine”. But open tasting is suspect because of the seeming lack of objectivity. Here is a possible solution: The critic tastes single blind, writes the notes, and then removes the cover, which can enable some discussion of context. Probably that’s too idealistic given the amount of wines that most critics have to taste.

  6. The beauty of blind tasting is that no classification, no previous performance gets in the way of judgment.

    And, it should be expected that a critic tasting Rioja Crianza or northern Rhone reds or CA Pinots or Aussie Semillon actually knows enough about the category to bring an informed frame of reference to the table. Otherwise, the critic is an amateur. Now, don’t get me wrong. We are all amateurs in something, and just as I would not judge South African Pinotage professionally, and was out of my element when asked to judge Aussie sparkling Shiraz, so too am I more likely to be in my element and thus able to taste blind when the subject is Cabernet Sauvignon or others with which I ought to be, and think I am, very familiar.

    I truly do not want to know which Cabs are in front of me. That will only introduce unwanted, unnecessary bias into the judgments being mad.

  7. It would be interesting to see how scores differ if a wine critic tastes the wine blind (single) and then does an open tasting. This might cause the critic to have palate fatigue. I have stated this before, but I think wine critics need to use a curve. If they are tasting 50 Napa Valley Cabernets, then only a certain % can receive scores of 90+. Great discussion.

  8. Not every critic tastes the same way but if their method provides consistent results I don’t see an issue. A good study over nearly a decade of tastings can be found at the Blackbird Vineyards website where critics who taste openly (including me) have essentially the same evaluation as those who taste blind (including you).

    I learned to evaluate wines as a retail buyer where I tasted fifteen – thirty wines over several hours during appointments with sales reps. It wasn’t uncommon to end one appointment with a Syrah, and five minutes later have a dry Sauvignon Blanc in my glass. Tasting like this helped make me a more disciplined writer. Wines were never bagged and many were from new producers. I still taste open, but the first thing I do after opening a box is take the technical paperwork and throw it in a drawer. Completing the evaluation of all the wine in an issue before knowing production or price removes those potential influences.

  9. Dusty Gillson says:

    The problem I see with single blind, is that is does not prevent bias as stated, it can only minimize it. No matter what things you know about the wines to be tasted (region, variety, vintage), it will inevitably color your opinions to a degree.

  10. Christopher says:

    This is an interesting discussion. How many other critical professions is the reviewer expected to judge completely impartially? It’s certainly not for movie reviewers. A movie critic knows the director (and their body of work), actors (their previous roles), DoP, etc. and is not expected to forget that information when reviewing a movie. A restaurant critic knows the chef’s history (in all likelihood) and previous meals from that establishment, and is not expected to “un-know” that information in their review. Heck, the movie reviewer and restaurant critic often bring up previous encounters with that director/chef in their reviews to compare and contrast with the current experience.

    Is there any other type of critic who is expected to give an objective review without their personal bias? I’m not saying that’s a bad thing, just interesting to me that wine is treated differently.

  11. doug wilder says:

    Christopher, you made an excellent point. Think if a restaurant critic was led blindfolded into a tented room in a restaurant and asked to evaluate the food, chef and ambiance based only on what is before them. or a film critic to know nothing of the director, producer, plot, location, length, budget of a picture.

  12. If it were possible to review movies or restaurants blind, that is how they would be reviewed.

    This is not apples and oranges. It is cars and corn flakes. One can be reviewed blind and the other cannot.

  13. Most winemakers I know taste openly, because the way the wine tastes today will be different in 2 months or 2 years or 10 years, so the context matters.

  14. Richard Fadeley says:

    I do a lot of blind tasting, but what I think we are missing in the analysis of accuracy is that you really can’t tell about the “quality” and durability of a wine until you get into the second glass. I’ve had countless wines that upon first sip, sniff, and swirl were captivating, but after a glass, and/or into the second glass the wine get’s pretty tiresome. With blind tasting you really don’t ge the chance to experience that degree of familiarization and therefore the results can/will be erroneous. IMO you can’t really tell if you have a “keeper” until you get well into you second glass and ask “is there anymore of this wine left?”. Then you know you have a 90 pt. or a 92 pt. wine.

  15. Gabe, you bring up a good point. There is no “one size fits all” for every part of the wine biz. In your job, context includes what the wine did last week, is there anything developing in the wine that calls for winemaker creativity, when will the wine be right for bottling, for release, does it live up to the expectations for the label, for the price, etc.

    Blind tasting helps at some point in the process but it gets in the way when you are looking at a specific wine and its components.

    Similarly, if I am a sommelier [ my kind of sommelier :-} ], I need to taste open a lot of the time because the context of what does my list need in terms of diversity, range, type, price, expectations of the customer base is better answered that way. But, I would also taste blind at times, especially when I feel the need to put some samples to the test.

    Bon Appetit Magazine has a very different set of needs than does The Spectator or Connoisseurs’ Guide. Our responsibilities to our readers are, in my opinion, best met by tasting blind.

    As for Mr. Fadeley’s thoughtful comments, I do agree, and it is why my tasting regimen takes an hour to get through eight wines blind, including tasting and retasting and why we also conclude many of tastings with lunch or dinner and retaste the wines there as well. I would suggest, however, and I apologize if this sounds arrogant or self-serving, that experience critics have developed a wide and varied tasting memory that includes how wines go with food, how they age and what components in wine aid in or detract from the likely enjoyment of a given wine.

  16. Blake Gray says:

    Really the last word? You promise?

  17. Bob Henry says:

    In an ideal world, there is one way: “single blind” tasting.

    The reviewer knows the single grape variety/grape variety blend and the vintage, but not the producer or the appellation/vineyard designate or the suggested retail selling price.

    [Some would argue for not revealing the vintage, but I think that's too severe a restriction. Knowing the grape, the state or country, and the vintage is sufficient "context" for any accomplished wine taster. Much more knowledge than any M.S. or M.W. exam taker is offered.

    As Lettie Teague writes in this weekend's Wall Street Journal "On Wine" column:

    "The practice [of "blind tasting"] purportedly evolved from a professional need to identify fraudulent wine (bottles didn’t come with labels long ago).

    Link: http://online.wsj.com/articles/how-to-blind-taste-wine-like-a-sommelier-1408118884

    But in the real world, no winery is going to pour you a complete vertical of their expensive, small production showcase wine. (For that experience, buy a seat at a Dr. Bipin Desai tasting weekend in Los Angeles.)

    At my organized tastings in Los Angeles . . .

    http://www.kirktech.com/bob_henry/

    . . . I insisted on “single blind” tasting: for each discrete “flight” and throughout the entire tasting event. All “flight” and “overall top three preference” wine votes were recorded before unbagging the bottles and encouraging “table talk.”

    (It is always a welcomed discovery when a bottle of California Cabernet from a little known producer stood shoulder-to-shoulder in appeal with one or more of the “cult” Cabernet brands. And doubly so when that little known producer’s wine costs less than the nearly 10% California sales tax on a bottle of Screaming Eagle.)

    Responding to the statement . . .

    “a wine critic also may choose to taste openly, as I know for a fact some of the world’s most famous English-speaking critics do. They will argue (and who am I to disagree?) that they are perfectly capable of disregarding their knowledge of the wine so that they arrive at an objective conclusion.”

    . . . I, for one, disagree. Backed by social science research [*] that attests to “price bias” and “label bias” in respondents.

    Regarding this statement . . .

    “The critic who tastes openly will be a lot more consistent in her reviews that the critic who tastes double-blind.”

    . . . too true — with this editing change: “a lot more CONSISTENTLY BIASED in her reviews . . .”

    And this statement . . .

    “If you value consistency in critics (and the winemakers I know say it’s the single most important thing they look for), then you probably want your critic to taste openly.”

    . . . my editing change: “then you probably want your critic to taste METHODICALLY CONSISTENTLY. AND THAT MEANS SINGLE BLIND.”

    If you don’t comprehend the intrinsic validity of this approach, then you don’t comprehend the “scientific method” (using “controls” and “variables” and “statistical sampling”) that has produced the great medical and technological advantages of our Western society.

    ~~ Bob

    Postscript. Steve, you have failed to cite another pertinent metaphor: “arranged” political marriages, when future spouses meet for the first time on their wedding day. That’s truly “double blind”!

    [*] Bibliography:

    From CNET News
    (January 14, 2008)

    “Study: $90 Wine Tastes Better Than the Same Wine at $10;
    CalTech and Stanford researchers have seen direct evidence that increasing the price of the same wine significantly increases the pleasure we get from it.”

    Link: http://www.news.com/8301-13580_3-9849949-39.html

    Posted by Stephen Shankland

    – AND –

    From The New York Times
    (January 5, 2008):

    “Raising a Wine’s Price Improves Its Taste, Study Suggests”

    Link: http://www.nytimes.com/2008/01/15/world/americas/15iht-wine.1.9221093.html?_r=0&pagewanted=print

    (Byline unknown)
    Associated Press Science Writer

  18. Bob Henry says:

    Another comment awaiting “moderation.”

    Erratum.

    If you don’t comprehend the intrinsic validity of this approach, then you don’t comprehend the “scientific method” (using “controls” and “variables” and “statistical sampling”) that has produced the great medical and technological ADVANCES of our Western society.

Leave a Reply

*

Recent Comments

Recent Posts

Categories

Archives