subscribe: Posts | Comments      Facebook      Email Steve

Talkin’ 100-point blues

23 comments

READERS: I’m still buried in meetings at Wine Enthusiast for our annual winter conference. Please enjoy this post, originally published in July, 2008.

* * *

There’s been a lot of chatter out there lately about the 100 point system. (Even my colleague at Wine Enthusiast, Paul Gregutt, has written skeptically about it.) While I might have thought this was a bit of a dead horse, the issue does shed light on, not just how some of us rate wine, but how we think about wine.

As a person of interest at Ground Zero of the 100-point scoring system, I’d like to offer my thoughts. What I meant by the debate shedding light on how we think about wine is this: Wine is something that people rank (consciously or not) on a qualitative basis. Other things we rank are films, automobiles and politicians. Things we don’t tend to rank are those we take for granted. Probably no one ranks paper clips.

We know all wine isn’t the same and even if we’re not wine drinkers we’re aware that some wines are better and more expensive than others. Once you get into the ranking game, you’re opening the door for experts to come in and decide what’s best, what’s better, and what’s not so good.

So the concept of wine critiquing works for me. As to how it’s done, it’s important to keep in mind that people want visual symbols to reference, not just text. A few years ago, the San Francisco Chronicle stopped using visual symbols in their wine reviews and went to text only. Readers revolted, and the paper had to restore the icons.

I guess there’s fundamentally no difference between a numerical score and puffs, stars, glasses or any other symbol, and so I can’t make an argument on logical grounds that the 100-point system is inherently better. I can only say why it works for me at Wine Enthusiast.

To begin with, it’s not really a 100-point system, it’s a 20-point system. We only publish wines with a rating of 80 or above. Everything else is given a code number, “22,” and consigned to the database’s bowels, where the public will never see them.

Since I work with a 20-point system, not a 100-point one, I don’t have to defend the extraordinary practice of giving a 67 to something instead of a 66 or a 68 or for that matter a 71. How you can slice the baloney that thin is a mystery to me and a little spurious.

So what’s the difference, you ask, between 82 and 83, or 91 or 92? It’s something you feel in your bones, head and heart. The bones are your first instinct. The head is your considered opinion based on further tasting and reflection, and the heart is when you’re sure you’re right and have nothing to be ashamed or afraid of, but can hold your head high and say, “This is what I believe.”

All this raises profound questions, which may be summed up by Alder Yarrow’s query at his blog, a few days ago: When it comes to wine critics, “whose perceptions and emotions do we trust?”

I’m not sure that this period of the public’s reliance on critics will be seen kindly by future generations (assuming there are any). We may one day be viewed as the equivalent of soothsayers or snake charmers or seers who read the entrails of beasts. But for now, wine critics are a vital part of the industry, along with the 100-point system. As for the who-do-you-trust part, I’ll leave that for others to decide.

  1. Since the way we feel varies from day to day and hour to hour, you ought to be following some predetermined criteria for how you award points. That 89 point feeling “in your bones” may actually be only arthritis.

  2. Dear He Said, I do have criteria, and you’re right to point out there should be criteria in the critical process. I’ve written before about what my criteria are, and will do so again — but I try to keep my posts to 300 or 400 words and to have added criteria would have pushed it way over. Thanks for your observation.

  3. Steve, Just a quick note to say thank you for publishing this blog. Excellent work. Keep it up. I found you via a recommendation on Dr. Vino in case you are interested.

  4. Can you offer a link to where these criteria are posted?

  5. Morton Leslie says:

    If it is a 20 point system why pretend it is a 100 point system? As you describe your use of points in scoring wines you are describing a hedonistic scale of 20 points, that there are 20 different levels of the degree to which you like or dislike a wine. The most I can imagine on a hedonistic basis is an eight point scale.
    I hate it
    I strongly dislike it
    I am not wild about it
    comme ceci, comme cela
    I’m slightly drawn to it
    I like it
    I love it
    I just wet my pants
    I would be interested to see how you can get 12 more levels of appreciation.
    Using points, however, implies a completely different method of scoring that is objective and not hedonistic. It implies that there are specific properties a wine has or lacks to which in sensory analysis you objectively give or take away points. It implies an objective regimen that is followed which is reproducible from one sitting to another because the points are assigned for specific things a wine either has or lacks. Unfortunately there are not 100 aspects of the wine that are judged and points objectively assigned for the presence or absence of each one, nor is there any attempt to do so. No one who uses a 100 point system will tell us objectively how the system works, how they use it, and how others could learn to use it. This is because it is a journalistic invention leaning on their reader’s high school experience with the scoring of exams which conveys to the reader the idea that something was objectively graded.

    Wine educators are valuable in their contributions. I am not sure critics who mislead the public with a pretend scoring system, rather than educate them about wine, are that vital to the industry. To an individual winery a score can be important, to the overall industry it makes no contribution.

  6. He Said, in Wine Enthusiast’s Buying Guide, you’ll see a fairly detailed explanation on the tear-out card that’s in the Buying Guide. It’s not online, yet, but it should be, and I’ll be taking care of that shortly. Thanks for the request. I’ll be commenting more on this on my blog.

  7. Morton, the real question is how many calibrations can fit on the head of a pin. I’m comfortable with 20 — you’re fine with 8 — others are fine with 5, or 50. I don’t want this to turn into a Talmudic debate. Don’t forget that I am employed by a magazine that uses the 100-point [or 20-point] system, and I am therefore bound to use it in my reviews. But I couldn’t say that, if I owned Wine Enthusiast, I would not keep the 100-point system. Despite its flaws (and EVERY system has flaws), it has proven at least one thing: Its usefulness. The readers like and understand it, and as a writer and critic, my readers are my #1 obligation to take care of. Now, if anyone would like to know why I scored a particular wine XX points, I’d be happy to explain my mental process.

  8. Morton,

    Nice to see your input.

    I actually have a set of fixed criteria for scoring. It is an imprefect system that ultimately is not the 100 point sytstm it appears. Thus I am going to be abandoning it when I launch the new redwinebuzz.com platform.

    If you are curious, you can see it here: http://www.redwinebuzz.com/wrs.htm

  9. Arthur, I’ve been thinking of the commotion that’s always stirred up with every discussion of this topic, and it occurred to me that wine criticism is like government: Every country has a slightly different way of governance. Even democracies can be quite unalike: Britain, the U.S., Israel and Russia are all democracies (or such is my understanding of the term) but in Britain or Israel a P.M. can fall while in the U.S. or Russia a President cannot (unless impeached). Not to wander too far afield, but each critic will always have his/her different approach, and ultimately I don’t think it’s possible to say which one is the “best.” All that we’ll be able to do is to determine which critics are the most popular — but heck, we can do that now.

  10. My colleague Lauren at Wine Enthusiast did a little digging and found the criteria we used to post online, which I’ve reproduced below; somehow, it was inadvertantly dropped during numerous site redesigns. We’ll be updating it shortly. Again, thanks to He Said for bringing this to my attention.

    WINE ENTHUSIAST SCORING CRITERIA

    Approximately 500 wine reviews are included each month in the Buying Guide. Each review contains a score on the 100-point scale, the full name of the wine, its suggested national retail price, and a tasting note. If price cannot be confirmed, $NA (not available) will be printed. Prices are for 750 ml bottles unless otherwise indicated.

    Tasting Methodology and Goals
    Tastings are conducted individually or in a group setting and performed blind or in accordance with accepted industry practices. Price is not a factor in assigning scores to wines. When possible, wines considered flawed or uncustomary are retasted. In some instances production figures are included as an aid to consumers.

    About the Scores
    Ratings reflect what our editors felt about the overall quality of a particular wine. Beyond the rating, we encourage you to read the accompanying tasting note to learn about a wine’s special characteristics.

    * Classic 98-100: The pinnacle of quality.
    * Superb 94-97: A great achievement.
    * Excellent 90-93: Highly recommended.
    * Very Good 87-89: Often good value; well recommended.
    * Good 83-86: Suitable for everyday consumption; often good value.
    * Acceptable 80-82: Can be employed in casual, less-critical circumstances.

    Wines receiving a rating below 80 are not reviewed.

    Special Qualifiers
    Editor’s Choice wines are those that offer excellent quality at a price above our Best Buy range, or a wine at any price with unique qualities that merit special attention.
    Cellar Selections are wines deemed highly collectible and/or requiring time in a wine cellar to reach their maximum potential. A Cellar Selection designation does not mean that a wine must be stored to be enjoyed, but that cellaring will probably result in a more enjoyable bottle. In general, an optimum time for cellaring will be indicated.
    Best Buys are wines that offer a high level of quality in relation to price. There are no specific guidelines or formulae for determining Best Buys, but they are generally priced below $15.

  11. Steve,

    How are the points accumulated?

    I created a rating sheet that I use for every wine. It follows set criteria and is a (loose?) variation of the UC Davis rating system.

    My experience has been that if I follow my rating sheet and give points for each category (color, aroma, flavor, finish, acids, tannins, age worthiness, alcohol balance, etc) as indicted by my scale, two very different wines can end up with the same total score.

    For that reason, I started to de-emphasize the summation of points from all categories. I publish the point breakdown (by category) so readers can reference that if they choose.

    My feeling is that, this way, they can apply their preferences to what I report about the individual wine and decide if it will suit their tastes and if they want to buy it.

  12. Arthur, I keep a copy of Wine Enthusiast’s criteria (which I reproduced above) on my desk. It is my Bible and determines the point score. If I can summarize my mental process, it’s roughly: Is the wine classic, superb, excellent, very good, good, acceptable, a or unacceptable? I suppose this actually constitutes a 7-point system, but we then drill a little further down. If it’s “excellent” (90-93 points), then how excellent, within those parameters? I realize someone may object to degrees of excellence, or classicism, or acceptability, but there it is. It’s our system and, like I said previously, it’s not perfect, but — as several people have told me, privately — no system is perfect. And I haven’t found a system that (a) works better, (b) is fairer and (c) is preferred by the vast majority of readers who care about wine reviewing.

  13. Not to be pedantic here, Steve, but what you posted is not a list of criteria for tabulating scores, it is a breakdown of total score ranges that you and the W.E. feel those ranges mean in terms of quality.

    My question (and I think that of earlier commenters) addresses the process by which points are accumulated and which precludes the dilemma of how excellent a wine is.

    Stated another way:

    “On the basis of which objective criteria resulting from the sensory assessment of the wine did you arrive at 93 points?”

    It would seem from what you are saying here that you taste a line up of wines and rank them based on your hedonic response rather than following a pre-set scoring paradigm that assigns a set of points to each wines based on the presence or absence of particular traits and attributes.

    I think that is what Morton is driving at: We should 1) strive to inform and educate our readers and 2) describe wines (and rate them) in a way that reflects the wine and not our individual enjoyment of it.

    That would require a common benchmark of quality. It exists. The only reason that people say that “it can’t be agreed on” is that very few in this industry have the minerals to come out and say that some wines are crappy and some are not and the distinction has nothing to do with individual enjoyment and preference but rather with absolute and objective standards of wine quality. Making these assertions is un-P.C. and people avoid them because they’re afraid it might cost them business relationships and readership and subscriptions…

  14. Arthur, I appreciate your tenacity. No one will ever be able to create a Manual for Scoring Wines, a sort of Municipal Code for determining a number. Oh, I don’t mean that no one will write one. Someone (you?) will, and maybe you’ll use it, but that doesn’t mean anyone else will use it, or trust it any more than they trust any other scoring mechanism. There is something subjective about scoring wine, and you’ll never be able to eliminate it by creating complex grids of attributes that you tell yourself are “objective.” If you really want a completely objective analysis of a wine, send it to one of those labs that tear it apart and give you a technical breakdown of everything about it. That’s “objective” but it’s not wine reviewing.

  15. Christopher O'Gorman says:

    Whether a critic is using the 20 point system, 100 points or stars/puffs, it is fairly easy to extrapolate into ones own preferences based on your palate similarites with reviewer.

    I judge at a lot of competitions, and when evaluating wine you just “know” right away whether some thing is a Bronze or a Gold, etc. and that realization usually comes very quickly. Or as Steve comments above, you just feel it in your bones.

  16. Actually, Steve, when one comes to understand what the findings achieved through organoleptic assessment indicate about the wine at hand, it becomes clear that wine reviewing is either an amateur’s account of a yum-yuck response or a learned assessment of the wine. The former is subjective. The latter relies on reporting the wine’s own characteristics and is far more objective.

    Because a wine’s traits and attributes fairly predictably result from vintage, weather, farming and vinification practices, and they also are very good indicators of age worthiness and food pairing, etc, this is the better and fairer way to *rate* wines. It follows time-proven track records of wines.

    One may enjoy a 100% varietal Napa cabernet sauvignon (or Sta. Rita Hills pinot noir) that tastes of plums, clocks in at 15.8%, reeks of new oak, lacks acids and has negligible tannins two years out of the vineyard, but that does not change the fact that it is a crappy wine. Enjoyment is the subjective part here. The belief that plum and vanilla represent regional or varietal typicity in Napa cabernet sauvignon is ignorance.

    The reader/consumer may not need any intricate process but rather a simple bottom line. That does not excuse us from following a process and arriving at a more intelligent recommendation which are focused on the wine and not our enjoyment of them. This may seem too “Six-Sigma” for some, but you have to substantiate your score or recommendation with something more than subjectivity.

    Because we tend to respond emotionally to aromas and flavors, pre-set criteria are important. If you go through your process and lay out in front of yourself the merits of the wine, then the final summary can only be based on the information you have collected. That, of course, depends on how honest one is willing o be with themselves and how willing they are to try to separate their preferences from the merits of the wine.

  17. Christopher,

    How do you communicate that feeling in the bones to your readers?
    How do you substantiate the bronze, silver or gold to them?

    You are either making tha decision based on a hedonic response or you follow criteria which reflect standards of quality wine.

  18. Morton Leslie says:

    If we took the last 100 wines that you tasted and scored, Steve and had you re-taste them next week, double blind, my guess is that on most of the wines you would not give them the same score. My guess is that in evaluating the standard deviation or score for the two judging we would find that it was somewhere between two and five points. My guess is that on a sizeable number of samples you would not be able to tell the grape variety. This would mean that you are actually tasting to a 4 to 10 point, not a hundred point scale. This is why it is not objective. You would not be alone. Anyone tasting and using such a hedonic system with arbitrary numbers would be in the same boat. Anyone tasting a mixture of 100 double blind wines would be doing good to get half of the varieties correct.

    If you sensory skills are such that you cannot identify grape varieties at least 95% of the time, what meaning does a 95 score really have?

    I have tasted with Robert Parker on a half dozen occasions. He spends somewhere between 15 seconds to 30 seconds on a wine. Jots down maybe a half dozen words on a pad and ….later after leaving the winery, maybe later that day, assigns points. My guess Steve, you do pretty much the same, except you taste blind and assign the points while the wine is in front of you. That is better, but it is not sensory analysis and the points you assign are meaningless aside from if it is in a particular range it means a wine is of one of six levels of quality.

    What you are giving your readers would be similar to your car dealer telling you bring the car in for service and they will give it a “100 point inspection” (implying that they will look at separate 100 maintenance issues.) The mechanic sits down in the car. Starts the engine. Listens to it. And concludes his hundred point inspection. The honest thing would be for him to tell the customer, “We start the engine and we listen to it. That’s all.” Problem is. You don’t need the mechanic to do this. You can do it yourself.

  19. Steve, this topic is hardly a dead horse. It’s more like a Trojan Horse, and people who continue to treat the 100-point scale like currency are perpetuating a whole culture of distraction.

    THe original idea of the 100-point scale was to provide a more accessible means of “guidance” for consumers. Back in the day, the universe of wines in the marketplace was smaller and quality was way more variable, so the guidance really needed to steer wine drinkers to “good” wines, which tended to be 80 points and up.

    But two phenomena have completely changed the landscape of wines and critics’ roles. In today’s market, not only is overall quality better but also there are just way too many wines for people (critics and consumers alike) to keep track of. In my role as a public speaker/educator/entertainer, I find that the guidance needed today is more about style, not quality, and ratings (even when accompanied by tasting notes) distract from that mission. Moreover, as we all know, functionally, the 100-point scale has devolved — in large part because so many WS and RP copycats have risen — into a two-point scale: a wine is either 90 or it’s not. At the same time, retailers and marketeres have become adept at making this sense event more ingrained by cherry-picking the scores they promote.

    THe system is broke. Buying Guides are a joke. Real people are paying less and less attention. I think we are approaching a tipping point; and I think within a few years the blogosphere will paly a huge role in developing a new way that people think about the whole process of sharing wine recommendations/opinions

  20. As I written on my own blog, everyone’s got a palate and knows what he or she truly enjoys.

    Evaluations by others – if we find a palate that’s similar to our own likes (or dislikes) is a helpful guide to what we might also enjoy… IN psychology it’s referred to as “membership.”

    Ratings won’t go away, though, until we stop evaluating, which we’ll never do. We’re taught as soon as we enter school that we’re working toward the report card. People compare and contrast themselves to others constantly… It’s human nature, and third party endorsements really help in support of wine sales.

    So, Steve, keep up your excellent work… however you do it it matters not. That you “do it,” is what truly matters.

    Also, that you are able to make a living from it is what differentiates you from others. I’ll give you a score of 95. (I don’t believe perfection exists in our world; but you’re darn close, as compared to others who haven’t yet found a way to have an opinion that helps to pay the rent.)

  21. Morton Leslie says:

    The reason a few of us get so exercised about critics is not that they judge our wines, it’s how little respect they give us when they do. We spend a year tending a vineyard, taking on the elements, sweating in the sun, shivering in the rain, fighting insects, mildew, birds, and deer, doing everything we can to bring in quality fruit. We harvest, lose sleep, ferment, worry, rack, fix pumps, repair forklifts, taste, ponder, rack, taste, assemble, reject, sell off bulk, blend, finish, sanitize, fiddle with labelers, and bottle. We literally put a year of our lives into the bottle.

    The critic lines up our wine with a bunch of others, makes no attempt to understand the wine, has no idea why it is the way it is, makes a brief judgement, jots a couple notes, assigns a score that has no meaning, and is done with it. This is not respectful to our wine. At best it is superficial, at worst, it is fraud.

    This is not really much different than Trump’s beauty contest and the lineup of women with the perfect teeth, posture, proportions, and aspirations for world peace, in front of a few celebrity judges. With superficial measures of beauty the contestants become exaggerated in stature, yet strangely they are sort of the same. Soulless. In the real world of women, if you go beyond the surface, take some time to get to know them, learn about their childhood, their interests, their aspirations, their values, their accomplishments, and their sense of humor it is easy to find something to love in each one. There is soul. It’s then that you see the superficiality of rating them and the importance of treating each with respect.

    This is why an increasing number of top wineries no longer send their wines to critics. What is interesting is when you tell the publication that you will not be submitting your wine, their response is, “If you don’t submit them we will go out and buy them. But do you really want us reviewing your wines under those circumstances?” … Revealing… Very revealing.

  22. Steve,

    What a can of worms you opened! I’ve been rallying around the whole “death to ratings” now for sometime because the biggest flaw I’ve discovered in the 100-point system is the fixation on BODY. It seems that the system is bent toward fuller-bodied wines, and wines meant to be light and delicate haven’t a chance in hell of anything above 90. Aside from being a retail buyer, I also have the opportunity to judge various wine festivals, and the criteria for them is typical of the 100-point system. It certainly misleads the public and I have had to do all in my power to educate my customers of the deceptiveness inherent in the system. It cannot be the be-all-end-all for decision making in purchasing wines, but I try to get my customers to use it as one piece of the determining puzzle. I also tell my customers of the inherent biases one can see in all reviewer’s writeups and scores, and how that too can help them determine if a particular wine is right for them. Arthur made a great point in abandoning the 100-point system, and spells out specific criteria to use when reviewing wines. If anyone needed a rulebook for reviewing, I think Arthur has it all wrapped up. And Tish I think is onto something when she points to blogs as being the new vanguard of wine reviews – the days of paying for information are numbered when the Internet continues to expand on its wealth of knowledge, free knowledge, offered up to the masses instantaneously.

  23. I want to thank all the people who took the time to express their views. I’m leaving this topic for now, but want to make 2 small points. First, to Morton Leslie — a great correspondant. He said: “The reason a few of us get so exercised about critics is not that they judge our wines, it’s how little respect they give us when they do.” This charge hurts me very much. Morton, I agonize over giving low scores to hard-working winemakers. I fully understand that people have mortgages, bank loans, college tuition bills, etc. to pay. Ever since I’ve been in this business I’ve tried to empathize with winemakers and other winery workers. I always worry that I don’t know enough about the technical end, even after all these years of having it explained to me and reading about it. But I have a job (same as you). Yours is making wine. Mine is writing about it. That’s what I get paid to do, and I try to do it with integrity and professionalism. I’ve said it before, and I’ll say it again: Someday Americans won’t need wine critics; they’ll make up their own minds. But that day hasn’t yet arrived.

    Finally, K2, Tish is a he, not a she.

    Bye!

Leave a Reply

*

Recent Comments

Recent Posts

Categories

Archives