More analysis of the 100 point scoring system
I sometimes come under fire from commenters on this blog when I say stuff like “Bottles vary in temperament, the human body varies in its receptivity to aromas and tastes on a daily basis, etc. etc. This is why it’s important to remind readers that a score is a photograph of the critic’s reaction at a particular moment in time.” That’s from my blog a couple days ago, when I owned up to the truth that “obviously a point score is not meant to be taken as a mathematical certainty.” That wasn’t the first time I’ve said so, but whenever I do, the point-score doubters pile on. Gregg Burke said my confession was “Kind of a problem wouldn’t you say?” David Honig said “if a wine can vary from 87 to 90, depending upon your mood, the temperature, what you had for breakfast, whether your dog threw up in the car, and whether you drink it before or after listening to Michelle Bachmann argue that the HPV vaccine causes retardation in young girls, then it is flawed.” Et cetera et cetera.
So, this being an extremely important issue, I want to explore it further. I am aware that some people will say “Steve is just writing about the 100 point system to increase his readership” but that isn’t true. My regular readers know that I write about whatever I’m thinking about that day, without regard to eyeballs.
I want to start by quoting Daniel Baron. He had been the winemaker at Dominus, and was at Silver Oak when he was interviewed in the wonderful 2004 book, The Winemaker’s Dance. (I haven’t researched where Daniel is now, but it doesn’t matter from the point of view of his quote.)
Daniel said: “You have to remember this when you think about judging wines. They’re alive and changing moment to moment; they have good days and bad; they show well in a particular glass or with particular food. Judging a wine at any particular moment in life is like giving a kid a letter grade based on his behavior in the supermarket.”
I’m not sure about that analogy concerning the kid in the supermarket. but other than that, Daniel’s message couldn’t be clearer. Read it again. It speaks for itself. This is the viewpoint of a professional winemaker, working at a top winery. He’s not trying to defend the 100 point system, or point scores in general, and he’s certainly not offering an apologia for critics. He’s just speaking a truth that, apparently, some critics of the 100 point system fail to understand.
Bottles do vary. Anybody who’s been in this business for more than two minutes knows that. There is an infinitude of reasons why bottles or, more exactly, bottle impressions vary, in addition to the ones Daniel listed, which I need not explain now, although I can if you want. This variation is why it is unlikely if not impossible that any particular review can stand the test of time, the way an empirical finding, such as a mathematical or chemical result, can.
So what is the conusmer’s touchstone of reality? I repeat what I said in my blog: “The more reliable the critic, the more trustworthy you may assume his reaction to be.” The consumer should not assume that any critic in the world–whether his last name is Parker or he has the letters M.W. following his name–is mathematically precise, or that his review is replicable across all times, spaces and conditions. That is utter nonsense. Instead, the consumer should react this way: “Okay. So-and-so rates such-and-such wine at [whatever] score, and then explains his score in the text. I trust so-and-so; therefore, if he felt that way when he tasted the wine, there’s a reasonable certainty that I will feel the same way when I taste it.”
That is not mathematical certainty. It’s not the same as saying, “Okay, my math teacher demonstrated on the blackboard that two plus two equals four. So when I add them, there’s a reasonable certainty I’ll arrive at the same conclusion.” No. Your math teacher is absolutely certain of his results, and he knows that he will arrive at the same result every time he does the calculation, for the rest of his life, and so will everyone else when they add the same numbers. Wine reviewing is not and cannot be like that. It is soft objectivity. A score, properly speaking, should be considered as guidance. And as with any guidance, it should be taken in a certain context.
I would be curious to know what percentage of the detractors of the 100pt system are in the wine business vs consumers. A lot of detractors I read sound an awful lot like sour grapes. As if they are embittered and feel slighted that their particular wines did not live up to their expectations when submitted for outside criticism. You have called their baby ugly. I hear arguments for a more simplified system (recommend or not, thumbs up or down, etc). But that would just make it is easier to appear to play on the same field as those making truly remarkable wine. There is not enough demarcation between a good wine and an out-of-this world wine. Sure that would help those in the supply chain dump middling wine on an unsuspecting public, but how does that help the consumer?
Anthony, I agree with you. The detractors tend to be professionals in the business, who seem to have difficulty separating their personal antagonisms from their judgment.
“a score is a photograph of the critic’s reaction at a particular moment in time.”
A photographer happens the catch a picture of some celebrity tackling a little old lady. That’s what he saw, it happened, so he runs it all over the press. “SO AND SO TACKLES LITTLE OLD LADY!!”
The “picture” was a snap shot of a particular moment in time. Well, because of the photographer “angle” that say, he couldn’t see that celebrity was actually pushing the little old lady out of the path of a quick moving car.
The point, the photographer saw what he saw, true, but there were some big gaps missing and once he published the picture, the damage is done to the reputation to the celebrity.
I don’t mind notes and descriptors with some commentary, but while an 89 vs. 91 might not really mean a huge difference between two $20 Syrahs, it can mean all the difference to consumers who lack the gumption to make up their own minds. When you factor in that the 89, could have been a 91/92 at some other “point in time,” well there is my problem with it. This is how people provide for their families after all and at the end of the day the critic gets paid all the same.
Regarding a non point based system: “There is not enough demarcation between a good wine and an out-of-this world wine.” Anthony, yes there is. It would look like this, following a description of the wine and a “highly recommended” type thing:
“This is a wine that should not be missed by anyone!” – Sir Critic
I am not saying we need to get rid of scores, but to say it is the only way to really identify an outstanding wine is false.
P.S. I don’t work in the industry and my friends who do score rather well. No sour grapes.
Wayne: a very interesting take on the “relativistic” simultaneity of what the photographer saw, as opposed to what another observer might have seen. But you make my point for me: it’s another reason why the consumer needs to take any critical review as a singularity — a particular space/time impression that describes only itself.
At last, an original topic!
What’s funny about all this is that almost no one outside the wine business, that is, the average consumer, spends any time worrying about or analyzing the 100 point scale. They’ve heard all the arguments for and against it, and they just don’t care. They use it as a reference because it works for them, or they ignore it because it’s meaningless to them. But it is the gold standard for navel-gazing in the industry.
And, in truth, the industry, wineries and lazy retailers and marketing people, made the 100 point scale successful, not Robert Parker. Even by endlessly talking about it we give it added importance. It’s not going away, too much advertising revenue, revenue that pays STEVE!’s salary, and, at times, mine, depends upon those scores. The 100 point scale, like so much else in life, is all about the almighty dollar–not integrity, not accuracy, not objectivity, not morality. It sells magazines and it sells wine. It isn’t required to be anything but financially viable, and it has proven to be that, more than Puffs and Davis scales and Badges.
I’d say we should stop beating this dead horse, but then blogs would only be about bloggers and how much they’ve changed the wine business. And that’s a far drearier subject.
Ron, you’re too tough on this salary thing. There’s nothing wrong with earning a living. We all do it. You too! And when it comes to their jobs, every worker buys into the system they work for. If they didn’t, they wouldn’t last long. So the harshness of your critique is unwarranted.
Try this. Have the same people taste the same 5 wines (double blind) on two successive days – or the same day with an activity lasting at least two hours w
between the tastings. Have no music in the background during the activity – play cards, walk, tour a village – whatever.
During one of the tastings play background music with some beat – something modern.
Wanna make people like a wine, play soft music – not jazz, rock, hip hop, etc.
AGS
At the other tasting play soft, gentle classical music – or even romantic piano songs (NOT the 1812 Overture).
The resulting scores on the wineswill vary significantly – one tasting to the other. Depend on it.
I don’t think I was being harsh, Steve, maybe cynical, but not that harsh.
If the 100 point scale didn’t resonate with consumers, didn’t make them buy wine and wine magazines, would it still be with us? I remember the early days of the Wine Spectator, when it was in a newspaper format, and they didn’t have scores, only descriptions and recommendations. This is what so many 100 point bashers long for, it seems. Problem is, it didn’t sell magazines and it didn’t sell wine and it didn’t sell advertising. And it made for incredibly boring reading. When they began emulating Parker’s success, they began to make money and get a lot more advertising dollars. Would they be so successful without the wine ratings? I think the answer to that is simple.
I’ve often made fun of the 100 point scale, but I don’t think it needs any defense. Nielsen ratings for TV are equally stupid, but they ruled TV for decades, still do, and those are simply, and only, about advertising money. Look at their methods closely, and it’s a joke. But the industry buys into it, so it remains important. This doesn’t make the people who use Nielsen ratings or wine scores frauds or disingenuous. I would never question your integrity, or Charlie’s, or Paul Gregutt’s (except in a satiric mode). You use the 100 point scale as you are employed to, and you use it as fairly and as incisively as that flawed system allows. No need to defend that. I don’t mean stop explaining how you use it, but defending it is a waste of time. It doesn’t need defending.
Humans are variable – it is inherent in the nature of being human. You are not the same day to day and you can vary throughout the day. Sensory scores can be analyzed using analysis of variance to indicate the degree of variation. Indeed, a score of 87 may not be STATISTICALLY SIGNIFICANT from a score of 90, yet we all know which score will sell more wine.
The glory of this blog is its wide ranging scope and the fact that the admin allows any sort of comment within reason, even when it is critical of the lone critic issuing pronouncements from the mountain top.
Our own “Wisdom of the Crowd” http://www.en.wikipedia.org/wiki/The_Wisdom_of_Crowds tasting methodology starts from two premises: more than one opinion should be included, as in tasting panels (after all Messrs Bonné and Asimov, plus many other newspaper critics, rely on a tasting panel). Having 15+ different scores tends to reduce if not eliminate statistical variation, and allows the wine with the widest appeal among different palates to emerge and thereby have a higher likelihood of appealing to more prospective buyers. And secondly, that the consumer or wine enthusiast should be the preferred “guidance counselor” since s/he buys their own wines (with social media this is happening in every domain of human activity today). This presumably is why Google bought Zagat.
There is a third reason in the context of this thread–the revered 100 pt. system emerges from the collective taste off, no matter what methodology the individual panelist is asked to use. We like the 5 star system, with half stars, because it doesn’t require of the participant to be unnaturally precise–it translates to a three point spread to allow for all sorts of variables. However, when the average/median score is determined it easily translates to a specific score within the 100 point score. So, for example, a 3.25 is equivalent to 86, or 4.04 to a 90.
Yes, the major critics tend to cover a greater number of wines; though it should be said that national and regional competitions judge a large cross section of current releases, e.g., Barbara Drady’s Pinot Shootout was able to critique over 350 Pinots this year.
Steve, I appreciate your courage in weighing in on this thing. It always generates batches of critical comments (incl. some from me). And because you admit the soft objectivity of it (what a great phrase that is), your cred goes up in my view.
Amazing – This blog post reads like a near-perfect explanation for why point scores are folly. And then it finishes with the obstinate view that point scores are still worthwhile, even though the preceding graphs blew apart any sense that they’re a good idea.
The 100 point system will not go away, we all know this, but as an industry can’t we figure out a way to make it more level then just one persons opinion at a time? Tom you nailed it with your comments. I also have mentioned the panel approach before when addressing the flaws of the 100 point system. Steve you are a talented writer and you have a good palate so why are you so unwilling to say that their are flaws in the system that need to be addressed? What the industry needs are people in position of power willing to take charge and address these flaws and come up with an alternative. I nominate you Steve.
Anthony you are correct most of the people who rage against the 100 point system are in the industry. It is not sour grapes or that some called their baby ugly. It is the fact that a small number of people have too much power to truly hurt a winery simply becuase they do not like the style of wine they make. Check out Coturri Winery for an example wonderful wines but confounding to most critics.
I am fine with the 100 point system. It’s not perfect but it works and isn’t going away anytime soon.
What I find curious is the lack of discussion about the tasting protocols the various critics employ. Does the reviewer simply pop the cork, pour some into a glass, grab a whiff or two, take a taste, a couple swishes in the mouth, spit it out and come up with a number and then move onto the next wine, all in the space of a few minutes?
Or is the wine opened and/or decanted and allowed to breathe for some time before evaluating. Does that same wine get re-evaluated over the course of two to five days before a number is finalized? With most young, high end wines, I find they evolve over a week and continue to open up each day until they become oxidized, which usually takes four to eight days with high end wines. I use this method to evaluate our young wines and I find they show best on the fourth or fifth day.
Steve, how do you do it?
Tim, I don’t expect you to have read every post I’ve put up over the last 3+ years. But if you had, you would know every single aspect of how I taste! I don’t believe any critic has been more transparent than I have.
Steve,
I agree, you are definitely the most transparent wine writer/critic around and I appreciate it as I’m sure most of your readers do. It’s unfortunate that your honesty is attacked so often. Out of curiosity, can you point me to your tasting protocol?
Tim, like I wrote, you’d have to go through three years of posts! I never put up a single “tasting protocol” because somebody always has an additional question that I have to answer later.
Steve,
In defense of the system, you miss the point of the criticism. It’s not that reviewers shouldn’t use a scale of some sort. It is that a scale of 100 points offers a lie, a precision that, by your own statements above, is not only impossible to achieve, but that is not actually even offered.
100 points from the Newbie wine taster, the Specter of Subjectivism, the Realism that pervades everything in our real world; has anyone looked at IMBd movie reviews lately? 6.5, and move the decimal.
The 100 point system is a “guide”, it’s not engraved on a tablet, but, especially to those new to wine, that rating system can help hone the palate (I, and most Newbies I know, like the 100 point system): I taste a wine, I think about what I’ve tasted, and now I look for what others have said and how they’ve scored that wine, it doesn’t change how much I like or dislike a wine, but is a directional signal; a reference of a 100 points is much better (To Me) than good, bad, or indifferent.
I try and aide family, friends, and neighbors toward wise wine purchases, and I have not made a penny doing it (Too bad).
I wonder how a section of people, and not just someone, attempts at quantifying feelings and memories. And how long they shall try to do so. And how strongly they’ll vouch for their delusions. And, finally, how much credibility they shall be awarded by powerful, political and like-minded “authorities”.
Insightful, as always, Steve.
Last time I checked not a single tasting room or wine retailer had a bouncing staff on hand strong arming people to buy wines that said critic gave 92 points for a Pinot over another producer’s 88 point Pinot. We live in a free society and if the masses choose to act like lemmings and rubber stamp their purchasing habits on score alone that is the consumer’s choice of their own volition.
We see enough BS in politics and in other arenas of life that if people are not willing to be free thinkers and drinkers, ie. drink what you like as opposed to what said critic thinks you should like, then that is the consumer’s fault. You want to trust the opinions of the critics that you might hold in high regard, but I have always believed that whether I am dealing with my finances, investments, or my wine purchases for that matter, I have the final say and should not give the power of attorney as to what’s in my cellar to any critic because he will not be paying for the wine nor serving it to me. What ever happened to the concept of free will?
I don’t like the 100 point system, but if the next person does, well, who am I to criticize. Recently, a gentleman pointed out to me that whilst I might know quite a bit about wine, he “needed to be told what was good to drink.” For people like that, the 100 point system is invaluable. And, how about Biodynamics? Maria Thun would probably suggest wine critics only review wines on ‘leaf’ and ‘flower’ days to lessen any variables. Anybody can buy a copy of Thun’s Biodynamics calendar and experiment for themselves.
Steve,
You’re totally right about the explosiveness of 100. It’s like the advent of a CD or email. Since there is so much wine out there, we need a system to triage wines and a 100 point scale even crosses most cultural barriers. All systems have faults and the 100 pt system has two: implications of mathematical certainty and lack of standardization.
Anyone who has looked at the ratings long enough would be able to eventually figure out that system is not a mathematical one. And that is the issue, the system uses numbers implies a mathematical certainty to a situation that cannot possibly be numbered and yet we do (you say don’t send me wine if you don’t believe in scores, I say don’t use a system based on numbers if you don’t can’t provide a mathematical certainty or don’t mind explaining this over and over and over again). It takes a certain type of person to think that they can number wines numerically (see Parker, who can “remember” every single wine he tasted but can’t re-ID them in a blind tasting or even which Bordeaux bank). You acknowledge these severe limitations, but even under your best case scenario requires me to subscribe to a magazine (or read a blog) with a reviewer who drinks enough of the same wines as you. I drink mostly Californian and am a WE subscriber and still don’t feel that I can trust yours or anyone’s pallet to spend $$$ without tasting it first. Wine scores let me what to investigate further. That’s it.
Wine scores are successful not because they are demystifying. Look at what goes in to a wine score: color, nose, favor, mouth feel, etc who knows. Every publication leaves very vague criteria without an actual breakdown of what makes a certain wine score. The reason 90 pts became so important because it sounds like a good wine no matter how critical (or not). In reality each person’s scale is completely different. Merry Edwards Sauvignon Blanc is ridiculously good, but it’s a little cloudy, does that dock you points? Who knows. You’re never going to get scores from a panel or standardize pallets, but you can at least formalize the score to make a useful tool even more so.
Scores are for those who must a have number. They exist and there is no changing their minds. I prefer wine reviews that describe the wine in simple language; its flavors and style and perhaps food affinities. If it is flawed, from wine making, it should not be given the time of day. One of my favorite quotes is a conversation between Emile Peynaud & a French Producer, Dubourdieu, (Bordeaux). “Actually, I never studied under Peynaud, but I admired his work and devoured his books, especially The Taste of Wine. He was a man for all seasons, at once scientist and practical hands-on winemaker. He could talk to the Chateau owner just as easily as the cellarmaster & convince them both. He said “The wine should taste more of the grape than the grape itself…wine is the revelation of the hidden character of the fruit.” He also said something that profoundly influenced me. He said “Better a genuine small wine than a fake grand wine.” Once at a dinner, I asked him what he thought of my wines. At first he did not answer. Finally, he said “That is of no interest.” I was completely dumbfounded! After a long pause, he explained, “We don’t belong to the same period. You must make the wines of your time.” His answer was particularly profound. A Wine, if it is to be human should be beyond time, in constant evolution, otherwise it becomes merely an interesting museum piece.
I’ve tasted wines that are rated a 94 and above that I love and I have also tasted 94+ wines that I do not like. Each person has different tastes and there is no way to objectively give wine a score that every single person will agree with.
I have noticed that my particular taste agrees more with some particular wine reviewers than with others. Naturally, I will pay more attention to those who share my tastes.
I meant to type fruit and flower days!
I’m ok with a point system, but the associated text has become really important to me as I have learned about my palate. I personally prefer low oak, moderate fruit, and good acidity. I look for plush, jammy, rich, and other adjectives to know the wines to avoid. A score alone is little help.
Also, I am very happy there are many wines made for many different palates. Life would be pretty dull if everyone had the same interests, views, and tastes!
Vinogirl – yeah that bioD bulls**t is confusing, isn’t it
Yes, Kathy, it certainly is 🙂
The best thing about it is that it sells wine. The worst thing about it is that it sells wine.
I don’t think the debate will ever die. There are haters of the point scales, and there are fans. I have for many years detested the ratings, not because of who is behind it, but because of how arbitrarily it seems to diminish the art, the science, the life inside the bottle. It’s the poet in me I guess. Yet as a retailer, it is a necessary (quasi) evil. Bottom line, it sells wine. And we as writers, bloggers, retailers, wholesalers, importers, winemakers, consumers, fans, etc. – if we aren’t really working toward selling wine, then what the hell are we doing? Yes, I am a member of the Scorevolution, but only because the score cannot be the beginning, middle and end of discussing a wine or wine overall, it can only serve to be the beginning. If it simplifies a consumer’s choice from one Napa Cab over another, or one Australian Riesling or another, so be it. If it serves as a gateway to get the consumer into a new varietal, wine region or style, all the better. As for being a blogger who would hop on the points-bashing train, no thanks. I have moved on. If for nothing else, no one has come up with a better alternative to draw people in to a particular wine. My only protest would be that, there aren’t enough wines being reviewed. What is the percentage of wines produced being reviewed Steve? What would you say, 10%? 20% Why not more? Why not 50%? Why not all? Impossible? Why? Doesn’t necessarily have to make it to print. That’s what the Web is for. The haters? Come work in a wine store. See what makes your job easier – simple shelf descriptors, or a freakin’ 90+ score. Then come bitch to me about the points. If you hate it so much, stop bitching about it and devise a way that will get the job done better. Solutions not complaints people.
k2, thanks for a thoughtful comment.
K2 – I’ve worked in a wine shop. It is more fun, interesting and satisfying if you do your job and sell wine rather than let points sell your wine for you. Taste the wine, ask the consumer what type of wines they enjoy and then describe a few wines to them. Sure, not all consumers will want to listen to you, but if you enjoy what you do then consider changing their views as something to accomplish. If you don’t enjoy your job and want scores to do you work for you, find a new job. And I agree with you, I do not think scores are going anywhere either. But we can all do our part to educate consumers so they no longer needs scores!
Winemaker Daniel Baron is currently the winemaker for Twomey Merlot and most recently the winemaker (and still consultant) for Twomey Sauvignon Blanc and Pinot Noir