subscribe: Posts | Comments      Facebook      Email Steve

Crowd-based reviews? A consideration



I’m not certain I agree that, in large, multi-judge competitions, “the best wines tend to rise to the top,” as Andy Perdue says in his column in Pacific NW Magazine says.

Andy’s contention is based on the fact that a big competition represents “a consensus of the judges who are tasting the wines” and, thus, this crowd-sourced opinion is more free of “biases” than would be the judgment of an individual taster.

This is a sort of “50 million Frenchmen can’t be wrong” analysis. (That slogan comes from a Broadway play of 1929 that contrasted the socially-liberal attitudes of 1920s Paris with the censorship and alcohol Prohibition then in effect in the U.S. Its usual meaning is, “Hey, if so many people like it, there must be something to it.”)

There is indeed something to be said about consensus among large groups when it comes to decision-making. Our elections are based on consensus. Just as we would not want a single individual to pick our political leaders, but prefer to leave that up to the collective will of the voters, so too the consensual approach to wine judging suggests it somehow arrives at findings that are more true and real and thus of greater lasting value.

However, Andy himself acknowledges that you can unearth new and exciting wines in at least five ways,” namely, wine competitions, wine critics, wine merchants, wineries/wine festivals and friend recommendations. The question is not, “Which is best?,” because each way obviously has its own special appeals. The question is, “Which of these five ways is on the increase and which is losing steam?”

We’re certainly familiar with the contention that the Big Critics are losing sway, and I think that’s true, in the sense that the more famous of them are aging out. But what does this mean for the template? Not much. The New York Yankees, much less Major League Baseball, didn’t phase away with Derek Jeter’s retirement, because talented younger players—Bryce Harper, Yasiel Puig, Mike Trout—are constantly coming up. In the same way, it’s becoming fashionable to predict the demise of the individual critic, but we’ve seen the stunning rise and success of Antonio Galloni’s Vinous, with its super-stable of critics, and other leading publications (including The Wine Advocate, which isn’t going anywhere), remain hugely influential. So it does not seem to me that wine critics are an especially endangered species.

The question, though, of individual judgment versus consensus-driven judgment remains interesting, albeit probably unresolvable. Having participated in huge tastings (although the last one was years ago), I came to the personal conclusion that there were too many people involved, too many egos, too much confusion and, at any rate, too much alcohol imbibed in too short a time, at least for my tastes. (Perhaps things have changed in the interim.) Then too, consensus usually means that outlier wines, at both the top and the bottom, are eliminated, resulting in a shift toward the median, or average. That’s not to say that a winner at a big competition is not a fine, even a great wine, but it does stand to reason that it will be a wine that offends the least number of people, while appealing to the greater esthetic. One could argue that such a wine would be more ordinary than special. But I’m acutely aware that every single one of my arguments could be turned on its head, and used to prove the opposite.

Another limitation of big competitions is that not all the top wineries even bother to enter them. In fact, many top wineries don’t, for the simple reason that they have more to lose than to gain, by getting a lower score than, say, a supermarket wine. This raises a fundamental limitation of any method of critical appraisal: no competition, no individual judge can possibly review all the wines out there. Even I, as a working critic who tasted thousands of wines each year, could barely keep up—and there were some wines I never tasted at all.

I will admit there’s a certain allure to a double gold winner from a big competition. I think to myself, “It certainly can’t be a bad wine, and it might even be a very special one.” It’s kind of the way I feel about Yelp: for all its well-publicized problems, Yelp does provide a kind of base-of-the-pyramid generalization of the conclusions of many people. If the majority of 40 reviews of a restaurant are positive, I’m more inclined to check it out. But—and this is just me—I still prefer the review of a single, trusted critic, like Michael Bauer, to that of the crowd.

  1. Bill Haydon says:

    I’ll speak only relative to my usual audience which is influential on and off premise buyers located predominately in NYC, Chicago, DC and Boston. Nobody is caring about critics anymore, so I dispute that the WA is still “hugely influential.” Sure, RMP is going to have his fanboys and subscribers among consumers, but I have to wonder as wine trends filter out from the so-called “trendsetter” markets to the rest of the country that the ongoing decay of influence will only accelerate.

    Now, these buyers aren’t replacing Parker with crowd sourced wine reviews anymore than they are looking at Yelp for their dining decisions. What I see is a return to the true pre-Parker wine merchant. Stores with no scores that buy based on the owner’s perception of the market and evaluation of the quality brought before him.

    Probably the best new store that I’ve seen come along in a long time is Independent Wine & Spirits in Chicago. Every wine has a handwritten shelf talker written solely by the owner and zero point scores. The owner neither asks for nor cares about scores. Everything is hand-selected solely based upon the region, quality and value. And the shop has become a huge success even in the shadow of the Binney’s leviathon.

  2. Judi Levens says:

    Very interesting…but wouldn’t it also depend on the qualification of the judges in a big competition? If they are very experienced than it should take away that “general crowd” kind of atmosphere and still elevate the best…I don’t have any experience in this area…just a thought

  3. Yeah, just like the best movies rise to the top and get Academy Awards.

  4. I’ve judged in one wine competition, and it was an absolute shitshow. I have much more faith in professional critics.

  5. doug wilder says:

    Bill Haydon, It isn’t the path that any other critic I know of took, but what I do now is a straight outgrowth from spending 18 years as a retail buyer for some of the leading retailers of their day. During that time I can’t recall writing more than a dozen shelf talkers. Fifteen years ago, there were 1300 facings at Dean & Deluca St. Helena and zero shelf talkers. We kept it all in our heads, however, that career taught me an important lesson. people do value specific guidance, and much prefer it from a pro (retailer,sommelier or critic). As a wine buyer I wrote tasting notes for my clients and since they understood scores, I adapted a 20 point scale to describe the wines I really liked. later shifting to what I use now.
    I have commented on other sites about the issues I find with crowd-sourced apps and my main point is that a consumer will likely never be the Alpha Case of a wine note that any reasonable person would think a wine professional should have written first. Take Vivino, for example. Look at the California reviews, the top 100 are dominated by large production corporate-owned wines that are staples in most large grocery chains, well-made but not close to representing the small fraction of wines most consumers seek professional guidance on. There are multiple thousands of wines released every year and it is impractical for any one person to taste them all. I don’t know off the top of my head the value of the wines I taste in a particular issue, but if a consumer wanted to duplicate that and make their own conclusions, it would likely cost mid to high 5 figures and they would need to approach it with serious concentration not to mention devotion of many hours. As Steve can attest wine criticism can be grueling. I think that is why you don’t see that many independents doing it at a high level, with any kind of quality mandate. The commitment requires all sorts of other compromises with other activities. I think it easy to criticize the critics because people only need to judge the results. Ex: It is simple to remember the lyrics to a song written by John Lennon, but could any of us have come close to writing them originally? Currently wine apps are the 5 second sound bite of the industry.

  6. Andy could provide helpful information as the journalist for this competition if he would be transparent with his readers about:

    —What is the exact percentage of those that won’t win any medals? If it is 40% (which he/they won’t tell you), it means that 40% of the wines at the competition are mediocre craps & pure craps;

    —10-12% of entries will win gold. How about the % for silver and bronze? So everyone wins something, and no one is a loser.

    —If each member of the 3-5 member judge panel selected a different gold-medal winner, who then would decide the one winner out of 3-5 different winners?

    Amazing that human & earth resources are continuing to be employed to churn out tons of mediocre craps.

  7. Steve:

    Thanks for bringing up the topic. Sorry I’m a bit late in commenting, but I spent much of the day judging wine.

    There does seem to be some misconceptions here about how competitions work. I wouldn’t necessarily call this “crowd-sourced reviews,” though one could argue that the old saw about “three’s a crowd” would make it so.

    Today, I judged with two superb wine experts who come from two different perspectives and backgrounds. As a result, each of us learned something today that we didn’t previously know. We worked together to taste, review, rate and reward wines based on our various experiences and insights. We talked through every wine and did our best to reward each according to its virtues. It was a really great day spent exploring Petite Sirah and dry Rieslings.

    To say that wine judgings are the be-all, end-all would be ludicrous, which of course is why I suggested to my readers in The Seattle Times that there are at least five ways for them to explore and discover new wines. All are methods I use myself as an enthusiastic consumer.

    You make an excellent point that many wineries do not enter competitions. That certainly is their prerogative, and it might not fit into their marketing strategies or budgets. But for those that do, it is a good opportunity to get their wines in front of accomplished professionals.

    I will tell you that as I judge wine, my one constant thought is that each entry cost the producer a fair bit of coin (at least $100 per entry, when you take into account entry fee, multiple bottles of wine and shipping). As such, each wine deserves my full attention and consideration. It is my experience that most judges operate this way.

    You might also find it interesting to know that many judges also are wine critics who will publish their own notes and results (sometimes using the 100-point system), regardless of how the panel votes. This is an interesting side benefit we don’t often think about. Indeed, I’ve been able to develop several story ideas and angles that I will pursue in the days and weeks to come.


    I’m sorry you had a poor experience. I participate in perhaps 15 competitions per year (in varying capacities). Most are professionally run. In my early years, I judged some pretty bad ones, too (“Why yes, you should probably rent more than 10 glasses.”).

    I don’t know that your one experience should sour you on all competition results.


    Thanks for the constructive feedback. In fact, I wasn’t trying to hide anything from my readers. From within the 500-word limit I have for my column each Sunday in The Seattle Times, I try to be as informative as possible. In this case, I was taking advantage of the timing of the Chronicle competition to inform readers about how they might use judging results. I don’t believe that flooding the column with statistics would serve any purpose, nor make me more transparent. Your opinion may differ, and I will defend your right to express it.

    Additionally, I apologize for appearing to churn out tons of mediocre crap. That certainly is not my intention. Your concise and constructive feedback will encourage me to improve.

  8. Bob Henry says:

    Some years ago, I judged the Mid-State County Fair Competition in Paso Robles.

    Chardonnays in the morning. Syrahs in the afternoon.

    Joining me at my four-person judging table were a UCLA English classics professor (whom I knew from local wine circles), and two winemakers from prominent Paso Robles/San Luis Obispo wineries. (Sorry, no names.)

    They made and loved “Butterball turkey” malolactic, oaky Chards. I like mine with palate-cleansing acidity.

    They made and loved very fruity, low acid Syrahs — with no hint of Brettanomyces. I take my stylistic lead from the Rhone Valley. (But concede that the “French funk” notes from Brettanomyces can be off-putting.)

    The UCLA professor shared my affinities.

    When a non-ML Monterey Chard exhibiting crisp green apple acidity appeared in a flight, the two winemakers downgraded it to a bronze medal. (No butter. No toasty oak.) The UCLA professor and I championed it, and graded it a gold medal.

    When a Santa Barbara County Syrah with mouthwatering acidity and just a scintilla of Brettanomyces appeared in a flight, the winemakers declared it “defective” and lobbied to have it eliminated from the competition. The UCLA professor and I pushed back, and it stayed in the judging. The winemakers gave it no award. The UCLA professor, judges at other tables, and I graded it a gold medal.

    During a break in the competition, I asked the two winemakers if they drink white Burgundies or red Rhones at home or at the office.

    “No,” was their reply.

    I asked if they had ever visited Burgundy or the Rhone Valley and sampled their offerings.

    Again, “no” was the reply.

    So I asked them how they could be so dismissive of California wines that emulate those historical paradigms.

    They had no good answer.

    (And they were not amused by my defending such wines — and grading them worthy of gold medals. Maybe it is just a coincidence, but I’ve not been invited back since to judge another competition.)

    My first wine mentor Robert Lawrence Balzer used to exclaim to his wine appreciation course students that “We like that best to which we are most accustomed.”

    These Paso/SLO winemakers only knew their homegrown wines. Ostensibly lacked the curiosity to sample wines from outside of their own backyard and explore the larger wine world.

    I close this comment with this “thought piece” article from a Caltech lecturer/professor on the inherent flaws in wine judging competitions:

    From The Wall Street Journal “Weekend” Section
    (November 20, 2009, Page W6):

    “A Hint of Hype, A Taste of Illusion;
    They pour, sip and, with passion and snobbery, glorify or doom wines.
    But studies say the wine-rating system is badly flawed.
    How the experts fare against a coin toss.”


    Essay by Leonard Mlodinow

  9. Bob Henry says:

    At the risk of appearing to be playing James Boswell to his Samuel Johnson (by citing his wine column in comments here), Dan Berger judges “more than a few” wine competitions each year.

    My next comment (awaiting “moderation”) extends the bibliography on Dan’s “take” on the credibility of wine competitions.

  10. Andy,

    Thanks for the thoughtful response. I will try to keep an open mind

  11. Andy, I appreciate your openness and that you’re conscientiously responsive. Although I didn’t get my answers, which can be addressed in 2 sentences, I read your enthusiasm and aspiration. If wine media is like the earth where we live in, I hope you’re the salt of it. Best wishes to you.

  12. Susan,

    Thank you for taking the time to respond. I misunderstood your comment as wishing I’d included this information in my column. Here are my best estimates:

    — What is the exact percentage of those that won’t win any medals? This depends on the wines entered and the judges. I’ve seen it as high as 50 percent and as low as 15 percent. In my experience, the number of “no medal” wines has fallen in the past 15 years as wines have generally improved through education and technology. A wine winning a “no medal” is not always an indication that it is “pure crap.” Rather, it could be a flaw or it could be that it is boring or lacks varietal definition. Last week at the Chronicle competition, I know of at least one wine that was awarded “no medal” because the winery quite obviously loaded up a Pinot Noir with Mega Purple to deepen its color. The judges on that panel were offended by this strategy and awarded the wine appropriately. (I wasn’t on that panel, but I observed it.)

    — 10-12% of entries will win gold. How about the % for silver and bronze? In recent years, there has been a trend to move a “high bronze” to a silver because a silver medal is more valuable. As such, I’ve seen about 25 percent of wines earning silvers and anywhere from 25 to 35 percent winning bronzes.

    —If each member of the 3-5 member judge panel selected a different gold-medal winner, who then would decide the one winner out of 3-5 different winners? It doesn’t work this way. Each wine is awarded gold, silver, bronze or no medal on its own merits. Technically, every wine entered could win a gold or be given a “no medal.” But that never happens. In many competitions, a “best of class” is awarded from among the gold medal winners. So let’s say a panel awards five gold medals out of 50 Cabernet Sauvignons. They then would select their favorite from those five for the “best of class” award. In the rare instance that all judges on a panel vote gold, then the wine is awarded a unanimous double gold medal.

    That said, every competition is not the same. Some do double gold medals, best of class and superlatives (best red, best white, etc.), and some do not.

    It would be nice if all competitions worked from the same set of rules, but they do not. However, most are consistent. And in California, you will see some of the same judges working different competitions. This helps with consistency, in my opinion. Up here in the Pacific Northwest, we have a few judges who work different competitions, but it’s not nearly as developed as in California, which has way more qualified wine professionals.

    I hope this helps.

  13. From my hoary archive of newsclippings, circa 2007.

    (No, April Fools’ Day didn’t come early this year. And this comment isn’t from HoseMaster . . .)

    From the Napa Valley Register “On Wine” Section
    (June 29, 2007, Page C6):

    “The Judgment of California:
    Charles Shaw Chardonnay is State’s Best”


    By Register Staff

    The Charles Shaw 2005 California chardonnay (yes, the $1.99 “Two Buck Chuck” made by Bronco Wine Company sold at Trader Joe’s) was judged Best Chardonnay from California at California State Fair Commercial Wine Competition.

    The chardonnay received 98 points, a double gold, with accolades of Best of California and Best of Class.

    “Since we judge all wines totally by variety without different brackets for price, this double-gold achievement by the Bronco winemakers is astounding,” said G.M. Pucilowski, chief judge and director of the competition.

    While the complete results of the competition are scheduled to be announced July 12, Renata Franzia, from Bronco’s Franzia family received the results Thursday.

    Dr. Richard Peterson, veteran winemaker and a State Fair judge for 20 years, said, “We have the most open judging I know. There is nothing to bias judging. We get numbered glasses. We don’t know the region, brand or price. We evaluate the judges frequently to make sure they’re tops in the field. Charles Shaw won because it is a fresh, fruity, well-balanced chardonnay that people and wine judges — though maybe not wine critics — will like.”

    Bronco president Fred Franzia said, “The customer has responded to our great values and now the judges to our quality. This is why we’re in business: To put the best wine possible on the customers’ tables at a reasonable price. I just wish some other retailers and restaurateurs had the courage and good sense to make these super-value wines available to more American wine consumers.”

Leave a Reply


Recent Comments

Recent Posts