moki
el Presidente
A discussion of Cigar Aficionado's cigar rating system appeared in this thread on another board. Accusations were flung around, and then Gordon Mott, Executive Editor of Cigar Aficionado stepped in to respond to some of the more... flamboyant accusations that were leveled, and explain the testing system that they use.
Here's the post, I think it is interesting reading.
.....
Gordon Mott
A Response from Cigar Aficionado - Today, 04:02 PM
Gentlemen,
This is Gordon Mott, the Executive Editor of Cigar Aficionado. I usually don't jump into forums, except our own, to explain how we taste and rate cigars. But when hugomarink borders on libel in the way he describes our blind tastings, I feel compelled to set the record straight. Many of you have heard me explain this in public forums, so for you, it is nothing new. But let me be very, very clear; just because Smoke magazine does their ratings the way Hugomarink says, it doesn't mean that Cigar Aficionado does it that way. Frankly, if what he says about Smoke is true, I would be humiliated to ever admit that I'd been part of such deception. Integrity starts at home. But it is clear he has no idea how a professional publishing company operates.
First of all, let me explain our process. I have a full-time employee who does nothing but prepare our blind tastings for us. He goes out into the New York retail market and purchases the cigars that we rate; yes, to answer your question, we have our methods to acquire Cuban cigars too but they are all purchased. Over the years, I would estimate that we have actually purchased more than 98 percent of the cigars we rate in the magazine--the goal is simple; re-create as closely as possible the buying experience of our reader. The tasting coordinator then removes the brand's bands, and replaces them with numbered bands; he and only he holds the key during the tasting period. I think right there you might see one of the fundamental differences with any other magazine; we devote the resources and manpower to ensure there is a firewall between the purchase and the smokers.
After the bands are numbered, they are placed in the humidors of the tasters: they include my self, Dave Savona and two other tasters here in the New York office. From time to time, we will include Marvin in the blind tastings and when Mr. Suckling is in New York, like for our final top 25 tasting this year, he participates also. But there are four tasters who are involved in every blind tasting. To suggest that I only get the good ones, and everyone else gets the dregs is just plain ignorant. They are divided among us equally and fairly and without regard to country of origin. If there are big discrepancies in the scores, we will re-taste a cigar but that's only fair.
Secondly, in 15 years, I have never seen a score manipulated, nor have we ever given a score to a cigar that we haven't smoked. I actually find that hard to believe that other magazines might make up scores out of thin air. Talk about dishonest.
His suspicion that we don't smoke all our cigars in house is also totally ignorant. We decided never to use outside panels because we don't want any unknown influence on the ratings, or for that matter, differing levels of amateurishness and tasting experience to cloud the ratings process. We now have tasters where the least experienced has been doing it for eight years. Dave has 13 years experience and I have 15 years in as a taster. While that doesn't automatically qualify us as Tasters Extraordinaires, it does allow us to lay claim to the reality that outside of a cigar factory there are very few people in the world who have smoked as many different cigars as we have. And, yes, I would say that it one of the hardest parts of the job because it never stops...we finish one tasting and we move on to next. But I can guarantee you this: every cigar is smoked blind inside the offices of M. Shanken by full-time employees of the company, all of whom love cigars and smoke them for pleasure when they can. In rare cases, someone may take a tasting cigar home to smoke over the weekend, but I don't encourage that; by reducing the external variables as much as possible, you at least approach something akin to the same circumstance for each cigar smoked, and maybe make it a more objective process.
Finally, the canard that just won't go away is that one that we massage ratings for our advertisers. I have tried to dispel that myth so many times that I'm blue in the face. But the facts are the facts, and they do speak for themselves. First of all, many of our highest rated cigars are Cubans; by law, we cannot take one cent of advertising from the Cubans. Secondly, for the first 10 years of the magazine, one of the lowest average scores for a cigar brand was earned by one of the top-selling brands of our biggest advertiser. If manufacturers out there are advertising because they think they can buy scores, they sure aren't getting much for it. Also look at some of the newer smaller brands that have earned high ratings in the last two or three years; the ratings long preceded their ability to even think about spending money on advertising.
One of our favorite sayings here is, "The score is the score" And, we tell manufacturers to make better cigars if they want higher scores. But we DO NOT change scores because they belong to an advertiser...and if you talk to people in the industry, instead of speculating because of what other magazines do, you would know that to be true. It is true that in recent years we have noticed a narrowing the range of scores, but in the early days, we gave a lot of scores under 80...we have been analyzing the scores and trying to figure out if its just that cigars have gotten better, or we've gotten softer...I think it's the former. But this is not a science...it is a subjective exercise that we are always examining in hopes of making it better.
For more than 30 years, M. Shanken Communications has based its reputation on the integrity of our blind ratings, both for the Wine Spectator and in the last 15 years, for Cigar Aficionado. In all those years, the same charge has been made that advertisers can buy a good score. And in all that time, no one has ever proven that to be true, nor has any advertiser ever come forward and said, Yes, I bought that score. There's a simple reason. It is not true. We take great pride in the honesty and integrity of our blind ratings system, and we put both the manpower and the training into it to make sure we maintain its quality.
People disagree with our ratings. That's what makes the world go around. But again, I can guarantee you that no one can fault the process.
It now seems apparent that others cannot make the same claim. That's too bad.
Here's the post, I think it is interesting reading.
.....
Gordon Mott
A Response from Cigar Aficionado - Today, 04:02 PM
Gentlemen,
This is Gordon Mott, the Executive Editor of Cigar Aficionado. I usually don't jump into forums, except our own, to explain how we taste and rate cigars. But when hugomarink borders on libel in the way he describes our blind tastings, I feel compelled to set the record straight. Many of you have heard me explain this in public forums, so for you, it is nothing new. But let me be very, very clear; just because Smoke magazine does their ratings the way Hugomarink says, it doesn't mean that Cigar Aficionado does it that way. Frankly, if what he says about Smoke is true, I would be humiliated to ever admit that I'd been part of such deception. Integrity starts at home. But it is clear he has no idea how a professional publishing company operates.
First of all, let me explain our process. I have a full-time employee who does nothing but prepare our blind tastings for us. He goes out into the New York retail market and purchases the cigars that we rate; yes, to answer your question, we have our methods to acquire Cuban cigars too but they are all purchased. Over the years, I would estimate that we have actually purchased more than 98 percent of the cigars we rate in the magazine--the goal is simple; re-create as closely as possible the buying experience of our reader. The tasting coordinator then removes the brand's bands, and replaces them with numbered bands; he and only he holds the key during the tasting period. I think right there you might see one of the fundamental differences with any other magazine; we devote the resources and manpower to ensure there is a firewall between the purchase and the smokers.
After the bands are numbered, they are placed in the humidors of the tasters: they include my self, Dave Savona and two other tasters here in the New York office. From time to time, we will include Marvin in the blind tastings and when Mr. Suckling is in New York, like for our final top 25 tasting this year, he participates also. But there are four tasters who are involved in every blind tasting. To suggest that I only get the good ones, and everyone else gets the dregs is just plain ignorant. They are divided among us equally and fairly and without regard to country of origin. If there are big discrepancies in the scores, we will re-taste a cigar but that's only fair.
Secondly, in 15 years, I have never seen a score manipulated, nor have we ever given a score to a cigar that we haven't smoked. I actually find that hard to believe that other magazines might make up scores out of thin air. Talk about dishonest.
His suspicion that we don't smoke all our cigars in house is also totally ignorant. We decided never to use outside panels because we don't want any unknown influence on the ratings, or for that matter, differing levels of amateurishness and tasting experience to cloud the ratings process. We now have tasters where the least experienced has been doing it for eight years. Dave has 13 years experience and I have 15 years in as a taster. While that doesn't automatically qualify us as Tasters Extraordinaires, it does allow us to lay claim to the reality that outside of a cigar factory there are very few people in the world who have smoked as many different cigars as we have. And, yes, I would say that it one of the hardest parts of the job because it never stops...we finish one tasting and we move on to next. But I can guarantee you this: every cigar is smoked blind inside the offices of M. Shanken by full-time employees of the company, all of whom love cigars and smoke them for pleasure when they can. In rare cases, someone may take a tasting cigar home to smoke over the weekend, but I don't encourage that; by reducing the external variables as much as possible, you at least approach something akin to the same circumstance for each cigar smoked, and maybe make it a more objective process.
Finally, the canard that just won't go away is that one that we massage ratings for our advertisers. I have tried to dispel that myth so many times that I'm blue in the face. But the facts are the facts, and they do speak for themselves. First of all, many of our highest rated cigars are Cubans; by law, we cannot take one cent of advertising from the Cubans. Secondly, for the first 10 years of the magazine, one of the lowest average scores for a cigar brand was earned by one of the top-selling brands of our biggest advertiser. If manufacturers out there are advertising because they think they can buy scores, they sure aren't getting much for it. Also look at some of the newer smaller brands that have earned high ratings in the last two or three years; the ratings long preceded their ability to even think about spending money on advertising.
One of our favorite sayings here is, "The score is the score" And, we tell manufacturers to make better cigars if they want higher scores. But we DO NOT change scores because they belong to an advertiser...and if you talk to people in the industry, instead of speculating because of what other magazines do, you would know that to be true. It is true that in recent years we have noticed a narrowing the range of scores, but in the early days, we gave a lot of scores under 80...we have been analyzing the scores and trying to figure out if its just that cigars have gotten better, or we've gotten softer...I think it's the former. But this is not a science...it is a subjective exercise that we are always examining in hopes of making it better.
For more than 30 years, M. Shanken Communications has based its reputation on the integrity of our blind ratings, both for the Wine Spectator and in the last 15 years, for Cigar Aficionado. In all those years, the same charge has been made that advertisers can buy a good score. And in all that time, no one has ever proven that to be true, nor has any advertiser ever come forward and said, Yes, I bought that score. There's a simple reason. It is not true. We take great pride in the honesty and integrity of our blind ratings system, and we put both the manpower and the training into it to make sure we maintain its quality.
People disagree with our ratings. That's what makes the world go around. But again, I can guarantee you that no one can fault the process.
It now seems apparent that others cannot make the same claim. That's too bad.
