Backstage: iLounge’s new grading system | iLounge Backstage

Backstage

Backstage: iLounge’s new grading system

Author's pic

By Jeremy Horwitz

Editor-in-Chief, iLounge
Published: Friday, August 13, 2004
Backstage Categories:

Over more than a dozen years of professionally reviewing (and twenty years of reading about) consumer electronics products, I think I’ve seen just about every type of rating system ever devised. The most common involve four or five stars, icons, or key phrases, while others prefer 10-point scales, 100-point scales, or something in-between (such as 10-point scales with single-digit decimals).

Each system has its problems. What does a 73 out of 100 (or a 7.3 out of 10) really tell you about a product? Would your answer to that question change if the publication never awarded scores below, say, 50 (or 5.0)? And what is a 3.5-star product? Numerically, that’s the equivalent of a 70% rating… if you’re using a 5-star scale. On a 4-star scale, that’s an 87.5% rating. What does –that- mean when the only higher score you can get is 100%?

And I won’t even get into some of the cool but decidedly non-mainstream Asian rating systems I’ve seen – multi-factoral graphs with scores that visually resemble starbursts. There are unquestionably lots of ways to review products. The question is: which is best? My answer, and an early explanation of our new rating system, follow in the “Read More” box below.

Of all the scales I’ve ever used, my favorite remains the letter grade system, which is very easy for most of our readers to understand instantly (and intuitively because of school), even though it has the flexibility of a 13-point scale. For that reason, we’ve implemented letter grades on iLounge in place of our previous four-icon system. The new letter grade system works like this.

A+, A, and A- grades are awarded to only the very best products, ones that we would highly recommend. While an A- grade indicates one or two small deficiencies that limit the product’s universal appeal, an A+ grade indicates that the product is too good to be true – a useful and virtually universally appealing item, more than reasonably priced, that represents a major step forward without even small flaws. For reference, even the original iPod would not have received an A+ grade from iLounge. We haven’t awarded one yet, and would be surprised if we ever do.

B+ and B grades are awarded to very good and good products, ones that we would recommend only to specific audiences. A B+ grade indicates that the product may not be universally appealing because of certain small to medium design limitations, decisions, or flaws, but will likely provide great satisfaction to a specific narrow audience. A B grade denotes a product that will mostly satisfy a specific narrow audience.

A B- grade is on the fine edge of “good” and “recommendable.” It indicates a product that is better than average, but has a few medium to large issues that crimp its appeal. We would typically recommend a B- product only to people who really need its functionality.

C graded products are “okay” or “average” ones that we neither recommend nor dislike. They perform all of their stated functions acceptably, but may not be attractively designed, well thought out, or appropriately priced. A C+ grade indicates that the product was a little bit better than average, but still not “good” or “recommendable.” The C- grade indicates that the product was on the edge of being contemptable, and though it worked substantially as promised, possessed some serious design issues.

D graded products are ones that we actively disliked for some reason. Typically, that reason is that the product provided such substantially below-average design, performance or pricing that it was nearly laughable by comparison to other available offerings. It typically will be something that is rendered unfit for a major stated purpose because of poor design. A D+ indicates that the product could have been brought up to C level if a single major issue was fixed. A D- means that the product was close to historically terrible.

F graded products are loathsome. They were either pointless or dangerous when we tested them, or otherwise quite substantially failed to conform to their promised performance levels. We do not award F+ or F- grades; a flat F is considered “failing.” We reserve the right to award either D or F grades to products that we receive for review in defective or materially damaged condition, when we believe that such defects may also appear in products received by end-users.

In the near future, we will be retrofitting all of iLounge’s prior four-icon ratings with letter grades, and providing a much-requested report card for all of the products we’ve reviewed. For the most part, “Excited” icons will translate to A grades, “Happy” icons will translate to B grades, “It’s OK” icons will translate to C grades, and “Sad” icons will translate to D or F grades. (There may be a few exceptions.) We also will likely change the old words “Good” and “Bad” at the bottom of our reviews to the words “Pros” and “Cons” or something similar, because as products approach the A level, “Bad” becomes more like “nitpicks” than “serious issues.”

The purpose of these changes is to make iLounge reviews more useful to our readers, and since this is appearing on Backstage first, my hope is that you might share comments/suggestions/thoughts on these changes. Anyone? Bueller?

« Backstage: An aside on portable games of 2004

Backstage: Griffin’s secret iPod products »

Related Stories

Comments

1

I think letter grading is a great idea. The four icon system isn’t quite as clear. 
I’d love to see a feature and grade comparison chart for each of the review categories, which I guess is what you’re referring to as the “report card”. 
Keep up the good work. You’re making a good thing even better.

Posted by akseez in Orem, UT on August 13, 2004 at 9:41 AM (PDT)

2

I mostly ignore ratings. The good and bad points of the product have much more value.

Posted by SouthPaw in PV, NY on August 13, 2004 at 10:07 AM (PDT)

3

Well—

When you are using a numerical value (e.g. 0 through 10, 100, whatever), the only way that the system can actually mean something is if the ratings are normally distributed… You have to have the same number of “F” grade products as “A” grade products… otherwise, what defines what an A or F grade product really is?  Almost everything has room for improvement (even *gasp* the iPod), and almost nothing is as bad as it could be.

Just my opinion,

Posted by James on August 15, 2004 at 10:43 AM (PDT)

4

I agree that the letter grade scale is very intuitive, and therefore is a great system for any new-comers to the ‘lounge.

I prefer the new precise ratings to the previous emoticons.  There are enough rating levels, but then again, not too many either.

To be blunt, thank you for the change.  Much like apple would do, you gave us an improvement that we didn’t realize we needed until it was offered.

Jason smile

Posted by StinkieDMB in Pittsburgh, PA on August 15, 2004 at 6:21 PM (PDT)

5

Thanks for all of the opinions so far… more are welcome.

James - yes, since “normal” distribution for grading purposes is supposed to be a bell curve, the perfect system would be one where there are far more “average” grades than A’s or F’s, and both As and Fs would be comparatively rare.

Our system will probably differ a bit from statistical perfection in that A will be paralleled more closely by C- and D grades, with F reserved for dangerous/defective products, and A+ reserved for perfection. I hesitate to call the system a “B curve” system… I think B-/C+ should be where grades tend to fall… but who knows what will happen. Thoughts?

Posted by Jeremy Horwitz on August 16, 2004 at 12:00 PM (PDT)

6

I’d like to use this site to make decesions about ipod accessories, but there appears to be a fair amount of reading involved.  A list of products with their grades in the “Review” section, would allow me to read only the A level reviews, and pick the best in a shorter amount of time.

This would also make it easier to evaluate the scale itself, I could make a reasonable guess at what an A or B grade meant.

Does this list exist, and I missed it?

Posted by redruM on August 18, 2004 at 5:45 AM (PDT)

If you have a comment, news tip, advertising inquiry, or coverage request, a question about iPods or accessories, or if you sell or market products, read iLounge's Comments + Questions policies before posting, and fully identify yourself if you do. We will delete comments containing advertising, astroturfing, trolling, personal attacks, offensive language, or other objectionable content, then ban and/or publicly identify violators. Wondering why we're talking about something other than iPods? Check the Archives: Backstage has been here and kicking it since 2004.

Commenting is not available in this section entry.
Sign up for the iLounge Weekly Newsletter

Email:

Recent News

Recent Reviews

Recent Articles

Sign up for the iLounge Weekly Newsletter

Email:

iLounge is an independent resource for all things iPod, iPhone, iPad, and beyond.
iPod, iPhone, iPad, iTunes, Apple TV, Mac, and the Apple logo are trademarks of Apple Inc.
iLounge is © 2001 - 2014 iLounge, Inc. All Rights Reserved. Terms of Use | Privacy Policy