Monday, March 14, 2016

Deceptive 5 stars rating systems

I noticed something funny, and illuminating, when browsing the Apple App Store. One game had this kind of rating:

In other words, 2.5 stars (which is even stated as text as well). This would indicate that the ratings are split pretty evenly. 50% approval rate.

However, the game had four 1-star ratings and two 5-star ratings. 1 star is the minimum rating, and 5 stars is the maximum.

Wait... That doesn't make any sense. That's not an even split. There are significantly more 1-star ratings than 5-star ratings (in fact, double the amount). That's not even close to an even split. Four people rated it at 1 star, and only two at 5 stars.

Is the calculation correct? Well, the weighted average is (4*1+2*5)/(4+2) = 2.333 ≈ 2.5 (we can allow rounding to the nearest half star.)

So the calculation is correct (allowing a small amount of rounding). It is indeed 2.5 stars. The graphic is correct.

But it still doesn't make any sense. How can 4x1 star + 2x5 stars give an even split? That's not possible. There are way more 1-star ratings than 5-star ratings. It can't be an even split! What's going on here?

The problem is that the graphic is misleading. The minimum vote is 1 star, not 0 stars. (If there were a possibility of 0-star ratings, then the graphic would actually be correct.)

The graphic becomes more intuitive if we remove the leftmost star:

Now it looks more intuitive. Now it looks like it better corresponds to the 4x1 - 2x5 split. In other words, a bit less than 50% rating.

Or to state it in another way: The problem is that the leftmost star is always "lit", regardless of what the actual ratings are, which gives a misleading and confusing impression.

In reality the rating system should be thought of as being in the 0-4 range (rather than 1-5), with only four stars, and the possibility of none of them being "lit". Then it becomes more intuitive and gives a better picture of how the ratings are split.

As it is, with a range of 1-5, with the leftmost star always "lit", it gives the false impression of the ratings being higher than they really are. I don't know if they do this deliberately, or if they just haven't thought of this.

No comments:

Post a Comment