Burntide Official Website › Blog › Reading a star rating

Reading A Star Rating: Why 4.7 Out Of 5 Is Not One Number

Reviewed by Dr. Chioma Okafor-Mbah, MD

What an average actually compresses

A mean takes every opinion in a set and returns one number. That is the whole job, and it is genuinely useful: it lets you compare two things at a glance without reading anything. What it costs is every piece of structure in the data.

For a supplement that structure is the interesting part, because the question a buyer is actually asking is not “how good is this on average”. It is “what are the chances I end up in the group that was disappointed, and what were they disappointed about”. An average cannot answer either half of that.

• It hides the spread: whether opinion is clustered or split.
• It hides the tail: how many people were actively unhappy.
• It hides the reason: what the unhappy group was unhappy about.
• It hides the sample: whether this is twenty people or two thousand.
• It hides the timing: whether reviews were written on day three or day fifty.

The same 4.7, two different products

The clearest way to see the problem is to build two distributions that produce the same average and describe two quite different purchases.

Those two figures are illustrative rather than measured, and they are deliberately rounded. The point is structural: product B carries a real chance of a bad experience and product A does not, and the headline number is identical.

Whichever of those a buyer would prefer, they cannot choose between them without the breakdown. So the first question to put to any review page is not about the score at all. It is whether the five numbers underneath it are published.

The number under the number

The second question is the sample, and it has two halves that are usually collapsed into one.

The first half is how many people wrote. A 4.7 from twenty reviews and a 4.7 from two thousand are different claims about the world, and only one of them would survive another fifty reviews arriving.

The second half is almost never published: how many people bought and did not write. This website publishes it, because the arithmetic is unflattering and interesting. There are 986 written reviews across 7,197 orders, which is roughly one buyer in seven.

Six buyers in seven said nothing. Some were happy and busy. Some stopped in week two and moved on. No review page in existence can tell you the split, and the group least likely to write anything is the group that quietly gave up, which means every aggregate in this category is drawn from a sample that over-represents people who kept going.

What a recommend rate adds

A recommend rate is a second measure of roughly the same thing, asked as a yes or no. 91 per cent of the reviewers behind this product said they would recommend it.

On its own that adds very little. Where it earns its place is when it disagrees with the distribution, because the two are measuring slightly different questions. A star score is a judgement of the experience. A recommend rate is a judgement about somebody else, and people are noticeably more cautious about the second.

So compare them. In this record, 87 per cent gave four stars or five and 91 per cent would recommend. Those two figures are close, which is the unremarkable and reassuring outcome. A page showing 92 per cent five stars and a 61 per cent recommend rate would be showing you something worth asking about.

Rounding, and why 4.7 is not 4.65

A small technical point that changes how much precision to read into a headline.

A figure printed as 4.7 could be anything from 4.65 to 4.749, and on a sample of 986 reviews a single additional one-star review moves the mean by about a thousandth. That means the second decimal place carries no information a buyer can use, and comparing two products at 4.7 and 4.6 is comparing noise.

The same applies to the distribution. Percentages that total exactly 100 have been rounded to get there, and a figure printed as 2 per cent might be 1.6 or 2.4. None of that is dishonest; it is what rounding is. It is a reason to read a distribution for its shape rather than for its digits.

• Treat the first decimal place as meaningful and the second as decoration.
• Treat a 0.1 difference between two products as no difference.
• Treat a difference in the shape of two distributions as real.
• Treat any page quoting a score to two decimal places as trying too hard.

The same caution applies to an order count. A figure like 7,197 is a snapshot taken on the day the page was last reviewed, and it moves every week. What it is good for is scale, thousands rather than dozens. What it is not good for is comparison against another website’s figure, taken on a different day from a different desk and describing a different set of orders.

How to read a distribution in ten seconds

The practical version, which works on any review page in any category.

The second row is the one people miss. A product where almost nobody gave four stars is unusual, because four stars is what a satisfied person with one small reservation gives, and most satisfied people have one small reservation.

The FTC's consumer guidance covers the same ground from the fraud side, and the endorsement guides set out what a seller is allowed to do with a testimonial. The question-and-answer version is the most readable of the three.

Applied to this product

Running this website’s own rating through the checklist above, which is the only way a post like this is worth publishing on a seller’s domain.

Seven of those read yes and one reads no, and the one that reads no is the most important row in the table. The reviews page explains what was actually supplied and why it is printed as three themes rather than eight testimonials, and the post on spotting a fake review puts this website through the full seven-point test and fails three rows of it.

The NIH fact sheet for this category is the neutral thing to read afterwards. It has no interest in any product’s star rating and it is considerably less impressed by the whole aisle than any review page will be.