With the recent addition of Jeannie Cho Lee MW as Wine Lister’s fourth partner critic, now seems a good opportunity to elucidate our approach to calculating Quality scores.

Wine Lister’s Quality scoring algorithm takes into account each critic’s rating scale and the relative generosity or severity of their scoring.

Vinous and Jeannie Cho Lee rate on a 100-point scale, whereas Jancis Robinson and Bettane+Desseauve rate on a 20-point scale.  But we also know that the former pair never (or very rarely) give a rating less than 70, while the latter never (or very rarely) give a rating less than 10.

The first step in our scoring algorithm is to put all ratings on a level playing field by rebasing them. We take each critic’s minimum and maximum wine ratings and spread these back out over our entire 100-point scale. The result is that, for example, a Bettane+Desseauve rating of 15 and a Jeannie Cho Lee rating of 85 both score 50 out of 100.

But it’s not quite that simple! We also know that the way the critics score is different. For example, we find that Jeannie Cho Lee tends to give fewer high scores than Vinous. Our algorithm takes into consideration these tendencies towards more or less generous ratings, and adjusts scores as necessary for a fair outcome. This is our score normalisation process.

Using nonparametric statistical techniques, for the same wine-vintages that the critics have rated, we look at each critic’s probability of awarding each score, and use the difference in probabilities to make necessary adjustments.

For example, we looked at the set of wine-vintages rated by both Jancis Robinson and Bettane+Desseauve, and found that wines are around 5% more likely to achieve a rating of 17 or higher from Bettane+Desseauve. Once normalised, this means that a wine such as Château Beauregard 2009, rated 17 by both Bettane+Desseauve and Jancis Robinson, achieves Wine Lister critic scores of 73 and 76 respectively. In other words, 17 is harder to get from Jancis Robinson than from Bettane+Desseauve. This is illustrated in the image above.

Once we’ve rebased and normalised the critics’ scores, we simply take an average of those scores, giving an equal rating to each of our four partner critics, each of whom represents one of the world’s key fine wine markets.

There’s still more… The Wine Lister Quality score also takes into consideration a wine’s ageing potential as defined by critics’ drinking windows – watch this space for a further explanation of how that works!