Comparing Different Ways to Calculate Accuracy

Comments on https://lichess.org/@/jk_182/blog/comparing-different-ways-to-calculate-accuracy/6aeYnbh8

Thanks for the interesting post.

The issue with rank-based scores (the percentile of master-level classical games with the same centipawn loss) is they amplify small differences in an objective metric. So if players play out a Berlin draw, then suffer very small centipawn loss, while a spicier game with sharp tactics and sacrifices will have a much larger centipawn loss. This is already a feature of the present metrics, but using a distribution function and extracting ranks, the spread will be amplified.

Additionally, clearly a system which would cause a spread in master-classical games to have accuracies from 0 to 100 with an even distribution would cause accuracies of the average Lichess rapid or blitz player to get dragged into the mud. An accuracy of 1 would be a solid game, with 0 more likely. One could use a player's rating to establish a cohort, but accuracy should be independent of player rating. You shouldn't need to know who the players are to objectively evaluate the game.

If I was trying to improve accuracy metrics, one thing I would like to consider is a quantitative assessment of the complexity of the position. There have been algorithms proposed for this. The limiting case is a forced move. That's the simplest possible complexity. Defending against Eric Rosen playing the Stafford Gambit would be closer to the opposite extreme.

A test of accuracy methods is how they correlate with game result. There should clearly be a correlation but the correlation shouldn't be perfect. For example, a player may systematically outplay an opponent only to blunder a fork. One move shouldn't dominate the entire score, not matter how poor.

Nice article, It gave a lot of useful insight for a project I am working on. I am trying to create a Firefox extension that takes Lichess game review and calculates a game score (similar to chess.com) This has proven to be a lot more difficult than I originally anticipated. Do you know much about CAPS2 algorithm? If so, I would love to hop on a call with you and ask you a few questions, and maybe play a game of chess while doing so. If not its all good.

Also,
@djconnel said ^

You shouldn't need to know who the players are to objectively evaluate the game.

Do you agree with this statement? Do you think its possible to calculate an estimated elo score for a game without any player elo information? If so, what accuracy system would you use.

I wish lichess would correct their logic to use WDL probabilities calculated by Stockfish.

The whole article on lichess that describes accuracy and its formula are wrong because they rest on the wrong assumption that SF eval is in centi pawns. It is NOT.

SF eval has been normalised for years. The function to calculate WDL proba from score is even coded in SF. You can ask SF to output directly the probabilities instead of a meaningless score (in arbitrary units).

Also, on github, they have another repo, used to recalibrate the model from time to time. There are some very smart people working on it, smarter than me and almost everyone here. It would be better to use this instead of an outdated formula calibrated years ago, based on assumptions that have become more and more wrong now.

See for yourself if you don't want to take my word for it:

https://github.com/official-stockfish/WDL_model

https://github.com/official-stockfish/Stockfish/blob/6088838797d6333711c17fe2c0962fa0858517ec/src/uci.cpp#L503

Also, when analysing with the engine, I wish lichess showed WDL proba, instead of SF eval. Or perhaps E(Score)=P(win)+0.5*P(draw) to reduce it to a single number.

Regarding the instability of engine evaluations: Wouldn't it be possible to "just" run stockfish on every board you could get to with 1 ply, choose the board with the highest eval as the engine's preferred move, and compare the move that was actually played to that? I get that it would require running the engine on an average of around 30 boards instead of 1, but with transpositions and all, it probably wouldn't be a full 30x slowdown, and we're certainly not talking hours of CPU time per position, as said in the blog post.

And since you're hardly ever gonna find that the move ranked 10th gets a better evaluation than the top move, you could probably get away with only getting stockfish to output its 3 top moves, then using the eval generated looking ahead from that original position to calculate accuracy, if one of the top 3 moves is chosen.

Lastly, maybe evaluating the position from which a move was played at e.g. depth 20 and the position that follows at depth 19 yields more stable results? I'm pretty certain that the exact results would not be equivalent to this "evaluating all possible moves and the boards that arise from them" brute-force approach, with all the pruning that stockfish does, but it might be worth a shot, given that it would be fairly economical in the grand scheme of things.

It's been years since I last played around with stockfish, so feel free to tell me that I've just got no clue what I'm talking about :D

@djconnel said ^

Additionally, clearly a system which would cause a spread in master-classical games to have accuracies from 0 to 100 with an even distribution would cause accuracies of the average Lichess rapid or blitz player to get dragged into the mud. An accuracy of 1 would be a solid game, with 0 more likely. One could use a player's rating to establish a cohort, but accuracy should be independent of player rating. You shouldn't need to know who the players are to objectively evaluate the game.

Personally, I prefer the granularity of values that are tailored to specific rating ranges and time controls. If we wanted one metric for all levels of play, we'd need to range from bullet games between beginners to correspondence games with engine use, which seems like a huge range.
In general, my goal was to give the accuracy value some meaning and I think that the meaning comes from comparisons and not from the most objective measure.

For example, a player may systematically outplay an opponent only to blunder a fork. One move shouldn't dominate the entire score, not matter how poor.

I guess that this is a somewhat philosophical question. Of course, we want to measure the accuracy of the whole game, but if one move has a larger impact on the outcome, it should also have a larger impact on the score.

@Br1ckedJon said ^

Nice article, It gave a lot of useful insight for a project I am working on. I am trying to create a Firefox extension that takes Lichess game review and calculates a game score (similar to chess.com) This has proven to be a lot more difficult than I originally anticipated. Do you know much about CAPS2 algorithm? If so, I would love to hop on a call with you and ask you a few questions, and maybe play a game of chess while doing so. If not its all good.

As far as I know, there isn't much public knowledge about the chesscom algorithms (unfortunately, not all chess platforms are open-source). The only information I could find is that they cluster the scores around 80, so that players don't get discouraged from seeing low values

@BananaBeaver said ^

I wish lichess would correct their logic to use WDL probabilities calculated by Stockfish.

The whole article on lichess that describes accuracy and its formula are wrong because they rest on the wrong assumption that SF eval is in centi pawns. It is NOT.

SF eval has been normalised for years. The function to calculate WDL proba from score is even coded in SF. You can ask SF to output directly the probabilities instead of a meaningless score (in arbitrary units).

I'm not sure how well the Stockfish WDL would work, as it reports the winning chances from Stockfish's point of view. So, it overestimates the winning chances in better positions, as Stockfish is better at converting these positions than humans.
Lichess uses their own win percentage function, which they derived from games on their site, and I think that this is the best way to do this.

@jk_182 said ^

Lichess uses their own win percentage function, which they derived from games on their site, and I think that this is the best way to do this.

As author of Lichess' code and someone following Stockfish development, it wouldn't surprise me if changes after SF renormalized its evaluation function could merit changing my function. Anyone could do the math on this...

@kcitkcab said ^

Regarding the instability of engine evaluations: Wouldn't it be possible to "just" run stockfish on every board you could get to with 1 ply, choose the board with the highest eval as the engine's preferred move, and compare the move that was actually played to that?

But you still have the problem that the evaluations of each move change as the depth of the engine increases.

Lastly, maybe evaluating the position from which a move was played at e.g. depth 20 and the position that follows at depth 19 yields more stable results? I'm pretty certain that the exact results would not be equivalent to this "evaluating all possible moves and the boards that arise from them" brute-force approach, with all the pruning that stockfish does, but it might be worth a shot, given that it would be fairly economical in the grand scheme of things.

This may be better for a fixed depth, but then there still remains the question if the evaluation shifts when analysing with a depth of 25, 30 or 40.

My main problem is that players may play the moves that Stockfish expects, but as the game moves down the line, Stockfish will be able to analyse the resulting position more deeply and thus may change the evaluation, even though the players played the "correct" moves.