What I recommend reading / skipping:

• Chapter 1: Introduction - Mostly motivation / first half.
• Chapter 2: Background - Read 2.4 - 2.6, and the rest only if unfamiliar with GPAI or calibration.
• Chapter 3: Evaluation as a Prediction Problem - Read fully, most important part.
• Chapter 4: Applications & Methods - Read or skim, but do pay attention to 4.3, especially on types of generalisation.
• Chapter 5: Language Model Score Predictors - Skip or skim.
• Chapter 6: Human Score Predictors - Skim
• Chapter 7: Conclusion - Read

Apart from the idea and some work on metrics at the beginning of the paper, all credits go to collaborators.

I do not recommend anyone reading this paper, which is why I do not have a free copy online. If you really want to read it, it is Chapter 4 of my PhD thesis.

Something easily missed is that the tasks get difficult quickly. Locality is just an obnoxiously difficult task. Just look at the human expectations of failure on the x-axis, for the easiest bin it is already 91%.

For addition, one of the easiest instances that GPT-4 v2 fails on is "12191 + 3187383000 =", which is in difficulty bin 1. This is quite easy, because there are no carry operations, but it looks scary and especially with tokenisation, its a challenge for LLMs.

There are actually no instances with the simplicity of "100 + 150 where" where the latest models fail. There probably are, but they were not included in our dataset. We should have been more explicit about this in the paper.

On the other hand, confirming our notion of difficulty, GPT-4 v2 also just fails on "Make the addition of 984733680321971 and 0."... There is a long way to go.