Reward hacking

KeyVerifiedUnverifiedDisputedCorrected* read off a figure or stated in wordsA value opens its source and history.

• Each of the 13 evaluations is its own test, with its own tasks, graders and definitions. Values from different evaluations describe the same property measured in different ways, and are not comparable with one another, even where two evaluations share a name.
• Values come from the documents of four developers. Developers use different tests, prompts and graders for the same property, so we do not rank them on these numbers: tables here are ordered by name or by kind of test, never by a reported value.
• In three evaluations the test changed between documents (Impossible tasks, Reward-hack-prone coding tasks and Training distribution). Values on the old and new definitions are never joined: the explorer breaks the line where the test changed.
• In two evaluations the explorer connects results from different documents that do not say whether the test stayed the same. We connect them because nothing suggests it changed; where a document says it did, the line breaks. Each value on such a line is marked “test assumed unchanged”.
• Two evaluations report values under more than one condition, such as reasoning effort, safeguards or number of attempts. Each condition is kept as a separate series.
• Numbers in this family are printed in three units: percent, counts and relative change in percent. We never convert one unit into another, so a chart shows one unit at a time.
• For some measures a higher value is worse and for others better, depending on whether the developer reports the behaviour or its absence. Each value’s record says which.
• One value is a restatement: a later document reporting an earlier model again, sometimes with a different value. A restatement is kept beside the model’s own value, never in place of it.

No finding is about this family yet.