Dangerous capabilities and risk determinations

KeyVerifiedUnverifiedDisputedCorrected* read off a figure or stated in wordsA value opens its source and history.

• Each of the 90 evaluations is its own test, with its own tasks, graders and definitions. Values from different evaluations describe the same property measured in different ways, and are not comparable with one another, even where two evaluations share a name.
• Values come from the documents of six developers. Developers use different tests, prompts and graders for the same property, so we do not rank them on these numbers: tables here are ordered by name or by kind of test, never by a reported value.
• Some names are used by more than one evaluation: “Cyber Range” by two, “CyberGym” by two, “Cybench” by three, “VCT” by two, “ExploitBench” by two, “CVE-Bench” by two, “CyScenarioBench” by two and “ProtocolQA Open-Ended” by two. They are different tests, run by different evaluators or reported by different developers, so this page names who ran each one.
• In eight evaluations the test changed between documents (among them Frontier Safety Framework determination, Cyber challenges · Irregular and Cyber Range · OpenAI). Values on the old and new definitions are never joined: the explorer breaks the line where the test changed.
• In 38 evaluations the explorer connects results from different documents that do not say whether the test stayed the same. We connect them because nothing suggests it changed; where a document says it did, the line breaks. Each value on such a line is marked “test assumed unchanged”.
• 32 evaluations report values under more than one condition, such as reasoning effort, safeguards or number of attempts. Each condition is kept as a separate series.
• Numbers in this family are printed in 14 units: percent, minutes, USD, counts, human-normalised score, scores from 0 to 1 and 8 others. We never convert one unit into another, so a chart shows one unit at a time.
• For some measures a higher value is worse and for others better, depending on whether the developer reports the behaviour or its absence. Each value’s record says which.
• 16 values are restatements: a later document reporting an earlier model again, sometimes with a different value. A restatement is kept beside the model’s own value, never in place of it.

No finding is about this family yet.