Side
Offense or defense describes the agent's objective. It does not identify who might use the evaluation.
Evaluation methodology
A cyber score needs a task, a threat model, and evidence. Our evaluation design makes those conditions explicit.
Specify the objective, starting access, allowed tools, operational constraints, and time and token budgets. The task definition sets what a successful run must demonstrate.
Give the agent a real system to act on. Record tool use and changes to the environment, with reproducible setup and a grader isolated from the agent.
Check the resulting system state against predefined checkpoints. Report success alongside cost, service disruption, and scope violations. Preserve evaluator errors as a separate outcome.
Report the model and harness configuration, environment and grader versions, trial count, and run budgets. Publish measured results with uncertainty and limitations supported by the actual trials.
Reading the catalog
Offense or defense describes the agent's objective. It does not identify who might use the evaluation.
Intrusion operations, incident response, or recovery describes the work being tested.
Enterprise, cloud and SaaS, or software systems describe where that work happens.
Capability labels describe coverage. Individual skills are only reported as subscores when they are independently measured.
Interpretation
Normalizing two scores to 0–100 does not make the tasks equally difficult. Compare runs within a benchmark version and under matched conditions; cross-track comparisons need evidence of calibration.
Repeated trials help distinguish reliable performance from a lucky outcome. Refusals, scope violations, grader failures, and infrastructure errors should remain visible rather than disappearing into one average.
Training and evaluation scenarios must remain separate. The same infrastructure can support both, but the held-out test set must stay outside the optimization loop.
Useful references