Counterweight Labs
01

Define the task

Specify the objective, starting access, allowed tools, operational constraints, and time and token budgets. The task definition sets what a successful run must demonstrate.

02

Run in an instrumented environment

Give the agent a real system to act on. Record tool use and changes to the environment, with reproducible setup and a grader isolated from the agent.

03

Verify the outcome

Check the resulting system state against predefined checkpoints. Report success alongside cost, service disruption, and scope violations. Preserve evaluator errors as a separate outcome.

04

Publish the context

Report the model and harness configuration, environment and grader versions, trial count, and run budgets. Publish measured results with uncertainty and limitations supported by the actual trials.

Reading the catalog

How benchmarks are organized

Side

Offense or defense describes the agent's objective. It does not identify who might use the evaluation.

Capability

Intrusion operations, incident response, or recovery describes the work being tested.

Environment

Enterprise, cloud and SaaS, or software systems describe where that work happens.

Capability labels describe coverage. Individual skills are only reported as subscores when they are independently measured.

Interpretation

Comparing results

Normalizing two scores to 0–100 does not make the tasks equally difficult. Compare runs within a benchmark version and under matched conditions; cross-track comparisons need evidence of calibration.

Repeated trials help distinguish reliable performance from a lucky outcome. Refusals, scope violations, grader failures, and infrastructure errors should remain visible rather than disappearing into one average.

Training and evaluation scenarios must remain separate. The same infrastructure can support both, but the held-out test set must stay outside the optimization loop.

Read the benchmark data policy

Useful references

Evaluation references