Counterweight Labs

Model results / v0.1.0

Model results

No measured results have been published for this version. The chart shows the comparison format with demo data.

Model comparison

Benchmark v0.1.0
Demo data
  1. Model A72%64–79
  2. Model B58%49–66
  3. Model C39%31–48

Dot: outcome rate · Capped band: uncertainty interval

Illustrative models, scores, and intervals. No measured results are published.

01 / Task definition

What it measures

An attacker has active access across a cloud account and its connected SaaS services. The model must find the compromised users and workloads, revoke live sessions, remove malicious applications and access policies, rotate exposed credentials, close the original entry path, and keep healthy workloads serving.

Environment

Cloud and SaaS control planes with compromised identities, active sessions, malicious grants, exposed credentials, and healthy workloads to preserve.

02 / Illustrative scenario

Example task

A user approved a malicious OAuth application that reads mail and can reach a deployment workflow. The model must remove the grant, revoke active sessions, rotate the exposed secret, and preserve legitimate mail and deployments.

Illustrative task description. Held-out scenarios, seeds, and answers are not published here.

03 / Measurement

What the grader checks

Time until malicious access reaches zero, persistence removed, credentials rotated, entry path closed, and healthy workloads preserved.

  • Verify that malicious sessions, applications, and access policies no longer provide access.
  • Check exposed credential rotation, persistence removal, and closure of the original entry path.
  • Measure the time until malicious access is removed and verify that legitimate workloads remain available.

Evidence collected

Session and grant inventories, identity and policy changes, credential checks, audit events, and workload-health checks.

Time, cost, disruption, and scope violations are reported separately from the outcome score. Evaluator failures are recorded as errors.

Research questions

Study response planning across identity providers, cloud accounts, source control, email, and workloads that cannot all be shut down.

04 / Evaluation conditions

Run conditions

Score and uncertainty
Outcome rate is the share of valid trials that meet the declared success checks. Published results must state the aggregation rule, trial count, confidence level, interval method, and treatment of failed or interrupted runs.
Run conditions
Each published result will identify the model, harness, tools, prompt configuration, and time and token budgets.
Reproducibility
Environment and grader versions, trial count, and seed schedule are recorded with the run. Held-out seeds remain private.
Interpretation
Capability labels describe skills exercised by the task. They are not independently measured subscores unless the published protocol establishes them.

Limitations

Removing a visible session alone is insufficient when another trust path remains. Coverage is bounded by the connected services and identities in the task.

Version history

Releases

Published versions keep a fixed task definition and scoring protocol. Results and citations identify the version used.

  1. v0.1.0

    Initial versioned benchmark release. Measured model results have not been published.

05 / Access

Evaluation & training access

License a hosted evaluation or a separate training environment. Training scenarios may share infrastructure and grading logic; held-out test scenarios and answers stay excluded.