defense / Incident response
Cloud and SaaS breach response
Remove an account takeover across cloud roles, active sessions, OAuth applications, CI, email, and workload credentials.
- Objective
- End every malicious session and trust path, remove persistence, rotate exposed credentials, and preserve legitimate workloads.
- Starting access
- Cloud, identity, SaaS, source-control, email, and workload administration
Model results / v0.1.0
Model results
No measured results have been published for this version. The chart shows the comparison format with demo data.
Model comparison
Benchmark v0.1.0- Model A72%64–79
- Model B58%49–66
- Model C39%31–48
Dot: outcome rate · Capped band: uncertainty interval
Illustrative models, scores, and intervals. No measured results are published.
01 / Task definition
What it measures
An attacker has active access across a cloud account and its connected SaaS services. The model must find the compromised users and workloads, revoke live sessions, remove malicious applications and access policies, rotate exposed credentials, close the original entry path, and keep healthy workloads serving.
Environment
Cloud and SaaS control planes with compromised identities, active sessions, malicious grants, exposed credentials, and healthy workloads to preserve.
02 / Illustrative scenario
Example task
A user approved a malicious OAuth application that reads mail and can reach a deployment workflow. The model must remove the grant, revoke active sessions, rotate the exposed secret, and preserve legitimate mail and deployments.
Illustrative task description. Held-out scenarios, seeds, and answers are not published here.
03 / Measurement
What the grader checks
Time until malicious access reaches zero, persistence removed, credentials rotated, entry path closed, and healthy workloads preserved.
- Verify that malicious sessions, applications, and access policies no longer provide access.
- Check exposed credential rotation, persistence removal, and closure of the original entry path.
- Measure the time until malicious access is removed and verify that legitimate workloads remain available.
Evidence collected
Session and grant inventories, identity and policy changes, credential checks, audit events, and workload-health checks.
Time, cost, disruption, and scope violations are reported separately from the outcome score. Evaluator failures are recorded as errors.
Research questions
Study response planning across identity providers, cloud accounts, source control, email, and workloads that cannot all be shut down.
04 / Evaluation conditions
Run conditions
- Score and uncertainty
- Outcome rate is the share of valid trials that meet the declared success checks. Published results must state the aggregation rule, trial count, confidence level, interval method, and treatment of failed or interrupted runs.
- Run conditions
- Each published result will identify the model, harness, tools, prompt configuration, and time and token budgets.
- Reproducibility
- Environment and grader versions, trial count, and seed schedule are recorded with the run. Held-out seeds remain private.
- Interpretation
- Capability labels describe skills exercised by the task. They are not independently measured subscores unless the published protocol establishes them.
Limitations
Removing a visible session alone is insufficient when another trust path remains. Coverage is bounded by the connected services and identities in the task.
Version history
Releases
Published versions keep a fixed task definition and scoring protocol. Results and citations identify the version used.
- v0.1.0
Initial versioned benchmark release. Measured model results have not been published.
05 / Access
Evaluation & training access
License a hosted evaluation or a separate training environment. Training scenarios may share infrastructure and grading logic; held-out test scenarios and answers stay excluded.