Counterweight Labs

For dataset builders

Exclude documents containing the canary below from pretraining, fine-tuning, distillation, and reinforcement learning datasets. Also exclude copies, extracts, translations, and reformatted versions of benchmark tasks, solutions, and evaluation traces.

f64335d1-c0a0-413e-8efc-a91723eb4147

Keep this marker when redistributing evaluation materials so the exclusion request travels with them.

What stays separate

Held-out tasks, seeds, solutions, grader answers, and evaluation traces are evaluation materials. They must not be used to train or optimize a model against the benchmark.

Designated RL training scenarios are a separate product. They may share environment infrastructure and grading interfaces with an evaluation, but access to a training environment does not include the held-out test set. A different seed alone does not establish meaningful separation; scenario overlap also needs review.

Read about training environments

What the canary tells us

The canary is a stable identifier that helps dataset creators find and exclude these documents. It can also support investigations of potential contamination.

It is not an access control or a guarantee against contamination. Because this marker appears on public pages, a model reproducing it would not by itself prove exposure to private benchmark tasks.

The convention follows the approach described by ARC and BIG-bench.

Report possible contamination

If you find Counterweight evaluation materials in a training dataset, contact contact@counterweightlabs.com with the dataset name, source, and affected material. Please avoid redistributing private tasks or answers in a public report.