Evaluation integrity
Keep the test
out of the training set.
A benchmark is useful only when it measures capability on tasks the model has not trained on. This policy explains how to treat Counterweight evaluation materials.
For dataset builders
Exclude documents containing the canary below from pretraining, fine-tuning, distillation, and reinforcement learning datasets. Also exclude copies, extracts, translations, and reformatted versions of benchmark tasks, solutions, and evaluation traces.
f64335d1-c0a0-413e-8efc-a91723eb4147Keep this marker when redistributing evaluation materials so the exclusion request travels with them.
What stays separate
Held-out tasks, seeds, solutions, grader answers, and evaluation traces are evaluation materials. They must not be used to train or optimize a model against the benchmark.
Designated RL training scenarios are a separate product. They may share environment infrastructure and grading interfaces with an evaluation, but access to a training environment does not include the held-out test set. A different seed alone does not establish meaningful separation; scenario overlap also needs review.
What the canary tells us
The canary is a stable identifier that helps dataset creators find and exclude these documents. It can also support investigations of potential contamination.
It is not an access control or a guarantee against contamination. Because this marker appears on public pages, a model reproducing it would not by itself prove exposure to private benchmark tasks.
The convention follows the approach described by ARC and BIG-bench.
Report possible contamination
If you find Counterweight evaluation materials in a training dataset, contact contact@counterweightlabs.com with the dataset name, source, and affected material. Please avoid redistributing private tasks or answers in a public report.