CodeContests

Properties

axis: coding
verification: det-test
whatItTests: A large-scale, temporally-split competitive-programming dataset (13,328 training / 117 validation / 165 test problems scraped from Codeforces plus mixed-in existing datasets) built for training and evaluating code generation systems like AlphaCode, with additional generated tests to sharply cut false-positive solution rates.
discriminates: [build-verify-reflect]
saturation: saturated
patternLabFit: The dataset's generated-test filtering to reduce false positives (62% to 4%) is itself a build-verify-reflect precedent at the dataset-construction level; AlphaCode's own large-scale-sampling-plus-filtering pipeline is a single-agent generate-and-filter approach, not a tested multi-agent collaboration pattern.
source: Competition-Level Code Generation with AlphaCode (arXiv:2203.07814)
notes: Distinct from livecodebench (one-time temporal split with generated tests to fix false positives vs. LiveCodeBench's continuously rolling post-cutoff window) and usaco (much larger scale, 13k+ train problems, built as a training resource as well as an eval set). Marked saturated: AlphaCode-era solve rates (per the abstract, top 54.3% ranking in 2022) have been far surpassed by modern frontier models on Codeforces-style problems, though no specific current-model figure is asserted here since it is not confirmed in this paper.

axis: coding
verification: det-test
whatItTests: A large-scale, temporally-split competitive-programming dataset (13,328 training / 117 validation / 165 test problems scraped from Codeforces plus mixed-in existing datasets) built for training and evaluating code generation systems like AlphaCode, with additional generated tests to sharply cut false-positive solution rates.
discriminates: [build-verify-reflect]
saturation: saturated
patternLabFit: The dataset's generated-test filtering to reduce false positives (62% to 4%) is itself a build-verify-reflect precedent at the dataset-construction level; AlphaCode's own large-scale-sampling-plus-filtering pipeline is a single-agent generate-and-filter approach, not a tested multi-agent collaboration pattern.
source: Competition-Level Code Generation with AlphaCode (arXiv:2203.07814)
notes: Distinct from livecodebench (one-time temporal split with generated tests to fix false positives vs. LiveCodeBench's continuously rolling post-cutoff window) and usaco (much larger scale, 13k+ train problems, built as a training resource as well as an eval set). Marked saturated: AlphaCode-era solve rates (per the abstract, top 54.3% ranking in 2022) have been far surpassed by modern frontier models on Codeforces-style problems, though no specific current-model figure is asserted here since it is not confirmed in this paper.