EvoCodeBench

Properties

axis: coding
verification: det-test
whatItTests: 275 function-level code generation tasks from 25 real-world repositories, annotated with reference code, dependencies and requirements, scored with Pass@k and Recall@k, and refreshed via an automatic pipeline from the latest repositories to resist data leakage.
saturation: open
patternLabFit: The dependency-aware Recall@k metric plus the evolving/anti-leakage update pipeline are a candidate for testing whether collaboration patterns hold up against a benchmark deliberately refreshed to defeat memorization. The paper evaluates single models only, so no collaboration-pattern discriminator is asserted.
source: EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories (arXiv:2404.00599)
notes: Distinct from repobench (line-level completion, static 2023 dataset) and livecodebench (competitive programming, not repo-derived): EvoCodeBench targets function-level generation anchored to real repo dependencies with a self-refreshing anti-contamination pipeline, the repo-code analog to LiveCodeBench's contamination defense.

axis: coding
verification: det-test
whatItTests: 275 function-level code generation tasks from 25 real-world repositories, annotated with reference code, dependencies and requirements, scored with Pass@k and Recall@k, and refreshed via an automatic pipeline from the latest repositories to resist data leakage.
saturation: open
patternLabFit: The dependency-aware Recall@k metric plus the evolving/anti-leakage update pipeline are a candidate for testing whether collaboration patterns hold up against a benchmark deliberately refreshed to defeat memorization. The paper evaluates single models only, so no collaboration-pattern discriminator is asserted.
source: EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories (arXiv:2404.00599)
notes: Distinct from repobench (line-level completion, static 2023 dataset) and livecodebench (competitive programming, not repo-derived): EvoCodeBench targets function-level generation anchored to real repo dependencies with a self-refreshing anti-contamination pipeline, the repo-code analog to LiveCodeBench's contamination defense.