Skip to content

README.md

Latest commit

 

History

History
47 lines (35 loc) · 2.45 KB

README.md

File metadata and controls

47 lines (35 loc) · 2.45 KB
Raw

CodeNet — dataset provenance and terms

200 C programs with differential tests, used to score C→Rust translation by pass@1.

Contents

raw_data/{argv,scanf}/*.c            the reference C programs (92 argv + 108 scanf = 200)
generated_tests/{argv,scanf}/*.json  input / expected-output cases per program
test_tasks/{argv,scanf}/*.json       the test command per program
manifest.json                        problem list, metric and decoding settings

argv programs read their input from command-line arguments, scanf programs from stdin. The harness passes that convention explicitly to the translation prompt, so a model is told where input comes from instead of inferring it from the C source.

Expected outputs were produced by compiling and running the reference C program, so a problem "passes" when the Rust translation reproduces the C program's behaviour on those inputs.

Provenance

  • Derived from IBM Project CodeNet (~120 of the 200): original C submissions taken from Project CodeNet, redistributed with their source content intact under new filenames (codenet_argv_NNN.c / codenet_scanf_NNN.c). Some of these retain their original authors' code comments verbatim, including non-English comments and one reference URL — that is upstream content, not part of this benchmark's annotations.
  • Written for this benchmark (~80 of the 200): additional C programs authored with LLM assistance, then compiled with gcc and executed to verify their I/O behaviour before inclusion.
  • Tests (generated_tests/, test_tasks/): produced for this benchmark.

Terms

The Project CodeNet–derived portion is licensed under the Community Data License Agreement – Permissive – Version 2.0 (CDLA-Permissive-2.0), which permits redistribution and derivative data sets provided this notice travels with the data. Project CodeNet's own source code is separately licensed Apache-2.0.

Material added by this benchmark (the newly written programs, the tests, the manifest) is released under the repository's Apache-2.0 license — see ../LICENSE and ../NOTICE.

A note on absolute paths

Each test_tasks/*.json embeds an absolute path to its generated_tests counterpart. They ship with a /__DATASET__/... placeholder; run bash ../fix_paths.sh once per checkout to rewrite them to your location. Without that step every problem fails with Invalid test command.