Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study

ADE — agents that develop data selection, reward and curriculum strategies through controlled experiments.

🏠 Homepage · 🚀 Quickstart · 🧭 Walkthrough · 📚 Documentation · 📄 Paper

LLM data engineering requires iterative experimentation on data selection, rewards, and curricula. Agentic Data Engineering (ADE) lets agents develop, revise, and combine strategies using experimental feedback and cumulative findings under fixed training and evaluation protocols. Across four tasks—math and code data selection, math reward design, and math curriculum learning—selected strategies achieve absolute gains over generic baselines of up to 13.34% on AIME25 Pass@32 and 5.85% averaged across four held-out benchmarks. In-loop scores improve over successive experiments, but later gains do not always translate into better held-out performance.

Start here: Install → Prepare inputs → Configure deployment → Give an Operation Prompt to the operator agent.

Framework

Agents develop strategies from Experiment Packages and Memory; the Engine executes controlled experiments. An external operator agent handles operation, monitoring and recovery using a repository Skill.

A Run first evaluates the base model and trains a baseline. ADE then proposes a Plan, implements its strategy, trains and evaluates a Trial, and records the analysis in Memory. Later Plans use that evidence to choose their next intervention. Model, training budget and evaluation rules stay fixed within a Run.

Quickstart

Every real Run starts with an Operation Prompt + an exact config, handed to one operator agent. ade-supervise-run owns admission, startup, monitoring, operational recovery and terminal acceptance. ADE launches its research-role workers internally.

1. Install the complete runtime with bash scripts/recreate_unified_vllm_env.sh after reading the prerequisites.
2. Prepare models/data, the local deployment and assigned Ray nodes using the walkthrough.
3. Fill baseline-operation.md with your checkout, environment, deployment and resource authorization; give it to the operator agent.
4. After baseline acceptance, fill ade-operation.md with its actual Run ID and hand off the separate N=1 ADE.

Use $ade-supervise-run to execute the completed Operation Prompt in
<absolute path to your local operation.md> through terminal acceptance.

The example directory provides Baseline, ADE N=1 and ADE N=3 prompts with their configurations. Prepare inputs and resources according to the selected config, then follow the Quickstart to launch and inspect results.

Want to inspect the workflow without models or GPUs? The optional CPU demo uses scripted responses and synthetic scores; it is separate from the supervised real experiment.

Tasks

Task	Agent-controlled strategy	Backend	Available domains
Data Selection	Selection code over a fixed candidate pool	LlamaFactory SFT	Math, code
Reward Design	Reward implementation	VERL RFT	Math
Curriculum Learning	Training-data schedule	VERL RFT	Math

The configuration matrix contains baseline, N=1 and N=3 entries. Each Run freezes its task contract, inputs, training/evaluation settings and ADE budget. Read results for Trial Records, rankings and Memory. See operator test and final generalization for held-out datasets and the final evaluation workflow.

Documentation

Guide	Documentation
Start with an operator agent	Quickstart
Operation Prompt → baseline → ADE → results	Walkthrough
Runtime and dependencies	Installation
Models and datasets	Inputs
Cluster and resources	Deployment · Ray
Operation and process ownership	CLI · Services
Implementation and extension	Components
Evidence, scores and Memory	Results
Help	FAQ

Repository

• ade/: Harness/Control, research Agents, Engine, tasks, evaluation and Memory.
• .agents/skills/: operator supervision and research-role Skills.
• examples/, configs/, prompts/: Operation Prompts, fixed experiment configs and model prompts.
• docs/: usage and component guides.
• scripts/, dataset/, requirements/: environment and input preparation.
• third_party/llamafactory/, third_party/verl/: included training backends.

See AGENTS.md for contributor-Agent instructions.

License

ADE is licensed under the MIT License. Third-party components retain their respective license files.

Citation

@misc{ding2026agenticdataengineeringllms,
  title={Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study},
  author={Ruomeng Ding and Qianli Shen and Zhaoyang Han and Meixin Chen and Daoyuan Chen and Yaliang Li},
  year={2026},
  eprint={2610.14238},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.14238},
}

The same entry is available in CITATION.bib.