feat(workflow): add cohort-based record retry

[nabinchha]

📋 Summary

This PR implements the cohort-based workflow retry design as an explicit alternative to the engine-native record-selection implementation in #817. It keeps a stable logical cohort, reruns only rejected slots, and publishes one ordinary durable stage artifact after the retry bounds are satisfied or exhausted.

The intent is comparison, not replacement: #817 should remain open while we evaluate the complexity and behavior of both approaches.

🔗 Related Issue

Related to #790. Alternative design to #817; this PR intentionally does not close either one.

🔄 Changes

✨ Added

• Add the full design and decision record at plans/790/cohort-based-record-retry.md, including prior design attempts, architecture diagrams, artifact layout, crash boundaries, limitations, and validation cases.
• Add RetryUntil to CompositeWorkflow.add_stage() with:
  • max_attempts and/or max_candidate_records bounds;
  • sampler_retry_mode="preserve" by default, with an explicit "resample" option;
  • on_exhausted="raise" or "return_partial" behavior.
• Add a workflow-level cohort runner with stable slot IDs. Accepted slots are never retried; rejected or null-predicate slots retain seed and, by default, sampler values while generated and expression columns are rerun.
• Add strict Boolean/null predicate classification for Boolean expressions and runtime-defined custom/plugin predicates. Structured and nested outputs must feed a Boolean expression. The built-in LLM-text generator returns strings, so direct LLM-text values require a Boolean expression in practice.
• Add immutable per-attempt inputs, accepted partitions, completion markers, canonical metadata, model-usage aggregation, and strict ALWAYS versus restartable IF_POSSIBLE resume behavior.
• Snapshot referenced local seed media with the base cohort, package accepted generated media under the canonical stage artifact, and remove internal retry identity columns before user processors run.
• Add explicit partial and zero-row terminal publication, while preserving typed early-shutdown and generation failures.

🔧 Changed

• Split the public DataDesigner.create() lifecycle from a private workflow seam so hidden base/attempt runs can skip profiling and the final accepted artifact can run the original processors/profilers exactly once.
• Keep CohortRetryRunner focused on orchestration (447 lines, down from 1,363). Durable Pydantic state and deterministic paths live in cohort_retry_state.py; classification, coalescing, media, usage, metadata, and resume checks are public stateless functions in cohort_retry_utils.py so they can be tested directly.
• Derive attempt paths, sampler-execution behavior, the fixed base path, and partial-result warnings instead of persisting redundant manifest fields.
• Build seedless cohorts as a direct N-row stable-slot frame when no seed or preserved sampler needs materialization, avoiding a synthetic UUID column and an unnecessary base generation run.
• Persist enough profiler and model-usage metadata to reconstruct completed workflow results without rerunning them.
• Prefer final profiler row counts in Hugging Face cards, with actual_num_records as the fallback for unprofiled artifacts such as a zero-row retry result.

⚖️ Comparison notes

• This keeps retry policy out of the engine scheduler and builds on ordinary workflow stages and DataDesigner.create() runs.

• Library LOC below counts files under packages/*/src/** and excludes Markdown, tests, plans, documentation, notebooks, and generated artifacts:

  Approach	Additions	Deletions	Net library LOC	Library files changed
  Engine-native selection (#817)	2,045	140	+1,905	23
  Workflow cohort retry (this PR)	2,249	105	+2,144	11

• The workflow implementation is therefore 239 net library LOC larger (+12.5%) than feat: add engine-native record selection #817: 204 more additions and 35 fewer deletions. Its structural distinction is concentration in the workflow/interface layer, an orchestration-only runner, and no selection-specific engine scheduler behavior—not a reduction in total library code.

• The current v1 materializes cohort and coalescing frames in pandas, retains attempt artifacts for auditability, and serializes attempts because each attempt depends on the prior acceptance result.

• Terminal processors must preserve row count. Generic plugin/external-media transactions remain outside the defined contract.

🧪 Testing

• Full package suite: 4,105 passed, 1 skipped
• Coverage: 92% (repository requirement: at least 90%)
• Cohort retry and composite-workflow focused suite: 205 passed
• Interface Ruff format/lint, pre-commit hooks, and repository license-header checks pass
• Unit tests added/updated
• E2E tests added/updated (not applicable; no external model/service dependency is required for this implementation)

✅ Checklist

• Follows commit message conventions
• Commit is signed off (DCO)
• Architecture plan updated

[github-actions]

Fern preview: https://nvidia-preview-pr-830.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.