INS-1714: Add DCTD Curated Datasets

[zdorman]

Add steps to process a curated TSV of datasets from the NCI Division of Cancer Treatment and Diagnosis Data Downloads through INS data packaging.

Use the final, processed TSV dctd_datasets_curated_clean.tsv for loading into INS.

[Copilot]

🟡 Changes recommended

Invalid TSV quote escaping corrupts a DCTD description in the final loading artifact.

3 open findings

• Escape embedded quotes in TSV description · New
• Add focused test for DCTD packaging · New
• Fix typo in dataset description · New

What changed in this PR

Adds curated DCTD datasets to the INS packaging workflow.

Changes:

• Adds DCTD configuration and packaging logic.
• Adds 13 curated and processed datasets.
• Updates documentation and dataset-type tests.

File	Description
README.md	Documents DCTD curation.
config.py	Defines DCTD schema and paths.
modules/​package_output_data.py	Packages DCTD datasets.
tests/​test_package_output_data.py	Tests DCTD type mapping.
data/​01_intermediate/​dctd/​2026-10-01/​dctd_datasets_curated.tsv	Adds curated source data.
data/​02_output/​dctd/​2026-10-01/​dctd_datasets_curated_clean.tsv	Adds processed loading data.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The integration and generated UUIDs are consistent, with only minor description copy edits noted.

0 open findings

3 resolved since last review

• Add focused test for DCTD packaging
• Escape embedded quotes in TSV description
• Fix typo in dataset description

Previously missed (4)

In code that hasn't changed since last review

Add missing article and hyphenate compound modifier

data/​01_intermediate/​dctd/​2026-10-01/​dctd_datasets_curated.tsv:8

The description is missing an article: “It is high-throughput amenable library” should read “It is a high-throughput-amenable library.” Correct the curated source and regenerate the clean TSV so the grammatical error is not loaded into INS.

Remove space before -supported compound modifier

data/​01_intermediate/​dctd/​2026-10-01/​dctd_datasets_curated.tsv:9

The space before “-supported” breaks the compound modifier in this user-facing description. Change “(OCCPR) -supported” to “(OCCPR)-supported” and regenerate the clean TSV.

Regenerate description after correcting missing article

data/​02_output/​dctd/​2026-10-01/​dctd_datasets_curated_clean.tsv:8

This generated description retains the missing article from the curated source: “It is high-throughput amenable library” should read “It is a high-throughput-amenable library.” Regenerate this file after correcting the intermediate TSV.

Regenerate artifact after fixing OCCPR-supported modifier

data/​02_output/​dctd/​2026-10-01/​dctd_datasets_curated_clean.tsv:9

The generated load artifact also contains the malformed “(OCCPR) -supported” modifier. Regenerate it after changing the curated source to “(OCCPR)-supported.”

🧠 Review effort: Balanced

[zdorman]

Low-priority issues noted in the last CoPilot review are valid finds but both are language decisions that reflect the source material. No change.