Fm/protected donor dcicutils

[aschroed]

Purpose

Add the dcicutils-side ProtectedDonor workbook analysis and selective transformation used by submitr.

Feature summary

• Supports the current protected-item sheets: Demographic, DeathCircumstances, FamilyHistory, MedicalHistory, and TissueCollection.
• Classifies plain Donor and ProtectedDonor references across actual workbook data rows.
• Allows mixed references and transforms only referenced Donors that still need conversion.
• Validates missing workbook Donors clearly.
• Accepts existing ProtectedDonors found in the workbook or through the portal.
• Leaves unreferenced Donor rows untouched and supports incremental transformation alongside existing ProtectedDonor sheets.
• Integrates with CustomExcel while keeping generic structured-data parsing separate.

Remediation updates

• Hardened transformed-workbook target protection so an earlier save cannot silently authorize a later overwrite of the same input/output pair.
• Preserved existing CustomExcel mapping behavior; mapped fields are materialized by the submitr data path rather than by the raw ProtectedDonor workbook transform.
• Added regression coverage for existing-target protection, repeated saves, and the relevant CustomExcel integration.
• Corrected the readthedocs declaration for dcicutils.submitr.donor_transformer.

Review focus

• Reference classification and incremental transformation behavior.
• Workbook and portal existence checks and error handling.
• Generated ProtectedDonor row contents and portal-unique fields.
• Existing-target safety and CustomExcel integration.
• Focused tests covering mixed, missing, portal-backed, already-transformed, and repeated-save cases.

The downstream submitr implementation and payload remediation are tracked in smaht-dac/submitr#48.

[coveralls]

Coverage Report for CI Build 32604258760

Coverage increased (+0.6%) to 75.19%

Details

• Coverage increased (+0.6%) from the base build.
• Patch coverage: 38 uncovered changes across 2 files (285 of 323 lines covered, 88.24%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
dcicutils/submitr/donor_transformer.py	272	241	88.6%
dcicutils/submitr/custom_excel.py	44	37	84.09%
Total (3 files)	323	285	88.24%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	15933
Covered Lines:	11980
Line Coverage:	75.19%
Coverage Strength:	0.75 hits per line

---

💛 - Coveralls

[aschroed]

Implemented and pushed generic JSON schema ordering fix in commit a61b3ac on fm/protected-donor-dcicutils.

Summary:

• Shared canonical Schema.type_name ordering between workbook sheets and multi-schema JSON top-level keys.
• Supplied order now reorders recognized JSON keys while preserving original key spelling and stable unknown-key order.
• No-order behavior, single-schema JSON behavior, and nested record field ordering remain unchanged; no ProtectedDonor-specific logic was added.

Tests:

• Focused ordering tests: 5 passed.
• StructuredDataSet + ProtectedDonor suites: 111 passed.
• flake8 on dcicutils/structured_data.py and test/test_structured_data.py: passed.

Adversarial review outcome:

• Challenged normalization, canonical-key collisions, unknown-key stability, no-order and single-schema paths, nested fields, workbook compatibility, and regression coverage.
• Added coverage for canonical spelling collisions and nested field order; all findings were addressed.