Skip to content

feat(ci): modernize ADO integration test pipeline - #912

Draft
Hangyi (HangyiWang) wants to merge 36 commits into
Azure:release/1.1.0-previewfrom
HangyiWang:users/hangyiwang/ado147-integration-parity-retries
Draft

Hangyi (HangyiWang) wants to merge 36 commits into
Azure:release/1.1.0-previewfrom
HangyiWang:users/hangyiwang/ado147-integration-parity-retries

Conversation

@HangyiWang

@HangyiWang Hangyi (HangyiWang) commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Since we're reusing ADO pipeline https://dev.azure.com/azureiotdevxp/aziotcli/_build?definitionId=147&_a=summary for FIC migration, this PR modernizes pipeline 147 / Azure IoT CLI - Integration Tests for DPS, HubControl, HubData, ADU and ADR, using the existing ADO service connection with renewable WIF and a simple Dry run / Integration tests form with Python, region and ARM-routing selection. The form defaults to Integration tests with the named DPS/Hub/ADR/ADU preset; clear the preset to select individual suites, or explicitly choose Dry run for planning only. Shipped CLI behavior, pipeline default branch, GitHub release/schedule callers and FICs are unchanged.

Build, Lint and Unit run in parallel, followed by parallel selected integration services against one immutable candidate wheel, with manual same-run recovery that preserves original results. Completed call failures rerun only the failed cases, including service/transport exceptions. When exact selection is unsafe or unavailable, the next manual attempt reruns the entire affected service/Python/region combination with fresh fixtures, including passing authentication phases. This covers fixture failures, incomplete execution, worker crashes, timeouts and installation/login failures. Full retries replace effective service results rather than hiding new failures behind old passes. Commit, candidate wheel, dependency fingerprint, target, ancestry and raw-artifact integrity remain enforced; changed immutable evidence requires a new run. Rejected attempts publish explicit failure reports, completed sibling receipts remain available, and startup gaps cannot reuse an older pass. Runs pinned to the older attempt format need a new run to use this policy.

ADO cleanup uses submission handoff rather than waiting for resource absence where supported. Accepted deletion is recorded separately from confirmed absence. Pending dependencies and failed submissions retain exact IDs in cleanup-status.json and ownership receipts for later operational cleanup, as requested; no extra cleanup job or new cleanup polling deadline is added. CSR targets stay retained while their namespace/device cleanup is unresolved. Pending resources may continue consuming quota. Ownership checks, worker termination and actual deletion assertions inside tests remain intact, and no generic ADR/ADU HTTP or inventory guard is introduced.

ADO runs are no longer blocked by other project builds or a fixed cleanup-time window; the separate GitHub live-run check remains. Standalone controllers and their subprocesses inherit candidate dependencies, with main/DPS interpreter startup checks before Azure login. ADU has no added per-test timeout, and DPS regular/service-SAS attempts retain seven-worker execution with worker-safe result collection; existing job/phase and process-shutdown limits, serial local-auth toggling and non-ADO cleanup requirements remain intact.

Based on release/1.1.0-preview, including merged #917. Shared unit-test improvements are excluded from this diff. This PR retains the ADO-specific four-shard Unit/UnitGate adapter, pipeline artifact wiring, and adapter regression checks. ADO cross-job pip caching remains disabled after measurement showed no net benefit.

Rollout: this PR does not complete the FIC cutover. Pipeline 147 still defaults to dev; adopting the migration on other active branches and switching GitHub integration/release callers remain separate follow-up work. No pipeline settings, schedules, or FICs are changed here.

Hangyi (HangyiWang) and others added 5 commits October 5, 2026 16:57
…retries

Scope pipeline 147 to supported services, add strict immutable attempt qualification and offline retry diagnostics, preserve owned fixture cleanup, and refresh ADO WIF credentials for long-running jobs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the selectable retry self-test while retaining offline regression coverage and service-level manual retry targets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope ADR/ADU cleanup inventory to ARM roots observed by each test process and its workers so parallel services do not falsely report one another's resources as leaks. Preserve strict cleanup, retry evidence and per-service matrix serialization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collect the full unit inventory in each serial pytest shard, assign files by measured duration, and require complete consistent results before merging coverage. Cache only pip downloads for lint and unit jobs and expose cache and tox timing measurements. Keep every integration service behind Build, Lint and Unit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@HangyiWang Hangyi (HangyiWang) changed the title feat(ci): modernize ADO 147 with manual failed-case retries feat(ci): modernize ADO integration test pipeline Oct 5, 2026
Hangyi (HangyiWang) and others added 11 commits October 5, 2026 20:55
Remove cross-job pip caching after cold/warm runs showed no net gain. Preserve real Digital Twins LRO success/failure coverage with immediate mocked Retry-After and bounded completion, cutting the two unit cases from141 seconds to0.1 seconds. Rebalance the four unit shards using the corrected timing weight.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep cleanup, integration, unknown definitions and changed revisions fail-closed; identify blocking builds in diagnostics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Carry extension dependency paths into standalone controllers and children after initial Azure login, and verify main/DPS interpreter startup during setup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the merge-definition revision exception along with the ADO build scan. Preserve GitHub live-run admission, immutable candidates, and manual failed-case retry history.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove HTTP interception and the coupled inventory gate. Retain existing service cleanup, owned Hub/DPS phase checks, immutable candidates, and manual retry evidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate immutable execution integrity from test recovery eligibility. Preserve all sibling phase evidence, validate worker completion and combined DPS stages, and publish actionable rejected-attempt reports.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep exact-case recovery for complete call failures; rerun every affected service phase after fixture, startup or execution failures. Record accepted, pending and failed cleanup separately without polling for deletion completion, retaining dependent leftovers for operational cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract reusable duration-balanced serial unit partitions and strict completeness/coverage aggregation from the ADO migration. Run four GitHub integration precheck shards alongside independent lint, with attempt-scoped evidence and CI-neutral run identity. Preserve the existing tox platform matrix and remove mocked Digital Twins polling sleeps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stack the migration on the standalone unit-test performance branch. Keep the reusable runner, GitHub wiring, profile and Digital Twins speedup identical to its new base, adapt ADO run identity to the shared contract, and keep ADO-specific assertions with the migration tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@HangyiWang
Hangyi (HangyiWang) changed the base branch from release/1.0.0-preview to users/hangyiwang/unit-test-parallelism October 6, 2026 17:23
Hangyi (HangyiWang) and others added 10 commits October 6, 2026 17:26
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hosted runners resolved Python 3.13 to both 3.13.15 and 3.13.16 in the same workflow. Validate the requested major/minor series while retaining exact versions in receipts and summaries; preserve all collection, execution and artifact checks. Add mixed-patch and incompatible-version regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract reusable duration-balanced serial unit partitions and strict completeness/coverage aggregation from the ADO migration. Run four GitHub integration precheck shards alongside independent lint, with attempt-scoped evidence and CI-neutral run identity. Preserve the existing tox platform matrix and remove mocked Digital Twins polling sleeps.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hosted runners resolved Python 3.13 to both 3.13.15 and 3.13.16 in the same workflow. Validate the requested major/minor series while retaining exact versions in receipts and summaries; preserve all collection, execution and artifact checks. Add mixed-patch and incompatible-version regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve all OS/Python combinations with four serial pytest shards each, lint once per combination, and per-combination coverage gates. Namespace artifacts and run identities to isolate PR matrix results from integration prechecks. Reuse the existing aggregator with an optional prefix and fix Windows UTF-8 workflow reads.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the legacy required contexts on complete aggregate results and explicitly fail gates when shards or lint fail instead of skipping them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@HangyiWang
Hangyi (HangyiWang) force-pushed the users/hangyiwang/unit-test-parallelism branch from 85eafd3 to 51ef7cc Compare October 6, 2026 21:39
@HangyiWang
Hangyi (HangyiWang) force-pushed the users/hangyiwang/ado147-integration-parity-retries branch from 4f76bb7 to da9d5d7 Compare October 6, 2026 21:39
Hangyi (HangyiWang) and others added 7 commits October 6, 2026 22:11
Keep pull-request coverage for every target while retaining push CI for long-lived branches and tags, plus manual dispatch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep Python 3.13 on all three operating systems and Python 3.10 on Ubuntu for PRs. Default reusable, push, manual, and release runs to the full matrix, preserving required gates and complete per-combination test coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the four GitHub PR unit combinations and preserve every non-unit Merge job, including manifest generation and CredScan. Leave full GitHub matrices and integration pipelines unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep fresh-attempt override rejection intact while isolating offline tests from legacy ADO sentinels. Use a high-resolution counter to order equal wall-clock cleanup events, retaining old receipt compatibility and append-only evidence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@HangyiWang
Hangyi (HangyiWang) deleted the branch Azure:release/1.1.0-preview October 7, 2026 17:53
Preserve the migration-only changes while incorporating the merged unit-test improvements and latest release fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@HangyiWang
Hangyi (HangyiWang) changed the base branch from users/hangyiwang/unit-test-parallelism to release/1.1.0-preview October 7, 2026 21:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant