Solution update fails at UpdateSecuredCore (ConfigureSystemGuardLaunch 2≠1, 0x82d0000c) - orphaned ApplyAndAutoCorrect machine configuration assignment enforcing DRTM=disabled on one node

[Sir-Jigston]

Suggested labels: Update, Security, TSG-Candidate

Environment

Item	Value
Product	Azure Local, version 24H2
OS build	10.0.26100.32995
Solution version (current)	12.2605.1003.210
Solution update being applied	12.2606.1003.205
ECE agent version (from stack trace)	Microsoft.AzureStack.Solution.ECEAgent.Service.10.2605.0.1303
AzureStackOSConfig nuget (from stack trace)	Microsoft.AS.Infra.Security.OSConfig.10.2606.0.1
AzureStackOSConfigAgent bundled OSConfig module	1.3.9 (Const.ps1 OSConfigModuleVersion)
Cluster size	2 nodes
Hardware	HPE ProLiant DL380 Gen11, Storage Spaces Direct
Secured-core state	TPM 2.0, Secure Boot, VBS, HVCI, Secure Launch (DRTM) all running on both nodes throughout the incident
Arc / management	Both nodes Arc-enabled; ARM securitySettings resource: drtmProtection: true, securedCoreComplianceAssignment: Audit
Affected node	1 of 2 (referred to as NODE02 below; NODE01 healthy)

This defeated, in order: Set-OSConfigDesiredConfiguration (0x82d0000c), Enable-AzsSecurity -FeatureName DRTM (15-retry enforcement failure), disabling drift control, and two ECE EnableOSConfigDRTM action plans. Every local registry correction reverted at the next GC consistency cycle (Event 819 "Delete policy" + Event 813 "Set policy int" pairs from EnrollmentID C429...).

Nothing in the existing Update/Security TSGs covers this failure mode, and no node-local diagnostic identifies guest configuration as the writer.

Contradictory signals (defining symptom)

All of the following were true simultaneously:

• systeminfo / Win32_DeviceGuard: Secure Launch configured and running on both nodes
• WAC Security: "All settings are on", 2 of 2 machines
• ARM securitySettings: drtmProtection: true, securedCoreCompliance: Compliant
• GC DSCResourceStateCache.mof: DRTM InDesiredState = True (deceptive - "in desired state" relative to the rogue assignment's own desired value of 2)
• Get-OSConfigDesiredConfiguration ... ConfigureSystemGuardLaunch: Default 1, Value 2, NotCompliant, IsCapable True
• MDM_Policy_Result01_DeviceGuard02.ConfigureSystemGuardLaunch = 2 on NODE02, = 1 on NODE01
• Get-AzsSecurity -FeatureName DRTM: False (ECE cluster intent had also diverged to disabled)
• Event 4031 loops: orchestrator conflict, uriPath(config/deviceguard/configuresystemguardlaunch), 0x82d0000c, Count(2, 1, 0)

Key diagnostic (the check that found it)

Diffing guest configuration assignments between the two nodes:

az rest --method GET --uri "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.HybridCompute/machines/<node>/providers/Microsoft.GuestConfiguration/guestConfigurationAssignments?api-version=2022-01-25"

NODE02 (broken) returned, in addition to three audit-mode assignments present on both nodes:

{
  "name": "SetSecuredCore",
  "properties": {
    "complianceStatus": "Compliant",
    "context": "Policy",
    "guestConfiguration": {
      "assignmentSource": null,
      "assignmentType": "ApplyAndAutoCorrect",
      "configurationParameter": [
        { "name": "Enforce Virtualization-Based Security (VBS);Value", "value": "1" },
        { "name": "Enforce Hypervisor-protected Code Integrity (HVCI);Value", "value": "2" },
        { "name": "Enforce System Guard Secure Launch (DRTM);Value", "value": "2" }
      ]
    }
  }
}

NODE01 (healthy) had no such assignment. On-box corroboration: C:\ProgramData\GuestConfig\Configuration\SetSecuredCore\ package present; gc_worker.log consistency runs ("Applying configuration from Current.mof" → "Sending report ... Compliant") timestamp-matched to the Event 819/813 revert pairs, recurring roughly daily.

Secondary persistence (why deleting the assignment alone was not enough)

After deleting the assignment and confirming the GC package was removed from the node, the stale value continued to be re-applied from cached WinDC state until all three of the following were removed in one dcsvc-stopped window:

1. HKLM\SOFTWARE\Microsoft\DMOrchestrator\<jobGUID> - an active job with EnrollmentId = C429BE2D-..., DocId = __csp__, StateMachineType = 0x1 (SET/enforcement type), which replayed the stale write on every dcsvc start
2. HKLM\SOFTWARE\Microsoft\DeclaredConfiguration\HostOS\Config\enrollments\C429BE2D-... - the enrollment state store (Device\state\__csp__\MostRecentVersion = the stale document hash), the rehydration source
3. HKLM\SOFTWARE\Microsoft\DeclaredConfiguration\HostOS\Current\providers\csp\.\device\vendor\msft\policy\config\deviceguard\configuresystemguardlaunch - the per-URI ownership record (authorityId = C429..., documentId = __csp__, value = 0x2)

Deleting only the ownership projection (item 3) or only PolicyManager provider values reverts within seconds of dcsvc restart. The stale document version hash makes an effective fingerprint for locating every persisted copy: reg query HKLM\SOFTWARE\Microsoft /s /f "<hash>" /d.

Remediation that worked (validated end-to-end)

1. Delete the SetSecuredCore guest configuration assignment (ARM DELETE); restart GCArcService; wait for the local package folder to disappear
2. Keep OSConfig drift control disabled for the remediation window
3. Stop dcsvc; remove the three stale-state artifacts above plus PolicyManager projections (SYSTEM-ACL'd keys; Invoke-DscResource -Name Registry ... Ensure=Absent in PS 5.1 executes as SYSTEM without psexec); start dcsvc
4. Enable-AzsSecurity -FeatureName DRTM → completes cleanly (reboot-required warning only)
5. Re-run the ECE action plan (Enable-ASOSConfigDRTMSetting, invoked in AzureStackOSConfigAgent module scope since it is not exported) → Success on all roles; Get-ASOSConfigDRTMSetting → True
6. Re-enable drift control; verify the value survives an OSConfigAlwaysRefresh cycle; drain, reboot, verify with boot-time-scoped event queries; resume the solution update

Product feedback / questions for the Azure Local team

1. Diagnosability: The UpdateSecuredCore / ASOSConfigEnableDRTM failure and the 0x82d0000c conflict give no indication that a machine configuration assignment is the conflicting writer. Event 4031 names only the requesting enrollment. Could the enforcement failure path enumerate the winning authority (and, when it is AzurePolicySourceId, surface active GC assignments) in the error output or a TSG?
2. Assignment provenance: The rogue assignment had assignmentSource: null, context: Policy, no backing policy assignment. What creates node-scoped SetSecuredCore ApplyAndAutoCorrect assignments (WAC security blade? a past portal remediation flow?), and should platform-owned secured-core enforcement ever coexist with the on-box SecuredCore WinDC document at higher precedence?
3. Intent reconciliation: ARM securitySettings said drtmProtection: true while ECE cluster intent (Get-ASOSConfigDRTMSetting) said False and the GC assignment enforced disabled. Three stores, three answers, no supported reconciliation path: node-scoped Enable-AzsSecurity does not update ECE, and the cluster-scoped Enable-ASOSConfigDRTMSetting is not exported from AzureStackOSConfigAgent. Is invoking it in module scope the intended remediation, and could it be exported/documented?
4. Stale WinDC cleanup: Is there a supported equivalent of dcsvc's internal document cleanup for a single stale enrollment authority (the __csp__ SET job + enrollment state store), short of manual registry removal?

Evidence available on request

• Two failed EnableOSConfigDRTM action plan progress XMLs (identical per-host failure) and the final successful run
• Event exports: 4031 conflict loops; 819/813 revert pairs timestamp-matched to gc_worker consistency runs
• gc_worker.log excerpts for the SetSecuredCore consistency cycles
• Full assignment JSON for both nodes (broken vs healthy)
• Registry exports of the DeclaredConfiguration/DMOrchestrator stale state (pre-eviction)
• ASSecurityOSConfigLogs for the failed and successful enforcement runs