Skip to content

Correlate installed software the way winget does - #13

Merged
TecharyJames merged 1 commit into
mainfrom
feat/winget-correlation
Sep 21, 2026
Merged

TecharyJames merged 1 commit into
mainfrom
feat/winget-correlation

Conversation

@TecharyJames

Copy link
Copy Markdown
Member

Detection had grown four name heuristics because it was reverse-engineering "is this package installed" from the registry. Each new application found another gap; the last one, Microsoft.Office, caused a reinstall loop on a production endpoint.

winget does not guess. Its list path is an exact-equality join on four keys, OR'd together (CompositeSource.cpp:937):

Key Shipped before Now
PackageFamilyName yes yes
ProductCode yes yes
UpgradeCode no yes
NormalizedNameAndPublisher no yes

The edit-distance code in ARPCorrelation.cpp is the post-install heuristic and is deliberately not ported.

Why name+publisher matters

6,443 of 14,896 packages declare no product code and no package family name. Microsoft.Office is one. But every package has a normalised name and publisher, so this key makes correlation total rather than a fallback.

A name match requires a publisher match, as winget's inner join does. That is why Git.Git cannot collide with GitHub CLI without any boundary rule, and it removes the prefix heuristic entirely.

The normaliser

Get-WgNormalized ports NormalizationVersion::Initial from NameNormalization.cpp. The subtle part: ICU applies case closure to character classes under UREGEX_CASE_INSENSITIVE and .NET does not, so every \p{Lu} is written as [\p{Lu}\p{Ll}\p{Lt}]. Without that the locale pattern never fires and Microsoft 365 Apps for enterprise - en-us does not reduce to the stored key.

Ambiguity

Weak matches on a name+publisher pair shared by several packages are dropped, reproducing the reverse-correlation veto. 614 pairs are affected; mozillathunderbird+mozilla alone is shared by 133 package ids.

Packages with a real identifier are unaffected, because a strong match returns first. Google Chrome shares its pair with Google.Chrome.EXE and still resolves, on UpgradeCode.

Verification

Against winget list --id X --exact for all 61 packages installed on a real machine:

AGREEMENT: 61/61 (100%)

ProductCode            32
PackageFamilyName      19
NameAndPublisher        7
UpgradeCode             2

That includes agreeing that Bitwarden.Bitwarden is not detectable, which winget also reports, because it has no ARP entry and no MSIX package. An earlier run of this harness used winget export as the oracle and wrongly counted that as a miss; export draws on the tracking catalogue, which is machine-local state and not what we reproduce.

Tests

Every defect this replaces reached an endpoint before anyone noticed, so both run on every pull request:

  • Tests/Test-Normalizer.ps1 runs winget's own 1,137-vector corpus (vendored under Tests/corpus, MIT) plus the architecture anchors from its unit tests. 1137/1137 and 6/6.
  • Tests/Test-Module.ps1 covers the structural faults that have actually happened here: a scope qualifier that stopped the module loading, an exported function with no file, and a control character that made a workflow unparseable.

Known limitations

  • ToLowerInvariant approximates ICU full case folding. It diverges on ß, ligatures and final sigma. The corpus passes, but a name containing those may not match.
  • 522 packages carry only arch-suffixed normalised names, so an ARP entry with no architecture token will not match them by name. winget fails identically.
  • The veto uses the ambiguous-pair list rather than a full reverse index. A weak match whose product code belongs to a different package is not detected.

Detection had grown four name heuristics - exact, bounded prefix, substring
and MSIX name - because it was reverse engineering "is this package
installed" from the registry. Each new application found another gap, and
the last one, Microsoft.Office, caused a reinstall loop on a production
endpoint.

winget does not guess. Its list path is an exact-equality join on four
keys, OR'd together: PackageFamilyName, ProductCode, UpgradeCode and
NormalizedNameAndPublisher (CompositeSource.cpp:937). There is no fuzzy
matching in it at all. The edit-distance code in ARPCorrelation.cpp is the
post-install heuristic and is deliberately not ported.

The winget source index already carries all four, and we were shipping two.
It now ships all four, including norm_names2 and norm_publishers2, which is
the only key that covers the 6,443 packages declaring no identifier at all.
Every one of the 14,896 packages has a normalised name and publisher, so
name correlation is total rather than a fallback.

Get-WgNormalized is a port of NormalizationVersion::Initial from
NameNormalization.cpp. The subtle part is that ICU applies case closure to
character classes under UREGEX_CASE_INSENSITIVE and .NET does not, so every
\p{Lu} is written out as [\p{Lu}\p{Ll}\p{Lt}]. Without that the locale
pattern never fires and "Microsoft 365 Apps for enterprise - en-us" does
not reduce to the stored key.

A name match requires a publisher match, as winget's inner join does. That
is why Git.Git cannot collide with GitHub CLI without any boundary rule,
and it removes the need for the prefix heuristic entirely.

Weak matches on a name+publisher pair shared by several packages are
dropped, reproducing the reverse-correlation veto. Packages carrying a real
identifier are unaffected because a strong match returns first: Google
Chrome shares its pair with Google.Chrome.EXE and still resolves, on
UpgradeCode.

UpgradeCode needs a second registry read. It is not a value on the
uninstall key but lives under Installer\UpgradeCodes in packed GUID form,
keyed the opposite way round, so the map is built once and inverted.

Tests, because every defect this replaces reached an endpoint before anyone
noticed. Tests/Test-Normalizer.ps1 runs winget's own 1,137 vector corpus,
vendored under Tests/corpus, plus the architecture anchors from its unit
tests. Tests/Test-Module.ps1 covers the structural faults that have
actually happened here: a scope qualifier that stopped the module loading,
an exported function with no file, and a control character that made a
workflow unparseable. Both run on every pull request.

Verified against "winget list --id X --exact" for all 61 packages installed
on a real machine: 61/61 agreement, by ProductCode 32, PackageFamilyName
19, NameAndPublisher 7, UpgradeCode 2. That includes agreeing that
Bitwarden.Bitwarden is NOT detectable, which winget also reports, because
it has no ARP entry and no MSIX package.
@TecharyJames
TecharyJames merged commit 3c76c22 into main Sep 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant