Correlate installed software the way winget does - #13
Merged
Merged
Conversation
Detection had grown four name heuristics - exact, bounded prefix, substring
and MSIX name - because it was reverse engineering "is this package
installed" from the registry. Each new application found another gap, and
the last one, Microsoft.Office, caused a reinstall loop on a production
endpoint.
winget does not guess. Its list path is an exact-equality join on four
keys, OR'd together: PackageFamilyName, ProductCode, UpgradeCode and
NormalizedNameAndPublisher (CompositeSource.cpp:937). There is no fuzzy
matching in it at all. The edit-distance code in ARPCorrelation.cpp is the
post-install heuristic and is deliberately not ported.
The winget source index already carries all four, and we were shipping two.
It now ships all four, including norm_names2 and norm_publishers2, which is
the only key that covers the 6,443 packages declaring no identifier at all.
Every one of the 14,896 packages has a normalised name and publisher, so
name correlation is total rather than a fallback.
Get-WgNormalized is a port of NormalizationVersion::Initial from
NameNormalization.cpp. The subtle part is that ICU applies case closure to
character classes under UREGEX_CASE_INSENSITIVE and .NET does not, so every
\p{Lu} is written out as [\p{Lu}\p{Ll}\p{Lt}]. Without that the locale
pattern never fires and "Microsoft 365 Apps for enterprise - en-us" does
not reduce to the stored key.
A name match requires a publisher match, as winget's inner join does. That
is why Git.Git cannot collide with GitHub CLI without any boundary rule,
and it removes the need for the prefix heuristic entirely.
Weak matches on a name+publisher pair shared by several packages are
dropped, reproducing the reverse-correlation veto. Packages carrying a real
identifier are unaffected because a strong match returns first: Google
Chrome shares its pair with Google.Chrome.EXE and still resolves, on
UpgradeCode.
UpgradeCode needs a second registry read. It is not a value on the
uninstall key but lives under Installer\UpgradeCodes in packed GUID form,
keyed the opposite way round, so the map is built once and inverted.
Tests, because every defect this replaces reached an endpoint before anyone
noticed. Tests/Test-Normalizer.ps1 runs winget's own 1,137 vector corpus,
vendored under Tests/corpus, plus the architecture anchors from its unit
tests. Tests/Test-Module.ps1 covers the structural faults that have
actually happened here: a scope qualifier that stopped the module loading,
an exported function with no file, and a control character that made a
workflow unparseable. Both run on every pull request.
Verified against "winget list --id X --exact" for all 61 packages installed
on a real machine: 61/61 agreement, by ProductCode 32, PackageFamilyName
19, NameAndPublisher 7, UpgradeCode 2. That includes agreeing that
Bitwarden.Bitwarden is NOT detectable, which winget also reports, because
it has no ARP entry and no MSIX package.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detection had grown four name heuristics because it was reverse-engineering "is this package installed" from the registry. Each new application found another gap; the last one,
Microsoft.Office, caused a reinstall loop on a production endpoint.winget does not guess. Its list path is an exact-equality join on four keys, OR'd together (
CompositeSource.cpp:937):PackageFamilyNameProductCodeUpgradeCodeNormalizedNameAndPublisherThe edit-distance code in
ARPCorrelation.cppis the post-install heuristic and is deliberately not ported.Why name+publisher matters
6,443 of 14,896 packages declare no product code and no package family name.
Microsoft.Officeis one. But every package has a normalised name and publisher, so this key makes correlation total rather than a fallback.A name match requires a publisher match, as winget's inner join does. That is why
Git.Gitcannot collide withGitHub CLIwithout any boundary rule, and it removes the prefix heuristic entirely.The normaliser
Get-WgNormalizedportsNormalizationVersion::InitialfromNameNormalization.cpp. The subtle part: ICU applies case closure to character classes underUREGEX_CASE_INSENSITIVEand .NET does not, so every\p{Lu}is written as[\p{Lu}\p{Ll}\p{Lt}]. Without that the locale pattern never fires andMicrosoft 365 Apps for enterprise - en-usdoes not reduce to the stored key.Ambiguity
Weak matches on a name+publisher pair shared by several packages are dropped, reproducing the reverse-correlation veto. 614 pairs are affected;
mozillathunderbird+mozillaalone is shared by 133 package ids.Packages with a real identifier are unaffected, because a strong match returns first. Google Chrome shares its pair with
Google.Chrome.EXEand still resolves, onUpgradeCode.Verification
Against
winget list --id X --exactfor all 61 packages installed on a real machine:That includes agreeing that
Bitwarden.Bitwardenis not detectable, which winget also reports, because it has no ARP entry and no MSIX package. An earlier run of this harness usedwinget exportas the oracle and wrongly counted that as a miss; export draws on the tracking catalogue, which is machine-local state and not what we reproduce.Tests
Every defect this replaces reached an endpoint before anyone noticed, so both run on every pull request:
Tests/Test-Normalizer.ps1runs winget's own 1,137-vector corpus (vendored underTests/corpus, MIT) plus the architecture anchors from its unit tests. 1137/1137 and 6/6.Tests/Test-Module.ps1covers the structural faults that have actually happened here: a scope qualifier that stopped the module loading, an exported function with no file, and a control character that made a workflow unparseable.Known limitations
ToLowerInvariantapproximates ICU full case folding. It diverges on ß, ligatures and final sigma. The corpus passes, but a name containing those may not match.