Index every winget package for detection, not just the curated catalogue - #7
Merged
Merged
Conversation
Two problems with detection as merged. The index was never fetched on an endpoint that only runs detection. -NoRefresh was implemented as "never download", but a detection-only machine installs nothing, so nothing else would ever fetch the index for it and ProductCode detection could never work there. It now means "do not re-download a copy we already have": absent means fetch, present means use what is there. Coverage was limited to the twelve packages in Index/Catalog.json. Detection of anything else fell back to name matching, which cannot resolve a winget package ID at all. The detection index is now built from Microsoft's own published winget source, which already carries the ARP product codes and MSIX package family names that detection matches on, for every package in the repository. That is one 3.5 MB CDN download in CI, with no api.github.com calls and no clone of winget-pkgs. Packages carrying neither a product code nor a package family name are omitted, because they cannot be identified this way. Detection now tries, in order: product codes and package family names from the full index, the product code from the curated index or the local manifest cache, an exact display name, a substring, then an MSIX name. Package family name matching uses -AllUsers when elevated, for the same reason as the uninstall path. The curated Index/Catalog.json stays, and still carries installer URLs and silent arguments so a common install needs no API call either. It was never an allow-list: Install-TecharyApp resolves any winget package live.
The separator in the SQL group_concat was char(31), and the matching jq split used a \u001f escape. That escape reached the workflow file as a raw unit separator byte, which YAML does not permit, so the workflow failed to parse and the run produced no jobs at all. Any separator safe inside a product code has to be a control character, so the query now emits one row per identifier and jq groups them. No separator, nothing to escape.
The winget source carries every product code a package has ever shipped:
Mozilla.Firefox alone has 5205, one per locale and version. Probing those
across three hives is 15,615 registry reads, measured at ~384 seconds, which
would exceed an N-central scan interval on its own.
The uninstall key names are now enumerated once into a case-insensitive
dictionary and each candidate is a hash lookup, so the cost is ~230 registry
reads regardless of how many codes a package carries. Firefox drops from
~384s to 0.47s.
Ordinal-ignore-case is required, not cosmetic: the source index stores codes
normalised to lower case ("7-zip") while the real key is "7-Zip".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #3 and #6, both merged. Fixes a blocking bug in the merged detection path and removes the coverage limit.
The index was never fetched on a detection-only endpoint
-NoRefreshwas implemented as "never download". A machine that only runs the detection service installs nothing, so nothing else would ever fetch the index for it — those machines were left permanently without one and ProductCode detection could never work there.Measured on a fresh endpoint before this change, with 7-Zip installed:
-NoRefreshnow means "do not re-download a copy we already have": absent means fetch, present means use what is there. Same result after:Coverage was 12 packages
Detection resolved a package ID only if it appeared in
Index/Catalog.json. Everything else fell back to name matching, which cannot resolve a winget package ID at all.The detection index is now built from Microsoft's own published winget source, which already carries the ARP product codes and MSIX package family names that detection matches on, for every package in the repository:
https://cdn.winget.microsoft.com/cache/source2.msixapi.github.comcallsTables used:
packages,productcodes2,pfns2. Packages carrying neither a product code nor a package family name are excluded — they cannot be identified this way, so they are dead weight in a file every endpoint downloads.Detection order
Package family name matching uses
-AllUserswhen elevated, for the same reason as #4.Scope note
Index/Catalog.jsonstays and still carries installer URLs and silent arguments, so a common install needs no API call either. It was never an allow-list —Install-TecharyAppresolves any winget package live from the manifest repository, and falls through toCustomApps.jsonfor non-winget apps.Verification
Client against an index in the exact shape the Action emits:
covering an ARP product-code hit, an MSIX package-family hit, and a true negative.
Note on the first push
The initial commit used
char(31)as agroup_concatseparator, with jq splitting on the matching unit-separator escape. That escape reached the workflow file as a raw 0x1F byte, which YAML does not permit, so the workflow failed to parse and produced no jobs at all. Any separator guaranteed safe inside a product code is a control character, so the query now emits one row per identifier and jq groups them - no separator, nothing to escape.Lookup cost
The source carries every product code a package has ever shipped —
Mozilla.Firefoxhas 5,205, one per locale and version. Probing those across three hives is 15,615 registry reads, measured at ~384 seconds, which would exceed a scan interval on its own.Uninstall key names are enumerated once into a case-insensitive dictionary and each candidate is a hash lookup, so cost is ~230 registry reads regardless of how many codes a package carries. Ordinal-ignore-case is required rather than cosmetic: the source stores codes normalised to lower case (
7-zip) while the real key is7-Zip.Timed against the real published index:
Zoom.ZoomreturnedFalseunder the 12-package curated index andTruehere, which is the coverage change in one line.Published index
The Action ran on this branch and published:
Packages whose manifests declare neither a product code nor a package family name are absent by design —
Valve.Steamis one — and fall back to name matching as before.