brief currently uses the github.com/git-pkgs/licensecheck fork for one job: read a LICENSE file and return an SPDX identifier (detect.detectLicenseType). Moving that call to github.com/git-pkgs/licenses would let us retire the fork, but importing this package pulls in the full embedded ScanCode corpus.
Measured on darwin/arm64, go1.26.7:
| binary |
size |
brief with git-pkgs/licensecheck v0.4.1 |
48.8 MB |
brief with git-pkgs/licenses v0.8.0 added |
60.5 MB |
The delta is the 11.5 MB internal/corpus/corpus.bin.gz embed. brief already depends on git-pkgs/magic, git-pkgs/manifests and git-pkgs/spdx, so the transitive graph adds nothing else.
The only public constructor is licenses.New, which unconditionally calls internal/corpus.Load, which references the //go:embed var. There is no way to construct a Matcher against a smaller corpus, and internal/corpus (which has Read/Write for the on-disk format) is not importable from outside the module.
For a LICENSE-file classifier, only a fraction of the corpus is relevant. Rule counts in the v0.8.0 embed (ScanCode 33.0.0rc1, 39,215 rules total):
| flag |
rules |
FlagLicenseText |
6,959 |
FlagLicenseNotice |
14,917 |
FlagLicenseReference |
11,893 |
FlagLicenseTag |
3,853 |
| other (intro/clue/false-positive) |
1,593 |
Proposal:
- Add
func NewFromReader(r io.Reader, options ...Option) (*Matcher, error) that decodes a corpus with internal/corpus.Read and builds a matchEngine from it, bypassing the embedded index. The Go linker drops an unreferenced //go:embed package var (verified: a 5 MB embed referenced only by an unreachable function is not linked), so a caller that uses only NewFromReader does not carry the 11.5 MB.
- Teach the corpus builder to emit a filtered index, e.g.
--rule-flags text or --rule-flags text,notice, which writes only the selected rules and rebuilds the Aho-Corasick automaton and vocabulary over that subset. internal/corpus.Write already serialises an arbitrary Index; the missing piece is regenerating Index.Automaton and trimming Index.Vocabulary/StopwordIDs for the reduced rule set. Callers embed the resulting corpus.bin.gz themselves and pass it to NewFromReader.
licensecheck's builtin.dfa is 2.9 MB, so a text-rules-only corpus in the same ballpark would let brief swap without growing, and dependents/downstream (which pull licensecheck in transitively via brief) would follow.
briefcurrently uses thegithub.com/git-pkgs/licensecheckfork for one job: read aLICENSEfile and return an SPDX identifier (detect.detectLicenseType). Moving that call togithub.com/git-pkgs/licenseswould let us retire the fork, but importing this package pulls in the full embedded ScanCode corpus.Measured on darwin/arm64, go1.26.7:
briefwithgit-pkgs/licensecheckv0.4.1briefwithgit-pkgs/licensesv0.8.0 addedThe delta is the 11.5 MB
internal/corpus/corpus.bin.gzembed.briefalready depends ongit-pkgs/magic,git-pkgs/manifestsandgit-pkgs/spdx, so the transitive graph adds nothing else.The only public constructor is
licenses.New, which unconditionally callsinternal/corpus.Load, which references the//go:embedvar. There is no way to construct aMatcheragainst a smaller corpus, andinternal/corpus(which hasRead/Writefor the on-disk format) is not importable from outside the module.For a LICENSE-file classifier, only a fraction of the corpus is relevant. Rule counts in the v0.8.0 embed (ScanCode 33.0.0rc1, 39,215 rules total):
FlagLicenseTextFlagLicenseNoticeFlagLicenseReferenceFlagLicenseTagProposal:
func NewFromReader(r io.Reader, options ...Option) (*Matcher, error)that decodes a corpus withinternal/corpus.Readand builds amatchEnginefrom it, bypassing the embedded index. The Go linker drops an unreferenced//go:embedpackage var (verified: a 5 MB embed referenced only by an unreachable function is not linked), so a caller that uses onlyNewFromReaderdoes not carry the 11.5 MB.--rule-flags textor--rule-flags text,notice, which writes only the selected rules and rebuilds the Aho-Corasick automaton and vocabulary over that subset.internal/corpus.Writealready serialises an arbitraryIndex; the missing piece is regeneratingIndex.Automatonand trimmingIndex.Vocabulary/StopwordIDsfor the reduced rule set. Callers embed the resultingcorpus.bin.gzthemselves and pass it toNewFromReader.licensecheck'sbuiltin.dfais 2.9 MB, so a text-rules-only corpus in the same ballpark would letbriefswap without growing, anddependents/downstream(which pulllicensecheckin transitively viabrief) would follow.