Support caller-supplied and filtered license corpora

[abhinavgautam01]

Related to #59. Further corpus size reductions remain follow-up work; this PR does not close the issue.

Summary

Applications currently must load the full embedded ScanCode corpus through licenses.New(). This adds the entire corpus to their binaries even when they only need a subset of license rules.

This change adds NewFromReader(io.Reader, ...Option) and corpus generation filtering, allowing applications to supply and embed their own reduced corpus.

Changes

• Add NewFromReader, supporting the same options as New.
• Build an independent matcher engine per call without initializing the shared embedded corpus.
• Leave reader ownership with the caller.
• Add corpusgen -rule-flags, selecting rules with any of the requested flags.
• Rebuild vocabulary, token IDs and the automaton from the selected rules.
• Preserve the complete stopword list and SPDX mappings.
• Keep the default corpus generation behavior unchanged.
• Document custom corpus generation, embedding, format compatibility and detection trade-offs.

Size measurements

Measured with Go 1.26.7 on darwin/arm64:

• Full corpus: 12,764,318 bytes.
• Text-only corpus: 10,350,064 bytes across 6,959 rules.
• Text-and-notice corpus: 12,200,280 bytes across 21,881 rules.

An external test program using only NewFromReader produced a binary of approximately 6.1 MB, compared with 19.0 MB using New. The full corpus bytes were absent from the reader-only binary.

Embedding the text-only corpus produced a binary of approximately 16.5 MB. This enables smaller, caller-controlled corpora, but text-only filtering does not reach the approximately 2.9 MB corpus size discussed in the issue.

Filtered corpora can change detection behavior because omitted rules and vocabulary are unavailable.

Validation

• Formatting, build, and go vet passed.
• Full race test suite passed.
• golangci-lint run: 0 issues.
• git diff --check passed.
• Default corpus regeneration remained byte-identical.
• Pinned ScanCode conformance remained 1,579/1,787 (88.36%), with zero skipped cases.
• External callers successfully detected MIT using supplied and embedded filtered corpora.

Regression tests cover independent matchers, options, reader ownership, malformed and truncated data, checksum and read errors, flag validation, deterministic filtering, token remapping, stopwords and included/excluded rule matching.

[andrew]

The direction matches what #59 proposed, and the implementation is in the right place: filtering runs before the vocabulary is built, so vocabulary, token IDs, stopword remapping and the automaton are all rebuilt over the reduced rule set, and the default all mask leaves generation byte-identical. corpus.Read already validates framing, counts, sortedness and token/stopword ranges, so exposing it through a public constructor does not open a decoding hole. A few things before merge.

Blocking:

• matcher.go:216 and the README both state that a program using only NewFromReader can omit the embedded corpus from its binary. That is the point of the change and the one claim with no test behind it. It also regresses invisibly: the day some shared code path calls corpus.Load, every reader-only caller silently gets the 12.7 MB back and no existing test notices. Add a guard that builds two fixture programs in t.TempDir() against the module with a replace directive, one calling New and one calling only NewFromReader, and asserts a multi-megabyte size delta (skip when no Go toolchain is available). internal/corpus/embed_test.go already budgets the embed size, so a size assertion fits the existing shape.

Non-blocking:

• reader_test.go covers code in matcher.go, and cmd/corpusgen/filter_test.go covers code in main.go. Test files here pair with their source file (scan.go/scan_test.go, spdx.go/spdx_test.go, filter.go/filter_test.go). Either move NewFromReader and configureMatcher into a new reader.go, or fold the tests into matcher_test.go and cmd/corpusgen/main_test.go.
• cmd/corpusgen/main.go:166: the comment about a zero mask describes the filter at line 216 rather than the parser it is attached to, and as a doc comment it should start with the function name. Move it to the if ruleFlags != 0 block.

One note on the numbers: at 10.35 MB for text-only, flag filtering does not get close to the 2.9 MB figure in #59, because the per-rule token arrays for the 6,959 license texts are the bulk of the corpus rather than the auxiliary rules. Reaching that size needs a different axis, so #59 should stay open after this lands.

[abhinavgautam01]

Thanks, added the binary-size guard and verified it fails when corpus.Load is accidentally retained. Also aligned the source/test files, moved the zero-mask comment and removed the closing reference to #59.