Skip to content

Make the pii occurrence pagination test independent of numeric labels - #137

Merged
Travmatth merged 1 commit into
mainfrom
claude/pagination-test-label-independent
Oct 13, 2026
Merged

Travmatth merged 1 commit into
mainfrom
claude/pagination-test-label-independent

Conversation

@Travmatth

Copy link
Copy Markdown
Contributor

Summary

test_occurences_pagination asserted that 12 random "The number is N" values all come back as NUMERIC_VALUE. With Solar's new English NER model (upbeat-leaf-281, TonicAI/solar#3838), some bare 3-digit numbers are labeled CREDIT_CARD:

  • The model's credit_card_number tag scores 0.51–0.69.
  • That beats the 0.5 regex NUMERIC_VALUE when the two overlap.

So the NUMERIC_VALUE count varied with the random numbers (5, 8 or 10 of 12) even though pagination worked. Reproduced on the #3838 PR app: on one seed, 7 of 12 numbers came back CREDIT_CARD, and the paginated and unpaginated fetches agreed.

The test now:

  • uses 30 distinct numbers;
  • compares paginated (limit=5) results to a single unpaginated fetch across all PII types, and for the mixed PiiType/str request;
  • checks every number was found under some label.

It no longer depends on which numeric label the model picks.

Test plan

  • Passed twice against the #3838 PR app running upbeat-leaf-281 (further runs were blocked by a local Border0 DNS outage)
  • Solar's Python SDK job on TonicAI/solar#3838 picks this up from the default branch once merged

🤖 Generated with Claude Code

The test asserted that 12 random "The number is N" values all came back as
NUMERIC_VALUE. A newer English NER model labels some bare 3-digit numbers
CREDIT_CARD (score > the 0.5 regex NUMERIC_VALUE, so it wins the overlap),
which made the count vary with the random numbers even though pagination
worked. Compare paginated results against a single unpaginated fetch, and
check every number was found under some label, instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Travmatth
Travmatth enabled auto-merge (squash) October 13, 2026 21:01
@Travmatth
Travmatth merged commit f6507fe into main Oct 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants