privacy: cover the postal half of "address", which no gate could see - #82
Conversation
…wo things An audit for residential addresses of real people found none, in this repo or in pdoom1, pdoom1-website and pdoom-dashboard. Every address-shaped string in the tracked tree is an organisational one -- ten of them, all university departments, a national library, an innovation centre and a defence contractor's published point of contact, carried in from academic paper title pages by the same unparsed-PDF import that caused the email exposures. No redaction was warranted, so this commit changes no data. What the audit did find is that nothing could have caught a home address if one had arrived. redact_emails.py gates the tracked zones, and RESIDUE requires an at-sign or the wreckage of one, so no setting of it could ever match a street. check_all.py labelled that check "no email addresses in tracked zones" while the tool itself printed "No address-shaped text in any tracked zone", and the word "address" doing double duty across those two lines is the whole reason the gap read as coverage for as long as it did. scan_postal.py covers the postal half: unit and apartment prefixes, Australian localities, coordinate pairs at dwelling resolution, personal handsets, and streets in both number-first and number-last order. The design problem was not detection, it was staying unmuted. A guard that fired on all ten institutional addresses would be switched off within a week, and test_privacy_gate.py already records why that is worse than no guard. So the discriminator is an INSTITUTION token in the surrounding window rather than a narrower street list: a published affiliation address travels with the name of the institution that published it and a leaked home address does not, which is a fact about how the two come to exist rather than about how they are spelled. The accepted bound is a home address within 160 characters of the word "University", named here rather than hidden. Two false-positive families were measured against the real corpus and both are "the digit run continues past the match": a float, "Training power draw (W)": "202329.0409413959", and a sha256 digest, each containing a well-formed Australian mobile number. A third was a year, "Released SA 2019", which is now rejected by checking the postcode against the block Australia Post actually issued to that state. All three are MUST_NOT_FIRE cases. The scanner prints paths and line numbers and never the matched text, because CI logs are public on a public repository. RED run observed per Workshop 2 ruling R6, on a deliberate bad input appended to a tracked doc: all five detector families fired and --ci exited 1. Green again on restore. check_all is 17 gating checks, all passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
932eff8 to
ef196ad
Compare
|
Rebased onto ("scan cross-reference check fires", ["tests/test_scan_cross_references.py"], True), # from main
("no dwelling addresses in tracked zones", ["scripts/privacy/scan_postal.py", "--ci"], True),
("postal gate can still fail", ["tests/test_postal_gate.py"], True),Verified locally after the rebase:
One gating check fails, and it is not this branch's. Also worth recording against this PR's escalation section: the Still holding for Pip's review per the body — not merging. |
|
Merged at cbfc25b with The red was inherited, not caused. The same job failed on main's two preceding pushes — runs What was failing was one gating finding, not the 114 warnings the log tail shows. Fixed on main in af03c68 as a This PR's two new gates were verified passing, which CI could not show because the reference check failed first: One thing this PR diagnosed and did not fix. The commit message identifies the root cause as wording — "check_all.py labelled that check 'no email addresses in tracked zones' while the tool itself printed 'No address-shaped text in any tracked zone', and the word 'address' doing double duty across those two lines is the whole reason the gap read as coverage" — but neither string changed. Current output: The overclaiming line now sits directly above the check that actually covers addresses, so a reader scanning output sees address coverage asserted twice and cannot tell which one means it. Also still open, from this PR's escalation section: |
1 participant
What this is not
An audit for residential addresses of real people found none -- not in this repo, and not in
pdoom1,pdoom1-websiteorpdoom-dashboard. No data file is changed by this PR and no redaction was warranted.Every address-shaped string in the tracked tree is organisational: 10 occurrences across 4 files, all
data/serveable/api/timeline_events/. They are university departments, a national library, an innovation centre and a defence contractor's published point of contact, extracted from academic paper title pages by the same unparsed-PDF import behind the three email tombstones. Categorised as NOT the exposure. Details, counts and severities are in the coordination report rather than here, since this page is public.What the audit actually found
Nothing could have caught a home address if one had arrived.
redact_emails.py --cigates the tracked zones, butRESIDUErequires an at-sign or the wreckage of one, so no setting of it could ever match a street.check_all.pylabelled that checkno email addresses in tracked zoneswhile the tool printedNo address-shaped text in any tracked zone. The word "address" doing double duty across those two lines is why the gap read as coverage.What this adds
scripts/privacy/scan_postal.py-- unit/apartment prefixes, Australian localities, coordinate pairs at dwelling resolution (4+ decimals is ~11m), personal handsets, and streets in number-first and number-last order.tests/test_postal_gate.py-- 13 must-fire, 15 must-not-fire.check_all.pyanddata-integrity.ymlas two gating checks.The scanner prints paths and line numbers only, never matched text, because CI logs are public.
The design problem was staying unmuted
A guard that fired on all 10 institutional addresses would be switched off within a week, and
test_privacy_gate.pyalready records why that is worse than no guard. So the discriminator is anINSTITUTIONtoken in the surrounding window, not a narrower street list: a published affiliation address travels with the name of the institution that published it; a leaked home address does not. That is a fact about how the two come to exist, not how they are spelled.Accepted bound, named rather than hidden: a home address within 160 characters of the word "University" would be missed.
Three false-positive families were measured against the real corpus and are locked in as must-not-fire -- a float and a sha256 digest each containing a well-formed Australian mobile number, and a year (
SA 2019), now rejected by checking the postcode against the block Australia Post issued to that state.Verification
check_all.pygatingRED run observed per Workshop 2 ruling R6: a deliberate bad input appended to a tracked doc made all five detector families fire and
--ciexit 1; green again on restore.History
Not rewritten, deliberately. The institutional addresses reach back to
5526156. History rewriting on a repo with clones is Pip's call, and there is no residential exposure that would motivate it.Escalation, outside this repo
pdoom1-websitepublishes 6 academic contact emails that this repo has already scrubbed -- inpublic/data/events.json, in 6 rendered HTML pages, and in OpenGraph meta tags. Its own guard has the brace-group bug that12c0455fixed here.pdoom1carries the same in its vendored copy. Separate PRs; flagged in the report.Do not merge without Pip's review.