Skip to content

fix: eliminate Linux LD_AUDIT crossref pair_deadline timeout without relaxing clocks or custody - #330

Open
docxology wants to merge 6 commits into
mainfrom
fix/linux-ld-audit-crossref-pair-deadline-20261119
Open

docxology wants to merge 6 commits into
mainfrom
fix/linux-ld-audit-crossref-pair-deadline-20261119

Conversation

@docxology

Copy link
Copy Markdown
Owner

Summary

Resolves the Linux LD_AUDIT Native Pandoc/Crossref 600s pair_deadline timeout observed in Native candidate preflight run 44436293395 / job 135265900452 (infrastructure/reporting/native_crossref_pandoc.py:108) without extending NATIVE_OBSERVATION_CEILING_SECONDS (600.0), weakening _correspondence_guard, or changing any custody/replay schema.

Root Cause (from artifact 14495955479 monotonic timeline)

  1. Pre-launch deadline burn (196.75s): source_crossref_pandoc computed pair_deadline = min(runtime.deadline, time.monotonic() + 600.0) before prepare_source_crossref_pair observed the two static Linux provider archives (slot-0 & slot-1, which already run under owner.deadline), leaving only 403.25s when baseline-source-renderer launched.
  2. Compounding redundant verification overhead (~9.55s per owner.verify()): Back-to-back owner.verify() / compiled.verify() / verify_correspondence() calls with no intervening subprocess (in observe_helper_memory, observe_crossref_provider_archive, prepare_source_crossref_filter, SourceCrossrefFilter.finish, SourceCrossrefPair.finish, qualify_native_loader_tool, source_crossref_pandoc, and native_crossref_worker) plus full email.parser METADATA body parsing in runtime_identity() and dual git ls-files passes in _inventory() pushed slot-0 + slot-1 past 1,000s (actual native tool + LD_AUDIT work was <60s).

Fix

  • native_crossref_pandoc.py & native_crossref_pair.py & native_crossref_filter.py: Observe static Linux provider archives under owner.deadline first, then stamp pair_deadline = min(owner.deadline, time.monotonic() + NATIVE_OBSERVATION_CEILING_SECONDS) right before freezing the live pair/filter specs and launching Pandoc.
  • Atomic _verification_guard consolidation: Bracket adjacent pure-DATA custody checks in native_crossref_{archive,filter,pair,pandoc,worker}.py and native_loader_{helper_memory,qualification}.py inside single owner._verification_guard(...) blocks (strictly stronger bracketing, eliminating redundant duplicate full-repo scans).
  • implementation_custody.py: Read Name and Version headers directly from METADATA/PKG-INFO in runtime_identity() (with exact fallback to item.metadata), cutting per-call cost ~25x with identical output.
  • public_reference_git.py: Combine git ls-files --stage -z and git ls-files -v -z into a single allowlisted git ls-files -v --stage -z pass with identical flag/stage/tree checks.
  • Tests: Added regression tests in tests/infra_tests/reporting/test_native_original_launch_correctness.py.

…relaxing clocks or custody

Diagnosed from Native candidate preflight run 44436293395 / job 135265900452 (artifact 14495955479):
- source_crossref_pandoc (native_crossref_pandoc.py:108) stamped the 600s pair_deadline before prepare_source_crossref_pair observed the two static provider archives, burning 196.75s before baseline Pandoc even launched.
- Repeated back-to-back owner.verify() / compiled.verify() / source.verify_correspondence() calls (~9.55s each on Ubuntu 24.04) across archive prep, pair/filter setup, observe_helper_memory, qualify_native_loader_tool, and pair/filter finish consumed >1,000s across slot-0 + slot-1.

Fixes (preserving NATIVE_OBSERVATION_CEILING_SECONDS == 600.0 and every custody/replay invariant):
- Defer pair_deadline stamping inside prepare_source_crossref_pair until after static Linux provider archive admission (which already runs under owner.deadline).
- Consolidate adjacent back-to-back verify() calls into single atomic owner._verification_guard() brackets across native_crossref_{archive,filter,pair,pandoc,worker}.py and native_loader_{helper_memory,qualification}.py.
- Extract package Name/Version headers directly in runtime_identity() (~25x faster than email-parsing multi-KB METADATA bodies, identical output with fallback).
- Combine git ls-files --stage -z and git ls-files -v -z into one git ls-files -v --stage -z invocation in public_reference_git._inventory().
…prepare_source_crossref_filter

- Merge current main (0899e19, PR #329) so rendered provenance rebind covers both PR #329 and PR #330.
- In prepare_source_crossref_filter, verify the pre-created slot directory contains only provider-archive instead of calling create_private_native_directory(fresh=True) a second time.
Parse the header block with the standard email parser without reading the description as MIME content. Preserve importlib handling of case, trailing whitespace and folded headers.
Apply the exact reviewed 25-exemplar report patch from run 44651798417, artifact 14596150349. All 25 output fingerprints are preserved and all three committed-tree publication gates passed on tree 0188d4f.
Share a single source census for the inventory and external fingerprint. Read new file bytes at fresh fingerprint and snapshot closure boundaries, including same-inode rewrites with unchanged metadata. Retain original provider pins and the existing 600-second ceiling.
Apply the reviewed patch from workflow run 44839104763, artifact 14685020940. The artifact gates passed for this exact tree, with 50 report modifications across 25 unchanged output fingerprints.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant