Repository navigation
Postgres conformance: row-storage lease check fails once real time passes the settling harness's fixed clock #1792
Description
tests/conformance/reporting/test_reporting_row_storage_object_pg.py::test_the_producer_stages_object_uploads_outside_its_publication_transaction fails on next, on main-based branches, and locally since 2026-11-01, with no related code change:
ReportingRowStoreError: Reporting row store: STATE_UNAVAILABLE: no live worker lease covers this row upload
The last green next CI run was 2026-10-31T22:11Z (run 41247073211); every run on 2026-11-01 fails this one test (seen on #1787, #1788, #1790, which touch no related code, and on a plain next worktree locally).
Cause. The test uses tests/test_reporting_settling.py::_harness, whose clock is pinned at datetime(2026, 11, 1, 2, 10, tzinfo=UTC). The producer stamps reporting_configurations.lease_expires_at from that fake clock, but adcp_reporting_storage/ledger/row_write.py verifies the lease against the database's real time:
... AND c.lease_expires_at > clock_timestamp()Once wall-clock time passes the fixed clock plus the lease length, no row upload is "covered" and the test fails. Any test that combines a frozen application clock with this DB-clock check has the same shelf life.
Options. Pin the database clock for the test (e.g. compare against the store's clock when one is injected, or widen the lease in the harness for PG runs), or move the harness clock to a date far in the future. Either way the check's production semantics (DB clock, so hosts agree) should stay.
Blocks "fully green" CI on every PR against next and main today.
Activity
bokelley commentedon Nov 1, 2026
Fixed on
nextby #1767 (c1c6d6eb):test_the_producer_stages_object_uploads_outside_its_publication_transactionnow runs the producer's turn at the wall clock. The lease the injected clock stamps then lies in the database's real future, androw_write.py'sclock_timestamp()check keeps its production semantics.Verified locally against PostgreSQL 16 at 2026-11-01T23:39Z, past the harness's frozen
02:10plus the lease:test_reporting_row_storage_object_pg.pyandtest_reporting_period_retirement_pg.py, the other row-storage file that uses_harness, give 40 passed. A full local run oftests/conformance/reporting/showed no other lease failure.next's CI hasn't finished a run since the merge (runners are queued); reopen if the Postgres lane still shows it.bokelley commentedon Nov 2, 2026
More evidence, from a local PostgreSQL 16 and from CI re-runs:
1 failed in 16s, twice), and the whole module fails only this test (1 failed, 22 passed).lease_period_close(now=<fake 2026-11-01T02:10Z>, lease_seconds=60)writeslease_expires_at = 02:11:00Z;row_write.pythen requiresc.lease_expires_at > clock_timestamp(), which has been false since 2026-11-01T02:11Z.gh run rerun --failed), and on plainnextlocally. One run did pass, on ci(release): gate publishes on a companion the built distribution works with (#1763) #1788's core lane started 2026-11-01T23:47Z (job 125015996166), which I cannot explain from the code; treat that as the exception.test(reporting): run the row-upload lease test at the wall clock (#1813)