Skip to content

fix(models): hard_delete() removes all it collected or rolls back; spares by closure; reads keys a WHERE cannot hold (#72, #71, #73, 2.15.0) - #75

Merged
Behnam-RK merged 10 commits into
mainfrom
fix/hard-delete-bundle
Oct 7, 2026
Merged

Behnam-RK merged 10 commits into
mainfrom
fix/hard-delete-bundle

Conversation

@Behnam-RK

Copy link
Copy Markdown
Owner

Three hard_delete() defects the review loop on #69 left as issues, in one minor release because one of them adds a public name.

#72: all or nothing. Instance hard_delete() under another tenant's scope committed half a tree: the row policy hid the root, so its DELETE removed nothing, while its children, on a table with no policy, were removed for good. Each DELETE that deletes keys collected beforehand now has to remove exactly that many rows, or it raises the new HardDeleteIncompleteError (a GuitarsError, exported from guitars.models) and the walk rolls back. That covers every table of the instance walk (an empty scope counts as zero removed), the MTI queryset form's own table and ancestors (a descendant table holds rows only for the keys that are one), and #73's key-read path. The count comes with each DELETE, so no extra statement. ADR 0032.

#71: sparing reads the whole closure. An owned target was spared only for a key into the target itself, so a plain key into a descendant aborted the walk at COMMIT. _cascade_closure now records which targets take each row: the recursive query carries its seed, and the level walk carries targets through each key. A row reached again with a new target is walked again, so a row two targets reach spares both. _still_referenced asks the referrers of every closure row, grouped by MTI tree, and spares exactly the targets they lead back to: one read per relation into each model the closure reaches, never one per row. Testapp gains CueNote (a plain key into a Cue) and an empty retirement host after its migrations.

#73: keys first where a WHERE cannot hold the filter. A window or single-table aggregate filter made the plain queryset form's DELETE invalid. It now reads the matched keys (as delete() does) and removes them in batches, each held to its count; a plain filter stays one DELETE. The new _hard_delete_by_key is denied on an unscoped tenant queryset.

Also. Queryset hard_delete() returns None in every form; the plain one returned a closed cursor. Dependencies: Django 5.2.18 and 6.0.9, sqlparse 0.6.0. This supersedes #44, whose lock, cut against 2.11.0, would have merged the two Django forks.

Tests. tests/test_hard_delete_incomplete.py (the #72 reproduction; a concurrent writer simulated mid-walk; the MTI chain; the key-read path), #71's reproduction at child and grandchild depth plus attribution unit tests (only the anchored target; a row two targets reach; a row reached along two relations; a row going anyway; a key into a non-Ledger closure row), and #73's window and aggregate filters with a one-statement check. Each guard was mutated away and a test failed.

Gate. pytest at 100% coverage, ruff, format, ty, bandit, uv lock --check, makemigrations --check, doc budget, and the Django 5.0 and 6.0 cells.

Closes #72, #71, #73. Supersedes #44.

🤖 Generated with Claude Code

Behnam-RK and others added 6 commits October 7, 2026 10:13
…ck (#72)

Instance hard_delete() under another tenant's scope committed half a tree: the
row policy hid the root, so its DELETE removed nothing, while the children's
policy-free table let theirs go. Each DELETE the walk issues now has to remove
exactly the rows collected for its table, or it raises the new
HardDeleteIncompleteError and the walk rolls back. The same holds for an empty
scope (a table compiling to nothing), another transaction removing a collected
row first, and the MTI queryset form's own chain, where a hidden row left the
chain half removed. A descendant's table, holding rows only for the keys that
are one, has no count to meet.

The plain queryset form's DELETE primitive now returns its row count; queryset
hard_delete() returns None in every form, where the plain one returned a closed
cursor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its closure (#71)

_still_referenced asked which surviving rows point at an owned target, but
never which point at the rows its cascade takes along. So a plain key from
outside into a descendant left the target unspared, its subtree was removed,
and the walk aborted at COMMIT. The closure now records, per row, which
targets take it: the recursive query carries the seed it descends from, the
level walk carries targets through each key, and a row reached again with a
new target is walked again so a row two targets reach spares both. Sparing
asks the referrers of every closure row, grouped by MTI tree, and spares
exactly the targets they lead back to. One query per relation, as before.

Testapp gains CueNote (a plain key into a Cue) so a closure row of a model
other than the target's is covered, and an empty retirement host after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#73)

The plain queryset form compiled its query as a DELETE, so a filter on a
window function or a single-table aggregate landed in the DELETE's WHERE and
PostgreSQL refused it, where delete() reads the keys first and succeeds. When
the WHERE holds an aggregate or a window, the form now reads the matched keys
(_matching_pks, as delete()'s fast path does) and removes them in batches on a
bare queryset, each batch held to removing every key it names. A plain filter
stays one DELETE. The new _hard_delete_by_key is denied on an unscoped tenant
queryset like its siblings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Supersedes dependabot #44, which was cut against 2.11.0's lock and would have
collapsed the Django 5.2 and 6.0 forks into one. Upgraded within each fork
(django<6.1) instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR 0032 records why hard_delete() rolls back when a DELETE removes fewer rows
than it collected, where it counts and why its error names no cause. The
soft-deletion, owned-relations and API reference docs say what 2.15.0 changed.
A test covers the key-read path reading nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the target alone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e a row already gone

Round 1 of the review loop on #75.

- A queryset filtered across a many-valued relation reads a key once per joined
  row, so the MTI form and the key-read path counted duplicates and raised
  HardDeleteIncompleteError over a clean delete (a regression in the MTI form).
  Both count distinct keys now.
- _with_self_descendants had become a wrapper over the (pk, origin) CTE, so the
  collection walk read one row per seed above each node, quadratic in depth
  (1.54 s against 0.11 s on a 1500-deep chain). It reads each row once again;
  only the sparing closure pays for origins.
- A row already gone, because its table has no soft-delete rule yet and
  Phase 1's delete() really removed it, is named among the causes; the walk
  still fails closed (maintainer's decision). ADR 0032 and the CHANGELOG say
  so, including that hard_delete() completed there through 2.14.
- A GenericRelation child is not in the sparing closure: filed as #76, and
  the docs and the comment that called the gap harmless are corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Behnam-RK

Copy link
Copy Markdown
Owner Author

Review round 1

Read-only reviewer over origin/main...a2dce3d. Fixes in 8d22ea4.

# Finding Class Origin Outcome
1 A queryset across a many-valued relation reads a key once per joined row: the MTI form and the key-read path counted duplicates and raised over a clean delete (the MTI form a regression from main) defect (high) this PR fixed: distinct keys; two tests, red before
2 _with_self_descendants became a wrapper over the (pk, origin) CTE, so the collection walk read one row per seed above each node: 1.54 s vs 0.11 s on a 1500-deep chain defect (perf) this PR fixed: the walk reads each row once again; a test counts the rows fetched (210 before, 19 after)
3 A GenericRelation child is not in the sparing closure, so #71's failure is still reachable through one; the docs said otherwise defect (fails closed) gap predates; claim is this PR's deferred to #76; docs and the comment calling the gap harmless corrected
4 A root already gone (no soft-delete rule yet, or Model(pk=<gone>)) now raises, and the message blamed scope or concurrency product decision this PR asked: keep failing closed; the message, ADR 0032 and the CHANGELOG name the cause and that 2.14 completed there

Checked clean: row counts across MTI levels, self-referential subtrees, rule rewriting; a 300-trial differential of _still_referenced against a per-target brute force (0 mismatches); #73's detection over Exists, subqueries and aggregates over joins.

Gate at 8d22ea4: pytest 100% coverage, ruff, format, ty, bandit, uv lock --check, makemigrations --check, doc budget, Django 5.0 and 6.0 cells.

🤖 Generated with Claude Code

…e walk's reader pinned

Round 2 of the review loop on #75.

- hard_delete()'s fallback for a model without _all_objects runs Django's
  collector, which removes a plain MTI child's parent rows with it; the
  parent's own entry, later in child-first order, then removed nothing and
  the row count aborted a clean walk. The fallback now credits those parent
  rows to their own entries. The branch is no longer `no cover`: testapp
  gains Rig (an owned target of its own), Roadie (owning it), and the plain
  chain Gear/Amp under it, with their migrations and an empty retirement host.
- A test pins that the collection walk never reads origin pairs.
- The error's docstring and docs/soft-deletion.md name all three causes; an
  owned key pointing at no row (db_constraint=False) fails closed too, kept
  and recorded in ADR 0032 as an accepted cost.
- A test docstring overclaimed what it reproduced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Behnam-RK

Copy link
Copy Markdown
Owner Author

Review round 2

Read-only reviewer over origin/main...8d22ea4, scoped at round 1's fixes and the files round 1 did not cite. Fixes in a6e0871.

# Finding Class Origin Outcome
1 A plain (non-soft-deletable) MTI chain under an owned row: the fallback's collector removes the parent rows with the child, so the parent's own entry removed nothing and the count aborted a clean walk defect this PR fixed: parent rows the collector took are credited to their entries; the branch is now tested (testapp Rig/Roadie/Gear/Amp), no longer no cover
2 An OwningForeignKey(db_constraint=False) pointing at no row: the target is collected from the key and its DELETE removes nothing, so the owner's hard_delete() aborts trade-off this PR kept failing closed; recorded in ADR 0032 (null the key first)
3 Nothing pinned that the collection walk uses the row-only reader test gap round 1 test: the walk never calls _self_descendant_origins
4 The error's docstring and docs/soft-deletion.md named two causes, the message three doc round 1 fixed
5 A test docstring claimed a "no rule yet" scenario it does not build test this PR docstring corrected

Loop-introduced: two (#3, #4). Checked clean: distinct keys everywhere a count is taken; the row-only and origin readers agree on 480 random graphs; 0069 regenerates byte-identically; nine mutants of the PR's guards each killed.

Gate at a6e0871: pytest 100% coverage, ruff, format, ty, bandit, uv lock --check, makemigrations --check, doc budget, Django 5.0 and 6.0 cells.

🤖 Generated with Claude Code

…ollector deletes

Round 3 of the review loop on #75. A plain (non-soft-deletable) model under an
owned row goes through Django's collector, which cascades by its own rules: a
plain model reached a second time removes children whose own entries come
later, and round 2's credit for MTI parents missed that shape, still aborting
clean walks. Such a table is no longer counted, as the maintainer decided: no
row policy hides a plain row, so the hidden-row case the count exists for
cannot occur there. The credit logic goes; the Rig/Gear/Amp test stays and
fails if a count comes back. ADR 0032, the CHANGELOG and the docs scope the
rule to soft-deletable tables and say why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Behnam-RK

Copy link
Copy Markdown
Owner Author

Review round 3

Read-only reviewer over origin/main...a6e0871, scoped at round 2's fallback credit, the new testapp models and a final read. Fix in ca4c034.

# Finding Class Origin Outcome
1 The fallback's count still aborted clean walks: Django's collector, reaching a plain model a second time, removes children (or MTI children) whose own entries come later; round 2 credited MTI parents only, and its comment said nothing else could happen defect (fails closed) this PR, incompletely fixed in round 2 asked: plain tables are no longer counted (no row policy hides a plain row); credit logic removed; ADR 0032, CHANGELOG and docs scope the rule to soft-deletable tables; the Amp/Gear test fails if a count returns

Checked clean: a three-level plain MTI chain, a plain child over a non-plain parent (impossible), by_collector across groups; 0072 regenerates byte-identically and moves no other model's SQL; full suite on a copy, 1830 passed.

Gate at ca4c034: pytest 100% coverage, ruff, format, ty, bandit, uv lock --check, makemigrations --check, doc budget, Django 5.0 and 6.0 cells.

🤖 Generated with Claude Code

… no policy

Round 4 of the review loop on #75. The comment and ADR 0032 said no row
policy hides a plain row; a plain model with tenanted_manager() gets one. What
holds instead: a plain row reaches the walk only through a SELECT under the
same policy as its DELETE, so a hidden row is never collected; a read-only
exempt role is the exception, failing at COMMIT on the deferred key. The
root-already-gone test now matches the table and counts, not a word every
message carries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Behnam-RK

Copy link
Copy Markdown
Owner Author

Review round 4 and ledger

Read-only reviewer over origin/main...ca4c034, asked whether the PR's own lines are done. Fix in 47b78d2.

# Finding Class Origin Outcome
1 The comment and ADR 0032 said "no row policy hides a plain row"; a plain model with tenanted_manager() gets one. The decision stands on the corrected reason: a plain row reaches the walk only through a SELECT under the same policy as its DELETE; a read-only exempt role fails at COMMIT on the deferred key instead doc round 3 corrected; the premise was also in the question put to the maintainer, who was told

Also: a test's match named a word every message carries; it now matches the table and counts. Checked clean: the uncounted collector cannot make a soft-deletable count misfire (probed: a cycle through a plain and a soft-deletable model, the collector's DELETE archived, the walk's own removes 1 of 1); every doc sentence against the code; no vacuous test.

Round Findings Pre-existing Loop-introduced Outcome
1 4 4 0 duplicate keys; quadratic walk read; #76 deferred; root-gone message (asked)
2 5 3 2 plain MTI chain false abort; dangling owned key (kept, ADR); test gap; wording
3 1 1 0 fallback count dropped (asked)
4 1 0 1 false rationale corrected

Stop: round 4 produced no new defect in the diff's lines, and loop-introduced findings outnumber pre-existing ones. Every file in the diff has been cited. Deferred: #76. Waiting on the maintainer's read of the prose (ADR 0032, CHANGELOG 2.15.0, docs/soft-deletion.md, docs/owned-relations.md, docs/api-reference.md).

Gate at 47b78d2: pytest 100% coverage; at ca4c034 also ruff, format, ty, bandit, uv lock --check, makemigrations --check, doc budget, Django 5.0 and 6.0 cells (47b78d2 changes a comment, docs and one test).

🤖 Generated with Claude Code

@Behnam-RK
Behnam-RK merged commit b374828 into main Oct 7, 2026
17 checks passed
@Behnam-RK
Behnam-RK deleted the fix/hard-delete-bundle branch October 7, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hard_delete() under another tenant's scope permanently deletes the root's untenanted children and leaves the root live

1 participant