Update deleted entities expectation

[sianteesdale]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

count_deleted_entities flagged entities that the LPA never supplied. It treated every entity belonging to an organisation as something that should appear in that organisation's active resources. But MHCLG seeds data on behalf of LPAs, so those entities belong to the LPA without ever being in the LPA's own data. They were always reported as "deleted".

For example, entity 5100307 belongs to local-authority:MAI, but MHCLG supplied it. Maidstone has never had a plan-timetable endpoint, so all 32 of its plan-timetable entities were flagged.

Fix

An entity now only counts as deleted if the organisation supplied it at some point (in any of its resources, current or ended) and it's missing from the organisation's active resources. This handles the organisation-to-organisation case. It also covers shared endpoints, e.g. Hackney brownfield-land entities supplied through the GLA/Harrow endpoint.

Changes in digital-land-python (digital_land/expectations/operations/dataset.py):

• Added fetch_all_resources_for_dataset, which fetches every resource each organisation has supplied, including ended ones. The checkpoint prefetches it once per dataset, like the existing active-resources cache.
• count_deleted_entities takes a new all_resources_cache and only considers entities with at least one fact from one of those resources.
• Updated the tests and added new ones for the third-party-supplied case and the new fetch.

Impact (old rule vs new rule)

Dataset	Before	After	No longer flagged
local-plan	770	8	759
plan-timetable	6,560	110	6,388
plan-timetable (MAI)	32	0	32

Related Tickets & Documents

• Ticket Link: deleted entities expectation should check data only from same org config#2986

Summary by CodeRabbit

• Bug Fixes
  • Deleted-entity counts now include only entities previously supplied by the organisation being checked, including those found in ended resources. Entities supplied only by other organisations are excluded.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 13d0a2e0-3ff2-4427-b213-9812ccedef46

📥 Commits

Reviewing files that changed from the base of the PR and between 876c77b and 350d43f.

📒 Files selected for processing: 4

• digital_land/expectations/checkpoints/dataset.py
• digital_land/expectations/operations/dataset.py
• tests/integration/expectations/checkpoints/test_dataset.py
• tests/integration/expectations/operations/test_dataset.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

Dataset expectation checks now prefetch active and historical resources. Deleted-entity checks use historical resource facts to restrict comparisons to entities previously supplied by the organisation.

Changes

Dataset expectation resource caches

Layer / File(s)	Summary
Fetch and pass resource caches digital_land/expectations/operations/dataset.py, digital_land/expectations/checkpoints/dataset.py, tests/integration/expectations/checkpoints/test_dataset.py, tests/integration/expectations/operations/test_dataset.py	Shared fetch logic retrieves active or historical resources. The checkpoint passes each cache only to operations whose signatures accept it. Tests cover historical resource retrieval and cache passing.

Deleted-entity comparison scope

Layer / File(s)	Summary
Scope deleted-entity comparisons digital_land/expectations/operations/dataset.py, tests/integration/expectations/operations/test_dataset.py	count_deleted_entities uses historical resources to identify entities previously linked to the organisation. Tests cover ended-resource facts and exclude entities found only in another organisation’s resources.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant DatasetCheckpoint
  participant DatasetResourceFetchers
  participant DatasetExpectationOperation
  DatasetCheckpoint->>DatasetResourceFetchers: Fetch active and historical resources
  DatasetResourceFetchers-->>DatasetCheckpoint: Return resource caches
  DatasetCheckpoint->>DatasetExpectationOperation: Pass supported resource caches

Loading

Suggested reviewers: tombrooks248

Merge Risk: ⚪ Minimal · up to 350d4

The deleted-entity check uses the required active and historical resource sets without the claimed uncached cost on the supported command path. No identified issue remains to resolve before merge.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 47.06% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly describes the main change to the deleted entities expectation. It is concise and specific.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}