Fix deleted entities bug for joint endpoints

[sianteesdale]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

Fixes count_deleted_entities flagging entities as deleted when organisations share a resource, such as joint plans.

The query that collects the entities in an organisation's active resources ended in group by reference, which returns one entity per reference. Organisations that publish the same resource each have their own entity for the same reference, so only one organisation's entity survived the grouping. The others were reported as deleted, even though they're in the active resource.

The query now uses select distinct f.entity instead. The result is only used as a set of entity IDs, so grouping by reference served no purpose.

Why

Reported for local-authority:SNO plan-timetable: all of its entities are in its latest resource (see check-transform), but the expectation flagged 12 as deleted.

• SNO and BRO share an endpoint (13d5fb57…), so they share a resource (7609ee1a…).
• That resource holds 12 BRO entities and 12 SNO entities, with the same references, e.g. GNLP2045-ADOPT → 5111030 (BRO) and 5111352 (SNO).
• group by reference kept BRO's 12, so SNO's 12 were flagged.

This isn't related to the paging fix in #611, or cause by code changes in #608. SNO's resource lists were complete, and plan-timetable is well under 1,000 rows.

Impact

On live data, every active resource shared by more than one organisation loses entities to the grouping:

Dataset	Shared active resources	Entities wrongly flagged	Currently flagged in total
plan-timetable	7	98	161
local-plan	3	5	11

Affected organisations are all joint plans: NDE/TOR, BRO/SNO, SHA/WDE, SOX/VAL, BAB/MSU(/BAB-MSU), NET/GAT-NET, MAV/WOC/WYC(/MAV-WOC-WYC) and NOW/BRO-NOW-SNO.

Expect fewer count_deleted_entities failures for these organisations after the next nightly run. Genuine deletions are still caught.

Related Tickets & Documents

• Ticket Link: deleted entities expectation should check data only from same org config#2986

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

• Yes
• No, and this is why: please replace this line with details on why tests
  have not been included
• I need help with writing tests

Summary by CodeRabbit

• Bug Fixes
  • Improved deleted-entity counts when organisations share an active resource. Each organisation’s entities are now assessed separately, so entities belonging to another organisation are not incorrectly treated as missing.
  • Confirmed that entities supplied only through an ended resource continue to be counted as deleted.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 20d85154-cc48-48c3-8bfe-711d78af936b

📥 Commits

Reviewing files that changed from the base of the PR and between f761751 and b5410bb.

📒 Files selected for processing: 2

• digital_land/expectations/operations/dataset.py
• tests/integration/expectations/operations/test_dataset.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

count_deleted_entities now selects distinct entity IDs from active resources. An integration test covers two organisations that share a resource and an entity that appears only in an ended resource.

Changes

Deleted entity counting

Layer / File(s)	Summary
Active entity selection and integration coverage digital_land/expectations/operations/dataset.py, tests/integration/expectations/operations/test_dataset.py	The query selects distinct entity IDs from active resources. The integration test checks that shared-resource entities remain active for both organisations and that entity 3002, found only in an ended resource, counts as deleted for organisation 304.

Priority: ⬆️ High

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: tombrooks248, kena-sl

Merge Risk: ⚪ Minimal · up to b5410

The change is mergeable after normal checks; no outstanding issue is identified.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarises the main change: fixing incorrect deleted-entity reporting for joint endpoints.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}