#711 16 to 19 Include journey (AB#301154)

AGENTS.md

@@ -181,6 +181,8 @@ Results enquiries and 16-19 windows: `docs/results-enquiry.md`, `docs/16-19-wind
   `x-ingress.source`. Never put a supplier column name in a C# property name.
 - **Inclusion is the `INCLUDED` stamp.** Ingress decides it once. Code that reads pupils must
   not know a key stage's inclusion codes.
+- **Who may be included is the `INCLUDABLE` stamp.** Ingress writes it from the schema's
+  `x-ingress.includable` rule (#711). The Include journey reads the stamp, never a code.
 - **The service is not live yet.** Do not write fallbacks for data written before a change.
   Treat missing new-format data as an error. (This does not change the migration rule above:
   that rule protects deployed databases.)

docs/blob-storage.md

@@ -44,7 +44,15 @@ reach the journey as `QAN`, `SYLLABUS`, `SESSION` and `QUAL_NAME`; in the 16-18
 schools see. Schools see the label, never the tag. A dataset's `SourceFile` is the exact analogue of its `Included`: `Included` stamps inclusion
 by file of origin, `SourceFile` stamps provenance, and both are stamped by the processor before
 validation and guarded by the schema declaring the column (`INCLUDED` / `SOURCE`), because
-`AllowAdditionalProperties` is false. A run writes
+`AllowAdditionalProperties` is false. A third stamp, `INCLUDABLE` (#711), says whether a school may
+ask to include the pupil: true when the record is not included, or when its code is one the
+schema's `x-ingress.includable` rule names (the 16-19 included file names `P_INCL` 502). It is
+derived from `INCLUDED`, so it follows the same guard. The Include journey reads it. A rule with
+no field, an undeclared field, no codes or a blank code fails the run by name before any row is
+read, and so does a schema that declares `INCLUDED` but not `INCLUDABLE`: it would otherwise
+offer nobody for inclusion. A pupil blob written before `INCLUDABLE` existed lacks it, on every
+key stage, and reads as nobody includable: the Include search finds nobody, and the "already
+included" check matches every typed name. Validate the exercise again to restamp it. A run writes
 `CheckingExerciseBlobPaths.DataBlobName(exercise, laestab)` — never `PupilsBlobName` directly: the
 two normalise the laestab differently on purpose. Every feed must carry a `LAESTAB` column, or the column its schema's `x-ingress.institutionKey` names
 (`LAESTAB_0` in the previously published file): it is what splits one supplier file into one blob per school; a file without it fails the run by name.

docs/domain-model.md

@@ -135,7 +135,7 @@ Session readiness for the journey requires only `SelectedWhatToChange` and `Chec
 - The controller holds **no `TimeProvider`** — every "is it open" question goes through `ICheckingExerciseService`.
 - Displays paged, searchable pupil lists for a given checking window and school (by `laestab`).
 - `CheckYourPupilDataRepository` loads the school's pupil JSON (cached) and filters/searches/sorts/pages **in memory** — sorted by surname/firstname. **The pupil search boxes share one rule, `PupilSearchTerms`** (parse the query once with `Parse`, then `Matches` per row): the query splits on spaces, tabs and commas, and each word must match the start of a word in some field (words start after a space, tab, hyphen or apostrophe), ignoring case, in any order ("John Smith", "Smith, John"; "ohn" finds nothing). Identifiers match from their start too ("A860" finds a UPN, "8604" does not). The journey autocomplete checks names, UPN/ULN, CYPMD ID and displayed DOB; the pupil tabs check names; the results table checks names, CYPMD ID and subject; the schema-driven tabs check the schema's `searchable` columns. Only the Add-pupil duplicate check (fuzzy on purpose, not a search box) does not use it.
-- Pupils are split into included (`IsIncluded`, the `INCLUDED` stamp) and non-included (all others) tabs.
+- Pupils are split into included (`IsIncluded`, the `INCLUDED` stamp) and non-included (all others) tabs. A second stamp, `INCLUDABLE` (#711), says who a school may ask to include; only the Include journey reads it.
 - **Since #466 the tabs can be exercise-driven.** `IExerciseTabBuilder` (Application) returns one
   tab per *visible* exercise — enabled, within `VisibleFrom`/`VisibleUntil` (exclusive at the end),
   and every exercise has a `TabName`. When no exercise is visible the list is empty and the page
@@ -176,7 +176,7 @@ Session readiness for the journey requires only `SelectedWhatToChange` and `Chec
 - Multi-step question flow for submitting a pupil data change request.
 - Session state (`RequestState`) tracks answers, page history, selected pupil(s), and reference number.
 - Pupil selection is an in-journey `PupilSearch` page (the first page of every flow config). `WhatToChangeController` redirects to `Journey.Page(firstPageId)` after saving the window and `WhatToChange` to session. The journey's `IsSessionReady` check requires only those two fields.
-- **What to change offers a change only when its flow file exists** (`IQuestionFlowService.GetPupilDataChangesAsync`: Merge, Include, Remove, Add, in that order, each only when `{WhatToChange}_{CheckingWindowType}.json` exists). The view renders `WhatToChangeViewModel.AvailableChanges` and names no change or window type; the POST refuses any change outside the same list. To give a window type a journey, add its flow file. There are no window-type lists.
+- **What to change offers a change only when its flow file exists** (`IQuestionFlowService.GetPupilDataChangesAsync`: Merge, Include, Remove, Add, in that order, each only when `{WhatToChange}_{CheckingWindowType}.json` exists). The view renders `WhatToChangeViewModel.AvailableChanges` and names no change or window type; the POST refuses any change outside the same list. To give a window type a journey, add its flow file. There are no window-type lists. 16-19 offers Include since #711: `Include_Post16.json`, whose search uses `pupilFilter: "Includable"` (the `INCLUDABLE` stamp — the non-included file plus the included file's P_INCL 502 students). Every 16-19 include goes to Scrutiny: the `Inclusion` rules name KS4 codes only.
 - On submission, `IRequestService.ConfirmRequestAsync` builds the `RequestDocument` (including an optional `MatchedPupil` for Merge) and saves it to blob storage (container named by `windowId`, blob `request_{referenceNumber}.json`). The close sweep sends this saved copy.
 - The Summary page shows "First record to merge" / "Second record to merge" rows (with Change links to the respective `PupilSearchPage`) when `MatchedPupil` is set; otherwise shows the standard "Pupil name" row.
 
@@ -240,7 +240,7 @@ through `IExerciseEarlyClosureService`, then sweeps). Once it has closed, the su
 Requests page is removed.
 
 **`ICheckYourPupilDataService`** — pupil data queries (partial)
-`GetPupilSuggestionsAsync(windowId, query, PupilFilter filter, Guid? excludeId, bool requireResults)` returns up to 10 `PupilSuggestionDto(Id, Label)` records. `PupilFilter.Included` restricts to pupils stamped `INCLUDED`; `PupilFilter.All` returns all pupils for the school. `excludeId` removes a specific pupil from results (used by the match-pupil page to prevent selecting the primary pupil again). `requireResults` restricts the search to students in `IResultsClient.GetCypmdIdsWithResultsAsync` (the same cached school results file the enquiry itself reads), and is passed down to `SearchPupilsAsync` as a CYPMD id allow-list so it applies **before** the ten-suggestion cap — filtering afterwards would drop the one student who holds results whenever ten who do not sort ahead of them. Served by `PupilSuggestionsController` at `GET /pupils/suggestions`. The `WhatToChange?`-based overload no longer exists — callers must pass an explicit `PupilFilter`.
+`GetPupilSuggestionsAsync(windowId, query, PupilFilter filter, Guid? excludeId, bool requireResults)` returns up to 10 `PupilSuggestionDto(Id, Label)` records. `PupilFilter.Included` restricts to pupils stamped `INCLUDED`; `PupilFilter.All` returns all pupils for the school; `PupilFilter.Includable` restricts to pupils stamped `INCLUDABLE` and `PupilFilter.NotIncludable` to the rest (#711). `excludeId` removes a specific pupil from results (used by the match-pupil page to prevent selecting the primary pupil again). `requireResults` restricts the search to students in `IResultsClient.GetCypmdIdsWithResultsAsync` (the same cached school results file the enquiry itself reads), and is passed down to `SearchPupilsAsync` as a CYPMD id allow-list so it applies **before** the ten-suggestion cap — filtering afterwards would drop the one student who holds results whenever ten who do not sort ahead of them. Served by `PupilSuggestionsController` at `GET /pupils/suggestions`. The `WhatToChange?`-based overload no longer exists — callers must pass an explicit `PupilFilter`.
 
 **`ICountryService`** — country/autocomplete lookup
 `SearchAsync(query)` returns up to 10 `CountrySuggestionDto(Code, Name)` records matching the query against `Country.Name` (case-insensitive `ILike`). Used by `CountryController.Suggestions` to serve the `Autocomplete` question type. The underlying `Countries` table is seeded from the FCDO Geographical Names Index and is not checking-window-specific.

docs/ks2-journeys.md

@@ -28,7 +28,7 @@ KS2 has no Merge, Include or Add journey. The requirements say so.
    | Window | Flow files | Options shown |
    |---|---|---|
    | KS4 June | Add, Include, Merge, Remove | All four (no change) |
-   | Post16 | Merge, Remove | Merge, Remove (no change) |
+   | Post16 | Include, Merge, Remove | Include (since #711), Merge, Remove |
    | KS4 Autumn | Add | Add only. Before, it also showed Merge, Include and Remove, which had no flow file and went nowhere. |
    | KS2 | Remove | Remove only |
 

docs/question-flows.md

@@ -36,13 +36,13 @@ The journey feature (pupil change requests) is driven by JSON config files that
 
 | Field | Required | Values | Notes |
 |---|---|---|---|
-| `pupilFilter` | yes | `"Included"` / `"All"` | `Included` limits results to pupils stamped `INCLUDED`; `All` returns every pupil for the school |
+| `pupilFilter` | yes | `"Included"` / `"All"` / `"Includable"` / `"NotIncludable"` | `Included` limits results to pupils stamped `INCLUDED`; `All` returns every pupil for the school; `Includable` returns pupils stamped `INCLUDABLE` — the Include journey's search, which on 16-19 adds the included file's P_INCL 502 students to the non-included file (#711); `NotIncludable` is the rest, which the Include journey's "already included" guards look a name up in |
 | `pupilKey` | yes | `"primary"` / `"match"` | `primary` saves to `RequestState.SelectedPupil*` and generates the reference number; `match` saves to `RequestState.MatchedPupil*` |
 | `requireResults` | no | `true` / `false` (default) | Limits the search to students the school holds a 16-19 result for. Independent of `pupilFilter`, which selects by inclusion status. Set on the results-enquiry student search — there is no grade to correct for a student with no result. The page must say so in its `subheading`, because the restriction silently hides students |
 | `nextPageId` | no | page id string | Absent means redirect to Summary after selection |
 | `validationFailure` | no | string (supports `{pupilName}`) | Error shown when no pupil is submitted. Falls back to `"Enter the name of the pupil"` |
 
-**Suggestions endpoint:** `GET /pupils/suggestions?windowId={id}&query={q}&filter=Included|All&excludePupilId={guid}&requireResults=true` (served by `PupilSuggestionsController`). The `match` pupil page automatically passes the primary pupil's ID as `excludePupilId` so it cannot be selected twice. `requireResults` is sent only when the page config sets it; it is a search restriction, never a permission — it can reach no pupil outside the signed-in school's own file. When it is on, the autocomplete's no-match text changes to "No students found with results" so a school can tell a typo from a student who holds nothing.
+**Suggestions endpoint:** `GET /pupils/suggestions?windowId={id}&query={q}&filter=Included|All|Includable|NotIncludable&excludePupilId={guid}&requireResults=true` (served by `PupilSuggestionsController`). The `match` pupil page automatically passes the primary pupil's ID as `excludePupilId` so it cannot be selected twice. `requireResults` is sent only when the page config sets it; it is a search restriction, never a permission — it can reach no pupil outside the signed-in school's own file. When it is on, the autocomplete's no-match text changes to "No students found with results" so a school can tell a typo from a student who holds nothing.
 
 **Navigation:** `JourneyController.Page` (GET) transparently dispatches to `PupilSearchPage` when the requested page has `Type == PupilSearch`, so all navigation (summary redirects, `GetNextPageId`, navigation guard) works uniformly.
 

docs/request-journey.md

@@ -120,7 +120,7 @@ All flow configs begin with one or more `PupilSearch` pages. These are full-page
 
 | Field | Required | Values | Notes |
 |---|---|---|---|
-| `pupilFilter` | yes | `"Included"` / `"All"` | `Included` limits results to Pincl codes `[401,403,414,421,431]`; `All` returns every pupil for the school |
+| `pupilFilter` | yes | `"Included"` / `"All"` / `"Includable"` / `"NotIncludable"` | `Included` limits results to pupils stamped `INCLUDED`; `All` returns every pupil for the school; `Includable` returns pupils stamped `INCLUDABLE` — the Include journey's search, which on 16-19 adds the included file's P_INCL 502 students to the non-included file (#711); `NotIncludable` is the rest, which the Include journey's "already included" guards look a name up in |
 | `pupilKey` | yes | `"primary"` / `"match"` | Controls which session field is populated (see below) |
 | `requireResults` | no | `true` / `false` (default) | Limits the search to students the school holds a 16-19 result for. Independent of `pupilFilter`, which selects by inclusion status. Used by the results enquiry — there is no grade to correct for a student with no result. A page that sets it must say so in its `subheading`, because the restriction hides students silently |
 | `pupilSearchField` | no | `"All"` / `"CypmdId"` (default `All`) | Which field(s) the query is matched against. `All` (or absent) matches forename, surname, split names, UPN/ULN and CYPMD ID, plus date of birth on 16-19. `CypmdId` matches **only** a trimmed, case-insensitive prefix of `Cypmd_Id` — a name, UPN or date of birth returns nothing. Set on `Merge_KS4June.json`'s `select-match-pupil`, whose `subheading` already reads "Start typing ID to search records"; had it said nothing, narrowing the search would hide records silently and read as a typo |
@@ -129,7 +129,7 @@ All flow configs begin with one or more `PupilSearch` pages. These are full-page
 
 `pupilSearchField` also changes what the suggestion text says. On a `CypmdId` page a KS4 suggestion is `"Surname, Firstname, DOB (Cypmd_Id)"`, because the KS4 suggestion leads with name and date of birth — which are identical across the duplicate records a merge exists to reconcile, so without the ID two rows in the dropdown cannot be told apart. `All` keeps the original `"Surname, Firstname, DOB"`; 16-19 is unchanged either way, as it already prints `CYPMD ID:`. The narrowed value is a property of one page: the merge journey's `select-pupil` page and every other journey keep `All`.
 
-**Suggestions endpoint:** `GET /pupils/suggestions?windowId={id}&query={q}&filter=Included|All&excludePupilId={guid}&requireResults=true&pupilSearchField=All|CypmdId`  
+**Suggestions endpoint:** `GET /pupils/suggestions?windowId={id}&query={q}&filter=Included|All|Includable|NotIncludable&excludePupilId={guid}&requireResults=true&pupilSearchField=All|CypmdId`  
 Served by `PupilSuggestionsController`. Reads the school's pupils from the cached per-school-window blob through `ICheckYourPupilDataService.GetPupilSuggestionsAsync` (`CheckYourPupilDataRepository`); it does **not** query PostgreSQL — the `Pupils` table was dropped when pupil data moved to blob storage. The `match` pupil page automatically passes the primary pupil's ID as `excludePupilId` so the same pupil cannot be selected twice.
 
 `requireResults` is sent only when the page config sets it. The service resolves the school's set of CYPMD ids with results (`IResultsClient.GetCypmdIdsWithResultsAsync`, served from the same cached results file the enquiry itself reads) and passes it to `SearchPupilsAsync` as an allow-list, which applies it **before** the ten-suggestion cap — filtering afterwards would drop the one student who holds results whenever ten who do not sort ahead of them. It is a search restriction, never a permission: it can reach no pupil outside the signed-in school's own file. With it on, the autocomplete's no-match text becomes "No students found with results", so a school can tell a typo from a student who holds nothing.
@@ -164,6 +164,8 @@ POST → saves MatchedPupil
      → no nextPageId → redirects to Summary
 ```
 
+**Include** uses one `PupilSearch` page with `pupilFilter: "Includable"` (the `INCLUDABLE` stamp: not included, or a code the schema names — 16-19 P_INCL 502), then an evidence page; `Include_KS4June.json` and `Include_Post16.json` (#711).
+
 **Session after primary PupilSearch page:**
 | Field | Value |
 |---|---|

docs/technical-stakeholders-functional-overview.md

@@ -78,7 +78,7 @@ The journey is data-driven from JSON flow configs:
 
 - **Types** (`Application/Journey/`): `QuestionFlowConfig` → `JourneyPage` (`PageType`: `Question`, `Content`, `EvidenceUpload`, `PupilSearch`) → `Question` (`Radio`, `FreeText`, `Date`, `FileUpload`, `TextArea`, `Autocomplete`; supports `Optional`, `CharacterLimit`, named `Validator`, per-option `NextPageId` branching, `VisibleWhen` conditions).
 - **Config source**: the files that ship in the release image at `Web/Data/QuestionFlows/{WhatToChange}_{CheckingWindowType}.json`, read via `IQuestionFlowConfigSource` / `FileSystemQuestionFlowClient` and cached in `IMemoryCache` with `NeverRemove` priority (flow edits need a redeploy). There is **no blob container and no seeding step** — a flow reaches an environment by deploying the commit that contains it. See `docs/question-flow-deployment.md`.
-- **Flows that exist today**: `Include_KS4June.json`, `Merge_KS4June.json`, `Remove_KS4June.json` (12-reason branch tree). **Only KS4 June has flows** — for KS2/Post16/KS4Autumn the config lookup returns null and `WhatToChangeController.Confirm` bounces back to the pupil list.
+- **Flows that exist today**: KS4 June — `Add_KS4June.json`, `Include_KS4June.json`, `Merge_KS4June.json`, `Remove_KS4June.json`; KS4 Autumn — `Add_KS4Autumn.json`; KS2 — `Remove_KS2.json`; 16-19 — `Include_Post16.json` (#711), `Merge_Post16.json`, `Remove_Post16.json`, plus the results-enquiry flows `IncorrectGrade_Post16.json`, `MissingQualification_Post16.json`, `ResultDoesNotBelong_Post16.json`. What to change offers exactly the pupil-data changes whose flow file exists for the window type (`IQuestionFlowService.GetPupilDataChangesAsync`); a posted change with no flow file is refused.
 - **State**: `RequestState` per window in session — answers keyed by question id, plus `QuestionHistory` (ordered visited pages). Session is a **Postgres distributed cache** (`AddDistributedPostgreSqlCache`, table `session_cache`) so journeys survive load-balancing across replicas. Auth cookie is 30-min sliding; there is **no auto-save** — drafts persist only on explicit "Save and continue later".
 - **Mechanics** (`JourneyController`, ~750 lines): navigation guard prevents URL-hacking ahead of answered history; changing a branch answer trims downstream history; `fromSummary=true` returns to Summary after single-page edits unless the branch changed. Evidence upload: **PDF only** (verified by PdfPig page counting in `Web/FileStorage/PdfPageCounter.cs`), **10 MB max per file**; the page-count cap exists but is **disabled** (`maxEvidencePages = 0` at `Application/DependencyManager.cs`). Validation failures emit `ValidationErrorEvent` analytics with coded reasons.
 - **Reference numbers**: `CYPMD_{CheckingWindowType}_{7-char-uppercase-GUID-fragment}` from `JourneyValidationService.GenerateReference`.