Add automatic OCI tag pagination to the library and CLI

README.md

@@ -71,6 +71,15 @@ List tags for a repository:
 regshape tag list -i docker.io/library/alpine
 ```
 
+Retrieve every page of tags:
+
+```bash
+regshape tag list -i docker.io/library/alpine --all
+```
+
+Use `--n` to set the all-pages request size and `--last` to start after a tag.
+See the [tag command reference](specs/cli/tag.md) for pagination and output options.
+
 ## Documentation
 
 ### Guides

specs/cli/tag.md

@@ -13,10 +13,8 @@ that map to the OCI Distribution Spec endpoints:
 Tag creation and movement are implicit side-effects of `manifest put` and are
 not exposed here.
 
-> **Note:** Until `libs/transport/` is implemented, all commands issue HTTP
-> requests directly via the `requests` library using the existing
-> `libs/auth/` helpers. Migration to `RegistryClient` is tracked as a
-> follow-up task.
+All requests use the shared `RegistryClient` transport and its authentication
+and telemetry support.
 
 ---
 
@@ -50,15 +48,16 @@ image reference syntax with the registry always embedded:
 
 ### `tag list`
 
-List all tags for a repository, with optional pagination.
+List one page of tags, or opt into automatic traversal of all remaining pages.
 
 #### Options
 
 | Option | Short | Type | Default | Description |
 |---|---|---|---|---|
 | `--image-ref` | `-i` | string | required | Repository reference with embedded registry (e.g., `registry/repo`, `registry/repo:tag`) |
-| `--n` | | integer | none | Maximum number of tags to return per page |
+| `--n` | | integer | none; 100 with `--all` | Tags requested per page; must be positive with `--all` |
 | `--last` | | string | none | Return tags lexicographically after this value (pagination cursor) |
+| `--all` | | flag | false | Retrieve and aggregate every remaining page after the optional cursor |
 | `--json` | | flag | false | Output the raw `TagList` JSON object instead of one tag per line |
 | `--output` | `-o` | path | stdout | Write output to this file instead of stdout |
 | `--time-methods` | | flag | false | Print execution time for individual method calls |
@@ -71,29 +70,68 @@ List all tags for a repository, with optional pagination.
    ignored if present).
 2. Resolve credentials from the Docker credential store for the extracted
    registry.
-3. Issue `GET /v2/<name>/tags/list`, appending `?n=<n>` and/or
-   `?last=<last>` when the respective options are provided.
+3. Without `--all`, issue one `GET /v2/<name>/tags/list`, appending `?n=<n>`
+   and/or `?last=<last>` when provided. Existing single-page behavior, including
+   `--n 0`, is unchanged.
+   With `--all`, call `list_tags_all()` with a positive page size (default 100)
+   and the optional initial tag cursor. Follow next links when present;
+   otherwise continue full pages using the last returned tag value.
 4. Parse the response body into a `TagList` model.
 5. If `--json` is set, print the JSON object. Otherwise, print one tag per
    line (suitable for scripting with `xargs` or other tools).
-6. If `--output` is given, write to that file instead of stdout.
+6. If `--output` is given, write to that file instead of stdout. All-pages
+   results are emitted only after the entire traversal succeeds; a later-page
+   failure does not print a successful partial list or overwrite an output file.
+   Repeated tags, including partial overlap between pages, fail with exit code 1
+   rather than producing duplicates or silently deduplicating the response.
+   A page that includes an excluded `--last` or server-provided cursor tag also
+   fails instead of returning that tag.
+   If concurrent repository changes cause overlap, retry once the listing is
+   stable.
+
+Pagination follows [OCI Distribution Specification v1.1.1, Listing
+Tags](https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#listing-tags).
+`Link` is optional; when present, its `next` target takes precedence even for
+short/empty pages. Without a next link, a full page advances using an exclusive
+tag-value cursor. The effective page size can change in a server-provided link.
+See the [library operation](../operations/tags.md#list_tags_all) for safety
+checks and stopping rules. The 100-tag default is a client policy, not an OCI
+requirement, and `--n` does not limit the total size of an all-pages result.
+Registry and continuation URL authorities must use ASCII spelling; use the
+punycode hostname for an international domain.
+
+As a client safety policy, `--all` fails on the **tenth consecutive empty page
+with a next link**, even when continuation links change. Receiving tags resets
+the counter; an empty page without a next link completes normally. This limit
+is not an OCI requirement or a cap on total pages. Failure exits with code 1,
+prints no partial list, and leaves output files untouched.
+All-pages results and `--last`/linked tag cursors must also satisfy the OCI
+tag-name grammar (at most 128 ASCII characters); malformed tags fail before
+output or cursor encoding. See the library reference above for the exact grammar.
 
 #### Exit Codes
 
 | Code | Meaning |
 |---|---|
 | 0 | Success |
-| 1 | HTTP error (repo not found, auth failure, connection error) |
+| 1 | Registry/transport error, invalid response, or unsafe/non-progressing pagination |
+| 2 | Usage error, including non-positive `--n` with `--all` |
 
 #### Examples
 
 ```bash
 # Log in first
 regshape auth login -r acr.io -u alice
 
-# List all tags for a repository
+# List one page using registry defaults
 regshape tag list -i acr.io/myrepo/myimage
 
+# Retrieve every tag (100 requested per page by default)
+regshape tag list -i acr.io/myrepo/myimage --all
+
+# Retrieve all remaining tags after v1.5, requesting 10 per page
+regshape tag list -i acr.io/myrepo/myimage --all --n 10 --last v1.5
+
 # List with pagination (first 10 tags)
 regshape tag list -i acr.io/myrepo/myimage --n 10
 
@@ -105,6 +143,9 @@ regshape tag list -i acr.io/myrepo/myimage --json
 
 # Save tag list to a file
 regshape tag list -i acr.io/myrepo/myimage -o tags.txt
+
+# Save the complete aggregated response
+regshape tag list -i acr.io/myrepo/myimage --all --json -o tags.json
 ```
 
 #### Output Format
@@ -134,6 +175,10 @@ v2.0
 | Repository not found | `Error [acr.io/myrepo/myimage]: Registry error: repository not found` |
 | Auth failure | `Error [acr.io/myrepo/myimage]: Authentication failed for acr.io` |
 | Connection error | `Error [acr.io/myrepo/myimage]: <transport error>` |
+| Invalid all-pages size | Usage error: `--n` must be positive with `--all` |
+| Unsafe next link | `Error [<ref>]: Unsafe or invalid tag pagination URL: <detail>` |
+| Repeated page/cursor | `Error [<ref>]: Tag pagination repeated a page/cursor for <registry>/<repo>` |
+| Empty-page limit | `Error [<ref>]: Tag pagination returned 10 consecutive empty linked pages` |
 
 ---
 

specs/operations/tags.md

@@ -14,6 +14,7 @@ It never calls `requests` directly — all HTTP traffic goes through a
 | Operation | Method | Endpoint |
 |---|---|---|
 | List tags | `GET` | `/v2/{repo}/tags/list[?n=<n>&last=<last>]` |
+| List all remaining tags | `GET` | Same endpoint, following links or tag cursors |
 | Delete tag | `DELETE` | `/v2/{repo}/manifests/{tag}` |
 
 ---
@@ -60,12 +61,129 @@ Fetches the tag list for a repository (single page).
 3. Call `client.get(path, params=params)`.
 4. Pass response to `_raise_for_list_error`.
 5. Deserialize and return `TagList.from_json(response.text)`.
-   Re-raises `TagError` directly; wraps all other exceptions in `TagError`.
+   Re-raises `TagError` directly; wraps field validation errors in `TagError`.
 
 **Decorator:** `@track_time`
 
 ---
 
+### `list_tags_all`
+
+```python
+@track_scenario("tag list all")
+def list_tags_all(
+    client: RegistryClient,
+    repo: str,
+    page_size: Optional[int] = None,
+    last: Optional[str] = None,
+) -> TagList:
+```
+
+Returns one `TagList` containing all remaining pages, in server order and with
+tag spelling unchanged. `page_size=None` uses 100 per request. An explicit size
+must be a positive integer; invalid sizes fail before network access. `last`
+is an optional exclusive starting tag value, not an offset.
+
+The protocol is defined by [OCI Distribution Specification v1.1.1, Listing
+Tags](https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#listing-tags).
+The helper and its default size are RegShape conveniences, not OCI requirements.
+
+**Traversal:**
+
+1. Request the first page with `n` and, when provided, `last`.
+2. Read pagination metadata from that page's returned response, not a separate
+   mutable client response slot.
+3. Prefer its `Link` with relation `next`, including on a short or empty page.
+   Parse [RFC 5988](https://www.rfc-editor.org/rfc/rfc5988.html) link parameters
+   without splitting commas inside URI references or quoted values. Resolve
+   relative targets against the actual response URL and preserve linked query
+   values and path. Follow [RFC 3986 reference resolution](https://www.rfc-editor.org/rfc/rfc3986.html#section-5.2)
+   for dot segments and query inheritance without collapsing empty path
+   segments: `pages/token//part` and `pages/token/part` are distinct paths.
+4. If there is no next link and the current request has a positive `n`, continue
+   a full-sized page using its last tag as the next exclusive cursor. Preserve
+   other query fields byte-for-byte, including opaque percent-encoded values
+   and bare flags; replace only `last`. Stop on a short/empty unlinked page;
+   exact multiples may need a final empty request.
+5. Use the effective `n` of each request, not the initial size. A linked request
+   may change `n` or omit it: with no `n` and no next link, its response is the
+   remaining unbounded list.
+6. Aggregate pages only on success. Propagate later-page failures without
+   returning a successful partial result.
+
+`Link` is optional in the OCI specification. Consequently, absence of a link
+alone does not terminate a full-sized cursor-paginated page. Conversely, page
+length alone cannot terminate a page with a next link. OCI permits `n=0` for
+single-page listing (empty result, no link); `list_tags()` retains that behavior,
+while all-pages mode rejects it as a client input policy.
+
+**Client safety policies:**
+
+- Continuations must remain on the configured scheme, hostname, and effective
+  port. Reject cross-origin links, scheme changes, embedded user credentials,
+  malformed ports, raw whitespace/control characters, and backslashes before
+  dispatch. Explicitly configured HTTP registries remain supported.
+  Authorities must be ASCII: use punycode for international domain names.
+  Reject Unicode authorities rather than allowing differences between
+  validator and HTTP-client domain-name mappings to bypass origin checks.
+- Disable automatic redirects on pagination requests, including authentication
+  probes and retries in both transport modes. Validate each `Location` before
+  following a 301/302/303/307/308 redirect so a foreign authentication challenge
+  cannot receive stored credentials. Follow at most 30 redirects per page;
+  reject missing locations and redirect cycles. Ordinary transport requests
+  retain their existing automatic-redirect default; cached responses are
+  separated by redirect policy so an auto-follow result cannot bypass an
+  explicit opt-out. Track redirect hops separately from completed tag pages,
+  allowing a same-origin redirect to canonicalize query ordering without being
+  mistaken for a repeated page.
+- If challenge discovery at `/v2/` redirects, both transport modes validate
+  and follow same-origin probe redirects explicitly before accepting a
+  challenge. Probe chains have the same cycle checks and 30-hop bound; unsafe
+  targets fail before any credential exchange.
+- Sanitize URL-parser failures and suppress their raw exception context:
+  malformed URI authorities may contain credentials even when parsing fails
+  before the userinfo check. Requests may prepare `Response.next` even when
+  automatic redirects are disabled, so the shared request wrapper converts
+  pre-return parsing failures into sanitized transport errors before logging.
+- Remove fragments for request/cycle identity. Detect repeated URLs/cursors and
+  nonempty pages containing no new tags; raise `TagError` rather than looping.
+  Empty pages with distinct valid continuation links may still be followed.
+  Normalize only known pagination parameters in page identity; opaque values
+  such as `%FF` and `%FE` must remain distinct.
+- Fail with `TagError` on the **tenth consecutive empty page with a next link**,
+  before requesting its continuation, even if opaque links keep changing.
+  Receiving tags resets the counter; an empty page without a next link completes
+  normally, including after nine empty linked pages. Redirect responses neither
+  increment nor reset this counter. The limit is a RegShape safety policy, not
+  an OCI requirement or a cap on total pages; failure returns no partial result.
+- Reject malformed or ambiguous pagination headers/parameters, conflicting next
+  links, and anchored next links whose alternate context is not supported.
+- Reject invalid tag values and pages belonging to another repository.
+  Returned tags and initial/linked `last` cursors must match the
+  [OCI tag grammar](https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#pulling-manifests):
+  `[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}` (at most 128 ASCII characters).
+  Validate before rendering or cursor encoding, so malformed strings produce
+  `TagError`, not extra output lines or an unhandled encoding failure.
+  This does not reinterpret opaque continuation fields or change single-page
+  `list_tags()` behavior.
+- Reject repeated tags within a page or across partially overlapping pages.
+  This is an all-pages consistency policy, not silent deduplication.
+- Reject any returned tag equal to an initial or linked exclusive cursor,
+  even on a short page. Retain requested cursor boundaries across redirects.
+
+There is no snapshot guarantee if tags change while listing. Aggregation does
+not re-sort, lower-case, or silently deduplicate returned tags. Concurrent
+changes that cause overlapping pages fail explicitly; retry against a stable
+repository rather than treating a duplicated list as a complete result.
+
+**Returns:** `TagList`
+
+**Errors:** `TagError` for invalid input, unsafe/non-progressing pagination,
+malformed responses, or registry errors; `AuthError` for authentication failures;
+transport exceptions propagate unchanged.
+
+---
+
 ### `delete_tag`
 
 ```python
@@ -100,6 +218,16 @@ is routed through the manifests endpoint.
 
 ## Private Helpers
 
+The single-page and all-pages operations share `_parse_tag_page()` for
+response parsing/error handling. `_next_tag_link()`, `_tag_page_parameters()`,
+`_tag_page_key()`, and `_safe_tag_page_url()` implement the tag-specific
+continuation rules above. The shared
+[`resolve_registry_url()`](../../src/regshape/libs/transport/urls.py) utility
+preserves URI semantics and enforces the same origin policy for pagination
+and authentication probes.
+Requests are dispatched as registry-relative paths so both middleware and legacy
+transport modes work.
+
 ### `_raise_for_list_error`
 
 | HTTP status | Exception | Message |
@@ -128,9 +256,9 @@ the first 200 characters of `response.text`.
 | Decorator | Applied to |
 |---|---|
 | `@track_time` | `list_tags`, `delete_tag` |
+| `@track_scenario` | `list_tags_all` |
 
-No `@track_scenario` is needed — neither operation is multi-step at the HTTP
-level.
+All page requests still use the shared transport's request tracing and metrics.
 
 ---
 
@@ -144,6 +272,7 @@ level.
 | `400` / `405` on delete | `TagError` "not supported" | `_raise_for_delete_error` |
 | Malformed JSON body | `TagError` | `TagList.from_json` (re-raised) |
 | Transport / connection failure | `requests.exceptions.RequestException` | propagated as-is |
+| Non-positive all-pages size, unsafe link, cycle, or invalid page | `TagError` | `list_tags_all` / pagination helpers |
 
 ---
 
@@ -156,6 +285,7 @@ level.
 - `regshape.libs.refs` — `format_ref`
 - `regshape.libs.transport` — `RegistryClient`
 - `regshape.libs.decorators.timing` — `track_time`
+- `regshape.libs.decorators.scenario` — `track_scenario`
 
 **External:**
 - `requests` — `requests.Response` type annotation only