Feat/cluster live demo - #1
Open
asmith987 wants to merge 17 commits into
Open
Conversation
asmith987
force-pushed
the
feat/cluster-live-demo
branch
from
August 31, 2026 14:13
2a1d741 to
3c1088f
Compare
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…ration Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: matt.garmon <mjg2790@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…P e2e
Adds `cluster-consumer`, a minimal Profile-3 gear that resolves
`ClusterCacheV1` from the ClientHub and does a cache put+get round-trip,
proving a cross-pod gRPC coordination call. It links no cluster code (no
`deps = [cluster]`): the framework's proxy-wiring phase registers a remote
`dyn ClusterClient` from cluster-sdk's `ConsumerRegistration` before the
gear's routes run. `cluster-sdk` is pulled with `grpc-client` so that
registration and `RemoteClusterClient` are actually compiled in.
Surface: an anonymous, edge-exposed `GET /cluster-consumer/v1/ping`
(cluster-free liveness, used to observe route-sync) and `POST
/cluster-consumer/v1/roundtrip` (the cache round-trip; returns 503 with the
underlying cluster error when the coordination plane is unreachable).
Deploy/test wiring:
- config/oop-cluster.yaml (standalone `demo` profile; own grpc-hub :50061,
probes :9092) and config/oop-cluster-consumer.yaml (:9093).
- testing/e2e/suites/oop: boots cluster-oop + cluster-consumer-oop, sets
POD_NAMESPACE (what the consumer derives its cluster endpoint from), and
adds two seam tests — both driven through the edge like the rest.
- Makefile `e2e-oop` builds both new binaries.
Loopback is seam-only by design: a consumer's cluster endpoint is derived
from Kubernetes DNS convention (cluster.{POD_NAMESPACE}.svc.cluster.local
:50051) and is non-configurable (invariant I9), so on 127.0.0.1 the
round-trip returns a typed Provider{ConnectionLost}, surfaced as 503 — the
seam proof. The successful round-trip is the Kubernetes demo's job.
make e2e-oop: 7 passed.
Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Deploys the cluster coordination gear and its consumer as their own pods on
Kubernetes, verified end-to-end on minikube: a cache round-trip travels edge ->
consumer pod -> gRPC (endpoint derived by DNS convention, authenticated by
TokenReview) -> one of 3 cluster replicas -> shared Postgres, and back.
POST /cluster-consumer/v1/roundtrip {"key":"seat/12","value":"held"}
=> 200 {"key":"seat/12","value":"held","version":1,"served_by":"cluster-consumer-oop (pid 1)"}
(row persisted to shared-postgres cluster_cache as `cluster-consumer/seat/12`)
Charts (both based on toolkit-common; no shared-chart changes needed — it already
supports a gRPC Service and injects POD_NAMESPACE from the downward API):
- deploy/helm/cluster: publishes the coordination gRPC plane on :50051 (the port
consumers derive), `fullnameOverride: cluster` so the derived Service DNS
resolves, TokenReview RBAC (system:auth-delegator), and the `demo` profile on
the postgres backend against the shared Postgres.
- deploy/helm/cluster-consumer: like hello; no RBAC (it only attaches tokens).
- toolkit-platform: umbrella deps + values-dev entries — cluster at 3 replicas on
postgres (shared state keeps reads/writes consistent across replicas), and
`cluster` added to postgres.databases.
cluster crate: add a `k8s-auth` feature. It MUST propagate to grpc-hub
(`grpc_hub/k8s-auth`) — the hub's inbound TokenReview validator is gated behind
its own feature (mirroring platform-host's `k8s`); without it the cluster pod
fails init with "grpc-hub internal_auth provider=kube requires the k8s-auth
feature". This was the one issue the live cluster surfaced.
oop-smoke.sh: GEARS specs gain a per-gear features field (cluster builds with
just `k8s-auth`; its OoP bin is unconditional). cluster + cluster-consumer added
to the image/rollout/smoke sets; the smoke asserts the real 200 round-trip.
README: build/load commands, a cluster-coordination section, and a macOS +
docker-driver note (minikube ip is not host-routable; use `minikube tunnel` or
`kubectl port-forward svc/platform-host 8087:8087`).
Verified on minikube v1.38 (docker driver) / helm v4 / kubectl v1.37: all 10
pods healthy, TokenReview enforcement active, postgres migrations applied.
Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
The smoke stage curled the edge Ingress at `minikube ip`, which is not routable
from the host on the docker driver (macOS/Windows) without `minikube tunnel` — so
`oop-smoke.sh --all` failed on `/healthz` there despite a healthy stack.
smoke() now probes the Ingress for up to 30s and, if the node IP is unreachable,
falls back to `kubectl port-forward svc/platform-host` (works on every driver;
torn down via an EXIT trap). The fallback uses an inert `--resolve`
(127.0.0.1->127.0.0.1) rather than an empty array, since macOS bash 3.2 under
`set -u` errors on expanding an empty "${arr[@]}".
Verified on minikube docker driver: 15 passed, 0 failed (incl. the cluster
round-trip, TokenReview enforcement, PEP-over-REST, OoP->OoP). README updated.
Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…round-trip
Expands the demo's coordination cycle from cache-only to all three cluster
primitives, resolved from the ClientHub and driven against the cluster pod over
gRPC:
1. DistributedLockV1 — acquire + release (no remote I/O while held, per
ADR-002: the cache write happens after release, the lock serializes it).
2. ClusterCacheV1 — put + get.
3. LeaderElectionV1 — join, settle leadership, resign.
Response gains lock_name / lock_released / is_leader / leader_status. Verified on
minikube (17/17 smoke): the consumer acquires+releases the lock, round-trips the
cache, and wins the election ('leader_status':'Leader'); loopback still 7/7
(now 503 'cluster coordination call failed').
Also surfaced (noted for the cluster PR, worked around here):
- Lock/leader coordination NAMES are validated by validate_cluster_name
([a-zA-Z0-9_-], no '/'), so DistributedLockV1::scoped()/LeaderElectionV1::
scoped() (which compose 'prefix/name') produce names the remote path rejects
with InvalidName. Cache .scoped() is fine (keys allow '/'). The demo uses a
flat, slash-free coordination name and does not scope lock/leader.
- That InvalidName error loses its 'reason' across the gRPC codec (renders
'<unknown>' instead of the rule).
CoordinationService replaces CacheRoundTripService.
Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Declares lock: { provider: postgres } on the demo profile so it uses
PostgresLockProvider (pg_advisory_lock + cluster_lock table) instead of the SDK
CAS-over-cache default. leader_election stays on the CAS default (no native
provider exists).
Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Now that the server validates scoped coordination names (fix B), switch the
lock and leader facades back to .scoped("cluster-consumer") so all three
primitives share one namespace. The lock/election wire name composes as
`cluster-consumer/reservation` (leaf `reservation` + the SCOPE prefix), the
exact scoped name that returned 503 InvalidName before the fix.
Verified live (17/17): response now carries
"lock_name":"cluster-consumer/reservation". Loopback still 7/7 (unchanged 503
path — the failure is at connect, before any name reaches a server).
Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…tion service
Replaces the single synthetic `POST /roundtrip` (which fired lock+cache+leader
in one artificial cycle) with a small but realistic seat-reservation service that
uses each cluster primitive the way ADR-002 intends.
Every seat's lifecycle lives in one cluster-cache key and moves
`available -> held -> booked -> available`:
- Cache + CAS is the correctness gate: every transition is a versioned
`compare_and_swap` with a bounded retry loop, so two pods racing for the same
seat cannot both win (the loser sees `CasConflict` and retries).
- The per-seat distributed lock is an advisory contention damper only, held with
no remote I/O inside (released before the CAS) per ADR-002.
- Leader election drives a singleton background sweeper (`run_while_leader`) that
seeds the roster and reclaims expired holds; followers idle. On loopback it
campaigns-and-backs-off rather than crashing.
Routes (were: one roundtrip + ping):
GET /ping liveness (cluster-free)
GET /status this pod's leader/sweeper role
POST /reservations hold a seat { seat, holder }
POST /reservations/{seat}/confirm book a held seat { holder }
DELETE /reservations/{seat} release { holder }
GET /reservations/{seat} read a seat
GET /inventory venue-wide availability
Structure: new `model.rs` (seat records + roster, unit-tested), `error.rs`
(idiomatic `#[resource_error]` domain errors mapped to canonical HTTP statuses),
`sweeper.rs` (leader loop + shared lock-free `NodeState`), and rewritten
`domain.rs`/`rest.rs`/`gear.rs`. The gear gains the `stateful` capability and a
`RunnableCapability` lifecycle (spawn the sweeper in `start`, cancel/join in
`stop`), capturing the `ClientHub` at `init` since `start` gets no `GearCtx`.
Also fixes a latent packaging bug: the crate used tokio unconditionally but
declared it optional, so `cargo check` (no features) was broken. `tokio` and
`tokio-util` are now non-optional; both build modes are clean.
Demo tooling updated for the new API: the `oop-smoke.sh` e2e drives
reserve -> confirm -> inventory -> release -> status, plus
`config/oop-cluster-consumer.yaml` and `deploy/README.md`.
Verified: fmt clean; clippy clean (workspace pedantic + restriction lints) with
and without `oop_module`; 6 model unit tests pass; the OoP binary builds.
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…OoP e2e Updates the `make e2e-oop` (loopback, no-k8s) suite for the consumer's new seat-reservation API: - Repoints the gRPC-seam ConnectionLost test from the removed `/roundtrip` to `POST /reservations` (a valid roster seat, so it passes input validation and reaches the cluster call, which 503s on loopback as the seam proof). - Adds a cluster-free `/status` cross-process test (proves the `stateful` sweeper route is wired; role is never `leader` on loopback). - Adds an unknown-seat -> 404 test: `reserve` validates the roster before any cluster call, so the new domain logic is exercised end-to-end through the edge without needing a reachable cluster. Route-sync readiness gate still probes the cluster-free `/ping`; comment reworded off the old round-trip route. Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
github-actions
Bot
force-pushed
the
feat/cluster-live-demo
branch
from
August 31, 2026 14:13
3c1088f to
58663ea
Compare
…esolving per call Each cluster primitive was resolved from the ClientHub on every request. After warmup that does no network I/O (the client caches the profile descriptor and reuses its long-lived channel), but it re-runs the requirement-registry recording, name/scope validation, and Arc allocation on every call -- pure churn on the reservation hot path. Resolve each scoped facade once via tokio::sync::OnceCell::get_or_try_init and return a cheap Arc-handle clone thereafter. Only a successful bind is memoized, so a transient warmup failure retries rather than pinning a broken facade. Binding mode (Profile 1 vs 3) is fixed at client registration, not per resolve, so the cached handle stays mode-oblivious. Signed-off-by: Andre Smith <andre.smith+oss@acronis.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.