diff --git a/AGENTS.md b/AGENTS.md index 8fc7d20..576d8ba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,6 +21,7 @@ machines whose "disk" is 20 GiB of tmpfs, next to a long tail of small repositor | Doc | Who / when to read it | |---|---| +| `docs/DEV_SETUP.md` | Local tools, build/test entry points, supported platform limits and local store ports. | | `GOAL.md` | Everyone, first. What walgit is for, the acceptance table, what we do not optimise for. | | `AGENTS.md` (this) | Everyone. Constraints §1, WAL §2, principles §3, decisions §4, working rules §5. | | `README.md` | The introduction: why (the Cursor lineage), what it does, how it works briefly, running it, invariants. | diff --git a/README.md b/README.md index 1206f82..9afb3e8 100644 --- a/README.md +++ b/README.md @@ -117,6 +117,8 @@ with its reasoning, the invariants, and the cost model (round trips to the bucke ## Running it +See [developer setup](docs/DEV_SETUP.md) for tool versions, local store ports and validation commands. + ```sh # build (needs rust per rust-toolchain.toml, protoc, node 24 + pnpm for the web UI) just web-build && cargo build --release -p walgit-cli diff --git a/compose.yaml b/compose.yaml index 26552e7..84af161 100644 --- a/compose.yaml +++ b/compose.yaml @@ -12,8 +12,8 @@ services: rustfs: image: rustfs/rustfs:latest ports: - - "9000:9000" - - "9001:9001" + - "127.0.0.1:${WALGIT_DEV_STORE_PORT:-9000}:9000" + - "127.0.0.1:${WALGIT_DEV_CONSOLE_PORT:-9001}:9001" environment: RUSTFS_ACCESS_KEY: walgit-dev RUSTFS_SECRET_KEY: walgit-dev-secret diff --git a/docs/DEV_SETUP.md b/docs/DEV_SETUP.md new file mode 100644 index 0000000..74d09a3 --- /dev/null +++ b/docs/DEV_SETUP.md @@ -0,0 +1,78 @@ +# Developer setup + +Start with `GOAL.md` and `AGENTS.md`. Linux is the CI reference platform; macOS +has local recipe support. Windows is not currently qualified (issue #16). + +## Tools and reproducibility + +`nix develop` uses the checked-in `flake.lock` and `rust-toolchain.toml` for the +devshell. Keep lockfiles unchanged when reproducing CI. Network downloads and +container images tagged `latest` mean the entire local workflow is not hermetic. + +| Tool | Repository requirement | +|---|---| +| Rust | `rust-toolchain.toml`, currently 1.97.1, with rustfmt and Clippy | +| Git / Git LFS | Server Git ≥2.47; install git-lfs for LFS integration tests | +| Node / pnpm | Node 24 and pnpm 10, matching CI; use `web/pnpm-lock.yaml` | +| Protobuf | `protoc` plus well-known imports (Debian: protobuf-compiler and libprotobuf-dev) | +| Native compiler | C/C++ compiler, pkg-config, CMake, Perl and platform development libraries | +| Shell tools | Bash, just, curl, ripgrep, GNU sort/comm/timeout; lsof or ss for port checks | +| TLC | Java 11+, checksum-pinned jar fetched by scripts/ensure-tla-tools.sh | +| Local S3 rig | Podman and its compose provider; macOS also needs a running podman machine | + +Without Nix, install the same tools through your platform package manager and +rustup. On macOS, GNU coreutils may need its `gnubin` directory on PATH for the +TLC runner; `gtimeout` alone only covers recipes that explicitly select it. +Homebrew protobuf supplies the imports; `podman machine init` is a one-time step, +and `podman machine start` starts its VM. The recipes give a useful error if that +VM is not running. No test should use or alter your personal Git configuration. + +## Build and run + +```sh +nix develop +just web-build +cargo build --locked --release -p walgit-cli +just dev-local +``` + +The web build produces the SPA and SDK embedded by the server. Open +`https://walgit.localhost:8080/`; standalone uses a self-signed development CA and +loopback-only unauthenticated access. Follow the served setup instructions for +trusting that CA. `PORT` changes the application listener, not the store listener. + +`just dev-store` starts the local RustFS service and ensures the test bucket +exists. It accepts a healthy existing store on repeat startup. A listener on the +selected ports that does not answer the health check produces an actionable error +before the container bootstrap. Compose owns the final bind and detects races. + +To avoid a port collision, change both the host binding and application endpoint +through these shared variables: + +```sh +export WALGIT_DEV_STORE_PORT=19100 +export WALGIT_DEV_CONSOLE_PORT=19101 +just dev-local +``` + +Compose binds those ports on loopback. `dev-local` derives its S3 endpoint from +the store port; changing only `WALGIT__STORE__S3__ENDPOINT` does not remap containers. +An explicit endpoint override instead uses a store you have already started. +The checked-in standalone TOML still defaults to port 9000 when invoked directly. +Stop the compose services with `just dev-store-stop`; it does not delete volumes. + +## Validate + +```sh +just ci # warnings, Clippy, fast tests, e2e, serial simulations, standalone smoke +just spec # bounded models, exact negative controls, fast reference +just spec-full # larger reference arms too; explicit JVM budgets in docs/spec/README.md +just test-slow # opt-in ignored stress/benchmark tests +just test-s3 # local store contract; start the local S3 rig first +``` + +For a nondefault local port, set `WALGIT_TEST_S3_ENDPOINT` to that endpoint when +running `just test-s3`. Credentials come from the test environment; the local +compose fixture uses synthetic development credentials. Full cloud/edge/client +qualification is separate from these local checks. Use `--locked` for ad-hoc Cargo +builds and the documented bounded test tiers rather than an unbounded workspace run. diff --git a/flake.nix b/flake.nix index c3a1e7e..cef594c 100644 --- a/flake.nix +++ b/flake.nix @@ -156,6 +156,9 @@ just git git-lfs + curl + lsof + jdk_headless coreutils jq ripgrep diff --git a/justfile b/justfile index f0449ab..f8253cf 100644 --- a/justfile +++ b/justfile @@ -26,8 +26,10 @@ dev-local config="walgit.standalone.toml": #!/usr/bin/env bash set -euo pipefail export AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID:-walgit-dev}" AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-walgit-dev-secret}" - if ! curl -sf http://127.0.0.1:9000/minio/health/live >/dev/null 2>&1; then - echo "rustfs not running on :9000 — starting it (just dev-store)" + local_endpoint="http://127.0.0.1:${WALGIT_DEV_STORE_PORT:-9000}" + export WALGIT__STORE__S3__ENDPOINT="${WALGIT__STORE__S3__ENDPOINT:-$local_endpoint}" + if [ "$WALGIT__STORE__S3__ENDPOINT" = "$local_endpoint" ] && ! curl --max-time 2 -sf "$local_endpoint/minio/health/live" >/dev/null 2>&1; then + echo "local store not healthy at $local_endpoint — starting it (just dev-store)" just dev-store fi # crates/walgit-server/build.rs:19 writes a placeholder index.html on any cargo build, so only @@ -38,7 +40,7 @@ dev-local config="walgit.standalone.toml": fi cargo build --release --bin walgit-server port="${PORT:-8080}" - echo "→ https://walgit.localhost:${port}/ (PORT=${port}, config {{config}}, store rustfs :9000, cache /tmp/walgit)" + echo "→ https://walgit.localhost:${port}/ (PORT=${port}, config {{config}}, store $WALGIT__STORE__S3__ENDPOINT, cache /tmp/walgit)" exec ./target/release/walgit-server --config {{config}} # Start rustfs (S3-compatible) for local dev via podman compose (rootless, no daemon group needed; @@ -50,6 +52,7 @@ dev-local config="walgit.standalone.toml": dev-store: #!/usr/bin/env bash set -euo pipefail + scripts/dev-store-preflight.sh # The rootless socket bootstrap is Linux-only: XDG_RUNTIME_DIR and /run/user do not exist on # macOS or the BSDs, and setsid is util-linux. There the socket lives in the podman machine VM. if [ "$(uname -s)" = Linux ]; then @@ -72,7 +75,7 @@ dev-store: podman compose up -d rustfs echo "Waiting for rustfs to be healthy..." podman compose run --rm create-bucket - echo "rustfs is running on http://127.0.0.1:9000 (console :9001)" + echo "rustfs is running on http://127.0.0.1:${WALGIT_DEV_STORE_PORT:-9000} (console :${WALGIT_DEV_CONSOLE_PORT:-9001})" echo "Credentials: walgit-dev / walgit-dev-secret" echo "Bucket: walgit-test" @@ -92,6 +95,7 @@ dev-store-stop: # Never run `cargo test --workspace --no-fail-fast` interactively: a single # hung test blocks for the whole timeout. Use `just e2e` / `just ci` below. test: + scripts/test-dev-store-preflight.sh {{t5}} cargo test --workspace --lib --bins {{t10}} cargo test -p walgit-store -p walgit-git -p walgit-wal --tests {{t10}} cargo test -p walgit-server --test web_api --test web_ui --test api_v1 --test static_http --test packfile_uri --test forward --test maintain --test routing_prefix --test lfs_upstream --test drain --test events --test follow --test policy @@ -159,10 +163,10 @@ store-test: test-s3: store-test-s3 store-test-s3: - WALGIT_TEST_S3_ENDPOINT=http://127.0.0.1:9000 \ - WALGIT_TEST_BUCKET=walgit-test \ - AWS_ACCESS_KEY_ID=walgit-dev \ - AWS_SECRET_ACCESS_KEY=walgit-dev-secret \ + WALGIT_TEST_S3_ENDPOINT="${WALGIT_TEST_S3_ENDPOINT:-http://127.0.0.1:${WALGIT_DEV_STORE_PORT:-9000}}" \ + WALGIT_TEST_BUCKET="${WALGIT_TEST_BUCKET:-walgit-test}" \ + AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID:-walgit-dev}" \ + AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-walgit-dev-secret}" \ cargo test -p walgit-store --test contract -- --nocapture # Run all walgit-store tests (memory + S3 if env set). diff --git a/scripts/dev-store-preflight.sh b/scripts/dev-store-preflight.sh new file mode 100755 index 0000000..2608ee7 --- /dev/null +++ b/scripts/dev-store-preflight.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash +# A healthy existing store is a valid repeat start, not a port conflict. +set -euo pipefail +store_port="${WALGIT_DEV_STORE_PORT:-9000}" +console_port="${WALGIT_DEV_CONSOLE_PORT:-9001}" +for port in "$store_port" "$console_port"; do + if [[ ! "$port" =~ ^[1-9][0-9]{0,4}$ ]] || ((10#$port > 65535)); then + echo "Invalid local store port: $port (expected 1..65535)" >&2 + exit 1 + fi +done +if [[ "$store_port" == "$console_port" ]]; then + echo 'Store and console ports must differ.' >&2 + exit 1 +fi +if curl --max-time 2 -sf "http://127.0.0.1:$store_port/minio/health/live" >/dev/null 2>&1; then + exit 0 +fi +for port in "$store_port" "$console_port"; do + busy=false + if command -v ss >/dev/null 2>&1; then + [[ -z "$(ss -tlnH "sport = :$port")" ]] || busy=true + elif command -v lsof >/dev/null 2>&1; then + if lsof -nP -iTCP:"$port" -sTCP:LISTEN -t >/dev/null 2>&1; then busy=true; fi + else + echo 'Install ss (iproute2) or lsof to check local store ports.' >&2 + exit 1 + fi + if $busy; then + echo "Port $port is occupied but the local store is not healthy." >&2 + echo 'Choose unused WALGIT_DEV_STORE_PORT and WALGIT_DEV_CONSOLE_PORT values; compose and dev-local use them together.' >&2 + exit 1 + fi +done diff --git a/scripts/test-dev-store-preflight.sh b/scripts/test-dev-store-preflight.sh new file mode 100755 index 0000000..3573b92 --- /dev/null +++ b/scripts/test-dev-store-preflight.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +set -euo pipefail +root="$(cd "$(dirname "$0")/.." && pwd)" +scratch="$(mktemp -d)" +trap 'rm -rf "$scratch"' EXIT +mkdir "$scratch/bin" +cat > "$scratch/bin/curl" <<'MOCK' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$CALLS" +exit "$HEALTH_STATUS" +MOCK +cat > "$scratch/bin/ss" <<'MOCK' +#!/usr/bin/env bash +printf '%s\n' "$*" >> "$CALLS" +if [[ "$BUSY" == true ]]; then echo 'LISTEN 0 128 127.0.0.1:19100'; fi +MOCK +chmod +x "$scratch/bin/"* +export PATH="$scratch/bin:$PATH" CALLS="$scratch/calls" +export WALGIT_DEV_STORE_PORT=19100 WALGIT_DEV_CONSOLE_PORT=19101 +export HEALTH_STATUS=0 BUSY=true +bash "$root/scripts/dev-store-preflight.sh" +[[ "$(wc -l < "$CALLS")" -eq 1 ]] # Healthy repeat start never probes busy ports. +export HEALTH_STATUS=1 BUSY=false +bash "$root/scripts/dev-store-preflight.sh" +export BUSY=true +if bash "$root/scripts/dev-store-preflight.sh" > "$scratch/error" 2>&1; then exit 1; fi +[[ "$(< "$scratch/error")" == *'Port 19100 is occupied'* ]] +for invalid in 0 65536 019100 nope; do + if WALGIT_DEV_STORE_PORT="$invalid" bash "$root/scripts/dev-store-preflight.sh" >/dev/null 2>&1; then exit 1; fi +done +if WALGIT_DEV_CONSOLE_PORT=19100 bash "$root/scripts/dev-store-preflight.sh" >/dev/null 2>&1; then exit 1; fi +[[ "$(< "$CALLS")" == *'http://127.0.0.1:19100/minio/health/live'* ]] +echo 'PASS healthy repeat start, free/occupied alternate ports and invalid settings'