Add Scala experiment log with http4s and doobie

.github/workflows/experiment-log.yml

@@ -0,0 +1,21 @@
+name: Experiment log
+on:
+  push:
+    paths: ['applications/experiment-log/**', '.github/workflows/experiment-log.yml']
+  pull_request:
+    paths: ['applications/experiment-log/**', '.github/workflows/experiment-log.yml']
+permissions:
+  contents: read
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: applications/experiment-log
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '21'
+      - run: scripts/sbtw clean scalafmtCheckAll compile test verifyDependencyLock

applications/experiment-log/.env.example

@@ -0,0 +1,10 @@
+# Copy values from your own dedicated Cloud Postgres receipt.
+PGHOST=your-service-hostname
+PGPORT=5432
+PGDATABASE=postgres
+PGUSER=experiment_app
+PGPASSWORD=replace-with-runtime-role-password
+PGSSLROOTCERT=/absolute/path/to/cloud-ca.pem
+PGSSLMODE=verify-full
+PROJECT_A_TOKEN=replace-with-a-distinct-random-URL-safe-token-at-least-32-characters
+PROJECT_B_TOKEN=replace-with-another-distinct-random-URL-safe-token-at-least-32-characters

applications/experiment-log/.gitignore

@@ -0,0 +1,11 @@
+target/
+project/target/
+project/project/
+.local/
+.bsp/
+.metals/
+.venv/
+__pycache__/
+.env
+*.env
+*.pem

applications/experiment-log/.scalafmt.conf

@@ -0,0 +1,3 @@
+version = "3.11.5"
+runner.dialect = scala3
+maxColumn = 100

applications/experiment-log/README.md

@@ -0,0 +1,124 @@
+# Experiment log
+
+A small Scala API records supplied experiment results in ClickHouse Managed Postgres. Two seeded projects have separate bearer tokens. Each immutable run contains a title, a bounded flat JSONB configuration and 1–10 exact decimal measurements. These are synthetic examples, not model training or scientific validation.
+
+## Requirements and verified versions
+
+Use a native Linux environment with OpenJDK 21, curl, PostgreSQL `psql`, and a dedicated Cloud Postgres service. The checked set is Scala 3.9.0, sbt 1.13.0, http4s/Ember 0.23.38, Cats Effect 3.7.1, doobie **1.0.0-RC13** (a maintained published release candidate, not stable 1.0), Circe 0.14.16, HikariCP 7.1.0 and pgJDBC 42.7.13. Exact direct versions and the selected transitive dependency record are committed. The wrapper verifies its pinned sbt launcher SHA-256; `verifyDependencyLock` rejects resolver drift. No snapshots are used.
+
+The native CPU build was checked before allocating the database. Testing used Ubuntu 24.04 ARM64, OpenJDK 21.0.12.1 and PostgreSQL 18.6 on 2 October 2026. See [Cloud Postgres documentation](https://clickhouse.com/docs/products/managed-postgres/) for provisioning and the service CA.
+
+## Dedicated database setup
+
+Run from this application directory. Keep receipt credentials and the downloaded Cloud CA outside the clone, restrict private files to mode 600, and do not commit them. Never use this bootstrap on a shared database: it changes PUBLIC database/schema creation privileges.
+
+Install/configure [clickhousectl](https://github.com/ClickHouse/clickhousectl) with your own Cloud API credentials. Verify current supported size/region and [pricing](https://clickhouse.com/pricing) before creation. This dedicated fixture used AWS us-east-1, `c6gd.large`, PostgreSQL 18 and no HA; it incurs compute/storage charges. Stopping the API does not remove the database.
+
+```bash
+mkdir -p $HOME/experiment-private
+chmod 700 $HOME/experiment-private
+clickhousectl cloud postgres create --org-id YOUR_ORG_ID \
+  --name experiment-log --region us-east-1 --provider aws \
+  --size c6gd.large --pg-version 18 --ha-type none \
+  --tag purpose=experiment-log --json > $HOME/experiment-private/create.json
+chmod 600 $HOME/experiment-private/create.json
+clickhousectl cloud postgres get YOUR_SERVICE_ID --org-id YOUR_ORG_ID --json
+```
+
+Take `YOUR_SERVICE_ID` from the private receipt. Wait until `get` reports running, with its hostname populated, before retrieving the CA:
+
+```bash
+clickhousectl cloud postgres certs get YOUR_SERVICE_ID \
+  --org-id YOUR_ORG_ID --output $HOME/experiment-private/ca.pem
+chmod 600 $HOME/experiment-private/ca.pem
+```
+
+Create four private shell environment files. Each includes the receipt hostname, port, database, `PGSSLROOTCERT` (absolute CA path), and `PGSSLMODE=verify-full`:
+
+* `admin.env`: receipt `PGUSER`/`PGPASSWORD`, plus `PG_MIGRATION_PASSWORD` and `PG_APP_PASSWORD` for two newly generated role passwords.
+* `migration.env`: `PGUSER=experiment_owner` and its password.
+* `runtime.env`: `PGUSER=experiment_app` and its password, plus distinct `PROJECT_A_TOKEN` and `PROJECT_B_TOKEN`.
+* `test.env`: `TEST_OWNER_USER=experiment_owner` and `TEST_OWNER_PASSWORD` for opt-in acceptance only.
+
+Generate each token with `python3 -c 'import secrets; print(secrets.token_urlsafe(32))'`. Do not put tokens in URLs or screenshots. The API uses explicit Authorization headers; there are no cookies, browser sign-in, CORS allowances or UI. Bind locally for this tutorial; use HTTPS and proper token distribution for a real deployment. Tokens identify fixed seeded project UUIDs and are loaded at process startup. They are long-lived bearer credentials; this example does not implement per-user identity, expiry, audit trails or token revocation storage.
+
+In separate subshells, use the correct role for each step:
+
+```bash
+(set -a; source $HOME/experiment-private/admin.env; set +a
+ psql -X -v ON_ERROR_STOP=1 -f scripts/bootstrap.sql)
+(set -a; source $HOME/experiment-private/migration.env; set +a
+ scripts/migrate.sh up
+ psql -X -v ON_ERROR_STOP=1 -f scripts/grants.sql
+ psql -X -v ON_ERROR_STOP=1 -f scripts/seed.sql)
+```
+
+The bootstrap creates `experiment_owner`, `experiment_app` and the schema. Runtime can SELECT and INSERT only the required columns; it cannot update/delete runs or children, set timestamps/transaction IDs, or create tables. The shared runtime database role is trusted across both projects; project isolation is enforced by the API's server-derived identity, not PostgreSQL row-level security.
+
+`migrate.sh up` takes an advisory transaction lock and verifies the recorded file checksum on repeat. Seed is repeat-safe. `migrate.sh down` removes app data and tables; use it only on this dedicated fixture. To check a fresh migration cycle before recording data, run down/up as owner, then reapply grants and seed. Keep administrator credentials out of the runtime environment.
+
+## Build and run
+
+```bash
+scripts/sbtw clean scalafmtCheckAll compile test verifyDependencyLock writeRuntimeClasspath
+```
+
+Start from a fresh shell containing **only** `runtime.env` and normal process variables. Do not source administrator, migration or test files in that shell:
+
+```bash
+set -a; source $HOME/experiment-private/runtime.env; set +a
+java -Xmx768m -XX:ActiveProcessorCount=2 \
+  -cp "$(cat .local/runtime-classpath.txt)" experimentlog.Main
+```
+
+The server binds `127.0.0.1:8080` and checks that the two seeded project IDs match its configured token identities. pgJDBC requires the actual Cloud CA with `sslmode=verify-full`, including hostname verification. Hikari owns four connections, with bounded connect/validation waits. SQL statements are limited to 10 seconds, locks to 8, socket reads to 15. Headers have a 5-second deadline, request JSON is at most 16 KiB with a 5-second body deadline, and the response middleware has a 30-second deadline. SQL parameter logging is disabled.
+
+## Record, replay and search
+
+```bash
+curl --fail-with-body http://127.0.0.1:8080/runs \
+  -H "Authorization: Bearer $PROJECT_A_TOKEN" \
+  -H 'Content-Type: application/json' --data-binary @sample-run.json
+curl --fail-with-body http://127.0.0.1:8080/runs/search \
+  -H "Authorization: Bearer $PROJECT_A_TOKEN" \
+  -H 'Content-Type: application/json' \
+  --data '{"config":{"optimizer":"adam"},"title":"comparison","limit":5}'
+```
+
+Registration returns 201; an identical retained request returns 200 and its original complete response. Reuse the sample UUID to observe replay, or generate a new UUID for another run. Reusing a UUID with changed semantic content returns 409. The UUID is retained for the lifetime of its project data, without expiry or deletion in this example. Whitespace around titles is trimmed; configuration object order, measurement array order and decimal trailing zeros do not change the semantic payload. Changing a configuration string's whitespace does change it.
+
+Measurement names match `[a-z][a-z0-9_]{0,31}` and are unique. Values are **plain decimal strings**, absolute value at most 1,000,000 and at most six fractional digits. They pass through Scala `BigDecimal`, doobie and unconstrained PostgreSQL `numeric`; a CHECK rejects excess scale instead of silently rounding it. Responses return decimal strings. No `Double` participates in the storage path.
+
+Configuration is an object of at most ten ASCII keys (`[A-Za-z][A-Za-z0-9_]{0,23}`), with bounded strings, booleans or exact JSON numbers of magnitude at most 1,000,000 and normalized scale at most six. Null, arrays, nested objects, controls, unpaired Unicode surrogates, duplicate object keys and oversized numeric/exponent lexemes are rejected. GET `/runs/:id` retrieves a project-scoped run; GET `/project` returns the authenticated project ID.
+
+Search applies parameterized JSONB `@>` containment plus an optional literal title substring. `%`, `_` and backslash are escaped. Empty `{}` matches every configuration. Page size is 1–20 (default 10), ordered by `(created_at,id)` descending. Pass `nextCursor` as `cursor` to continue. This is a validated opaque keyset cursor over a **live** listing, not a frozen snapshot: later inserts can appear above an existing cursor. Populated pages use two prepared SELECT statements, loading all selected child rows together; immutable committed runs make the pair consistent. The GIN `jsonb_path_ops` index supports containment; a small fixture may correctly use a sequential scan. No speed claim is made.
+
+The parent and complete measurement set are one `ConnectionIO` with one `transact` at the HTTP boundary. The unique `(project_id,request_id)` key arbitrates races; the losing `ON CONFLICT DO NOTHING` statement is followed by a fresh READ COMMITTED lookup. A deferred constraint checks complete children at commit, and a transaction-ID trigger prevents attaching later measurements to an already committed run. This database role is trusted to submit its own consistent initial payload; it is not a sandbox for arbitrary hostile SQL.
+
+## Verification
+
+CI runs only formatting, compile, six boundary/semantic unit tests and dependency-record verification, without database credentials. Opt-in live checks need the running API, runtime env and test env:
+
+```bash
+python3 -m venv .venv
+.venv/bin/pip install -r tests/requirements.txt
+.venv/bin/pip check
+(set -a; source $HOME/experiment-private/runtime.env; source $HOME/experiment-private/test.env; set +a
+ .venv/bin/python tests/acceptance.py)
+scripts/sbtw Test/compile writeTestClasspath
+(set -a; source $HOME/experiment-private/runtime.env; set +a
+ java -Xmx512m -cp "$(cat .local/test-classpath.txt)" experimentlog.LiveProbe)
+```
+
+Acceptance uses independent HTTP clients with observed database lock contention, a controlled owner-installed after-child failure and runtime privilege checks. The test-only TLS hostname control passes the actual TLS session to pgJDBC's native verifier with a substituted incorrect name; the application retains its default verifier. LiveProbe also performs actual doobie SQL type analysis, observes two prepared statements for a populated search, and prints the actual GIN definition and unforced plan. Its test classes are absent from the production classpath. Live checks are never silently skipped by CI.
+
+## Stop and remove the dedicated fixture
+
+Stop the JVM first. Delete only the exact Cloud service ID you created, using the explicit organization:
+
+```bash
+clickhousectl cloud postgres delete YOUR_SERVICE_ID --org-id YOUR_ORG_ID --json
+clickhousectl cloud postgres list --org-id YOUR_ORG_ID --json
+```
+
+Deletion is asynchronous; verify that exact ID becomes absent. If retaining the service, use the administrator receipt (not runtime credentials) to drop the dedicated schema/roles after stopping all connections. Never run cleanup against another service. The VM can be stopped and preserved; source and private evidence should be saved separately before shutdown.

applications/experiment-log/build.sbt

@@ -0,0 +1,41 @@
+ThisBuild / scalaVersion := "3.9.0"
+ThisBuild / organization := "examples.clickhouse"
+ThisBuild / version := "0.1.0"
+
+val http4sVersion = "0.23.38"
+val doobieVersion = "1.0.0-RC13" // Maintained published release candidate, not stable 1.0.
+
+libraryDependencies ++= Seq(
+  "org.http4s" %% "http4s-ember-server" % http4sVersion,
+  "org.http4s" %% "http4s-dsl" % http4sVersion,
+  "org.http4s" %% "http4s-circe" % http4sVersion,
+  "org.typelevel" %% "cats-effect" % "3.7.1",
+  "org.typelevel" %% "doobie-core" % doobieVersion,
+  "org.typelevel" %% "doobie-hikari" % doobieVersion,
+  "org.typelevel" %% "doobie-postgres" % doobieVersion,
+  "org.typelevel" %% "doobie-postgres-circe" % doobieVersion,
+  "io.circe" %% "circe-core" % "0.14.16",
+  "io.circe" %% "circe-parser" % "0.14.16",
+  "com.zaxxer" % "HikariCP" % "7.1.0",
+  "org.postgresql" % "postgresql" % "42.7.13",
+  "org.slf4j" % "slf4j-simple" % "2.0.17",
+  "org.scalameta" %% "munit" % "1.3.6" % Test
+)
+
+// Record selected modules from the actual resolver, including test dependencies.
+val resolvedModules = taskKey[String]("Canonical selected dependency record")
+resolvedModules := (Compile / update).value.configurations.flatMap(_.modules)
+  .filterNot(_.evicted).map(m => s"${m.module.organization}:${m.module.name}:${m.module.revision}")
+  .distinct.sorted.mkString("", "\n", "\n")
+val writeDependencyLock = taskKey[Unit]("Write resolved dependency record intentionally")
+writeDependencyLock := IO.write(baseDirectory.value / "dependency-lock.txt", resolvedModules.value)
+val verifyDependencyLock = taskKey[Unit]("Fail when resolved dependencies differ from committed record")
+verifyDependencyLock := {
+  val path = baseDirectory.value / "dependency-lock.txt"
+  require(path.exists && IO.read(path) == resolvedModules.value, "Dependency record differs; review before writeDependencyLock")
+}
+val writeRuntimeClasspath = taskKey[Unit]("Write native production JVM classpath")
+writeRuntimeClasspath := IO.write(baseDirectory.value / ".local/runtime-classpath.txt", (Runtime / fullClasspath).value.files.mkString(java.io.File.pathSeparator))
+
+val writeTestClasspath = taskKey[Unit]("Write native live-check JVM classpath")
+writeTestClasspath := IO.write(baseDirectory.value / ".local/test-classpath.txt", (Test / fullClasspath).value.files.mkString(java.io.File.pathSeparator))

applications/experiment-log/dependency-lock.txt

@@ -0,0 +1,109 @@
+co.fs2:fs2-core_3:3.14.0
+co.fs2:fs2-io_3:3.14.0
+com.comcast:ip4s-core_3:3.8.0
+com.fasterxml.jackson.core:jackson-annotations:2.21
+com.fasterxml.jackson.core:jackson-core:2.13.4
+com.fasterxml.jackson.core:jackson-databind:2.13.4.2
+com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.13.2
+com.twitter:hpack:1.0.2
+com.vladsch.flexmark:flexmark-ext-anchorlink:0.64.8
+com.vladsch.flexmark:flexmark-ext-autolink:0.64.8
+com.vladsch.flexmark:flexmark-ext-emoji:0.64.8
+com.vladsch.flexmark:flexmark-ext-gfm-strikethrough:0.64.8
+com.vladsch.flexmark:flexmark-ext-gfm-tasklist:0.64.8
+com.vladsch.flexmark:flexmark-ext-ins:0.64.8
+com.vladsch.flexmark:flexmark-ext-superscript:0.64.8
+com.vladsch.flexmark:flexmark-ext-tables:0.64.8
+com.vladsch.flexmark:flexmark-ext-wikilink:0.64.8
+com.vladsch.flexmark:flexmark-ext-yaml-front-matter:0.64.8
+com.vladsch.flexmark:flexmark-jira-converter:0.64.8
+com.vladsch.flexmark:flexmark-util-ast:0.64.8
+com.vladsch.flexmark:flexmark-util-builder:0.64.8
+com.vladsch.flexmark:flexmark-util-collection:0.64.8
+com.vladsch.flexmark:flexmark-util-data:0.64.8
+com.vladsch.flexmark:flexmark-util-dependency:0.64.8
+com.vladsch.flexmark:flexmark-util-format:0.64.8
+com.vladsch.flexmark:flexmark-util-html:0.64.8
+com.vladsch.flexmark:flexmark-util-misc:0.64.8
+com.vladsch.flexmark:flexmark-util-options:0.64.8
+com.vladsch.flexmark:flexmark-util-sequence:0.64.8
+com.vladsch.flexmark:flexmark-util-visitor:0.64.8
+com.vladsch.flexmark:flexmark-util:0.64.8
+com.vladsch.flexmark:flexmark:0.64.8
+com.zaxxer:HikariCP:7.1.0
+io.circe:circe-core_3:0.14.16
+io.circe:circe-jawn_3:0.14.16
+io.circe:circe-numbers_3:0.14.16
+io.circe:circe-parser_3:0.14.16
+io.get-coursier:interface:1.0.29-M4
+junit:junit:4.13.2
+nl.big-o:liqp:0.9.2.3
+org.checkerframework:checker-qual:3.55.1
+org.hamcrest:hamcrest-core:1.3
+org.http4s:http4s-circe_3:0.23.38
+org.http4s:http4s-core_3:0.23.38
+org.http4s:http4s-crypto_3:0.2.5
+org.http4s:http4s-dsl_3:0.23.38
+org.http4s:http4s-ember-core_3:0.23.38
+org.http4s:http4s-ember-server_3:0.23.38
+org.http4s:http4s-jawn_3:0.23.38
+org.http4s:http4s-server_3:0.23.38
+org.jetbrains:annotations:24.0.1
+org.jline:jline-native:4.0.14
+org.jline:jline-reader:4.0.14
+org.jline:jline-terminal-jni:4.0.14
+org.jline:jline-terminal:4.0.14
+org.jsoup:jsoup:1.22.2
+org.log4s:log4s_3:1.10.0
+org.nibor.autolink:autolink:0.6.0
+org.portable-scala:portable-scala-reflect_2.13:1.1.3
+org.postgresql:postgresql:42.7.13
+org.scala-lang.modules:scala-asm:9.9.0-scala-1
+org.scala-lang:scala-library:3.9.0
+org.scala-lang:scala3-compiler_3:3.9.0
+org.scala-lang:scala3-directives-parser_3:3.9.0
+org.scala-lang:scala3-interfaces:3.9.0
+org.scala-lang:scala3-library_3:3.9.0
+org.scala-lang:scala3-repl_3:3.9.0
+org.scala-lang:scala3-tasty-inspector_3:3.9.0
+org.scala-lang:scaladoc_3:3.9.0
+org.scala-lang:tasty-core_3:3.9.0
+org.scala-sbt:compiler-interface:1.12.0
+org.scala-sbt:test-interface:1.0
+org.scala-sbt:util-interface:1.11.5
+org.scalameta:junit-interface:1.3.6
+org.scalameta:munit-diff_3:1.3.6
+org.scalameta:munit_3:1.3.6
+org.scodec:scodec-bits_3:1.2.5
+org.slf4j:slf4j-api:1.7.36
+org.slf4j:slf4j-api:2.0.17
+org.slf4j:slf4j-simple:2.0.17
+org.snakeyaml:snakeyaml-engine:3.0.1
+org.tpolecat:typename_3:1.1.2
+org.typelevel:algebra_3:2.13.0
+org.typelevel:case-insensitive_3:1.5.0
+org.typelevel:cats-collections-core_3:0.9.10
+org.typelevel:cats-core_3:2.13.0
+org.typelevel:cats-effect-kernel_3:3.7.1
+org.typelevel:cats-effect-std_3:3.7.1
+org.typelevel:cats-effect_3:3.7.1
+org.typelevel:cats-free_3:2.13.0
+org.typelevel:cats-kernel_3:2.13.0
+org.typelevel:cats-mtl_3:1.7.0
+org.typelevel:cats-parse_3:1.1.0
+org.typelevel:doobie-core_3:1.0.0-RC13
+org.typelevel:doobie-free_3:1.0.0-RC13
+org.typelevel:doobie-hikari_3:1.0.0-RC13
+org.typelevel:doobie-postgres-circe_3:1.0.0-RC13
+org.typelevel:doobie-postgres_3:1.0.0-RC13
+org.typelevel:idna4s-core_3:0.1.0
+org.typelevel:jawn-fs2_3:2.6.0
+org.typelevel:jawn-parser_3:1.7.0
+org.typelevel:literally_3:1.2.0
+org.typelevel:log4cats-core_3:2.8.0
+org.typelevel:log4cats-slf4j_3:2.8.0
+org.typelevel:vault_3:3.7.0
+tools.jackson.core:jackson-core:3.1.2
+tools.jackson.core:jackson-databind:3.1.2
+tools.jackson.dataformat:jackson-dataformat-yaml:3.1.2
+ua.co.k:strftime4j:1.0.6

applications/experiment-log/migrations/001-down.sql

@@ -0,0 +1,11 @@
+BEGIN;
+SELECT pg_advisory_xact_lock(hashtextextended('experiment_log:001', 0));
+DROP TABLE experiment_log.measurements;
+DROP TABLE experiment_log.runs;
+DROP TABLE experiment_log.projects;
+DROP FUNCTION experiment_log.complete_run();
+DROP FUNCTION experiment_log.initial_measurement();
+DROP FUNCTION experiment_log.valid_measurement_payload(jsonb);
+DROP FUNCTION experiment_log.valid_config(jsonb);
+DELETE FROM experiment_log.schema_migrations WHERE version=1;
+COMMIT;