diff --git a/.gitignore b/.gitignore index 380efe19e..622aebe01 100644 --- a/.gitignore +++ b/.gitignore @@ -60,3 +60,7 @@ frontend/.astro # Bundler/vendored dependencies /vendor/ + +# Local bundle builds +/dist/ + diff --git a/.rubocop.yml b/.rubocop.yml index ee5e2c04c..328acf9b4 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -5,6 +5,7 @@ plugins: - rubocop-thread_safety AllCops: + TargetRubyVersion: 4.0 DisplayCopNames: true NewCops: enable Exclude: @@ -38,6 +39,10 @@ Style/Documentation: AllowedConstants: - App +Style/ItBlockParameter: + Enabled: true + EnforcedStyle: allow_single_line + RSpec/SpecFilePathFormat: Exclude: - 'spec/html2rss/web/app/*_spec.rb' diff --git a/AGENTS.md b/AGENTS.md index 07b31e815..c73412f4e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,21 +54,63 @@ See [docs/design-system.md](docs/design-system.md) for visual rules. - **No host execution:** All commands MUST run inside the Dev Container via `make` or `bundle exec`. - **No skipped quality gate:** Opening a PR without a green Dev Container gate is forbidden. If the gate cannot run, do not open the PR; fix the environment or hand off with explicit blocker + next command for the user. +## Ruby 4 Style + +**Ruby 4.0+ only** (see `.tool-versions`). No Ruby 3.x backward-compat shims, guards, or dual-path APIs. + +### Baseline + +- `# frozen_string_literal: true` on every `.rb` file +- Plain Ruby — no ActiveSupport +- Keyword arguments for public multi-arg APIs +- Typed YARD on public methods in `app/` (`@param`, `@return`) — enforced by `make yard-verify-public-docs` + +### Modern syntax (prefer consistently) + +| Idiom | Use instead of | +| --- | --- | +| Leading `&&` / `\|\|` at line start (Ruby 4) | Trailing operators on long wrapped conditions | +| `it` in single-parameter blocks | `{ \|x\| x.foo }` when block has one arg only | +| Pattern matching (`in`, `case … in`) | Deep `if/elsif` chains on shape | +| `Data.define` | OpenStruct / hand-rolled structs | +| `filter_map`, `index_by`, `then`, `match?` | Verbose `map`/`compact`, nested `if`, `=~` | +| Endless `def` | One-line pure helpers when RuboCop allows | +| Core `Set` (no `require 'set'`) | Array membership/diff on growing collections | + +### Performance (agent defaults) + +- **Set** for catalog/diff/membership when sizes can grow +- **Memoize** repeated `ENV.fetch` / pure computations on hot paths +- **One owner** for duplicated helpers — dedupe before splitting into new files +- **Functional iterators** over imperative loops +- **No metric-driven micro-methods** whose only purpose is satisfying RuboCop metrics +- Do **not** document ZJIT/Ruby Box/Ractor as defaults + +### Web-specific deltas + +- Prefer `class << self` + `private` over `module_function` (see docs/README Architectural Constraints) +- Do not use `send(...)` to reach private APIs in app code or specs +- Specs: table-drive matrices; `:aggregate_failures` for discriminating multi-assert examples +- LOC: dedupe/unify before extracting — new files only when they buy a real seam or test surface + ## Config catalog API -Public feed-directory metadata for embedded and local configs. +Public feed-directory metadata from verified registry bundles (including the built-in `local` path bundle when `catalog: true`). | Item | Detail | | --- | --- | | Endpoint | `GET /api/v1/configs` | | Flag | `CONFIG_CATALOG_ENABLED` (default `true`; set `false` to disable) | | Disabled response | `404` with `{ "error": "catalog_disabled" }` | -| Embedded entries | `Html2rss::Configs::Catalog.entries` — do not re-walk YAML in the handler | -| Local entries | `Catalog::Merge` includes `feeds.yml` feeds only when `directory.title` is set | +| Registry entries | `Registry::Index.current.catalog_rows` — loads signed/path bundles from `config/registries.yml`; adds `source: registry`, `registry: ` | +| Local registry | Built-in `local` at `data/registries/local` (`catalog: false` by default); populate with `html2rss-web registry add FILE_OR_DIR` | +| Per-registry privacy | `catalog: false` in `registries.yml` omits that registry from the API (feeds still served) | | Starter feeds (UI) | Frontend `selectStarterFeeds` when feed creation is disabled; catalog find uses full catalog when enabled | | Catalog find | `findCatalogEntries` → multi-hit list under create URL; links via `catalogFeedHref` (path + defaults) | | CORS | Route-scoped on `/api/v1/configs` only (`GET`, `OPTIONS`) | -| Root metadata | `GET /api/v1/` exposes `instance.catalog: { enabled, url }` | +| Root metadata | `GET /api/v1/` exposes `instance.catalog: { enabled, url }` and `instance.registries` sync status | | Contract SSOT | Request specs under `spec/html2rss/web/api/v1_spec.rb` and generated `public/openapi.yaml` | +Registry sync: `bin/html2rss-web registry status`; embedded bundle + optional runtime sync via `Registry::Sync.boot!`. See [docs/README.md](docs/README.md#registry-sync-runbook). + After handler or envelope changes: `make openapi` and `make ci-ready`. diff --git a/CONTEXT.md b/CONTEXT.md index 81c2ec430..e6685160f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -54,3 +54,12 @@ Audit channel: snake_case `security_event` with IP / user-agent / token hash. Au ### LogEvent Shared emit plumbing for both channels (`RequestContext`, `LogSanitizer`, `AppLogger` / Sentry). Not a third public facade. + +### Registry Index +Sole feed repository over registry bundles (official sync/path plus the built-in `local` path bundle). Builds catalog wire rows, enforces load-time trust and channel-domain allowlists, and serves `config_for` / `catalog_rows` / `status`. + +### Registry Sync +Backend orchestration for fetch → verify → stage/promote of signed registry bundles. Owns boot initialization, background refresh, CLI exit codes, and catalog-change telemetry after promotion. + +### Sync Transport +Backend HTTPS fetch, sync URL resolution (GitHub releases and channel defaults), and manifest version gating used by `Registry::Sync` and parse-time config resolution. diff --git a/Dockerfile b/Dockerfile index d9bc8e0a5..0f16d5a53 100644 --- a/Dockerfile +++ b/Dockerfile @@ -38,7 +38,26 @@ RUN apk add --no-cache \ /usr/local/bundle/bundler/gems/*/.git \ /usr/local/bundle/cache/bundler/git -# Stage 3: Runtime +# Stage 3: Official Registry Artifact Builder & Verifier +FROM ${RUBY_BASE_IMAGE} AS registry-builder + +ARG REGISTRY_BUNDLE_URL="https://github.com/html2rss/html2rss-configs/releases/latest/download/registry-bundle.tar.gz" + +WORKDIR /app + +COPY --from=builder /usr/local/bundle /usr/local/bundle +COPY bin ./bin +COPY app ./app +COPY config ./config + +# hadolint ignore=DL3018 +RUN apk add --no-cache curl tar ca-certificates \ + && curl -fsSL -o registry-bundle.tar.gz "${REGISTRY_BUNDLE_URL}" \ + && mkdir -p /build/official \ + && tar -xzf registry-bundle.tar.gz -C /build/official \ + && bin/html2rss-web registry verify --registry official --dir /build/official + +# Stage 4: Runtime FROM ${RUBY_BASE_IMAGE} LABEL maintainer="Gil Desmarais " @@ -51,13 +70,14 @@ ARG GIT_SHA ENV PORT=4000 \ RACK_ENV=production \ RUBY_YJIT_ENABLE=1 \ + PATH="/app/bin:$PATH" \ BUILD_TAG=${BUILD_TAG} \ GIT_SHA=${GIT_SHA} EXPOSE $PORT HEALTHCHECK --interval=30m --timeout=60s --start-period=5s \ - CMD ["/app/bin/docker-healthcheck"] + CMD ["/app/bin/html2rss-web", "healthcheck"] ARG USER=html2rss ARG UID=991 @@ -78,6 +98,7 @@ RUN apk add --no-cache \ && mkdir -p /app \ && mkdir -p /app/tmp/rack-cache-body \ && mkdir -p /app/tmp/rack-cache-meta \ + && mkdir -p /app/data/registries \ && chown "$USER":"$USER" -R /app WORKDIR /app @@ -85,10 +106,11 @@ WORKDIR /app USER 991 COPY --from=builder /usr/local/bundle /usr/local/bundle -COPY --chown=$USER:$USER bin/docker-healthcheck ./bin/docker-healthcheck +COPY --chown=$USER:$USER bin ./bin COPY --chown=$USER:$USER Gemfile Gemfile.lock app.rb config.ru ./ COPY --chown=$USER:$USER app ./app COPY --chown=$USER:$USER config ./config +COPY --from=registry-builder --chown=$USER:$USER /build/official ./registries/official COPY --chown=$USER:$USER public ./public COPY --from=frontend-builder --chown=$USER:$USER /app/frontend/dist ./frontend/dist diff --git a/Gemfile b/Gemfile index 5816b002d..85e42a7e8 100644 --- a/Gemfile +++ b/Gemfile @@ -4,13 +4,8 @@ source 'https://rubygems.org' git_source(:github) { |repo_name| "https://github.com/#{repo_name}" } -gem 'html2rss', '~> 0.27' -# gem 'html2rss', github: 'html2rss/html2rss', branch: 'master' -gem 'html2rss-configs', github: 'html2rss/html2rss-configs' - -# Use these instead of the two above (uncomment them) when developing locally: -# gem 'html2rss', path: '../html2rss' -# gem 'html2rss-configs', path: '../html2rss-configs' +# Until rubygems 0.28.0: git branch; local monorepo: BUNDLE_LOCAL__HTML2RSS=/path/to/html2rss +gem 'html2rss', github: 'html2rss/html2rss', branch: 'feat/registry-v1' gem 'base64' gem 'rack-cache' diff --git a/Gemfile.lock b/Gemfile.lock index 9e2e0b168..17812a510 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,9 +1,29 @@ GIT - remote: https://github.com/html2rss/html2rss-configs - revision: f888a8dc5bb260c998c5ae8f39bb7d45320638ee + remote: https://github.com/html2rss/html2rss + revision: 9794632bb4328a7c8462ab76633c51dfd048e774 + branch: feat/registry-v1 specs: - html2rss-configs (0.2.0) - html2rss + html2rss (0.28.0) + addressable (~> 2.7) + brotli + dry-validation + faraday (> 2.0.1, < 3.0) + faraday-follow_redirects + faraday-gzip (~> 3) + kramdown + mcp (~> 1.2) + mime-types (> 3.0) + nokogiri (>= 1.10, < 2.0) + rack (~> 3.0) + rackup (~> 2.0) + regexp_parser + reverse_markdown (~> 3.0) + rss + sanitize + thor + tzinfo + webrick (~> 1.9) + zeitwerk GEM remote: https://rubygems.org/ @@ -41,7 +61,7 @@ GEM public_suffix (>= 2.0.2, < 8.0) ast (2.4.3) base64 (0.3.0) - bigdecimal (4.1.2) + bigdecimal (4.1.3) brotli (0.8.0) builder (3.3.0) climate_control (1.2.0) @@ -90,40 +110,19 @@ GEM zeitwerk (~> 2.6) erb (6.0.7) erubi (1.13.1) - faraday (2.14.3) + faraday (2.14.4) faraday-net_http (>= 2.0, < 3.5) json logger faraday-follow_redirects (0.5.0) faraday (>= 1, < 3) - faraday-gzip (3.1.0) + faraday-gzip (3.2.0) faraday (>= 2.0, < 3) zlib (~> 3.0) faraday-net_http (3.4.4) net-http (~> 0.5) hana (1.3.7) hashdiff (1.2.1) - html2rss (0.27.2) - addressable (~> 2.7) - brotli - dry-validation - faraday (> 2.0.1, < 3.0) - faraday-follow_redirects - faraday-gzip (~> 3) - kramdown - mcp (~> 1.2) - mime-types (> 3.0) - nokogiri (>= 1.10, < 2.0) - rack (~> 3.0) - rackup (~> 2.0) - regexp_parser - reverse_markdown (~> 3.0) - rss - sanitize - thor - tzinfo - webrick (~> 1.9) - zeitwerk i18n (1.15.2) concurrent-ruby (~> 1.0) io-console (0.9.2) @@ -146,17 +145,20 @@ GEM loofah (2.25.2) crass (~> 1.0.2) nokogiri (>= 1.12.0) - mcp (1.3.0) + mcp (1.7.0) json_schemer (>= 2.4) mime-types (3.7.0) logger mime-types-data (~> 3.2025, >= 3.2025.0507) - mime-types-data (3.2026.0701) + mime-types-data (3.2026.1006) minitest (6.0.6) drb (~> 2.0) prism (~> 1.5) - net-http (0.9.1) + net-http (0.10.0) + net-protocol (>= 0.3.0) uri (>= 0.11.1) + net-protocol (0.4.0) + timeout nio4r (2.7.5) nokogiri (1.19.4-aarch64-linux-gnu) racc (~> 1.4) @@ -215,7 +217,7 @@ GEM prism (>= 1.6.0) rbs (>= 4.0.0) tsort - regexp_parser (2.12.0) + regexp_parser (2.13.1) reline (0.7.0) io-console (~> 0.5) reverse_markdown (3.0.2) @@ -285,16 +287,17 @@ GEM concurrent-ruby (~> 1.0, >= 1.0.2) logger simplecov (1.1.1) - simpleidn (0.2.3) + simpleidn (0.3.0) stackprof (0.2.28) thor (1.5.0) + timeout (0.6.1) tsort (0.2.0) tzinfo (2.0.6) concurrent-ruby (~> 1.0) unicode-display_width (3.2.0) unicode-emoji (~> 4.1) unicode-emoji (4.2.0) - uri (1.1.1) + uri (1.1.2) useragent (0.16.11) vcr (6.4.0) webmock (3.26.2) @@ -304,7 +307,7 @@ GEM webrick (1.9.2) yard (0.9.45) zeitwerk (2.8.3) - zlib (3.2.3) + zlib (3.2.4) PLATFORMS aarch64-linux-gnu @@ -319,8 +322,7 @@ PLATFORMS DEPENDENCIES base64 climate_control - html2rss (~> 0.27) - html2rss-configs! + html2rss! irb puma rack-cache @@ -351,7 +353,7 @@ CHECKSUMS addressable (2.9.0) sha256=7fdf6ac3660f7f4e867a0838be3f6cf722ace541dd97767fa42bc6cfa980c7af ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383 base64 (0.3.0) sha256=27337aeabad6ffae05c265c450490628ef3ebd4b67be58257393227588f5a97b - bigdecimal (4.1.2) sha256=53d217666027eab4280346fba98e7d5b66baaae1b9c3c1c0ffe89d48188a3fbd + bigdecimal (4.1.3) sha256=61ebe1e5e559bdc3cc6f2c0ee7f427321fc838f59611c294356eb04d6e21cf66 brotli (0.8.0) sha256=0c5a42046b3b603fb109656881147fd76064c034b7d19c1b4fcc32a093a4d55d builder (3.3.0) sha256=497918d2f9dca528fdca4b88d84e4ef4387256d984b8154e9d5d3fe5a9c8835f climate_control (1.2.0) sha256=36b21896193fa8c8536fa1cd843a07cf8ddbd03aaba43665e26c53ec1bd70aa5 @@ -371,14 +373,13 @@ CHECKSUMS dry-validation (1.11.1) sha256=70900bb5a2d911c8aab566d3e360c6bff389b8bf92ea8e04885ce51c41ff8085 erb (6.0.7) sha256=c5ca6dc25b0ef974a44dc8f59fe847577122483b1968a38dec305c60bf91ee92 erubi (1.13.1) sha256=a082103b0885dbc5ecf1172fede897f9ebdb745a4b97a5e8dc63953db1ee4ad9 - faraday (2.14.3) sha256=1882247e6766615c8220b4392bf1d27f6ebb63d8e28267587cef1fb0bf37f278 + faraday (2.14.4) sha256=9bb4408c44621b0dbaaaa39e5dbfc4dbe58cc6751b6eb94d16ae9e1f2f2fab6e faraday-follow_redirects (0.5.0) sha256=5cde93c894b30943a5d2b93c2fe9284216a6b756f7af406a1e55f211d97d10ad - faraday-gzip (3.1.0) sha256=320783690be169f9b7ddde11598b77156951343753f66a9ab98b1f6694433ff8 + faraday-gzip (3.2.0) sha256=bedae115512201df9b9b13c95cb9a1aa4cb8a848286212d593b20e9456c8578f faraday-net_http (3.4.4) sha256=0e78af151747ed1b00f33e25973b4bc220d7f16c00c39676817c8b12331eb588 hana (1.3.7) sha256=5425db42d651fea08859811c29d20446f16af196308162894db208cac5ce9b0d hashdiff (1.2.1) sha256=9c079dbc513dfc8833ab59c0c2d8f230fa28499cc5efb4b8dd276cf931457cd1 - html2rss (0.27.2) sha256=82b28308c023cb669704bf9b0612aa2a00730be4b53d88f957d456ba5e631ac8 - html2rss-configs (0.2.0) + html2rss (0.28.0) i18n (1.15.2) sha256=00f9eb62412fe593b2a65a97daa75300d37abb8f7202ec748e94b6d46a9dd1b5 io-console (0.9.2) sha256=efa74f891dd03c0939a931dfc6e74c2813d904763d456ea9762b0525e748db08 irb (1.18.0) sha256=de9454a0703a54704b9811a5ef31a60c86949fbf4013fcf244fabc7c775248e3 @@ -389,11 +390,12 @@ CHECKSUMS lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87 logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203 loofah (2.25.2) sha256=2007f746959ac65552456e04b433e83deb22759ab38c838b4445c70e43425918 - mcp (1.3.0) sha256=9395aa3a054eb8986b7714ec8abb25b533729af896f4875da4c4cb7a3024fbae + mcp (1.7.0) sha256=bdf3261177b948b0777a6f247c57aad5f64c84830c02d126d4bfea188b0cebb7 mime-types (3.7.0) sha256=dcebf61c246f08e15a4de34e386ebe8233791e868564a470c3fe77c00eed5e56 - mime-types-data (3.2026.0701) sha256=cd8811e1fb89d836499ba0582368a10ee74cef929ba956d1d5ddca045e6a730f + mime-types-data (3.2026.1006) sha256=657973f3fdf857f493584e05f9d093417456836133fd5b30c5a078ee54e4ae2f minitest (6.0.6) sha256=153ea36d1d987a62942382b61075745042a2b3123b1cd48f4c3675af9cc7d6f1 - net-http (0.9.1) sha256=25ba0b67c63e89df626ed8fac771d0ad24ad151a858af2cc8e6a716ca4336996 + net-http (0.10.0) sha256=1a2ff6e37a05724973c43f2bf7bf105f59c36a4cd0a7e9b1c021a0a566987c27 + net-protocol (0.4.0) sha256=f9f90c8c6f64794a45160bf422177a336db12809de3beeeaa26b88fcd2e8c853 nio4r (2.7.5) sha256=6c90168e48fb5f8e768419c93abb94ba2b892a1d0602cb06eef16d8b7df1dca1 nokogiri (1.19.4-aarch64-linux-gnu) sha256=1269fb644a6de405057a53dd5c762b1209b43ca7424f839454d3dbc677c31a8f nokogiri (1.19.4-aarch64-linux-musl) sha256=35c65b9ce72b3bb03207bdbe7067915019dc18c1b9b59139684bd6690fdd01af @@ -423,7 +425,7 @@ CHECKSUMS rake (13.4.2) sha256=cb825b2bd5f1f8e91ca37bddb4b9aaf345551b4731da62949be002fa89283701 rbs (4.1.3) sha256=0c4474a9751cdc14364bfad0b3e53678323bbdc2c31683b0445932867dbab8c4 rdoc (8.0.0) sha256=03bf8c08a9639658855a0cfd77c0abca8325c227693f7f33f82957811348c469 - regexp_parser (2.12.0) sha256=35a916a1d63190ab5c9009457136ae5f3c0c7512d60291d0d1378ba18ce08ebb + regexp_parser (2.13.1) sha256=5aedb6b7c35688f51e86eef17a15374dfd287c83f4ca644b2c5f3ec50a33b44b reline (0.7.0) sha256=5b012d8e55dbf9d450f12bde2cf7d15ff546ae80b3f8f3b30e570d431815583d reverse_markdown (3.0.2) sha256=818ebb92ce39dbb1a291690dd1ec9a6d62530d4725296b17e9c8f668f9a5b8af rexml (3.4.4) sha256=19e0a2c3425dfbf2d4fc1189747bdb2f849b6c5e74180401b15734bc97b5d142 @@ -447,21 +449,22 @@ CHECKSUMS securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1 sentry-ruby (6.7.0) sha256=b4b915d79a0395ad02106d4012aa1914641456e042ce36393f15bd1630d8db42 simplecov (1.1.1) sha256=25825ef13f0b2e74694d769817dad6ab8e90131dabdaa666e522fea105521e78 - simpleidn (0.2.3) sha256=08ce96f03fa1605286be22651ba0fc9c0b2d6272c9b27a260bc88be05b0d2c29 + simpleidn (0.3.0) sha256=12ca730bed2f3db04d11e9bfd1bca3e11fb37f55b21eb2e9793fb5814bf54d03 stackprof (0.2.28) sha256=4ec2ace02f386012b40ca20ef80c030ad711831f59511da12e83b34efb0f9a04 thor (1.5.0) sha256=e3a9e55fe857e44859ce104a84675ab6e8cd59c650a49106a05f55f136425e73 + timeout (0.6.1) sha256=78f57368a7e7bbadec56971f78a3f5ecbcfb59b7fcbb0a3ed6ddc08a5094accb tsort (0.2.0) sha256=9650a793f6859a43b6641671278f79cfead60ac714148aabe4e3f0060480089f tzinfo (2.0.6) sha256=8daf828cc77bcf7d63b0e3bdb6caa47e2272dcfaf4fbfe46f8c3a9df087a829b unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42 unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f - uri (1.1.1) sha256=379fa58d27ffb1387eaada68c749d1426738bd0f654d812fcc07e7568f5c57c6 + uri (1.1.2) sha256=89292e98eabbef6248b0a0ed5e7e398e152f83f17560023a6d94a7f492b169fb useragent (0.16.11) sha256=700e6413ad4bb954bb63547fa098dddf7b0ebe75b40cc6f93b8d54255b173844 vcr (6.4.0) sha256=077ac92cc16efc5904eb90492a18153b5e6ca5398046d8a249a7c96a9ea24ae6 webmock (3.26.2) sha256=774556f2ea6371846cca68c01769b2eac0d134492d21f6d0ab5dd643965a4c90 webrick (1.9.2) sha256=beb4a15fc474defed24a3bda4ffd88a490d517c9e4e6118c3edce59e45864131 yard (0.9.45) sha256=52e211493f7cb8a3ebf7e104a25a1e73937a3103092545d34cb88fafebb3dc51 zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5 - zlib (3.2.3) sha256=5bd316698b32f31a64ab910a8b6c282442ca1626a81bbd6a1674e8522e319c20 + zlib (3.2.4) sha256=a0d0e5845a18d839c078fb5b6e4ae523ffb918601c2004a77c081ee211f9b038 BUNDLED WITH 4.0.9 diff --git a/README.md b/README.md index 987776f24..beb64e2d7 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,8 @@ Test drive the app with these steps: 3. Open [`http://localhost:4000/`](http://localhost:4000/) in your browser 4. When prompted for a token, use `CHANGE_ME_ADMIN_TOKEN` +Add a custom feed YAML with `html2rss-web registry add path/to/feed.yml` (writes into the built-in `local` registry bundle under `data/registries/local`). + > [!IMPORTANT] > This is a first-run demo path, not a production-ready setup. diff --git a/app/web/api/v1/configs.rb b/app/web/api/v1/configs.rb index 900b9b9b9..694b46252 100644 --- a/app/web/api/v1/configs.rb +++ b/app/web/api/v1/configs.rb @@ -17,7 +17,7 @@ def index(_router) entries, duration_ms = build_entries emit_success(entries.size, duration_ms) success_payload(entries) - rescue Html2rss::Configs::Catalog::MissingDirectoryTitle => error + rescue Html2rss::Registry::CatalogBuilder::MissingDirectoryTitle => error emit_failure(error) raise end @@ -25,12 +25,16 @@ def index(_router) private def build_entries - started = Process.clock_gettime(Process::CLOCK_MONOTONIC) - entries = Html2rss::Web::Catalog::Merge.call - duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round + started = monotonic_now + entries = Registry::Index.current.catalog_rows + duration_ms = ((monotonic_now - started) * 1000).round [entries, duration_ms] end + def monotonic_now + Process.clock_gettime(Process::CLOCK_MONOTONIC) + end + def emit_success(count, duration_ms) Observability.emit( event_name: 'catalog.build', diff --git a/app/web/api/v1/health.rb b/app/web/api/v1/health.rb index 0e1effdc4..7e236a04f 100644 --- a/app/web/api/v1/health.rb +++ b/app/web/api/v1/health.rb @@ -94,7 +94,7 @@ def bearer_token(request) # @return [void] def verify_configuration! - LocalConfig.load_snapshot + Registry::Index.current rescue StandardError raise Html2rss::Web::HealthCheckFailedError end diff --git a/app/web/api/v1/root_metadata.rb b/app/web/api/v1/root_metadata.rb index 562f5a86b..e3d38096e 100644 --- a/app/web/api/v1/root_metadata.rb +++ b/app/web/api/v1/root_metadata.rb @@ -27,16 +27,41 @@ def build(router) # @return [Hash{Symbol=>Object}] def instance_payload(router) { - feed_creation: { - enabled: Flags.auto_source_enabled?, - access_token_required: Flags.auto_source_enabled? - }, - catalog: { - enabled: Flags.config_catalog_enabled?, - url: "#{router.base_url}/api/v1/configs" - } + feed_creation: feed_creation_payload, + catalog: catalog_payload(router), + registries: registry_status_rows + } + end + + # @return [Hash{Symbol => Object}] + def feed_creation_payload + auto_source = Flags.auto_source_enabled? + { + enabled: auto_source, + access_token_required: auto_source + } + end + + # @param router [Roda::RodaRequest] + # @return [Hash{Symbol => Object}] + def catalog_payload(router) + { + enabled: Flags.config_catalog_enabled?, + url: "#{router.base_url}/api/v1/configs" } end + + # @return [Array Object}>] + def registry_status_rows + Registry::Index.current.status.map do |entry| + { + id: entry.id, + version: entry.version, + updated_at: entry.updated_at&.utc&.iso8601, + sync_mode: entry.mode.to_s + } + end + end end end end diff --git a/app/web/boot.rb b/app/web/boot.rb index deed80ba6..ceba52f75 100644 --- a/app/web/boot.rb +++ b/app/web/boot.rb @@ -60,7 +60,10 @@ def build_loader def configure_loader(new_loader) new_loader.push_dir(app_root, namespace: Html2rss) collapsed_web_dirs.each { |path| new_loader.collapse(path) } - new_loader.inflector.inflect('api_v1' => 'ApiV1') + new_loader.inflector.inflect( + 'api_v1' => 'ApiV1', + 'cli' => 'CLI' + ) end # @return [Array] diff --git a/app/web/boot/development_reloader.rb b/app/web/boot/development_reloader.rb index 82e28a436..e4e2ea589 100644 --- a/app/web/boot/development_reloader.rb +++ b/app/web/boot/development_reloader.rb @@ -61,7 +61,7 @@ def watched_files # @return [void] def reset_runtime_caches! - Html2rss::Web::LocalConfig.reload!(reason: 'code_reload') if defined?(Html2rss::Web::LocalConfig) + Html2rss::Web::Registry::Index.reload! if defined?(Html2rss::Web::Registry::Index) Html2rss::Web::AccountManager.reload!(reason: 'code_reload') if defined?(Html2rss::Web::AccountManager) end end diff --git a/app/web/boot/setup.rb b/app/web/boot/setup.rb index 55ac05b34..7c95660ff 100644 --- a/app/web/boot/setup.rb +++ b/app/web/boot/setup.rb @@ -24,6 +24,7 @@ def call! configure_request_service! configure_runtime_logging! configure_gem_defaults! + configure_registry! log_startup! end @@ -31,10 +32,9 @@ def call! # @return [void] def configure_gem_defaults! - global_config = LocalConfig.global Html2rss.configure do |config| - config.headers = global_config[:headers] if global_config[:headers] - config.stylesheets = global_config[:stylesheets] if global_config[:stylesheets] + config.headers = Feeds::FeedChrome.headers + config.stylesheets = Feeds::FeedChrome.stylesheets end end @@ -76,6 +76,11 @@ def configure_runtime_logging! Rack::Timeout::Logger.logger = AppLogger.logger end + # @return [void] + def configure_registry! + Registry::Sync.boot! + end + # @return [void] def log_startup! AppLogger.logger.info( diff --git a/app/web/catalog/merge.rb b/app/web/catalog/merge.rb deleted file mode 100644 index d7e6b7734..000000000 --- a/app/web/catalog/merge.rb +++ /dev/null @@ -1,101 +0,0 @@ -# frozen_string_literal: true - -require 'html2rss/configs' - -module Html2rss - module Web - ## - # Merges embedded catalog entries with local feed configs for the public catalog API. - module Catalog - module Merge - STARTER_FEED_IDS = %w[ - microsoft.com/azure-products - phys.org/weekly - softwareleadweekly.com/issues - ].freeze - - module_function - - ## - # @return [Array Object}>] - def call - embedded = Html2rss::Configs::Catalog.entries.map(&:to_h) - local = local_entries - (embedded + local).sort_by { |entry| entry.fetch(:id) } - end - - ## - # @return [Array Object}>] - def starter_entries - entries = call - selected = STARTER_FEED_IDS.filter_map { |id| entries.find { |entry| entry.fetch(:id) == id } } - selected.empty? ? entries.first(3) : selected - end - - ## - # @return [Array Object}>] - def local_entries - LocalConfig.feeds.filter_map do |feed_name, feed_config| - build_local_entry(feed_name, feed_config) - end - end - - ## - # @param feed_name [String, Symbol] - # @param feed_config [Hash] - # @return [Hash{Symbol => Object}, nil] - def build_local_entry(feed_name, feed_config) - directory = feed_config[:directory] || {} - title = directory[:title] - return nil if title.to_s.strip.empty? - - id = feed_name.to_s - channel = feed_config[:channel] || {} - - local_entry(id, directory, title, channel) - end - - ## - # @param id [String] - # @param directory [Hash] - # @param title [String] - # @param channel [Hash] - # @return [Hash{Symbol => Object}] - def local_entry(id, directory, title, channel) - { - id:, - path: "/#{id}.rss", - source: 'local', - directory: local_directory(directory, title), - channel: local_channel(channel, title), - parameters: { schema: {}, defaults: {} } - } - end - - ## - # @param directory [Hash] - # @param title [String] - # @return [Hash{Symbol => Object}] - def local_directory(directory, title) - { - title: title.to_s, - summary: directory[:summary], - topics: Array(directory[:topics]) - }.compact - end - - ## - # @param channel [Hash] - # @param title [String] - # @return [Hash{Symbol => Object}] - def local_channel(channel, title) - { - url: channel.fetch(:url), - language: channel[:language], - title: channel[:title] || title.to_s - }.compact - end - end - end - end -end diff --git a/app/web/cli.rb b/app/web/cli.rb new file mode 100644 index 000000000..798d70191 --- /dev/null +++ b/app/web/cli.rb @@ -0,0 +1,258 @@ +# frozen_string_literal: true + +require 'net/http' +require 'optparse' +require 'uri' + +module Html2rss + module Web + ## + # Unified command-line interface for operator workflows and container management. + module CLI # rubocop:disable Metrics/ModuleLength + STATUS_HEADERS = %w[registry mode version staged_version updated_at sync_url last_error].freeze + + module_function + + ## + # Dispatches command-line arguments to the appropriate handler. + # + # @param argv [Array] command-line arguments + # @param out [IO] standard output stream + # @param err [IO] standard error stream + # @return [Integer] process exit code (0 for success, 1 for failure) + def run(argv = ARGV, out: $stdout, err: $stderr) # rubocop:disable Metrics/MethodLength + case argv.first + when 'registry' then run_registry(argv[1..] || [], out:, err:) + when 'healthcheck' then run_healthcheck(out:, err:) + when 'version', '-v', '--version' then run_version(out:) + when '-h', '--help', 'help', nil + print_root_help(out:) + 0 + else + err.puts "Unknown command: #{argv.first.inspect}. Run with --help for usage." + 1 + end + end + + ## + # @param args [Array] + # @param out [IO] + # @param err [IO] + # @return [Integer] + def run_registry(args, out:, err:) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength + case args.first + when 'status' then registry_status(args[1..] || [], out:) + when 'sync' then registry_sync(args[1..] || []) + when 'promote' then registry_promote(args[1..] || []) + when 'verify' then registry_verify(args[1..] || [], out:, err:) + when 'add' then registry_add(args[1..] || [], out:, err:) + when '-h', '--help', 'help', nil + print_registry_help(out:) + 0 + else + err.puts "Unknown registry command: #{args.first.inspect}. Run with --help for usage." + 1 + end + end + + ## + # @param args [Array] + # @param out [IO] + # @return [Integer] + def registry_status(args, out:) + options = { registry_id: nil } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry status [options]' + opts.on('--registry ID', 'Inspect a single registry ID') { options[:registry_id] = it } + end.parse!(args) + + print_status_table(options[:registry_id], out:) + Registry::Sync.cli_exit_code + end + + ## + # @param args [Array] + # @return [Integer] + def registry_sync(args) + options = { registry_id: nil, dry_run: false } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry sync [options]' + opts.on('--registry ID', 'Sync a single registry ID') { options[:registry_id] = it } + opts.on('--dry-run', 'Fetch and verify without swapping active bundle') { options[:dry_run] = true } + end.parse!(args) + + target_registry_ids(options[:registry_id]).each do |registry_id| + Registry::Sync.run(registry_id:, dry_run: options[:dry_run]) + end + Registry::Sync.cli_exit_code + end + + ## + # @param args [Array] + # @return [Integer] + def registry_promote(args) + options = { registry_id: nil } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry promote [options]' + opts.on('--registry ID', 'Promote a single registry ID') { options[:registry_id] = it } + end.parse!(args) + + target_registry_ids(options[:registry_id]).each do |registry_id| + Registry::Sync.promote_staged!(registry_id:) + end + Registry::Sync.cli_exit_code + end + + ## + # @param args [Array] + # @param out [IO] + # @param err [IO] + # @return [Integer] + def registry_verify(args, out:, err:) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + options = { registry_id: 'official', dir: nil } + OptionParser.new do |opts| + opts.banner = 'Usage: html2rss-web registry verify [options]' + opts.on('--registry ID', 'Registry ID from config (default: official)') { options[:registry_id] = it } + opts.on('--dir PATH', 'Directory containing bundle to verify') { options[:dir] = it } + end.parse!(args) + + registry_id = options[:registry_id] + definition = Registry::Config.entry(registry_id) + raise Registry::Errors::ConfigError, "Unknown registry '#{registry_id}'" unless definition + + dir = options[:dir] || Registry::Store.active_dir(registry_id) || Registry::Store.embedded_dir(registry_id) + raise Registry::Errors::LoadError, "Bundle directory not found: #{dir}" unless dir && File.directory?(dir) + + trust = definition.mode == :path ? :integrity_only : :signed + public_keys = definition.mode == :path ? {} : { definition.public_key_id => definition.public_key }.compact + manifest = Html2rss::Registry::Verifier.verify!(dir, trust:, public_keys:) + out.puts "Verified registry bundle '#{registry_id}' (#{manifest.version}) at #{dir}" + 0 + rescue StandardError => error + err.puts "Registry verification failed: #{error.message}" + 1 + end + + ## + # @param args [Array] + # @param out [IO] + # @param err [IO] + # @return [Integer] + def registry_add(args, out:, err:) # rubocop:disable Metrics/MethodLength + source = args.first + if source.to_s.strip.empty? + err.puts 'Usage: html2rss-web registry add FILE_OR_DIR' + return 1 + end + + result = Registry::LocalBundle.add!(source) + out.puts "Added #{result.ids.join(', ')} to local registry at #{result.path}" + 0 + rescue StandardError => error + err.puts "Registry add failed: #{error.message}" + 1 + end + + ## + # @param out [IO] + # @param err [IO] + # @return [Integer] + def run_healthcheck(out:, err:) # rubocop:disable Metrics/MethodLength + port = ENV.fetch('PORT', 4000) + uri = URI.parse("http://127.0.0.1:#{port}/api/v1/health/live") + response = Net::HTTP.get_response(uri) + if response.is_a?(Net::HTTPSuccess) + out.puts 'OK' + 0 + else + err.puts "Healthcheck failed: HTTP #{response.code}" + 1 + end + rescue StandardError => error + err.puts "Healthcheck failed: #{error.message}" + 1 + end + + ## + # @param out [IO] + # @return [Integer] + def run_version(out:) + out.puts "html2rss-web build=#{RuntimeEnv.build_tag} sha=#{RuntimeEnv.git_sha} ruby=#{RUBY_VERSION}" + 0 + end + + ## + # @param out [IO] + # @return [void] + def print_root_help(out:) # rubocop:disable Metrics/MethodLength + out.puts <<~HELP + Usage: html2rss-web [command] [options] + + Commands: + registry status [--registry ID] Show status of configured registries + registry sync [--registry ID] [--dry-run] Fetch, verify, and sync registry bundles + registry promote [--registry ID] Promote verified staged bundle to active + registry verify [--registry ID] [--dir DIR] Verify registry bundle signature and integrity + registry add FILE_OR_DIR Add feed YAML into the local registry bundle + healthcheck Verify container process liveness + version Print version and runtime information + HELP + end + + ## + # @param out [IO] + # @return [void] + def print_registry_help(out:) + out.puts <<~HELP + Usage: html2rss-web registry [subcommand] [options] + + Subcommands: + status Show registry sync status table + sync Fetch and verify remote registry bundle(s) + promote Promote staged verified bundle to active + verify Verify bundle signature and file integrity against pinned configuration + add Add feed YAML file(s) into the built-in local registry bundle + HELP + end + + ## + # @param registry_id [String, nil] + # @return [Array] + def target_registry_ids(registry_id) + return Array(registry_id) if registry_id + + Registry::Config.precedence + end + + ## + # @param registry_id [String, nil] + # @param out [IO] + # @return [void] + def print_status_table(registry_id, out:) # rubocop:disable Metrics/MethodLength + rows = Registry::Sync.status(registry_id:) + out.puts STATUS_HEADERS.join("\t") + rows.each do |row| + out.puts [ + row.id, + row.mode, + row.version || '-', + row.staged_version || '-', + format_time(row.updated_at), + row.sync_url || '-', + row.last_error || '-' + ].join("\t") + end + end + + ## + # @param value [Time, nil] + # @return [String] + def format_time(value) = value ? value.utc.iso8601 : '-' + + private_class_method :run_registry, :registry_status, :registry_sync, :registry_promote, + :registry_verify, :registry_add, :run_healthcheck, :run_version, + :print_root_help, :print_registry_help, :target_registry_ids, + :print_status_table, :format_time + end + end +end diff --git a/app/web/config/config_snapshot.rb b/app/web/config/config_snapshot.rb deleted file mode 100644 index 0a8fc03c5..000000000 --- a/app/web/config/config_snapshot.rb +++ /dev/null @@ -1,120 +0,0 @@ -# frozen_string_literal: true - -module Html2rss - module Web - ## - # Typed immutable snapshot built from feeds YAML. - # - # This keeps parsing/validation in one place while letting runtime callers - # progressively migrate away from dynamic hash contracts. - module ConfigSnapshot - ## - # Immutable stylesheet entry model. - StylesheetEntry = Data.define(:href, :media, :type) - ## - # Immutable auth account model. - AuthAccount = Data.define(:username, :token, :allowed_urls) - ## - # Immutable feed config boundary model. - FeedConfig = Data.define(:name, :raw) - ## - # Immutable root snapshot model. - Snapshot = Data.define(:global, :feeds, :accounts) - - class << self - # @param yaml_hash [Hash{Symbol=>Object}] - # @return [Snapshot] - def load(yaml_hash) - raise ArgumentError, 'Configuration root must be a hash' unless yaml_hash.is_a?(Hash) - - feeds_hash = normalize_feeds(yaml_hash.fetch(:feeds, {})) - global_hash = yaml_hash.reject { |key| key == :feeds } - accounts = normalize_accounts(global_hash.dig(:auth, :accounts)) - normalized_global = normalized_global_hash(global_hash, accounts) - - Snapshot.new( - global: normalized_global.freeze, - feeds: feeds_hash.freeze, - accounts: accounts.freeze - ) - end - - private - - # @param raw_feeds [Hash, Object] - # @return [Hash{Symbol=>FeedConfig}] - def normalize_feeds(raw_feeds) - return {} unless raw_feeds.is_a?(Hash) - - raw_feeds.each_with_object({}) do |(name, config), memo| - memo[name.to_sym] = FeedConfig.new(name: name.to_sym, raw: deep_dup(config).freeze) - end - end - - # @param raw_accounts [Array, Object] - # @return [Array] - def normalize_accounts(raw_accounts) - Array(raw_accounts).map do |account| - account_hash = account.to_h.transform_keys(&:to_sym) - AuthAccount.new( - username: account_hash.fetch(:username).to_s, - token: account_hash.fetch(:token).to_s, - allowed_urls: Array(account_hash[:allowed_urls]).map(&:to_s).freeze - ) - end - end - - # @param global_hash [Hash{Symbol=>Object}] - # @param accounts [Array] - # @return [Hash{Symbol=>Object}] - def normalized_global_hash(global_hash, accounts) - normalized = deep_dup(global_hash) - return normalized unless normalized.key?(:auth) - - normalized[:auth] = normalized_auth_hash(normalized[:auth], accounts) - normalized - end - - # @param auth_hash [Hash, Object] - # @param accounts [Array] - # @return [Hash{Symbol=>Object}] - def normalized_auth_hash(auth_hash, accounts) - auth = auth_hash.to_h - auth[:accounts] = accounts.map do |account| - { username: account.username, token: account.token, allowed_urls: account.allowed_urls.dup } - end - auth - end - - # @param value [Object] - # @return [Object] - def deep_dup(value) - case value - when Hash - deep_dup_hash(value) - when Array - deep_dup_array(value) - when String - value.dup - else - value - end - end - - # @param value [Hash] - # @return [Hash] - def deep_dup_hash(value) - value.each_with_object({}) do |(key, val), memo| - memo[key.is_a?(String) ? key.dup : key] = deep_dup(val) - end - end - - # @param value [Array] - # @return [Array] - def deep_dup_array(value) - value.map { |element| deep_dup(element) } - end - end - end - end -end diff --git a/app/web/config/local_config.rb b/app/web/config/local_config.rb deleted file mode 100644 index fcf4f98d3..000000000 --- a/app/web/config/local_config.rb +++ /dev/null @@ -1,150 +0,0 @@ -# frozen_string_literal: true - -require 'erb' -require 'yaml' -require_relative 'runtime_env' -begin - require 'html2rss/configs' -rescue LoadError => error - warn "[html2rss-web] Failed to load 'html2rss/configs': #{error.message}" - raise -end - -module Html2rss - module Web - ## - # Loads and normalizes feed configuration from disk. - # - # Keeping lookup/defaulting here gives the rest of the app one predictable - # config shape instead of repeating file parsing and fallback logic. - module LocalConfig - @mutex = Mutex.new - @snapshot = nil - - ## - # raised when the local config wasn't found - class NotFound < RuntimeError; end - ## - # raised when the local config shape is invalid - class InvalidConfig < RuntimeError; end - FEED_EXTENSION_PATTERN = /\.(json|rss|xml)\z/ - EMBEDDED_FEED_NAME_PATTERN = %r{\A[^/]+/.+\z} - - # Path to local feed configuration file. - CONFIG_FILE = 'config/feeds.yml' - - class << self - ## - # @param name [String, Symbol, #to_sym] - # @return [Hash] - def find(name) - normalized_name = normalize_name(name) - config_hash = local_feed_config(normalized_name) || embedded_feed_config(normalized_name) - raise NotFound, "Did not find local feed config at '#{normalized_name}'" unless config_hash - - config_hash - end - - ## - # @return [Hash] - def feeds - snapshot.feeds.transform_values { |feed| deep_dup(feed.raw) } - end - - ## - # @return [Hash] - def global - deep_dup(snapshot.global) - end - - ## - # @return [Html2rss::Web::ConfigSnapshot::Snapshot] - def snapshot - @mutex.synchronize { @snapshot ||= load_snapshot } - rescue KeyError, TypeError, ArgumentError => error - raise InvalidConfig, "Invalid local config: #{error.message}" - end - - ## - # Reparses the current config file without touching memoized runtime - # state. Health checks use this path so config drift shows up without - # forcing live request handlers onto a reload path. - # - # @return [Hash] - def load_yaml - template = File.read(CONFIG_FILE) - YAML.safe_load(ERB.new(template, trim_mode: '-').result, symbolize_names: true).freeze - rescue Errno::ENOENT => error - raise NotFound, "Configuration file not found: #{error.message}" - end - - ## - # Reparses and normalizes the current config file without mutating the - # memoized runtime snapshot. - # - # @return [Html2rss::Web::ConfigSnapshot::Snapshot] - def load_snapshot - ConfigSnapshot.load(load_yaml) - rescue KeyError, TypeError, ArgumentError => error - raise InvalidConfig, "Invalid local config: #{error.message}" - end - - ## - # @param reason [String] - # @return [nil] - def reload!(reason: 'manual') - @mutex.synchronize { @snapshot = nil } - Observability.emit( - event_name: 'cache.lifecycle', - outcome: 'success', - details: { component: 'local_config', event: 'reload', reason: } - ) - nil - end - - private - - # @param normalized_name [String] - # @return [Hash{Symbol=>Object}, nil] - def local_feed_config(normalized_name) - config = snapshot.feeds[normalized_name.to_sym] - return nil unless config - - deep_dup(config.raw) - end - - # @param normalized_name [String] - # @return [Hash{Symbol=>Object}, nil] - def embedded_feed_config(normalized_name) - return nil unless defined?(Html2rss::Configs) - return nil unless normalized_name.match?(EMBEDDED_FEED_NAME_PATTERN) - - deep_dup(Html2rss::Configs.find_by_name(normalized_name)) - rescue Html2rss::Configs::ConfigNotFound - nil - end - - # @param name [String, Symbol, #to_s] - # @return [String] path without feed extension for feed lookup. - def normalize_name(name) - name.to_s.delete_prefix('/').sub(FEED_EXTENSION_PATTERN, '') - end - - # Deep-duplicates nested config structures to avoid mutating shared data. - # - # @param value [Object] - # @return [Object] - def deep_dup(value) - case value - when Hash - value.transform_values { |val| deep_dup(val) } - when Array - value.map { |element| deep_dup(element) } - else - value - end - end - end - end - end -end diff --git a/app/web/config/structured_data.rb b/app/web/config/structured_data.rb new file mode 100644 index 000000000..d829d6d5c --- /dev/null +++ b/app/web/config/structured_data.rb @@ -0,0 +1,29 @@ +# frozen_string_literal: true + +module Html2rss + module Web + ## + # Shared helpers for cloning nested Hash/Array config structures. + module StructuredData + module_function + + ## + # @param value [Object] + # @return [Object] + def deep_dup(value) # rubocop:disable Metrics/MethodLength + case value + when Hash + value.each_with_object({}) do |(key, val), memo| + memo[key.is_a?(String) ? key.dup : key] = deep_dup(val) + end + when Array + value.map { deep_dup(it) } + when String + value.dup + else + value + end + end + end + end +end diff --git a/app/web/feeds/feed_chrome.rb b/app/web/feeds/feed_chrome.rb new file mode 100644 index 000000000..206c9218f --- /dev/null +++ b/app/web/feeds/feed_chrome.rb @@ -0,0 +1,46 @@ +# frozen_string_literal: true + +module Html2rss + module Web + module Feeds + ## + # Instance-wide RSS chrome: stylesheets and outbound User-Agent. + # + # Previously loaded from +config/feeds.yml+ global keys; now owned here so + # boot and token feed generation share one source. + module FeedChrome + STYLESHEETS = [ + { href: '/rss.xsl', media: 'all', type: 'text/xsl' }.freeze + ].freeze + + HEADERS = { + 'User-Agent' => + 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) ' \ + 'Chrome/108.0.0.0 Safari/537.36' + }.freeze + + class << self + ## + # @return [Array String}>] + def stylesheets + STYLESHEETS + end + + ## + # @return [Hash{String => String}] + def headers + HEADERS + end + + ## + # Slice used by token feed generator input. + # + # @return [Hash{Symbol => Object}] + def generator_defaults + { stylesheets:, headers: } + end + end + end + end + end +end diff --git a/app/web/feeds/source_resolver.rb b/app/web/feeds/source_resolver.rb index a90ae65ee..9f35de1ec 100644 --- a/app/web/feeds/source_resolver.rb +++ b/app/web/feeds/source_resolver.rb @@ -12,13 +12,14 @@ class << self # @param feed_request [Html2rss::Web::Feeds::Contracts::Request] # @return [Html2rss::Web::Feeds::Contracts::ResolvedSource] def call(feed_request) - case feed_request.target_kind + target_kind = feed_request.target_kind + case target_kind when :static resolve_static(feed_request) when :token resolve_token(feed_request) else - raise Html2rss::Web::BadRequestError, "Unsupported feed target: #{feed_request.target_kind}" + raise Html2rss::Web::BadRequestError, "Unsupported feed target: #{target_kind}" end end @@ -27,31 +28,31 @@ def call(feed_request) # @param feed_request [Html2rss::Web::Feeds::Contracts::Request] # @return [Html2rss::Web::Feeds::Contracts::ResolvedSource] def resolve_static(feed_request) - config = LocalConfig.find(feed_request.feed_name) - generator_input = static_generator_input(config, feed_request.params) + feed_name = feed_request.feed_name + params = feed_request.params + config = Registry::Index.current.config_for(feed_name) || raise(Html2rss::Web::NotFoundError) + generator_input = static_generator_input(config, params) resolved_source_for( source_kind: :static, - cache_identity: static_cache_identity(feed_request.feed_name, feed_request.params), - generator_input: generator_input, + cache_identity: static_cache_identity(feed_name, params), + generator_input:, ttl_seconds: Cache.seconds_from_minutes(generator_input.dig(:channel, :ttl)) ) - rescue Html2rss::Web::LocalConfig::NotFound - raise Html2rss::Web::NotFoundError end # @param feed_request [Html2rss::Web::Feeds::Contracts::Request] # @return [Html2rss::Web::Feeds::Contracts::ResolvedSource] def resolve_token(feed_request) ensure_auto_source_enabled! - feed_token = authorize_feed_token!(feed_request.token) - strategy = resolved_strategy(feed_token) - generator_input = token_generator_input(feed_token.url, strategy) + token = feed_request.token + feed_token = authorize_feed_token!(token) + generator_input = token_generator_input(feed_token.url, resolved_strategy(feed_token)) resolved_source_for( source_kind: :token, - cache_identity: token_cache_identity(feed_request.token), - generator_input: generator_input, + cache_identity: token_cache_identity(token), + generator_input:, ttl_seconds: Cache.seconds_from_minutes(generator_input.dig(:channel, :ttl), default: 300) ) end @@ -131,7 +132,7 @@ def resolved_strategy(feed_token) # @param strategy [String] # @return [Hash{Symbol=>Object}] def token_generator_input(url, strategy) - LocalConfig.global.slice(:stylesheets, :headers).merge( + FeedChrome.generator_defaults.merge( channel: { url: }, auto_source: {}, strategy: strategy.to_sym diff --git a/app/web/registry/channel_resolver.rb b/app/web/registry/channel_resolver.rb new file mode 100644 index 000000000..93d6b0dc4 --- /dev/null +++ b/app/web/registry/channel_resolver.rb @@ -0,0 +1,52 @@ +# frozen_string_literal: true + +require 'json' + +module Html2rss + module Web + module Registry + ## + # Resolves download URLs for official registry channels and GitHub release tags. + module ChannelResolver + OFFICIAL_GITHUB_RELEASES_API = 'https://api.github.com/repos/html2rss/html2rss-configs/releases/latest' + OFFICIAL_GITHUB_TAG_RELEASES_API = 'https://api.github.com/repos/html2rss/html2rss-configs/releases/tags/%s' + OFFICIAL_ASSET_NAME = 'registry-bundle.tar.gz' + + module_function + + ## + # @param definition [Definition] + # @return [String] download URL + def resolve(definition) + sync_url = definition.sync_url + return sync_url if sync_url && !sync_url.empty? + + if definition.sync_channel == Config::DEFAULT_OFFICIAL_SYNC_CHANNEL + pin = definition.sync_policy.pin_version + return Config::OFFICIAL_RELEASE_URL if pin.to_s.empty? + + api_url = format(OFFICIAL_GITHUB_TAG_RELEASES_API, tag: pin) + return resolve_github_asset(api_url, tag: pin) + end + + raise Errors::SyncError, "Registry '#{definition.id}' has no sync URL" + end + + ## + # @param api_url [String] + # @param tag [String, nil] + # @return [String] + def resolve_github_asset(api_url, tag: nil) + release = JSON.parse(HttpTransport.fetch!(api_url), symbolize_names: true) + url = Array(release[:assets]).find { it[:name] == OFFICIAL_ASSET_NAME }&.dig(:browser_download_url) + return url if url + + tag_suffix = " for #{tag}" if tag + raise Errors::SyncError, "Official asset '#{OFFICIAL_ASSET_NAME}' not found#{tag_suffix}" + rescue JSON::ParserError => error + raise Errors::SyncError, "Invalid GitHub release metadata: #{error.message}" + end + end + end + end +end diff --git a/app/web/registry/config.rb b/app/web/registry/config.rb new file mode 100644 index 000000000..438676e65 --- /dev/null +++ b/app/web/registry/config.rb @@ -0,0 +1,201 @@ +# frozen_string_literal: true + +require 'yaml' +require 'openssl' + +module Html2rss + module Web + module Registry + ## + # Sync policy parsed from registry YAML. + SyncPolicy = Data.define(:pin_version, :max_version, :auto_promote) + + ## + # Registry definition parsed from {Config::REGISTRIES_FILE}. + Definition = Data.define( + :id, + :mode, + :path, + :sync_channel, + :sync_url, + :catalog, + :public_key_id, + :public_key, + :sync_policy, + :allowed_channel_domains + ) do + ## + # @return [Hash{String => OpenSSL::PKey::PKey}] + def public_keys + public_key ? { public_key_id => public_key } : {} + end + end + + ## + # Parses registry configuration and applies zero-config defaults. + module Config # rubocop:disable Metrics/ModuleLength + REGISTRIES_FILE = 'config/registries.yml' + DEFAULT_PRECEDENCE = %w[local official].freeze + DEFAULT_LOCAL_PATH = 'data/registries/local' + DEFAULT_OFFICIAL_SYNC_CHANNEL = 'html2rss-official' + OFFICIAL_RELEASE_URL = 'https://github.com/html2rss/html2rss-configs/releases/latest/download/registry-bundle.tar.gz' + DEFAULT_PUBLIC_KEY_PEM = <<~PEM + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- + PEM + + ## + # Registry configuration document containing precedence order and registry definitions. + Document = Data.define(:precedence, :entries) + + @mutex = Mutex.new + @current = nil + + class << self + ## + # @return [Array] + def precedence + current.precedence + end + + ## + # @param registry_id [String, Symbol] + # @return [Definition] + def entry(registry_id) + current.entries.fetch(registry_id.to_s) do + raise Errors::UnknownRegistry, "Unknown registry '#{registry_id}'" + end + end + + ## + # @param registry_id [String, Symbol] + # @return [Boolean] + def catalog_enabled?(registry_id) + entry(registry_id).catalog + end + + ## + # @return [Document] + def current + @mutex.synchronize { @current ||= parse_document } + end + + ## + # @return [nil] + def reload! + @mutex.synchronize { @current = nil } + nil + end + + private + + ## + # @return [Document] + def parse_document + raw_doc = load_yaml + precedence = parse_precedence(raw_doc[:precedence]) + registries = raw_doc[:registries] || {} + + entries = precedence.to_h do |id| + raw = registries[id.to_sym] || registries[id] || {} + [id, build_definition(id, raw)] + end + + Document.new(precedence:, entries:) + end + + def parse_precedence(raw_precedence) + list = Array(raw_precedence).map(&:to_s).reject(&:empty?) + list.empty? ? DEFAULT_PRECEDENCE : list + end + + ## + # @return [Hash{Symbol => Object}] + def load_yaml + path = ENV.fetch('REGISTRIES_CONFIG', REGISTRIES_FILE) + return default_document unless File.file?(path) + + YAML.safe_load_file(path, symbolize_names: true) || {} + rescue Psych::SyntaxError => error + raise Errors::ConfigError, "Invalid #{path}: #{error.message}" + end + + ## + # @return [Hash{Symbol => Object}] + def default_document # rubocop:disable Metrics/MethodLength + { + precedence: DEFAULT_PRECEDENCE, + registries: { + 'local' => { + path: DEFAULT_LOCAL_PATH, + catalog: false + }, + 'official' => { + sync: { channel: DEFAULT_OFFICIAL_SYNC_CHANNEL }, + catalog: true, + public_key_id: 'html2rss:registry:2026', + public_key: DEFAULT_PUBLIC_KEY_PEM + } + } + } + end + + ## + # @param id [String] + # @param raw [Hash{Symbol => Object}] + # @return [Definition] + def build_definition(id, raw) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + sync_raw = raw[:sync] + sync = sync_raw.is_a?(Hash) ? sync_raw : {} + path = raw[:path]&.to_s + mode = path && !path.empty? ? :path : :sync + public_key = parse_public_key(raw[:public_key]&.to_s) + + definition = Definition.new( + id:, + mode:, + path: mode == :path ? File.expand_path(path, Dir.pwd) : nil, + sync_channel: sync[:channel]&.to_s || DEFAULT_OFFICIAL_SYNC_CHANNEL, + sync_url: sync[:url]&.to_s, + catalog: raw.fetch(:catalog, true), + public_key_id: raw[:public_key_id]&.to_s || 'html2rss:registry:2026', + public_key:, + sync_policy: build_sync_policy(sync, raw), + allowed_channel_domains: Array(raw[:allowed_channel_domains]).map(&:to_s).reject(&:empty?) + ) + + if definition.mode == :sync && !definition.public_key + raise Errors::ConfigError, "Sync registry '#{id}' requires a pinned public_key" + end + + definition + end + + ## + # @param sync [Hash{Symbol => Object}] + # @param raw [Hash{Symbol => Object}] + # @return [SyncPolicy] + def build_sync_policy(sync, raw) + SyncPolicy.new( + pin_version: sync[:pin_version]&.to_s, + max_version: sync[:max_version]&.to_s, + auto_promote: raw[:auto_promote] == true + ) + end + + ## + # @param key_pem [String, nil] + # @return [OpenSSL::PKey::PKey, nil] + def parse_public_key(key_pem) + return nil if key_pem.to_s.strip.empty? + + OpenSSL::PKey.read(key_pem) + rescue OpenSSL::PKey::PKeyError => error + raise Errors::ConfigError, "Invalid registry public_key: #{error.message}" + end + end + end + end + end +end diff --git a/app/web/registry/errors.rb b/app/web/registry/errors.rb new file mode 100644 index 000000000..bb917c753 --- /dev/null +++ b/app/web/registry/errors.rb @@ -0,0 +1,26 @@ +# frozen_string_literal: true + +module Html2rss + module Web + module Registry + ## + # Actionable registry runtime errors for operators and API handlers. + module Errors + # Base error for web registry operations. + class Error < StandardError; end + + # Raised when registry configuration is invalid or incomplete. + class ConfigError < Error; end + + # Raised when a registry bundle cannot be loaded. + class LoadError < Error; end + + # Raised when registry synchronization fails. + class SyncError < Error; end + + # Raised when a registry id is unknown. + class UnknownRegistry < Error; end + end + end + end +end diff --git a/app/web/registry/http_transport.rb b/app/web/registry/http_transport.rb new file mode 100644 index 000000000..392cc8d14 --- /dev/null +++ b/app/web/registry/http_transport.rb @@ -0,0 +1,124 @@ +# frozen_string_literal: true + +require 'net/http' +require 'uri' + +module Html2rss + module Web + module Registry + ## + # HTTPS client for fetching remote registry bundles. + module HttpTransport + DEFAULT_ALLOWED_HOSTS = %w[ + api.github.com + github.com + objects.githubusercontent.com + release-assets.githubusercontent.com + ].freeze + OPEN_TIMEOUT_SECONDS = 10 + READ_TIMEOUT_SECONDS = 60 + MAX_RESPONSE_BYTES = 52_428_800 + DEFAULT_MAX_REDIRECTS = 5 + + module_function + + ## + # @param url [String] + # @param max_redirects [Integer] + # @param allowed_hosts [Array] + # @return [String] response body + def fetch!(url, max_redirects: DEFAULT_MAX_REDIRECTS, allowed_hosts: default_allowed_hosts) # rubocop:disable Metrics/MethodLength + uri = parse_https_uri!(url) + hops = 0 + + loop do + ensure_allowed_host!(uri.host, allowed_hosts) + response = perform_request(uri) + + case response + in Net::HTTPRedirection + hops += 1 + raise Errors::SyncError, 'Registry sync exceeded redirect limit' if hops > max_redirects + + uri = handle_redirect(uri, response['location']) + in Net::HTTPSuccess + return read_body(response) + else + raise Errors::SyncError, "Registry sync fetch failed with HTTP #{response.code}" + end + end + end + + ## + # @param current_uri [URI::HTTPS] + # @param location [String, nil] + # @return [URI::HTTPS] + def handle_redirect(current_uri, location) + raise Errors::SyncError, 'Registry sync redirect missing Location header' if location.to_s.empty? + + parse_https_uri!(URI.join(current_uri, location).to_s) + end + + ## + # @return [Array] + def default_allowed_hosts + extra = ENV.fetch('REGISTRY_SYNC_ALLOWED_HOSTS', '').split(',').map(&:strip).reject(&:empty?) + DEFAULT_ALLOWED_HOSTS + extra + end + + ## + # @param url [String] + # @return [URI::HTTPS] + def parse_https_uri!(url) + uri = URI(url) + unless uri.is_a?(URI::HTTPS) + raise Errors::SyncError, "Registry sync requires HTTPS URLs (got #{uri.scheme.inspect})" + end + + uri + end + + ## + # @param host [String] + # @param allowed_hosts [Array] + # @return [void] + def ensure_allowed_host!(host, allowed_hosts) + return if allowed_hosts.include?(host) + + raise Errors::SyncError, "Registry sync host not allowed: #{host}" + end + + ## + # @param uri [URI::HTTPS] + # @return [Net::HTTPResponse] + def perform_request(uri) # rubocop:disable Metrics/MethodLength + Net::HTTP.start( + uri.host, + uri.port, + use_ssl: true, + open_timeout: OPEN_TIMEOUT_SECONDS, + read_timeout: READ_TIMEOUT_SECONDS + ) do |http| + request = Net::HTTP::Get.new(uri) + request['Accept'] = 'application/octet-stream' + request['User-Agent'] = 'html2rss-web/registry-sync' + http.request(request) + end + end + + ## + # @param response [Net::HTTPResponse] + # @return [String] + def read_body(response) + body = response.body.to_s + if body.bytesize > MAX_RESPONSE_BYTES + raise Errors::SyncError, + "Response exceeds max bytes (#{MAX_RESPONSE_BYTES})" + end + + body + end + end + end + end +end diff --git a/app/web/registry/index.rb b/app/web/registry/index.rb new file mode 100644 index 000000000..8a6ab1f39 --- /dev/null +++ b/app/web/registry/index.rb @@ -0,0 +1,232 @@ +# frozen_string_literal: true + +require 'uri' +require_relative '../config/structured_data' + +module Html2rss + module Web + module Registry + ## + # Unified status model for registry state across API, CLI, and index. + Status = Data.define( + :id, + :mode, + :version, + :staged_version, + :updated_at, + :sync_url, + :last_error + ) do + ## + # @param id [String, Symbol] + # @param mode [Symbol] + # @param version [String, nil] + # @param staged_version [String, nil] + # @param updated_at [Time, nil] + # @param sync_url [String, nil] + # @param last_error [String, nil] + def initialize(id:, mode:, version: nil, staged_version: nil, updated_at: nil, sync_url: nil, last_error: nil) + super + end + end + + ## + # Sole feed repository over registry bundles. + class Index # rubocop:disable Metrics/ClassLength + FEED_EXTENSION_PATTERN = /\.(json|rss|xml)\z/ + + ## + # Immutable container holding a loaded registry bundle's manifest, configs, and catalog entries. + RegistryBundle = Data.define(:registry_id, :manifest, :configs, :catalog_entries) + + @mutex = Mutex.new + @current = nil + + class << self + ## + # @return [Index] + def current + @mutex.synchronize do + if @current.nil? || @current.stale? + @current = new + else + @current + end + end + end + + ## + # @return [nil] + def reload! + @mutex.synchronize { @current = nil } + Config.reload! + nil + end + end + + def initialize + @loaded_bundles = Config.precedence.to_h { [it, load_bundle(it)] }.compact + @manifest_mtimes = current_manifest_mtimes + end + + ## + # @return [Boolean] + def stale? + @manifest_mtimes != current_manifest_mtimes + end + + ## + # @param feed_id [String, Symbol] + # @return [Hash{Symbol => Object}, nil] + def config_for(feed_id) + id = normalize_feed_id(feed_id) + + loaded_bundles.each_value do |bundle| + config = bundle.configs[id] + return StructuredData.deep_dup(config) if config + end + + nil + end + + ## + # @return [Array] + def catalog_entries + build_registry_catalog_entries.values.sort_by(&:id) + end + + ## + # Wire rows for the catalog API (+source+ / +registry+ stamped here). + # + # @return [Array Object}>] + def catalog_rows + build_registry_catalog_rows.values.sort_by { it.fetch(:id) } + end + + ## + # @param registry_id [String, Symbol] + # @return [RegistryBundle, nil] + def bundle_for(registry_id) + loaded_bundles[registry_id.to_s] + end + + ## + # @return [Array] + def status # rubocop:disable Metrics/MethodLength + Config.precedence.map do |registry_id| + definition = Config.entry(registry_id) + bundle = loaded_bundles[registry_id] + sync_state = Store.sync_state(registry_id) + mode = definition.mode + sync_url = mode == :sync ? ChannelResolver.resolve(definition) : nil + + Status.new( + id: registry_id, + mode:, + version: bundle&.manifest&.version, + staged_version: Store.staged_version(registry_id), + updated_at: Store.manifest_mtime(bundle_directory(definition)), + sync_url:, + last_error: sync_state.last_error + ) + end + end + + ## + # @param registry_id [String, Symbol] + # @return [Status, nil] + def status_entry_for(registry_id) + status.find { it.id == registry_id.to_s } + end + + private + + attr_reader :loaded_bundles, :manifest_mtimes + + def current_manifest_mtimes + Config.precedence.to_h do |registry_id| + definition = Config.entry(registry_id) + directory = bundle_directory(definition) + [registry_id, Store.manifest_mtime(directory)] + end + end + + def build_registry_catalog_entries + merge_registry_catalog { |entry, _registry_id| entry } + end + + def build_registry_catalog_rows + merge_registry_catalog do |entry, registry_id| + entry.to_h.merge(source: 'registry', registry: registry_id) + end + end + + def merge_registry_catalog + Config.precedence.each_with_object({}) do |registry_id, rows| + next unless Config.catalog_enabled?(registry_id) + + bundle = loaded_bundles[registry_id] + next unless bundle + + bundle.catalog_entries.each do |entry| + rows[entry.id] ||= yield(entry, registry_id) + end + end + end + + def load_bundle(registry_id) # rubocop:disable Metrics/MethodLength + definition = Config.entry(registry_id) + directory = bundle_directory(definition) + return nil unless directory && File.directory?(directory) && Store.bundle_present_at?(directory) + + trust_opts = if definition.mode == :path + { trust: :integrity_only } + else + { trust: :signed, public_keys: definition.public_keys } + end + bundle_data = Html2rss::Registry::Bundle.load(directory, **trust_opts) + bundle = RegistryBundle.new( + registry_id:, + manifest: bundle_data.manifest, + configs: bundle_data.configs, + catalog_entries: bundle_data.catalog_entries + ) + enforce_scrape_policy!(definition, bundle) + bundle + rescue Html2rss::Registry::Error => error + raise Errors::LoadError, "Failed to load registry '#{registry_id}': #{error.message}" + end + + def enforce_scrape_policy!(definition, bundle) + allowed = definition.allowed_channel_domains + return if allowed.empty? + + bundle.configs.each do |feed_id, config| + next if host_allowed?(config.dig(:channel, :url), allowed) + + raise Errors::LoadError, "Registry '#{definition.id}' config '#{feed_id}' host not allowed" + end + end + + def host_allowed?(url, allowed) + host = URI.parse(url.to_s).host&.downcase rescue nil # rubocop:disable Style/RescueModifier + return false unless host + + allowed.any? { |domain| domain_matches?(host, domain.downcase) } + end + + def domain_matches?(host, domain) + host == domain || host.end_with?(".#{domain}") + end + + def bundle_directory(definition) + definition.mode == :path ? definition.path : Store.active_dir(definition.id) + end + + def normalize_feed_id(feed_id) + feed_id.to_s.delete_prefix('/').sub(FEED_EXTENSION_PATTERN, '') + end + end + end + end +end diff --git a/app/web/registry/local_bundle.rb b/app/web/registry/local_bundle.rb new file mode 100644 index 000000000..ceb468af3 --- /dev/null +++ b/app/web/registry/local_bundle.rb @@ -0,0 +1,190 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +module Html2rss + module Web + module Registry + ## + # Mutates the built-in path-mode +local+ registry bundle. + # + # Validates each feed with +Html2rss::Config::Validator+, upserts by + # +registry.id+, and atomically replaces the on-disk bundle directory. + module LocalBundle # rubocop:disable Metrics/ModuleLength + REGISTRY_ID = 'local' + PUBLIC_KEY_ID = 'local' + CONFIGS_PREFIX = Html2rss::Registry::BundleRelativePath::CONFIGS_PREFIX + + ## + # Outcome of a successful {.add!}. + Result = Data.define(:ids, :path) + + class << self + ## + # Adds one feed YAML file or every +.yml+/+.yaml+ file under a directory + # into the local registry bundle. Directory adds are all-or-nothing. + # + # @param source [String] path to a feed file or directory of feed files + # @return [Result] + def add!(source) + definition = local_definition! + feeds = load_and_validate_sources!(source) + install_bundle!(definition.path, feeds) + Index.reload! + Result.new(ids: feeds.keys.sort, path: definition.path) + end + + private + + ## + # @return [Definition] + def local_definition! + definition = Config.entry(REGISTRY_ID) + unless definition.mode == :path && definition.path + raise Errors::ConfigError, "Registry '#{REGISTRY_ID}' must be path mode" + end + + definition + end + + ## + # @param source [String] + # @return [Hash{String => String}] registry.id => YAML contents + def load_and_validate_sources!(source) + path = File.expand_path(source) + raise Errors::LoadError, "Path not found: #{source}" unless File.exist?(path) + + paths = source_files(path) + raise Errors::LoadError, "No YAML feed files found in #{source}" if paths.empty? + + paths.each_with_object({}) do |file_path, memo| + id, contents = validate_feed_file!(file_path) + raise Errors::LoadError, "Duplicate registry.id '#{id}' in add set" if memo.key?(id) + + memo[id] = contents + end + end + + ## + # @param path [String] + # @return [Array] + def source_files(path) + if File.directory?(path) + Dir[File.join(path, '**', '*.{yml,yaml}')] + else + [path] + end + end + + ## + # @param file_path [String] + # @return [Array<(String, String)>] registry.id and file contents + def validate_feed_file!(file_path) # rubocop:disable Metrics/MethodLength + contents = File.read(file_path) + config = YAML.safe_load(contents, symbolize_names: true) + raise Errors::LoadError, "Invalid YAML in #{file_path}" unless config.is_a?(Hash) + + result = Html2rss::Config::Validator.new.call(config) + unless result.success? + message = result.errors(full: true).map(&:text).join('; ') + raise Errors::LoadError, "Invalid feed config #{file_path}: #{message}" + end + + id = config.dig(:registry, :id).to_s + raise Errors::LoadError, "Missing registry.id in #{file_path}" if id.empty? + + [id, contents] + end + + ## + # @param target [String] + # @param feeds [Hash{String => String}] + # @return [void] + def install_bundle!(target, feeds) + FileUtils.mkdir_p(File.dirname(target)) + staging = sibling_staging_dir(target) + begin + assemble_bundle!(staging, target, feeds) + Html2rss::Registry::Bundle.load(staging, trust: :integrity_only) + Store.replace_directory!(staging, target) + ensure + FileUtils.rm_rf(staging) + end + end + + ## + # @param staging [String] + # @param existing_path [String] + # @param feeds [Hash{String => String}] + # @return [void] + def assemble_bundle!(staging, existing_path, feeds) + merged = existing_feed_contents(existing_path).merge(feeds) + raise Errors::LoadError, 'Local registry bundle would be empty' if merged.empty? + + relative_paths = write_feed_files!(staging, merged) + write_manifest!(staging, relative_paths) + end + + ## + # @param staging [String] + # @param feeds [Hash{String => String}] + # @return [Array] + def write_feed_files!(staging, feeds) + feeds.keys.sort.map do |id| + relative = "#{CONFIGS_PREFIX}#{id}.yml" + absolute = File.join(staging, relative) + FileUtils.mkdir_p(File.dirname(absolute)) + File.write(absolute, feeds.fetch(id)) + relative + end + end + + ## + # @param existing_path [String] + # @return [Hash{String => String}] + def existing_feed_contents(existing_path) + return {} unless Store.bundle_present_at?(existing_path) + + manifest = Html2rss::Registry::Verifier.load_manifest(existing_path) + manifest.files.each_key.with_object({}) do |relative, memo| + absolute = File.join(existing_path, relative) + next unless File.file?(absolute) + + config = YAML.safe_load_file(absolute, symbolize_names: true) + id = config.is_a?(Hash) ? config.dig(:registry, :id).to_s : '' + next if id.empty? + + memo[id] = File.read(absolute) + end + end + + ## + # @param staging [String] + # @param relative_paths [Array] + # @return [void] + def write_manifest!(staging, relative_paths) + file_index = Html2rss::Registry::Manifest.file_index(staging, relative_paths) + manifest = Html2rss::Registry::Manifest.build( + file_index:, + registry_id: REGISTRY_ID, + version: Time.now.utc.strftime('%Y%m%d%H%M%S'), + public_key_id: PUBLIC_KEY_ID + ) + File.write(File.join(staging, Html2rss::Registry::Manifest::MANIFEST_FILE), manifest.to_json) + end + + ## + # Staging directory on the same filesystem as +target+ so +File.rename+ works. + # + # @param target [String] + # @return [String] + def sibling_staging_dir(target) + stamp = Process.clock_gettime(Process::CLOCK_MONOTONIC_RAW, :nanosecond) + File.join(File.dirname(target), ".local-staging-#{Process.pid}-#{stamp}") + end + end + end + end + end +end diff --git a/app/web/registry/store.rb b/app/web/registry/store.rb new file mode 100644 index 000000000..d63c7704d --- /dev/null +++ b/app/web/registry/store.rb @@ -0,0 +1,239 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'json' + +module Html2rss + module Web + module Registry + ## + # Transactional storage engine for registry bundles and sync state. + module Store # rubocop:disable Metrics/ModuleLength + DEFAULT_DATA_ROOT = 'tmp/registry-data' + DEFAULT_EMBEDDED_ROOT = '/app/registries' + SYNC_STATE_FILE = '.sync-state.json' + + ## + # Immutable registry sync state. + SyncState = Data.define(:last_error, :last_sync_at) + + class << self # rubocop:disable Metrics/ClassLength + ## + # @return [String] + def data_root + File.expand_path(ENV.fetch('REGISTRY_DATA_ROOT', DEFAULT_DATA_ROOT)) + end + + ## + # @return [String] + def embedded_root + File.expand_path(ENV.fetch('REGISTRY_EMBEDDED_ROOT', DEFAULT_EMBEDDED_ROOT)) + end + + ## + # @param registry_id [String, Symbol] + # @return [String] + def registry_dir(registry_id) + File.join(data_root, registry_id.to_s) + end + + ## + # @param registry_id [String, Symbol] + # @return [String] + def embedded_dir(registry_id) + File.join(embedded_root, registry_id.to_s) + end + + ## + # Resolves the active bundle directory, preferring synced runtime data + # and falling back to the baked image bundle. + # + # @param registry_id [String, Symbol] + # @return [String, nil] + def active_dir(registry_id) + synced = registry_dir(registry_id) + return synced if bundle_present_at?(synced) + + embedded = embedded_dir(registry_id) + return embedded if bundle_present_at?(embedded) + + nil + end + + ## + # @param registry_id [String, Symbol] + # @return [String] + def staging_dir(registry_id) + File.join(registry_dir(registry_id), '.staging') + end + + ## + # @param registry_id [String, Symbol] + # @return [Boolean] + def bundle_present?(registry_id) + bundle_present_at?(registry_dir(registry_id)) || bundle_present_at?(embedded_dir(registry_id)) + end + + ## + # @param path [String, nil] + # @return [Boolean] + def bundle_present_at?(path) + return false unless path && File.directory?(path) + + File.file?(File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE)) + end + + ## + # @param registry_id [String, Symbol] + # @param staged_dir [String] + # @return [String] + def stage_bundle!(registry_id, staged_dir) + target = staging_dir(registry_id) + FileUtils.mkdir_p(registry_dir(registry_id)) + FileUtils.rm_rf(target) + FileUtils.cp_r(staged_dir, target) + target + end + + ## + # @param registry_id [String, Symbol] + # @return [Boolean] + def staged_present?(registry_id) + bundle_present_at?(staging_dir(registry_id)) + end + + ## + # @param registry_id [String, Symbol] + # @return [String, nil] + def staged_version(registry_id) + manifest_version_at(staging_dir(registry_id)) + end + + ## + # Atomically swaps a directory into the active registry directory. + # + # @param registry_id [String, Symbol] + # @param source_dir [String] + # @return [String] active directory + def swap!(registry_id, source_dir) + target = registry_dir(registry_id) + FileUtils.mkdir_p(File.dirname(target)) + + temp_target = staging_swap_dir(registry_id) + FileUtils.rm_rf(temp_target) + FileUtils.cp_r(source_dir, temp_target) + + replace_directory!(temp_target, target) + target + ensure + FileUtils.rm_rf(temp_target) + end + + ## + # Promotes staging directory to active directory. + # + # @param registry_id [String, Symbol] + # @return [String] active directory + def promote_staged!(registry_id) + staged = staging_dir(registry_id) + raise Errors::LoadError, "No staged bundle for '#{registry_id}'" unless staged_present?(registry_id) + + temp_root = Dir.mktmpdir('registry-promote-') + temp_staged = File.join(temp_root, 'bundle') + FileUtils.mv(staged, temp_staged) + + swap!(registry_id, temp_staged) + registry_dir(registry_id) + ensure + FileUtils.rm_rf(temp_root) if temp_root + end + + ## + # @param registry_id [String, Symbol] + # @return [SyncState] + def sync_state(registry_id) + raw = read_sync_state.fetch(registry_id.to_s, {}) + last_sync = raw['last_sync_at'] + SyncState.new( + last_error: raw['last_error'], + last_sync_at: last_sync ? Time.parse(last_sync) : nil + ) + end + + ## + # @param registry_id [String, Symbol] + # @param last_error [String, nil] + # @param last_sync_at [Time, nil] + # @return [void] + def write_sync_state!(registry_id, last_error:, last_sync_at: Time.now.utc) + state = read_sync_state + state[registry_id.to_s] = { + 'last_error' => last_error, + 'last_sync_at' => last_sync_at&.iso8601 + }.compact + FileUtils.mkdir_p(data_root) + File.write(sync_state_path, JSON.generate(state)) + end + + ## + # @param path [String, nil] + # @return [Time, nil] + def manifest_mtime(path) + return nil unless path && File.directory?(path) + + manifest_file = File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE) + File.file?(manifest_file) ? File.mtime(manifest_file) : nil + end + + ## + # Atomically replaces +target+ with +source+ (backup + rename). + # + # @param source [String] + # @param target [String] + # @return [void] + def replace_directory!(source, target) # rubocop:disable Metrics/MethodLength + FileUtils.mkdir_p(File.dirname(target)) + backup = "#{target}.backup.#{Process.pid}" + FileUtils.mv(target, backup) if File.exist?(target) + begin + File.rename(source, target) + rescue StandardError + FileUtils.mv(backup, target) if File.exist?(backup) && !File.exist?(target) + raise + ensure + FileUtils.rm_rf(backup) + end + end + + private + + def sync_state_path + File.join(data_root, SYNC_STATE_FILE) + end + + def read_sync_state + return {} unless File.file?(sync_state_path) + + JSON.parse(File.read(sync_state_path)) + rescue JSON::ParserError + {} + end + + def manifest_version_at(path) + manifest_file = File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE) + return nil unless File.file?(manifest_file) + + Html2rss::Registry::Manifest.parse(File.read(manifest_file)).version + rescue Html2rss::Registry::ManifestError + nil + end + + def staging_swap_dir(registry_id) + stamp = Process.clock_gettime(Process::CLOCK_MONOTONIC_RAW, :nanosecond) + File.join(data_root, ".tmp-swap-#{registry_id}-#{Process.pid}-#{stamp}") + end + end + end + end + end +end diff --git a/app/web/registry/sync.rb b/app/web/registry/sync.rb new file mode 100644 index 000000000..af16fcb84 --- /dev/null +++ b/app/web/registry/sync.rb @@ -0,0 +1,205 @@ +# frozen_string_literal: true + +require 'fileutils' + +module Html2rss + module Web + module Registry + ## + # Fetches, verifies, and stores signed registry bundles. + module Sync # rubocop:disable Metrics/ModuleLength + @boot_mutex = Mutex.new + @boot_started = false + @timer_started = false + REGISTRY_MUTEXES = Hash.new { |mutexes, registry_id| mutexes[registry_id] = Mutex.new } + + class << self # rubocop:disable Metrics/ClassLength + ## + # @param registry_id [String, Symbol] + # @return [String] + def sync_url_for(registry_id) + definition = Config.entry(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' is path mode" if definition.mode == :path + + ChannelResolver.resolve(definition) + end + + ## + # @param registry_id [String, Symbol] + # @param dry_run [Boolean] + # @return [Status] + def run(registry_id:, dry_run: false) + REGISTRY_MUTEXES[registry_id.to_s].synchronize { run_sync!(registry_id:, dry_run:) } + end + + ## + # @param registry_id [String, Symbol] + # @return [Status] + def promote_staged!(registry_id:) # rubocop:disable Metrics/MethodLength + REGISTRY_MUTEXES[registry_id.to_s].synchronize do + definition = Config.entry(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' is path mode" if definition.mode == :path + unless Store.staged_present?(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' has no staged bundle" + end + + previous_bundle = Index.current.bundle_for(registry_id) + Store.promote_staged!(registry_id) + Index.reload! + report_catalog_change!(registry_id, previous_bundle) + Store.write_sync_state!(registry_id, last_error: nil) + Index.current.status_entry_for(registry_id) + end + end + + ## + # @param registry_id [String, Symbol] + # @return [Array] + def status(registry_id: nil) + rows = Index.current.status + registry_id ? rows.select { it.id == registry_id.to_s } : rows + end + + ## + # Returns process exit code for CLI invocations: 0 on success, 1 on failure. + # + # @return [Integer] + def cli_exit_code + unusable = Index.current.status.any? { it.last_error || (it.mode == :sync && it.version.nil?) } + unusable ? 1 : 0 + end + + ## + # @return [void] + def boot! # rubocop:disable Metrics/MethodLength + @boot_mutex.synchronize do + return if @boot_started + + @boot_started = true + end + return if ENV.fetch('RACK_ENV', 'development') == 'test' + + if ENV.fetch('REGISTRY_SYNC_ON_BOOT', 'false') == 'true' + Config.precedence.each do |id| + definition = Config.entry(id) + next unless definition.mode == :sync + + Thread.new { run(registry_id: id) } # rubocop:disable ThreadSafety/NewThread + end + end + + start_background_timer! + end + + ## + # @return [void] + def start_background_timer! # rubocop:disable Metrics/MethodLength + interval = Integer(ENV.fetch('REGISTRY_SYNC_INTERVAL_HOURS', '24')) + return if interval <= 0 + + @boot_mutex.synchronize do + return if @timer_started + + @timer_started = true + end + + Thread.new do # rubocop:disable ThreadSafety/NewThread + loop do + sleep(interval * 3600) + Config.precedence.each { |id| run(registry_id: id) if Config.entry(id).mode == :sync rescue nil } # rubocop:disable Style/RescueModifier + end + end + end + + private + + def run_sync!(registry_id:, dry_run:) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + definition = Config.entry(registry_id) + raise Errors::SyncError, "Registry '#{registry_id}' is path mode" if definition.mode == :path + + staging_root = Dir.mktmpdir('registry-sync-') + staged_dir = File.join(staging_root, 'bundle') + FileUtils.mkdir_p(staged_dir) + + download_url = ChannelResolver.resolve(definition) + tarball = HttpTransport.fetch!(download_url) + tarball_path = File.join(staging_root, 'download.tar.gz') + File.binwrite(tarball_path, tarball) + + File.open(tarball_path, 'rb') do |io| + Html2rss::Registry::Archive.extract!(io, into: staged_dir) + end + + manifest = Html2rss::Registry::Verifier.verify!(staged_dir, trust: :signed, + public_keys: definition.public_keys) + enforce_max_version!(definition, manifest) + + unless dry_run + if definition.sync_policy.auto_promote + previous = Index.current.bundle_for(registry_id) + Store.swap!(registry_id, staged_dir) + Index.reload! + report_catalog_change!(registry_id, previous) + else + Store.stage_bundle!(registry_id, staged_dir) + end + Store.write_sync_state!(registry_id, last_error: nil) + end + + Index.current.status_entry_for(registry_id) + rescue Html2rss::Registry::Error => error + msg = error.message + log_signature_failure!(registry_id, msg) if msg.match?(/signature|public_key_id/i) + Store.write_sync_state!(registry_id, last_error: msg) unless dry_run + raise Errors::SyncError, msg + rescue StandardError => error + Store.write_sync_state!(registry_id, last_error: error.message) unless dry_run + raise + ensure + FileUtils.rm_rf(staging_root) if staging_root + end + + def log_signature_failure!(registry_id, message) + SecurityLogger.log_registry_signature_failure(registry_id, message) + rescue StandardError + nil + end + + def enforce_max_version!(definition, manifest) + max = definition.sync_policy.max_version + return if max.to_s.empty? + + version = manifest.version + return unless Html2rss::Registry::Manifest.exceeds_max?(version, max) + + raise Errors::SyncError, "Manifest version '#{version}' exceeds max_version '#{max}'" + end + + def report_catalog_change!(registry_id, previous) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity + current = Index.current.bundle_for(registry_id) + return unless current + + current_version = current.manifest.version + prev_ids = previous ? Set.new(previous.catalog_entries.map(&:id)) : Set.new + curr_ids = Set.new(current.catalog_entries.map(&:id)) + added = (curr_ids - prev_ids).to_a.sort + removed = (prev_ids - curr_ids).to_a.sort + return if added.empty? && removed.empty? && previous&.manifest&.version == current_version + + Observability.emit( + event_name: 'registry.catalog_changed', + outcome: 'success', + level: :warn, + details: { + registry_id:, + version: current_version, + added_count: added.size, + removed_count: removed.size + } + ) + end + end + end + end + end +end diff --git a/app/web/security/account_manager.rb b/app/web/security/account_manager.rb index e4ab9dad2..5c9be520a 100644 --- a/app/web/security/account_manager.rb +++ b/app/web/security/account_manager.rb @@ -3,10 +3,10 @@ module Html2rss module Web ## - # Thread-safe account snapshot cache. + # Thread-safe account snapshot cache built from RuntimeEnv tokens. # - # Keeps config reads cheap by materializing one immutable snapshot and - # exposing narrow lookup helpers for auth and authorization flows. + # Admin always exists (full URL access). Health-check exists only when + # +HEALTH_CHECK_TOKEN+ is set to a non-placeholder value. module AccountManager @mutex = Mutex.new @snapshot = nil @@ -54,8 +54,7 @@ def snapshot end def build_snapshot - raw_accounts = LocalConfig.global.dig(:auth, :accounts) - accounts = normalized_accounts(raw_accounts) + accounts = build_accounts token_index = index_accounts(accounts, :token) username_index = index_accounts(accounts, :username) @@ -67,10 +66,33 @@ def build_snapshot { accounts: accounts, token_index: token_index, username_index: username_index }.freeze end - # @param raw_accounts [Array, nil] # @return [ArrayObject}>] - def normalized_accounts(raw_accounts) - Array(raw_accounts).map { |account| account.transform_keys(&:to_sym).freeze }.freeze + def build_accounts + accounts = [admin_account] + health = health_check_account + accounts << health if health + accounts.freeze + end + + # @return [Hash{Symbol=>Object}] + def admin_account + { + username: 'admin', + token: RuntimeEnv.admin_access_token, + allowed_urls: ['*'].freeze + }.freeze + end + + # @return [Hash{Symbol=>Object}, nil] + def health_check_account + token = RuntimeEnv.health_check_token + return nil if token == RuntimeEnv::HEALTH_CHECK_TOKEN_PLACEHOLDER + + { + username: 'health-check', + token:, + allowed_urls: [].freeze + }.freeze end # @param accounts [ArrayObject}>] diff --git a/app/web/security/security_logger.rb b/app/web/security/security_logger.rb index 1349e8155..894345e81 100644 --- a/app/web/security/security_logger.rb +++ b/app/web/security/security_logger.rb @@ -69,6 +69,20 @@ def log_blocked_request(ip, reason, endpoint) log_event('blocked_request', { ip:, reason:, endpoint: }, severity: :warn) end + # @param registry_id [String] + # @param reason [String] + # @return [void] + def log_registry_signature_failure(registry_id, reason) + log_event('registry_signature_failure', { registry_id:, reason: }, severity: :warn) + end + + # @param registry_id [String] + # @param details [Hash{Symbol => Object}] + # @return [void] + def log_registry_catalog_changed(registry_id, details) + log_event('registry_catalog_changed', { registry_id:, **details }, severity: :warn) + end + private # @param event_type [String] diff --git a/bin/docker-build b/bin/docker-build index 7b4e5f9d7..b88cbc487 100755 --- a/bin/docker-build +++ b/bin/docker-build @@ -2,3 +2,4 @@ set -eux docker build --no-cache -t html2rss/web -f Dockerfile . + diff --git a/bin/docker-healthcheck b/bin/docker-healthcheck deleted file mode 100755 index 617c99ceb..000000000 --- a/bin/docker-healthcheck +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -require 'uri' -require 'net/http' - -port = ENV.fetch('PORT', '4000') -token = ENV.fetch('HEALTH_CHECK_TOKEN', nil) -token = 'CHANGE_ME_HEALTH_CHECK_TOKEN' if token.nil? || token.empty? - -uri = URI("http://localhost:#{port}/api/v1/health") -request = Net::HTTP::Get.new(uri) -request['Authorization'] = "Bearer #{token}" - -begin - response = Net::HTTP.start(uri.hostname, uri.port) { |http| http.request(request) } - exit(response.is_a?(Net::HTTPSuccess) ? 0 : 1) -rescue StandardError - exit(1) -end diff --git a/bin/html2rss-web b/bin/html2rss-web new file mode 100755 index 000000000..aad85bd4f --- /dev/null +++ b/bin/html2rss-web @@ -0,0 +1,11 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +require 'bundler/setup' + +ENV['RACK_ENV'] ||= 'development' + +require_relative '../app' +require_relative '../app/web/cli' + +exit Html2rss::Web::CLI.run(ARGV) if $PROGRAM_NAME == __FILE__ diff --git a/bin/prepare-registry-seed b/bin/prepare-registry-seed new file mode 100755 index 000000000..acbd8b74b --- /dev/null +++ b/bin/prepare-registry-seed @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# frozen_string_literal: true + +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)" +CONFIGS_REPO="${HTML2RSS_CONFIGS_ROOT:-$ROOT_DIR/../html2rss-configs}" +OUTPUT_TAR="${1:-$ROOT_DIR/dist/registry-bundle.tar.gz}" + +if [[ ! -d "$CONFIGS_REPO/configs" ]]; then + echo "Missing configs tree at $CONFIGS_REPO/configs" >&2 + exit 1 +fi + +mkdir -p "$(dirname "$OUTPUT_TAR")" +echo "Building registry bundle from $CONFIGS_REPO into $OUTPUT_TAR" +( + cd "$CONFIGS_REPO" + BUNDLE_GEMFILE=tool/Gemfile bundle exec ruby tool/registry-build --output "$OUTPUT_TAR" +) + +echo "Registry bundle ready at $OUTPUT_TAR" + diff --git a/config/feeds.yml b/config/feeds.yml deleted file mode 100644 index d9639e67a..000000000 --- a/config/feeds.yml +++ /dev/null @@ -1,39 +0,0 @@ -auth: - accounts: - - username: "admin" - token: "<%= Html2rss::Web::RuntimeEnv.admin_access_token %>" - allowed_urls: - - "*" # Full access - - username: "demo" - token: "CHANGE_ME_DEMO_TOKEN" - allowed_urls: - - "https://www.chip.de/testberichte" - - "https://news.ycombinator.com" - - "https://github.com/trending" - - username: "health-check" - token: "<%= Html2rss::Web::RuntimeEnv.health_check_token %>" - allowed_urls: [] # Health check doesn't need URL access - -stylesheets: - - href: "/rss.xsl" - media: "all" - type: "text/xsl" - -headers: - "User-Agent": "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" - -feeds: - # your custom feeds go here: - example: - channel: - url: https://github.com/html2rss/html2rss-web/commits/main - title: Latest commits on html2rss-web's main branch. - ttl: 120 - selectors: - items: - selector: "li > div" - title: - selector: "h4" - url: - selector: "a" - extractor: "href" diff --git a/config/registries.yml b/config/registries.yml new file mode 100644 index 000000000..271531127 --- /dev/null +++ b/config/registries.yml @@ -0,0 +1,25 @@ +precedence: + - local + - official + +registries: + local: + path: data/registries/local + catalog: false + official: + sync: + channel: html2rss-official + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- + # Pin the publisher Ed25519 public key for signed network sync and embedded bundle verification. + # Production: keep auto_promote false and promote manually after review. + # auto_promote: false + # sync: + # pin_version: v2026.08.22 + # max_version: v2026.08.22 + # allowed_channel_domains: + # - anthropic.com diff --git a/docker-compose.yml b/docker-compose.yml index c098fb039..a76c843c6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,22 +24,10 @@ services: RACK_TIMEOUT_SERVICE_TIMEOUT: 55 BOTASAURUS_SCRAPE_TIMEOUT_SECONDS: 45 BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS: 30 + REGISTRY_DATA_ROOT: /app/data/registries BOTASAURUS_SCRAPER_URL: http://botasaurus:4010 - # Trial runs use the image's bundled config/feeds.yml. - # Uncomment the block below when you want to replace it with your own file. - # volumes: - # - type: bind - # source: ./config/feeds.yml - # target: /app/config/feeds.yml - # read_only: true - - watchtower: - image: containrrr/watchtower - restart: unless-stopped volumes: - - /var/run/docker.sock:/var/run/docker.sock - - "${HOME}/.docker/config.json:/config.json" - command: --cleanup --interval 7200 + - registry-data:/app/data/registries botasaurus: image: html2rss/botasaurus-scrape-api:latest @@ -53,3 +41,19 @@ services: SCRAPE_WORK_TIMEOUT_SECONDS: 30 ports: - "127.0.0.1:4010:4010" + + registry-sync: + image: html2rss/web + restart: unless-stopped + command: ["sh", "-c", "while true; do html2rss-web registry sync; sleep $${REGISTRY_SYNC_INTERVAL_SECONDS:-86400}; done"] + env_file: + - path: .env + required: false + environment: + REGISTRY_DATA_ROOT: /app/data/registries + REGISTRY_SYNC_INTERVAL_SECONDS: ${REGISTRY_SYNC_INTERVAL_SECONDS:-86400} + volumes: + - registry-data:/app/data/registries + +volumes: + registry-data: diff --git a/docs/README.md b/docs/README.md index 480eb526b..7320f8567 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,7 +26,7 @@ Welcome! This is the canonical source of truth for contributing to `html2rss-web - **Runtime behavior**: Application code plus tests. - **HTTP contract**: Request specs plus generated OpenAPI. -- **Config catalog API**: `GET /api/v1/configs` — embedded data from `Html2rss::Configs::Catalog`, merged with local `feeds.yml` entries that include `directory.title`. Disabled when `CONFIG_CATALOG_ENABLED=false` (`404`, `catalog_disabled`). CORS is enabled on this route only. +- **Config catalog API**: `GET /api/v1/configs` — catalog rows from `Registry::Index` (verified registry bundles per `config/registries.yml`; the built-in `local` path bundle is omitted while `catalog: false`). Disabled when `CONFIG_CATALOG_ENABLED=false` (`404`, `catalog_disabled`). CORS is enabled on this route only. - **This file**: Contributor conventions and current project rules. --- @@ -167,12 +167,15 @@ Search these pages for examples, plugins, and configuration options: ## Architectural Constraints - **No Persistence**: Do not add databases, ORMs, or background job systems. -- **Backend Style**: +- **Backend Style** (Ruby **4.0+** only — see [AGENTS.md](../AGENTS.md#ruby-4-style)): - Keep the main `app.rb` thin; organize routes in `Html2rss::Web::Routes::*`. - For helpers, use `class << self` and `private` methods. Avoid `module_function`. - Use YARD doc comments for all public methods in `app/`. - Add `# frozen_string_literal: true` to all Ruby files. - Do not use `send(...)` to reach into private APIs; expose what is needed at the module level. + - Prefer leading `&&` / `||` at line start for wrapped conditions; `it` in single-parameter blocks; pattern matching over deep `if/elsif` chains. + - Prefer `Data.define`, `filter_map`, `index_by`, `then`, `match?`, and core `Set` (no `require 'set'`) over OpenStruct, verbose `map`/`compact`, nested `if`, `=~`, and array membership on growing collections. + - Dedupe helpers before extracting new files; use `Set` and memoization on hot paths; table-drive specs with `:aggregate_failures` for multi-assert outcomes. - **Frontend Style**: - Follow visual and CSS rules in [design-system.md](design-system.md). - Use Preact components in `frontend/src/`. @@ -284,6 +287,107 @@ Tune alert thresholds from sustained `request.error` or `feed.render` failure sp --- +## Registry sync runbook + +For end-to-end release and deployment steps (maintainers and operators), see [registry-go-live.md](registry-go-live.md). + +Signed feed registries replace the embedded `html2rss-configs` gem. Each registry is defined in `config/registries.yml` (override path with `REGISTRIES_CONFIG`). + +### Check sync status + +Inside the Dev Container or Docker container: + +```bash +bin/html2rss-web registry status +``` + +Columns: `registry`, `mode`, `version`, `staged_version`, `updated_at`, `sync_url`, `last_error`. Exit code is non-zero when any sync-mode registry lacks a usable on-disk bundle. + +Sync, dry-run, or promote a staged bundle: + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry sync --registry official --dry-run +bin/html2rss-web registry promote --registry official +``` + +Production recommendation: keep `auto_promote: false` (default), pin `sync.pin_version` to the approved configs tag, run sync to stage a verified bundle, then promote manually after review. Use `sync.max_version` as an incident freeze cap. + +In Docker Compose, the dedicated `registry-sync` service runs periodic background updates cleanly outside Puma/Ruby. + +Optional hardening: `allowed_channel_domains` suffix-matches every registry config `channel.url` host at bundle load time. + +### Boot behavior + +`Registry::Sync.boot!` runs during app boot (see `app/web/boot/setup.rb`): + +1. **Instant index** — loads the embedded official bundle (140+ feeds) directly from `/app/registries/official` (or `REGISTRY_DATA_ROOT/official` if an updated synced bundle exists). Zero copying, zero startup network traffic. +2. **Sync on boot** — when `REGISTRY_SYNC_ON_BOOT=true`, triggers an asynchronous background sync. +3. **Background refresh** — in standalone single-process mode, `REGISTRY_SYNC_INTERVAL_HOURS` (default `24`) re-syncs on a periodic timer. Set to `0` when using the `registry-sync` Compose service or in static/offline environments. + +Network sync and embedded bundle verification verify Ed25519 signatures using the `public_key` pinned in `registries.yml`. Local `path:` mounts use integrity-only verification. + +### Add a corporate registry + +```yaml +precedence: + - official + - corp + +registries: + official: + sync: + channel: html2rss-official + pin_version: v2026.08.22 # optional + max_version: v2026.08.22 # optional incident freeze + auto_promote: false # default; verified bundles stage until --promote + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- + + corp: + sync: + url: https://registry.example.com/registry-bundle.tar.gz + catalog: false # feeds served; omitted from GET /api/v1/configs + public_key_id: corp:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- +``` + +- **`precedence`** — merge order for feed lookup; first match wins. +- **`sync.url`** — direct tarball URL, or use `sync.channel: html2rss-official` for the default GitHub release asset. +- **`sync.pin_version` / `sync.max_version`** — fetch a specific tag or reject manifests newer than the cap. +- **`auto_promote: false`** — default; verified bundles land in `REGISTRY_DATA_ROOT//.staging/` until `bin/html2rss-web registry promote`. +- **`allowed_channel_domains`** — optional suffix allowlist enforced when the bundle loads. +- **`catalog: false`** — private registry: configs are served at `/{registry.id}.rss` but excluded from the public catalog API (privacy for internal feeds). +- Restrict outbound hosts with `REGISTRY_SYNC_ALLOWED_HOSTS` (comma-separated hostnames). + +### Air-gapped / offline path mount + +For environments without outbound network access, mount a verified bundle directory: + +```yaml +registries: + official: + path: /opt/html2rss/registry/official + catalog: true +``` + +The directory must contain `manifest.json`, optional `manifest.sig`, and `configs/`. Path mode skips network sync; run `bin/html2rss-web registry status` to confirm `mode` is `path`. + +### Key rotation + +1. Publish a new bundle signed with the new key (`public_key_id` in `manifest.json`). +2. Update `public_key_id` and `public_key` in `registries.yml` on every instance **before** or together with the first release that requires the new key. +3. Re-sync (`bin/html2rss-web registry sync`) and promote when `auto_promote: false` (`bin/html2rss-web registry promote`). Failed verification leaves the previous bundle active and records `last_error`. + +--- + ## Documentation Policy - Prefer deleting stale docs over archiving them in-place. diff --git a/docs/architecture.md b/docs/architecture.md index 19530d7c9..ecc5d03ba 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,9 +22,9 @@ flowchart TD Requests enter via `app.rb` and are dispatched to `app/web/routes/`. - **Static feed pages (`/`)**: Routed by `app/web/routes/feed_pages.rb` and resolved as `target_kind: :static`. - - Source: static config in `config/feeds.yml` (via `LocalConfig.find`). + - Source: registry bundle configs via `Registry::Index.config_for` (official + built-in `local`). - Auth boundary: no feed token required on this route. - - Failure mode: unknown feed names fail at static config lookup. + - Failure mode: unknown feed names fail at index lookup. - **Token-backed feed reads (`/api/v1/feeds/:token`)**: Routed by `app/web/routes/api_v1/feed_routes.rb` and resolved as `target_kind: :token`. - Token scope: `FeedAccess.authorize_feed_token!` validates signature/expiry and re-checks account URL access. - Constraint: disabled when AutoSource is off (`ForbiddenError` from `SourceResolver.ensure_auto_source_enabled!`). @@ -34,8 +34,8 @@ Requests enter via `app.rb` and are dispatched to `app/web/routes/`. The `Html2rss::Web::Feeds::SourceResolver` determines where feed configuration comes from based on route target: -- **Static (`target_kind: :static`)**: Pre-defined in `config/feeds.yml`. -- **Token (`target_kind: :token`)**: Generated from validated feed token payload + AutoSource globals. +- **Static (`target_kind: :static`)**: Pre-defined in a registry bundle (add custom feeds with `html2rss-web registry add`). +- **Token (`target_kind: :token`)**: Generated from validated feed token payload + `FeedChrome` defaults. ### 3. Fetching & Rendering @@ -58,3 +58,23 @@ Strategies are defined by the `html2rss` gem but can be configured here. - **Botasaurus**: Used for JavaScript-heavy websites or anti-bot protected pages (`BOTASAURUS_SCRAPER_URL`). To add or configure strategies, see `app/web/feeds/source_resolver.rb` and the `html2rss` gem documentation. + +## Feed Registries & Storage Architecture + +Feed configs are dynamically resolved from curated registry bundles (including the built-in `local` path bundle) via `Registry::Index`: + +```mermaid +flowchart TD + req["Request (/{feed_id}.rss or /api/v1/configs)"] --> index["Registry::Index.current"] + index --> store["Registry::Store.active_dir(id)"] + store --> synced{"Synced Bundle in data_root?"} + synced -- Yes --> data["/app/data/registries/"] + synced -- No --> embedded["/app/registries/ (Baked Image Bundle)"] + data --> bundle["Html2rss::Registry::Bundle.load"] + embedded --> bundle + bundle --> config["Feed Config"] +``` + +- **Baked Embedded Bundles:** Immutable image layers at `/app/registries/official` provide offline, instant-boot feeds without startup copying. +- **Runtime Synced Bundles:** Periodic synchronization writes verified bundles to `/app/data/registries/`, dynamically taking precedence over the baked image bundle. + diff --git a/docs/registry-go-live.md b/docs/registry-go-live.md new file mode 100644 index 000000000..cfa7dd835 --- /dev/null +++ b/docs/registry-go-live.md @@ -0,0 +1,505 @@ +# Registry go-live manual + +Step-by-step guide to ship and operate signed `registry.v1` bundles across the html2rss org repos. + +**Related docs** + +- Bundle format: [`html2rss/lib/html2rss/registry/README.md`](../../html2rss/lib/html2rss/registry/README.md) +- Operator runbook (sync flags, custom registries): [README.md — Registry sync runbook](./README.md#registry-sync-runbook) + +--- + +## 1. Maintainers — merge and release order + +Work in this order when a change touches registry contracts or feed configs. + +### 1.1 `html2rss` (core gem) + +Source of truth for `registry.v1` schema, `Html2rss::Registry::Verifier`, `CatalogBuilder`, and archive limits. + +1. Merge registry-related changes to `main`. +2. Regenerate schema if config schema changed: + ```bash + cd html2rss + mise exec -- make ready + ``` +3. Tag/release the gem when the contract is stable (web depends on this). + +### 1.2 `html2rss-configs` (signed bundle publisher) + +Source of truth for feed YAML and signed release artifacts. + +1. Merge config changes to `master`. +2. Run the configs quality gate: + ```bash + cd html2rss-configs + make ready + ``` +3. Tag the release (tag name becomes `manifest.json` `version` in CI via `REGISTRY_VERSION: ${{ github.ref_name }}`): + ```bash + git tag v2026.08.22 # example; use your release tag + git push origin v2026.08.22 + ``` +4. Tag push triggers [`.github/workflows/release.yml`](../../html2rss-configs/.github/workflows/release.yml): + - `make registry-build -- --sign` + - uploads `dist/registry-bundle.tar.gz` to a **draft** GitHub Release (human publishes after review) + +**Draft → publish:** CI creates a draft release. Maintainers review the asset, release notes, and tag diff, then click **Publish release** in GitHub. Draft releases are invisible to `/releases/latest` until published. + +### 1.3 `html2rss-web` (runtime + Docker image) + +Consumes verified bundles; bakes the official release bundle into the Docker image at build time. + +1. Bump the `html2rss` gem dependency if the core contract changed. +2. Update `config/registries.yml` when the signing key or sync channel changes (see sections 2 and 5). +3. Build the image with the official release artifact baked (section 3). +4. Merge to `main`, wait for CI (`ci` workflow) to pass. +5. Publish a GitHub Release on `html2rss-web` — [`.github/workflows/release.yml`](../../html2rss-web/.github/workflows/release.yml) builds and pushes `html2rss/web` tags (`latest`, semver, major, commit SHA). + +**Rule of thumb:** core contract → configs signed release → web image that embeds/verifies that release. + +--- + +## 2. First signed release — signing key, tag, verify artifact + +### 2.1 Generate an Ed25519 key pair + +Use OpenSSL (same algorithm as `tool/registry-build` and `Html2rss::Registry::Verifier`): + +```bash +openssl genpkey -algorithm ED25519 -out registry-signing.pem +openssl pkey -in registry-signing.pem -pubout -out registry-signing.pub +``` + +Keep the private key offline except where signing happens. + +### 2.2 Configure `html2rss-configs` CI secrets + +In the `html2rss-configs` GitHub repo, add secrets on the **`registry-release`** environment: + +| Secret | Purpose | +| --- | --- | +| `REGISTRY_SIGNING_KEY` | Full PEM contents of `registry-signing.pem` (private key; used by `make registry-build -- --sign`) | +| `REGISTRY_PUBLIC_KEY_PEM` | Full PEM contents of `registry-signing.pub` (public key; used by the release verify step — not a repo fixture) | + +Release CI reads them here: + +```yaml +env: + REGISTRY_VERSION: ${{ github.ref_name }} + REGISTRY_SIGNING_KEY: ${{ secrets.REGISTRY_SIGNING_KEY }} +run: make registry-build -- --sign +``` + +The verify step loads the public key from `REGISTRY_PUBLIC_KEY_PEM` in-process (`OpenSSL::PKey.read(ENV.fetch('REGISTRY_PUBLIC_KEY_PEM'))`); no committed PEM file in the configs repo. + +At go-live, the same public key must appear in `html2rss-web/config/registries.yml` (`public_key` / `public_key_id`) so runtime sync verification matches CI. + +### 2.3 Pin the public key in `html2rss-web` + +Add the public key to `config/registries.yml` before instances need to sync signed bundles over the network: + +```yaml +registries: + official: + sync: + channel: html2rss-official + pin_version: v2026.08.22 # optional: fetch this tag only + max_version: v2026.08.22 # optional: reject newer manifests (incident freeze) + auto_promote: false # default false; verified bundles stage until manual promote + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- + allowed_channel_domains: # optional suffix allowlist for channel.url domains + - anthropic.com + - github.com +``` + +- `public_key_id` must match the value in signed `manifest.json` (default in `tool/registry-build`: `html2rss:registry:2026`). +- Network sync and embedded bundle loading use `:signed` trust and require this pin. Local `path:` mounts use `:integrity_only` trust (no signature check on load). +- **`auto_promote: false`** (default) writes verified bundles to `REGISTRY_DATA_ROOT//.staging/` without changing the active catalog. Promote after review with `bin/html2rss-web registry promote --registry official`. +- **`pin_version`** resolves the GitHub tag release API instead of `/releases/latest`. +- **`max_version`** rejects sync when the verified manifest version is newer than the cap (incident freeze). +- **`allowed_channel_domains`** rejects bundle load when any config `channel.url` host is outside the suffix allowlist. + +### 2.4 Tag and publish the configs release + +```bash +cd html2rss-configs +git tag # e.g. 2026.08.22 +git push origin +``` + +Wait for the Release workflow to finish. The asset name is always **`registry-bundle.tar.gz`**. + +### 2.5 Verify the release artifact + +Download the asset from the GitHub Release, then verify locally: + +```bash +mkdir -p /tmp/registry-verify && tar -xzf registry-bundle.tar.gz -C /tmp/registry-verify + +# Inspect manifest +jq . /tmp/registry-verify/manifest.json +# Confirm: format=registry.v1, registry_id=official, version=, public_key_id matches pin + +# Confirm signature file exists +test -f /tmp/registry-verify/manifest.sig + +# Optional: verify with core gem (from html2rss checkout) +cd html2rss +mise exec -- bundle exec ruby -rhtml2rss -ropenssl -e " + pk = OpenSSL::PKey.read(File.read('path/to/registry-signing.pub')) + Html2rss::Registry::Verifier.verify!( + '/tmp/registry-verify', + trust: :signed, + public_keys: { 'html2rss:registry:2026' => pk } + ) + puts 'OK' +" +``` + +Unsigned local builds (no `--sign`) are valid for integrity-only use only: + +```bash +cd html2rss-configs +make registry-build # writes dist/registry-bundle.tar.gz without manifest.sig +``` + +--- + +## 3. Web image build — artifact baking and Docker + +### 3.1 How the official registry gets into the image + +| Step | What happens | +| --- | --- | +| `Dockerfile` (`registry-builder`) | Downloads the official release `registry-bundle.tar.gz`, extracts it, and verifies its Ed25519 signature & file digests against `config/registries.yml` | +| `Dockerfile` (Runtime) | `COPY --from=registry-builder /build/official /app/registries/official` | +| Boot | `Registry::Index.current` loads `/app/registries/official` directly. Zero copying, zero network calls. | + +### 3.2 Build locally + +To build a production image fetching the latest official release: + +```bash +cd html2rss-web +docker build -t html2rss/web -f Dockerfile . +``` + +To build pinning a specific configs release tag: + +```bash +docker build \ + --build-arg REGISTRY_RELEASE_TAG=v2026.08.22 \ + --build-arg REGISTRY_BUNDLE_URL=https://github.com/html2rss/html2rss-configs/releases/download/v2026.08.22/registry-bundle.tar.gz \ + --build-arg BUILD_TAG=1.2.3 \ + --build-arg GIT_SHA="$(git rev-parse HEAD)" \ + -t html2rss/web \ + -f Dockerfile . +``` + +### 3.3 What operators get in the image + +- **`/app/registries/official/`** — Pre-verified official release bundle baked at build time (immutable image layer). +- **`/app/config/registries.yml`** — Default official registry configuration with pinned public key. +- **`/app/data/registries`** — Empty at build; used at runtime for network synchronization. + +--- + +## 4. Operators — default path (official registry) + +### 4.1 Pull a new image (simplest update path) + +`docker-compose.yml` defaults: + +- Image: `html2rss/web` +- Volume: `registry-data:/app/data/registries` (`REGISTRY_DATA_ROOT`) + +```bash +docker compose pull html2rss-web +docker compose up -d html2rss-web +``` + +A new image updates the **embedded bundle** inside the container layers. + +### 4.2 Zero-config first boot + +With the stock `config/registries.yml`, no extra registry env vars are required. + +On boot (`Registry::Sync.boot!`): + +1. **Instant index** — loads directly from `/app/registries/official` (or `REGISTRY_DATA_ROOT/official` if an updated synced bundle exists). +2. **Sync on boot** — runs only when `REGISTRY_SYNC_ON_BOOT=true`. +3. **Background refresh** — when `REGISTRY_SYNC_INTERVAL_HOURS` > 0 (default **24**), re-syncs on a periodic timer. Set to `0` to disable. + +Official sync URL (from `config/registries.yml` + `Registry::Config`): + +`https://github.com/html2rss/html2rss-configs/releases/latest/download/registry-bundle.tar.gz` + +Allowed outbound hosts (built-in): `api.github.com`, `github.com`, `objects.githubusercontent.com`. + +### 4.3 Existing instance — volume retained + +The named volume preserves runtime-synced bundles across container restarts. When present and valid, `Store.active_dir` prefers the volume copy over the embedded image copy. + +- Old bundle stays active until a successful sync swaps it. +- Use `bin/html2rss-web registry sync` or wait for background refresh to pick up a new configs release without rebuilding the image (section 7). + +--- + +## 5. Operators — custom corporate registry + +Override or extend `config/registries.yml` (or set `REGISTRIES_CONFIG` to an alternate file path). + +Example from the [registry sync runbook](./README.md#add-a-corporate-registry): + +```yaml +precedence: + - official + - corp + +registries: + official: + sync: + channel: html2rss-official + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- + + corp: + sync: + url: https://registry.example.com/registry-bundle.tar.gz + catalog: false + public_key_id: corp:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + ... + -----END PUBLIC KEY----- +``` + +| Field | Purpose | +| --- | --- | +| `precedence` | Feed lookup merge order; first match wins | +| `sync.url` | Direct HTTPS tarball URL for network sync | +| `sync.channel: html2rss-official` | Resolves to the official GitHub release asset | +| `catalog: false` | Feeds served at `/{feed_id}.rss`; **omitted** from `GET /api/v1/configs` | +| `public_key` / `public_key_id` | Required for `:signed` network sync verification | + +For hosts outside the default GitHub allowlist: + +```bash +REGISTRY_SYNC_ALLOWED_HOSTS=registry.example.com,cdn.example.com +``` + +Mount a custom registries file in Compose: + +```yaml +volumes: + - ./config/registries.yml:/app/config/registries.yml:ro +``` + +--- + +## 6. Verify live + +Run these checks after deploy or sync. + +### 6.1 Registry sync status (CLI) + +Inside the running container (or Dev Container): + +```bash +bin/html2rss-web registry status +``` + +Tab-separated columns: `registry`, `mode`, `version`, `staged_version`, `updated_at`, `sync_url`, `last_error`. + +- Exit code **0** — all sync-mode registries have a usable on-disk bundle. +- Exit code **1** — at least one sync registry lacks a bundle (see `Registry::Sync.unusable_sync_registries`). + +Single registry, dry-run, or promote staged bundle: + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry sync --registry official --dry-run +bin/html2rss-web registry promote --registry official +``` + +### 6.2 Instance metadata API + +```bash +curl -sS http://127.0.0.1:4000/api/v1/ | jq '.data.instance' +``` + +Confirm: + +- **`instance.registries`** — array with `id`, `version`, `updated_at`, `sync_mode` per configured registry +- **`instance.catalog`** — `{ "enabled": true, "url": ".../api/v1/configs" }` (unless `CONFIG_CATALOG_ENABLED=false`) + +### 6.3 Catalog API + +```bash +curl -sS http://127.0.0.1:4000/api/v1/configs | jq '.data.configs[0]' +``` + +Expect rows with `source: "registry"`, `registry: "official"`, and a `path` like `/anthropic.com/news.rss`. + +When `CONFIG_CATALOG_ENABLED=false`, expect `404` with `{ "error": "catalog_disabled" }`. + +### 6.4 Sample static feed + +Pick a feed id from the catalog (`id` field) or from `configs/.yml` in the bundle. Request: + +```bash +curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4000/anthropic.com/news.rss +``` + +Expect HTTP **200** and valid RSS XML. + +--- + +## 7. Update configs without pulling a new Docker image + +Configs releases are independent of web image releases. To refresh feeds on a running instance: + +### 7.1 Manual sync and promote + +Production recommendation: keep `auto_promote: false`, set `sync.pin_version` to the approved tag, fetch with sync, then promote manually after review. + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry status # staged_version shows verified bundle +bin/html2rss-web registry promote --registry official +``` + +Fetches the pinned or latest signed tarball, verifies signature + digests, and either stages (`auto_promote: false`) or atomically swaps the active bundle (`auto_promote: true`). + +### 7.2 Incident freeze + +To block uptake of a newer configs release without disabling sync entirely: + +```yaml +registries: + official: + sync: + channel: html2rss-official + max_version: v2026.08.21 + auto_promote: false +``` + +Set `REGISTRY_SYNC_INTERVAL_HOURS=0` to pause background refresh while investigating. + +### 7.3 Automatic refresh via Docker Compose + +In production Compose environments, registry synchronization is handled cleanly outside the Ruby/Puma process lifecycle via the dedicated `registry-sync` Compose service in `docker-compose.yml`: + +```yaml + registry-sync: + image: html2rss/web + restart: unless-stopped + command: ["sh", "-c", "while true; do html2rss-web registry sync; sleep $${REGISTRY_SYNC_INTERVAL_SECONDS:-86400}; done"] + environment: + REGISTRY_DATA_ROOT: /app/data/registries + REGISTRY_SYNC_INTERVAL_SECONDS: 86400 + volumes: + - registry-data:/app/data/registries +``` + +When new bundles are synced to disk, the `html2rss-web` server detects the updated `manifest.json` mtimes automatically on subsequent requests. + +To run an on-demand sync or check status with Docker Compose: + +```bash +docker compose exec html2rss-web html2rss-web registry status +docker compose exec html2rss-web html2rss-web registry sync --registry official +``` + +--- + +## 8. Troubleshooting + +### 8.1 Signature verification failure + +Symptoms in `bin/html2rss-web registry status`: + +- `last_error` contains `Unknown public_key_id`, `Invalid manifest signature`, or `Missing manifest.sig` + +Checks: + +1. `public_key_id` and `public_key` in `registries.yml` match the signed release. +2. Deploy web config **before** or **with** the first bundle signed by a new key (key rotation). +3. Downloaded asset is `registry-bundle.tar.gz` from the expected release (not an unsigned local build). + +Signature-related failures are logged via `SecurityLogger.log_registry_signature_failure`. + +Verify without swapping the active bundle: + +```bash +bin/html2rss-web registry sync --registry official --dry-run +``` + +### 8.2 Sync failure keeps the old bundle + +By design: + +- `Registry::Sync.run` only calls `Store.swap!` after fetch **and** verification succeed. +- On failure, `last_error` is recorded; the previous bundle under `REGISTRY_DATA_ROOT//` remains served. +- `Store.promote_bundle!` rolls back on swap failure. + +If sync fails on first boot with no prior bundle in the volume, the instance serves the embedded official bundle (140+ feeds) directly from the container image. + +### 8.3 Network / host errors + +| Error pattern | Likely cause | +| --- | --- | +| `Registry sync host not allowed` | Add host to `REGISTRY_SYNC_ALLOWED_HOSTS` | +| `Registry sync rejects HTTP redirects` | Publish a direct HTTPS asset URL (`sync.url`) | +| `Registry sync fetch failed with HTTP …` | Release missing, URL wrong, or GitHub outage | +| `Registry sync requires HTTPS URLs` | Use `https://` in `sync.url` | + +Default allowed hosts cover official GitHub release downloads (`api.github.com`, `github.com`, `objects.githubusercontent.com`, `release-assets.githubusercontent.com`). + +### 8.4 Air-gapped / offline (`path` mode) + +For environments without outbound sync, mount a verified bundle directory and skip network sync: + +```yaml +registries: + official: + path: /opt/html2rss/registry/official + catalog: true +``` + +Requirements: + +- Directory contains `manifest.json`, `configs/`, and optionally `manifest.sig` +- `bin/html2rss-web registry sync` is not applicable (`path mode; sync is not applicable`) +- `bin/html2rss-web registry status` should show `mode: path` + +Load path uses `:integrity_only` trust (disk/image trust boundary). + +In Docker Compose, bind-mount the bundle and optionally set `REGISTRIES_CONFIG`: + +```yaml +volumes: + - /opt/html2rss/registry/official:/opt/html2rss/registry/official:ro +environment: + REGISTRIES_CONFIG: /app/config/registries.yml +``` + +### 8.5 CLI exit code non-zero with empty `last_error` + +`bin/html2rss-web registry status` exits **1** when a sync-mode registry has **no usable bundle** (neither in the volume nor embedded in the image). Run a sync or verify the embedded bundle is present: + +```bash +bin/html2rss-web registry sync --registry official +bin/html2rss-web registry status +``` + diff --git a/frontend/src/api/generated/types.gen.ts b/frontend/src/api/generated/types.gen.ts index 2939a0d7b..8a300e216 100644 --- a/frontend/src/api/generated/types.gen.ts +++ b/frontend/src/api/generated/types.gen.ts @@ -13,7 +13,7 @@ export type GetApiMetadataData = { export type GetApiMetadataResponses = { /** - * returns catalog pointer metadata + * returns registry status metadata */ 200: { data: { @@ -31,6 +31,12 @@ export type GetApiMetadataResponses = { access_token_required: boolean; enabled: boolean; }; + registries: Array<{ + id: string; + sync_mode: string; + updated_at?: string; + version?: string; + }>; }; }; success: boolean; @@ -77,39 +83,14 @@ export type GetConfigCatalogResponses = { id: string; parameters: { defaults: { - blog?: string | null; - id?: string | null; - region?: string | null; - repository?: string | null; - section?: string | null; - user_id?: string | null; - username?: string | null; + [key: string]: unknown; }; schema: { - blog?: { - type: string; - } | null; - id?: { - type: string; - } | null; - region?: { - type: string; - } | null; - repository?: { - type: string; - } | null; - section?: { - type: string; - } | null; - user_id?: { - type: string; - } | null; - username?: { - type: string; - } | null; + [key: string]: unknown; }; }; path: string; + registry?: string; source: string; }>; }; diff --git a/public/openapi.yaml b/public/openapi.yaml index 863713c24..f87837ca3 100644 --- a/public/openapi.yaml +++ b/public/openapi.yaml @@ -64,9 +64,28 @@ paths: - enabled - access_token_required type: object + registries: + items: + properties: + id: + type: string + sync_mode: + type: string + updated_at: + type: string + nullable: true + version: + type: string + nullable: true + required: + - id + - sync_mode + type: object + type: array required: - feed_creation - catalog + - registries type: object required: - api @@ -78,7 +97,7 @@ paths: - success - data type: object - description: returns catalog pointer metadata + description: returns registry status metadata security: - {} summary: API metadata @@ -132,101 +151,10 @@ paths: parameters: properties: defaults: - properties: - blog: - type: - - string - - 'null' - id: - type: - - string - - 'null' - region: - type: - - string - - 'null' - repository: - type: - - string - - 'null' - section: - type: - - string - - 'null' - user_id: - type: - - string - - 'null' - username: - type: - - string - - 'null' + properties: {} type: object schema: - properties: - blog: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - id: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - region: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - repository: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - section: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - user_id: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' - username: - properties: - type: - type: string - required: - - type - type: - - object - - 'null' + properties: {} type: object required: - schema @@ -234,15 +162,17 @@ paths: type: object path: type: string + registry: + type: string source: type: string required: - id - path - - source - directory - channel - parameters + - source type: object type: array required: diff --git a/spec/fixtures/registries/keys/test-key.pem b/spec/fixtures/registries/keys/test-key.pem new file mode 100644 index 000000000..d1583ef45 --- /dev/null +++ b/spec/fixtures/registries/keys/test-key.pem @@ -0,0 +1,3 @@ +-----BEGIN PRIVATE KEY----- +MC4CAQAwBQYDK2VwBCIEIKWA7CdQvmMCa06H6jojjHE30xRV9Ps823kjgqWKvfSE +-----END PRIVATE KEY----- diff --git a/spec/fixtures/registries/keys/test-key.pub b/spec/fixtures/registries/keys/test-key.pub new file mode 100644 index 000000000..236aab386 --- /dev/null +++ b/spec/fixtures/registries/keys/test-key.pub @@ -0,0 +1,3 @@ +-----BEGIN PUBLIC KEY----- +MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= +-----END PUBLIC KEY----- diff --git a/spec/fixtures/registries/official/configs/anthropic.com/news.yml b/spec/fixtures/registries/official/configs/anthropic.com/news.yml new file mode 100644 index 000000000..af1cbc812 --- /dev/null +++ b/spec/fixtures/registries/official/configs/anthropic.com/news.yml @@ -0,0 +1,23 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json +registry: + id: anthropic.com/news +directory: + topics: + - tech + - research + title: "Anthropic — News" + summary: "Product and research announcements from Anthropic." +channel: + title: "Anthropic — News" + url: https://www.anthropic.com/news + language: en + time_zone: UTC + ttl: 360 +selectors: + items: + selector: 'ul[class*="PublicationList"] > li > a[href^="/news/"]' + enhance: false + title: + selector: '[class*="__title"]' + url: + extractor: href diff --git a/spec/fixtures/registries/official/configs/deepmind.google/blog.yml b/spec/fixtures/registries/official/configs/deepmind.google/blog.yml new file mode 100644 index 000000000..6d3733c1e --- /dev/null +++ b/spec/fixtures/registries/official/configs/deepmind.google/blog.yml @@ -0,0 +1,27 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/html2rss/html2rss/refs/heads/master/schema/html2rss-config.schema.json +registry: + id: deepmind.google/blog +directory: + topics: + - science + - research + title: "Google DeepMind — Blog" + summary: "Research and product posts from Google DeepMind." + +channel: + title: "Google DeepMind — Blog" + url: https://deepmind.google/blog/ + language: en + time_zone: UTC + ttl: 360 +selectors: + items: + selector: .card__inner + enhance: false + title: + selector: h3 + url: + selector: .card__overlay-link + extractor: href + published_at: + selector: time diff --git a/spec/fixtures/registries/official/manifest.json b/spec/fixtures/registries/official/manifest.json new file mode 100644 index 000000000..82f32f845 --- /dev/null +++ b/spec/fixtures/registries/official/manifest.json @@ -0,0 +1,11 @@ +{ + "format": "registry.v1", + "registry_id": "official", + "version": "test-fixture", + "public_key_id": "test", + "files": { + "configs/anthropic.com/news.yml": "679b3c28b6dd7e08bbfdaf8564e077e7c2ac4140eacbc9543756591aaf0621f5", + "configs/deepmind.google/blog.yml": "3dbc30f109bb169f1de06372236a2d352584808d11be8243ba93befcd94a7b41" + } +} + diff --git a/spec/fixtures/registries/private/configs/secret.example/private.yml b/spec/fixtures/registries/private/configs/secret.example/private.yml new file mode 100644 index 000000000..14a119e7f --- /dev/null +++ b/spec/fixtures/registries/private/configs/secret.example/private.yml @@ -0,0 +1,17 @@ +registry: + id: secret.example/private +directory: + title: "Private Corp Feed" + summary: "Hidden from catalog." +channel: + title: "Private Corp Feed" + url: https://secret.example/private + ttl: 60 +selectors: + items: + selector: article + title: + selector: h1 + url: + selector: a + extractor: href diff --git a/spec/fixtures/registries/private/manifest.json b/spec/fixtures/registries/private/manifest.json new file mode 100644 index 000000000..6d901dbbf --- /dev/null +++ b/spec/fixtures/registries/private/manifest.json @@ -0,0 +1,9 @@ +{ + "format": "registry.v1", + "registry_id": "private", + "version": "test-fixture", + "public_key_id": "test", + "files": { + "configs/secret.example/private.yml": "f3b4f6d567e44960be89c171a915a5451266998090ca9ffdb394050bcbbd0dd2" + } +} diff --git a/spec/fixtures/registries/registries.yml b/spec/fixtures/registries/registries.yml new file mode 100644 index 000000000..b505e5794 --- /dev/null +++ b/spec/fixtures/registries/registries.yml @@ -0,0 +1,11 @@ +precedence: + - official + - private + +registries: + official: + path: spec/fixtures/registries/official + catalog: true + private: + path: spec/fixtures/registries/private + catalog: false diff --git a/spec/fixtures/registries/sync/bundle/manifest.json b/spec/fixtures/registries/sync/bundle/manifest.json new file mode 100644 index 000000000..8a0bf8a6d --- /dev/null +++ b/spec/fixtures/registries/sync/bundle/manifest.json @@ -0,0 +1,11 @@ +{ + "format": "registry.v1", + "registry_id": "official", + "version": "test-fixture", + "public_key_id": "test-key", + "files": { + "configs/anthropic.com/news.yml": "679b3c28b6dd7e08bbfdaf8564e077e7c2ac4140eacbc9543756591aaf0621f5", + "configs/deepmind.google/blog.yml": "3dbc30f109bb169f1de06372236a2d352584808d11be8243ba93befcd94a7b41" + } +} + diff --git a/spec/fixtures/registries/sync/registries.yml b/spec/fixtures/registries/sync/registries.yml new file mode 100644 index 000000000..4f1ab3dff --- /dev/null +++ b/spec/fixtures/registries/sync/registries.yml @@ -0,0 +1,14 @@ +precedence: + - official + +registries: + official: + sync: + url: https://registry.test.example/registry-bundle.tar.gz + auto_promote: true + catalog: true + public_key_id: test-key + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- diff --git a/spec/html2rss/web/account_manager_spec.rb b/spec/html2rss/web/account_manager_spec.rb index b8252e4fb..a85a3cb6c 100644 --- a/spec/html2rss/web/account_manager_spec.rb +++ b/spec/html2rss/web/account_manager_spec.rb @@ -1,54 +1,95 @@ # frozen_string_literal: true require 'spec_helper' +require 'climate_control' require_relative '../../../app/web/security/account_manager' RSpec.describe Html2rss::Web::AccountManager do describe '.get_account' do - it 'returns account by token' do - allow(Html2rss::Web::LocalConfig).to receive(:global).and_return( - auth: { accounts: [{ username: 'alice', token: 'token-1', allowed_urls: ['*'] }] } - ) + it 'returns the admin account by token from RuntimeEnv' do + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'token-1') do + described_class.reload! - account = described_class.get_account('token-1') + account = described_class.get_account('token-1') - expect(account).to include(username: 'alice', token: 'token-1') + expect(account).to include(username: 'admin', token: 'token-1', allowed_urls: ['*']) + end + end + + it 'falls back to the admin placeholder when HTML2RSS_ACCESS_TOKEN is unset' do + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => nil) do + described_class.reload! + + account = described_class.get_account(Html2rss::Web::RuntimeEnv::ADMIN_ACCESS_TOKEN_PLACEHOLDER) + + expect(account).to include(username: 'admin', allowed_urls: ['*']) + end end end describe '.reload!' do it 'keeps memoized snapshot before reload' do - allow(Html2rss::Web::LocalConfig).to receive(:global).and_return( - { auth: { accounts: [{ username: 'alice', token: 'token-1', allowed_urls: ['*'] }] } }, - { auth: { accounts: [{ username: 'bob', token: 'token-2', allowed_urls: ['*'] }] } } - ) + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'token-1') do + described_class.reload! + described_class.get_account('token-1') + end - described_class.get_account('token-1') - expect(described_class.get_account('token-2')).to be_nil + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'token-2') do + expect(described_class.get_account('token-2')).to be_nil + end end it 'clears memoized snapshot after reload' do - allow(Html2rss::Web::LocalConfig).to receive(:global).and_return( - { auth: { accounts: [{ username: 'alice', token: 'token-1', allowed_urls: ['*'] }] } }, - { auth: { accounts: [{ username: 'bob', token: 'token-2', allowed_urls: ['*'] }] } } - ) - - described_class.get_account('token-1') - described_class.reload! - expect(described_class.get_account('token-2')).to include(username: 'bob') + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'token-1') do + described_class.reload! + described_class.get_account('token-1') + end + + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'token-2') do + described_class.reload! + expect(described_class.get_account('token-2')).to include(username: 'admin') + end end end describe '.get_account_by_username' do it 'returns account by username from the memoized snapshot' do - allow(Html2rss::Web::LocalConfig).to receive(:global).and_return( - auth: { accounts: [{ username: 'alice', token: 'token-1', allowed_urls: ['*'] }] } - ) + ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'token-1') do + described_class.reload! + + account = described_class.get_account_by_username('admin') + + expect(account).to include(username: 'admin', token: 'token-1') + end + end + end + + describe 'health-check account' do + it 'omits health-check when HEALTH_CHECK_TOKEN is unset or placeholder', :aggregate_failures do + ClimateControl.modify('HEALTH_CHECK_TOKEN' => nil) do + described_class.reload! + expect(described_class.get_account_by_username('health-check')).to be_nil + end + + ClimateControl.modify( + 'HEALTH_CHECK_TOKEN' => Html2rss::Web::RuntimeEnv::HEALTH_CHECK_TOKEN_PLACEHOLDER + ) do + described_class.reload! + expect(described_class.get_account_by_username('health-check')).to be_nil + end + end - account = described_class.get_account_by_username('alice') + it 'includes health-check when HEALTH_CHECK_TOKEN is a real value' do + ClimateControl.modify('HEALTH_CHECK_TOKEN' => 'real-health-token') do + described_class.reload! - expect(account).to include(username: 'alice', token: 'token-1') + expect(described_class.get_account_by_username('health-check')).to include( + username: 'health-check', + token: 'real-health-token', + allowed_urls: [] + ) + end end end end diff --git a/spec/html2rss/web/api/v1_spec.rb b/spec/html2rss/web/api/v1_spec.rb index e037715f7..f51a3e50d 100644 --- a/spec/html2rss/web/api/v1_spec.rb +++ b/spec/html2rss/web/api/v1_spec.rb @@ -196,6 +196,21 @@ def relative_feed_link_header(token) ) end + it 'returns registry status metadata', :aggregate_failures do + get '/api/v1' + + expect(last_response.status).to eq(200) + json = expect_success_response(last_response) + official = json.dig('data', 'instance', 'registries').find { |row| row['id'] == 'official' } + + expect(official).to include( + 'id' => 'official', + 'version' => 'test-fixture', + 'sync_mode' => 'path' + ) + expect(official['updated_at']).to be_a(String) + end + it 'returns API information with trailing slash', :aggregate_failures do get '/api/v1/' @@ -221,7 +236,10 @@ def relative_feed_link_header(token) json = expect_success_response(last_response) expect(json.dig('meta', 'catalog_version')).to eq(1) expect(json.dig('data', 'configs')).to be_an(Array) - expect(json.dig('data', 'configs').first).to include('id', 'path', 'source', 'directory', 'channel', 'parameters') + first = json.dig('data', 'configs').first + expect(first).to include('id', 'path', 'source', 'directory', 'channel', 'parameters') + expect(first['source']).to eq('registry') + expect(first).to include('registry' => 'official') end it 'returns 404 when the catalog is disabled', :aggregate_failures do @@ -334,7 +352,7 @@ def relative_feed_link_header(token) it 'returns error when configuration fails', :aggregate_failures do allow(Html2rss::Web::Auth).to receive(:authenticate).and_return({ username: 'health-check' }) - allow(Html2rss::Web::LocalConfig).to receive(:load_snapshot).and_raise(StandardError, 'boom') + allow(Html2rss::Web::Registry::Index).to receive(:current).and_raise(StandardError, 'boom') header 'Authorization', "Bearer #{health_token}" get '/api/v1/health' diff --git a/spec/html2rss/web/app_integration_spec.rb b/spec/html2rss/web/app_integration_spec.rb index 181ab59b8..e2fbcf10c 100644 --- a/spec/html2rss/web/app_integration_spec.rb +++ b/spec/html2rss/web/app_integration_spec.rb @@ -27,14 +27,10 @@ } end - let(:accounts_config) do - { - auth: { - accounts: [account, { username: 'admin', token: 'admin-token-xyz789', allowed_urls: ['*'] }] - } - } + let(:admin_account) do + { username: 'admin', token: 'admin-token-xyz789', allowed_urls: ['*'] } end - let(:config_snapshot) { Html2rss::Web::ConfigSnapshot.load(accounts_config) } + let(:accounts) { [account, admin_account] } let(:json_headers) { { 'CONTENT_TYPE' => 'application/json' } } let(:auth_headers) { json_headers.merge('HTTP_AUTHORIZATION' => "Bearer #{account[:token]}") } @@ -56,7 +52,13 @@ end before do - allow(Html2rss::Web::LocalConfig).to receive_messages(global: config_snapshot.global, snapshot: config_snapshot) + allow(Html2rss::Web::AccountManager).to receive(:accounts).and_return(accounts) + allow(Html2rss::Web::AccountManager).to receive(:get_account) { |token| + accounts.find { it[:token] == token } + } + allow(Html2rss::Web::AccountManager).to receive(:get_account_by_username) { |username| + accounts.find { it[:username] == username } + } stub_const('Html2rss::FeedChannel', Class.new { attr_reader :ttl }) stub_const('Html2rss::Feed', Class.new { attr_reader :channel }) allow(Html2rss::Web::Flags).to receive(:auto_source_enabled?).and_return(true) diff --git a/spec/html2rss/web/app_spec.rb b/spec/html2rss/web/app_spec.rb index ead4b1827..294b3ebed 100644 --- a/spec/html2rss/web/app_spec.rb +++ b/spec/html2rss/web/app_spec.rb @@ -24,7 +24,7 @@ def static_feed_json end def stub_static_feed(rss_body: '', json_body: static_feed_json, ttl: 180) - allow(Html2rss::Web::LocalConfig).to receive(:find).and_return({ channel: { ttl: ttl } }) + allow(Html2rss::Web::Registry::Index.current).to receive(:config_for).and_return({ channel: { ttl: ttl } }) stub_static_renderers(static_feed_result(ttl:), rss_body:, json_body:) end @@ -59,8 +59,8 @@ def static_service_error_result end def stub_static_service_error(feed_name) - allow(Html2rss::Web::LocalConfig) - .to receive(:find) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for) .with(feed_name) .and_return({ channel: { ttl: 180 } }) allow(Html2rss::Web::Feeds::Service).to receive(:call).and_return(static_service_error_result) @@ -145,7 +145,8 @@ def app = described_class end it 'serves nested static feed routes' do - allow(Html2rss::Web::LocalConfig).to receive(:find).with('team/releases').and_return({ channel: { ttl: 180 } }) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for).with('team/releases').and_return({ channel: { ttl: 180 } }) stub_static_renderers(static_feed_result(ttl: 180), rss_body: '', json_body: static_feed_json) get '/team/releases.xml' @@ -157,7 +158,8 @@ def app = described_class it 'serves HEAD requests for static feed routes with negotiated headers only' do feed_name = "legacy-head-#{SecureRandom.hex(4)}" - allow(Html2rss::Web::LocalConfig).to receive(:find).with(feed_name).and_return({ channel: { ttl: 180 } }) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for).with(feed_name).and_return({ channel: { ttl: 180 } }) stub_static_renderers(static_feed_result(ttl: 180), rss_body: '', json_body: static_feed_json) head "/#{feed_name}" @@ -176,7 +178,8 @@ def app = described_class it 'coerces string ttl values before cache expiry math' do feed_name = "legacy-ttl-#{SecureRandom.hex(4)}" - allow(Html2rss::Web::LocalConfig).to receive(:find).with(feed_name).and_return({ channel: { ttl: '180' } }) + allow(Html2rss::Web::Registry::Index.current) + .to receive(:config_for).with(feed_name).and_return({ channel: { ttl: '180' } }) stub_static_renderers(static_feed_result(ttl: '180'), rss_body: '', json_body: static_feed_json) get "/#{feed_name}" diff --git a/spec/html2rss/web/boot/development_reloader_spec.rb b/spec/html2rss/web/boot/development_reloader_spec.rb index 173076531..c88cd5a94 100644 --- a/spec/html2rss/web/boot/development_reloader_spec.rb +++ b/spec/html2rss/web/boot/development_reloader_spec.rb @@ -11,9 +11,9 @@ let(:instance) { described_class.new(loader:, app_provider: -> { rack_app }) } before do - stub_const('Html2rss::Web::LocalConfig', Class.new { def self.reload!(reason:); end }) + stub_const('Html2rss::Web::Registry::Index', Class.new { def self.reload!; end }) stub_const('Html2rss::Web::AccountManager', Class.new { def self.reload!(reason:); end }) - allow(Html2rss::Web::LocalConfig).to receive(:reload!) + allow(Html2rss::Web::Registry::Index).to receive(:reload!) allow(Html2rss::Web::AccountManager).to receive(:reload!) end @@ -27,7 +27,7 @@ status, headers, body = instance.call({}) expect(loader).to have_received(:reload).once - expect(Html2rss::Web::LocalConfig).to have_received(:reload!).with(reason: 'code_reload').once + expect(Html2rss::Web::Registry::Index).to have_received(:reload!).once expect(Html2rss::Web::AccountManager).to have_received(:reload!).with(reason: 'code_reload').once expect([status, headers['Content-Type'], body.each.to_a]).to eq([200, 'text/plain', ['ok']]) end diff --git a/spec/html2rss/web/boot/setup_spec.rb b/spec/html2rss/web/boot/setup_spec.rb index 116b60811..d98558843 100644 --- a/spec/html2rss/web/boot/setup_spec.rb +++ b/spec/html2rss/web/boot/setup_spec.rb @@ -21,6 +21,7 @@ before do allow(Html2rss::Web::Flags).to receive(:validate!) allow(Html2rss::Web::Boot::Sentry).to receive(:configure!) + allow(Html2rss::Web::Registry::Sync).to receive(:boot!) end describe '.call!' do @@ -33,6 +34,7 @@ expect(Html2rss::Web::EnvironmentValidator).to have_received(:validate_environment!).once expect(Html2rss::Web::EnvironmentValidator).to have_received(:validate_production_security!).once expect(Html2rss::Web::Flags).to have_received(:validate!).once + expect(Html2rss::Web::Registry::Sync).to have_received(:boot!).once end it 'routes rack-timeout logs through the shared app logger' do diff --git a/spec/html2rss/web/cli_spec.rb b/spec/html2rss/web/cli_spec.rb new file mode 100644 index 000000000..59d8c489a --- /dev/null +++ b/spec/html2rss/web/cli_spec.rb @@ -0,0 +1,262 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'stringio' +require 'climate_control' +require_relative '../../../app' +require_relative '../../../app/web/cli' + +RSpec.describe Html2rss::Web::CLI do + describe '.run' do + let(:stdout) { StringIO.new } + let(:stderr) { StringIO.new } + + context 'with root help and version commands' do + it 'prints root help for help options', :aggregate_failures do + exit_code = described_class.run(['--help'], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('Usage: html2rss-web [command] [options]') + expect(stdout.string).to include('registry status') + end + + it 'prints root help when argv is empty', :aggregate_failures do + exit_code = described_class.run([], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('Usage: html2rss-web [command] [options]') + end + + it 'prints version and runtime info', :aggregate_failures do + exit_code = described_class.run(['version'], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to match(/html2rss-web build=.* sha=.* ruby=.*/) + end + + it 'handles unknown commands', :aggregate_failures do + exit_code = described_class.run(['invalid-cmd'], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Unknown command: "invalid-cmd"') + end + end + + context 'with healthcheck command' do + let(:success_response) { instance_double(Net::HTTPSuccess, is_a?: true) } + let(:error_response) { instance_double(Net::HTTPInternalServerError, is_a?: false, code: '500') } + + it 'returns 0 when HTTP healthcheck succeeds', :aggregate_failures do + allow(Net::HTTP).to receive(:get_response).and_return(success_response) + + exit_code = described_class.run(['healthcheck'], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('OK') + end + + it 'returns 1 when HTTP healthcheck returns non-200', :aggregate_failures do + allow(Net::HTTP).to receive(:get_response).and_return(error_response) + + exit_code = described_class.run(['healthcheck'], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Healthcheck failed: HTTP 500') + end + + it 'returns 1 when HTTP connection fails', :aggregate_failures do + allow(Net::HTTP).to receive(:get_response).and_raise(Errno::ECONNREFUSED) + + exit_code = described_class.run(['healthcheck'], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Healthcheck failed') + end + end + + context 'with registry subcommands' do + it 'prints registry help for --help', :aggregate_failures do + exit_code = described_class.run(%w[registry --help], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include('Usage: html2rss-web registry [subcommand]') + end + + it 'prints registry status table', :aggregate_failures do + exit_code = described_class.run(%w[registry status], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(stdout.string).to include("registry\tmode\tversion\tstaged_version\tupdated_at\tsync_url\tlast_error") + expect(stdout.string).to include('official') + end + + it 'runs registry sync', :aggregate_failures do + allow(Html2rss::Web::Registry::Sync).to receive_messages(run: nil, cli_exit_code: 0) + + exit_code = described_class.run(%w[registry sync --registry official --dry-run], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(Html2rss::Web::Registry::Sync).to have_received(:run).with(registry_id: 'official', dry_run: true) + end + + it 'runs registry promote', :aggregate_failures do + allow(Html2rss::Web::Registry::Sync).to receive_messages(promote_staged!: nil, cli_exit_code: 0) + + exit_code = described_class.run(%w[registry promote --registry official], out: stdout, err: stderr) + expect(exit_code).to eq(0) + expect(Html2rss::Web::Registry::Sync).to have_received(:promote_staged!).with(registry_id: 'official') + end + + it 'verifies valid registry bundles', :aggregate_failures do + exit_code = described_class.run( + ['registry', 'verify', '--registry', 'official', '--dir', 'spec/fixtures/registries/official'], + out: stdout, + err: stderr + ) + expect(exit_code).to eq(0) + expect(stdout.string).to include("Verified registry bundle 'official' (test-fixture)") + end + + it 'reports failure on invalid registry verification', :aggregate_failures do + exit_code = described_class.run( + ['registry', 'verify', '--registry', 'official', '--dir', 'tmp/non-existent'], + out: stdout, + err: stderr + ) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Registry verification failed') + end + + it 'handles unknown registry subcommands', :aggregate_failures do + exit_code = described_class.run(%w[registry invalid-sub], out: stdout, err: stderr) + expect(exit_code).to eq(1) + expect(stderr.string).to include('Unknown registry command: "invalid-sub"') + end + end + + context 'with registry add' do + let(:local_root) { File.join(Dir.pwd, 'tmp', 'cli-local-registry') } + let(:config_path) { File.join(Dir.pwd, 'tmp', 'cli-local-registries.yml') } + + before do + FileUtils.rm_rf(local_root) + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - local + registries: + local: + path: #{local_root} + catalog: false + YAML + end + + after do + FileUtils.rm_rf(local_root) + FileUtils.rm_f(config_path) + end + + def with_local_registry(&) + ClimateControl.modify('REGISTRIES_CONFIG' => config_path) do + Html2rss::Web::Registry::Index.reload! + yield + end + end + + def write_feed(path, id:, title: 'Example') # rubocop:disable Metrics/MethodLength + FileUtils.mkdir_p(File.dirname(path)) + File.write(path, <<~YAML) + registry: + id: #{id} + directory: + title: #{title} + channel: + url: https://example.com/#{id} + title: #{title} + selectors: + items: + selector: article + title: + selector: h1 + url: + extractor: href + YAML + end + + def run_add(path) + described_class.run(['registry', 'add', path], out: stdout, err: stderr) + end + + it 'accepts one valid feed file', :aggregate_failures do + feed = File.join(Dir.pwd, 'tmp', 'cli-add-feed.yml') + write_feed(feed, id: 'example.com/feed') + + with_local_registry do + expect(run_add(feed)).to eq(0) + expect(stdout.string).to include('example.com/feed', local_root) + expect(Html2rss::Web::Registry::Index.current.config_for('example.com/feed')) + .to include(channel: hash_including(title: 'Example')) + end + ensure + FileUtils.rm_f(feed) + end + + it 'rejects missing registry.id', :aggregate_failures do # rubocop:disable RSpec/ExampleLength + feed = File.join(Dir.pwd, 'tmp', 'cli-add-missing-id.yml') + File.write(feed, <<~YAML) + channel: + url: https://example.com/feed + title: Example + selectors: + items: + selector: article + title: + selector: h1 + url: + extractor: href + YAML + + with_local_registry do + expect(run_add(feed)).to eq(1) + expect(stderr.string).to match(/registry\.id|Registry add failed/) + expect(File).not_to exist(File.join(local_root, 'manifest.json')) + end + ensure + FileUtils.rm_f(feed) + end + + it 'rejects invalid YAML', :aggregate_failures do + feed = File.join(Dir.pwd, 'tmp', 'cli-add-invalid.yml') + File.write(feed, "registry:\n id: [broken") + + with_local_registry do + expect(run_add(feed)).to eq(1) + expect(stderr.string).to include('Registry add failed') + end + ensure + FileUtils.rm_f(feed) + end + + it 'upserts the same registry.id', :aggregate_failures do + feed = File.join(Dir.pwd, 'tmp', 'cli-add-upsert.yml') + write_feed(feed, id: 'example.com/feed', title: 'First') + + with_local_registry do + run_add(feed) + write_feed(feed, id: 'example.com/feed', title: 'Second') + expect(run_add(feed)).to eq(0) + expect(Html2rss::Web::Registry::Index.current.config_for('example.com/feed')) + .to include(channel: hash_including(title: 'Second')) + end + ensure + FileUtils.rm_f(feed) + end + + it 'leaves the previous bundle when a directory add fails', :aggregate_failures do + dir = File.join(Dir.pwd, 'tmp', 'cli-add-dir') + good = File.join(dir, 'good.yml') + write_feed(good, id: 'example.com/good', title: 'Keep Me') + + with_local_registry do + run_add(good) + File.write(File.join(dir, 'bad.yml'), "registry:\n id: [broken") + expect(run_add(dir)).to eq(1) + expect(Html2rss::Web::Registry::Index.current.config_for('example.com/good')) + .to include(channel: hash_including(title: 'Keep Me')) + end + ensure + FileUtils.rm_rf(dir) + end + end + end +end diff --git a/spec/html2rss/web/feeds/responder_spec.rb b/spec/html2rss/web/feeds/responder_spec.rb index 1d8e1a9d5..0ebd91e54 100644 --- a/spec/html2rss/web/feeds/responder_spec.rb +++ b/spec/html2rss/web/feeds/responder_spec.rb @@ -25,7 +25,7 @@ end before do - allow(Html2rss::Web::LocalConfig).to receive(:find).with('example').and_return(static_config) + allow(Html2rss::Web::Registry::Index.current).to receive(:config_for).with('example').and_return(static_config) allow(Html2rss::Web::Observability).to receive(:emit) allow(Html2rss::Web::SentryOps).to receive(:emit_failure_telemetry) end diff --git a/spec/html2rss/web/feeds/source_resolver_spec.rb b/spec/html2rss/web/feeds/source_resolver_spec.rb index fcb43545f..d58aa708f 100644 --- a/spec/html2rss/web/feeds/source_resolver_spec.rb +++ b/spec/html2rss/web/feeds/source_resolver_spec.rb @@ -27,7 +27,7 @@ def resolved_tuple(resolved) end before do - allow(Html2rss::Web::LocalConfig).to receive(:find).with('legacy').and_return(config) + allow(Html2rss::Web::Registry::Index.current).to receive(:config_for).with('legacy').and_return(config) end it 'normalizes the static source into shared generator input without forcing a default strategy', @@ -92,7 +92,7 @@ def resolved_tuple(resolved) .with({ username: 'admin' }, 'https://example.com/private').and_return(true) allow(Html2rss::Web::Flags).to receive(:auto_source_enabled?).and_return(true) allow(Html2rss::RequestService).to receive(:strategy_names).and_return([:faraday]) - allow(Html2rss::Web::LocalConfig).to receive(:global) + allow(Html2rss::Web::Feeds::FeedChrome).to receive(:generator_defaults) .and_return({ headers: { 'User-Agent' => 'html2rss-web' } }) end @@ -101,7 +101,8 @@ def resolved_tuple(resolved) expect(resolved_tuple(resolved)).to match( [:token, start_with('token:'), 300, - include(strategy: :faraday, channel: { url: 'https://example.com/private' }, auto_source: {})] + include(strategy: :faraday, channel: { url: 'https://example.com/private' }, auto_source: {}, + headers: { 'User-Agent' => 'html2rss-web' })] ) expect(resolved).to have_attributes( url: 'https://example.com/private', diff --git a/spec/html2rss/web/local_config_spec.rb b/spec/html2rss/web/local_config_spec.rb deleted file mode 100644 index aa0a10d35..000000000 --- a/spec/html2rss/web/local_config_spec.rb +++ /dev/null @@ -1,150 +0,0 @@ -# frozen_string_literal: true - -require 'spec_helper' -require 'climate_control' -require 'tempfile' - -require_relative '../../../app' - -RSpec.describe Html2rss::Web::LocalConfig do - let(:empty_snapshot) { Html2rss::Web::ConfigSnapshot::Snapshot.new(global: {}, feeds: {}, accounts: []) } - - def titles_for(*names) - names.map { |name| described_class.find(name)[:title] } - end - - def with_config_file(contents) - Tempfile.create(['feeds', '.yml']) do |file| - write_config(file, contents) - stub_const("#{described_class}::CONFIG_FILE", file.path) - yield file - end - end - - def write_config(file, contents) - file.rewind - file.truncate(0) - file.write(contents) - file.flush - end - - def account_token(snapshot) - snapshot.accounts.first.token - end - - before do - described_class.reload! - end - - describe '.find' do - it 'strips feed extensions before lookup' do - allow(described_class).to receive(:snapshot).and_return( - Html2rss::Web::ConfigSnapshot::Snapshot.new( - global: {}, - feeds: { - example: Html2rss::Web::ConfigSnapshot::FeedConfig.new(name: :example, raw: { title: 'Example' }) - }, - accounts: [] - ) - ) - - expect(titles_for('example.json', 'example.rss', 'example.xml')).to eq(%w[Example Example Example]) - end - - it 'falls back to embedded configs when the feed is not in local yaml' do - stub_const('Html2rss::Configs', Module.new do - def self.find_by_name(_name); end - end) - stub_const('Html2rss::Configs::ConfigNotFound', Class.new(StandardError)) - allow(Html2rss::Configs) - .to receive(:find_by_name) - .with('support.apple.com/en_gb_ht201222') - .and_return({ channel: { title: 'Apple security releases' } }) - allow(described_class).to receive(:snapshot).and_return(empty_snapshot) - - config = described_class.find('support.apple.com/en_gb_ht201222.rss') - - expect(config).to include(channel: { title: 'Apple security releases' }) - end - - it 'returns not found for malformed embedded config paths instead of depending on gem error messages' do - stub_const('Html2rss::Configs', Module.new do - def self.find_by_name(_name); end - end) - stub_const('Html2rss::Configs::ConfigNotFound', Class.new(StandardError)) - allow(Html2rss::Configs).to receive(:find_by_name) - allow(described_class).to receive(:snapshot).and_return(empty_snapshot) - - expect { described_class.find('/broken-name.rss') } - .to raise_error(described_class::NotFound, "Did not find local feed config at 'broken-name'") - expect(Html2rss::Configs).not_to have_received(:find_by_name) - end - end - - describe '.snapshot' do - let(:yaml_fixture) do - { - auth: { accounts: [{ username: 'alice', token: 'token-1', allowed_urls: ['*'] }] }, - feeds: {} - } - end - - it 'builds typed account models from configuration' do - allow(described_class).to receive(:load_yaml).and_return(yaml_fixture) - described_class.reload! - - expect(described_class.snapshot.accounts.first.username).to eq('alice') - end - - it 'evaluates ERB before parsing YAML so runtime helpers become effective config' do - with_config_file(erb_backed_config) do - ClimateControl.modify('HTML2RSS_ACCESS_TOKEN' => 'runtime-access-token') do - described_class.reload! - expect(account_token(described_class.snapshot)).to eq('runtime-access-token') - end - end - end - end - - describe '.load_snapshot' do - it 'reparses current config without mutating the memoized runtime snapshot', :aggregate_failures do - with_config_file(config_for('cached-token')) do |file| - described_class.reload! - - cached_snapshot = described_class.snapshot - - write_config(file, config_for('fresh-token')) - - expect(described_class.load_snapshot).to have_attributes( - accounts: contain_exactly(have_attributes(token: 'fresh-token')) - ) - expect(account_token(described_class.snapshot)).to eq('cached-token') - expect(account_token(cached_snapshot)).to eq('cached-token') - end - end - end - - def erb_backed_config - <<~YAML - auth: - accounts: - - username: admin - token: <%= Html2rss::Web::RuntimeEnv.admin_access_token %> - allowed_urls: - - "*" - feeds: {} - YAML - end - - def config_for(token) - <<~YAML - auth: - accounts: - - username: admin - token: #{token} - allowed_urls: - - "*" - feeds: {} - YAML - end -end diff --git a/spec/html2rss/web/registry/config_spec.rb b/spec/html2rss/web/registry/config_spec.rb new file mode 100644 index 000000000..2845da4e9 --- /dev/null +++ b/spec/html2rss/web/registry/config_spec.rb @@ -0,0 +1,117 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'climate_control' +require 'spec_helper' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Config do + describe '.entry' do + let(:config_path) { File.join(Dir.pwd, 'tmp', 'missing-public-key-registries.yml') } + + before do + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + sync: + url: https://registry.test.example/registry-bundle.tar.gz + catalog: true + YAML + end + + after do + FileUtils.rm_f(config_path) + end + + it 'requires a pinned public key for sync-mode registries' do + ClimateControl.modify('REGISTRIES_CONFIG' => config_path) do + expect do + described_class.reload! + described_class.entry('official') + end.to raise_error(Html2rss::Web::Registry::Errors::ConfigError, /requires a pinned public_key/) + end + end + end + + describe 'sync policy parsing' do + let(:config_path) { File.join(Dir.pwd, 'tmp', 'sync-policy-registries.yml') } + + before do + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + sync: + channel: html2rss-official + pin_version: v2026.08.22 + max_version: v2026.08.21 + auto_promote: true + allowed_channel_domains: + - phys.org + catalog: true + public_key_id: html2rss:registry:2026 + public_key: | + -----BEGIN PUBLIC KEY----- + MCowBQYDK2VwAyEAiMbg/04MyC5azBdM/aeY0mNuA8JbP5/jOiNRwJ2KJHE= + -----END PUBLIC KEY----- + YAML + end + + after do + FileUtils.rm_f(config_path) + end + + it 'parses sync policy and domain allowlist fields', :aggregate_failures do + ClimateControl.modify('REGISTRIES_CONFIG' => config_path) do + described_class.reload! + entry = described_class.entry('official') + + expect(entry.sync_policy).to have_attributes( + pin_version: 'v2026.08.22', + max_version: 'v2026.08.21', + auto_promote: true + ) + expect(entry.allowed_channel_domains).to eq(['phys.org']) + end + end + + it 'defaults auto_promote to false for security' do + ClimateControl.modify('REGISTRIES_CONFIG' => nil) do + described_class.reload! + entry = described_class.entry('official') + + expect(entry.sync_policy.auto_promote).to be(false) + end + end + end + + describe 'built-in local registry' do + it 'includes local before official in the zero-config document', :aggregate_failures do + ClimateControl.modify('REGISTRIES_CONFIG' => nil) do + described_class.reload! + + expect(described_class.precedence).to eq(%w[local official]) + expect(described_class.entry('local')).to have_attributes( + mode: :path, + catalog: false, + path: File.expand_path('data/registries/local', Dir.pwd) + ) + end + end + + it 'loads local from config/registries.yml', :aggregate_failures do + ClimateControl.modify('REGISTRIES_CONFIG' => 'config/registries.yml') do + described_class.reload! + + expect(described_class.precedence).to eq(%w[local official]) + expect(described_class.entry('local')).to have_attributes(mode: :path, catalog: false) + end + end + end +end diff --git a/spec/html2rss/web/registry/index_spec.rb b/spec/html2rss/web/registry/index_spec.rb new file mode 100644 index 000000000..c2048d2e7 --- /dev/null +++ b/spec/html2rss/web/registry/index_spec.rb @@ -0,0 +1,195 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'spec_helper' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Index do + describe '#config_for' do + it 'returns registry configs by feed id' do + config = described_class.current.config_for('anthropic.com/news') + + expect(config).to include(channel: hash_including(title: 'Anthropic — News')) + end + + it 'returns nil for unknown ids' do + expect(described_class.current.config_for('missing.example/feed')).to be_nil + end + + it 'resolves feeds by alias' do # rubocop:disable RSpec/ExampleLength + fake_bundle = described_class::RegistryBundle.new( + registry_id: 'official', + manifest: nil, + configs: { + 'anthropic.com/news' => { channel: { title: 'Anthropic — News' } }, + 'anthropic.com/legacy-news' => { channel: { title: 'Anthropic — News' } } + }, + catalog_entries: [] + ) + allow(described_class.current).to receive(:loaded_bundles).and_return('official' => fake_bundle) + + expect(described_class.current.config_for('anthropic.com/legacy-news')).to include( + channel: hash_including(title: 'Anthropic — News') + ) + end + + it 'still resolves catalog-disabled registry feeds by id' do + config = described_class.current.config_for('secret.example/private') + + expect(config).to include(channel: hash_including(url: 'https://secret.example/private')) + end + end + + describe '.current' do + it 'reloads when manifest mtime changes' do + initial_index = described_class.current + allow(Html2rss::Web::Registry::Store).to receive(:manifest_mtime).and_return('2099-01-01T00:00:00Z') + + expect(described_class.current).not_to be(initial_index) + end + end + + describe '#catalog_rows' do + it 'includes registry rows with source and registry fields' do + rows = described_class.current.catalog_rows + anthropic = rows.find { it.fetch(:id) == 'anthropic.com/news' } + + expect(anthropic).to include( + source: 'registry', + registry: 'official', + path: '/anthropic.com/news.rss' + ) + end + + it 'omits catalog-disabled registries from the catalog API rows' do + rows = described_class.current.catalog_rows + + expect(rows.map { it.fetch(:id) }).not_to include('secret.example/private') + end + + it 'prefers the first registry in precedence for duplicate feed ids' do + rows = described_class.current.catalog_rows + deepmind = rows.find { it.fetch(:id) == 'deepmind.google/blog' } + + expect(deepmind.fetch(:registry)).to eq('official') + end + end + + describe '#status' do + it 'reports loaded registry metadata' do + status = described_class.current.status + official = status.find { it.id == 'official' } + + expect(official).to have_attributes( + version: 'test-fixture', + mode: :path + ) + end + end + + describe 'allowed_channel_domains' do + let(:config_path) { File.join(Dir.pwd, 'tmp', 'domain-allowlist-registries.yml') } + + after do + FileUtils.rm_f(config_path) + end + + it 'allows suffix-matching channel domains via config load' do # rubocop:disable RSpec/ExampleLength + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + path: spec/fixtures/registries/official + catalog: true + allowed_channel_domains: + - anthropic.com + - deepmind.google + YAML + ENV['REGISTRIES_CONFIG'] = config_path + described_class.reload! + + expect(described_class.current.config_for('anthropic.com/news')).to include( + channel: hash_including(url: 'https://www.anthropic.com/news') + ) + end + + it 'rejects bundles with channel URLs outside the allowlist' do # rubocop:disable RSpec/ExampleLength + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, <<~YAML) + precedence: + - official + registries: + official: + path: spec/fixtures/registries/official + catalog: true + allowed_channel_domains: + - blocked.example + YAML + ENV['REGISTRIES_CONFIG'] = config_path + described_class.reload! + + expect { described_class.current.config_for('anthropic.com/news') } + .to raise_error(Html2rss::Web::Registry::Errors::LoadError, /anthropic\.com/) + end + end + + describe 'embedded bundle resolution' do + let(:embedded_config_path) { File.join(Dir.pwd, 'tmp', 'embedded-registries.yml') } + let(:embedded_root) { File.join(Dir.pwd, 'tmp', 'embedded-root') } + + before do + FileUtils.mkdir_p(File.dirname(embedded_config_path)) + File.write( + embedded_config_path, + YAML.dump( + 'precedence' => ['official'], + 'registries' => { + 'official' => { + 'sync' => { 'channel' => 'html2rss-official' }, + 'catalog' => true, + 'public_key_id' => 'test-key', + 'public_key' => RegistrySyncTestHelpers::TEST_PUBLIC_KEY + } + } + ) + ) + ENV['REGISTRIES_CONFIG'] = embedded_config_path + ENV['REGISTRY_DATA_ROOT'] = File.join(Dir.pwd, 'tmp', 'empty-data-root') + ENV['REGISTRY_EMBEDDED_ROOT'] = embedded_root + FileUtils.rm_rf(ENV.fetch('REGISTRY_DATA_ROOT')) + FileUtils.rm_rf(embedded_root) + + embedded_official = File.join(embedded_root, 'official') + FileUtils.mkdir_p(embedded_official) + FileUtils.cp_r(File.join(RegistryTestHelpers::FIXTURES_ROOT, 'official', 'configs'), embedded_official) + FileUtils.cp( + File.join(RegistrySyncTestHelpers::SYNC_FIXTURES_ROOT, 'bundle', Html2rss::Registry::Manifest::MANIFEST_FILE), + File.join(embedded_official, Html2rss::Registry::Manifest::MANIFEST_FILE) + ) + manifest = Html2rss::Registry::Manifest.parse( + File.read(File.join(embedded_official, Html2rss::Registry::Manifest::MANIFEST_FILE)) + ) + Html2rss::Registry::Signer.sign!( + manifest, + key_pem: RegistrySyncTestHelpers::TEST_PRIVATE_KEY, + bundle_dir: embedded_official + ) + described_class.reload! + end + + after do + FileUtils.rm_f(embedded_config_path) + FileUtils.rm_rf(embedded_root) + FileUtils.rm_rf(ENV.fetch('REGISTRY_DATA_ROOT', nil)) + end + + it 'loads configs directly from the embedded root when no runtime sync data exists' do + expect(described_class.current.config_for('anthropic.com/news')).to include( + channel: hash_including(title: 'Anthropic — News') + ) + end + end +end diff --git a/spec/html2rss/web/registry/store_spec.rb b/spec/html2rss/web/registry/store_spec.rb new file mode 100644 index 000000000..d6cd8a77f --- /dev/null +++ b/spec/html2rss/web/registry/store_spec.rb @@ -0,0 +1,115 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'json' +require 'digest' +require 'spec_helper' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Store do + let(:registry_id) { 'store-test' } + let(:data_root) { File.join(Dir.pwd, 'tmp', 'store-spec-data') } + + let(:embedded_root) { File.join(Dir.pwd, 'tmp', 'store-spec-embedded') } + + before do + ENV['REGISTRY_DATA_ROOT'] = data_root + ENV['REGISTRY_EMBEDDED_ROOT'] = embedded_root + FileUtils.rm_rf(data_root) + FileUtils.rm_rf(embedded_root) + end + + describe '.stage_bundle! and .promote_staged!' do + it 'stages a verified bundle and promotes it to active', :aggregate_failures do + source = build_bundle_dir('stage-source') + active_before = build_bundle_dir('active-before') + + FileUtils.mkdir_p(File.dirname(described_class.registry_dir(registry_id))) + FileUtils.cp_r(active_before, described_class.registry_dir(registry_id)) + + described_class.stage_bundle!(registry_id, source) + + expect(described_class.staged_present?(registry_id)).to be(true) + expect(described_class.staged_version(registry_id)).to eq('stage-source') + expect(described_class.bundle_present?(registry_id)).to be(true) + expect(read_manifest_version(described_class.registry_dir(registry_id))).to eq('active-before') + + described_class.promote_staged!(registry_id) + + expect(described_class.staged_present?(registry_id)).to be(false) + expect(read_manifest_version(described_class.registry_dir(registry_id))).to eq('stage-source') + end + end + + describe '.bundle_present?' do + it 'requires manifest.json instead of any non-empty directory' do + path = described_class.registry_dir(registry_id) + + FileUtils.rm_rf(path) + FileUtils.mkdir_p(path) + File.write(File.join(path, 'placeholder.txt'), 'content') + + expect(described_class.bundle_present?(registry_id)).to be(false) + + File.write(File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + expect(described_class.bundle_present?(registry_id)).to be(true) + end + end + + describe '.active_dir' do + let(:embedded_root) { File.join(Dir.pwd, 'tmp', 'store-spec-embedded') } + + before do + ENV['REGISTRY_EMBEDDED_ROOT'] = embedded_root + FileUtils.rm_rf(embedded_root) + end + + it 'returns nil when neither synced nor embedded bundle exists' do + expect(described_class.active_dir(registry_id)).to be_nil + end + + it 'falls back to embedded bundle directory when synced bundle is absent' do + embedded_dir = described_class.embedded_dir(registry_id) + FileUtils.mkdir_p(embedded_dir) + File.write(File.join(embedded_dir, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + + expect(described_class.active_dir(registry_id)).to eq(embedded_dir) + end + + it 'prefers synced runtime bundle over embedded bundle' do + embedded_dir = described_class.embedded_dir(registry_id) + FileUtils.mkdir_p(embedded_dir) + File.write(File.join(embedded_dir, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + + synced_dir = described_class.registry_dir(registry_id) + FileUtils.mkdir_p(synced_dir) + File.write(File.join(synced_dir, Html2rss::Registry::Manifest::MANIFEST_FILE), '{}') + + expect(described_class.active_dir(registry_id)).to eq(synced_dir) + end + end + + def build_bundle_dir(version) # rubocop:disable Metrics/MethodLength + dir = Dir.mktmpdir("registry-store-#{version}") + config_path = File.join(dir, 'configs', 'example.com', 'feed.yml') + FileUtils.mkdir_p(File.dirname(config_path)) + File.write(config_path, "channel:\n url: https://example.com/\n") + digest = Digest::SHA256.file(config_path).hexdigest + File.write( + File.join(dir, Html2rss::Registry::Manifest::MANIFEST_FILE), + { + format: 'registry.v1', + registry_id: 'store-test', + version:, + public_key_id: 'test', + files: { 'configs/example.com/feed.yml' => digest } + }.to_json + ) + dir + end + + def read_manifest_version(path) + JSON.parse(File.read(File.join(path, Html2rss::Registry::Manifest::MANIFEST_FILE))).fetch('version') + end +end diff --git a/spec/html2rss/web/registry/sync_spec.rb b/spec/html2rss/web/registry/sync_spec.rb new file mode 100644 index 000000000..d0ac20012 --- /dev/null +++ b/spec/html2rss/web/registry/sync_spec.rb @@ -0,0 +1,259 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'climate_control' +require 'spec_helper' +require 'webmock/rspec' + +require_relative '../../../../app' + +RSpec.describe Html2rss::Web::Registry::Sync do + describe '.sync_url_for' do + it 'resolves the official release URL for sync-mode defaults' do + ClimateControl.modify('REGISTRIES_CONFIG' => nil) do + Html2rss::Web::Registry::Index.reload! + + expect(described_class.sync_url_for('official')).to eq( + Html2rss::Web::Registry::Config::OFFICIAL_RELEASE_URL + ) + end + end + end + + describe '.cli_exit_code', :registry_sync do + let(:download_url) { 'https://registry.test.example/registry-bundle.tar.gz' } + let(:tarball) { RegistrySyncTestHelpers.build_signed_tarball } + + before do + stub_request(:get, download_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + end + + it 'returns 1 when a sync registry is not yet synced' do + expect(described_class.cli_exit_code).to eq(1) + end + + it 'returns 0 when all registries are synced and healthy' do + described_class.run(registry_id: 'official') + expect(described_class.cli_exit_code).to eq(0) + end + + it 'returns 1 when any registry has last_error' do + described_class.run(registry_id: 'official') + allow(Html2rss::Web::Registry::Store).to receive(:sync_state).and_return( + Html2rss::Web::Registry::Store::SyncState.new(last_error: 'Boom', last_sync_at: nil) + ) + + expect(described_class.cli_exit_code).to eq(1) + end + end + + describe '.run', :registry_sync do + let(:download_url) { 'https://registry.test.example/registry-bundle.tar.gz' } + let(:tarball) { RegistrySyncTestHelpers.build_signed_tarball } + + before do + stub_request(:get, download_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + end + + it 'fetches, verifies, and stores a signed bundle', :aggregate_failures do + status = described_class.run(registry_id: 'official') + + expect(status.version).to eq('test-fixture') + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(true) + expect(Html2rss::Web::Registry::Index.current.config_for('anthropic.com/news')).to include( + channel: hash_including(title: 'Anthropic — News') + ) + end + + it 'supports dry-run verification without swapping the active bundle' do + expect do + described_class.run(registry_id: 'official', dry_run: true) + end.not_to(change { Html2rss::Web::Registry::Store.bundle_present?('official') }) + end + + it 'follows bounded redirects to allowed CDN hosts', :aggregate_failures do + cdn_url = 'https://release-assets.githubusercontent.com/registry-bundle.tar.gz' + stub_request(:get, download_url) + .to_return(status: 302, headers: { 'Location' => cdn_url }) + stub_request(:get, cdn_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + + status = described_class.run(registry_id: 'official') + + expect(status.version).to eq('test-fixture') + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(true) + end + + it 'rejects redirects to disallowed hosts' do + stub_request(:get, download_url) + .to_return(status: 302, headers: { 'Location' => 'https://evil.example/bundle.tar.gz' }) + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /host not allowed/i) + end + + it 'rejects excessive redirect chains' do + (1..6).each do |hop| + from = hop == 1 ? download_url : "#{download_url}?hop=#{hop - 1}" + to = "#{download_url}?hop=#{hop}" + stub_request(:get, from).to_return(status: 302, headers: { 'Location' => to }) + end + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /redirect limit/i) + end + + it 'logs signature verification failures to the security logger' do + allow(Html2rss::Web::SecurityLogger).to receive(:log_registry_signature_failure) + stub_request(:get, download_url).to_return(status: 200, body: 'not-a-tarball') + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError) + + expect(Html2rss::Web::SecurityLogger).not_to have_received(:log_registry_signature_failure) + end + + it 'keeps the previous bundle when sync fails' do + described_class.run(registry_id: 'official') + stub_request(:get, download_url).to_return(status: 500, body: 'fail') + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /HTTP 500/) + + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(true) + end + end + + describe '.run' do + it 'rejects path-mode registries' do + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /path mode/) + end + end + + describe '.status', :registry_sync do + it 'includes sync metadata and last error state' do + row = described_class.status(registry_id: 'official').first + + expect(row).to have_attributes( + id: 'official', + mode: :sync, + sync_url: 'https://registry.test.example/registry-bundle.tar.gz', + staged_version: nil + ) + end + end + + describe 'sync policy', :registry_sync do + let(:download_url) { 'https://registry.test.example/registry-bundle.tar.gz' } + let(:tarball) { RegistrySyncTestHelpers.build_signed_tarball } + let(:policy_config_path) { File.join(Dir.pwd, 'tmp', 'sync-policy-official.yml') } + + before do + FileUtils.mkdir_p(File.dirname(policy_config_path)) + stub_request(:get, download_url) + .to_return(status: 200, body: tarball, headers: { 'Content-Type' => 'application/octet-stream' }) + end + + after do + FileUtils.rm_f(policy_config_path) + end + + def write_policy_config(yaml) + File.write(policy_config_path, yaml) + ENV['REGISTRIES_CONFIG'] = policy_config_path + Html2rss::Web::Registry::Index.reload! + end + + it 'stages verified bundles when auto_promote is false', :aggregate_failures do + write_policy_config( + RegistrySyncTestHelpers.policy_registry_yaml( + download_url:, + auto_promote: false, + sync_extra: { max_version: 'test-fixture' } + ) + ) + + status = described_class.run(registry_id: 'official') + + expect(Html2rss::Web::Registry::Store.staged_present?('official')).to be(true) + expect(Html2rss::Web::Registry::Store.bundle_present?('official')).to be(false) + expect(status.staged_version).to eq('test-fixture') + end + + it 'promotes staged bundles and emits catalog change telemetry', :aggregate_failures do + allow(Html2rss::Web::Observability).to receive(:emit) + + write_policy_config( + RegistrySyncTestHelpers.policy_registry_yaml(download_url:, auto_promote: false) + ) + + described_class.run(registry_id: 'official') + status = described_class.promote_staged!(registry_id: 'official') + + expect(status.version).to eq('test-fixture') + expect(Html2rss::Web::Registry::Store.staged_present?('official')).to be(false) + expect(Html2rss::Web::Observability).to have_received(:emit).with( + hash_including(event_name: 'registry.catalog_changed', outcome: 'success') + ) + end + + it 'rejects manifests newer than max_version' do + write_policy_config( + RegistrySyncTestHelpers.policy_registry_yaml( + download_url:, + auto_promote: true, + sync_extra: { max_version: '0.0.1' } + ) + ) + + expect { described_class.run(registry_id: 'official') } + .to raise_error(Html2rss::Web::Registry::Errors::SyncError, /exceeds max_version/) + end + end + + describe '.boot!' do + it 'does not run in the test environment' do + allow(described_class).to receive(:start_background_timer!) + + described_class.boot! + + expect(described_class).not_to have_received(:start_background_timer!) + end + end + + describe Html2rss::Web::Registry::ChannelResolver do + describe '.resolve' do + it 'uses the GitHub tag release API when pin_version is set', :aggregate_failures do # rubocop:disable RSpec/ExampleLength + tag_api = format( + described_class::OFFICIAL_GITHUB_TAG_RELEASES_API, + tag: 'v2026.08.22' + ) + download_url = 'https://release-assets.githubusercontent.com/registry-bundle.tar.gz' + stub_request(:get, tag_api).to_return( + status: 200, + body: { + assets: [{ name: described_class::OFFICIAL_ASSET_NAME, browser_download_url: download_url }] + }.to_json + ) + + definition = Html2rss::Web::Registry::Definition.new( + id: 'official', + mode: :sync, + path: nil, + sync_channel: Html2rss::Web::Registry::Config::DEFAULT_OFFICIAL_SYNC_CHANNEL, + sync_url: nil, + catalog: true, + public_key_id: 'html2rss:registry:2026', + public_key: nil, + sync_policy: Html2rss::Web::Registry::SyncPolicy.new('v2026.08.22', nil, false), + allowed_channel_domains: [] + ) + + expect(described_class.resolve(definition)).to eq(download_url) + end + end + end +end diff --git a/spec/smoke/docker_spec.rb b/spec/smoke/docker_spec.rb index 115d2b76b..c056a2624 100644 --- a/spec/smoke/docker_spec.rb +++ b/spec/smoke/docker_spec.rb @@ -112,4 +112,13 @@ def expect_json_feed_response(path) expect(body.dig('error', 'code')).to eq('FORBIDDEN') expect(body.dig('error', 'message')).to eq('Auto source feature is disabled') end + + it 'exposes the config catalog without authentication', :aggregate_failures do + response, payload = get_json('/api/v1/configs') + + expect(response).to be_a(Net::HTTPOK) + expect(payload.fetch('success')).to be(true) + expect(payload.dig('data', 'configs')).to be_an(Array) + expect(payload.dig('meta', 'catalog_version')).to eq(1) + end end diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 4a7811d74..f037e688a 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -8,13 +8,17 @@ require 'simplecov' SimpleCov.start do - add_filter '/spec/' - add_filter '/config/' + enable_coverage :branch + primary_coverage :branch - track_files '**/*.rb' + add_group 'App', 'app' + add_group 'Config', 'config' - minimum_coverage 80 unless ENV['OPENAPI'] - maximum_coverage_drop 5 unless ENV['OPENAPI'] + add_filter %r{/spec/} + add_filter %r{/config/} + + minimum_coverage line: 80, branch: 70 unless ENV['OPENAPI'] + maximum_coverage_drop line: 5, branch: 5 unless ENV['OPENAPI'] end end @@ -52,7 +56,6 @@ Html2rss::Web::SecurityLogger.reset_logger! if defined?(Html2rss::Web::SecurityLogger) Html2rss::Web::RuntimeEnv.reset! if defined?(Html2rss::Web::RuntimeEnv) Html2rss::Web::AccountManager.reload! if defined?(Html2rss::Web::AccountManager) - Html2rss::Web::LocalConfig.reload! if defined?(Html2rss::Web::LocalConfig) end config.after do diff --git a/spec/support/openapi.rb b/spec/support/openapi.rb index 30ea6e7e1..766d9159a 100644 --- a/spec/support/openapi.rb +++ b/spec/support/openapi.rb @@ -233,5 +233,21 @@ else spec[:tags] = tags end + + paths = spec['paths'] || spec[:paths] + catalog_configs_required = paths&.dig('/configs', 'get', 'responses', '200', 'content', + 'application/json', 'schema', 'properties', 'data', 'properties', 'configs', + 'items', 'required') + catalog_configs_required&.delete('registry') if catalog_configs_required.is_a?(Array) + + root_registries = paths&.dig('/', 'get', 'responses', '200', 'content', + 'application/json', 'schema', 'properties', 'data', 'properties', 'instance', + 'properties', 'registries', 'items') + if root_registries.is_a?(Hash) + root_registries['properties']&.fetch('version', {})&.merge!('nullable' => true) + root_registries['properties']&.fetch('updated_at', {})&.merge!('nullable' => true) + root_registries['required']&.delete('version') + root_registries['required']&.delete('updated_at') + end end end diff --git a/spec/support/registry_sync.rb b/spec/support/registry_sync.rb new file mode 100644 index 000000000..dd19dd05f --- /dev/null +++ b/spec/support/registry_sync.rb @@ -0,0 +1,101 @@ +# frozen_string_literal: true + +require 'fileutils' +require 'stringio' +require 'yaml' + +module RegistryTestHelpers + FIXTURES_ROOT = File.expand_path('../fixtures/registries', __dir__) + DEFAULT_REGISTRIES_CONFIG = File.join(FIXTURES_ROOT, 'registries.yml') + + module_function + + def configure_registry_fixtures! + ENV['REGISTRIES_CONFIG'] = DEFAULT_REGISTRIES_CONFIG + ENV['REGISTRY_DATA_ROOT'] = File.join(Dir.pwd, 'tmp', 'test-registry-data') + FileUtils.rm_rf(ENV.fetch('REGISTRY_DATA_ROOT', nil)) + end + + def reset_registry! + Html2rss::Web::Registry::Index.reload! + end +end + +module RegistrySyncTestHelpers + FIXTURE_KEYS_ROOT = File.expand_path('../fixtures/registries/keys', __dir__) + TEST_PUBLIC_KEY = File.read(File.join(FIXTURE_KEYS_ROOT, 'test-key.pub')) + TEST_PRIVATE_KEY = File.read(File.join(FIXTURE_KEYS_ROOT, 'test-key.pem')) + SYNC_FIXTURES_ROOT = File.expand_path('../fixtures/registries/sync', __dir__) + OFFICIAL_FIXTURES_ROOT = File.join(RegistryTestHelpers::FIXTURES_ROOT, 'official') + SYNC_REGISTRIES_CONFIG = File.join(SYNC_FIXTURES_ROOT, 'registries.yml') + SYNC_DATA_ROOT = File.join(Dir.pwd, 'tmp', 'sync-registry-data') + + module_function + + def configure_sync_registry! + ENV['REGISTRIES_CONFIG'] = SYNC_REGISTRIES_CONFIG + ENV['REGISTRY_DATA_ROOT'] = SYNC_DATA_ROOT + ENV['REGISTRY_SYNC_ALLOWED_HOSTS'] = 'registry.test.example,release-assets.githubusercontent.com' + FileUtils.rm_rf(SYNC_DATA_ROOT) + Html2rss::Web::Registry::Index.reload! + end + + def build_signed_tarball # rubocop:disable Metrics/MethodLength + bundle_dir = Dir.mktmpdir('signed-registry-bundle') + manifest_target = File.join(bundle_dir, Html2rss::Registry::Manifest::MANIFEST_FILE) + FileUtils.cp_r(File.join(OFFICIAL_FIXTURES_ROOT, 'configs'), File.join(bundle_dir, 'configs')) + FileUtils.cp( + File.join(SYNC_FIXTURES_ROOT, 'bundle', Html2rss::Registry::Manifest::MANIFEST_FILE), + manifest_target + ) + manifest = Html2rss::Registry::Manifest.parse(File.read(manifest_target)) + Html2rss::Registry::Signer.sign!(manifest, key_pem: TEST_PRIVATE_KEY, bundle_dir:) + + pack_bundle_dir(bundle_dir) + ensure + FileUtils.rm_rf(bundle_dir) + end + + def pack_bundle_dir(bundle_dir) + dir = File.dirname(tarball_path = File.join(Dir.mktmpdir('registry-sync-tarball'), 'bundle.tar.gz')) + env = { 'COPYFILE_DISABLE' => '1' } + # BusyBox tar (Alpine Dev Container) has no --format=ustar; GNU tar accepts omitting it. + success = system(env, 'tar', '-czf', tarball_path, '-C', bundle_dir, '.', exception: false) + raise "Failed to pack registry test bundle from #{bundle_dir}" unless success + + File.binread(tarball_path) + ensure + FileUtils.rm_rf(dir) if dir + end + + def policy_registry_yaml(download_url:, auto_promote:, sync_extra: {}) # rubocop:disable Metrics/MethodLength + sync = { 'url' => download_url }.merge(sync_extra.transform_keys(&:to_s)) + YAML.dump( + { + 'precedence' => ['official'], + 'registries' => { + 'official' => { + 'sync' => sync, + 'auto_promote' => auto_promote, + 'catalog' => true, + 'public_key_id' => 'test-key', + 'public_key' => TEST_PUBLIC_KEY + } + } + } + ) + end +end + +RSpec.configure do |config| + config.before do + RegistryTestHelpers.configure_registry_fixtures! + RegistryTestHelpers.reset_registry! + end + + config.before do |example| + next unless example.metadata[:registry_sync] + + RegistrySyncTestHelpers.configure_sync_registry! + end +end