From 46abcf77e02815667ffd87c294efff1c041191e2 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 20 Sep 2026 11:33:14 +0200 Subject: [PATCH 1/2] docs(web): document selector refinement recovery --- .../guides/refine-selectors.mdx | 29 +++++++++++ .../reference/env-variables.mdx | 51 ++++++++++--------- 2 files changed, 55 insertions(+), 25 deletions(-) create mode 100644 src/content/docs/web-application/guides/refine-selectors.mdx diff --git a/src/content/docs/web-application/guides/refine-selectors.mdx b/src/content/docs/web-application/guides/refine-selectors.mdx new file mode 100644 index 00000000..983d35f8 --- /dev/null +++ b/src/content/docs/web-application/guides/refine-selectors.mdx @@ -0,0 +1,29 @@ +--- +title: "Refine a generated feed" +description: "Recover a generated feed when it finds the wrong list, misses fields, or starts from the wrong page." +--- + +Use **Refine selectors** after automatic feed generation when the result is empty, follows the wrong list of links, or misses the title, link, or published date. Refinement is a recovery step for one generated feed, not a YAML authoring workflow. + +## Before You Start + +Your `html2rss-web` instance must have `STUDIO_ENABLED=true`, and you must have entered a valid access token. Live suggestions and previews also require `AUTO_SOURCE_ENABLED=true`. + +If refinement is unavailable, the result page does not show the action. Opening an old refinement link on an instance where the studio is disabled returns you to feed creation. + +## Refine the Result + +1. Generate a feed from a listing page and open **Refine selectors** from the result. +2. Check the page URL at the top. If you started from the wrong listing, edit it first; the studio requests a new suggestion for that page. +3. Check **Items selector**. It should identify the repeated article or entry container, such as `article` or `.post`. +4. Use the suggested selector when it identifies the correct list. Enable enhanced extraction only when the page needs additional semantic field detection. +5. Open a missing field to set a selector for its title, link, or published date. +6. Check the sample items, then choose **Save and generate**. + +The result is successful when the sample contains the intended entries and the generated feed preserves their useful title and link. + +## Keep the Recovery Focused + +Prefer changing the listing URL before adding more selectors. A category, newsroom, changelog, or archive page usually gives better results than a generic homepage. + +**View as YAML** shows the current selectors fragment for inspection. It is not a complete custom-feed config and does not publish a Feed Directory entry. For a reviewable, version-controlled scraping rule, use [Creating Custom Feeds](/creating-custom-feeds/). diff --git a/src/content/docs/web-application/reference/env-variables.mdx b/src/content/docs/web-application/reference/env-variables.mdx index 82c9712d..15535d13 100644 --- a/src/content/docs/web-application/reference/env-variables.mdx +++ b/src/content/docs/web-application/reference/env-variables.mdx @@ -5,31 +5,32 @@ description: "Configuration reference for html2rss-web environment variables." ## Supported ENV variables -| Name | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default | -| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted | -| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks | -| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version | -| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit | -| `SENTRY_DSN` | optional; enables Sentry errors/logs when set | -| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass | -| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise | -| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` | -| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` | -| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction | -| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting | -| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window | -| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds | -| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts | -| `PORT` | app listen port; compose uses `4000` | -| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | -| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` | -| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes | -| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | -| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api | -| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose | -| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` | +| Name | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `HTML2RSS_SECRET_KEY` | required in production; development/test gets a temporary default | +| `HTML2RSS_ACCESS_TOKEN` | token for the main feed-creation flow; paste the same value into the web UI when prompted | +| `HEALTH_CHECK_TOKEN` | optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks | +| `BUILD_TAG` | release metadata used in logs; published Docker images set this to the release version | +| `GIT_SHA` | deployed commit metadata used in logs; published Docker images set this to the released commit | +| `SENTRY_DSN` | optional; enables Sentry errors/logs when set | +| `BOTASAURUS_SCRAPER_URL` | optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass | +| `AUTO_SOURCE_ENABLED` | `true` by default in development/test, `false` otherwise | +| `STUDIO_ENABLED` | `true` by default in development/test, `false` otherwise; enables token-holder selector refinement; live preview and suggestions also require `AUTO_SOURCE_ENABLED=true` | +| `ASYNC_FEED_REFRESH_ENABLED` | optional boolean; default `false` | +| `ASYNC_FEED_REFRESH_STALE_FACTOR` | optional integer `>= 1`; default `3` | +| `FEEDS_CACHE_MAX_SIZE` | optional integer `>= 1`; default `500`; maximum number of feeds to cache in memory before eviction | +| `RATE_LIMIT_ENABLED` | optional boolean; default `true` (except `false` in `test`); enables or disables client IP rate limiting | +| `RATE_LIMIT_MAX_REQUESTS` | optional integer `>= 1`; default `60`; maximum requests allowed per IP per window | +| `RATE_LIMIT_WINDOW_SECONDS` | optional integer `>= 1`; default `60`; rate limiter sliding window duration in seconds | +| `RETRY_AFTER_TIMEOUT_SECONDS` | optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts | +| `PORT` | app listen port; compose uses `4000` | +| `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | +| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` | +| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes | +| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose | +| `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` | ## Timeout Chain From 24d2a73e8ce52c945e76281d5dcebe2c9ebf1dad Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Thu, 15 Oct 2026 23:03:00 +0200 Subject: [PATCH 2/2] docs: remove the feeds.yml multi-feed file --- src/content/docs/creating-custom-feeds.mdx | 34 +++++---- .../ruby-gem/guides/managing-feed-configs.mdx | 69 +++++++------------ .../docs/ruby-gem/reference/cli-reference.mdx | 6 +- .../docs/troubleshooting/troubleshooting.mdx | 2 +- .../docs/web-application/deployment.mdx | 15 ++-- src/content/docs/web-application/index.mdx | 2 +- 6 files changed, 59 insertions(+), 69 deletions(-) diff --git a/src/content/docs/creating-custom-feeds.mdx b/src/content/docs/creating-custom-feeds.mdx index 6a0ffcea..2e41f643 100644 --- a/src/content/docs/creating-custom-feeds.mdx +++ b/src/content/docs/creating-custom-feeds.mdx @@ -1,6 +1,6 @@ --- title: "Advanced Feeds (Escape Hatch)" -description: "Write YAML configs when auto-source is not enough; mount feeds.yml; contribute to html2rss-configs." +description: "Write YAML configs when auto-source is not enough; add them with registry add; contribute to html2rss-configs." sidebar: order: 2 --- @@ -12,7 +12,7 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory] ## Recommended workflow @@ -21,15 +21,19 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory] 2. Optionally draft: `html2rss capture https://example.com/articles > your-config.yml` 3. Validate: `html2rss validate your-config.yml` 4. Live-check: `html2rss test your-config.yml`, then ship with `html2rss apply your-config.yml` -5. Mount into `html2rss-web` or contribute to html2rss-configs +5. Add into `html2rss-web` with `registry add`, or contribute to html2rss-configs 6. Keep `strategy: auto`; if results are incomplete, confirm the companion scraper is configured before adding site-specific controls `html2rss feed` is a Thor alias for `apply`. `html2rss auto` aliases `scrape` (one-shot, no YAML). ## Minimal config +One YAML file is one feed. Include `registry.id` when you will add the file to `html2rss-web`: + + +On Docker Compose, copy the file into the container first, then add: + + -Feeds are served as `.rss` / `.json` paths on your instance (for example `/example.com/blog.rss`). +The feed is served as `/{registry.id}.rss` / `.json` on your instance (for example `/example.com/blog.rss`). ## Contribute to the Feed Directory diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index d8d515a2..d12d446a 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -5,47 +5,35 @@ description: "Learn how to manage feed configurations with YAML files. Organize, import { Code } from "@astrojs/starlight/components"; -For easier management, especially when using the CLI or `html2rss-web`, you can store your feed configurations in a YAML file. +For easier management, especially when using the CLI or `html2rss-web`, store each feed configuration in its own YAML file. -## Global and Feed-Specific Configurations +## One file, one feed -You can define global settings that apply to all feeds, and then define individual feed configurations under the `feeds` key. +Each YAML file is a single feed. Put channel settings, headers, selectors, and other options in that file — there is no multi-feed map. +Include a `registry.id` when you will add the file to `html2rss-web` with `registry add`. See [Advanced Feeds](/creating-custom-feeds/). + ## Building Feeds from a YAML File ### Ruby @@ -53,14 +41,10 @@ You can define global settings that apply to all feeds, and then define individu @@ -88,7 +72,7 @@ For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same ` diff --git a/src/content/docs/ruby-gem/reference/cli-reference.mdx b/src/content/docs/ruby-gem/reference/cli-reference.mdx index 9115faaa..c07d8961 100644 --- a/src/content/docs/ruby-gem/reference/cli-reference.mdx +++ b/src/content/docs/ruby-gem/reference/cli-reference.mdx @@ -109,7 +109,6 @@ Loads a YAML config, builds the feed, and prints the RSS XML to stdout. (`html2r -Command: `html2rss apply YAML_FILE [feed_name]` +Command: `html2rss apply YAML_FILE` Available options: @@ -209,13 +208,12 @@ Validates a config with the runtime validator without generating a feed. -Command: `html2rss validate YAML_FILE [feed_name]` +Command: `html2rss validate YAML_FILE` Available options: diff --git a/src/content/docs/troubleshooting/troubleshooting.mdx b/src/content/docs/troubleshooting/troubleshooting.mdx index f2cc4b23..c6cbdf5a 100644 --- a/src/content/docs/troubleshooting/troubleshooting.mdx +++ b/src/content/docs/troubleshooting/troubleshooting.mdx @@ -163,7 +163,7 @@ If you are getting a "command not found" error, try the following: - Try a more specific listing, newsroom, changelog, or updates URL before changing infrastructure - If your first-run stack includes Botasaurus, ensure the `botasaurus` service is running -- Check the feed configuration in `feeds.yml` for typos or invalid selectors +- Check your feed YAML (and `registry.id` if you used `registry add`) for typos or invalid selectors - Look for parsing errors in the logs: diff --git a/src/content/docs/web-application/deployment.mdx b/src/content/docs/web-application/deployment.mdx index d32230dd..91cafed2 100644 --- a/src/content/docs/web-application/deployment.mdx +++ b/src/content/docs/web-application/deployment.mdx @@ -79,21 +79,20 @@ Example `.env` beside Compose: After `docker compose up -d`, check `docker compose logs caddy --tail 20` for certificate issuance. -## Custom feeds mount +## Custom feeds -Auto-source is the default. To add curated YAML feeds, bind-mount `feeds.yml` (see [Advanced Feeds](/creating-custom-feeds/)): +Auto-source is the default. To add curated YAML feeds, use `html2rss-web registry add` into the built-in `local` registry (see [Advanced Feeds](/creating-custom-feeds/)): +Each YAML file is one feed and must include `registry.id`. Feeds are served at `/{registry.id}.rss` on your instance. + ## Operate - `GET /api/v1/health/ready` for readiness diff --git a/src/content/docs/web-application/index.mdx b/src/content/docs/web-application/index.mdx index 31b1ab1a..b9f57e29 100644 --- a/src/content/docs/web-application/index.mdx +++ b/src/content/docs/web-application/index.mdx @@ -13,7 +13,7 @@ sidebar: - Web UI for page-URL auto-source - Access-controlled create API (`HTML2RSS_ACCESS_TOKEN`) - Feed Directory catalog from your instance (`GET /api/v1/configs`) -- Optional custom YAML via `feeds.yml` mount +- Optional custom YAML via `html2rss-web registry add` (built-in local registry) - OpenAPI at `/openapi.yaml` The extraction engine is the [`html2rss`](https://github.com/html2rss/html2rss) gem.