diff --git a/.agents/skills/html2rss-config/reference/curation-verbs.md b/.agents/skills/html2rss-config/reference/curation-verbs.md
index 2bf7a83b..aa73fce5 100644
--- a/.agents/skills/html2rss-config/reference/curation-verbs.md
+++ b/.agents/skills/html2rss-config/reference/curation-verbs.md
@@ -32,8 +32,8 @@ Batch: `batch_inspect`, `batch_recon`, `batch_scrape`. CLI aliases: `feed` → `
| Tool | `strategy: auto` behavior |
| --- | --- |
-| `scrape`, `capture` | Faraday → Botasaurus fallback — **do not** retry explicit `faraday` after `auto` |
-| `inspect` | Faraday only (cheap); pin `botasaurus` when you need browser rendering |
+| `scrape`, `capture` | Default (HTTPX) → Botasaurus fallback — **do not** retry explicit `default` after `auto` |
+| `inspect` | Default only (cheap); pin `botasaurus` when you need browser rendering |
## Configs-repo note
diff --git a/.agents/skills/html2rss-config/reference/pitfalls.md b/.agents/skills/html2rss-config/reference/pitfalls.md
index b703af5d..7734ebf6 100644
--- a/.agents/skills/html2rss-config/reference/pitfalls.md
+++ b/.agents/skills/html2rss-config/reference/pitfalls.md
@@ -19,14 +19,14 @@ Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../
- Run `probe_rss` / `batch_recon` on the same URL as `channel.url`, not just the homepage.
- Exit `3` / ledger `DEFER` → native RSS found; defer unless curated value is clearly higher.
-## Faraday vs JS
+## Static HTTP vs JS
- `html2rss scrape` returning 0 items ≠ empty HTML — fetch HTML before declaring Botasaurus-only.
- Prefer nested title selectors (`h6`, `aria-label`) over whole-card text.
## Redirects and schemes
-- Faraday rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`.
+- Default HTTP strategy rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`.
- If only stable surface downgrades: try Botasaurus once, then drop.
## Botasaurus
@@ -39,7 +39,7 @@ Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../
## MCP / CLI drift
- Cursor catalog stale after gem upgrade: old `*_url` tool names, `-32602` on calls, new `inspect` “not found” — reload MCP/Cursor; use CLI until catalog shows bare verbs ([curation-verbs.md](curation-verbs.md)).
-- `scrape` with `auto`: do not retry explicit `faraday` after empty auto (chain already ran).
+- `scrape` with `auto`: do not retry explicit `default` after empty auto (chain already ran).
## Selectors
diff --git a/.agents/skills/html2rss-config/reference/repair.md b/.agents/skills/html2rss-config/reference/repair.md
index 888a858a..0f1864cc 100644
--- a/.agents/skills/html2rss-config/reference/repair.md
+++ b/.agents/skills/html2rss-config/reference/repair.md
@@ -6,7 +6,7 @@ Fix one existing config. Runtime debugging: [AGENTS.md](../../../../AGENTS.md)
1. Read YAML: `channel.url`, selectors, `strategy`.
2. `scripts/check_config ` (or `html2rss validate` + `apply`).
-3. Zero items on Faraday → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`.
+3. Zero items on default strategy → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`.
4. Still wrong → Chrome MCP snapshot; confirm item boundary and post-redirect URL.
5. CLI `apply` OK but fetch spec fails → request-strategy mismatch; prefer Botasaurus or drop.
6. Botasaurus 422/504 → [pitfalls.md](pitfalls.md).
diff --git a/.agents/skills/html2rss-config/scripts/add_config b/.agents/skills/html2rss-config/scripts/add_config
index 9fddd6a9..4708fc0a 100755
--- a/.agents/skills/html2rss-config/scripts/add_config
+++ b/.agents/skills/html2rss-config/scripts/add_config
@@ -6,8 +6,8 @@ require 'uri'
require 'fileutils'
require 'json'
require 'nokogiri'
-require 'faraday'
-require 'faraday/follow_redirects'
+require 'httpx'
+require 'httpx/plugins/follow_redirects'
require 'public_suffix'
require_relative 'html2rss_api'
@@ -163,15 +163,18 @@ def infer_topics(url, title)
end
def fetch_html(url)
- conn = Faraday.new(ssl: { verify: false }, request: { open_timeout: 4, timeout: 8 }) do |f|
- f.response :follow_redirects, limit: 3
- f.adapter Faraday.default_adapter
- end
- res = conn.get(url, {}, {
- 'User-Agent' => USER_AGENT,
- 'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
- 'Accept-Language' => 'en-US,en;q=0.9,de;q=0.8'
+ session = HTTPX.plugin(:follow_redirects).with(
+ ssl: { verify_mode: OpenSSL::SSL::VERIFY_NONE },
+ timeout: { connect_timeout: 4, request_timeout: 8 },
+ max_redirects: 3
+ )
+ res = session.get(url, headers: {
+ 'user-agent' => USER_AGENT,
+ 'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+ 'accept-language' => 'en-US,en;q=0.9,de;q=0.8'
})
+ return ['', 0] if res.is_a?(HTTPX::ErrorResponse)
+
[res.body.to_s, res.status]
rescue StandardError
['', 0]
diff --git a/AGENTS.md b/AGENTS.md
index e4dd3e13..01d42de0 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -79,11 +79,11 @@ Prefer `user-html2rss` when the tool catalog shows **bare verbs** (`inspect`, `c
**After gem upgrade:** If old names (`inspect_url`, `capture_config`) still appear in Cursor or calls return `-32602`, the server may be fine — **reload MCP / Cursor** until the catalog matches. Until then, use CLI (`html2rss …` from `../html2rss`) or skill scripts — authoritative fallback.
-**Strategy:** `scrape`/`capture` + `auto` run Faraday → Botasaurus (don’t retry `faraday` after `auto`). `inspect` + `auto` uses Faraday only; pin `botasaurus` for JS-heavy inspect.
+**Strategy:** `scrape`/`capture` + `auto` run default (HTTPX) → Botasaurus (don’t retry `default` after `auto`). `inspect` + `auto` uses default only; pin `botasaurus` for JS-heavy inspect.
## Chrome MCP
-Use Chrome MCP when the static HTML is unclear, the page is hydrated, or Faraday fetch returns zero items while the browser shows a valid list.
+Use Chrome MCP when the static HTML is unclear, the page is hydrated, or default fetch returns zero items while the browser shows a valid list.
Recommended sequence:
@@ -105,7 +105,7 @@ If Chrome MCP is unavailable (`Transport closed` or page-lock errors), do this r
Use Botasaurus when:
- the page is JS-rendered
-- Faraday fetch returns zero items but Chrome shows a valid repeated list
+- default fetch returns zero items but Chrome shows a valid repeated list
- the site is bot-sensitive enough that static fetch is unreliable
Local Botasaurus notes:
@@ -132,7 +132,7 @@ Assume the `html2rss` CLI is available on `PATH` when working against the siblin
6. Tighten selectors until the feed output is clean.
7. Run repo validation and non-fetch tests.
8. Run the appropriate fetch lane:
- - plain fetch for static or Faraday-backed configs
+ - plain fetch for static or default-backed configs
- Botasaurus fetch for JS-heavy or Botasaurus-backed configs
## Quality Gate
@@ -175,7 +175,7 @@ make test
6. Focused fetch verification:
-- Faraday-backed candidate:
+- Default-backed candidate:
```bash
bundle exec rspec --tag fetch --example 'example.com/feed.yml' spec/html2rss/configs_dynamic_spec.rb
@@ -219,7 +219,7 @@ Use the core CLI as the authority for single-config debugging. The quickest loop
4. adjust selectors
5. rerun
-If Botasaurus works but Faraday does not, keep the config narrow and classify it as Botasaurus-backed instead of trying to rescue it with brittle tweaks.
+If Botasaurus works but default strategy does not, keep the config narrow and classify it as Botasaurus-backed instead of trying to rescue it with brittle tweaks.
Additional high-value checks:
@@ -240,7 +240,7 @@ html2rss scrape 'https://example.com'
Use it to:
- discover likely repeated item selectors
-- compare Faraday and Botasaurus behavior quickly
+- compare default and Botasaurus behavior quickly
- decide whether a site belongs in the curated set at all
Do not ship raw scrape output without manual tightening into a curated config.
diff --git a/Gemfile b/Gemfile
index 038f3d6b..9d1d3033 100644
--- a/Gemfile
+++ b/Gemfile
@@ -4,7 +4,7 @@ source 'https://rubygems.org'
git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
-gem 'html2rss', github: 'html2rss/html2rss', ref: '9ac61d13'
+gem 'html2rss', github: 'html2rss/html2rss', ref: '4260c842'
group :development do
gem 'nokogiri'
diff --git a/Gemfile.lock b/Gemfile.lock
index 16de031f..0d965c4e 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -1,15 +1,13 @@
GIT
remote: https://github.com/html2rss/html2rss
- revision: 9ac61d13b96ff86cf2a843ba25e35584134696e1
- ref: 9ac61d13
+ revision: 4260c842a5632fbef712cf3bee3a4fddcfee3654
+ ref: 4260c842
specs:
- html2rss (0.28.0)
+ html2rss (0.29.1)
addressable (~> 2.7)
brotli
dry-validation
- faraday (> 2.0.1, < 3.0)
- faraday-follow_redirects
- faraday-gzip (~> 3)
+ httpx (~> 1.8)
kramdown
mcp (~> 1.2)
mime-types (> 3.0)
@@ -17,7 +15,6 @@ GIT
rack (~> 3.0)
rackup (~> 2.0)
regexp_parser
- reverse_markdown (~> 3.0)
rss
sanitize
thor
@@ -77,18 +74,10 @@ GEM
dry-initializer (~> 3.2)
dry-schema (~> 1.14)
zeitwerk (~> 2.6)
- faraday (2.14.3)
- faraday-net_http (>= 2.0, < 3.5)
- json
- logger
- faraday-follow_redirects (0.5.0)
- faraday (>= 1, < 3)
- faraday-gzip (3.1.0)
- faraday (>= 2.0, < 3)
- zlib (~> 3.0)
- faraday-net_http (3.4.4)
- net-http (~> 0.5)
hana (1.3.7)
+ http-2 (1.2.2)
+ httpx (1.8.3)
+ http-2 (>= 1.2.0)
json (2.21.2)
json_schemer (2.5.0)
bigdecimal
@@ -106,8 +95,6 @@ GEM
logger
mime-types-data (~> 3.2025, >= 3.2025.0507)
mime-types-data (3.2026.0701)
- net-http (0.9.1)
- uri (>= 0.11.1)
nokogiri (1.19.4-arm64-darwin)
racc (~> 1.4)
nokogiri (1.19.4-x86_64-darwin)
@@ -126,8 +113,6 @@ GEM
rack (>= 3)
rainbow (3.1.1)
regexp_parser (2.12.0)
- reverse_markdown (3.0.2)
- nokogiri
rexml (3.4.4)
rspec (3.13.2)
rspec-core (~> 3.13.0)
@@ -174,10 +159,8 @@ GEM
tzinfo (2.0.6)
concurrent-ruby (~> 1.0)
unicode-display_width (2.6.0)
- uri (1.1.1)
webrick (1.9.2)
zeitwerk (2.8.3)
- zlib (3.2.3)
PLATFORMS
arm64-darwin-23
diff --git a/lib/html2rss/configs/asml.com/press-releases.yml b/lib/html2rss/configs/asml.com/press-releases.yml
index 5dcc32fe..5628d61d 100644
--- a/lib/html2rss/configs/asml.com/press-releases.yml
+++ b/lib/html2rss/configs/asml.com/press-releases.yml
@@ -16,6 +16,6 @@ selectors:
selector: 'a.related-content-card[href*="/en/news/press-releases/20"]'
enhance: false
title:
- selector: span.related-content-card-title
+ selector: .related-content-card-title
url:
extractor: href
diff --git a/spec/bin/validate_configs_spec.rb b/spec/bin/validate_configs_spec.rb
index cb8aa075..a7537ea9 100644
--- a/spec/bin/validate_configs_spec.rb
+++ b/spec/bin/validate_configs_spec.rb
@@ -20,16 +20,28 @@
end
end
- def with_temp_config
+ it 'fails validation when config violates schema' do
+ with_temp_config('channel: {}') do |dir|
+ expect { run_script(dir) }.to raise_error(an_instance_of(SystemExit).and(having_attributes(status: 1)))
+ end
+ end
+
+ it 'fails validation when YAML is malformed' do
+ with_temp_config('invalid: [yaml: {') do |dir|
+ expect { run_script(dir) }.to raise_error(an_instance_of(SystemExit).and(having_attributes(status: 1)))
+ end
+ end
+
+ def with_temp_config(content = valid_config)
Dir.mktmpdir do |dir|
- write_config(dir)
+ write_config(dir, content)
yield dir
end
end
- def write_config(dir)
+ def write_config(dir, content = valid_config)
FileUtils.mkdir_p(File.join(dir, File.dirname(config_path)))
- File.write(File.join(dir, config_path), valid_config)
+ File.write(File.join(dir, config_path), content)
end
def run_script(dir)