diff --git a/.agents/skills/html2rss-config/reference/curation-verbs.md b/.agents/skills/html2rss-config/reference/curation-verbs.md index 2bf7a83b..aa73fce5 100644 --- a/.agents/skills/html2rss-config/reference/curation-verbs.md +++ b/.agents/skills/html2rss-config/reference/curation-verbs.md @@ -32,8 +32,8 @@ Batch: `batch_inspect`, `batch_recon`, `batch_scrape`. CLI aliases: `feed` → ` | Tool | `strategy: auto` behavior | | --- | --- | -| `scrape`, `capture` | Faraday → Botasaurus fallback — **do not** retry explicit `faraday` after `auto` | -| `inspect` | Faraday only (cheap); pin `botasaurus` when you need browser rendering | +| `scrape`, `capture` | Default (HTTPX) → Botasaurus fallback — **do not** retry explicit `default` after `auto` | +| `inspect` | Default only (cheap); pin `botasaurus` when you need browser rendering | ## Configs-repo note diff --git a/.agents/skills/html2rss-config/reference/pitfalls.md b/.agents/skills/html2rss-config/reference/pitfalls.md index b703af5d..7734ebf6 100644 --- a/.agents/skills/html2rss-config/reference/pitfalls.md +++ b/.agents/skills/html2rss-config/reference/pitfalls.md @@ -19,14 +19,14 @@ Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../ - Run `probe_rss` / `batch_recon` on the same URL as `channel.url`, not just the homepage. - Exit `3` / ledger `DEFER` → native RSS found; defer unless curated value is clearly higher. -## Faraday vs JS +## Static HTTP vs JS - `html2rss scrape` returning 0 items ≠ empty HTML — fetch HTML before declaring Botasaurus-only. - Prefer nested title selectors (`h6`, `aria-label`) over whole-card text. ## Redirects and schemes -- Faraday rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`. +- Default HTTP strategy rejects HTTPS→HTTP downgrades; confirm with `curl -I -L`. - If only stable surface downgrades: try Botasaurus once, then drop. ## Botasaurus @@ -39,7 +39,7 @@ Invariants for `new` / `repair` / batch campaigns. Quality gate: [AGENTS.md](../ ## MCP / CLI drift - Cursor catalog stale after gem upgrade: old `*_url` tool names, `-32602` on calls, new `inspect` “not found” — reload MCP/Cursor; use CLI until catalog shows bare verbs ([curation-verbs.md](curation-verbs.md)). -- `scrape` with `auto`: do not retry explicit `faraday` after empty auto (chain already ran). +- `scrape` with `auto`: do not retry explicit `default` after empty auto (chain already ran). ## Selectors diff --git a/.agents/skills/html2rss-config/reference/repair.md b/.agents/skills/html2rss-config/reference/repair.md index 888a858a..0f1864cc 100644 --- a/.agents/skills/html2rss-config/reference/repair.md +++ b/.agents/skills/html2rss-config/reference/repair.md @@ -6,7 +6,7 @@ Fix one existing config. Runtime debugging: [AGENTS.md](../../../../AGENTS.md) 1. Read YAML: `channel.url`, selectors, `strategy`. 2. `scripts/check_config ` (or `html2rss validate` + `apply`). -3. Zero items on Faraday → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`. +3. Zero items on default strategy → Botasaurus (`check_config … --botasaurus`); if works → `strategy: botasaurus` + `register_botasaurus`. 4. Still wrong → Chrome MCP snapshot; confirm item boundary and post-redirect URL. 5. CLI `apply` OK but fetch spec fails → request-strategy mismatch; prefer Botasaurus or drop. 6. Botasaurus 422/504 → [pitfalls.md](pitfalls.md). diff --git a/.agents/skills/html2rss-config/scripts/add_config b/.agents/skills/html2rss-config/scripts/add_config index 9fddd6a9..4708fc0a 100755 --- a/.agents/skills/html2rss-config/scripts/add_config +++ b/.agents/skills/html2rss-config/scripts/add_config @@ -6,8 +6,8 @@ require 'uri' require 'fileutils' require 'json' require 'nokogiri' -require 'faraday' -require 'faraday/follow_redirects' +require 'httpx' +require 'httpx/plugins/follow_redirects' require 'public_suffix' require_relative 'html2rss_api' @@ -163,15 +163,18 @@ def infer_topics(url, title) end def fetch_html(url) - conn = Faraday.new(ssl: { verify: false }, request: { open_timeout: 4, timeout: 8 }) do |f| - f.response :follow_redirects, limit: 3 - f.adapter Faraday.default_adapter - end - res = conn.get(url, {}, { - 'User-Agent' => USER_AGENT, - 'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', - 'Accept-Language' => 'en-US,en;q=0.9,de;q=0.8' + session = HTTPX.plugin(:follow_redirects).with( + ssl: { verify_mode: OpenSSL::SSL::VERIFY_NONE }, + timeout: { connect_timeout: 4, request_timeout: 8 }, + max_redirects: 3 + ) + res = session.get(url, headers: { + 'user-agent' => USER_AGENT, + 'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', + 'accept-language' => 'en-US,en;q=0.9,de;q=0.8' }) + return ['', 0] if res.is_a?(HTTPX::ErrorResponse) + [res.body.to_s, res.status] rescue StandardError ['', 0] diff --git a/AGENTS.md b/AGENTS.md index e4dd3e13..01d42de0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,11 +79,11 @@ Prefer `user-html2rss` when the tool catalog shows **bare verbs** (`inspect`, `c **After gem upgrade:** If old names (`inspect_url`, `capture_config`) still appear in Cursor or calls return `-32602`, the server may be fine — **reload MCP / Cursor** until the catalog matches. Until then, use CLI (`html2rss …` from `../html2rss`) or skill scripts — authoritative fallback. -**Strategy:** `scrape`/`capture` + `auto` run Faraday → Botasaurus (don’t retry `faraday` after `auto`). `inspect` + `auto` uses Faraday only; pin `botasaurus` for JS-heavy inspect. +**Strategy:** `scrape`/`capture` + `auto` run default (HTTPX) → Botasaurus (don’t retry `default` after `auto`). `inspect` + `auto` uses default only; pin `botasaurus` for JS-heavy inspect. ## Chrome MCP -Use Chrome MCP when the static HTML is unclear, the page is hydrated, or Faraday fetch returns zero items while the browser shows a valid list. +Use Chrome MCP when the static HTML is unclear, the page is hydrated, or default fetch returns zero items while the browser shows a valid list. Recommended sequence: @@ -105,7 +105,7 @@ If Chrome MCP is unavailable (`Transport closed` or page-lock errors), do this r Use Botasaurus when: - the page is JS-rendered -- Faraday fetch returns zero items but Chrome shows a valid repeated list +- default fetch returns zero items but Chrome shows a valid repeated list - the site is bot-sensitive enough that static fetch is unreliable Local Botasaurus notes: @@ -132,7 +132,7 @@ Assume the `html2rss` CLI is available on `PATH` when working against the siblin 6. Tighten selectors until the feed output is clean. 7. Run repo validation and non-fetch tests. 8. Run the appropriate fetch lane: - - plain fetch for static or Faraday-backed configs + - plain fetch for static or default-backed configs - Botasaurus fetch for JS-heavy or Botasaurus-backed configs ## Quality Gate @@ -175,7 +175,7 @@ make test 6. Focused fetch verification: -- Faraday-backed candidate: +- Default-backed candidate: ```bash bundle exec rspec --tag fetch --example 'example.com/feed.yml' spec/html2rss/configs_dynamic_spec.rb @@ -219,7 +219,7 @@ Use the core CLI as the authority for single-config debugging. The quickest loop 4. adjust selectors 5. rerun -If Botasaurus works but Faraday does not, keep the config narrow and classify it as Botasaurus-backed instead of trying to rescue it with brittle tweaks. +If Botasaurus works but default strategy does not, keep the config narrow and classify it as Botasaurus-backed instead of trying to rescue it with brittle tweaks. Additional high-value checks: @@ -240,7 +240,7 @@ html2rss scrape 'https://example.com' Use it to: - discover likely repeated item selectors -- compare Faraday and Botasaurus behavior quickly +- compare default and Botasaurus behavior quickly - decide whether a site belongs in the curated set at all Do not ship raw scrape output without manual tightening into a curated config. diff --git a/Gemfile b/Gemfile index 038f3d6b..9d1d3033 100644 --- a/Gemfile +++ b/Gemfile @@ -4,7 +4,7 @@ source 'https://rubygems.org' git_source(:github) { |repo_name| "https://github.com/#{repo_name}" } -gem 'html2rss', github: 'html2rss/html2rss', ref: '9ac61d13' +gem 'html2rss', github: 'html2rss/html2rss', ref: '4260c842' group :development do gem 'nokogiri' diff --git a/Gemfile.lock b/Gemfile.lock index 16de031f..0d965c4e 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,15 +1,13 @@ GIT remote: https://github.com/html2rss/html2rss - revision: 9ac61d13b96ff86cf2a843ba25e35584134696e1 - ref: 9ac61d13 + revision: 4260c842a5632fbef712cf3bee3a4fddcfee3654 + ref: 4260c842 specs: - html2rss (0.28.0) + html2rss (0.29.1) addressable (~> 2.7) brotli dry-validation - faraday (> 2.0.1, < 3.0) - faraday-follow_redirects - faraday-gzip (~> 3) + httpx (~> 1.8) kramdown mcp (~> 1.2) mime-types (> 3.0) @@ -17,7 +15,6 @@ GIT rack (~> 3.0) rackup (~> 2.0) regexp_parser - reverse_markdown (~> 3.0) rss sanitize thor @@ -77,18 +74,10 @@ GEM dry-initializer (~> 3.2) dry-schema (~> 1.14) zeitwerk (~> 2.6) - faraday (2.14.3) - faraday-net_http (>= 2.0, < 3.5) - json - logger - faraday-follow_redirects (0.5.0) - faraday (>= 1, < 3) - faraday-gzip (3.1.0) - faraday (>= 2.0, < 3) - zlib (~> 3.0) - faraday-net_http (3.4.4) - net-http (~> 0.5) hana (1.3.7) + http-2 (1.2.2) + httpx (1.8.3) + http-2 (>= 1.2.0) json (2.21.2) json_schemer (2.5.0) bigdecimal @@ -106,8 +95,6 @@ GEM logger mime-types-data (~> 3.2025, >= 3.2025.0507) mime-types-data (3.2026.0701) - net-http (0.9.1) - uri (>= 0.11.1) nokogiri (1.19.4-arm64-darwin) racc (~> 1.4) nokogiri (1.19.4-x86_64-darwin) @@ -126,8 +113,6 @@ GEM rack (>= 3) rainbow (3.1.1) regexp_parser (2.12.0) - reverse_markdown (3.0.2) - nokogiri rexml (3.4.4) rspec (3.13.2) rspec-core (~> 3.13.0) @@ -174,10 +159,8 @@ GEM tzinfo (2.0.6) concurrent-ruby (~> 1.0) unicode-display_width (2.6.0) - uri (1.1.1) webrick (1.9.2) zeitwerk (2.8.3) - zlib (3.2.3) PLATFORMS arm64-darwin-23 diff --git a/lib/html2rss/configs/asml.com/press-releases.yml b/lib/html2rss/configs/asml.com/press-releases.yml index 5dcc32fe..5628d61d 100644 --- a/lib/html2rss/configs/asml.com/press-releases.yml +++ b/lib/html2rss/configs/asml.com/press-releases.yml @@ -16,6 +16,6 @@ selectors: selector: 'a.related-content-card[href*="/en/news/press-releases/20"]' enhance: false title: - selector: span.related-content-card-title + selector: .related-content-card-title url: extractor: href diff --git a/spec/bin/validate_configs_spec.rb b/spec/bin/validate_configs_spec.rb index cb8aa075..a7537ea9 100644 --- a/spec/bin/validate_configs_spec.rb +++ b/spec/bin/validate_configs_spec.rb @@ -20,16 +20,28 @@ end end - def with_temp_config + it 'fails validation when config violates schema' do + with_temp_config('channel: {}') do |dir| + expect { run_script(dir) }.to raise_error(an_instance_of(SystemExit).and(having_attributes(status: 1))) + end + end + + it 'fails validation when YAML is malformed' do + with_temp_config('invalid: [yaml: {') do |dir| + expect { run_script(dir) }.to raise_error(an_instance_of(SystemExit).and(having_attributes(status: 1))) + end + end + + def with_temp_config(content = valid_config) Dir.mktmpdir do |dir| - write_config(dir) + write_config(dir, content) yield dir end end - def write_config(dir) + def write_config(dir, content = valid_config) FileUtils.mkdir_p(File.join(dir, File.dirname(config_path))) - File.write(File.join(dir, config_path), valid_config) + File.write(File.join(dir, config_path), content) end def run_script(dir)