GopherScraper 🐹

A high-performance, configuration-as-code CLI crawling framework written in Go.

GopherScraper is a versatile command-line tool that allows you to extract structured data from HTML, JSON, XML, and YAML using a single declarative configuration file. By decoupling the scraping logic from the code, you can build scalable and maintainable data pipelines in minutes instead of writing throwaway crawler scripts.

Features ✨

• Multi-Format Resolvers: Built-in parsers for HTML (CSS Selectors), JSON (GJSON Path), XML (XPath), and YAML.
• Configuration-as-Code: Define your scraping tasks and data schemas entirely in YAML.
• High-Performance Scheduler: Concurrent worker-pool design with built-in rate limiting and delay configurations.
• Data Pipeline & Sanitization: Native HTML sanitization (powered by bluemonday) and text normalization out of the box.
• Smart Content-Type Detection: Automatically infers the parser type from HTTP response headers.
• Flexible Exporting: Export extracted results cleanly to stdout or files in JSON or YAML formats.
• Robust Observability: Powered by zerolog, featuring multi-tier verbosity levels (-v, -vv, -vvv) and automatic metadata tracking (elapsed_ms, timestamp).

---

Installation 🚀

Make sure you have Go 1.26+ installed on your system.

git clone https://github.com/j75689/goscraper.git
cd goscraper
go build -o goscraper cmd/goscraper/main.go

# Optional: Move to your bin path
sudo mv goscraper /usr/local/bin/

Quick Start 📖

1. Create a Configuration File

Create a file named config.yaml to define your crawling rules.

global:
  max_concurrency: 5
  headers:
    "User-Agent": "GopherScraper/1.0"

tasks:
  - task_id: "example_html"
    url: "https://example.com"
    parser_type: "html" # Optional: GopherScraper can auto-detect this!
    delay: 500
    fields:
      - name: "title"
        selector: "h1"
        clean_html: true
      - name: "link"
        selector: "a"
        attr: "href"

  - task_id: "httpbin_json"
    url: "https://httpbin.org/get"
    # parser_type omitted to test auto-detection
    fields:
      - name: "origin_ip"
        selector: "origin"

2. Run the Scraper

Execute GopherScraper and output the results directly to the terminal:

goscraper --config config.yaml --output json

Save to a File:

goscraper -c config.yaml -o yaml -f output.yaml

CLI Reference ⚙️

Usage:
  goscraper [flags]

Flags:
  -c, --config string   Path to the YAML configuration file (Required)
  -f, --file string     Output file path (default: stdout)
  -h, --help            help for goscraper
  -o, --output string   Output format (json, yaml) (default "json")
  -t, --task string     Run only a specific task ID
  -v, --verbose count   Enable verbose logging (e.g., -v for info, -vv for debug, -vvv for trace)

Verbosity Levels

• (Default): Silent/Warn/Error. Only outputs task failures or panics. Perfect for cronjobs.
• -v: Info. Outputs task start summaries, target URLs, and parsing success counts.
• -vv: Debug. Outputs worker state changes, concurrency tracking, and HTTP Request Headers.
• -vvv: Trace. Outputs raw HTTP Body (JSON/HTML/XML payloads), Callstacks, and deep internal state.

Configuration Schema 📜

global (Optional)

• max_concurrency (int): Maximum number of concurrent tasks (default: 10).
• headers (map): Global HTTP headers applied to all tasks.

tasks (Required)

• task_id (string): Unique identifier for the task. Injected into the exported data as _meta.task_id.
• url (string): The target URL to fetch.
• parser_type (string): One of html, json, xml, or yaml. If omitted, GopherScraper infers it from the HTTP Content-Type header.
• headers (map): Task-specific headers (overrides global headers).
• delay (int): Rate limiting delay before fetching in milliseconds.
• fields (list): A list of extraction rules.

fields

• name (string): The output key in the generated JSON/YAML result.
• selector (string): The query to extract data.
  • HTML: CSS Selector (h1, .class, #id).
  • JSON/YAML: GJSON Path (data.users.0.name).
  • XML: XPath (//book/author).
• attr (string): (HTML Only) Extract a specific attribute (e.g., href, src). If omitted, extracts the text content.
• clean_html (bool): If true, heavily sanitizes the extracted string, stripping out harmful or unnecessary HTML tags.

Output Format

GopherScraper automatically merges all parsed fields and injects a standard _meta object containing observability data:

[
  {
    "_meta": {
      "elapsed_ms": 321,
      "task_id": "example_html",
      "timestamp": "2026-05-14T13:00:00+08:00"
    },
    "link": "https://iana.org/domains/example",
    "title": "Example Domain"
  }
]

Architecture 🏗️

GopherScraper leverages a fully decoupled architecture:

1. Config Loader: Validates and unmarshals YAML definitions.
2. Scheduler: Manages Goroutine worker pools, channels, delays, and context-based graceful shutdowns.
3. Downloader: Handles context timeouts, HTTP execution, and Header merging.
4. Resolver Dispatcher: A Factory pattern that routes payloads to specific parsers (goquery, gjson, xmlquery).
5. Data Pipeline & Transformer: Normalizes strings, unescapes entities, and runs bluemonday sanitization.
6. Exporter: Serializes the in-memory map arrays into JSON or YAML.

License 📄

This project is licensed under the MIT License.