Skip to content
softwarecorporationPublic

About

Provider-neutral async search client + crate for agents.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

145 Commits

Folders and files

Repository files navigation

Darash

Darash is an all-in-one research crate: provider-neutral async web search, page fetching, and HTML extraction in one dependency, with no API keys. Search runs on a small in-process multi-source backend by default and can also query a remote SearxNG endpoint; fetch and extraction turn HTTP(S) pages into markdown, outlines, selector matches, rows, and tables — local, deterministic, and shaped for a context window.

External SearxNG

The official SearxNG Docker Compose setup is the quickest local instance. It requires Docker with Compose:

mkdir -p searxng/core-config
cd searxng
curl -fsSL \
  -O https://raw.githubusercontent.com/searxng/searxng/master/container/docker-compose.yml \
  -O https://raw.githubusercontent.com/searxng/searxng/master/container/.env.example
cp .env.example .env
docker compose up -d

SearxNG is then available at http://localhost:8080. Stop it with docker compose down. See the SearxNG container documentation for configuration and maintenance.

Use the crate

Add Darash from crates.io:

[dependencies]
darash = "0.7.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Create a client, build a query, and search asynchronously:

use darash::{SearchClient, SearchQuery, SafeSearch, TimeRange};

#[tokio::main]
async fn main() -> Result<(), darash::Error> {
    let client = SearchClient::new("http://localhost:8080")?;
    let query = SearchQuery::new("rust async")
        .with_categories("general,news")
        .with_engines(["brave", "duckduckgo"])
        .with_language("en-US")
        .with_page(1)
        .with_safe_search(SafeSearch::Moderate)
        .with_time_range(TimeRange::Month);

    let response = client.search(&query).await?;
    for citation in response.citations() {
        println!("{} — {}", citation.title, citation.url);
    }
    Ok(())
}

SearchResponse also exposes the raw SearxNG query, result count, results, answers, corrections, and suggestions. Its optional answer is a backend answer when one is supplied; Darash does not call an AI provider. sources and cited_sources() expose the cited Citation values for host-owned synthesis. Each SearchResult includes its title, URL, content, engines, category, publication date, and score.

The Vane-compatible request contract carries a search mode and source selection without adding provider credentials:

use darash::{SearchClient, SearchMode, SearchRequest, SearchSource};

#[tokio::main]
async fn main() -> Result<(), darash::Error> {
    let request = SearchRequest::new("rust async")
        .with_mode(SearchMode::Quality)
        .with_sources([SearchSource::Web, SearchSource::Academic]);
    let response = SearchClient::local()?.search_request(&request).await?;
    for source in response.cited_sources() {
        println!("{}: {}", source.title, source.url);
    }
    Ok(())
}

SearchClient::local() runs Darash's provider adapters directly in the current process. The default backend queries DuckDuckGo, OpenAlex, and Hacker News as needed; it does not start a separate search service. SearchMode supports Speed, Balanced (the default), and Quality. SearchSource supports Web (the default), Academic, and Discussions. JSON requests may omit mode and sources; they default to balanced and web. The host can synthesize an answer from the returned sources with its own model.

The local providers are selected from the requested sources and run concurrently:

  • web queries DuckDuckGo HTML results.
  • academic queries OpenAlex works.
  • discussions queries Hacker News Algolia results.

Each selected provider contributes up to 5 results in speed mode and up to 10 in balanced or quality mode before URL deduplication and relevance ranking. SearchQuery::with_engines selects the local provider names duckduckgo/ddg, openalex, and hacker-news/hn; source categories remain the convenient default selection. SearchResponse::provider_status preserves success and failure information when one provider is unavailable.

SearchResponse::coverage reports which local providers were attempted, skipped by the mode limit, blocked, or unrecognized, and how requested language and time range constraints were handled. forwarded means Darash sent a parameter; it does not prove the provider enforced it. not_forwarded means the adapter did not send it, not_attempted means the provider was skipped, and unknown means forwarding cannot be established. Remote SearxNG responses set providers_known to false because the server does not establish complete per-provider coverage. Websurfx and older serialized responses may have no coverage report, which also means unknown. Coverage on a cache hit describes the original search; an all-provider failure remains an error without a report.

SearchMode limits provider selection and retrieval as well as the returned results and sources: speed selects one provider and returns at most 5 results, balanced selects up to two and returns at most 10, and quality selects all requested providers and returns at most 20. number_of_results is not changed by that cap.

SafeSearch supports levels 0 through 4. Remote requests use the SearxNG values; local DuckDuckGo requests use DuckDuckGo's native values and date-range codes. Configure local level-3/4 filtering with SearchConfig::with_blocklist and with_allowlist. The response exposes the applied level and filtered/disallowed flags in SearchFilters. Local responses use a bounded in-memory TTL cache by default; configure it with with_cache or call clear_cache on the client.

The local backend is a direct in-process adapter. It does not start an HTTP listener, expose a Websurfx server, spawn a search subprocess, or read Websurfx configuration or assets. The dependency-free Websurfx compatibility types and URL builder are available for hosts that already run Websurfx; they do not embed Websurfx or add its AGPL dependency tree. Use SearchClient::search_websurfx when a configured endpoint is a Websurfx server; it maps Websurfx's engine and error metadata into Darash's response model.

Fetch and extract

Search finds sources; darash::fetch reads them. Every fetch returns a FetchReport with the status, final URL, redirect flag, elapsed time, content type, size, and body — never silent, even for error statuses. Bodies are capped at 2 MiB.

The default fetch is a fast HTTP request; it does not execute JavaScript. Main-content extraction is available without a browser or extra dependencies:

use darash::{fetch, main_content::{extract_main_content, MainContentFallback}};

let content = extract_main_content(
    "<nav>Menu</nav><main><h1>Guide</h1><p>Useful content.</p></main>",
    MainContentFallback::Error,
)?;
let markdown = fetch::to_markdown(&content.html);
# Ok::<(), darash::main_content::MainContentError>(())

It prefers <main>/role="main", then article landmarks, then a conservative prose heuristic that compares the deepest qualifying containers, including wrapped paragraphs. It removes navigation, ordinary forms, complementary content, dialogs and explicitly hidden nodes while keeping links, code and tables. The returned source identifies the selection. Unrecognized pages return an error unless you explicitly choose MainContentFallback::FullDocument, which returns the original HTML. to_markdown and to_text continue to render the full document.

For JavaScript pages, enable the optional browser feature and supply an installed Chromium or Google Chrome executable. No browser is downloaded, and no user profile is opened. Browser support currently requires macOS or Linux.

darash = { version = "0.7.0", features = ["browser"] }
use darash::{browser::{fetch_rendered, BrowserOptions}, fetch};

let mut options = BrowserOptions::new("/usr/bin/google-chrome");
options.wait_for = Some("main".into());
let report = fetch_rendered("https://example.com", &options).await?;
let markdown = fetch::to_markdown(&report.body);

Rendering waits for DOM readiness, an optional selector, and 250 ms of unchanged DOM, under one 30-second deadline including startup and requests. For delayed applications, select an element that indicates the content is ready. Arbitrary long-running or continuously changing pages may time out. BrowserOptions exposes the deadline, settling interval, 2 MiB resource/DOM cap, 8 MiB aggregate resource cap, 128-request cap and 10-document-navigation cap. Cancellation and timeouts kill the owned browser process group and remove its temporary profile. FetchReport retains the HTTP status and final URL; browser body is the rendered DOM with computed hidden content annotated for extraction, and bytes is its UTF-8 size. Non-2xx HTTP pages still produce reports with ok == false.

Browser rendering supports unauthenticated public HTTP(S) GET/HEAD resources. Each intercepted resource and redirect is checked for public IP addresses; the HTTP client pins DNS results, disables ambient proxies and never forwards cookies, authorization or user headers. Non-public and mixed public/private DNS results are rejected. A dedicated sink proxy prevents Chromium's other HTTP traffic from contacting destination servers; this is not an OS network sandbox. Subframes, WebSockets, service workers, worker-backed applications, downloads and non-GET/HEAD requests are unsupported. Browser runs bypass the CLI's static-fetch cache. Authentication and bot protections are not bypassed.

cargo install darash --features browser
darash fetch https://example.com --md --main-content
darash fetch https://example.com --md --main-content --main-fallback full
darash fetch https://example.com --render --browser /usr/bin/google-chrome \
  --wait-for main --md --main-content --budget 4000

To validate browser rendering against synthetic fixtures only, explicitly set an installed executable and run the ignored tests. The test-only synthetic host-to-loopback mapping is unavailable in production builds:

DARASH_TEST_BROWSER=/usr/bin/google-chrome cargo test --locked --features browser \
  browser::tests::chromium -- --ignored --test-threads=1
use darash::fetch;

#[tokio::main]
async fn main() -> Result<(), darash::Error> {
    let report = fetch::fetch("https://example.com", &[]).await?;
    println!("{}", report.summary());

    // Agent-shaped output, all local and deterministic:
    let markdown = fetch::to_markdown(&report.body);   // page as markdown
    let outline = fetch::outline(&report.body);        // repeating structures
    let titles = fetch::select_texts(&report.body, "h2")?; // selector matches
    let rows = fetch::rows(&report.body, ".item", "title=h2, url=a@href")?;
    let tables = fetch::tables(&report.body);          // keyed table rows
    let tokens = fetch::estimate_tokens(&markdown);    // ~4 chars per token
    Ok(())
}

apply_budget cuts a list of items at item boundaries to fit an estimated token budget, always keeping at least one item. fetch::fetch(url, headers) is a thin wrapper over fetch::fetch_with(url, FetchOptions { .. }), which carries a method, request body, basic auth, insecure-TLS opt-in, timeout, and body cap. fetch::read_source(path) reads a local file through the same pipeline. fetch::locate(html, text) finds the deepest elements that hold a piece of text and returns a CSS selector path plus a short snippet. fetch::paginate(total, offset, limit) computes the PageMeta state (more, complete, past_end) the CLI uses. The filter module compiles the safe --where expression language, and disk_cache holds the CLI's short-lived on-disk body cache.

CLI

The darash binary ships with the crate (cargo install darash). Search starts the in-process backend by default; pass --url only when using another SearxNG-compatible endpoint:

darash search "rust async"
darash search "rust async" --mode quality --source academic --url http://localhost:9090
darash search "rust async" --json

fetch is the research half of the CLI. Without extraction flags it prints the full report as JSON (including the body); --body prints only the body. With a mode it prints extracted data on stdout and a one-line report on stderr:

darash fetch https://example.com                          # full report, JSON
darash fetch https://example.com --body                   # response body only
darash fetch https://example.com --md                     # page as markdown
darash fetch https://example.com --md --budget 800        # markdown within ~800 tokens
darash fetch https://example.com --outline                # repeating structures
darash fetch https://example.com --select "h2" --limit 10 # selector matches
darash fetch https://example.com --select "h2" --count    # just the match count
darash fetch https://example.com --select ".item" --row "title=h2, url=a@href"
darash fetch https://example.com --table --json           # keyed rows as JSON
darash fetch https://example.com --locate "Example Domain" # which selector holds it
darash fetch page.html --md                               # local files work too

Rows and single tables print as TSV by default (a header line once, then values; tabs and newlines inside a value fold to spaces, and objects become JSON); --json switches them to JSON rows, and multiple tables always print as JSON. --locate TEXT reports each deepest element whose text or an attribute contains TEXT, as a CSS selector path and an 80-character snippet.

Structured output (--select, --row, --table, --locate) is paginated: --limit (default 50) caps a page and --offset starts it. In plain output the CLI announces N more result(s) hidden — continue with --offset … or that the offset is past the end; --json-envelope instead prints {"data": …, "meta": {state, total, offset, returned, next_offset}}, where state is more, complete, or past_end. --where EXPR filters rows and tables with a safe expression language (no eval): comparisons (`== != ~ !~ >

= < <=), &&/||/!, numeric-string coercion, /regex/[flags]`, backtick literal column names, and dotted paths that prefer a literally named column. When a filter matches nothing the CLI says so on stderr.

fetch also accepts the usual curl reflexes: -X/--method, -d/--data (@file/@- strip CR/LF; --data-raw never reads @ as a file; --data-binary keeps bytes), -u user:pass, -I/--head, -o FILE (atomic write), -k/--insecure, -m/--max-time, --max-bytes, and -f/--fail (HTTP error exits 22 after printing the report). -L, -i, -s, and -S are accepted no-ops. Fetched URL bodies are cached for about two minutes under the user's cache directory and reused by parse modes; --fresh refetches and re-caches, --no-cache never reads or writes, and credential-bearing URLs, Basic Auth, custom headers, non-GET methods, and request bodies bypass the cache. Responses with Cache-Control: no-store are not stored. Cache hits announce their age on stderr.

Extraction output is capped at 50 items by default (--limit) and can be further bounded with --budget (estimated tokens, cut at item boundaries); omissions are announced on stderr, never silent. --json wraps any mode as {"data": …, "meta": {status, ok, url, ms, bytes, count, omitted}}.

AI synthesis remains a host responsibility; no MCP server is needed for this in-process tool.

Use SearchConfig when the endpoint needs a custom timeout:

use std::time::Duration;
use darash::{SearchClient, SearchConfig};

fn main() -> Result<(), darash::Error> {
    let config = SearchConfig::new("https://search.example.test")?
        .with_timeout(Duration::from_secs(5));
    let _client = SearchClient::from_config(config)?;
    Ok(())
}

Limits and errors

  • Requests time out after 15 seconds by default; configure this with SearchConfig::with_timeout.
  • Response bodies are capped at 256 KiB, including streamed responses.
  • Queries must contain non-whitespace text, and page numbers start at 1.
  • Endpoints must use http or https and cannot contain embedded credentials.
  • Redirects are disabled. Point the client at the final SearxNG endpoint.
  • Error distinguishes invalid configuration, request failures, non-success HTTP responses, oversized or invalid responses, and JSON decode failures.

Native quality checks

Run these commands from the crate root:

cargo fmt --all -- --check
cargo build --locked
cargo test --locked
cargo clippy --all-targets --all-features -- -D warnings
cargo package --locked

Darash is licensed under the ISC license.

About

Provider-neutral async search client + crate for agents.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages