Darash is an all-in-one research crate: provider-neutral async web search, page fetching, and HTML extraction in one dependency, with no API keys. Search runs on a small in-process multi-source backend by default and can also query a remote SearxNG endpoint; fetch and extraction turn HTTP(S) pages into markdown, outlines, selector matches, rows, and tables — local, deterministic, and shaped for a context window.
The official SearxNG Docker Compose setup is the quickest local instance. It requires Docker with Compose:
mkdir -p searxng/core-config
cd searxng
curl -fsSL \
-O https://raw.githubusercontent.com/searxng/searxng/master/container/docker-compose.yml \
-O https://raw.githubusercontent.com/searxng/searxng/master/container/.env.example
cp .env.example .env
docker compose up -dSearxNG is then available at http://localhost:8080. Stop it with
docker compose down. See the SearxNG container documentation
for configuration and maintenance.
Add Darash from crates.io:
[dependencies]
darash = "0.7.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }Create a client, build a query, and search asynchronously:
use darash::{SearchClient, SearchQuery, SafeSearch, TimeRange};
#[tokio::main]
async fn main() -> Result<(), darash::Error> {
let client = SearchClient::new("http://localhost:8080")?;
let query = SearchQuery::new("rust async")
.with_categories("general,news")
.with_engines(["brave", "duckduckgo"])
.with_language("en-US")
.with_page(1)
.with_safe_search(SafeSearch::Moderate)
.with_time_range(TimeRange::Month);
let response = client.search(&query).await?;
for citation in response.citations() {
println!("{} — {}", citation.title, citation.url);
}
Ok(())
}SearchResponse also exposes the raw SearxNG query, result count, results,
answers, corrections, and suggestions. Its optional answer is a backend
answer when one is supplied; Darash does not call an AI provider. sources and
cited_sources() expose the cited Citation values for host-owned synthesis.
Each SearchResult includes its title, URL, content, engines, category,
publication date, and score.
The Vane-compatible request contract carries a search mode and source selection without adding provider credentials:
use darash::{SearchClient, SearchMode, SearchRequest, SearchSource};
#[tokio::main]
async fn main() -> Result<(), darash::Error> {
let request = SearchRequest::new("rust async")
.with_mode(SearchMode::Quality)
.with_sources([SearchSource::Web, SearchSource::Academic]);
let response = SearchClient::local()?.search_request(&request).await?;
for source in response.cited_sources() {
println!("{}: {}", source.title, source.url);
}
Ok(())
}SearchClient::local() runs Darash's provider adapters directly in the current
process. The default backend queries DuckDuckGo, OpenAlex, and Hacker News as
needed; it does not start a separate search service. SearchMode supports
Speed, Balanced (the default), and Quality.
SearchSource supports Web (the default), Academic, and Discussions.
JSON requests may omit mode and sources; they default to balanced and
web.
The host can synthesize an answer from the returned sources with its own model.
The local providers are selected from the requested sources and run concurrently:
webqueries DuckDuckGo HTML results.academicqueries OpenAlex works.discussionsqueries Hacker News Algolia results.
Each selected provider contributes up to 5 results in speed mode and up to 10
in balanced or quality mode before URL deduplication and relevance ranking.
SearchQuery::with_engines selects the local provider names
duckduckgo/ddg, openalex, and hacker-news/hn; source categories remain
the convenient default selection. SearchResponse::provider_status preserves
success and failure information when one provider is unavailable.
SearchResponse::coverage reports which local providers were attempted, skipped
by the mode limit, blocked, or unrecognized, and how requested language and time
range constraints were handled. forwarded means Darash sent a parameter; it
does not prove the provider enforced it. not_forwarded means the adapter did
not send it, not_attempted means the provider was skipped, and unknown
means forwarding cannot be established. Remote SearxNG responses set
providers_known to false because the server does not establish complete
per-provider coverage. Websurfx and older serialized responses may have no
coverage report, which also means unknown. Coverage on a cache hit describes
the original search; an all-provider failure remains an error without a report.
SearchMode limits provider selection and retrieval as well as the returned
results and sources: speed selects one provider and returns at most 5
results, balanced selects up to two and returns at most 10, and quality selects
all requested providers and returns at most 20. number_of_results is not
changed by that cap.
SafeSearch supports levels 0 through 4. Remote requests use the SearxNG
values; local DuckDuckGo requests use DuckDuckGo's native values and date-range
codes. Configure local level-3/4 filtering with
SearchConfig::with_blocklist and with_allowlist. The response exposes the
applied level and filtered/disallowed flags in SearchFilters. Local
responses use a bounded in-memory TTL cache by default; configure it with
with_cache or call clear_cache on the client.
The local backend is a direct in-process adapter. It does not start an HTTP
listener, expose a Websurfx server, spawn a search subprocess, or read
Websurfx configuration or assets. The dependency-free Websurfx compatibility
types and URL builder are available for hosts that already run Websurfx; they
do not embed Websurfx or add its AGPL dependency tree. Use
SearchClient::search_websurfx when a configured endpoint is a Websurfx
server; it maps Websurfx's engine and error metadata into Darash's response
model.
Search finds sources; darash::fetch reads them. Every fetch returns a
FetchReport with the status, final URL, redirect
flag, elapsed time, content type, size, and body — never silent, even for
error statuses. Bodies are capped at 2 MiB.
The default fetch is a fast HTTP request; it does not execute JavaScript. Main-content extraction is available without a browser or extra dependencies:
use darash::{fetch, main_content::{extract_main_content, MainContentFallback}};
let content = extract_main_content(
"<nav>Menu</nav><main><h1>Guide</h1><p>Useful content.</p></main>",
MainContentFallback::Error,
)?;
let markdown = fetch::to_markdown(&content.html);
# Ok::<(), darash::main_content::MainContentError>(())It prefers <main>/role="main", then article landmarks, then a conservative
prose heuristic that compares the deepest qualifying containers, including
wrapped paragraphs. It removes navigation, ordinary forms, complementary content, dialogs
and explicitly hidden nodes while keeping links, code and tables. The returned
source identifies the selection. Unrecognized pages return an error unless
you explicitly choose MainContentFallback::FullDocument, which returns the
original HTML. to_markdown and to_text continue to render the full document.
For JavaScript pages, enable the optional browser feature and supply an
installed Chromium or Google Chrome executable. No browser is downloaded, and
no user profile is opened. Browser support currently requires macOS or Linux.
darash = { version = "0.7.0", features = ["browser"] }use darash::{browser::{fetch_rendered, BrowserOptions}, fetch};
let mut options = BrowserOptions::new("/usr/bin/google-chrome");
options.wait_for = Some("main".into());
let report = fetch_rendered("https://example.com", &options).await?;
let markdown = fetch::to_markdown(&report.body);Rendering waits for DOM readiness, an optional selector, and 250 ms of unchanged
DOM, under one 30-second deadline including startup and requests. For delayed
applications, select an element that indicates the content is ready. Arbitrary
long-running or continuously changing pages may time out. BrowserOptions
exposes the deadline, settling interval, 2 MiB resource/DOM cap, 8 MiB aggregate
resource cap, 128-request cap and 10-document-navigation cap. Cancellation and
timeouts kill the owned browser process group and remove its temporary profile.
FetchReport retains the HTTP status and final URL; browser body is the
rendered DOM with computed hidden content annotated for extraction, and bytes
is its UTF-8 size. Non-2xx HTTP pages still produce reports with ok == false.
Browser rendering supports unauthenticated public HTTP(S) GET/HEAD resources. Each intercepted resource and redirect is checked for public IP addresses; the HTTP client pins DNS results, disables ambient proxies and never forwards cookies, authorization or user headers. Non-public and mixed public/private DNS results are rejected. A dedicated sink proxy prevents Chromium's other HTTP traffic from contacting destination servers; this is not an OS network sandbox. Subframes, WebSockets, service workers, worker-backed applications, downloads and non-GET/HEAD requests are unsupported. Browser runs bypass the CLI's static-fetch cache. Authentication and bot protections are not bypassed.
cargo install darash --features browser
darash fetch https://example.com --md --main-content
darash fetch https://example.com --md --main-content --main-fallback full
darash fetch https://example.com --render --browser /usr/bin/google-chrome \
--wait-for main --md --main-content --budget 4000To validate browser rendering against synthetic fixtures only, explicitly set an installed executable and run the ignored tests. The test-only synthetic host-to-loopback mapping is unavailable in production builds:
DARASH_TEST_BROWSER=/usr/bin/google-chrome cargo test --locked --features browser \
browser::tests::chromium -- --ignored --test-threads=1use darash::fetch;
#[tokio::main]
async fn main() -> Result<(), darash::Error> {
let report = fetch::fetch("https://example.com", &[]).await?;
println!("{}", report.summary());
// Agent-shaped output, all local and deterministic:
let markdown = fetch::to_markdown(&report.body); // page as markdown
let outline = fetch::outline(&report.body); // repeating structures
let titles = fetch::select_texts(&report.body, "h2")?; // selector matches
let rows = fetch::rows(&report.body, ".item", "title=h2, url=a@href")?;
let tables = fetch::tables(&report.body); // keyed table rows
let tokens = fetch::estimate_tokens(&markdown); // ~4 chars per token
Ok(())
}apply_budget cuts a list of items at item boundaries to fit an estimated
token budget, always keeping at least one item. fetch::fetch(url, headers) is
a thin wrapper over fetch::fetch_with(url, FetchOptions { .. }), which carries
a method, request body, basic auth, insecure-TLS opt-in, timeout, and body cap.
fetch::read_source(path) reads a local file through the same pipeline.
fetch::locate(html, text) finds the deepest elements that hold a piece of text
and returns a CSS selector path plus a short snippet. fetch::paginate(total, offset, limit) computes the PageMeta state (more, complete, past_end)
the CLI uses. The filter module compiles the safe --where expression
language, and disk_cache holds the CLI's short-lived on-disk body cache.
The darash binary ships with the crate (cargo install darash). Search
starts the in-process backend by default; pass --url only when using another
SearxNG-compatible endpoint:
darash search "rust async"
darash search "rust async" --mode quality --source academic --url http://localhost:9090
darash search "rust async" --jsonfetch is the research half of the CLI. Without extraction flags it prints the
full report as JSON (including the body); --body prints only the body. With a
mode it prints extracted data on stdout and a one-line report on stderr:
darash fetch https://example.com # full report, JSON
darash fetch https://example.com --body # response body only
darash fetch https://example.com --md # page as markdown
darash fetch https://example.com --md --budget 800 # markdown within ~800 tokens
darash fetch https://example.com --outline # repeating structures
darash fetch https://example.com --select "h2" --limit 10 # selector matches
darash fetch https://example.com --select "h2" --count # just the match count
darash fetch https://example.com --select ".item" --row "title=h2, url=a@href"
darash fetch https://example.com --table --json # keyed rows as JSON
darash fetch https://example.com --locate "Example Domain" # which selector holds it
darash fetch page.html --md # local files work tooRows and single tables print as TSV by default (a header line once, then values;
tabs and newlines inside a value fold to spaces, and objects become JSON);
--json switches them to JSON rows, and multiple tables always print as JSON.
--locate TEXT reports each deepest element whose text or an attribute contains
TEXT, as a CSS selector path and an 80-character snippet.
Structured output (--select, --row, --table, --locate) is paginated:
--limit (default 50) caps a page and --offset starts it. In plain output the
CLI announces N more result(s) hidden — continue with --offset … or that the
offset is past the end; --json-envelope instead prints
{"data": …, "meta": {state, total, offset, returned, next_offset}}, where
state is more, complete, or past_end. --where EXPR filters rows and
tables with a safe expression language (no eval): comparisons (`== != ~ !~ >
= < <=
),&&/||/!, numeric-string coercion,/regex/[flags]`, backtick literal column names, and dotted paths that prefer a literally named column. When a filter matches nothing the CLI says so on stderr.
fetch also accepts the usual curl reflexes: -X/--method, -d/--data
(@file/@- strip CR/LF; --data-raw never reads @ as a file;
--data-binary keeps bytes), -u user:pass, -I/--head, -o FILE (atomic
write), -k/--insecure, -m/--max-time, --max-bytes, and -f/--fail (HTTP
error exits 22 after printing the report). -L, -i, -s, and -S are
accepted no-ops. Fetched URL bodies are cached for about two minutes under the
user's cache directory and reused by parse modes; --fresh refetches and
re-caches, --no-cache never reads or writes, and credential-bearing URLs,
Basic Auth, custom headers, non-GET methods, and request bodies bypass the
cache. Responses with Cache-Control: no-store are not stored. Cache hits
announce their age on stderr.
Extraction output is capped at 50 items by default (--limit) and can be
further bounded with --budget (estimated tokens, cut at item boundaries);
omissions are announced on stderr, never silent. --json wraps any mode as
{"data": …, "meta": {status, ok, url, ms, bytes, count, omitted}}.
AI synthesis remains a host responsibility; no MCP server is needed for this in-process tool.
Use SearchConfig when the endpoint needs a custom timeout:
use std::time::Duration;
use darash::{SearchClient, SearchConfig};
fn main() -> Result<(), darash::Error> {
let config = SearchConfig::new("https://search.example.test")?
.with_timeout(Duration::from_secs(5));
let _client = SearchClient::from_config(config)?;
Ok(())
}- Requests time out after 15 seconds by default; configure this with
SearchConfig::with_timeout. - Response bodies are capped at 256 KiB, including streamed responses.
- Queries must contain non-whitespace text, and page numbers start at 1.
- Endpoints must use
httporhttpsand cannot contain embedded credentials. - Redirects are disabled. Point the client at the final SearxNG endpoint.
Errordistinguishes invalid configuration, request failures, non-success HTTP responses, oversized or invalid responses, and JSON decode failures.
Run these commands from the crate root:
cargo fmt --all -- --check
cargo build --locked
cargo test --locked
cargo clippy --all-targets --all-features -- -D warnings
cargo package --lockedDarash is licensed under the ISC license.