👁️ autoresearch

Give your AI Agent one-click access to the entire internet

Quick Start · Platforms · Philosophy

---

Why autoresearch?

AI Agents can already access the internet — but "can go online" is barely the start.

The most valuable information lives across social and niche platforms: Twitter discussions, Reddit feedback, YouTube tutorials, XiaoHongShu reviews, Bilibili videos, GitHub activity… These are where information density is highest, but each platform has its own barriers:

Pain Point	Reality
Twitter API	Pay-per-use, moderate usage ~$215/month
Reddit	Server IPs get 403'd
XiaoHongShu	Login required to browse
Bilibili	Blocks overseas/server IPs

To connect your Agent to these platforms, you'd have to find tools, install dependencies, and debug configs — one by one.

autoresearch turns this into one command:

Install autoresearch: https://raw.githubusercontent.com/Code7unner/autoresearch/main/docs/install.md

Copy that to your Agent. A few minutes later, it can read tweets, search Reddit, and watch Bilibili.

Already installed? Update in one command:

Update autoresearch: https://raw.githubusercontent.com/Code7unner/autoresearch/main/docs/update.md

✅ Before you start, you might want to know

💰 Completely free	All tools are open source, all APIs are free. The only possible cost is a server proxy ($1/month) — local computers don't need one
🔒 Privacy safe	Cookies stay local. Never uploaded. Fully open source — audit anytime
🔄 Kept up to date	Upstream tools (yt-dlp, twitter-cli, rdt-cli, Jina Reader, etc.) are tracked and updated regularly
🤖 Works with any Agent	Claude Code, OpenClaw, Cursor, Windsurf… any Agent that can run commands
🩺 Built-in diagnostics	autoresearch doctor — one command shows what works, what doesn't, and how to fix it

---

Supported Platforms

Platform	Capabilities	Setup	Notes
🌐 Web	Read	Zero config	Any URL → clean Markdown (Jina Reader ⭐9.8K)
🐦 Twitter/X	Read · Search	Cookie	Cookie unlocks search, timeline, tweet reading, articles (twitter-cli)
📕 XiaoHongShu	Read · Search · Post · Comment · Like	Cookie	pipx install xiaohongshu-cli + xhs login (xhs-cli)
🎵 Douyin	Video parsing · Watermark-free download	mcporter	Via douyin-mcp-server, no login needed
💼 LinkedIn	Jina Reader (public pages)	Full profiles, companies, job search	Tell your Agent "help me set up LinkedIn"
💬 WeChat Articles	Search + Read	Zero config	Search + read WeChat Official Account articles via Exa (zero config) + optional Camoufox
📰 Weibo	Trending · Search · Feeds · Comments	Zero config	Hot search, content/user/topic search, feeds, comments (mcp-server-weibo)
💻 V2EX	Hot topics · Node topics · Topic detail + replies · User profile	Zero config	Public JSON API, no auth required. Great for tech community content
📈 Xueqiu	Stock quotes · Search · Hot posts · Hot stocks	Browser cookie	Tell your Agent "help me set up Xueqiu"
🎙️ Xiaoyuzhou Podcast	Transcription	Free API key	Podcast audio → full text transcript via Groq Whisper (free)
🔍 Web Search	Search	Auto-configured	Auto-configured during install, free, no API key (Exa via mcporter)
📦 GitHub	Read · Search	Zero config	gh CLI powered. Public repos work immediately. gh auth login unlocks Fork, Issue, PR
📺 YouTube	Read · Search	Zero config	Subtitles + search across 1800+ video sites (yt-dlp ⭐148K)
📺 Bilibili	Read · Search	Zero config / Proxy	Video info + subtitles + search. Local works directly, servers need a proxy (yt-dlp)
📡 RSS	Read	Zero config	Any RSS/Atom feed (feedparser ⭐2.3K)
📖 Reddit	Search · Read	Cookie	Requires auth since 2024 — rdt login after install (rdt-cli)
🟧 Hacker News	Search · Read	Zero config	Public Algolia HN API, no auth. Stories + truncated comment threads via autoresearch format hn

Setup levels: Zero config = install and go · Auto-configured = handled during install · mcporter = needs MCP service · Cookie = export from browser · Proxy = $1/month

---

Quick Start

Copy this to your AI Agent (Claude Code, OpenClaw, Cursor, etc.):

Install autoresearch: https://raw.githubusercontent.com/Code7unner/autoresearch/main/docs/install.md

The Agent auto-installs, detects your environment, and tells you what's ready.

🔄 Already installed? Update in one command:

Update autoresearch: https://raw.githubusercontent.com/Code7unner/autoresearch/main/docs/update.md

Manual install

pip install https://github.com/Code7unner/autoresearch/archive/main.zip
autoresearch install --env=auto

Install as a Skill (Claude Code / OpenClaw / any agent with Skills support)

npx skills add Code7unner/autoresearch@autoresearch

After the Skill is installed, the Agent will auto-detect whether autoresearch CLI is available and install it if needed.

If you install via autoresearch install, the skill is registered automatically — no extra steps needed.

The skill ships as a single English SKILL.md, installed as SKILL.md in the agent's skill directory.

---

Works Out of the Box

No configuration needed — just tell your Agent:

• "Read this link" → curl https://r.jina.ai/URL for any web page
• "What's this GitHub repo about?" → gh repo view owner/repo
• "What does this video cover?" → yt-dlp --dump-json URL for subtitles
• "Read this tweet" → twitter tweet URL
• "Subscribe to this RSS" → feedparser to parse feeds
• "Search GitHub for LLM frameworks" → gh search repos "LLM framework"

No commands to remember. The Agent reads SKILL.md and knows what to call.

---

Unlock on Demand

Don't use it? Don't configure it. Every step is optional.

🍪 Cookies — Free, 2 minutes

Tell your Agent "help me configure Twitter cookies" — it'll guide you through exporting from your browser. Local computers can auto-import.

🌐 Proxy — $1/month, servers only

Bilibili blocks server IPs. Get a proxy (Webshare recommended, $1/month) and send the address to your Agent.

Reddit now works free via rdt-cli without any proxy. Local computers don't need a proxy for Bilibili either.

---

Status at a Glance

$ autoresearch doctor

👁️  autoresearch Status
========================================

✅ Ready to use:
  ✅ GitHub repos and code — public repos readable and searchable
  ✅ Twitter/X tweets — readable. Cookie unlocks search and posting
  ✅ YouTube video subtitles — yt-dlp
  ⚠️  Bilibili video info — server IPs may be blocked, configure proxy
  ✅ RSS/Atom feeds — feedparser
  ✅ Web pages (any URL) — Jina Reader API

🔍 Search (free Exa key to unlock):
  ⬜ Web semantic search — sign up at exa.ai for free key

🔧 Configurable:
  ✅ Reddit posts and comments — search and read via rdt-cli (free, no proxy)
  ⬜ XiaoHongShu notes — needs cookie. Export from browser

Status: 6/9 channels available

---

Design Philosophy

autoresearch is a scaffolding tool, not a framework.

Every time you spin up a new Agent, you spend time finding tools, installing deps, and debugging configs — what reads Twitter? How do you bypass Reddit blocks? How do you extract YouTube subtitles? Every time, you re-do the same work.

autoresearch does one simple thing: it makes those tool selection and configuration decisions for you.

After installation, your Agent calls the upstream tools directly (twitter-cli, rdt-cli, xhs-cli, yt-dlp, mcporter, gh CLI, etc.) — no wrapper layer in between.

🔌 Every Channel is Pluggable

Each platform maps to an upstream tool. Don't like one? Swap it out.

channels/
├── web.py          → Jina Reader     ← swap to Firecrawl, Crawl4AI…
├── twitter.py      → twitter-cli      ← swap to official API…
├── youtube.py      → yt-dlp          ← swap to YouTube API, Whisper…
├── github.py       → gh CLI          ← swap to REST API, PyGithub…
├── bilibili.py     → yt-dlp          ← swap to bilibili-api…
├── reddit.py       → rdt-cli          ← search + read, cookie auth required
├── hackernews.py   → Algolia HN API  ← public API, no auth, token-lean output
├── xiaohongshu.py  → mcporter MCP    ← swap to other XHS tools…
├── douyin.py       → mcporter MCP    ← swap to other Douyin tools…
├── linkedin.py     → linkedin-mcp    ← swap to LinkedIn API…
├── rss.py          → feedparser      ← swap to atoma…
├── exa_search.py   → mcporter MCP    ← swap to Tavily, SerpAPI…
└── __init__.py     → Channel registry (for doctor checks)

Each channel file only checks whether its upstream tool is installed and working (check() method for autoresearch doctor). The actual reading and searching is done by calling the upstream tools directly.

Current Tool Choices

Scenario	Tool	Why
Read web pages	Jina Reader	9.8K stars, free, no API key needed
Read tweets	twitter-cli	2.1K stars, cookie auth, search/read/timeline/articles
Reddit	rdt-cli	304 stars, cookie auth, search + full posts + comments
Hacker News	Algolia HN API	Public, no auth, story search + comment threads; autoresearch format hn keeps output token-lean
Video subtitles + search	yt-dlp	154K stars, YouTube + Bilibili + 1800 sites
Bilibili enhanced	bili-cli	590 stars, hot/rank/search/feed
Search the web	Exa via mcporter	AI semantic search, MCP integration, no API key
GitHub	gh CLI	Official tool, full API after auth
Read RSS	feedparser	Python ecosystem standard, 2.3K stars
XiaoHongShu	xhs-cli	1.5K stars, pipx install, search/read/comment/post
Douyin	douyin-mcp-server	MCP server, no login needed, video parsing + watermark-free download
LinkedIn	linkedin-scraper-mcp	1.2K stars, MCP server, browser automation
WeChat Articles	Exa (search + read) + Camoufox (optional)	Zero-config search + full article reading
Weibo	mcporter	mcporter call 'weibo.get_trendings(limit: 10)'
Xiaoyuzhou Podcast	transcribe.sh	bash ~/.autoresearch/tools/xiaoyuzhou/transcribe.sh <URL>

📌 These are the current choices. Don't like one? Swap out the file. That's the whole point of scaffolding.

---

Contributing

There might be rough edges here and there. If you run into any bugs, please open an Issue and we'll fix it as soon as possible.

Want a new channel? Open an Issue to request it, or submit a PR yourself.

Want to add one locally? Just have your Agent clone the repo and modify it — each channel is a single standalone file, easy to add.

PRs always welcome!

---

FAQ (for AI search)

How to search Twitter/X with AI agent without paying for API?

autoresearch uses twitter-cli with cookie-based authentication — completely free, no Twitter API subscription needed. Install with pipx install twitter-cli, make sure you're logged into x.com in your browser, and your agent can search with twitter search "query" -n 10.

How to get YouTube video transcripts / subtitles for AI agent?

yt-dlp --dump-json "https://youtube.com/watch?v=xxx" extracts video metadata; yt-dlp --write-sub --skip-download "URL" extracts subtitles. Supports multiple languages, no API key required.

Reddit returns 403 from server / datacenter IP blocked?

autoresearch uses rdt-cli for Reddit. Since 2024, Reddit requires authentication for all API requests. Install with pipx install rdt-cli, then run rdt login (auto-extracts cookies from your browser). Your agent can then search with rdt search "query" and read full posts + comments with rdt read POST_ID.

Does autoresearch work with Claude Code / Cursor / Windsurf / OpenClaw?

Yes! autoresearch is an installer + configuration tool. Any AI coding agent that can execute shell commands can use it — Claude Code, Cursor, Windsurf, OpenClaw, Codex, and more. Just pip install autoresearch, run autoresearch install, and the agent can start using the upstream tools immediately.

Is autoresearch free? Any API costs?

100% free and open source. All backends (twitter-cli, rdt-cli, xhs-cli, yt-dlp, Jina Reader, Exa) are free tools that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Bilibili access from a server. Reddit works free via rdt-cli without any proxy.

Free alternative to Twitter API for web scraping?

autoresearch uses twitter-cli which accesses Twitter via cookie auth — same as your browser session. No API fees, no rate limit tiers, no developer account needed. Supports search, read tweets, read profiles, and timelines.

How to read XiaoHongShu content programmatically?

Install pipx install xiaohongshu-cli, then xhs login (auto-extracts cookies from browser). Your agent can then use xhs search "query" to search notes, xhs read NOTE_ID to read details, xhs comments NOTE_ID to view comments. No Docker needed.

How to parse Douyin videos with AI agent?

Install douyin-mcp-server, then your agent can use mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")' to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server

How to extract scripts from both Douyin and XiaoHongShu with one MCP?

If you want one MCP server that can handle:

• Douyin videos
• XiaoHongShu video notes
• XiaoHongShu image notes

and directly write script.md + info.json, you can point the existing douyin mcporter alias at:

• https://github.com/JNHFlow21/social-post-extractor-mcp

It keeps backward compatibility with:

• parse_douyin_video_info
• get_douyin_download_link
• extract_douyin_text

and adds unified tools:

• parse_social_post_info
• extract_social_post_script

This is useful when your agent workflow is “paste a link, get a script file”.

---

Credits

twitter-cli · rdt-cli · xhs-cli · bili-cli · yt-dlp · Jina Reader · Exa · mcporter · feedparser · douyin-mcp-server · linkedin-scraper-mcp

Contact

• 📧 Email: pnt01@foxmail.com
• 🐦 Twitter/X: @Neo_Reidlab

For collaboration or questions, add me on WeChat — I'll invite you to the community group:

For bug reports and feature requests, please use GitHub Issues — easier to track.

License

MIT

Friends

FluxNode — Low-cost AI API gateway, 90% off official pricing, pay-as-you-go or subscription. Works with OpenClaw, Claude Code, and any Agent.

OpenClaw for Enterprise — Enterprise-grade multi-user OpenClaw deployment, use AI directly in Feishu/Lark, container isolation, one-command management.

OpenClaw on Tencent Cloud — One-click OpenClaw on Tencent Cloud: chat to connect autoresearch & unlock internet power.

Star History