diff --git a/README.md b/README.md index 29299d5..ec1c179 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,41 @@ # Browser RPC -> High-performance browser automation RPC service based on Playwright + gRPC with powerful anti-detection capabilities +> High-performance browser automation RPC service based on Playwright + gRPC with powerful anti-detection capabilities. Now supports Distributed Architecture and Docker Deployment. [![CI](https://github.com/LBatsoft/browser_rpc/actions/workflows/ci.yml/badge.svg)](https://github.com/LBatsoft/browser_rpc/actions/workflows/ci.yml) ## ✨ Features - πŸ›‘οΈ **Powerful Anti-Detection**: playwright-stealth + custom scripts to bypass common bot detection -- πŸ”Œ **gRPC Interface**: 13 standardized APIs supporting remote calls +- πŸ”Œ **Unified Gateway**: Distributed architecture with Gateway as single entry point (Load Balancing & Routing) +- 🐳 **Docker Ready**: One-click deployment with Docker Compose +- πŸ–₯️ **Remote Control UI**: Real-time browser remote control with multi-tab support - πŸ“‘ **Network Interception**: Complete request/response capture capabilities - πŸš€ **High Performance**: Supports multi-session concurrency with resource pool management -- 🎨 **Easy to Use**: Clean Python client API +- πŸ”’ **Security**: API Key authentication for clients + Cluster Secret for internal communication -## πŸš€ Quick Start +## πŸš€ Quick Start (Docker) - Recommended -### One-Click Test (Recommended) +The easiest way to start the distributed cluster (Gateway + Redis + Worker Nodes). + +### 1. Requirements + +- Docker & Docker Compose + +### 2. Start Cluster ```bash -cd /path/to/browser_rpc -./scripts/quick_test.sh +docker compose up --build -d ``` -This will automatically: install browser β†’ start server β†’ run tests β†’ cleanup +### 3. Verify -## πŸ“¦ Installation +Access the remote control interface: +http://localhost:8000/static/remote.html + +> Note: By default, `API_KEY` is set to `dev-test-key`. You might need to configure headers if using the UI directly, or disable auth in `docker-compose.yml` for testing. + +## πŸ“¦ Installation (Local Development) ### 1. Install Dependencies @@ -37,7 +49,7 @@ pip install -r requirements.txt python -m grpc_tools.protoc -I./proto --python_out=. --grpc_python_out=. ./proto/spider.proto ``` -### 3. Install Browser (First Time) +### 3. Install Browser ```bash playwright install chromium @@ -45,276 +57,108 @@ playwright install chromium ## πŸ’» Usage -### Start Server - -**gRPC Server:** -```bash -./scripts/start_rpc_server.sh -# or -python rpc_server.py -``` - -**HTTP Server:** -```bash -./scripts/start_http_server.sh -# or -python http_server.py -``` - -The HTTP server provides: -- **API Documentation**: http://localhost:8000/docs (Swagger UI) -- **ReDoc**: http://localhost:8000/redoc -- **REST API**: http://localhost:8000/api/* +### HTTP API (via Gateway) -### Basic Example (gRPC) +The system provides a RESTful API via the Gateway (Default port: 8000). -```python -import asyncio -from rpc_client import BrowserRPCClient - -async def main(): - client = BrowserRPCClient() - await client.connect() - - # Create session - await client.create_session(headless=True) - - # Navigate to page - await client.navigate('https://www.example.com') - - # Get content - html = await client.get_page_content() - - await client.close() - -asyncio.run(main()) -``` - -### Basic Example (HTTP) - -```python -import asyncio -from http_client import BrowserHTTPClient - -async def main(): - client = BrowserHTTPClient(base_url='http://localhost:8000') - - try: - # Create session - await client.create_session(headless=True) - - # Navigate to page - await client.navigate('https://www.example.com') - - # Get content - html = await client.get_page_content() - - # Take screenshot - await client.take_screenshot(save_path='page.png', full_page=True) - - finally: - await client.close() - -asyncio.run(main()) -``` +**Authentication Headers:** +- `X-API-Key`: `dev-test-key` (Default in docker-compose.yml) -### Advanced Example +#### Basic Python Example ```python -# Set headers -await client.set_headers({ - 'Authorization': 'Bearer token' -}) - -# Set cookies -await client.set_cookies([{ - 'name': 'session', - 'value': 'abc123', - 'domain': '.example.com' -}]) - -# Wait and click element -await client.wait_for_element('button#submit') -await client.click_element('button#submit') - -# Type text -await client.type_text('input#username', 'myname') - -# Take screenshot -await client.take_screenshot(save_path='page.png', full_page=True) - -# Get network requests -requests = await client.get_network_requests(url_pattern=r'/api/') -for req in requests: - print(f"{req['method']} {req['url']}") - if req.get('response'): - print(f"Response: {req['response']['body']}") +import requests + +GATEWAY_URL = "http://localhost:8000" +HEADERS = {"X-API-Key": "dev-test-key"} + +# 1. Create Session +resp = requests.post(f"{GATEWAY_URL}/api/sessions", json={"headless": True}, headers=HEADERS) +session_id = resp.json()["session_id"] +print(f"Session Created: {session_id}") + +# 2. Navigate +requests.post( + f"{GATEWAY_URL}/api/sessions/{session_id}/navigate", + json={"url": "https://www.google.com"}, + headers=HEADERS +) + +# 3. Take Screenshot +requests.post( + f"{GATEWAY_URL}/api/sessions/{session_id}/screenshot", + json={"full_page": False}, + headers=HEADERS +) + +# 4. Close Session +requests.delete(f"{GATEWAY_URL}/api/sessions/{session_id}", headers=HEADERS) ``` -### HTTP API Endpoints - -The HTTP server provides RESTful API endpoints: - -- `POST /api/sessions` - Create browser session -- `DELETE /api/sessions/{session_id}` - Close session -- `POST /api/sessions/{session_id}/navigate` - Navigate to URL -- `POST /api/sessions/{session_id}/execute` - Execute JavaScript -- `GET /api/sessions/{session_id}/content` - Get page HTML -- `POST /api/sessions/{session_id}/network` - Get network requests -- `POST /api/sessions/{session_id}/wait` - Wait for element -- `POST /api/sessions/{session_id}/click` - Click element -- `POST /api/sessions/{session_id}/type` - Type text -- `POST /api/sessions/{session_id}/screenshot` - Take screenshot -- `POST /api/sessions/{session_id}/headers` - Set headers -- `POST /api/sessions/{session_id}/cookies` - Set cookies -- `GET /api/sessions/{session_id}/cookies` - Get cookies - -Visit http://localhost:8000/docs for interactive API documentation. - -## πŸ“‹ Available RPC APIs - -| API | Description | -|-----|-------------| -| `CreateSession` | Create browser session | -| `CloseSession` | Close session | -| `Navigate` | Navigate to URL | -| `ExecuteScript` | Execute JavaScript | -| `GetPageContent` | Get page HTML | -| `GetNetworkRequests` | Get network requests | -| `WaitForElement` | Wait for element | -| `ClickElement` | Click element | -| `TypeText` | Type text | -| `TakeScreenshot` | Take screenshot | -| `SetHeaders` | Set request headers | -| `SetCookies` | Set cookies | -| `GetCookies` | Get cookies | +### API Documentation -## πŸ›‘οΈ Anti-Detection Capabilities - -### Automatically Hidden Features - -- βœ… `navigator.webdriver` -- βœ… `window.chrome` object -- βœ… `plugins` and `mimeTypes` -- βœ… Automation traces (`cdc_*` variables) -- βœ… WebGL fingerprint consistency -- βœ… Permission state simulation +Once started, visit: http://localhost:8000/docs -## πŸ”§ Configuration +## πŸ—οΈ Architecture -### Server Configuration (`config.py`) +The system has evolved into a distributed microservices architecture: -```python -RPC_HOST = '0.0.0.0' # Listen address -RPC_PORT = 50051 # Listen port -MAX_SESSIONS = 10 # Max sessions -SESSION_TIMEOUT = 3600 # Session timeout (seconds) +```mermaid +graph TD + Client[Client / Frontend] -->|X-API-Key| Gateway[Gateway :8000] + Gateway -->|Service Discovery| Redis[(Redis)] + Gateway -->|Load Balanced| Node1[Worker Node 1] + Gateway -->|Load Balanced| Node2[Worker Node 2] + Node1 -->|Heartbeat| Redis + Node2 -->|Heartbeat| Redis ``` -### Client Configuration +- **Gateway**: Handles Auth, Load Balancing, and Request Routing. +- **Worker Node**: Executes browser automation tasks (Playwright). +- **Redis**: Stores node registry, heartbeats, and session mapping. -```python -# Connect to remote server -client = BrowserRPCClient(host='192.168.1.100', port=50051) -``` +## πŸ”§ Configuration -## πŸ“Š Performance +Configuration is managed via Environment Variables (or `config.py` defaults). -- **Single session memory**: 100-200MB -- **Startup time**: 1-3 seconds -- **Default timeout**: 30 seconds -- **Max concurrency**: 10 sessions (configurable) +| Variable | Description | Default | +|----------|-------------|---------| +| `REDIS_URL` | Redis connection string | `redis://localhost:6379/0` | +| `HTTP_PORT` | Service listening port | `8000` | +| `MAX_SESSIONS` | Max concurrent sessions per node | `10` | +| `API_KEY` | Client access token | `None` (Disabled) | +| `CLUSTER_SECRET` | Internal communication secret | `None` (Disabled) | ## πŸ—‚οΈ Project Structure ``` browser_rpc/ -β”œβ”€β”€ proto/ -β”‚ └── spider.proto # gRPC service definition -β”œβ”€β”€ spider_pb2.py # Generated protobuf code -β”œβ”€β”€ spider_pb2_grpc.py # Generated gRPC code -β”œβ”€β”€ docs/ # Documentation -β”‚ β”œβ”€β”€ README_CN.md -β”‚ β”œβ”€β”€ anti_detection_notes.md -β”‚ └── startup_guide.md -β”œβ”€β”€ resources/ -β”‚ β”œβ”€β”€ screenshots/ # Test screenshots -β”‚ └── stealth/ # Stealth scripts -β”œβ”€β”€ scripts/ -β”‚ β”œβ”€β”€ install.sh -β”‚ β”œβ”€β”€ quick_test.sh -β”‚ β”œβ”€β”€ start_rpc_server.sh -β”‚ └── start_server.sh -β”œβ”€β”€ cdp_client.py # CDP client implementation -β”œβ”€β”€ rpc_server.py # gRPC server -β”œβ”€β”€ rpc_client.py # gRPC client wrapper -β”œβ”€β”€ config.py # Configuration -└── requirements.txt # Dependencies +β”œβ”€β”€ core/ +β”‚ └── registry.py # Service discovery & Load balancing logic +β”œβ”€β”€ proto/ # gRPC definitions +β”œβ”€β”€ static/ # Remote Control UI +β”œβ”€β”€ scripts/ # Helper scripts +β”œβ”€β”€ gateway.py # API Gateway entry point +β”œβ”€β”€ http_server.py # Worker Node entry point +β”œβ”€β”€ cdp_client.py # Playwright/CDP wrapper +β”œβ”€β”€ config.py # Configuration loader +β”œβ”€β”€ docker-compose.yml # Docker orchestration +└── Dockerfile # Container definition ``` -## 🎯 Use Cases - -- βœ… Scraping JavaScript-rendered pages -- βœ… Bypassing anti-bot detection -- βœ… Capturing AJAX/API request data -- βœ… Automated testing -- βœ… Page screenshot service -- βœ… Form auto-filling - -## ⚠️ Troubleshooting - -### Q: Connection refused error? -**A:** Server not started, run `./scripts/start_rpc_server.sh` - -### Q: Browser not installed? -**A:** Run `playwright install chromium` - -### Q: Port already in use? -**A:** `lsof -ti:50051 | xargs kill -9` - -### Q: Module import error? -**A:** `pip install -r requirements.txt` - -## πŸ“š Documentation +## πŸ›‘οΈ Anti-Detection Capabilities -- **δΈ­ζ–‡ζ–‡ζ‘£**: [docs/README_CN.md](docs/README_CN.md) -- **API Definition**: [proto/spider.proto](proto/spider.proto) -- **Startup Guide**: [docs/startup_guide.md](docs/startup_guide.md) -- **Anti-Detection Notes**: [docs/anti_detection_notes.md](docs/anti_detection_notes.md) +- βœ… `navigator.webdriver` hidden +- βœ… `window.chrome` object mocked +- βœ… Automation traces removed +- βœ… WebGL fingerprint consistency +- βœ… Permission state simulation ## πŸ“„ License MIT License - see [LICENSE](LICENSE) file for details -## πŸ”„ CI/CD - -This project uses GitHub Actions for continuous integration and deployment: - -- **CI Workflow** (`.github/workflows/ci.yml`): - - Tests on Python 3.9, 3.10, and 3.11 - - Installs dependencies and Playwright browsers - - Compiles Proto files - - Verifies imports and code syntax - - Tests server startup - - Runs code linting (flake8, pylint) - -- **CD Workflow** (`.github/workflows/cd.yml`): - - Creates release archives for version tags - - Optional GitHub Pages deployment (commented out by default, see workflow file for instructions) - -The CI pipeline runs automatically on every push and pull request to the `master` branch. - -## 🀝 Contributing - -Contributions are welcome! Please feel free to submit a Pull Request. - -Before submitting, please ensure: -- Code passes all CI checks -- Follows the project's code style (configured in `.flake8`) -- Includes appropriate tests if applicable - --- -**Version**: 1.0.0 +**Version**: 2.0.0 (Distributed) **Status**: βœ… Production Ready - diff --git a/requirements.txt b/requirements.txt index 785cc92..6c37fb5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1,12 @@ +fastapi==0.104.1 +uvicorn==0.24.0 redis==5.0.1 +playwright==1.40.0 +httpx==0.25.2 +websockets==12.0 +grpcio==1.59.3 +grpcio-tools==1.59.3 +protobuf==4.25.1 +pydantic==2.5.2 +python-multipart==0.0.6 +jinja2==3.1.2