基于 LLM 的语义化网页爬虫系统
课程:计算机网络课程设计
技术栈:Python + FastAPI + Socket + GLM-4
Smart Crawler 是一个基于大语言模型(LLM)的智能网页爬虫系统。与传统爬虫依赖固定规则不同,本系统通过 LLM 理解网页语义,自动导航网站结构,智能提取用户所需信息。
| 特性 | 说明 |
|---|---|
| 语义导航 | LLM 理解页面内容,智能选择爬取路径 |
| 意图识别 | 自动解析用户模糊需求,生成精确任务 |
| 自动提取 | 根据目标描述自动识别和提取结构化数据 |
| 智能剪枝 | 自动放弃无效分支,提高爬取效率 |
| 格式化输出 | 自动清洗、规范化输出数据 |
| Socket 通信 | 基于 TCP Socket 的自定义协议(课设重点) |
┌─────────────────────────────────────────────────────────────┐
│ Client Layer │
│ (Socket Client / REST Client) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Communication Layer │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Socket Server │ │ FastAPI REST │ │
│ │ (TCP:9527) │ │ (HTTP:8000) │ │
│ └─────────────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Orchestrator Layer │
│ (核心调度引擎) │
└─────────────────────────────────────────────────────────────┘
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ State Manager │ │ Task Queue │ │ Result Store │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Agent Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Intent │ │Navigator │ │Extractor │ │ Pruner │ │
│ │Recognizer│ │ Agent │ │ Agent │ │ Agent │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ ┌──────────┐ ┌──────────┐ │
│ │Formatter │ │Evaluator │ │
│ │ Agent │ │ Agent │ │
│ └──────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ LLM Layer │
│ (GLM-4 API Client) │
└─────────────────────────────────────────────────────────────┘
用户输入 (模糊需求)
│
▼
┌─────────────────────┐
│ Intent Recognizer │ ── 意图识别
│ Agent │ 生成候选 Prompt
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ 用户确认/选择 │ ── 交互确认
│ 增强后的 Prompt │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Orchestrator │ ── 调度器
└──────────┬──────────┘
│
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Fetcher │ │ Fetcher │ │ Fetcher │
│ (Socket) │ │ (Socket) │ │ (Socket) │
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│Preprocessor│ │Preprocessor│ │Preprocessor│
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└─────────────────────────┼─────────────────────────┘
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Navigator │ │ Extractor │ │ Pruner │
│ Agent │ │ Agent │ │ Agent │
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
│ 选择URL │ 提取数据 │ 剪枝判断
│ │ │
└──────────────────┼──────────────────┘
│
▼
┌─────────────────┐
│ 循环直到完成 │
└────────┬────────┘
│
▼ 原始数据
┌─────────────────┐
│ Formatter │ ── 格式化输出
│ Agent │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Evaluator │ ── 质量评估
│ Agent │
└────────┬────────┘
│
▼
最终格式化结果
│
▼
┌─────────────────┐
│ Socket/REST │ ── 返回给客户端
│ Response │
└─────────────────┘
┌─────────────┐
│ CrawlConfig │
└──────┬──────┘
│
▼
┌───────────┐ ┌─────────────────────┐ ┌───────────┐
│ Fetcher │──────▶│ Orchestrator │◀──────│ GLMClient │
└───────────┘ └─────────────────────┘ └───────────┘
│
┌──────────┬───────────┼───────────┬──────────┐
▼ ▼ ▼ ▼ ▼
┌──────────┐┌─────────┐┌──────────┐┌─────────┐┌──────────┐
│Preprocess││ State ││ Queue ││ Results ││ Agents │
└──────────┘└─────────┘└──────────┘└─────────┘└──────────┘
# ==================== core/types.py ====================
from dataclasses import dataclass, field
from typing import List, Dict, Optional, Any
from enum import Enum
from datetime import datetime
class TaskStatus(Enum):
"""任务状态"""
PENDING = "pending"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
TIMEOUT = "timeout"
@dataclass
class CrawlConfig:
"""爬取配置"""
goal: str # 用户需求描述
root_url: str # 起始 URL
max_depth: int = 5 # 最大爬取深度
max_branches: int = 3 # 每层最多展开的分支数
max_pages: int = 100 # 最多爬取的页面总数
max_concurrent: int = 5 # 最大并发数
timeout_per_page: int = 30 # 单页超时时间(秒)
total_timeout: int = 600 # 任务总超时时间(秒)
same_domain_only: bool = True # 是否限制同域名
render_js: bool = False # 是否渲染 JavaScript
respect_robots_txt: bool = True # 是否遵守 robots.txt
request_delay: int = 1000 # 请求间隔(毫秒)
output_schema: Optional[dict] = None # 期望的输出数据结构
@dataclass
class CrawlTask:
"""爬取任务"""
url: str
depth: int
parent_url: Optional[str] = None
priority: float = 0.5 # 0-1, 越高越优先
metadata: Dict[str, Any] = field(default_factory=dict)
@dataclass
class Link:
"""页面中的链接"""
url: str
anchor_text: str # 链接文字
context: str # 链接周围的文字
@dataclass
class CleanedPage:
"""清洗后的页面"""
url: str
title: str
text_content: str # 纯文本内容
links: List[Link] # 结构化的链接列表
raw_html: str # 原始 HTML 备用
@dataclass
class ScoredURL:
"""带评分的 URL"""
url: str
score: float # 0-1, 找到目标的可能性
reason: str # 选择理由
@dataclass
class ExtractionResult:
"""提取结果"""
found: bool
data: List[dict]
confidence: float
@dataclass
class PruneDecision:
"""剪枝决策"""
should_prune: bool
confidence: float
reason: str
@dataclass
class FormattedResult:
"""格式化后的结果"""
data: List[dict]
summary: str
quality_score: float
warnings: List[str]
@dataclass
class CrawlProgress:
"""爬取进度"""
pages_crawled: int = 0
pages_queued: int = 0
pages_pruned: int = 0
current_depth: int = 0
items_found: int = 0
current_url: Optional[str] = None
@dataclass
class CrawlResult:
"""爬取最终结果"""
task_id: str
status: TaskStatus
goal: str
data: List[dict]
summary: str
quality_score: float
statistics: Dict[str, int]
trace: List[dict]
created_at: datetime
completed_at: Optional[datetime] = None
error: Optional[str] = None┌────────────┬────────────┬────────────┬──────────┬─────────────────┐
│ Magic (2) │ Version(1) │ Type (1) │Length (4)│ Payload (N) │
│ 0xCAFE │ 0x01 │ MsgType │ uint32 │ JSON UTF-8 │
└────────────┴────────────┴────────────┴──────────┴─────────────────┘
│
└─► 总长度 = 8 + N bytes
class MessageType(Enum):
"""消息类型枚举"""
# ===== 客户端 -> 服务端 =====
# 意图识别
INTENT_ANALYZE = 0x01 # 请求意图分析
INTENT_SELECT = 0x02 # 用户选择候选
INTENT_CONFIRM = 0x03 # 确认最终 Prompt
# 任务管理
TASK_CREATE = 0x11 # 创建任务
TASK_QUERY = 0x12 # 查询状态
TASK_CANCEL = 0x13 # 取消任务
TASK_RESULT = 0x14 # 获取结果
# 通用
HEARTBEAT = 0x20 # 心跳包
# ===== 服务端 -> 客户端 =====
# 意图识别响应
INTENT_RESULT = 0x31 # 意图分析结果
INTENT_CONFIRMED = 0x32 # 确认成功
# 任务响应
TASK_CREATED = 0x41 # 任务已创建
TASK_STATUS = 0x42 # 状态响应
TASK_PROGRESS = 0x43 # 进度推送
TASK_COMPLETED = 0x44 # 任务完成
TASK_FAILED = 0x45 # 任务失败
ITEM_FOUND = 0x46 # 发现新数据
# 通用响应
ERROR = 0xFF # 错误响应# ==================== network/protocol.py ====================
import struct
import json
from dataclasses import dataclass
from typing import Any
@dataclass
class Message:
"""协议消息"""
msg_type: MessageType
payload: dict
MAGIC = 0xCAFE
VERSION = 0x01
HEADER_SIZE = 8 # 2 (magic) + 1 (version) + 1 (type) + 4 (length)
def encode(self) -> bytes:
"""编码为字节流"""
payload_bytes = json.dumps(
self.payload,
ensure_ascii=False
).encode('utf-8')
header = struct.pack(
'>HBBI', # big-endian: ushort, ubyte, ubyte, uint
self.MAGIC,
self.VERSION,
self.msg_type.value,
len(payload_bytes)
)
return header + payload_bytes
@classmethod
def decode(cls, data: bytes) -> 'Message':
"""从字节流解码"""
if len(data) < cls.HEADER_SIZE:
raise ValueError("数据长度不足")
magic, version, msg_type, length = struct.unpack(
'>HBBI',
data[:cls.HEADER_SIZE]
)
if magic != cls.MAGIC:
raise ValueError(f"无效的魔数: {magic:#x}")
if version != cls.VERSION:
raise ValueError(f"不支持的版本: {version}")
payload_bytes = data[cls.HEADER_SIZE:cls.HEADER_SIZE + length]
payload = json.loads(payload_bytes.decode('utf-8'))
return cls(
msg_type=MessageType(msg_type),
payload=payload
)
@classmethod
def read_from_socket(cls, sock) -> 'Message':
"""从 socket 读取完整消息"""
# 读取头部
header = cls._recv_exact(sock, cls.HEADER_SIZE)
magic, version, msg_type, length = struct.unpack('>HBBI', header)
if magic != cls.MAGIC:
raise ValueError(f"无效的魔数: {magic:#x}")
# 读取消息体
payload_bytes = cls._recv_exact(sock, length)
payload = json.loads(payload_bytes.decode('utf-8'))
return cls(msg_type=MessageType(msg_type), payload=payload)
@staticmethod
def _recv_exact(sock, n: int) -> bytes:
"""精确接收 n 字节"""
data = b''
while len(data) < n:
chunk = sock.recv(n - len(data))
if not chunk:
raise ConnectionError("连接已关闭")
data += chunk
return data# ==================== network/socket_server.py ====================
import socket
import threading
import logging
from typing import Dict, Callable, Optional
from .protocol import Message, MessageType
logger = logging.getLogger(__name__)
class ClientConnection:
"""客户端连接封装"""
def __init__(self, sock: socket.socket, addr: tuple):
self.socket = sock
self.address = addr
self.subscribed_tasks: set = set()
self.is_alive = True
self._lock = threading.Lock()
def send_message(self, message: Message) -> bool:
"""发送消息到客户端"""
with self._lock:
try:
self.socket.sendall(message.encode())
return True
except Exception as e:
logger.error(f"发送消息失败: {e}")
self.is_alive = False
return False
def receive_message(self) -> Message:
"""接收客户端消息"""
return Message.read_from_socket(self.socket)
def close(self):
"""关闭连接"""
self.is_alive = False
try:
self.socket.close()
except:
pass
class CrawlerSocketServer:
"""
爬虫 Socket 服务器
职责:
1. 管理客户端 TCP 连接
2. 接收和解析请求消息
3. 分发请求到对应处理器
4. 向客户端推送实时进度
"""
def __init__(self, host: str = '0.0.0.0', port: int = 9527):
self.host = host
self.port = port
self.server_socket: Optional[socket.socket] = None
self.clients: Dict[str, ClientConnection] = {}
self.handlers: Dict[MessageType, Callable] = {}
self.running = False
self._lock = threading.Lock()
def register_handler(self, msg_type: MessageType, handler: Callable):
"""注册消息处理器"""
self.handlers[msg_type] = handler
def start(self):
"""启动服务器"""
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.server_socket.bind((self.host, self.port))
self.server_socket.listen(5)
self.running = True
logger.info(f"Socket 服务器启动: {self.host}:{self.port}")
# 启动接受连接的线程
accept_thread = threading.Thread(target=self._accept_loop, daemon=True)
accept_thread.start()
def stop(self):
"""停止服务器"""
self.running = False
for client in self.clients.values():
client.close()
if self.server_socket:
self.server_socket.close()
def _accept_loop(self):
"""接受新连接的循环"""
while self.running:
try:
client_socket, addr = self.server_socket.accept()
logger.info(f"新连接: {addr}")
connection = ClientConnection(client_socket, addr)
addr_key = f"{addr[0]}:{addr[1]}"
with self._lock:
self.clients[addr_key] = connection
# 为每个客户端启动处理线程
client_thread = threading.Thread(
target=self._handle_client,
args=(connection, addr_key),
daemon=True
)
client_thread.start()
except Exception as e:
if self.running:
logger.error(f"接受连接失败: {e}")
def _handle_client(self, connection: ClientConnection, addr_key: str):
"""处理单个客户端的消息"""
try:
while connection.is_alive and self.running:
try:
message = connection.receive_message()
self._dispatch_message(connection, message)
except ConnectionError:
break
except Exception as e:
logger.error(f"处理消息失败: {e}")
error_msg = Message(
msg_type=MessageType.ERROR,
payload={"error": str(e)}
)
connection.send_message(error_msg)
finally:
with self._lock:
self.clients.pop(addr_key, None)
connection.close()
logger.info(f"连接断开: {addr_key}")
def _dispatch_message(self, connection: ClientConnection, message: Message):
"""分发消息到对应处理器"""
handler = self.handlers.get(message.msg_type)
if handler:
response = handler(message.payload, connection)
if response:
connection.send_message(response)
else:
logger.warning(f"未注册的消息类型: {message.msg_type}")
def broadcast_to_task(self, task_id: str, message: Message):
"""向订阅了特定任务的客户端广播消息"""
with self._lock:
for connection in self.clients.values():
if task_id in connection.subscribed_tasks:
connection.send_message(message)
def notify_progress(self, task_id: str, progress: dict):
"""通知任务进度"""
message = Message(
msg_type=MessageType.TASK_PROGRESS,
payload={"task_id": task_id, "progress": progress}
)
self.broadcast_to_task(task_id, message)
def notify_item_found(self, task_id: str, item: dict):
"""通知发现新数据"""
message = Message(
msg_type=MessageType.ITEM_FOUND,
payload={"task_id": task_id, "item": item}
)
self.broadcast_to_task(task_id, message)
def notify_completed(self, task_id: str, result: dict):
"""通知任务完成"""
message = Message(
msg_type=MessageType.TASK_COMPLETED,
payload={"task_id": task_id, "result": result}
)
self.broadcast_to_task(task_id, message)# ==================== network/socket_client.py ====================
import socket
import threading
import uuid
from typing import Callable, Optional, Dict, Any
from .protocol import Message, MessageType
class CrawlerSocketClient:
"""
爬虫 Socket 客户端
用法:
client = CrawlerSocketClient('localhost', 9527)
client.connect()
# 意图识别
result = client.analyze_intent("找一下老师的邮箱")
# 选择候选
selected = result['enhanced_prompts'][0]
# 创建任务
task_id = client.create_task(
goal=selected['prompt'],
root_url="<https://example.edu.cn>"
)
# 注册回调
client.on_progress(lambda p: print(f"进度: {p}"))
# 等待完成
result = client.wait_for_result(task_id)
"""
def __init__(self, host: str, port: int):
self.host = host
self.port = port
self.socket: Optional[socket.socket] = None
self.connected = False
self._callbacks: Dict[str, list] = {
'progress': [],
'item_found': [],
'completed': [],
'error': []
}
self._pending_responses: Dict[str, dict] = {}
self._response_events: Dict[str, threading.Event] = {}
self._lock = threading.Lock()
self._recv_thread: Optional[threading.Thread] = None
def connect(self):
"""建立连接"""
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((self.host, self.port))
self.connected = True
# 启动接收线程
self._recv_thread = threading.Thread(
target=self._receive_loop,
daemon=True
)
self._recv_thread.start()
def disconnect(self):
"""断开连接"""
self.connected = False
if self.socket:
self.socket.close()
# ===== 回调注册 =====
def on_progress(self, callback: Callable):
self._callbacks['progress'].append(callback)
def on_item_found(self, callback: Callable):
self._callbacks['item_found'].append(callback)
def on_completed(self, callback: Callable):
self._callbacks['completed'].append(callback)
def on_error(self, callback: Callable):
self._callbacks['error'].append(callback)
# ===== 意图识别 =====
def analyze_intent(
self,
prompt: str,
target_url: str = None
) -> dict:
"""分析用户意图"""
return self._send_and_wait(
MessageType.INTENT_ANALYZE,
{"prompt": prompt, "target_url": target_url}
)
def select_intent(self, selected_prompt: str, schema: dict = None) -> dict:
"""选择候选 Prompt"""
return self._send_and_wait(
MessageType.INTENT_SELECT,
{"selected_prompt": selected_prompt, "schema": schema}
)
def confirm_intent(self, prompt: str, schema: dict) -> dict:
"""确认最终 Prompt"""
return self._send_and_wait(
MessageType.INTENT_CONFIRM,
{"prompt": prompt, "schema": schema}
)
# ===== 任务管理 =====
def create_task(
self,
goal: str,
root_url: str,
config: dict = None
) -> str:
"""创建爬取任务"""
response = self._send_and_wait(
MessageType.TASK_CREATE,
{"goal": goal, "root_url": root_url, "config": config or {}}
)
return response.get("task_id")
def query_task(self, task_id: str) -> dict:
"""查询任务状态"""
return self._send_and_wait(
MessageType.TASK_QUERY,
{"task_id": task_id}
)
def cancel_task(self, task_id: str) -> bool:
"""取消任务"""
response = self._send_and_wait(
MessageType.TASK_CANCEL,
{"task_id": task_id}
)
return response.get("success", False)
def get_result(self, task_id: str) -> dict:
"""获取任务结果"""
return self._send_and_wait(
MessageType.TASK_RESULT,
{"task_id": task_id}
)
def wait_for_result(self, task_id: str, timeout: int = 600) -> dict:
"""阻塞等待任务完成并返回结果"""
event = threading.Event()
result = {}
def on_complete(data):
if data.get("task_id") == task_id:
result.update(data)
event.set()
self.on_completed(on_complete)
if event.wait(timeout):
return result
else:
raise TimeoutError(f"任务 {task_id} 超时")
# ===== 内部方法 =====
def _send_and_wait(
self,
msg_type: MessageType,
payload: dict,
timeout: int = 30
) -> dict:
"""发送消息并等待响应"""
request_id = str(uuid.uuid4())
payload["_request_id"] = request_id
event = threading.Event()
with self._lock:
self._response_events[request_id] = event
message = Message(msg_type=msg_type, payload=payload)
self.socket.sendall(message.encode())
if event.wait(timeout=timeout):
with self._lock:
return self._pending_responses.pop(request_id, {})
else:
raise TimeoutError("请求超时")
def _receive_loop(self):
"""接收消息循环"""
while self.connected:
try:
message = Message.read_from_socket(self.socket)
self._handle_message(message)
except Exception as e:
if self.connected:
for cb in self._callbacks['error']:
cb({"error": str(e)})
break
def _handle_message(self, message: Message):
"""处理接收到的消息"""
payload = message.payload
# 检查是否是响应消息
request_id = payload.get("_request_id")
if request_id:
with self._lock:
self._pending_responses[request_id] = payload
event = self._response_events.pop(request_id, None)
if event:
event.set()
return
# 处理推送消息
callbacks_map = {
MessageType.TASK_PROGRESS: 'progress',
MessageType.ITEM_FOUND: 'item_found',
MessageType.TASK_COMPLETED: 'completed',
MessageType.ERROR: 'error'
}
callback_key = callbacks_map.get(message.msg_type)
if callback_key:
for cb in self._callbacks[callback_key]:
cb(payload)# ==================== network/socket_fetcher.py ====================
import socket
import ssl
from urllib.parse import urlparse
from dataclasses import dataclass
from typing import Optional, Dict
@dataclass
class FetchResult:
"""获取结果"""
url: str
status_code: int
headers: Dict[str, str]
body: str
error: Optional[str] = None
class SocketHttpFetcher:
"""
基于原始 Socket 的 HTTP 获取器
为体现计网课设要求,使用原始 Socket 实现 HTTP 协议
"""
def __init__(self, timeout: int = 30, user_agent: str = None):
self.timeout = timeout
self.user_agent = user_agent or "SmartCrawler/1.0"
def fetch(self, url: str) -> FetchResult:
"""
获取网页内容
手动实现 HTTP/1.1 协议:
1. 解析 URL
2. 建立 TCP 连接 (HTTPS 则包装 SSL)
3. 发送 HTTP 请求
4. 接收并解析 HTTP 响应
"""
try:
# 1. 解析 URL
parsed = urlparse(url)
host = parsed.hostname
port = parsed.port or (443 if parsed.scheme == 'https' else 80)
path = parsed.path or '/'
if parsed.query:
path += '?' + parsed.query
# 2. 建立 TCP 连接
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(self.timeout)
# 3. 如果是 HTTPS,包装 SSL
if parsed.scheme == 'https':
context = ssl.create_default_context()
sock = context.wrap_socket(sock, server_hostname=host)
sock.connect((host, port))
# 4. 构造并发送 HTTP 请求
request = self._build_request(host, path)
sock.sendall(request.encode('utf-8'))
# 5. 接收响应
response = self._receive_response(sock)
# 6. 解析响应
status_code, headers, body = self._parse_response(response)
sock.close()
return FetchResult(
url=url,
status_code=status_code,
headers=headers,
body=body
)
except Exception as e:
return FetchResult(
url=url,
status_code=0,
headers={},
body='',
error=str(e)
)
def _build_request(self, host: str, path: str) -> str:
"""构造 HTTP 请求"""
lines = [
f"GET {path} HTTP/1.1",
f"Host: {host}",
f"User-Agent: {self.user_agent}",
"Accept: text/html,application/xhtml+xml,*/*;q=0.8",
"Accept-Language: zh-CN,zh;q=0.9,en;q=0.8",
"Accept-Encoding: identity",
"Connection: close",
"",
""
]
return "\\r\\n".join(lines)
def _receive_response(self, sock: socket.socket) -> bytes:
"""接收完整的 HTTP 响应"""
response = b''
while True:
try:
chunk = sock.recv(4096)
if not chunk:
break
response += chunk
except socket.timeout:
break
return response
def _parse_response(self, response: bytes) -> tuple:
"""解析 HTTP 响应"""
# 分离头部和正文
header_end = response.find(b'\\r\\n\\r\\n')
if header_end == -1:
raise ValueError("无效的 HTTP 响应")
header_bytes = response[:header_end]
body_bytes = response[header_end + 4:]
# 解析状态行和头部
header_text = header_bytes.decode('utf-8', errors='ignore')
lines = header_text.split('\\r\\n')
# 状态行: HTTP/1.1 200 OK
status_line = lines[0]
parts = status_line.split(' ', 2)
status_code = int(parts[1])
# 解析头部字段
headers = {}
for line in lines[1:]:
if ':' in line:
key, value = line.split(':', 1)
headers[key.strip().lower()] = value.strip()
# 处理 chunked 编码
if headers.get('transfer-encoding') == 'chunked':
body_bytes = self._decode_chunked(body_bytes)
# 解码正文
charset = self._get_charset(headers)
body = body_bytes.decode(charset, errors='ignore')
return status_code, headers, body
def _decode_chunked(self, data: bytes) -> bytes:
"""解码 chunked 传输编码"""
result = b''
while data:
line_end = data.find(b'\\r\\n')
if line_end == -1:
break
size_str = data[:line_end].decode('ascii')
chunk_size = int(size_str.split(';')[0], 16)
if chunk_size == 0:
break
chunk_start = line_end + 2
chunk_end = chunk_start + chunk_size
result += data[chunk_start:chunk_end]
data = data[chunk_end + 2:]
return result
def _get_charset(self, headers: dict) -> str:
"""从 Content-Type 获取字符集"""
content_type = headers.get('content-type', '')
if 'charset=' in content_type:
charset = content_type.split('charset=')[-1].split(';')[0].strip()
return charset
return 'utf-8'# ==================== llm/glm_client.py ====================
import httpx
import json
import time
from typing import List, Dict, Optional, Generator
from dataclasses import dataclass
from abc import ABC, abstractmethod
@dataclass
class ChatMessage:
"""对话消息"""
role: str # "user" / "assistant" / "system"
content: str
@dataclass
class GLMResponse:
"""GLM 响应"""
content: str
usage: Dict[str, int]
finish_reason: str
class LLMClient(ABC):
"""LLM 客户端抽象接口"""
@abstractmethod
def chat(self, messages: List[ChatMessage], **kwargs) -> GLMResponse:
pass
@abstractmethod
def chat_json(self, messages: List[ChatMessage], **kwargs) -> dict:
pass
class GLMClient(LLMClient):
"""
智谱 GLM API 客户端
文档: <https://open.bigmodel.cn/dev/api>
"""
API_BASE = "<https://open.bigmodel.cn/api/paas/v4>"
def __init__(
self,
api_key: str,
model: str = "glm-4",
timeout: int = 60,
max_retries: int = 3
):
self.api_key = api_key
self.model = model
self.timeout = timeout
self.max_retries = max_retries
self.client = httpx.Client(timeout=timeout)
def _get_headers(self) -> Dict[str, str]:
return {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json"
}
def chat(
self,
messages: List[ChatMessage],
temperature: float = 0.7,
max_tokens: int = 2000,
**kwargs
) -> GLMResponse:
"""发送对话请求"""
payload = {
"model": self.model,
"messages": [
{"role": m.role, "content": m.content}
for m in messages
],
"temperature": temperature,
"max_tokens": max_tokens,
**kwargs
}
for attempt in range(self.max_retries):
try:
response = self.client.post(
f"{self.API_BASE}/chat/completions",
headers=self._get_headers(),
json=payload
)
response.raise_for_status()
data = response.json()
return GLMResponse(
content=data["choices"][0]["message"]["content"],
usage=data.get("usage", {}),
finish_reason=data["choices"][0].get("finish_reason", "stop")
)
except httpx.HTTPStatusError as e:
if e.response.status_code == 429:
time.sleep(2 ** attempt)
continue
raise
except Exception as e:
if attempt == self.max_retries - 1:
raise
time.sleep(1)
raise RuntimeError("GLM API 调用失败")
def chat_json(
self,
messages: List[ChatMessage],
**kwargs
) -> dict:
"""发送对话请求,返回解析后的 JSON"""
response = self.chat(messages, **kwargs)
content = response.content.strip()
# 提取 JSON 代码块
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end].strip()
elif "```" in content:
start = content.find("```") + 3
end = content.find("```", start)
content = content[start:end].strip()
return json.loads(content)
def chat_stream(
self,
messages: List[ChatMessage],
**kwargs
) -> Generator[str, None, None]:
"""流式对话(SSE)"""
payload = {
"model": self.model,
"messages": [
{"role": m.role, "content": m.content}
for m in messages
],
"stream": True,
**kwargs
}
with self.client.stream(
"POST",
f"{self.API_BASE}/chat/completions",
headers=self._get_headers(),
json=payload
) as response:
for line in response.iter_lines():
if line.startswith("data: "):
data = line[6:]
if data == "[DONE]":
break
chunk = json.loads(data)
delta = chunk["choices"][0].get("delta", {})
if "content" in delta:
yield delta["content"]# ==================== agents/base.py ====================
from abc import ABC, abstractmethod
from typing import Any
from dataclasses import dataclass
from ..llm.glm_client import LLMClient, ChatMessage
@dataclass
class AgentConfig:
"""Agent 配置"""
temperature: float = 0.7
max_tokens: int = 2000
max_retries: int = 2
class BaseAgent(ABC):
"""
智能体基类
所有 Agent 继承此类,实现:
- 系统提示词
- 输入格式化
- 输出解析
"""
def __init__(self, llm_client: LLMClient, config: AgentConfig = None):
self.llm = llm_client
self.config = config or AgentConfig()
@abstractmethod
def get_system_prompt(self) -> str:
"""返回系统提示词"""
pass
@abstractmethod
def format_input(self, *args, **kwargs) -> str:
"""格式化输入为用户消息"""
pass
@abstractmethod
def parse_output(self, response: str) -> Any:
"""解析模型输出"""
pass
def run(self, *args, **kwargs) -> Any:
"""执行 Agent"""
user_message = self.format_input(*args, **kwargs)
messages = [
ChatMessage(role="system", content=self.get_system_prompt()),
ChatMessage(role="user", content=user_message)
]
response = self.llm.chat(
messages,
temperature=self.config.temperature,
max_tokens=self.config.max_tokens
)
return self.parse_output(response.content)
def run_json(self, *args, **kwargs) -> dict:
"""执行 Agent,返回 JSON"""
user_message = self.format_input(*args, **kwargs)
messages = [
ChatMessage(role="system", content=self.get_system_prompt()),
ChatMessage(role="user", content=user_message)
]
return self.llm.chat_json(
messages,
temperature=self.config.temperature,
max_tokens=self.config.max_tokens
)# ==================== agents/intent_recognizer.py ====================
from typing import List, Optional
from dataclasses import dataclass, field
from enum import Enum
from .base import BaseAgent
class IntentType(Enum):
"""意图类型"""
CONTACT_INFO = "contact_info"
PERSONNEL_LIST = "personnel_list"
ARTICLE_CONTENT = "article_content"
DOCUMENT_FILE = "document_file"
EVENT_INFO = "event_info"
PRODUCT_INFO = "product_info"
GENERAL_DATA = "general_data"
UNKNOWN = "unknown"
@dataclass
class ExtractedEntity:
"""提取的实体"""
entity_type: str
value: str
confidence: float
@dataclass
class EnhancedPrompt:
"""增强后的 Prompt"""
prompt: str
description: str
specificity_score: float
inferred_schema: Optional[dict]
tags: List[str] = field(default_factory=list)
@dataclass
class IntentRecognitionResult:
"""意图识别结果"""
original_prompt: str
intent_type: IntentType
confidence: float
entities: List[ExtractedEntity]
ambiguities: List[str]
enhanced_prompts: List[EnhancedPrompt]
analysis: str
class IntentRecognizerAgent(BaseAgent):
"""
意图识别智能体
职责:
1. 分析用户的模糊输入
2. 识别意图类型和关键实体
3. 检测歧义和不明确之处
4. 生成多个具体化的候选 Prompt
5. 推断可能的输出数据结构
"""
def get_system_prompt(self) -> str:
return """你是一个专业的意图识别和需求分析专家。分析用户的网页数据抓取需求,识别模糊之处,并生成更具体的任务描述。
## 工作流程
### 1. 意图分类
- contact_info: 联系方式(邮箱、电话等)
- personnel_list: 人员列表
- article_content: 文章、新闻
- document_file: 文档文件
- event_info: 活动信息
- product_info: 产品信息
- general_data: 其他数据
### 2. 实体识别
- organization: 组织名称
- person_type: 人员类型
- data_field: 数据字段
- time_range: 时间范围
- scope: 范围限定
- condition: 筛选条件
### 3. 生成候选 Prompt
生成 3-5 个不同具体程度的候选
## 输出格式(JSON)
{
"original_prompt": "用户原始输入",
"intent_type": "contact_info",
"confidence": 0.85,
"entities": [
{"entity_type": "organization", "value": "信息学院", "confidence": 0.95}
],
"ambiguities": ["'联系方式'具体指哪些信息不明确"],
"enhanced_prompts": [
{
"prompt": "获取信息学院全体教师的电子邮箱地址",
"description": "仅邮箱",
"specificity_score": 0.9,
"inferred_schema": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string", "format": "email"}
}
}
},
"tags": ["简洁", "邮箱"]
}
],
"analysis": "分析说明"
}"""
def format_input(
self,
user_prompt: str,
target_url: Optional[str] = None,
context: Optional[str] = None
) -> str:
url_section = f"\\n## 目标网站\\n{target_url}" if target_url else ""
context_section = f"\\n## 额外上下文\\n{context}" if context else ""
return f"""## 用户输入
"{user_prompt}"
{url_section}{context_section}
请分析这个需求,识别意图,并生成多个具体化的候选任务描述。"""
def parse_output(self, response: str) -> IntentRecognitionResult:
import json
content = response.strip()
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end]
data = json.loads(content)
entities = [
ExtractedEntity(**e) for e in data.get("entities", [])
]
enhanced_prompts = [
EnhancedPrompt(
prompt=p["prompt"],
description=p.get("description", ""),
specificity_score=p.get("specificity_score", 0.8),
inferred_schema=p.get("inferred_schema"),
tags=p.get("tags", [])
)
for p in data.get("enhanced_prompts", [])
]
try:
intent_type = IntentType(data.get("intent_type", "unknown"))
except ValueError:
intent_type = IntentType.UNKNOWN
return IntentRecognitionResult(
original_prompt=data.get("original_prompt", ""),
intent_type=intent_type,
confidence=data.get("confidence", 0.0),
entities=entities,
ambiguities=data.get("ambiguities", []),
enhanced_prompts=enhanced_prompts,
analysis=data.get("analysis", "")
)
async def recognize(
self,
user_prompt: str,
target_url: Optional[str] = None
) -> IntentRecognitionResult:
"""识别用户意图"""
return self.run(user_prompt, target_url)
async def quick_enhance(
self,
user_prompt: str,
num_candidates: int = 4
) -> List[EnhancedPrompt]:
"""快速生成增强的候选 Prompt"""
result = await self.recognize(user_prompt)
return result.enhanced_prompts[:num_candidates]# ==================== agents/navigator.py ====================
from typing import List
from .base import BaseAgent
from ..core.types import CleanedPage, ScoredURL
class NavigatorAgent(BaseAgent):
"""
导航智能体
职责:分析当前页面,选出最可能找到目标信息的 N 个链接
"""
def get_system_prompt(self) -> str:
return """你是一个专业的网页导航专家。分析页面内容和链接,找出最可能包含目标信息的链接。
输出 JSON 格式:
{
"analysis": "简要分析",
"candidates": [
{"url": "完整URL", "score": 0.95, "reason": "选择理由"}
]
}
注意:
- score 越高表示越可能找到目标
- 只选择真正相关的链接"""
def format_input(
self,
page: CleanedPage,
goal: str,
top_n: int = 3
) -> str:
links_text = "\\n".join([
f"- [{link.anchor_text}]({link.url})\\n 上下文: {link.context}"
for link in page.links[:50]
])
return f"""## 用户目标
{goal}
## 当前页面
标题: {page.title}
URL: {page.url}
## 页面内容摘要
{page.text_content[:2000]}
## 页面链接
{links_text}
请选出最多 {top_n} 个最可能找到目标信息的链接。"""
def parse_output(self, response: str) -> List[ScoredURL]:
import json
content = response.strip()
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end]
data = json.loads(content)
return [
ScoredURL(
url=item["url"],
score=item["score"],
reason=item["reason"]
)
for item in data.get("candidates", [])
]
async def navigate(
self,
page: CleanedPage,
goal: str,
top_n: int = 3
) -> List[ScoredURL]:
return self.run(page, goal, top_n)# ==================== agents/extractor.py ====================
from typing import Optional
from .base import BaseAgent
from ..core.types import CleanedPage, ExtractionResult
class ExtractorAgent(BaseAgent):
"""
提取智能体
职责:判断页面是否包含目标信息,如果有则提取
"""
def get_system_prompt(self) -> str:
return """你是一个专业的信息提取专家。从网页内容中识别并提取用户需要的信息。
输出 JSON 格式:
{
"found": true/false,
"confidence": 0.95,
"data": [{"字段1": "值1", "字段2": "值2"}],
"notes": "补充说明"
}
注意:
- found 为 true 时必须提供 data
- 确保提取的数据准确"""
def format_input(
self,
page: CleanedPage,
goal: str,
schema: Optional[dict] = None
) -> str:
schema_text = ""
if schema:
import json
schema_text = f"\\n## 期望格式\\n```json\\n{json.dumps(schema, ensure_ascii=False, indent=2)}\\n```"
return f"""## 用户目标
{goal}
{schema_text}
## 当前页面
标题: {page.title}
URL: {page.url}
## 页面内容
{page.text_content[:4000]}
请判断页面是否包含目标信息,如果有请提取。"""
def parse_output(self, response: str) -> ExtractionResult:
import json
content = response.strip()
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end]
data = json.loads(content)
return ExtractionResult(
found=data.get("found", False),
data=data.get("data", []),
confidence=data.get("confidence", 0.0)
)
async def extract(
self,
page: CleanedPage,
goal: str,
schema: Optional[dict] = None
) -> ExtractionResult:
return self.run(page, goal, schema)# ==================== agents/pruner.py ====================
from typing import List
from .base import BaseAgent
from ..core.types import CleanedPage, PruneDecision
class PrunerAgent(BaseAgent):
"""
剪枝智能体
职责:判断当前分支是否值得继续探索
"""
def get_system_prompt(self) -> str:
return """你是一个搜索效率专家。判断当前的网页爬取分支是否还有可能找到目标信息。
判断依据:
1. 页面主题是否与目标相关
2. 页面链接是否可能通向目标
3. 是否已经偏离目标方向
输出 JSON 格式:
{
"should_prune": true/false,
"confidence": 0.85,
"reason": "判断理由"
}
注意:宁可多探索也不要误剪掉有价值的分支"""
def format_input(
self,
page: CleanedPage,
goal: str,
path: List[str]
) -> str:
path_text = " → ".join(path[-5:])
return f"""## 用户目标
{goal}
## 访问路径
{path_text}
## 当前页面
标题: {page.title}
URL: {page.url}
## 页面内容摘要
{page.text_content[:1500]}
## 链接数量
{len(page.links)} 个链接
请判断是否应该剪枝。"""
def parse_output(self, response: str) -> PruneDecision:
import json
content = response.strip()
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end]
data = json.loads(content)
return PruneDecision(
should_prune=data.get("should_prune", False),
confidence=data.get("confidence", 0.0),
reason=data.get("reason", "")
)
async def should_prune(
self,
page: CleanedPage,
goal: str,
path: List[str]
) -> PruneDecision:
return self.run(page, goal, path)# ==================== agents/formatter.py ====================
from typing import List, Optional
from .base import BaseAgent
from ..core.types import FormattedResult
class FormatterAgent(BaseAgent):
"""
格式化智能体
职责:处理原始数据,使其符合用户期望的格式
功能:
1. 数据清洗和规范化
2. 字段重命名和映射
3. 去重和合并
4. 格式转换
5. 质量评估
"""
def get_system_prompt(self) -> str:
return """你是一个专业的数据格式化专家。将爬虫提取的原始数据处理成用户期望的最终格式。
职责:
1. **数据清洗**: 移除无效、不完整或错误的数据
2. **格式规范化**: 统一日期、电话、邮箱格式等
3. **字段映射**: 将原始字段名映射到期望的字段名
4. **去重合并**: 识别并合并重复记录
5. **质量评估**: 评估数据的完整性和可靠性
输出 JSON 格式:
{
"data": [{"标准字段1": "规范化的值1"}],
"summary": "数据处理总结",
"quality_score": 0.92,
"warnings": ["警告信息"],
"statistics": {
"total_raw": 50,
"total_valid": 45,
"duplicates_removed": 3
}
}"""
def format_input(
self,
raw_data: List[dict],
goal: str,
output_schema: Optional[dict] = None
) -> str:
import json
raw_data_text = json.dumps(raw_data, ensure_ascii=False, indent=2)
schema_text = ""
if output_schema:
schema_text = f"\\n## 期望输出格式\\n```json\\n{json.dumps(output_schema, ensure_ascii=False, indent=2)}\\n```"
return f"""## 用户原始目标
{goal}
{schema_text}
## 原始提取数据
共 {len(raw_data)} 条记录
```json
{raw_data_text}请处理这些数据,使其符合用户期望格式。"""
def parse_output(self, response: str) -> FormattedResult:
import json
content = response.strip()
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end]
data = json.loads(content)
return FormattedResult(
data=data.get("data", []),
summary=data.get("summary", ""),
quality_score=data.get("quality_score", 0.0),
warnings=data.get("warnings", [])
)
async def format(
self,
raw_data: List[dict],
goal: str,
output_schema: Optional[dict] = None
) -> FormattedResult:
return self.run(raw_data, goal, output_schema)
### 6.7 Evaluator Agent(评估)
```python
# ==================== agents/evaluator.py ====================
from typing import List
from dataclasses import dataclass
from .base import BaseAgent
@dataclass
class EvaluationResult:
"""评估结果"""
is_complete: bool
completeness_score: float
missing_info: List[str]
suggestions: List[str]
class EvaluatorAgent(BaseAgent):
"""
评估智能体
职责:评估爬取结果是否满足用户需求
"""
def get_system_prompt(self) -> str:
return """你是一个数据质量评估专家。评估爬取结果是否满足用户需求。
评估维度:
1. **完整性**: 是否获取了足够的数据
2. **准确性**: 数据是否看起来正确
3. **覆盖度**: 是否覆盖了目标范围
输出 JSON 格式:
{
"is_complete": true/false,
"completeness_score": 0.85,
"missing_info": ["缺失信息1"],
"suggestions": ["改进建议1"],
"analysis": "详细分析"
}"""
def format_input(self, results: List[dict], goal: str) -> str:
import json
return f"""## 用户目标
{goal}
## 已获取的数据
共 {len(results)} 条记录
```json
{json.dumps(results[:20], ensure_ascii=False, indent=2)}
{"(仅显示前20条)" if len(results) > 20 else ""}
请评估这些数据是否满足用户需求。"""
def parse_output(self, response: str) -> EvaluationResult:
import json
content = response.strip()
if "```json" in content:
start = content.find("```json") + 7
end = content.find("```", start)
content = content[start:end]
data = json.loads(content)
return EvaluationResult(
is_complete=data.get("is_complete", False),
completeness_score=data.get("completeness_score", 0.0),
missing_info=data.get("missing_info", []),
suggestions=data.get("suggestions", [])
)
async def evaluate(
self,
results: List[dict],
goal: str
) -> EvaluationResult:
return self.run(results, goal)
### 6.8 Agent 总览
| Agent | 阶段 | 职责 | 输入 | 输出 |
|-------|------|------|------|------|
| **IntentRecognizer** | 前置 | 意图识别、Prompt增强 | 用户模糊输入 | 候选Prompt列表 |
| **Navigator** | 爬取中 | 选择最可能的URL | 页面+目标 | 带评分URL列表 |
| **Extractor** | 爬取中 | 提取目标数据 | 页面+目标 | 提取结果 |
| **Pruner** | 爬取中 | 判断是否剪枝 | 页面+目标+路径 | 剪枝决策 |
| **Formatter** | 后置 | 格式化输出 | 原始数据+目标 | 格式化结果 |
| **Evaluator** | 后置 | 质量评估 | 结果+目标 | 评估报告 |
---
## 七、核心组件
### 7.1 HTML 预处理器
```python
# ==================== preprocessors/html_cleaner.py ====================
import re
from typing import List
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from ..core.types import CleanedPage, Link
class HTMLPreprocessor:
"""
HTML 预处理器
职责:清洗 HTML,提取结构化内容
"""
def __init__(self, context_chars: int = 50):
self.context_chars = context_chars
self.remove_tags = ['script', 'style', 'noscript', 'iframe', 'svg']
def process(self, html: str, url: str) -> CleanedPage:
"""处理 HTML,返回清洗后的页面"""
soup = BeautifulSoup(html, 'html.parser')
# 移除无用标签
for tag in self.remove_tags:
for element in soup.find_all(tag):
element.decompose()
# 移除注释
for comment in soup.find_all(string=lambda t: isinstance(t, Comment)):
comment.extract()
# 提取标题
title = self._extract_title(soup)
# 提取链接
links = self._extract_links(soup, url)
# 提取文本内容
text_content = self._extract_text(soup)
return CleanedPage(
url=url,
title=title,
text_content=text_content,
links=links,
raw_html=html
)
def _extract_title(self, soup: BeautifulSoup) -> str:
"""提取页面标题"""
if soup.title and soup.title.string:
return soup.title.string.strip()
h1 = soup.find('h1')
if h1:
return h1.get_text(strip=True)
return ""
def _extract_links(self, soup: BeautifulSoup, base_url: str) -> List[Link]:
"""提取所有链接"""
links = []
for a in soup.find_all('a', href=True):
href = a['href']
# 跳过无效链接
if href.startswith(('#', 'javascript:', 'mailto:', 'tel:')):
continue
# 转换为绝对 URL
absolute_url = urljoin(base_url, href)
# 获取锚文本
anchor_text = a.get_text(strip=True)
if not anchor_text:
anchor_text = a.get('title', '') or href
# 获取上下文
context = self._get_context(a)
links.append(Link(
url=absolute_url,
anchor_text=anchor_text,
context=context
))
return links
def _get_context(self, element) -> str:
"""获取元素周围的上下文文本"""
parent = element.parent
if not parent:
return ""
text = parent.get_text(strip=True)
return text[:self.context_chars * 2]
def _extract_text(self, soup: BeautifulSoup) -> str:
"""提取页面纯文本"""
text = soup.get_text(separator='\\n', strip=True)
# 合并多个空行
text = re.sub(r'\\n{3,}', '\\n\\n', text)
return text
# ==================== storage/state_manager.py ====================
from typing import Set, Dict, Optional
from dataclasses import dataclass, field
import threading
@dataclass
class CrawlStats:
"""爬取统计"""
total_visited: int = 0
total_in_progress: int = 0
total_failed: int = 0
class StateManager:
"""
状态管理器
职责:维护爬取状态,线程安全
"""
def __init__(self):
self.visited: Set[str] = set()
self.in_progress: Set[str] = set()
self.failed: Dict[str, str] = {}
self._lock = threading.Lock()
def mark_in_progress(self, url: str) -> bool:
"""标记 URL 正在处理"""
with self._lock:
if url in self.visited or url in self.in_progress:
return False
self.in_progress.add(url)
return True
def mark_visited(self, url: str):
"""标记 URL 已访问"""
with self._lock:
self.in_progress.discard(url)
self.visited.add(url)
def mark_failed(self, url: str, error: str):
"""标记 URL 失败"""
with self._lock:
self.in_progress.discard(url)
self.failed[url] = error
def is_visited(self, url: str) -> bool:
"""检查是否已访问"""
with self._lock:
return url in self.visited
def can_visit(self, url: str) -> bool:
"""检查是否可以访问"""
with self._lock:
return url not in self.visited and url not in self.in_progress
def get_stats(self) -> CrawlStats:
"""获取统计信息"""
with self._lock:
return CrawlStats(
total_visited=len(self.visited),
total_in_progress=len(self.in_progress),
total_failed=len(self.failed)
)
def reset(self):
"""重置状态"""
with self._lock:
self.visited.clear()
self.in_progress.clear()
self.failed.clear()# ==================== storage/task_queue.py ====================
import heapq
from typing import Optional, List
from dataclasses import dataclass, field
import threading
from ..core.types import CrawlTask
class PriorityTaskQueue:
"""
优先级任务队列
按 priority 降序排列(priority 越高越先执行)
"""
def __init__(self):
self._heap: List[tuple] = []
self._counter = 0
self._lock = threading.Lock()
def push(self, task: CrawlTask):
"""添加任务"""
with self._lock:
# 使用负数实现降序(heapq 是最小堆)
heapq.heappush(
self._heap,
(-task.priority, self._counter, task)
)
self._counter += 1
def pop(self) -> Optional[CrawlTask]:
"""取出最高优先级任务"""
with self._lock:
if self._heap:
_, _, task = heapq.heappop(self._heap)
return task
return None
def push_batch(self, tasks: List[CrawlTask]):
"""批量添加任务"""
for task in tasks:
self.push(task)
def is_empty(self) -> bool:
"""检查是否为空"""
with self._lock:
return len(self._heap) == 0
def size(self) -> int:
"""获取队列大小"""
with self._lock:
return len(self._heap)
def clear(self):
"""清空队列"""
with self._lock:
self._heap.clear()# ==================== storage/result_store.py ====================
from typing import List, Optional
from dataclasses import dataclass
import threading
import json
@dataclass
class StoredItem:
"""存储的数据项"""
data: dict
source_url: str
extracted_at: str
class ResultStore:
"""
结果存储
职责:存储和聚合爬取结果
"""
def __init__(self):
self._items: List[StoredItem] = []
self._lock = threading.Lock()
def add(self, data: dict, source_url: str):
"""添加数据"""
from datetime import datetime
with self._lock:
self._items.append(StoredItem(
data=data,
source_url=source_url,
extracted_at=datetime.now().isoformat()
))
def add_batch(self, items: List[dict], source_url: str):
"""批量添加"""
for item in items:
self.add(item, source_url)
def get_all(self) -> List[dict]:
"""获取所有数据"""
with self._lock:
return [item.data for item in self._items]
def get_all_with_metadata(self) -> List[dict]:
"""获取所有数据(含元数据)"""
with self._lock:
return [
{
**item.data,
"_metadata": {
"source_url": item.source_url,
"extracted_at": item.extracted_at
}
}
for item in self._items
]
def count(self) -> int:
"""获取数据条数"""
with self._lock:
return len(self._items)
def is_goal_satisfied(self, min_count: int = 1) -> bool:
"""检查是否满足目标"""
return self.count() >= min_count
def export_json(self) -> str:
"""导出为 JSON"""
return json.dumps(
self.get_all_with_metadata(),
ensure_ascii=False,
indent=2
)
def clear(self):
"""清空存储"""
with self._lock:
self._items.clear()Base URL: http://localhost:8000/api/v1
| 模块 | 端点 | 说明 |
|---|---|---|
| 意图识别 | /intent/* |
Prompt 增强 |
| 任务管理 | /tasks/* |
创建、查询、取消 |
| 结果查询 | /tasks/{id}/results |
获取和导出 |
| 配置管理 | /config/* |
默认配置、验证 |
| 系统状态 | /health, /usage |
健康检查、统计 |
POST /intent/analyze
请求体:
{
"prompt": "找一下信息学院老师的联系方式",
"target_url": "<https://www.example-university.edu.cn>"
}
响应:
{
"original_prompt": "找一下信息学院老师的联系方式",
"intent_type": "contact_info",
"confidence": 0.85,
"entities": [
{"entity_type": "organization", "value": "信息学院", "confidence": 0.95},
{"entity_type": "person_type", "value": "教师", "confidence": 0.80}
],
"ambiguities": [
"'联系方式'具体指哪些信息不明确",
"'老师'的范围不明确"
],
"enhanced_prompts": [
{
"prompt": "获取信息学院全体教师的电子邮箱地址",
"description": "仅邮箱 - 最基础的联系方式",
"specificity_score": 0.9,
"inferred_schema": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string", "format": "email"}
}
}
},
"tags": ["简洁", "邮箱"]
},
{
"prompt": "获取信息学院全体教师的姓名、职称、邮箱、办公电话",
"description": "标准联系信息",
"specificity_score": 0.95,
"tags": ["标准", "推荐"]
}
],
"analysis": "用户希望获取信息学院教师的联系方式..."
}
POST /intent/quick-enhance
请求体:
{
"prompt": "找老师邮箱",
"num_candidates": 4
}
POST /intent/confirm
请求体:
{
"selected_prompt": "获取信息学院全体教师的姓名、职称、邮箱",
"inferred_schema": {...}
}
POST /tasks
请求体:
{
"goal": "获取信息学院所有教师的姓名、职称和邮箱",
"root_url": "<https://www.example-university.edu.cn>",
"config": {
"max_depth": 4,
"max_branches": 5,
"same_domain_only": true
},
"output_schema": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"title": {"type": "string"},
"email": {"type": "string", "format": "email"}
}
}
}
}
响应: 201 Created
{
"task_id": "task_a1b2c3d4e5f6",
"status": "pending",
"created_at": "2024-01-15T10:30:00Z",
"message": "任务已创建"
}
GET /tasks/{task_id}
响应:
{
"task_id": "task_a1b2c3d4e5f6",
"status": "running",
"goal": "获取信息学院所有教师的姓名、职称和邮箱",
"progress": {
"pages_crawled": 15,
"pages_queued": 8,
"pages_pruned": 3,
"current_depth": 2,
"items_found": 12
},
"created_at": "2024-01-15T10:30:00Z",
"started_at": "2024-01-15T10:30:05Z"
}
GET /tasks?status=running&limit=20&offset=0
POST /tasks/{task_id}/cancel
GET /tasks/{task_id}/results?format=json&limit=100
响应:
{
"task_id": "task_a1b2c3d4e5f6",
"total": 45,
"data": [
{
"name": "张三",
"title": "教授",
"email": "zhangsan@example.edu.cn",
"_metadata": {
"source_url": "<https://www.example.edu.cn/faculty/zhangsan>",
"extracted_at": "2024-01-15T10:33:12Z",
"confidence": 0.95
}
}
]
}
POST /tasks/{task_id}/results/export
请求体:
{
"format": "xlsx",
"include_metadata": true
}
GET /tasks/{task_id}/trace
GET /config/defaults
POST /config/validate-url
| 状态 | 说明 |
|---|---|
pending |
等待执行 |
running |
正在执行 |
completed |
已完成 |
failed |
执行失败 |
cancelled |
已取消 |
timeout |
超时终止 |
wss://localhost:8000/ws?token=<your_token>
{"action": "subscribe", "task_id": "task_a1b2c3d4e5f6"}
{
"event": "task.progress",
"task_id": "task_a1b2c3d4e5f6",
"data": {
"pages_crawled": 15,
"items_found": 12,
"current_url": "https://..."
}
}
{
"event": "task.item_found",
"task_id": "task_a1b2c3d4e5f6",
"data": {
"item": {"name": "张三", "email": "..."},
"source_url": "https://..."
}
}
{
"event": "task.completed",
"task_id": "task_a1b2c3d4e5f6",
"data": {
"status": "completed",
"total_items": 45
}
}
{
"error": {
"code": "VALIDATION_ERROR",
"message": "请求参数验证失败",
"details": [
{"field": "root_url", "message": "必须是有效的 URL 格式"}
],
"request_id": "req_abc123xyz"
}
}
| 状态码 | 错误码 | 描述 |
|---|---|---|
| 400 | VALIDATION_ERROR |
请求参数验证失败 |
| 401 | UNAUTHORIZED |
未认证 |
| 403 | QUOTA_EXCEEDED |
配额已用尽 |
| 404 | TASK_NOT_FOUND |
任务不存在 |
| 422 | URL_NOT_ACCESSIBLE |
目标 URL 无法访问 |
| 429 | RATE_LIMITED |
请求频率超限 |
| 500 | INTERNAL_ERROR |
服务器内部错误 |
| 错误码 | 描述 |
|---|---|
CRAWL_TIMEOUT |
爬取超时 |
PAGE_LOAD_FAILED |
页面加载失败 |
LLM_ERROR |
LLM 服务异常 |
ALL_BRANCHES_PRUNED |
所有分支被剪枝 |
NO_DATA_FOUND |
未找到任何数据 |
smart_crawler/
├── core/
│ ├── __init__.py
│ ├── types.py # 数据类型定义
│ ├── config.py # 配置管理
│ └── exceptions.py # 自定义异常
│
├── network/ # ⭐ Socket 编程核心
│ ├── __init__.py
│ ├── protocol.py # 消息协议定义
│ ├── socket_server.py # TCP Socket 服务端
│ ├── socket_client.py # TCP Socket 客户端
│ └── socket_fetcher.py # 基于 Socket 的 HTTP 获取器
│
├── llm/ # ⭐ GLM 集成
│ ├── __init__.py
│ ├── base.py # LLM 抽象接口
│ └── glm_client.py # 智谱 GLM 客户端
│
├── agents/ # ⭐ 智能体
│ ├── __init__.py
│ ├── base.py # Agent 基类
│ ├── intent_recognizer.py # 意图识别
│ ├── navigator.py # 导航
│ ├── extractor.py # 提取
│ ├── pruner.py # 剪枝
│ ├── formatter.py # 格式化
│ └── evaluator.py # 评估
│
├── preprocessors/
│ ├── __init__.py
│ ├── base.py # 预处理器接口
│ └── html_cleaner.py # HTML 清洗器
│
├── storage/
│ ├── __init__.py
│ ├── state_manager.py # 状态管理
│ ├── task_queue.py # 任务队列
│ └── result_store.py # 结果存储
│
├── api/ # FastAPI REST 层
│ ├── __init__.py
│ ├── main.py # FastAPI 应用
│ ├── routes/
│ │ ├── intent.py # 意图识别路由
│ │ ├── tasks.py # 任务路由
│ │ ├── results.py # 结果路由
│ │ └── config.py # 配置路由
│ └── schemas.py # Pydantic 模型
│
├── orchestrator.py # 核心调度器
├── server.py # 服务启动入口
├── client.py # 客户端工具
├── requirements.txt
└── README.md
from smart_crawler import SmartCrawlerClient
# 连接服务器
client = SmartCrawlerClient(host='localhost', port=9527)
client.connect()
# 1. 意图识别
result = client.analyze_intent(
prompt="找一下信息学院老师的联系方式",
target_url="<https://www.example-university.edu.cn>"
)
# 2. 显示候选
print("候选任务:")
for i, p in enumerate(result.enhanced_prompts, 1):
print(f" [{i}] {p.prompt}")
print(f" {p.description}")
# 3. 选择并创建任务
selected = result.enhanced_prompts[1]
task_id = client.create_task(
goal=selected.prompt,
root_url="<https://www.example-university.edu.cn>",
config={"max_depth": 4, "max_branches": 5}
)
# 4. 注册回调
client.on_progress(lambda p: print(f"进度: {p['items_found']} 条"))
client.on_item_found(lambda item: print(f"新数据: {item['name']}"))
# 5. 等待完成
results = client.wait_for_result(task_id, timeout=600)
print(f"完成!共 {len(results['data'])} 条数据")
# 6. 断开连接
client.disconnect()# 交互式爬取
$ smart-crawler crawl <https://www.example-university.edu.cn>
🔍 请输入您的需求: 找一下信息学院老师的联系方式
📋 意图分析完成!
📝 候选任务:
[1] 获取信息学院全体教师的邮箱地址
[2] 获取信息学院全体教师的姓名、职称、邮箱、办公电话 ⭐推荐
[3] 获取信息学院全体教师的完整信息
[4] 仅获取信息学院教授的联系方式
请选择 (1-4): 2
✅ 已确认: 获取信息学院全体教师的姓名、职称、邮箱、办公电话
开始爬取...
[▓▓▓▓░░░░░░] 40% | 已爬取: 15 | 已找到: 12
✅ 完成!共找到 45 条数据
📄 结果已保存至: ./results/teachers.json
# 意图分析
curl -X POST <http://localhost:8000/api/v1/intent/analyze> \\
-H "Content-Type: application/json" \\
-d '{
"prompt": "找一下信息学院老师的联系方式",
"target_url": "<https://www.example-university.edu.cn>"
}'
# 创建任务
curl -X POST <http://localhost:8000/api/v1/tasks> \\
-H "Content-Type: application/json" \\
-d '{
"goal": "获取信息学院全体教师的姓名、职称、邮箱",
"root_url": "<https://www.example-university.edu.cn>",
"config": {"max_depth": 4}
}'
# 查询状态
curl <http://localhost:8000/api/v1/tasks/task_abc123>
# 获取结果
curl <http://localhost:8000/api/v1/tasks/task_abc123/results>
| 知识点 | 实现位置 | 说明 |
|---|---|---|
| TCP Socket 编程 | socket_server.py |
服务端多客户端管理 |
| TCP 客户端 | socket_client.py |
长连接、心跳机制 |
| HTTP 协议实现 | socket_fetcher.py |
手动构造请求、解析响应 |
| 自定义协议 | protocol.py |
魔数、版本、消息帧 |
| 多线程并发 | socket_server.py |
每客户端独立线程 |
| SSL/TLS | socket_fetcher.py |
HTTPS 支持 |
| 粘包处理 | protocol.py |
固定帧头 + 长度字段 |
┌────────────┬────────────┬────────────┬──────────┬─────────────────┐
│ Magic (2) │ Version(1) │ Type (1) │Length (4)│ Payload (N) │
│ 0xCAFE │ 0x01 │ MsgType │ uint32 │ JSON UTF-8 │
└────────────┴────────────┴────────────┴──────────┴─────────────────┘
- 魔数校验: 识别有效数据包
- 版本控制: 协议升级兼容
- 类型字段: 区分不同消息
- 长度字段: 解决粘包问题
# socket_fetcher.py 中手动实现:
# 1. TCP 连接
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
# 2. 构造 HTTP 请求
request = f"GET {path} HTTP/1.1\\r\\nHost: {host}\\r\\n\\r\\n"
sock.sendall(request.encode())
# 3. 接收并解析响应
# - 解析状态行
# - 解析头部字段
# - 处理 chunked 编码
# - 解码正文- LLM 驱动的启发式搜索: 用语义理解代替规则匹配
- 意图识别 + Prompt 增强: 自动理解模糊需求
- 智能剪枝: 减少无效爬取
- 6 个专业化 Agent: 各司其职、协同工作
- Socket + REST 双通道: 灵活的通信方式
# 1. 安装依赖
pip install -r requirements.txt
# 2. 配置环境变量
export GLM_API_KEY="your_api_key"
# 3. 启动服务
python server.py
# 4. 运行客户端
python client.py crawl <https://example.edu.cn>
文档版本: 1.0.0
最后更新: 2025-11-26