Skip to content

About

计算机网络课程设计

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Smart Crawler 智能爬虫系统

基于 LLM 的语义化网页爬虫系统

课程:计算机网络课程设计

技术栈:Python + FastAPI + Socket + GLM-4


目录

  1. 项目概述
  2. 系统架构
  3. 核心数据结构
  4. Socket 通信层
  5. GLM API 集成
  6. 智能体设计
  7. 核心组件
  8. RESTful API
  9. WebSocket 实时通信
  10. 错误处理
  11. 项目结构
  12. 使用示例
  13. 课设亮点

一、项目概述

1.1 项目简介

Smart Crawler 是一个基于大语言模型(LLM)的智能网页爬虫系统。与传统爬虫依赖固定规则不同,本系统通过 LLM 理解网页语义,自动导航网站结构,智能提取用户所需信息。

1.2 核心特性

特性 说明
语义导航 LLM 理解页面内容,智能选择爬取路径
意图识别 自动解析用户模糊需求,生成精确任务
自动提取 根据目标描述自动识别和提取结构化数据
智能剪枝 自动放弃无效分支,提高爬取效率
格式化输出 自动清洗、规范化输出数据
Socket 通信 基于 TCP Socket 的自定义协议(课设重点)

1.3 技术架构

┌─────────────────────────────────────────────────────────────┐
│                      Client Layer                           │
│              (Socket Client / REST Client)                  │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Communication Layer                       │
│         ┌─────────────────┐    ┌─────────────────┐          │
│         │  Socket Server  │    │   FastAPI REST  │          │
│         │   (TCP:9527)    │    │   (HTTP:8000)   │          │
│         └─────────────────┘    └─────────────────┘          │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Orchestrator Layer                        │
│                     (核心调度引擎)                            │
└─────────────────────────────────────────────────────────────┘
                              │
          ┌───────────────────┼───────────────────┐
          ▼                   ▼                   ▼
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│  State Manager  │  │   Task Queue    │  │  Result Store   │
└─────────────────┘  └─────────────────┘  └─────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      Agent Layer                             │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐       │
│  │ Intent   │ │Navigator │ │Extractor │ │ Pruner   │       │
│  │Recognizer│ │  Agent   │ │  Agent   │ │  Agent   │       │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘       │
│  ┌──────────┐ ┌──────────┐                                  │
│  │Formatter │ │Evaluator │                                  │
│  │  Agent   │ │  Agent   │                                  │
│  └──────────┘ └──────────┘                                  │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      LLM Layer                               │
│                   (GLM-4 API Client)                         │
└─────────────────────────────────────────────────────────────┘


二、系统架构

2.1 完整处理流程

                              用户输入 (模糊需求)
                                     │
                                     ▼
                          ┌─────────────────────┐
                          │   Intent Recognizer │ ── 意图识别
                          │        Agent        │    生成候选 Prompt
                          └──────────┬──────────┘
                                     │
                                     ▼
                          ┌─────────────────────┐
                          │   用户确认/选择      │ ── 交互确认
                          │    增强后的 Prompt   │
                          └──────────┬──────────┘
                                     │
                                     ▼
                          ┌─────────────────────┐
                          │    Orchestrator     │ ── 调度器
                          └──────────┬──────────┘
                                     │
          ┌──────────────────────────┼──────────────────────────┐
          ▼                          ▼                          ▼
    ┌───────────┐             ┌───────────┐             ┌───────────┐
    │  Fetcher  │             │  Fetcher  │             │  Fetcher  │
    │ (Socket)  │             │ (Socket)  │             │ (Socket)  │
    └─────┬─────┘             └─────┬─────┘             └─────┬─────┘
          │                         │                         │
          ▼                         ▼                         ▼
    ┌───────────┐             ┌───────────┐             ┌───────────┐
    │Preprocessor│            │Preprocessor│            │Preprocessor│
    └─────┬─────┘             └─────┬─────┘             └─────┬─────┘
          │                         │                         │
          └─────────────────────────┼─────────────────────────┘
                                    │
                 ┌──────────────────┼──────────────────┐
                 ▼                  ▼                  ▼
          ┌───────────┐      ┌───────────┐      ┌───────────┐
          │ Navigator │      │ Extractor │      │  Pruner   │
          │   Agent   │      │   Agent   │      │   Agent   │
          └─────┬─────┘      └─────┬─────┘      └─────┬─────┘
                │                  │                  │
                │      选择URL     │   提取数据       │  剪枝判断
                │                  │                  │
                └──────────────────┼──────────────────┘
                                   │
                                   ▼
                          ┌─────────────────┐
                          │ 循环直到完成     │
                          └────────┬────────┘
                                   │
                                   ▼ 原始数据
                          ┌─────────────────┐
                          │   Formatter     │ ── 格式化输出
                          │     Agent       │
                          └────────┬────────┘
                                   │
                                   ▼
                          ┌─────────────────┐
                          │   Evaluator     │ ── 质量评估
                          │     Agent       │
                          └────────┬────────┘
                                   │
                                   ▼
                             最终格式化结果
                                   │
                                   ▼
                          ┌─────────────────┐
                          │ Socket/REST     │ ── 返回给客户端
                          │   Response      │
                          └─────────────────┘

2.2 组件依赖图

                         ┌─────────────┐
                         │ CrawlConfig │
                         └──────┬──────┘
                                │
                                ▼
┌───────────┐       ┌─────────────────────┐       ┌───────────┐
│  Fetcher  │──────▶│    Orchestrator     │◀──────│ GLMClient │
└───────────┘       └─────────────────────┘       └───────────┘
                                │
         ┌──────────┬───────────┼───────────┬──────────┐
         ▼          ▼           ▼           ▼          ▼
   ┌──────────┐┌─────────┐┌──────────┐┌─────────┐┌──────────┐
   │Preprocess││  State  ││  Queue   ││ Results ││  Agents  │
   └──────────┘└─────────┘└──────────┘└─────────┘└──────────┘


三、核心数据结构

3.1 基础类型定义

# ==================== core/types.py ====================

from dataclasses import dataclass, field
from typing import List, Dict, Optional, Any
from enum import Enum
from datetime import datetime

class TaskStatus(Enum):
    """任务状态"""
    PENDING = "pending"
    RUNNING = "running"
    COMPLETED = "completed"
    FAILED = "failed"
    CANCELLED = "cancelled"
    TIMEOUT = "timeout"

@dataclass
class CrawlConfig:
    """爬取配置"""
    goal: str                           # 用户需求描述
    root_url: str                       # 起始 URL
    max_depth: int = 5                  # 最大爬取深度
    max_branches: int = 3               # 每层最多展开的分支数
    max_pages: int = 100                # 最多爬取的页面总数
    max_concurrent: int = 5             # 最大并发数
    timeout_per_page: int = 30          # 单页超时时间(秒)
    total_timeout: int = 600            # 任务总超时时间(秒)
    same_domain_only: bool = True       # 是否限制同域名
    render_js: bool = False             # 是否渲染 JavaScript
    respect_robots_txt: bool = True     # 是否遵守 robots.txt
    request_delay: int = 1000           # 请求间隔(毫秒)
    output_schema: Optional[dict] = None  # 期望的输出数据结构

@dataclass
class CrawlTask:
    """爬取任务"""
    url: str
    depth: int
    parent_url: Optional[str] = None
    priority: float = 0.5               # 0-1, 越高越优先
    metadata: Dict[str, Any] = field(default_factory=dict)

@dataclass
class Link:
    """页面中的链接"""
    url: str
    anchor_text: str                    # 链接文字
    context: str                        # 链接周围的文字

@dataclass
class CleanedPage:
    """清洗后的页面"""
    url: str
    title: str
    text_content: str                   # 纯文本内容
    links: List[Link]                   # 结构化的链接列表
    raw_html: str                       # 原始 HTML 备用

@dataclass
class ScoredURL:
    """带评分的 URL"""
    url: str
    score: float                        # 0-1, 找到目标的可能性
    reason: str                         # 选择理由

@dataclass
class ExtractionResult:
    """提取结果"""
    found: bool
    data: List[dict]
    confidence: float

@dataclass
class PruneDecision:
    """剪枝决策"""
    should_prune: bool
    confidence: float
    reason: str

@dataclass
class FormattedResult:
    """格式化后的结果"""
    data: List[dict]
    summary: str
    quality_score: float
    warnings: List[str]

@dataclass
class CrawlProgress:
    """爬取进度"""
    pages_crawled: int = 0
    pages_queued: int = 0
    pages_pruned: int = 0
    current_depth: int = 0
    items_found: int = 0
    current_url: Optional[str] = None

@dataclass
class CrawlResult:
    """爬取最终结果"""
    task_id: str
    status: TaskStatus
    goal: str
    data: List[dict]
    summary: str
    quality_score: float
    statistics: Dict[str, int]
    trace: List[dict]
    created_at: datetime
    completed_at: Optional[datetime] = None
    error: Optional[str] = None

四、Socket 通信层

4.1 协议设计

消息帧格式

┌────────────┬────────────┬────────────┬──────────┬─────────────────┐
│  Magic (2) │ Version(1) │ Type (1)   │Length (4)│   Payload (N)   │
│   0xCAFE   │    0x01    │  MsgType   │  uint32  │   JSON UTF-8    │
└────────────┴────────────┴────────────┴──────────┴─────────────────┘
                          │
                          └─► 总长度 = 8 + N bytes

消息类型定义

class MessageType(Enum):
    """消息类型枚举"""

    # ===== 客户端 -> 服务端 =====
    # 意图识别
    INTENT_ANALYZE = 0x01       # 请求意图分析
    INTENT_SELECT = 0x02        # 用户选择候选
    INTENT_CONFIRM = 0x03       # 确认最终 Prompt

    # 任务管理
    TASK_CREATE = 0x11          # 创建任务
    TASK_QUERY = 0x12           # 查询状态
    TASK_CANCEL = 0x13          # 取消任务
    TASK_RESULT = 0x14          # 获取结果

    # 通用
    HEARTBEAT = 0x20            # 心跳包

    # ===== 服务端 -> 客户端 =====
    # 意图识别响应
    INTENT_RESULT = 0x31        # 意图分析结果
    INTENT_CONFIRMED = 0x32     # 确认成功

    # 任务响应
    TASK_CREATED = 0x41         # 任务已创建
    TASK_STATUS = 0x42          # 状态响应
    TASK_PROGRESS = 0x43        # 进度推送
    TASK_COMPLETED = 0x44       # 任务完成
    TASK_FAILED = 0x45          # 任务失败
    ITEM_FOUND = 0x46           # 发现新数据

    # 通用响应
    ERROR = 0xFF                # 错误响应

4.2 消息协议实现

# ==================== network/protocol.py ====================

import struct
import json
from dataclasses import dataclass
from typing import Any

@dataclass
class Message:
    """协议消息"""
    msg_type: MessageType
    payload: dict

    MAGIC = 0xCAFE
    VERSION = 0x01
    HEADER_SIZE = 8  # 2 (magic) + 1 (version) + 1 (type) + 4 (length)

    def encode(self) -> bytes:
        """编码为字节流"""
        payload_bytes = json.dumps(
            self.payload,
            ensure_ascii=False
        ).encode('utf-8')

        header = struct.pack(
            '>HBBI',  # big-endian: ushort, ubyte, ubyte, uint
            self.MAGIC,
            self.VERSION,
            self.msg_type.value,
            len(payload_bytes)
        )
        return header + payload_bytes

    @classmethod
    def decode(cls, data: bytes) -> 'Message':
        """从字节流解码"""
        if len(data) < cls.HEADER_SIZE:
            raise ValueError("数据长度不足")

        magic, version, msg_type, length = struct.unpack(
            '>HBBI',
            data[:cls.HEADER_SIZE]
        )

        if magic != cls.MAGIC:
            raise ValueError(f"无效的魔数: {magic:#x}")
        if version != cls.VERSION:
            raise ValueError(f"不支持的版本: {version}")

        payload_bytes = data[cls.HEADER_SIZE:cls.HEADER_SIZE + length]
        payload = json.loads(payload_bytes.decode('utf-8'))

        return cls(
            msg_type=MessageType(msg_type),
            payload=payload
        )

    @classmethod
    def read_from_socket(cls, sock) -> 'Message':
        """从 socket 读取完整消息"""
        # 读取头部
        header = cls._recv_exact(sock, cls.HEADER_SIZE)
        magic, version, msg_type, length = struct.unpack('>HBBI', header)

        if magic != cls.MAGIC:
            raise ValueError(f"无效的魔数: {magic:#x}")

        # 读取消息体
        payload_bytes = cls._recv_exact(sock, length)
        payload = json.loads(payload_bytes.decode('utf-8'))

        return cls(msg_type=MessageType(msg_type), payload=payload)

    @staticmethod
    def _recv_exact(sock, n: int) -> bytes:
        """精确接收 n 字节"""
        data = b''
        while len(data) < n:
            chunk = sock.recv(n - len(data))
            if not chunk:
                raise ConnectionError("连接已关闭")
            data += chunk
        return data

4.3 Socket 服务端

# ==================== network/socket_server.py ====================

import socket
import threading
import logging
from typing import Dict, Callable, Optional

from .protocol import Message, MessageType

logger = logging.getLogger(__name__)

class ClientConnection:
    """客户端连接封装"""

    def __init__(self, sock: socket.socket, addr: tuple):
        self.socket = sock
        self.address = addr
        self.subscribed_tasks: set = set()
        self.is_alive = True
        self._lock = threading.Lock()

    def send_message(self, message: Message) -> bool:
        """发送消息到客户端"""
        with self._lock:
            try:
                self.socket.sendall(message.encode())
                return True
            except Exception as e:
                logger.error(f"发送消息失败: {e}")
                self.is_alive = False
                return False

    def receive_message(self) -> Message:
        """接收客户端消息"""
        return Message.read_from_socket(self.socket)

    def close(self):
        """关闭连接"""
        self.is_alive = False
        try:
            self.socket.close()
        except:
            pass

class CrawlerSocketServer:
    """
    爬虫 Socket 服务器

    职责:
    1. 管理客户端 TCP 连接
    2. 接收和解析请求消息
    3. 分发请求到对应处理器
    4. 向客户端推送实时进度
    """

    def __init__(self, host: str = '0.0.0.0', port: int = 9527):
        self.host = host
        self.port = port
        self.server_socket: Optional[socket.socket] = None
        self.clients: Dict[str, ClientConnection] = {}
        self.handlers: Dict[MessageType, Callable] = {}
        self.running = False
        self._lock = threading.Lock()

    def register_handler(self, msg_type: MessageType, handler: Callable):
        """注册消息处理器"""
        self.handlers[msg_type] = handler

    def start(self):
        """启动服务器"""
        self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.server_socket.bind((self.host, self.port))
        self.server_socket.listen(5)
        self.running = True

        logger.info(f"Socket 服务器启动: {self.host}:{self.port}")

        # 启动接受连接的线程
        accept_thread = threading.Thread(target=self._accept_loop, daemon=True)
        accept_thread.start()

    def stop(self):
        """停止服务器"""
        self.running = False
        for client in self.clients.values():
            client.close()
        if self.server_socket:
            self.server_socket.close()

    def _accept_loop(self):
        """接受新连接的循环"""
        while self.running:
            try:
                client_socket, addr = self.server_socket.accept()
                logger.info(f"新连接: {addr}")

                connection = ClientConnection(client_socket, addr)
                addr_key = f"{addr[0]}:{addr[1]}"

                with self._lock:
                    self.clients[addr_key] = connection

                # 为每个客户端启动处理线程
                client_thread = threading.Thread(
                    target=self._handle_client,
                    args=(connection, addr_key),
                    daemon=True
                )
                client_thread.start()

            except Exception as e:
                if self.running:
                    logger.error(f"接受连接失败: {e}")

    def _handle_client(self, connection: ClientConnection, addr_key: str):
        """处理单个客户端的消息"""
        try:
            while connection.is_alive and self.running:
                try:
                    message = connection.receive_message()
                    self._dispatch_message(connection, message)
                except ConnectionError:
                    break
                except Exception as e:
                    logger.error(f"处理消息失败: {e}")
                    error_msg = Message(
                        msg_type=MessageType.ERROR,
                        payload={"error": str(e)}
                    )
                    connection.send_message(error_msg)
        finally:
            with self._lock:
                self.clients.pop(addr_key, None)
            connection.close()
            logger.info(f"连接断开: {addr_key}")

    def _dispatch_message(self, connection: ClientConnection, message: Message):
        """分发消息到对应处理器"""
        handler = self.handlers.get(message.msg_type)
        if handler:
            response = handler(message.payload, connection)
            if response:
                connection.send_message(response)
        else:
            logger.warning(f"未注册的消息类型: {message.msg_type}")

    def broadcast_to_task(self, task_id: str, message: Message):
        """向订阅了特定任务的客户端广播消息"""
        with self._lock:
            for connection in self.clients.values():
                if task_id in connection.subscribed_tasks:
                    connection.send_message(message)

    def notify_progress(self, task_id: str, progress: dict):
        """通知任务进度"""
        message = Message(
            msg_type=MessageType.TASK_PROGRESS,
            payload={"task_id": task_id, "progress": progress}
        )
        self.broadcast_to_task(task_id, message)

    def notify_item_found(self, task_id: str, item: dict):
        """通知发现新数据"""
        message = Message(
            msg_type=MessageType.ITEM_FOUND,
            payload={"task_id": task_id, "item": item}
        )
        self.broadcast_to_task(task_id, message)

    def notify_completed(self, task_id: str, result: dict):
        """通知任务完成"""
        message = Message(
            msg_type=MessageType.TASK_COMPLETED,
            payload={"task_id": task_id, "result": result}
        )
        self.broadcast_to_task(task_id, message)

4.4 Socket 客户端

# ==================== network/socket_client.py ====================

import socket
import threading
import uuid
from typing import Callable, Optional, Dict, Any

from .protocol import Message, MessageType

class CrawlerSocketClient:
    """
    爬虫 Socket 客户端

    用法:
        client = CrawlerSocketClient('localhost', 9527)
        client.connect()

        # 意图识别
        result = client.analyze_intent("找一下老师的邮箱")

        # 选择候选
        selected = result['enhanced_prompts'][0]

        # 创建任务
        task_id = client.create_task(
            goal=selected['prompt'],
            root_url="<https://example.edu.cn>"
        )

        # 注册回调
        client.on_progress(lambda p: print(f"进度: {p}"))

        # 等待完成
        result = client.wait_for_result(task_id)
    """

    def __init__(self, host: str, port: int):
        self.host = host
        self.port = port
        self.socket: Optional[socket.socket] = None
        self.connected = False
        self._callbacks: Dict[str, list] = {
            'progress': [],
            'item_found': [],
            'completed': [],
            'error': []
        }
        self._pending_responses: Dict[str, dict] = {}
        self._response_events: Dict[str, threading.Event] = {}
        self._lock = threading.Lock()
        self._recv_thread: Optional[threading.Thread] = None

    def connect(self):
        """建立连接"""
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.socket.connect((self.host, self.port))
        self.connected = True

        # 启动接收线程
        self._recv_thread = threading.Thread(
            target=self._receive_loop,
            daemon=True
        )
        self._recv_thread.start()

    def disconnect(self):
        """断开连接"""
        self.connected = False
        if self.socket:
            self.socket.close()

    # ===== 回调注册 =====

    def on_progress(self, callback: Callable):
        self._callbacks['progress'].append(callback)

    def on_item_found(self, callback: Callable):
        self._callbacks['item_found'].append(callback)

    def on_completed(self, callback: Callable):
        self._callbacks['completed'].append(callback)

    def on_error(self, callback: Callable):
        self._callbacks['error'].append(callback)

    # ===== 意图识别 =====

    def analyze_intent(
        self,
        prompt: str,
        target_url: str = None
    ) -> dict:
        """分析用户意图"""
        return self._send_and_wait(
            MessageType.INTENT_ANALYZE,
            {"prompt": prompt, "target_url": target_url}
        )

    def select_intent(self, selected_prompt: str, schema: dict = None) -> dict:
        """选择候选 Prompt"""
        return self._send_and_wait(
            MessageType.INTENT_SELECT,
            {"selected_prompt": selected_prompt, "schema": schema}
        )

    def confirm_intent(self, prompt: str, schema: dict) -> dict:
        """确认最终 Prompt"""
        return self._send_and_wait(
            MessageType.INTENT_CONFIRM,
            {"prompt": prompt, "schema": schema}
        )

    # ===== 任务管理 =====

    def create_task(
        self,
        goal: str,
        root_url: str,
        config: dict = None
    ) -> str:
        """创建爬取任务"""
        response = self._send_and_wait(
            MessageType.TASK_CREATE,
            {"goal": goal, "root_url": root_url, "config": config or {}}
        )
        return response.get("task_id")

    def query_task(self, task_id: str) -> dict:
        """查询任务状态"""
        return self._send_and_wait(
            MessageType.TASK_QUERY,
            {"task_id": task_id}
        )

    def cancel_task(self, task_id: str) -> bool:
        """取消任务"""
        response = self._send_and_wait(
            MessageType.TASK_CANCEL,
            {"task_id": task_id}
        )
        return response.get("success", False)

    def get_result(self, task_id: str) -> dict:
        """获取任务结果"""
        return self._send_and_wait(
            MessageType.TASK_RESULT,
            {"task_id": task_id}
        )

    def wait_for_result(self, task_id: str, timeout: int = 600) -> dict:
        """阻塞等待任务完成并返回结果"""
        event = threading.Event()
        result = {}

        def on_complete(data):
            if data.get("task_id") == task_id:
                result.update(data)
                event.set()

        self.on_completed(on_complete)

        if event.wait(timeout):
            return result
        else:
            raise TimeoutError(f"任务 {task_id} 超时")

    # ===== 内部方法 =====

    def _send_and_wait(
        self,
        msg_type: MessageType,
        payload: dict,
        timeout: int = 30
    ) -> dict:
        """发送消息并等待响应"""
        request_id = str(uuid.uuid4())
        payload["_request_id"] = request_id

        event = threading.Event()
        with self._lock:
            self._response_events[request_id] = event

        message = Message(msg_type=msg_type, payload=payload)
        self.socket.sendall(message.encode())

        if event.wait(timeout=timeout):
            with self._lock:
                return self._pending_responses.pop(request_id, {})
        else:
            raise TimeoutError("请求超时")

    def _receive_loop(self):
        """接收消息循环"""
        while self.connected:
            try:
                message = Message.read_from_socket(self.socket)
                self._handle_message(message)
            except Exception as e:
                if self.connected:
                    for cb in self._callbacks['error']:
                        cb({"error": str(e)})
                break

    def _handle_message(self, message: Message):
        """处理接收到的消息"""
        payload = message.payload

        # 检查是否是响应消息
        request_id = payload.get("_request_id")
        if request_id:
            with self._lock:
                self._pending_responses[request_id] = payload
                event = self._response_events.pop(request_id, None)
                if event:
                    event.set()
            return

        # 处理推送消息
        callbacks_map = {
            MessageType.TASK_PROGRESS: 'progress',
            MessageType.ITEM_FOUND: 'item_found',
            MessageType.TASK_COMPLETED: 'completed',
            MessageType.ERROR: 'error'
        }

        callback_key = callbacks_map.get(message.msg_type)
        if callback_key:
            for cb in self._callbacks[callback_key]:
                cb(payload)

4.5 基于 Socket 的 HTTP Fetcher

# ==================== network/socket_fetcher.py ====================

import socket
import ssl
from urllib.parse import urlparse
from dataclasses import dataclass
from typing import Optional, Dict

@dataclass
class FetchResult:
    """获取结果"""
    url: str
    status_code: int
    headers: Dict[str, str]
    body: str
    error: Optional[str] = None

class SocketHttpFetcher:
    """
    基于原始 Socket 的 HTTP 获取器

    为体现计网课设要求,使用原始 Socket 实现 HTTP 协议
    """

    def __init__(self, timeout: int = 30, user_agent: str = None):
        self.timeout = timeout
        self.user_agent = user_agent or "SmartCrawler/1.0"

    def fetch(self, url: str) -> FetchResult:
        """
        获取网页内容

        手动实现 HTTP/1.1 协议:
        1. 解析 URL
        2. 建立 TCP 连接 (HTTPS 则包装 SSL)
        3. 发送 HTTP 请求
        4. 接收并解析 HTTP 响应
        """
        try:
            # 1. 解析 URL
            parsed = urlparse(url)
            host = parsed.hostname
            port = parsed.port or (443 if parsed.scheme == 'https' else 80)
            path = parsed.path or '/'
            if parsed.query:
                path += '?' + parsed.query

            # 2. 建立 TCP 连接
            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.settimeout(self.timeout)

            # 3. 如果是 HTTPS,包装 SSL
            if parsed.scheme == 'https':
                context = ssl.create_default_context()
                sock = context.wrap_socket(sock, server_hostname=host)

            sock.connect((host, port))

            # 4. 构造并发送 HTTP 请求
            request = self._build_request(host, path)
            sock.sendall(request.encode('utf-8'))

            # 5. 接收响应
            response = self._receive_response(sock)

            # 6. 解析响应
            status_code, headers, body = self._parse_response(response)

            sock.close()

            return FetchResult(
                url=url,
                status_code=status_code,
                headers=headers,
                body=body
            )

        except Exception as e:
            return FetchResult(
                url=url,
                status_code=0,
                headers={},
                body='',
                error=str(e)
            )

    def _build_request(self, host: str, path: str) -> str:
        """构造 HTTP 请求"""
        lines = [
            f"GET {path} HTTP/1.1",
            f"Host: {host}",
            f"User-Agent: {self.user_agent}",
            "Accept: text/html,application/xhtml+xml,*/*;q=0.8",
            "Accept-Language: zh-CN,zh;q=0.9,en;q=0.8",
            "Accept-Encoding: identity",
            "Connection: close",
            "",
            ""
        ]
        return "\\r\\n".join(lines)

    def _receive_response(self, sock: socket.socket) -> bytes:
        """接收完整的 HTTP 响应"""
        response = b''
        while True:
            try:
                chunk = sock.recv(4096)
                if not chunk:
                    break
                response += chunk
            except socket.timeout:
                break
        return response

    def _parse_response(self, response: bytes) -> tuple:
        """解析 HTTP 响应"""
        # 分离头部和正文
        header_end = response.find(b'\\r\\n\\r\\n')
        if header_end == -1:
            raise ValueError("无效的 HTTP 响应")

        header_bytes = response[:header_end]
        body_bytes = response[header_end + 4:]

        # 解析状态行和头部
        header_text = header_bytes.decode('utf-8', errors='ignore')
        lines = header_text.split('\\r\\n')

        # 状态行: HTTP/1.1 200 OK
        status_line = lines[0]
        parts = status_line.split(' ', 2)
        status_code = int(parts[1])

        # 解析头部字段
        headers = {}
        for line in lines[1:]:
            if ':' in line:
                key, value = line.split(':', 1)
                headers[key.strip().lower()] = value.strip()

        # 处理 chunked 编码
        if headers.get('transfer-encoding') == 'chunked':
            body_bytes = self._decode_chunked(body_bytes)

        # 解码正文
        charset = self._get_charset(headers)
        body = body_bytes.decode(charset, errors='ignore')

        return status_code, headers, body

    def _decode_chunked(self, data: bytes) -> bytes:
        """解码 chunked 传输编码"""
        result = b''
        while data:
            line_end = data.find(b'\\r\\n')
            if line_end == -1:
                break

            size_str = data[:line_end].decode('ascii')
            chunk_size = int(size_str.split(';')[0], 16)

            if chunk_size == 0:
                break

            chunk_start = line_end + 2
            chunk_end = chunk_start + chunk_size
            result += data[chunk_start:chunk_end]
            data = data[chunk_end + 2:]

        return result

    def _get_charset(self, headers: dict) -> str:
        """从 Content-Type 获取字符集"""
        content_type = headers.get('content-type', '')
        if 'charset=' in content_type:
            charset = content_type.split('charset=')[-1].split(';')[0].strip()
            return charset
        return 'utf-8'

五、GLM API 集成

5.1 GLM Client

# ==================== llm/glm_client.py ====================

import httpx
import json
import time
from typing import List, Dict, Optional, Generator
from dataclasses import dataclass
from abc import ABC, abstractmethod

@dataclass
class ChatMessage:
    """对话消息"""
    role: str  # "user" / "assistant" / "system"
    content: str

@dataclass
class GLMResponse:
    """GLM 响应"""
    content: str
    usage: Dict[str, int]
    finish_reason: str

class LLMClient(ABC):
    """LLM 客户端抽象接口"""

    @abstractmethod
    def chat(self, messages: List[ChatMessage], **kwargs) -> GLMResponse:
        pass

    @abstractmethod
    def chat_json(self, messages: List[ChatMessage], **kwargs) -> dict:
        pass

class GLMClient(LLMClient):
    """
    智谱 GLM API 客户端

    文档: <https://open.bigmodel.cn/dev/api>
    """

    API_BASE = "<https://open.bigmodel.cn/api/paas/v4>"

    def __init__(
        self,
        api_key: str,
        model: str = "glm-4",
        timeout: int = 60,
        max_retries: int = 3
    ):
        self.api_key = api_key
        self.model = model
        self.timeout = timeout
        self.max_retries = max_retries
        self.client = httpx.Client(timeout=timeout)

    def _get_headers(self) -> Dict[str, str]:
        return {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json"
        }

    def chat(
        self,
        messages: List[ChatMessage],
        temperature: float = 0.7,
        max_tokens: int = 2000,
        **kwargs
    ) -> GLMResponse:
        """发送对话请求"""
        payload = {
            "model": self.model,
            "messages": [
                {"role": m.role, "content": m.content}
                for m in messages
            ],
            "temperature": temperature,
            "max_tokens": max_tokens,
            **kwargs
        }

        for attempt in range(self.max_retries):
            try:
                response = self.client.post(
                    f"{self.API_BASE}/chat/completions",
                    headers=self._get_headers(),
                    json=payload
                )
                response.raise_for_status()
                data = response.json()

                return GLMResponse(
                    content=data["choices"][0]["message"]["content"],
                    usage=data.get("usage", {}),
                    finish_reason=data["choices"][0].get("finish_reason", "stop")
                )

            except httpx.HTTPStatusError as e:
                if e.response.status_code == 429:
                    time.sleep(2 ** attempt)
                    continue
                raise
            except Exception as e:
                if attempt == self.max_retries - 1:
                    raise
                time.sleep(1)

        raise RuntimeError("GLM API 调用失败")

    def chat_json(
        self,
        messages: List[ChatMessage],
        **kwargs
    ) -> dict:
        """发送对话请求,返回解析后的 JSON"""
        response = self.chat(messages, **kwargs)
        content = response.content.strip()

        # 提取 JSON 代码块
        if "```json" in content:
            start = content.find("```json") + 7
            end = content.find("```", start)
            content = content[start:end].strip()
        elif "```" in content:
            start = content.find("```") + 3
            end = content.find("```", start)
            content = content[start:end].strip()

        return json.loads(content)

    def chat_stream(
        self,
        messages: List[ChatMessage],
        **kwargs
    ) -> Generator[str, None, None]:
        """流式对话(SSE)"""
        payload = {
            "model": self.model,
            "messages": [
                {"role": m.role, "content": m.content}
                for m in messages
            ],
            "stream": True,
            **kwargs
        }

        with self.client.stream(
            "POST",
            f"{self.API_BASE}/chat/completions",
            headers=self._get_headers(),
            json=payload
        ) as response:
            for line in response.iter_lines():
                if line.startswith("data: "):
                    data = line[6:]
                    if data == "[DONE]":
                        break
                    chunk = json.loads(data)
                    delta = chunk["choices"][0].get("delta", {})
                    if "content" in delta:
                        yield delta["content"]

六、智能体设计

6.1 Agent 基类

# ==================== agents/base.py ====================

from abc import ABC, abstractmethod
from typing import Any
from dataclasses import dataclass

from ..llm.glm_client import LLMClient, ChatMessage

@dataclass
class AgentConfig:
    """Agent 配置"""
    temperature: float = 0.7
    max_tokens: int = 2000
    max_retries: int = 2

class BaseAgent(ABC):
    """
    智能体基类

    所有 Agent 继承此类,实现:
    - 系统提示词
    - 输入格式化
    - 输出解析
    """

    def __init__(self, llm_client: LLMClient, config: AgentConfig = None):
        self.llm = llm_client
        self.config = config or AgentConfig()

    @abstractmethod
    def get_system_prompt(self) -> str:
        """返回系统提示词"""
        pass

    @abstractmethod
    def format_input(self, *args, **kwargs) -> str:
        """格式化输入为用户消息"""
        pass

    @abstractmethod
    def parse_output(self, response: str) -> Any:
        """解析模型输出"""
        pass

    def run(self, *args, **kwargs) -> Any:
        """执行 Agent"""
        user_message = self.format_input(*args, **kwargs)

        messages = [
            ChatMessage(role="system", content=self.get_system_prompt()),
            ChatMessage(role="user", content=user_message)
        ]

        response = self.llm.chat(
            messages,
            temperature=self.config.temperature,
            max_tokens=self.config.max_tokens
        )

        return self.parse_output(response.content)

    def run_json(self, *args, **kwargs) -> dict:
        """执行 Agent,返回 JSON"""
        user_message = self.format_input(*args, **kwargs)

        messages = [
            ChatMessage(role="system", content=self.get_system_prompt()),
            ChatMessage(role="user", content=user_message)
        ]

        return self.llm.chat_json(
            messages,
            temperature=self.config.temperature,
            max_tokens=self.config.max_tokens
        )

6.2 Intent Recognizer Agent(意图识别)

# ==================== agents/intent_recognizer.py ====================

from typing import List, Optional
from dataclasses import dataclass, field
from enum import Enum

from .base import BaseAgent

class IntentType(Enum):
    """意图类型"""
    CONTACT_INFO = "contact_info"
    PERSONNEL_LIST = "personnel_list"
    ARTICLE_CONTENT = "article_content"
    DOCUMENT_FILE = "document_file"
    EVENT_INFO = "event_info"
    PRODUCT_INFO = "product_info"
    GENERAL_DATA = "general_data"
    UNKNOWN = "unknown"

@dataclass
class ExtractedEntity:
    """提取的实体"""
    entity_type: str
    value: str
    confidence: float

@dataclass
class EnhancedPrompt:
    """增强后的 Prompt"""
    prompt: str
    description: str
    specificity_score: float
    inferred_schema: Optional[dict]
    tags: List[str] = field(default_factory=list)

@dataclass
class IntentRecognitionResult:
    """意图识别结果"""
    original_prompt: str
    intent_type: IntentType
    confidence: float
    entities: List[ExtractedEntity]
    ambiguities: List[str]
    enhanced_prompts: List[EnhancedPrompt]
    analysis: str

class IntentRecognizerAgent(BaseAgent):
    """
    意图识别智能体

    职责:
    1. 分析用户的模糊输入
    2. 识别意图类型和关键实体
    3. 检测歧义和不明确之处
    4. 生成多个具体化的候选 Prompt
    5. 推断可能的输出数据结构
    """

    def get_system_prompt(self) -> str:
        return """你是一个专业的意图识别和需求分析专家。分析用户的网页数据抓取需求,识别模糊之处,并生成更具体的任务描述。

## 工作流程

### 1. 意图分类
- contact_info: 联系方式(邮箱、电话等)
- personnel_list: 人员列表
- article_content: 文章、新闻
- document_file: 文档文件
- event_info: 活动信息
- product_info: 产品信息
- general_data: 其他数据

### 2. 实体识别
- organization: 组织名称
- person_type: 人员类型
- data_field: 数据字段
- time_range: 时间范围
- scope: 范围限定
- condition: 筛选条件

### 3. 生成候选 Prompt
生成 3-5 个不同具体程度的候选

## 输出格式(JSON)
{
    "original_prompt": "用户原始输入",
    "intent_type": "contact_info",
    "confidence": 0.85,
    "entities": [
        {"entity_type": "organization", "value": "信息学院", "confidence": 0.95}
    ],
    "ambiguities": ["'联系方式'具体指哪些信息不明确"],
    "enhanced_prompts": [
        {
            "prompt": "获取信息学院全体教师的电子邮箱地址",
            "description": "仅邮箱",
            "specificity_score": 0.9,
            "inferred_schema": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "email": {"type": "string", "format": "email"}
                    }
                }
            },
            "tags": ["简洁", "邮箱"]
        }
    ],
    "analysis": "分析说明"
}"""

    def format_input(
        self,
        user_prompt: str,
        target_url: Optional[str] = None,
        context: Optional[str] = None
    ) -> str:
        url_section = f"\\n## 目标网站\\n{target_url}" if target_url else ""
        context_section = f"\\n## 额外上下文\\n{context}" if context else ""

        return f"""## 用户输入
"{user_prompt}"
{url_section}{context_section}

请分析这个需求,识别意图,并生成多个具体化的候选任务描述。"""

    def parse_output(self, response: str) -> IntentRecognitionResult:
        import json

        content = response.strip()
        if "```json" in content:
            start = content.find("```json") + 7
            end = content.find("```", start)
            content = content[start:end]

        data = json.loads(content)

        entities = [
            ExtractedEntity(**e) for e in data.get("entities", [])
        ]

        enhanced_prompts = [
            EnhancedPrompt(
                prompt=p["prompt"],
                description=p.get("description", ""),
                specificity_score=p.get("specificity_score", 0.8),
                inferred_schema=p.get("inferred_schema"),
                tags=p.get("tags", [])
            )
            for p in data.get("enhanced_prompts", [])
        ]

        try:
            intent_type = IntentType(data.get("intent_type", "unknown"))
        except ValueError:
            intent_type = IntentType.UNKNOWN

        return IntentRecognitionResult(
            original_prompt=data.get("original_prompt", ""),
            intent_type=intent_type,
            confidence=data.get("confidence", 0.0),
            entities=entities,
            ambiguities=data.get("ambiguities", []),
            enhanced_prompts=enhanced_prompts,
            analysis=data.get("analysis", "")
        )

    async def recognize(
        self,
        user_prompt: str,
        target_url: Optional[str] = None
    ) -> IntentRecognitionResult:
        """识别用户意图"""
        return self.run(user_prompt, target_url)

    async def quick_enhance(
        self,
        user_prompt: str,
        num_candidates: int = 4
    ) -> List[EnhancedPrompt]:
        """快速生成增强的候选 Prompt"""
        result = await self.recognize(user_prompt)
        return result.enhanced_prompts[:num_candidates]

6.3 Navigator Agent(导航)

# ==================== agents/navigator.py ====================

from typing import List
from .base import BaseAgent
from ..core.types import CleanedPage, ScoredURL

class NavigatorAgent(BaseAgent):
    """
    导航智能体

    职责:分析当前页面,选出最可能找到目标信息的 N 个链接
    """

    def get_system_prompt(self) -> str:
        return """你是一个专业的网页导航专家。分析页面内容和链接,找出最可能包含目标信息的链接。

输出 JSON 格式:
{
    "analysis": "简要分析",
    "candidates": [
        {"url": "完整URL", "score": 0.95, "reason": "选择理由"}
    ]
}

注意:
- score 越高表示越可能找到目标
- 只选择真正相关的链接"""

    def format_input(
        self,
        page: CleanedPage,
        goal: str,
        top_n: int = 3
    ) -> str:
        links_text = "\\n".join([
            f"- [{link.anchor_text}]({link.url})\\n  上下文: {link.context}"
            for link in page.links[:50]
        ])

        return f"""## 用户目标
{goal}

## 当前页面
标题: {page.title}
URL: {page.url}

## 页面内容摘要
{page.text_content[:2000]}

## 页面链接
{links_text}

请选出最多 {top_n} 个最可能找到目标信息的链接。"""

    def parse_output(self, response: str) -> List[ScoredURL]:
        import json

        content = response.strip()
        if "```json" in content:
            start = content.find("```json") + 7
            end = content.find("```", start)
            content = content[start:end]

        data = json.loads(content)

        return [
            ScoredURL(
                url=item["url"],
                score=item["score"],
                reason=item["reason"]
            )
            for item in data.get("candidates", [])
        ]

    async def navigate(
        self,
        page: CleanedPage,
        goal: str,
        top_n: int = 3
    ) -> List[ScoredURL]:
        return self.run(page, goal, top_n)

6.4 Extractor Agent(提取)

# ==================== agents/extractor.py ====================

from typing import Optional
from .base import BaseAgent
from ..core.types import CleanedPage, ExtractionResult

class ExtractorAgent(BaseAgent):
    """
    提取智能体

    职责:判断页面是否包含目标信息,如果有则提取
    """

    def get_system_prompt(self) -> str:
        return """你是一个专业的信息提取专家。从网页内容中识别并提取用户需要的信息。

输出 JSON 格式:
{
    "found": true/false,
    "confidence": 0.95,
    "data": [{"字段1": "值1", "字段2": "值2"}],
    "notes": "补充说明"
}

注意:
- found 为 true 时必须提供 data
- 确保提取的数据准确"""

    def format_input(
        self,
        page: CleanedPage,
        goal: str,
        schema: Optional[dict] = None
    ) -> str:
        schema_text = ""
        if schema:
            import json
            schema_text = f"\\n## 期望格式\\n```json\\n{json.dumps(schema, ensure_ascii=False, indent=2)}\\n```"

        return f"""## 用户目标
{goal}
{schema_text}

## 当前页面
标题: {page.title}
URL: {page.url}

## 页面内容
{page.text_content[:4000]}

请判断页面是否包含目标信息,如果有请提取。"""

    def parse_output(self, response: str) -> ExtractionResult:
        import json

        content = response.strip()
        if "```json" in content:
            start = content.find("```json") + 7
            end = content.find("```", start)
            content = content[start:end]

        data = json.loads(content)

        return ExtractionResult(
            found=data.get("found", False),
            data=data.get("data", []),
            confidence=data.get("confidence", 0.0)
        )

    async def extract(
        self,
        page: CleanedPage,
        goal: str,
        schema: Optional[dict] = None
    ) -> ExtractionResult:
        return self.run(page, goal, schema)

6.5 Pruner Agent(剪枝)

# ==================== agents/pruner.py ====================

from typing import List
from .base import BaseAgent
from ..core.types import CleanedPage, PruneDecision

class PrunerAgent(BaseAgent):
    """
    剪枝智能体

    职责:判断当前分支是否值得继续探索
    """

    def get_system_prompt(self) -> str:
        return """你是一个搜索效率专家。判断当前的网页爬取分支是否还有可能找到目标信息。

判断依据:
1. 页面主题是否与目标相关
2. 页面链接是否可能通向目标
3. 是否已经偏离目标方向

输出 JSON 格式:
{
    "should_prune": true/false,
    "confidence": 0.85,
    "reason": "判断理由"
}

注意:宁可多探索也不要误剪掉有价值的分支"""

    def format_input(
        self,
        page: CleanedPage,
        goal: str,
        path: List[str]
    ) -> str:
        path_text = " → ".join(path[-5:])

        return f"""## 用户目标
{goal}

## 访问路径
{path_text}

## 当前页面
标题: {page.title}
URL: {page.url}

## 页面内容摘要
{page.text_content[:1500]}

## 链接数量
{len(page.links)} 个链接

请判断是否应该剪枝。"""

    def parse_output(self, response: str) -> PruneDecision:
        import json

        content = response.strip()
        if "```json" in content:
            start = content.find("```json") + 7
            end = content.find("```", start)
            content = content[start:end]

        data = json.loads(content)

        return PruneDecision(
            should_prune=data.get("should_prune", False),
            confidence=data.get("confidence", 0.0),
            reason=data.get("reason", "")
        )

    async def should_prune(
        self,
        page: CleanedPage,
        goal: str,
        path: List[str]
    ) -> PruneDecision:
        return self.run(page, goal, path)

6.6 Formatter Agent(格式化)

# ==================== agents/formatter.py ====================

from typing import List, Optional
from .base import BaseAgent
from ..core.types import FormattedResult

class FormatterAgent(BaseAgent):
    """
    格式化智能体

    职责:处理原始数据,使其符合用户期望的格式

    功能:
    1. 数据清洗和规范化
    2. 字段重命名和映射
    3. 去重和合并
    4. 格式转换
    5. 质量评估
    """

    def get_system_prompt(self) -> str:
        return """你是一个专业的数据格式化专家。将爬虫提取的原始数据处理成用户期望的最终格式。

职责:
1. **数据清洗**: 移除无效、不完整或错误的数据
2. **格式规范化**: 统一日期、电话、邮箱格式等
3. **字段映射**: 将原始字段名映射到期望的字段名
4. **去重合并**: 识别并合并重复记录
5. **质量评估**: 评估数据的完整性和可靠性

输出 JSON 格式:
{
    "data": [{"标准字段1": "规范化的值1"}],
    "summary": "数据处理总结",
    "quality_score": 0.92,
    "warnings": ["警告信息"],
    "statistics": {
        "total_raw": 50,
        "total_valid": 45,
        "duplicates_removed": 3
    }
}"""

    def format_input(
        self,
        raw_data: List[dict],
        goal: str,
        output_schema: Optional[dict] = None
    ) -> str:
        import json

        raw_data_text = json.dumps(raw_data, ensure_ascii=False, indent=2)

        schema_text = ""
        if output_schema:
            schema_text = f"\\n## 期望输出格式\\n```json\\n{json.dumps(output_schema, ensure_ascii=False, indent=2)}\\n```"

        return f"""## 用户原始目标
{goal}
{schema_text}

## 原始提取数据
共 {len(raw_data)} 条记录

```json
{raw_data_text}

请处理这些数据,使其符合用户期望格式。"""

def parse_output(self, response: str) -> FormattedResult:
    import json

    content = response.strip()
    if "```json" in content:
        start = content.find("```json") + 7
        end = content.find("```", start)
        content = content[start:end]

    data = json.loads(content)

    return FormattedResult(
        data=data.get("data", []),
        summary=data.get("summary", ""),
        quality_score=data.get("quality_score", 0.0),
        warnings=data.get("warnings", [])
    )

async def format(
    self,
    raw_data: List[dict],
    goal: str,
    output_schema: Optional[dict] = None
) -> FormattedResult:
    return self.run(raw_data, goal, output_schema)


### 6.7 Evaluator Agent(评估)

```python
# ==================== agents/evaluator.py ====================

from typing import List
from dataclasses import dataclass
from .base import BaseAgent

@dataclass
class EvaluationResult:
    """评估结果"""
    is_complete: bool
    completeness_score: float
    missing_info: List[str]
    suggestions: List[str]

class EvaluatorAgent(BaseAgent):
    """
    评估智能体

    职责:评估爬取结果是否满足用户需求
    """

    def get_system_prompt(self) -> str:
        return """你是一个数据质量评估专家。评估爬取结果是否满足用户需求。

评估维度:
1. **完整性**: 是否获取了足够的数据
2. **准确性**: 数据是否看起来正确
3. **覆盖度**: 是否覆盖了目标范围

输出 JSON 格式:
{
    "is_complete": true/false,
    "completeness_score": 0.85,
    "missing_info": ["缺失信息1"],
    "suggestions": ["改进建议1"],
    "analysis": "详细分析"
}"""

    def format_input(self, results: List[dict], goal: str) -> str:
        import json

        return f"""## 用户目标
{goal}

## 已获取的数据
共 {len(results)} 条记录

```json
{json.dumps(results[:20], ensure_ascii=False, indent=2)}

{"(仅显示前20条)" if len(results) > 20 else ""}

请评估这些数据是否满足用户需求。"""

def parse_output(self, response: str) -> EvaluationResult:
    import json

    content = response.strip()
    if "```json" in content:
        start = content.find("```json") + 7
        end = content.find("```", start)
        content = content[start:end]

    data = json.loads(content)

    return EvaluationResult(
        is_complete=data.get("is_complete", False),
        completeness_score=data.get("completeness_score", 0.0),
        missing_info=data.get("missing_info", []),
        suggestions=data.get("suggestions", [])
    )

async def evaluate(
    self,
    results: List[dict],
    goal: str
) -> EvaluationResult:
    return self.run(results, goal)


### 6.8 Agent 总览

| Agent | 阶段 | 职责 | 输入 | 输出 |
|-------|------|------|------|------|
| **IntentRecognizer** | 前置 | 意图识别、Prompt增强 | 用户模糊输入 | 候选Prompt列表 |
| **Navigator** | 爬取中 | 选择最可能的URL | 页面+目标 | 带评分URL列表 |
| **Extractor** | 爬取中 | 提取目标数据 | 页面+目标 | 提取结果 |
| **Pruner** | 爬取中 | 判断是否剪枝 | 页面+目标+路径 | 剪枝决策 |
| **Formatter** | 后置 | 格式化输出 | 原始数据+目标 | 格式化结果 |
| **Evaluator** | 后置 | 质量评估 | 结果+目标 | 评估报告 |

---

## 七、核心组件

### 7.1 HTML 预处理器

```python
# ==================== preprocessors/html_cleaner.py ====================

import re
from typing import List
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

from ..core.types import CleanedPage, Link

class HTMLPreprocessor:
    """
    HTML 预处理器

    职责:清洗 HTML,提取结构化内容
    """

    def __init__(self, context_chars: int = 50):
        self.context_chars = context_chars
        self.remove_tags = ['script', 'style', 'noscript', 'iframe', 'svg']

    def process(self, html: str, url: str) -> CleanedPage:
        """处理 HTML,返回清洗后的页面"""
        soup = BeautifulSoup(html, 'html.parser')

        # 移除无用标签
        for tag in self.remove_tags:
            for element in soup.find_all(tag):
                element.decompose()

        # 移除注释
        for comment in soup.find_all(string=lambda t: isinstance(t, Comment)):
            comment.extract()

        # 提取标题
        title = self._extract_title(soup)

        # 提取链接
        links = self._extract_links(soup, url)

        # 提取文本内容
        text_content = self._extract_text(soup)

        return CleanedPage(
            url=url,
            title=title,
            text_content=text_content,
            links=links,
            raw_html=html
        )

    def _extract_title(self, soup: BeautifulSoup) -> str:
        """提取页面标题"""
        if soup.title and soup.title.string:
            return soup.title.string.strip()

        h1 = soup.find('h1')
        if h1:
            return h1.get_text(strip=True)

        return ""

    def _extract_links(self, soup: BeautifulSoup, base_url: str) -> List[Link]:
        """提取所有链接"""
        links = []

        for a in soup.find_all('a', href=True):
            href = a['href']

            # 跳过无效链接
            if href.startswith(('#', 'javascript:', 'mailto:', 'tel:')):
                continue

            # 转换为绝对 URL
            absolute_url = urljoin(base_url, href)

            # 获取锚文本
            anchor_text = a.get_text(strip=True)
            if not anchor_text:
                anchor_text = a.get('title', '') or href

            # 获取上下文
            context = self._get_context(a)

            links.append(Link(
                url=absolute_url,
                anchor_text=anchor_text,
                context=context
            ))

        return links

    def _get_context(self, element) -> str:
        """获取元素周围的上下文文本"""
        parent = element.parent
        if not parent:
            return ""

        text = parent.get_text(strip=True)
        return text[:self.context_chars * 2]

    def _extract_text(self, soup: BeautifulSoup) -> str:
        """提取页面纯文本"""
        text = soup.get_text(separator='\\n', strip=True)
        # 合并多个空行
        text = re.sub(r'\\n{3,}', '\\n\\n', text)
        return text

7.2 状态管理器

# ==================== storage/state_manager.py ====================

from typing import Set, Dict, Optional
from dataclasses import dataclass, field
import threading

@dataclass
class CrawlStats:
    """爬取统计"""
    total_visited: int = 0
    total_in_progress: int = 0
    total_failed: int = 0

class StateManager:
    """
    状态管理器

    职责:维护爬取状态,线程安全
    """

    def __init__(self):
        self.visited: Set[str] = set()
        self.in_progress: Set[str] = set()
        self.failed: Dict[str, str] = {}
        self._lock = threading.Lock()

    def mark_in_progress(self, url: str) -> bool:
        """标记 URL 正在处理"""
        with self._lock:
            if url in self.visited or url in self.in_progress:
                return False
            self.in_progress.add(url)
            return True

    def mark_visited(self, url: str):
        """标记 URL 已访问"""
        with self._lock:
            self.in_progress.discard(url)
            self.visited.add(url)

    def mark_failed(self, url: str, error: str):
        """标记 URL 失败"""
        with self._lock:
            self.in_progress.discard(url)
            self.failed[url] = error

    def is_visited(self, url: str) -> bool:
        """检查是否已访问"""
        with self._lock:
            return url in self.visited

    def can_visit(self, url: str) -> bool:
        """检查是否可以访问"""
        with self._lock:
            return url not in self.visited and url not in self.in_progress

    def get_stats(self) -> CrawlStats:
        """获取统计信息"""
        with self._lock:
            return CrawlStats(
                total_visited=len(self.visited),
                total_in_progress=len(self.in_progress),
                total_failed=len(self.failed)
            )

    def reset(self):
        """重置状态"""
        with self._lock:
            self.visited.clear()
            self.in_progress.clear()
            self.failed.clear()

7.3 任务队列

# ==================== storage/task_queue.py ====================

import heapq
from typing import Optional, List
from dataclasses import dataclass, field
import threading

from ..core.types import CrawlTask

class PriorityTaskQueue:
    """
    优先级任务队列

    按 priority 降序排列(priority 越高越先执行)
    """

    def __init__(self):
        self._heap: List[tuple] = []
        self._counter = 0
        self._lock = threading.Lock()

    def push(self, task: CrawlTask):
        """添加任务"""
        with self._lock:
            # 使用负数实现降序(heapq 是最小堆)
            heapq.heappush(
                self._heap,
                (-task.priority, self._counter, task)
            )
            self._counter += 1

    def pop(self) -> Optional[CrawlTask]:
        """取出最高优先级任务"""
        with self._lock:
            if self._heap:
                _, _, task = heapq.heappop(self._heap)
                return task
            return None

    def push_batch(self, tasks: List[CrawlTask]):
        """批量添加任务"""
        for task in tasks:
            self.push(task)

    def is_empty(self) -> bool:
        """检查是否为空"""
        with self._lock:
            return len(self._heap) == 0

    def size(self) -> int:
        """获取队列大小"""
        with self._lock:
            return len(self._heap)

    def clear(self):
        """清空队列"""
        with self._lock:
            self._heap.clear()

7.4 结果存储

# ==================== storage/result_store.py ====================

from typing import List, Optional
from dataclasses import dataclass
import threading
import json

@dataclass
class StoredItem:
    """存储的数据项"""
    data: dict
    source_url: str
    extracted_at: str

class ResultStore:
    """
    结果存储

    职责:存储和聚合爬取结果
    """

    def __init__(self):
        self._items: List[StoredItem] = []
        self._lock = threading.Lock()

    def add(self, data: dict, source_url: str):
        """添加数据"""
        from datetime import datetime

        with self._lock:
            self._items.append(StoredItem(
                data=data,
                source_url=source_url,
                extracted_at=datetime.now().isoformat()
            ))

    def add_batch(self, items: List[dict], source_url: str):
        """批量添加"""
        for item in items:
            self.add(item, source_url)

    def get_all(self) -> List[dict]:
        """获取所有数据"""
        with self._lock:
            return [item.data for item in self._items]

    def get_all_with_metadata(self) -> List[dict]:
        """获取所有数据(含元数据)"""
        with self._lock:
            return [
                {
                    **item.data,
                    "_metadata": {
                        "source_url": item.source_url,
                        "extracted_at": item.extracted_at
                    }
                }
                for item in self._items
            ]

    def count(self) -> int:
        """获取数据条数"""
        with self._lock:
            return len(self._items)

    def is_goal_satisfied(self, min_count: int = 1) -> bool:
        """检查是否满足目标"""
        return self.count() >= min_count

    def export_json(self) -> str:
        """导出为 JSON"""
        return json.dumps(
            self.get_all_with_metadata(),
            ensure_ascii=False,
            indent=2
        )

    def clear(self):
        """清空存储"""
        with self._lock:
            self._items.clear()

八、RESTful API

8.1 API 概览

Base URL: http://localhost:8000/api/v1

模块 端点 说明
意图识别 /intent/* Prompt 增强
任务管理 /tasks/* 创建、查询、取消
结果查询 /tasks/{id}/results 获取和导出
配置管理 /config/* 默认配置、验证
系统状态 /health, /usage 健康检查、统计

8.2 意图识别 API

分析意图

POST /intent/analyze

请求体:

{
    "prompt": "找一下信息学院老师的联系方式",
    "target_url": "<https://www.example-university.edu.cn>"
}

响应:

{
    "original_prompt": "找一下信息学院老师的联系方式",
    "intent_type": "contact_info",
    "confidence": 0.85,
    "entities": [
        {"entity_type": "organization", "value": "信息学院", "confidence": 0.95},
        {"entity_type": "person_type", "value": "教师", "confidence": 0.80}
    ],
    "ambiguities": [
        "'联系方式'具体指哪些信息不明确",
        "'老师'的范围不明确"
    ],
    "enhanced_prompts": [
        {
            "prompt": "获取信息学院全体教师的电子邮箱地址",
            "description": "仅邮箱 - 最基础的联系方式",
            "specificity_score": 0.9,
            "inferred_schema": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "email": {"type": "string", "format": "email"}
                    }
                }
            },
            "tags": ["简洁", "邮箱"]
        },
        {
            "prompt": "获取信息学院全体教师的姓名、职称、邮箱、办公电话",
            "description": "标准联系信息",
            "specificity_score": 0.95,
            "tags": ["标准", "推荐"]
        }
    ],
    "analysis": "用户希望获取信息学院教师的联系方式..."
}

快速增强

POST /intent/quick-enhance

请求体:

{
    "prompt": "找老师邮箱",
    "num_candidates": 4
}

确认 Prompt

POST /intent/confirm

请求体:

{
    "selected_prompt": "获取信息学院全体教师的姓名、职称、邮箱",
    "inferred_schema": {...}
}

8.3 任务管理 API

创建任务

POST /tasks

请求体:

{
    "goal": "获取信息学院所有教师的姓名、职称和邮箱",
    "root_url": "<https://www.example-university.edu.cn>",
    "config": {
        "max_depth": 4,
        "max_branches": 5,
        "same_domain_only": true
    },
    "output_schema": {
        "type": "array",
        "items": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "title": {"type": "string"},
                "email": {"type": "string", "format": "email"}
            }
        }
    }
}

响应: 201 Created

{
    "task_id": "task_a1b2c3d4e5f6",
    "status": "pending",
    "created_at": "2024-01-15T10:30:00Z",
    "message": "任务已创建"
}

查询任务状态

GET /tasks/{task_id}

响应:

{
    "task_id": "task_a1b2c3d4e5f6",
    "status": "running",
    "goal": "获取信息学院所有教师的姓名、职称和邮箱",
    "progress": {
        "pages_crawled": 15,
        "pages_queued": 8,
        "pages_pruned": 3,
        "current_depth": 2,
        "items_found": 12
    },
    "created_at": "2024-01-15T10:30:00Z",
    "started_at": "2024-01-15T10:30:05Z"
}

任务列表

GET /tasks?status=running&limit=20&offset=0

取消任务

POST /tasks/{task_id}/cancel

8.4 结果查询 API

获取结果

GET /tasks/{task_id}/results?format=json&limit=100

响应:

{
    "task_id": "task_a1b2c3d4e5f6",
    "total": 45,
    "data": [
        {
            "name": "张三",
            "title": "教授",
            "email": "zhangsan@example.edu.cn",
            "_metadata": {
                "source_url": "<https://www.example.edu.cn/faculty/zhangsan>",
                "extracted_at": "2024-01-15T10:33:12Z",
                "confidence": 0.95
            }
        }
    ]
}

导出结果

POST /tasks/{task_id}/results/export

请求体:

{
    "format": "xlsx",
    "include_metadata": true
}

获取爬取路径

GET /tasks/{task_id}/trace

8.5 配置 API

获取默认配置

GET /config/defaults

验证 URL

POST /config/validate-url

8.6 任务状态说明

状态 说明
pending 等待执行
running 正在执行
completed 已完成
failed 执行失败
cancelled 已取消
timeout 超时终止

九、WebSocket 实时通信

9.1 连接

wss://localhost:8000/ws?token=<your_token>

9.2 订阅任务

{"action": "subscribe", "task_id": "task_a1b2c3d4e5f6"}

9.3 事件类型

task.progress

{
    "event": "task.progress",
    "task_id": "task_a1b2c3d4e5f6",
    "data": {
        "pages_crawled": 15,
        "items_found": 12,
        "current_url": "https://..."
    }
}

task.item_found

{
    "event": "task.item_found",
    "task_id": "task_a1b2c3d4e5f6",
    "data": {
        "item": {"name": "张三", "email": "..."},
        "source_url": "https://..."
    }
}

task.completed

{
    "event": "task.completed",
    "task_id": "task_a1b2c3d4e5f6",
    "data": {
        "status": "completed",
        "total_items": 45
    }
}

十、错误处理

10.1 错误响应格式

{
    "error": {
        "code": "VALIDATION_ERROR",
        "message": "请求参数验证失败",
        "details": [
            {"field": "root_url", "message": "必须是有效的 URL 格式"}
        ],
        "request_id": "req_abc123xyz"
    }
}

10.2 HTTP 错误码

状态码 错误码 描述
400 VALIDATION_ERROR 请求参数验证失败
401 UNAUTHORIZED 未认证
403 QUOTA_EXCEEDED 配额已用尽
404 TASK_NOT_FOUND 任务不存在
422 URL_NOT_ACCESSIBLE 目标 URL 无法访问
429 RATE_LIMITED 请求频率超限
500 INTERNAL_ERROR 服务器内部错误

10.3 任务执行错误码

错误码 描述
CRAWL_TIMEOUT 爬取超时
PAGE_LOAD_FAILED 页面加载失败
LLM_ERROR LLM 服务异常
ALL_BRANCHES_PRUNED 所有分支被剪枝
NO_DATA_FOUND 未找到任何数据

十一、项目结构

smart_crawler/
├── core/
│   ├── __init__.py
│   ├── types.py              # 数据类型定义
│   ├── config.py             # 配置管理
│   └── exceptions.py         # 自定义异常
│
├── network/                   # ⭐ Socket 编程核心
│   ├── __init__.py
│   ├── protocol.py           # 消息协议定义
│   ├── socket_server.py      # TCP Socket 服务端
│   ├── socket_client.py      # TCP Socket 客户端
│   └── socket_fetcher.py     # 基于 Socket 的 HTTP 获取器
│
├── llm/                       # ⭐ GLM 集成
│   ├── __init__.py
│   ├── base.py               # LLM 抽象接口
│   └── glm_client.py         # 智谱 GLM 客户端
│
├── agents/                    # ⭐ 智能体
│   ├── __init__.py
│   ├── base.py               # Agent 基类
│   ├── intent_recognizer.py  # 意图识别
│   ├── navigator.py          # 导航
│   ├── extractor.py          # 提取
│   ├── pruner.py             # 剪枝
│   ├── formatter.py          # 格式化
│   └── evaluator.py          # 评估
│
├── preprocessors/
│   ├── __init__.py
│   ├── base.py               # 预处理器接口
│   └── html_cleaner.py       # HTML 清洗器
│
├── storage/
│   ├── __init__.py
│   ├── state_manager.py      # 状态管理
│   ├── task_queue.py         # 任务队列
│   └── result_store.py       # 结果存储
│
├── api/                       # FastAPI REST 层
│   ├── __init__.py
│   ├── main.py               # FastAPI 应用
│   ├── routes/
│   │   ├── intent.py         # 意图识别路由
│   │   ├── tasks.py          # 任务路由
│   │   ├── results.py        # 结果路由
│   │   └── config.py         # 配置路由
│   └── schemas.py            # Pydantic 模型
│
├── orchestrator.py           # 核心调度器
├── server.py                 # 服务启动入口
├── client.py                 # 客户端工具
├── requirements.txt
└── README.md


十二、使用示例

12.1 Python SDK

from smart_crawler import SmartCrawlerClient

# 连接服务器
client = SmartCrawlerClient(host='localhost', port=9527)
client.connect()

# 1. 意图识别
result = client.analyze_intent(
    prompt="找一下信息学院老师的联系方式",
    target_url="<https://www.example-university.edu.cn>"
)

# 2. 显示候选
print("候选任务:")
for i, p in enumerate(result.enhanced_prompts, 1):
    print(f"  [{i}] {p.prompt}")
    print(f"      {p.description}")

# 3. 选择并创建任务
selected = result.enhanced_prompts[1]
task_id = client.create_task(
    goal=selected.prompt,
    root_url="<https://www.example-university.edu.cn>",
    config={"max_depth": 4, "max_branches": 5}
)

# 4. 注册回调
client.on_progress(lambda p: print(f"进度: {p['items_found']} 条"))
client.on_item_found(lambda item: print(f"新数据: {item['name']}"))

# 5. 等待完成
results = client.wait_for_result(task_id, timeout=600)
print(f"完成!共 {len(results['data'])} 条数据")

# 6. 断开连接
client.disconnect()

12.2 命令行

# 交互式爬取
$ smart-crawler crawl <https://www.example-university.edu.cn>

🔍 请输入您的需求: 找一下信息学院老师的联系方式

📋 意图分析完成!

📝 候选任务:
  [1] 获取信息学院全体教师的邮箱地址
  [2] 获取信息学院全体教师的姓名、职称、邮箱、办公电话 ⭐推荐
  [3] 获取信息学院全体教师的完整信息
  [4] 仅获取信息学院教授的联系方式

请选择 (1-4): 2

✅ 已确认: 获取信息学院全体教师的姓名、职称、邮箱、办公电话

开始爬取...
  [▓▓▓▓░░░░░░] 40% | 已爬取: 15 | 已找到: 12

✅ 完成!共找到 45 条数据
📄 结果已保存至: ./results/teachers.json

12.3 cURL

# 意图分析
curl -X POST <http://localhost:8000/api/v1/intent/analyze> \\
  -H "Content-Type: application/json" \\
  -d '{
    "prompt": "找一下信息学院老师的联系方式",
    "target_url": "<https://www.example-university.edu.cn>"
  }'

# 创建任务
curl -X POST <http://localhost:8000/api/v1/tasks> \\
  -H "Content-Type: application/json" \\
  -d '{
    "goal": "获取信息学院全体教师的姓名、职称、邮箱",
    "root_url": "<https://www.example-university.edu.cn>",
    "config": {"max_depth": 4}
  }'

# 查询状态
curl <http://localhost:8000/api/v1/tasks/task_abc123>

# 获取结果
curl <http://localhost:8000/api/v1/tasks/task_abc123/results>

十三、课设亮点

13.1 Socket 编程体现

知识点 实现位置 说明
TCP Socket 编程 socket_server.py 服务端多客户端管理
TCP 客户端 socket_client.py 长连接、心跳机制
HTTP 协议实现 socket_fetcher.py 手动构造请求、解析响应
自定义协议 protocol.py 魔数、版本、消息帧
多线程并发 socket_server.py 每客户端独立线程
SSL/TLS socket_fetcher.py HTTPS 支持
粘包处理 protocol.py 固定帧头 + 长度字段

13.2 自定义协议设计

┌────────────┬────────────┬────────────┬──────────┬─────────────────┐
│  Magic (2) │ Version(1) │ Type (1)   │Length (4)│   Payload (N)   │
│   0xCAFE   │    0x01    │  MsgType   │  uint32  │   JSON UTF-8    │
└────────────┴────────────┴────────────┴──────────┴─────────────────┘

  • 魔数校验: 识别有效数据包
  • 版本控制: 协议升级兼容
  • 类型字段: 区分不同消息
  • 长度字段: 解决粘包问题

13.3 HTTP 协议手动实现

# socket_fetcher.py 中手动实现:

# 1. TCP 连接
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))

# 2. 构造 HTTP 请求
request = f"GET {path} HTTP/1.1\\r\\nHost: {host}\\r\\n\\r\\n"
sock.sendall(request.encode())

# 3. 接收并解析响应
# - 解析状态行
# - 解析头部字段
# - 处理 chunked 编码
# - 解码正文

13.4 技术创新点

  1. LLM 驱动的启发式搜索: 用语义理解代替规则匹配
  2. 意图识别 + Prompt 增强: 自动理解模糊需求
  3. 智能剪枝: 减少无效爬取
  4. 6 个专业化 Agent: 各司其职、协同工作
  5. Socket + REST 双通道: 灵活的通信方式

附录:快速启动

# 1. 安装依赖
pip install -r requirements.txt

# 2. 配置环境变量
export GLM_API_KEY="your_api_key"

# 3. 启动服务
python server.py

# 4. 运行客户端
python client.py crawl <https://example.edu.cn>

文档版本: 1.0.0

最后更新: 2025-11-26

About

计算机网络课程设计

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages