From dd390646dfa66c35eb691dfda7f1a439de608418 Mon Sep 17 00:00:00 2001 From: unknown Date: Sat, 31 Oct 2026 08:50:10 +0900 Subject: [PATCH 1/4] docs: add quality improvements and VyOS integration roadmap --- docs/QUALITY-ROADMAP.zh.md | 82 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 docs/QUALITY-ROADMAP.zh.md diff --git a/docs/QUALITY-ROADMAP.zh.md b/docs/QUALITY-ROADMAP.zh.md new file mode 100644 index 0000000..065712c --- /dev/null +++ b/docs/QUALITY-ROADMAP.zh.md @@ -0,0 +1,82 @@ +# 质量提升与 VyOS 联动建议 + +本文是待实施的审查建议，不代表问题已经修复或功能已经交付。 +审查基线：`a0c7c54c0cdfed21d96ee6a146fb4bf632889c05`。 + +## 结论与验证范围 + +项目已有模块化结构、权限测试、协议回归测试和跨平台 CI 基础。要达到世界级目标，建议优先保证数据可信、故障可恢复、账号安全，再扩大功能和规模。 + +在 Linux Docker / Go 1.27.3 下，`go test ./...`、`go vet ./...`，以及 collector、decode、pipeline、store、api 的竞态测试通过。该工具链不同于项目声明的 go1.26.9；结果不替代声明工具链验证、真实 VyOS 联调或生产负载测试。 + +另外两个隔离的审查测试复现了 sFlow 边界问题：expanded source index 的 `1` 与 `0x01000001` 被压缩成同一个 SourceID；零长度 counter sample 没有返回错误。其余问题为静态审查发现，未做故障注入或应用级复现。 + +## 十项改进建议 + +### 1. 写入失败可恢复（优先） + +`internal/pipeline/pipeline.go:207–218,249–259` 在数据库写入前移除待写批次，失败后没有恢复路径。 +建议保留未提交批次，采用有界重试与明确的事务或幂等边界，避免明细和汇总分叉。 +**验收：**模拟写入中途失败，恢复后不丢失、不重复，相关汇总一致。 + +### 2. 时间序列和采样率保持准确（优先） + +`internal/decode/nf/totals.go:32–34,58–82` 的累计转增量状态不记录上次结束时间；结合 `fields.go:495–509` 和 pipeline 的时间分摊，新流量可能被摊回旧时间段。 +此外，`fields.go:467–475`、`decoder.go:332–335` 的地址级采样率借用需要检查跨 observation domain 冲突。 +**验收：**长连接连续上报的分钟桶准确；不同域采样率冲突时不猜测倍率；覆盖乱序和重启。 + +### 3. 补齐 sFlow 协议边界（优先） + +`internal/decode/sflow/sflow.go:140–145` 截断 expanded source index；`305–347` 未完整拒绝截断的 counter sample。两项均已用隔离测试复现。 +建议完整保存数据源类型与索引，统一长度校验，并增加 fuzz 和真实设备报文回归。 +**验收：**不同 expanded 数据源不碰撞；截断数据明确报错；合法流量不回归。 + +### 4. 统一密码更新与会话撤销（优先） + +`internal/api/accounts.go:141–173` 的管理员重置密码路径没有像用户自行改密一样撤销已有会话。 +`internal/auth/auth.go:103–154` 的账号文件读改写还需要处理并发更新与固定临时文件竞争。 +**验收：**管理员重置后目标账号旧会话失效；并发账号变更不丢失；CLI 与服务的操作边界明确。 + +### 5. 将删分支操作移出发布流程（优先） + +`.github/workflows/ci.yml:259–274` 在手动运行时独立清理分支，未检查是否已合并；开放 PR 列表未分页。 +建议独立维护流程，默认只预览，完整分页，并按合并状态、保留时间和保护规则筛选。 +**验收：**发布不附带删分支；未合并或仍有开放 PR 的分支不会被清理。 + +### 6. 给采集状态设置资源预算 + +`internal/collector/collector.go:247–279` 限制 agent 数，但持续活跃 agent 的 subagent/source 状态仍需独立限制。 +建议设置每设备与全局容量、独立过期策略，以及超限指标。 +**验收：**不断变化的数据源不会导致内存持续增长；异常设备不能拖垮其他设备的采集。 + +### 7. 批量写入接口计数，补齐丢弃指标 + +`internal/pipeline/pipeline.go:115–122,169–187` 的 counter 队列满时静默丢弃，写入和维护任务共用消费循环；`internal/store/store.go:318–324` 每条 counter 都 Flush。 +建议批量提交，控制维护耗时，并暴露队列深度、丢弃数和写入延迟。 +**验收：**高接口数和归档期间有可复现的吞吐、延迟、内存及丢弃报告。 + +### 8. 防止旧请求覆盖新页面 + +`internal/web/static/app.js:2497–2514,2616–2622` 的成功响应需要在渲染前检查请求代次，不能仅保护错误处理。 +建议引入取消请求或代次校验，并补浏览器测试。 +**验收：**快速切换时间范围、接口和页面，旧响应不能覆盖当前选择。 + +### 9. 把质量要求变成持续门禁 + +保留现有测试优势，增加针对性 race、协议 fuzz、浏览器回归、故障恢复测试和固定负载基准。 +修正 `internal/web/i18n_test.go:41–46` 仅比较占位符数量的检查，改为比较完整名称。 +**验收：**核心数据口径、恢复能力和性能回退有可重复验证；不以测试数量代替覆盖质量。 + +### 10. 与 vyos-sflow 建立开放的互操作契约 + +建议 traffic66 专注接收、分析、存储和展示；vyos-sflow 专注 VyOS 原生配置适配、诊断和测试。 +以标准 sFlow 为主，明确 agent/subagent/source 身份、采样率、接口计数、重启和双向采样语义。设备元数据可独立扩展，不绑定私有数据协议。 +**验收：**真实 VyOS 导出的流量和接口计数可解释、可对账；不重复计算入出方向；其他标准 collector 仍可接入。 + +## 建议实施顺序 + +1. 先处理上述数据、安全和发布风险，逐项提交小 PR，并加入对应回归测试。 +2. 再补资源预算、性能基准和浏览器质量门禁。 +3. 最后以一套真实 VyOS 环境完成端到端联调，再逐步扩大版本和数据平面支持范围。 + +不建议为了“世界级”过早引入分布式复杂度；先证明单机可靠性、容量边界和可恢复性。 From 3a817ffcf87a700bdacd1e09b4106fbccebac103 Mon Sep 17 00:00:00 2001 From: unknown Date: Sat, 31 Oct 2026 09:11:45 +0900 Subject: [PATCH 2/4] docs: align VyOS roadmap with shared Linux probe core --- docs/QUALITY-ROADMAP.zh.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/QUALITY-ROADMAP.zh.md b/docs/QUALITY-ROADMAP.zh.md index 065712c..1d6bb46 100644 --- a/docs/QUALITY-ROADMAP.zh.md +++ b/docs/QUALITY-ROADMAP.zh.md @@ -67,11 +67,15 @@ 修正 `internal/web/i18n_test.go:41–46` 仅比较占位符数量的检查，改为比较完整名称。 **验收：**核心数据口径、恢复能力和性能回退有可重复验证；不以测试数量代替覆盖质量。 -### 10. 与 vyos-sflow 建立开放的互操作契约 +### 10. 提取 Linux 探针子集，两个产品共享一个核心 -建议 traffic66 专注接收、分析、存储和展示；vyos-sflow 专注 VyOS 原生配置适配、诊断和测试。 -以标准 sFlow 为主，明确 agent/subagent/source 身份、采样率、接口计数、重启和双向采样语义。设备元数据可独立扩展，不绑定私有数据协议。 -**验收：**真实 VyOS 导出的流量和接口计数可解释、可对账；不重复计算入出方向；其他标准 collector 仍可接入。 +**架构决定：vyos-sflow 是 traffic66 的 Linux 轻量探针子集，不是原生 hsflowd 包装。** 路由器端保留包采样、真实接口计数、sFlow 导出和健康状态；DuckDB、历史存储、Web/TUI、地理库与威胁分析留在 traffic66。 + +当前 `internal/capture/capture.go` 输出的是聚合 flow.Record，已丢掉原始包头；`internal/encode/encode.go` 主要服务模拟器，不能直接认定为生产导出器。需要先分离原始包源与聚合器，再提取生产采样/编码核心。不能从聚合记录重造包样本。 + +共享核心拟放在 traffic66 仓库的独立 `probe` Go 子模块，发布版本后供两个产品依赖，解决跨仓库不能直接导入 internal 包的问题，避免长期复制分叉。模块尚未实现。VyOS 配置和服务生命周期单独适配；原生 hsflowd 仅作互操作、冲突检查和性能对照。 + +**验收：**探针可无 CGO 构建且不链接数据库/分析模块；真实 Linux 采样→sFlow→traffic66 可对账；独立解码器验证报文；首版仅支持内核网络路径，VPP 后置。依赖编译成功不等于真实采样或 VyOS 适配已经完成。 ## 建议实施顺序 From 80a343e00e1fa32293a0572bb6b58e39edb7bea5 Mon Sep 17 00:00:00 2001 From: unknown Date: Sat, 31 Oct 2026 09:22:02 +0900 Subject: [PATCH 3/4] docs: add reproducible review probes and verified evidence --- docs/QUALITY-ROADMAP.zh.md | 8 +- docs/review/README.zh.md | 51 +++++++++++ docs/review/evidence/verified-probes.txt | 91 +++++++++++++++++++ .../internal/auth/review_accounts_test.go.txt | 45 +++++++++ .../decode/nf/review_lending_test.go.txt | 40 ++++++++ .../nf/review_totals_interval_test.go.txt | 29 ++++++ .../decode/sflow/review_protocol_test.go.txt | 48 ++++++++++ docs/review/reproduce.py | 70 ++++++++++++++ 8 files changed, 378 insertions(+), 4 deletions(-) create mode 100644 docs/review/README.zh.md create mode 100644 docs/review/evidence/verified-probes.txt create mode 100644 docs/review/probes/internal/auth/review_accounts_test.go.txt create mode 100644 docs/review/probes/internal/decode/nf/review_lending_test.go.txt create mode 100644 docs/review/probes/internal/decode/nf/review_totals_interval_test.go.txt create mode 100644 docs/review/probes/internal/decode/sflow/review_protocol_test.go.txt create mode 100644 docs/review/reproduce.py diff --git a/docs/QUALITY-ROADMAP.zh.md b/docs/QUALITY-ROADMAP.zh.md index 1d6bb46..381fd7c 100644 --- a/docs/QUALITY-ROADMAP.zh.md +++ b/docs/QUALITY-ROADMAP.zh.md @@ -9,7 +9,7 @@ 在 Linux Docker / Go 1.27.3 下，`go test ./...`、`go vet ./...`，以及 collector、decode、pipeline、store、api 的竞态测试通过。该工具链不同于项目声明的 go1.26.9；结果不替代声明工具链验证、真实 VyOS 联调或生产负载测试。 -另外两个隔离的审查测试复现了 sFlow 边界问题：expanded source index 的 `1` 与 `0x01000001` 被压缩成同一个 SourceID；零长度 counter sample 没有返回错误。其余问题为静态审查发现，未做故障注入或应用级复现。 +首轮两个隔离探针复现了 sFlow 边界问题。补充审查在 Windows / Go 1.27.3 的精确基线归档上独立复跑：nf、sflow、auth 原测试通过，新增六个修复目标断言失败，确认五类条件触发问题。累计流总量守恒，但增量时间区间未推进；跨域采样冲突时仍错误借值；另在 Windows 复现了并发账号更新冲突。测试源、完整输出与复跑脚本见 [可复现审查证据](review/README.zh.md)。未执行的存储、API、浏览器、VyOS 与性能测试仍明确列为未验证，不与上述结果混用。 ## 十项改进建议 @@ -19,10 +19,10 @@ 建议保留未提交批次，采用有界重试与明确的事务或幂等边界，避免明细和汇总分叉。 **验收：**模拟写入中途失败，恢复后不丢失、不重复，相关汇总一致。 -### 2. 时间序列和采样率保持准确（优先） +### 2. 时间序列和采样率保持准确（P2，条件触发） -`internal/decode/nf/totals.go:32–34,58–82` 的累计转增量状态不记录上次结束时间；结合 `fields.go:495–509` 和 pipeline 的时间分摊，新流量可能被摊回旧时间段。 -此外，`fields.go:467–475`、`decoder.go:332–335` 的地址级采样率借用需要检查跨 observation domain 冲突。 +`internal/decode/nf/totals.go:32–34,58–82` 的累计转增量状态不记录上次结束时间；结合 `fields.go:495–509` 和 pipeline 的时间分摊，新流量可能被摊回旧时间段。decoder 实测三条增量各为6000 bytes，但跨度为1、2、3分钟；**总量正确，问题是时间归属**，尚未完成实际落库分钟桶验证。 +此外，`fields.go:467–475`、`decoder.go:332–335` 的地址级采样率借用已复现跨域冲突：声明100/1000的两个域可使未声明倍率的第三域借用最后更新值；不能据此断言第三域真实倍率或固定误差。 **验收：**长连接连续上报的分钟桶准确；不同域采样率冲突时不猜测倍率；覆盖乱序和重启。 ### 3. 补齐 sFlow 协议边界（优先） diff --git a/docs/review/README.zh.md b/docs/review/README.zh.md new file mode 100644 index 0000000..674728c --- /dev/null +++ b/docs/review/README.zh.md @@ -0,0 +1,51 @@ +# 可复现审查证据 + +代码基线：`a0c7c54c0cdfed21d96ee6a146fb4bf632889c05`。本目录提供审查探针，不是问题修复。规划见 [质量建议](../QUALITY-ROADMAP.zh.md)。 + +## 本轮独立复跑结果 + +环境：Windows amd64、Go 1.27.3、CGO_ENABLED=0。使用现有工具链，不启动 Docker；从精确 Git 提交归档到隔离目录，生产工作树不变。 + +1. nf、sflow、auth 原有测试全部通过。 +2. nf 行为刻画模式通过，证明当前代码确实表现出记录的行为；这不代表行为正确。 +3. 修复目标模式下，新增 **6 个不变量测试失败**，对应下方五类问题（采样冲突有两个测试）。 + +完整输出：[verified-probes.txt](evidence/verified-probes.txt)。仅将本地绝对工作目录替换成占位符，结果未改写。工具链不是项目指定的 go1.26.9；下载该工具链成功不算已用它完成测试。 + +| 发现 | 实际结果 | 证据范围 / 不应扩大解释 | +|---|---|---| +| IPFIX 累计流增量仍带原始 Start | 三次输出各 6000 bytes / 60 packets；区间却分别为 1、2、3 分钟 | 合计 18000 bytes 正确；不是本案例的总量重复累计。DuckDB 分钟桶尚未实测 | +| 跨 observation domain 采样借用 | 同地址两个域声明 100/1000 时，第三域借 1000；交换声明值则借 100 | 第三域真实倍率未知，不能断言必然高估十倍。相同倍率和显式自身倍率对照通过 | +| expanded sFlow source identity 截断 | 输入索引 1 与 16777217 均得到 SourceID=1 | 合法边界身份碰撞；不代表常见低索引设备均受影响 | +| 截断 counter sample 被接受 | 缺少全部必填字段的零长度 sample 未报错 | 完整零 record sample 的正向对照通过；没有证明崩溃或代码执行 | +| 并发账号更新冲突 | 16 个并发新增加原账号，应有17个；本次最终仅2个，9次调用报错 | Windows 文件读改写/临时文件竞争实测；调度敏感，不能要求每次丢失数量相同，也不是 Linux 实测 | + +累计流时间归属和跨域采样建议按 **P2、中等、条件触发** 处理，不能把它们描述为所有流量统计都错误。 + +## 如何复跑 + +要求 Python 3.12+、Git、已有 Go 工具链。先取得包含上述基线提交的 traffic66 checkout，不需要修改生产代码。提供一个全新或空的工作目录： + +```bash +python docs/review/reproduce.py \ + --go /absolute/path/to/go \ + --work /absolute/path/to/empty-review-workspace \ + --output /absolute/path/to/review-output.txt +``` + +Windows 可传入 `C:/.../go.exe`。可用 `--cache` 指定现有 Go build cache；脚本不会安装工具或重启 Docker。运行时固定 GOTOOLCHAIN=local，因此日志中的真实版本才是验证版本。 + +脚本依次归档精确基线、跑原包测试、复制 `.go.txt` 探针到隔离源目录、运行行为刻画和修复目标断言。最后一步在当前基线上预期退出1；**这个失败表示复现到缺陷，不是质量门禁通过**。基线检查失败返回2，其他环境或准备异常也可能产生非零退出；必须查看完整日志，不能将任意失败都当作成功复现。 + +NF 探针复用基线原测试中的 IPFIX 模板和 helper，因此应在完整测试包中运行，不是单独一个 go 文件。行为模式通过记录当前表现；设置 REVIEW_EXPECT_FIXED=1 后检查期望修复结果。 + +## 仍未完成的验证 + +- 管理员重置密码后的会话撤销：目前是静态路径发现，未完成 HTTP 级探针执行。 +- 写入失败丢批次、部分汇总提交：静态路径发现；故障探针未完成编译/执行，不能声称已做磁盘故障注入。 +- collector 子状态无独立预算：静态设计风险；未获得堆增长或 OOM 实测。 +- 前端旧响应覆盖：静态竞态路径，未完成浏览器响应乱序测试。 +- 原 CI 分支清理风险：静态工作流检查，没有执行删除操作。 +- Linux 无 CGO 子集构建、真实包采样、VyOS 配置/重启/升级、生产性能：未完成，不计为通过。 + +首轮 Linux / Go 1.27.3 全量 test、vet 及目标包 race 通过；本轮 Docker 引擎故障使追加 Linux 验证中断。存储探针另遇执行审批阻断，没有重试或绕过。以上限制不应被这次纯 Go 包测试覆盖掉。 diff --git a/docs/review/evidence/verified-probes.txt b/docs/review/evidence/verified-probes.txt new file mode 100644 index 0000000..912d09a --- /dev/null +++ b/docs/review/evidence/verified-probes.txt @@ -0,0 +1,91 @@ +Absolute workspace paths redacted; test results otherwise unchanged. +Baseline: a0c7c54c0cdfed21d96ee6a146fb4bf632889c05 +Platform: win32 + +=== toolchain === +command: ['[REVIEW_WORKSPACE]/deep-data/toolchain/go/bin/go.exe', 'version'] +REVIEW_EXPECT_FIXED= +go version go1.27.3 windows/amd64 + +exit=0 + +=== unmodified package baseline === +command: ['[REVIEW_WORKSPACE]/deep-data/toolchain/go/bin/go.exe', 'test', '-count=1', './internal/decode/nf', './internal/decode/sflow', './internal/auth'] +REVIEW_EXPECT_FIXED= +ok github.com/githubflyideas/traffic66/internal/decode/nf 5.381s +ok github.com/githubflyideas/traffic66/internal/decode/sflow 1.141s +ok github.com/githubflyideas/traffic66/internal/auth 2.902s + +exit=0 + +=== NF observed characterization === +command: ['[REVIEW_WORKSPACE]/deep-data/toolchain/go/bin/go.exe', 'test', '-count=1', '-v', '-run', '^TestReview', './internal/decode/nf'] +REVIEW_EXPECT_FIXED= +=== RUN TestReviewCrossDomainLending + review_lending_test.go:22: domain9=100 domain10=1000 borrowed_by_domain11=1000 cached=1000 raw_bytes=1500 scaled_bytes=1.5e+06 + review_lending_test.go:22: domain9=1000 domain10=100 borrowed_by_domain11=100 cached=100 raw_bytes=1500 scaled_bytes=150000 + review_lending_test.go:22: domain9=100 domain10=100 borrowed_by_domain11=100 cached=100 raw_bytes=1500 scaled_bytes=150000 +--- PASS: TestReviewCrossDomainLending (0.00s) +=== RUN TestReviewAmbiguousSessionCanBeOverwritten + review_lending_test.go:37: domain9 rates=[100 1000], domain10 rate=100, borrower=100 cache=100 +--- PASS: TestReviewAmbiguousSessionCanBeOverwritten (0.00s) +=== RUN TestReviewCumulativeInterval + review_totals_interval_test.go:22: report=1 bytes=6000 packets=60 Start=2023-11-15T07:13:00+09:00 End=2023-11-15T07:14:00+09:00 duration=1m0s + review_totals_interval_test.go:22: report=2 bytes=6000 packets=60 Start=2023-11-15T07:13:00+09:00 End=2023-11-15T07:15:00+09:00 duration=2m0s + review_totals_interval_test.go:22: report=3 bytes=6000 packets=60 Start=2023-11-15T07:13:00+09:00 End=2023-11-15T07:16:00+09:00 duration=3m0s +--- PASS: TestReviewCumulativeInterval (0.05s) +PASS +ok github.com/githubflyideas/traffic66/internal/decode/nf 1.142s + +exit=0 + +=== desired invariants (failure expected on reviewed baseline) === +command: ['[REVIEW_WORKSPACE]/deep-data/toolchain/go/bin/go.exe', 'test', '-count=1', '-v', '-run', '^TestReview', './internal/decode/nf', './internal/decode/sflow', './internal/auth'] +REVIEW_EXPECT_FIXED=1 +=== RUN TestReviewCrossDomainLending + review_lending_test.go:22: domain9=100 domain10=1000 borrowed_by_domain11=1000 cached=1000 raw_bytes=1500 scaled_bytes=1.5e+06 + review_lending_test.go:24: borrowed 1000 want 0 + review_lending_test.go:22: domain9=1000 domain10=100 borrowed_by_domain11=100 cached=100 raw_bytes=1500 scaled_bytes=150000 + review_lending_test.go:24: borrowed 100 want 0 + review_lending_test.go:22: domain9=100 domain10=100 borrowed_by_domain11=100 cached=100 raw_bytes=1500 scaled_bytes=150000 +--- FAIL: TestReviewCrossDomainLending (0.00s) +=== RUN TestReviewAmbiguousSessionCanBeOverwritten + review_lending_test.go:37: domain9 rates=[100 1000], domain10 rate=100, borrower=100 cache=100 + review_lending_test.go:39: borrowed 100 want 0 +--- FAIL: TestReviewAmbiguousSessionCanBeOverwritten (0.00s) +=== RUN TestReviewCumulativeInterval + review_totals_interval_test.go:22: report=1 bytes=6000 packets=60 Start=2023-11-15T07:13:00+09:00 End=2023-11-15T07:14:00+09:00 duration=1m0s + review_totals_interval_test.go:22: report=2 bytes=6000 packets=60 Start=2023-11-15T07:13:00+09:00 End=2023-11-15T07:15:00+09:00 duration=2m0s + review_totals_interval_test.go:26: delta Start=2023-11-15 07:13:00 +0900 JST want previous End=2023-11-15 07:14:00 +0900 JST + review_totals_interval_test.go:22: report=3 bytes=6000 packets=60 Start=2023-11-15T07:13:00+09:00 End=2023-11-15T07:16:00+09:00 duration=3m0s + review_totals_interval_test.go:26: delta Start=2023-11-15 07:13:00 +0900 JST want previous End=2023-11-15 07:15:00 +0900 JST +--- FAIL: TestReviewCumulativeInterval (0.05s) +FAIL +FAIL github.com/githubflyideas/traffic66/internal/decode/nf 0.499s +=== RUN TestReviewExpandedSourceCollision + review_protocol_test.go:34: expanded input indices: 1 and 16777217; decoded IDs: 1 and 1 + review_protocol_test.go:36: distinct expanded data sources collapse to the same identity +--- FAIL: TestReviewExpandedSourceCollision (0.00s) +=== RUN TestReviewTruncatedCounterRejected + review_protocol_test.go:46: zero-length counter sample accepted without error (mandatory fields absent) +--- FAIL: TestReviewTruncatedCounterRejected (0.00s) +FAIL +FAIL github.com/githubflyideas/traffic66/internal/decode/sflow 1.088s +=== RUN TestReviewConcurrentAccountUpdates + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: rename [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password: Access is denied. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:34: Set returned error: open [REVIEW_WORKSPACE]\parent-native-review\tmp\TestReviewConcurrentAccountUpdates888739420\001\password.tmp: The process cannot access the file because it is being used by another process. + review_accounts_test.go:41: requested=17 completed-with-error=9 stored=2 + review_accounts_test.go:43: concurrent account changes are not preserved: errors=9 stored=2 want=17 +--- FAIL: TestReviewConcurrentAccountUpdates (0.38s) +FAIL +FAIL github.com/githubflyideas/traffic66/internal/auth 0.993s +FAIL + +exit=1 diff --git a/docs/review/probes/internal/auth/review_accounts_test.go.txt b/docs/review/probes/internal/auth/review_accounts_test.go.txt new file mode 100644 index 0000000..99ded7e --- /dev/null +++ b/docs/review/probes/internal/auth/review_accounts_test.go.txt @@ -0,0 +1,45 @@ +package auth + +import ( + "fmt" + "sync" + "testing" +) + +// This is a concurrent filesystem stress probe, not a deterministic scheduler. +func TestReviewConcurrentAccountUpdates(t *testing.T) { + dir := t.TempDir() + if err := Set(dir, "admin", "admin-review-password"); err != nil { + t.Fatal(err) + } + const workers = 16 + start := make(chan struct{}) + errs := make(chan error, workers) + var wg sync.WaitGroup + for i := 0; i < workers; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + <-start + errs <- Set(dir, fmt.Sprintf("review-user-%02d", i), "review-password") + }(i) + } + close(start) + wg.Wait() + close(errs) + failed := 0 + for err := range errs { + if err != nil { + failed++ + t.Logf("Set returned error: %v", err) + } + } + entries, err := Load(dir) + if err != nil { + t.Fatalf("Load after concurrent updates: %v", err) + } + t.Logf("requested=%d completed-with-error=%d stored=%d", workers+1, failed, len(entries)) + if failed != 0 || len(entries) != workers+1 { + t.Fatalf("concurrent account changes are not preserved: errors=%d stored=%d want=%d", failed, len(entries), workers+1) + } +} diff --git a/docs/review/probes/internal/decode/nf/review_lending_test.go.txt b/docs/review/probes/internal/decode/nf/review_lending_test.go.txt new file mode 100644 index 0000000..a4f53a2 --- /dev/null +++ b/docs/review/probes/internal/decode/nf/review_lending_test.go.txt @@ -0,0 +1,40 @@ +package nf + +import ( + "os" + "testing" + "github.com/githubflyideas/traffic66/internal/encode" +) + +func reviewDeclare(t *testing.T,d *Decoder,domain,id,rate,seq uint32) { + t.Helper(); opt,vals:=sampler(id,rate) + m:=encode.NewIPFIX();m.AddTemplate(opt);m.AddData(opt.ID,[]encode.Values{vals}) + if _,err:=d.Decode(m.IPFIX(uint32(base.Unix()),seq,domain),exp,base);err!=nil {t.Fatal(err)} +} + +func TestReviewCrossDomainLending(t *testing.T) { + for _,rates:=range [][2]uint32{{100,1000},{1000,100},{100,100}} { + d:=NewDecoder() + reviewDeclare(t,d,9,1,rates[0],0) + reviewDeclare(t,d,10,1,rates[1],0) + // The third domain supplies no rate; different domains declare conflicting rates. + got:=decodeOne(t,d,11,nil,nil,ipfixFlow(1,2)) + t.Logf("domain9=%d domain10=%d borrowed_by_domain11=%v cached=%d raw_bytes=1500 scaled_bytes=%v",rates[0],rates[1],got,d.byAddr[exp],1500*got) + want:=float64(rates[1]); if rates[0]!=rates[1] && os.Getenv("REVIEW_EXPECT_FIXED")=="1" {want=0} + if got!=want {t.Errorf("borrowed %v want %v",got,want)} + // A domain's explicit own rate still wins. This is not corruption of every domain. + own:=decodeOne(t,d,9,nil,nil,ipfixFlow(1,2)) + if own!=float64(rates[0]) {t.Errorf("own rate=%v want=%d",own,rates[0])} + } +} + +func TestReviewAmbiguousSessionCanBeOverwritten(t *testing.T) { + d:=NewDecoder() + reviewDeclare(t,d,9,1,100,0);reviewDeclare(t,d,9,2,1000,1) + if got:=d.byAddr[exp];got!=0 {t.Fatalf("same-domain conflict should remove lender, got %d",got)} + reviewDeclare(t,d,10,1,100,0) + got:=decodeOne(t,d,11,nil,nil,ipfixFlow(1,2)) + t.Logf("domain9 rates=[100 1000], domain10 rate=100, borrower=%v cache=%d",got,d.byAddr[exp]) + want:=float64(100);if os.Getenv("REVIEW_EXPECT_FIXED")=="1" {want=0} + if got!=want {t.Errorf("borrowed %v want %v",got,want)} +} diff --git a/docs/review/probes/internal/decode/nf/review_totals_interval_test.go.txt b/docs/review/probes/internal/decode/nf/review_totals_interval_test.go.txt new file mode 100644 index 0000000..b7f1e1d --- /dev/null +++ b/docs/review/probes/internal/decode/nf/review_totals_interval_test.go.txt @@ -0,0 +1,29 @@ +package nf + +import ( + "net/netip" + "os" + "testing" + "time" + "github.com/githubflyideas/traffic66/internal/encode" +) + +func TestReviewCumulativeInterval(t *testing.T) { + d:=NewDecoder();d.Unsampled=map[netip.Addr]bool{exp:true} + start:=base.Truncate(time.Minute) + var sum uint64 + for i:=0;i<3;i++ { + end:=start.Add(time.Duration(i+1)*time.Minute) + m:=encode.NewIPFIX();m.AddTemplate(totalTmpl) + m.AddData(totalTmpl.ID,[]encode.Values{totRec(uint64(start.UnixMilli()),uint64(end.UnixMilli()),uint64(6000*(i+1)),uint64(60*(i+1)))}) + res,err:=d.Decode(m.IPFIX(uint32(end.Unix()),uint32(i),1),exp,end) + if err!=nil || len(res.Records)!=1 {t.Fatalf("decode %d: %+v %v",i,res,err)} + r:=res.Records[0];sum+=r.Bytes + t.Logf("report=%d bytes=%d packets=%d Start=%s End=%s duration=%s",i+1,r.Bytes,r.Packets,r.Start.Format(time.RFC3339),r.End.Format(time.RFC3339),r.End.Sub(r.Start)) + if r.Bytes!=6000 || r.Packets!=60 {t.Errorf("delta counters wrong %+v",r)} + want:=start + if os.Getenv("REVIEW_EXPECT_FIXED")=="1" {want=start.Add(time.Duration(i)*time.Minute)} + if !r.Start.Equal(want) {t.Errorf("delta Start=%s want previous End=%s",r.Start,want)} + } + if sum!=18000 {t.Errorf("total=%d want 18000",sum)} +} diff --git a/docs/review/probes/internal/decode/sflow/review_protocol_test.go.txt b/docs/review/probes/internal/decode/sflow/review_protocol_test.go.txt new file mode 100644 index 0000000..592d223 --- /dev/null +++ b/docs/review/probes/internal/decode/sflow/review_protocol_test.go.txt @@ -0,0 +1,48 @@ +package sflow + +import ( + "encoding/binary" + "testing" + "time" +) + +func reviewWords(words ...uint32) []byte { + b := make([]byte, 4*len(words)) + for i, word := range words { + binary.BigEndian.PutUint32(b[i*4:], word) + } + return b +} + +func reviewDatagram(format uint32, body []byte) []byte { + header := reviewWords(5, 1, 0xc0000201, 0, 1, 1000, 1, format, uint32(len(body))) + return append(header, body...) +} + +func TestReviewExpandedSourceCollision(t *testing.T) { + get := func(index uint32) uint32 { + // expanded flow sample: sequence, type, full index, rate, pool, + // drops, input format/value, output format/value, record count. + body := reviewWords(1, 0, index, 100, 1000, 0, 0, 1, 0, 2, 0) + d, err := Decode(reviewDatagram(3, body), time.Unix(1000, 0)) + if err != nil || len(d.Flows) != 1 { + t.Fatalf("decode expanded index %d: result=%v error=%v", index, d, err) + } + return d.Flows[0].SourceID + } + a, b := get(1), get(0x01000001) + t.Logf("expanded input indices: 1 and 16777217; decoded IDs: %d and %d", a, b) + if a == b { + t.Fatal("distinct expanded data sources collapse to the same identity") + } +} + +func TestReviewTruncatedCounterRejected(t *testing.T) { + if _, err := Decode(reviewDatagram(2, reviewWords(1, 1, 0)), time.Unix(1000, 0)); err != nil { + t.Fatalf("control: a complete zero-record counter sample must decode: %v", err) + } + _, err := Decode(reviewDatagram(2, nil), time.Unix(1000, 0)) + if err == nil { + t.Fatal("zero-length counter sample accepted without error (mandatory fields absent)") + } +} diff --git a/docs/review/reproduce.py b/docs/review/reproduce.py new file mode 100644 index 0000000..e4d9743 --- /dev/null +++ b/docs/review/reproduce.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +"""Replay pure-Go review probes against an archived, exact source commit. + +Requires Python 3.12+, Git and an existing Go toolchain. Does not download or +install tools, touch the checkout's production files, or run storage probes. +The final test command is expected to fail on the reviewed baseline. Inspect +the saved log: compilation or environment failures are not bug reproductions, +and an expected invariant failure is not a passing quality gate. +""" +import argparse +import io +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tarfile + +BASELINE = "a0c7c54c0cdfed21d96ee6a146fb4bf632889c05" +PACKAGES = ["internal/decode/nf", "internal/decode/sflow", "internal/auth"] + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--go", default="go") + parser.add_argument("--work", required=True, type=Path, help="new, empty workspace; never use a source checkout") + parser.add_argument("--output", required=True, type=Path) + parser.add_argument("--cache", type=Path, help="optional existing Go build cache") + args = parser.parse_args() + root = Path(__file__).resolve().parents[2] + work = args.work.resolve() + if work.exists() and any(work.iterdir()): + parser.error("--work must be empty to avoid overwriting any files") + work.mkdir(parents=True, exist_ok=True) + paths = PACKAGES + ["internal/encode", "internal/flow", "internal/packet", "internal/wire", "go.mod", "go.sum"] + archive = subprocess.run(["git", "-C", str(root), "archive", BASELINE, "--", *paths], check=True, stdout=subprocess.PIPE).stdout + with tarfile.open(fileobj=io.BytesIO(archive)) as source: + source.extractall(work, filter="data") + temp = work / "tmp" + temp.mkdir() + env = dict(os.environ, GOTOOLCHAIN="local", CGO_ENABLED="0", TMP=str(temp), TEMP=str(temp), TMPDIR=str(temp)) + env.pop("REVIEW_EXPECT_FIXED", None) + if args.cache: + env["GOCACHE"] = str(args.cache.resolve()) + logs = ["Baseline: " + BASELINE, "Platform: " + sys.platform] + + def run(label, arguments): + command = [args.go, *arguments] + result = subprocess.run(command, cwd=work, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding="utf-8", errors="replace", timeout=180) + logs.extend(["\n=== " + label + " ===", "command: " + repr(command), "REVIEW_EXPECT_FIXED=" + env.get("REVIEW_EXPECT_FIXED", ""), result.stdout, "exit=" + str(result.returncode)]) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text("\n".join(logs) + "\n", encoding="utf-8") + print(label + ": exit=" + str(result.returncode), flush=True) + return result.returncode + + if run("toolchain", ["version"]): + return 2 + if run("unmodified package baseline", ["test", "-count=1", *["./" + p for p in PACKAGES]]): + return 2 + for package in PACKAGES: + for probe in sorted((root / "docs/review/probes" / package).glob("*_test.go.txt")): + shutil.copyfile(probe, work / package / probe.name.removesuffix(".txt")) + if run("NF observed characterization", ["test", "-count=1", "-v", "-run", "^TestReview", "./internal/decode/nf"]): + return 2 + env["REVIEW_EXPECT_FIXED"] = "1" + return run("desired invariants (failure expected on reviewed baseline)", ["test", "-count=1", "-v", "-run", "^TestReview", *["./" + p for p in PACKAGES]]) + + +if __name__ == "__main__": + sys.exit(main()) From e05f348b369983f9be82726c459bf781b3e6447f Mon Sep 17 00:00:00 2001 From: unknown Date: Sat, 31 Oct 2026 09:55:29 +0900 Subject: [PATCH 4/4] docs: restrict Linux probe subset to sFlow --- docs/QUALITY-ROADMAP.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/QUALITY-ROADMAP.zh.md b/docs/QUALITY-ROADMAP.zh.md index 381fd7c..a434f79 100644 --- a/docs/QUALITY-ROADMAP.zh.md +++ b/docs/QUALITY-ROADMAP.zh.md @@ -69,7 +69,7 @@ ### 10. 提取 Linux 探针子集，两个产品共享一个核心 -**架构决定：vyos-sflow 是 traffic66 的 Linux 轻量探针子集，不是原生 hsflowd 包装。** 路由器端保留包采样、真实接口计数、sFlow 导出和健康状态；DuckDB、历史存储、Web/TUI、地理库与威胁分析留在 traffic66。 +**架构决定：vyos-sflow 是 traffic66 的 Linux 轻量探针子集，只支持 sFlow，不实现或预留 NetFlow/IPFIX，也不是原生 hsflowd 包装。** 路由器端保留包采样、真实接口计数、sFlow 导出和健康状态；DuckDB、历史存储、Web/TUI、地理库与威胁分析留在 traffic66。traffic66 原有的多协议分析能力不随子集一起迁入。 当前 `internal/capture/capture.go` 输出的是聚合 flow.Record，已丢掉原始包头；`internal/encode/encode.go` 主要服务模拟器，不能直接认定为生产导出器。需要先分离原始包源与聚合器，再提取生产采样/编码核心。不能从聚合记录重造包样本。