Skip to content

feat(automata): add ground-truth evaluation framework for classification accuracy - #816

Merged
AlexMikhalev merged 1 commit into
mainfrom
task/576-automata-evaluation-framework
Apr 16, 2026
Merged

AlexMikhalev merged 1 commit into
mainfrom
task/576-automata-evaluation-framework

Conversation

@AlexMikhalev

Copy link
Copy Markdown
Contributor

Summary

  • Generic evaluation framework for measuring automata classification accuracy against human-labeled ground truth
  • Computes micro-averaged precision, recall, F1 overall and per-term
  • Detects systematic errors (false positives appearing across multiple documents)
  • load_ground_truth() for JSON-based ground truth files
  • All types are serde-serializable for JSON report output

New types

GroundTruthDocument, ExpectedMatch, ClassificationMetrics, TermReport, EvaluationResult, SystematicError

Core API

pub fn evaluate(ground_truth: &[GroundTruthDocument], thesaurus: Thesaurus) -> EvaluationResult
pub fn load_ground_truth(path: &Path) -> Result<Vec<GroundTruthDocument>, std::io::Error>

Test plan

  • 14 tests pass (all new, real thesaurus construction, no mocks)
  • cargo clippy -p terraphim_automata clean
  • cargo fmt clean
  • CI green

Refs #576

…ion accuracy

Refs #576

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@AlexMikhalev
AlexMikhalev merged commit 3fa9fb7 into main Apr 16, 2026
32 of 34 checks passed
@AlexMikhalev
AlexMikhalev deleted the task/576-automata-evaluation-framework branch April 16, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant