Skip to content

Fix weighted evaluation score aggregation - #89

Open
zjn20030811 wants to merge 1 commit into
OpenMOSS:mainfrom
zjn20030811:fix/complete-evaluation-score
Open

Fix weighted evaluation score aggregation#89
zjn20030811 wants to merge 1 commit into
OpenMOSS:mainfrom
zjn20030811:fix/complete-evaluation-score

Conversation

@zjn20030811

Copy link
Copy Markdown

Summary

  • include the skill-with-entity and exact-match metrics in total_score
  • add regression tests for partial and perfect evaluation scores

Details

get_final_score computed four metrics and exposed all four in its return value,
but its weighted sum iterated only over skill_entity_scores. The two additional
metrics therefore had no effect and the reported maximum was capped at 80. This
change builds one score mapping before applying the existing weights, preserving
the public result fields and the configured 40/40/10/10 weighting.

Validation

  • python -m pytest -q tests/test_evaluation_utils.py (2 passed)
  • python -m compileall -q VLABench/evaluation/utils.py tests/test_evaluation_utils.py
  • git diff --check

Fixes #77

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug Report: ”total_score“ calculation ignores partial metrics in ”get_final_score“

1 participant