Skip to content

fix(server): Preserve supplied answers across exercise layouts - #458

Merged
shenyankm merged 4 commits into
mainfrom
fix/source-evidence-exercise-layouts
Oct 29, 2026
Merged

shenyankm merged 4 commits into
mainfrom
fix/source-evidence-exercise-layouts

Conversation

@shenyankm

@shenyankm shenyankm commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Final import merge could discard supplied translation or composite answers and retain unverified answers when option text, passage context or item identities changed. Preserve supplied fields through verified prompts/material, complete parent passages, unique blank numbers and scoped local or printed answer keys while retaining questions and review flags.

Numbered composite children keep their own source proofs. Parent scopes extend only over complete child clauses whose printed record and blank numbers match. Complete option contents and leading/trailing passage context must match; neighboring stem prefixes cannot supply answers. Printed item labels govern both JSON and plain-text ordering/matching answers, including implicit or positional IDs. All 21 existing text evaluation sources retain their supplied answers.

Patch the existing langgraph-sdk dependency to 0.4.6 for CVE-2026-118667, which blocked dependency audit on both CI Python versions.

AI assistance: Codex reproduced the six review findings, repaired the shared source proofs, checked direct callers and added regression cases. Validation uses provider/model substitutes and disposable PostgreSQL/RabbitMQ, without paid model calls.

Linked issue

Follow-up to merged #454, including the translation/composite review findings. N/A for a separate issue: this is a focused bug fix permitted by CONTRIBUTING.md.

Affected areas

AI-service answer provenance, its regression tests and evaluation documentation. Changes cover the source-proof helper, regression tests, evaluation documentation and service lockfile. Existing shared result contracts and null/review fields are reused.

Validation

Choose relevant checks from CONTRIBUTING.md.

Check or manual flow Platform / environment Result and evidence
PYTHONPATH=$PWD/server/src make verify AI_PYTHON=/Users/sheny/.local/share/uv/python/cpython-3.14-macos-aarch64-none/bin/python3.14 macOS arm64, CPython 3.14.7, SDK 0.4.6, disposable PostgreSQL/RabbitMQ, provider/model substitutes Passed on ec2c2476: 3,398 tests, 94% total coverage; lock consistency, Ruff, Pyright, 30 evaluation fixtures, recovery probes, sdist/wheel build and package integrity.
make install-locked; make audit AI_PYTHON=/Users/sheny/.local/share/uv/python/cpython-3.14-macos-aarch64-none/bin/python3.14 macOS ARM64, locked SDK 0.4.6, CI-pinned uv 0.12.13 Passed: runtime/development and build-requirement scans both report no known vulnerabilities. Only the SDK package record changed; generated contracts and 51 shared fixtures also passed.
Source-format and rejection regressions Same environment Passed: 296 provenance cases, including 35 review-follow-up cases; 473 broader source/fidelity regressions. All 21 text evaluation sources retain supplied answers. Existing numbered-bank continuations pass actual text and OCR graph paths; omitted context, truncated options, foreign fields and mismatched item labels remain reviewable.
Review-fix diff and direct callers; git diff --check Local Git/source review Passed. Reviewed source scope, blank/item identity, field independence and the final merge call sites. Retained the first failing reports and corrected the numbered-bank regression before the final complete verification.
Hosted CI and fresh automated review GitHub current head Pending for ec2c2476. Previous Python 3.14/3.15 CI failed on the SDK 0.4.4 vulnerability audit; current local audit and complete verification pass with 0.4.6.
Live Tencent/model evaluation External providers Not run: provider/model substitutes only, without paid calls or real-document accuracy claims.
App/Web browser/native acceptance Unchanged UI/native implementation Not run: no frontend/native behavior changes. Shared contracts and 51 fixtures passed under the patched Python environment; current-head App CI remains separate.

Risk and rollout

  • Data, contracts, and backup compatibility: No schema/API/backup change or rewrite of saved banks or immutable practice snapshots. Nulls, source associations and review flags remain intact.
  • Configuration, dependencies, and security: No new dependencies, settings, credentials, model calls or automatic retries. The existing SDK is patched from 0.4.4 to 0.4.6; regenerate installs/images from the updated lock. Proofs use existing checksum-verified source artifacts.
  • Main risk and rollout or recovery steps: Unknown or ambiguous layouts may still require manual recovery from the retained original source. Deploy the updated service without migration; rollback uses the previous service build. These checks establish source attribution, not printed-answer correctness or OCR accuracy.

Checklist

  • I reviewed the full diff and kept it focused, including any required generated files and documentation.
  • I recorded relevant validation results and explained remaining gaps.
  • I checked code, fixtures, logs, and attachments for secrets and personal data.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-29T03:25:53.135128Z
We couldn’t complete this review. Comment @codex review to try again.
ec2c247 New commits
🔒 Security Review ✅ Completed 2026-10-29T03:28:35.569031Z ec2c247 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8c7d879a02

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread server/src/practiq_ai/graphs/answer_evidence.py Outdated
Comment thread server/src/practiq_ai/graphs/answer_evidence.py
Comment thread server/src/practiq_ai/graphs/answer_evidence.py
Comment thread server/src/practiq_ai/graphs/answer_evidence.py Outdated
Retain explicitly scoped composite-child answers, verify complete option
contents and item identities, and reject fields borrowed from later questions.

Co-authored-by: Codex <codex@openai.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f0a85e59f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread server/src/practiq_ai/graphs/answer_evidence.py Outdated
Comment thread server/src/practiq_ai/graphs/answer_evidence.py
shenyankm and others added 2 commits October 29, 2026 11:03
Verify leading and trailing passage boundaries and extend parent scopes
only over complete child clauses with matching numbered blanks. Preserve
existing numbered-bank continuations in text and OCR imports.

Co-authored-by: Codex <codex@openai.com>
Resolve CVE-2026-118667 reported by the locked-dependency audit.
Regenerate the lock with the CI-pinned uv without changing other packages.

Co-authored-by: Codex <codex@openai.com>
@shenyankm
shenyankm merged commit aefbf71 into main Oct 29, 2026
13 checks passed
@shenyankm
shenyankm deleted the fix/source-evidence-exercise-layouts branch October 29, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant