Skip to content

test(api): replace change-detector assertions with behavior contracts - #45298

Open
GYJ99 wants to merge 4 commits into
langgenius:mainfrom
GYJ99:test/41418-change-detector-cleanup
Open

GYJ99 wants to merge 4 commits into
langgenius:mainfrom
GYJ99:test/41418-change-detector-cleanup

Conversation

@GYJ99

@GYJ99 GYJ99 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Check emitted telemetry events, counters, deduplication, payload rehydration, and degraded behavior through EnterpriseMetricHandler.handle().
  • Assert meaningful mail recipients/template data/log content and real Celery queue routing instead of incidental internal call counts or task attributes.
  • Validate generated OpenAPI schemas and operation contracts, with accepted/rejected request payloads. Batch schema-root validation and inspect descendants once, preserving required-field, reference, and discriminator checks.
  • Add four nested-schema negative cases. Give the aggregate five-document OpenAPI check a bounded 60-second timeout under coverage; ordinary tests retain the global CI timeout.
  • Sync upstream main at 8d812b938af6, including the stable xdist parameter IDs merged in fix(api): pass lint, type check and unit tests on the new dev dependencies #45308.

Refs #41418. This is a selected four-module slice of the broader cleanup.

Screenshots

N/A (test-only change).

Validation

Executed locally on macOS / Python 3.13.11 against the tree committed as 67710432275c8d3e1fb42664bacdd8b65998eae4:

  • uv run --project api --python 3.13 --frozen --dev pytest api/tests/unit_tests/tasks/test_generate_summary_index_task.py api/tests/unit_tests/tasks/test_mail_inner_task.py api/tests/unit_tests/enterprise/telemetry/test_metric_handler.py api/tests/unit_tests/dev/test_generate_swagger_specs.py --timeout 20 --durations=5 -q — 110 passed in 189.23s. The aggregate OpenAPI contract test took 13.01s in this run.
  • UV_PYTHON=3.13 UV_FROZEN=1 make lint — passed; no unrelated file changes; 82 import contracts kept.
  • UV_PYTHON=3.13 UV_FROZEN=1 make type-check — passed.
  • Ruff checks and all original staged commit hooks passed. The hooks used UV_PYTHON=3.13 UV_NO_SYNC=1 to reuse the environment already synchronized from the committed lockfile.
  • git diff --check upstream/main...HEAD — clean; the PR still changes only its original four test files.

Before the follow-up, the aggregate OpenAPI check reproducibly exceeded the CI 20-second limit locally and in upstream CI on head 6f49055e45 (that unit shard otherwise passed 12,726 tests). The follow-up removes repeated recursive validation and budgets the remaining real work explicitly. The console export alone is about 4.4 MB with 2,113 component schemas; the test checks all five exports. Nested malformed-schema, missing-required-property, dangling-reference, and discriminator failures remain covered.

Upstream checks on 6771043227 have completed: 35 successful, 14 skipped, no failed or pending checks (verified Nov 3, 2026), including API unit/controller/integration jobs, Python style, and Codecov. GitHub reports MERGEABLE; maintainer review is still required.

The full repository test suite and Docker-backed integration suites were not run locally for this test-only PR. No production code, generated contracts, dependency, or lockfile changes are included.

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues.
  • I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
  • I've updated the documentation accordingly. (N/A: test-only change.)
  • I ran make lint && make type-check (backend), with the environment recorded above. vp staged is N/A (no frontend changes).

From Codex

Assert observable outcomes instead of internal call sequences across telemetry dispatch, mail sending, queue routing, and OpenAPI export tests, per langgenius#41418.
@GYJ99
GYJ99 requested a review from QuantumGhost as a code owner November 2, 2026 03:55
@github-actions github-actions Bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 83.00% 83.00% -0.00%
Strict coverage 82.66% 82.65% -0.00%
Typed symbols 85,596 85,582 -14
Untyped symbols 17,603 17,605 +2
Modules 4373 4373 0

@codecov

codecov Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.77%. Comparing base (8d812b9) to head (6771043).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #45298      +/-   ##
==========================================
- Coverage   89.77%   89.77%   -0.01%     
==========================================
  Files        6239     6239              
  Lines      390924   390924              
  Branches    79291    79291              
==========================================
- Hits       350950   350946       -4     
- Misses      33875    33877       +2     
- Partials     6099     6101       +2     
Flag Coverage Δ
api 89.63% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

GYJ99 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

CI triage for the current head (8921fbd): the Python Style job (including the previously failing No New Cast Guard) now passes. The remaining Main CI failure is in API Unit Tests (3.13, shard 1/2) during pytest-xdist collection, before test execution:

Different tests were collected between gw1 and gw0.
-test_valid_credentials_do_not_acquire_a_lock[1793599135]
+test_valid_credentials_do_not_acquire_a_lock[1793599134]

This comes from the wall-clock-derived parametrization in api/tests/unit_tests/services/credentials/test_tool_runtime.py, which is outside this PR's diff. There is already a targeted upstream fix, #45243 (explicit stable parameter IDs), with green workflow runs. See the failed CI job.

No unrelated test change is needed in this PR. I could not rerun the upstream job with my fork's Actions permissions; a maintainer rerun (or a fresh run after #45243 merges) should verify this head once the collection flake is addressed. This assessment is from CI logs and source inspection; I have not claimed a local Dify test run.

Update (Nov 2, 2026): Upstream #45308 has since merged and its change to test_tool_runtime.py supplies stable parametrization IDs on current main, independently addressing the same xdist collection flake (#45243 remains open but now overlaps). The recorded failed run for this PR predates that upstream change; it has not been rerun, so this is not a claim of green CI. A maintainer rerun or fresh check against the updated base is still needed.

GYJ99 added 2 commits November 3, 2026 09:51
Merge upstream main at 8d812b9 to include the stable xdist parameter IDs from langgenius#45308. Preserve the existing four-file test-only scope for langgenius#41418.

Validated the merged tree with 106 focused tests, 10 xdist regression tests, full lint and type-check. AI-assisted with Codex.
Validate schema roots together instead of recursively rechecking each descendant. Preserve nested required-field, reference and discriminator checks, with four negative regression cases. Give the aggregate five-document check a 60-second budget under coverage.

The previous head fails under the CI 20-second limit. The updated four-module suite passes 110 tests with the same global timeout; full lint and type-check pass. AI-assisted with Codex.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant