[ci] Auto-rerun flaky integration-test jobs on a fresh runner

[weiqingy]

Linked issue: #893

Purpose of change

The integration-test jobs (it-python, it-java, cross-language) run against a live local Ollama server, and the llama-server process intermittently segfaults mid-run. Once it does, every subsequent inference in that job fails, so the in-suite reruns already configured (pytest --reruns 2, surefire rerunFailingTestsCount=2) cannot recover it — they retry into the same wedged process.

The crash tracks the runner environment rather than anything in the repo: it is not reproducible in isolation, the installed Ollama version has been constant across both the green and the red days, and the failures roam between matrix jobs from run to run rather than pinning to one. The reliable way to clear it is a fresh runner — not a larger in-suite retry count, an in-process restart, or a version pin.

Because the run goes red if any single integration job fails, one segfaulted job is enough to redden an otherwise-green matrix. Recent example on main (run 29227640397): 10 of the 11 integration jobs passed, and the single failure — it-python [python-3.11] [flink-2.0] — was purely the segfault (llama-server process has terminated: signal: segmentation fault). Re-running the whole workflow re-rolls all 11 jobs; re-running just the failed one is enough.

This adds a workflow_run companion that, when the CI run finishes in failure, re-runs only the failed integration-test jobs on a fresh runner, which resets the whole job environment. Unit-test, lint, and build failures are never re-run, so a real regression there stays red immediately. The retry is bounded by run_attempt (at most two re-runs) so a genuinely broken job — which fails on every fresh runner — still goes red instead of looping.

An earlier version of this description also attributed a PyFlink py4j gateway failure to the environment. That one was an in-repo test bug and has since been fixed by #896, so it is out of scope here — this PR targets the Ollama segfault alone. How often that remains frequent enough to be worth automating is being tracked on #893.

Tests

The mechanism was validated end-to-end on a fork: a failed job matching the integration-test filter is re-run on a fresh runner and the run flips green, a non-matching job is left untouched, and the run_attempt cap stops the retry chain. The job-name filter was checked against the current CI job names, so only it-python / it-java / cross-language match; the unit-test jobs (ut-python / ut-java), Code Style Check, install.sh tests, and ut-build-backend are excluded.

Two operational notes for reviewers: the companion needs actions: write on the workflow token to re-run jobs (the existing labeler companions already run on write-scoped tokens), and because workflow_run companions execute from the default branch, this takes effect after merge and cannot be exercised on this PR's own CI.

API

No public API changes; this only affects CI.

Documentation

• doc-needed
• doc-not-needed
• doc-included

[weiqingy]

Note on this PR's own CI, to avoid a misread: the two red legs (it-java [flink-1.20] and it-java [java-21] [flink-2.0]) are the llama-server segfault flake this PR targets — llama-server process has terminated: signal: segmentation fault (core dumped). This change only adds a workflow file and touches no test or build path, so it cannot itself cause a test failure.

This PR cannot demonstrate the fix on its own CI: a workflow_run companion runs only from the copy of the file on the default branch, so it is not active for this PR's run. Once merged it activates for every subsequent "Flink Agents CI" run, including pull-request runs — a failed integration-test leg is re-run on a fresh runner and its check flips to green, while a genuine failure that reproduces on every fresh runner stays red. That is the intended signal: a flaky failure auto-heals, a real one persists.

I validated the mechanism end-to-end on a fork (companion on the fork's default branch): a failed integration-test-named job was re-run on a fresh runner and the run flipped green, a non-matching job was left untouched, and the run_attempt cap stopped the retry chain. This mirrors the existing pr_labeler / issue_labeler companions, which likewise run from the default branch via workflow_run.

[weiqingy]

Rebased onto current main (which now includes #896) and updated the description.

The scope of this PR has narrowed. It previously covered two failure modes; the py4j gateway failure turned out to be an in-repo test bug rather than an environmental one, and #896 has fixed it. What remains is the Ollama llama-server segfault alone, which is unaffected by that fix.

The case for the rerun is now easier to state precisely: on the first post-#896 main run (29227640397), 10 of the 11 integration jobs passed and the single failure was purely the segfault — enough on its own to turn the run red. Re-running the workflow re-rolls all 11 jobs; this companion re-runs only the one that failed.

The earlier per-job failure-rate figures counted both modes and have been dropped from the description rather than restated. Whether the residual segfault fires often enough to justify automating the rerun is worth judging from the next few main runs, and that is being tracked on #893.