[tools] Update install.sh for Flink Agents 0.3.0 and Python 3.12 by Olawoyin007 · Pull Request #888 · apache/flink-agents

Olawoyin007 · 2026-07-09T08:17:12Z

Linked issue: #863

Purpose of change

install.sh did not offer the 0.3.0 release (shipped June 19th), and it rejected Python 3.12 even though 0.3.0 supports it.

Add 0.3.0 to FLINK_AGENTS_SUPPORTED_VERSIONS and make it the recommended default, matching https://flink.apache.org/downloads/#apache-flink-agents (0.3.0 ships JARs for Flink 1.20 / 2.0 / 2.1 / 2.2, same matrix as 0.2.x).
Make the Python upper bound follow the selected release instead of a hardcoded <3.12: 0.1.x/0.2.x keep >=3.10,<3.12, while 0.3.0+ accepts Python 3.12 (its wheels publish requires-python >=3.10,<3.13). The version-mismatch messages now print the bound that actually applies.

The version choice happens in stage 1 (plan_flink_agents) before Python resolution in stage 2 (plan_pyflink), so FLINK_AGENTS_VERSION is always settled when validate_python_bin runs.

One known edge left as-is: on Python 3.12 with Flink 1.20/2.0, apache-flink has no cp312 wheel on PyPI, so pip falls back to the sdist. That combination fails at pip time with a clear resolver message rather than being pre-blocked by the installer.

Tests

Reworked the 3.12/3.13 cases in tools/test/unit/validate_python_bin.bats to cover both bounds (3.12 rejected for 0.2.x, accepted for 0.3.0+, 3.13 still rejected) and updated the default-version assertion in dry_run_extra.bats.
bash tools/test/run.sh: all 149 tests pass.
Manually verified on Linux with Python 3.12.3: accepted with the 0.3.0 default, correctly rejected with --flink-agents-version 0.2.1.

API

No public API changes; installer script only.

Documentation

doc-needed
doc-not-needed (quickstart docs already state "Python 3.10, 3.11 or 3.12")
doc-included

🤖 Generated with Claude Code

Add 0.3.0 to the supported versions (new recommended default) and make the Python upper bound follow the selected release: 0.1.x/0.2.x keep >=3.10,<3.12, while 0.3.0+ accepts Python 3.12 per its requires-python metadata (>=3.10,<3.13).

Olawoyin007 · 2026-07-09T09:56:52Z

The it-java [flink-2.1] failure looks like CI infrastructure rather than this change: the Ollama runner crashed mid-suite (OllamaException: llama-server process has terminated: signal: segmentation fault (core dumped)), and all three surefire retries of the 5 erroring tests hit the same dead server. This PR only touches tools/install.sh and its bats tests (covered by the separate install.sh tests jobs, green on both ubuntu and macos); the Java IT job doesn't execute install.sh. The same leg passes on recent main runs. A re-run should clear it - happy to rebase instead if you prefer.

weiqingy · 2026-07-10T06:34:02Z

    py_minor="${py_version_output##*.}"

-    if [[ "$py_major" -ne 3 ]] || [[ "$py_minor" -lt 10 ]] || [[ "$py_minor" -ge 12 ]]; then
+    if [[ "$py_major" -ne 3 ]] || [[ "$py_minor" -lt 10 ]] || [[ "$py_minor" -ge "$(python_minor_ceiling)" ]]; then


This line is where the interpreter check starts depending on the selected release — and once resolve_python has run, the confirm screen can still change that release out from under it. edit_plan_interactive's flink_agents_version) arm reassigns FLINK_AGENTS_VERSION (install.sh:1112-1114 → 693) without re-checking the already-resolved PYTHON_BIN, unlike the enable_pyflink) arm right below it, which re-runs resolve_python when its dependent changes (install.sh:1153-1160). Concretely: a user on Python 3.12 accepts the new 0.3.0 default (validates here, ceiling 13), then at the confirm screen edits the version down to 0.2.1; setup_python_env then runs pip install flink-agents==0.2.1 (install.sh:1434), whose requires-python is <3.12, and the install fails after the plan looked good. Before this PR the interpreter was independent of the release, so this window didn't exist. Would re-validating PYTHON_BIN after a version edit — mirroring the enable_pyflink) arm — catch the mismatch at plan time rather than mid-install?

Good catch - fixed in c4f81b4. The flink_agents_version) arm now calls a new revalidate_python_for_agents_version after the version prompt: if PyFlink is enabled and the already-resolved PYTHON_BIN no longer satisfies the new release's ceiling, it warns and re-runs resolve_python (same semantics as the enable_pyflink) arm - and since PYTHON_BIN is part of the edit-state dump, the re-resolved interpreter propagates back to the plan). Your exact scenario (3.12 accepted under 0.3.0, then edited down to 0.2.1) is covered by a new bats test, plus four more for the keep/no-op paths.

🤖 Generated with Claude Code

wenjin272 · 2026-07-10T08:23:42Z

Minor test fixture drift: reset_install_sh_state in load.bash still resets the Flink Agents default/recommended/supported versions to the old 0.2.1 matrix, while install.sh now defaults to 0.3.0. Could we update the helper so sourced unit tests exercise the same defaults as the real installer?

The Python ceiling now depends on the selected release, so editing the version at the confirm screen could invalidate an interpreter that resolve_python had already accepted, failing later in setup_python_env. Mirror the enable_pyflink edit arm: re-resolve at plan time.

reset_install_sh_state still carried the 0.2.1 version matrix, so sourced unit tests exercised different defaults than the installer ships. Align it with the 0.3.0 defaults.

Olawoyin007 · 2026-07-10T09:22:52Z

Thanks - synced in ece24ba: reset_install_sh_state now carries the 0.3.0 default/recommended/supported matrix, matching install.sh. The full suite still passes (154 tests, now including 5 for the confirm-screen re-validation weiqingy flagged above).

🤖 Generated with Claude Code

wenjin272

Hi, @Olawoyin007, I left two new comments.

wenjin272 · 2026-07-13T06:27:22Z

+    case "$FLINK_AGENTS_VERSION" in
+        0.1.*|0.2.*) printf '12' ;;
+        *) printf '13' ;;
+    esac


Python 3.12 compatibility also depends on the selected Flink version, not only the Flink Agents version. The installation docs require Flink 2.1+ for Python 3.12, but 0.3.0 + Flink 1.20/2.0 + Python 3.12 currently passes this validation and only fails later when apache-flink==$FLINK_VERSION is installed.

Could we derive the Python constraint from both versions and revalidate it whenever the Agents version, Flink version, or detected FLINK_HOME changes?

Good catch - fixed in 1839dcd. python_minor_ceiling now derives the ceiling from both versions and returns the stricter of the two: the Flink Agents axis (0.1.x/0.2.x → <3.12, 0.3.0+ → <3.13) and a new Flink axis (Flink <2.1 → <3.12, since Python 3.12 needs Flink 2.1+). The Flink comparison is major-then-minor numeric (so 1.20.x isn't mistaken for ≥2.1) and reuses flink_major_minor, so snapshots/rc/source builds like 2.1-SNAPSHOT resolve correctly; an unparseable FLINK_VERSION leaves the Flink axis unrestricted (fail open). Because validate_python_bin and resolve_python both consult the ceiling, they pick this up automatically.

For revalidation on change: revalidate_python_for_agents_version is renamed revalidate_python_constraint and now also runs on the flink_version and install_flink edit arms (both mutate FLINK_VERSION, and install_flink's No branch re-detects from FLINK_HOME). Your exact case - 0.3.0 + Flink 2.0.x + Python 3.12 - is covered by new bats cases in validate_python_bin.bats and revalidate_python_constraint.bats.

One scoping note: the standalone flink_home edit arm currently only repoints FLINK_HOME and deliberately does not re-detect the Flink version (unlike the install_flink arm), so I left its semantics unchanged rather than widen it here. Happy to add version re-detection to that arm too if you'd prefer it fully symmetric.

wenjin272 · 2026-07-13T06:29:17Z

+        return 0
+    fi
+    ui_warn "The selected Python (${PYTHON_BIN}) is incompatible with Flink Agents ${FLINK_AGENTS_VERSION} (requires Python >=3.10 and <3.$(python_minor_ceiling))"
+    PYTHON_BIN=""


This revalidates PYTHON_BIN, but an existing VENV_DIR may still use the incompatible interpreter. For example, after changing Agents from 0.3.0 to 0.2.1, this can select Python 3.11 while setup_python_env later reuses and activates an existing Python 3.12 venv. Pip then still runs under Python 3.12 and flink-agents==0.2.1 fails.

Could we also validate $VENV_DIR/bin/python and ask the user to select or recreate the venv when it is incompatible? A regression test using an existing Python 3.12 venv would cover this path.

Fixed in 1839dcd. You're right that re-resolving PYTHON_BIN doesn't help if an existing venv is reused with its own interpreter. Two layers now:

revalidate_existing_venv (called from revalidate_python_constraint, so it runs on every version edit) checks a reused venv's bin/python against the current ceiling at plan time. If it's incompatible it offers to recreate it or point at a different path; recreation is deferred to setup_python_env via a RECREATE_VENV flag (so a later cancel never destroys the venv) and guarded to a directory carrying a pyvenv.cfg marker, so it only ever removes a real venv. Non-interactive callers get an actionable error instead of a mid-install failure.

setup_python_env re-validates a reused venv's interpreter as a final guard for any path that skips plan-time revalidation (e.g. an explicit VENV_DIR in non-interactive mode), failing early with a clear message rather than at pip install.

Your regression scenario (Agents 0.3.0 → 0.2.1, existing Python 3.12 venv) is covered by new cases in revalidate_python_constraint.bats, including the non-interactive die path. Full suite is 162 passing.

…on and existing venvs Two plan-time gaps left Python 3.12 selections that validate cleanly but fail later at pip install: 1. python_minor_ceiling considered only FLINK_AGENTS_VERSION. Python 3.12 also requires Flink 2.1+, so 0.3.0 + Flink 2.0.x/1.20.x + Python 3.12 passed validation and failed when apache-flink was installed. The ceiling now takes the stricter of the Agents and Flink axes (major-then-minor numeric compare; unparseable Flink versions leave the Flink axis unrestricted, i.e. fail open). resolve_python and validate_python_bin inherit this automatically. revalidate_python_for_agents_version is renamed revalidate_python_constraint and now also runs on the flink_version and install_flink edit arms, which mutate FLINK_VERSION. 2. Re-resolving PYTHON_BIN did not cover an existing venv: setup_python_env reuses a venv as-is, so its own interpreter is what pip runs under. After editing the Agents version down, an existing 3.12 venv would still be reused and fail. revalidate_existing_venv now checks the reused venv's interpreter at plan time and offers to recreate it (deferred to setup_python_env via RECREATE_VENV, guarded to a pyvenv.cfg marker so it only ever removes a real venv) or point at a different path; non-interactive callers get an actionable error. setup_python_env also re-checks a reused venv's interpreter as a final guard for paths that skip plan-time revalidation (e.g. an explicit VENV_DIR in non-interactive mode). Tests: Flink-axis ceiling cases in validate_python_bin.bats; Flink-version re-resolution and existing-venv revalidation cases in the renamed revalidate_python_constraint.bats. Full suite 162 passing.

Olawoyin007 · 2026-07-13T13:31:11Z

Heads up on the three red integration jobs on this push (it-python 3.12/flink-2.1, it-python 3.12/flink-2.2, it-java flink-1.20) - they're the same Ollama infra flake as before, not this change:

FAILED flink_agents/integrations/chat_models/tests/test_ollama_chat_model.py::test_ollama_chat
  - ollama._types.ResponseError: llama-server process has terminated:
    signal: segmentation fault (core dumped) (status code: 500)

The runner's llama-server segfaulted mid-suite. A couple of things confirm it's unrelated to this PR: this change only touches tools/install.sh and its bats tests (no Python/Java source), the install.sh tests jobs on ubuntu and macos both passed, and the same segfault is hitting main right now on different matrix cells (nondeterministic). Happy to push an empty commit to trigger a fresh run if that's easier than a re-run on your side.

github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 9, 2026

github-actions Bot added doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs labels Jul 9, 2026

weiqingy reviewed Jul 10, 2026

View reviewed changes

Olawoyin007 added 2 commits July 10, 2026 10:22

[tools] Sync test fixture version defaults with install.sh

ece24ba

reset_install_sh_state still carried the 0.2.1 version matrix, so sourced unit tests exercised different defaults than the installer ships. Align it with the 0.3.0 defaults.

wenjin272 reviewed Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[tools] Update install.sh for Flink Agents 0.3.0 and Python 3.12#888

[tools] Update install.sh for Flink Agents 0.3.0 and Python 3.12#888
Olawoyin007 wants to merge 4 commits into
apache:mainfrom
Olawoyin007:install-sh-0.3.0

Olawoyin007 commented Jul 9, 2026 •

edited

Loading

Uh oh!

Olawoyin007 commented Jul 9, 2026

Uh oh!

weiqingy Jul 10, 2026

Uh oh!

wenjin272 Jul 10, 2026

Uh oh!

Olawoyin007 Jul 10, 2026

Uh oh!

wenjin272 commented Jul 10, 2026

Uh oh!

Olawoyin007 commented Jul 10, 2026

Uh oh!

wenjin272 left a comment

Uh oh!

wenjin272 Jul 13, 2026

Uh oh!

Olawoyin007 Jul 13, 2026

Uh oh!

wenjin272 Jul 13, 2026

Uh oh!

Olawoyin007 Jul 13, 2026

Uh oh!

Olawoyin007 commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Olawoyin007 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of change

Tests

API

Documentation

Uh oh!

Olawoyin007 commented Jul 9, 2026

Uh oh!

weiqingy Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Olawoyin007 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 commented Jul 10, 2026

Uh oh!

Olawoyin007 commented Jul 10, 2026

Uh oh!

wenjin272 left a comment

Choose a reason for hiding this comment

Uh oh!

wenjin272 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Olawoyin007 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Olawoyin007 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Olawoyin007 commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Olawoyin007 commented Jul 9, 2026 •

edited

Loading