diff --git a/docs/how-tos/airflow/airflow-3-migration-check.mdx b/docs/how-tos/airflow/airflow-3-migration-check.mdx new file mode 100644 index 0000000..80c0d86 --- /dev/null +++ b/docs/how-tos/airflow/airflow-3-migration-check.mdx @@ -0,0 +1,83 @@ +--- +title: Check your DAGs for Airflow 3 migration issues +sidebar_label: Airflow 3 migration check +description: "Use the datacoves airflow3-check command to find and auto-fix Airflow 2 to 3 migration issues in your DAG code before upgrading your environment." +sidebar_position: 39 +--- +# Check your DAGs for Airflow 3 migration issues + +:::info +The `datacoves airflow3-check` command is available in Datacoves 6.1 and later. It works regardless of the Airflow version your environment runs, so you can (and should) use it while still on Airflow 2. +::: + +When an environment is upgraded from Airflow 2 to Airflow 3, DAG code written for Airflow 2 may rely on imports, arguments, or behaviors that were removed or deprecated in Airflow 3, for example: + +| Airflow 2 pattern | Airflow 3 replacement | +| --- | --- | +| `from airflow.decorators import dag, task` | `from airflow.sdk import dag, task` | +| `from airflow.operators.bash import BashOperator` | `from airflow.providers.standard.operators.bash import BashOperator` | +| `schedule_interval="@daily"` | `schedule="@daily"` | +| `email_on_failure` / `email_on_retry` in `default_args` | SMTP notifications (`SmtpNotifier`) | +| Runtime-varying values in DAG definitions (e.g. `datetime.today()` in `start_date`) | Static values; Airflow 3 creates a new DAG version on every parse otherwise | + +Finding every occurrence across a repository by hand is tedious. The Datacoves code-server image ships the `datacoves airflow3-check` command, which scans your DAG code for all of these using [ruff's Airflow lint rules](https://docs.astral.sh/ruff/rules/#airflow-air), maintained alongside the Airflow project itself. + +## Run the check + +Open a terminal in your Datacoves workbench and run: + +```bash +datacoves airflow3-check +``` + +By default it checks the DAGs folder configured for your environment. Each finding shows the file, line, offending pattern, and the suggested replacement: + +``` +airflow3-suggested-update: `airflow.decorators.dag` is removed in Airflow 3.0 + --> orchestrate/dags/daily_run.py:13:2 +help: `dag` has been moved to `airflow.sdk` since Airflow 3.0 +``` + +The command exits with code `0` when no issues are found and `1` when there are findings, so you can also use it in scripts or CI. + +To check a different directory (for example a scratch folder, or a repo with a non-standard layout), pass it explicitly: + +```bash +datacoves airflow3-check path/to/dags +``` + +## Apply automatic fixes + +Some issues have unambiguous, safe fixes (for example `schedule_interval` -> `schedule`). Apply them with: + +```bash +datacoves airflow3-check --fix +``` + +Changes that could alter behavior, such as rewriting imports to their new locations, are reported but not applied automatically; review each finding's `help:` line and update the code by hand. + +:::tip +Commit or stash your work before running `--fix` so you can review the changes with `git diff`. +::: + +## Recommended workflow + +1. Run `datacoves airflow3-check` on your repository while the environment is still on Airflow 2 and work through the findings. Fixes like `schedule_interval` -> `schedule` already work on recent Airflow 2 releases; for new-location imports such as `airflow.sdk` (Airflow 3 only), use a portable pattern while both versions are in play: + + ```python + try: + # Airflow 3 + from airflow.sdk import dag, task + except ImportError: + # Airflow 2 + from airflow.decorators import dag, task + ``` + + The check recognizes this pattern and will not flag the Airflow 2 import inside the `except` branch. + +2. Re-run it until it reports no issues. +3. Coordinate the environment upgrade with your Datacoves administrator. + +:::note +The check covers Python DAG files. DAGs generated from YAML definitions (`dbt-coves generate airflow-dags`) are regenerated by the tooling and do not usually need manual migration. +:::