diff --git a/README.md b/README.md index dd3cb401d..2329d3be5 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ For the most detailed and up-to-date documentation please visit our Instinct Doc ## Introduction -AMD GPU Operator simplifies the deployment and management of AMD Instinct GPU accelerators within Kubernetes clusters. This project enables seamless configuration and operation of GPU-accelerated workloads, including machine learning, Generative AI, and other GPU-intensive applications. +AMD GPU Operator simplifies the deployment and management of AMD Instinct™ and AMD Radeon™ GPU accelerators within Kubernetes clusters. This project enables seamless configuration and operation of GPU-accelerated workloads, including machine learning, Generative AI, and other GPU-intensive applications. ## Components diff --git a/docs-internal/knowledge/plans/2026-07-09-gpuop-969-argo-finalizers-rbac.md b/docs-internal/knowledge/plans/2026-07-09-gpuop-969-argo-finalizers-rbac.md deleted file mode 100644 index 21a69be62..000000000 --- a/docs-internal/knowledge/plans/2026-07-09-gpuop-969-argo-finalizers-rbac.md +++ /dev/null @@ -1,111 +0,0 @@ -# Fix ANR Remediation Workflow Test Step on OpenShift — Add `workflows/finalizers` RBAC - -- **Date:** 2026-07-09 -- **Author:** Uday Bhaskar Biluri -- **Related PR(s):** #1593 -- **Related issue(s) / JIRA:** [GPUOP-969](https://pensando.atlassian.net/browse/GPUOP-969); regression from [KUBE-35](https://amd.atlassian.net/browse/KUBE-35) / PR #1575 (see `docs-internal/knowledge/plans/2026-07-06-kube-35-argo-rbac-least-privilege.md`) - -## Context - -All ANR test cases that run a full remediation workflow fail on OpenShift. The -workflow `test` step — which creates the test-runner `Job` and `ConfigMap` — is -rejected by the API server immediately: - -``` -Error from server (Forbidden): configmaps "-cm" is forbidden: - cannot set blockOwnerDeletion if an ownerReference refers to a resource - you can't set finalizers on: , -Error from server (Forbidden): jobs.batch "-test" is forbidden: ... -``` - -Affected tests (all OpenShift clusters — MI210, MI350P, W7900; oc-7 CI 2026-07-07): - -- `test_node_remediation::test_anr_workflow` -- `test_node_remediation::test_tester_image_frameworks[RVS]` -- `test_node_remediation::test_tester_image_frameworks[AGFHC]` -- `test_node_remediation::test_node_label_removal` - -**Root cause.** `internal/controllers/remediation/scripts/test.sh` creates the -`ConfigMap` and `Job` with an `ownerReference` back to the parent Argo `Workflow` -using `blockOwnerDeletion: true` / `controller: true`. Kubernetes' garbage-collector -admission plugin performs a single authorization check when `blockOwnerDeletion` -is set: the requester must have `update` on the owner type's `finalizers` -subresource — here `argoproj.io/workflows/finalizers`. - -The least-privilege Argo RBAC introduced by KUBE-35 (PR #1575) replaced the prior -wildcard grant with an explicit rule set that covers `workflows` and -`workflowtemplates` but not the distinct `workflows/finalizers` subresource. The -operator runs the workflow pods under the same `controller-manager` SA, so the -`test.sh` create calls fail the finalizers check. OpenShift enforces this strictly; -vanilla Kubernetes CI does not, which is why the regression went unnoticed. - -## Approach - -Add a single rule granting `update` on the `workflows/finalizers` subresource to -both Argo RBAC source files (both bound to the same operator `controller-manager` -SA and which must stay in sync): - -```yaml -- apiGroups: [argoproj.io] - resources: [workflows/finalizers] - verbs: [update] -``` - -This is both necessary and sufficient: the blockOwnerDeletion admission check only -requires the `update` verb on the `finalizers` subresource — no `create`/`patch`/ -`delete` on finalizers, and no additional verbs on `jobs`/`configmaps` (those -creates were failing only because of the finalizers check, not missing job/CM -permissions). - -After editing the two source files, regenerate the derived artifacts (do not -hand-edit the generated CSV / Helm template — they are protected generated files): - -- `make helm-k8s` — copies `hack/k8s-patch/template-patch/*` into - `helm-charts-k8s/templates/` and rebuilds the chart `*.tgz`. -- `make bundle-build` — regenerates the RBAC block in the OLM CSV. - -### Alternatives considered - -- **Grant broader verbs on `workflows/finalizers` (e.g. `create`, `patch`)** — - rejected: the admission check only exercises `update`; anything more re-widens - the grant the KUBE-35 hardening deliberately narrowed. -- **Revert to the pre-KUBE-35 wildcard Argo RBAC** — rejected: reintroduces the - SEC-00455 security finding. -- **Drop `blockOwnerDeletion` from `test.sh` owner references** — rejected: the - owner reference provides automatic garbage collection of the test `Job`/`ConfigMap` - when the parent workflow is deleted; changing runtime behavior is riskier than a - one-line RBAC addition. - -## Scope - -- **In scope:** - - `config/manifests/argo-workflow-rbac.yaml` (OLM bundle source) - - `hack/k8s-patch/template-patch/argo-rbac.yaml` (Helm source) - - Regenerated artifacts: `helm-charts-k8s/templates/argo-rbac.yaml`, chart `*.tgz`, - and `bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml`. -- **Out of scope:** - - Argo workflow-controller RBAC (`remediation-deployment.yaml`) — different SA. - - Any change to `test.sh` owner-reference semantics. - - Namespace-scoping / dedicated workflow SA (tracked as KUBE-35 follow-ups). - -## Validation - -- **Static review:** confirm the `workflows/finalizers: [update]` rule appears in - both source files, the regenerated Helm template, and the CSV RBAC block; confirm - nothing else changed. -- **Integration / e2e tests:** on an OpenShift cluster with `remediation.enabled: true`, - run `tests/pytests/openshift/gpu-operator/test_node_remediation.py::test_anr_workflow` - (plus the tester-image-framework and node-label-removal cases) and confirm the - `test` step creates the `ConfigMap` + `Job`, the runner completes, and the workflow - reaches a Succeeded state with no `Forbidden` error. -- **Install paths:** verify both a Helm install and an OLM bundle install come up - with the updated role and remediation functions. - -## Risks and rollback - -- **Known risks:** Low — purely additive RBAC (one subresource, single `update` - verb). Minor risk that the two source files drift; mitigated by regenerating and - reviewing the CSV/Helm diff against the source edits. -- **Rollback plan:** revert the two source-file edits and regenerate the artifacts - (or revert the PR). No cluster/runtime state persists; the change is purely RBAC - manifest content. diff --git a/docs/index.md b/docs/index.md index efac65c61..a0ba824a4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,6 @@ # AMD GPU Operator Documentation -The AMD GPU Operator simplifies the deployment and management of AMD Instinct GPU accelerators within Kubernetes clusters. This project enables seamless configuration and operation of GPU-accelerated workloads, including machine learning, Generative AI, and other GPU-intensive applications. +The AMD GPU Operator simplifies the deployment and management of AMD Instinct™ and AMD Radeon™ GPU accelerators within Kubernetes clusters. This project enables seamless configuration and operation of GPU-accelerated workloads, including machine learning, Generative AI, and other GPU-intensive applications. ## Features @@ -45,23 +45,23 @@ Below is a matrix of supported Operating systems and the corresponding Kubernete Ubuntu 22.04 LTS - 1.29-1.35 + 1.29-1.36 Ubuntu 24.04 LTS - 1.29-1.35 + 1.29-1.36 Debian 12 - 1.29-1.35 (driver management not supported) + 1.29-1.36 (driver management not supported) Red Hat Core OS (RHCOS) - 4.16—4.20 + 4.16—4.22 diff --git a/docs/installation/kubernetes-helm.md b/docs/installation/kubernetes-helm.md index e5d2fa352..499c3b823 100644 --- a/docs/installation/kubernetes-helm.md +++ b/docs/installation/kubernetes-helm.md @@ -16,7 +16,7 @@ This guide walks through installing the AMD GPU Operator on a Kubernetes cluster ### System Requirements -- Kubernetes cluster v1.29.0 or later +- Kubernetes cluster v1.29.0 to v1.36.x - Helm v3.2.0 or later - `kubectl` command-line tool configured with access to the cluster - Cluster admin privileges diff --git a/docs/releasenotes.md b/docs/releasenotes.md index d9fb986ef..30353b296 100644 --- a/docs/releasenotes.md +++ b/docs/releasenotes.md @@ -1,23 +1,26 @@ # Release Notes -## GPU Operator v1.5.1 (beta) Release Notes +## GPU Operator v1.5.1 Release Notes -The AMD GPU Operator v1.5.1 (beta) release extends hardware support to the **AMD Instinct MI350P** and **AMD Radeon AI Pro** GPU families, and updates the Device Metrics Exporter with configurable exporter arguments. +The AMD GPU Operator v1.5.1 release extends hardware support to the **AMD Instinct™ MI350P** and **AMD Radeon™ AI PRO** GPU families, and updates the Device Metrics Exporter with configurable exporter arguments. ### Release Highlights -- **AMD Instinct MI350P Platform Support** - - The GPU Operator now supports AMD Instinct MI350P GPUs on both vanilla Kubernetes and OpenShift. +- **AMD Instinct™ MI350P Platform Support** + - The GPU Operator now supports AMD Instinct™ MI350P GPUs on both vanilla Kubernetes and OpenShift. - Requires ROCm 7.13+ (amdgpu driver 6.19+). - All core operator features are supported: driver management (KMM), device plugin, node labeller, metrics exporter, test runner, DCM, DRA driver, and NPD integration. - - > **Note:** Auto Node Remediation (ANR) is not fully supported on MI350P in this beta release. + - > **Note:** Auto Node Remediation (ANR) is not fully supported on MI350P in this release. -- **AMD Radeon AI Pro Platform Support** - - The GPU Operator now supports AMD Radeon AI Pro workstation GPUs (RDNA 4 architecture, e.g. Radeon AI Pro R9700S) on Kubernetes. +- **AMD Radeon™ AI PRO Platform Support** + - The GPU Operator now supports AMD Radeon™ AI PRO workstation GPUs (RDNA 4 architecture, e.g. Radeon AI PRO R9700/R9700S) on Kubernetes. - Requires ROCm 7.13+ (amdgpu driver 6.19+). - **Supported features:** driver management, device plugin workload scheduling, and metrics monitoring. - **Not supported:** GPU partitioning (DCM), XGMI topology-aware scheduling, and full ECC metrics — these are Instinct (CDNA) platform features and do not apply to Radeon (RDNA) GPUs. - - > **Note:** Auto Node Remediation (ANR) is not fully supported on Radeon AI platforms in this beta release. + - > **Note:** Auto Node Remediation (ANR) is not fully supported on Radeon AI PRO platforms in this release. + +- **ROCm 7.14 Managed Stack** + - The metrics exporter, test runner, and DCM operand images are packaged with ROCm 7.14 runtime libraries in this release. - **Device Metrics Exporter: Configurable Exporter Arguments** - The Helm chart now supports passing arbitrary arguments to the exporter at deploy time via new deployment options: @@ -27,12 +30,13 @@ The AMD GPU Operator v1.5.1 (beta) release extends hardware support to the **AMD ### Platform Support -- **AMD Instinct MI350P**: Kubernetes 1.29 – 1.35, OpenShift 4.21 -- **AMD Radeon AI Pro**: Kubernetes 1.29 – 1.35 +- **AMD Instinct™ MI350P**: Kubernetes 1.29 – 1.36, OpenShift 4.21 – 4.22 +- **AMD Radeon™ AI PRO**: Kubernetes 1.29 – 1.36 ### Fixes -- Platform support additions for MI350P and Radeon AI Pro GPU families. +- Platform support additions for MI350P and Radeon AI PRO GPU families. +- **Driver upgrade reboot could leave operands with `exec format error`** — the driver-upgrade reboot pod now flushes filesystems (`sync`) before rebooting the node, and the DRA driver is now included in the pre-reboot drain list, preventing truncated operand executables after an operator-managed upgrade reboot. [[ROCm/gpu-operator#606]](https://github.com/ROCm/gpu-operator/pull/606) ### Known Limitations @@ -45,10 +49,6 @@ The AMD GPU Operator v1.5.1 (beta) release extends hardware support to the **AMD - **Node Labeller reports incorrect product name for Radeon AI GPUs** - On Radeon AI platforms, the `amd.com/gpu.product-name` node label is populated as `AMD_Radeon_Graphics` instead of the specific model name (e.g., `AMD Radeon AI Pro R9700S`). This is a known issue and will be fixed in a future release. -- **Device Metrics Exporter: CPER-based health detection and `GPU_AFID_ERRORS` disabled in this release** - - CPER-based health detection is not functional and the `GPU_AFID_ERRORS` field is disabled in this release for **all platforms**, including MI350P, Radeon AI, and existing MI2xx/MI3xx deployments. - - **Users on MI2xx/MI3xx platforms who rely on CPER-based health detection or `GPU_AFID_ERRORS` should not upgrade the Device Metrics Exporter to `v1.5.1-beta.0` until this is resolved in a future release.** - - **Device Metrics Exporter: `gfx_activity` idle uptick on Radeon AI** - On Radeon AI platforms, `gfx_activity` may show a spurious uptick when the GPU is idle. This is a cosmetic issue and does not indicate actual GPU activity.