Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
---
kind:
- Troubleshooting
products:
- Alauda Container Platform
ProductsVersion:
- '4.3.x,4.4.x'
---

# Resolve Host FRR and MetalLB FRR Conflicts by Using the Native BGP Backend

## Issue

On a bare-metal Alauda Container Platform cluster, a customer-managed FRR service runs as a systemd unit on the nodes and has established BGP sessions. After the MetalLB plugin is installed, the MetalLB Speaker Pods start their own FRR processes and the host's main routing table can lose BGP routes. The two FRR instances can also interfere with each other's BGP sessions.

This solution applies when the host FRR service and MetalLB Speakers run on the same nodes. It does not apply to OpenShift clusters.

## Environment

- Alauda Container Platform 4.3.x or 4.4.x.
- A bare-metal, non-OpenShift cluster.
- The MetalLB plugin is installed and configured for BGP advertisement.
- A customer-managed FRR service runs on one or more nodes as a systemd unit.

## Root Cause

MetalLB Speakers use `hostNetwork: true`. With the `frr` BGP backend, each Speaker Pod also runs the MetalLB-managed `frr`, `reloader`, and `frr-metrics` containers. These processes share the node network namespace with the systemd-managed FRR service, so both FRR instances can modify the host routing table and manage overlapping BGP state.

For non-OpenShift clusters, MetalLB uses the `frr` backend when `spec.bgpBackend` is not set. The MetalLB `MetalLB` custom resource supports the `native` backend, which establishes BGP sessions without deploying the MetalLB FRR containers.

## Resolution

Use the `native` BGP backend and give MetalLB its own BGP session identity.

:::warning
Updating the `MetalLB` resource rolls the Speaker DaemonSet and can briefly interrupt MetalLB BGP advertisements. Perform the change during a maintenance window and confirm the advertised VIP routes after the rollout.
:::

### 1. Confirm the current MetalLB backend

The plugin creates a `MetalLB` resource named `metallb` in the `metallb-system` namespace by default. Run the following commands before changing it:

```bash
kubectl -n metallb-system get metallb
kubectl -n metallb-system get metallb metallb \
-o jsonpath='{.spec.bgpBackend}{"\n"}'
kubectl -n metallb-system get daemonset speaker \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
```

If the `bgpBackend` output is empty on a non-OpenShift cluster, the Operator uses `frr` by default. If the container list includes `frr`, the MetalLB FRR process is running in the Speaker Pod. Replace `metallb` in the commands if the resource has a different name.

### 2. Configure independent BGP sessions

In the target cluster, go to **Administrator -> Network Management -> BGP Peers** and create or edit the BGP peers used by MetalLB. Configure values reserved for MetalLB:

- **Local AS Number**: The local AS number for the MetalLB session. Do not reuse the host FRR local AS number when the upstream router requires separate sessions.
- **Peer AS Number** and **Peer IP**: The values configured on the upstream router for the MetalLB session.
- **Local IP**: A source address that is different from the source address used by the host FRR session.
- **RouterID**: A router ID that is different from the host FRR router ID and other BGP instances on the node.
- **BGP-Connected Node**: Only the nodes that have the MetalLB source address and should run the MetalLB Speaker.

If MetalLB and the host FRR service use the same upstream router, configure the router to accept both sessions. MetalLB and host FRR must not reuse the same local address, router ID, or advertised prefixes.

### 3. Configure the BGP external address pool

Go to **Administrator -> Network Management -> External IP Address Pool** and create or edit the pool used by the LoadBalancer Services:

1. Set **Type** to **BGP**.
2. Enter the MetalLB VIP range in **IP Resources**.
3. Associate the MetalLB BGP peer.
4. Select only the nodes that are allowed to advertise the VIP range.

The VIP range must not overlap with prefixes advertised by the host FRR service.

### 4. Switch MetalLB to the native BGP backend

The console does not expose the `spec.bgpBackend` field. A platform administrator must set it with `kubectl`:

```bash
kubectl -n metallb-system patch metallb metallb \
--type=merge \
-p '{"spec":{"bgpBackend":"native"}}'
```

The command should report that the resource was configured. The Operator then rolls the Speaker DaemonSet and removes the MetalLB-managed FRR containers. It does not stop or reconfigure the host FRR systemd service.

### 5. Verify the result

Wait for the Speaker rollout to complete:

```bash
kubectl -n metallb-system rollout status daemonset/speaker
```

Confirm that the backend is `native` and the Speaker template no longer contains the MetalLB FRR containers:

```bash
kubectl -n metallb-system get metallb metallb \
-o jsonpath='{.spec.bgpBackend}{"\n"}'
kubectl -n metallb-system get daemonset speaker \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
kubectl -n metallb-system get pods -l app=metallb,component=speaker -o wide
```

The first command must return `native`. The container list must not include `frr`, `reloader`, `frr-metrics`, or `metrics-auth-proxy-frr`. All Speaker Pods should be `Running` and `Ready`. Pod readiness alone does not prove that BGP sessions are established or that VIP prefixes are advertised; confirm both the MetalLB session and the existing host FRR sessions with the upstream router or the customer's normal network monitoring tools.

The `MetalLB` resource stores this setting. Recheck `spec.bgpBackend` after a MetalLB plugin upgrade or reinstall, because a resource recreation or reset can restore the default backend.

## Diagnostic Steps

Use the following checks to determine whether the host FRR and MetalLB FRR conflict is present:

```bash
kubectl -n metallb-system get metallb metallb \
-o jsonpath='{.spec.bgpBackend}{"\n"}'
kubectl -n metallb-system get daemonset speaker \
-o jsonpath='{.spec.template.spec.hostNetwork}{"\n"}{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
kubectl -n metallb-system get bgppeers,bgpadvertisements,ipaddresspools
```

An empty backend on a non-OpenShift cluster and an `frr` container in the Speaker template indicate that the default MetalLB FRR backend is active. `hostNetwork` should be `true`. Compare the BGP peer source address, router ID, local AS number, and advertised prefixes with the host FRR configuration and the upstream router configuration.

## Rollback

If the native backend cannot meet the BGP requirements, restore the FRR backend:

```bash
kubectl -n metallb-system patch metallb metallb \
--type=merge \
-p '{"spec":{"bgpBackend":"frr"}}'
kubectl -n metallb-system rollout status daemonset/speaker
```

After the rollout, confirm that the required FRR containers are present and validate the BGP sessions. Do not roll back while the original host FRR conflict is unresolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
---
kind:
- Troubleshooting
products:
- Alauda Container Platform
ProductsVersion:
- '4.3.x,4.4.x'
---

# 使用 Native BGP 后端解决主机 FRR 与 MetalLB FRR 冲突

## 问题

在裸金属 Alauda Container Platform 集群中,客户自行维护的 FRR 服务以 systemd 单元的方式运行在节点上,并且已经建立 BGP 会话。安装 MetalLB 插件后,MetalLB Speaker Pod 会启动自身的 FRR 进程,可能导致主机主路由表中的 BGP 路由丢失。两套 FRR 实例还可能相互干扰 BGP 会话。

当主机 FRR 服务与 MetalLB Speaker 运行在相同节点上时,适用本解决方案。本方案不适用于 OpenShift 集群。

## 环境

- Alauda Container Platform 4.3.x 或 4.4.x。
- 裸金属、非 OpenShift 集群。
- 已安装 MetalLB 插件,并已配置 BGP 宣告。
- 一个或多个节点上以 systemd 单元的方式运行客户自行维护的 FRR 服务。

## 根本原因

MetalLB Speaker 使用 `hostNetwork: true`。当 BGP 后端为 `frr` 时,每个 Speaker Pod 还会运行由 MetalLB 管理的 `frr`、`reloader` 和 `frr-metrics` 容器。这些进程与 systemd 管理的 FRR 服务共享节点网络命名空间,因此两套 FRR 实例都可能修改主机路由表并管理相互重叠的 BGP 状态。

在非 OpenShift 集群中,如果未设置 `spec.bgpBackend`,MetalLB 默认使用 `frr` 后端。MetalLB 自定义资源支持 `native` 后端;该后端建立 BGP 会话时不会部署 MetalLB FRR 容器。

## 解决方案

使用 `native` BGP 后端,并为 MetalLB 配置独立的 BGP 会话身份。

:::warning
更新 `MetalLB` 资源会滚动更新 Speaker DaemonSet,可能短暂中断 MetalLB 的 BGP 宣告。请在维护窗口内执行变更,并在滚动更新完成后确认 VIP 路由已重新宣告。
:::

### 1. 确认当前 MetalLB 后端

插件默认会在 `metallb-system` 命名空间中创建名为 `metallb` 的 `MetalLB` 资源。在修改资源前执行以下命令:

```bash
kubectl -n metallb-system get metallb
kubectl -n metallb-system get metallb metallb \
-o jsonpath='{.spec.bgpBackend}{"\n"}'
kubectl -n metallb-system get daemonset speaker \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
```

在非 OpenShift 集群中,如果 `bgpBackend` 输出为空,表示 Operator 默认使用 `frr`。如果容器列表中包含 `frr`,表示 MetalLB FRR 进程正在 Speaker Pod 中运行。如果资源名称不同,请在后续命令中将 `metallb` 替换为实际资源名。

### 2. 配置独立的 BGP 会话

在目标集群中,进入 **管理员 -> 网络管理 -> BGP 对等体**,创建或编辑 MetalLB 使用的 BGP 对等体。配置为 MetalLB 保留的参数:

- **本地 AS 号**:MetalLB 会话使用的本地 AS 号。当上游路由器要求使用独立会话时,不要复用主机 FRR 的本地 AS 号。
- **对端 AS 号** 和 **对端 IP**:上游路由器为 MetalLB 会话配置的参数。
- **本地 IP**:与主机 FRR 会话使用的源地址不同的地址。
- **RouterID**:与主机 FRR Router ID 以及节点上的其他 BGP 实例不同的 Router ID。
- **BGP 连接节点**:仅选择配置了 MetalLB 源地址并且需要运行 MetalLB Speaker 的节点。

如果 MetalLB 和主机 FRR 服务使用同一个上游路由器,请在路由器上配置并接受两个会话。MetalLB 与主机 FRR 不得复用相同的本地地址、Router ID 或宣告前缀。

### 3. 配置 BGP 外部地址池

进入 **管理员 -> 网络管理 -> 外部 IP 地址池**,创建或编辑 LoadBalancer 服务使用的地址池:

1. 将 **类型** 设置为 **BGP**。
2. 在 **IP 资源** 中填写 MetalLB VIP 范围。
3. 关联 MetalLB BGP 对等体。
4. 仅选择允许宣告该 VIP 范围的节点。

VIP 范围不得与主机 FRR 服务宣告的前缀重叠。

### 4. 将 MetalLB 切换到 Native BGP 后端

控制台不提供 `spec.bgpBackend` 字段。平台管理员必须使用 `kubectl` 设置该字段:

```bash
kubectl -n metallb-system patch metallb metallb \
--type=merge \
-p '{"spec":{"bgpBackend":"native"}}'
```

命令应报告资源已被配置。随后 Operator 会滚动更新 Speaker DaemonSet,并移除由 MetalLB 管理的 FRR 容器。该操作不会停止或重新配置主机上的 FRR systemd 服务。

### 5. 验证结果

等待 Speaker 滚动更新完成:

```bash
kubectl -n metallb-system rollout status daemonset/speaker
```

确认后端为 `native`,并且 Speaker 模板中不再包含 MetalLB FRR 容器:

```bash
kubectl -n metallb-system get metallb metallb \
-o jsonpath='{.spec.bgpBackend}{"\n"}'
kubectl -n metallb-system get daemonset speaker \
-o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
kubectl -n metallb-system get pods -l app=metallb,component=speaker -o wide
```

第一条命令必须返回 `native`。容器列表中不得包含 `frr`、`reloader`、`frr-metrics` 或 `metrics-auth-proxy-frr`。所有 Speaker Pod 都应处于 `Running` 和 `Ready` 状态。Pod 就绪不能单独证明 BGP 会话已经建立或 VIP 前缀已经宣告;请通过上游路由器或客户现有的网络监控工具,同时确认 MetalLB 会话和原有主机 FRR 会话。

`MetalLB` 资源会保存此设置。MetalLB 插件升级或重装后,请重新检查 `spec.bgpBackend`,因为资源重建或重置可能恢复默认后端。

## 诊断步骤

使用以下检查确定是否存在主机 FRR 与 MetalLB FRR 冲突:

```bash
kubectl -n metallb-system get metallb metallb \
-o jsonpath='{.spec.bgpBackend}{"\n"}'
kubectl -n metallb-system get daemonset speaker \
-o jsonpath='{.spec.template.spec.hostNetwork}{"\n"}{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}'
kubectl -n metallb-system get bgppeers,bgpadvertisements,ipaddresspools
```

在非 OpenShift 集群中,后端输出为空且 Speaker 模板中包含 `frr`,表示 MetalLB 默认 FRR 后端处于启用状态。`hostNetwork` 应为 `true`。将 BGP 对等体的源地址、Router ID、本地 AS 号和宣告前缀与主机 FRR 配置及上游路由器配置进行比较。

## 回滚

如果 Native 后端无法满足 BGP 要求,请恢复 FRR 后端:

```bash
kubectl -n metallb-system patch metallb metallb \
--type=merge \
-p '{"spec":{"bgpBackend":"frr"}}'
kubectl -n metallb-system rollout status daemonset/speaker
```

滚动更新完成后,确认所需的 FRR 容器已恢复,并验证 BGP 会话。原有主机 FRR 冲突未解决时,不要回滚到 `frr`。
Loading