Proposal: Pre-validate haproxy.cfg before Secret update using a short-lived Job

[kenichi-mashiyama]

Hi !

To improve the safety and reliability of the Operator, I'm currently considering implementing the mechanism described below.
Would you be open to accepting a pull request for this once I have it ready?

If this approach doesn't align with the project's design direction, please feel free to reject the idea—no worries at all!

Configuration Validation Before Secret Update

Before the operator updates the runtime configuration Secret (<instance>-haproxy-config), it validates the generated haproxy.cfg using a short-lived Kubernetes Job:

1. Build the generated haproxy.cfg and all referenced certificate/config files.

2. Create a temporary validation Secret.

3. Start a short-lived Job using the same image as the target HAProxy instance (spec.image, fallback haproxy:latest) and run:

   haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg

4. If validation succeeds, update the runtime configuration Secret.

5. If validation fails, do not update the runtime configuration Secret, and set the Error status on the Instance and related configuration resources.

6. Clean up temporary validation resources (Job and Secret) automatically (via TTL) and explicitly by the operator.

Looking forward to hearing your thoughts!

[m-terra]

hi @kenichi-mashiyama

thanks for the suggestion. we do use https://github.com/snorwin/haproxy-reload-wrapper for this purpose.
please have a look.

andy

[kenichi-mashiyama]

Hi @m-terra

Thank you for sharing the link! I took a look at haproxy-reload-wrapper, and it looks like a good tool.

Combining my proposed pre-validation in the HAProxy Operator with haproxy-reload-wrapper actually provides two layers of protection across the configuration lifecycle:

1. At the Operator level: Before updating the runtime Secret, the Operator validates the generated haproxy.cfg using haproxy -c. If validation fails, the Secret remains untouched, ensuring the runtime configuration stays in a known-good state.
2. At the Pod level: Inside the HAProxy Pod, haproxy-reload-wrapper acts as a second safeguard. When the mounted configuration changes, it re-validates the config before launching a new HAProxy process. If validation or startup fails, the existing HAProxy process continues running and serving traffic without disruption.

This approach delivers true defense in depth:

• The Operator prevents invalid configurations from ever being persisted to the runtime Secret.
• The reload wrapper prevents invalid configurations from disrupting the active HAProxy process during a reload.

Together, they ensure end-to-end safety for both configuration state and runtime reloads.

To improve the safety and reliability of the Operator, I'm currently considering implementing the mechanism described above. Would you be open to accepting a pull request for this once I have it ready?

ken