[thomas-mattsson]

Adds IBM Bob on-premises support: Ansible roles for install/uninstall/air-gap, cert-manager integration, Openshift AI installation and supported model inferencing configuratoin, smoke tests, sample configs, docs, and a CI workflow.

[fketelaars]

Findings:

• When deploying on a TZ cluster, which has a ca-signed certificate, the Bob endpoints have self-signed certificates
• If bob[].state is set to removed and deployer is run, no check for confirm_destroy is performed
• Bob is removed correctly, but the ibm-bob-models namespace remains after destroy
• Ideally the sample configurations should be less verbose and the different settings should go into the documentation. Consider creating multiple configs, one for in-cluster models, one for gateways, or simplify the config file.
• I got a IO Error: No space left on device (os error 28) as part of hf download. Can we avoid using local disk if --skip-portable-registry is used?

[thomas-mattsson]

@fketelaars, last commit has the latest changes, re your comments:

When deploying on a TZ cluster, which has a ca-signed certificate, the Bob endpoints have self-signed certificates

Bob has the project name as a sub-domain (like bob.<project>.<apps-domain>) so the default ingress certificate can't be reused (plus there's the rotation issue). I've added support for referring to a ClusterIssuer by name or creating let's encrypt certs with http-01 challenges.

If bob[].state is set to removed and deployer is run, no check for confirm_destroy is performed

Fixed

Bob is removed correctly, but the ibm-bob-models namespace remains after destroy

This was by design to not delete the large model weights PVCs, but I changed this so that the PVCs and the namespace is deleted by default now, but the previous behavior can be kept by adding remove_model_weights: false.

Ideally the sample configurations should be less verbose and the different settings should go into the documentation. Consider creating multiple configs, one for in-cluster models, one for gateways, or simplify the config file.

Cleaned them up and added a reference to the documentation instead

I got a IO Error: No space left on device (os error 28) as part of hf download. Can we avoid using local disk if --skip-portable-registry is used?

This would only happen with env download for air-gapped mode so no way to avoid it. When on a connected cluster a job will download directly into the PVC. I've added support for the nvidia nemotron super now as well which is smaller to test with. And this only happens when using openshift_ai mode, a custom model gateway config would jump over this step.

Besides the review comments, I've also changed the integration tests and added a test harness with cp-deploy.sh test. For Bob it runs a full test against a cluster with connected and both air-gapped modes. I have a cp4d version on a separate branch that I will test later once this one is done.

Also made some smaller updates and added support to the nvidia nemotron super model that now is supported.