Bootstrap kubeconfig does not include named serving certificate CAs

[amasolov]

Description

When a HostedCluster uses Route-based service publishing (servicePublishingStrategy.type: Route)
and the KAS serving certificate is signed by a non-HyperShift CA (e.g. a corporate PKI configured
via spec.configuration.apiServer.servingCerts.namedCertificates), worker nodes fail to connect to
the API server with:

tls: failed to verify certificate: x509: certificate signed by unknown authority

Root Cause

The worker node's kubelet kubeconfig (/var/lib/kubelet/kubeconfig) embeds only the HyperShift
root-ca as its certificate-authority-data. On Route-based clusters the local
kube-apiserver-proxy (HAProxy at 172.20.0.1:6443) does TCP passthrough to the KAS Route, so
the kubelet must verify the KAS TLS certificate directly.

spec.additionalTrustBundle correctly flows into the MCO user CA bundle on the node
(/etc/pki/ca-trust/source/anchors/), but the kubelet does not use the system trust store for API
server verification; it uses only the CA embedded in its kubeconfig.

The code path that builds the bootstrap kubeconfig (adaptBootstrapKubeconfigSecret in
control-plane-operator/.../v2/kas/kubeconfig.go) uses GenerateKubeConfig, which only fetches
root-ca. Compare with adaptCustomAdminKubeconfigSecret, which already calls
combineRootCAWithServingCerts to include named certificate CAs. The bootstrap kubeconfig does
not receive the same treatment.

Expected Behaviour

The bootstrap kubeconfig's certificate-authority-data should include:

1. The HyperShift root-ca
2. CAs from spec.configuration.apiServer.servingCerts.namedCertificates (consistent with the admin kubeconfig)

This would allow worker kubelets to verify KAS serving certificates signed by external CAs when
using Route-based service publishing.

Steps to Reproduce

1. Create a HostedCluster with servicePublishingStrategy.type: Route
2. Configure spec.configuration.apiServer.servingCerts.namedCertificates with a certificate
   signed by a non-HyperShift CA
3. Provision a worker node
4. Observe kubelet logs: tls: failed to verify certificate: x509: certificate signed by unknown authority

Workaround

A MachineConfig can be applied via the NodePool to inject a combined CA bundle (HyperShift root-ca +
external CA) into the kubelet kubeconfigs at boot time using a systemd oneshot service ordered
before kubelet.service. This is fragile and requires manual maintenance when CAs rotate.

Related

• Previous upstream issue: Unable to Register New Nodes in HostedCluster When Adding API Server Certificate #3985 (closed by bot inactivity, no code fix)
• Jira: OCPBUGS-41853 (resolved with docs-only fix in 4.19.0)
• KCS: 7111089
• Fix PR: OCPBUGS-84555: fix(cpo): include serving cert CAs in bootstrap and external admin kubeconfigs #8311

[amasolov]

Prior Art

This is a known issue that was previously tracked but never received a code fix:

• Upstream issue: Unable to Register New Nodes in HostedCluster When Adding API Server Certificate #3985 (closed by bot inactivity, no code fix merged)
• Jira: OCPBUGS-41853 (closed as "Done" in 4.19.0, but the resolution was documentation only via PR OCPBUGS-41853: Add documentation for API Server custom certificate in HCP #6001, not a code change)
• KCS: 7111089 ("When adding custom API server certificate to the Hosted Cluster, the bootstrap-kubeconfig is no longer trusted")

The documented workaround in KCS 7135847 (using kubeAPIServerDNSName + namedCertificates) generates a custom-admin-kubeconfig that includes the named cert CAs. However, this only fixes admin access. The bootstrap kubeconfig used by worker kubelets remains unpatched and only contains the HyperShift root-ca.

PR #8311 provides the actual code fix by extending adaptBootstrapKubeconfigSecret to call combineRootCAWithServingCerts, the same function already used by adaptCustomAdminKubeconfigSecret.