Created attachment 1756017 [details]
apiserver pod

The workaround for this is to force a non-empty cloud config. This can be done at install time by updating the manifests/cloud-provider-config.yaml to include `config` data.

Example,
> manifests/cloud-provider-config.yaml

apiVersion: v1
data:
  config: |
    [Global]
  ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
<--snip-->
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cloud-provider-config
  namespace: openshift-config

Plan to look into this next sprint.

When we consider this bz, we should keep in mind that the behavior for cloud configs in aws will change in 4.9 with out-of-tree providers and CCM. We should confirm this is actually a bug in 4.9, or whether it just needs to be updated in 4.8 & 4.7.

Needs further review to determine action plan.

Will review for a future sprint.

Any updates on this related to OpenShift 4.9? I have this fix in my ansible for installing on C2S so I haven't distinguished between if it is still an issue in 4.9 or the fix is still working. I can remove the patch and test this in C2S if that's needed. Thank you.

(In reply to Dan Clark from comment #8)
> Any updates on this related to OpenShift 4.9? I have this fix in my ansible
> for installing on C2S so I haven't distinguished between if it is still an
> issue in 4.9 or the fix is still working. I can remove the patch and test
> this in C2S if that's needed. Thank you.

No update. We have not done any work on testing whether the patch is still needed.

It sounds like this might or might not be fixed in newer OpenShift versions.  Dan, if you want to test without that patch that would be a helpful datapoint.  You could test on 4.9, or wait a few weeks for 4.10.

OK, I'll test it by removing the step in the process where I add in the cloud-provider-config.yaml and see what happens.

I tested this in us-iso-east-1 by using the default cloud-provider-config.yaml created by the official openshift-install-4.9.17 binary with the same error result. 

kube apiserver pods still complain about this file missing:

/etc/kubernetes/static-pod-resources/configmaps/cloud-config/config

I believe the issue lies in the code here: pkg/asset/manifests/cloudproviderconfig.go

```
        case awstypes.Name:
                // Store the additional trust bundle in the ca-bundle.pem key if the cluster is being installed on a C2S region.
                trustBundle := installConfig.Config.AdditionalTrustBundle
                if trustBundle == "" || !awstypes.C2SRegions.Has(installConfig.Config.AWS.Region) {
                        return nil
                }
                cm.Data[cloudProviderConfigCABundleDataKey] = trustBundle
```

   For internet connected OpenShift cluster installs, the additionalTrustBundle in the install-config.yaml is probably empty which means the Generate function returns nil. I'm testing now but returning nil probably means the ConfigMap doesn't get created at all and the pods handle that. What they can't handle is the case where the CM gets created but only partially. The AWS case is the only case that can return the configMap with only the trustBundle filled out.

verified. PASS.
OCP version: 4.11.0-0.nightly-2022-02-07-154612

note:
in C2S region, installer will generate cloud-provider-config.yaml which includes .data.config and .data."ca-bundle.pem".

install succeeded without following workaround:
cat << EOF > ${manifests_dir}/cloud-provider-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-provider-config
  namespace: openshift-config
data:
  ca-bundle.pem: |
${CA_CONTENT}
  config: |
    [Global]
EOF

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069