I wanted to test Secret Store Service with a workload in my VMware Cloud Foundation 9.1.0 lab. The injector added the Vault Agent init container to the Pod, but its image could not be pulled into my VKS cluster. The workflow stopped before the agent could start, authenticate or retrieve a secret.

The application Pod stayed in Init:ImagePullBackOff. The failing container was vault-agent-init, which VCF had automatically injected into the Pod.

What follows is the path I took through that failure: first separating the image pull from the Secret Store configuration, then overriding the injected image reference so the same VMware image was pulled from the Regional Harbor depot already present in the VCF 9.1 environment.

Tracing the failure to the injected image path

The image reference for vault-agent-init used mgmt-image-proxy.kube-system.svc.cluster.local. The Pod event showed containerd timing out while trying to resolve that hostname through the node-local resolver at 127.0.0.53. Because the injector had already added the init container successfully, I could narrow the problem to the image-pull path rather than the injection process itself.

The resolver timeout showed that the request never reached the Management Proxy.

I tested name resolution from the VKS cluster. CoreDNS was running, and both kubernetes.default.svc.cluster.local and public names resolved normally. mgmt-image-proxy.kube-system.svc.cluster.local returned NXDOMAIN, while the worker’s node-local resolver timed out on the same lookup.

On the Supervisor, the mgmt-image-proxy Service was present and had ready EndpointSlices. This confirmed that the Service and its backends existed on the Supervisor, but it did not establish that the image route was healthy end to end.

I still wanted to rule out a general registry or worker-networking problem, so I deployed a plain nginx:stable-alpine Pod. Setting imagePullPolicy: Always forced the worker to contact the public registry instead of relying on a locally cached image.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pull-control
  namespace: demo
spec:
  restartPolicy: Never
  containers:
    - name: nginx
      image: docker.io/library/nginx:stable-alpine
      imagePullPolicy: Always

That Pod pulled the image and reached Running. The worker could resolve a public registry, establish HTTPS, download and unpack an image, and start a container. The failing path was specific to the image injected by the platform.

Overriding the injected image to use Regional Harbor

Starting with VCF 9.1, Supervisor Services that integrate with VCF Automation require Regional Harbor or an equivalent registry. As described in Broadcom KB 446991, installing Regional Harbor establishes the Supervisor Cluster Container Registry at depot.kube-system.svc.

Regional Harbor was already part of my lab and available to the VKS worker nodes.

A contact at Broadcom suggested using the vault.hashicorp.com/agent-image annotation to override the injected image reference. I pointed it to the Regional Harbor path while preserving the digest from the default injected image:

metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "secret-store-lab-fvtkz-secret-store-vks"
    vault.hashicorp.com/agent-inject-secret-lab-app.txt: "secret/data/secret-store-lab-fvtkz/secret-store-vks-lab-app-credential"
    vault.hashicorp.com/auth-path: "auth/kubernetes-secret-store-lab-fvtkz-secret-store-vks"
    vault.hashicorp.com/agent-image: "depot.kube-system.svc/vcf/vcf-service-secret-store/ga/9.1.0.0100/vcf-service-secret-store@sha256:c5f1a7f8c9f3dca295e39c0af92f00d38b29e48ea1ef1bffd66ce5058ddd9840"
spec:
  serviceAccountName: secret-lab
  containers:
    - name: secret-consumer
      image: docker.io/library/nginx:stable-alpine

The existing Vault role, authentication path and service account remained unchanged. Only the image reference moved from the unusable Management Proxy hostname to the existing Regional Harbor depot.

After deploying both versions, I checked the actual vault-agent-init image references in the resulting Pods and extracted their digests:

DEFAULT_IMAGE="$(kubectl -n demo get pod secret-consumer-default-route \
  -o jsonpath='{.spec.initContainers[?(@.name=="vault-agent-init")].image}')"
REGIONAL_IMAGE="$(kubectl -n demo get pod secret-consumer-working \
  -o jsonpath='{.spec.initContainers[?(@.name=="vault-agent-init")].image}')"
printf 'Management Proxy digest: %s\n' "${DEFAULT_IMAGE##*@}"
printf 'Regional Harbor digest:  %s\n' "${REGIONAL_IMAGE##*@}"

Management Proxy digest: sha256:c5f1a7f8c9f3dca295e39c0af92f00d38b29e48ea1ef1bffd66ce5058ddd9840
Regional Harbor digest:  sha256:c5f1a7f8c9f3dca295e39c0af92f00d38b29e48ea1ef1bffd66ce5058ddd9840

Both resulting Pod specifications referenced the same digest. The registry route changed; the image identity did not.

Clearing the image-pull failure exposed a separate certificate trust issue in the Secret Store configuration. That issue was unrelated to the registry path and is outside the scope of this article.

Verifying the image override

The image override cleared Init:ImagePullBackOff and allowed the injected Vault Agent to start. After I separately completed the required certificate trust configuration, the Pod reached Running with both the application and injected Vault Agent containers ready (2/2). Vault authentication succeeded, and I verified that the injected file was present without exposing its contents.

The result confirmed that the existing Regional Harbor depot provided a working route to the required VMware image and removed the blocker that had prevented the Secret Store workflow from starting. The image override did not repair the original Management Proxy and DNS integration; it bypassed that path.

I would not try to get around the original failure by forcing an untrusted Management Proxy package, adding arbitrary static DNS entries or substituting an unsupported Vault Agent image. Those changes move the environment outside the supported platform lifecycle without fixing the integration problem.

I plan to retest the default Secret Store injection behavior with a future VCF version. For now, this workaround applies only to the issue I observed with VCF 9.1.0.

References

• Broadcom KB 446991 — Cannot install VKS Cluster Management: 405 Method Not Allowed.
• Broadcom TechDocs: Inject secrets into Pods on VKS clusters.