Declarative VMs on vSphere aren’t new. VM Service has let you describe a VM as a Kubernetes object since vSphere 7.0 Update 2a, and with the VCF Automation All Apps model the platform team sets the guardrails while the developer gets a Kubernetes API. Add Git and a reconciler, and you have GitOps for VMs, built from VCF components plus a Git repo. So I built the full loop in my VCF 9.1.1 lab: a namespace class in VCF Automation, VM manifests in GitHub, the Argo CD Supervisor Service reconciling them, and day-2 changes, drift and policy checks on top. Below is the setup, the tests, and what surprised me along the way.

Why VMs as code

Containers went declarative years ago. VMs mostly didn’t. Even in my own lab, most of them are still born from a wizard in the vSphere Client or from a PowerCLI script I keep forgetting to update. That works until you need to know why a VM looks the way it does, who changed it, or how to rebuild it exactly.

Describing a VM as a manifest in Git fixes that. Every change becomes a commit with an author and a reason, and a reconciler keeps the VM in line with what’s written down.

The lab

• VCF 9.1.1 with VCF Automation and an All Apps organization
• A namespace class with two VM classes (best-effort-small, best-effort-medium), one storage class and an Ubuntu 24.04 cloud image
• VM Service through the vmoperator.vmware.com/v1alpha5 API
• A public GitHub repository with the VM manifests
• The Argo CD Supervisor Service, with an Argo CD instance in the same namespace as the VMs
• A VCF Automation IaaS resource policy for the VM guardrails

Step 1: Guardrails first, in VCF Automation

Everything starts in the VCF Automation organization portal. As org admin, I created a namespace class under Manage & Govern → Namespace Class: a 10 GHz CPU and 10,000 MiB memory limit, two best-effort VM classes, and one storage class with a 100 GB limit. Then, as the lab user, I created a namespace from that class in the project. VCF Automation adds a random suffix to the name, so my namespace is kubevm-lab-hth89. Use the full name everywhere from here on.

As the lab user, everything after this is kubectl. The VCF CLI creates a context against VCF Automation with an API token, and each namespace you have access to shows up as its own context:

vcf context create kubevm-lab --endpoint https://[vcfa-fqdn] \
  --type cci --auth-type basic \
  --tenant-name [org-name] --api-token $API_TOKEN
vcf context use kubevm-lab:[namespace]:[project]
kubectl get virtualmachineclass
kubectl get vmi
kubectl api-resources | grep "virtualmachines "

Note the API group in the output: vmoperator.vmware.com/v1alpha5. That’s the VM Service API, and every manifest in this post uses it.

Step 2: The VM as code

A VM is two objects: a Secret with cloud-init user data (hostname, a user with my SSH public key, and nginx) and the VirtualMachine itself, which references that Secret. The VM manifest picks a VM class, an image and a storage class, all of which the namespace class allows. The app label is what the load balancer in Step 3 uses to find the VM:

apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachine
metadata:
  name: web01
  namespace: [namespace]
  labels:
    expires-on: "2026-11-30"
    app: web01
spec:
  className: best-effort-small
  imageName: [vmi-name]
  storageClass: [storage-class]
  powerState: PoweredOn
  bootstrap:
    cloudInit:
      rawCloudConfig:
        name: web01-cloudinit
        key: user-data

Replace the bracketed values with your own: the namespace name from VCF Automation, an image from kubectl get vmi and the storage class from your namespace class. My values are in the screenshot further down and in the lab repository.

Field names shift between API versions, so check yours with kubectl explain vm.spec --api-version=vmoperator.vmware.com/v1alpha5, or fill in the Create VM wizard in VCF Automation and copy the YAML it generates. Before committing, I validate the manifests with kubectl apply --dry-run=server -f vms/. The cloud-init Secret lives in Git too. That’s fine here, since it only holds a public SSH key and a package list. Anything really sensitive belongs in SOPS, Sealed Secrets or External Secrets instead.

Step 3: Letting Argo CD take over

I run Argo CD as the Argo CD Supervisor Service. It’s installed once on the Supervisor, and each namespace that needs one gets its own Argo CD instance inside the same guardrails as the VMs, so there’s no extra cluster and nothing running on my workstation.

The Argo CD Service on the Supervisor

The VI admin installs Argo CD Service 1.2.0 on the Supervisor from the vSphere Client, like any other Supervisor Service. Once it’s running, kubectl get argocdversion argocd-supported-versions -o yaml lists the Argo CD versions an instance can use, with 3.4.4 and 3.3.12 recommended.

I kept the default, Supervisor Service mode, on purpose. In that mode an Argo CD instance targets a namespace directly, which keeps every moving part of this walkthrough visible. The vcfaIntegrated install option switches the service to VCF Service mode, where destinations reference VCF Automation projects instead. That fits an All Apps organization naturally, and it’s a topic for another post.

An Argo CD instance in the namespace

Creating an instance is one small resource:

apiVersion: argocd-service.vsphere.vmware.com/v1alpha1
kind: ArgoCD
metadata:
  name: argocd
  namespace: [namespace]
spec:
  version: 3.4.4+vmware.1-vks.1

Shortly after, the Argo CD pods run in my namespace and argocd-server has a load balancer IP. Broadcom ships a matching argocd CLI alongside the service on the support portal.

Next, Argo CD needs to know where it may deploy. The usual argocd cluster add doesn’t fit here, because it creates a service account with cluster-wide rights, which a namespace user can’t grant, and shouldn’t. The service has its own answer, a ManagedEntity, which generates the destination for you:

apiVersion: argocd-service.vsphere.vmware.com/v1alpha1
kind: ManagedEntity
metadata:
  name: kubevm-lab
  namespace: [namespace]
spec:
  clusterName: kubevm-lab
  targetRef:
    apiGroup: ""
    kind: Namespace
    name: [namespace]

The ManagedEntity uses the service account argocd-k8s-sa, which has no rights in the namespace until you grant them. That’s by design, so you decide what Argo CD may touch. I bound it to a Role that only covers what this repo needs:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-manager-role
  namespace: [namespace]
rules:
  - apiGroups: ["vmoperator.vmware.com"]
    resources: ["virtualmachines", "virtualmachineservices"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-k8s-sa-binding
  namespace: [namespace]
subjects:
  - kind: ServiceAccount
    name: argocd-k8s-sa
    namespace: [namespace]
roleRef:
  kind: Role
  name: argocd-manager-role
  apiGroup: rbac.authorization.k8s.io

Apply the Role and RoleBinding before the ManagedEntity. Once its phase is Ready, Argo CD lists the namespace as a destination at https://kubernetes.default.svc/?context=[namespace], which is the server URL to use in the Application.

The Application itself lives in the same namespace as the Argo CD instance and points at the vms/ folder of the repo, with automated sync, pruning and self-healing:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kubevm-lab
  namespace: [namespace]
spec:
  project: default
  source:
    repoURL: https://github.com/rutgerblom/vcf9-vm-gitops-lab.git
    targetRevision: main
    path: vms
  destination:
    server: https://kubernetes.default.svc/?context=[namespace]
    namespace: [namespace]
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

After the first sync, the VM was created and powered on. Argo CD pulls the repo from inside the Supervisor, so the Supervisor needs outbound access to GitHub, not just your workstation. Be careful with pruning. With it on, removing a manifest from Git makes Argo CD delete the VirtualMachine object, and VM Service then deletes the VM. For lab VMs that come and go, that’s the intent. For VMs you care about, add the annotation argocd.argoproj.io/sync-options: Prune=false to the VM, or use sync windows.

Reaching the VM

The VM gets an IP on the namespace’s internal network, so nothing outside can reach it yet. Exposing it is one more manifest in vms/, a VirtualMachineService of type LoadBalancer that selects the VM by its app: web01 label.

apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachineService
metadata:
  name: web01-lb
  namespace: [namespace]
spec:
  type: LoadBalancer
  selector:
    app: web01
  ports:
    - name: ssh
      protocol: TCP
      port: 22
      targetPort: 22
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

Push, and Argo CD creates it. Within seconds it had an external IP, and both SSH and the nginx page answered. Two notes. In anything beyond a lab, use loadBalancerSourceRanges to limit who can connect (or leave SSH out). And Argo CD shows no health status for a VirtualMachineService unless you add a custom health check.

Step 4: Day 2, changing a VM through Git

Creating a VM from Git is one thing. I wanted to see how changes behave, both the ones that go through Git and the ones that don’t.

Resize through a pull request. web01 needs more memory, so I change className from best-effort-small to best-effort-medium in vms/web01.yaml, open a pull request and merge it. Argo CD picks up the commit, patches the VirtualMachine and reports Synced. The VM itself doesn’t change, though. kubectl describe vm web01 shows why:

Spec:
  Class Name:  best-effort-medium
Status:
  Class:
    Name:  best-effort-small
  Conditions:
    Reason:  ClassNameChanged
    Status:  False
    Type:    VirtualMachineClassConfigurationSynced
  Hardware:
    Cpu:     Total: 2
    Memory:  Total: 4096M

VM Service accepted the new class, but it doesn’t apply it to a VM that’s powered on. VirtualMachineClassConfigurationSynced stays False, and the VM keeps its 2 CPUs and 4 GB until the next power cycle. So I powered web01 off from the vSphere Client. Within seconds it was running again, because the VirtualMachine object still says powerState: PoweredOn and VM Service turned it back on. This time it came back with 8 GB (2 CPUs, as best-effort-medium defines). status.class now reads best-effort-medium and the condition is True.

Argo CD showed the app as Progressing while the VM was down, but it stayed Synced, because Git and the object never disagreed. So Synced means the object matches Git, not that the VM does. The real state is in the VM’s status conditions, and a resize needs a power cycle you plan for. The commit is still the change record, showing who resized the VM, when, and why.

Self-heal. Now a change that skips Git. I power the VM off with kubectl patch vm web01 --type merge -p '{"spec":{"powerState":"PoweredOff"}}'. With selfHeal: true, the change was undone before I could look at it. To show what Argo CD sees, I turned self-heal off for a moment and repeated the patch. In the screenshots below, web01 is OutOfSync against the last commit, and the diff is one line.

Then I turned self-heal back on. argocd app get kubevm-lab --show-operation shows a sync that took one second, against the same commit (d217931). No new commit means no new desired state, so this was self-heal putting powerState: PoweredOn back, and the VM’s message, web01 configured, is Argo CD re-applying the object from Git. A change that bypasses Git lasts seconds, so anything that should stick has to go through a pull request.

Step 5: Platform policy meets GitOps

The namespace class isn’t the only guardrail a platform team can set. It can also set rules for the VMs themselves, and the question for GitOps is whether those rules still hold when Argo CD, not a person, does the deploying. My example rule is that every VirtualMachine carries an expires-on label in YYYY-MM-DD format. Rather than bringing my own admission controller, I used what VCF Automation already has, IaaS resource policies.

As org admin, under Manage & Govern → Policies, I created an IaaS resource policy called require-vm-expiry and assigned it to my namespace. It starts from the built-in Label enforcement template, written in CEL. I only changed the validation:

failurePolicy: Fail
matchConstraints:
  resourceRules:
    - apiGroups:
        - vmoperator.vmware.com
      apiVersions:
        - "*"
      operations:
        - CREATE
        - UPDATE
      resources:
        - virtualmachines
validations:
  - expression: >-
      has(object.metadata.labels) &&
      'expires-on' in object.metadata.labels &&
      object.metadata.labels['expires-on'].matches('^[0-9]{4}-[0-9]{2}-[0-9]{2}$')
    message: Every VirtualMachine needs an expires-on label (YYYY-MM-DD)
validationActions:
  - Deny

The has() and in checks come first for a reason. Reading a label that isn’t there is an error in CEL, and the user would see that error instead of my message. The regex only checks the format. CEL has no clock, so the policy can’t compare the date to today and reject one that’s already passed. The label itself doesn’t do anything either. Nothing in VCF acts on it, so the VM keeps running after the date. Here it’s just a rule the platform team wants enforced, and acting on the date is a separate job.

VCF Automation turns the policy into a ValidatingAdmissionPolicy, plus a binding for my namespace. That means the Kubernetes API server enforces it, not just the VCF Automation UI. To test, I copied web01’s manifest to web02.yaml and left out the label. A server-side dry run is refused:

$ kubectl apply --dry-run=server -f vms/web02.yaml
The virtualmachines "web02" is invalid: : ValidatingAdmissionPolicy 'require-vm-expiry-7973d8616d' with binding 'require-vm-expiry-7973d8616d-kubevm-lab-hth89' denied request: Every VirtualMachine needs an expires-on label (YYYY-MM-DD)

Pushed to Git, the same rule stops Argo CD. The sync fails on web02, while web01, its Secret and the load balancer stay Synced:

Two things I noticed. The policy also matches UPDATE, so removing the label from a VM that already exists fails too. I found that by accident, when web02.yaml still said name: web01 and the dry run tried to update web01 without its label. And Argo CD doesn’t give up. It keeps retrying the sync with backoff (the operation was still Running on attempt 4 when I checked), while App Health stays Healthy, because everything that exists is healthy. Only Sync Status shows the problem, so that’s the one to alert on.

The fix goes through Git like any other change. I added expires-on to web02.yaml and pushed. No manual sync was needed. Argo CD picked up the new commit on its own, the sync took four seconds, the policy let web02 through, and VM Service created and powered it on. The app showed Progressing while the VM booted.

Verdict

• What worked: the whole loop, built only from VCF components and a Git repo. The namespace class sets the limits, the VMs live as manifests in Git, and Argo CD runs as a Supervisor Service that can only touch one namespace. Changes that skip Git are undone within seconds, and the IaaS resource policy is enforced by the API server, so it stops kubectl and Argo CD alike.
• Where it leaked: Synced means the object matches Git, not the VM. A resize waits for a power cycle, and a sync blocked by policy still shows the app as Healthy. On a smaller scale, argocd app diff failed in the CLI (the UI diff works), and a VirtualMachineService has no health status without a custom check.
• Use it today for: lab, dev and test VMs that come and go, where a commit is the change record and deleting a manifest is the cleanup. For long-lived VMs, I’d turn off pruning, plan resizes as maintenance, and alert on sync status, not just health.

Existing VMs aren’t covered here. Bringing them under VM Service comes with its own conditions, and I’ll leave that for a separate post.

Looking ahead: KubeVM

Everything in this post runs on VM Service as it ships in VCF 9.1 today. Broadcom has since announced KubeVM Service, which builds on the same foundation. VM Operator becomes the vSphere provider behind a vendor-neutral VirtualMachine API, proposed for the CNCF sandbox. The announcement also mentions adopting existing VMs and lifecycle policies. It isn’t available yet, so I’ll revisit this lab once it is. The workflow itself, with manifests in Git, pull requests as the change process and Argo CD reconciling, doesn’t depend on which VM API sits underneath, so I expect it to carry over.

References

• vcf9-vm-gitops-lab, the lab repository with all manifests used in this post
• Create a Namespace Class in VCF Automation, Broadcom TechDocs
• Using the Argo CD Supervisor Service, Broadcom TechDocs
• Configure IaaS Resource Policies in VCF Automation, Broadcom TechDocs