When Terraform Plans Lie: Two GKE Issues and How to Catch Them

Here is a sprint that looks routine. A platform team runs four GKE clusters from Terraform. One is a production Autopilot cluster in us-central1 running stateful services that take a long time to rebalance after a node restart, so it carries a disruption budget: at least 45 days between minor upgrades and 3 days between patch upgrades. The other three are Standard private clusters built from the community module's beta-private-cluster-update-variant. CI posts a plan on every pull request, applies on merge, and a nightly job runs terraform plan -detailed-exitcode on all four stacks and pages someone if the plan is not empty.

Two changes go in that sprint. The production cluster moves to a recurring_maintenance_window, the newer window type with an explicit start time and duration that landed in the provider source in July 2026. The private clusters get enable_private_endpoint = true, so the control plane is reachable only from inside the VPC. Both plans look right. Both applies succeed.

Afterwards, the production cluster has quietly lost its disruption budget, and the three private clusters show the same one-line diff every night that no apply ever clears. That is roughly 90 drift pages a month (3 stacks, 30 nights) that mean nothing, plus one missing setting on the cluster that matters most.

This is a representative scenario, not a customer engagement, but both failures are real and open upstream: terraform-provider-google #30554 and terraform-google-kubernetes-engine #2697, both filed on 3 November 2026. Below we work through each one: what the reporter saw, what the source code says, how to reproduce it in a throwaway project, which workaround we would pick and what it costs, and the post-apply gate that catches both.

Bug 1: the budget the provider never sends

What the reporter saw

The report is against hashicorp/google v8.10.0 and Terraform v1.14.0, reproduced on a real Autopilot cluster. The steps: start with a cluster that has a disruption budget, confirm it in the maintenance policy, configure recurring_maintenance_window on the google_container_cluster resource, apply, and inspect the policy again. The budget is gone. The reporter's config sets both blocks:

resource "google_container_cluster" "example" {
  name             = "example-cluster"
  location         = "us-central1"
  enable_autopilot = true

  maintenance_policy {
    recurring_maintenance_window {
      window_start_time {
        hours   = 2
        minutes = 0
        seconds = 0
      }
      window_duration = "4h"
      recurrence      = "FREQ=DAILY"
    }

    disruption_budget {
      minor_version_disruption_interval = "3888000s" # 45 days
      patch_version_disruption_interval = "259200s"  # 3 days
    }
  }
}

The reporter traced it to two functions in the provider source and noted that the equivalent handling already exists for daily_maintenance_window and recurring_window. They did not post plan output, so what Terraform shows before and after the apply is not part of the report. That gap matters, and we come back to it below.

What the source says

We read resource_container_cluster.go.tmpl in magic-modules on main as of 5 November 2026, which is where the provider's container code is generated from. The reporter's diagnosis holds. expandMaintenancePolicy() builds a budget value once, then returns from one of three branches. The daily and recurring_window branches both set DisruptionBudget: budget. The recurring_maintenance_window branch builds its policy without it:

// recurring_window branch: budget included
return &container.MaintenancePolicy{
    Window: &container.MaintenanceWindow{ /* RecurringWindow ... */ },
    DisruptionBudget: budget,
    ResourceVersion:  resourceVersion,
}

// recurring_maintenance_window branch: budget computed above, never attached
policy := &container.MaintenancePolicy{
    Window: &container.MaintenanceWindow{ /* RecurringMaintenanceWindow ... */ },
    ResourceVersion: resourceVersion,
}

flattenMaintenancePolicy() mirrors it. It builds a budget list from the API response, adds "disruption_budget": budget to the daily and recurring_window results, and leaves it out of the recurring_maintenance_window result. The fix is two lines, one in each function.

One more detail explains why an existing budget disappears rather than just failing to update. Changes to maintenance_policy go through GKE's SetMaintenancePolicy call with the whole policy object. A policy sent without a budget replaces one that had a budget. GKE then falls back to its defaults, which the cluster disruption budget docs give as 24 hours between patch upgrades and 30 days between minor upgrades. In our scenario that means patch upgrades can land a day apart instead of three, and minor upgrades 30 days apart instead of 45.

Two cases, two symptoms

Whether the plan shows anything depends on where the budget lives. disruption_budget is Optional but not Computed in the schema. Read that against the flatten bug and you get two different symptoms:

The first case is the dangerous one: a setting someone added by hand, perhaps during an incident, is removed by an unrelated Terraform change and nothing says so. The second case is noisier but still misleading. The diff says Terraform is about to add the budget. The apply succeeds, and the budget still is not there.

Reproduce it, then confirm a fix

Do this in a throwaway project, not by experimenting on production. An Autopilot cluster is enough. Pin the provider to the version you run (8.10.0 in the report), and keep the reproduction to one resource so nothing else in your stack affects the result:

# 0. Baseline: daily window plus budget. This code path carries the budget.
terraform apply -auto-approve
gcloud container clusters describe repro --location us-central1 \
  --format='yaml(maintenancePolicy.disruptionBudget)'
# expect minorVersionDisruptionInterval: 3888000s, patchVersionDisruptionInterval: 259200s

# 1. Case A: swap daily_maintenance_window for recurring_maintenance_window, keep the budget block.
terraform apply -auto-approve
gcloud container clusters describe repro --location us-central1 \
  --format='yaml(maintenancePolicy.disruptionBudget)'
# bug present: empty output
terraform plan -detailed-exitcode; echo "exit=$?"
# bug present (by our reading): exit=2 with "+ disruption_budget"

# 2. Case B: delete the budget block from config, set it out of band, then change the window.
gcloud container clusters update repro --location us-central1 \
  --maintenance-minor-version-disruption-interval=3888000s \
  --maintenance-patch-version-disruption-interval=259200s
# edit window_duration from "4h" to "5h"
terraform apply -auto-approve
gcloud container clusters describe repro --location us-central1 \
  --format='yaml(maintenancePolicy.disruptionBudget)'
# bug present: empty output, and the plan above it showed only the window change

A fixed provider passes when step 1 shows the budget in gcloud and the follow-up plan exits 0. Case B will still remove an out-of-band budget even after the fix, because the policy is replaced as a whole. That is how SetMaintenancePolicy behaves, not a bug. If a setting matters, it belongs in config.

To test a patched provider before a release ships, build it from a branch and point Terraform at the binary with a dev_overrides block in your CLI config. Run the same script against it. That is fine for proving a fix. It is not how you should run production.

Workarounds, and what each one costs

In the scenario, the production cluster's weekend window becomes this, which keeps the budget in the code path that works:

maintenance_policy {
  recurring_window {
    start_time = "2026-11-07T02:00:00Z"
    end_time   = "2026-11-07T06:00:00Z"
    recurrence = "FREQ=WEEKLY;BYDAY=SA,SU"
  }

  disruption_budget {
    minor_version_disruption_interval = "3888000s" # 45 days
    patch_version_disruption_interval = "259200s"  # 3 days
  }
}

Two weekend windows of 4 hours give about 104 hours of maintenance availability in any 92-day period, comfortably above the 48 hours GKE requires, and each window meets the 4-hour minimum from the maintenance window docs. Switching window types sends a full SetMaintenancePolicy, so run the verification script below straight after this apply too.

Bug 2: the empty block that never lands

What the reporter saw

The report is against beta-private-cluster-update-variant v44.3.0 with hashicorp/google 7.35.0 and Terraform 1.14. The same condition is in v47.0.0 and master. With enable_private_endpoint = true and neither master_authorized_networks nor gcp_public_cidrs_access_enabled set, every plan shows:

  ~ resource "google_container_cluster" "primary" {
        ...
      + master_authorized_networks_config {}
    }

Applying changes nothing, and the next plan shows the same diff. The cause, from reading the module, is the for_each that decides whether to render the block:

dynamic "master_authorized_networks_config" {
  for_each = var.enable_private_endpoint || var.gcp_public_cidrs_access_enabled != null || length(var.master_authorized_networks) > 0 ? [true] : []
  content {
    gcp_public_cidrs_access_enabled = var.gcp_public_cidrs_access_enabled
    dynamic "cidr_blocks" { ... }
  }
}

The private endpoint flag alone makes the condition true, so an empty block goes out. For the affected cluster, gcloud ... --format='yaml(masterAuthorizedNetworksConfig)' returns nothing, so state never holds the block and each plan wants to add it. The reporter proposes dropping var.enable_private_endpoint || or rendering the block only when it carries a setting. They are careful about what they did not check: they do not know why the clause was added, they did not verify whether the provider or API also contributes to the diff, and they only read this one variant.

Filling in what the report left open

We followed up on those three open points on 5 November 2026.

• Other variants: the condition comes from the shared template autogen/main/cluster.tf.tmpl, which adds var.enable_private_endpoint || only for private clusters. On master it is present in private-cluster, private-cluster-update-variant, beta-private-cluster, beta-private-cluster-update-variant and beta-autopilot-private-cluster. The root module and beta-public-cluster do not have it.
• Why the clause exists: it arrived in #2058, merged in August 2024, titled "allow enable_private_endpoint with no master_authorized_networks". The PR description does not say what failed before it. So dropping the clause could regress whatever #2058 fixed. A fork needs to test cluster creation from scratch, not only plans on existing clusters.
• Related work: open PR #2687 stops rendering the block when ip_endpoints_enabled = false (DNS-only control planes), following the same symptom in #2460. By its own table it leaves the default ip_endpoints_enabled = null case unchanged, so it would not fix the configuration in #2697.
• The provider side: in the provider, an empty block expands to an authorized networks config with Enabled: true and no CIDR blocks. On read, flatten returns nothing when the API's config is absent or disabled. The provider reads it from controlPlaneEndpointsConfig.ipEndpointsConfig.authorizedNetworksConfig, while the reporter checked the top-level masterAuthorizedNetworksConfig. If the API does not keep enabled: true with zero CIDRs for this cluster shape, the result is the diff the reporter saw. We have not confirmed which layer drops it. The raw-resource reproduction below is how you would tell.

Reproduce it, and isolate the layer

Two runs separate the module from the provider and API. First, the raw resource, with no module involved:

resource "google_container_cluster" "repro" {
  name               = "mancfg-repro"
  location           = "us-central1"
  initial_node_count = 1
  # network, subnetwork and ip_allocation_policy as in your environment

  private_cluster_config {
    enable_private_nodes    = true
    enable_private_endpoint = true
    master_ipv4_cidr_block  = "172.16.0.32/28"
  }

  master_authorized_networks_config {}
}

Apply, then run terraform plan -detailed-exitcode from a host inside the VPC. If it exits 2 with + master_authorized_networks_config {}, the empty block does not survive the provider and API round trip on its own, and any module that emits it will never converge. If it exits 0, the problem is specific to how the module builds the block. Either way, also dump both API paths with gcloud ... --format=json | jq '{top: .masterAuthorizedNetworksConfig, nested: .controlPlaneEndpointsConfig.ipEndpointsConfig.authorizedNetworksConfig}' so you know what GKE actually stored.

Second, the module run: the reporter's module block at v44.3.0, then the same block pointed at a vendored copy with the clause removed. The fix passes when the vendored copy plans to exit 0 on the existing cluster and creates a fresh cluster cleanly, which is the #2058 regression check.

Workarounds, and what each one costs

Because master_authorized_networks_config is Optional and Computed in the provider schema, removing the block from config produces no diff even if the cluster has authorized networks enabled. Terraform simply stops managing it. That makes the fork safe to roll out, but it also means Terraform will not notice if someone later enables authorized networks by hand. If that matters, set the networks explicitly instead.

In the scenario we would vendor the module with the clause removed, since the team does not want authorized networks on a VPC-only endpoint. We would pin the vendored copy by commit, link it to #2697, and add a check that fails when the upstream release containing a fix is available, so the fork does not outlive its reason.

Why this class of bug exists

Both bugs live in the round trip between three representations: your HCL config, the API request and response, and Terraform state. Providers built on the Terraform plugin SDK convert between them with hand-written expand functions (config to API) and flatten functions (API to state). terraform plan only compares config with state. If a value is lost on the way out, or an empty value cannot come back, the plan stops describing the cluster.

For a budget set outside Terraform, which is the reporter's reproduction, and for the empty authorized networks block, the round trip looks like this:

Four mechanics produce almost every case we have seen of this class:

1. Branchy expand and flatten. When a block has mutually exclusive variants (ExactlyOneOf across daily_maintenance_window, recurring_window and recurring_maintenance_window), each variant gets its own branch, and shared siblings like disruption_budget have to be wired into every one. A newly added branch is where a sibling gets missed.
2. Whole-object update calls. SetMaintenancePolicy replaces the policy. Anything the request omits is deleted, including settings Terraform was never told about.
3. Empty nested blocks. An empty block is present in HCL, but the Go API clients omit zero values unless a field is listed in ForceSendFields, and an API may normalise a config with nothing in it back to absent. The provider authors know this: a comment in expandMasterAuthorizedNetworksConfig notes that "the GKE API pays attention to the field presence". Presence that the API does not keep cannot round-trip.
4. Schema flags that decide visibility. Optional without Computed means state lacking a configured value shows as a diff. Optional plus Computed means omitting a block is silent, so Terraform stops owning it. Neither flag helps when flatten drops the value: the plan is either always dirty or always clean, and neither matches the cluster.

The post-apply gate

The guard is not more careful plan review. It is two independent checks run straight after every apply: read the live resource through a path that does not use the provider, and require a second plan to be empty. This script does both for the scenario's production cluster:

#!/usr/bin/env bash
# verify-gke.sh CLUSTER LOCATION WANT_MINOR WANT_PATCH
# Run in the Terraform working directory, straight after apply.
set -euo pipefail
CLUSTER=$1 LOCATION=$2 WANT_MINOR=$3 WANT_PATCH=$4
fail=0

desc=$(gcloud container clusters describe "$CLUSTER" --location "$LOCATION" --format=json)

# 1. Live disruption budget, read from the API, not from Terraform state.
minor=$(jq -r '.maintenancePolicy.disruptionBudget.minorVersionDisruptionInterval // "unset"' <<<"$desc")
patch=$(jq -r '.maintenancePolicy.disruptionBudget.patchVersionDisruptionInterval // "unset"' <<<"$desc")
[[ "$minor" == "$WANT_MINOR" ]] || { echo "FAIL minor interval: want $WANT_MINOR, live $minor"; fail=1; }
[[ "$patch" == "$WANT_PATCH" ]] || { echo "FAIL patch interval: want $WANT_PATCH, live $patch"; fail=1; }

# 2. Authorized networks at both API paths, for the record.
jq -c '{top: .masterAuthorizedNetworksConfig,
        nested: .controlPlaneEndpointsConfig.ipEndpointsConfig.authorizedNetworksConfig}' <<<"$desc"

# 3. Convergence: a second plan must be empty.
set +e
terraform plan -detailed-exitcode -input=false -out=recheck.tfplan >/dev/null
rc=$?
set -e
case $rc in
  0) echo "OK plan converged" ;;
  2) echo "FAIL plan did not converge:"
     terraform show -json recheck.tfplan | jq -r '
       .resource_changes[]
       | select(.change.actions == ["update"])
       | .change as $c
       | .address + ": " + ([$c.after | keys[]
           | select($c.before[.] != $c.after[.] and ($c.after_unknown[.] // false) != true)]
           | join(", "))'
     fail=1 ;;
  *) echo "ERROR plan failed"; fail=1 ;;
esac
exit $fail

Run as ./verify-gke.sh prod-autopilot us-central1 3888000s 259200s, a cluster hit by Case A or Case B fails on step 1, whatever the plan says. A private cluster hit by #2697 fails on step 3 and prints module.gke.google_container_cluster.primary: master_authorized_networks_config, which names the culprit instead of dumping the whole plan. Compare duration strings in the form GKE returns them; if your config uses a different form, normalise before comparing.

If you would rather keep the check in Terraform, a check block works, as long as it reads the API directly with the http data source and not through google_container_cluster:

data "google_client_config" "current" {}

check "gke_disruption_budget" {
  data "http" "cluster" {
    url = "https://container.googleapis.com/v1/projects/my-project/locations/us-central1/clusters/prod-autopilot"
    request_headers = {
      Authorization = "Bearer ${data.google_client_config.current.access_token}"
    }
  }

  assert {
    condition = try(
      jsondecode(data.http.cluster.response_body).maintenancePolicy.disruptionBudget.minorVersionDisruptionInterval,
      "unset"
    ) == "3888000s"
    error_message = "Live cluster has no 45-day minor disruption budget. See hashicorp/terraform-provider-google#30554."
  }
}

The tradeoff is in how Terraform treats checks. A failed assertion, and even an error in the scoped data source, is reported as a warning and does not stop the plan or apply. That is good for visibility, because the warning sits in every PR plan where reviewers will see it. It is not a gate. Keep the script as the step that fails the pipeline, and use the check block so humans see the problem in context.

Around the gate, three habits close the remaining gaps:

• Upgrade providers and modules on a copy first. Apply to a non-production cluster, run the gate, and diff the gcloud JSON before and after, not only the plan.
• Write down every exception. Anything set outside Terraform, and every forked module, gets a comment in code and a ticket linked to the upstream issue, so it is removed once a fix ships.
• Treat a recurring diff as an incident, not noise. In the scenario, 90 meaningless pages a month is how the next real diff gets ignored.

Spotting the same class in other resources

Neither bug is specific to GKE. When you adopt a new resource, a new block type or a new module, these are the questions we ask, roughly in order of how often they find something:

1. Does a second plan right after the first apply exit 0? It is the cheapest test there is, and it catches every non-converging case.
2. Does the block have mutually exclusive variants? Open the provider source and check that every shared sibling field appears in every branch of expand and flatten. A quick grep -n for the sibling's API field name, counted against the number of branches, finds gaps like #30554 in minutes.
3. Is the update a whole-object replace? Look for Set*Policy, set*Config or a full PUT. If so, anything you set outside Terraform will be deleted by the next unrelated change.
4. Can the config produce an empty nested block? Check every dynamic block whose for_each depends on a flag that is not one of the block's own attributes, like enable_private_endpoint here.
5. Is the block Optional and Computed? If so, omitting it means Terraform does not own it. Decide whether that is what you want.
6. Was the field or variant added recently? New branches have the least test coverage. recurring_maintenance_window was about four months old when #30554 was filed. Read the expand and flatten of anything that new before you rely on it.
7. Can you read the live value without going through the provider? If not, you cannot verify the setting. Find a gcloud, REST or http path before you depend on it.

How MigrateX handles this

MigrateX is our platform and service for moving software, data and infrastructure from one technology to another. It combines automation with experienced engineers. The automation does the repetitive work quickly, and the engineers make the tricky decisions.

Infrastructure moves are where bugs like these do the most damage. When you move EKS clusters to GKE, upgrade a major module version, or rewrite hand-built infrastructure in Terraform, hundreds of settings change at once. A plan that is quietly wrong, or one that is always noisy, is what lets a lost setting slip through. The gate above is the core of how MigrateX deals with it:

1. Record what the cluster really does. Before we change anything, we capture each cluster's live settings from the cloud API: maintenance windows and budgets, endpoints, authorized networks, node pools and IAM. That is the baseline the new setup has to match, not the old Terraform code.
2. Check the round trip. After each apply, we read the live resources back through the API and compare them field by field with the baseline. A missing disruption budget shows up as a failed check, even when Terraform reports success.
3. Require a clean second plan. Every workload must reach an empty plan before it moves on. A diff that never goes away, like the empty master_authorized_networks_config, blocks the step until it is fixed or recorded as a known exception.
4. Track known issues. When a provider or module bug affects a setting, we record the version, the workaround and the upstream issue. When the fix ships, we upgrade, remove the workaround and run the gate again.
5. Flag what needs a person. Automation finds the mismatch. Our engineers decide whether it is a bug, an intended change or a gap in the baseline, and agree it with your team before the cluster carries traffic.

What you get along the way:

• No silent changes. If a setting your cluster relies on goes missing, a check fails before production finds out.
• Plans you can trust again. Clean plans mean real changes stand out in review.
• Provider upgrades without surprises. New versions are tested against the live baseline, not just the plan output.
• A clear record. Every workaround is tied to an upstream issue and removed once it is no longer needed.