GKE config connector vs deployment manager

Overview

Managing infrastructure as code has always been a challenge. Google cloud deployment manager and Hashicorp Terraform have been the top choices for creating reusable templates and modules to create the organizational folders, projects, and project resources. As you can see in this simple VPC example, deployment manager template is able to reference other resources with $(ref.Resource_Name.selfLink). Deployment manager has a great feature which GKE config connector lacks: reference a resource’s output by its properties. The reason that’s important is because in a deployment, a resource’s output is often another resource’s input. Using $(ref.RESOURCE.OUTPUT), you’d be able to reference a compute engine instance’s IP by defining the outputs:

outputs:
- name: ip
  value: f"$(ref.{vm-instance}.networkInterfaces[0].natIp)"

Pass $(ref.VM-instance-resource-name.ip) as a property to create a firewall rule that allows port 8080 to the IP. Deployment manager knows to create the VM first then the firewall later sequentially. You could also use the depends on syntax to order resource creation if deployment manager can’t figure out the resource creation order from the syntax.

GKE config connector

is surprisingly lacking the output features above. I had to create a folder, get the folder ID, then create a project in that folder. Passing the project ID to create a GCS bucket is not a problem because the project ID is defined in the yaml as metadata-name. I believe it’s optimized to create resources in a project given that the project has already been created in deployment manager or Terraform. I’m giving the examples below to demonstrate the problem.

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Folder
metadata:
  annotations:
    cnrm.cloud.google.com/organization-id: "000000000000"
  labels:
    use: "test"
  name: test-folder-gke-config-connector
spec:
  displayName: "GKE Config-Connector"
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
  name: iampolicy-folder
spec:
  resourceRef:
    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
    kind: Folder
    external: "folders/777777777777"
  bindings:
    - members:
        - serviceAccount:gke-config-connector@PROJECT_ID.iam.gserviceaccount.com
      role: roles/resourcemanager.folderIamAdmin
    - members:
        - serviceAccount:gke-config-connector@PROJECT_ID.iam.gserviceaccount.com
      role: roles/resourcemanager.folderAdmin
---
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
  annotations:
    cnrm.cloud.google.com/folder-id: "777777777777"
    cnrm.cloud.google.com/auto-create-network: "true"
  labels:
    use: "test"
  name: metadata-name
spec:
  name: "spec-name"
---
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/project-id : metadata-name
  name: gke-config-connector-hil-test
spec:
  location: us-central1
  lifecycleRule:
    - action:
        type: Delete
      condition:
        age: 4 # days

As you can see the 1st yaml section generates a folder in a organization. The 2nd yaml section creates a IAM policy binding in the folder. The problem is that the folder-id is unknown as you write the yaml file. There is no deployment manager equivalent way of coding $(ref.test-folder-gke-config-connector.outputs.id) to reference the prior resource’s output. Any kind of resource that generates output such as compute engine instance or Folder ID that can’t be acquired at code push time would need to be created first. Then the dependent resources would need to be created later. Such manual steps are not scalable when you need to replicate a series of resource creation across dev, test, prod environment folders in cloud build. Google cloud support told me that’s tracked in this github issue.

IAMPolicy in GKE config connector

One good use case of GKE config connector is GKE workload identity. Typically, the infrastructure engineer would need to execute Terraform code to create a Google service account and bind workload identity user role for the namespace/Kubernetes_service_account in a project. Then push a commit to cloud source repository to annotate the Kubernetes_service_account if GKE config sync is configured. GKE config connector has the sample code to achieve all the steps. I created another sample below for only binding the Workload identity user without creating service accounts. Replace ${SA}, ${PROJECT_ID}, ${KSA0}, ${KSA1}, ${NS0}, ${NS1}

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
  name: iampolicy-workloadidentity-test
  labels:
    use: workload-identity
spec:
  resourceRef:
    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    external: projects/${PROJECT_ID}/serviceAccounts/${SA}@${PROJECT_ID}.iam.gserviceaccount.com
  bindings:
    - role: roles/iam.workloadIdentityUser
      members:
        - serviceAccount:${PROJECT_ID}.svc.id.goog[${NS0}/${KSA0}]
        - serviceAccount:${PROJECT_ID}.svc.id.goog[${NS1}/${KSA1}]