Creating Google Storage Buckets with Config Connector

We are trying to build an API to store files for user profile images. To do that in our current infrastructure, we need to use Google Cloud Storage. Since we are already using Kubernetes, we can automate the process by defining the buckets what we are about to use, configure it and make it available to our application.

What is Config Connector?

Config Connector is a Kubernetes addon that allows you to manage Google Cloud resources through Kubernetes. Therefore, instead of manually creating them in Console, we can just define these resources as part of the deployment process in a nice and handy YAML file.

Discovering available Google Cloud resources

We are only interested with Google Cloud Storage right now. We already have a working Google Pub/Sub configuration that uses Config Connector so it must be easy to add another type of resource. Wrong! We need to get familiar with the resource we are trying to configure before we can proceed.

List resources and filter the resources that we are interested in because there are so many of them.

kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true | grep storage

You should get something like this:

storagebucketaccesscontrols.storage.cnrm.cloud.google.com 2019-12-25T14:34:52Z
storagebuckets.storage.cnrm.cloud.google.com 2019-12-25T14:34:53Z
storagedefaultobjectaccesscontrols.storage.cnrm.cloud.google.com 2019-12-25T14:34:53Z
storagenotifications.storage.cnrm.cloud.google.com 2019-12-25T14:34:54Z

Next, desribe the resource, ie: the storagebuckets.resource.

kubectl describe crd storagebuckets.storage.cnrm.cloud.google.com

The output is a large YAML document. Just read it to familiarize with it. Take note of the the spec versions as it is important. In my case, I got the v1alpha2 version. This is important because we need to match it in our YAML document.

Describe the buckets

We already have a nice example here: https://cloud.google.com/config-connector/docs/reference/resource-docs/storage/storagebucket

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/force-destroy: "false"
  labels:
    label-one: "value-one"
  # StorageBucket names must be globally unique. Replace ${PROJECT_ID?} with your project ID.
  name: ${PROJECT_ID?}-sample
spec:
  lifecycleRule:
    - action:
        type: Delete
      condition:
        age: 7
  versioning:
    enabled: true
  cors:
    - origin: ["http://example.appspot.com"]
      responseHeader: ["Content-Type"]
      method: ["GET", "HEAD", "DELETE"]
      maxAgeSeconds: 3600

I only need a simple private bucket with no CORS configration and no complex permissions. Here is my configuration.

charts/app-name/templates/buckets.yaml:

apiVersion: storage.cnrm.cloud.google.com/v1alpha2
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/force-destroy: "false"
  name: my-unique-bucket-name
spec:
  versioning:
    enabled: true
  location: "US"
  storageClass: "STANDARD"
---
apiVersion: storage.cnrm.cloud.google.com/v1alpha2
kind: StorageBucket
metadata:
  annotations:
    cnrm.cloud.google.com/force-destroy: "false"
  name: my-other-unique-bucket-name
spec:
  versioning:
    enabled: true
  location: "US"
  storageClass: "STANDARD"

Notice that I changed the version from v1beta1 to v1alpha2. This is to match the resource with the current working version for the API.

Apply the configuration

To manually apply the configuration, simply run:

kubectl --namespace $(gcloud config get-value project) apply -f charts/app-name/templates/buckets.yaml

Delete the resource managed by Config Connector

It is worth noting that you can delete the buckets in Google Cloud Console. However, the bucket will get recreated since the configuration ensures that what is defined gets maintained. Therefore, the only way to permanently delete these managed buckets is to delete the configuration. It is important to keep the YAML file intact and unchanged so that you can properly remove the configuration.

To remove the resource, run the following:

kubectl --namespace $(gcloud config get-value project) delete -f charts/app-name/templates/buckets.yaml

I made a mistake while doing this and I ended up having those unwanted buckets and I have to recreate the previous YAML file just to remove the resources permanently.

Run Config Connector as part of Jenkins-X deployment

Instead of running kubectl apply manually, we added the step during the promote stage on pullRequest pipeline. This ensures that the resource is available to preview environments and even staging and production.

jenkins-x.yaml:

buildPack: typescript
pipelineConfig:
  pipelines:
    overrides:
      - pipeline: pullRequest
        stage: promote
        name: make-preview
        steps:
          - command: kubectl --namespace $(gcloud config get-value project) apply -f charts/${REPO_NAME}/templates/pubsub.yaml
            dir: /workspace/source
            image: google/cloud-sdk:latest
            name: apply-config-connector-pubsub
        type: before

What it does is to apply the configuration during creation of preview for PRs. This can be improved though but this just demonstrates how we can integrate Config Connector with Jenkins-X pipeline. We should probably add it on a different pipeline but this is it for now.

Resources: