Our kubernetes cluster is more than a year old and I’m the only one left to maintain it. Several things happened almost all at once and things started to break one by one.
Versions
- kubernetes – 1.16.15-gke.4300
- kubectl client – v1.16.15
- kubectl server – v1.16.15-gke.4300
- jx – 2.1.150
- helm client – v2.17.0
- helm server – v2.17.0
Kubernetes Upgrade
Our Google Cloud Kubernetes cluster control plan has been automatically upgraded to 1.16.15-gke.4300 (previously 1.15.x) with notice months before. We know the upgrade was applied when we notice build errors referring to apiVersion: extensions/v1beta1.
The fix was to simply change it to: apiVersion: apps/v1 and we live quite life for a week or two.
We eventually manually upgraded our nodes to 1.16.15-gke.4300 too to match with the control plane.
Helm Charts Deprecation
The helm charts repository that we are using for mongodb and dex are being deprecated and was moved to a new location.
Github project: https://github.com/helm/charts
Blog announcement: https://helm.sh/blog/helm-2-becomes-unsupported/
We started getting errors in our preview environments that mongodb won’t load. The fix was to change the repository url:
From: https://kubernetes-charts.storage.googleapis.com
To: https://charts.helm.sh/stable
And we live quite for another week.
To be continued
We still need to migrate to helm version 3 at some point or even upgrade jenkins-x but this is too much for me to handle since I don’t really have the knowledge on this stuff.
I think I will just gradually learn this things moving forward. I hope we can still do deployments until we can fully fix things .
I currently have this issue that I don’t know yet how to fix:
error: failed to add chart repositories: failed to add the Helm repository with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io': failed to add the repository 'storage.googleapis.com' with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io': failed to run 'helm repo add storage.googleapis.com https://storage.googleapis.com/chartmuseum.jenkins-x.io' command in directory '.', output: 'Error: Couldn't load repositories file (/builder/home/.helm/repository/repositories.yaml)
God save our cluster!
Will this work?