Another month, another Jenkins-X and Kubernetes issue. Google cloud recently notified us that they are going to start automatic upgrades from Kubernetes v1.25 to v1.26. Wait what? We are not even on v1.25 yet and v1.25 will reach its end of life on February 2024!
We have encountered several issues with the changes to these two ecosystem and so far we’ve survived. This time, I upgraded to v1.25 without checking if we have incompatible configurations. Turns out Jenkins-X still uses the CronJob api version batch/v1beta on jx-preview and lighthouse. Updating the version stream doesn’t work yet since there are a lot of work involved affecting multiple components and these updates may not reach a stable version anytime soon.
Below are the problematic cron jobs:
- jx-preview-gc-jobs
- lighthouse-gc-jobs
Here are the list of deprecated API in Kubernetes v1.25
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-25
Initial Fix
Although stable version fixes are not yet available officially, there are already fixes deployed to address the Kubernetes v1.25 issue. I have to do some manual update. I checked the helm chart repo index for newer versions:
https://jenkins-x-charts.github.io/repo/index.yaml
I made some manual upgrades on the following helm charts on helmfiles/jx/helmfiles.yaml:
jx-preview– v0.1.0lighthouse– v1.12.1 – pre-release as of this writing
The update indeed removes all references to batch/v1beta1 but unfortunately, we are still getting the following error on jx-boot:
error: error retrieving RESTMappings to prune: invalid resource batch/v1beta1, Kind=CronJob, Namespaced=true: no matches for kind "CronJob" in version "batch/v1beta1"
It doesn’t make sense! We already removed all references to “batch/v1beta”.
Another Quick Fix
Because we really need to deploy our changes to our apps, we need to address the jx-boot issue. A temporary workaround is to just run the deployment manually against the latest config-root from the main branch.
# Bulk apply - be very careful with this as this can wipe out your cluster
cd /path/to/gitops-repo
# Remove the dry run option to run it
kubectl apply --prune -l=gitops.jenkins-x.io/pipeline=cluster -R -f config-root/cluster --dry-run=client;
kubectl apply --prune -l=gitops.jenkins-x.io/pipeline=namespaces -R -f config-root/namespaces --dry-run=client;
It happily runs and deployed all our application changes. The “batch/v1beta1” issue must have only exists in our cluster so we waited for more updates from the Jenkins-X team.
Tekton Update
We also updated tekton-pipeline but it doesn’t seem to fix the boot job issue. We kept the update anyway. We changed the following file:
helmfiles/tekton-pipelines/helmfile.yaml
and upgrade tekton-pipeline to v0.6.0.
12 days later – jx-gitops plugin update
Recently, there was a fix on jx-gitops plugin which addresses the issue we currently have. They fixed it by upgrading kubectl so that it won’t use batch/v1beta anymore.
- https://github.com/jenkins-x-plugins/jx-gitops/pull/932
- https://github.com/jenkins-x-plugins/jx-gitops/releases/tag/v0.14.9
- https://github.com/jenkins-x/jx/releases/tag/v3.10.86
I even asked how am I supposed to apply this fixes in our cluster. Due to time zone difference, I don’t expect an answer for at least 12 hours so I tried to find a away.
Based on this guide: https://jenkins-x.io/v3/about/how-it-works/#git-operator, I figured out I just need to manually update the following files:
versionStream/git-operator/job-azurekeyvault.yamlversionStream/git-operator/job-gsm.yamlversionStream/git-operator/job-vault.yamlversionStream/git-operator/job.yaml
We changed the jx-boot image version to the pre-release version 3.10.86.
This update finally fixes our deployment issues and boot job is now able to finish. We will just monitor the version stream releases in case they release the stable versions.
Featured image by Kaique Rocha.