Another month, another Jenkins-X and Kubernetes issue. Google cloud recently notified us that they are going to start automatic upgrades from Kubernetes v1.25 to v1.26. Wait what? We are not even on v1.25 yet and v1.25 will reach its end of life on February 2024!
We have encountered several issues with the changes to these two ecosystem and so far we’ve survived. This time, I upgraded to v1.25 without checking if we have incompatible configurations. Turns out Jenkins-X still uses the
CronJob api version
lighthouse. Updating the version stream doesn’t work yet since there are a lot of work involved affecting multiple components and these updates may not reach a stable version anytime soon.
Below are the problematic cron jobs:
Here are the list of deprecated API in Kubernetes v1.25
Although stable version fixes are not yet available officially, there are already fixes deployed to address the Kubernetes v1.25 issue. I have to do some manual update. I checked the helm chart repo index for newer versions:
I made some manual upgrades on the following helm charts on
The update indeed removes all references to
batch/v1beta1 but unfortunately, we are still getting the following error on
error: error retrieving RESTMappings to prune: invalid resource batch/v1beta1, Kind=CronJob, Namespaced=true: no matches for kind "CronJob" in version "batch/v1beta1"
It doesn’t make sense! We already removed all references to “batch/v1beta”.
Another Quick Fix
Because we really need to deploy our changes to our apps, we need to address the jx-boot issue. A temporary workaround is to just run the deployment manually against the latest
config-root from the main branch.
# Bulk apply - be very careful with this as this can wipe out your cluster cd /path/to/gitops-repo # Remove the dry run option to run it kubectl apply --prune -l=gitops.jenkins-x.io/pipeline=cluster -R -f config-root/cluster --dry-run=client; kubectl apply --prune -l=gitops.jenkins-x.io/pipeline=namespaces -R -f config-root/namespaces --dry-run=client;
It happily runs and deployed all our application changes. The “batch/v1beta1” issue must have only exists in our cluster so we waited for more updates from the Jenkins-X team.
We also updated tekton-pipeline but it doesn’t seem to fix the boot job issue. We kept the update anyway. We changed the following file:
tekton-pipeline to v0.6.0.
12 days later – jx-gitops plugin update
Recently, there was a fix on
jx-gitops plugin which addresses the issue we currently have. They fixed it by upgrading
kubectl so that it won’t use
I even asked how am I supposed to apply this fixes in our cluster. Due to time zone difference, I don’t expect an answer for at least 12 hours so I tried to find a away.
Based on this guide: https://jenkins-x.io/v3/about/how-it-works/#git-operator, I figured out I just need to manually update the following files:
We changed the jx-boot image version to the pre-release version
This update finally fixes our deployment issues and boot job is now able to finish. We will just monitor the version stream releases in case they release the stable versions.
Featured image by Kaique Rocha.