Kubernetes, Jenkins-X and Helm Madness part 2

This post is a continuation of my journey on kubernetes, jenkins-x and helm updates and deprecation issues. I’ll be focusing on the following error:

error: failed to add chart repositories: failed to add the Helm repository with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io': failed to add the repository 'storage.googleapis.com' with URL 'https://storage.googleapis.com/chartmuseum.jenkins-x.io': failed to run 'helm repo add storage.googleapis.com https://storage.googleapis.com/chartmuseum.jenkins-x.io' command in directory '.', output: 'Error: Couldn't load repositories file (/builder/home/.helm/repository/repositories.yaml)

Versions

  • kubernetes – 1.16.15-gke.4300
  • kubectl client – v1.16.15
  • kubectl server – v1.16.15-gke.4300
  • jx – 2.1.150
  • helm client – v2.17.0
  • helm server – v2.17.0

Upgrading Helm

Our helm version was not v2.17.0 from the start. It was a bit lower. Based on the previous annoumcement, helm v2 users must upgrade to v2.17.0 which was created to handle the transition to the new stable repo url.

https://github.com/helm/helm/releases/tag/v2.17.0

To upgrade helm to v2.17.0, we just have to download the executable as specified in the URL above. This should upgrade (or downgrade) the helm client (local helm version).

To upgrade the server version, run the following command:

helm init --upgrade

To verify if we have installed the correct version, run helm version

> helm version
Client: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.17.0", GitCommit:"a690bad98af45b015bd3da1a41f6218b1a451dbe", GitTreeState:"clean"}

This step is required so that we can apply our patch/overrides to helm during the build processes.

Make Preview

The error occurred during the make preview step: step-promote-make-preview and the error is specified above. I searched around and tried some suggestions but they didn’t work. I see another error message that gave me some hint.

You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)

What I did was add a command on Makefile for the preview environment for the helm init tweak as suggested.

File: charts/preview/Makefile – added a line above jx step helm build

helm init --client-only --stable-repo-url https://charts.helm.sh/stable
jx step helm build

It works and the error goes away.

File: charts/app-name/Makefile – this file also contains a helm init command. I applied the same fix just in case it will throw and error in the future. This is under the release: clean section.

helm init --client-only --stable-repo-url https://charts.helm.sh/stable

Helm Release

Now that preview pushes through, I now encountered the error on helm release step on promote stage. I may need to add the helm init again but it looks like I have to create a pipeline override.

This is the original format based on our TypeScript/JavaScript build pack.

https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes/blob/master/packs/javascript/pipeline.yaml

In my project file jenkins-x.yaml, I have this overrides:

buildPack: typescript
pipelineConfig:
  pipelines:
    overrides:
      - pipeline: release
        stage: promote
        name: helm-release
        steps:
          - comment: release the helm chart
            sh: helm init --client-only --stable-repo-url https://charts.helm.sh/stable && jx step helm release
            name: helm-release
        type: replace
      - pipeline: release
        stage: promote
        name: jx-promote
        steps:
          - comment: promote through all 'Auto' promotion Environments
            sh: helm init --client-only --stable-repo-url https://charts.helm.sh/stable && jx promote -b --all-auto --timeout 1h --version \$(cat ../../VERSION)
            name: jx-promote
        type: replace

I added an override on both the helm-release and jx-promote step as both of they throw that helm error.

Staging Environment Helm Build

As the application is being promoted to staging, I, once again, encountered the helm error at the step: step-build-helm-build. This is where a pull request is created in staging environment so that the new application version can be deployed to staging.

There is a Makefile where a helm init is also present. I modified it again so I add the stable repo url.

helm init --stable-repo-url https://charts.helm.sh/stable

However, the patch on Makefile does not fix the issue. Maybe it is for another step. I kept the changes but look for other places where I might find the solution.

Looking at the step-build-helm-build, I can’t seem to find anything in our current staging environment repo. Is there an override for environment build pack?

Looks like the answer is yes.

https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes/blob/master/packs/environment/pipeline.yaml

I modified the environment’s jenkins-x.yaml file to add the override.

env:
- name: DEPLOY_NAMESPACE
  value: jx-staging
pipelineConfig:
  env:
  - name: DEPLOY_NAMESPACE
    value: jx-staging
  pipelines:
    overrides:
      - pipeline: release
        stage: build
        name: helm-apply
        steps:
          - sh: helm init --client-only --stable-repo-url https://charts.helm.sh/stable && jx step helm apply
            name: helm-apply
        type: replace
      - pipeline: pullRequest
        stage: build
        name: helm-build
        steps:
          - sh: helm init --client-only --stable-repo-url https://charts.helm.sh/stable && jx step helm build
            name: helm-build
        type: replace

Note: I also added an override to helm-build-step-apply as it throws the same error once the pull request is approved.

After all these changes, our build process from preview to staging went through smoothly as expected.

Development Environment Helm Build

I made some changes to the development environment too or the jx-boot-config repository for our development environment. I’m not really sure if it made any difference but I applied these changes earlier.

File: env/Makefile

init:
	helm init --client-only --stable-repo-url https://charts.helm.sh/stable
	helm repo add jenkins-x https://storage.googleapis.com/chartmuseum.jenkins-x.io
	helm repo add bitnami https://charts.bitnami.com/bitnami
	helm repo add stable https://charts.helm.sh/stable

build: clean init
	helm init --client-only --stable-repo-url https://charts.helm.sh/stable
	jx step helm build --boot --provider-values-dir=../kubeProviders
	helm lint .

clean: 
	rm -rf charts
	rm -rf requirements.lock

I tweaked the helm init so that it is supplied with stable-repo-url.

To be continued…

Posted in jenkins-x, kubernetes | Tagged | Leave a comment

Kubernetes, Jenkins-X and Helm Madness part 1

Our kubernetes cluster is more than a year old and I’m the only one left to maintain it. Several things happened almost all at once and things started to break one by one. Versions kubernetes – 1.16.15-gke.4300 kubectl client – … Continue reading

Posted in jenkins-x, kubernetes | Tagged | 1 Comment

Running Jest Debug on Windows – SyntaxError: missing ) after argument list

I’m using NestJS and the regular tests works smooth even in my Windows machine. However, when I tried running the debugging mode tests, especially since I’m trying to debug a memory leak, I get this error: basedir=$(dirname “$(echo “$0” | … Continue reading

Posted in NodeJS | Tagged | 1 Comment

The Dreaded node-gyp installation problem on Windows 10

Time and time again, I encounter this dreaded node-gyp installation problem for Windows. I used to get away with in but when the package depends on it, I have no choice but to fix it. I just followed this instructions: … Continue reading

Posted in Uncategorized | Tagged | Leave a comment

Creating Google Storage Buckets with Config Connector

We are trying to build an API to store files for user profile images. To do that in our current infrastructure, we need to use Google Cloud Storage. Since we are already using Kubernetes, we can automate the process by … Continue reading

Posted in jenkis-x, kubernetes | Tagged , | Leave a comment

Kubernetes – Add vault secrets into environment variables

We already have a full kubernetes and Jenkins-X stack for our application’s CI/CD. For a fully setup application, I’m already able to tweak the secrets settings and how the secrets are populated from vault to environment variable. However, for an … Continue reading

Posted in jenkis-x, kubernetes | Tagged , | Leave a comment

React TypeScript – Props not recognized when component is wrapped in some high order components

It is so annoying when you developed your component nicely but when you need to wrap it with some high order components from some cool plugins, props suddenly doesn’t get recognized by the TypeScript compiler. Without knowing the proper way … Continue reading

Posted in React, TypeScript | Tagged | Leave a comment

Viewing Container Logs in Kubernetes

I’m currently working as a developer but also manages the infrastructure of our project which is hosted in Google Cloud Platform. GCP has web console for viewing logs for each of our services and using the web is a bit … Continue reading

Posted in kubernetes | Tagged | Leave a comment

Jenkins-X – Customize release version

We have an existing infrastructure in GCP using Kubernetes and Jenkis-X. When one of the owner decided to leave the business due to personal reasons, we ended up moving all the repositories to the other owner. This caused the releases … Continue reading

Posted in jenkis-x | Leave a comment

Mongo Read hangs up indefinitely when readPreference is secondary

We have a backend API app in Google Cloud using all the Kubernetes goodness. For some reason, we don’t use a Google Cloud backed MongoDB service. Instead, we spin our own MongoDB replicaSet using one of the stable helm charts … Continue reading

Posted in kubernetes, MongoDB | Leave a comment