The Continued Adventures of Jenkins X
I recently wrote a blog post about how we use Jenkins X to make use of best in class GitOps, DevOps, and all the other BuzzWordsOps practices at Mentor Medier. After I published the blog, Cloudbees contacted me and wanted to chat about our experiences. In particular — what problems we had experienced using Jenkins X. Ironically, the story I tell in this blog happened right after I talked with Cloudbees. So — here is a report on some of the problems using Jenkins X — a bleeding-edge project, with rapid development, and continuous change.
Trigger warning: There will be (virtual) blood.
Adding multiple production environments
In our project, to create tools for migrating the publishing system for Mentor Medier into using Arc Publishing, we need to build integrations on several systems. One of these is the migration of the old content — this will not be public-facing, and we will shut it down once we finally switch all systems over to Arc. Another integration is with the existing customer management system, handling users and subscriptions. These will be public-facing and will continue to run under the new Arc system. Thus, we figured that it makes sense to create two separate production environments in Kubernetes.
Quite confident this would be a walk in the park, I went to work. To create a new environment in Jenkins X, I remembered to have used the “jx create env” command. I ran the command, answered some questions about naming and promotion, and very soon, I had what seemed to be a brand new working environment — for production!!! But as I started to look at the details, I saw that I could not access the applications using https, so somehow neither TLS nor domains were correctly set up. Then I realized: Of course! “jx create env” is the old (and bad) way to create environments. Now, in these modern days, we, of course, use “jx boot” for everything, so that we can have everything documented in git! GITOPS!!! “jx boot,” for those unfamiliar with Jenkins X, was created some months ago, replacing the old way of installing Jenkins X, which involved using an interactive “jx install” command.
So the “jx create env” creates a Kubernetes environment but does not update configuration in jx boot git repository, and that is where you should configure the handling of DNS and TLS.
Adding a new environment when you first run “jx boot” is as simple as adding a new key under the list of environments. However, my experiments proved that merely adding a new key (“acme-production”) in this list, would not do anything at all. So, the two must be combined, using the cumbersome process described in this FAQ. Not being overly impressed by this procedure, I created an issue in GitHub — it would be nice if this process were simple.
Running Jx Boot
Continuing my quest — I have now created the environment using the “jx create env” command, and have copied out the SourceRepository and Environment definitions CRD into my jx boot git repository. To make the magic happen and have Jenkins X magically start the process of adding TLS and setup domains — I have to run the “jx boot” command.
It starts up, logging all the usual steps, setting up IRSA service accounts, using CloudFormation in AWS, external-dns, cert-manager, etc. After it’s been running for some minutes, I get a failure message about Nexus: Can not downgrade disk size. It dawns on me what this is about — It’s a long time since last I ran “jx boot” — the Jenkins X installation has been running very smooth lately, and we are busy developing — since we are soon going into production! I recollect the problem with Nexus — since we are developing rapidly, we had to increase the disk size of Nexus — from the initial meager 8GB to 100GB, just to be sure. Since I could not find where to edit this in the jx boot repo, I did the change manually. (I already created an issue for this back in December 2019) A workaround I used earlier allowed me to run “jx boot “ anyway — by configuring the repository type to something other than nexus in the jx boot config. That would make jx boot skip Nexus installation altogether, and leave my existing installation as it was, which some may say is another bug, but it helped me.
Updated JX versions
As I now was getting deep into things, I made sure to update everything to the latest versions, just to be sure that we were up to date. Perhaps even some of our bugs were fixed.
Now, when I ran “jx boot,” I found the environments were created — with TLS and domains set up correctly!!! HOORAY!!! However, not long after, I figured that the entire Nexus installation had disappeared. Perhaps a bug was fixed, or maybe I did something different this time. Gone it was.
With Nexus gone — all our shared libraries (Yes, I know — we are not doing microservices strictly) were gone as well, and there was no trace of the old disk. And the backups don’t work (see issue), so I had to rerun the library builds, and update the dependents with the new versions. However, this process went quite straight forward and quick. All I had to was run “jx start pipeline” on the library projects, and updating version numbers in the projects that used the libraries.
Now Nexus worked, and I was happy to inform my coworkers that everything was okay. Good vibrations, time to celebrate the joys of open source with a nice IPA.
Tekton permissions to ECR
The next surprise was that the AWS docker registry ECR was not accepting our images. We got HTTP status code 401, Access denied, on trying to push, which indicates some permissions problem. Digging around on the policies attached to the service account connected to the tekton bot — which is the guy responsible for running our pipelines — revealed that we did not have any permissions set to use ECR. Adding the full set of ECR permissions fixed it — I created an issue in the jx repo, and almost immediately, Daniel Gozalo, one of the helpful contributors, created a pull request with a proposed fix.
Results
The results are that we have two working production Kubernetes environments, we have Nexus, we have pipelines running and publishing to Docker. Jenkins X is an inspiring project, there is a large helpful community, with a very active Slack channel that I have used extensively, as well as reporting bugs on the Github repository.
Jenkins X is a very ambitious project, attempting to help people quickly get started using CI/CD on Kubernetes. It supports multiple cloud providers, numerous programming languages, creates Tekton build pipelines, and handles deployment of applications to environments.
An overarching problem could be to define what the core functionality is. Should it include Nexus, for instance?
We are following the developments closely, and are looking forwards to whatever the excellent Jenkins X team decides to be the future of CI/CD on Kubernetes.
