Kubernetes does not accept my certificate

05 Jan, 2024

error: You must be logged in to the server (the server has asked for the client to provide credentials)

We try to do things right at work. So with that we create user specific certificates to be allowed to run deployments in our Kubernetes cluster.

It worked. We tested it on staging. We then also ran the kubectl apply commands on production … and it stopped working on staging.

I am writing this because when I looked around the internet with that error, most results were about Amazon EKS. Our case was pure Kubernetes.

So what happened? And how did we solve it?

The crux was with how we organise different clusters on our machines. We have separate kubeconfig files. These are joined by setting all their paths in a KUBECONFIG environment variable and letting kubectl do the heavy lifting.

But we missed one important point when RTFM:

Example: If two files specify a red-user, use only values from the first file's red-user. Even if the second file has non-conflicting entries under red-user, discard them.

(From Organizing Cluster Access Using kubeconfig Files: Merging kubeconfig files.)

In both cases my user was being identified as simply “martijn”. This meant only one pair of credentials was kept and was then tried for both clusters.

We have updated setup flow to prepend the cluster identifier to the user name, and everything works again!