k8s里各种证书配置关系解释

Today, let’s talk about Kubernetes private/public keys & certificate authorities!

This blog post is about how to take your own requirements about how certificate authorities + private keys should be organized and set up your Kubernetes cluster the way you need to.

The various Kubernetes components have a TON of different places where you can put in a certificate/certificate authority. When we were setting up a cluster I felt like there were like 10 billion different command line arguments for certificates and keys and certificate authorities and I didn’t understand how they all fit together.

There are not actually 10 billion command line arguments but there are quite a lot. For example! Let’s just look at the command line arguments to the API server.

The API server has more than 16 different command line arguments to do with certificates or keys (I actually deleted a bunch to cut it down to this list).

and here are the arguments for the controller manager:

and for the kubelet:

In this post I’m going to assume you know basically how TLS certificates and certificate authorities (“CAs”) work. When setting this up, I knew how certs work but I did not understand how all the different Kubernetes certificate authorities fit together.

In this post I’ll explain some of the main CAs you can set in Kubernetes and how they fit together.

I’ll also tell you about a few things I learned while setting all of this up:

• You can’t use a CA to check the validity of the service account key. The service account key is a weird key which is handled differently from literally every other key.
• You can (and should!!) use an authenticating proxy if the way Kubernetes maps client certificates to usernames and groups doesn’t work for you
• Setting up the API server to support too many different authentication methods (“more than 2”) makes things confusing (though maybe this isn’t too surprising :))

(as usual, let me know what mistakes you find in here, I think most of this is right but it’s a complicated topic!)

PKI & Kubernetes

When I started reading about kubernetes I saw this term “PKI” a lot and I wasn’t sure what it meant.

If you have a Kubernetes cluster, you might have hundreds or thousands of private & public keys (in client certificates, server certificates, anywhere!). That is a lot of private keys!

If you just had thousands of unrelated independent keys that would be chaos. Chaos is not great for security. So instead the way you manage private/public keys is you have certificate authorities (“CAs”) issue certificates saying “hey, this public key is OK, it’s from me, you should trust it”.

Your PKI (“public key infrastructure”) is how you organize all of your keys – which keys are signed by which certificate authorities.

For example:

• You could have one CA per Kubernetes cluster, that’s responsible for signing all the private keys in that cluster (this is the model Kubernetes docmentation usually assumes)
• You could have one global CA that’s responsible for signing ALL your private keys
• You could have one CA you use for services that are externally visibile and another CA you use for internal-only services
• … something else

I’m not a security expert and I’m not gonna try to tell you how you should manage the private keys + certificate authorities in your infrastructure. But! No matter what PKI model you want to use, I’m pretty sure you can make it work with Kubernetes.

This blog post is about how to take your own requirements about how certificate authorities + private keys should be organized and set up your Kubernetes cluster the way you need to.

Does a Kubernetes cluster have to have a single root certificate authority? (no)

If you read a lot of guides to how to set up Kubernetes, you’ll see a step like “set up a certificate authority”. Kelsey Hightower’s great “kubernetes the hard way” doc has it as Step 2, and the Trusting TLS in a cluster guide says:

Every Kubernetes cluster has a cluster root Certificate Authority (CA). The CA is generally used by cluster components to validate the API server’s certificate, by the API server to validate kubelet client certificates, etc.

The way this basically works is:

1. Set up a certificate authority
2. Use that certificate authority to generate a bunch of different certificates that you’ll give to different parts of the Kubernetes infrastructure.

But what if you don’t want to set up a new certificate authority for each kubernetes cluster? We didn’t want to do this for various reasons that I won’t go into, and at first I was worried that you had to set up a single root cluster CA.

It turns out that this sentence (“every Kubernetes cluster has a [single] cluster root Certificate Authority”) is not true – you can actually use certificates issued by several completely different CAs to manage your Kubernetes cluster and it’s fine.

To be clear – I’m not saying you shouldn’t have a single root certificate for your Kubernetes cluster, I’m just saying you don’t have to if for whatever reason that way doesn’t meet your requirements.

Let’s break down some of these command line arguments about certificates and how they relate to each other. Each of these sections is about one certificate authority you can define. They’re all independent – none of them need to be the same as any other one. (though in practice you will probably want to make some of them the same, you don’t want to be managing like 6 CAs probably).

The API server’s TLS certificate (and certificate authority)

You’re probably using TLS to connect to your Kubernetes API server. These two options (to the API server) let you pick what certificate the API server should use.

Once you set a TLS cert, you’ll need to set up a kubeconfig file for the components (like the kubelet and kubectl) that want to talk to the API server.

The kubeconfig file will look something like this:

One thing I found surprising about this is – almost everything else in the universe that uses TLS will look in /etc/ssl to find a list of certificate authorities the computer trusts by default. But Kubernetes doesn’t do that, instead it mandates “no, you have to say exactly which CA issued the API server’s TLS cert”.

You can pass this kubeconfig file into any kubernetes component with --kubeconfig /path/to/kubeconfig.yaml

So! We’ve met our first certificate authority: the CA that issues the API server’s TLS cert. This CA doesn’t need to be the same as any of the other certificate authorities we’re going to discuss.

The API server client certificate authority (+ certificates)

One way for Kubernetes components to authenticate to the API server is withclient certificates.

All of these client certs should be issued by the same CA (which, again, doesn’t need to be the same as the CA that issued the API server’s server TLS cert).

When using a kubeconfig file (like we talked about above), you set the client certificates in that file, like this:

Kubernetes makes a lot of assumptions about how you’ve set up your client certificates. (it sets the user to be the Common Name field and the group to be the Organization field). If those assumptions don’t match what you want, the right thing to do is to not use client cert auth and instead use an authenticating proxy.

The request header certificate authority (or: using an authenticating proxy)

Another way to set up Kubernetes auth is with an authenticating proxy. If you have a lot of opinions about what usernames and groups should be sent to the API server, you can set up a proxy which passes usernames & groups to the API server in a HTTP header.

The docs basically explain how this works – the proxy uses a client cert to identify itself, and the --requestheader-client-ca-file tells the API server which CA to use to verify that client cert.

I don’t have too much to say about this except – we learned pretty quickly that having too many auth methods in your API server (“accept client certificates OR an authenticating proxy OR a token OR…“) makes things confusing. It’s probably better to pick a small number (like 1 or 2?) of authentication methods your API server supports because it makes it easier to debug problems and understand your security model.

serviceaccount private keys (which aren’t signed by a certificate authority)

The controller manager signs serviceaccount tokens with a private key. Unlike every other private key that Kubernetes supports, the serviceaccount key doesn’t support “hey use a CA to check if this is the right key”. This means you have to give exactly the same private key file to every controller manager.

Anyway this key does not need to have a certificate and does not need to be signed by any certificate authority. You can just generate a key with

and distribute it to every controller manager / API server.

Using --tls-private-key-file for this seems generally fine to me though, as long as you give every API server the same TLS key (which I think you usually would?). (I’m assuming here that you have a HA setup where you run more than one API server and more than one controller manager)

If you give 2 different controller managers 2 different keys, they’ll sign serviceaccount tokens with different keys and you’ll end up with invalid serviceaccount tokens (see this issue). I think this isn’t ideal (Kubernetes should probably support these keys being issued from a CA like it does for ~every other private key). From reading the source code I think the reason it’s set up this way is that jwt-go doesn’t support using a CA to check signatures.

kubelet certificate authorities

Let’s talk about the kubelet! Here are the relevant command line arguments for the API server & kubelet:

It’s useful to authenticate requests to the kubelet because the kubelet can execute arbitrary code on your machines :) (in fact that’s its job)

There are actually 2 CAs here. I won’t go into too much detail because this is the same as the setup for the APi server – the kubelet has a TLS cert (that it uses to serve TLS requests) and also supports client cert authentication.

You tell the API server what certificate authority to use to check the kubelet’s TLS cert, and what client certificate to use when talking to the kubelet.

Again these 2 CAs could be totally different from each other.