kubernetes-the-hard-way/docs/13-certificate-renewal.md

Configuring Certificate Renewal

Prerequisites

The commands in this section must be run on every instance: controller-0, controller-1, controller-2, worker-0, worker-1, and worker-2. Login to each instance using the gcloud command. Example:

gcloud compute ssh controller-0

Download certificate management tools

Run each command on every node.

Download the step CLI binary:

wget -q --show-progress --https-only --timestamping \
  "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.1/step_linux_0.18.1_amd64.tar.gz"

Install the binary:

tar -xvf step_linux_0.18.1_amd64.tar.gz
sudo mv step_0.18.1/bin/step /usr/local/bin/

Bootstrap with the CA

Configure the host to trust your Certificate Authority:

{
STEP_CA_URL=$(gcloud compute project-info describe --format='get(commonInstanceMetadata.items.STEP_CA_URL)')
STEP_CA_FINGERPRINT=$(gcloud compute project-info describe --format='get(commonInstanceMetadata.items.STEP_CA_FINGERPRINT)')
sudo step ca bootstrap \
    --ca-url "${STEP_CA_URL}" \
    --fingerprint "${STEP_CA_FINGERPRINT}"
}

Output:

The root certificate has been saved in /root/.step/certs/root_ca.crt.
The authority configuration has been saved in /root/.step/config/defaults.json.

Set up the certificate renewal timer

We'll use a systemd timer to renew certificates when they are 2/3rds of the way through their validity period.

Install the systemd certificate renewal service and timer.

cat << EOF | sudo tee /etc/systemd/system/cert-renewer@.service
[Unit]
Description=Certificate renewer for %I
After=network-online.target
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
StartLimitIntervalSec=0

[Service]
Type=oneshot
User=root

Environment=STEPPATH=/etc/step-ca \\
            CERT_LOCATION=/etc/step/certs/%i.crt \\
            KEY_LOCATION=/etc/step/certs/%i.key

; ExecCondition checks if the certificate is ready for renewal,
; based on the exit status of the command.
; (In systemd <242, you can use ExecStartPre= here.)
ExecCondition=/usr/local/bin/step certificate needs-renewal \${CERT_LOCATION}

; ExecStart renews the certificate, if ExecStartPre was successful.
ExecStart=/usr/local/bin/step ca renew --force \${CERT_LOCATION} \${KEY_LOCATION}

[Install]
WantedBy=multi-user.target
EOF

Install the timer:

cat << EOF | sudo tee /etc/systemd/system/cert-renewer@.timer
[Unit]
Description=Certificate renewal timer for %I
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production

[Timer]
Persistent=true

; Run the timer unit every 5 minutes.
OnCalendar=*:1/5

; Always run the timer on time.
AccuracySec=1us

; Add jitter to prevent a "thundering hurd" of simultaneous certificate renewals.
RandomizedDelaySec=5m

[Install]
WantedBy=timers.target
EOF

Controller Certificate Renewal

Prerequisites

The commands in this section must be run on every controller: controller-0, controller-1, controller-2. Login to each instance using the gcloud command. Example:

gcloud compute ssh controller-0

Configure certificate renewal for etcd

Create and start a certificate renewal timer for etcd:

sudo mkdir /etc/systemd/system/cert-renewer@etcd.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@etcd.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/etc/etcd/kubernetes.pem \\
            KEY_LOCATION=/etc/etcd/kubernetes-key.pem
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@etcd.timer

Configure certificate renewal for kube-controller-manager

Create and start a certificate renewal timer for kube-controller-manager. This one will use kubectl to embed the renewed certificate and key into the kubeconfig file before restarting the controller manager. Run:

sudo mkdir /etc/systemd/system/cert-renewer@kube-controller-manager.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@kube-controller-manager.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/var/lib/kubernetes/kube-controller-manager.pem
 \\
            KEY_LOCATION=/var/lib/kubernetes/kube-controller-manager-key.pem

ExecStartPost=kubectl config set-credentials system:kube-controller-manager \\
    --client-certificate=\${CERT_LOCATION} \\
    --client-key=\${KEY_LOCATION} \\
    --embed-certs=true \\
    --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig

ExecStartPost=systemctl restart kube-controller-manager.service
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@kube-controller-manager.timer

Configure certificate renewal for kube-scheduler

Create and start a certificate renewal timer for kube-scheduler:

sudo mkdir /etc/systemd/system/cert-renewer@kube-scheduler.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@kube-scheduler.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/var/lib/kubernetes/kube-scheduler.pem
 \\
            KEY_LOCATION=/var/lib/kubernetes/kube-scheduler-key.pem

ExecStartPost=kubectl config set-credentials system:kube-scheduler \\
    --client-certificate=\${CERT_LOCATION} \\
    --client-key=\${KEY_LOCATION} \\
    --embed-certs=true \\
    --kubeconfig=/var/lib/kubernetes/kube-scheduler.kubeconfig

ExecStartPost=systemctl restart kube-scheduler.service
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@kube-scheduler.timer

Configure certificate renewal for kube-apiserver

Create and start a certificate renewal timer for kube-apiserver:

sudo mkdir /etc/systemd/system/cert-renewer@kube-apiserver.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@kube-apiserver.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/var/lib/kubernetes/kubernetes.pem  \\
            KEY_LOCATION=/var/lib/kubernetes/kubernetes-key.pem 

ExecStartPost=systemctl restart kube-apiserver.service
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@kube-apiserver.timer

Configure service account certificate renewal timer

The service account certificate and key is used by the API server, so we will need to restart it when the certificate file is updated:

sudo mkdir /etc/systemd/system/cert-renewer@kube-service-account.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@kube-service-account.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/var/lib/kubernetes/service-account.pem
 \\
            KEY_LOCATION=/var/lib/kubernetes/service-account-key.pem

; Restart services that use the service account certificate or key
ExecStartPost=systemctl restart kube-apiserver.service
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@kube-service-account.timer

Remember to run the above commands on each controller node: controller-0, controller-1, and controller-2.

Worker Certificate Renewal

Prerequisites

The commands in this section must be run on every worker: worker-0, worker-1, and worker-2. Login to each instance using the gcloud command. Example:

gcloud compute ssh worker-0

Configure Certificate Renewal for kubelet.service

Install the a renewal service that will restart kubelet.service when the certificate is renewed:

sudo mkdir /etc/systemd/system/cert-renewer@kubelet.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@kubelet.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/var/lib/kubelet/${HOSTNAME}.pem
 \\
            KEY_LOCATION=/var/lib/kubelet/${HOSTNAME}-key.pem

; Restart services that use the service account certificate or key
ExecStartPost=systemctl restart kubelet.service
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@kubelet.timer

Configure Certificate Renewal for kube-proxy.service

Install a renewal service that will rebuild the kubeconfig file and restart kube-proxy when the certificate is renewed:

sudo mkdir /etc/systemd/system/cert-renewer@kube-proxy.service.d
cat <<EOF | sudo tee /etc/systemd/system/cert-renewer@kube-proxy.service.d/override.conf
[Service]
Environment=STEPPATH=/root/.step \\
            CERT_LOCATION=/var/lib/kube-proxy/kube-proxy.pem
 \\
            KEY_LOCATION=/var/lib/kube-proxy/kube-proxy.pem

ExecStartPost=kubectl config set-credentials system:kube-proxy \\
    --client-certificate=\${CERT_LOCATION} \\
    --client-key=\${KEY_LOCATION} \\
    --embed-certs=true \\
    --kubeconfig=/var/lib/kube-proxy/kubeconfig

ExecStartPost=systemctl restart kube-proxy.service
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewer@kube-proxy.timer

Remember to run the above commands on each controller node: worker-0, worker-1, and worker-2.