From 397ca243436dea6501cea852caeae48a4a1de15b Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Mon, 31 Jan 2022 14:09:51 -0800 Subject: [PATCH 01/10] CA install WIP --- docs/02-client-tools.md | 58 ++--- docs/04-certificate-authority.md | 354 +++++++++++++++++++++++-------- 2 files changed, 282 insertions(+), 130 deletions(-) diff --git a/docs/02-client-tools.md b/docs/02-client-tools.md index 94d93be..1590210 100644 --- a/docs/02-client-tools.md +++ b/docs/02-client-tools.md @@ -1,72 +1,56 @@ # Installing the Client Tools -In this lab you will install the command line utilities required to complete this tutorial: [cfssl](https://github.com/cloudflare/cfssl), [cfssljson](https://github.com/cloudflare/cfssl), and [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl). +In this lab you will install the command line utilities required to complete this tutorial: [step](https://github.com/smallstep/cli), and [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl). ## Install CFSSL -The `cfssl` and `cfssljson` command line utilities will be used to provision a [PKI Infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure) and generate TLS certificates. +The `step` command line utility will be used to provision a [PKI Infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure) and generate TLS certificates. -Download and install `cfssl` and `cfssljson`: +Download and install `step`: ### OS X -``` -curl -o cfssl https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/darwin/cfssl -curl -o cfssljson https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/darwin/cfssljson -``` +For Intel chips: ``` -chmod +x cfssl cfssljson +curl -L https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_darwin_0.18.0_amd64.tar.gz | tar xz +sudo mv step_0.18.0/bin/step /usr/local/bin/ ``` -``` -sudo mv cfssl cfssljson /usr/local/bin/ -``` - -Some OS X users may experience problems using the pre-built binaries in which case [Homebrew](https://brew.sh) might be a better option: +For Apple Silicon: ``` -brew install cfssl +curl -L https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_darwin_0.18.0_arm64.tar.gz | tar xz +sudo mv step_0.18.0/bin/step /usr/local/bin/ +``` + +Or, if you'd like to use [Homebrew](https://brew.sh): + +``` +brew install step ``` ### Linux ``` -wget -q --show-progress --https-only --timestamping \ - https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssl \ - https://storage.googleapis.com/kubernetes-the-hard-way/cfssl/1.4.1/linux/cfssljson -``` - -``` -chmod +x cfssl cfssljson -``` - -``` -sudo mv cfssl cfssljson /usr/local/bin/ +curl -L https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz +sudo mv step_0.18.0/bin/step /usr/local/bin/ ``` ### Verification -Verify `cfssl` and `cfssljson` version 1.4.1 or higher is installed: +Verify `step` version 0.18.0 or higher is installed: ``` -cfssl version +step version ``` > output ``` -Version: 1.4.1 -Runtime: go1.12.12 -``` - -``` -cfssljson --version -``` -``` -Version: 1.4.1 -Runtime: go1.12.12 +Smallstep CLI/0.18.0 (linux/amd64) +Release Date: 2021-11-17 21:15 UTC ``` ## Install kubectl diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index 1510993..da38fe2 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -1,100 +1,280 @@ -# Provisioning a CA and Generating TLS Certificates +# Provisioning a CA -In this lab you will provision a [PKI Infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure) using CloudFlare's PKI toolkit, [cfssl](https://github.com/cloudflare/cfssl), then use it to bootstrap a Certificate Authority, and generate TLS certificates for the following components: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy. +In this lab you will provision a [PKI Infrastructure](https://en.wikipedia.org/wiki/Public_key_infrastructure) using Smallstep's CA server, [`step-ca`](https://github.com/smallstep/certificates), and generate TLS certificates for the following components: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy. ## Certificate Authority -In this section you will provision a Certificate Authority that can be used to generate additional TLS certificates. +In this section you will provision a `step-ca` Certificate Authority that can be used to generate additional TLS certificates. The CA will only run on `controller-0`. While it's possible to run a high-availability CA across multiple nodes, it's not necessary in a small-to-medium sized Kubernetes cluster. The CA service would have to be down for several days before having any negative impact on the cluster. -Generate the CA configuration file, certificate, and private key: +Connect to `controller-0`: + +``` +gcloud compute ssh controller-0 +``` + +Download the `step` client and `step-ca` server binaries, and the `jq` command: ``` { - -cat > ca-config.json < /dev/null +< /dev/urandom tr -dc A-Za-z0-9 | head -c40 > provisioner-password +umask 002 +} +``` + +Initialize your PKI: + +``` +{ +INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \ +http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip) +EXTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip) +INTERNAL_HOSTNAME=$(hostname -f) +sudo -E step ca init --name="admin" \ + --dns="$INTERNAL_IP,$INTERNAL_HOSTNAME,$EXTERNAL_IP" \ + --address=":4443" --provisioner="kubernetes" \ + --password-file="$(step path)/password" \ + --provisioner-password-file="provisioner-password" +sudo -E step ca provisioner add acme --type ACME +} +``` + +Configure the CA provisioner to issue 90-day certificates: + +``` +{ +sudo jq '(.authority.provisioners[]) += { + "claims": { + "maxTLSCertDuration": "2160h", + "defaultTLSCertDuration": "2160h" + } +}' /etc/step-ca/config/ca.json > ca-new.json +sudo mv ca-new.json /etc/step-ca/config/ca.json +} +``` + +Put the CA configuration into place, and add the CA to systemd: + +``` +{ +sudo chown -R step:step /etc/step-ca +cat < ca-csr.json < /dev/null +``` + +Finally, start the CA service: + +``` { - "CN": "Kubernetes", - "key": { - "algo": "rsa", - "size": 2048 - }, - "names": [ - { - "C": "US", - "L": "Portland", - "O": "Kubernetes", - "OU": "CA", - "ST": "Oregon" - } - ] -} -EOF - -cfssl gencert -initca ca-csr.json | cfssljson -bare ca - +sudo systemctl daemon-reload +sudo systemctl enable --now step-ca } ``` -Results: +## Verification + +Check the CA health, then request and save the CA root certificate: + +``` +sudo -u step -E step ca health +``` + +Output: + +``` +ok +``` + +You can now sign out of `controller-0`. + +# Generating certificates + +## Bootstrapping with the CA + +### Bootstrapping your local machine + +Run the following on your local machine. + +Download your CA's root certificate: + +``` +gcloud compute scp controller-0:ca.pem controller-0:provisioner-password . +``` + +Result: ``` -ca-key.pem ca.pem +provisioner-password ``` +Now bootstrap with your CA: + +``` +{ +CA_IP=$(gcloud compute instances describe controller-0 \ + --format='get(networkInterfaces[0].accessConfigs[0].natIP)') +step ca bootstrap --ca-url "https://$CA_IP:4443/" --fingerprint $(step certificate fingerprint ca.pem) +} +``` + +Output: + +``` +The root certificate has been saved in /home/carl/.step/authorities/XX.XXX.XXX.XXX/certs/root_ca.crt. +The authority configuration has been saved in /home/carl/.step/authorities/XX.XXX.XXX.XXX/config/defaults.json. +The profile configuration has been saved in /home/carl/.step/profiles/XX.XXX.XXX.XXX/config/defaults.json. +``` + +Add your CA URL and fingerprint to the project metadata on GCP, so instances can bootstrap: + +``` +gcloud compute project-info add-metadata --metadata="STEP_CA_URL=https://10.240.0.10:4443,STEP_CA_FINGERPRINT=$(step certificate fingerprint ca.pem)" +``` + +Output: + +``` +Updated [https://www.googleapis.com/compute/v1/projects/project-id-xxxxxx]. +``` + +### Bootstrapping remote instances + + +Run each command on every node: + +``` +{ +for i in 0 1 2; do + gcloud compute ssh worker-${i} -- \ + step ca bootstrap \ + --ca-url "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_URL)" \ + --fingerprint "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_FINGERPRINT)" + gcloud compute ssh worker-${i} -- \ + step ca bootstrap \ + --ca-url "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_URL)" \ + --fingerprint "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_FINGERPRINT)" +done +} +``` + +Output: + +``` +The root certificate has been saved in /home/carl/.step/certs/root_ca.crt. +The authority configuration has been saved in /home/carl/.step/config/defaults.json. +``` + + ## Client and Server Certificates In this section you will generate client and server certificates for each Kubernetes component and a client certificate for the Kubernetes `admin` user. ### The Admin Client Certificate -Generate the `admin` client certificate and private key: +On your local machine, generate the `admin` client certificate and private key: ``` { - -cat > admin-csr.json < ${instance}-csr.json < Date: Tue, 1 Feb 2022 13:06:06 -0800 Subject: [PATCH 02/10] More edits --- docs/04-certificate-authority.md | 223 ++++++------------------------- 1 file changed, 39 insertions(+), 184 deletions(-) diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index da38fe2..98fba24 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -300,7 +300,12 @@ EXTERNAL_IP=$(gcloud compute instances describe ${instance} \ INTERNAL_IP=$(gcloud compute instances describe ${instance} \ --format 'value(networkInterfaces[0].networkIP)') -step ca certificate "system:node:${instance}" ${instance}.pem ${instance}-key.pem --san "${instance}" --san "${EXTERNAL_IP}" --san "${INTERNAL_IP}" --provisioner "kubernetes" --provisioner-password-file "provisioner-password" +step ca certificate "system:node:${instance}" ${instance}.pem ${instance}-key.pem \ + --san "${instance}" \ + --san "${EXTERNAL_IP}" \ + --san "${INTERNAL_IP}" \ + --provisioner "kubernetes" \ + --provisioner-password-file "provisioner-password" done ``` @@ -315,51 +320,24 @@ worker-2-key.pem worker-2.pem ``` - - - - - - - - - - - - ### The Controller Manager Client Certificate -Generate the `kube-controller-manager` client certificate and private key: +Generate the `kube-controller-manager`, `kube-proxy`, and `kube-scheduler` client certificates and private keys: ``` { - -cat > kube-controller-manager-csr.json < kube-proxy-csr.json < kube-scheduler-csr.json < kubernetes-csr.json < service-account-csr.json < Date: Tue, 1 Feb 2022 17:50:06 -0800 Subject: [PATCH 03/10] Starting on cert renewal pieces --- docs/03-compute-resources.md | 4 +- docs/04-certificate-authority.md | 89 +++++++++++-------- ...08-bootstrapping-kubernetes-controllers.md | 3 +- docs/{13-smoke-test.md => 14-smoke-test.md} | 0 docs/{14-cleanup.md => 15-cleanup.md} | 0 5 files changed, 54 insertions(+), 42 deletions(-) rename docs/{13-smoke-test.md => 14-smoke-test.md} (100%) rename docs/{14-cleanup.md => 15-cleanup.md} (100%) diff --git a/docs/03-compute-resources.md b/docs/03-compute-resources.md index a5402bb..f93dff3 100644 --- a/docs/03-compute-resources.md +++ b/docs/03-compute-resources.md @@ -43,11 +43,11 @@ gcloud compute firewall-rules create kubernetes-the-hard-way-allow-internal \ --source-ranges 10.240.0.0/24,10.200.0.0/16 ``` -Create a firewall rule that allows external SSH, ICMP, and HTTPS: +Create a firewall rule that allows external SSH, ICMP, HTTPS, and step-ca traffic: ``` gcloud compute firewall-rules create kubernetes-the-hard-way-allow-external \ - --allow tcp:22,tcp:6443,icmp \ + --allow tcp:22,tcp:4443,tcp:6443,icmp \ --network kubernetes-the-hard-way \ --source-ranges 0.0.0.0/0 ``` diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index 98fba24..5844642 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -17,9 +17,10 @@ Download the `step` client and `step-ca` server binaries, and the `jq` command: ``` { wget -q --show-progress --https-only --timestamping \ - "https://dl.step.sm/gh-release/certificates/gh-release-header/v0.18.0/step-ca_linux_0.18.0_amd64.tar.gz" -wget -q --show-progress --https-only --timestamping \ - "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" + "https://dl.step.sm/gh-release/certificates/gh-release-header/v0.18.0/step-ca_linux_0.18.0_amd64.tar.gz" \ + "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" \ + "https://raw.githubusercontent.com/smallstep/cli/master/systemd/cert-renewer%40.service" \ + "https://raw.githubusercontent.com/smallstep/cli/master/systemd/cert-renewer%40.timer" sudo apt update sudo apt install -y jq } @@ -70,21 +71,53 @@ sudo -E step ca init --name="admin" \ --address=":4443" --provisioner="kubernetes" \ --password-file="$(step path)/password" \ --provisioner-password-file="provisioner-password" -sudo -E step ca provisioner add acme --type ACME } ``` +Add an X509 certificate template file: + +``` +mkdir -p /etc/step-ca/templates/x509 + +# Server cert template. +cat < /etc/step-ca/templates/x509/kubernetes.tpl +{ + "subject": { +{{- if .Insecure.User.Organization }} + "organization": {{ toJson .Insecure.User.Organization }}, +{{- end }} + "commonName": {{ toJson .Subject.CommonName }}, + "organizationalUnit": {{ toJson .OrganizationalUnit }} + }, + "sans": {{ toJson .SANs }}, +{{- if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }} + "keyUsage": ["keyEncipherment", "digitalSignature"], +{{- else }} + "keyUsage": ["digitalSignature"], +{{- end }} + "extKeyUsage": ["serverAuth", "clientAuth"] +} +EOF +``` + Configure the CA provisioner to issue 90-day certificates: ``` { -sudo jq '(.authority.provisioners[]) += { +cat <<< $(jq '(.authority.provisioners[] | select(.name == "kubernetes")) += { "claims": { "maxTLSCertDuration": "2160h", "defaultTLSCertDuration": "2160h" + }, + "options": { + "x509": { + "templateFile": "templates/x509/kubernetes.tpl", + "templateData": { + "OrganizationalUnit": "Kubernetes The Hard Way" + } + } } -}' /etc/step-ca/config/ca.json > ca-new.json -sudo mv ca-new.json /etc/step-ca/config/ca.json + }' /etc/step-ca/config/ca.json) > /etc/step-ca/config/ca.json } ``` @@ -234,34 +267,6 @@ Output: Updated [https://www.googleapis.com/compute/v1/projects/project-id-xxxxxx]. ``` -### Bootstrapping remote instances - - -Run each command on every node: - -``` -{ -for i in 0 1 2; do - gcloud compute ssh worker-${i} -- \ - step ca bootstrap \ - --ca-url "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_URL)" \ - --fingerprint "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_FINGERPRINT)" - gcloud compute ssh worker-${i} -- \ - step ca bootstrap \ - --ca-url "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_URL)" \ - --fingerprint "$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/attributes/STEP_CA_FINGERPRINT)" -done -} -``` - -Output: - -``` -The root certificate has been saved in /home/carl/.step/certs/root_ca.crt. -The authority configuration has been saved in /home/carl/.step/config/defaults.json. -``` - - ## Client and Server Certificates In this section you will generate client and server certificates for each Kubernetes component and a client certificate for the Kubernetes `admin` user. @@ -272,9 +277,11 @@ On your local machine, generate the `admin` client certificate and private key: ``` { -step ca certificate admin admin.pem admin-key.pem \ - --provisioner="kubernetes" \ - --provisioner-password-file="provisioner-password" + step ca certificate admin admin.pem admin-key.pem \ + --provisioner="kubernetes" \ + --provisioner-password-file="provisioner-password" \ + --set "Organization=system:masters" \ + --kty RSA } ``` @@ -304,6 +311,7 @@ step ca certificate "system:node:${instance}" ${instance}.pem ${instance}-key.pe --san "${instance}" \ --san "${EXTERNAL_IP}" \ --san "${INTERNAL_IP}" \ + --set "Organization=system:nodes" \ --provisioner "kubernetes" \ --provisioner-password-file "provisioner-password" done @@ -328,14 +336,17 @@ Generate the `kube-controller-manager`, `kube-proxy`, and `kube-scheduler` clien { step ca certificate "system:kube-controller-manager" kube-controller-manager.pem kube-controller-manager-key.pem \ --kty RSA \ + --set "Organization=system:kube-controller-manager" \ --provisioner "kubernetes" \ --provisioner-password-file "provisioner-password" step ca certificate "system:kube-proxy" kube-proxy.pem kube-proxy-key.pem \ --kty RSA \ + --set "Organization=system:node-proxier" \ --provisioner "kubernetes" \ --provisioner-password-file "provisioner-password" step ca certificate "system:kube-scheduler" kube-scheduler.pem kube-scheduler-key.pem \ --kty RSA \ + --set "Organization=system:kube-scheduler" \ --provisioner "kubernetes" \ --provisioner-password-file "provisioner-password" } @@ -376,6 +387,7 @@ step ca certificate "kubernetes" kubernetes.pem kubernetes-key.pem \ --san 10.240.0.12 \ --san ${KUBERNETES_PUBLIC_ADDRESS} \ --san 127.0.0.1 \ + --set "Organization=Kubernetes" \ --provisioner "kubernetes" \ --provisioner-password-file "provisioner-password" } @@ -400,6 +412,7 @@ Generate the `service-account` certificate and private key: { step ca certificate "service-accounts" service-account.pem service-account-key.pem \ --kty RSA \ + --set "Organization=Kubernetes" \ --provisioner "kubernetes" \ --provisioner-password-file "provisioner-password" } diff --git a/docs/08-bootstrapping-kubernetes-controllers.md b/docs/08-bootstrapping-kubernetes-controllers.md index ede6ce0..7bf1384 100644 --- a/docs/08-bootstrapping-kubernetes-controllers.md +++ b/docs/08-bootstrapping-kubernetes-controllers.md @@ -49,7 +49,7 @@ Install the Kubernetes binaries: { sudo mkdir -p /var/lib/kubernetes/ - sudo mv ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \ + sudo mv ca.pem kubernetes-key.pem kubernetes.pem \ service-account-key.pem service-account.pem \ encryption-config.yaml /var/lib/kubernetes/ } @@ -142,7 +142,6 @@ ExecStart=/usr/local/bin/kube-controller-manager \\ --cluster-cidr=10.200.0.0/16 \\ --cluster-name=kubernetes \\ --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\ - --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\ --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\ --leader-elect=true \\ --root-ca-file=/var/lib/kubernetes/ca.pem \\ diff --git a/docs/13-smoke-test.md b/docs/14-smoke-test.md similarity index 100% rename from docs/13-smoke-test.md rename to docs/14-smoke-test.md diff --git a/docs/14-cleanup.md b/docs/15-cleanup.md similarity index 100% rename from docs/14-cleanup.md rename to docs/15-cleanup.md From e8537b3a902ae5e222c5a19cc66f744249bf4b4c Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Tue, 1 Feb 2022 17:51:48 -0800 Subject: [PATCH 04/10] Starting on cert renewal --- docs/13-certificate-renewal.md | 78 ++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/13-certificate-renewal.md diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md new file mode 100644 index 0000000..d0c77e0 --- /dev/null +++ b/docs/13-certificate-renewal.md @@ -0,0 +1,78 @@ +## Prerequisites + +The commands in this lab must be run on each controller instance: `controller-0`, `controller-1`, and `controller-2`. Login to each controller instance using the `gcloud` command. Example: + +``` +gcloud compute ssh controller-0 +``` + +## Download certificate management tools + +Download the `step` CLI binary and renewal utility for systemd: + +``` +wget -q --show-progress --https-only --timestamping \ + "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" \ + "https://files.smallstep.com/cert-renewer%40.service" \ + "https://files.smallstep.com/cert-renewer%40.timer" +``` + +Install the binary and renewal utility files: + +``` +tar -xvf step_linux_0.18.0_amd64.tar.gz +sudo mv step_0.18.0/bin/* /usr/local/bin/ +sudo mv cert-renewer@.service /etc/systemd/system +sudo mv cert-renewer@.timer /etc/systemd/system +sudo systemctl daemon-reload +``` + +### Bootstrapping the CA on your controllers + +Run each command on every node: + +``` +{ +STEP_CA_URL=$(gcloud compute project-info describe --format='get(commonInstanceMetadata.items.STEP_CA_URL)') +STEP_CA_FINGERPRINT=$(gcloud compute project-info describe --format='get(commonInstanceMetadata.items.STEP_CA_FINGERPRINT)') +sudo step ca bootstrap \ + --ca-url "${STEP_CA_URL}" \ + --fingerprint "${STEP_CA_FINGERPRINT}" +} +``` + +Output: + +``` +The root certificate has been saved in /root/.step/certs/root_ca.crt. +The authority configuration has been saved in /root/.step/config/defaults.json. +``` + +## Configure certificate renewal for etcd + +Create and start a certificate renewal timer for etcd: + +``` +sudo mkdir /etc/systemd/system/cert-renewer@etcd.service.d +cat < Remember to run the above commands on each controller node: `controller-0`, `controller-1`, and `controller-2`. From ea551e5278b8ee6c2c13f51682572a57d3abf0eb Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Wed, 2 Feb 2022 15:54:16 -0800 Subject: [PATCH 05/10] Lots of updates on cert renewal --- docs/04-certificate-authority.md | 13 +- ...08-bootstrapping-kubernetes-controllers.md | 3 + docs/13-certificate-renewal.md | 152 +++++++++++++++++- 3 files changed, 154 insertions(+), 14 deletions(-) diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index 5844642..99e7673 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -18,9 +18,7 @@ Download the `step` client and `step-ca` server binaries, and the `jq` command: { wget -q --show-progress --https-only --timestamping \ "https://dl.step.sm/gh-release/certificates/gh-release-header/v0.18.0/step-ca_linux_0.18.0_amd64.tar.gz" \ - "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" \ - "https://raw.githubusercontent.com/smallstep/cli/master/systemd/cert-renewer%40.service" \ - "https://raw.githubusercontent.com/smallstep/cli/master/systemd/cert-renewer%40.timer" + "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" sudo apt update sudo apt install -y jq } @@ -31,9 +29,9 @@ Install the binaries: ``` { tar -xvf step-ca_linux_0.18.0_amd64.tar.gz -sudo mv step-ca_0.18.0/bin/* /usr/local/bin/ +sudo mv step-ca_0.18.0/bin/step-ca /usr/local/bin/ tar -xvf step_linux_0.18.0_amd64.tar.gz -sudo mv step_0.18.0/bin/* /usr/local/bin/ +sudo mv step_0.18.0/bin/step /usr/local/bin/ } ``` @@ -441,7 +439,10 @@ Copy the appropriate certificates and private keys to each controller instance: ``` for instance in controller-0 controller-1 controller-2; do gcloud compute scp ca.pem kubernetes-key.pem kubernetes.pem \ - service-account-key.pem service-account.pem ${instance}:~/ + service-account-key.pem service-account.pem \ + kube-controller-manager-key.pem kube-controller-manager.pem \ + kube-proxy-key.pem kube-proxy.pem kube-scheduler-key.pem \ + kube-scheduler.pem ${instance}:~/ done ``` diff --git a/docs/08-bootstrapping-kubernetes-controllers.md b/docs/08-bootstrapping-kubernetes-controllers.md index 7bf1384..d3d90fe 100644 --- a/docs/08-bootstrapping-kubernetes-controllers.md +++ b/docs/08-bootstrapping-kubernetes-controllers.md @@ -51,6 +51,9 @@ Install the Kubernetes binaries: sudo mv ca.pem kubernetes-key.pem kubernetes.pem \ service-account-key.pem service-account.pem \ + kube-proxy.pem kube-proxy-key.pem \ + kube-controller-manager.pem kube-controller-manager-key.pem \ + kube-scheduler.pem kube-scheduler-key.pem \ encryption-config.yaml /var/lib/kubernetes/ } ``` diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md index d0c77e0..338892b 100644 --- a/docs/13-certificate-renewal.md +++ b/docs/13-certificate-renewal.md @@ -21,9 +21,7 @@ Install the binary and renewal utility files: ``` tar -xvf step_linux_0.18.0_amd64.tar.gz -sudo mv step_0.18.0/bin/* /usr/local/bin/ -sudo mv cert-renewer@.service /etc/systemd/system -sudo mv cert-renewer@.timer /etc/systemd/system +sudo mv step_0.18.0/bin/step /usr/local/bin/ sudo systemctl daemon-reload ``` @@ -48,6 +46,66 @@ The root certificate has been saved in /root/.step/certs/root_ca.crt. The authority configuration has been saved in /root/.step/config/defaults.json. ``` +## Set up the certificate renewal timer + +We'll use a systemd timer to renew certificates when they are 2/3rds of the way through their validity period. + +Install the systemd certificate renewal service and timer. + +``` +cat << EOF | sudo tee /etc/systemd/system/cert-renewer@.service +[Unit] +Description=Certificate renewer for %I +After=network-online.target +Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production +StartLimitIntervalSec=0 + +[Service] +Type=oneshot +User=root + +Environment=STEPPATH=/etc/step-ca \ + CERT_LOCATION=/etc/step/certs/%i.crt \ + KEY_LOCATION=/etc/step/certs/%i.key + +; ExecCondition checks if the certificate is ready for renewal, +; based on the exit status of the command. +; (In systemd <242, you can use ExecStartPre= here.) +ExecCondition=/usr/local/bin/step certificate needs-renewal ${CERT_LOCATION} + +; ExecStart renews the certificate, if ExecStartPre was successful. +ExecStart=/usr/local/bin/step ca renew --force ${CERT_LOCATION} ${KEY_LOCATION} + +[Install] +WantedBy=multi-user.target +EOF +``` + +Install the timer: + +``` +cat << EOF | sudo tee /etc/systemd/system/cert-renewer@.timer +[Unit] +Description=Certificate renewal timer for %I +Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production + +[Timer] +Persistent=true + +; Run the timer unit every 5 minutes. +OnCalendar=*:1/5 + +; Always run the timer on time. +AccuracySec=1us + +; Add jitter to prevent a "thundering hurd" of simultaneous certificate renewals. +RandomizedDelaySec=5m + +[Install] +WantedBy=timers.target +EOF +``` + ## Configure certificate renewal for etcd Create and start a certificate renewal timer for etcd: @@ -59,20 +117,98 @@ cat < Remember to run the above commands on each controller node: `controller-0`, `controller-1`, and `controller-2`. From b9bf0744bbeb5aac8dedfdd08acee44aef1b741e Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Wed, 2 Feb 2022 16:39:18 -0800 Subject: [PATCH 06/10] Coverage for all cert renewal --- docs/04-certificate-authority.md | 2 - docs/13-certificate-renewal.md | 74 +++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 17 deletions(-) diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index 99e7673..2c2b2cd 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -38,9 +38,7 @@ sudo mv step_0.18.0/bin/step /usr/local/bin/ Now create a `step` user and the paths for `step-ca`: ``` -{ sudo useradd --system --home /etc/step-ca --shell /bin/false step -} ``` Create a CA configuration folder and generate passwords for the CA root key and the CA provisioner: diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md index 338892b..fb517c3 100644 --- a/docs/13-certificate-renewal.md +++ b/docs/13-certificate-renewal.md @@ -1,6 +1,8 @@ +# Configuring Certificate Renewal + ## Prerequisites -The commands in this lab must be run on each controller instance: `controller-0`, `controller-1`, and `controller-2`. Login to each controller instance using the `gcloud` command. Example: +The commands in this section must be run on every instance: `controller-0`, `controller-1`, `controller-2`, `worker-0`, `worker-1`, and `worker-2`. Login to each instance using the `gcloud` command. Example: ``` gcloud compute ssh controller-0 @@ -8,26 +10,25 @@ gcloud compute ssh controller-0 ## Download certificate management tools -Download the `step` CLI binary and renewal utility for systemd: +Run each command on every node. + +Download the `step` CLI binary: ``` wget -q --show-progress --https-only --timestamping \ - "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" \ - "https://files.smallstep.com/cert-renewer%40.service" \ - "https://files.smallstep.com/cert-renewer%40.timer" + "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" ``` -Install the binary and renewal utility files: +Install the binary: ``` tar -xvf step_linux_0.18.0_amd64.tar.gz sudo mv step_0.18.0/bin/step /usr/local/bin/ -sudo systemctl daemon-reload ``` -### Bootstrapping the CA on your controllers +### Bootstrapping with the CA -Run each command on every node: +Configure the host to trust your Certificate Authority: ``` { @@ -64,17 +65,17 @@ StartLimitIntervalSec=0 Type=oneshot User=root -Environment=STEPPATH=/etc/step-ca \ - CERT_LOCATION=/etc/step/certs/%i.crt \ +Environment=STEPPATH=/etc/step-ca \\ + CERT_LOCATION=/etc/step/certs/%i.crt \\ KEY_LOCATION=/etc/step/certs/%i.key ; ExecCondition checks if the certificate is ready for renewal, ; based on the exit status of the command. ; (In systemd <242, you can use ExecStartPre= here.) -ExecCondition=/usr/local/bin/step certificate needs-renewal ${CERT_LOCATION} +ExecCondition=/usr/local/bin/step certificate needs-renewal \${CERT_LOCATION} ; ExecStart renews the certificate, if ExecStartPre was successful. -ExecStart=/usr/local/bin/step ca renew --force ${CERT_LOCATION} ${KEY_LOCATION} +ExecStart=/usr/local/bin/step ca renew --force \${CERT_LOCATION} \${KEY_LOCATION} [Install] WantedBy=multi-user.target @@ -106,6 +107,16 @@ WantedBy=timers.target EOF ``` +# Controller Certificate Renewal + +## Prerequisites + +The commands in this section must be run on every controller: `controller-0`, `controller-1`, `controller-2`. Login to each instance using the `gcloud` command. Example: + +``` +gcloud compute ssh controller-0 +``` + ## Configure certificate renewal for etcd Create and start a certificate renewal timer for etcd: @@ -194,6 +205,8 @@ sudo systemctl enable --now cert-renewer@kube-apiserver.timer ## Configure service account certificate renewal timer +The service account certificate and key is used by the API server, so we will need to restart it when the certificate file is updated: + ``` sudo mkdir /etc/systemd/system/cert-renewer@kube-service-account.service.d cat < Remember to run the above commands on each controller node: `controller-0`, `controller-1`, and `controller-2`. + +# Worker Certificate Renewal + +## Prerequisites + +The commands in this section must be run on every worker: `worker-0`, `worker-1`, and `worker-2`. Login to each instance using the `gcloud` command. Example: + +``` +gcloud compute ssh worker-0 +``` + +## Configure Certificate Renewal for `kubelet.service` + +Run: + +``` +sudo mkdir /etc/systemd/system/cert-renewer@kubelet.service.d +cat < Remember to run the above commands on each controller node: `worker-0`, `worker-1`, and `worker-2`. From 46eec77fc889ec88c9a496ca5ab2db85729d8b62 Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Wed, 2 Feb 2022 16:50:26 -0800 Subject: [PATCH 07/10] Instructions for worker certificate renewals --- docs/04-certificate-authority.md | 6 ++--- docs/09-bootstrapping-kubernetes-workers.md | 1 + docs/13-certificate-renewal.md | 28 ++++++++++++++++++++- 3 files changed, 31 insertions(+), 4 deletions(-) diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index 2c2b2cd..9dae315 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -428,7 +428,8 @@ Copy the appropriate certificates and private keys to each worker instance: ``` for instance in worker-0 worker-1 worker-2; do - gcloud compute scp ca.pem ${instance}-key.pem ${instance}.pem ${instance}:~/ + gcloud compute scp ca.pem ${instance}-key.pem ${instance}.pem \ + kube-proxy-key.pem kube-proxy.pem ${instance}:~/ done ``` @@ -439,8 +440,7 @@ for instance in controller-0 controller-1 controller-2; do gcloud compute scp ca.pem kubernetes-key.pem kubernetes.pem \ service-account-key.pem service-account.pem \ kube-controller-manager-key.pem kube-controller-manager.pem \ - kube-proxy-key.pem kube-proxy.pem kube-scheduler-key.pem \ - kube-scheduler.pem ${instance}:~/ + kube-scheduler-key.pem kube-scheduler.pem ${instance}:~/ done ``` diff --git a/docs/09-bootstrapping-kubernetes-workers.md b/docs/09-bootstrapping-kubernetes-workers.md index 9958f88..f756dd9 100644 --- a/docs/09-bootstrapping-kubernetes-workers.md +++ b/docs/09-bootstrapping-kubernetes-workers.md @@ -244,6 +244,7 @@ EOF ``` sudo mv kube-proxy.kubeconfig /var/lib/kube-proxy/kubeconfig +sudo mv kube-proxy.pem kube-proxy-key.pem /var/lib/kube-proxy ``` Create the `kube-proxy-config.yaml` configuration file: diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md index fb517c3..e37a00b 100644 --- a/docs/13-certificate-renewal.md +++ b/docs/13-certificate-renewal.md @@ -237,7 +237,7 @@ gcloud compute ssh worker-0 ## Configure Certificate Renewal for `kubelet.service` -Run: +Install the a renewal service that will restart `kubelet.service` when the certificate is renewed: ``` sudo mkdir /etc/systemd/system/cert-renewer@kubelet.service.d @@ -255,4 +255,30 @@ sudo systemctl daemon-reload sudo systemctl enable --now cert-renewer@kubelet.timer ``` +## Configure Certificate Renewal for `kube-proxy.service` + +Install a renewal service that will rebuild the kubeconfig file and restart kube-proxy when the certificate is renewed: + +``` +sudo mkdir /etc/systemd/system/cert-renewer@kube-proxy.service.d +cat < Remember to run the above commands on each controller node: `worker-0`, `worker-1`, and `worker-2`. From c73f1a7d8540829bf09382dac6b994a82e1b2850 Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Wed, 2 Feb 2022 16:52:14 -0800 Subject: [PATCH 08/10] Typofix --- docs/13-certificate-renewal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md index e37a00b..2246621 100644 --- a/docs/13-certificate-renewal.md +++ b/docs/13-certificate-renewal.md @@ -26,7 +26,7 @@ tar -xvf step_linux_0.18.0_amd64.tar.gz sudo mv step_0.18.0/bin/step /usr/local/bin/ ``` -### Bootstrapping with the CA +## Bootstrap with the CA Configure the host to trust your Certificate Authority: From 300f2fc77cea553b21d3524f63313f66833ac71a Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Fri, 4 Feb 2022 08:47:43 -0800 Subject: [PATCH 09/10] Update step tools to v0.18.1 --- docs/04-certificate-authority.md | 13 ++++++------- docs/13-certificate-renewal.md | 6 +++--- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/04-certificate-authority.md b/docs/04-certificate-authority.md index 9dae315..141af37 100644 --- a/docs/04-certificate-authority.md +++ b/docs/04-certificate-authority.md @@ -17,8 +17,8 @@ Download the `step` client and `step-ca` server binaries, and the `jq` command: ``` { wget -q --show-progress --https-only --timestamping \ - "https://dl.step.sm/gh-release/certificates/gh-release-header/v0.18.0/step-ca_linux_0.18.0_amd64.tar.gz" \ - "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" + "https://dl.step.sm/gh-release/certificates/gh-release-header/v0.18.1/step-ca_linux_0.18.1_amd64.tar.gz" \ + "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.1/step_linux_0.18.1_amd64.tar.gz" sudo apt update sudo apt install -y jq } @@ -28,10 +28,10 @@ Install the binaries: ``` { -tar -xvf step-ca_linux_0.18.0_amd64.tar.gz -sudo mv step-ca_0.18.0/bin/step-ca /usr/local/bin/ -tar -xvf step_linux_0.18.0_amd64.tar.gz -sudo mv step_0.18.0/bin/step /usr/local/bin/ +tar -xvf step-ca_linux_0.18.1_amd64.tar.gz +sudo mv step-ca_0.18.1/bin/step-ca /usr/local/bin/ +tar -xvf step_linux_0.18.1_amd64.tar.gz +sudo mv step_0.18.1/bin/step /usr/local/bin/ } ``` @@ -47,7 +47,6 @@ Create a CA configuration folder and generate passwords for the CA root key and { export STEPPATH=/etc/step-ca umask 077 -sudo mkdir -p $(step path)/db < /dev/urandom tr -dc A-Za-z0-9 | head -c40 | sudo tee $(step path)/password > /dev/null < /dev/urandom tr -dc A-Za-z0-9 | head -c40 > provisioner-password umask 002 diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md index 2246621..e38f890 100644 --- a/docs/13-certificate-renewal.md +++ b/docs/13-certificate-renewal.md @@ -16,14 +16,14 @@ Download the `step` CLI binary: ``` wget -q --show-progress --https-only --timestamping \ - "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.0/step_linux_0.18.0_amd64.tar.gz" + "https://dl.step.sm/gh-release/cli/gh-release-header/v0.18.1/step_linux_0.18.1_amd64.tar.gz" ``` Install the binary: ``` -tar -xvf step_linux_0.18.0_amd64.tar.gz -sudo mv step_0.18.0/bin/step /usr/local/bin/ +tar -xvf step_linux_0.18.1_amd64.tar.gz +sudo mv step_0.18.1/bin/step /usr/local/bin/ ``` ## Bootstrap with the CA From 340050478d51fc6497152a0dc7a105f6910f676c Mon Sep 17 00:00:00 2001 From: Carl Tashian Date: Tue, 22 Mar 2022 14:54:26 -0700 Subject: [PATCH 10/10] Fix bug in renewer code --- docs/13-certificate-renewal.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/docs/13-certificate-renewal.md b/docs/13-certificate-renewal.md index e38f890..dca9e1f 100644 --- a/docs/13-certificate-renewal.md +++ b/docs/13-certificate-renewal.md @@ -142,8 +142,7 @@ sudo mkdir /etc/systemd/system/cert-renewer@kube-controller-manager.service.d cat <