kubernetes-the-hard-way/docs/03-etcd.md

167 lines
4.8 KiB
Markdown
Raw Normal View History

2016-07-07 18:06:53 +03:00
# Bootstrapping a H/A etcd cluster
2016-07-07 17:15:59 +03:00
2016-07-07 18:06:53 +03:00
In this lab you will bootstrap a 3 node etcd cluster. The following virtual machines will be used:
2016-07-07 17:15:59 +03:00
2016-09-27 15:23:35 +03:00
* controller0
* controller1
* controller2
2016-07-07 18:06:53 +03:00
2016-07-07 18:25:27 +03:00
## Why
All Kubernetes components are stateless which greatly simplifies managing a Kubernetes cluster. All state is stored
in etcd, which is a database and must be treated specially. To limit the number of compute resource to complete this lab etcd is being installed on the Kubernetes controller nodes, although some people will prefer to run etcd on a dedicated set of machines for the following reasons:
2016-07-07 18:25:27 +03:00
* The etcd lifecycle is not tied to Kubernetes. We should be able to upgrade etcd independently of Kubernetes.
* Scaling out etcd is different than scaling out the Kubernetes Control Plane.
* Prevent other applications from taking up resources (CPU, Memory, I/O) required by etcd.
However, all the e2e tested configurations currently run etcd on the master nodes.
2016-07-07 18:06:53 +03:00
## Provision the etcd Cluster
2016-07-07 17:15:59 +03:00
2016-09-27 15:23:35 +03:00
Run the following commands on `controller0`, `controller1`, `controller2`:
2016-07-07 17:15:59 +03:00
2016-09-27 15:23:35 +03:00
### TLS Certificates
The TLS certificates created in the [Setting up a CA and TLS Cert Generation](02-certificate-authority.md) lab will be used to secure communication between the Kubernetes API server and the etcd cluster. The TLS certificates will also be used to limit access to the etcd cluster using TLS client authentication. Only clients with a TLS certificate signed by a trusted CA will be able to access the etcd cluster.
Copy the TLS certificates to the etcd configuration directory:
2016-07-07 17:15:59 +03:00
```
sudo mkdir -p /etc/etcd/
```
```
2016-09-27 15:23:35 +03:00
sudo cp ca.pem kubernetes-key.pem kubernetes.pem /etc/etcd/
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
### Download and Install the etcd binaries
Download the official etcd release binaries from `coreos/etcd` GitHub project:
2016-07-07 18:06:53 +03:00
2016-07-07 17:15:59 +03:00
```
2017-03-24 05:48:14 +03:00
wget https://github.com/coreos/etcd/releases/download/v3.1.4/etcd-v3.1.4-linux-amd64.tar.gz
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
Extract and install the `etcd` server binary and the `etcdctl` command line client:
2016-07-07 17:15:59 +03:00
```
2017-03-24 05:48:14 +03:00
tar -xvf etcd-v3.1.4-linux-amd64.tar.gz
2016-07-07 17:15:59 +03:00
```
```
2017-03-24 05:48:14 +03:00
sudo mv etcd-v3.1.4-linux-amd64/etcd* /usr/bin/
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
All etcd data is stored under the etcd data directory. In a production cluster the data directory should be backed by a persistent disk. Create the etcd data directory:
2016-07-07 17:15:59 +03:00
```
sudo mkdir -p /var/lib/etcd
```
2016-09-11 06:00:31 +03:00
### Set The Internal IP Address
2016-09-27 15:23:35 +03:00
The internal IP address will be used by etcd to serve client requests and communicate with other etcd peers.
2016-09-11 06:00:31 +03:00
#### GCE
2016-07-07 17:15:59 +03:00
```
2016-09-11 13:07:28 +03:00
INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
2016-07-08 20:26:32 +03:00
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
2016-07-07 17:15:59 +03:00
```
2016-09-11 06:00:31 +03:00
#### AWS
```
2016-09-11 13:07:28 +03:00
INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
2016-09-11 06:00:31 +03:00
```
---
2016-09-27 15:23:35 +03:00
Each etcd member must have a unique name within an etcd cluster. Set the etcd name:
2016-09-11 06:00:31 +03:00
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
ETCD_NAME=controller$(echo $INTERNAL_IP | cut -c 11)
2016-07-07 17:15:59 +03:00
```
2017-03-24 05:48:14 +03:00
The etcd server will be started and managed by systemd. Create the etcd systemd unit file:
2016-09-27 15:23:35 +03:00
2016-07-07 17:15:59 +03:00
```
2017-03-24 05:48:14 +03:00
cat > etcd.service <<EOF
[Unit]
Description=etcd
Documentation=https://github.com/coreos
2016-07-07 17:15:59 +03:00
2017-03-24 05:48:14 +03:00
[Service]
ExecStart=/usr/bin/etcd \\
--name ${ETCD_NAME} \\
--cert-file=/etc/etcd/kubernetes.pem \\
--key-file=/etc/etcd/kubernetes-key.pem \\
--peer-cert-file=/etc/etcd/kubernetes.pem \\
--peer-key-file=/etc/etcd/kubernetes-key.pem \\
--trusted-ca-file=/etc/etcd/ca.pem \\
--peer-trusted-ca-file=/etc/etcd/ca.pem \\
--initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
--listen-peer-urls https://${INTERNAL_IP}:2380 \\
--listen-client-urls https://${INTERNAL_IP}:2379,http://127.0.0.1:2379 \\
--advertise-client-urls https://${INTERNAL_IP}:2379 \\
--initial-cluster-token etcd-cluster-0 \\
--initial-cluster controller0=https://10.240.0.10:2380,controller1=https://10.240.0.11:2380,controller2=https://10.240.0.12:2380 \\
--initial-cluster-state new \\
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
Once the etcd systemd unit file is ready, move it to the systemd system directory:
2016-07-07 17:15:59 +03:00
```
2016-07-08 20:26:32 +03:00
sudo mv etcd.service /etc/systemd/system/
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
Start the etcd server:
2016-07-07 18:06:53 +03:00
2016-07-07 17:15:59 +03:00
```
sudo systemctl daemon-reload
2016-09-27 15:23:35 +03:00
```
```
2016-07-07 17:15:59 +03:00
sudo systemctl enable etcd
2016-09-27 15:23:35 +03:00
```
```
2016-07-07 17:15:59 +03:00
sudo systemctl start etcd
```
2016-07-09 03:14:11 +03:00
2016-07-08 20:26:32 +03:00
### Verification
2016-07-07 17:15:59 +03:00
```
2016-07-08 20:26:32 +03:00
sudo systemctl status etcd --no-pager
2016-07-07 17:15:59 +03:00
```
2016-09-27 15:23:35 +03:00
> Remember to run these steps on `controller0`, `controller1`, and `controller2`
2016-07-09 03:14:31 +03:00
2016-07-08 20:26:32 +03:00
## Verification
2016-07-07 18:06:53 +03:00
2016-07-08 20:26:32 +03:00
Once all 3 etcd nodes have been bootstrapped verify the etcd cluster is healthy:
2016-07-07 17:15:59 +03:00
2016-09-27 15:23:35 +03:00
* On one of the controller nodes run the following command:
2016-07-07 17:15:59 +03:00
```
2017-03-24 05:48:14 +03:00
etcdctl \
--ca-file=/etc/etcd/ca.pem \
--cert-file=/etc/etcd/kubernetes.pem \
--key-file=/etc/etcd/kubernetes-key.pem \
cluster-health
2016-07-07 17:15:59 +03:00
```
```
member 3a57933972cb5131 is healthy: got healthy result from https://10.240.0.12:2379
member f98dc20bce6225a0 is healthy: got healthy result from https://10.240.0.10:2379
member ffed16798470cab5 is healthy: got healthy result from https://10.240.0.11:2379
cluster is healthy
2016-09-11 06:00:31 +03:00
```