From 28bb2663c42c44089ce2b03315c9ee856e32ffde Mon Sep 17 00:00:00 2001 From: Jonathan Carter Date: Sat, 14 Jan 2017 10:00:06 -0800 Subject: [PATCH 1/5] Adding Azure --- README.md | 8 ++++++- docs/01-infrastructure.md | 3 ++- docs/02-certificate-authority.md | 26 ++++++++++++++++++-- docs/03-etcd.md | 8 ++++++- docs/04-kubernetes-controller.md | 29 ++++++++++++++++++++++ docs/05-kubernetes-worker.md | 1 - docs/06-kubectl.md | 10 +++++++- docs/07-network.md | 41 ++++++++++++++++++++++++++++++++ docs/09-smoke-test.md | 23 ++++++++++++++++++ docs/10-cleanup.md | 6 +++++ 10 files changed, 148 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index fc07bb9..d9be0e2 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,7 @@ This tutorial is optimized for learning, which means taking the long route to he * [Google Compute Engine](https://cloud.google.com/compute) * [Amazon EC2](https://aws.amazon.com/ec2) +* [Microsoft Azure VMs](https://azure.microsoft.com/en-us/services/virtual-machines/?b=16.51b) > The results of this tutorial should not be viewed as production ready, and may receive limited support from the community, but don't let that prevent you from learning! @@ -47,16 +48,21 @@ AWS * The us-west-2 region will be used +Azure + +* The westus region will be used + ## Platforms This tutorial assumes you have access to one of the following: * [Google Cloud Platform](https://cloud.google.com) and the [Google Cloud SDK](https://cloud.google.com/sdk/) (125.0.0+) * [Amazon Web Services](https://aws.amazon.com), the [AWS CLI](https://aws.amazon.com/cli) (1.10.63+), and [jq](https://stedolan.github.io/jq) (1.5+) +* [Microsoft Azure](https://azure.com) and the [Azure CLI](https://github.com/azure/azure-cli) ## Labs -While GCP or AWS will be used for basic infrastructure needs, the things learned in this tutorial apply to every platform. +While GCP, AWS or Azure will be used for basic infrastructure needs, the things learned in this tutorial apply to every platform. * [Cloud Infrastructure Provisioning](docs/01-infrastructure.md) * [Setting up a CA and TLS Cert Generation](docs/02-certificate-authority.md) diff --git a/docs/01-infrastructure.md b/docs/01-infrastructure.md index bd4ae69..6c5c9c1 100644 --- a/docs/01-infrastructure.md +++ b/docs/01-infrastructure.md @@ -1,8 +1,9 @@ # Cloud Infrastructure Provisioning -Kubernetes can be installed just about anywhere physical or virtual machines can be run. In this lab we are going to focus on [Google Cloud Platform](https://cloud.google.com/) and [Amazon Web Services](https://aws.amazon.com). +Kubernetes can be installed just about anywhere physical or virtual machines can be run. In this lab we are going to focus on [Google Cloud Platform](https://cloud.google.com/), [Amazon Web Services](https://aws.amazon.com) and [Microsoft Azure](https://azure.com). This lab will walk you through provisioning the compute instances required for running a H/A Kubernetes cluster. * [Cloud Infrastructure Provisioning - Google Cloud Platform](01-infrastructure-gcp.md) * [Cloud Infrastructure Provisioning - Amazon Web Services](01-infrastructure-aws.md) +* [Cloud Infrastructure Provisioning - Microsoft Azure](01-infrastructure-azure.md) \ No newline at end of file diff --git a/docs/02-certificate-authority.md b/docs/02-certificate-authority.md index b8facb2..fb187a0 100644 --- a/docs/02-certificate-authority.md +++ b/docs/02-certificate-authority.md @@ -38,7 +38,6 @@ chmod +x cfssljson_darwin-amd64 sudo mv cfssljson_darwin-amd64 /usr/local/bin/cfssljson ``` - ### Linux ``` @@ -137,6 +136,13 @@ KUBERNETES_PUBLIC_ADDRESS=$(aws elb describe-load-balancers \ jq -r '.LoadBalancerDescriptions[].DNSName') ``` +#### Azure + +```shell +KUBERNETES_PUBLIC_ADDRESS=$(az network public-ip show -g kubernetes \ + -n kubernetes-pip --query "ipAddress" -otsv) +``` + --- Create the `kubernetes-csr.json` file: @@ -229,7 +235,7 @@ done The following command will: * Extract the public IP address for each Kubernetes host * Copy the TLS certificates and keys to each Kubernetes host using `scp` - + ``` for host in ${KUBERNETES_HOSTS[*]}; do PUBLIC_IP_ADDRESS=$(aws ec2 describe-instances \ @@ -239,3 +245,19 @@ for host in ${KUBERNETES_HOSTS[*]}; do ubuntu@${PUBLIC_IP_ADDRESS}:~/ done ``` + +### Azure + +The following command will: + * Extract the public IP address for each Kubernetes host + * Copy the TLS certificates and keys to each Kubernetes host using `scp` + +```shell +for host in ${KUBERNETES_HOSTS[*]}; do + PUBLIC_IP_ADDRESS=$(az network public-ip show -g kubernetes \ + -n ${host}-pip --query "ipAddress" -otsv) + + scp ca.pem kubernetes-key.pem kubernetes.pem \ + $(whoami)@${PUBLIC_IP_ADDRESS}:~/ +done +``` \ No newline at end of file diff --git a/docs/03-etcd.md b/docs/03-etcd.md index f64271a..3f8e824 100644 --- a/docs/03-etcd.md +++ b/docs/03-etcd.md @@ -107,6 +107,12 @@ INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \ INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) ``` +#### Azure + +```shell +INTERNAL_IP=$(ip addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}') +``` + --- Each etcd member must have a unique name within an etcd cluster. Set the etcd name: @@ -167,4 +173,4 @@ member 3a57933972cb5131 is healthy: got healthy result from https://10.240.0.12: member f98dc20bce6225a0 is healthy: got healthy result from https://10.240.0.10:2379 member ffed16798470cab5 is healthy: got healthy result from https://10.240.0.11:2379 cluster is healthy -``` +``` \ No newline at end of file diff --git a/docs/04-kubernetes-controller.md b/docs/04-kubernetes-controller.md index 05c1e88..e894a97 100644 --- a/docs/04-kubernetes-controller.md +++ b/docs/04-kubernetes-controller.md @@ -138,6 +138,12 @@ INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \ INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) ``` +#### Azure + +```shell +INTERNAL_IP=$(ip addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}') +``` + --- Create the systemd unit file: @@ -340,3 +346,26 @@ aws elb register-instances-with-load-balancer \ --load-balancer-name kubernetes \ --instances ${CONTROLLER_0_INSTANCE_ID} ${CONTROLLER_1_INSTANCE_ID} ${CONTROLLER_2_INSTANCE_ID} ``` + +### Azure + +```shell +az network lb probe create -g kubernetes \ + -n kubernetes-apiserver-check \ + --lb-name kubernetes-lb \ + --protocol http \ + --port 8080 \ + --path /healthz +``` + +```shell +az network lb rule create -g kubernetes \ + -n kubernetes-apiserver-rule \ + --protocol tcp \ + --lb-name kubernetes-lb \ + --frontend-ip-name LoadBalancerFrontEnd \ + --frontend-port 6443 \ + --backend-pool-name kubernetes-lb-pool \ + --backend-port 6443 \ + --probe-name kubernetes-apiserver-check +``` \ No newline at end of file diff --git a/docs/05-kubernetes-worker.md b/docs/05-kubernetes-worker.md index 52ac597..c6c8128 100644 --- a/docs/05-kubernetes-worker.md +++ b/docs/05-kubernetes-worker.md @@ -15,7 +15,6 @@ Kubernetes worker nodes are responsible for running your containers. All Kuberne Some people would like to run workers and cluster services anywhere in the cluster. This is totally possible, and you'll have to decide what's best for your environment. - ## Provision the Kubernetes Worker Nodes Run the following commands on `worker0`, `worker1`, `worker2`: diff --git a/docs/06-kubectl.md b/docs/06-kubectl.md index c75b17b..44b9036 100644 --- a/docs/06-kubectl.md +++ b/docs/06-kubectl.md @@ -36,6 +36,14 @@ KUBERNETES_PUBLIC_ADDRESS=$(aws elb describe-load-balancers \ --load-balancer-name kubernetes | \ jq -r '.LoadBalancerDescriptions[].DNSName') ``` + +#### Azure + +```shell +KUBERNETES_PUBLIC_ADDRESS=$(az network public-ip show -g kubernetes \ + -n kubernetes-pip --query "ipAddress" -otsv) +``` + --- Recall the token we setup for the admin user: @@ -95,4 +103,4 @@ NAME STATUS AGE worker0 Ready 7m worker1 Ready 5m worker2 Ready 2m -``` +``` \ No newline at end of file diff --git a/docs/07-network.md b/docs/07-network.md index 5b99809..31e6657 100644 --- a/docs/07-network.md +++ b/docs/07-network.md @@ -118,3 +118,44 @@ aws ec2 create-route \ --destination-cidr-block 10.200.2.0/24 \ --instance-id ${WORKER_2_INSTANCE_ID} ``` + +### Azure + +```shell +az network route-table create -g kubernetes \ + -n kubernetes-routes +``` + +```shell +az network vnet subnet update -g kubernetes \ + -n kubernetes-subnet \ + --vnet-name kubernetes-vnet \ + --route-table kubernetes-routes +``` + +```shell +az network route-table route create -g kubernetes \ + -n kubernetes-route-10-200-0-0-24 \ + --route-table-name kubernetes-routes \ + --address-prefix 10.200.0.0/24 \ + --next-hop-ip-address 10.240.0.20 \ + --next-hop-type VirtualAppliance +``` + +```shell +az network route-table route create -g kubernetes \ + -n kubernetes-route-10-200-1-0-24 \ + --route-table-name kubernetes-routes \ + --address-prefix 10.200.1.0/24 \ + --next-hop-ip-address 10.240.0.21 \ + --next-hop-type VirtualAppliance +``` + +```shell +az network route-table route create -g kubernetes \ + -n kubernetes-route-10-200-2-0-24 \ + --route-table-name kubernetes-routes \ + --address-prefix 10.200.2.0/24 \ + --next-hop-ip-address 10.240.0.22 \ + --next-hop-type VirtualAppliance +``` \ No newline at end of file diff --git a/docs/09-smoke-test.md b/docs/09-smoke-test.md index a812108..cdc818f 100644 --- a/docs/09-smoke-test.md +++ b/docs/09-smoke-test.md @@ -79,6 +79,29 @@ NODE_PUBLIC_IP=$(aws ec2 describe-instances \ jq -j '.Reservations[].Instances[].PublicIpAddress') ``` +#### Azure + +```shell +az network nsg rule create -g kubernetes \ + -n kubernetes-allow-nginx \ + --access allow \ + --destination-address-prefix '*' \ + --destination-port-range ${NODE_PORT} \ + --direction inbound \ + --nsg-name kubernetes-nsg \ + --protocol tcp \ + --source-address-prefix '*' \ + --source-port-range '*' \ + --priority 1002 +``` + +Grab the `EXTERNAL_IP` for one of the worker nodes: + +``` +NODE_PUBLIC_IP=$(gcloud compute instances describe worker0 \ + --format 'value(networkInterfaces[0].accessConfigs[0].natIP)') +``` + --- Test the nginx service using cURL: diff --git a/docs/10-cleanup.md b/docs/10-cleanup.md index 452b58a..63d97c6 100644 --- a/docs/10-cleanup.md +++ b/docs/10-cleanup.md @@ -205,3 +205,9 @@ DHCP_OPTION_SET_ID=$(aws ec2 describe-dhcp-options \ aws ec2 delete-dhcp-options \ --dhcp-options-id ${DHCP_OPTION_SET_ID} ``` + +## GCP + +```shell +az group delete -n kubernetes +``` \ No newline at end of file From fe3a3070a554ca1261444d5faf50778a052b37c0 Mon Sep 17 00:00:00 2001 From: Jonathan Carter Date: Sat, 14 Jan 2017 10:02:55 -0800 Subject: [PATCH 2/5] Fixing CLI reference --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d9be0e2..b766003 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ This tutorial assumes you have access to one of the following: * [Google Cloud Platform](https://cloud.google.com) and the [Google Cloud SDK](https://cloud.google.com/sdk/) (125.0.0+) * [Amazon Web Services](https://aws.amazon.com), the [AWS CLI](https://aws.amazon.com/cli) (1.10.63+), and [jq](https://stedolan.github.io/jq) (1.5+) -* [Microsoft Azure](https://azure.com) and the [Azure CLI](https://github.com/azure/azure-cli) +* [Microsoft Azure](https://azure.com) and the [Azure CLI 2.0](https://github.com/azure/azure-cli) ## Labs From 25e4cdd35882acce10f52ad8060111c77bde44ea Mon Sep 17 00:00:00 2001 From: Jonathan Carter Date: Sat, 14 Jan 2017 10:03:31 -0800 Subject: [PATCH 3/5] Adding Azure provisioning --- docs/01-infrastructure-azure.md | 175 ++++++++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 docs/01-infrastructure-azure.md diff --git a/docs/01-infrastructure-azure.md b/docs/01-infrastructure-azure.md new file mode 100644 index 0000000..5d7159d --- /dev/null +++ b/docs/01-infrastructure-azure.md @@ -0,0 +1,175 @@ +# Cloud Infrastructure Provisioning - Microsoft Azure + +This lab will walk you through provisioning the compute instances required for running a H/A Kubernetes cluster. A total of 6 virtual machines will be created. + +The guide assumes you've installed the [Azure CLI 2.0](https://github.com/azure/azure-cli#installation), and will be creating resources in the `westus` region, within a resource group named `kubernetes`. To create this resource group, simply run the following command: + +```shell +az group create -n kubernetes -l westus +``` + +After completing this guide you should have the following compute instances: + +```shell +az vm list --query "[].{name:name,provisioningState:provisioningState}" +``` + +```shell +Name ProvisioningState +----------- ------------------- +controller0 Succeeded +controller1 Succeeded +controller2 Succeeded +worker0 Succeeded +worker1 Succeeded +worker2 Succeeded +``` + +> All machines will be provisioned with fixed private IP addresses to simplify the bootstrap process. + +To make our Kubernetes control plane remotely accessible, a public IP address will be provisioned and assigned to a load balancer that will sit in front of the 3 Kubernetes controllers. + +## Networking + +Create a virtual network and subnet for the Kubernetes cluster: + +```shell +az network vnet create -g kubernetes \ + -n kubernetes-vnet \ + --address-prefix 10.240.0.0/16 \ + --subnet-name kubernetes-subnet +``` + +### Firewall Rules + +Create a firewall ("network security group"), assign it to the subnet, and configure it to allow the necessary incoming traffic: + +```shell +az network nsg create -g kubernetes -n kubernetes-nsg +``` + +```shell +az network vnet subnet update -g kubernetes \ + -n kubernetes-subnet \ + --vnet-name kubernetes-vnet \ + --network-security-group kubernetes-nsg +``` + +```shell +az network nsg rule create -g kubernetes \ + -n kubernetes-allow-ssh \ + --access allow \ + --destination-address-prefix '*' \ + --destination-port-range 22 \ + --direction inbound \ + --nsg-name kubernetes-nsg \ + --protocol tcp \ + --source-address-prefix '*' \ + --source-port-range '*' \ + --priority 1000 +``` + +```shell +az network nsg rule create -g kubernetes \ + -n kubernetes-allow-api-server \ + --access allow \ + --destination-address-prefix '*' \ + --destination-port-range 6443 \ + --direction inbound \ + --nsg-name kubernetes-nsg \ + --protocol tcp \ + --source-address-prefix '*' \ + --source-port-range '*' \ + --priority 1001 +``` + +```shell +az network nsg rule list -g kubernetes --nsg-name kubernetes-nsg --query "[].{Name:name, Port:destinationPortRange}" +``` + +``` +Name Port +--------------------- ------ +kube-allow-ssh 22 +kube-allow-api-server 6443 +``` + +### Kubernetes Public Address + +Create a public IP address that will be used by remote clients to connect to the Kubernetes control plane: + +```shell +az network lb create -g kubernetes \ + -n kubernetes-lb \ + --backend-pool-name kubernetes-lb-pool \ + --public-ip-address kubernetes-pip \ + --public-ip-address-allocation static +``` + +## Provision Virtual Machines + +All the VMs in this lab will be provisioned using Ubuntu 16.04 mainly because it runs a newish Linux Kernel that has good support for Docker. + +### Virtual Machines + +#### Kubernetes Controllers + +```shell +az vm availability-set create -g kubernetes -n controller-as +``` + +```shell +for num in {0..2}; do + echo "[Controller ${num}] Creating public IP..." + az network public-ip create -n controller${num}-pip -g kubernetes > /dev/null + + echo "[Controller ${num}] Creating NIC..." + az network nic create -g kubernetes \ + -n controller${num}-nic \ + --private-ip-address 10.240.0.1${num} \ + --public-ip-address controller${num}-pip \ + --vnet kubernetes-vnet \ + --subnet kubernetes-subnet \ + --ip-forwarding \ + --lb-name kubernetes-lb \ + --lb-address-pools kubernetes-lb-pool > /dev/null + + echo "[Controller ${num}] Creating VM..." + az vm create -g kubernetes \ + -n controller${num} \ + --image Canonical:UbuntuServer:16.04.0-LTS:16.04.201609210 \ + --nics controller${num}-nic \ + --availability-set controller-as \ + --nsg '' > /dev/null +done +``` + +#### Kubernetes Workers + +```shell +az vm availability-set create -g kubernetes -n worker-as +``` + +```shell +for num in {0..2}; do + echo "[Worker ${num}] Creating public IP..." + az network public-ip create -n worker${num}-pip -g kubernetes > /dev/null + + echo "[Worker ${num}] Creating NIC..." + az network nic create -g kubernetes \ + -n worker${num}-nic \ + --private-ip-address 10.240.0.2${num} \ + --public-ip-address worker${num}-pip \ + --vnet kubernetes-vnet \ + --subnet kubernetes-subnet \ + --ip-forwarding > /dev/null + + echo "[Worker ${num}] Creating VM..." + az vm create -g kubernetes \ + -n worker${num} \ + --image Canonical:UbuntuServer:16.04.0-LTS:16.04.201609210 \ + --nics worker${num}-nic \ + --availability-set worker-as \ + --nsg '' > /dev/null +done +``` \ No newline at end of file From 256125ecf8196ecd342ac3cf4a440fe99b4c49ae Mon Sep 17 00:00:00 2001 From: Jonathan Carter Date: Sat, 14 Jan 2017 10:18:32 -0800 Subject: [PATCH 4/5] Fixing typos --- docs/04-kubernetes-controller.md | 1 - docs/09-smoke-test.md | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/04-kubernetes-controller.md b/docs/04-kubernetes-controller.md index e894a97..9e48e85 100644 --- a/docs/04-kubernetes-controller.md +++ b/docs/04-kubernetes-controller.md @@ -237,7 +237,6 @@ sed -i s/INTERNAL_IP/$INTERNAL_IP/g kube-controller-manager.service sudo mv kube-controller-manager.service /etc/systemd/system/ ``` - ``` sudo systemctl daemon-reload sudo systemctl enable kube-controller-manager diff --git a/docs/09-smoke-test.md b/docs/09-smoke-test.md index cdc818f..f361624 100644 --- a/docs/09-smoke-test.md +++ b/docs/09-smoke-test.md @@ -98,8 +98,8 @@ az network nsg rule create -g kubernetes \ Grab the `EXTERNAL_IP` for one of the worker nodes: ``` -NODE_PUBLIC_IP=$(gcloud compute instances describe worker0 \ - --format 'value(networkInterfaces[0].accessConfigs[0].natIP)') +NODE_PUBLIC_IP=$(az network public-ip show -g kubernetes \ + -n worker0-pip --query "ipAddress" -otsv) ``` --- From 9f31aca0e4724844bd23113b3a423b02c0b8afc3 Mon Sep 17 00:00:00 2001 From: Jonathan Carter Date: Sun, 22 Jan 2017 14:27:15 -0800 Subject: [PATCH 5/5] Fixing header --- docs/10-cleanup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/10-cleanup.md b/docs/10-cleanup.md index 63d97c6..8289480 100644 --- a/docs/10-cleanup.md +++ b/docs/10-cleanup.md @@ -206,8 +206,8 @@ aws ec2 delete-dhcp-options \ --dhcp-options-id ${DHCP_OPTION_SET_ID} ``` -## GCP +## Azure ```shell az group delete -n kubernetes -``` \ No newline at end of file +```