mirrors
/
kubernetes-the-hard-way
mirror of https://github.com/kelseyhightower/kubernetes-the-hard-way
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
kubernetes-the-hard-way/docs/01-infrastructure-azure.md

16 KiB
Raw Blame History

Cloud Infrastructure Provisioning - Azure

This lab will walk you through provisioning the compute instances required for running a H/A Kubernetes cluster. A total of 10 virtual machines will be created.

The guide assumes you'll be creating resources in the West Us region as a single Azure Resource Manager resource group.

All machines will be provisioned with fixed private IP addresses to simplify the bootstrap process.

The cluster VMs are only accessible via a jump box (a VM with publicly accessible ssh endpoint). The workers machines are exposed via external load balancer that carries both an public IP and public FQDN.

Variables

Change the following values as needed.

# dns for jumpbox is <jumpboxDnsLabel>.westus.cloudapp.azure.com
jumpboxDnsLabel="the-hard-way-jumpbox" 

# dns for workers is <workersDnsLabel>.westus.cloudapp.azure.com
workersDnsLabel="the-hard-way" 

#storage account used by jumpbox + controllers + Etcd VMs
controlPlaneStorageAccount="thehardwaycsa"

#storage account used by workers VMs
workersStorageAccount="thehardwaywsa"

# all vms are using ubunut 16.4 LTS 
#(Publisher)
imageUrn="Canonical:UbuntuServer:16.04.0-LTS:latest"

Create Resource Group

azure group create \
	--name the-hard-way \
	--location "West Us"

Networking

Create Routing Table

azure network route-table create \
	--resource-group the-hard-way \
	--name the-hard-way-rtable \
	--location "West Us"

Create Network Security Group (NSG)

azure network nsg create \
	--resource-group the-hard-way \
	--name the-hard-way-nsg \
	--location "West Us"

Create NSG rule allowing SSH to the jumpbox

azure network nsg rule create \
	--resource-group the-hard-way \
	--nsg-name the-hard-way-nsg \
	--name allow-ssh-jumpbox \
	--protocol tcp \
	--access allow  \
	--destination-address-prefix 10.0.0.5 \
	--destination-port-range 22 \
	--priority 100 \
	--direction inbound

Create VNET + Subnet

Cluster VNET

azure network vnet create \
	--resource-group the-hard-way \
	--name the-hard-way-net \
	--address-prefixes 10.0.0.0/8 \
	--location "West Us"

Create subnets

# Azure UDR "user defined routes" in custom routing tables 
# routes traffic leaving the subnet.
# Workers & pods (IPs) have to be in two separate subnets

azure network vnet subnet create \
	--resource-group the-hard-way \
	--vnet-name the-hard-way-net \
	--name kubernetes-mgmt \
	--address-prefix 10.0.0.0/16


azure network vnet subnet create \
	--resource-group the-hard-way \
	--vnet-name the-hard-way-net \
	--name kubernetes \
	--address-prefix 10.224.0.0/11

Associate the routing table and NSG to Kubernetes/-mgmt subnets

azure network vnet subnet set \
	--resource-group the-hard-way \
	--vnet-name the-hard-way-net \
	--name kubernetes-mgmt \
	--network-security-group-name the-hard-way-nsg \
	--route-table-name the-hard-way-rtable


azure network vnet subnet set \
	--resource-group the-hard-way \
	--vnet-name the-hard-way-net \
	--name kubernetes \
	--network-security-group-name the-hard-way-nsg \
	--route-table-name the-hard-way-rtable

Create public IP + DNS label for the jumpbox

azure network public-ip create \
	--resource-group the-hard-way \
	--name the-hard-way-jumpbox  \
	--allocation-method Dynamic \
	--domain-name-label $jumpboxDnsLabel \
	--location "West Us"

Virtual Machines

Create SSH keys (Used by All VMs)

mkdir keys
ssh-keygen -t rsa -f ./keys/cluster

Storage Accounts

Create storage account for control plane VMs (Etcd & Controllers)

azure storage account create $controlPlaneStorageAccount \
	--resource-group the-hard-way \
	--kind storage \
	--sku-name LRS \
	--location "West Us"

Create storage account for workers VMs

azure storage account create $workersStorageAccount \
	--resource-group the-hard-way \
	--kind storage \
	--sku-name LRS \
	--location "West Us"

Jump Box

Create NIC (Private IP + Public IP + FQDN)

azure network nic create \
	--resource-group the-hard-way \
	--name jumpbox-nic \
	--private-ip-address "10.0.0.5" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes-mgmt  \
	--public-ip-name the-hard-way-jumpbox \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
	--name jumpbox \
	--vm-size Standard_A1 \
	--nic-name jumpbox-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes-mgmt  \
	--os-type linux \
	--image-urn $imageUrn \
	--storage-account-name $controlPlaneStorageAccount \
	--storage-account-container-name vhds \
	--os-disk-vhd jumpbox.vhd \
	--admin-username thehardway \
	--ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Etcd

Etcd 0

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name etcd-0-nic \
	--private-ip-address "10.240.0.10" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
	--name etcd0 \
	--vm-size Standard_D4 \
	--nic-name etcd-0-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
	--os-type linux \
	--image-urn $imageUrn \
	--storage-account-name $controlPlaneStorageAccount \
	--storage-account-container-name vhds \
	--os-disk-vhd etcd-0.vhd \
	--admin-username thehardway \
	--ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Etcd 1

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name etcd-1-nic \
	--private-ip-address "10.240.0.11" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
	--name etcd1 \
	--vm-size Standard_D4 \
	--nic-name etcd-1-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
	--os-type linux \
	--image-urn $imageUrn \
	--storage-account-name $controlPlaneStorageAccount \
	--storage-account-container-name vhds \
	--os-disk-vhd etcd-1.vhd \
	--admin-username thehardway \
	--ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Etcd 2

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name etcd-2-nic \
	--private-ip-address "10.240.0.12" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
	--name etcd2 \
	--vm-size Standard_D4 \
	--nic-name etcd-2-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
	--os-type linux \
	--image-urn $imageUrn \
	--storage-account-name $controlPlaneStorageAccount \
	--storage-account-container-name vhds \
	--os-disk-vhd etcd-2.vhd \
	--admin-username thehardway \
	--ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Kubernetes Controllers

Controllers Internal Load Balancer

Create load balancer

azure network lb create \
	--resource-group the-hard-way \
	--name the-hard-way-clb \
	--location "West Us"

Create & assign the front-end private IP to the internal load balancer

azure network lb frontend-ip create \
	--resource-group the-hard-way \
	--name the-hard-way-cfe  \
	--lb-name the-hard-way-clb \
	--private-ip-address "10.240.0.4" \
	--subnet-vnet-name the-hard-way-net \
	--subnet-name kubernetes

Create a backend address pool for the load balancer

clbbackendPoolId=$(azure network lb address-pool create \
	--resource-group the-hard-way \
	--lb-name the-hard-way-clb \
	--name backend-pool \
	--json | jq -r '.id')

Create controllers availability set

azure availset create \
	--resource-group the-hard-way \
	--name controllers-availset \
	--location "West Us"

Controller 0

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name controller-0-nic \
	--private-ip-address "10.240.0.20" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--lb-address-pool-ids $clbbackendPoolId \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
    --name controller0 \
    --vm-size Standard_D4 \
    --nic-name controller-0-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
	--availset-name controllers-availset \
    --os-type linux \
    --image-urn $imageUrn \
    --storage-account-name $controlPlaneStorageAccount \
    --storage-account-container-name vhds \
    --os-disk-vhd controller-0.vhd \
    --admin-username thehardway \
    --ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Controller 1

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name controller-1-nic \
	--private-ip-address "10.240.0.21" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--lb-address-pool-ids $clbbackendPoolId \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
    --name controller1 \
    --vm-size Standard_D4 \
    --nic-name controller-1-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
	--availset-name controllers-availset \
    --os-type linux \
    --image-urn $imageUrn \
    --storage-account-name $controlPlaneStorageAccount \
    --storage-account-container-name vhds \
    --os-disk-vhd controller-1.vhd \
    --admin-username thehardway \
    --ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Controller 2

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name controller-2-nic \
	--private-ip-address "10.240.0.22" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--lb-address-pool-ids $clbbackendPoolId \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
    --name controller2 \
    --vm-size Standard_D4 \
    --nic-names controller-2-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
	--availset-name controllers-availset \
    --os-type linux \
    --image-urn $imageUrn \
    --storage-account-name $controlPlaneStorageAccount \
    --storage-account-container-name vhds \
    --os-disk-vhd controller-2.vhd \
    --admin-username thehardway \
    --ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Kubernetes Workers

Workers External Load Balancer

Create public IP + DNS label for workers ingestion external load balancer

azure network public-ip create \
	--resource-group the-hard-way \
	--name the-hard-way-workers  \
	--allocation-method Dynamic \
	--domain-name-label $workersDnsLabel \
	--location "West Us"

Create load balancer

azure network lb create \
	--resource-group the-hard-way \
	--name the-hard-way-lb \
	--location "West Us"

Assign the front-end public IP + FQDN to the load balancer

azure network lb frontend-ip create \
	--resource-group the-hard-way \
	--name the-hard-way-fe  \
	--lb-name the-hard-way-lb \
	--public-ip-name the-hard-way-workers

Create a backend address pool for the load balancer

wlbbackendPoolId=$(azure network lb address-pool create \
	--resource-group the-hard-way \
	--lb-name the-hard-way-lb \
	--name backend-pool \
	--json | jq -r '.id')

Create Workers Availablity set

azure availset create \
	--resource-group the-hard-way \
	--name workers-availset \
	--location "West Us"

Worker 0

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name worker-0-nic \
	--private-ip-address "10.240.0.30" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--enable-ip-forwarding "true" \
	--lb-address-pool-ids $wlbbackendPoolId \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
    --name worker0 \
    --vm-size Standard_D4 \
    --nic-name worker-0-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
    --availset-name workers-availset \
	--os-type linux \
    --image-urn $imageUrn \
    --storage-account-name $workersStorageAccount \
    --storage-account-container-name vhds \
    --os-disk-vhd worker-0.vhd \
    --admin-username thehardway \
    --ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Worker 1

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name worker-1-nic \
	--private-ip-address "10.240.0.31" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--enable-ip-forwarding "true" \
	--lb-address-pool-ids $wlbbackendPoolId \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
    --name worker1 \
    --vm-size Standard_D4 \
    --nic-name worker-1-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
    --availset-name workers-availset \
	--os-type linux \
    --image-urn $imageUrn \
    --storage-account-name $workersStorageAccount \
    --storage-account-container-name vhds \
    --os-disk-vhd worker-1.vhd \
    --admin-username thehardway \
    --ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Worker 2

Create Nic

azure network nic create \
	--resource-group the-hard-way \
	--name worker-2-nic \
	--private-ip-address "10.240.0.32" \
	--subnet-vnet-name the-hard-way-net  \
	--subnet-name kubernetes  \
	--enable-ip-forwarding "true" \
	--lb-address-pool-ids $wlbbackendPoolId \
	--location "West Us"

Create VM

azure vm create \
	--resource-group the-hard-way \
    --name worker2 \
    --vm-size Standard_D4 \
    --nic-name worker-2-nic \
	--vnet-name the-hard-way-net  \
	--vnet-subnet-name kubernetes  \
    --availset-name workers-availset \
	--os-type linux \
    --image-urn $imageUrn \
    --storage-account-name $workersStorageAccount \
    --storage-account-container-name vhds \
    --os-disk-vhd worker-2.vhd \
    --admin-username thehardway \
    --ssh-publickey-file ./keys/cluster.pub \
	--location "West US"

Verify

azure vm list --resource-group the-hard-way

Expected Output

info:    Executing command vm list
+ Getting virtual machines                                                     
data:    ResourceGroupName  Name          ProvisioningState  PowerState  Location  Size       
data:    -----------------  ------------  -----------------  ----------  --------  -----------
data:    the-hard-way       controller-0  Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       controller-1  Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       controller-2  Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       etcd-0        Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       etcd-1        Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       etcd-2        Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       jumpbox       Succeeded          VM running  westus    Standard_A1
data:    the-hard-way       worker-0      Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       worker-1      Succeeded          VM running  westus    Standard_D4
data:    the-hard-way       worker-2      Succeeded          VM running  westus    Standard_D4
info:    vm list command OK

Using The Jumpbox

Connect to Jumpbox

ssh -i ./keys/cluster \
	thehardway@$jumpboxDnsLabel.westus.cloudapp.azure.com

Copy the cluster private key to jumpbox

scp -i ./keys/cluster \
	./keys/cluster \
	thehardway@$jumpboxDnsLabel.westus.cloudapp.azure.com:~/cluster

Connecting to Other VMs

# on the jumpbox 
#connect to the second controller

ssh -i ./cluster \
	thehardway@10.240.0.31  

#or
ssh -i ./cluster \
	thehardway@controller-1