diff --git a/README.md b/README.md index d33275a..8df2853 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,9 @@ -# Kubernetes The Hard Way +# Kubernetes The Hard Way on OCI + +This is a translation of the extremely useful [Kubernetes the Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way) +by Kelsey Hightower, using [Oracle Cloud Infrastructure](https://www.oracle.com/cloud/) instead of Google Cloud Platform. + +***** This tutorial walks you through setting up Kubernetes the hard way. This guide is not for people looking for a fully automated command to bring up a Kubernetes cluster. If that's you then check out [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine), or the [Getting Started Guides](https://kubernetes.io/docs/setup). @@ -27,7 +32,7 @@ Kubernetes The Hard Way guides you through bootstrapping a highly available Kube ## Labs -This tutorial assumes you have access to the [Google Cloud Platform](https://cloud.google.com). While GCP is used for basic infrastructure requirements the lessons learned in this tutorial can be applied to other platforms. +This tutorial assumes you have access to [OCI](https://www.oracle.com/cloud/). While OCI is used for basic infrastructure requirements the lessons learned in this tutorial can be applied to other platforms. * [Prerequisites](docs/01-prerequisites.md) * [Installing the Client Tools](docs/02-client-tools.md) diff --git a/docs/01-prerequisites.md b/docs/01-prerequisites.md index 1e47b1b..c9073ce 100644 --- a/docs/01-prerequisites.md +++ b/docs/01-prerequisites.md @@ -1,54 +1,75 @@ # Prerequisites -## Google Cloud Platform +## Oracle Cloud Infrastructure -This tutorial leverages the [Google Cloud Platform](https://cloud.google.com/) to streamline provisioning of the compute infrastructure required to bootstrap a Kubernetes cluster from the ground up. [Sign up](https://cloud.google.com/free/) for $300 in free credits. +This tutorial leverages [OCI](https://www.oracle.com/cloud/) to streamline provisioning of the compute infrastructure required to bootstrap a Kubernetes cluster from the ground up. [Sign up](https://www.oracle.com/cloud/free/) for $300 in free credits. -[Estimated cost](https://cloud.google.com/products/calculator#id=873932bc-0840-4176-b0fa-a8cfd4ca61ae) to run this tutorial: $0.23 per hour ($5.50 per day). +[Estimated cost](https://www.oracle.com/cloud/cost-estimator.html) to run this tutorial: $0.38 per hour ($9.23 per day). -> The compute resources required for this tutorial exceed the Google Cloud Platform free tier. +> The compute resources required for this tutorial exceed the OCI free tier. -## Google Cloud Platform SDK +## OCI CLI -### Install the Google Cloud SDK +### Install the OCI SDK -Follow the Google Cloud SDK [documentation](https://cloud.google.com/sdk/) to install and configure the `gcloud` command line utility. +Follow the OCI CLI [documentation](https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm) to install and configure the `oci` command line utility. -Verify the Google Cloud SDK version is 301.0.0 or higher: +Verify the OCI CLI version is 2.17.0 or higher: ``` -gcloud version +oci --version ``` -### Set a Default Compute Region and Zone +### Capture OCIDs and Generate Required Keys -This tutorial assumes a default compute region and zone have been configured. +Follow the documentation [here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm) to fetch your tenancy and user OCIDs and generate an RSA key pair, which are necessary to use the OCI CLI. -If you are using the `gcloud` command-line tool for the first time `init` is the easiest way to do this: +### Create OCI Config File + +Follow the documentation [here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm) to create an OCI config file `~/.oci/config`. Here's an example of what it will look like: ``` -gcloud init +[DEFAULT] +user=ocid1.user.oc1.. +fingerprint= +key_file=~/.oci/oci_api_key.pem +tenancy=ocid1.tenancy.oc1.. +region=us-ashburn-1 ``` -Then be sure to authorize gcloud to access the Cloud Platform with your Google user credentials: +### Set a Default Region + +The above example uses "us-ashburn-1" as the region, but you can replace this with any available region. For best +performance running the commands from this tutorial, pick a region close to your physical location. To list +the available regions: ``` -gcloud auth login +oci iam region list ``` -Next set a default compute region and compute zone: +### Create a Compartment + +Create yourself an OCI compartment, within which we'll create all the resources in this tutorial. In the +following command, you will need to fill in your tenancy OCID and your OCI [Home Region](https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/managingregions.htm#The). +Your Home Region will be indicated to you when you first create your tenancy. You can also determine it +like [this](https://docs.oracle.com/en-us/iaas/Content/GSG/Reference/faq.htm#How). ``` -gcloud config set compute/region us-west1 +oci iam compartment create --name kubernetes-the-hard-way --description "Kubernetes the Hard Way" \ + --compartment-id --region ``` -Set a default compute zone: +### Set this Compartment as the Default + +Note the compartment `id` from the output of the above command, and create a file `~/.oci/oci_cli_rc` with +the following content: ``` -gcloud config set compute/zone us-west1-c +[DEFAULT] +compartment-id= ``` -> Use the `gcloud compute zones list` command to view additional regions and zones. +From this point on, all `oci` commands we run will target the above compartment. ## Running Commands in Parallel with tmux diff --git a/docs/02-client-tools.md b/docs/02-client-tools.md index 6a89963..7e73cb3 100644 --- a/docs/02-client-tools.md +++ b/docs/02-client-tools.md @@ -1,7 +1,44 @@ # Installing the Client Tools -In this lab you will install the command line utilities required to complete this tutorial: [cfssl](https://github.com/cloudflare/cfssl), [cfssljson](https://github.com/cloudflare/cfssl), and [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl). +In this lab you will install the command line utilities required to complete this tutorial: [jq](https://stedolan.github.io/jq/download/), [cfssl](https://github.com/cloudflare/cfssl), [cfssljson](https://github.com/cloudflare/cfssl), [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl), and set up a few shell functions. +## Install jq + +Install jq: + +### OS X + +``` +curl -o jq -L https://github.com/stedolan/jq/releases/download/jq-1.6/jq-osx-amd64 +``` + +``` +chmod +x jq +``` + +``` +sudo mv jq /usr/local/bin/ +``` + +Some OS X users may experience problems using the pre-built binaries in which case [Homebrew](https://brew.sh) might be a better option: + +``` +brew install jq +``` + +### Linux + +``` +curl -o jq -L https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 +``` + +``` +chmod +x jq +``` + +``` +sudo mv jq /usr/local/bin/ +``` ## Install CFSSL @@ -115,4 +152,34 @@ kubectl version --client Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} ``` +## Shell Helper Functions + +In your terminal, run the following to define a few shell helper functions that we'll use throughout the tutorial: + +``` +function oci-ssh(){ + # Helper function to ssh into a named OCI compute instance + if [ -z "$1" ] + then + echo "Usage: oci-ssh " + else + ocid=$(oci compute instance list --lifecycle-state RUNNING --display-name $1 | jq -r .data[0].id) + ip=$(oci compute instance list-vnics --instance-id $ocid | jq -r '.data[0]["public-ip"]') + ssh -i kubernetes_ssh_rsa ubuntu@$ip $2 + fi +} + +function oci-scp(){ + # Helper function to scp a set of local files to a named OCI compute instance + if [ -z "$3" ] + then + echo "Usage: oci-scp " + else + ocid=$(oci compute instance list --lifecycle-state RUNNING --display-name ${@: (-2):1} | jq -r .data[0].id) + ip=$(oci compute instance list-vnics --instance-id $ocid | jq -r '.data[0]["public-ip"]') + scp -i kubernetes_ssh_rsa "${@:1:$#-2}" ubuntu@$ip:${@: -1} + fi +} +``` + Next: [Provisioning Compute Resources](03-compute-resources.md) diff --git a/docs/03-compute-resources.md b/docs/03-compute-resources.md index 0602cf9..b5d910d 100644 --- a/docs/03-compute-resources.md +++ b/docs/03-compute-resources.md @@ -10,107 +10,81 @@ The Kubernetes [networking model](https://kubernetes.io/docs/concepts/cluster-ad > Setting up network policies is out of scope for this tutorial. -### Virtual Private Cloud Network +### Virtual Cloud Network -In this section a dedicated [Virtual Private Cloud](https://cloud.google.com/compute/docs/networks-and-firewalls#networks) (VPC) network will be setup to host the Kubernetes cluster. +In this section a dedicated [Virtual Cloud Network](https://www.oracle.com/cloud/networking/virtual-cloud-network/) (VCN) network will be setup to host the Kubernetes cluster. -Create the `kubernetes-the-hard-way` custom VPC network: +Create the `kubernetes-the-hard-way` custom VCN: ``` -gcloud compute networks create kubernetes-the-hard-way --subnet-mode custom +VCN_ID=$(oci network vcn create --display-name kubernetes-the-hard-way --dns-label vcn --cidr-block 10.240.0.0/24 | jq -r .data.id) ``` -A [subnet](https://cloud.google.com/compute/docs/vpc/#vpc_networks_and_subnets) must be provisioned with an IP address range large enough to assign a private IP address to each node in the Kubernetes cluster. +A [subnet](https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/managingVCNs_topic-Overview_of_VCNs_and_Subnets.htm#Overview) must be provisioned with an IP address range large enough to assign a private IP address to each node in the Kubernetes cluster. -Create the `kubernetes` subnet in the `kubernetes-the-hard-way` VPC network: +Create the `kubernetes` subnet in the `kubernetes-the-hard-way` VCN, along with a Route Table and Internet Gateway allowing traffic to the internet. ``` -gcloud compute networks subnets create kubernetes \ - --network kubernetes-the-hard-way \ - --range 10.240.0.0/24 +INTERNET_GATEWAY_ID=$(oci network internet-gateway create --vcn-id $VCN_ID --is-enabled true \ + --display-name kubernetes-the-hard-way | jq -r .data.id) +ROUTE_TABLE_ID=$(oci network route-table create --vcn-id $VCN_ID --display-name kubernetes-the-hard-way \ + --route-rules "[{\"cidrBlock\":\"0.0.0.0/0\",\"networkEntityId\":\"$INTERNET_GATEWAY_ID\"}]" | jq -r .data.id) +SUBNET_ID=$(oci network subnet create --display-name kubernetes --dns-label subnet --vcn-id $VCN_ID \ + --cidr-block 10.240.0.0/24 --route-table-id $ROUTE_TABLE_ID | jq -r .data.id) ``` > The `10.240.0.0/24` IP address range can host up to 254 compute instances. -### Firewall Rules - -Create a firewall rule that allows internal communication across all protocols: - -``` -gcloud compute firewall-rules create kubernetes-the-hard-way-allow-internal \ - --allow tcp,udp,icmp \ - --network kubernetes-the-hard-way \ - --source-ranges 10.240.0.0/24,10.200.0.0/16 -``` - -Create a firewall rule that allows external SSH, ICMP, and HTTPS: - -``` -gcloud compute firewall-rules create kubernetes-the-hard-way-allow-external \ - --allow tcp:22,tcp:6443,icmp \ - --network kubernetes-the-hard-way \ - --source-ranges 0.0.0.0/0 -``` - -> An [external load balancer](https://cloud.google.com/compute/docs/load-balancing/network/) will be used to expose the Kubernetes API Servers to remote clients. - -List the firewall rules in the `kubernetes-the-hard-way` VPC network: - -``` -gcloud compute firewall-rules list --filter="network:kubernetes-the-hard-way" -``` - -> output - -``` -NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED -kubernetes-the-hard-way-allow-external kubernetes-the-hard-way INGRESS 1000 tcp:22,tcp:6443,icmp False -kubernetes-the-hard-way-allow-internal kubernetes-the-hard-way INGRESS 1000 tcp,udp,icmp Fals -``` - -### Kubernetes Public IP Address - -Allocate a static IP address that will be attached to the external load balancer fronting the Kubernetes API Servers: - -``` -gcloud compute addresses create kubernetes-the-hard-way \ - --region $(gcloud config get-value compute/region) -``` - -Verify the `kubernetes-the-hard-way` static IP address was created in your default compute region: - -``` -gcloud compute addresses list --filter="name=('kubernetes-the-hard-way')" -``` - -> output - -``` -NAME ADDRESS/RANGE TYPE PURPOSE NETWORK REGION SUBNET STATUS -kubernetes-the-hard-way XX.XXX.XXX.XXX EXTERNAL us-west1 RESERVED -``` +:warning: **Note**: For simplicity and to stay close to the original kubernetes-the-hard-way, we will be using a single subnet, shared between the Kubernetes worker nodes, controller plane nodes, and LoadBalancer. A production-caliber setup would consist of at least: +- A dedicated public subnet for the public LoadBalancer. +- A dedicated private subnet for controller plan nodes. +- A dedicated private subnet for worker nodes. This setup would not allow NodePort access to services. ## Compute Instances The compute instances in this lab will be provisioned using [Ubuntu Server](https://www.ubuntu.com/server) 20.04, which has good support for the [containerd container runtime](https://github.com/containerd/containerd). Each compute instance will be provisioned with a fixed private IP address to simplify the Kubernetes bootstrapping process. +:warning: **Note**: For simplicity in this tutorial, we will be accessing controller and worker nodes over SSH, using public addresses. A production-caliber setup would instead run controller and worker nodes in _private_ subnets, with any direct SSH access done via [Bastions](https://docs.oracle.com/en-us/iaas/Content/Resources/Assets/whitepapers/bastion-hosts.pdf) when required. + +### Create SSH Keys + +Generate an RSA key pair, which we'll use for SSH access to our compute nodes: + +``` +ssh-keygen -b 2048 -t rsa -f kubernetes_ssh_rsa +``` + +Enter a passphrase at the prompt to continue: +``` +Generating public/private rsa key pair. +Enter passphrase (empty for no passphrase): +Enter same passphrase again: +``` + +Results: +``` +kubernetes_ssh_rsa +kubernetes_ssh_rsa.pub +``` + ### Kubernetes Controllers Create three compute instances which will host the Kubernetes control plane: ``` +IMAGE_ID=$(oci compute image list --operating-system "Canonical Ubuntu" --operating-system-version "20.04" | jq -r .data[0].id) for i in 0 1 2; do - gcloud compute instances create controller-${i} \ - --async \ - --boot-disk-size 200GB \ - --can-ip-forward \ - --image-family ubuntu-2004-lts \ - --image-project ubuntu-os-cloud \ - --machine-type e2-standard-2 \ - --private-network-ip 10.240.0.1${i} \ - --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \ - --subnet kubernetes \ - --tags kubernetes-the-hard-way,controller + # Rudimentary spreading of nodes across Availability Domains and Fault Domains + NUM_ADS=$(oci iam availability-domain list | jq -r .data | jq length) + AD_NAME=$(oci iam availability-domain list | jq -r .data[$((i % NUM_ADS))].name) + NUM_FDS=$(oci iam fault-domain list --availability-domain $AD_NAME | jq -r .data | jq length) + FD_NAME=$(oci iam fault-domain list --availability-domain $AD_NAME | jq -r .data[$((i % NUM_FDS))].name) + + oci compute instance launch --display-name controller-${i} --assign-public-ip true \ + --subnet-id $SUBNET_ID --shape VM.Standard.E3.Flex --availability-domain $AD_NAME \ + --fault-domain $FD_NAME --image-id $IMAGE_ID --shape-config '{"memoryInGBs": 8.0, "ocpus": 2.0}' \ + --private-ip 10.240.0.1${i} --freeform-tags \'{"project": "kubernetes-the-hard-way","role":"controller"}' \ + --metadata "{\"ssh_authorized_keys\":\"$(cat kubernetes_ssh_rsa.pub)\"}" done ``` @@ -123,99 +97,86 @@ Each worker instance requires a pod subnet allocation from the Kubernetes cluste Create three compute instances which will host the Kubernetes worker nodes: ``` +IMAGE_ID=$(oci compute image list --operating-system "Canonical Ubuntu" --operating-system-version "20.04" | jq -r .data[0].id) for i in 0 1 2; do - gcloud compute instances create worker-${i} \ - --async \ - --boot-disk-size 200GB \ - --can-ip-forward \ - --image-family ubuntu-2004-lts \ - --image-project ubuntu-os-cloud \ - --machine-type e2-standard-2 \ - --metadata pod-cidr=10.200.${i}.0/24 \ - --private-network-ip 10.240.0.2${i} \ - --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \ - --subnet kubernetes \ - --tags kubernetes-the-hard-way,worker + # Rudimentary spreading of nodes across Availability Domains and Fault Domains + NUM_ADS=$(oci iam availability-domain list | jq -r .data | jq length) + AD_NAME=$(oci iam availability-domain list | jq -r .data[$((i % NUM_ADS))].name) + NUM_FDS=$(oci iam fault-domain list --availability-domain $AD_NAME | jq -r .data | jq length) + FD_NAME=$(oci iam fault-domain list --availability-domain $AD_NAME | jq -r .data[$((i % NUM_FDS))].name) + + oci compute instance launch --display-name worker-${i} --assign-public-ip true \ + --subnet-id $SUBNET_ID --shape VM.Standard.E3.Flex --availability-domain $AD_NAME \ + --fault-domain $FD_NAME --image-id $IMAGE_ID --shape-config '{"memoryInGBs": 8.0, "ocpus": 2.0}' \ + --private-ip 10.240.0.2${i} --freeform-tags '{"project": "kubernetes-the-hard-way","role":"worker"}' \ + --metadata "{\"ssh_authorized_keys\":\"$(cat kubernetes_ssh_rsa.pub)\",\"pod-cidr\":\"10.200.${i}.0/24\"}" \ + --skip-source-dest-check true done ``` ### Verification -List the compute instances in your default compute zone: +List the compute instances in our compartment: ``` -gcloud compute instances list --filter="tags.items=kubernetes-the-hard-way" +oci compute instance list --sort-by DISPLAYNAME --lifecycle-state RUNNING --all | jq -r .data[] | jq '{"display-name","lifecycle-state"}' ``` > output ``` -NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS -controller-0 us-west1-c e2-standard-2 10.240.0.10 XX.XX.XX.XXX RUNNING -controller-1 us-west1-c e2-standard-2 10.240.0.11 XX.XXX.XXX.XX RUNNING -controller-2 us-west1-c e2-standard-2 10.240.0.12 XX.XXX.XX.XXX RUNNING -worker-0 us-west1-c e2-standard-2 10.240.0.20 XX.XX.XXX.XXX RUNNING -worker-1 us-west1-c e2-standard-2 10.240.0.21 XX.XX.XX.XXX RUNNING -worker-2 us-west1-c e2-standard-2 10.240.0.22 XX.XXX.XX.XX RUNNING +{ + "display-name": "controller-0", + "lifecycle-state": "RUNNING" +} +{ + "display-name": "controller-1", + "lifecycle-state": "RUNNING" +} +{ + "display-name": "controller-2", + "lifecycle-state": "RUNNING" +} +{ + "display-name": "worker-0", + "lifecycle-state": "RUNNING" +} +{ + "display-name": "worker-1", + "lifecycle-state": "RUNNING" +} +{ + "display-name": "worker-2", + "lifecycle-state": "RUNNING" +} ``` -## Configuring SSH Access +Rerun the above command until all of the compute instances we created are listed above as "Running". -SSH will be used to configure the controller and worker instances. When connecting to compute instances for the first time SSH keys will be generated for you and stored in the project or instance metadata as described in the [connecting to instances](https://cloud.google.com/compute/docs/instances/connecting-to-instance) documentation. +## Verifying SSH Access -Test SSH access to the `controller-0` compute instances: +Our subnet was created with a default Security List that allows public SSH access, so we can verify at this point that SSH is working: ``` -gcloud compute ssh controller-0 +oci-ssh controller-0 ``` -If this is your first time connecting to a compute instance SSH keys will be generated for you. Enter a passphrase at the prompt to continue: - +The first time SSHing into a node, you'll see something like the following, at which point enter "yes": ``` -WARNING: The public SSH key file for gcloud does not exist. -WARNING: The private SSH key file for gcloud does not exist. -WARNING: You do not have an SSH key for gcloud. -WARNING: SSH keygen will be executed to generate a key. -Generating public/private rsa key pair. -Enter passphrase (empty for no passphrase): -Enter same passphrase again: +The authenticity of host 'XX.XX.XX.XXX (XX.XX.XX.XXX )' can't be established. +ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxx. +Are you sure you want to continue connecting (yes/no/[fingerprint])? ``` -At this point the generated SSH keys will be uploaded and stored in your project: - ``` -Your identification has been saved in /home/$USER/.ssh/google_compute_engine. -Your public key has been saved in /home/$USER/.ssh/google_compute_engine.pub. -The key fingerprint is: -SHA256:nz1i8jHmgQuGt+WscqP5SeIaSy5wyIJeL71MuV+QruE $USER@$HOSTNAME -The key's randomart image is: -+---[RSA 2048]----+ -| | -| | -| | -| . | -|o. oS | -|=... .o .o o | -|+.+ =+=.+.X o | -|.+ ==O*B.B = . | -| .+.=EB++ o | -+----[SHA256]-----+ -Updating project ssh metadata...-Updated [https://www.googleapis.com/compute/v1/projects/$PROJECT_ID]. -Updating project ssh metadata...done. -Waiting for SSH key to propagate. -``` - -After the SSH keys have been updated you'll be logged into the `controller-0` instance: - -``` -Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-1019-gcp x86_64) +Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1029-oracle x86_64) ... ``` Type `exit` at the prompt to exit the `controller-0` compute instance: ``` -$USER@controller-0:~$ exit +ubuntu@controller-0:~$ exit ``` > output @@ -224,4 +185,121 @@ logout Connection to XX.XX.XX.XXX closed ``` +### Security Lists + +For use in later steps of the tutorial, we'll create Security Lists to allow: +- Intra-VCN communication between worker and controller nodes. +- Public access to the NodePort range. +- Public access to the LoadBalancer port. + +``` +INTRA_VCN_SECURITY_LIST_ID=$(oci network security-list create --vcn-id $VCN_ID --display-name intra-vcn --ingress-security-rules '[ +{ + "icmp-options": null, + "is-stateless": true, + "protocol": "all", + "source": "10.240.0.0/24", + "source-type": "CIDR_BLOCK", + "tcp-options": null, + "udp-options": null +}]' --egress-security-rules '[]' | jq -r .data.id) + +WORKER_SECURITY_LIST_ID=$(oci network security-list create --vcn-id $VCN_ID --display-name worker --ingress-security-rules '[ +{ + "icmp-options": null, + "is-stateless": false, + "protocol": "6", + "source": "0.0.0.0/0", + "source-type": "CIDR_BLOCK", + "tcp-options": { + "destination-port-range": { + "max": 32767, + "min": 30000 + }, + "source-port-range": null + }, + "udp-options": null +}]' --egress-security-rules '[]' | jq -r .data.id) + +LB_SECURITY_LIST_ID=$(oci network security-list create --vcn-id $VCN_ID --display-name load-balancer --ingress-security-rules '[ +{ + "icmp-options": null, + "is-stateless": false, + "protocol": "6", + "source": "0.0.0.0/0", + "source-type": "CIDR_BLOCK", + "tcp-options": { + "destination-port-range": { + "max": 6443, + "min": 6443 + }, + "source-port-range": null + }, + "udp-options": null +}]' --egress-security-rules '[]' | jq -r .data.id) +``` + +We'll add these Security Lists to our subnet: +``` +DEFAULT_SECURITY_LIST_ID=$(oci network security-list list --display-name "Default Security List for kubernetes-the-hard-way" | jq -r .data[0].id) +oci network subnet update --subnet-id $SUBNET_ID --security-list-ids \ + "[\"$DEFAULT_SECURITY_LIST_ID\",\"$INTRA_VCN_SECURITY_LIST_ID\",\"$WORKER_SECURITY_LIST_ID\",\"$LB_SECURITY_LIST_ID\"]" --force +``` + +### Firewall Rules + +And similarly, we'll open up the firewall of the worker and controller nodes to allow intra-VCN traffic. + +``` +for instance in controller-0 controller-1 controller-2; do + oci-ssh ${instance} "sudo ufw allow from 10.240.0.0/24;sudo iptables -A INPUT -i ens3 -s 10.240.0.0/24 -j ACCEPT;sudo iptables -F" +done +for instance in worker-0 worker-1 worker-2; do + oci-ssh ${instance} "sudo ufw allow from 10.240.0.0/24;sudo iptables -A INPUT -i ens3 -s 10.240.0.0/24 -j ACCEPT;sudo iptables -F" +done +``` + +### Provision a Network Load Balancer + +> An [OCI Load Balancer](https://docs.oracle.com/en-us/iaas/Content/Balance/Concepts/balanceoverview.htm) will be used to expose the Kubernetes API Servers to remote clients. + +Create the Load Balancer: + +``` +LOADBALANCER_ID=$(oci lb load-balancer create --display-name kubernetes-the-hard-way \ + --shape-name 100Mbps --wait-for-state SUCCEEDED --subnet-ids "[\"$SUBNET_ID\"]" | jq -r .data.id) +``` + +Create a Backend Set, with Backends for the our 3 controller nodes: +``` +cat > backends.json < ${instance}-csr.json < ${instance}-csr.json < kubernetes-csr.json < The `/healthz` API server endpoint does not require authentication by default. @@ -215,8 +216,7 @@ sudo apt-get install -y nginx ``` cat > kubernetes.default.svc.cluster.local < The compute instances created in this tutorial will not have permission to complete this section. **Run the following commands from the same machine used to create the compute instances**. - - -### Provision a Network Load Balancer - -Create the external load balancer network resources: - -``` -{ - KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \ - --region $(gcloud config get-value compute/region) \ - --format 'value(address)') - - gcloud compute http-health-checks create kubernetes \ - --description "Kubernetes Health Check" \ - --host "kubernetes.default.svc.cluster.local" \ - --request-path "/healthz" - - gcloud compute firewall-rules create kubernetes-the-hard-way-allow-health-check \ - --network kubernetes-the-hard-way \ - --source-ranges 209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 \ - --allow tcp - - gcloud compute target-pools create kubernetes-target-pool \ - --http-health-check kubernetes - - gcloud compute target-pools add-instances kubernetes-target-pool \ - --instances controller-0,controller-1,controller-2 - - gcloud compute forwarding-rules create kubernetes-forwarding-rule \ - --address ${KUBERNETES_PUBLIC_ADDRESS} \ - --ports 6443 \ - --region $(gcloud config get-value compute/region) \ - --target-pool kubernetes-target-pool -} -``` +Now we'll verify external connectivity to our cluster via the external Load Balancer ### Verification > The compute instances created in this tutorial will not have permission to complete this section. **Run the following commands from the same machine used to create the compute instances**. -Retrieve the `kubernetes-the-hard-way` static IP address: +Retrieve the `kubernetes-the-hard-way` Load Balancer public IP address: ``` -KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \ - --region $(gcloud config get-value compute/region) \ - --format 'value(address)') +KUBERNETES_PUBLIC_ADDRESS=$(oci lb load-balancer list --all | jq '.data[] | select(."display-name"=="kubernetes-the-hard-way")' | jq -r '."ip-addresses"[0]."ip-address"') ``` Make a HTTP request for the Kubernetes version info: diff --git a/docs/09-bootstrapping-kubernetes-workers.md b/docs/09-bootstrapping-kubernetes-workers.md index 289b9fa..9d42f2b 100644 --- a/docs/09-bootstrapping-kubernetes-workers.md +++ b/docs/09-bootstrapping-kubernetes-workers.md @@ -4,16 +4,18 @@ In this lab you will bootstrap three Kubernetes worker nodes. The following comp ## Prerequisites -The commands in this lab must be run on each worker instance: `worker-0`, `worker-1`, and `worker-2`. Login to each worker instance using the `gcloud` command. Example: +The commands in this lab must be run on each worker instance: `worker-0`, `worker-1`, and `worker-2`. Login to each worker instance using our `oci-ssh` shell function. Example: ``` -gcloud compute ssh worker-0 +oci-ssh worker-0 ``` ### Running commands in parallel with tmux [tmux](https://github.com/tmux/tmux/wiki) can be used to run commands on multiple compute instances at the same time. See the [Running commands in parallel with tmux](01-prerequisites.md#running-commands-in-parallel-with-tmux) section in the Prerequisites lab. +**Note**: please import the shell functions defined [here](02-client-tools.md#shell-functions) in each tmux window/pane, as we will make use of them. + ## Provisioning a Kubernetes Worker Node Install the OS dependencies: @@ -21,7 +23,7 @@ Install the OS dependencies: ``` { sudo apt-get update - sudo apt-get -y install socat conntrack ipset + sudo apt-get -y install socat conntrack ipset jq } ``` @@ -37,7 +39,7 @@ Verify if swap is enabled: sudo swapon --show ``` -If output is empthy then swap is not enabled. If swap is enabled run the following command to disable swap immediately: +If output is empty then swap is not enabled. If swap is enabled run the following command to disable swap immediately: ``` sudo swapoff -a @@ -45,6 +47,17 @@ sudo swapoff -a > To ensure swap remains off after reboot consult your Linux distro documentation. +### Set net.bridge.bridge-nf-call-iptables=1 + +To allow pods to access service endpoints whose backend pods run on the same node, run: + +``` +sudo modprobe br-netfilter +sudo sysctl -w net.bridge.bridge-nf-call-iptables=1 +``` + +See [here](https://github.com/kelseyhightower/kubernetes-the-hard-way/issues/561) for reference. + ### Download and Install Worker Binaries ``` @@ -90,8 +103,7 @@ Install the worker binaries: Retrieve the Pod CIDR range for the current compute instance: ``` -POD_CIDR=$(curl -s -H "Metadata-Flavor: Google" \ - http://metadata.google.internal/computeMetadata/v1/instance/attributes/pod-cidr) +POD_CIDR=$(curl -H "Authorization: Bearer Oracle" -L http://169.254.169.254/opc/v2/instance | jq -r '.metadata["pod-cidr"]') ``` Create the `bridge` network configuration file: @@ -224,6 +236,7 @@ Requires=containerd.service [Service] ExecStart=/usr/local/bin/kubelet \\ + --hostname-override=${HOSTNAME}.subnet.vcn.oraclevcn.com \\ --config=/var/lib/kubelet/kubelet-config.yaml \\ --container-runtime=remote \\ --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\ @@ -297,8 +310,7 @@ EOF List the registered Kubernetes nodes: ``` -gcloud compute ssh controller-0 \ - --command "kubectl get nodes --kubeconfig admin.kubeconfig" +oci-ssh controller-0 "kubectl get nodes --kubeconfig admin.kubeconfig" ``` > output diff --git a/docs/10-configuring-kubectl.md b/docs/10-configuring-kubectl.md index 888013b..9ec2ddb 100644 --- a/docs/10-configuring-kubectl.md +++ b/docs/10-configuring-kubectl.md @@ -12,9 +12,7 @@ Generate a kubeconfig file suitable for authenticating as the `admin` user: ``` { - KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \ - --region $(gcloud config get-value compute/region) \ - --format 'value(address)') +KUBERNETES_PUBLIC_ADDRESS=$(oci lb load-balancer list --all | jq '.data[] | select(."display-name"=="kubernetes-the-hard-way")' | jq -r '."ip-addresses"[0]."ip-address"') kubectl config set-cluster kubernetes-the-hard-way \ --certificate-authority=ca.pem \ diff --git a/docs/11-pod-network-routes.md b/docs/11-pod-network-routes.md index 415db25..1c39d05 100644 --- a/docs/11-pod-network-routes.md +++ b/docs/11-pod-network-routes.md @@ -8,14 +8,16 @@ In this lab you will create a route for each worker node that maps the node's Po ## The Routing Table -In this section you will gather the information required to create routes in the `kubernetes-the-hard-way` VPC network. +In this section you will gather the information required to create routes in the `kubernetes-the-hard-way` VCN. Print the internal IP address and Pod CIDR range for each worker instance: ``` for instance in worker-0 worker-1 worker-2; do - gcloud compute instances describe ${instance} \ - --format 'value[separator=" "](networkInterfaces[0].networkIP,metadata.items[0].value)' + NODE_ID=$(oci compute instance list --lifecycle-state RUNNING --display-name $instance | jq -r .data[0].id) + PRIVATE_IP=$(oci compute instance list-vnics --instance-id $NODE_ID | jq -r '.data[0]["private-ip"]') + POD_CIDR=$(oci compute instance list --lifecycle-state RUNNING --display-name $instance | jq -r '.data[0].metadata["pod-cidr"]') + echo "$PRIVATE_IP $POD_CIDR" done ``` @@ -29,32 +31,50 @@ done ## Routes -Create network routes for each worker instance: +Here, we'll update our Route Table to include, for each worker node, a route from the worker node's pod CIDR to the worker node's private address: ``` -for i in 0 1 2; do - gcloud compute routes create kubernetes-route-10-200-${i}-0-24 \ - --network kubernetes-the-hard-way \ - --next-hop-address 10.240.0.2${i} \ - --destination-range 10.200.${i}.0/24 -done -``` +{ + ROUTE_TABLE_ID=$(oci network route-table list --display-name kubernetes-the-hard-way --vcn-id $VCN_ID | jq -r .data[0].id) -List the routes in the `kubernetes-the-hard-way` VPC network: - -``` -gcloud compute routes list --filter "network: kubernetes-the-hard-way" -``` - -> output - -``` -NAME NETWORK DEST_RANGE NEXT_HOP PRIORITY -default-route-6be823b741087623 kubernetes-the-hard-way 0.0.0.0/0 default-internet-gateway 1000 -default-route-cebc434ce276fafa kubernetes-the-hard-way 10.240.0.0/24 kubernetes-the-hard-way 0 -kubernetes-route-10-200-0-0-24 kubernetes-the-hard-way 10.200.0.0/24 10.240.0.20 1000 -kubernetes-route-10-200-1-0-24 kubernetes-the-hard-way 10.200.1.0/24 10.240.0.21 1000 -kubernetes-route-10-200-2-0-24 kubernetes-the-hard-way 10.200.2.0/24 10.240.0.22 1000 + # Fetch worker-0's private IP OCID + NODE_ID=$(oci compute instance list --lifecycle-state RUNNING --display-name worker-0 | jq -r .data[0].id) + VNIC_ID=$(oci compute instance list-vnics --instance-id $NODE_ID | jq -r '.data[0]["id"]') + PRIVATE_IP_WORKER_0=$(oci network private-ip list --vnic-id $VNIC_ID | jq -r '.data[0]["id"]') + # Fetch worker-1's private IP OCID + NODE_ID=$(oci compute instance list --lifecycle-state RUNNING --display-name worker-1 | jq -r .data[0].id) + VNIC_ID=$(oci compute instance list-vnics --instance-id $NODE_ID | jq -r '.data[0]["id"]') + PRIVATE_IP_WORKER_1=$(oci network private-ip list --vnic-id $VNIC_ID | jq -r '.data[0]["id"]') + # Fetch worker-2's private IP OCID + NODE_ID=$(oci compute instance list --lifecycle-state RUNNING --display-name worker-2 | jq -r .data[0].id) + VNIC_ID=$(oci compute instance list-vnics --instance-id $NODE_ID | jq -r '.data[0]["id"]') + PRIVATE_IP_WORKER_2=$(oci network private-ip list --vnic-id $VNIC_ID | jq -r '.data[0]["id"]') + + INTERNET_GATEWAY_ID=$(oci network internet-gateway list --vcn-id $VCN_ID | jq -r '.data[0]["id"]') + + oci network route-table update --rt-id $ROUTE_TABLE_ID --force --route-rules "[ + { + \"destination\": \"0.0.0.0/0\", + \"destination-type\": \"CIDR_BLOCK\", + \"network-entity-id\": \"$INTERNET_GATEWAY_ID\" + }, + { + \"destination\": \"10.200.0.0/24\", + \"destination-type\": \"CIDR_BLOCK\", + \"network-entity-id\": \"$PRIVATE_IP_WORKER_0\" + }, + { + \"destination\": \"10.200.1.0/24\", + \"destination-type\": \"CIDR_BLOCK\", + \"network-entity-id\": \"$PRIVATE_IP_WORKER_1\" + }, + { + \"destination\": \"10.200.2.0/24\", + \"destination-type\": \"CIDR_BLOCK\", + \"network-entity-id\": \"$PRIVATE_IP_WORKER_2\" + } +]" +} ``` Next: [Deploying the DNS Cluster Add-on](12-dns-addon.md) diff --git a/docs/13-smoke-test.md b/docs/13-smoke-test.md index ab14fe5..16a6128 100644 --- a/docs/13-smoke-test.md +++ b/docs/13-smoke-test.md @@ -16,8 +16,8 @@ kubectl create secret generic kubernetes-the-hard-way \ Print a hexdump of the `kubernetes-the-hard-way` secret stored in etcd: ``` -gcloud compute ssh controller-0 \ - --command "sudo ETCDCTL_API=3 etcdctl get \ +oci-ssh controller-0 \ + "sudo ETCDCTL_API=3 etcdctl get \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/etcd/ca.pem \ --cert=/etc/etcd/kubernetes.pem \ @@ -181,19 +181,11 @@ NODE_PORT=$(kubectl get svc nginx \ --output=jsonpath='{range .spec.ports[0]}{.nodePort}') ``` -Create a firewall rule that allows remote access to the `nginx` node port: - -``` -gcloud compute firewall-rules create kubernetes-the-hard-way-allow-nginx-service \ - --allow=tcp:${NODE_PORT} \ - --network kubernetes-the-hard-way -``` - Retrieve the external IP address of a worker instance: ``` -EXTERNAL_IP=$(gcloud compute instances describe worker-0 \ - --format 'value(networkInterfaces[0].accessConfigs[0].natIP)') +NODE_ID=$(oci compute instance list --lifecycle-state RUNNING --display-name worker-0 | jq -r .data[0].id) +EXTERNAL_IP=$(oci compute instance list-vnics --instance-id $NODE_ID | jq -r '.data[0]["public-ip"]') ``` Make an HTTP request using the external IP address and the `nginx` node port: diff --git a/docs/14-cleanup.md b/docs/14-cleanup.md index 07be407..89493ea 100644 --- a/docs/14-cleanup.md +++ b/docs/14-cleanup.md @@ -7,50 +7,51 @@ In this lab you will delete the compute resources created during this tutorial. Delete the controller and worker compute instances: ``` -gcloud -q compute instances delete \ - controller-0 controller-1 controller-2 \ - worker-0 worker-1 worker-2 \ - --zone $(gcloud config get-value compute/zone) +for instance in controller-0 controller-1 controller-2 worker-0 worker-1 worker-2; do + NODE_ID=$(oci compute instance list --lifecycle-state RUNNING --display-name $instance | jq -r .data[0].id) + oci compute instance terminate --instance-id $NODE_ID --force +done ``` ## Networking -Delete the external load balancer network resources: +Delete the Load Balancer: ``` { - gcloud -q compute forwarding-rules delete kubernetes-forwarding-rule \ - --region $(gcloud config get-value compute/region) - - gcloud -q compute target-pools delete kubernetes-target-pool - - gcloud -q compute http-health-checks delete kubernetes - - gcloud -q compute addresses delete kubernetes-the-hard-way + LOAD_BALANCER_ID=$(oci lb load-balancer list --display-name kubernetes-the-hard-way | jq -r .data[0].id) + oci lb load-balancer delete --load-balancer-id $LOAD_BALANCER_ID --force --wait-for-state SUCCEEDED } ``` -Delete the `kubernetes-the-hard-way` firewall rules: - -``` -gcloud -q compute firewall-rules delete \ - kubernetes-the-hard-way-allow-nginx-service \ - kubernetes-the-hard-way-allow-internal \ - kubernetes-the-hard-way-allow-external \ - kubernetes-the-hard-way-allow-health-check -``` - -Delete the `kubernetes-the-hard-way` network VPC: +Delete all resources within the `kubernetes-the-hard-way` VCN: ``` { - gcloud -q compute routes delete \ - kubernetes-route-10-200-0-0-24 \ - kubernetes-route-10-200-1-0-24 \ - kubernetes-route-10-200-2-0-24 + VCN_ID=$(oci network vcn list --display-name kubernetes-the-hard-way | jq -r .data[0].id) - gcloud -q compute networks subnets delete kubernetes + SUBNET_ID=$(oci network subnet list --display-name kubernetes --vcn-id $VCN_ID | jq -r .data[0].id) + oci network subnet delete --subnet-id $SUBNET_ID --force - gcloud -q compute networks delete kubernetes-the-hard-way + ROUTE_TABLE_ID=$(oci network route-table list --display-name kubernetes-the-hard-way --vcn-id $VCN_ID | jq -r .data[0].id) + oci network route-table delete --rt-id $ROUTE_TABLE_ID --force + + INTERNET_GATEWAY_ID=$(oci network internet-gateway list --display-name kubernetes-the-hard-way --vcn-id $VCN_ID | jq -r .data[0].id) + oci network internet-gateway delete --ig-id $INTERNET_GATEWAY_ID --force + + SECURITY_LIST_ID=$(oci network security-list list --vcn-id $VCN_ID --display-name intra-vcn | jq -r .data[0].id) + oci network security-list delete --security-list-id $SECURITY_LIST_ID --force + + SECURITY_LIST_ID=$(oci network security-list list --vcn-id $VCN_ID --display-name load-balancer | jq -r .data[0].id) + oci network security-list delete --security-list-id $SECURITY_LIST_ID --force + + SECURITY_LIST_ID=$(oci network security-list list --vcn-id $VCN_ID --display-name worker | jq -r .data[0].id) + oci network security-list delete --security-list-id $SECURITY_LIST_ID --force } ``` + +And finally, the VCN itself: + +``` +oci network vcn delete --vcn-id $VCN_ID --force +``` \ No newline at end of file