Difference between revisions of "Kubernetes/the-hard-way"

Revision as of 20:20, 8 August 2019

This article will show how to setup Kubernetes The Hard Way, as originally developed by Kelsey Hightower. I will add my own additions, changes, alterations, etc. to the process (and this will be continually expanded upon).

Install the client tools

Note: See here for how to install on other OSes.

In this section, we will install the command line utilities required to complete this tutorial:

• cfssl
• cfssljson
• kubectl

Install CFSSL

The cfssl and cfssljson command line utilities will be used to provision a PKI Infrastructure and generate TLS certificates.

• Download and install cfssl and cfssljson from the cfssl repository:

$ wget -q --show-progress --https-only --timestamping \
    https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 \
    https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

$ chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
$ sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

• Verify cfssl version 1.2.0 or higher is installed:

$ cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6

Note: The cfssljson command line utility does not provide a way to print its version.

Install kubectl

The kubectl command line utility is used to interact with the Kubernetes API Server.

• Download and install kubectl from the official release binaries:

$ K8S_VERSION=$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)
$ curl -LO https://storage.googleapis.com/kubernetes-release/release/${K8S_VERSION}/bin/linux/amd64/kubectl
$ chmod +x kubectl
$ sudo mv kubectl /usr/local/bin/

• Verify kubectl version 1.12.0 or higher is installed:

$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Provisioning compute resources

Networking

Virtual Private Cloud Network (VPC)

In this section, a dedicated Virtual Private Cloud (VPC) network will be setup to host the Kubernetes cluster.

• Create the kubernetes-the-hard-way custom VPC network:

$ gcloud compute networks create kubernetes-the-hard-way --subnet-mode custom
Created [https://www.googleapis.com/compute/v1/projects/<project-name>/global/networks/kubernetes-the-hard-way].

$ gcloud compute networks list --filter="name~'.*hard.*'"
NAME                     SUBNET_MODE  BGP_ROUTING_MODE  IPV4_RANGE  GATEWAY_IPV4
kubernetes-the-hard-way  CUSTOM       REGIONAL

A subnet must be provisioned with an IP address range large enough to assign a private IP address to each node in the Kubernetes cluster.

• Create the kubernetes subnet in the kubernetes-the-hard-way VPC network:

$ gcloud compute networks subnets create kubernetes \
  --network kubernetes-the-hard-way \
  --range 10.240.0.0/24
Created [https://www.googleapis.com/compute/v1/projects/<project-name>/regions/us-west1/subnetworks/kubernetes].

$ gcloud compute networks subnets list --filter="network ~ kubernetes-the-hard-way"
NAME        REGION    NETWORK                  RANGE
kubernetes  us-west1  kubernetes-the-hard-way  10.240.0.0/24

Note: The 10.240.0.0/24 IP address range can host up to 254 compute instances.

Firewall rules

• Create a firewall rule that allows internal communication across all protocols:

$ gcloud compute firewall-rules create kubernetes-the-hard-way-allow-internal \
  --allow tcp,udp,icmp \
  --network kubernetes-the-hard-way \
  --source-ranges 10.240.0.0/24,10.200.0.0/16

• Create a firewall rule that allows external SSH, ICMP, and HTTPS:

$ gcloud compute firewall-rules create kubernetes-the-hard-way-allow-external \
  --allow tcp:22,tcp:6443,icmp \
  --network kubernetes-the-hard-way \
  --source-ranges 0.0.0.0/0

Note: An external load balancer will be used to expose the Kubernetes API Servers to remote clients.

• List the firewall rules in the kubernetes-the-hard-way VPC network:

$ gcloud compute firewall-rules list --filter="network:kubernetes-the-hard-way"
NAME                                    NETWORK                  DIRECTION  PRIORITY  ALLOW                 DENY  DISABLED
kubernetes-the-hard-way-allow-external  kubernetes-the-hard-way  INGRESS    1000      tcp:22,tcp:6443,icmp        False
kubernetes-the-hard-way-allow-internal  kubernetes-the-hard-way  INGRESS    1000      tcp,udp,icmp                False

Kubernetes public IP address

• Allocate a static IP address that will be attached to the external load balancer fronting the Kubernetes API Servers:

$ gcloud compute addresses create kubernetes-the-hard-way \
  --region $(gcloud config get-value compute/region)

• Verify that the kubernetes-the-hard-way static IP address was created in your default compute region:

$ gcloud compute addresses list --filter="name=('kubernetes-the-hard-way')"
NAME                     ADDRESS/RANGE  TYPE      PURPOSE  NETWORK  REGION    SUBNET  STATUS
kubernetes-the-hard-way  XX.XX.XX.XX    EXTERNAL                    us-west1          RESERVED

Compute instances

The compute instances will be provisioned using Ubuntu Server 18.04, which has good support for the containerd container runtime. Each compute instance will be provisioned with a fixed private IP address to simplify the Kubernetes bootstrapping process.

Kubernetes Controllers

• Create three compute instances, which will host the Kubernetes control plane:

for i in 0 1 2; do
  gcloud compute instances create controller-${i} \
    --async \
    --boot-disk-size 200GB \
    --can-ip-forward \
    --image-family ubuntu-1804-lts \
    --image-project ubuntu-os-cloud \
    --machine-type n1-standard-1 \
    --private-network-ip 10.240.0.1${i} \
    --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
    --subnet kubernetes \
    --tags kubernetes-the-hard-way,controller
done

Kubernetes Workers

Each worker instance requires a pod subnet allocation from the Kubernetes cluster CIDR range. The pod subnet allocation will be used to configure container networking further down. The pod-cidr instance metadata will be used to expose pod subnet allocations to compute instances at runtime.

Note: The Kubernetes cluster CIDR range is defined by the Controller Manager's --cluster-cidr flag. The cluster CIDR range will be set to 10.200.0.0/16, which supports 254 subnets.

• Create three compute instances, which will host the Kubernetes worker nodes:

for i in 0 1 2; do
  gcloud compute instances create worker-${i} \
    --async \
    --boot-disk-size 200GB \
    --can-ip-forward \
    --image-family ubuntu-1804-lts \
    --image-project ubuntu-os-cloud \
    --machine-type n1-standard-1 \
    --metadata pod-cidr=10.200.${i}.0/24 \
    --private-network-ip 10.240.0.2${i} \
    --scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
    --subnet kubernetes \
    --tags kubernetes-the-hard-way,worker
done

Verification

• List the compute instances in your default compute zone:

$ gcloud compute instances list --filter="tags:kubernetes-the-hard-way"
NAME          ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP  STATUS
controller-0  us-west1-a  n1-standard-1               10.240.0.10  XX.XX.XX.XX  RUNNING
controller-1  us-west1-a  n1-standard-1               10.240.0.11  XX.XX.XX.XX  RUNNING
controller-2  us-west1-a  n1-standard-1               10.240.0.12  XX.XX.XX.XX  RUNNING
worker-0      us-west1-a  n1-standard-1               10.240.0.20  XX.XX.XX.XX  RUNNING
worker-1      us-west1-a  n1-standard-1               10.240.0.21  XX.XX.XX.XX  RUNNING
worker-2      us-west1-a  n1-standard-1               10.240.0.22  XX.XX.XX.XX  RUNNING

• SSH into the instances:

$ gcloud compute ssh controller-0

Provisioning a CA and Generating TLS Certificates

In this section, we will provision a PKI Infrastructure using CloudFlare's PKI toolkit, cfssl (which we installed above), then use it to bootstrap a Certificate Authority, and generate TLS certificates for the following components: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy.

Certificate Authority

Provision a Certificate Authority that can be used to generate additional TLS certificates.

• Generate the CA configuration file, certificate, and private key:

{

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

}

• Results:

ca-key.pem
ca.pem

Client and Server Certificates

In this section, we will generate client and server certificates for each Kubernetes component and a client certificate for the Kubernetes admin user.

The Admin Client Certificate

• Generate the admin client certificate and private key:

{

cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "system:masters",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin

}

• Results:

admin-key.pem
admin.pem

The Kubelet Client Certificates

Kubernetes uses a special-purpose authorization mode called "Node Authorizer", which specifically authorizes API requests made by Kubelets. In order to be authorized by the Node Authorizer, Kubelets must use a credential that identifies them as being in the system:nodes group, with a username of system:node:<nodeName>. In this section, we will create a certificate for each Kubernetes worker node that meets the Node Authorizer requirements.

• Generate a certificate and private key for each Kubernetes worker node:

for instance in worker-0 worker-1 worker-2; do
cat > ${instance}-csr.json <<EOF
{
  "CN": "system:node:${instance}",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "system:nodes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

EXTERNAL_IP=$(gcloud compute instances describe ${instance} \
  --format 'value(networkInterfaces[0].accessConfigs[0].natIP)')

INTERNAL_IP=$(gcloud compute instances describe ${instance} \
  --format 'value(networkInterfaces[0].networkIP)')

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=${instance},${EXTERNAL_IP},${INTERNAL_IP} \
  -profile=kubernetes \
  ${instance}-csr.json | cfssljson -bare ${instance}
done

• Results:

worker-0-key.pem
worker-0.pem
worker-1-key.pem
worker-1.pem
worker-2-key.pem
worker-2.pem

The Controller Manager Client Certificate

• Generate the kube-controller-manager client certificate and private key:

{

cat > kube-controller-manager-csr.json <<EOF
{
  "CN": "system:kube-controller-manager",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "system:kube-controller-manager",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

}

• Results:

kube-controller-manager-key.pem
kube-controller-manager.pem

The Kube Proxy Client Certificate

• Generate the kube-proxy client certificate and private key:

{

cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "system:node-proxier",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-proxy-csr.json | cfssljson -bare kube-proxy

}

• Results:

kube-proxy-key.pem
kube-proxy.pem

The Scheduler Client Certificate

• Generate the kube-scheduler client certificate and private key:

{

cat > kube-scheduler-csr.json <<EOF
{
  "CN": "system:kube-scheduler",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "system:kube-scheduler",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  kube-scheduler-csr.json | cfssljson -bare kube-scheduler

}

• Results:

kube-scheduler-key.pem
kube-scheduler.pem

The Kubernetes API Server Certificate

The kubernetes-the-hard-way static IP address will be included in the list of subject alternative names for the Kubernetes API Server certificate. This will ensure the certificate can be validated by remote clients.

• Generate the Kubernetes API Server certificate and private key:

{

KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  --region $(gcloud config get-value compute/region) \
  --format 'value(address)')

cat > kubernetes-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=10.32.0.1,10.240.0.10,10.240.0.11,10.240.0.12,${KUBERNETES_PUBLIC_ADDRESS},127.0.0.1,kubernetes.default \
  -profile=kubernetes \
  kubernetes-csr.json | cfssljson -bare kubernetes

}

• Results:

kubernetes-key.pem
kubernetes.pem

The Service Account Key Pair

The Kubernetes Controller Manager leverages a key pair to generate and sign service account tokens as described in the managing service accounts documentation.

• Generate the service-account certificate and private key:

{

cat > service-account-csr.json <<EOF
{
  "CN": "service-accounts",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Seattle",
      "O": "Kubernetes",
      "OU": "Kubernetes The Hard Way",
      "ST": "Washington"
    }
  ]
}
EOF

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  service-account-csr.json | cfssljson -bare service-account

}

• Results:

service-account-key.pem
service-account.pem

Distribute the Client and Server Certificates

• Copy the appropriate certificates and private keys to each worker instance:

for instance in worker-0 worker-1 worker-2; do
  gcloud compute scp ca.pem ${instance}-key.pem ${instance}.pem ${instance}:~/
done

• Copy the appropriate certificates and private keys to each controller instance:

for instance in controller-0 controller-1 controller-2; do
  gcloud compute scp ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
    service-account-key.pem service-account.pem ${instance}:~/
done

Note: The kube-proxy, kube-controller-manager, kube-scheduler, and kubelet client certificates will be used to generate client authentication configuration files in the next section.

Generating the Data Encryption Config and Key

Kubernetes stores a variety of data including cluster state, application configurations, and secrets. Kubernetes supports the ability to encrypt cluster data at rest.

In this section, we will generate an encryption key and an encryption config suitable for encrypting Kubernetes Secrets.

• Create the encryption-config.yaml encryption config file:

cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: $(head -c 32 /dev/urandom | base64 -i -)
      - identity: {}
EOF

• Copy the encryption-config.yaml encryption config file to each controller instance:

for instance in controller-0 controller-1 controller-2; do
  gcloud compute scp encryption-config.yaml ${instance}:~/
done

Bootstrapping the etcd Cluster

Kubernetes components are stateless and store cluster state in etcd. In this section, we will bootstrap a three-node etcd cluster and configure it for high availability and secure remote access.

Prerequisites

The commands in this section must be run on each controller instance: controller-0, controller-1, and controller-2.

Using tmux, split your shell into 3 x panes (ctrl + b ") and then log into each controller instance:

$ gcloud compute ssh controller-0
ctrl + b o
$ gcloud compute ssh controller-1
ctrl + b o
$ gcloud compute ssh controller-2

Now, set tmux to run all subsequent commands (unless otherwise stated) in parallel on all 3 x controller instances (in all 3 x tmux panes):

ctrl + b :
set synchronize-panes on # off
#~OR~
setw synchronize-panes # toggles on/off

Download and Install the etcd Binaries

Download the official etcd release binaries from the coreos/etcd GitHub project:

ETCD_VER=v3.3.13
# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

wget -q --show-progress --https-only --timestamping \
  "${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz"

• Extract and install the etcd server and the etcdctl command line utility:

{
  tar -xvf etcd-${ETCD_VER}-linux-amd64.tar.gz
  sudo mv etcd-${ETCD_VER}-linux-amd64/etcd* /usr/local/bin/
  rm -rf etcd-${ETCD_VER}-linux-amd64*
}

Configure the etcd Server

{
  sudo mkdir -p /etc/etcd /var/lib/etcd
  sudo cp ca.pem kubernetes-key.pem kubernetes.pem /etc/etcd/
}

The instance internal IP address will be used to serve client requests and communicate with etcd cluster peers.

• Retrieve the internal IP address for the current compute instance:

INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)

Each etcd member must have a unique name within an etcd cluster. Set the etcd name to match the hostname of the current compute instance:

$ ETCD_NAME=$(hostname -s)

• Create the etcd.service systemd unit file:

cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos

[Service]
ExecStart=/usr/local/bin/etcd \\
  --name ${ETCD_NAME} \\
  --cert-file=/etc/etcd/kubernetes.pem \\
  --key-file=/etc/etcd/kubernetes-key.pem \\
  --peer-cert-file=/etc/etcd/kubernetes.pem \\
  --peer-key-file=/etc/etcd/kubernetes-key.pem \\
  --trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-trusted-ca-file=/etc/etcd/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth \\
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-peer-urls https://${INTERNAL_IP}:2380 \\
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \\
  --advertise-client-urls https://${INTERNAL_IP}:2379 \\
  --initial-cluster-token etcd-cluster-0 \\
  --initial-cluster controller-0=https://10.240.0.10:2380,controller-1=https://10.240.0.11:2380,controller-2=https://10.240.0.12:2380 \\
  --initial-cluster-state new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

See also

• Kubernetes

External links

• Kubernetes the Hard Way — on GitHub
• CFSSL — CloudFlare's PKI/TLS toolkit on GitHub