Scaling Out a Kubernetes Cluster with more VM nodes

Scaling Out a Kubernetes Cluster with more VM nodes
Page content

Intro

Following up on my last post where I built a Kubernetes cluster with virtual machines now the time has come to scale out the cluster with both master and worker nodes.

This post (and the previous one) is part of my study and preparation towards the Certified Kubernetes Administrator (CKA) exam where hands-on experience is key.

A word of warning: This post should not be used for anything production and it is not a comprehensive write up of Kubernetes

The current state of the nation

In the last post we left off with a Master node and two worker nodes.

We tested the cluster by deploying a simple nginx application which we both scaled to more replicas, and we managed to access it from outside the cluster.

Scale-out the cluster

As mentioned we will add more nodes to the cluster, namely two Master nodes and one more worker node. Three masters is the recommended minimum for a production cluster to support High availability.

I have already prepared my Ubuntu VMs as part of the last post, please reference that if needed.. In short the VMs have been provisioned, and we have installed the required packages as well as the Kubernetes binaries.

Load balancer

Kubernetes documentation reference

Kubernetes haproxy configuration reference

We also provisioned a VM to act as a load balancer for our masters in the previous post. We will start of by configuring that now before installing more nodes.

The load balancer is a haproxy VM that will balance between the three masters’ kube-api-server component which runs on port 6443. If you’ve followed the previous post the VM should be up and running and have haproxy installed so now let’s go ahead and configure it.

To configure the haproxy we will open the /etc/haproxy/haproxy.cfg file

sudo vi /etc/haproxy/haproxy.cfg

I’ll use the example config from the Kubernetes GitHub repo and just modify with the correct address to my master nodes

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
    bind *:6443
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin 
        server kube-a-01 192.168.100.151:6443 check
	    server kube-a-02 192.168.100.152:6443 check
	    server kube-a-03 192.168.100.153:6443 check

Now we’ll restart the haproxy service

sudo systemctl restart haproxy

If we run a status on the service we will get some error messages because our new nodes hasn’t been installed yet

sudo systemctl status haproxy
HAProxy status failed

Now let’s test that the load balancer is responding on our port

HAProxy test

So far so good. Next step is to change the hosts file on our worker nodes to point to the new loadbalancer IP.

In the /etc/hosts file we’ve already added the loadbalancer IP so it’s just a matter of commenting out that line (192.168.100.150 in my case), and comment the IP pointing to node 1 directly (192.168.100.151 in my case). This way we can quickly revert if anything fails

sudo vi /etc/hosts
192.168.100.150 kube-master
#192.168.100.151 kube-master

Now let’s verify from the node that the loadbalancer works on both IP and name

Load balancer test from worker

Remember to change this on all nodes

Finally we’ll do a quick kubectl get nodes to see that there’s no errors on our nodes

kubectl get nodes
Kubernetes node status

Add master nodes

Now we can finally add in our new master nodes!

In the previous post we installed docker and got the Kubernetes binaries. Now we’ll have to run the kubeadm utility to join our nodes to the cluster.

Fetch certificate key, hash and token

Kubernetes documentation reference

As we did in the previous post, when we added the worker nodes, we need to have a token and it’s certificate hash for this command. When it comes to adding more masters (control-plane) we also need the key of the control-plane certificates. This was generated when the cluster was installed, and is uploaded to the kubeadm-certs secret, but we can regenerate this key by re-uploading the certs

sudo kubeadm init phase upload-certs --upload-certs
Upload new control-plane certs

Let’s generate a new discovery token like we did in the previous post

sudo kubeadm token create
Create token

And fetch the certificate hash

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | 
  openssl rsa -pubin -outform der 2>/dev/null | 
  openssl dgst -sha256 -hex | sed 's/^.* //'
Get token cert hash

Join new node to the cluster

Now we have all we need to build our kubadm join command which we will run on our two new master nodes. Per the documentation it is supported to run this in parallel from 1.15 onwards, but since I only have two new nodes I’ll run it one by one

sudo kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<cert hash> --control-plane --certificate-key <cert key>

Note that the difference between this join command and the one used for worker nodes is that for a new master we add the --control-plane and --certificate-key parameters

Join second master

After a while the master have joined the new cluster and we get a success message

Second master joined

We’ll run the three commands on the new master to be able to run kubectl from this node as well

Kubectl config on new master

Let’s test that we can use kubectl from this node by checking the node status

kubectl get nodes
Kubectl status on new master

Remember also to set up bash completion on your nodes

echo 'source <(kubectl completion bash)' >>~/.bashrc
source <(kubectl completion bash)

If all is good we’ll replicate this on the second of our two new masters, and finally check the current status of our cluster

Kubectl status on second new master

Add worker nodes

With those two new masters our cluster consists of three masters and two workers. Now we’ll add a third worker to the cluster. The process is very similar to the process of adding masters, we just leave out the --control-plane and certificate-key parameters from the join command.

We added workers in the previous post so we’ll quickly replicate that. We’ll run the kubeadm join command with our new token and cert hash.

sudo kubeadm join --token <token> <control-plane-host>:<control-plane-port> --discovery-token-ca-cert-hash sha256:<hash>
Added third worker

Let’s check the status of our nodes with the third master

Node status with third worker

To test our new worker we’ll scale up our existing nginx deployment with some new pods.

Deployment status

We’ll add one new replica and check which nodes the pods are running on

Deployment scaled

In my example the pods are spread to all three worker nodes. I have seen that one of the nodes didn’t get that third replica, but when scaling up to four replicas it got a pod. Not sure what caused that, but if you’re testing, it can be wise to test scaling additional replicas to make sure.

Summary

To summarize this post we have seen how we can scale out a Kubernetes cluster with additional master and worker nodes. Again this post is part of my preparations towards the CKA exam and not a production ready cluster.

There’s quite a few manual steps in this process which is fine when preparing for an exam and you want to really work through the setup, but in a normal environment you should look at automating this with something like Ansible or the likes.

For us VMware guys Mark Brookfield (@virtualhobbit) have written a blog post on how to deploy this with vRealize Automation which is something I’m planning to look at later on.

Thanks for reading and reach out if you have any questions or comments.

This page was modified on August 14, 2020: Added missing link