Tanzu Backup and Restore with Standalone Velero
In this post we will take a look at backup and restore of workloads inside of vSphere with Tanzu. In this setup where we have vSphere with Tanzu setup with vSphere networking, we will use the Standalone Velero and Restic solution as opposed to the more integrated Velero operator that can be utilized when using NSX-T as the networking stack.
Backing up and restoring vSphere with Tanzu
First a note on backups of a vSphere with Tanzu environment..
vSphere with Tanzu consists of multiple components. Currently there's no solution that does a complete backup of all these components in one go (and maybe we'll never have one).
We have vCenter it self, and it's configuration, that will have it's own backup process, you'll have the Load Balancer if using the vSphere networking stack that should have it's backup process, NSX-T will have it's own process, and you'll have the Tanzu stuff which is handled differently based on what type of object (and a few other things).
VMware provides a table of the different components and the Tools that can be used which explains in a high-level what considerations we must take when implementing a backup and restore strategy
Note that, as mentioned, there are differences in the plugins/tools that can be used based on if we're using vSphere networking or NSX-T in our environment. The latter uses a Velero Plugin for vSphere which is more integrated than the standalone version we'll have to use without NSX-T, but this will also tie it to Tanzu and backups cannot necessarily be ported to other clusters. For a run through of using the Velero plugin, check out this blog post by Cormac Hogan
The focus of this post will be on backup and restore of workloads running in vSphere with Tanzu, i.e. what is running inside a Tanzu Kubernetes Cluster (TKC).
The environment used in my setup consists of a cluster of three hosts running on vSphere 7.0 U2. There's a couple of vSphere Namespaces configured, and there's also a couple of Tanzu Kubernetes Clusters (guest clusters) running in the environment. The load balancer used in the environment is the NSX Advanced Load Balancer (Avi Vantage).
The tool used for doing backup and restore inside the TKC clusters is Velero. This is an open-source, Cloud Native Computing Foundation backed project for performing disaster recovery of Kubernetes cluster resources. It is also used for migrating resources and persistent volumes.
Velero supports a few storage providers, but again since the vSphere provider requires NSX-T as the networking stack we'll go for an object store and in this setup we'll use MinIO as the object store provider
Note that to run the Velero commands we'll see later on in this post we'll have to have the Velero CLI installed. And be aware that to be eligible for support from VMware we'll have to use the VMware signed binary. Check the VMware documentation for more information
MinIO can be installed in a container, on bare-metal or on virtual machines. In this setup we'll install it on a Ubuntu 20.04 VM.
VMware provides documentation on how to configure it, but we'll go through it step-by-step with a few additions and clarifications.
First we'll download the MinIO executable
We'll create a directory for storing the MinIO data
We'll start MinIO manually first, later on we'll create a service for it
1chmod +x minio 2./minio server /MINIO-DATA
The output will give us the root user and password which we'll need. Note that this can be set with environment variables, which also is true for the console endpoint port. In my lab environment I'll leave it with the defaults. In a production environment this should taken care of.
We can verify the MinIO server by navigating to the console address (if you use the 9000 port you'll be redirected). Use the credentials recorded from the console output to log in
Let's stop the server and prepare a service for it.
First we'll create a user to run the service
1useradd -M minio-user
Set the minio-user as the owner of the data directory
1chown -R minio-user:minio-user /MINIO-DATA
Copy the executable to /usr/local/bin
1cp minio /usr/local/bin
Create a service file
Add the following to the service file
1[Unit] 2Description=MinIO 3Documentation=https://docs.min.io 4Wants=network-online.target 5After=network-online.target 6AssertFileIsExecutable=/usr/local/bin/minio 7 8[Service] 9WorkingDirectory=/usr/local/ 10 11User=minio-user 12Group=minio-user 13ProtectProc=invisible 14 15ExecStart=/usr/local/bin/minio server /MINIO-DATA 16 17# Let systemd restart this service always 18Restart=always 19 20# Specifies the maximum file descriptor number that can be opened by this process 21LimitNOFILE=65536 22 23# Specifies the maximum number of threads this process can create 24TasksMax=infinity 25 26# Disable timeout logic and wait until process is stopped 27TimeoutStopSec=infinity 28SendSIGKILL=no 29 30[Install] 31WantedBy=multi-user.target
Copy the service file to /etc/systemd/system
1cp minio.service /etc/systemd/system
Reload service daemon
Start the MinIO service and check that it's running
1systemctl start minio 2systemctl status minio
Verify that you can access the MinIO UI and let's create a user account and a bucket to be used for our Tanzu backups!
Create MinIO user and bucket
Note that this could easily be done with the MinIO CLI tools, but we'll use the UI for demonstration purposes
First we'll create a user named
velero and give it
readwrite access. We could (and in prod should) restrict it to only the buckets used for backups, but for now I'm find with a generic access
Now, we'll create a Bucket for our Tanzu backups.
Before you go ahead and create the buckets, you might put some considerations into our bucket structure.
In this post I'm doing a Velero install per TKC cluster so a bucket per cluster could be a strategy. Likewise, if you want to utilize the MinIO server for more than just backups then maybe you want to separate backups from other data and will end up with a specific backup bucket per cluster. In short, do some thinking before creating buckets for all of your clusters!
In this case I'm creating one bucket per cluster and I'll give the bucket the same name as the TKC cluster
Also, verify that the bucket access is R/W
And lastly that the user account we created has access to it
Now we can carry on with installing Velero and Restic on our Tanzu Kubernetes Cluster
First let's take a quick look on the environment. We have two vSphere namespaces in this setup as we saw earlier in this blog post, and in one of the namespaces we have two TKC clusters
Let's install Velero in the
First we'll change to/verify that we are working with the correct context. The Velero CLI will automatically work with the current context
1kubectl config use-context tkc-cluster-2
Let's also create a credentials file for the Velero installer to use where we specify the details for the user we created earlier
1[default] 2aws_access_key_id = velero 3aws_secret_access_key = <secret-key-for-velero-user>
I'm saving this in a file called
Note that the VMware documentation doesn't include that
[default]header in the credentials file. In my setup I got error messages like "Error getting a backup store" and "NoCredentialProviders: no valid providers in chain" if I left that header out. I got a clue from this discussion on Github on how to solve it
Now we're ready to install Velero and Restic in our TKC cluster. (Make sure you change the bucket, secret-file, s3Url and publicUrl parameters to reflect your environment)
1velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.0.0 --bucket tkc-cluster-2 --secret-file ./credentials-minio --use-volume-snapshots=false --use-restic --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.100.41:9000,publicUrl=http://192.168.100.41:9000
After a few minutes we can run the
kubectl logs deployment/velero -n velero command to check the status of the install. The message we want to see is the
Backup storage location valid, marking as available
With Velero installed we can try to run a backup in our cluster.
Create a backup with Velero
First, let's check out a simple workload that are running in our cluster
Now we'll create a backup
1velero backup create my-app-backup
Over in our MinIO server we can check our bucket and see that our backup objects have been created there
Delete app and do a restore
Now let's delete our namespace mimicing that our app has been deleted and needs to be restored
1kubectl delete ns my-app
Now let's run a restore and see if we can ge our app back
1velero restore create --from-backup my-app-backup
As we can see, our namespace is back and also our nginx deployment and the Load balanced service!
Ok, so what about losing an entire cluster? Or maybe we want to migrate our app to a different cluster?
Restore to different cluster
Well, let's take a look at our
Now, if we'd want to restore our backup taken from the different cluster we'll just install Velero and point it to our MinIO bucket. In fact we can use the exact same install command as we did in the first cluster.
The cool thing now is that Velero will automatically sync our object store which we can verify by listing available backups
So, let's try a restore of that backup in this cluster!
There's a lot more to Velero (and Restic) than what we've looked at here, but hopefully this has been an intro to some of the capabilities in the product, and also shown that it's quite easy to get started with it.
Thanks for reading, please reach out if you have any questions or comments!