Skip to content

talos-prod-vbox

This is an example of how to set up a Talos Linux (Kubernetes) cluster using VirtualBox VMs, following production-ready best-practices as much as possible. Of course, for real production, you shouldn't use VirtualBox: this tutorial serves just as inspiration, and to see what the process looks like.

Note: if you want to quickly spin up a Talos Linux cluster using Docker containers instead, just to try it out, you should refer to Quickstart - Talos Linux. It should be as simple as running something like:

Bash
1
talosctl cluster create --controlplanes 3 --workers 3

This example is heavily inspired by the official Talos Linux documentation. In particular:

Overview

The goal of this tutorial is to create a Talos Linux cluster with 3 control plane nodes and 3 worker nodes.

A VirtualBox NAT Network will be used for network communication. The control host will be able to access the Talos and Kubernetes APIs via port forwarding rules.

IP address Type Talos API access from host Kubernetes API access from host
192.168.10.10 VIP shared between control plane nodes - 127.0.0.1:6010
192.168.10.11 Control plane node 127.0.0.1:5011 127.0.0.1:6011
192.168.10.12 Control plane node 127.0.0.1:5012 127.0.0.1:6012
192.168.10.13 Control plane node 127.0.0.1:5013 127.0.0.1:6013
192.168.10.21 Worker node 127.0.0.1:5021 -
192.168.10.22 Worker node 127.0.0.1:5022 -
192.168.10.23 Worker node 127.0.0.1:5023 -

Note that we also set up a VIP (Virtual IP address) for control plane nodes using Talos Linux's Virtual (shared) IP feature.

Control host tools

First of all, you need to install some utilities on your host. This example has been tested with:

  • VirtualBox version 7.1.4
  • talosctl version 1.9.1
  • kubectl version 1.32.0
  • Helm version 3.17.0

VirtualBox NAT Network

Create the VirtualBox NAT Network for the nodes. You can use a command similar to the following:

Bash
1
vboxmanage natnetwork add --netname mynat01 --network 192.168.10.0/24 --enable --dhcp on

To create the port forwarding rules:

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
vboxmanage natnetwork modify --netname mynat01 \
    --port-forward-4 "Kubernetes10:tcp:[127.0.0.1]:6010:[192.168.10.10]:6443"
for i in {11..13}; do
    vboxmanage natnetwork modify --netname mynat01 \
        --port-forward-4 "Talos$i:tcp:[127.0.0.1]:50$i:[192.168.10.$i]:50000" \
        --port-forward-4 "Kubernetes$i:tcp:[127.0.0.1]:60$i:[192.168.10.$i]:6443"
done
for i in {21..23}; do
    vboxmanage natnetwork modify --netname mynat01 \
        --port-forward-4 "Talos$i:tcp:[127.0.0.1]:50$i:[192.168.10.$i]:50000"
done

VMs

Download the metal-amd64.iso ISO file from https://github.com/siderolabs/talos/releases.

Then you can leverage the create-vbox-vm-headless.sh script to create the virtual machines (replace metal-amd64.iso with the correct path of the ISO file):

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
while read -r name cpus mem; do
    ./create-vbox-vm-headless.sh -n"$name" -oLinux_64 \
        -c"$cpus" -m"$mem" -d102400 -i metal-amd64.iso

    vboxmanage modifyvm "$name" --nic1 natnetwork --nat-network1 mynat01
done << 'EOF'
TalosCtrl11   4   2048
TalosCtrl12   4   2048
TalosCtrl13   4   2048
TalosWork21   2   1024
TalosWork22   2   1024
TalosWork23   2   1024
EOF

Note: you may also want to adjust some values based on System Requirements - Talos Linux.

Nodes setup

We are now ready to generate the Talos Linux cluster configuration files:

Bash
1
2
3
4
5
6
7
8
talosctl gen config mycluster https://192.168.10.10:6443

for i in {11..13}; do
    talosctl machineconfig patch controlplane.yaml -p"@patch-controlplane-$i.yaml" -o "controlplane-$i.yaml"
done
for i in {21..23}; do
    talosctl machineconfig patch worker.yaml -p"@patch-worker-$i.yaml" -o "worker-$i.yaml"
done

Now start all the VMs.

Once started, change their network configuration to make them reachable by the host via the port-forwarding rules created before. In general, you need to set the following:

  • DNS Servers: 1.1.1.1 1.0.0.1
  • Interface: enp0s3
  • Mode: Static
  • Addresses: 192.168.10.XX/24 (replace XX with the proper number for each VM)
  • Gateway: 192.168.10.1

Note: you can access the network configuration screen by pressing the F3 key in the VM.

Note: if the main network interface name is different, please remember to change it in the configuration files too.

We should now be ready to apply the cluster configuration to the nodes:

Bash
1
2
3
4
5
6
for i in {11..13}; do
    talosctl apply-config -in "127.0.0.1:50$i" -f "controlplane-$i.yaml"
done
for i in {21..23}; do
    talosctl apply-config -in "127.0.0.1:50$i" -f "worker-$i.yaml"
done

They should reboot automatically. After that, you can bootstrap the Kubernetes cluster:

Bash
1
talosctl --talosconfig=talosconfig bootstrap -e127.0.0.1:5011 -n192.168.10.11

Wait a few minutes for the Kubernetes cluster to be set up, and then you can get the kubeconfig:

Bash
1
2
3
4
talosctl --talosconfig=talosconfig kubeconfig ./kubeconfig -e127.0.0.1:5011 -n192.168.10.11
sed -Ei 's/^(\s+server:\s+https:\/\/).+$/\1127.0.0.1:6010/' kubeconfig

kubectl --kubeconfig=kubeconfig get nodes

At this point, even if the Kubernetes cluster is now working, it's a good idea to detach the ISO disks from the VMs. If you don't do so, they will throw an error on the next boot saying that "Talos is already installed to disk but booted from another media", "Please reboot from the disk".

To detach the ISOs:

Bash
1
2
3
for i in Talos{Ctrl{11..13},Work{21..23}}; do
    vboxmanage storageattach "$i" --storagectl IDE --port 0 --device 0 --type dvddrive --medium none
done

Finally, if you want, you can also set the endpoints in your talosconfig file, so you won't have to pass the -e flag anymore on every talosctl invocation:

Bash
1
2
3
talosctl --talosconfig=talosconfig config endpoint 127.0.0.1:50{11..13}

talosctl --talosconfig=talosconfig -n192.168.10.11 get disks

Storage

To set up persistent storage in your cluster, you have the following options.

Local storage

With a local storage solution, each PersistentVolume you create (and its data) will be bound to a specific node. It's a simple and lightweight approach, and often it's just enough.

For example, one such solution is Rancher Local Path Provisioner; to set it up, see https://www.talos.dev/v1.9/kubernetes-guides/configuration/local-storage/#local-path-provisioner.

Note that, since you can choose the directory on the host (node) in which to save the data, you can also configure it to save to a partition on a secondary disk, by leveraging this Talos Linux feature: machine.disks[].partitions[]

Replicated storage

With a replicated persistent storage solution, the data of each PersistentVolume can be replicated on many nodes. This approach is often more complicated and resource-intensive.

For example, one such solution is OpenEBS Replicated PV Mayastor; to set it up, see https://www.talos.dev/v1.9/kubernetes-guides/configuration/storage/#openebs-mayastor-replicated-storage.

If you choose to set up this solution, please make sure that your cluster satisfies the minimum requirements for Mayastor.

Ingress Controller

You can set up the Ingress-Nginx Controller in your cluster by following this guide: Ingress-Nginx Quick start. I recommend using Helm as the installation method, as it's the most simple and flexible one.

Since we are working on a bare-metal Kubernetes cluster, to actually make Ingress-Nginx available, we need to rely on NodePorts. Please refer to this section of the official documentation: Bare-metal Ingress-Nginx over a NodePort Service.

I suggest setting the following Helm values, to make the port numbers constant:

Key Value
controller.service.nodePorts.http 30080
controller.service.nodePorts.https 30443

Then we can create some additional port forwarding rules to make the node ports available outside the VirtualBox NAT Network:

Bash
1
2
3
4
5
for i in {21..23}; do
    vboxmanage natnetwork modify --netname mynat01 \
        --port-forward-4 "HTTP$i:tcp:[127.0.0.1]:38$i:[192.168.10.$i]:30080" \
        --port-forward-4 "HTTPS$i:tcp:[127.0.0.1]:34$i:[192.168.10.$i]:30443"
done

Then you should be able to access the exposed node ports from your host like this:

Bash
1
2
curl http://127.0.0.1:3821/
curl https://127.0.0.1:3421/ --insecure

Observability

Observability is a crucial part of a production-ready Kubernetes cluster.

Here are some advices that may be useful to set up an observability system in your cluster. We will use Victoria* products.

They also have some playgrounds (public demos) available, to let you quickly explore the UIs: https://docs.victoriametrics.com/#playgrounds

Metrics

You can use VictoriaMetrics as monitoring solution. They say it's more performant and easier to operate than Prometheus.

It supports HA, zero-downtime upgrades, backups to S3, metrics discovery via CRDs such as ServiceMonitor, PodMonitor, etc. (via the VictoriaMetrics operator) and much more.

Logs

You can use VictoriaLogs as logs solution. They say it's more performant and easier to operate than Grafana Loki.

Next steps

You might want to do some additional setup. For example:

To set up a custom CNI (Container Network Interface) (useful for example if you need Network Policies):

To perform upgrades: