mini-lab

The mini-lab is a small, virtual setup to locally run the metal-stack. It deploys the metal control plane and a metal-stack partition with two simulated leaf switches. The lab can be used for trying out metal-stack, demonstration purposes or development.

Figure 1: Simplified illustration of the mini-lab.

This project can also be used as a template for writing your own metal-stack deployments.

• Requirements
• Known Limitations
• Try it out
  • Power management
• Flavors
• Network topology
• V2 Quickstart

Requirements

• Linux machine with hardware virtualization support
• kvm as hypervisor for the VMs (you can check through the kvm-ok command)
• docker >= 24.x.y (for using kind and our deployment base image)
• kind == v0.23.0 (for hosting the metal control plane)
• containerlab >= v0.56.0
• the lab creates a docker network on your host machine with the address block 203.0.113.0/24, designated as TEST-NET-3 for documentation and examples.
• (recommended) haveged to have enough random entropy (only needed if the PXE process does not work)

Here is some code that should help you to set up most of the requirements:

# If UFW enabled.
# Disable the firewall or allow traffic through Docker network IP range.
sudo ufw status
sudo ufw allow from 172.17.0.0/16

# Install required tools / kvm
sudo apt install -y git curl qemu qemu-kvm haveged jq

# Install Docker
curl -fsSL https://get.docker.com | sh
# if you want to be on the safe side, follow the original installation
# instructions at https://docs.docker.com/engine/install/ubuntu/

# Ensure that your user is member of the group "docker"
# you need to login again in order to make this change take effect
sudo usermod -G docker -a ${USER}

# Install containerlab
bash -c "$(curl -sL https://get.containerlab.dev)"

# Install kind (kubernetes in docker), for more details see https://kind.sigs.k8s.io/docs/user/quick-start/#installation
sudo curl -Lo /usr/local/bin/kind "https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64"
sudo chmod +x /usr/local/bin/kind

The following ports are used statically on your host machine:

Port	Bind Address	Description
6443	0.0.0.0	kube-apiserver of the kind cluster
4443	0.0.0.0	HTTPS ingress
4150	0.0.0.0	nsqd
8080	0.0.0.0	HTTP ingress

Known Limitations

• to keep the demo small there is no EVPN
• machines have to be restarted manually with make power-reset-<machine>
• login to the machines is possible with virsh console, login to the firewall is possible with SSH from your local machine

Try it out

git clone https://github.com/metal-stack/mini-lab.git
cd mini-lab

Start the mini-lab with a kind cluster, a metal-api instance as well as two containers wrapping leaf switches and another container that hosts two user-allocatable machines:

make MONITORING_ENABLED=false
# containerlab will ask you for root permissions (https://github.com/srl-labs/containerlab/issues/669)
# if you explicitly want to see the monitoring stack you can also run "make" without passing any params

Before the upcoming steps, you need to bind some environment variables using the following command. This ensures metalctl kubectl are able to communicate with your mini-lab.

eval $(make dev-env)

After the deployment and waiting for a short amount of time, two machines in status PXE booting become visible through metalctl machine ls:

docker compose run --rm metalctl machine ls

ID                                          LAST EVENT   WHEN     AGE  HOSTNAME  PROJECT  SIZE          IMAGE  PARTITION
00000000-0000-0000-0000-000000000001        PXE Booting  3s
00000000-0000-0000-0000-000000000002        PXE Booting  5s

Wait until the machines reach the waiting state:

docker compose run --rm metalctl machine ls

ID                                          LAST EVENT   WHEN     AGE  HOSTNAME  PROJECT  SIZE          IMAGE  PARTITION
00000000-0000-0000-0000-000000000001        Waiting      8s                               v1-small-x86         mini-lab
00000000-0000-0000-0000-000000000002        Waiting      8s                               v1-small-x86         mini-lab

Create a firewall and a machine with:

make firewall
make machine

Alternatively, you may want to issue the metalctl commands on your own:

docker compose run --rm metalctl network allocate \
        --partition mini-lab \
        --project 00000000-0000-0000-0000-000000000000 \
        --name user-private-network

# lookup the network ID and create a machine
docker compose run --rm metalctl machine create \
        --description test \
        --name machine \
        --hostname machine \
        --project 00000000-0000-0000-0000-000000000000 \
        --partition mini-lab \
        --image ubuntu-24.4 \
        --size v1-small-x86 \
        --networks <network-ID>

# create a firewall that is also connected to the virtual internet-mini-lab network
docker compose run --rm metalctl firewall create \
        --description fw \
        --name fw \
        --hostname fw \
        --project 00000000-0000-0000-0000-000000000000 \
        --partition mini-lab \
        --image firewall-ubuntu-3.0 \
        --size v1-small-x86 \
        --networks internet-mini-lab,<network-ID>

See the installation process in action

make console-machine01 # or console-machine02
...
Ubuntu 24.04 machine ttyS0

machine login:

Two machines are now installed and have status "Phoned Home"

docker compose run --rm metalctl machine ls
ID                                          LAST EVENT   WHEN   AGE     HOSTNAME  PROJECT                               SIZE          IMAGE               PARTITION
00000000-0000-0000-0000-000000000001        Phoned Home  2s     21s     machine   00000000-0000-0000-0000-000000000000  v1-small-x86  Ubuntu 24.04        mini-lab
00000000-0000-0000-0000-000000000002        Phoned Home  8s     18s     fw        00000000-0000-0000-0000-000000000000  v1-small-x86  Firewall 3 Ubuntu   mini-lab

Login with user name metal and the console password from

docker compose run --rm metalctl machine consolepassword 00000000-0000-0000-0000-000000000001

To remove the kind cluster, the switches and machines, run:

make cleanup

Power management

There are make targets to handle the power state of a machine:

make power-<on,reset,off>-<machine name>

Flavors

There are four flavors of the mini-lab environment:

• sonic: runs two Community SONiC switches
• dell_sonic: runs two Enterprise SONiC switches with a locally built vrnetlab image
• capms: runs the dell_sonic flavor but with four instead of two machines (this is used for cluster-provider-metal-stack in order to have dedicated hosts for control plane / worker / firewall)
• gardener: runs the sonic flavor and installs the Gardener in the mini-lab

In order to start specific flavor, you can define the flavor as follows:

export MINI_LAB_FLAVOR=sonic
make

Network topology

An Nginx is running inside of the www container to allow automatic testing of outgoing connections.

Figure 2: mini-lab network topology illustration.

V2 Quickstart

Login with cli:

$ metalctlv2 login --provider openid-connect

User: olli.owner@metal-stack.io Password: Olli.Owner123!

User: gerrit.guest@metal-stack.io Password: Gerrit.Guest123!

Zitadel Admin:

User: admin@metal-stack.zitadel.172.17.0.1.nip.io Password: Password1!