Kubernetes Cluster in Container Nodes

The kind plugin implements a Kubernetes-in-Docker (KinD) cluster for containerlab deployments.

Using the Plugin

  • Add plugin: [ kind ] to lab topology.

  • Add a node with device: kind to the lab topology. The kind plugin will automatically create control-plane and worker nodes using the kind-node device.

  • Set the number of worker nodes in the cluster with the kind.workers node parameter. Without this parameter, the plugin deploys a single-node Kubernetes cluster

For example, the following topology deploys a 3-node cluster with no external connectivity (the cluster is connected to the kind Docker network created by containerlab):

provider: clab
plugin: [ kind ]

nodes:
  kc:
    device: kind
    kind.workers: 2

The names of the cluster members are derived from the cluster name. In our example, the three containers would be named kc-control-plane, kc-worker and kc-worker2:

$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS          PORTS                       NAMES
49f3a87d7742   kindest/node:v1.34.3           "/usr/local/bin/entr…"   11 minutes ago   Up 10 minutes   127.0.0.1:35301->6443/tcp   kc-control-plane
832e14fe9842   kindest/node:v1.34.3           "/usr/local/bin/entr…"   11 minutes ago   Up 10 minutes                               kc-worker2
509551e5ca05   kindest/node:v1.34.3           "/usr/local/bin/entr…"   11 minutes ago   Up 10 minutes                               kc-worker

Lab Connectivity

You can connect the KinD cluster node to any number of netlab links. When creating the control-plane and worker nodes, the kind plugin automatically connects them to the same links if the links are stub or LAN links. For point-to-point links, the kind plugin replaces the link between the cluster node and another node with a series of point-to-point links.

For example, netlab creates three links when given this topology:

provider: clab
plugin: [ kind ]

nodes:
  kc:
    device: kind
    kind.workers: 2
  sw:
    device: frr

links: [ kc-sw ]

Link Name

Origin Device

Origin Port

Destination Device

Destination Port

kc-control-plane

eth1

sw

eth1

kc-worker

eth1

sw

eth2

kc-worker2

eth1

sw

eth3

On the other hand, the following topology using a LAN link results in all three cluster nodes connected to the same switch port:

provider: clab
plugin: [ kind ]

nodes:
  kc:
    device: kind
    kind.workers: 2
  sw:
    device: frr

links: 
- interfaces: [ kc, sw ]
  type: lan

Origin Device

Origin Port

Link Name (NET)

Description

sw

eth1

X_1

sw -> [kc-control-plane,kc-worker,kc-worker2]

kc-control-plane

eth1

X_1

kc-control-plane -> [sw,kc-worker,kc-worker2]

kc-worker

eth1

X_1

kc-worker -> [sw,kc-control-plane,kc-worker2]

kc-worker2

eth1

X_1

kc-worker2 -> [sw,kc-control-plane,kc-worker]

Other Networking Parameters

  • IPv4/IPv6 forwarding is enabled on the KinD containers. To disable it, set the netlab_ip_forwarding parameter of the cluster node to False (more details), for example:

provider: clab
plugin: [ kind ]

nodes:
  kc:
    device: kind
    kind.workers: 2
    netlab_ip_forwarding: False

  • KinD containers do not have loopback interfaces. To add the loopback interface to KinD containers, set the loopback parameter of the cluster node to True.

Additional Cluster Node Parameters

You can influence the parameters of the control-plane or worker nodes using the attributes of the cluster node. The following parameters are copied from the cluster node (device kind) to KinD containers:

  • The box, role, and loopback parameters

  • The routing dictionary (allowing you to specify custom static routes)

  • All parameters starting with netlab_ (allowing you to disable IP forwarding).

Setting Parameters on Individual KinD Containers

You can also change the parameters of individual KinD containers:

  • Add them to the lab topology (use the node names the plugin will use)

  • Set the node device to kind-node if you haven’t specified the default device in the lab topology (the plugin overrides this value anyway).

  • Set the parameters you want to change, for example, clab.exec to execute commands on individual containers once they’re started.

  • You can also use the config node parameter to execute scripts on individual KinD containers after they’ve been configured.

Example:

provider: clab
plugin: [ kind ]

nodes:
  kc:
    device: kind
    kind.workers: 2
  kc-control-plane:
    clab.exec:
    - echo "kubectl get services"
    config: [ kconf ]
    device: kind-node

Warning

  • The clab.exec commands are executed after ‌containerlab starts the container but before the containers are configured with ‌netlab initial

  • The ‌config scripts are executed as the last step in the ‌netlab initial process (after the IP addressing and routing have been configured)

  • You cannot use ‌clab.binds or ‌clab.config_templates parameters; containerlab does not use these parameters on members of KinD clusters.

Using kubectl As Part of Lab Configuration

Use clab.exec or config parameter on the control-plane node to configure Kubernetes in your KinD cluster. The clab.exec commands are shell commands (for example, ‌ kubectl get services), the config custom configuration templates are Linux scripts, for example:

#!/bin/bash
kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml

With a creative shebang (requiring coreutils 8.30 or later), you can use kubectl to apply the Kubernetes configuration from the custom configuration template, for example:

#!/usr/bin/env -S kubectl apply -f
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80