DOCSEARCHbyAlgolia
Submit Docs

virtink Documentation

repository·main·Indexed 20 days ago

https://github.com/smartxworks/virtink

Virtink is a virtualization platform leveraging Cloud Hypervisor to manage Virtual Machines. It supports advanced features such as direct kernel booting, container-based root filesystems (containerRootfs), and dedicated CPU placement via the Kubernetes CPU manager. The platform consists of several components: virt-controller for resource management and high availability, virt-daemon for VM reconciliation and device plugins, and virt-prerunner for preparing VM configurations and host-side resources.

Tokens
7.2K
Snippets
24
Records
34
Agent score
70%

What's inside virtink

  1. How VM networking works in Virtink

    main

    Connecting a VM to a network requires two distinct configuration parts in the VirtualMachine spec:

    1. spec.networks: Defines the backend network (logical or physical) that the VM can access.
    2. spec.instance.interfaces: Defines the virtual network interfaces seen by the guest OS.

    Crucially, every interface defined in spec.instance.interfaces must have a corresponding network with the exact same name defined in spec.networks.

    spec:
  instance:
    interfaces:
      - name: my-net
        bridge: {}
  networks:
    - name: my-net
      pod: {}

  2. Use `containerRootfs` for VM root filesystems

    main
    The containerRootfs volume type allows you to store and distribute VM root filesystems directly in a container image registry using the Docker toolchain. This avoids the need for raw or QCOW2 images. When using direct kernel boot, you specify the containerRootfs image and its size under the volumes section of the VirtualMachine spec.

  3. How disks and volumes work together in Virtink

    main

    To make persistent storage accessible to a VM, you must define two separate parts in the VirtualMachine specification:

    1. Volumes: Defined in spec.volumes. Each volume must have a unique name and a valid volume source (e.g., containerDisk, persistentVolumeClaim).
    2. Disks: Defined in spec.instance.disks. Each disk entry must have a name that matches a name defined in spec.volumes.

    This two-part approach decouples the storage definition from the VM's hardware configuration.

    spec:
  instance:
    disks:
      - name: my-disk-name
      - name: cloud-init
  volumes:
    - name: my-disk-name
      containerDisk:
        image: my-registry/my-image:latest
    - name: cloud-init
      cloudInit:
        userData: "#cloud-config\npassword: password"

  4. Configure Network Types in `spec.networks`

    main

    Networks are defined in spec.networks. You must specify the network type using one of the following fields:

    TypeDescription
    podThe default Kubernetes network (the pod's eth0 interface).
    multusA secondary network provided via Multus CNI.

    When using multus, you must provide the networkName which corresponds to a NetworkAttachmentDefinition in your Kubernetes cluster.

  5. Enable Dedicated CPU Placement for Virtual Machines

    main

    To ensure low latency and high performance by pinning guest vCPUs to host pCPUs, you can request dedicated CPU resources for a Virtual Machine.

    Prerequisites: Virtink relies on the Kubernetes CPU manager. For dedicated CPU placement to work, the Kubernetes CPU manager policy must be set to static. The default none policy provides no affinity beyond standard OS scheduling. Refer to the Kubernetes documentation for cluster-level configuration of the CPU manager policy.

    Implementation:

    1. Set spec.instance.cpu.dedicatedCPUPlacement to true in your VirtualMachine specification.
    2. Define the total number of vCPUs by configuring the guest topology using sockets and coresPerSocket within spec.instance.cpu.
    apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    cpu:
      sockets: 2
      coresPerSocket: 1
      dedicatedCPUPlacement: true

  6. Setup and use SR-IOV mode

    main

    SR-IOV mode provides high-performance networking by passing through an SR-IOV PCI device via VFIO.

    Prerequisites

    1. Install Multus CNI, SR-IOV CNI, and SR-IOV Network Device Plugin.
    2. Create Virtual Functions (VFs) on the host device.

    Step 1: Expose VFs to Virtink

    Change the VF driver to vfio-pci on the host:

    export VF_ADDR=0000:58:01.2 # change to your VF's PCI address
modprobe vfio_pci
export DRIVER=$(lspci -s $VF_ADDR -k | grep driver | awk '{print $5}')
echo $VF_ADDR > /sys/bus/pci/drivers/$DRIVER/unbind
export VENDOR_ID=$(lspci -s $VF_ADDR -Dn | awk '{split($3,a,":"); print a[1]}')
export DEVICE_ID=$(lspci -s $VF_ADDR -Dn | awk '{split($3,a,":"); print a[2]}')
echo $VENDOR_ID $DEVICE_ID > /sys/bus/pci/drivers/vfio-pci/new_id

    Step 2: Configure Device Plugin

    Create a ConfigMap for the SR-IOV device plugin to capture the VF as a node resource:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
      "resourceList": [{
        "resourceName": "intel.com/mellanox_SRIOV_25G",
        "selectors": {
          "vendors": ["$VENDOR_ID"],
          "devices": ["$DEVICE_ID"],
          "drivers": ["vfio-pci"]
        }
      }]
    }

    Step 3: Create NetworkAttachmentDefinition

    apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: mellanox-sriov-25g
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/mellanox_SRIOV_25G
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "sriov"
    }

    Step 4: Start the VM

    apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    interfaces:
      - name: sriov
        sriov: {}
  networks:
    - name: sriov
      multus:
        networkName: mellanox-sriov-25g

  7. Use the `multus` network type with secondary networks

    main

    To connect a VM to secondary networks, use the multus field in spec.networks and provide the networkName of an existing NetworkAttachmentDefinition.

    Example Workflow:

    1. Create a NetworkAttachmentDefinition (e.g., using Open vSwitch CNI).
    2. Reference that definition in the VM spec.
    # 1. The NetworkAttachmentDefinition
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ovs-br1
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br1"
    }

---

# 2. The VirtualMachine using it
apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    interfaces:
      - name: ovs
        bridge: {}
  networks:
    - name: ovs
      multus:
        networkName: ovs-br1

  8. Use Direct Kernel Boot with Virtink

    main

    Virtink supports direct kernel boot into a vmlinux ELF kernel using Cloud Hypervisor. This method does not require an EFI system partition and works with rootfs from most distributions. To use this feature, you must provide both a kernel image and a rootfs volume in your VirtualMachine specification.

    Configure the kernel using spec.instance.kernel, specifying the image name and the Linux cmdline arguments (e.g., console=ttyS0 root=/dev/vda rw).

    apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    kernel:
      image: smartxworks/virtink-kernel-5.15.12
      cmdline: "console=ttyS0 root=/dev/vda rw"
    disks:
      - name: ubuntu
  volumes:
    - name: ubuntu
      containerRootfs:
        image: smartxworks/virtink-container-rootfs-ubuntu
        size: 4Gi

  9. Use `containerDisk` for ephemeral image-based storage

    main

    The containerDisk volume source allows you to store and distribute VM disk images via a container registry. The disks are pulled from the registry and reside on the local node hosting the VM.

    Best Practices:

    • Use for: Replicating large numbers of VM workloads that do not require persistent data (ephemeral storage).
    • Avoid for: Workloads requiring persistent root disks across VM restarts.
    • Format: Raw and QCOW2 formats are supported. QCOW2 is recommended to reduce container image size.
    • Requirement: Disks must be placed at exactly the /disk path within the container image. Images should be based on smartxworks/virtink-container-disk-base.
    FROM smartxworks/virtink-container-disk-base
ADD https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img /disk

  10. Use `containerRootfs` for direct kernel booting

    main

    The containerRootfs volume source allows you to use a container image's filesystem directly as the VM's rootfs. This is typically used with Virtink's direct kernel boot feature.

    Key Details:

    • Requirement: The root of the rootfs must be placed at exactly the /rootfs path in the container image.
    • Base Image: Must be based on smartxworks/virtink-container-rootfs-base.
    • Use Case: Ideal for ephemeral workloads where you want to avoid disk partitioning/formatting. It is not suitable for persistent data.
    • Recommended Packages: Install systemd, cloud-init, and openssh-server within the image to ensure a functional VM environment.
    FROM ubuntu:jammy AS rootfs
RUN apt-get update -y && \
    apt-get install -y --no-install-recommends systemd-sysv udev lsb-release cloud-init sudo openssh-server && \
    rm -rf /var/lib/apt/lists/*

FROM smartxworks/virtink-container-rootfs-base
COPY --from=rootfs / /rootfs
RUN ln -sf ../run/systemd/resolve/stub-resolv.conf /rootfs/etc/resolv.conf

  11. Build and use a custom kernel for direct boot

    main

    You can use your own kernel for direct kernel booting. To ensure compatibility with Cloud Hypervisor, your kernel must be a vmlinux file.

    To package your kernel for Virtink:

    1. Build your kernel following the Cloud Hypervisor documentation.
    2. Create a Docker image using smartxworks/virtink-kernel-base as the base.
    3. Copy your vmlinux file to the exact path /vmlinux within that image.
    FROM smartxworks/virtink-kernel-base
COPY vmlinux /vmlinux

  12. Use `dataVolume` for automated disk importing via CDI

    main

    The dataVolume volume source integrates with the Containerized Data Importer (CDI) project. It automates the process of creating a PVC and importing data into it from a source.

    Requirements:

    • CDI must be installed in the cluster.

    Behavior:

    • You can submit the VM manifest before the DataVolume is created or while it is still importing.
    • Virtink will automatically wait until the referenced DataVolume has finished its clone and import phases before starting the VM.
    apiVersion: virt.virtink.smartx.com/v1alpha1
kind: VirtualMachine
spec:
  instance:
    disks:
      - name: ubuntu
  volumes:
    - name: ubuntu
      dataVolume:
        volumeName: ubuntu