Virtual switching, and how packets reach a VM fast

A virtual switch is a software Layer 2 switch inside a server: VMs reach each other and the physical network through it, by their virtual NICs. A server can run several vSwitches, but traffic cannot flow directly between them and they cannot share a physical NIC. Because every packet costs CPU in software, OVS-DPDK, PCI passthrough and SR-IOV speed the path up.

In Daveth Pro (Network)7 min video

By , founder of Daveth · Updated

Opening frame of the video walkthrough of Virtual switching, and how packets reach a VM fast

Watch this lesson

7 min · Video walkthrough

In Daveth Pro (Network)

The full 7 min walkthrough is in Daveth Pro (Network). See what the plan opens

What this page answers

What is a virtual switch (vSwitch)?

A software-based Layer 2 switch that operates like a physical Ethernet switch, letting VMs talk to each other inside the server through their vNICs and reach the physical network through the server's physical NICs. Read it in “4.1 The switch inside the server”

Can two vSwitches on one host talk directly?

No. A server can have several vSwitches, but traffic cannot flow directly from one to another within the same host, and they cannot share the same physical NIC. Read it in “4.1 The switch inside the server”

4.1 The switch inside the server

The VMs in 04-7 still need a network. "A virtual switch (vSwitch) is a software-based Layer 2 switch that operates like a physical Ethernet switch" (quiz Q4's answer: a software version of a physical Layer 2 switch); it lets VMs, through their virtual NICs (vNICs), talk to each other inside the server and reach the physical network through the server's physical NICs (pNICs). The vSwitch works like a building's internal post room: letters between flats never leave the building, and anything for outside goes out through the front door.

Two rules shape everything after this. You can create several vSwitches on one server (so quiz Q5's "only one" is false), but "network traffic cannot flow directly from one vSwitch to another vSwitch within the same host, and the vSwitches cannot share the same pNIC". The popular ones: Cisco Nexus 1000VE, Cisco Application Virtual Switch, Open vSwitch (OVS), IBM DVS 5000v and the vSphere Switch.

Look at Figure 27-5 with those rules in mind. vSwitch 1 reaches the physical network, vSwitch 2 has no pNIC at all, and the only thing on both is a virtual firewall, the NGFWv (what a next-generation firewall inspects: 08-5). So traffic from VM1 to the outside, or to VM0, "needs to flow through the virtual next-generation firewall". That's the design: a firewall you can't walk around. You can think of it as a building where one wing's only corridor goes through the security desk.

Before running D1, predict: the NGFWv stops. Of the VMs on vSwitches 1 and 2, which can still reach Switch 1?

D1

Network drawing: virtualized server, VM0, VM1, VM2, NGFWv, vSwitch 1, vSwitch 2, vSwitch 3, pNIC 1, pNIC 2, pNIC 3, Switch 1, Switch 2
VM1 → Switch 1: VM1 → vSwitch 2 → NGFWv → vSwitch 1 → pNIC 1 → Switch 1. It crosses 2 vSwitches, and the only way from one to the other is through the NGFWv, which is exactly the guide’s point: traffic can’t go straight from one vSwitch to another, so it has to be forwarded by something with a vNIC on each.

Two vSwitches in the same server that can't talk to each other directly is the most realistic thing in this chapter; every office has two departments that only speak through a third.

A standard vSwitch lives in one host, so in a cluster "every vSwitch ... needs to be configured individually in every virtual host". Distributed virtual switching "aggregates vSwitches together from a cluster of virtualized servers and treats them as a single distributed virtual switch". The benefits the guide lists: one place to manage the vSwitch configuration for every host; a VM's networking statistics and policies move with it during a live migration; and the same configuration everywhere. It runs like a chain of shops with one head office setting the prices, instead of every branch manager writing their own labels.

Configuring the same vSwitch by hand on forty hosts is how networking teams discovered that copy and paste has a typo rate.

The live-migration benefit ties back to 04-7: when a VM moves, its networking statistics and policies move with it. If you've ever moved to a new branch of your bank and found your account settings already there, that's the idea.

Quiz Q6 says containers don't need vSwitches; that's false. "Like VMs, containers rely on vSwitches (also known as virtual bridges)". Docker creates a bridge the guide calls Docker0 by default (Docker's own documentation names the default network bridge, and on Linux the device is usually shown in lower case) and gives it 172.17.0.1/16; the subnet can be customized, and user-defined bridges can be used as well. Every container gets a virtual Ethernet interface (veth) on Docker0, which "appears to the container as eth0", addressed from the bridge's subnet (Figure 27-6). It's like a shared house with an internal phone line: every room has an extension, and they can all ring each other.

The limit: containers "can then communicate with each other only if they are within the same node"; reaching containers on other nodes takes routing at the OS level or an overlay network, and a second Docker node left at its defaults ends up with the same addressing as the first, which has to be sorted out node by node. The guide's better answer is an orchestrator such as Kubernetes. Docker's own documentation says the same about crossing hosts, and adds that a bridge network "has an IPv4 subnet and, optionally, an IPv6 subnet". It's a bit like two houses with identical extension numbers: fine inside each house, confusing the moment they try to call each other.

Every default Docker host thinks it's 172.17.0.1, which is the container world's version of every hotel room being called "Room 1".

A VNF, a virtual network function, is "the virtual or software version of a physical network function (NF) such as a firewall", and it typically runs on a hypervisor as a VM; the NGFWv is one. Two traffic shapes: north-south comes in from the wire, goes to a VNF and back out; east-west goes from one VNF to another inside the server, perhaps through several (service chaining), before leaving; and a VNF can use both, north-south for users and east-west to a VNF that only collects statistics or logs. The guide says these patterns "are important to understand when deciding which technology to use". Then the terms: I/O is a computer's traffic with the outside world, an I/O device is something like a NIC, and a device driver is the program that lets the CPU talk to it; an IRQ is a device telling the CPU it has data, and the CPU "saves its current state, temporarily stops what it’s doing, and runs an interrupt handler"; DMA lets a device write to memory "bypassing the CPU"; the kernel runs in protected kernel space, apps in user space. An interrupt is like a doorbell: every ring means stopping what you're doing and walking to the door.

Without virtualization, a packet goes from the pNIC through kernel space to an app in user space (Figure 27-9). In a virtualized server, "there are pNICs and virtual NICs (vNICs) and a hypervisor with a virtual switch in between them", and that layer "creates bottlenecks and reduces I/O packet throughput". The guide's 11 steps for standard OVS show why: the pNIC DMAs the packet into memory and interrupts the CPU; the driver pushes it up the host's network stack; it's copied to OVS; OVS forwards it to the VM, "switching the packet between the kernel and user space, which is expensive in terms of CPU cycles"; then, inside the VM, the vNIC does the whole thing again, interrupt and all, before the app gets a copy (Figure 27-10). And: "The number of interrupts increases when using high-speed NICs (for example, 40 Gbps) and the packet size is small", because more packets have to be processed per second, and every interrupt means stopping, saving state, handling it and restoring. If you've ever tried to cook while the doorbell rang for every single parcel, you know the problem.

D2

  1. Path of a packet through a host running standard OVS, seven stages from the physical NIC to the application in a VM; nothing has moved yet.

    Step 1 of 8 standard OVS: 7 stages from the wire to the app.

  2. Standard OVS path, stage 1 lit: the packet enters the physical NIC's receive queue and is copied by DMA into host memory.

    Step 2 of 8 PNIC Rx queue, DMA to host memory.

  3. Standard OVS path, stage 2 lit: the physical NIC interrupts the host CPU, whose driver pushes the packet up the host network stack.

    Step 3 of 8 The pNIC interrupts the host CPU: it saves its state, stops what it was doing and runs the pNIC driver, which pushes the packet up the host network stack into a socket buffer. The number of these interrupts rises with fast NICs (40 Gbps) and small packets.

  4. Standard OVS path, stage 3 lit: the packet is copied from the socket buffer to Open vSwitch.

    Step 4 of 8 Copied from the socket buffer to OVS.

  5. Standard OVS path, stage 4 lit: OVS forwards the packet to the VM, crossing between kernel space and user space.

    Step 5 of 8 OVS forwards the packet to the VM, which means switching it between kernel space and user space: "expensive in terms of CPU cycles".

  6. Standard OVS path, stage 5 lit: the packet lands in the virtual NIC's queue and is copied by DMA into the VM's memory.

    Step 6 of 8 VNIC queue, DMA to the VM’s memory.

  7. Standard OVS path, stage 6 lit: inside the VM the virtual NIC interrupts the vCPU and the guest driver and stack take over.

    Step 7 of 8 Inside the VM the whole thing happens again: the vNIC interrupts the vCPU and the guest driver and stack take over.

  8. Standard OVS path, all seven stages lit: the packet is delivered to the application inside the VM.

    Step 8 of 8 Delivered to the app. Interrupts to the host CPU, the host network stack and the kernel/user crossing are what the other three technologies, OVS-DPDK, PCI passthrough and SR-IOV, remove.

At 40 Gbps of small packets, the CPU spends so much time answering the door that dinner never gets finished.

The guide's answer is three technologies, and all of them need a pNIC that supports them:

What it removesWhat it costs
OVS-DPDKthe host's interrupts and network stack, keeping a software switch: OVS runs "entirely in user space", and the DPDK Poll Mode Driver "polls for data that comes into the pNIC"; the packet is already in user space, so it's switched straight to the VNF"one or more CPU cores dedicated to polling"
PCI passthroughthe hypervisor: the VNF gets "direct access" to a pNIC. The guide's advantages: "Exclusive one-to-one mapping", bypassed hypervisor, direct access to I/O resources, reduced CPU utilization, reduced latency, increased I/O throughput"the entire pNIC is dedicated to a single VNF", so the number of VNFs is limited by the number of pNICs
SR-IOVthe hypervisor, like passthrough, but many VNFs share one pNIC: the pNIC emulates several PCIe devices, virtual functions (VFs), on its physical function (PF), and each VNF uses a VF by PCI passthrough (quiz Q10)the guide names no downside; like the others, it needs a pNIC that supports it

DPDK's own guide says its poll-mode drivers work "without relying on interrupts (except for Link Status Change notifications)". Polling is like a tenant checking the mailbox constantly instead of waiting for the doorbell: no interruptions, but somebody is always standing at the mailbox.