VXLAN
Server virtualization outgrew the 12-bit VLAN ID, MAC tables and spanning tree. VXLAN carries a whole Ethernet frame inside UDP port 4789 across a routed underlay, tagged with a 24-bit VNI that allows up to 16 million segments. The VTEP at each edge wraps and unwraps the frame, for about 50 bytes of overhead on an IPv4 underlay.
By Arif, founder of Daveth · Updated
The full 7 min walkthrough is in Daveth Pro (Network). See what the plan opens
What this page answers
What is VXLAN?
An overlay that extends Layer 2 and Layer 3 networks over a Layer 3 underlay with MAC-in-IP/UDP tunnelling: a whole Ethernet frame inside UDP 4789, tagged with a 24-bit VXLAN network identifier, the VNI. Read it in “4.1 Why VXLAN exists”
What is a VNI in VXLAN?
The 24-bit VXLAN network identifier, allowing up to 16 million segments against the VLAN ID's 12 bits. Hosts in different VNIs cannot talk at Layer 2, even behind the same VTEP; joining them takes routing. Read it in “4.1 Why VXLAN exists”
How does VXLAN flood and learn work?
A VTEP that does not yet know which remote VTEP a MAC sits behind floods the frame, for example to the VNI's multicast group or by ingress replication to each remote VTEP, and learns from the replies. The lesson compares it with a control plane that tells VTEPs directly. Read it in the lesson
4.1 Why VXLAN exists
Server virtualization broke some old Layer 2 assumptions. One physical server now runs many VMs and containers, each with its own MAC address, and the guide lists what that did to traditional Layer 2: "The 12-bit VLAN ID yields 4000 VLANs, which are insufficient"; MAC tables have to hold "hundreds of thousands of VMs and containers"; STP blocks links to stop loops, leaving many of them idle; ECMP isn't available across an STP-blocked Layer 2 network; and moving hosts is hard. The data centre ended up like the office building that ran out of room numbers the week every desk got split into four.
The guide says 4000 VLANs and twelve bits give 4094 usable, which only matters until you've used them all, and then neither number is enough.
VXLAN's answer is "an overlay data plane encapsulation scheme": it "extends Layer 2 and Layer 3 overlay networks over a Layer 3 underlay network, using MAC-in-IP/UDP tunneling", and each overlay is a VXLAN segment. The routed underlay keeps its ECMP and loses STP; the segments ride on top. You can think of it as keeping the building's corridors (the routed network) and giving each team its own numbered floor plan that works on any floor.
The VTEP (the box at each edge of the tunnel; section 4.3 has the detail) takes the original Ethernet frame, header and all, and puts it inside four new headers (Figure 16-13):
| Part | Holds |
|---|---|
| outer Ethernet header | the next hop in the underlay |
| outer IP header | source: the sending VTEP; destination: the far VTEP, or the VNI's multicast group for a flood (section 4.4); IPv4 or IPv6 |
| outer UDP header | destination port 4789; a source port the VTEP picks, usually a hash of the inner frame's headers as RFC 7348 recommends, so ECMP spreads the flows (in 49152-65535) |
| VXLAN header (8 bytes) | flags with the I flag set for a valid VNI, the others reserved; the 24-bit VNI; reserved bits |
| original Ethernet header, IP header, data | the frame as the host sent it |
It's like moving house with a whole room packed into one crate: the removal van only reads the label on the crate, and when it's unpacked the furniture is exactly where it was. Adding up RFC 7348's formats, the wrapping costs 50 bytes over an IPv4 underlay (outer Ethernet 14, IP 20, UDP 8, VXLAN 8, with no VLAN tags), 70 over IPv6; VTEPs "MUST NOT fragment VXLAN packets", so RFC 7348 recommends raising the underlay's MTU to fit. The byte counts and the source-port hash go beyond the guide; they're here because they explain the MTU warning you'll meet in practice.
The port: the IANA "assigned to VXLAN the UDP destination port 4789; the default UDP destination port used by Linux is 8472". Linux implemented VXLAN before the port was assigned, used 8472 because many vendors already did, and kept it so existing deployments wouldn't break (quiz Q10: 4789). RFC 7348 says the port "SHOULD be configurable" for exactly that reason.
Linux still defaults to 8472 because changing it would break everyone who copied the early vendors, which is the most backward-compatible reason a port number has ever had.
The VNI is 24 bits, "which allows for up to 16 million VXLAN segments" (the guide adds, "more commonly known as overlay networks") against the VLAN ID's 12 bits; it sits in the VXLAN header and provides "segmentation for Layer 2 and Layer 3 traffic". Hosts in different VNIs "cannot communicate with each other" at Layer 2: the same MAC address can exist in two VNIs, and traffic never crosses between them, even when both sit behind the same VTEP; joining them takes routing. An overlay is the virtual network on top; the underlay is the physical routed network it rides on. It's a bit like flat numbers in a building: flat 12 on one staircase and flat 12 on another are different homes, and the VNI is the staircase.
The memory hook: VXLAN: the whole frame, UDP 4789, 24-bit VNI, and somebody else tells you where.
VTEPs "are entities that originate or terminate VXLAN tunnels. They map Layer 2 and Layer 3 packets to the VNI". The guide and Cisco expand the name as virtual tunnel endpoint; RFC 7348 says "VXLAN Tunnel End Point"; same box. Each has two sides (Figure 16-14):
- local LAN interfaces, which bridge between the local hosts;
- an IP interface, core-facing, whose address identifies the VTEP: it's the outer source of everything the VTEP sends, and the outer destination of unicast VXLAN sent to it.
VTEPs are also how VNIs are discovered across the underlay, which is how the guide introduces them: they "facilitate the discovery of VNIs over the underlay Layer 3 network".
RFC 7348 draws the VTEP inside the hypervisor (04-7) on each server; in Cisco's Catalyst 9300 BGP EVPN VXLAN design, the leaf switches (leaf-spine: 06-1) are the VTEPs. A VTEP acts kind of like a site's post room: letters for residents are handed over inside, and anything for another site goes out in a bag addressed to that site's post room.
A VXLAN gateway "is a VTEP device that combines a VXLAN segment and a classic VLAN segment into one common Layer 2 domain", for devices that only speak VLANs; Cisco calls this a Layer 2 VXLAN gateway, mapping a VNI to a VLAN. It's a separate role because in RFC 7348's picture the VTEP lives in the hypervisor, so a physical server with no VXLAN-capable hypervisor needs a switch to do the joining for it. The gateway works like the adapter plug you carry abroad: the socket on the wall doesn't change, you just make the two fit.
A VXLAN gateway is how you tell an old server "don't worry, you're still on VLAN 10" without mentioning the 16 million segments behind it.
Here's the surprise. "The VXLAN standard defines VXLAN as a data plane protocol, but it does not define a VXLAN control plane" (quiz Q11: false). So when A sends a frame to B's MAC, what tells VTEP1 that B is behind VTEP2?
Before reading on, predict, with flood and learn (D1's default): after A's ARP broadcast, which VTEPs know where A is, and does VTEP3 ever learn where B is? D1 lets you check.
RFC 7348's own answer is flood and learn, and it isn't a separate protocol: VTEPs learn from the data packets themselves, which the RFC calls "the data plane learning scheme as the control plane for VXLAN". That's why Cisco lists VXLAN's learning methods under "control and data planes": it mixes data-plane methods with real control-plane protocols. A broadcast, such as A's ARP, is sent to the IP multicast group mapped to the VNI: VTEP1 sends one packet and the underlay copies it to every VTEP with that VNI; each receiving VTEP learns the inner source MAC against the outer source IP, the sending VTEP. B's reply is unicast, because VTEP2 just learnt where A is, and VTEP1 learns B from it. Unknown-destination and multicast frames (together with broadcasts, BUM traffic) flood the same way. Flooding is like a question shouted up a stairwell: everyone on the staircase hears it, the one who answers comes to your door, and next time you know which flat to knock on. Cisco's verdict: it "limits scalability".
The alternative is a control plane that tells the other VTEPs. With BGP EVPN, once a VTEP has learnt its own local hosts, it advertises their MAC and IP addresses in route type 2 updates. A route target is a tag on those updates, and a VTEP imports a route only when the tag matches, which RFC 8365 uses "to limit the propagation and import of routes to only the NVEs that are interested in a given VNI". Broadcasts can then go by ingress replication (announced in route type 3): VTEP1 itself sends a unicast copy to each VTEP in the VNI, instead of one multicast packet the underlay copies. The payoff is that unknown MACs don't have to be flooded to be found: control-plane learning "does not require any flooding". Think of it as a residents' list at the front desk: nobody has to shout to find out where someone lives, though a notice for everyone still goes to every door. Route types, route targets and ingress replication go beyond the guide; they're the minimum that makes D1's second mode make sense.
D1

Step 1 of 4 Nothing has been sent, so no VTEP knows any remote MAC. A wants to talk to B and doesn’t know B’s MAC.

Step 2 of 4 A doesn’t know B’s MAC, so it sends an ARP broadcast in VNI 10100. VTEP1 wraps it in VXLAN and sends one packet to the multicast group 239.1.1.1, which carries VNI 10100; the underlay copies it to every VTEP with a host in VNI 10100 (RFC 7348 section 4.2). Each receiving VTEP learns A’s MAC against VTEP1’s address, the outer source IP (RFC 7348 section 4.1). It reaches B and C; D sits on VTEP3 too, but in VNI 10200, so VTEP3 doesn’t hand it over.

Step 3 of 4 B answers with a unicast ARP reply to A’s MAC. VTEP2 learnt that MAC from the flood, so it sends one unicast VXLAN packet straight to VTEP1: no flooding. VTEP1 learns B’s MAC from it (RFC 7348 sections 4.1-4.2). VTEP3 never sees the reply, so it never learns B.

Step 4 of 4 A’s first real frame to B: VTEP1 looks up B’s MAC in VNI 10100, finds VTEP2, and sends unicast VXLAN, outer 10.255.0.1 → 10.255.0.2, UDP 4789. VTEP2 checks the VNI, strips the headers and hands the original Ethernet frame to B, which never knows VXLAN was involved (RFC 7348 section 4.1).
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 9 more after it.





