DMVPN Phase 3 with dual hub

Replace a tunnel destination with tunnel mode gre multipoint and one interface reaches every site, but now the router must find each packet's far end. NHRP, an ARP-like protocol for NBMA networks, solves it: the hubs are next hop servers and the spokes register with them. Each hub advertises one summary, so the first packet goes via a hub, which sends an NHRP redirect.

In Daveth Pro (Network)9 min video · lab: DMVPN

By , founder of Daveth · Updated

Opening frame of the video walkthrough of DMVPN Phase 3 with dual hub

Watch this lesson

9 min · Video walkthrough

In Daveth Pro (Network)

The full 9 min walkthrough is in Daveth Pro (Network). See what the plan opens

Upcoming practical lab

Lab C12: DMVPN

What the lab covers

What this page answers

What is NHRP in DMVPN?

An ARP-like protocol that maps tunnel addresses to real public addresses on a non-broadcast network. The hubs are next hop servers and the spokes are next hop clients that register with them. Read it in “The hub is a database, and the spokes are the ones who fill it in”

How does a DMVPN Phase 3 spoke reach another spoke?

Its first packet follows the hub's summary through the hub. The hub, seeing traffic go back into the same cloud, sends the spoke an NHRP redirect, and the spoke resolves the other spoke's address to build a direct tunnel. Read it in “The hub forwards the packet and tells the sender to stop”

What routes does a DMVPN Phase 3 spoke learn?

Each hub advertises one summary, and each spoke advertises only its own LAN, so no spoke learns a specific route to another spoke: everything not its own is behind a hub. Read it in “The first packet, and the second lookup that no other interface needs”

Delete one line, and the tunnel has to work out where to send every packet

A point-to-point GRE tunnel is configured with a tunnel destination. That address has to be routable, the path to it must not run back through the tunnel, and the whole interface depends on it. Take the line away and put tunnel mode gre multipoint in its place, and one interface can now reach every other site instead of one.

That is the scaling argument, and it creates the problem this page is about. With many possible far ends and no configured destination, the router has to decide per packet which real address to send to. By the end you will be able to read a spoke's cache and its routing table together and say, for any destination, whether the traffic is about to take a shortcut or go the long way round, and when it goes the long way, which of five routers is the reason.

State 1 What actually exists

What is physically joined to what, before any tunnel is configured?

Network drawing: 203.0.113.11, 203.0.113.12, 203.0.113.21, 203.0.113.22, 203.0.113.23, HUB1, NHS, HUB2, the internet, NBMA · 203.0.113.0/24, SPOKE1, NHC, SPOKE2, SPOKE3, PC1, PC3

Five routers, each with one link to the internet and one public address on it. There is no circuit between any two sites. Those public addresses are the only way any of these routers can reach any other, and that layer has a name worth knowing: NBMA, non-broadcast multi-access. The name is doing real work: a network where you cannot broadcast is a network where ARP cannot help you, which is exactly why a different resolution protocol has to exist.

SPOKE1# show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

C       10.0.0.0/24          is directly connected, Tunnel0
C       192.168.1.0/24       is directly connected, GigabitEthernet2
C       203.0.113.0/24       is directly connected, GigabitEthernet1
! one connected network, and nothing else: no site can see any other site's LAN

The hub is a database, and the spokes are the ones who fill it in

The protocol that solves the lookup is NHRP, and the published description says what to carry across from something you already know: it is an Address Resolution Protocol (ARP)-like protocol that dynamically maps a Non-Broadcast Multi-Access (NBMA) network. ARP maps an IP address to a MAC address on a segment you can broadcast on. NHRP maps a tunnel address to a real public address on one you cannot.

It has two roles. The hub is the next hop server, the spokes are next hop clients, and the hub maintains an NHRP database of the public interface addresses of each spoke. Read what the hub is in that sentence. Not a concentrator, not a forwarder: a place where every spoke's real address is known, because every spoke said so.

The hubs first. They need no mapping configured at all, because they are the ones being registered with.

HUB1> enable
HUB1# configure terminal
HUB1(config)# interface Tunnel0
HUB1(config-if)# ip address 10.0.0.11 255.255.255.0
HUB1(config-if)# ipv6 address 2001:DB8:FFFF::11/64
HUB1(config-if)# tunnel source GigabitEthernet1
HUB1(config-if)# tunnel mode gre multipoint
HUB1(config-if)# ip nhrp network-id 100
HUB1(config-if)# ip nhrp holdtime 360
HUB1(config-if)# ip nhrp authentication DMVPN-KEY
HUB1(config-if)# ip nhrp map multicast dynamic
HUB1(config-if)# ip nhrp redirect
HUB1(config-if)# end

The tunnel comes up on its source address alone: an mGRE interface does not need a far end to be up, which is the first thing that separates it from the point-to-point kind. ip nhrp holdtime 360 is the number to hold on to, because the registration interval is derived from it rather than configured: registrations go out every one third of the holdtime, so 360 seconds means one every 120, and three of them fit inside every holdtime. That is the ordinary lose two and still be fine margin.

Then the spokes, which need one thing the hubs do not.

SPOKE1# configure terminal
SPOKE1(config)# interface Tunnel0
SPOKE1(config-if)# ip nhrp nhs 10.0.0.11 nbma 203.0.113.11 multicast
SPOKE1(config-if)# ip nhrp nhs 10.0.0.12 nbma 203.0.113.12 multicast
SPOKE1(config-if)# ip nhrp shortcut
SPOKE1(config-if)# end

Those two lines are the bootstrap, and they are the only mappings in this network that a router is given rather than learns. Without them a spoke has a multipoint tunnel and no idea where to send anything, including the message that would tell it. Note that each spoke is given both hubs: a spoke registers with every server it has, continuously, and there is no primary and no standby in NHRP's sense.

State 2 SPOKE1 registers, twice

What does a hub know afterwards that it did not know before, and what does the spoke know?

Network drawing: HUB1, NHS, HUB2, the internet, SPOKE1, NHC, 203.0.113.21, 203.0.113.11, a tunnel, not a wire

The spoke sends a registration to each server over the tunnel, inside a GRE packet addressed to the public address the static mapping gave it, and each hub writes down the pair.

HUB1# show ip nhrp
10.0.0.21/32 via 10.0.0.21
   Tunnel0 created 00:00:12, expire 00:05:48
   Type: dynamic
   NBMA address: 203.0.113.21
   ! the real address SPOKE1 registered from: this line is the database
10.0.0.22/32 via 10.0.0.22
   Tunnel0 created 00:00:12, expire 00:05:48
   Type: dynamic
   NBMA address: 203.0.113.22
10.0.0.23/32 via 10.0.0.23
   Tunnel0 created 00:00:12, expire 00:05:48
   Type: dynamic
   NBMA address: 203.0.113.23

Now read the same command on the spoke, and read it for what is absent.

SPOKE1# show ip nhrp
10.0.0.11/32 via 10.0.0.11
   Tunnel0 created 00:00:12
   Type: static   ! static: this was configured, not learned
   NBMA address: 203.0.113.11
10.0.0.12/32 via 10.0.0.12
   Tunnel0 created 00:00:12
   Type: static
   NBMA address: 203.0.113.12

Two entries, both static, both about hubs. A spoke knows nothing about any other spoke. Every hub holds the complete directory; nobody who actually needs it has any of it.

What these outputs show, and what they leave out. Every line above is computed from the modelled network. Two things this course could not source are therefore left out rather than invented: a real device prints a Flags line on each cache entry, and priority and cluster columns on the server list, and no page reached here defines that vocabulary. The administrative distance shown later on an NHRP route is in the same position. The written lesson records the same gap.

Proves: each hub has a mapping from every spoke's tunnel address to its public address, learned from a registration, and each spoke has reached both of its servers.
Does not prove: that any spoke can reach any other spoke directly: no spoke holds a mapping for another spoke, and none has been asked for one.

The first packet, and the second lookup that no other interface needs

Routing over the tunnel is deliberately dull. Each hub advertises one summary to the spokes, each spoke advertises only its own LAN, and no spoke learns a specific route to any other spoke. So every spoke believes the same correct thing: everything that is not mine is behind a hub.

SPOKE1# show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

C       10.0.0.0/24          is directly connected, Tunnel0
D       192.168.0.0/16       [90/26880256] via 10.0.0.11, Tunnel0
! one summary, next hop a hub: this is the whole of SPOKE1's knowledge of the other sites
C       192.168.1.0/24       is directly connected, GigabitEthernet2
C       203.0.113.0/24       is directly connected, GigabitEthernet1

PC1 sends a packet to PC3. SPOKE1 does the lookup any router would do and gets 192.168.0.0/16 via 10.0.0.11, out Tunnel0. Then it has to do a second one, and this is where DMVPN stops resembling ordinary routing: Tunnel0 has no destination, so a next hop of 10.0.0.11 is not yet something you can send to. It has to become a real address first.

State 3 The long way

The packet reaches PC3. How many times did it cross the internet, and why?

Network drawing: HUB1, NHS, the internet, SPOKE1, NHC, SPOKE3, 203.0.113.21, 203.0.113.11, 203.0.113.23

Twice. Spoke to hub, then hub to spoke, because the only mapping SPOKE1 held was the one it was configured with. To every router in between this is an ordinary packet from 203.0.113.21 to 203.0.113.11; the tunnel, the private addresses and the encryption are payload. That is why the NBMA layer can be anything at all (broadband, an MPLS service, four different providers) as long as those two public addresses can reach each other.

The hub takes the packet out of its GRE wrapper, looks the destination up, finds SPOKE3's LAN learned over the tunnel, and wraps it again for SPOKE3's public address. Nothing is broken and nothing is slow enough to complain about. It is simply going a long way round.

HUB1# show ip route 192.168.3.10
Routing entry for 192.168.3.0/24
  Known via "eigrp 1", distance 90, metric 26880256
  Routing Descriptor Blocks:
  * 10.0.0.23, from 10.0.0.23, via Tunnel0
! the hub does know where SPOKE3 is: it learned the LAN over the tunnel
      Route metric is 26880256, traffic share count is 1

Proves: the path works end to end, and the hub is forwarding between two spokes.
Does not prove: that this is how it will stay. The next thing that happens is caused by exactly this: a hub that finds itself forwarding between two spokes.

The hub forwards the packet and tells the sender to stop

Having forwarded it, HUB1 notices something about itself: it is not the exit point, and the traffic it just handled is going back out into the same cloud it came from. So it sends an NHRP redirect to SPOKE1. There is something quietly excellent about a device whose main contribution here is to say not me.

SPOKE1 reacts in two steps, and the first one is easy to skim past. It inserts an incomplete cache entry for 192.168.3.0/24. Read the word: the spoke now knows it needs a mapping and does not have one, so the entry exists as a place to put an answer. It adds no route to its routing table, and until the answer arrives the traffic keeps going through the hub.

SPOKE1# show ip nhrp
10.0.0.11/32 via 10.0.0.11
   Tunnel0 created 00:00:12
   Type: static
   NBMA address: 203.0.113.11
10.0.0.12/32 via 10.0.0.12
   Tunnel0 created 00:00:12
   Type: static
   NBMA address: 203.0.113.12
192.168.3.0/24 via 0.0.0.0
   Tunnel0 created 00:00:12, expire 00:00:03
   Type: incomplete
   ! incomplete: the question has been asked and not yet answered

Then it asks. Notice the path the question takes: the resolution request travels over the existing tunnel to the hub, because that is still the only path SPOKE1 has to anywhere in this cloud. The one thing it is trying to escape is the one thing carrying the escape request.

SPOKE3 receives the request, and the request carried SPOKE1's public address. So SPOKE3 has what it needs to build the tunnel and it builds it, then replies across it. The end that received the question initiates, and predicting that wrong sends you to the wrong device's logs.

State 4 The short way

Same frame, same devices. What is different, and what did it cost?

Network drawing: HUB1, NHS, the internet, SPOKE1, NHC, SPOKE3, 203.0.113.21, 203.0.113.23

One arc, and one line in each table. The incomplete entry became a real mapping with a real address on it, and, this is the part worth slowing down for, a route was added, not just a cache entry.

SPOKE1# show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

C       10.0.0.0/24          is directly connected, Tunnel0
D       192.168.0.0/16       [90/26880256] via 10.0.0.11, Tunnel0
C       192.168.1.0/24       is directly connected, GigabitEthernet2
H       192.168.3.0/24       [250/1] via 10.0.0.23, Tunnel0
! the NHRP route: more specific than the summary, so longest match takes it
C       203.0.113.0/24       is directly connected, GigabitEthernet1

Both entries are in the table at once. The summary still points at the hub and is still perfectly valid; it just stopped being the longest match. Nothing was overridden or withdrawn: a more specific route simply arrived, which is the ordinary rule you already trust, doing the whole job.