WAN Architecture: SOHO and Metro Ethernet

A company rarely owns cable across town; it buys WAN services. Metro Ethernet carries Ethernet frames between sites, in three shapes the MEF defines: E-Line, point to point; E-LAN, a full mesh; and E-Tree, hub and spoke. An MPLS VPN works one layer up, routing the customer's IP packets and keeping each customer's routes separate with labels.

In Daveth Pro (Network)12 min video

By , founder of Daveth · Updated

Opening frame of the video walkthrough of WAN Architecture: SOHO and Metro Ethernet

Watch this lesson

12 min · Video walkthrough

In Daveth Pro (Network)

The full 12 min walkthrough is in Daveth Pro (Network). See what the plan opens

What this page answers

What is Metro Ethernet?

A WAN service that carries Ethernet frames between a customer's sites across a provider's network. The MEF defines three shapes: E-Line, point to point; E-LAN, full mesh; and E-Tree, hub and spoke. Read it in “Three shapes of service”

What is a WAN (wide area network)?

The links that join a company's sites, usually services bought from a provider rather than cable the company owns, such as Metro Ethernet lines or an MPLS VPN. Read it in “What the WAN links really are”

What is an MPLS VPN?

A provider service that routes the customer's IP packets between sites, forwarding by labels inside its network so each customer's routes stay separate, even when customers use the same private addresses. Read it in “A service that routes: MPLS VPNs”

What the WAN links really are

Friday, month nine, and the backup line's last day: its lease ends tonight, as planned back in month seven (08.1.4). Before anything else changes, Maya writes down what the company's three WAN links actually are. They have been drawn as cables since they were built:

Network drawing: E-Line, 1 Gbps, E-Line (the fibre), 1 Gbps, E-Line (backup), 10 Mbps: ends today, R1, R3, BR1

The company's WAN: R1 (the office), R3 (head office) and BR1 (the branch), joined by three E-Lines.

This company owns no cable across town. Each link is a Metro Ethernet service bought from a provider: a Layer 2 WAN service that forwards Ethernet frames between the customer's sites, as though the provider were one very large Ethernet switch. Each site has its own access link, an Ethernet link to the provider's switch in a nearby point of presence (PoP), which the provider places close to as many customers as it can. What happens on that link is defined as the user network interface (UNI): the user is the customer, the network is the provider's. The provider's switch reads the frames' MAC addresses and 802.1Q VLAN tags; beyond that, its network stays hidden.

The UNI allows any IEEE Ethernet standard on the access link, but because the link must reach the PoP it normally uses one of the longer-distance standards over fibre, rather than those for UTP cable: 100BASE-LX10 (100 Mbps, 10 km), 1000BASE-LX (1 Gbps, 5 km), 1000BASE-LX10 (1 Gbps, 10 km), 1000BASE-ZX (1 Gbps, 100 km), 10GBASE-LR (10 Gbps, 10 km) or 10GBASE-ER (10 Gbps, 40 km).

All three of the company's links are the simplest service, an E-Line (Ethernet Line Service): point-to-point, like a leased line, between two pieces of customer premises equipment (CPE). An Ethernet virtual connection (EVC) defines which customer devices may talk to which, and an E-Line's EVC is point-to-point. For the routers it looks like a long crossover cable: Ethernet interfaces, addresses in one subnet (each link its own /30), and routing neighbours, just as OSPF found them in month three. A central site with a hundred remote sites would otherwise need a hundred router interfaces, so one fast access link can carry several E-Lines instead, one VLAN each over 802.1Q, with a subinterface per E-Line. Routers traditionally sit at the WAN edge because the LAN and WAN used different links and data-link protocols; with Metro Ethernet both sides speak Ethernet, so a switch, including a Layer 3 switch, can sit there too. Other names you may meet, when a provider builds these services over MPLS inside, are VPWS for an E-Line and VPLS for an E-LAN.

If you've ever posted a parcel, an E-Line will feel familiar: you hand it over at the counter near you, and it turns up at the other end; the route in between is the courier's business. (Where it stops: a courier delivers anywhere; an E-Line delivers only between its two ends.)

Three shapes of service

E-Line isn't the only shape. The standards body for Metro Ethernet, the MEF, defines three:

  1. E-Line (Ethernet Line Service): point-to-point. Two sites exchange frames.
  2. E-LAN (Ethernet LAN Service): full mesh, a direct path for every pair of sites, as if on one LAN.
  3. E-Tree (Ethernet Tree Service): hub and spoke, also called point-to-multipoint or partial mesh (a direct path for only some pairs). A root site can send to every leaf site; leaf sites can send only to the root. It suits one central site with many remote ones.

The shape matters as soon as the building across the car park (09.2) is more than a what-if. Two questions first.

Interactive tool: Joining the sites Sites 4 Design E-Lines between every pair of sites Services to buy 6 WAN subnets to plan 6 Routing neighbours of a site other than head office 3 Routing neighbours of head office's router 3 One of the other sites reaches another directly Four sites: the office (R1), head office (R3), the branch (BR1) and, if it happens, the building across the car park.

Each E-Line is its own subnet, and its two routers are neighbours. Joining N sites all to each other with E-Lines needs N(N-1)/2 of them: 3 for three sites, 6 for four, 10 for five, and 190 for twenty. With E-Lines from head office only, each other site has one neighbour, head office's router, and reaches the rest through it: the next hop for another site's subnet is head office's address on that E-Line. An E-LAN behaves like one switch: one service, one subnet for every router's WAN address, and every router a neighbour of every other, sending straight to the next hop's address in that subnet. For the company's OSPF that would mean a change: its WAN ports use the point-to-point network type (05.2.5), and a shared E-LAN would take them back to the broadcast type, with a DR elected again.

A working service can still carry nothing useful. The access link can be up at both ends while the frames are tagged with the wrong VLAN for the service, or the router has no route to use it. Check the access link, the service and the routing separately.

A train can be running beautifully to the wrong platform.

A service that routes: MPLS VPNs

The other large WAN service works one layer up. An MPLS VPN routes the customer's IP packets between sites. Inside, the provider forwards by labels (the name is Multiprotocol Label Switching), which lets it keep each customer's routes and traffic separate even though customers share its routers. It needs that: many customers use the same private addresses, such as 10.0.0.0, and a plain shared IP network would mix their overlapping routes. The provider's edge routers add an MPLS header to each packet on the way in and remove it on the way out, and the devices inside forward by its label; because that header sits between the data-link and IP headers, MPLS is sometimes called Layer 2.5. That is what private means here: separate, not encrypted. The routers are named by where they sit: customer edge (CE) at the customer's site, provider edge (PE) at the far end of the access link.

Because the PE routes, discarding each incoming frame's data-link header and building a new one, the access link can be almost anything: Metro Ethernet where it is offered (a good fit from about 10 Mbps up), a serial line for remote sites, cable or 4G/5G. The company picks the type and speed for each site, and agrees the routing design and quality of service with the provider. MPLS was the first WAN service with effective quality of service: voice needs low loss, low delay and low jitter (variation in delay), so the customer marks voice packets with DSCP EF and the provider's network sends them first, as agreed in the contract (07.4).

A CE router becomes a routing neighbour of its PE, and not of the other CE routers; the pair can use RIPv2, EIGRP, OSPF or eBGP (BGP, the internet's routing protocol, run between organisations). The PEs carry the customer's routes between them with multiprotocol BGP, which marks which routes belong to which customer, so every CE learns every site's routes, each with its own PE as the next hop. When the CE-PE protocol isn't BGP, the PE uses route redistribution, taking routes from one routing protocol process and injecting them into another inside the same router. With Metro Ethernet, the provider knows nothing about the customer's addressing; with MPLS, it has to.