NSX-T Meets FRRouting – Part 2

Welcome back! We’re in the process of building an NSX-T Edge – FRRouting environment.

In part 1 we prepared the FRR routers by doing he following:

  • Installed two Debian Linux servers
    • Installed VLAN support
    • Enabled packet forwarding
    • Configured network interfaces
  • Installed and configured VRRP
  • Installed FRRouting

In this second part we will first deploy the NSX-T Edge components and then set up BGP routing. There’s a lot to do so let’s get started!

Target topology

As a refresher here is the big picture once more:

We’ll use this diagram as our blueprint. Scroll back up here any time you wonder what the heck it is we’re doing down there.

Deploy NSX-T Edge

Let’s begin by getting the NSX-T Edge on par with the FRR routers.

Create NSX-T segments

The FRR routers, frr-01 and frr-02, were configured with local “peering” VLANs 1657 and 1658 respectively. Corresponding VLAN-backed segments are needed for L2 adjacency with the FRR routers.

Creating the “vlan-1658” segment:

Both segments in place:

Uplink profile

Create an uplink profile for the edge transport nodes containing settings for teamings, transport VLAN, and MTU:

The transport VLAN has id 1659 and MTU size is 9000.

Deploy Edge VMs

Instead of walking through the Edge node deployment, the table below summarizes the settings I used during the deployment. Have a look at the Single N-VDS per Edge VM article for a detailed Edge node deployment walkthrough.

SettingEdge Node 1Edge Node 2
Nameen01en02
FQDNen01.lab.localen02.lab.local
Form FactorSmallSmall
Mgmt IP172.16.11.61/24172.16.11.62/24
Mgmt InterfacePG-MGMT (VDS)PG-MGMT (VDS)
Default Gateway172.16.11.1172.16.11.1
Transport ZoneTZ-VLAN, TZ-OVERLAYTZ-VLAN, TZ-OVERLAY
Static IP List172.16.59.71, 172.16.59.81172.16.59.72, 172.16.59.82
Gateway172.16.59.1172.16.59.1
Mask255.255.255.0255.255.255.0
DPDK InterdaceUplink1 > Trunk1 (VDS)
Uplink2 > Trunk2 (VDS)
Uplink1 > Trunk1 (VDS)
Uplink2 > Trunk2 (VDS)

The two Edge nodes are up and running:

We add both Edge nodes to an Edge cluster:

Create Tier-0 gateway

With the Edge nodes in place we can create a Tier-0 gateway. I’m configuring it with Active-Standby HA Mode:

We add four external interfaces to the Tier-0:

NameIP addressSegmentEdge Node
en1-uplink1172.16.57.2/29vlan-1657en1
en1-uplink2172.16.58.2/29vlan-1658en1
en2-uplink1172.16.57.3/29vlan-1657en2
en2-uplink2172.16.58.3/29vlan-1658en2

The four Tier-0 interfaces are in place:

Test connectivity

Now is a good time to verify the L2 adjacency between the FRR routers and the Tier-0 interfaces.

A ping from frr-01 to the Tier-0 interfaces in VLAN 1657:

And a ping from frr-02 to the Tier-0 interfaces in VLAN 1658:

Successful pings. We’re good!

Configure BGP

Moving up an OSI layer, we continue with setting up BGP.

Tier-0 gateway

The Tier-0 is configured with the following BGP settings:

SettingValue
Local AS65000
BGPOn
Graceful RestartDisable
ECMPOn

The settings in NSX Manager:

We add two BGP neighbors to the Tier-0: 172.16.57.1 (frr-01) and 172.16.58.1 (frr-02). Make sure to enable BFD for these neighbors too:

The neighbor status will be “Down” at this point which is expected as we didn’t configure BGP on the FRR routers yet.

For route re-distribution I choose to re-distribute from all the available sources into the BGP process:

FRR routers

Configuration of BGP in FRRouting can be done by editing configuration files directly or through VTY shell which is FRRouting’s CLI frontend. We’ll use VTY shell today.

frr-01

Run the vtysh command to start VTY shell:

After changing to the configuration mode with conf t, we enable the BGP process with:

router bgp 65001

Next, we configure the router ID and the BGP/BFD neighbors which are the Tier-0’s interfaces in VLAN 1657 on frr-01:

bgp router-id 172.16.57.1
neighbor 172.16.57.2 remote-as 65000
neighbor 172.16.57.2 bfd
neighbor 172.16.57.3 remote-as 65000
neighbor 172.16.57.3 bfd

We want frr-01 to advertise itself as the default gateway to its BGP neighbors which is accomplished with:

address-family ipv4 unicast
neighbor 172.16.57.2 default-originate
neighbor 172.16.57.3 default-originate

Run end followed by wr to save the configuration:

If all went well we should now see active BGP and BFD sessions between frr-01 and the Tier-0 interfaces in VLAN 1657. Let’s verify this with:

show bgp summary

BGP neighbor sessions are looking good. How about BFD?

show bfd peers

BFD sessions are up.

frr-02

We repeat the exact same configuration steps on frr-02. The configuration for frr-02 looks like this:

router bgp 65001
bgp router-id 172.16.58.1
neighbor 172.16.58.2 remote-as 65000
neighbor 172.16.58.2 bfd
neighbor 172.16.58.3 remote-as 65000
neighbor 172.16.58.3 bfd
!
address-family ipv4 unicast
neighbor 172.16.58.2 default-originate
neighbor 172.16.58.3 default-originate
exit-address-family

Let’s check the BGP/BFD status at frr-02:

show bgp summary
show bfd peers

BGP and BFD sessions are looking good.

Routing

After a lot of deploying and configuring it’s finally time to see if we can actually route any traffic.

FRR routing tables

We begin by having a look at the FRR routing tables. Run the following command in VTY shell on the FRR routers:

show ip route bgp

frr-01:

frr-02:

The FRR routers have learned about each other’s /29 subnets via the NSX-T Tier-0. More specifically, they were learned from neighbor 172.16.57.2 and 172.16.58.2. This tells us that the active Tier-0 SR is hosted on Edge node 1.

Is the standby Tier-0 SR completely out of the picture then? Let’s see:

show bgp detail

The standby Tier-0 SR on Edge node 2 also advertises routes for the same /29 subnets, but as you can see the ASN (65000) is added to the path three more times and packets won’t be routed over these longer paths.

Tier-0 routing table

Run the following command on the Edge node hosting the active Tier-0 SR:

get route bgp

Here we see two equal cost routes for 0.0.0.0/0, one to each FRR router. This tells us that “default-originate” did its job. Both routes also ended up in the FIB which means ECMP is working.

From overlay to physical

It’s now time for the ultimate test. We create an overlay segment, 192.168.10.0/24, connected to the Tier-0 gateway:

The BGP process on the Tier-0 advertises the 192.168.10.0/24 network to its neighbors. Let’s check if they ended up there:

show ip route bgp

frr-01:

frr-02:

A route to the overlay network is indeed present in both of the FRR routers routing table.

Now we connect a VM to the overlay segment and run a traceroute from this VM to an IP address north of the FRR routers:

traceroute 10.2.129.10 -n -q 2

The VM on the overlay segment can reach the physical network. By doing two probes per hop we also see that the Tier-0 offers two paths to the destination: one via frr-01 (172.16.57.1) and one via frr-02 (172.16.58.1).

It’s a wrap

It’s been quite a project, but we got ourselves a working NSX-T Edge – FRRouting environment and it wasn’t that hard to set up, right?

This all started with me looking for a more enterprise like virtual top-of-rack solution for my NSX-T lab. Having these FRR routers north of the Tier-0 certainly feels like a big step towards that goal. Perhaps not fully showcased in these articles, but FRRouting’s feature set is pretty much on par with today’s data center leaf-spine switches. As a matter of fact it’s already being used there. Have a look at Cumulus Networks for example.

For more information about features and possibilities surrounding BGP have a look at the official NSX-T and FRRouting documentation. Most of all I recommend that you set this up yourself. Hopefully these two articles will help you get started with that.

Thanks for reading!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.