Setting up VRF Lite in NSX-T 3.0

NSX-T version 3.0 brings a new routing construct to the table: VRF Lite. With VRF Lite we are able to configure per tenant data plane isolation all the way up to the physical network. Creating dedicated tenant Tier-0 Gateways for this particular use case is now a thing of the past!

With 100 VRFs per Tier-0 Gateway we’re also looking at a quite substantial improvement from a scalability point of view.

A closer look

VRF Lite in NSX-T 3.0 has the following main features and characteristics:

  • Each VRF maintains its own routing table.
  • A VRF acts as a virtual Tier-0 Gateway associated with a “parent” Tier-0 Gateway.
  • Inter-VRF traffic is either routed through the physical fabric or directly by using static routes.

As a child object of a Tier-0 Gateway, the VRF inherits some attributes and configuration from its parent. Edge cluster, HA mode, BGP local AS number, and BGP graceful restart settings are inherited and can’t be changed at the VRF level. All other configuration is managed independently within the VRF. This includes external interfaces, BGP state, BGP neighbors, routes, route filters, route redistribution, NAT, and Edge firewall.

Caveats

There are some things to keep in mind when working with NSX-T VRF Lite:

  • Bandwidth is shared across all Tier-0 Gateways and VRFs.
  • The Tier-0 Gateway’s HA mode (A/A or A/S) is inherited by the VRF. This is an important consideration when talking stateful services on the VRF level.
  • Inter-SR routing for VRF routing instances is not possible today.
  • Inter-VRF static routing does not work with NAT. Route through the physical fabric instead.

Not too bad.

Setting up VRF Lite

Let’s have a look at how to set up VRF Lite for two new tenants: Blue and Green.

In this simple walkthrough the assumption is that a Tier-0 Gateway with external interfaces is already configured. BGP should be enabled but peering or neighbor entries are not required. As a reference, in my lab environment the starting point looks like this:

A Tier-0 Gateway configured with Active/Standby HA mode and four external interfaces (two per Edge node) connecting it to the two ToRs.

Step 1 – Create VLAN segments

Just like its parent, a VRF needs VLAN-based uplink segments for establishing connectivity with the physical network:

Note that we configure a VLAN range indicating that the segment will be trunking VLANs within that range. Trunking segments are required for VRF uplinks.

In total we create four uplink segments for our two VRFs:

SegmentVLAN rangeUplink Teaming Policy
seg-blue-ext-011-50teaming-1
seg-blue-ext-021-50teaming-2
seg-green-ext-0151-100teaming-1
seg-green-ext-0251-100teaming-2

The uplink teaming policies make sure that traffic from each segment is steered towards specific Edge node N-VDS uplinks. This is done to establish a deterministic routing path.

Step 2 – Create the VRFs

Creating VRFs in NSX Manager is done under Networking > Connectivity > Tier-0 Gateways > Add Gateway > VRF:

When creating a VRF we initially only need to specify a name and a parent Tier-0 Gateway:

After repeating this process for the “Green” VRF we have our two VRFs as well as the Tier-0 Gateway in place:

tier-0 gateways

Are we done now? No.

Step 3 – Create VRF external interfaces

Just like Tier-0 Gateways, VRFs need external interfaces to connect to the physical network.

In this scenario each VRF is configured with four external interfaces as specified in the table below:

VRFInterface NameIP AddressSegmentAccess VLAN
Blueen01-uplink110.203.28.2/24seg-blue-ext-0128
Blueen01-uplink210.203.29.2/24seg-blue-ext-0229
Blueen02-uplink110.203.28.3/24seg-blue-ext-0128
Blueen02-uplink210.203.29.3/24seg-blue-ext-0229
Greenen01-uplink110.203.58.2/24seg-green-ext-0158
Greenen01-uplink210.203.59.2/24seg-green-ext-0259
Greenen02-uplink110.203.58.3/24seg-green-ext-0158
Greenen02-uplink210.203.59.3/24seg-green-ext-0259

As you remember, the segments that the external interfaces connect into are configured as trunk segments. Therefore we use the “Access VLAN” property on the VRF external interfaces to specify the BGP peering VLANs.

Step 4 – Configure BGP

With the L2 connectivity in place we can move our focus to L3. As stated earlier, VRFs inherit their BGP local AS number and some other BGP settings from their parent Tier-0, but BGP neighbor configuration is done within each VRF.

Configuring BGP neighbors within a VRF is done exactly the same way as on a Tier-0 Gateway:

There’s not much to explain here. In this particular scenario each VRF gets two BGP neighbor entries (one for each ToR):

Once the neighbor configurations are in place we can have a look at things from the Edge node CLI:

get logical-router

Here we see the two VRFs that we just configured.

After connecting a Tier-1 Gateway to each of the VRFs we can see that DR components are being instantiated for the VRFs:

get logical-router

From within a VRF context we can check things like the BGP neighbor status:

vrf 5
get bgp neighbor summary

And the BGP routing table for this particular VRF:

get bgp ipv4

Inevitably, VRFs add some complexity to NSX-T Edge routing. I recommend using the Network Topology map in NSX Manager which is a pretty nice tool for keeping an overview of the routing configuration:

Summary

The new VRF Lite feature introduced in NSX-T 3.0 is a great addition to the platform. It gives customers scalable data plane isolation all the way into the physical network. VRF Lite is easy to set up and maintain and will definitely become the go-to configuration in NSX-T multitenancy environments.

Thanks for reading.

12 Comments

  1. Danny
    Permalink

    Can a tier1 be connected to both a tier0 that’s part of a vrf and a tier0 in the “global” routing table?

    Like

    Reply
    • rutgerblom
      Permalink

      Hi Danny,

      If you mean “connect” in the sense of within the NSX-T management plane connecting a T1 construct to both a T0 construct and a VRF construct the answer is no. You can however “connect” through the data plane by using static routes.

      Cheers

      Like

      Reply
      • Danny
        Permalink

        So from the perspective of the Tier1, would I statically route the networks that I want to reach via the VRF to the Tier0 in that VRF and the auto-configured default route would hit the other Tier0 instance?

        Like

      • rutgerblom
        Permalink

        When you connect a Tier1 to a Tier0 or a VRF a default route is auto-plumbed on the Tier1 pointing to either the Tier0 or the VRF as next hop, but not both.
        From the Tier1 point of view a Tier0 and a VRF are separate Tier0 gateways.

        Like

      • Danny
        Permalink

        I see. If you want to provide both Internet services via NAT and access to a VRF for a virtual server behind a Tier1, is it possible to do that?

        Like

      • rutgerblom
        Permalink

        Yes that’s fine. One thing to keep in mind is the HA mode. If you want to do NAT on the VRF its parent Tier0 must have been configured with active/standby HA mode.

        Like

  2. Danny
    Permalink

    I guess I’m confused as to how that would be accomplished if you can only connect a Tier1 to a single Tier0. The connection would either be in the VRF or in the ‘global/default’ routing table.

    Like

    Reply
    • rutgerblom
      Permalink

      To a single Tier0 or a single VRF. From the Tier1 point of view it does not make any difference. If it’s connected to a Tier0 you configure all of your stateful services on the Tier0 and if it’s connected to a VRF you configure them there. Either way your VM behind the Tier1 gets its stateful service (like NAT) and access to the physical network.

      Like

      Reply
  3. Redouane BALI
    Permalink

    Hello, First of all thanks for the nice example and how you explain that. Do you know how the configuration should be done if we want to do the vrf leaking like between green vrf and T0 global. I did that with static route and scope but it doesnt work, i even created a Lo in both of them but no way.

    Like

    Reply
    • Redouane BALI
      Permalink

      Actually just resolve the prb, the next hope of the VRF should be the next hope of the T0 and the next hope of the T0 should be the next hope of vrf to T1 which is 100.64.64.x/31

      Liked by 1 person

      Reply
  4. Ronald de Jong
    Permalink

    Thanks for the excellent write-up @Rutger, have gone back to this article multiple times during a design I am writing, with my own test-environment in transit :).

    Liked by 1 person

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.