Around the NSX-T Table(s)

The NSX-T Central Control Plane (CCP) is building and maintaining a central repository for some tables that make NSX-T the unique network virtualization solution it is. More specifically I’m talking about:

  • The Global MAC address table
  • The Global ARP table

In today’s article I’ll have a closer look at these two tables.

MAC Address Table

As soon as a virtual machine’s vNIC is connected to the NSX-T Data Plane, its MAC address as well as the Tunnel End Point (TEP) used to reach that MAC address are registered with the CCP. Now, when the Data Plane receives a frame destined to an unknown MAC address, besides flooding the frame, it will also query the CCP’s MAC address table to see if it can find a matching entry there. The CCP’s MAC address table is also used to pre-populate the local MAC address tables on Transport Nodes before they receive any traffic.

There are two exceptions where MAC addresses of connected vNICs are not registered with the CCP. The first exception is when a vNIC is allowed to send traffic from several source MAC addresses. The second exception is when MAC addresses are learned from an Edge bridge connected to a physical layer 2 network. This is by design and protects the CCP from injection of an arbitrarily large number of MAC addresses into in the network.

So, that’s a pretty cool table, right? One that you might want to have a look at yourself now and then perhaps.

Querying the MAC Address Table

There’s more than one way to retrieve entries from the CCP’s MAC address table. In this article I will show you how it’s done using the Manager CLI. Another option would be to leverage the NSX-T API using curl for example.

We query the MAC address table on a per NSX-T segment basis. To see the learned MAC addresses and their associated TEPs for a segment we first need to know that segment’s Virtual Network Identifier (VNI). From the Manager CLI we run the following command to list segments and their VNIs:

get logical-switches

This gives the following result:

VNI     UUID                                  Name   Type      
65542   e1b15ca9-4c04-4692-8926-a4cd769b4776  Web    DEFAULT
65538   0058ae01-04cd-4992-9c2c-60fb764bbad1  App    DEFAULT

In this case we’re interested in the “App” segment which has VNI 65538. We can now run the following command to see the learned MAC address entries for the “App” segment:

get logical-switch 65538 mac-table

The output of this command in my tiny lab:

VNI     MAC               VTEP-IP        TransportNode-ID                            
65538   00:50:56:a4:b6:e5 10.81.234.16   4173bca0-4e6e-4ffb-8300-f2f4bed88b29
65538   00:50:56:a2:bd:ec 10.81.234.17   78eeca52-69a8-44f6-a795-f1a1ecc7bdf7

As we can see, the table contains two MAC addresses that belong to vNICs connected to the “App” segment. The MAC addresses are reachable via two different TEP IP addresses which are shown in the “VTEP-IP” column.

Each entry also contains a value for the Transport Node ID. This tells us on which transport node the MAC address is connected and basically discloses on which host the virtual machine is running. To translate a Transport Node ID to an ESXi Management IP address we would run:

get transport-node <TransportNode-ID> status 

ARP Table

The Central Control Plane also maintains a global ARP table. Thanks to this table we enjoy things like ARP suppression on our NSX-T Data Plane. It’s populated by snooping DHCP and ARP traffic. The snooping itself happens on the individual transport nodes and results are reported back to the CCP.

Querying the ARP Table

Retrieving information from the CCP’s ARP table can be done in different ways as well, but we’ll stick to the Manager CLI today.

As with the MAC address table, querying the ARP table is done on a per segment basis. If we for example would like to see the ARP entries for the “Web” segment, we first need to know that segment’s VNI. In this case we already know that the “Web” segment’s VNI is 65542 and continue by running:

get logical-switch 65542 arp-table

ARP entries are displayed:

VNI     IP          MAC                TransportNode-ID            
65542   10.80.1.20  00:50:56:a2:bd:ec  78eeca52-69a8-44f6-a795-f1a1ecc7bdf7  
65542   10.80.1.21  00:50:56:a4:b6:e5  4173bca0-4e6e-4ffb-8300-f2f4bed88b29

The two ARP entries showing MAC address and IP address of the connected vNICs. The Transport Node ID is attached as well and can be used to find out on which Transport Node the IP/MAC (virtual machine) is connected.

Modifying Output

With just a couple of entries in these tables, finding relevant information is easy. It becomes a whole different story when hundreds or thousands of virtual machines are connected to a single segment. Luckily we have the option to modify the output:

get logical-switch 65542 arp-table | 
   count     Count number of entities
   find      Only show lines that contain regex pattern
   first     Show first N lines of output
   ignore    Ignore lines that contain regex pattern
   json      Show output in JSON format
   last      Show last N lines of output
   more      Show output one page at a time
   sort      Sort command output

To use a very simple example. If we just want to know the number of ARP entries for the “Web” segment that has a 10.80.1.0/24 CIDR, something like this would do the trick:

get logical-switch 65542 arp-table | count 10.80.1.

Output is modified and now looks like this:

Number of lines that match pattern '10.80.1.': 2

Regex patterns are used to filter the command output. Depending on your regex skills (I suck at it) you could construct a pretty advanced query and extract exactly the information that you are looking for.

Summary

In today’s article I went back to some NSX-T basics and looked closer at two important tables that are living within the NSX-T Central Control Plane. The information in these tables is likely available in other systems around your SDDC (vCenter, vROps, vRLI, vRNI, physical network, etc), but personally I think it’s important to know how to extract this information at the source (NSX-T). Sooner or later you will end up in a situation where you depend on that knowledge. 😉

Thanks for reading.

References and resources:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.