Posted by The NSX-T Central Control Plane (CCP) is building and maintaining a central repository for some tables that make NSX-T the unique network virtualization solution it is. More specifically I’m talking about:
- The Global MAC address table
- The Global ARP table
In today’s article I’ll have a closer look at these two tables.
MAC Address Table
As soon as a virtual machine’s vNIC is connected to the NSX-T Data Plane, its MAC address as well as the Tunnel End Point (TEP) used to reach that MAC address are registered with the CCP. Now, when the Data Plane receives a frame destined to an unknown MAC address, besides flooding the frame, it will also query the CCP’s MAC address table to see if it can find a matching entry there. The CCP’s MAC address table is also used to pre-populate the local MAC address tables on Transport Nodes before they receive any traffic.
There are two exceptions where MAC addresses of connected vNICs are not registered with the CCP. The first exception is when a vNIC is allowed to send traffic from several source MAC addresses. The second exception is when MAC addresses are learned from an Edge bridge connected to a physical layer 2 network. This is by design and protects the CCP from injection of an arbitrarily large number of MAC addresses into in the network.
So, that’s a pretty cool table, right? One that you might want to have a look at yourself now and then perhaps.
Querying the MAC Address Table
There’s more than one way to retrieve entries from the CCP’s MAC address table. In this article I will show you how it’s done using the Manager CLI. Another option would be to leverage the NSX-T API using curl for example.
We query the MAC address table on a per NSX-T segment basis. To see the learned MAC addresses and their associated TEPs for a segment we first need to know that segment’s Virtual Network Identifier (VNI). From the Manager CLI we run the following command to list segments and their VNIs:
get logical-switches
This gives the following result:
VNI UUID Name Type 65542 e1b15ca9-4c04-4692-8926-a4cd769b4776 Web DEFAULT 65538 0058ae01-04cd-4992-9c2c-60fb764bbad1 App DEFAULT
In this case we’re interested in the “App” segment which has VNI 65538. We can now run the following command to see the learned MAC address entries for the “App” segment:
get logical-switch 65538 mac-table
The output of this command in my tiny lab:
VNI MAC VTEP-IP TransportNode-ID 65538 00:50:56:a4:b6:e5 10.81.234.16 4173bca0-4e6e-4ffb-8300-f2f4bed88b29 65538 00:50:56:a2:bd:ec 10.81.234.17 78eeca52-69a8-44f6-a795-f1a1ecc7bdf7
As we can see, the table contains two MAC addresses that belong to vNICs connected to the “App” segment. The MAC addresses are reachable via two different TEP IP addresses which are shown in the “VTEP-IP” column.
Each entry also contains a value for the Transport Node ID. This tells us on which transport node the MAC address is connected and basically discloses on which host the virtual machine is running. To translate a Transport Node ID to an ESXi Management IP address we would run:
get transport-node <TransportNode-ID> status
ARP Table
The Central Control Plane also maintains a global ARP table. Thanks to this table we enjoy things like ARP suppression on our NSX-T Data Plane. It’s populated by snooping DHCP and ARP traffic. The snooping itself happens on the individual transport nodes and results are reported back to the CCP.
Querying the ARP Table
Retrieving information from the CCP’s ARP table can be done in different ways as well, but we’ll stick to the Manager CLI today.
As with the MAC address table, querying the ARP table is done on a per segment basis. If we for example would like to see the ARP entries for the “Web” segment, we first need to know that segment’s VNI. In this case we already know that the “Web” segment’s VNI is 65542 and continue by running:
get logical-switch 65542 arp-table
ARP entries are displayed:
VNI IP MAC TransportNode-ID 65542 10.80.1.20 00:50:56:a2:bd:ec 78eeca52-69a8-44f6-a795-f1a1ecc7bdf7 65542 10.80.1.21 00:50:56:a4:b6:e5 4173bca0-4e6e-4ffb-8300-f2f4bed88b29
The two ARP entries showing MAC address and IP address of the connected vNICs. The Transport Node ID is attached as well and can be used to find out on which Transport Node the IP/MAC (virtual machine) is connected.
Modifying Output
With just a couple of entries in these tables, finding relevant information is easy. It becomes a whole different story when hundreds or thousands of virtual machines are connected to a single segment. Luckily we have the option to modify the output:
get logical-switch 65542 arp-table | count Count number of entities find Only show lines that contain regex pattern first Show first N lines of output ignore Ignore lines that contain regex pattern json Show output in JSON format last Show last N lines of output more Show output one page at a time sort Sort command output
To use a very simple example. If we just want to know the number of ARP entries for the “Web” segment that has a 10.80.1.0/24 CIDR, something like this would do the trick:
get logical-switch 65542 arp-table | count 10.80.1.
Output is modified and now looks like this:
Number of lines that match pattern '10.80.1.': 2
Regex patterns are used to filter the command output. Depending on your regex skills (I suck at it) you could construct a pretty advanced query and extract exactly the information that you are looking for.
Summary
In today’s article I went back to some NSX-T basics and looked closer at two important tables that are living within the NSX-T Central Control Plane. The information in these tables is likely available in other systems around your SDDC (vCenter, vROps, vRLI, vRNI, physical network, etc), but personally I think it’s important to know how to extract this information at the source (NSX-T). Sooner or later you will end up in a situation where you depend on that knowledge. 😉
Thanks for reading.
References and resources:
rutgerblom.com 