Posted by By deploying multiple ESG appliances and enabling Equal Cost Multi-Path (ECMP) on the Distributed Logical Router (DLR), NSX Data Center provides scalability and redundancy for northbound traffic in our SDDC.
Setting up ECMP in NSX Data Center is fairly easy. Let me show you.
The diagram
A high level overview of the NSX infrastructure for this excercise.
The goal in this exercise is to provide two equal cost routes on he DLR for northbound traffic. For this we are going to deploy two ESG appliances, configure dynamic routing between these and the DLR, and then finally enable ECMP.
Let’s get started!
Deploying the ESG appliances
Create the first ESG appliance.
Add two interfaces: One connected to a VLAN-backed port group leading to the physical network, the other one connected to the logical transit switch leading to the DLR.
Don’t forget to configure a default gateway pointing to the next hop on the physical network.
Once deployed we can go ahead and deploy the second ESG appliance in the same way.
Create the same interfaces as on “Edge-01”. Use unique IP addresses though. 😉
Configure dynamic routing on the DLR
There won’t be much multi-pathing going on without a working routing process between the DLR and the ESGs so let’s make sure that is in place first.
Starting with the DLR. We first need to add an uplink interface that connects the DLR to the transit switch:
We’ll be doing dynamic routing using BGP in this exercise so navigate to the DLR’s “Routing” tab and set the router ID:
Next navigate to “BGP”, set a private ASN (64 512 – 65 534), and enable BGP:
While we’re here we also add our two future BGP neighbors. Click the green “plus” sign under “Neighbors” to add the first BGP neighbor:
Now what’s all of this?
First we need to specify the IP address of our BGP neighbor (Edge-01 in the above screen). Next we have a forwarding address (data plane) which is the IP-address configured on the DLR’s uplink interface (2ESG in my case). The protocol address is used for the BGP peering/neighbor relationship which can be any unused IP address in the transit network (192.168.100.0/24 in my case). Finally we supply the remote ASN which is the private ASN of the BGP neighbor (65501 in my case).
Click “OK” and then add the second BGP neighbor (Edge-02). We use the same protocol address as for the first neighbor (one protocol address per uplink). Also type the same remote ASN as for the first BGP neighbor. Once done it should look something like this:
The last thing we need to configure on the DLR is “Route Redistribution”.
Enable route redistribution for BGP. Also add BGP to the route redistribution table and configure it to learn connected routes.
Configure dynamic routing on the ESGs
With BGP configured on the DLR we move over to to our ESGs.
Just like on the DLR we need a router ID on our ESGs. Starting with “Edge-01” go to the “Routing” tab and configure the the router ID:
Next we configure the BGP which is almost identical to the way we configure BGP on the DLR.
Enable BGP and type a private ASN for the ESG AS (65501 in my case). We might want to enable “Default Originate” so that the ESG will advertise a default gateway to the DLR. Otherwise we’ll have to configure a default gateway on the DLR manually.
We continue with adding the DLR as a BGP neighbor:
Here things look slightly different compared to the DLR. No protocol address and no forwarding address are needed. Why not?
With the DLR the control plane and the data plane are separate from each other. What this means is that the forwarding address lives in the data plane, which is a distributed plane (ESXi hosts). The protocol address on the other hand is tied to the DLR control VM. The ESG is a central component where data plane and control plane both reside in the ESG appliance.
Next configure BGP route redistribution the same way as we did on the ESGs.
Before we move on with the second ESG, let us just check that we have a working BGP neighbor relationship between the DLR and Edge-01 at this point.
Open the console of the DLR control VM and log in. Issue the “show ip bgp neigbors summary” command. At this point we should see one established BGP neighbor relationship:
Next type “show ip route bgp”. Here we should see one (or two if default originate was enabled) BGP derived routes in the routing table:
If everything looks as expected we can go ahead and configure BGP on Edge-02. The configuration steps are identical to the ones we did on Edge-01.
With Edge-02 configured and peered with the DLR, the BGP neighbor table on the DLR should now show two BGP neighbor relationships:
ECMP
What we’ve built so far gives us better availability. When Edge-01 goes down, the BGP process on the DLR will after some time, depending on the configured “keep alive” and “hold down” timers, use the routes advertised by Edge-02 instead and traffic starts flowing again.
But with this setup we only have one active path from the DLR to the edge and beyond. To activate multi-pathing we simply need to enable ECMP on the DLR.
Head over to the DLR management and click the “Routing” tab > “Global Configuration” and start ECMP.
At the DLR console issue the “show ip route bgp” command once again. We’ll notice a difference:
We now have two BGP routes for every destination. Each of the Edge’s route advertisements end up as equal cost routes in the DLR’s routing table. The DLR will now use both these paths simultaneously. It does this by using a (non-configurable) hashing algorithm based on source and destination IP address.
Conclusion
Multiple ESG appliances combined with ECMP provides scalability and availability for northbound traffic. If your SDDC grows the size where a dedicated Edge cluster becomes best practice design, ECMP should be considered. Up to 8 equal paths (8 ESG appliances) can be provisioned to make sure the necessary bandwidth and redundancy are available to this critical part of the NSX infrastructure.
rutgerblom.com 