Welcome back! In part two we deployed the data plane for the East-West traffic in our cross-vCenter NSX environment. In this part we continue with the setting up the components for the North-South traffic.
A quick look at the current state of our cross-vCenter NSX lab environment.
The management plane and the control plane are operational and configured for cross-vCenter NSX. We have deployed the data plane components for East-West traffic; two universal logical switches and a universal distributed logical router.
The NSX edge is responsible for the central on-ramp/off-ramp routing between the NSX logical networks (VXLANs) and the physical network. It’s a central function and is deployed as virtual appliances (edge services gateways) by NSX Manager.
There are some design considerations surrounding the NSX edge. In a cross-vCenter NSX environment spanning multiple physical sites one consideration is of particular interest: should each site use a local edge for North-South egress (local egress), or should one site act as the central edge for the other sites?
You might remember from part two that we deployed the universal distributed router with local egress enabled. So what we’ll do, is set up our fictional sites “DC-SE” and “DC-US” with active/active egress. This is actually a less common/popular type of cross-vCenter NSX deployment as it introduces asynchronous traffic flows (traffic egresses one site and ingresses at another site) which need to be dealt with somehow.
When local egress is a requirement we need to deploy some additional components. We need a UDLR control VM at each site. Each site will also use a separate transit universal logical switch.
So, let’s create the transit logical switches first and come back to the control VMs a little later.
Log in at the DC-SE’s vCenter and navigate to Networking and Security > Logical Switches. Click the “+ Add” button. Type a name for the logical switch. In my case I’ll call it “ULS Transit SE”. This logical switch will use the universal transport zone “UTZ” that we created in part one. Click “Add“. Repeat these steps to add the second transit universal logical switch. I’m calling this one “ULS Transit US”.
With the transit switches created we continue with the deployment of the ESG appliances. For this lab we’ll deploy just one ESG per site. At DC-SE navigate to Networking and Security > NSX Edges. Click the “+Add” button and choose “Edge Services Gateway”.
Give the ESG a name. In my lab it’s called “esg-se”.
Configure a user name and password. Enable SSH.
Configure the appliance VM deployment. A compact sized appliance will do for this lab.
Next we configure two interfaces on the ESG: one uplink and one internal. Starting with the uplink interface that we connect to a VLAN-backed distributed port group (VLAN 70 at my DC-SE site). This one connects the ESG with the pfSense router. I’m assigning IP address 10.0.70.2/24 to this interface (the pfSense router’s interface is configured with 10.0.70.1/24).
The internal interface connects the ESG with the “ULS Transit SE” logical switch. It is the link between the ESG and the UDLR. I’m assigning 192.168.100.1/29 to this interface.
We also configure a default gateway on “esg-se” which points to the next-hop router on the “physical” network. In my case this is 10.0.70.1 (pfSense router).
Review the configuration and click “Finish” to deploy the ESG appliance.
At the DC-US’s vCenter we basically repeat these steps for the ESG deployment over there. The unique settings for the ESG at DC-US in my lab are:
|Uplink interface IP (VLAN 700)||10.1.70.2/24|
|Internal interface connected to||ULS Transit US|
|Internal interface IP||192.168.101.1/29|
Universal Distributed Logical Router
We need to revisit the UDLR to deploy the additional UDLR control VM at DC-US as well as configure connectivity between the UDLR and the new transit logical switches.
Log in at vCenter in DC-US and navigate to Networking and Security > NSX Edges and click on the universal distributed router. Select Configure > Appliance Settings. Click on “Add Edge Appliance VM“.
Configure the placement parameters and deploy the VM.
Once the control VM has been deployed we head over to vCenter at DC-SE and navigate to Networking and Security > NSX Edges and open up the universal distributed router. Select Configure > Interfaces and add a new uplink interface that connects to the “ULS Transit SE” logical switch. I configure this interface with IP address 192.168.100.2/29.
Add a second uplink interface to the UDLR and connect it to the “ULS Transit US” logical switch. I configure this one with IP address 192.168.101.2/29.
With the UDLR uplinks configured we’ll do a quick connectivity test. Open an SSH session to the ESGs and ping the IP address of the UDLR uplink interface connected to that ESG.
Now that we have a working L2 connection between the ESGs and the UDLR, we can focus on setting up the North-South routing in our cross-vCenter NSX environment.
In my lab I’ve configured an iBGP peering between the UDLR control VMs and their respective ESG appliance. I also configured an eBGP peering between the ESGs and the pfSense routers at each site.
I won’t go through setting up dynamic routing with BGP, but here’s a summary of the configuration that I used in my lab:
|pfSense Local AS||65502||65502|
|ESG default gateway IP||10.0.70.1||10.1.70.1|
|ESG Local AS||65510||65110|
|ESG BGP neighbors||10.0.70.1, 192.168.100.3||10.1.70.1, 192.168.101.3|
|UDLR default gateway||192.168.100.1||192.168.101.1|
|UDLR Local AS||65510||65510|
|UDLR protocol address||192.168.100.3||192.168.101.3|
|UDLR BGP neighbors||192.168.100.1||192.168.101.1|
Setting up dynamic routing in cross-vCenter NSX with local egress can be quite a job. Even in a small lab like this we need to configure routing parameters at 6 places: pfSense x 2, ESG x 2, UDLR control VM x 2.
The big picture
Now that we have configured North-South routing let’s have another look at the environment from above.
Let’s do a simple test with traceroute and see if local egress works.
From AppServer-SE (192.168.0.20) we are egressing via “esg-se”:
From AppServer-US (192.168.0.22) we are egressing via “esg-us”:
A simple test, but local egress seems to work!
This concludes the blogpost series on cross-vCenter NSX. During these three posts we went through the process of setting up cross-vCenter NSX with local egress. We deployed and configured the different components and had a look at the data plane functions: universal East-West routing, universal distributed firewall and finally North-South routing in a cross-vCenter NSX environment.
All in all setting this up is a pretty straight forward process. Especially in a lab where we have control over the entire environment. 😉