Cross-vCenter NSX – Part 2

Welcome back! In part one we prepared the management and control plane for cross-vCenter NSX. In this part we’re going to deploy the data plane for the East-West traffic in a cross-vCenter NSX environment.

Universal logical switch

We start by deploying logical switching between our two fictional sites “DC-SE” and “DC-US”. To accomplish this we create a “stretched” logical switch. In cross-vCenter NSX terminology this is called a universal logical switch.

With cross-vCenter NSX, the vCenter system paired with the primary NSX Manager is the point of administration for NSX universal constructs. In my environment the vCenter at DC-SE is paired with the primary NSX Manager.

Log in at the DC-SE’s vCenter and navigate to Networking and Security > Logical Switches. Click the “+ Add” button. Type a name for the logical switch,. In my lab I’ll call it “ULS App”. This logical switch will use the universal transport zone “UTZ” that was created in part one. Click “Add“.

By picking a universal transport zone for a logical switch, the logical switch itself becomes universal and will be synced to secondary NSX Managers.

Testing connectivity

We now have a logical switch that exists on both of the sites. The VXLAN tunneling protocol and the VTEPs are responsible for carrying the frames over layer 3 between the sites. If we now connect virtual machines on both sites to the same universal logical switch, we should have layer 2 connectivity between them.

Let’s test this by deploying a virtual machine at each site and connect them to the “ULS App” universal logical switch.
I like Photon OS for lab purposes so I’m going to deploy two of these using the OVA.

Deploy the first virtual machine in DC-SE.

I’ll call it “AppServer-SE”.

And connect it to the “ULS App” logical switch.

Repeat this process over at DC-US. In my lab that virtual machine is called “AppServer-US”.

The first time we boot Photon OS deployed via the OVA, we need to change the root password. The default password is “changeme”.

I also want to set a hostname to avoid any confusion later on. Reboot the virtual machines after this.

At this point let’s have a look what the universal controller cluster knows about this logical segment. Start an SSH session to one of the NSX Managers and run the following command (replace “25000” with the VNI of your universal logical switch):

show logical-switch controller master vni 25000 mac

This looks good. The control plane has picked up the MAC addresses of both virtual machines. We can also see that the VTEPs of ESXi hosts on both sites are involved.

Do we have ARP entries?

show logical-switch controller master vni 25000 arp

Empty as expected. Let’s configure an IP address on the virtual machines.

AppServer-SE:

ifconfig eth0 192.168.0.10 netmask 255.255.255.0

AppServer-US:

ifconfig eth0 192.168.0.11 netmask 255.255.255.0

On both virtual machines we need to create iptables rules to allow ICMP:

iptables -A OUTPUT -p icmp -j ACCEPT
iptables -A INPUT -p icmp -j ACCEPT

And let’s see if the virtual machines can ping each other:

It works! Do we have ARP entries at the control plane now?

show logical-switch controller master vni 25000 arp

There they are!

So, we have verified L2 connectivity between the virtual machines which are located at different sites. Multi-site VXLAN in action!

Step 2 – Universal distributed logical router

East-West routing in a cross-vCenter NSX scenario is accomplished by deploying a universal distributed logical router. So let’s do that.

Log in at the DC-SE vCenter system and navigate to Networking and Security > NSX Edges. Click the “+Add” button and choose “Universal Distributed Logical Router”.

Give it a name like “UDLR-01”. Make sure you enable “Local Egress” which we’ll talk about in part three. Also tick the “Deploy Control VMs” box.

Set a password and enable SSH.

Configure the control VM deployment and choose a connection for the HA interface.

At this point we’ll just add one internal interface to the UDLR and attach it to the “ULS-App” logical switch that we created earlier. In my environment I call the interface “App-Tier” and assign it IP address 192.168.0.1 with a 24 prefix.

Disable “Configure Default Gateway” and complete the deployment wizard.

Once the UDLR is deployed we can test connectivity from one of the virtual machines by running a ping to the “App-Tier” interface IP address.

That should work as expected. Let’s add another universal logical switch to the environment so we can test some basic routing.

Once again in the vCenter system paired with the primary NSX Manager, navigate to Networking and Security > Logical Switches. Click the “+ Add” button. Type a name for the universal logical switch. I call it “ULS DB” in my environment. Once again choose “UTZ” as the transport zone. Click “Add“

Once created select the new “ULS DB” switch and click on Actions > Connect Edge. Select the “UDLR-01” edge.

Give the interface a name. I call it “DB-Tier” and select “Internal” as the interface type. Make sure the connectivity status is set to “Connected“. Assign IP address 192.168.1.1 with a 24 prefix and click “Add“.

Deploy a new virtual machine at one of the sites and connect it to the “ULS DB” logical switch.

Once the virtual machine is deployed, set a hostname. Mine is called “DBServer-SE” and reboot. Next configure it with IP address 192.168.1.10/24. Add a rule to iptables to allow ICMP. You should now be able to ping the DB-Tier UDLR-01 interface (192.168.1.1):

Can we reach any of the virtual machines connected to the App Tier at this point? Yes, but as none of the virtual machines have a default gateway configured it won’t work. Let’s quickly fix this:

For the App server virtual machines:

route add default gw 192.168.0.1 eth0

For the DBServer virtual machine:

route add default gw 192.168.1.1 eth0

And now routing between the two segments should work:

We can also use the NSX traceflow utility to visualize the path of the ICMP packets. From any of the vCenter systems navigate to Networking and Security > Tools > Traceflow. Do a trace from the DBServer-SE to AppServer-US for example:

Step 3 – Universal distributed firewall

Now that we have cross-vCenter routing and switching working it’s time to look at security.

Just as with universal distributed logical routing and universal logical switching, we use universal objects for security when we want to propagate them cross-vCenter. It is worth noting that not all NSX security features are available for cross-vCenter NSX. The things that are not supported are:

  • Exclude list
  • SpoofGuard
  • Flow monitoring for aggregate flows
  • Network service insertion
  • Edge Firewall
  • Service Composer

Universal security groups can contain the following:

  • Universal IP Sets
  • Universal MAC Sets
  • Universal Security Groups
  • Universal Security Tags
  • Dynamic criteria

As you see with cross-vCenter NSX security we’re not able to use vCenter constructs when defining security policies. Remember there’s a 1:1 relationship between NSX Manager and vCenter. One NSX Manager has no clue about the vCenter objects of a vCenter system paired with another NSX Manager.

Let’s create a universal firewall rule for our newly deployed virtual machines. Log in at the DC-SE vCenter system and navigate to Networking and Security > Security > Firewall > General. We start by creating a new universal firewall section. Click on “Add Section“. Call it “Universal Rules” and make sure you enable “Universal Synchronization“.

Once we have a universal firewall section we can start creating universal firewall rules in it. Click on “Add Rule” so a new empty rule shows up in the universal section.

Click on “Enter rule name” and write “Allow MariaDB”. Edit the source field and in the “Specify Source” click on “Create new IP Set“. Name the IP set “App-Tier” and add the 192.168.0.0/24 CIDR.

Select “App-Tier as the source for our “Allow MariaDB” rule:

Repeat this same procedure to specify the destination in the firewall rule. Call the IP set “DB-Tier” and specify the 192.168.1.0/24 CIDR.

Next we need to specify the service in this rule. Search for and add the “MySQL” service. Click “Save“.

The universal firewall rule should look like this now.

Click “Publish” to save and activate the changes.

Now let’s see if we can hit on the rule. We won’t actually install any MariaDB/MySQL server. Instead we’ll run a simple netcat listener on DBServer-SE and do a netcat connect from AppServer-US as a quick proof of concept.

First we need to allow traffic on TCP port 3306 through the iptables firewall on DBServer-SE:

iptables -A INPUT -p tcp --dport 3306 -j ACCEPT

Next start a netcat listener on TCP port 3306:

nc -l -p 3306

Over at the AppServer-US we can try to connect to port 3306:

nc -zv 192.168.0.20 3306

With no “block” rule in our distributed firewall anywhere this is the expected result (of course). But let’s have a look at the stats for our MySQL firewall rule. This can be looked up in a couple of places: vCenter GUI, central CLI, syslog or on the ESXi host. Today I’ll look this up on the ESXi host. Log in to the ESXi host where AppServer-US is running.

We first need the name of the filter at IOChain slot 2 of the AppServer-U vNic:

summarize-dvfilter | grep AppServer-U -A5

In my environment the filter’s name is “nic-2110503-eth0-vmware-sfw.2”

We use the “vsipioctl” command to see statistics for the rules in this filter:

vsipioctl getrules -f nic-2110503-eth0-vmware-sfw.2 -s

The ID for the MySQL firewall rule in my environment is “2147483648”. As we can see, the simple netcat connectivity test resulted in exactly 1 hit on the MySQL firewall rule we created earlier.

The ID of a firewall rule can be found at several places as well. The DFW GUI in vCenter is one of them.

Wrapping up

In this part we deployed the data plane components for facilitating East-West traffic in a cross-vCenter NSX environment: universal logical switching and universal distributed logical routing. We touched upon universal distributed firewalling as well.

In part three we will look at the components and configuration involved with the North-South traffic in a cross-vCenter NSX environment: The Edge tier.

Cross-vCenter NSX – Part 1

One of those scenarios where the NSX platform really shines is in multi-site environments. Here NSX together with vSphere is the infrastructure that delivers on business requirements like workload mobility, resource pooling, and consistent security.

Since NSX version 6.2 we can roll out NSX in a vSphere environment managed by more than one vCenter system. This type of deployment is called cross-vCenter NSX.
With cross-vCenter NSX we are able to centrally deploy and manage networking and security constructs regardless of the management domain architecture.

In preparation for some assignments involving cross-vCenter NSX, I’ve been busy with a cross-vCenter NSX lab. I thought I’d do a little writeup in two parts on setting this up.

In this first post we’ll prepare the management and control plane for cross-vCenter NSX. In part 2 we’ll have a closer look at how to deploy the data plane in a cross-vCenter NSX environment.

The lab environment

The following components are the building blocks I used for this simple cross-vCenter NSX lab:

  • 8 x nested ESXi 6.7 U1 hosts
  • 2 x vCenter 6.7 U1 systems
  • vSAN storage
  • NSX 6.4.4
  • 2 x pfSense routers

Just so we spend time focusing on the relevant stuff I’ve done some preparation in advance.

I set up two fictional sites: DC-SE and DC-US. Each with its own, non-linked, vCenter server system, four ESXi hosts, vSAN storage, and a standalone NSX Manager.
The ESXi hosts are prepared for NSX (VIBs installed) and a segment ID pool and transport zone are configured. DC-SE is running a controller cluster.

Each site has a pfSense machine acting as the perimeter router. Static routing is set up so each site’s management, vMotion and VTEP subnets can reach each other. Both sites also have a VLAN for vSAN plus one for ESG uplink which will be used in part two.

VLANDC-SEDC-US
Management10.0.10.0/2410.1.10.0/24
vMotion10.0.20.0/2410.1.20.0/24
vSAN10.0.30.0/2410.1.30.0/24
VXLAN transport10.0.50.0/2410.1.50.0/24
Uplink10.0.70.0/2410.1.70.0/24
High-level overview of the lab environment before cross-vCenter NSX is implemented

Please keep in mind that this is a simple lab environment design and in no way a design for a production environment. Have a look at VMware Validated Designs if you want to learn more about SDDC designs including NSX for production environments.

Step 1 – Assign primary role to NSX Manager

There’s a 1:1 relationship between vCenter server and NSX Manager. This is true when setting up cross-vCenter NSX as well, but here the NSX managers involved are assigned roles.
The NSX manager that should be running the controller cluster is assigned the primary role. Additional NSX Managers participating in cross-vCenter NSX (up to 7) are assigned the secondary role.

So let’s start by assigning the NSX Manager in DC-SE the primary role. In vCenter, go to Networking and Security > Installation and Upgrade > Management > NSX Managers. Select the NSX Manager and click on Actions > Assign Primary Role.

As you can see the role has changed from Standalone to Primary.

When we assign the primary role to a NSX Manager, its controller cluster automatically becomes the universal controller cluster. It is the one and only controller cluster in cross-vCenter NSX and provides control plane functionality (MAC, ARP, VTEP tables) for both the primary and secondary NSX Managers.

Step 2 – Configure Logical Network settings

While we’re at DC-SE we continue with the configuration of the logical network settings for cross-vCenter NSX.

We begin with defining a Universal Segment ID pool. These segment IDs (VNIs) are assigned to universal logical switches. Universal logical switches are logical switches that are synced to the secondary NSX Managers. We will look more at this in part two.

Go to Networking and Security > Installation and Upgrade > Logical Network Settings > VXLAN Settings > Segment IDs and click Edit.

Configure a unique range for the universal segment ID pool.

Create a universal transport zone and add CL01-SE

Next to VXLAN Settings we find Transport Zones. Click it and click Add to start adding an Universal Transport Zone.

Give it a name like “UTZ” and switch Universal Synchronization to On and add the CL01-SE vSphere cluster to the transport zone.

Step 3 – Assign secondary role to NSX Manager

Assigning the secondary role to the NSX Manager located in DC-US is done from the primary NSX Manager in DC-SE.

In vCenter, navigate to Networking and Security > Installation and Upgrade > Management > NSX Managers. Select the NSX Manager and click on Actions > Add Secondary Manager.

Here you enter the information of the NSX Manager at DC-US and click Add.

The NSX Manager at DC-US now has the secondary role. We can verify this by logging in to vCenter at DC-US and navigate to Networking and Security > Installation and Upgrade > Management > NSX Managers.

As you can see the NSX Manager now has the Secondary role.

Add CL01-US to the universal transport zone

While still logged in to vCenter at DC-US navigate to Networking and Security > Installation and Upgrade > Logical Network Settings > Transport Zones. The transport zone that we created over at the primary NSX Manager in DC-SE, shows up here too. Mark the transport zone and click on the “Connect Clusters” button. Add the CL01-US cluster to the transport zone.

Wrapping up

This completes the preparation of the management and control plane for our simple cross-vCenter NSX lab.

We started by assigning the primary role to the NSX Manager at DC-SE. By doing so we got ourselves a universal controller cluster. Next we configured the logical network settings necessary for cross-vCenter NSX. Finally we paired primary NSX Manager at DC-SE with the standalone NSX Manager at DC-US by assigning it the secondary role. Along the way we also added the vSphere clusters on both sites to the same universal transport zone.

In part 2 we will set up the data plane in our cross-vCenter NSX lab. We’ll have a look at how logical switching, distributed logical routing, and distributed security works in a cross-vCenter NSX environment.

3V0-643

3V0-643 is the exam code for the VMware Certified Advanced Professional 6 – Network Virtualization Deployment Exam. I recently passed this exam and thought I’d write a short blog post about it.

About the exam

The exam preparation guide for this exam contains the following description:
” The 3VO-643 exam tests candidates on their skills and abilities in implementation of a VMware NSX solution, including deployment, administration, optimization, and troubleshooting.”

As you might have guessed, this is a lab-based exam. Meaning you connect to a remote lab environment and work your way through a number of scenarios. You will be doing actual deployment, administration, optimization, and troubleshooting of a live VMware NSX platform.

Exam prerequisites

There is no course requirement for VCAP6-NV. That being said, to achieve the VCAP6-NV certification you must hold a valid VCP6-NV certification which does have a course requirement.

VMware recommends candidates to take the VMware NSX: Troubleshooting and Operations [V6.2] course as a preparation for this exam. I did the NSX version 6.4 of that course and although the course itself was very valuable, I can not say that it was really relevant as preparation for the 3V0-643 exam.

Another VMware recommendation is that the candidate should have two years of experience deploying NSX. Experience is a good thing of course. I would say this depends very much on how much and in which way you work with NSX in your day-to-day job role. My two cents:

  • You can skip the recommended course, but be sure you know your NSX including basic NSX troubleshooting.
  • Experience with deploying and managing all components of NSX is a must. Remember that you can gain experience in a lab environment as well.

Exam preparation

If you’ve ever been doing a VMware Hands-on Lab, the exam’s interface will be familiar to you. The controls, menus, and layout of the exam interface are very similar if not identical to the ones in the hands-on lab environment. I recommend doing a couple of NSX hands-on labs just to get used to the interface. HOLs VMware’s free, high quality labs that offer a convenient way for you to try out all kind of VMware tech and solutions. So check them out.

As always with VMware exams, the exam blueprint is your friend. It clearly states what areas of NSX (all) and scenarios you should prepare for. For 3V0-643 I would say the exam blueprint is spot on. On the exam I was faced with questions and scenarios linked to all of the objectives in the exam blueprint.

An important thing to keep in mind is that at the time of this writing the 3V0-643 lab environment is still based on vSphere 6.0 and NSX version 6.2. You should take this into account as you prepare for the exam.

  • Do your hands-on training in a NSX 6.2 environment. Build a nested lab with vSphere 6.0/NSX 6.2 if necessary. There are differences between the 6.2 and 6.4 UIs. If you’re not familiar with the 6.2 way of doing things you end up spending a lot of precious time navigating around in the (slow) vCenter UI trying to find things.
  • Study the NSX version 6.2 official documentation. It’s not meaningful to study anything else than version 6.2 documentation for this exam.
  • Study and practice all the objectives that are listed in the exam blueprint. Make sure you are comfortable accomplishing all of the objectives before you consider taking this exam.

What to expect?

23 questions/scenarios to be completed within 205 minutes. All of this in an outdated and slow vSphere 6.0/NSX 6.2 remote lab environment. It wasn’t the most exciting experience to be honest. The scenarios and problems you need to solve on the other hand were realistic and quite fun.  It’s a mixture of  tasks and objectives. One scenario you’ll complete with a couple of mouse clicks or commands, while the next will take you more than 30 minutes to complete.

  • Time is an issue. In my experience 205 minutes is not a lot of time. You easily end up with time pressure. You should be able to accomplish the objectives relatively fast. There is not much time for clicking around, finding your way, or thinking for that matter. Knowing beforehand how to accomplish objectives is key.
  • Time is not an issue. Don’t get stuck. You can skip questions and come back to them later if there’s time left. Correctly answered questions add to your score. Focus on that instead of getting stuck trying to answer a question and running out of time. This is especially true when you’re early on in the exam. Also, know that you can pass this exam even when you run out of time.

Conclusion

In my opinion passing 3V0-643 is an accomplishment. This lab-based NSX exam will thoroughly test your ability to successfully deploy, manage, optimize, and troubleshoot all aspects of a NSX platform. All that under time pressure.

A certification alone doesn’t make you a subject matter expert, but it’s probably fair to say that you know a thing or two about NSX when you pass 3V0-643.

Last but not least it would be great if VMware would update the exam’s lab environment to for instance vSphere 6.5/NSX 6.4, but I doubt that will happen. My guess is that we’ll see NSX-T exams instead, but time will tell.

Site-to-site IPsec VPN with NSX

With NSX Data Center for vSphere establishing site-to-site IPsec VPN connections is done on the Edge instances. One X-Large ESG instance supports up to 6000 IPsec tunnels which gives you an idea about scalability.

In this exercise I’ll show you how to configure a site-to-site IPsec VPN connection between two sites that are both using NSX Edge instances. This native NSX scenario is probably the easiest and most straightforward, but it’ll hopefully give you an idea of what the IPsec VPN configuration process looks like.

Diagram

As always let’s start with a picture of what we’re trying to achieve:

NSX IPSEC.png

The goal is simple. These two organisations want to establish a secure communications channel over the Internet so that specific logical networks at each site can communicate with each other. More specifically, in this scenario the two VMs in the diagram should be able to communicate with each other.

For this exercise I already deployed the NSX Edges and the DLRs. I also configured routing between ESG and DLR on each site. We will just focus on configuring the site-to-site IPsec VPN tunnel in this article.

Configuring the Edges

Under the “Manage” tab of the ESG click on “VPN” and choose “IPsec VPN”. To start adding a new IPsec configuration click the green plus sign.

Skärmavbild 2018-10-30 kl. 20.35.02

Let’s briefly touch upon the form here.

Enabled – Wether this IPsec VPN is active or not.
Enable perfect forward secrecy (PFS) – Enabled by default. Ensures each new cryptographic key is unrelated to any previous key.
Responder Only – When enabled the IPsec VPN will never initiate a connection.

Name – A name for the IPsec VPN.
Local Id – This is the IP address assigned to the ESG’s uplink interface.
Local Endpoint –  When using pre-shared key fo authentication this can the same as “Local Id”.
Local Subnets – The IP subnets on this side that will be part of the IPsec VPN
Peer Id – A unique ID for the peer site. When using pre-shared key for authentication this can be anything although it is recommended to use the public IP address/FQDN of the remote endpoint. When using certificates for authentication this is the common name (CN) in the peer’s certificate.
Peer Endpoint – This is the IP address of the remote endpoint.
Peer Subnets – The subnets on the remote site that should be part of the IPsec VPN.

IKE Version – Internet Key Exchange version. Used to set up a security association (SA).
Digest Algorithm – The authentication algorithm to use.
Encryption Algorithm – The encryption algorithm to use.
Authentication – This can be either pre-shared key (PSK) or certificate.
Diffie-Hellman Group – The key exchange algorithm.
Extension – Aadditional parameters we can add to the IPsec VPN.

Using the IP information in the diagram above, the IPsec VPN configuration for organisation A’s ESG looks like this:

Skärmavbild 2018-11-03 kl. 12.53.45.png

And for organisation B’s ESG it looks like this:

Skärmavbild 2018-11-03 kl. 13.31.04.png

Please note that in my environment the uplink interfaces of both ESGs connect to the same L2 segment, configured with IP subnet 10.2.128.0/24.

Establishing the IPsec tunnel

With the IPsec configurations in place we can now move on with the tunnel establishment.

Start the IPsec VPN service on both edges:

Skärmavbild 2018-11-03 kl. 13.34.13.png

Let’s verify that the IPsec tunnel got established. Mark the IPsec VPN and click on “Show IPsec Statistics”:

Skärmavbild 2018-11-03 kl. 19.54.29.png

Channel status looks good. We can click on “View Details” to get some more state information. As you see not much is going on at this point. We have established an IPsec tunnel, but it’s not being used.

We can also view the IPsec VPN status at the edge instance CLI by issuing the “show service ipsec” command:

Skärmavbild 2018-11-03 kl. 20.14.30.png

Testing communication

At this point organisation A’s virtual machines connected to “LS Web” should be able to reach organisation B’s virtual machines connected to “LS Web” and vice versa. Let’s try it out.

Open a SSH or console session to one of the VMs and ping the virtual machine at the other site:

Skärmavbild 2018-11-03 kl. 23.16.52.pngAs you can see I’m pinging 172.16.1.20 from 172.16.0.20 and receiving replies.

Looking once more at the tunnel state, we can now see packets have been sent and received. It works!

Skärmavbild 2018-11-03 kl. 23.19.08.png

Conclusion

Nothing really fancy here. Good old IPsec VPN doing its thing but on top of NSX components. Site-to-site IPsec VPN is an important component of any enterprise network platform and it’s good to see NSX Data Center for vSphere delivers the functionality as well as making it easy to configure.

Equal Cost Multi-Pathing in NSX Data Center

By deploying multiple ESG appliances and enabling Equal Cost Multi-Path (ECMP) on the Distributed Logical Router (DLR), NSX Data Center provides scalability and redundancy for northbound traffic in our SDDC.

Setting up ECMP in NSX Data Center is fairly easy. Let me show you.

The diagram

A high level overview of the NSX infrastructure for this excercise.

NSX ECMP.png

The goal in this exercise is to provide two equal cost routes on he DLR for northbound traffic. For this we are going to deploy two ESG appliances, configure dynamic routing between these and the DLR, and then finally enable ECMP.

Let’s get started!

Deploying the ESG appliances

Create the first ESG appliance.

Skärmavbild 2018-10-05 kl. 20.25.32.png

Add two interfaces: One connected to a VLAN-backed port group leading to the physical network, the other one connected to the logical transit switch leading to the DLR.

Skärmavbild 2018-10-05 kl. 20.33.12.png

Don’t forget to configure a default gateway pointing to the next hop on the physical network.

Once deployed we can go ahead and deploy the second ESG appliance in the same way.

Skärmavbild 2018-10-05 kl. 20.40.54.png

Create the same interfaces as on “Edge-01”. Use unique IP addresses though. 😉

Skärmavbild 2018-10-05 kl. 20.45.17.png

Configure dynamic routing on the DLR

There won’t be much multi-pathing going on without a working routing process between the DLR and the ESGs so let’s make sure that is in place first.

Starting with the DLR. We first need to add an uplink interface that connects the DLR to the transit switch:

Skärmavbild 2018-10-05 kl. 21.03.55

We’ll be doing dynamic routing using BGP in this exercise so navigate to the DLR’s “Routing” tab and set the router ID:

Skärmavbild 2018-10-05 kl. 21.11.15

Next navigate to “BGP”, set a private ASN (64 512 – 65 534), and enable BGP:

Skärmavbild 2018-10-05 kl. 21.19.14.png

While we’re here we also add our two future BGP neighbors. Click the green “plus” sign under “Neighbors” to add the first BGP neighbor:

Skärmavbild 2018-10-06 kl. 10.47.57.png

Now what’s all of this?
First we need to specify the IP address of our BGP neighbor (Edge-01 in the above screen). Next we have a forwarding address (data plane) which is the IP-address configured on the DLR’s uplink interface (2ESG in my case). The protocol address is used for the BGP peering/neighbor relationship which can be any unused IP address in the transit network (192.168.100.0/24 in my case). Finally we supply the remote ASN which is the private ASN of the BGP neighbor (65501 in my case).

Click “OK” and then add the second BGP neighbor (Edge-02). We use the same protocol address as for the first neighbor (one protocol address per uplink). Also type the same remote ASN as for the first BGP neighbor. Once done it should look something like this:

Skärmavbild 2018-10-05 kl. 21.49.06.png

The last thing we need to configure on the DLR is “Route Redistribution”.

Skärmavbild 2018-10-05 kl. 21.55.32.png

Enable route redistribution for BGP. Also add BGP to the route redistribution table and configure it to learn connected routes.

Configure dynamic routing on the ESGs

With BGP configured on the DLR we move over to to our ESGs.

Just like on the DLR we need a router ID on our ESGs. Starting with “Edge-01” go to the “Routing” tab and configure the the router ID:

Skärmavbild 2018-10-06 kl. 10.55.51.png

Next we configure the BGP which is almost identical to the way we configure BGP on the DLR.

Skärmavbild 2018-10-06 kl. 10.59.54

Enable BGP and type a private ASN for the ESG AS (65501 in my case). We might want to enable “Default Originate” so that the ESG will advertise a default gateway to the DLR. Otherwise we’ll have to configure a default gateway on the DLR manually.

We continue with adding the DLR as a BGP neighbor:

Skärmavbild 2018-10-06 kl. 16.54.30.png

Here things look slightly different compared to the DLR. No protocol address and no forwarding address are needed. Why not?
With the DLR the control plane and the data plane are separate from each other. What this means is that the forwarding address lives in the data plane, which is a distributed plane (ESXi hosts). The protocol address on the other hand is tied to the DLR control VM. The ESG is a central component where data plane and control plane both reside in the ESG appliance.

Next configure BGP route redistribution the same way as we did on the ESGs.

Skärmavbild 2018-10-06 kl. 16.56.40.png

Before we move on with the second ESG, let us just check that we have a working BGP neighbor relationship between the DLR and Edge-01 at this point.

Open the console of the DLR control VM and log in. Issue the “show ip bgp neigbors summary” command. At this point we should see one established BGP neighbor relationship:

Skärmavbild 2018-10-07 kl. 09.40.31.png

Next type “show ip route bgp”. Here we should see one (or two if default originate was enabled) BGP derived routes in the routing table:

Skärmavbild 2018-10-07 kl. 09.44.17

If everything looks as expected we can go ahead and configure BGP on Edge-02. The configuration steps are identical to the ones we did on Edge-01.

With Edge-02 configured and peered with the DLR, the BGP neighbor table on the DLR should now show two BGP neighbor relationships:

Skärmavbild 2018-10-07 kl. 09.55.10.png

ECMP

What we’ve built so far gives us better availability. When Edge-01 goes down, the BGP process on the DLR will after some time, depending on the configured “keep alive” and “hold down” timers, use the routes advertised by Edge-02 instead and traffic starts flowing again.

But with this setup we only have one active path from the DLR to the edge and beyond. To activate multi-pathing we simply need to enable ECMP on the DLR.

Head over to the DLR management and click the “Routing” tab > “Global Configuration” and start ECMP.

Skärmavbild 2018-10-06 kl. 19.43.24.png

At the DLR console issue the “show ip route bgp” command once again. We’ll notice a difference:

Skärmavbild 2018-10-07 kl. 10.10.25.png

We now have two BGP routes for every destination. Each of the Edge’s route advertisements end up as equal cost routes in the DLR’s routing table. The DLR will now use both these paths simultaneously. It does this by using a (non-configurable) hashing algorithm based on source and destination IP address.

Conclusion

Multiple ESG appliances combined with ECMP provides scalability and availability for northbound traffic. If your SDDC grows the size where a dedicated Edge cluster becomes best practice design, ECMP should be considered. Up to 8 equal paths (8 ESG appliances) can be provisioned to make sure the necessary bandwidth and redundancy are available to this critical part of the NSX infrastructure.

NSX: Bridging between VXLAN and VLAN

After you prepared your vSphere clusters for VXLAN you’re eager to start building your SDDC network. You provision some logical switches, a distributed logical router and maybe even an edge services gateway. Before you know it you are doing full-fledged network virtualization. It’s simple enough, right?

But then you realize you still have other virtual workloads and possibly all kind of physical equipment residing on that other network: The VLAN-based network.

Of course this is (hopefully) not the way you rollout VXLAN in your SDDC. Just like any other major change in your network (logical or physical) some kind of planning is required here. You should at least create a solid design for the VXLAN structure, the IP subnets, IP routing, and how it all connects and propagates to the physical network before you start implementing VXLAN. Things might get really complicated if you don’t.

But even with all the planning in the world you still might end up with workloads and equipment that for various reasons are stuck on VLANs. On top of that, some of these workloads require to be on the same L2 segment as the virtual workloads that you planned on migrating to VXLANs. This can be a short term (transitioning etc) or a long term requirement.

A helping hand

NSX BridgeOne component of NSX-V that comes in handy in a situation like this is the L2 bridge. The L2 bridge has a number of use cases including:

  • Migration: Physical to virtual, or virtual to virtual  without requiring re-IP.
  • Connectivity: Physical workloads not suitable for virtualization can maintain connectivity with virtual workloads inside of NSX.

  • Service insertion: Transparent integration of any physical appliance such as a router, load balancer or firewall into NSX.

There are some prerequisites and limitations:

  • L2 bridging requires a distributed logical router with a control VM
  • The VXLAN network and VLAN-backed port groups must be on the same distributed virtual switch and use the same physical NICs.
  • VLAN-backed port group must be configured with a VLAN ID (between 1 and 4094).
  • Don’t use a L2 bridge to connect a logical switch to another logical switch, a VLAN network to another VLAN network, or to interconnect data centers.
  • You can’t use a Universal logical router to configure bridging and you cannot add a bridge to a universal logical switch (cross-vCenter NSX objects).
  • A logical router can have multiple bridging instances, however, the routing and bridging instances cannot share the same VXLAN/VLAN network. Traffic to and from the bridged VLAN and bridged VXLAN cannot be routed to the bridged network and vice versa.
  • The recommended maximum is 500 bridging instances per distributed logical router. A number you’ll hopefully never need.

Configuring a L2 Bridge

The L2 bridge is configured with a couple of clicks over at the distributed logical router. Yes, a couple of API calls does the trick too.

At the DLR click the Manage tab and then Bridging. Click the green plus sign to add a bridge:

Skärmavbild 2018-09-08 kl. 19.48.50.pngType a name for the bridge and select a logical switch (VXLAN) and a distributed virtual port group (VLAN-backed). Click OK and don’t forget to publish your changes.

That’s all there is to it. You’re now bridging between a VXLAN and a VLAN.

Conclusion

VXLAN-VLAN bridging is not necessarily something you want to do over a long period of time as it adds some complexity to your environment. That being said, there are scenarios (mentioned above) where the L2 bridge is the right solution and it’s good to know that setting this up in NSX-V is a breeze.

VMware NSX 6.4.2 Released

nsxVMware has released NSX Data Center for vSphere 6.4.2 and it comes with some nice improvements.

Multicast routing is the big one in this release of course. It’s good to see that once again more NSX components can be managed/used with the vSphere Client (H5). Interesting as well is the addition of two new administrative roles: Network Engineer and Security Engineer.

Here’s a list of all that’s new:

Networking and Edge Services

  • Multicast Support: Adds ability to configure L3 IPv4 multicast on Distributed Logical Router and Edge Service Gateway through support of IGMPv2 and PIM Sparse Mode.
  • Default Limit of MAC identifiers: Increases from 2048 to 4096
  • Hardware VTEP: Added multi PTEP cluster capability to facilitate environments with multiple vCenters

Security Services

  • Context-Aware Firewall: Additional Layer 7 Application Context Support (EPIC, MSSQL, BLAST AppIDs)
  • Firewall Rule Hit Count: Monitor rule usage and easily identify unused rules for clean-up
  • Firewall Section Locking: Enables multiple security administrators to work concurrently on the firewall
  • NSX Application Rule Manager: Improved scale to 100 vNICs per session, further simplifying the process of creating security groups and whitelisting firewall rules for existing applications.

NSX User Interface

Operations and Troubleshooting

  • Authentication & Authorization: Introduces 2 new roles (Network Engineer and Security Engineer). Adds ability to enable/disable basic authentication.
  • NSX Scale Dashboard: Provides visibility into 25 new metrics. Adds ability to edit usage warning thresholds and filter for objects exceeding limits.
  • NSX Controller Cluster Settings: Specify common settings (DNS, NTP, Syslog) to apply to NSX Controller Cluster.
  • Support for VM Hardware version 11 for NSX components: For new installs of NSX 6.4.2, NSX appliances (Manager, Controller, Edge, Guest Introspection) are installed with VM HW version 11. For upgrades to NSX 6.4.2, please see Upgrade Notes for further details.

You can find the full release notes over here.

Updating my lab to 6.4.2 using the Upgrade Coordinator was a breeze. 🙂