Deploying NSX-T in a Stretched Cluster – Part 1

A stretched cluster architecture facilitates for higher levels of availability and things like inter-site load balancing. It’s a common multisite solution and also part of VMware’s Validated Design for SDDCs with multiple availability zones.

Traditionally compute networking in an active-active multisite setup has had its challenges, but with vSAN storage and NSX networking technologies that’s a thing of the past.

In the coming two articles I want to have a closer look at NSX-T in an active-active multisite environment. Specifically I want to learn more about how the different NSX-T components are deployed and how the data plane is configured in a stretched cluster.

In this first part I will deploy the NSX-T 2.5 platform and perform the necessary configurations and preparations so that in part two I can focus solely on the data plane (north-south and east-west).

This is going to be quite an exercise so let’s get right to it!

The lab environment

Below a high level overview of the lab environment as it looks right now:

A vSAN cluster consisting of eight ESXi hosts stretched to a second site. A third site is hosting the vSAN witness appliance. A completely separate vSphere management cluster is only hosting the vCenter server right now.

A quick look at the vSphere environment then. I’m running vSphere 6.7 U3:

The hosts have two physical 10Gbit NICs:

Three vmkernel adapters have been configured: Management, vMotion, and vSAN:

As mentioned, this is a vSAN stretched cluster:

The following tables list the VLANs and the associated IP subnets that are currently configured per site:

Site A:

VLAN Function	VLAN ID	Subnet	Gateway
ESXi Management	1641	172.16.41.0/24	172.16.41.253
vMotion	1642	172.16.42.0/24	172.16.42.253
vSAN	1643	172.16.43.0/24	172.16.43.253

Site B:

VLAN Function	VLAN ID	Subnet	Gateway
ESXi Management	1651	172.16.51.0/24	172.16.51.253
vMotion	1652	172.16.52.0/24	172.16.52.253
vSAN	1653	172.16.53.0/24	172.16.53.253

Witness Site:

VLAN Function	VLAN ID	Subnet	Gateway
ESXi Management	1711	172.17.11.0/24	172.17.11.253
vSAN	1713	172.17.13.0/24	172.17.13.253

Management Cluster:

VLAN Function	VLAN ID	Subnet	Gateway
SDDC Management	1611	172.16.11.0/24	172.16.11.253

NSX-T is not deployed yet, but that’s about to change pretty soon 😉

Deploying the NSX-T manager cluster

Installing NSX-T 2.5 always starts with deploying the manager cluster. It consists of three manager nodes and an optional virtual IP (VIP).

I will deploy the NSX manager cluster nodes in the vSphere management cluster and connect them to the SDDC Management VLAN (1611).

The IP plan for the NSX manager cluster looks like this:

Hostname	IP Address
nsxmanager01	172.16.11.82
nsxmanager02	172.16.11.83
nsxmanager03	172.16.11.84
nsxmanager	172.16.11.81 (virtual IP)

First manager node

I deploy the first manager node from the OVA package:

Filling out the configuration details and then kicking off the deployment.

When the first manager node is up and running I’m logging in to the NSX Manager UI:

Second and third manager nodes

The second and third manager nodes can be deployed from the NSX Manager UI. Before I can do that I need to add my vCenter server under System > Fabric > Compute Manager:

Now I’m able to deploy the second and third manager nodes via System > Appliances > Add Nodes.

Once done the three nodes are shown in the UI and the cluster connectivity is up:

Assign virtual IP address

I finalize the manager cluster deployment by configuring a virtual IP address. This is done under System > Appliances > Virtual IP:

A couple of minutes later the virtual IP is active:

Configuring the NSX-T data plane

Now that the NSX-T management plane is fully operational I will continue with the data plane preparations and configurations.

More VLANs

First I need to provision some more VLANs in the TORs at the data sites. At each site I need two VLANs for overlay and another two for connecting NSX with the physical network later on:

Site A:

VLAN Function	VLAN ID	Subnet	Gateway
Host overlay	1644	172.16.44.0/24	172.16.44.253
Uplink01	1647	172.16.47.0/24	172.16.47.253
Uplink02	1648	172.16.48.0/24	172.16.48.253
Edge overlay	1649	172.16.49.0/24	172.16.49.253

Site B:

VLAN Function	VLAN ID	Subnet	Gateway
Host overlay	1654	172.16.54.0/24	172.16.54.253
Uplink01	1657	172.16.57.0/24	172.16.57.253
Uplink02	1658	172.16.58.0/24	172.16.58.253
Edge overlay	1659	172.16.59.0/24	172.16.59.253

Transport zones

Two transport zones should do it I believe. I create them using the following details:

Name	N-VDS Name	Traffic Type
tz-vlan	nvds01	VLAN
tz-overlay	nvds01	Overlay

Transport zones are managed under System > Fabric > Transport Zones:

Uplink profiles

Next, I need to create four uplink profiles. The table below shows the configuration details for each of them:

Name	Teaming Policy	Active Uplinks	Transport VLAN	MTU
up-site-a-esxi	Load Balance Source	uplink-1, uplink-2	1644	9000
up-site-a-edge	Load Balance Source	uplink-1, uplink-2	1649	9000
up-site-b-esxi	Load Balance Source	uplink-1, uplink-2	1654	9000
up-site-b-edge	Load Balance Source	uplink-1, uplink-2	1659	9000

Uplink profiles are managed under System > Fabric > Profiles > Uplink Profiles:

In order to achieve VLAN pinning, deterministic routing, and ECMP I need to add two named teaming policies to the uplink profiles that I just created:

Name	Teaming Policy	Active Uplinks
Uplink01	Failover Order	uplink-1
Uplink02	Failover Order	uplink-2

Adding the named teaming policies to the uplink profiles:

I also need to add the Uplink01 and Uplink02 named teaming policies to transport zone tz-vlan. This so that they can be selected on segments belonging to that transport zone later on:

Network I/O Control profile

To allocate bandwidth to different types of network traffic I create a network I/O control profile. After long and hard thinking I decided to call it nioc-profile and it has the following settings:

Traffic Type / Traffic Name	Shares
Fault Tolerance (FT) Traffic	25
vSphere Replication (VR) Traffic	25
iSCSI Traffic	25
Management Traffic	50
NFS Traffic	25
vSphere Data Protection Backup Traffic	25
Virtual Machine Traffic	100
vMotion Traffic	25
vSAN Traffic	100

Network I/O control profiles are managed under System > Fabric > Profiles > NIOC Profiles:

Segments

VLAN-backed segments are needed for system, uplink/transit, and overlay traffic. The table below lists the segments with their settings that I will create:

Segment Name	Uplink & Type	Transport Zone	VLAN
site-a-nvds01-management	none	tz-vlan	1641
site-a-nvds01-vmotion	none	tz-vlan	1642
site-a-nvds01-vsan	none	tz-vlan	1643
site-a-edge-transit01	none	tz-vlan	1647
site-a-edge-transit02	none	tz-vlan	1648
site-b-nvds01-management	none	tz-vlan	1651
site-b-nvds01-vmotion	none	tz-vlan	1652
site-b-nvds01-vsan	none	tz-vlan	1653
site-b-edge-transit01	none	tz-vlan	1657
site-b-edge-transit02	none	tz-vlan	1658
edge-uplink1	none	tz-vlan	0-4094
edge-uplink2	none	tz-vlan	0-4094

Segments are managed under Networking > Connectivity > Segments:

Uplink teaming policy

The uplink teaming policy for segments edge-uplink1 and edge-uplink2 need to be modified so that the named teaming policies Uplink01 and Uplink02 are used instead of the default.

For this I have to edit these segments under Advanced Networking & Security > Networking > Switching:

Configure ESXi hosts

Now the time has come to configure the ESXi hosts and turn them into NSX-T transport nodes!

In the NSX Manager UI I navigate to System > Fabric > Nodes and change the “Managed by” to my vCenter server. The ESXi hosts are listed:

Unfortunately, I can’t make use of a transport node profiles here as these are assigned at the vSphere cluster level. I will therefore configure my hosts one at a time.

The ESXi transport nodes in Site A will be configured with the following settings:

Setting	Values
Transport Zone	tz-vlan, tz-overlay
N-VDS Name	nvds01
NIOC Profile	nioc-profile
Uplink Profile	up-site-a-esxi
LLDP Profile	LLDP [Send Packet Disabled]
IP Assignment	Use DHCP
Physical NICS	vmnic0 – uplink-1 vmnic1 – uplink-2
vmk0	site-a-nvds01-management
vmk1	site-a-nvds01-vmotion
vmk2	site-a-nvds01-vsan

ESXi transport nodes in Site B use slightly different settings:

Setting	Values
Transport Zone	tz-vlan, tz-overlay
N-VDS Name	nvds01
NIOC Profile	nioc-profile
Uplink Profile	up-site-b-esxi
LLDP Profile	LLDP [Send Packet Disabled]
IP Assignment	Use DHCP
Physical NICS	vmnic0 – uplink-1 vmnic1 – uplink-2
vmk0	site-b-nvds01-management
vmk1	site-b-nvds01-vmotion
vmk2	site-b-nvds01-vsan

Selecting one host at a time clicking Configure NSX:

The network mappings for install for vmkernel adapter migration:

When I click Finish the NSX installation and configuration process starts on the selected ESXi host. NSX bits are installed, the host receives the N-VDS, and the vmkernel adapters are migrated from VDS port groups to the N-VDS segments.

When all hosts have been configured I quickly check the status of the transport nodes:

And in vCenter I notice there’s now an N-VDS with a bunch of opaque port groups:

Summary

Most of the NSX-T platform is in place now and I think this is a good point to take a small break.

I started by deploying and configuring the NSX manager cluster (aka the central management plane). Next, I prepared the environment for the NSX data plane by provisioning some VLANs, profiles, and segments. Lastly, I prepared the ESXi hosts in the stretched cluster by installing the NSX VIBs and configuring them as NSX transport nodes. vSphere system networking (vmkernel adapters) was migrated to the N-VDS.

In the next part I will continue with the installation of the data plane and more specifically deployment and configuration of the NSX Edge as well as the logical networking components.

Stay tuned!