Deploying NSX-T in a Stretched Cluster – Part 2

1. Hi,
   You have deployed completely on the Site A and Site B , but what about the Witness site . For the witness site the reach-ability be from traditional network or we can implement it on the NSX-T.

   Just curious to know.

   Thank you

   LikeLike

   Reply

   1. Hi Manoj,

      NSX-T was not deployed at the witness site as it was a site dedicated for the witness functionality.

      Thank you for reading my blog.

      Rutger

      LikeLike

      Reply

2. Hi
   Thank you for the really nice blog! As I understand you route everything over the DCI except the SDDC Management VLAN, this one is really streched between both sites (L2), right?

   Where do you attach the two Host-Overlay vlans, vmotion vlans and esx vlans for routing between the to datacenter sites in real world?

   Thanks for help

   LikeLike

   Reply

   1. Thanks for reading my blog.

      It will depend on your network topology but with a leaf-spine topology these VLANs would be terminated in the leaves (ToRs) and traffic routed to a spine. From that spine traffic would then be routed to the leaf where the destination host is connected.
      Hope this makes sense.

      LikeLike

      Reply

      1. Thanks for clarification but then for me its unclear why you split the esxi per site into different vlans?
         You really want to route the HA traffic between the datacenters? Isnt it the better way to just move all esxi which are on the same vcenter per site into the same vlan?

         LikeLike

      2. In this case the scenario is a stretched cluster. There will be some traffic (L3 and L2) crossing the data center interconnect as a result of that design. The primary reason for using different VLANs per site is to create smaller layer 2 fault domains.
         On a side note this is a vSAN stretched cluster and HA traffic goes over the vSAN network.

         LikeLike

3. Hi, thanks for your brilliant guide on stretched clusters. One thing that is still puzzling me is say for example CustomerA had a presence at Site A with some VM’s and they also have some VM’s at Site B, is it possible for these customer VM’s to be on the same subnet? I’m just thinking from a failure scenario of the Site A hardware or a loss of the hardware at Site A for whatever reason – of course they would need the relevant vSAN policy applied to be able to tolerate this level of failure.

   Any ideas would be much appreciated.

   Thanks

   LikeLike

   Reply

   1. Thanks Richard,
      The VMs can be on the same logical network regardless of on which site they’re running. This is accomplished using Geneve overlay in NSX-T.

      LikeLike

      Reply

4. Hi,

   The North-South trace shows Site-B Tier-0 in the path. Is there a reason for that? Also the VMware documentation says with active-active setup Tier-0 plums to all the Edge nodes in a cluster with ECMP. Can that be overcome with inbound as-path prepend that you have applied?

   Thank you.

   LikeLike

   Reply

   1. Hi,

      The traceflow was done from a VM on site B. I realize that wasn’t clear at all.
      By AS-path prepending at site B, inbound traffic will enter through site A (under normal circumstances).

      Cheers

      LikeLike

      Reply

5. Hi,

   briliant work, thx…

   Do you have plans to update this lab solution with newest version of NSX 3.1

   I have plans to deploy NSX-T 3.1 to the prod system with existing vSAN 7 with stretched cluster.
   So, it has VDS 6.6 and where is no any management cluster, so just one consolidated compute cluster with management components like vCenter, NSX management cluster and Edge must be implimented with a 10 ESXi hosts.

   Any suggestions welcome…

   LikeLike

   Reply

   1. Thanks. That should be fine. Any reason the VDS is still on 6.6? How many pNICs do your host have?

      LikeLike

      Reply

   2. Is there any supported way to configure local egress in stretched clusters. For Example : traffic from a VMin site be to a physical server in site be should be in and out through tor in site b

      LikeLike

      Reply

      1. With NSX-T Multisite it’s only possible when you have distinct logical networks in each site. If the logical network is stretched, local egress can not ne enforced. With NSX-T Federation you can configure local egress by making the T0 “Primary” in each location.

         LikeLike

6. Thanks for the blog post, very helpful. Because the T0 is configured in ECMP will the traffic not traverse all links within the edge cluster regardless of the site?

   LikeLike

   Reply

   1. Hi Ben. With this particular set up that’s indeed the case for traffic from NSX to the physical network. Incoming traffic will be routed to the site with the shortest BGP AS path. If you would also like to pin the outgoing traffic to a certain site you can do so by using route priorities.

      LikeLike

      Reply

7. Hi Rutger, thanks for a very good guide on this. We are deploying a VCF solution and are having trouble understanding where to route the vSAN and vMotion VLANs. Is your suggestion to do this L3-routing in the ToR instead of doing it through NSX overlay?

   LikeLike

   Reply

   1. Thanks Andreas.
      This will be dictated by VCF and depends on the version of VCF you are deploying and some of the design decisions. In the latest versions you will have everything routed except for the management network which is still stretched. So with VCF, regardless of what you and I think is the best way of doing it, it’s very important to stick to that intended design for supportability reasons.

      Cheers

      LikeLike

      Reply

8. Hi Rutger,

   Thank you for the Explanation. We have been trying to get the same type of configuration running but have hit into a few issues however unsure if they are related to the newer NSX-T version 3.1.1 we are running or if we are missing something.

   Outbound
   Advertising the routes out bound with a prepend from Site B seems to be working as expected meaning inbound to Tier 1 is going via Site A as expected. under normal circumstances and fails over to B in event of DR.

   Inbound
   We are advertising a default route from the upstream platform and can see that the route map is adding the prepends via the uplinks on Site B increasing the Path but this dose not seem to permit the route over the ISR to be installed. As long as a peer is up in Site B the default route is being learnt over it:
   > * 0.0.0.0/0 [20/0] via A.B.C.D, uplink-277

   if we actively remove the Site B peering we then see the Default routes coming over the isr:
   isr> * 0.0.0.0/0 [200/0] via 169.254.0.130, inter-sr-275, 00:00:05
   isr> * 0.0.0.0/0 [200/0] via 169.254.0.131, inter-sr-275, 00:00:05

   from the numbers we are seeing it looks like the external peering has the AD 20 vs isr 200, have you seen this in your configuration / worked around it ?

   Cheers,
   James

   LikeLike

   Reply

9. Hi Rutger,

   Thank you for the Explanation. We have been trying to get the same type of configuration running but have hit into a few issues however unsure if they are related to the newer NSX-T version 3.1.1 we are running or if we are missing something.

   Outbound
   Advertising the routes out bound with a prepend from Site B seems to be working as expected meaning inbound to Tier 1 is going via Site A as expected. Under normal circumstances and fails over to B in event of DR.

   Inbound
   We are advertising a default route from the upstream platform and can see that the route map is adding prepends via the uplinks on Site B increasing the Path but this does not seem to permit the route over the ISR to be installed. As long as a peer is up in Site B the default route is being learnt over it:
   > * 0.0.0.0/0 [20/0] via A.B.C.D, uplink-277

   If we actively remove, the Site B peering we then see the Default routes coming over the isr:
   isr> * 0.0.0.0/0 [200/0] via 169.254.0.130, inter-sr-275, 00:00:05
   isr> * 0.0.0.0/0 [200/0] via 169.254.0.131, inter-sr-275, 00:00:05

   From the numbers we are seeing it looks like the external peering has the AD 20 vs isr 200, have you seen this in your configuration / worked around it?

   Cheers,
   James

   LikeLike

   Reply

   1. Hi James,

      I’m currently investigating a similar issue in an NSX-T 3.1.1 environment (in this article NSX-T 2.5 was used). Some things have changed and might have affected the way this is configured. I will let you know.

      LikeLike

      Reply

      1. Hi Rutger,
         Did you manage to find a fix for 3.1 ?

         LikeLike

      2. It’s simple you have to know the BGP protocol and how attributes handled in BGP (EBGP and IBGP).
         EBGP peering between different AS number
         IBGP peering between same AS number

         Outbound: Thats correct with as-prepend on site B peering with external AS.

         Inbound: use local-pref instead of as-prepend on inbound route prefixes on site B
         Mark the routes with a local-pref lower than 100 for example 90. Default is 100.
         That will steer how you in your AS will exit your own AS to other AS in this case over site A if everything is up.
         Highest local-pref is prefered and is local to your as and will be used in IBGP to set best route.
         All your edges peering bgp (ibgp) with each other in the same as.

         LikeLike

10. Really great article and exercise, I haven’t really managed VMware for a while, it was pretty good to catch up with the most recent on the networking aspects. Quick (dumb) architectural question: is it common for customers running Workload VMs and Edge Nodes VMs in the same ESXi hosts? Do all ESXi have at least one L0 and L1 Gw VM? How’s that distribution work in a real-life/enterprise-scale environment? I’m supposing they would have a cluster for Edge Nodes separate from regular Workload VMs, and then in this case the ESXis reserved for Workloads still run L0s and L1s? TIA

    LikeLike

    Reply

11. Great article on High Availability for multi-site environment. Trying to understand how the flow goes between the 4 Edge nodes. There are 2 in Site-A and 2 in site-B. A question about the network flow: If there is a VM at site-A that initiates North-South traffic, which Edge VM will be primarily used? Will it be any Edge VM from Site-A or it is random and we can end up from Site-B? Same when a VM initiates North-South traffic from Site-B, which site’s Edge VMs will be used? Is there anyway that we can prioritize Edge VM based on the VM’s site? It would be great if VMs at site-A always user Edge VM at that side, same for Site-B. Otherwise it means we need to route traffic from an ESXi at site-A to and Edge VM at an ESXi at Site-B and from there we go to a ToR (in Site-B) and internet. Not optimal, which is a trade-off

    LikeLike

    Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.