Prepare for the worst and oh, no need to hope for the best… (part 1/3)

If you’ve been working long enough in IT, you know that disasters do happen. It may not be a tornado or thunder blowing up your datacenter, but something will happen sooner or later which will require you to execute on that DR plan which has been gathering dust on your desk.

Whether Disaster Recovery (DR) plans are executed planned or unplanned, there are always things that go wrong, no matter how prepared you think you are. There are simply too many moving parts and variables: will the network failover properly? Will I have enough compute capacity at my recovery site to start everything? How will I recover my data? Will I have knowledgeable staff trained on the failover procedures? How will I failback once the primary site has been recovered?

One thing is for sure: things have changed. Whereas reliance on tape backups is still very much a thing, technology has evolved and infrastructure has gotten seemingly simpler.

Virtualization has helped overcome hardware compatibility issues. Networking has evolved and made IP reconfiguration a thing of the past. Storage replication has become synchronous enabling zero recovery point objective (RPO). Software defined infrastructure has enabled end to end automation enabling near zero recovery time objective (RTO). Metro clusters are invading datacenters enabling active-active configurations.

Still, all those new shiny technologies require skills and deep integration. Compatibility matrixes are still fairly complex and evolving faster than you can document it.

Then there is hyper-convergence. Merging storage and compute into a single software-defined solution most definitely makes things easier because you don’t have to figure out those things separately from one another. If in addition you have a nice intuitive interface to manage it which itself is always highly available, then you’ve hit the jackpot. You see where I’m going with this…

Nutanix Xtreme Computing Platform (XCP) has had built-in asynchronous and synchronous replication for a while now and this series will explore those options in more details, specifically on the VMware vSphere virtualization platform.

In part 1, we will cover the basic features of data replication in Nutanix Acropolis Base Software (formerly known as Nutanix Operating System or NOS). We will explain what Async DR is and how to set it up. We will also explain what Metro Availability is and how to set it up (with a bonus Powershell script to assist with some of the more tedious vSphere DRS configuration tasks) and finally, we’ll cover Sync DR as well.

In part 2, we will look at how to do planned failovers and failbacks for Async DR, Sync DR and Metro Availability.

Finally, in part 3, we will look at how to execute unplanned DR with Async DR, Sync DR and Metro Availability, including how to failback once your failed site has been recovered.

Comparing protection domain types

All right, enough with the chitchat and on with the replication features available in Nutanix.

Nutanix with vSphere as a virtualization platform currently has several options available for data replication: Async DR, Sync DR and Metro Availability. Think of it as asynchronous vs synchronous with some nuances.

First, some Nutanix specific terminology: data protection is implemented using what we call “Protection Domains”. A protection domain (PD) can be of either type Async DR or Metro Availability (Sync DR is configured in the same place as Metro Availability).

When a PD is of type Async DR, you can protect individual VMs and you can replicate every hour as the smallest replication interval.

With Async DR PDs, you normally failover those VMs to a different vSphere cluster which itself lives on a different Nutanix cluster (see fig. 1 below).

Async DR essentially means that you will snapshot your VMs locally, then replicate those snapshots to that other Nutanix cluster. You define the replication interval and the retention policy. You can also specify multiple replication targets, which are known as “Remote Sites”. That remote cluster may or may not have the same VLANs and subnets as your original site, so Async DR can be complemented with VMware Site Recovery Manager to automate the failover and failback and VM network reconfiguration.

 

Fig. 3 - Sync DR Overview

With a metro availability PD, you replicate an entire container (what is seen as an NFS datastore on your ESXi hosts) to another Nutanix cluster, but nodes (aka servers or hosts) which make up both of those replicating Nutanix clusters are part of the same compute cluster (one single HA/DRS cluster; see fig. 2 above). That means the VLANs and subnets are stretched across both locations and VMware HA is used to automatically restart VMs on the remaining site in case of a disaster. The requirement for Metro Availability (MA) is to have a maximum of 5 ms latency on average between all the Nutanix nodes on the network.

It is important to understand that as shown in Fig.2 above, there are then two copies of the container, one in each Nutanix cluster. One copy is “Active” meaning it services read and write I/Os while the other is in “Standby”, meaning it is read-only and does not service ANY I/O request (yes, even read I/O requests will go to the Active container).

That means in normal conditions, you don’t really want virtual machines which have their storage in the active container in site A to be running on hosts in site B. While that is technically possible, it will potentially be bad for your network link between both sites. That’s why Nutanix recommends using DRS affinity rules to keep virtual machines local to the hosts holding the active copy of the container (more on that later).

Note that the Prism user interface (UI) does not distinguish between Metro Availability and Sync DR, so don’t look for the Sync DR option, it’s not there. Essentially, if you are running two separate vSphere Clusters, or if you’re running on Hyper-V and you add a Metro Availability Protection Domain, it is by nature a Sync DR, even though it is called Metro Availability in the UI. I know, this is a bit confusing, but hopefully looking at Fig. 3 above will clear this up for you.

The table below highlights the main differences between Async DR, Sync DR and MA protection domains.

Async DR

Metro Availability

Sync DR

Quantity of Nutanix Clusters

2

2

2

Quantity of vSphere Clusters

2

1

2

Stretched VLANs

No1

Yes

No1

Protected Entity

Virtual Machine

Container

Container

Targets

One to one, one to many, many to one

One to one

One to one

Type of Replication

Asynchronous

Synchronous

Synchronous

RPO

>= 1 hour

0

0

RTO

Minutes to hours2

Minutes

Minutes to hours2

Network Requirement

N/A

5 ms latency

5 ms latency

Supported Hypervisors3

vSphere, Hyper-V, Acropolis

vSphere

vSphere, Hyper-V

1: While stretched VLANs are not required, they will shorten your RTO by making network reconfiguration unnecessary. If you do not have stretched VLANs, you can use VMware Site Recovery Manager with Async DR as Nutanix has a Storage Replication Agent (SRA) available, but you cannot use SRM with Sync DR.

2: This is mostly depending on whether network reconfiguration is required.

3: As of Acropolis Software version 4.5.0.2 (October 2015)

With all that being said, does this mean that you absolutely need two vSphere clusters to do Async DR between two Nutanix clusters? Absolutely not. You could very well have a metro cluster in which you choose to have two type of VMs: those which are protected by MA with an RPO of 0 and those which are protected by Async DR with an RPO of 1 hour or more. I’m just giving you what are typical scenarios.

The only strict rule is that it can’t be really called a Metro Availability setup if you have more than one compute cluster, because then it is Sync DR, not MA. And remember that MA is currently only supported on VMware vSphere.

Configuring Async DR

Configuring Async DR is very straightforward:

  1. Decide which containers are going to be used at the source and remote site for replicating the snapshots
  1. Create the remote site at each end and define a container/vstore mapping based on what you determined in step 1
  2. On the source cluster, add a Protection Domain of type Async DR, select the virtual machine you want to protect and if you want to regroup them in consistency groups (more on that later)
  3. Define a schedule for your Async DR PD, a target remote site, and a retention policy for both local and remote targets
  4. That’s it!

So, what are consistency groups? Consistency Groups (CG) are a logical way for you to define a group of VMs which will have snapshots taken at the exact same moment (as opposed to sequentially). This may be important if you have a multi-tier application and you want to make sure the snapshot data is consistent.

When you run on vSphere, you also have the option on enabling VSS consistent snapshots which will use Microsoft Volume Shadow Services thru the VMware tools to notify the OS that a storage snapshot is about to occur and that pending I/Os should be committed to disk so that the data is not just crash consistent.

So, to include screenshots (which visual people like me enjoy since seeing is believing), I will set this up in my lab.

My setup is as follows:

  • I have two Nutanix clusters: c02 and c06. Each Nutanix cluster has three nodes (c02nodea, c02nodeb and c02nodec for c02, and c06nodea, c06nodeb and c06nodec for c06).
  • All hosts are running vSphere 5.5 and Nutanix Acropolis Software 4.5.0.2
  • The VMs use VLAN92 which is stretched (so no need for IP reconfiguration when failing over VMs).
  • C02 hosts make up the c02 compute cluster. It uses the container/datastore c02-ct1. This cluster hosts VMs tinyvm01 to tinyvm10.
  • C06 hosts make up the c06 compute cluster. It uses the container/datastore c06-ct1. This cluster hosts VMs tinyvm11 to tinyvm20.
  • We will protect the VMs running in c06 (tinyvm11 to tinyvm20) using an Async DR protection domain which will replicate every hour.

Figure 4 gives you an overview of that setup:

Step 1: Configure the remote sites at each end

  1. On c02, in Prism, go to the “Protection Domain” view and select “+ Remote Site > Physical Cluster”:

  1. Name the remote site (in this example, we’ll call it c06) and give it the Nutanix cluster virtual IP address (if you haven’t configured one yet, you can do that by simply clicking the Nutanix cluster name in the upper left corner in Prism):

  1. Click on “Add Site”. Note that you can choose to modify options for the remote site such as bandwidth throttling if you want to, but you don’t have to.
  2. Now on c06, create the c02 remote site in the same fashion, except this time after clicking “Add Site”, we will configure a vstore (aka container) mapping from c06-ct1 to c02-ct1 (so that the system knows in which container it will replicate snapshots for VMs that are in the c06-ct1 container):

Make sure you create the opposite vstore mapping on the c06 remote site definition for c02 (this will be important when you’ll want to fail back).

Step 2: Create the Async DR protection domain with an hourly schedule

Now that we’ve got the remote sites sorted out, let’s create the protection domain.

  1. On c06, in the “Protection Domain” view, click on “+ Protection Domain > Async DR”:

  1. We’ll call our protection domain “c06-aspd” (for c06 async protection domain):

  1. Select the VMs you want to protect and click “Protect selected VMs”. Note that this is on a per-vm basis, not a per-container basis. You can also choose to regroup specific VMs in consistency group and enable application consistent snapshots if you are running vSphere and you have VMware Tools installed. When you’re done, click “Next”:

  1. Click on “New Schedule”:

  1. Select the schedule you want (here we selected every 1 hour). Then select a local retention policy (we left the default of 1) and select the remote site for replication as well as a remote site retention (we changed it to 2 in this example). When done, click on “Create Schedule”:

  1. Review your schedule and click on “Close”:

  1. When you select your protection domain, you’ll see that you can view which VMs are included, which snapshots are local or remote, all in the bottom part of the screen:

  1. On c02, you’ll see the Async DR PD as well, except the little circle on the left will be grey instead of green to show it is not active:

That’s it, you’ve configured your Async DR! Give yourself a pat on the back.

Configuring Metro Availability

Configuring Metro Availability is easy. At a high level, the workflow is as follows (assuming you already have your single vSphere cluster configured and ready):

  1. You create a new container on the target site which has the exact same name as the container at the source
  2. You define the target remote site on the source cluster in the Data Protection screen, then you define the source remote site on the target cluster also in the Data Protection screen
  3. On the source cluster, you add a Protection Domain of type Metro Availability, define the source container, select the remote site and select the VM Availability policy (more on that in a second)
  4. That’s it! Well, not quite… Remember that you still need to define DRS affinity rules to make sure that VMs run on hosts which have the active copy of the container in order to avoid cross site I/Os… That means you need to figure out which hosts are in each Nutanix cluster, which VMs are in which datastores, which datastore is active on which Nutanix cluster, then make sure VMs are on hosts which have the active container that maps to their datastore. Sounds complex but it really isn’t and I have even included a Powershell script to automate that for you. It’s available below when I go thru a specific example of configuring MA in my lab.

So what the hell is a VM Availability policy you ask? Good question.

A Metro Availability Protection Domain can have one of two roles and several different statuses. The roles are “Active” or “Standby”. As you have probably guessed, under normal circumstances, the role is Active on the source cluster and Standby on the target cluster.

The statuses are “Pending”, “Enabled in Sync”, “Remote Unreachable”, “Disabled” and “Decoupled”. “Pending” means a replication is occurring and success or failure has not been determined yet. “Enabled in Sync” means replication is on-going without any issue. This is the normal status under normal conditions. “Remote Unreachable” means that the remote site cannot be contacted. When this happens on a Standby PD, it just means data isn’t replicated anymore. On the Active PD however, I/Os will be suspended until the status is changed to “Disabled” or connectivity to the remote site is restored. “Disabled” means that the Protection Domain is no longer replicating, either because the remote site is unavailable, or because an administrator manually disabled the protection domain. “Decoupled” means that the protection domain is active both at the source and target and therefore out of sync. This is what happens after an unplanned site failure and can be easily remedied, but more on that in part 3 of this blog series.

Back to VM Availability policy: when a Metro Availability Protection Domain is in the “Remote Unreachable” status, all VM write I/Os are suspended in order to maintain consistency and avoid the “Decoupled” status. To resume VM write I/Os (which is what you would want if a disaster had occurred at the site to which you were replicating), the protection domain has to be in the “Disabled” status (meaning replication is stopped). That can be either a manual action (an admin logs in and disables the protection domain), or automatic (the Nutanix cluster will automatically set the status to disabled after a given number of seconds). Which option you choose is what defines the VM Availability policy. Note that this is NOT for standby protection domains, but active protection domains which have lost their replication partners. Think of it this way: replication is down, so any new data written will not be protected anymore, therefore someone needs to decide what to do: either this is legitimate because there was a site failure and therefore we can resume I/Os by changing the status to disabled, or we are in a split brain situation, in which case it may be better to power off VMs. Changing your VM Availability policy lets you choose whether you want that decision to be automatic or manual.

The table below summarizes the statuses for a Metro Availability Protection Domain:

Status

What does it mean?

What should I do?

Pending

A replication is occurring and status has not yet been determined

Nothing other than wait to see what the next status will be

(“Enabled in Sync” or “Remote Unreachable”)

Enabled in Sync

Replication is working

Nothing, everything is peachy

Remote Unreachable

A replication failed

If this is your active PD and your VM availability policy is manual, then you will need to decide if you prefer to power off your VMs or change the PD status to “Disabled” (which is what you want if you know for sure the remote site has crashed)

Disabled

Replication has been disabled

Determine which site should be Active and re-enable replication from that site on the PD after you’ve determined that everything is back to normal

Decoupled

PD is active on both sites

Shut down your VMs, determine which site has the most recent copy (should be the secondary site since the decoupled status is what happens after the primary site has recovered from failure), then change the status to “Disabled” on the other site, and re-enable on the site with the valid copy of the data.

So if you’re like me, after reading this you’re thinking “Muh, sounds easy enough, but I’d like to see what it really looks like”. Fear not, for I have included below monkey-proof screenshots.

My setup is as follows:

  • I have two Nutanix clusters: c02 and c06. Each Nutanix cluster has three nodes (c02nodea, c02nodeb and c02nodec for c02, and c06nodea, c06nodeb and c06nodec for c06).
  • All hosts are running vSphere 5.5 and pooled into a single compute cluster which I called “metro-cluster”
  • Nutanix cluster c02 will have an active protection domain of type MA called C02toC06 and which will apply to container c02-ma. This container will host VMs tinyvm01 to tinyvm10.
  • Nutanix cluster c06 will have an active protection domain of type MA called C06toC02 and which will apply to container c06-ma. This container will host VMs tinyvm11 to tinyvm20.
  • DRS affinity rules will be created to make sure tinyvm01 to tinyvm10 should run on hosts that make up c02 and that tinyvm11 to tinyvm20 should run on hosts that make up c06.

Figure 5 gives you an overview of that setup:

Now on with the actual configuration steps with screenshots:

Step 1: Create matching containers on each Nutanix cluster

First step is to make sure that the c02-ma container exists also on the c06 Nutanix cluster and that the c06-ma container exists also on the c02 Nutanix cluster. Remember, for MA to work, the replicated container must have a container on the target name with the same identical name.

  1. From c06’s Prism interface, we navigate to the Storage > Table > Containers view. Then click on the “+ Container” menu in the upper right corner:

  1. Create the c02-ma container, and make sure you do not mount it on any of the hosts (otherwise your compute cluster may create data files on it in which case you won’t be able to select it as a replication target)

  1. Once the container has been created, you should see both containers listed on c06:

  1. Create c06-ma on c02 now by following the same procedure. Once done, this is what you will see on c02:

Step 2: Creating the remote sites on each Nutanix cluster

Now that we have set up our containers, we need to configure remote sites so that each Nutanix cluster becomes aware of the other. This is done in the “Protection Domain” view in Prism on each cluster. Note that we will use the Nutanix cluster virtual IP address (configured at installation time, or which can be configured in Prism by clicking the cluster name in the upper left corner).

  1. On c06 Prism interface, add a “Remote Site” of type “Physical Cluster”:

  1. Give that site a meaningful name and put in the Nutanix cluster virtual IP address for c02:

  1. You can choose to configure specific settings (such as bandwidth throttling) on the next screen, but you don’t have to.
  2. Once added, you should see the remote site defined on c06:

  1. Repeat the procedure on c02 in order to define the c06 remote site:

Step 3: Creating the Metro Availability Protection Domains

So now we have our containers set up as well as our remote sites. We are ready to create our Metro Availability Protection Domains. We will create one on c02 which will replicate the c02-ma container to c06, and one on c06 which will replicate the c06-ma container to c02.

  1. On c02, add a protection domain of type “Metro Availability”:

  1. We will call that protection domain “C02toC06”:

  1. Select c02-ma as the container you want to replicate:

  1. Select c06 as the target site. Note that if network latency is too high, or if you have data in the target container, or if there is no existing target container defined, then the remote site will not be selectable as a target. Hover your mouse over whatever is causing the issue and resolve it in order to be able to select the remote site as a target:

  1. Configure a VM Availability policy (manual or automatic). In this example, we will select automatic after 20 seconds. Remember, this is what will happen if the replication partner is lost. By choosing automatic, we are telling the Nutanix cluster to stop replication by changing the status of the protection domain to disabled so that VM write i/o can resume on the container.

  1. Review the configuration before clicking “Create”:

  1. Note that the system will warn you that you are about to overwrite anything that is on the c02-ma container at the target site (c06). This is because replication will start immediately:

  1. After initial replication is performed, you will see that the protection domain is in the “Enabled in sync” status, which is what you want:

Note that if we look at what it looks like from c06, we’ll see pretty much the same view, except the protection domain will be marked as “Standby” instead of “Active”:

  1. We are now ready to mount the c02-ma container on c06. From c06 Prism interface, go to the “Storage > Table > Container” view and update the c02-ma container:

  1. Select the “Mount on all ESXi hosts” option and click “Save”:

  1. Repeat the procedure on c06 to create a metro availability protection domain for the c06-ma container which will replicate to c02 as the target site. When done, you should see the following from c06 protection domain, metro availability view:

Note that C06toC02 is active on c06 while C02toC06 is standby. Both as “Enabled in sync”. If we look on c02, we’ll see that the active and standby protection domains are reversed:

Step 4: Creating the vSphere DRS affinity rules with a Powershell script

So now our protection domains have been created, what’s next? Well, if you remember what was said earlier in this post, we still need to create DRS affinity rules on the compute cluster so that VMs which have their storage on the c02 Nutanix cluster don’t run on hosts which are in the c06 Nutanix cluster (which only has a standby copy of the container/datastore) and vice versa.

That means we need to understand:

  1. Which compute hosts belong to which Nutanix cluster
  2. Which datastores are replicated as part of a Metro Availability protection domain
  3. Which VMs are in those datastores

Then we will need to:

  1. Create DRS host and VM groups
  2. Create “should” VM to host affinity rules

That can be a somewhat tedious task, but thankfully, I have written a script for you that will automate all that.

Before we get into the script, let’s review what our environment looks like from the vSphere client.

  1. We have a single compute cluster regrouping all the Nutanix nodes and with HA and DRS enabled:

  1. Any host in that compute cluster can see both replicated datastores:

  1. Some VMs are running in the c02-ma datastore while others are running in the c06-ma datastore:


Given this configuration, we want tinyvm01 to tinyvm10 to run on c02 hosts and tinyvm11 to tinyvm20 to run on c06 hosts. This will make sure that there is no cross-site i/o traffic and that the cross-site network bandwidth will be used exclusively for replication.

For that purpose, we can use the add-DRSAffinityRulesForMA.ps1 script which you can download below. The script will prompt you for what it needs (the Nutanix clusters IP addresses and credentials, as well as the vCenter server IP address) and will create the DRS host and vm groups and vm to host rules:

Note that to correlate Nutanix hosts to vSphere compute hosts, it uses the management IP address (vmk0) as a key. If you don’t like the DRS groups or rules names, you can always change them later.

Once done, the script has created the following groups:

In addition, the following rules have been created:

Here is the script for you to download: add-DRSAffinityRulesForMA.ps1

Note that it requires PowerCLI and the Nutanix cmdlets to be installed. For instructions on how to set that up, refer to this blog post.

I have also included below the source code (JULY 2016 EDIT: I updated the script to be able to deal with updating the rules. Note that the object naming convention changed to make the objects more readable as well):

Configuring Sync DR

A few caveats for Sync DR:

  1. You won’t need DRS rules since your compute cluster isn’t stretched.
  2. When you create the Sync DR PD, make sure the container is empty and not mounted on the target site as vSphere may write stuff on it. Once the PD is online and ready, mount the countainer on all the hosts at the target site.
  3. If you do not complete the PD creation for any reason, you may end up with an empty Async DR PD definition in Prism. Delete it and start over after fixing whatever prevented you from creating it in the first place.
  4. Because the replicated container is mounted and visible on the target site, there is always the possibility that some admins will provision VMs to that container which will be registered on the target site. This should be avoided as it complicates matters when failing over and back.  I will not be covering some of those advanced scenarios due to lack of time and resources, so I encourage you to review them in the Prism guide.

In part 2, we will see how to do planned failover for each type of protection domains on VMware vSphere.

Stephane Bourdeaud

I am a Senior Consultant for Nutanix working out of France. I have over 18 years of experience in the IT industry including 15 as an infrastructure architect at IBM. I am a virtualization specialist holding VCP3, VCP4, VCP5, VCAP5-DCA and VCAP5-DCD certifications.

  • Brad Burgess

    I cant find part 2 and 3? have they not been posted yet?

  • Chris

    Hello Stephane,

    This is a great series – thanks alot for you this very valuable information!!! I have a somewhat more complex setup…could you have a look at it?

    I have 2 datacenters 1 and 2 with two zones A and B each. In all 4 zones are Nutanix deployments. In both datacenters zones A should be replicated synchronously to zones B. In addition datacenter 1 should be relplicated asychronously to datacenter 2.

    My assumptions are:
    – There should be 4 Nutanix clusters; one in each zone
    – On top of the Nutanix deployments there should be 2 VMware HA clusters
    – I configure PDs as follows: 1A(primary) to 2A(remote), 1B(primary) to 2B(remote) and also vice versa, 2A(primary) to 1A(remote) and 2B(primary) to 1B(remote) (or would it make more sense to replicate crosswise?)

    Would that be a reasonable setup?

    Thanks alot for your kind answer,
    Best Regards: Chris

    • Hello Chris and thank you for your feedback.
      First of all, let me apologize for not responding earlier. Nutanix currently has its Sales Kick-Off event in Vegas and I didn’t notice you had posted a comment.

      Now on to your actual question:
      So if I understand correctly, you have 4 Nutanix clusters. 2 are in datacenter A and 2 are in datacenter B. I’m not sure what you mean by “zones”, but I’ll assume they are just different rooms within the same datacenter.
      Your requirement is synchronous replication from zone A to zone B in each datacenter. I don’t completely understand the asynchronous replication requirement, but I’ll assume it is for zone B in DC1 to somewhere in DC2.

      So you would have:
      DC1 Zone A —-sync repl—> DC2 Zone B
      DC2 Zone A —-sync repl—> DC1 Zone B
      DC1 Zone B —-async repl–> DC2 Zone A or B

      Did I understand your requirements correctly?
      I guess the point is that yes, you can have multiple Metro Availability Protection Domains in a single cluster. Each MA PD can only replicate to a single another Nutanix cluster.
      You can mix MA and Async DR at your leisure.
      Async DR can have multiple replication targets.

      Ultimately, I think the things to keep in mind are:
      1. Starting with AOS 4.6, it is not recommended to have more than 50 VMs in a container which is protected with an MA PD. This is because hourly snapshots are done in order to reduce the amount of data to replicate when an MA PD is paused and resumed (such as when doing failover and failback)
      2. The same limitation applies to Async DR PD (no more than 50 VMs as a best practice)

      Let me know if that answered your questions, and if not, please clarify your requirements for me.

      Regards,

      Stephane

    • Stephane Bourdeaud

      Hello Chris and thank you for your feedback.
      First of all, let me apologize for not responding earlier. Nutanix currently has its Sales Kick-Off event in Vegas and I didn’t notice you had posted a comment.

      Now on to your actual question:
      So if I understand correctly, you have 4 Nutanix clusters. 2 are in datacenter A and 2 are in datacenter B. I’m not sure what you mean by “zones”, but I’ll assume they are just different rooms within the same datacenter.
      Your requirement is synchronous replication from zone A to zone B in each datacenter. I don’t completely understand the asynchronous replication requirement, but I’ll assume it is for zone B in DC1 to somewhere in DC2.

      So you would have:
      DC1 Zone A —-sync repl—> DC2 Zone B
      DC2 Zone A —-sync repl—> DC1 Zone B
      DC1 Zone B —-async repl–> DC2 Zone A or B

      Did I understand your requirements correctly?
      I guess the point is that yes, you can have multiple Metro Availability Protection Domains in a single cluster. Each MA PD can only replicate to a single another Nutanix cluster.
      You can mix MA and Async DR at your leisure.
      Async DR can have multiple replication targets.

      Ultimately, I think the things to keep in mind are:
      1. Starting with AOS 4.6, it is not recommended to have more than 50 VMs in a container which is protected with an MA PD. This is because hourly snapshots are done in order to reduce the amount of data to replicate when an MA PD is paused and resumed (such as when doing failover and failback)
      2. The same limitation applies to Async DR PD (no more than 50 VMs as a best practice)

      Let me know if that answered your questions, and if not, please clarify your requirements for me.

      Regards,

      Stephane

      • Chris

        Hi Stephane,

        thanks for your answer. I know…almost all Nutanix folk are in Vegas currently – have fun! To clearify:

        DC1 Zone A DC1 Zone B
        DC2 Zone A DC2 Zone B

        …additionally the customer imagines an async replication like that:

        DC1 Zone A,B DC2 Zone A,B

        Regarding the deployment size we probably talk about 24 nodes in each zone. So your approach would be to put all the 96 nodes in one large Nutanix cluster and create MA / DR PDs according to the customers requirements while keeping in mind not to put more than 50 VMs into one of them but rather create another PD?

        For a disaster szenario: Would you think that there is any need to integrate VMware SRM into the solution architecture?

        Best Regards,
        Chris

        • Stephane Bourdeaud

          Chris,
          So if you have already synchronous replication going in each DC, what is the value of async DR (which would be yet another copy of the data?): doing cross DC recovery?

          Also, assuming you are running at Replication Factor 2, if you have 24 nodes per zone, that means Nutanix clusters of 48 nodes when using MA in a single DC, which is quite a large fault domain. We usually recommend using Replication Factor 3 (which allows for the loss of two components simultaneously) for clusters with more than 16 nodes.

          So no, the recommendation would not be to pull all nodes in a single cluster. As I understand it, you would use MA to do recovery within a single DC, and async DR across DCs.
          So you would have 1 Nutanix cluster in DC1, and 1 Nutanix cluster in DC2. Given you would have 48 nodes in each cluster, ideally you would use replication factor 3.
          You can then setup async DR between DCs. For both MA replicated containers and Async DR protection domains, the best practice is to keep them at less than or equal 50 VMs each.

          VMware SRM would provide additional value if you need to:
          1. Change IP addresses and VLANs from one DC to another
          2. If you use distributed virtual switches
          3. If you need to have a specific boot order of your VMs during a failover

          • Chris

            Hi Stephane,

            great answer, thank you! The async replication between the two regionally separated DCs (250km) is wanted to help the customers company survive large scale disasters. Only the top-tear VMs and their data will be replicated. The standard use case is to access the cross-replicated top-tier-VMs only locally from each branches DC. If these are more than 50, We’ll use serveral PDs. During disaster recovery the surviving DC is planned to serve all request. BTW: For RF-3 each site would have to consist of even more nodes 🙂

            Best Regards,
            Chris