Replication Related

Important! The information provided in this section is for general Troubleshooting guidance only. The information is provided on "AS IS" basis, with no guarantee of completeness, accuracy or timeliness, and without warranty or representations of any kind, expressed or implied. In no event will CloudEndure and/or its subsidiaries and/or their employees or service providers be liable to you or anyone else for any decision made or action taken in reliance on the information provided above or for any direct, indirect, consequential, special or similar damages (including any kind of loss), even if advised of the possibility of such damages. CloudEndure is not responsible for the update, validation or support of troubleshooting information.

What do Lag and Backlog mean during replication?

During replication you may see a server falls out of Continuous Data Protection (CDPA method of replicating data in real time from one location to another. CDP works by capturing every change to any block on a storage volume in real time, and sending it immediately to a remote location. Traditional enterprise-grade disaster recovery solutions, such as the ones provided by IBM, EMC, NetApp and Microsoft all use CDP for storage or hypervisor replication.) mode. This may occur for various reasons, typically related to the network throughput or interruption.

How do I test Replication Speed?

Note: The following instructions demonstrate how to run a speed test through the iperf3 utility for both Linux and Windows. However, we strongly recommend utilizing the SSL Connectivity and Bandwidth test for Windows.

The replication speed depends on 4 key factors:

To test your uplink speed, you can you utility such as the iperf3 utility as follows:

  1. Install a vanilla Linux machineA physical or virtual computer. (m4.xlarge on AWS) in the same subnet of the CloudEndure Replication ServersThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location..
  2. On that machineA physical or virtual computer., install iperf3 utility using: sudo apt-get install iperf3
  3. Note: If you're using Red Hat/CentOS use yum install instead. Note that epel-release package may be required.

  4. Then run: iperf3 -s -p 1500
  5. On your source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine., install iperf3 as well:
    1. Windows: Download the right zip file from here and extract it.

      Note: We recommend using the SSL Connectivity and bandwidth test for Windows instead of iperf.


    2. Linux: Install as mentioned above.
  6. Then, on terminal window run:

    iperf3.exe -i 10 -c Vanilla_linux_server_ip -p 1500 -t 60 > iperf60.log
    iperf3.exe -i 10 -c Vanilla_linux_server_ip -p 1500 -t 1800 > iperf1800.log

    or

    iperf3 -i 10 -c Vanilla_linux_server_ip -p 1500 -t 60 > iperf60.log
    iperf3 -i 10 -c Vanilla_linux_server_ip -p 1500 -t 1800 > iperf1800.log

Here is a sample output:

In this output, you can see that the uplink is 23.4Mbps which means that a 100GB (idle) server should be replicated in about 10 hours. You can use this calculator, for example.

Note, if your server writes to disk an average of 20GB/day you need to take it into account when calculating as follows: 20GB/day --> ~2Mbps --> which leave us with only 21.4Mbps available for the initial 100GB.

How can I perform an SSL Connectivity and Bandwidth Test?

Note: This tool is for AWS only.

You can use our SSL bandwidth tool to check for replication bandwidth availability.

1. In your TargetThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery). region, launch an m4.large test server using the public AMI named CE-ssl-speedtest.

2. Select the same Subnet as the Subnet used in the Replication SettingsIn the Console UI, a set of settings affecting the data replication process, including Replication Servers subnet and IP, data throttling, use dedicated replicator, use multiple connections, etc. of your Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine..

3. Make sure that the Security GroupThe Security Group is an AWS feature that acts as a virtual firewall, which controls the inbound and outbout traffic of the Staging area. allows TCP Port 1500 inbound access.

4. On the Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine., browse to:

https://<test_server_ip>:1500/speedtest

5. Click on the Start button.

Note: Browse to the web page using the test server public or private IP according to what you set in your Replication Settings.

Note: The following are the AMI details per region.
ami-0eddaeafca15cb884 - Oregon
ami-0033acde7a667158d - N. Virginia
ami-0dd7efdc1e4d5153e - N. California
ami-0bf4515d4631c75b5 - Ohio
ami-021cf011e0fc597da - Sao Paolo
ami-098aed360e88e0bcf - Paris
ami-0f3c8e54a8783917f - London
ami-00e94f0aea23fd7b5 - Ireland
ami-057465f1ca6b64cc0 - Frankfurt
ami-036418f2316bf1c3d - Canada central
ami-0e4f14260e0f99d2f - Sydney
ami-037594652ca46896c - Singapore
ami-04a3c77d2f2f108ea - Mumbai
ami-08ec28115b395d148 - Seoul
ami-027f828fe74e61e42 - Tokyo
ami-0d8234c52b18b1281 - Osaka-Local
ami-06bf7ff42ec99x97e6 - Stockholm

Note: Ensure that the Security Group are configured to permit connectivity on inbound port 1500.

What are the Initial Data Replication Steps?

The replication steps are the following (the steps involved in the automatic creation of a Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. in the Staging AreaA part of the Target location; includes the Replication Servers’ subnet, IPs, and the Replication Servers and their disks., over TCP Port 1500)

  1. Firewall rules created - Creating firewall rules in the TargetThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery). infrastructure.
  2. Replication Server created – the TargetThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery). cloud allocates a virtual machineA physical or virtual computer. for the creation of a Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location..
  3. Replication Server booted – the allocated virtual machineA physical or virtual computer., which will function as a Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location., is launched.
  4. Resolved CloudEndure Service Manager address in the Replication Server – until this stage, the CloudEndure Service ManagerThe CloudEndure server. communicated with the TargetThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery). cloud, and not directly with the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location.. Now, the launched Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. asks for the public IP address of the CloudEndure Service ManagerThe CloudEndure server. from the DNS server of the cloud, in order to communicate with it directly.
  5. Authenticated the Replication Server with the CloudEndure Service Manager – direct communication between the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. and the CloudEndure Service ManagerThe CloudEndure server. is established.
  6. Downloaded the CloudEndure replication software to the Replication Server – the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. contacts an external storage unit that stores CloudEndure replication softwareThe software that is downloaded from the Console by the Replication Servers. It runs on the Replication Servers and it is responsible for writing the replicated data to the Staging area disks., and downloads it. This replication softwareThe software that is downloaded from the Console by the Replication Servers. It runs on the Replication Servers and it is responsible for writing the replicated data to the Staging area disks. will write the incoming replicated data to the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. disks.
  7. Created staging disks - the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. creates replication disks to store the incoming replicated data. The number of replication disks that are created depends on the size of the replicated data.
  8. Attached the staging disks to the Replication Server – in the previous step, the replication disks were created independently, without being attached to a specific Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location.. Now, they are attached to a Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location..
  9. Pair the CloudEndure Agent with the Replication Server –until now, the CloudEndure Service ManagerThe CloudEndure server. managed the communication between the AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. on the SourceThe location of the Source machine; Currently either a specific Region or Other Infrastructure. infrastructure and the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. on the Target infrastructure. Now, the CloudEndure Service ManagerThe CloudEndure server. knows that all the initiation steps have been completed successfully. Therefore, it provides the AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. and the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. information about each other, so that they could start communicating with each other directly.
  10. Establish communication between the CloudEndure Agent and the Replication Server – the AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. and the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. begin communicating with each other directly.

What are the regular Data Replication Steps?

Once communication is established between the AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. and the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location., Data ReplicationThe process of copying all data blocks from selected disks on a Source Machine to Staging Disks. begins. The following are the Data ReplicationThe process of copying all data blocks from selected disks on a Source Machine to Staging Disks. steps:

  1. Replication Initiation/Initiating Data Replication
  2. Initial Sync – in this stage, the progress bar will appear. The block replication of the content of the Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine. will begin, and the content will be copied to the Staging AreaA part of the Target location; includes the Replication Servers’ subnet, IPs, and the Replication Servers and their disks.. (Recovery Points are created so it will be possible to continue the replication from the last point in case of a disconnect.)


  3. Finalizing Initial Synchronization - once the Initial Sync reaches 100%, meaning, the data on the SourceThe location of the Source machine; Currently either a specific Region or Other Infrastructure. and the data on the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. totally match and are in a complete synchronizationThe process of producing an exact copy of the Source contents on the Target., another state appears - Finalizing Initial Sync. When reaching 100% for the first time, the machineA physical or virtual computer. will still not be launchable (the Finalizing Initial Sync step consists of two sub-stages):

How do I start stop or pause Data Replication?

You can start, stop, and pause Data ReplicationThe process of copying all data blocks from selected disks on a Source Machine to Staging Disks. from the right-hand MACHINE ACTIONS menu within the Machines tab.

Is the replicated data encrypted?

CloudEndure encrypts all the data in transit.

How is the Replication Server provisioned and managed in the Staging Area?

CloudEndure provisions the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location.(s) and automatically manages the addition and removal of the servers as necessary.

What type of storage is used in the Staging Area?

CloudEndure utilizes low cost storage in the Staging AreaA part of the Target location; includes the Replication Servers’ subnet, IPs, and the Replication Servers and their disks..

AWS

CloudEndure utilizes EBS for CloudEndure MigrationThe CloudEndure solution that allows you to move data, applications, and other business elements from an onsite network or a cloud environment to another physical location or cloud environment. and Disaster RecoveryThe CloudEndure solution that enables the recovery or continuation of vital technology infrastructure and systems in case of a crippling event..

For MigrationThe CloudEndure solution that allows you to move data, applications, and other business elements from an onsite network or a cloud environment to another physical location or cloud environment. & Disaster RecoveryThe CloudEndure solution that enables the recovery or continuation of vital technology infrastructure and systems in case of a crippling event., EBS magnetic is utilized for volumes less than "<" 500GiB. For volumes greater than ">" 500GiB, CloudEndure utilizes GP2, GP3 or ST1, depending on the disk type. The typical ratio of volumes to replication serversThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. is 15:1 for MigrationThe CloudEndure solution that allows you to move data, applications, and other business elements from an onsite network or a cloud environment to another physical location or cloud environment. & Disaster RecoveryThe CloudEndure solution that enables the recovery or continuation of vital technology infrastructure and systems in case of a crippling event..

What type of Replication Server is utilized in the CloudEndure Staging Area?

AWS

CloudEndure provisions a t3.Small server for MigrationThe CloudEndure solution that allows you to move data, applications, and other business elements from an onsite network or a cloud environment to another physical location or cloud environment. and Disaster RecoveryThe CloudEndure solution that enables the recovery or continuation of vital technology infrastructure and systems in case of a crippling event. replication. The typical ratio of volumes to replication serversThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. is 15:1 for MigrationThe CloudEndure solution that allows you to move data, applications, and other business elements from an onsite network or a cloud environment to another physical location or cloud environment. & Disaster RecoveryThe CloudEndure solution that enables the recovery or continuation of vital technology infrastructure and systems in case of a crippling event..

Does CloudEndure compress data during replication?

Yes, CloudEndure utilizes LZW compression during transit resulting in 60-70% compression depending on the type of data.

Are events that are generated by the CloudEndure servers logged in Cloudtrail in AWS?

Yes, CloudEndure generates standard AWS API calls that are visible in CloudTrail.

You will need to confirm that you are filtering on the CloudEndure IAM user in CloudTrail.

How many snapshots does CloudEndure create?

For each disk, there are two groups of snapshots:

  1. Internal CloudEndure snapshots needed for replication - 5-7 for each disk. Frequency and exact number depend on various factors, such as change rate on the Source machineA physical or virtual computer. and network stability.
  2. For Point-in-Time RecoveryDisaster Recovery feature that allows to recover your systems to a specific Recovery Point in the past., the frequency and number of Recovery Points are exact - every 10 minutes in the last hour, every hour in the last day and every day in the last month (60 in total per disk).

There is currently no mechanism for users to adjust the frequency and number of snapshots.

Does CloudEndure Delete Snapshots and Staging Volumes?

CloudEndure automatically deletes snapshots and staging volumesThe disks attached to the Replication Servers that are no longer used (such as those left over after SourceThe location of the Source machine; Currently either a specific Region or Other Infrastructure. machines have been removed from the User ConsoleCloudEndure SaaS User Interface. A web-based UI for setting up, managing, and monitoring the Migration and Disaster Recovery solutions..)

How much capacity is allocated to the Staging Area?

A volume is created for each volume in the SourceThe location of the Source machine; Currently either a specific Region or Other Infrastructure. infrastructure of the same size.

Why is 0.0.0.0:1500 added to inbound rules in the Staging Area?

CloudEndure uses TCP Port 1500 for replication between the SourceThe location of the Source machine; Currently either a specific Region or Other Infrastructure. AgentsThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. and the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location.. The connection is open for all IPs and can be managed by ACLs or networks controls to limit inbound IPs.

Can CloudEndure replicate Oracle ASM?

Replication of Oracle with ASM is fully supported.

CloudEndure replication works directly on block devices and not on mounts. ASM volumes are presented as block devices and therefore replicated transparently by CloudEndure.

How do I know if the initial sync is over?

The Data Replication Progress status bar should show Continuous Data Protection and LagA delay between the changes that occur on the Source and the replication of that changes to the Target. Lag appears when replication does not occur for a certain period of time. should be None.

How long does a rescan take?

A rescan may occur after a reboot of the Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine.. The rescan time will vary depending on the size of the SourceThe location of the Source machine; Currently either a specific Region or Other Infrastructure. disks. The time depends on the performance of the disks (linear read), target disksThe disks attached to the Target machines. (linear read), and the rate of write operations on the Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine. (which are sent in parallel with the re-scan.) The re-scan is functioning normally as long as its moving forward and is not "stuck".

Note: A rescan usually only occurs after a graceful reboot in a Linux environment. During the rescan, all contents of the disk will be read to ensure that no changes were made to the disk. During the read time, RPO and Lag experienced may grow - the larger the disk, the larger the RPO and Lag.

Is CloudEndure replication crash consistent?

Yes, CloudEndure is crash-consistent.

How do I replicate Clusters with CloudEndure?

Replicating your Cluster nodes can be done in one of the following ways:

Option A

  1. Installing the CloudEndure AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. on all nodes.
  2. Upon cluster node failoverA backup operational mode in which the functions of a system component are assumed by secondary system components when the primary component becomes unavailable through either failure or scheduled down time., the data disks from the original node will be automatically removed from replication.
  3. Reinstalling on the new node after a failoverA backup operational mode in which the functions of a system component are assumed by secondary system components when the primary component becomes unavailable through either failure or scheduled down time., the data disk will start replicating from scratch on that new node.

Option B

  1. Installing the CloudEndure agentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. on one node only and disabling the CloudEndureVolumeUpdater service.
  2. Upon cluster node failoverA backup operational mode in which the functions of a system component are assumed by secondary system components when the primary component becomes unavailable through either failure or scheduled down time., that node will not have the data disks so the RPOA metric indicating of the amount of data at risk of being lost. Determined by the amount of time between data protection events and reflects the amount of data that potentially could be lost during a disaster recovery. The maximum targeted period in which data might be lost from an IT service due to a major incident. grows.
  3. When the cluster fails back, restart the CloudEndure service to rescan the disks and replicate the incremental data.

Option C

  1. Installing the CloudEndure AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. on all nodes and disabling the CloudEndureVolumeUpdater service.
  2. Upon cluster node failoverA backup operational mode in which the functions of a system component are assumed by secondary system components when the primary component becomes unavailable through either failure or scheduled down time., restart the CloudEndure service on the new active node, the disks will be rescanned and the incremental data (from the last time this node was active) will be replicated.
  3. The limitation is that if the source cluster is X machines that have Y storage each, CloudEndure will maintain the Y storage X times on the cloud for replication (resulting in X times Y used storage).

The Volume Updater service is responsible for:

  1. Adjusting the replication volumes when a disk size on the Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine. is changed.
  2. Removing from replication any disks that are no longer attached to the machineA physical or virtual computer..

When replicating Clusters, the disks may be detached from the machineA physical or virtual computer. temporarily when the cluster fails over, and therefore disabling that service is recommended.

When the VolumeUpdater service is disabled, note:

  1. When changing the size of a disk, re-run the CloudEndure installer on the machineA physical or virtual computer. manually.
  2. When re-installing the CloudEndure AgentThe Agent is a software program that is installed on machines in the Source location that need to be migrated or replicated to the Target location. (either manually, or after an upgrade for example), the service will become active again and need to be disabled if you decided to go with options B or C above.

How do I control the Replication Server IP address range?

You can set the Replication ServerThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location. IP address range by using a VPN connection to determine the IP addresses within your internal network. Ensure that you have already set up a VPN connection in the AWS Console prior to enabling this setting within the Replication SettingsIn the Console UI, a set of settings affecting the data replication process, including Replication Servers subnet and IP, data throttling, use dedicated replicator, use multiple connections, etc. tab in the CloudEndure User ConsoleCloudEndure SaaS User Interface. A web-based UI for setting up, managing, and monitoring the Migration and Disaster Recovery solutions..

Can I replicate to more than one Target at once?

No, you can only replicate to a single TargetThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery). at once with CloudEndure. To replicate to multiple TargetsThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery)., you will have to create multiple ProjectsA Project is the basic organizational unit for running a CloudEndure solution. - one for each TargetThe location where the Replication Server will be located and where Target machines will be created (as a result of Test, Cutover or Recovery)..

Can I replicate machines from an AWS Source to an AWS Target across multiple AWS Accounts?

Yes.

What does Lag Exceeds Maximum Threshold Mean?

This means that the machineA physical or virtual computer. has been experiencing lagA delay between the changes that occur on the Source and the replication of that changes to the Target. Lag appears when replication does not occur for a certain period of time. for longer than 48 hours.

Can I migrate BYOL machine licenses into AWS with CloudEndure?

When migrating Linux machines into AWS with CloudEndure, all licenses are BYOL. Any RHEL, SUSE or Debian licenses will be transferred in their current form to the migrated machineA physical or virtual computer.. The customer is responsibly to ensure that the terms of their licenses allow this license transfer.

When migrating Windows machines into AWS, you can Bring Your Own (BYOL) Windows LicenseAllows the installation of the Agent on one machine. One License enables one Agent. Licenses are a part of the License Package. by setting a Dedicated Host in the machineA physical or virtual computer. Blueprint. When this is done, the licenses from the Source machineThe computer, physical or virtual machine that needs to be protected by replication (Disaster Recovery) or migrated (Migration) The CloudEndure Agent is installed on the Source machine. will be automatically transferred to the Target machineThe Machine created during Test, Cutover or Recovery. in AWS.

Note: For MSDN Windows licenses, you can use the Dedicated Instance option in addition to the Dedicated Host option. Selecting the Dedicated Instance option in the machine Blueprint does not allow Windows BYOL by default. To be able to transfer MSDN Windows licenses over to Dedicated Instances, contact Support.

Can I use CloudEndure to migrate my RHEL 5 and 6 machines to AWS Nitro instances?

No. AWS Nitro instances do not support RHEL 5 and 6. Choose XEN-based instances instead.

Does CloudEndure support the EBS volume multi-attach feature?

No. CloudEndure does not currently support replicating EC2 instances that utilize the EBS volume multi-attach feature.

How does CloudEndure replicate Source machines that have more than 15 disks?

The regular ratio for CloudEndure is 15 disks for each single replication serverThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location.. Any single machineA physical or virtual computer. with more than 15 disks will have its own replication serverThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location.. The disks will not be split into separate replication serversThe CloudEndure Machine to which Staging Disks are attached and to which data is replicated; launched on the Target location..


©2020 COPYRIGHT CloudEndure - Terms of Service - Privacy Policy - AWS Vulnerability Reporting Guidelines - Report a Security Issue