DR Design Guides

Failover Advanced mode Configuration - Parallel thread and failover jobs


Failover Advanced mode Configuration - Parallel thread , failover jobs & Concurrent Failovers Overview

These configurations are aimed at customers that have greater than 50 policies for business reasons and require faster failover option to maintain SLA on data recovery.

3 features exist:

  1. Parallel threads - allows make writable and resync prep to operate in parallel up to thread limit of 10.  This means 10 policies will be executed at a time for all SyncIQ steps, and Eyeglass will ensure that at least 10 policies are executing at a time throughout the failover process across all failover jobs.
  2. Parallel Jobs - used to allow more active jobs to share threads for failover.  This value and the threads value should be set to the same values.  It defaults to 10 jobs.
  3. Concurrent Failovers - A default of 5 is configured but can be increased with the parameters above to increase the number of active failover jobs. In a multi tenant scenario this my be required when users are able to submit failover jobs.

Parallel Failover Job and Concurrent Failover Configuration

This feature allows multiple failover jobs of any type to be failed over in parallel .  This means multiple failovers can be running at the same time.  This feature still has a 10 thread limit for all failover jobs that are shared across all jobs.  This can be combined with the parallel threads feature to increase each failover jobs parallelization.  Testing this in advance of a failover is mandatory step.  3 different values must be changed to increase the parallel jobs, threads and concurrent failover limits.   NOTE:  Do NOT start more than 4 concurrent failovers with releases < 2.5.6.  A failover job can be any type of failover with any number of policies in each failover job.  

How to increase Fast Failover parallel threads, increase Failover jobs and concurrent failover limit

These options default to 10 failover jobs and 10 threads in a pool.  This can be increased in release 2.5.6 which has been tested to 50 failover jobs and 50 threads. It is NOT recommended to increase beyond these limits.   The default for concurrent failovers is 5 but can be increased.  See steps below.

  1. Login via ssh as admin to Eyeglass.
  2. sudo -s
  3. Enter admin password to become root user.
  4. Type: nano /opt/superna/sca/data/parallelTaskLimits.json
    1. To increase to 20 failover jobs and 20 threads change the values below to 20 and 20.  NOTE:  increasing the job count requires increasing the thread count to the same number.
    2. {"parallelJobs":20,"totalParallelTasks":20
    3. Use the arrow keys to move, and delete key to change the values.
    4. Then to save and exit press CTRL+x,  answer Y to save the changes .
  5. To change the default concurrent failover limit.  This is required if the parallel jobs count is increased or if 5 concurrent fail overs is not enough.
    1. Type: nano  /opt/superna/sca/data/system.xml .
    2. Add a tag inside the <process> tag .
    3. Add a new line and paste this tag into the file and change the yellow value.  This has been tested to 50 concurrent failovers and it is NOT recommended to change to a higher value. NOTE: Do not increase the value above the parallel jobs value.
    4. <max_concurrent_failovers>5</max_concurrent_failovers> .
    5. Then to save and exit press CTRL+x  answer Y to save the changes .
  1. After making changes above the SCA process must be restarted:
    1. systemctl restart sca
  2. The feature is now enabled after the restart command above.
  3. To submit parallel Access Zone, policy, DFS or IP Pool jobs use the DR Assistant to start a failover job.  
  4. Close DR Assistant, re-open it and start another failover.
  5. Repeat, the above step to submit more parallel failover jobs.
  6. Monitor all failovers from DR Assistant running failover tab.
  7. NOTE:  Cluster resources may be exhausted and testing is mandatory prior to attempting a very large number of failovers.  

Parallel threads for Failover Mode Configuration (Legacy Mode)

This mode switches to parallel policy with up to 10 threads for make all steps.  This defaults to enabled in all current releases.  No need to change unless directed by support.

Key differences between default sequential and parallel mode:

  1. For 8.x clusters, 50 policies can run at a time and Eyeglass will use a maximum of 10 threads allow 10 policy make writable or resync prep commands to be sent in at once. For 7.2 clusters only 5 will execute and 5 are queued.  If one policy completes, another policy is started with the goal of keeping maximum number queued at all times.  
  2. Testing has shown 3x to 4x improvements in overall time to complete make writable.  Results in production may vary.

How to enable Fast Failover parallel threads

This runs steps with multi threads, and runs steps in parallel .  NOTE: 2.5.4 and later is enabled by default.

  1. igls adv failovermode set --parallel=true .
  2. Done. The change affects all failover jobs.
  3. Disable with:
    1. igls adv failovermode set --parallel=false .
© Superna Inc