All Product Installation and Upgrade Guides

Eyeglass Warm Standby Direct Sync Guide

Home

Overview


This procedure is to protect the production Eyeglass appliance using a 2nd Eyeglass appliance.  This solution offers sync of Eyeglass backup file to the 2nd appliance that can take over operations under the following conditions:

  1. Controlled failover - switching the active appliance from one data center to the other

  2. Uncontrolled failover protection - the 2nd appliance can be used for failover operations since it has a current near real time synced status and copy of all policies and configuration data (shares, exports and quotas) needed to complete a failover to the surviving cluster.

    1. NOTE: If you declare an uncontrolled failover is required. You must ensure the active appliance is at the site where the data will be become active. The Warm standby appliance procedure should be used if the active appliance is at the site that had the disaster. The appliance should be co-located with the data that will be writeable.


Definitions:

  1. Active Appliance - Responsible for syncing configuration data and is the primary appliance for all failover operations

  2. Active Sync Appliance - 2nd appliance that has a backup synced from the active appliance


Operating Considerations

  1. If you execute a planned controlled failover you should generate a backup from the active appliance after the backup completes.  This backup reflects the current DR state after failover.  The daily backup and sync process will generate a backup but you require a current view of the clusters DR status.  Always generate a backup after a planned failover.
  2. Best Practise:
    1. Enable phone home and request support to enable daily appliance backup to create an off site backup of your appliance.  This backup is retained for 14 days before it is automatically deleted. This provides an off site backup of your DR state.
    2. After a planned failover push an off site backup using direct to support option.
      1. Select the Support backup you generated with the check box, select the "Upload selected file directly Superna support".  This will push an off site backup that can be requested back if needed.
    3. OR If phone home is disabled and firewall or proxy does not allow the direct upload to support option,  then download a newly created support backup from the Active appliance (after the failover), then upload to the support page following steps here.   This backup is retained for 14 days before it is automatically deleted. 
    4. In addition to the above options, download the support backup file and store a copy at the opposite site from where the active appliance is located.   Document this location internally to use with the recover options below if it is determined this backup is the most recent backup.
    5. NOTE: The daily sync backup enabled by direct sync is 24 hours old and is typically the backup required in most cases.  

Deploy 2nd Appliance & Prep for Sync

  1. Follow the install guide to deploy a 2nd appliance.  Guide link.

Configure Keyless SSH on Active Appliance


  1. Login via ssh as the admin user
  2. type: ssh-keygen (hit enter to all prompts)
  3. ssh-copy-id -i /home/admin/.ssh/id_rsa.pub admin@x.x.x.x (where x.x.x.x is the ip address of the 2nd warm appliance,  enter yes to accept the ssh finger print,  enter the remote warm standby appliance admin user password)
  4. Test keyless ssh was successful
    1. ssh admin@x.x.x.x  (where x.x.x.x is the ip address of the remote warm standby appliance)
    2. if no password prompt then everything worked 
  5. done


Configure Scheduled Cron Sync from Active to Warm Standby

  1. Login to the Warm StandbyAppliance as admin user over ssh and set the permissions
    1. sudo mkdir -p /opt/superna/var/backup/ 
    2. sudo setfacl -m u:admin:rwx /opt/superna/var/backup/ 
  2. Login to the Active Appliance to Test copy backup files
    1. Login as the admin user
    2. scp /opt/superna/var/backup/* admin@x.x.x.x:/opt/superna/var/backup/  (x.x.x.x is the ip of the warm standby appliance)
    3. if successful continue, if not debug the steps above.
  3. Create the script on the Active Appliance
    1. Login as the admin user over ssh
    2. nano /home/admin/warmstandby.sh
    3. paste the string below into the file and replace x.x.x.x with ip of the warm standby appliance:
      1. scp /opt/superna/var/backup/* admin@x.x.x.x:/opt/superna/var/backup/ 
    4. Press control+x  answer yes to save the file
    5. chmod 777 /home/admin/warmstandby.sh 
  4. Schedule the script to run daily at noon with cron on the Active Appliance
    1. Login to the Active Appliance over ssh as the admin user
    2. sudo -s (enter admin password) 
    3. cd /etc/cron.d
    4. type this command:
      1. echo "0 8 * * * admin /usr/bin/timeout 6h /home/admin/warmstandby.sh" > iglsstandby
    5. Restart the cron service to pickup the new script
      1. systemctl restart cron
  5. Verify backup copy script is running
    1. Wait until the next scheduled copy
    2. Login to the warm standby appliance as admin user and list the folder to verify files have the correct daily date stamp
    3. ls -ls /opt/superna/var/backup/ 


How to Restore the Warm Standby Appliance to become Active Procedures (Requires 2.5.6 release or later)

  1. You can restore a backup to your second Warm Standby Eyeglass appliance to make the standby the active appliance. SSH as the admin user into the warm standby appliance and execute the following steps:
    1. ssh to as admin user on the Warm Standby Eyeglass appliance
    2. Run the restore command
      1. NOTE that command must include full path to backup zip file or use path only and restore command will detect the most recent backup it discovers on the path based on the time stamp on the backup zip files. 
      2. Option #1 Auto detect the most recent file
        1. igls app restore /opt/superna/var/backup/ 
          1. This option will auto detect the most recent backup files and displays it for confirmation to use the suggested file, accept the file selection.  You will be prompted to enter the admin password again to become the root user after entering the command.
      3. Option #2 Use a specific file
        1. igls app restore  /opt/superna/var/backup/<name_of_backup.zip>
          1. NOTE: you will be prompted to enter the admin password again to become the root user after entering the command.
    3. You will be prompted with a confirmation of yes/no.
      1. For Testing Only  Answer No to exit the process.
        1. This step should be used to test the restore procedure without actually running the restore.
        2. This will not restore the database and will not make the standby appliance the active appliannce.
    4. For Production Active appliance switch to the Warm Standby Answer answer Yes
      1. Continue and monitor the command execution until it completes before trying to login to the GUI, it may take 15-20 seconds before you can login to the web UI.
      2. IMPORTANT: Original Eyeglass appliance should be powered off if switching to the Warm Standby Appliance. NEVER have two appliances operating against the same clusters. This can cause a conflict and is not supported.  Verify your backup appliance is up and running after a restoration. Open a Browser to  ip address of warm standby appliance.
    5. Once the restore process completes, you can login to the GUI to start a failover job following the documentation for executing a failover. see Guide here.


How to Complete Appliance Switch Test Procedures


  1. Install 2 appliances as per above steps in this guide
  2. Shutdown the active appliance VM (power off)
  3. Follow instructions to switch to the Warm standby Appliance
  4. Verify the login to the GUI and open the jobs icon , and running jobs tab to verify normal configuration sync jobs are running, view the DR Dashboard after waiting at least 15 minutes to get a current view of DR Readiness.
  5. Test completed
  6. How to revert to the Active Appliance follow these steps:
    1. Login warm standby appliance
    2. Factory reset the warm standby appliance
      1. Login via ssh as admin user
      2. run this command
        1. ./opt/superna/sbin/reset.sh
        2. This command will delete the database and remove all clusters added to the appliance and return to a default state.
    3. Power on the Active Appliance VM again
    4. Login and verify jobs, and DR dashboard
    5. Done
© Superna LLC