Eyeglass All Product Installation and Upgrade Guides Publication

Eyeglass Warm Standby Direct Sync Guide

Home

Overview

This procedure is to protect the production Eyeglass appliance using a 2nd Eyeglass appliance. This solution offers sync of Eyeglass backup file to the 2nd appliance that can take over operations under the following conditions:

Controlled failover - switching the active appliance from one data center to the other
Uncontrolled failover protection - the 2nd appliance can be used for failover operations since it has a current near real time synced status and copy of all policies and configuration data (shares, exports and quotas) needed to complete a failover to the surviving cluster.
1. NOTE: If you declare an uncontrolled failover is required. You must ensure the active appliance is at the site where the data will be become active. The Warm standby appliance procedure should be used if the active appliance is at the site that had the disaster. The appliance should be co-located with the data that will be writeable.

Definitions:

Active Appliance - Responsible for syncing configuration data and is the primary appliance for all failover operations
Active Sync Appliance - 2nd appliance that has a backup synced from the active appliance

Operating Considerations

If you execute a planned controlled failover you should generate a backup from the active appliance after the backup completes. This backup reflects the current DR state after failover. The daily backup and sync process will generate a backup but you require a current view of the clusters DR status. Always generate a backup after a planned failover.
Best Practise:
1. Enable phone home and request support to enable daily appliance backup to create an off site backup of your appliance. This backup is retained for 14 days before it is automatically deleted. This provides an off site backup of your DR state.
2. After a planned failover push an off site backup using direct to support option.
  1. Select the Support backup you generated with the check box, select the "Upload selected file directly Superna support". This will push an off site backup that can be requested back if needed.
3. OR If phone home is disabled and firewall or proxy does not allow the direct upload to support option, then download a newly created support backup from the Active appliance (after the failover), then upload to the support page following steps here. This backup is retained for 14 days before it is automatically deleted.
4. In addition to the above options, download the support backup file and store a copy at the opposite site from where the active appliance is located. Document this location internally to use with the recover options below if it is determined this backup is the most recent backup.
5. NOTE: The daily sync backup enabled by direct sync is 24 hours old and is typically the backup required in most cases.

Deploy 2nd Appliance & Prep for Sync

Follow the install guide to deploy a 2nd appliance. Guide link.

Configure Keyless SSH on Active Appliance

Set password for root on the warm standby
1. login as admin on the warm standby appliance
2. sudo -s (enter admin password)
3. passwd (set a password for root)
Login via ssh as the admin user on the active appliance
1. sudo -s (enter admin password)
2. cd /root
3. type: ssh-keygen (hit enter to all prompts)
4. ssh-copy-id -i /root/.ssh/id_rsa.pub root@x.x.x.x (where x.x.x.x is the ip address of the 2nd warm appliance, enter yes to accept the ssh fingerprint, enter the remote warm standby appliance root user password set in the steps above)
Test keyless ssh was successful
1. From the active appliance ssh session logged
2. whoami (very you are the root user)
3. ssh root@x.x.x.x (where x.x.x.x is the ip address of the remote warm standby appliance)
4. if no password prompt then everything worked
done

Configure Scheduled Cron Sync from Active to Warm Standby

Login to the Warm StandbyAppliance as admin user over ssh and set the permissions
1. sudo mkdir -p /opt/superna/var/backup/
2. sudo setfacl -m u:admin:rwx /opt/superna/var/backup/
Login to the Active Appliance to Test copy backup files
1. Login as the admin user
2. sudo -s (enter the admin password)
3. rsync -auv -e "ssh -i ~/.ssh/id_rsa" --delete -og --chown=sca:users /opt/superna/var/backup/ root@x.x.x.x:/opt/superna/var/backup/
  1. (x.x.x.x is the ip of the warm standby appliance)
4. if successful continue, if not debug the steps above.
Create the script on the Active Appliance
1. Login as the admin user over ssh
2. sudo -s (enter the admin password)
3. nano /root/warmstandby.sh
4. paste the string below into the file and replace x.x.x.x with ip of the warm standby appliance:
  1. rsync -auv -e "ssh -i ~/.ssh/id_rsa" --delete -og --chown=sca:users /opt/superna/var/backup/ root@x.x.x.x:/opt/superna/var/backup/
5. Press control+x answer yes to save the file
6. chmod 777 /root/warmstandby.sh
Create an Advanced Script with failure detection and email notification on the Active appliance
1. Skip down to the advanced script section and once completed, continue with Step #5 below.
Schedule the script to run daily at noon with cron on the Active Appliance
1. Login to the Active Appliance over ssh as the admin user
2. sudo -s (enter admin password)
3. cd /etc/cron.d
4. type this command:
  1. echo "0 8 * * * root /usr/bin/timeout 6h /root/warmstandby.sh" > iglsstandby
5. Restart the cron service to pick up the new script
  1. systemctl restart cron
Verify backup copy script is running
1. Wait until the next scheduled copy
2. Login to the warm standby appliance as admin user and list the folder to verify files have the correct daily date stamp
3. ls -ls /opt/superna/var/backup/

How to create an script with failure detection and email alerting

You must configure postfix email with eyeglass to use email alerting script. Follow this guide to configure eyeglass and the eyeglass production VM to support email. Setup Eyeglass OS SMTP to Send mail to your mail server. Then return to this guide to finish creation of the script.
Login as the admin user over ssh
nano /home/admin/warmstandby.sh
Copy & paste the text below into the file and replace x.x.x.x with ip of the warm standby appliance and replace email@example.com with the group email to receive alerts when sync from the production eyeglass fails to sync backups to the warm standby.
Press control+x answer yes to save the file
chmod 777 /home/admin/warmstandby.sh

Script text (sample only must be edited)

if ! rsync -auv -e "ssh -i ~/.ssh/id_rsa" --delete -og --chown=sca:users /opt/superna/var/backup/ root@x.x.x.x:/opt/superna/var/backup/

then

echo Sync failed

echo sending error email

mail -s "Sync Failed to standby Eyeglass Appliance" -r email@example.com < /dev/null

exit

echo sync was successful

mail -s "Sync was successful to standby Eyeglass Appliance" -r email@example.com < /dev/null

exit

How to Restore the Warm Standby Appliance to become Active Procedures (Requires 2.5.6 release or later)

You can restore a backup to your second Warm Standby Eyeglass appliance to make the standby the active appliance. SSH as the admin user into the warm standby appliance and execute the following steps:
1. ssh to as admin user on the Warm Standby Eyeglass appliance
2. Run the restore command
  1. NOTE that command must include full path to backup zip file or use path only and restore command will detect the most recent backup it discovers on the path based on the time stamp on the backup zip files.
  2. Option #1 Auto detect the most recent file
    1. igls app restore /opt/superna/var/backup/
      1. This option will auto detect the most recent backup files and displays it for confirmation to use the suggested file, accept the file selection. You will be prompted to enter the admin password again to become the root user after entering the command.
  3. Option #2 Use a specific file
    1. igls app restore /opt/superna/var/backup/<name_of_backup.zip>
      1. NOTE: you will be prompted to enter the admin password again to become the root user after entering the command.
3. You will be prompted with a confirmation of yes/no.
  1. For Testing Only Answer No to exit the process.
    1. This step should be used to test the restore procedure without actually running the restore.
    2. This will not restore the database and will not make the standby appliance the active appliance.
4. For Production Active appliance switch to the Warm Standby Answer answer Yes
  1. Continue and monitor the command execution until it completes before trying to login to the GUI, it may take 15-20 seconds before you can login to the web UI.
  2. IMPORTANT: Original Eyeglass appliance should be powered off if switching to the Warm Standby Appliance. NEVER have two appliances operating against the same clusters. This can cause a conflict and is not supported. Verify your backup appliance is up and running after a restoration. Open a Browser to ip address of warm standby appliance.
5. Once the restore process completes, you can login to the GUI to start a failover job following the documentation for executing a failover. see Guide here.

How to Complete Appliance Switch Test Procedures

Install 2 appliances as per above steps in this guide
Shutdown the active appliance VM (power off)
Follow instructions to switch to the Warm standby Appliance
Verify the login to the GUI and open the jobs icon , and running jobs tab to verify normal configuration sync jobs are running, view the DR Dashboard after waiting at least 15 minutes to get a current view of DR Readiness.
Test completed
How to revert to the Active Appliance follow these steps:
1. Login warm standby appliance
2. Factory reset the warm standby appliance
  1. Login via ssh as admin user
  2. run this command
    1. sudo /opt/superna/sbin/reset.sh
    2. This command will delete the database and remove all clusters added to the appliance and return to a default state. If asked about which databases select the All option.
3. Power on the Active Appliance VM again
4. Login and verify jobs, and DR dashboard
5. Done