Failover Release Notes

1. The latest version of Eyeglass has been installed, we add enhancements to each release based on customer failovers to prevent or document anything that will block or impact failover. Not upgrading to latest release affects your entitlement to support for a planned failover event. Excludes real DR failovers.
2. If you are planning failover and want Eyeglass DR readiness assessment, we require 7 days advanced notice and support logs submitted.
3. If your Eyeglass installation is N-2 releases where N is the currently published GA release, you may choose to stay on a release that is N-2 without affecting support only if the following steps are completed:
   1. Open a case with support and upload support logs.  Then follow instructions below.
      1. State planned failover will use a release that is within N-2 and update the case.
      2. State the Failover Features of N release has been reviewed here and update the case to state confirmation.
      3. Support will request confirmation in the case that N, N-1 and N-2 "failover section" of the release notes (example here) has been reviewed and that you are confirming and accepting the risk in your environment. 
      4. NOTE: Failure to complete #1-3, will affect support entitlement of using N-2 release for a planned failover.
4. Target DR Releases if already running one of these releases
   1. Check with support for target release or consult software matrix above.
   2. NOTE: If not running these releases upgrade to the latest GA release is required.
   3. NOTE: if using one of these releases, all release notes apply and assumed read and accepted prior to any planned failover. 

(Release 1.6.3 >) Snapshot schedule expiration offset has OneFS API bug that adds extra time to creation of the snapshot schedule.

1. This results in an expiration on the DR cluster, that can be greater than entered on the source cluster.   example expire in 20 days will be 22 days on the target cluster.  Different units of off set all result in a value greater than entered.   After failover the DR (target cluster) value will be synced back to the source (Prod cluster).  Thereby losing the original expiry off set  and extending the expire time by a new offset from the API error.  This has been raised with EMC as SR to resolve.
2. Work around:  Before failover ensure a cluster report has been generated (cluster reports icon), or an existing emailed cluster report exists. Post Failover re-enter the original values on the DR snapshot schedules using the cluster report values from the source cluster as a reference.
   1.  Another option is disable Snapshot Sync jobs in the jobs window if the above workaround does not meet your needs to preserve expiry of snapshot settings.

(All Releases) SyncIQ file filters not supported 

1. File pattern filters are NOT synced on failover, these pattern filters can result in unprotected data during failover and failback. Failover and failback work flows require customer testing for their own use case.  All file filter scenario’s are untested without support for custom workflows related to file filters failover failback issue not present under normal failover and failback workflow

(Release < 1.8.0) Technical Advisory #10

1. For the case where PowerScale clusters have been added to Eyeglass using FQDN, uncontrolled failover for case where source cluster is not reachable does not start and gives the error ""Error performing zone failover: Cannot find associated source network element for zone". This issue will be addressed in a 1.8.1 patch.  Eyeglass installations using FQDN to add clusters must upgrade to this patch once available. Workaround: please refer to Technical Advisory #10

(All Releases) OneFS Failover and Failback without waiting for quota scan job to complete

1. In OneFS 8 quota scan job is started as soon as a quota is created (cannot be disabled on OneFS 8).  Resync Prep on failover or failback will fail when Quota scan job is active on a path on the target cluster. Do not add/edit quotas before or during failover. If you have Quotas with snapshot overhead enabled, deleting a snapshot may trigger a quota scan.   Also, after Eyeglass failover quotas are created by Eyeglass and quota scan will start.  If failback is attempted right away (typically testing only scenario) without waiting for quota scan to complete the resync prep step is blocked from running due to domain lock from the quota scan.   Workaround:  Wait for quota scan job to complete before attempting failover or failback.   Use cluster running jobs UI to verify if quota scan is running or not before attempting to failover.

1. For the case where OneFS 8 is configured with multiple groupnets and different AD provider between groupnets, the SPN update during failover does not succeed for non-default groupnet AD providers. SPN's are not deleted for source cluster and are not created for the target cluster. The failover log indicates success. This is due to a OneFS8 defect with multiple AD providers.  NOTE: SPN delete / create for the AD provider defined in groupnet0 is successful. Workaround: Manually delete and create the SPN for the Smartconnect Zones that were moved from AD ADSI Edit interface.

1. DFS mode uses parallel threads to rename shares for all policies involved in the failover.  
2. If share renaming is failed for all the shares from a cluster, then failover status is error. Failover is stopped and Users are not redirected to target cluster.  Make writeable and Resync prep does not run and data is active on source cluster still.
3. If share renaming is failed only for some shares from the source cluster, then failover status is warning AND failover will continue to run make writeable and resync prep.  
4. Summary:  In this scenario it is best to attempt the failover of some shares fail rename.  If all fail abort failover and stop.

1. This feature has been available for some time and should be understood how it works.
2. During make writeable step Eyeglass will send API to target cluster to start the make write able step. 
3. At this point in the failover smartconnect networking and SPN failover has been completed and dual delegation will mean new mount requests will be handled by the target cluster and SPN authentication will be handled by the target cluster.
4. If the make writeable step Succeeds on at least ONE policy of N (of all policies involved in the Access zone),  the failover logic will continue.  This means you are partially failed over for some of the data in the access zone. It also means all networking and SPN's are failed over.  Next step is to resolve failed make write step on policies to get the file system writeable.  This often requires EMC SR to resolve root cause of failover on the target cluster.
5. If NONE of the policies pass the make writeable step AUTOMATIC rollback of Smartconnect networking and SPN's are reverted to the source cluster. 
   1. The failover log shows if networking rollback is initiated. If you find this in the failover log, Your failover is aborted and all data remains writeable on the source cluster.
   2. Example Log entry 2017-06-08 22:49:00::260 INFO Starting Step: "Networking Updates Rollback"
   3. To validate source cluster data access to the following:
      1. nslookup to the smartconnect name(s) involved in the failover (use failover log for full list).  IP returned should be from the source cluster
      2. Test share and NFS mount access to the source cluster and verify you can mount and write
      3. This will validate SPN authentication for shares as well.
      4. Determine root cause , which may require EMC SR to resolve before rescheduling the failover
6. Summary: This logic determines the best option automatically.  If some data succeeds to failover its best to resolve only the failed policies than aborting the entire failover.  If no data succeeds at the make writeable step it is best to revert and abort the failover.    Eyeglass handles this decision automatically.