Administration Guides

Eyeglass Jobs

Home

Understanding how Eyeglass Jobs Work

Jobs are the basis for all automation in Eyeglass and job types dictate the type of automation that will be performed. Currently the job types supported are the following:

Unconfigured (new in 2.5.6 or later releases) - new policy detected that needs a type set.
Configuration Replication (shares, exports, permissions, nfs alias).
DFS mode enabled config replication policies (shares only).
Skip Share, Export, Alias replication.
Quota Replication (quotas - all types).
Snapshot schedule and Dedupe settings.
Access Zone replication.

Failover related job types:

Failover jobs (SyncIQ policy or Access Zone or DFS).

Created by DR Assistant.
- Failover Readiness.
Runbook Robot jobs.
Disaster Recovery Testing - LiveOPS DR test mode jobs.
Access Zone data migration
- Created by Data Migration Icon

Further to the above jobs, the following modes can be set on jobs:

Automatic - These are built by Eyeglass after auto detection of SyncIQ policies which are then used to detect which shares, exports, nfs alias and quotas should be replicated. They are automatically created.
DFS Mode - These are set by the administrator and will be used when configuring DFS mode failover. This job type will rename SMB shares on the read-only cluster automatically. You can switch a job from DFS mode back to sync mode.
AutoSkipConfig Mode - These are set by the administrator and will be used for the case where you do not want SMB shares and NFS exports to be synced to the target cluster, but still want to be able to failover the data. You can switch a job from AutoSkipConfig back to sync mode.
Custom:
1. These are user created and allow detection of shares, exports, quotas in a path in the file system, to be detected and added to a job, that will replicate the configuration to a target cluster without needing a SyncIQ policy to exist.
2. The data and path must exist on the target cluster

Job Types

Configuration Replication Jobs Automatically Created by Eyeglass

0) Type Unconfigured

Purpose: New in 2.5.6 and later releases, all new detected policies will by shown in a new section called Unconfigured. These jobs must have a type set of Auto or DFS and then enabled before any sync actions will occur.

A new alarm is raised for each new policy detected that requires configuration applied before the policy enters into production DR status.

How to Set Type on Unconfigured Jobs

Open the Jobs icon and select an Unconfigured Job
Click Bulk Actions menu
2. Then select AUTO or AUTODFS or SKIPCONFIG
Then select the policy in the AUTO or AUTODFS or SKIPCONFIG section and choose Bulk Actions menu and enable it to put the policy into production sync state. If left in User Disabled state the policy is not synced and not available for DR in DR assistant.

New Unconfigured Alarm

When a new Synciq policy is created steps must be taken in Eyeglass to set the type and enable it for DR readiness. A new alarm is raised for any synciq policy that is detected during the inventory process that runs every 5 minutes by default. The new policy alarm names the cluster where the policy was detected.

1) Share, Export, Alias replication (Type: AUTO)

Purpose:

Identify shares / exports / nfs alias that are related to SyncIQ Policies detected based on SyncIQ policy source path.
Synchronize these configuration items so that they exist on both clusters:
- associated shares & their configuration
- associated exports & their configuration
- associated nfs alias & their configuration

Eyeglass Configuration Replication Job Name convention: < PowerScale Cluster name >_< SyncIQ Policy name >

Job Creation:

The Share, Export, Alias replication jobs are auto created by Eyeglass after auto detection of SyncIQ policies which are then used to auto-detect which shares, exports, nfs alias, Access Zones and quotas should be replicated by Eyeglass.

Schedule: All Eyeglass share/export/alias configuration replication Jobs execute on a 5 minute schedule.

Initialstate: User Disabled (does not run), Must be enabled by the administrator in the Jobs icon

2) Access Zone Replication (Type: ZONES)

Note: This replication occurs when the associated Zone is NOT the System Zone.

Note: Recommendation to leave disabled unless directed by support.

Note: Access zones are case sensitive when syncing.

Purpose:

Identify Access Zone that is related to SyncIQ Policies detected based on SyncIQ policy source path.
Synchronize Access Zone so that it exists on both clusters:

Associated Zone & it's configuration, example user access mappings are synced.

Note: Deleted Zones on a source cluster are not deleted on the target cluster.

Eyeglass Zone Job Name convention: < PowerScale Cluster name >_< SyncIQ Policy name >-< ZONES >

Job Creation:

The Access Zone replication jobs are auto created by Eyeglass based on the SyncIQ Policy of the same name.

Schedule: Zone replication jobs execute on a 5 minute schedule.

Initialstate: User Disabled (does not run).

3) Quota replication (Type: QUOTA)

Background:

Quota jobs that are created based on SyncIQ autodetection are placed in a pending state. This state prevents quotas policies, that are collected and shown in the Inventory tree, from being applied to target clusters paths protected by SyncIQ policies. This is a best practice due to some scenarios that result in errors when quotas are applied to a target cluster file system.

The scenarios to apply quota policies are below.

In a failover event, the quota job can be selected and "Run Now" option used AFTER the target cluster file system is writable as a result of SyncIQ failover. This is run automatically under normal conditions. Cluster migrations is another use case where applying to target without a delete on source is desirable.
Custom jobs can replicate quota policies on a schedule for a path selected in the job. The quotas are applied successfully only when the target file system path on the target cluster already exists. Any new quota created under the selected job path, will be detected and replicated only if the target path also exists.

Note: When you run a QUOTA Job associated with an AUTO share/export configuration replication Job, the Job is based on the Eyeglass current inventory view. If you have made a change in OneFS to a quota, the Eyeglass Inventory Task must have run (runs on a 5 minute schedule) prior to running the QUOTA job in order for the change to be applied on the target.

Warning review quota failover limitations:

This section should be reviewed when planning quota failover solutions.

Review Dell EMC Quota EMC KB - https://support.emc.com/kb/88602

Some combinations of quota domain settings and SyncIQ source and target settings are incompatible. There are several scenarios where this might occur:

Multiple quota domains span SyncIQ target subtrees. SyncIQ translates source operations such as file or directory moves (mv) into similar actions on the target cluster. Moving files or directories across quota domains is not supported, and the syncs will fail.
Multiple quota domains span SyncIQ source subtrees. If the source cluster will be used for failback, the failback operation could error if multiple quota domains span SyncIQ source subtrees.
If the source cluster will not be used for failback, multiple quota domains can exist on the source.
Quota domains exist in directories other than the top-level directory of the SyncIQ policy locations. If a SmartQuotas quota domain overlaps with a SyncIQ policy domain, and failback is desired, then the quotas created must exist only on the top-level directory of the SyncIQ policy source and target locations.
- A QuotaScan job is still running when sync job starts. The quota scan identifies statistics about the files in a quota domain. If the quota scan does not finish identifying all the files that belong in the quota domain before the sync job starts, the sync will fail.
- Nested subdirectories receiving new files below an applied quota path at the target side. As part of the transfer process, SyncIQ first creates files and directories in a temporary directory in the target path, and then later moves (renames) them into the final destination. If the final destination has a quota domain configured, this will run into the quota limitation of not being able to move directories into and out of quota domains.

Purpose:

Identify Quotas that are related to SyncIQ Policies detected based on SyncIQ policy source path.
Synchronize Quotas so that they exist on both clusters:

Associated quotas & their configuration.

Eyeglass Quota Job Name convention: <PowerScale Cluster name>_<SyncIQ Policy name>_quotas

Job Creation:

The Quota Failover jobs are auto created by Eyeglass based on the SyncIQ Policy of the same name.

Schedule: Auto-created quota configuration replication Jobs do not run automatically. They are run on-demand as part of a failover.

Initialstate: User Disabled (does not run)

4) Configuration Replication: Snapshot Schedules (Type: FILESYSTEM)

Purpose:

Sync Snapshot schedules found on SyncIQ paths. Syncs the schedule as per SyncIQ policy paths defined. Will also read dedup path settings (scan and actual) for SyncIQ Policies that match only, and apply the path (corrected by policy path) to target cluster. Can be disabled independently of snapshots using igls command.

Job Creation:

Automatically build, user disabled by default must be enabled.

Schedule: The FILESYSTEM Jobs are run automatically on a 5 minute schedule.

Initialstate: The FILESYSTEM Jobs are disabled and must be enabled, execute at configuration replication scheduled defined.

Note: Snapshot schedule config sync does not overwrite existing Snapshot schedules on the target cluster which have a different Snapshot schedule name.

Configuration Replication Jobs not Auto Created by Eyeglass

1) CUSTOM (Type: CUSTOM)

Purpose:

Use when config data is not protected by SyncIQ policies. Scan path to find config data and replicates to target cluster and path defined in the job. Once created this Job will:

Identify shares / exports / nfs alias that are related to the Custom Job path.
Synchronize these configuration items so that they exist on both clusters:

Associated shares & their configuration.
Associated exports & their configuration.
Associated nfs alias & their configuration.

Job Creation:

If you see Job Type “CUSTOM” it means that this is a share/export/nfs alias/quota configuration replication Job that was created manually in the Eyeglass web page. A CUSTOM job that was not created based on a SyncIQ Policy and a SyncIQ Policy is not required or allowed.

Note: Multiple Eyeglass configuration replication jobs where paths overlap is not supported. (i.e. A Custom Job path cannot overlap with another Custom Job path or an “AUTO” Configuration Replication Job).

Note: Eyeglass custom job where path is the parent of another job is not supported.

Schedule:

The CUSTOM Jobs are run automatically on a 5 minute schedule.
QUOTA Jobs associated with a CUSTOM share/export configuration replication Job are run automatically on same replication schedule as the associated CUSTOM share/export replication Job.

Initialstate: User Disabled (does not run)

2) Configuration Replication: DFS Mode (Type: AUTODFS)

Purpose:

Please refer to Eyeglass SyncIQ Failover and Failback with Microsoft DFS for more information.

Job Creation:

DFS Mode is enabled manually from the Eyeglass web page. Please refer to Eyeglass SyncIQ Failover and Failback with Microsoft DFS for more information.

Schedule: The AUTODFS Jobs are run automatically on a 5 minute schedule.

Initialstate: The AUTODFS Jobs when enabled will have the same state as the AUTO Job that it came from.

3) Configuration Replication: Skip Share, Export, Alias replication Mode (Type: AUTOSKIPCONFIG)

Purpose:

Use this configuration mode for the case where the Shares, Exports or NFS Aliases purposely have different properties and permissions on source on target cluster and need to remain unique. In this case the Job remains enabled in the Jobs window but does not perform and configuration sync operations during Configuration Replication. However, because the Job is still enabled the associated SyncIQ policy can still be failed over thus executing other failover steps such as allow writes and resync prep.

Job Creation:

Skip Mode is enabled manually from the Eyeglass web page: Jobs -> select Job -> Select a bulk action -> Enable/Disable Skip Config Replication

Schedule: The AUTOSKIPCONFIG Jobs are run automatically on a 5 minute schedule.

In the Running Jobs view the Job will appear in the execution list but will not list any Shares or Exports. The Info confirms that the configuration data is not synced.

Initialstate: The AUTOSKIPCONFIG Jobs when enabled will have the same state as the AUTO Job that it came from.

4) Disaster Recovery Testing: (Type: AUTOMATIC)

Purpose:

3RD Copy LiveOPS feature, sync’s config from prod to DR test access zone if enabled.

Job Creation:

The mode is auto, and is automatically built if Eyeglass detects DR test mode policy, user disabled by default must be enabled.

Schedule: The AUTOMATIC Jobs are run automatically on a 5 minute schedule.

Initialstate: The AUTOMATIC Jobs are disabled and must be enabled, execute at configuration replication scheduled defined.

5) Runbook Robot: (Type: RUNBOOKROBOT and AUTOMATIC)

Purpose:

Continuous DR feature to failover and back daily to DR readiness

Job Creation:

Mode is auto since it's automatically built if Eyeglass detects robot access zone or policy name, and user disabled by default must be enabled. Igls command to set the schedule

Schedule: The RUNBOOKROBOT Jobs are run automatically on a 24 hour schedule.

Initialstate: The RUNBOOKROBOT Jobs are disabled and must be enabled, execute at configuration replication scheduled defined.

6) Failover Readiness Jobs (Type: AUTOMATIC)

Purpose:

Analyzes Access Zones against failover readiness criteria (data, config, SPN, network SmartConnect mapping). Updates the DR dashboard with Access Zone readiness and criteria readiness status. Collects subnet and pool data to display failover mapping on the Access Zone Readiness panel on the DR Dashboard. Failover Readiness Jobs are created between replicating cluster pairs, one job for each direction.

Note: Involved in failover operations, it is used to update DR assistant on readiness, update DR dashboard for users to correct errors, alarms Zone Readiness status to alert administrator.

Job Creation:

Mode is auto and is built automatically by Eyeglass and set to user disabled. The job can be enabled to analyze access zone readiness, if access zone failover is not planned it can remain disabled.

Failover Readiness Job Name convention: PowerScale Cluster name_PowerScale Cluster name

Schedule: Execute on a 15 minutes schedule.

Initialstate: Disabled

7) How to use Runbook Robot

The Runbook robot is an automated test of your failover infrastructure. It will failover an isolated syncIQ policy or access zone with only test data to ensure that the failover of production data is successful if and when that should be necessary.

Runbook Robot does the following:

Runs a failover.
1. during failover, the data will be synced to the DR cluster, and the DR side
Record results as a pass/fail.

How RunbookRobot job works with SMB

Runbook Robot job with SMB protocol:

Active Directory Credentials while adding Isilon Cluster to Eyeglass from Add Managed Device. They are used for Share creation and set permissions to the Share.

Note: There is also an option in Edit Managed Device for adding Active Directory Credentials.

Checklist:1. Open Eyeglass from SSH

cat /opt/superna/sca/conf/system.xml
Make sure this tag is set to <runbookrobot_mount_type>smb</runbookrobot_mount_type>
cat /opt/superna/sca/conf/runbookRobotConfig.json
Make sure it looks like this {"mount":true,"createExportOrShare":true}

Process:

Creates an SMB Share on the Product cluster with given Active Directory credentials
Writes a timestamp to a file on that SMB Share.
Runs a failover
1. Config Replication is run during failover to replicate the newly created share on the DR side.
2. The data will be synced to the DR cluster.
During failover validation, we read the timestamp on the file and ensured it was the same timestamp written at the job's start.
We write to the file again to ensure the DR copy is active.

Default runbook mode as SMB

Robot mount is enabled by default.

Command to enable:

admin@eye28:~> igls adv runbookrobot set --mount true

{

"success": true

}

admin@eye28:~> igls adv runbookrobot

{

"mount": true,

"createExportOrShare": false

}

admin@eye28:~> igls adv runbookrobot set --createExportOrShare true

{

"success": true

}

admin@eye28:~> igls adv runbookrobot

{

"mount": true,

"createExportOrShare": true

}

Note: Both need to be true/false to enable or disable without any errors