Administration Guides

AirGap 2.0 Guide

Home


Overview

Superna offers several products to protect data from Ransomware or unauthorized access.  We recommend the Ransomware Defender product as the primary tool to protect data since it covers all requirements for detection, prevention, and recovery.   Ransomware Defender includes an AirGap 2.0 solution that provides the only solution on the market that integrates user behavior detection of the protected data to suspend data updates to the secure vault copy until administrators take action on the alarms.    Complete role based administration solution with split roles for user behavior monitoring and separate AirGap administration.

Golden Copy File to Object Integration

Expanding on file based projection solution for Isilon is the introduction of Golden Copy Advanced that can copy files to Objects off site locations example Amazon S3 or Azure to allow.   This integration allows Golden copy to get real-time updates on source cluster data threats from Ransomware Defender user behavior monitoring and suspend syncing to the S3 targets.

Easy Auditor Integration Enables Custom Vault Open Criteria 

  1. Customers that own the Easy Auditor platform can extend the vault auto close replication criteria using Easy Auditor active auditing.   This extends the security of protecting the vault data by using builtin triggers for DLP , Mass delete or custom triggers to control vault replication.
    1. This enables security teams to apply user aware, network aware policies that will stop replication for any active events in Easy Auditor.    
    2. This provides a powerful capability to customize when the vault replication should occur depending on the data that is being protected.  No other solution offers fully customizable real time triggers to control vault replication.
    3. The guide on active auditing can be referenced here

Key Features

  1. Unified data protection at the source and Automated AirGap in a single product.  
  2. Integrated user behavior detection "Smart AirGap"  ensures data sync is suspended when suspicious user behavior is detected on the source production cluster.
  3. Split roles between AirGap management and Vault cluster access and passwords and day to day monitoring of data protection of the source production cluster.
  4. Fastest incremental always sync solution on the market powered by SyncIQ, keeps the AirGap open the least amount of time to sync the block changes of individual files into the Vault.
  5. Lowest Cost storage with longest retention of Vault data with SnapshotIQ that only stores the block differences of the changes made to the file system in production.  This translates to the longest retention of your versioned data with the lowest cost.
  6. The only solution that offers < 2 hour repaid recovery of any quantity of data.   Protect Peta bytes of data and bring it online < 2 hours provides unparalleled recovery speed to handle the worst case data recovery scenarios.
    1. Eyeglass DR mirrors shares, exports and quotas to the vault cluster to ensure your rapid recovery solution has the exact data access security as production.    No other solution considers configuration data  as a critical component of a Cyber Vault.
  7. Easy Auditor extends the solution to include full file auditing solution to monitor suspicious user behavior covering Data Loss Prevention,  Mass Delete detection and custom triggers along with historical searching of audit data.
  8. Full User auditing within the Unified Desktop 
  9. Future API integration with Golden Copy to allow Golden Copy copy/sync control when active threat to source cluster is detected.   For more information on license requirements for Golden copy click here.
  10. Integration with Easy Auditor will block replication to the Vault cluster if Mass delete, DLP or other custom triggers have active events.
  11. Inside vault Smart Airgap solution provides a VM inside the vault to automate the Airgap with the same functionality as the virtual Airgap mode.   


Solution Summary diagram


AirGap Location Independent Solution


How protect high change rate and low change rate data 


FAQ - Buyers Guide to Cyber Vaults

  1. How will the data be protected in the vault?
    1. The 3rd PowerScale is the vault since SyncIQ locks the replicated data in a read-only state, this data cannot be deleted or modified even by the root user on the cluster.
  2. Can I use my DR cluster as the target?
    1. We recommend a 3rd cluster because DR replication should be as fast as possible ie 5 minutes to protect against site failure.  A  3rd Cyber vault copy should be slower example 24 hours to allow detection time of data issues that would require the 3rd copy.  This buffer time is important to ensure compromised data is not replicated into the vault.
      1. The 2nd reason is an AirGap means fire-walling and securing the 3rd copy device.   The DR should be available and reachable for a DR event and blocking all access to the DR site would compromise a DR solution effectiveness and readiness.   The firewall would still require Eyeglass access to the DR cluster so this solution is not as secure as a dedicated Vault cluster device.
      2. In summary, a 3rd copy AirGap copy of data and DR have opposite requirements that prevent using the DR cluster as the AirGap device due to sync interval and security impact of fire-walling your DR site.
  3. Can the Airgap be opened and closed from the inside of the cyber vault?
    1. Yes Ransomware Defender supports 2 modes of operation with an outside the vault automation and an inside the vault automation that requires additional resources inside the cyber vault. 
  4. Is the data immutable? What is the granularity?
    1. Yes, the data is immutable and cannot be modified regardless of the permission applied to the data
    2. The entire source cluster, or only certain paths, can be replicated to the vaulted PowerScale.  This is a matter of creating more SyncIQ policies to sync specific paths of data to the vault.
  5. How will the AirGap be managed? Internal to vault or external? What are the requirements?
    1. The recommendation is the vault PowerScale is located next to the source cluster since this is not a DR copy of the data.  Off-site remote is supported using SyncIQ which allows the vault PowerScale to be located anywhere.  This will add AirGap networking requirements to ensure no IP routing to the remotely located cluster, and will not allow any access other than from the source production PowerScale.  This is a networking implementation with firewalls but is fully supported.
      1. Considerations for location of the vault onsite versus remote site means you are trading off rapid recovery with the onsite device versus remote that offers a longer recovery of data to an onsite device.
      2. Recommendation is always use an on site solution since DR is designed to protect against site failures and a Cyber vault is protecting data from an on site threat.
  6. How will a customer get into the vault to make future modifications?
    1. Inline SSH access to through the production cluster.  This will allow access to the vault PowerScale over SSH with a command that can open the AirGap for maintenance, and will automatically close the AirGap after X minutes.
    2. Optionally a physical management interface can be connected to the vault PowerScale for GUI access. This would require physical access to connect and disconnect the management interface.  The recommendation is to use the inline SSH access since the PowerScale CLI provides all management commands.
  7. What type of maintenance should be expected?
    1. A feature will collect alarm data via the AirGap when a copy is running, and forward the data through the Normal Eyeglass alarm feature (email, syslog snmp etc..).  This allows full alarm event monitoring of the vault PowerScale even when it is fully disconnected most of the time.    Any serious alarms detected would then require remediation, or CLI access to debug the issue on the Vault PowerScale.
    2. Benefits:  inline AirGap vault aware alarm monitoring avoids most scheduled maintenance on the vault PowerScale until a critical alarm requires action.
  8. What is needed in the vault to perform analytics or reporting?
    1. All reporting and analytics are done by Ransomware Defender from the source cluster.  All data copy jobs are monitored, trended, and reported over 24 hours, 30 days, and 60 days. A report is emailed daily that covers success, failure, throughput, average AirGap open time, and more along with a CSV with raw per path and policy reporting. 
    2. Benefit:  Fully automated monitoring and reporting.
  9. How will data be made available outside the vault in a recovery scenario?
    1. The vault PowerScale can have data exposed for 3 different scenarios and offers flexibility that no other vault backup solution offers since PowerScale can serve data over SMB, NFS as well as act a vault.
      1. Access to a subset of the vaulted and immutable data.  Connect the management interface, create an SMB share on the data path that is required for recovery.  NOTE: The share data will be read-only regardless of the permissions since it is locked by SyncIQ.
      2. Access or recover all the data.    SyncIQ steps can be executed to reverse replicate the data from the vault PowerScale back to the production cluster and will use SyncIQ speed and performance advantage to restore data and ACL permissions to the source cluster.  This would be used when the volume of data that is needed to be recovered is a very large % of the data.
      3. Emergency Operations mode. This mode highlights the advantages of a vault PowerScale solution. This scenario turns the vault PowerScale into the production cluster to serve data directly from the vault without a recovery phase.  This allows getting operational as fast as possible to operate the business.  High-level steps for Emergency Operations Mode are outlined below.
        1. Connect management interface of the Vault PowerScale to the network
        2. Eyeglass DR can copy the shares, exports, quotas to the vault PowerScale (requires the Vault PowerScale to be added to Eyeglass, delete DR cluster first and add Vault PowerScale)
        3. Pre-staged IP pools, SmartConnect names from the production cluster needed to get access to some of the data that is urgently needed.   Connect interfaces to the network.  Update DNS to point at the new vault cluster with NS recored edit.
        4. Execute a failover from Eyeglass to the Vault cluster and sync all shares, exports and quotas to re-secure the vault data.
        5. Start accessing the PowerScale data.
        6. Done.
  10. How will testing of the data recovery be done?
    1. In vault:
      1. Via ssh from source cluster -   Use Open AirGap command to ssh to the vault cluster and then use scp command to copy the files to the production cluster.
    2. External:
      1. On a quarterly basis - Connect the management interface on the vault cluster to the network, create an SMB test share mount, and test read the protected data.  NOTE:  no write access will be allowed in this test mode.
      2. Delete the share and disconnect the management interface to complete the test.
      3. Benefit: vault replication is still active during a test with the source cluster still in full production mode.  No downtime is needed for this test.
  11. Who is creating the recovery runbooks?
    1. Everything is documented in this guide to operate and test the vault data, and most management is automated with no day to day tasks needed.  The professional service also offers customers assistance in the design, implementation and operations.  See here.
    2. No Run books are needed since most tasks are automated.   Pre-staged rapid recovery option is recommended.
  12. Is there any dependence on NTP or other services?
    1. No, the cluster time can free run from its own clock.  
  13. Are there any additional hardware/software components that may be recommended or required to make the overall solution work?
    1. 2 AirGap Options are available:
      1. Virtual AirGap - Requires a layer 2/3 switch to route between the source PowerScale and the Vault PowerScale.  All other requirements are within Ransomware Defender to manage static routes to reach the Vault PowerScale.
  14. Is there any visibility in production that a copy is being sent to a vault?
    1. Yes, full reporting 24 hour, 30 day and 60 day reporting on all copied data with success, failure, and throughput metrics along with AirGap average open time.  This is the time the Gap is open and should be minimized at all times. The solution reports on this daily, or an on-demand report can be created.

 

Requirements & Prerequisites 

  1. License Requirements
    1. Ransomware Defender license for each source cluster that is protected by the vault.
    2. Airgap Enterprise -  Agent VM license for inside the vault automation.
      1. No DR license for the Vault cluster is required.
      2. Future roadmap to include Smartlock automation feature inside the vault.
    3. AirGap Basic - This is builtin to the Ransomware Defender license. This will only support virtual AirGap with outside the vault automation.
      1. Vault cluster DR license with maintenance is required for a supported AirGap.
  2. A PowerScale cluster, any make model with OneFS release matching source cluster, sized for the data set and change rate and retention of data required.   Sizing can be done with assistance from the sales team.
  3. Dedicated Airgap Ethernet switch
    1. Minimum of 4 x 10G ports for synciq port connections.
    2. NOTE: It is not recommended to use the front end ethernet switches to connect to the vault cluster using a vlan.   Physical connections offers the best practice network separation and reduces the attack surface.
  4. OR Firewall - Enhanced network option for the vault networking
    1. This option provides additional control of ports and data flow into the and out of the vault cluster.  This provides enhanced security to reduce the potential attack surface and provides logging.  The inside vault switch can be a firewall to lock down this network. 
  5. Powerscale Production Node Connections


    1. Best Practise with High Availability:  At least 2 nodes and 1 interface per node on the production cluster and 2 nodes on the Vault cluster connected to the AirGap Ethernet switch.
    2. Next Best option without High Availability:  One node and 1 interface connected from production cluster to the Airgap Ethernet network switch.
    3. NOTE: if the production cluster has no available ports the choices above offer lower port count for physical separation connections to the AirGap network.   It is also possible to add nodes to the production cluster that are dedicated to connect to the AirGap network.   

Firewall Vault Network

  1. Production cluster --> to the vault cluster with vault network firewall deployment
    1. ssh (maintenance only)
    2. 8080 https TCP --> optional used for vault cluster hardware alarm collection and free space reporting only during data syncing
    3. synciq ports TCP ports 5666, 5667, 2097, 2098, 3147 and 3148  
  2. Vault cluster --> to protected cluster network firewall
    1. 8080 https TCP -->  to connect to protected cluster from the vault cluster
    2. ssh tunnel  --> to connect from the vault cluster to the production cluster.  Used to build secure tunnel to prod cluster to communicate with Ransomware Defender

Additional Requirements for Enterprise AirGap Licensed Deployments

  1. Hardware Recommendation: 
    1. A dual socket server with 128 GB of RAM, 1 and 10G Ethernet interface options,  2 - 4T B of local flash storage.
      1. Hardware should be future proofed to allow additional VM's to run for cyber security protection solutions and Windows desktop(s) for administrators that has key tools installed to allow a guaranteed clean, secure OS desktop to be used for used recovery operations or upgrades to the Isilon hardware, firmware and software.
  2. Ransomware Defender VM Agent
    1. Vmware ESX host server that will run a single Ransomware Defender VM with 16G ram 130G disk and 4 x vcpu.  This ESX host only needs to run this one VM but can be used to run other applications inside the vault.
  3. Networking - Basic Airgap option more secure option below
    1. Ethernet switch to connect the VMware ESX host to the management ports on the vault cluster.  The 1G Ethernet interfaces can be used to connect to the ESX host using the system zone management interfaces.
    2. Allows device expansion in the vault for future equipment
  4. OR Firewall - Enhanced network option for the vault networking
    1. This option provides additional control of ports and data flow into the and out of the vault cluster.  This provides enhanced security to reduce the potential attack surface and provides logging.  The inside vault switch can be a firewall to lock down this network. 

High Level Configuration Steps

  1. Install 3rd PowerScale at the same location as the cluster with data to be protected. 
    1. Best practice deployment:
      1. Use a bastion host (VM connected to private IP vault management network) and complete all configuration of the vault cluster through this bastion host.  This avoids connecting the vault cluster to the corporate network during commissioning steps.  The vault cluster should never be exposed to the network directly.
    2. NOTE:  The airgap can be located at the DR location using the DR copy as the source of the data to copy to the vault.
    3. Vault PowerScale Requirements on Deployment :
      1. Cyber Recovery RunBook:  As much of the pre-configuration, labeling, Ethernet port planning (VLAN's,) cabling and logic configuration as possible should be completed at deployment time to speed up recovery scenarios described in this guide.
        1. The configuration steps completed should be documented in a Cyber Recovery RunBook.  This will be used along with this guide when executing a cyber recovery scenario.  This guide documents the high-level steps needed to complete recovery.  These high-level steps should be turned into detail specific steps for your environment and added to the Cyber Recovery Runbook.
      2. Management System zone access network configured but should be disconnected physically after installation.
      3. Vault Cluster High level hardening
        1. Note:  The Advanced service will provide detailed hardening of the vault cluster. This service scope is out lined here.   The information below is not the complete solution and only identifies high level steps.
        2. Delete all default shares and NFS exports on the cluster.
        3. Stage and plan physical ports, or VLANs required for the Vault cluster nodes, to be connected to the production network in the event a rapid recovery scenario is required.  These cables should be physically in place but not connected, with labeling applied to each cables Ethernet port connection.   The node Ethernet interfaces should be the minimal configuration needed to serve data for production IP pools and Access Zones.
        4. Stop the SMB and NFS services.
        5. Add NTP server (even though it will not be reachable). Used for Rapid Recovery. 
        6. Add DNS servers (even though they will not be reachable).  Used for Rapid Recovery.
      4. For additional hardening consult Dell documentation on how to apply additional changes for hardening.
        1. NOTE:  The Airgap Design And Implementation Service includes hardening of the vault cluster based on Dell documentation.
      5. Production Powerscale IP Address space and IP pool for replication with SyncIQ and management IP pool
        1. Use the default groupnet and subnet (enable vlan tagging on the subnet).  
        2. A syncIQ pool and management pool will be created in this subnet
        3. Review the layer 3 vs layer 2 vault network pros and cons.
      6. Production Powerscale Management IP pool 
        1. Create a new IP pool in the new private IP subnet, and configure at least 2 nodes to join the management  IP pool and set the pool mode to dynamic for HA IP address failover.
        2. NOTE: Make sure the IP pool is set to System Access zone
        3. NOTE: Vault cluster does not require a management IP pool facing the vault replication network
      7. Vault and Production PowerScale SyncIQ replication IP pool 
        1. Create a new IP pool in the default subnet, and configure at least 2 nodes to join the replication SyncIQ IP pool (static IP pool), for HA replication access to the vault PowerScale from the source PowerScale.
        2. NOTE: Make sure the IP pool is set to System Access zone
      8. Vault cluster Inside Airgap
        1. If the inside Airgap solution is used an ethernet switch , ESX host and VM deployment are required for inside the Airgap automation.
        2. Ransomware Defender vault VM requires the eyeglass minimum permissions configured on the vault cluster.  See guide.
        3. The vault cluster is added to the VM using the eyeglass service account. 
        4. SSH tunnel to a production cluster to allow communications with Ransomware Defender from within the vault.  See the Enterprise license Vault Agent configuration in this guide.
  2. Source PowerScale
    1. Create a new IP pool called "Vault Replication", and add at least 2 nodes, and 2 interfaces to this pool. No SmartConnect name is required for this pool.  The pool must be in the system zone.
  3. Layer 3 Vault Replication switch
    1. The network between the source PowerScale and the vault PowerScale will require a layer 3 device between the clusters.  The Interface on the source PowerScale IP pool will have a static route with a next-hop of the layer 3 vault switch added, to reach the private subnet created on the vault PowerScale.
    2. NOTE: This does not need to be a managed device and should be a statically configured routing device.  It can be a larger switch using VLAN routing, but this exposes the potential for misconfiguration and allowing routing into the vault network.  This is a business cost decision as a VLAN routing configuration can also be used.   
    3. Best Practice:  Use a dedicated switch with physical separation from production networks, and do not enable management of this switch, or leave the management port of the switch disconnected.
  4. Ransomware Defender and Eyeglass steps
    1. Vault policies are created on the source PowerScale with the a policy name prefixed with rw-airgap-xxxx where xxxx can be any text to describe the policy, more than one policy can be created if required.
      1. SyncIQ Policy details:
        1. The source path should NOT be a path that is used as a DR replication path on your production cluster.
          1. Example: DR policy /ifs/data/zone1, the AirGap policy can use a policy path above or below this DR policy source path. i.e. above would be: /ifs/data, or below: /ifs/data/zone1/somepath.
          2. Reason: In a full re-sync recover from the Vault PowerScale, DR cluster mirror policies will cause an overlapping SyncIQ condition with 2 clusters trying to write data into the same path on the production cluster.  This will block the Vault cluster policies from running successfully in a full recovery scenario.  
          3. Solution: Avoid the overlapping condition by using non-overlapping paths when creating the AirGap policies.  This may mean creating more policies to replicate all the data and to avoid the overlap with DR.  This is the best option to avoid several manual steps in a full re-sync recover scenario and will make recovery simpler, faster, and less complicated.   
        2. Schedule = set to manual (note Ransomware Defender manages the policy)
        3. Target host - is the IP address of an IP on the vault PowerScale replication pool.   
        4. Restrict at source option enabled, and select the vault replication Pool created above to force replication traffic to use the vault pool node interfaces. This is the same pool that will have the static route applied for virtual Airgap mode.
        5. Create the policy with the same target path used on the source path
    2. Configure the policy replication schedule on the Eyeglass appliance, the recommended schedule is daily at midnight.
    3. Virtual Airgap mode
      1. Add the static route to the Eyeglass Ransomware AirGap GUI.
      2. The static route will be the next hop of the layer 3 vault switch, and target network will be the private network subnet created on the vault PowerScale replication pool.
    4. Inside the vault Airgap mode with Vault Agent VM
      1. This requires the Vault Agent VM to be deployed on a dedicated ESX host that is secured inside the vault with the vault cluster.
      2. Configure inside the vault agent vm to connect to the vault cluster with minimum permissions user
      3. Add management IP pool to source cluster mapping information (see guide for more details)
      4. Verify Ransomware Defender reachability with test command to verify Airgap interfaces can be opened closed and remote API calls to the Ransomware Defender are functional.
    5. Reporting requires no steps other than configuring email on Eyeglass to receive the daily Airgap sync reports.
    6. Vault PowerScale alarm monitoring requires a service account user on the vault cluster.  This simplifies management and monitoring of the Vault PowerScale if any hardware faults are detected.  Alarm collection is completed during replication windows when the network is open.  This means alarms will only be collected once a day if the replication schedule is daily.


Security Configuration of Components

The sections below outline additional security configuration that should be implemented when deploying the AirGap feature.

Eyeglass VM Security

  1. Implement the hardening guidelines and password complexity and password management using this guide.
    1. Implement the fail to ban feature to auto ban and firewall failed login attempts to eyeglass using the hardening guide above.
    2. Configure 2 factor SSH on Eyeglass, ECA and the Vault Agent VM following this guide.
  2. Firewall ssh and https access to the Eyeglass VM to management network jump box (administration VM) that has 2 factor authentication.
  3. Using the ECA firewall requirements restrict ECA ports to only be authorized between Eyeglass and ECA and allow ssh access to the ECA from a network jump-box only.  See firewall guide here.
  4. Configure Role for AirGap management and configuration separately from Ransomware Defender management.   see next section.
  5. Full user UI access and configuration auditing covered in this guide.


Vault Cluster Configuration

  1. NOTE: The AirGap Design and implementation service covers more hardening, the items below are the minimum changes that should be applied to the cluster.
  2. Use only local accounts and no AD provider.  This simplifies security infrastructure and ensure a lower attack surface to the device itself.
  3. Enable Configuration Auditing to track all changes made to the cluster configuration.
  4. Disable all non essential services
    1. SMB, NFS
    2. Delete default SMB share and NFS export
  5. Disable all built in users accounts except for the root user
    1. The password should be a random password 20 characters or longer with upper case, lower case, numbers and at least 1 special character
    2. This password should be created and managed by the senior management within the security team and should not be shared with anyone outside of the security group.
  6. Create the Eyeglass service account with minimum permissions for vault alarm collection, see minimum permissions guide.



Role Based Management of the AirGap Feature

  1. The AirGap feature is added to the Ransomware Defender builtin role
  2. This allows the AirGap management to be separate from day to day Ransomware Defender management.  See example of dedicated role option that can be removed from the Ransomware Defender role and added to a custom role. 
  3. Recommendation:  CSO or senior security management personnel should be assigned this role.  The personnel with this role should be separate from the Ransomware Defender personnel. 
  4.  

Detailed Deployment Diagrams

Network Considerations for Layer 2 or Layer 3 Vault Network

Overview

The vault network itself can be designed using layer 2 or layer 3 between the prod and vault clusters.   The choices and best practices are as follows.

Layer 2 or Layer 3 - Fan-In cluster protection (Enterprise Airgap License)

  1. Highest Secure networking option - Inside the vault VM agent  with Enterprise Airgap license.
  2. The Vault network the connects each protected cluster to the vault can use layer 2 flat vlan or a layer 3 network with routing between the protected clusters and the vault cluster.
    1. Example below is a layer 3 network example 
    2.  
    3. Example below is layer 2 vault network
  3.  Best Practise:
    1. For multiple source production clusters, a single vault network subnet allows all clusters to attach to a single layer 2 network and a single vlan can be used between all the clusters. 
    2. A layer 3 vault network allows a firewall to be used between the source protected clusters and the vault cluster to add additional traffic firewall rules between the clusters.



  1. Virtual AirGap Mode - Layer 3 Vault Network (Basic Airgap License)

    1.  The diagram below shows the networking required for source and vault PowerScale clusters, and how the vault switch and ip static routes should be configured for initial setup and configuration.   The static route added on the source PowerScale will be added to the Ransomware Defender configuration to open and close the virtual AirGap.
    2.  
  2. Inside the Vault mode Deployment (Enterprise AirGap License Required)

Overview

This provides an alternate mode of operation with an inside the vault host and VM that opens and closes the vault from within the vault.   This requires the Enterprise Airgap license key. This is done by removing the replication interfaces from the IP pool which removes the IP address from the interfaces. This mode places a VM inside the vault and disables the IP stack that connects the vault cluster to any outside network.  This mode offers Smart Airgap feature and all the same automation and enables this through a hardened Linux OS that autonomously manages the Airgap and verifies if is safe to replicate data.  The slides below show how this mode is deployed.


Inside the Vault Physical Topology - Layer 3 example


How to Enable Inside the Vault Agent VM (Enterprise AirGap)

  1. Requirements:
    1. Airgap Enterprise agent VM license key is installed during deployment to enable this mode.
  2. To enable Airgap policies to be managed by the secure hardened inside the Vault VM agent the Airgap administrator must switch from Virtual Airgap mode defaults to inside the vault mode.  
  3. The inside the vault VM agent will collect schedules configured in the Airgap UI and import them during initial setup.
    1. Once activated all vault open and close operations are managed by the Vault VM agent vm securing the vault access and shutting the vault networking if Ransomware defender or Easy Auditor triggers have alarms raised.
    2. Smart Airgap - Active alarms will cause the vault vm agent to shut the network down without replicating data.
    3. NOTE: a static route will still need to be added, a fake route can be used that has no relevance to replication network.  This requirement will be removed in a future release.  example route 192.168.1.0/24 next hope 192.168.1.1 (NOTE: this assumes you are not using 192.168.x.x ip ranges)
  4. Open the Airgap Icon
  5. Click on Settings
  6. Enable "Managed by vault agent" check box and click save
  7.   
  8. Done


Operations of Vault Data Replication

  1. When using the inside the vault VM the CLI commands to force open the vault for maintenance are not supported and physical console access is required to gain access to the vault cluster or ESX host and VM.  A physical keyboard mouse inside the locked cabinet will be required.    This is a more secure operating mode.
  2. 2 modes exist on the vault agent that allow a 2 hour maintenance heartbeat API request from the vault agent VM.  This heartbeat API checks for a request for a maintenance access window using Airgap CLI command on the Eyeglass VM.
    1. This is defaulted to disabled but can be enabled to check for a request every 2 hours to open the vault for maintenance for a timed window in minutes.
    2. When the request is for 60 minutes the vault will close automatically after 60 minutes.
  3. Ransomware Defender Smart Airgap API reachability failure is a fail safe for the vault.   If the Eyeglass Smart Airgap API end point for safe replication cannot be reached the inside vault agent VM will fail safe and will close the vault on any failures keeping the data safe inside the vault.

Data Flow Example for Data Replication with Enterprise Airgap

  1.  


Vault Management Data Flow Example for Enterprise Airgap

Configuration Steps for AirGap Setup

Overview video

How to setup syncIQ policies for AirGap

  1. The AirGap policies are created on the Isilon and use restrict at source pool created in the physical configuration outlined in this guide.  This ensures vault cluster replication traffic will use the correct nodes and physical interfaces.  This also ensures Virtual AirGap to control the static route on this replication IP pool.
  2. Select the source path based on your data protection requirements that select the data that should be protected in the vault.   NOTE:  multiple policies can be created to protect different paths and change the replication schedule for each policy within the AirGap management GUI.
  3. The Name of the policy must use the following naming to be treated as an AirGap policy
    1. rw-airgap-xxxx where xxxx is unique part of the policy name.  See Advanced settings to change the policy name prefix.
  4. Synciq Policy Property requirements
    1. sync mode
    2. no schedule set leave at manual
    3. Mandatory - restrict at source pool set to the AirGap pool for synciq replication.  This is required for for Basic Airgap to add the static route to the correct pool
  5. Data Retention 
    1. This is an important consideration to provide maximum protection and options to recover data in a worst case data recovery scenario.
    2. Longer SyncIQ data Retention will require more space with longer retention.  Data change rates will determine how many days of retention.
    3. When creating the policy enable Target Snapshots mode and set the retention in days.  See example below.
    4.  
  6. New policies will appear in the AirGap icon AirGap Config tab.
    1. Configuration replication inventory defaults to 5 minutes to detect new SyncIQ policies
  7. Locate the policy in the Un-configured section of the jobs icon.  The policy must be run once before it will move to the Airgap section of the jobs window.
    1.  
  8. Verify the new policy appears in the AirGap icon
    1.  
  9. done.

How to Configure AirGap policies and setup Virtual AirGap (Basic Airgap License)

  1. The SyncIQ AirGap policy should be configured as per the above section.  Open the AirGap Icon to verify you can see the policy.  NOTE: The schedule is not set and the policy is not managed in this state.
  2.  
  3. Enter the Virtual AirGap subnet range , network mask bits and next hop gateway.   
    1. subnet = the network that the vault Isilon cluster IP pool that is configured for SyncIQ replication.
    2. The subnet mask bits to apply to the subnet entered.  example 24 bits for a 255.255.255.0 subnet
    3. The next hop gateway IP address will be the IP address of a router between the production Isilon IP pool for AirGap and the Vault Isilon.  Refer to the diagrams above on how to network the clusters together on a private network that is only reachable by the production cluster via the IP pool configured for the AirGap.  See example vault cluster subnet of 192.168.0.0/24 and next hope of 192.168.1.1
    4. NOTE:  You must enter a valid subnet that starts with the broadcast address for the subnet example 192.168.1.0/24 is the start of the subnet.  An invalid subnet would be 192.168.1.1/24 since this does not include the broadcast address.
    5.  
    6. Next configure the schedule by clicking the calendar icon and completing the scheduling.
    7.  
    8. The defaults radio button at the top if the windows allows simply setup for daily , weekly, monthly 
    9. Select Other to have a custom schedule and complete all fields to complete the custom schedule.
    10. Recommendation: Always enable pause data replication when active ransomware events detected.  This is the Intelligent data protection option that overcomes limitations on other backup based cyber vaults that allow encrypted comprised data to be copied into the vault.   
      1. Note the check box "Pause data replication when active Ransomware events detected"  This enables Smart AirGap mode that will monitor user behaviors for any activity that could be considered Ransomware this includes warning, major or critical detections.
      2. If these alarms are not cleared or managed as resolved in Ransomware Defender Icon the copy schedule will be skipped until an administrator makes addressed the alarms. 
      3. If Easy Auditor is installed all Active Auditor trigger active alarms will also block replication to the vault and must be cleared to allow replication.
        1. DLP, Mass Delete or custom triggers all block vault replication.   
        2. Suggested Configuration to enable a honeypot trigger to monitor snooping of open SMB shares.  See the guide.
    11. Best Practices:  This option should always be enabled to offer the highest protection level of your data and ensures no copies are stopped until an administrator makes a decision on the events.   When the events are cleared by an administrator AirGap will resume copies on the next schedule incremental update schedule.  Consult support if you plan to disable this check box.  If disabled the schedule will run regardless of what alerts are present in Ransomware Defender.
    12. Target cluster Credentials
    13.  
    14. The user and password should be the service account created above section for vault cluster configuration.  This service account is a minimum privilege user to collect alarm data only.
      1. These credentials are used to retrieve alarms from the vault Isilon in-band while the AirGap is open and proxies alarms on the Vault cluster to administrators to monitor physical hardware issues that may occur.
      2. This ensures an automated solution that is lights out at all times to secure the vault data.
    15. Then submit the save button and the state should now change to show next schedule replication and AirGap state 
    16. The AirGap policy is now in production mode.


How to test an Airgap Policy job

  1. Open the Jobs icon and click the run now icon to start the job and then monitor the job from job history tab of the airgap Icon.
  2.  

How Alarms from the vault Isilon are Viewed and Forwarded

  1. Configuring the Target cluster credentials allows remote alarm collection during incremental AirGap copies using the in-band replication network to collect alarms. 
  2. Alarms are forwarded through email only and will not display in the Active Alarms icon that is reserved for Eyeglass alarms only.  The history alarm will display on the Managed Cluster Alerts tab of the Alarms Icon.
  3. Sample email proxy alarm
  4. To route alarms to a specific email address use the Eyeglass custom email routing guide here.
  5. Example Tab in Alarms

How to Expand the Airgap Sync Job Timeout and the Airgap job prefix name 

  1. The default timeout is 240 minutes or 4 hours and will fail a sync job that takes longer. This only applies to incremental syncs. These steps can also be used to change the default prefix that is used to match the airgap synciq policy as an airgap policy.
  2. To change this timeout value
    1. On the eyeglass vm login as admin
    2. nano /opt/superna/sca/data/system.xml
    3. Add an airgap section with tags as per below and change the policy prefix value and or the timeout value in minutes.
    4. Save the file with control+x  and answer yes to save and exit. 
    5. <airgap>
      <policyPrefix>rw-airgap-</policyPrefix>
      <logsMaxAgeInDays>7</logsMaxAgeInDays>
      <airgapJobTimeout>240</airgapJobTimeout>
      </airgap>

       

Operational Procedures for AirGap Management 

  1. After the initial configuration, running the AirGap policies manually will start the large first full sync of the data. This can be done from the Onefs GUI SyncIQ tab.
  2. Monitor the initial data sync phase, and then enable AirGap on Ransomware Defender to take over the sync schedule and manage the AirGap replication automatically.
  3. Day to Day Administration  
    1. The Vault PowerScale is monitored in-band by Ransomware Defender to collect alarms. This allows administrators to monitor the vault PowerScale without needing to expose the vault PowerScale to the external network. When the AirGap is open to sync data, the in-band management is done over SSH from the production PowerScale to the vault PowerScale.


How to stop AirGap Replication in an Emergency

  1. If you suspect you IT environment has been compromised in any way it is important to shutdown the AirGap permanently to protect the AirGap copy of the data.
  2. See the AirGap CLI command in the CLI guide here
    1. ssh to Eyeglass as the admin user and issue this command to disable and isolate the AirGap data.
    2. igls AirGap disable

How to monitor replication AirGap policy success failure

  1. Login to Eyeglass and open the AirGap icon and click on the Jobs History tab to review the history of the replication jobs
  2.  


How to Monitor AirGap Replication Reports

  1. The SyncIQ jobs are managed and reported on by a dedicated AirGap report.  Setting up report notification in notification center with an email recipient set to reports , will receive the AirGap replication report.  Consult Eyeglass admin guide on how to configure email and recipients.

How to enable or disable the Airgap daily summary report or change the schedule

  1. igls admin schedules list (to check the current schedule)
  2. igls admin schedules set --id AirGapReportsTask --interval 7D (to change schedule to every week)
  3. igls admin schedules set --id AirGapReportsTask --enabled false (to disable the report)

How to pause all AirGap policies to complete Vault cluster maintenance

  1. This mode should be used to complete network or vault cluster maintenance and stops policy replication 
  2. See the AirGap CLI command in the CLI guide here

How to Pause the AirGap policies for maintenance with a timed auto close of the AirGap Network

  1. This option uses the igls AirGap connect, and disconnect command and operate separately on a per policy basis and sets a timer to keep the AirGap network open for X minutes or hours.    This ensures the AirGap network is not left open accidentally and automatically closes the AirGap network after the timer expires.
  2. See the AirGap CLI command in the CLI guide here

How to Configure  Enterprise AirGap Ransomware Defender Enterprise Airgap Agent

Overview 

This section covers how to configure the vault agent VM on the ESX host that is deployed inside the secure vault.  This VM manages the vault cluster and orchestrates all replication from protected source clusters.   The vault agent uses a secure SSH tunnel from the vault cluster to a source protected cluster to reach Ransomware Defender VM to send secure messages to orchestrate replication tasks,  upload logs,  download new policies or protected clusters and updates to schedules configured in the AirGap UI.


Topology and Communications


Prerequisites 

  1. Enterprise Airgap license
  2. ECA single VM deployed
    1. After first boot run this command
    2. ovf set-value -f mode=vault-agent
    3. How to startup the software
    4. ecactl cluster up


Configuration Steps 

  1. Install the Eyeglass Vault Agent (EVA) license in Eyeglass
    1. Login to Eyeglass, open the license manager Icon and click upload new license zip file.  This license is required to enable the managed by vault check box in the Airgap icon. 
  2. Configure Keyless ssh from Vault cluster to each protected source cluster to allow an ssh tunnel to be created for communications between the Vault Agent and the Eyeglass VM.
    1. NOTE:  The minimum permissions user eyeglass should be created on all clusters that are protected or the vault cluster.    The minimum permissions guide lists the ISI commands to verify have been applied to the eyeglass user.
    2. Login to the protected cluster that will be used for the SSH tunnel as the eyeglass user.
      1. mkdir .ssh
      2. procedure done
    3. Login to the vault cluster over ssh as the eyeglass user
      1. create an ssh key pair
        1. run this command: ssh-keygen -t rsa
        2. Hit Enter for default path
        3. Hit Enter for passphrase
        4. An ssh key pair should be created in /ifs/home/eyeglass/.ssh
        5. Copy public key (id_rsa.pub) to the primary protected cluster that will be used for Eyeglass communications.
          1. scp /ifs/home/eyeglass/.ssh/id_rsa.pub eyeglass@x.x.x.x:/ifs/home/eyeglass/.ssh/id_rsa.pub
      2. Complete protected cluster keyless SSH configuration
        1. ssh as eyeglass user to the production cluster
        2. cd .ssh
        3. cat id_rsa.pub >> authorized_keys
        4. chmod 600 authorized_keys
        5. done
      3. Test keyless SSH from vault to production cluster
        1. ssh to the vault cluster as eyeglass user
        2. ssh again to the production cluster
        3. If no password is requested then keyless ssh was successful,  if a password prompt is presented it means a step was missed and review all steps above were completed.
      4. done.
  3. Add the vault cluster IP and ip address for the SSH tunnel to this cluster.  (Requires license key applied)  
    1. ecactl isilons add --vaulthost x.x.x.x --user eyeglass --vaultPoolName groupnet0.subnet0.synciq --vaultsynciqexternalInterface 1:ext-1, 2:ext-1, 3:ext-1, 4:ext-1 
    2. --vaulthost  - isilon management IP address in system zone x.x.x.x is the system pool IP address on the inside of the vault.
    3.  --vaultPoolName  - the IP pool on the vault cluster used to receive synciq data from a protected cluster 
    4. --user - service account created on the vault cluster for the vault agent VM
    5. --vaultsynciqexternalInterface   - This is the list of interfaces in the synciq pool.   enter the node and interface name in a comma separated list example 1:ext-1, 2:ext-1, 3:ext-1, 4:ext-1 
  4. Add a protected cluster to the EVA VM to create the secure tunnel to reach Eyeglass Ransomware Defender
    1. NOTE: This cluster must be reachable over the vault cluster synciq IP pool interfaces configured in the vault cluster add command.
    2. NOTE:  This cluster will be used for all communications via a secure SSH tunnel from the vault cluster and will be used to send api calls to Ransomeware Defender.
    3. ecactl isilons add --protectedhost x.x.x.x --user eyeglass
  5. Test API communications over the secure tunnel to a protected cluster
    1. NOTE: This tests the ssh secure tunnel from the vault cluster to the named protected cluster and issues a test api to Ransomware Defender to verify end to end communications.  
    2. ecactl isilons list (to get the cluster name from the add command)
    3. ecactl airgap check --prod <protected cluster name>
    4. Example output
    5. Opening vault connection..
      Command succeeded
      Running command on vault.. whoami; hostname
      eyeglass
      Prod-cluster
      Running command on prod.. whoami; hostname
      eyeglass
      Prod-cluster
      Running command on eyeglass.. ' 'https://172.25.49.15/sera/v1/healthcheck' -k
      "\"Wed Aug 18 07:16:00 EDT 2021\""
      Closing vault connection
      Command succeeded
      DONE!

  6.  Modify a cluster configuration
    1. ecactl isilons modify --name ISL-EASEE-8-2-1-0-172-25-47-73 --vaultsynciqexternalInterface 1:ext-1, 2:ext-1, 3:ext-1, 4:ext-1, 5:ext-1 
  7.  List clusters of type vault and protected 
    1. ecactl isilons list
  8. Remove a cluster
    1. ecactl isilons remove --name ISL-EASEE-8-2-1-0-172-25-47-75
  9. List Synciq Jobs between the vault and a remote cluster
    1. ecactl airgap list
  10. Retrieves the airgap policies configured in Ransomware Defender, retrieves the schedules configured for each policy and saves this information locally to run on a schedule.  NOTE: New airgap policies and schedule changes are checked each time the vault opens to run a job.   A secondary schedule can be configured to check for new configuration or schedule changes independently of scheduled airgap sync schedules.
    1. ecactl airgap schedules
  11. Run an Airgap job on demand to test an incremental sync of data into the vault.
    1. NOTE:  Use the ecactl airgap schedules command to get the names of remote cluster synciq policies configured within the Airgap Icon in Eyeglass.   This will retrieve from all Ransomware Defender managed clusters.
    2. ecactl airgap runjob  --job ISL-EASEE-8-2-1-0-172-25-47-73_rw-airgap-test3
      1. --job  - this is the name of the policy returned from the list command above.


Operational Procedures Enterprise Airgap

How To reach outside the vault through the vault cluster it is possible to open and close the vault with cli commands

  1. ecactl vault open   - this will open the vault interfaces from the IP pool
  2. ecactl vault close  - this will close and remove the vault interfaces from the IP pool

How to Open the Airgap for maintenance from Ransomware Defender CLI

  1. NOTE: This command is default disabled on the vault agent and must be explicitly enabled.  This is for doing maintenance or temporary access to the vault and then disabling this feature after work is completed.  The vault agent will Not open the vault be default even if this command is used.
  2. In order to enable this feature on the vault agent.
    1. nano /opt/superna/eca/eca-env-common.conf
    2. add this variable
    3. export EYEGLASS_OPEN_VAULT_ENABLED=true
    4. save the file control+x
    5. ecactl cluster down
    6. ecactl cluster up 
  3. igls airgap vaultaccessrequest --interval=x  (request to open the airgap for x minutes, after this time the airgap will auto close,  the heartbeat check for pending requests is every 2 hours by default and will open the airgap for x minutes only once the pending request has been seen)
  4. igls airgap vaultaccessview   (view pending requests to open)
  5. igls airgap vaultaccesscancel  (cancel a pending request)

How to list running jobs

  1. ecactl jobs running

How to run an Airgap job from the Vault agent VM

  1. ecactl airgap startjob --job <job name>  (use ecactl airgap list to get job names)

How to monitor a running airgap job

  1. ecactl jobs view --follow --id job-1630432432546-879575052 (replace with job name)

How to check the remaining time of a maintenance window request on the vault agent

  1. Use this command if you have requested a timed maintenance window from the eyeglass vm.  This command runs on the vault agent.   NOTE:  The maintenance window time will survive and vault agent upgrade or cluster restart.
  2. ecactl airgap checkopen
    1.     


How to configure Vault cluster Log Gather Automation for Hardware Support

  1. These steps enable automation to allow automation to collect a log gather and place this on the production cluster to allow Dell Support the ability to verify the health of the vault cluster, if any alarms are proxied by the vault agent through Eyeglass managed devices.
  2. Requirements:
    1. 2.5.8 build 228 or later
  3. Configuration:
    1. ecactl airgap startisilonloggather -> to start the job now
    2. ecactl airgap isilonloggather -> to read the schedule
    3. Recommended for all deployments
      1. ecactl airgap isilonloggather --setschedule "0 0 * * *" => to set the schedule for the job every day at midnight
  4. Logging will output location of the log gather gz file
    1. Starting the vault gather job. Will find it under the production cluster /ifs/data/home/eyeglass/IsilonLogs-<vault_cluster_name>.tgz
    2. The file can be provided to Dell Support to monitor or investigate vault hardware cluster issues. 
  5. How to change production cluster path for the log gather in eca-env-common.conf
    1. export EVA_VAULT_LOG_GATHER_PATH_ON_PROD=/ifs/xxxx  (change xxxx to path to store the log gather in a different location)



Advanced Vault Agent and Airgap Eyeglass Configurations


Scheduled check for new or changed Airgap policies

  1. Configuration check schedule can be changed using a variable in the conf file.  This scheduled task will open the vault and download new policies or schedule changes and then close the vault.  The vault will only be open for a few seconds.
    1. on the vault agent VM
    2. nano /opt/superna/eca/eca-env-common.conf
    3. add this line with a cron string interval.    This is a 5 minute example.
    4. export TASKMASTER_AIRGAP_SCHEDULING_CRON="*/5 * * * *"
    5. control + x to save
    6. ecactl containers restart taskmaster

How to change the name of the Airgap policies.

  1. Use this procedure to change the name of the airgap policy prefix
  2. Login to eyeglass vm over ssh
  3. nano /opt/superna/data/system.xml
  4. add the tag below in a new section
  5. <airgap><policyPrefix>rw-airgap-</policyPrefix></airgap>
  6. Edit the yellow value to a string that will prefix all airgap policies.
  7. AIRGAP jobs are listed in Jobs window and AirGap Config'


How to enable Vault Agent Open delay

  1. This variable can be changed to increase a timer to wait x seconds after the vault is opened with the IP interfaces to allow time for the cluster to bring up the interfaces.  The default is 15 seconds.
  2. Login to the vault agent as ecaadmin
  3. nano /opt/superna/eca/eca-env-common.conf
  4. add this variable
    1. export EVA_WAIT_FOR_OPEN_VAULT_SEC=x       (x is seconds and can change to higher value than 15 second defaul)
  5. control + x to save and exit
  6. ecactl cluster down
  7. ecactl cluster up 



Security

Airgap Audit log 

  1. The Eyeglass VM audits the reachability of the vault cluster and logs this information to a dedicated log.
  2. This log can be found on Eyeglass below, the check is run every 5 minutes.   
  3. cat  /opt/superna/sca/logs/AirgapAudit.log 






Recovery Scenarios

Depending on the recovery requirements, the following three scenarios describe the high-level steps to gain access to the vault data, that are possible with the AirGap 2.0 solution.

Considerations 

  1. This procedures in this section should never be started until the all clear has been declared.  This would require all supporting infrastructure (AD, DNS etc..) with no threat present in the environment.  CSO or similar role in your organization should not request this procedure start until the environment has been cleaned of all active threats.   The risk that this final copy is comprised due to active threat still present in the environment.

Partial Vault Data Recovery Scenario 

  1. In this scenario, you require access to some of the data in the vault, due to an issue with the production data affecting a subset of the data protected by the vault.   The PowerScale vault makes this very simple.
    1. Method #1- Windows Explorer + SMB Share
      1. Physically connect the PowerScale management interface to the vault Ethernet switch management VLAN or port. 
      2. Connect to the PowerScale WebUI, and login.
      3. Enable the SMB protocol on the cluster. Protocols tab, enable SMB protocol.
      4. Create an SMB share on the path that stored the data you require.  Since there is no AD provider on the vault  cluster, a local user will be needed to authenticate to the SMB share.  The admin user can be used for this authentication.
      5. NOTE:  The data is locked by SyncIQ and immutable while connected to the network.
      6. Copy the data from the recovery SMB share using a Windows PC connected to the production cluster folder location.  Using Windows Explorer, copy the files or folders to the production cluster.
      7. Once the data restore is completed:
        1. Delete the SMB Share.
        2. Disable the SMB protocol.
        3. Disconnect the Management port Ethernet cable.
        4. Done.
    2. Method #2 - SCP in-band copy
      1. This method is best when a path of data needs to be restored and may require a long copy process. NOTE: SSH needs to be enabled on the vault cluster this may have been disabled with hardening steps.  This will prevent this method from being used.
      2. SSH to the production cluster.
      3. Open the AirGap - Use the Ransomware Defender CLI command to open the AigGap with the timed open command.  This command will open the AirGap for X minutes and will automatically close when the timer expires.
      4. Remotely copy data from the Vault PowerScale to the production PowerScale using the SCP command. Use the secure in-band replication network to copy data.
      5. This is syntax of the command that needs to be customized for your environment:
        1. scp -rp <user>@x.x.x.x:/ifs/data/yyy/* /ifs/data/ccc
        2. The user should be the admin.
        3. x.x.x.x is the ip address of the remote vault PowerScale replication pool IP address (pick any IP address in the pool).
        4. yyy - is the path on the remote vault PowerScale that contains the data to be restored.
        5. ccc - is the location on the production PowerScale where the data should be copied.
        6. NOTE: -rp  means recursive copy and -p preserves the date stamps on the files when copied.
      6. NOTE: The timed AirGap open command should be set long enough to ensure the copy completes.   Monitor the SCP command progress until it completes.  
      7. The AirGap will auto close after the timer expires.
      8. Done.


Complete Vault Data Recovery Scenario

  1. This scenario covers recovery of all the data on a production cluster protected by the AirGap SyncIQ policies.  This would be a worst-case scenario where the data in the Vault is determined to be the best copy of the data to be used for recovery.
  2. This scenario assumes that the production PowerScale itself is in a usable state, and simply needs data recovered to get back into an operational state.
  3. NOTE:  This is a recovery of last resort. This option should only be considered, if after a full evaluation of the data state is completed, it is determined the production data and its snapshots offer no restore option.   It is strongly suggested that you open a case with support, and open a Dell SR for a joint meeting to make sure this procedure should be used.
  4. Procedures to reverse Replicate Vault PowerScale data back to Production:
    1. Run the igls CLI command to disable AirGap SyncIQ policies from replicating.
    2. Run the igls CLI command to open the AirGap and open it for hours. (Estimate the time needed to complete reverse replication. It is best to use a large number of hours to ensure the copy can complete without the AirGap closing during the copy process. Example: 1000 hours used on the CLI command)
    3. Issue the SyncIQ CLI command for resync prep on the production cluster AirGap policies (repeat for each policy if you have more than one AirGap policy configured).  Consult Dell PowerScale documentation on the command syntax.  Verify the command completes successfully using the view jobs CLI command for SyncIQ.
    4. SSH to the vault PowerScale using an IP address of the remote vault PowerScale replication IP pool.   
    5. Issue the SyncIQ command to list the SyncIQ policies.  You should now see a <AirGap policy name>_mirror policy created by the Resync prep process (if you do not see any mirror policies, open a Dell SR for assistance with SyncIQ)
    6. The <AirGap policy name>_mirror can now be used to replicate the data in the vault back to the production cluster.  NOTE: This assumes the production PowerScale is fully operational and can serve the data as required, once the data is re-synced from the Vault PowerScale back to the production PowerScale. (Repeat this step for each AirGap policy)
      1. NOTE Data Impacting Steps:  Make sure you have confirmed all data protected by the AirGap policy needs to be restored. Their is no way back after starting a resync from the vault PowerScale to the production cluster, and resync Prep will make the data on the production cluster read-only blocking all IO to the data. 
      2. Run the <AirGap policy name>_mirror to reverse replicate the data from the vault PowerScale to the production cluster.  Use the cluster ISI cli view synciq command to monitor the progress of the copy job.
      3. Once the SyncIQ job(s) complete exit the SSH session with Vault PowerScale cluster.  You should now have an SSH session on the production cluster, verify by looking at the command prompt cluster name before you continue.
      4. Data SyncIQ Allow writes step is required to allow the data to be accessible to users, and applications since it is locked by SyncIQ.
        1. Issue the SyncIQ isi or OneFS GUI command to allow writes on the AirGap SyncIQ policy path(s).
          1. Repeat this step on all AirGap policies on the production cluster.
          2. This will mark the data as writable.
          3. If SMB shares are in place the data is now accessible and usable by end-users and applications.
          4. See Dell Documentation if you are unfamiliar with these commands or steps.
      5. Recovery of all fault data is now complete. Note: To reconfigure the AirGap policies to re-protect production data, open a Support case to get the best method to re-configure the AigGap policies.  It might be best to re-sync a full copy into the vault.

DR Vault Data Access Scenario - Rapid Recovery


  1. Unlike backup strategies that use a backup and restore workflow,  the Ransomware Defender AirGap 2.0 PowerScale Vault solution stands as the only rapid recovery solution.  The PowerScale Vault can vault and lock data in an immutable state and restore data with replication.   It is the only AirGap solution that can avoid data recovery completely by using the PowerScale as the file serving device.
    1. It is recommended to build a RunBook for the steps below based on your production cluster.  A well-documented recovery process would allow full data recovery and user/application access in less than 2 hours.  NOTE: It is possible to pre-configure most of the steps below to save time during a recovery.
    2. Benefits of Rapid Recovery
      1. Eliminates the data copy step by converting the vault into the file severing device.   This is the fastest possible recovery option available to customers, that need to get operational in the shortest possible time frame.
      2. If a DR license key is purchased Rapid Recovery allows syncing production shares, exports and quotas.
      3. Integrates file serving and vaulting in a single device.
      4. Supports partial recovery scenario with in-band or out of band recovery options.
      5. Supports immutable locked data.
      6. Supports data integrity during copy operations.
      7. Supports versioning of data that provides multiple recovery time periods.
  2. How to perform a Rapid Recovery
    1. Vault Cluster Physical Steps
      1. Connect Ethernet cables from each node to the production network, on the correct ports or VLAN's needed to reach production networks used by the production cluster.
    2. Configuration Data Sync Steps Eyeglass:
      1. SSH to Eyeglass and run the open AirGap command for 1 hour.
      2. Physically connect the vault PowerScale management interface to the network. 
      3. Login to the production protected cluster and disable the airgap SyncIQ policies
      4. In Eyeglass, add the vault PowerScale using the Add Network Element menu option (note this requires a DR license for the vault).
        1. Enter the management IP address, and supply the eyeglass user name and password. Wait for inventory to complete by viewing the Inventory icon.
      5. Open Data & Configuration Migration Icon in Eyeglass
        1. Configure a migration policy that matches the source path for each airgap policy and enter the target path used on the airgap policies if different from the source cluster.
        2. Complete the access zone and cluster selection to select the source production cluster and access zone and target vault cluster and access zone.  The access zone is where the configuration data is stored that should be copied to the vault.
        3. IMPORTANT:  Click the configuration only check box.  
        4.   
        5. Once ready to copy the configuration, Click the Submit button to start the copy function.
    3. Data Write Access Steps
      1. Data on the Vault PowerScale is locked by SyncIQ and will need to be unlocked to provide access to users and applications.
      2. Login to the OneFS management IP address connected in the previous steps.  Navigate to Data Protection SyncIQ menu, select the local targets tab, and select the More button and allow writes option on each of the AirGap policy paths.  This will allow the locked copy to become the writable copy of the data.
    4. Authentication Providers,  Access Zones, IP Pools 
      1. Verify all the Access Zones were created on the Access Zone tab in Onefs GUI
      2. Create all IP pools that existed on the production cluster
        1. Assign the SmartConnect names to each pool that existed on the production cluster.
        2. Assign nodes to each pool as required.
        3. Configure new IP ranges or re-use production cluster ip ranges (NOTE: this will require removing interfaces on the production cluster IP pools, so that IP addresses can be re-used on another cluster without IP address conflicts in the network).  Set other settings on the pool as needed.
        4. Edit each IP pool and assign the correct Access Zone to each IP pool and save the pool.
      3. DNS - Update DNS SmartConnect name delegations to point at the vault PowerScale subnet service IP.
      4. Active Directory - To speed up the SPN recovery steps, the fastest method is to delete the production cluster AD computer object that will remove all SPN's (Service Principal Names) from AD and the global Catalog. This is required to allow the Vault cluster to register all the SmartConnect name SPN's to its own AD computer object.
      5. Active Directory AD Providers - Add the AD provider(s) required for your AD configuration that was used on the production cluster.  AD providers cannot be added in advance since SPN registration conflict would occur with the production cluster.  This step must be done at this phase of recovery to simplify SPN bulk registration in AD.
        1. Add each AD provider with AD administrator credentials.  This will create the computer object and register all SmartConnect names currently configured on the IP pools.
        2. Add each Authentication provider to each Access Zone that exists on the vault cluster and set the order with AD provider listed first in the list for each Access Zone.
    5. Test Data Access
      1. At this phase of the recovery, sufficient cluster configuration has been completed to start testing SMB and NFS data access over IP and SmartConnect names.
      2. Attempt IP mount of an SMB share first before testing SmartConnect name access. Then test NFS mount via IP and then SmartConnect names.   It is likely debug efforts will be required before going into production.
      3. All teams should be involved  AD administrators, DNS administrators, Network Administrators, NAS administrators.
    6. Recovery Complete
      1. This configuration is temporary until a production cluster can be assessed for production use.  The Vault PowerScale is expected to operate at lower throughput levels and is designed to provide critical application recovery while planning the full recovery of production systems.
© Superna LLC