• Overview
• Key Features
• Overview Video
• Dependencies & FAQ
• Performance Considerations
• Load Balancer Support
• Deployment Diagram
• Firewall Requirements
• Configuration
• How to Create ECSSync Jobs
• Airgap Operations
• Monitoring Vault ECS cluster alarms and free space
• Monitoring ECSsync job success and failures and policy sync job reports
• Monitoring Job History
• Vault Agent CLI Commands for ECS Sync
• Add ECS production & vault cluster to the vault agent
• Manage ECSSync jobs, schedules, run jobs, push support logs, update eyeglass
• Performance table for CAS files

Overview

Key Features

1. S3 to S3 Airgap support.
2. CAS to CAS Airgap future support.
3. Available with inside-the-vault automation.
4. Enterprise Airgap - Inside the vault hardened solution offers in band management and full automation from a VM within the cyber vault.
5. Leverages Smart Airgap technology to only sync data when it’s safe to replicate.
6. Per S3 bucket level replication.
7. Supports immutability with ECS object lock and bucket versioning.
8. Rapid recovery allows the vault ECS cluster to present an immutable copy of data at PB scale. The object lock feature keeps the object data safe from modifications in a recovery scenario.
9. Many to-one support for protecting multiple source ECS clusters to a single ECS Vault cluster.
10. Defender for ECS supports for IAM user lockout.

Overview Video

Dependencies & FAQ

1. Dell Server model PowerEdge R450   with vmware 7.x,   raid 1 SSD 800GB, 128GB RAM,  dual 10G nic, dual socket 48 threads, 128GB RAM (cart number 3000111987102) 

   1. 3 year support NBD OR

   2. pro-deploy plus

2. Vault ECS

   1. Sized to the quantity of data needed for vault protection

3. Potential Optional Equipment Needed

   1. Production network 

      1. Ethernet switch or Firewall to connect Vault Agent Host 

   2. Vault Network

      1. Ethernet switch to connect Vault Agent host to private network connected to the Vault ECS cluster.

4. Progress Kemp LoadMaster

   1. Installation Service Required

5. Dell Services

   1. Physical installation

      1. Dell Server host

      2. ECS Vault cluster + Hardening configuration applied

6. Superna Services

   1. Airgap Design and implementation Service

   2. Ransomware Defender for ECS Installation, Configuration and Knowledge Transfer  

7. Scalability

   1. 1, 3 or 6 Vault Agent VM’s in a cluster for high availability and high throughput object to object Aysnc copying

8. Superna Software

   1. Ransomware Defender for ECS

   2. Enterprise Airgap

 Performance Considerations

1. To get the most up to date performance estimates follow this link for ECSSync. https://github.com/EMCECS/ecs-sync/wiki/General-Performance-Metrics
2. Recommendation and best practice
   1. Use 5 job definitions per Airgap ECSSync instance for optimal performance.
   2. Possible throughput rates are listed on the link above per ECSSync instance and Enterprise Airgap can scale out as required from 3 to 99 instances.

Load Balancer Support

If there is a load balancer in front of your ECS, you can configure the IP of the load balancer during the Add Managed Device step. This will associate the IP of the load balancer with the ECS, and enable airgap jobs managed by the vault agent to be associated with the correct ECS cluster. 

Deployment Diagram

NOTE: The eyeglass appliance must have network connectivity to both the management and data networks on the ECS.

Firewall Requirements

Configuration

1. Follow the Ransomware Defender for ECS installation guide. 
2. Deploy the Dell Vault hardware with vmware pre-installed
3. Deploy Enterprise Vault Agent ECA cluster to the vault hardware.  See guide.
4. Configure firewall as per port table above.
5. Install Enterprise Airgap license key in Eyeglass using the License Manager Icon
6. Enter eyeglass IP address and API token on the vault agent
   1. Login to Eyeglass
      1. click main menu and select Eyeglass REST API
      2. create new token and name it vault
      3. copy the new token by clicking it to use with yyyyy value below 
   2. Login to the vault agent as ecaadmin
      1. nano /opt/superna/eca/eca-env-common.conf
      2. export EYEGLASS_LOCATION=x.x.x.x
      3. export EYEGLASS_API_TOKEN=yyyyyy
      4. Add the Object Services for Enterprise Airgap
      5. export ECS_SYNC_CFG=true
      6. control + x (save and exit)
      7. ecactl cluster down
      8. ecactl cluster up
7. Follow these steps to configure the Vault Agent
   1. Add ecs clusters to the vault agent
      1. ecactl ecs add --host x.x.x.x  --user  <user>
         1. ip of management interface and service account user (see the guide)
      2. repeat for each managed production and vault ECS cluster
      3. verify 
         1. ecactl ecs list
   2. OneTime ECS Sync instance configuration
      1. https://x.x.x.x/ecssyncui
         1. login with ecaadmin and default password 3y3gl4ss
         2. Click Config tab and enter the config path /opt/emc/ecs-sync/config and an email address (this email will not be used for any alerting but is a required input)
         3. Uncheck (automatically archive completed syncs) 
         4.  
   3. Configure ECS Sync Jobs
      1. https://x.x.x.x/ecssyncui  (each vault agent has an ecssync container,  enter the ip address of each instance to add jobs to the copy engine)
      2. login with ecaadmin and default password 3y3gl4ss
         1. NOTE:  Always change the default password following steps here.   
      3. See detailed steps below for adding jobs
      4. Save the jobs
   4. Add ECSSync Jobs to the vault agent to control them and push to Eyeglass
      1. ecactl ecssync listjobs  (to get list of configured jobs)
      2. ecactl ecssync addjob --job xxxx (enter the job name from the list command,  this will add the job to the managed job list)
   5. Push ECS job definitions to Eyeglass to register them
      1. This command will push all the jobs defined and added to the vault agent to be managed, monitored and scheduled.
      2. ecactl ecssync updatejobs
   6. Login to Eyeglass
      1. Open Airgap Icon
      2. Click ECS Sync Config Tab
      3. The new jobs should appear automatically and show status of Not Scheduled.
      4. Click the checkbox to enable Ransomware Defender smart airgap control and set the schedule for each ECS Sync job listed and click save after making changes to any policy.
      5.  
      6. Set the schedule and save
      7.  
      8. Repeat for each policy that displays not scheduled.
   7. Open the jobs icon to enable the job.  The default state is disabled and no sync jobs will run until enabled.
      1. 
         
   8. Verify Vault agent has detected the schedule change
      1. login to the vault agent
      2. ecactl ecssync schedules
      3. The schedule should be displayed for each policy configuration.
      4. Done
   9. Test a job sync - this will force run a job from the vault agent cli
      1. ecactl ecssync startjob  --job xxxx (where xxxx is the name of the policy configured in ecactl ecssync checkjobs command
      2. The job should indicate it was started,  You can verify from the ECSSync GUI
      3.  
      4. NOTE:  Do not archive jobs, they are managed by Vault agent.
      5. List all jobs

How to Create ECSSync Jobs

1. Login to each instance of ecssync on each Vault Agent vm
2. https://x.x.x.x/ecssyncui/  (each ECA vault vm hosts an instance of ecssync to load balance the workloads)
   1. Select the Status Tab and click the New Sync Button
   2. NOTE: the job name is mandatory and MUST be unique across all instances of ecssync
   3. NOTE: Select ECS S3 for source and destination.  See suggested parameter highlighted below.   
   4. NOTE: selecting the version check box will increase copy time significantly
   5. NOTE: Key prefix allows selecting a location within a bucket
   6. NOTE:  The source and target host ip address should specify the load balancer DNS name or ip address for the source production cluster and the load balancer in the secure network for the Vault ECS. 
3.    
4. To enable high performance multipart copies for large objects click the Hide Advanced button.  This will accelerate the copy with multiple threads per object.  Always recommended for larger objects. 
   1. 
      
5. Click the start button to save the configuration and start the copy process.
6. If the Vault is open the job will succeed and start the copy.  
7. If the Vault is closed the job will fail and can be scheduled by Eyeglass for running at a later time.

Airgap Operations

1. Monitoring Vault ECS cluster alarms and free space 

   1. Login into the eyeglass vm to review Alarms retrieved from the vault cluster

2. Monitoring ECSsync job success and failures and policy sync job reports

   1. Login to eyeglass and open the airgap icon and click the Airgap Reports
   2. Click the open button to view the reports.   The reports are also emailed.

3. Monitoring Job History

   1. Login to eyeglass and open the airgap icon and click the Job History
   2. Click on each job to view details

Vault Agent CLI Commands for ECS Sync

1. Add ECS production & vault cluster to the vault agent

   1. ecactl ECS add –host <host> –user <user>   
      1. --host management ip address of the ecs
      2. --user service account created on the ecs 
   2. ecactl ecs list
   3. ecactl ecs delete
   4. ecsctl ecs pushalerts
      1. Retrieves the alerts from all ecs registered and forwards it to eyeglass.

2. Manage ECSSync jobs, schedules, run jobs, push support logs, update eyeglass

   1. ecactl ecssync {startjob, checkjobs, pushvaultagentlogs, schedules, updatejobs)
      1. ecactl ecssync starjob --job xxx (xxx is the the name of the ecssync job)  This will start the copy job immediately on the ecssync that owns the job
      2. ecactl ecssync checkjobs - Retrieves the jobs from all ecssync instances and displays summary and status of the job
      3. ecactl ecssync schedules - Retrieves the schedules set in Eyeglass
      4. ecactl ecssync pushvaultagentlogs - push vault agent logs to eyeglass
      5. ecactl  ecssync updatejobs - push jobs created in ecssync GUI to eyeglass to be managed and scheduled.  

Performance table for CAS files

File count/size	Time	rate (size per second)	rate (object per second)	ECSsync number of threads allocated during the copy	CPU usage and memory usage on all containers in ECS sync during the copy	CPU usage of VM during the copy	Overall memory usage on the VM during the copy (used and free)	Number of CPU to copy the test data
787,095 files (8B each)	3h 23m 57s	73.42 KB/s	62 files/s	1	CPU % usage: ecssync 74.58% vaultagent 0.47% taskmaster 0.60% mariadb 34.35% zookeeper 0.19% nginx 0.00% dns 0.00% Memory % usage: ecssync 44.47% vaultagent 4,51% taskmaster 7.23% mariadb 17.97% nginx 0.50% dns 2.27%			1