AWS DR Solutions with Golden Copy

• Mountpoint for Amazon S3
• Description
• Installation
• Mounting
• Overview - Cloud Compute and Application DR - Read Write Hybrid Cloud Solution
• Requirements
• Limitations
• Topology
• Configuration
• Testing your Application DR Failover Process
• Summary and Conclusion
• Overview - How to Copy Golden Copy Data into AWS FSx SMB Service
• Requirements
• High Level Steps
• Golden Copy Steps
• AWS DR Application & S3 to FSx SMB Share Configuration Steps
• Active Directory Deployment Options in AWS
• Create FSx File system for the Application Server Data
• Application Server AMI in EC2
• AWS DataSync Service configuration
• Create a DataSync Task to sync S3 to FSx Filesystem Share
• Test DataSync Copy
• Test Mount FSx Share from the ECA Application Server VM
• Application DR Recovery Procedures
• How to Initiate DR Test or Actual DR of the application to AWS EC2 and FSx File System

Mountpoint for Amazon S3

Description

Mountpoint for Amazon S3 is an application that runs on our compute node, such as an Amazon EC2 instance, and mounts an S3 bucket (AWS S3 bucket used as Golden Copy target)  to a directory on that EC2 instance local file system.

Installation

Reference: https://github.com/awslabs/mountpoint-s3

The alpha release of Mountpoint for Amazon S3 is only available by building from source, and only supports Linux. Mountpoint for Amazon S3 is currently read-only.

The following steps are based on the Amazon Linux 2, but other Linux distributions should be similar.

1. Install dependencies, including the Rust compiler via rustup:

sudo yum install fuse fuse-devel cmake3 clang-devel

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

2. Clone this repository and its submodules:

git clone --recurse-submodules https://github.com/awslabs/mountpoint-s3.git

3. Compile the client:

cd mountpoint-s3

cargo build --release

4. The final binary will be at sub-directory:  target/release/mount-s3

Mounting

1. To use the client, first ensure we have access to valid AWS credentials. (Configure an EC2 instance with an IAM role, or configure by using aws configure)

2. Create the mount point (or use existing mount point) 

mkdir ~/my-mount-point

3. To mount, use mount-s3 command (The path of this  command is at sub-directory: target/release/mount-s3) my-gc-target-s3-bucket-name ⇒ specify the AWS S3 bucket used as the Golden Copy target

mount-s3 my-gc-target-s3-bucket-name ~/my-mount-point

Now we can access S3 bucket from that mount point

Overview - Cloud Compute and Application DR - Read Write Hybrid Cloud Solution

This solution allows a read and write copy of S3 data within AWS EC2.  The solution components requires S3 Fuse file system and is not intended for high performance read , write use case.  This solution allows mounting PB of S3 data for simple easy access to live synced Golden Copy on premise data.   The edits to the S3 data can be synced back to on premise Powerscale.

Requirements

1. Linux OS in EC2 instance
2. Install s3 fuse
3. Golden Copy licensed appliance with Advanced license key
4. The S3 bucket name used by Golden Copy and access and secret keys to gain read and write access to the bucket

Limitations

1. Performance of S3 mounted storage will not perform like a traditional file system.   A reasonable throughput is 35-40 MBs.
2. Limited file locking for concurrent access to files.

Topology

Configuration

Testing your Application DR Failover Process

1. Now you can test the failover process for your application.  This example will use text files to show how the solution works.
2. This is the Golden Copy target bucket listing from the EC2 instance that mounted the S3 bucket.
   1.  
   2. View of the custom metadata on the file cow.txt 
      1.  
   3. Now we will edit this file and add some text and save it from the EC2 instance.
      1.  
      2. The save operation by S3 Fuse preserved the metadata information on the file , used by Golden Copy
      3.  
   4. Create new file from the EC2 Instance
      1. nano ec2newfile.txt
      2. add some text to the file and save it  control+x
   5. Note that fuse adds its own custom properties, these will need be used by Golden Copy but will allow this new data to be recalled back to the Powerscale.
      1.  
   6. A recall job was started from Cloud Browser GUI and the below screenshot shows the new data created in AWS EC2 host is recalled back to Powerscale.
   7.  

Summary and Conclusion

1. This completes the end to end testing showing how Golden Copy is a key tool to build a robust hybrid cloud solution with on premise PowerScale and AWS S3 and EC2 Fuse mounted bucket allowing read-write access in the Cloud and full data recall to on premise storage

Overview - How to Copy Golden Copy Data into AWS FSx SMB Service 

This solution explains how Golden Copy Amazon S3 sync mode can be used for application specific DR recovery in AWS.  The solution uses S3 storage and the FSx SMB Share Service in AWS to allow recovery of one or more applications in AWS.  This solution will cover the high level steps needed to get the data prepared for an application server running in AWS.  There are many aspects to accessing and securing data within AWS that are not covered in this guide.  Consult with AWS documentation on aspects of running and accessing applications within AWS for your requirements.

Requirements

1. AWS S3 , FSx Configured Service 
2. PowerScale Onefs 8.1.x or 8.2.x
3. Golden Copy
4. This example is only covering within a single availability zone

High Level Steps

1. Sync Isilon file system data to Amazon S3 for applications that need DR
2. Deploy EC2 application instances that will be used for DR.  They can be left powered off.
3. Configure FSx SMB service in AWS to hold the SMB application data within AWS to be used by EC2 application VM
4. Configure Datasync service in AWS to sync S3 data to FSx SMB share, in the event of a disaster recovery scenario
5. Start EC2 application server to recovery your application in AWS.

Golden Copy Steps

These steps explain how to protect a single application's data stored on PowerScale with the goal of providing a DR recover option for the application in AWS.

1. Create Amazon S3 Bucket to store the application data.
   1. These steps are covered here.
   2. In this example the storage bucket created is GCDEMOSYSTEM.
   3. You will need the access and secret key to authenticate to the bucket.
2. Install Golden Copy and configure sync mode on an application folder to sync the path on the PowerScale where the application data is stored and is required for DR recovery.
   1. AWS example in the configuration guide to add a folder.
   2. Set an incremental schedule on the folder following guide here.
   3. Start the Sync job full sync to copy the application data to the blob storage in AWS.
      1. searchctl archivedfolders archive --id  (use searchctl archivedfolders list to get the folder ID for  /ifs/data/applicationdata) 
   4. You can verify the data is visible in the S3 bucket from the AWS Console
      1.  
3. Now move on to AWS DR steps.

AWS DR Application & S3 to FSx SMB Share Configuration Steps

1. Active Directory Deployment Options in AWS

   1. AWS Managed AD instance - guide here - This option requires the AWS AD to have a trust in place to the on premise Active Directory Domain to ensure on premise and cloud authentication is the same.

   2. 
      

      1. 

2. Create FSx File system for the Application Server Data

   1. The FSx File System is a managed SMB server in AWS that can present SMB shares to applications servers.  This File System can store data for multiple applications and have multiple SMB shares for each application.  This example guide only presents a single SMB share for 1 application. 

   2. Make sure to create the Windows FSx File System  large enough to store the application data.  The FSx file system must also be connected to Active Directory.  The disk can be extended to add space from the AWS console if required.

   3. Follow the FSx guide to configure a new file system.

   4. 
      

   5. Retrieve the FSx File System DNS end point for mounting Shares from Application servers or Management Servers
      1. Login to the FSx management page in the AWS console and record the DNS endpoint, ip address and power shell endpoint needed to mount SMB shares within AWS VPC.  See example below.
      2. 
   6. Create or Manage SMB shares in FSx File Systems
      1. To create or manage SMB shares in an FSX File System requires a management server in AWS to ensure access to manage the file system is available in a disaster.  
      2. Best Practice:  Create a management server to connect to FSX File Systems.   
      3. Follow the guide to use power shell or Windows Management server GUI tool 
      4. RDP to the management Windows server logged in as domain admin user that has permissions to the File System FSX shares.  Consult FSx documentation on the Managed AD domain group that allows management of FSx File Systems.
         1. open fsmgmt.msc tool to create or modify SMB shares and set permissions
         2. 
            
            

3. Application Server AMI in EC2

   1. Application virtual machines to mount the SMB shares. Your application server should be pre-staged in AWS EC2 and joined to Active Directory so that correct SMB share permissions will allow the application to mount the data.
      1. NOTE: The scope of this is outside this documents intended purpose, and may require multiple VM's and a resource familiar with AWS services and application server clone and migration to AWS should be consulted.

4. AWS DataSync Service configuration

   1. This service copies data from S3 AWS buckets and a specific path of data within the bucket to a destination FSx SMB share.   This copy configuration can be preconfigured in advance of a DR event.  The steps below cover some of the key steps needed to setup and configure the Datasync service.  To learn more about DataSync see the documentation page here.
   2. Deploy the Datasync Agent following this guide.  The Datasync EC2 instance should be deployed in the same region as your storage bucket and FSx file system.
   3. The next step involves creating an endpoint.  This is documented here.  Since the DataSync service is only copying data within AWS choose the VPC endpoint option.
   4. Continue to activate the agent following this procedure 

5. Create a DataSync Task to sync S3 to FSx Filesystem Share

   1. The documentation can be found here. A location is required for FSx and S3 bucket. Using the links below create a location in DataSync to authenticate to your FSX file share for your application, this requires an AD user with permissions to the FSx filesystem share.
   2. Creating a Location for Amazon FSx for Windows File Server
   3. Creating a Location for Amazon S3
   4. Create the task that defines the source (s3) to target FSX following steps here

6. Test DataSync Copy

   1. Perform a test copy to verify the task is setup correctly. Once the test is complete you can delete the data in the share. This requires a management VM running Windows to mount the share and verify the data was copied and also to delete the data.
   2. This is done by selecting the task and starting the job.
   3.  
   4.  

7. Test Mount FSx Share from the ECA Application Server VM

   1.  From a the Windows application server in EC2 mount the SMB Share Configured for this application server to verify the DataSync Job has copied the data successfully. See example below.  
   2.  
      1. 

Application DR Recovery Procedures

1. Overview: The process would be used in the event of a disaster or if you plan to exercise a DR test.  The high level steps below would be used to move an application to operate from within EC2 AWS VPC. These steps can take time to complete depending on the quantity of data in Amazon S3 that needs to be copied into the FSx file System.

2. How to Initiate DR Test or Actual DR of the application to AWS EC2 and FSx File System

   1. Open AWS DataSync console
   2. Start the application datasync task to sync data into the FSx Share location target.  This process can take time depending on how much data needs to be copied.  
   3. Monitor the DataSync task's copy performance and completion status on the console.
   4. Proceed to startup up the EC2 instance of the application server
   5. Login to the application server over RDP and verify the SMB mount.  Data should already appear in the mount as the copy progresses.
   6. Startup the application software once the DataSync task has completed.
   7. DR test completed.