Administration Guides

Eyeglass Search & Recover - Archive Solution with Dell EMC ECS from Linux Host

Home

Eyeglass Search & Recover - Archive Solution with Dell EMC ECS

Overview

This solution allows simple scripted upload of data into ECA using Search’s powerful searching and script creation feature to simplify the process of bulk copy or move operations into object storage. The tested solution uses NFS to read the files and send over the S3 protocol to ECS storage bucket for archive.

Solution Test Environment

Eyeglass Search & Recover
Dell EMC PowerScale (Source)
Dell EMC ECS v3.3 (Archive)
s3curl

ECS Configuration

Create a new test user: test1

(Manage → Users → New Object Users)

Next to Add password → Generate & Add Secret Key (S3)

Show Secret Key to get the key

Create Bucket

Manage → Buckets → New Bucket

Set the Bucket Name, e.g. Bucket2

And set the owner → test1 user

S3curl installation

ssh to ECA node 1
sudo su -
cd /tmp
git clone https://github.com/EMCECS/s3curl.git
Set the profile and credentials file, and change the mode of the file

cd $HOME

touch ~/.s3curl

chmod 600 ~/.s3curl

vi ~/.s3curl

Add these lines into the profile file - specify the userid and secret key

%awsSecretAccessKeys = (

# ECS account

ecsid => {

id => 'test1',

key => 'a4TMiLkD2S1CRu/vhmVnMrZv87vlOlakYWdq1Do7',

);

push @endpoints , (

'172.22.4.25',

);

Create PowerScale NFS mount

Create PowerScale NFS mount for the folder’s contents will be archived to ECS.

Configure the ECA-1 IP address as the root client.

Example of the path: /ifs/data/search2/folder101

Mount PowerScale NFS to ECA-1

This mount is used to read the data from PowerScale that will be archived. The Search cluster node 1 can be used for the read and archive funtion

ssh to ECA-1
sudo su -
cd /mnt
mkdir folder101
Mount. Example

mount -t nfs -o vers=3 172.22.4.166:/ifs/data/search2/folder101 /mnt/folder101

Create Scripts from Eyeglass Search & Recover to push to ECS and remove from PowerScale

The search used to locate files can be any search including content aware searches or age based search using last accessed or last modified data stamps on files or any combination.

Login to Eyeglass Search & Recover
Search the folder that the contents will be archive. Example: folder01
Click the CMD Writer icon

Define the Script content as follow:

cmd	Select	cmd
./s3curl.pl --id=ecsid --put=	Full Path	-- -s http://ecs-node-ip:9020/bucket-name/

Example:

cmd	Select	cmd
./s3curl.pl --id=ecsid --put=	Full Path	-- -s http://172.22.4.25:9020/bucket2/

Click “Create For All” button
Open and Modify the script use text editor tools to have the correct naming for the file location from the mount directory and the file name that will be placed in ECS bucket

Example:

Before modification :

#!/bin/sh

# Solr Query Summary:

# -Content: folder101

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file03.txt -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file005.xls -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file04.txt -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file08.txt -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file01.txt -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file02.txt -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file07.txt -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file006.xls -- -s http://172.22.4.25:9020/bucket2/

After modification:

#!/bin/sh

# Solr Query Summary:

# -Content: folder101

./s3curl.pl --id=ecsid --put=/mnt/folder101/file03.txt -- -s http://172.22.4.25:9020/bucket2/file03.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file005.xls -- -s http://172.22.4.25:9020/bucket2/file005.xls

./s3curl.pl --id=ecsid --put=/mnt/folder101/file04.txt -- -s http://172.22.4.25:9020/bucket2/file04.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file08.txt -- -s http://172.22.4.25:9020/bucket2/file08.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file01.txt -- -s http://172.22.4.25:9020/bucket2/file01.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file02.txt -- -s http://172.22.4.25:9020/bucket2/file02.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file07.txt -- -s http://172.22.4.25:9020/bucket2/file07.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file006.xls -- -s http://172.22.4.25:9020/bucket2/file006.xls

Save the modified file and copy to the ECA-1 node under the same path of the s3curl command (example: /tmp/s3curl/archivescript1.sh)
Change mode to executable

Chmod +x archivescript1.sh

Execute the script

./archivescript1.sh

To verify the content of the bucket, after file copy has been completed, run this command:

./s3curl.pl --id=ecsid -- -s http://172.22.4.25:9020/bucket2 | xmllint --format -

Example of the output:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<Name>bucket2</Name>

<IsTruncated>false</IsTruncated>

<ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>

<ETag>"3ce27fddbbf9a902dd7bbace9ffd5a1b"</ETag>