Administration Guides

Eyeglass Search & Recover - Archive Solution with Dell EMC ECS from Linux Host

Home

Eyeglass Search & Recover - Archive Solution with Dell EMC ECS



Overview

This solution allows simple scripted upload of data into ECA using Search’s powerful searching and script creation feature to simplify the process of bulk copy or move operations into object storage.  The tested solution uses NFS to read the files and send over the S3 protocol to ECS storage bucket for archive.

Solution Test Environment

  • Eyeglass Search & Recover
  • Dell EMC PowerScale (Source)
  • Dell EMC ECS v3.3 (Archive)
  • s3curl

ECS Configuration

  1. Create a new test user: test1

(Manage →  Users → New Object Users)

  1. Next to Add password → Generate & Add Secret Key (S3)

Show Secret Key to get the key

  1. Create Bucket

Manage → Buckets → New Bucket

Set the Bucket Name, e.g. Bucket2

And set the owner → test1 user

S3curl installation

  1. ssh to ECA node 1
  2. sudo su -
  3. cd /tmp
  4. git clone https://github.com/EMCECS/s3curl.git
  5. Set the profile and credentials file, and change the mode of the file

cd $HOME

touch ~/.s3curl

chmod 600 ~/.s3curl

vi ~/.s3curl

Add these lines into the profile file - specify the userid and secret key

%awsSecretAccessKeys = (

 # ECS account

 ecsid => {

 id => 'test1',

 key => 'a4TMiLkD2S1CRu/vhmVnMrZv87vlOlakYWdq1Do7',

 },

);

push @endpoints , (

    '172.22.4.25',

);

Create PowerScale NFS mount

Create PowerScale NFS mount for the folder’s contents will be archived to ECS.

Configure the ECA-1 IP address as the root client.

Example of the path: /ifs/data/search2/folder101

Mount PowerScale NFS to ECA-1

This mount is used to read the data from PowerScale that will be archived.  The Search cluster node 1 can be used for the read and archive funtion

  1. ssh to ECA-1
  2. sudo su -
  3. cd /mnt
  4. mkdir folder101
  5. Mount. Example

mount -t nfs -o vers=3 172.22.4.166:/ifs/data/search2/folder101 /mnt/folder101

Create Scripts from Eyeglass Search & Recover to push to ECS and remove from PowerScale

The search used to locate files can be any search including content aware searches or age based search using last accessed or last modified data stamps on files or any combination.

  1. Login to Eyeglass Search & Recover
  2. Search the folder that the contents will be archive. Example: folder01
  3. Click the CMD Writer icon

  1. Define the Script content as follow:

cmd

 Select

cmd

./s3curl.pl --id=ecsid --put=

Full Path

 -- -s http://ecs-node-ip:9020/bucket-name/

Example:

cmd

 Select

cmd

./s3curl.pl --id=ecsid --put=

Full Path

 -- -s http://172.22.4.25:9020/bucket2/

  1. Click “Create For All” button
  2. Open and Modify the script use text editor tools to have the correct naming for the file location from the mount directory and the file name that will be placed in ECS bucket

Example:

Before modification :

#!/bin/sh

# Solr Query Summary:

# -Content: folder101

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file03.txt  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file005.xls  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file04.txt  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file08.txt  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file01.txt  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file02.txt  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file07.txt  -- -s http://172.22.4.25:9020/bucket2/

./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file006.xls  -- -s http://172.22.4.25:9020/bucket2/

After modification:

#!/bin/sh

# Solr Query Summary:

# -Content: folder101

./s3curl.pl --id=ecsid --put=/mnt/folder101/file03.txt  -- -s http://172.22.4.25:9020/bucket2/file03.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file005.xls  -- -s http://172.22.4.25:9020/bucket2/file005.xls

./s3curl.pl --id=ecsid --put=/mnt/folder101/file04.txt  -- -s http://172.22.4.25:9020/bucket2/file04.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file08.txt  -- -s http://172.22.4.25:9020/bucket2/file08.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file01.txt  -- -s http://172.22.4.25:9020/bucket2/file01.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file02.txt  -- -s http://172.22.4.25:9020/bucket2/file02.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file07.txt  -- -s http://172.22.4.25:9020/bucket2/file07.txt

./s3curl.pl --id=ecsid --put=/mnt/folder101/file006.xls  -- -s http://172.22.4.25:9020/bucket2/file006.xls

  1. Save the modified file and copy to the ECA-1 node under the same path of the s3curl command (example: /tmp/s3curl/archivescript1.sh)
  2. Change mode to executable

Chmod +x archivescript1.sh

  1. Execute the script

./archivescript1.sh

  1. To verify the content of the bucket, after file copy has been completed, run this command:

./s3curl.pl --id=ecsid -- -s http://172.22.4.25:9020/bucket2 | xmllint --format -

Example of the output:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

  <Name>bucket2</Name>

  <Prefix/>

  <Marker/>

  <MaxKeys>1000</MaxKeys>

  <IsTruncated>false</IsTruncated>

  <ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>

  <Contents>

    <Key>file005.xls</Key>

    <LastModified>2019-06-14T10:04:27.802Z</LastModified>

    <ETag>"3ce27fddbbf9a902dd7bbace9ffd5a1b"</ETag>

    <Size>247296</Size>

    <StorageClass>STANDARD</StorageClass>

    <Owner>

      <ID>test1</ID>

      <DisplayName>test1</DisplayName>

    </Owner>

  </Contents>

  <Contents>

    <Key>file01.txt</Key>

    <LastModified>2019-06-21T07:54:37.736Z</LastModified>

    <ETag>"cbc2d26585417e941c806eaf563a685b"</ETag>

    <Size>26</Size>

    <StorageClass>STANDARD</StorageClass>

    <Owner>

      <ID>test1</ID>

      <DisplayName>test1</DisplayName>

    </Owner>

  </Contents>

  <Contents>

    <Key>file02.txt</Key>

    <LastModified>2019-06-17T06:29:50.966Z</LastModified>

    <ETag>"a7bfc78c55cf0a6bffa3cc5f898aa050"</ETag>

    <Size>1481</Size>

    <StorageClass>STANDARD</StorageClass>

    <Owner>

      <ID>test1</ID>

      <DisplayName>test1</DisplayName>

    </Owner>

  </Contents>

  <Contents>

    <Key>file03.txt</Key>

    <LastModified>2019-06-21T08:39:17.591Z</LastModified>

    <ETag>"0bb572b08b954ee1b9047e603f2326b2"</ETag>

    <Size>69</Size>

    <StorageClass>STANDARD</StorageClass>

    <Owner>

      <ID>test1</ID>

      <DisplayName>test1</DisplayName>

    </Owner>

  </Contents>

  <Contents>

    <Key>file04.txt</Key>

    <LastModified>2019-06-21T08:39:18.558Z</LastModified>

    <ETag>"ca9f024b6bae43f9c05a0357f645c0c2"</ETag>

    <Size>2059</Size>

    <StorageClass>STANDARD</StorageClass>

    <Owner>

      <ID>test1</ID>

      <DisplayName>test1</DisplayName>

    </Owner>

  </Contents>

  <Contents>

    <Key>myfile.txt</Key>

    <LastModified>2019-06-14T09:38:30.256Z</LastModified>

    <ETag>"327922bccbb2249a049608ebda97197c"</ETag>

    <Size>11</Size>

    <StorageClass>STANDARD</StorageClass>

    <Owner>

      <ID>test1</ID>

      <DisplayName>test1</DisplayName>

    </Owner>

  </Contents>

</ListBucketResult>

  1. Verify that the files are copied to ECS bucket
  2. Create another script to remove the files from PowerScale NFS folder
  3. Repeat step#3, and define the command as follows:

cmd

Select

cmd

rm -f

Full Path

 

  1. Click “Create For All” button
  2. Open and Modify the script use text editor tools to have the correct naming for the file location from the mount directory

Example:

Before modification :

#!/bin/sh

# Solr Query Summary:

# -Content: folder101

rm -f  /ifs/data/search2/folder101/file03.txt  

rm -f  /ifs/data/search2/folder101/file005.xls  

rm -f  /ifs/data/search2/folder101/file04.txt  

rm -f  /ifs/data/search2/folder101/file08.txt  

rm -f  /ifs/data/search2/folder101/file01.txt  

rm -f  /ifs/data/search2/folder101/file02.txt  

rm -f  /ifs/data/search2/folder101/file07.txt  

rm -f  /ifs/data/search2/folder101/file006.xls  

After modification:

#!/bin/sh

# Solr Query Summary:

# -Content: folder101

rm -f  /mnt/folder101/file03.txt  

rm -f  /mnt/folder101/file005.xls  

rm -f  /mnt/folder101/file04.txt  

rm -f  /mnt/folder101/file08.txt  

rm -f  /mnt/folder101/file01.txt  

rm -f  /mnt/folder101/file02.txt  

rm -f  /mnt/folder101/file07.txt  

rm -f  /mnt/folder101/file006.xls  

  1. Copy the script to ECA node 1, change to executable and run it to remove file from PowerScale.
  2. Verify that file is no longer there.

© Superna LLC