Eyeglass Search & Recover - Archive Solution with Dell EMC ECS from Linux Host
Eyeglass Search & Recover - Archive Solution with Dell EMC ECS
- Overview
- Solution Test Environment
- ECS Configuration
- S3curl installation
- Create PowerScale NFS mount
- Mount PowerScale NFS to ECA-1
- Create Scripts from Eyeglass Search & Recover to push to ECS and remove from PowerScale
Overview
This solution allows simple scripted upload of data into ECA using Search’s powerful searching and script creation feature to simplify the process of bulk copy or move operations into object storage. The tested solution uses NFS to read the files and send over the S3 protocol to ECS storage bucket for archive.
Solution Test Environment
- Eyeglass Search & Recover
- Dell EMC PowerScale (Source)
- Dell EMC ECS v3.3 (Archive)
- s3curl
ECS Configuration
- Create a new test user: test1
(Manage → Users → New Object Users)
- Next to Add password → Generate & Add Secret Key (S3)
Show Secret Key to get the key
- Create Bucket
Manage → Buckets → New Bucket
Set the Bucket Name, e.g. Bucket2
And set the owner → test1 user
S3curl installation
- ssh to ECA node 1
- sudo su -
- cd /tmp
- git clone https://github.com/EMCECS/s3curl.git
- Set the profile and credentials file, and change the mode of the file
cd $HOME
touch ~/.s3curl
chmod 600 ~/.s3curl
vi ~/.s3curl
Add these lines into the profile file - specify the userid and secret key
%awsSecretAccessKeys = (
# ECS account
ecsid => {
id => 'test1',
key => 'a4TMiLkD2S1CRu/vhmVnMrZv87vlOlakYWdq1Do7',
},
);
push @endpoints , (
'172.22.4.25',
);
Create PowerScale NFS mount
Create PowerScale NFS mount for the folder’s contents will be archived to ECS.
Configure the ECA-1 IP address as the root client.
Example of the path: /ifs/data/search2/folder101
Mount PowerScale NFS to ECA-1
This mount is used to read the data from PowerScale that will be archived. The Search cluster node 1 can be used for the read and archive funtion
- ssh to ECA-1
- sudo su -
- cd /mnt
- mkdir folder101
- Mount. Example
mount -t nfs -o vers=3 172.22.4.166:/ifs/data/search2/folder101 /mnt/folder101
Create Scripts from Eyeglass Search & Recover to push to ECS and remove from PowerScale
The search used to locate files can be any search including content aware searches or age based search using last accessed or last modified data stamps on files or any combination.
- Login to Eyeglass Search & Recover
- Search the folder that the contents will be archive. Example: folder01
- Click the CMD Writer icon
- Define the Script content as follow:
cmd | Select | cmd |
./s3curl.pl --id=ecsid --put= | Full Path | -- -s http://ecs-node-ip:9020/bucket-name/ |
Example:
cmd | Select | cmd |
./s3curl.pl --id=ecsid --put= | Full Path | -- -s http://172.22.4.25:9020/bucket2/ |
- Click “Create For All” button
- Open and Modify the script use text editor tools to have the correct naming for the file location from the mount directory and the file name that will be placed in ECS bucket
Example:
Before modification :
#!/bin/sh
# Solr Query Summary:
# -Content: folder101
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file03.txt -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file005.xls -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file04.txt -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file08.txt -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file01.txt -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file02.txt -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file07.txt -- -s http://172.22.4.25:9020/bucket2/
./s3curl.pl --id=ecsid --put= /ifs/data/search2/folder101/file006.xls -- -s http://172.22.4.25:9020/bucket2/
After modification:
#!/bin/sh
# Solr Query Summary:
# -Content: folder101
./s3curl.pl --id=ecsid --put=/mnt/folder101/file03.txt -- -s http://172.22.4.25:9020/bucket2/file03.txt
./s3curl.pl --id=ecsid --put=/mnt/folder101/file005.xls -- -s http://172.22.4.25:9020/bucket2/file005.xls
./s3curl.pl --id=ecsid --put=/mnt/folder101/file04.txt -- -s http://172.22.4.25:9020/bucket2/file04.txt
./s3curl.pl --id=ecsid --put=/mnt/folder101/file08.txt -- -s http://172.22.4.25:9020/bucket2/file08.txt
./s3curl.pl --id=ecsid --put=/mnt/folder101/file01.txt -- -s http://172.22.4.25:9020/bucket2/file01.txt
./s3curl.pl --id=ecsid --put=/mnt/folder101/file02.txt -- -s http://172.22.4.25:9020/bucket2/file02.txt
./s3curl.pl --id=ecsid --put=/mnt/folder101/file07.txt -- -s http://172.22.4.25:9020/bucket2/file07.txt
./s3curl.pl --id=ecsid --put=/mnt/folder101/file006.xls -- -s http://172.22.4.25:9020/bucket2/file006.xls
- Save the modified file and copy to the ECA-1 node under the same path of the s3curl command (example: /tmp/s3curl/archivescript1.sh)
- Change mode to executable
Chmod +x archivescript1.sh
- Execute the script
./archivescript1.sh
- To verify the content of the bucket, after file copy has been completed, run this command:
./s3curl.pl --id=ecsid -- -s http://172.22.4.25:9020/bucket2 | xmllint --format -
Example of the output:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>bucket2</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<ServerSideEncryptionEnabled>false</ServerSideEncryptionEnabled>
<Contents>
<Key>file005.xls</Key>
<LastModified>2019-06-14T10:04:27.802Z</LastModified>
<ETag>"3ce27fddbbf9a902dd7bbace9ffd5a1b"</ETag>
<Size>247296</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>test1</ID>
<DisplayName>test1</DisplayName>
</Owner>
</Contents>
<Contents>
<Key>file01.txt</Key>
<LastModified>2019-06-21T07:54:37.736Z</LastModified>
<ETag>"cbc2d26585417e941c806eaf563a685b"</ETag>
<Size>26</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>test1</ID>
<DisplayName>test1</DisplayName>
</Owner>
</Contents>
<Contents>
<Key>file02.txt</Key>
<LastModified>2019-06-17T06:29:50.966Z</LastModified>
<ETag>"a7bfc78c55cf0a6bffa3cc5f898aa050"</ETag>
<Size>1481</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>test1</ID>
<DisplayName>test1</DisplayName>
</Owner>
</Contents>
<Contents>
<Key>file03.txt</Key>
<LastModified>2019-06-21T08:39:17.591Z</LastModified>
<ETag>"0bb572b08b954ee1b9047e603f2326b2"</ETag>
<Size>69</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>test1</ID>
<DisplayName>test1</DisplayName>
</Owner>
</Contents>
<Contents>
<Key>file04.txt</Key>
<LastModified>2019-06-21T08:39:18.558Z</LastModified>
<ETag>"ca9f024b6bae43f9c05a0357f645c0c2"</ETag>
<Size>2059</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>test1</ID>
<DisplayName>test1</DisplayName>
</Owner>
</Contents>
<Contents>
<Key>myfile.txt</Key>
<LastModified>2019-06-14T09:38:30.256Z</LastModified>
<ETag>"327922bccbb2249a049608ebda97197c"</ETag>
<Size>11</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>test1</ID>
<DisplayName>test1</DisplayName>
</Owner>
</Contents>
</ListBucketResult>
- Verify that the files are copied to ECS bucket
- Create another script to remove the files from PowerScale NFS folder
- Repeat step#3, and define the command as follows:
cmd | Select | cmd |
rm -f | Full Path |
|
- Click “Create For All” button
- Open and Modify the script use text editor tools to have the correct naming for the file location from the mount directory
Example:
Before modification :
#!/bin/sh
# Solr Query Summary:
# -Content: folder101
rm -f /ifs/data/search2/folder101/file03.txt
rm -f /ifs/data/search2/folder101/file005.xls
rm -f /ifs/data/search2/folder101/file04.txt
rm -f /ifs/data/search2/folder101/file08.txt
rm -f /ifs/data/search2/folder101/file01.txt
rm -f /ifs/data/search2/folder101/file02.txt
rm -f /ifs/data/search2/folder101/file07.txt
rm -f /ifs/data/search2/folder101/file006.xls
After modification:
#!/bin/sh
# Solr Query Summary:
# -Content: folder101
rm -f /mnt/folder101/file03.txt
rm -f /mnt/folder101/file005.xls
rm -f /mnt/folder101/file04.txt
rm -f /mnt/folder101/file08.txt
rm -f /mnt/folder101/file01.txt
rm -f /mnt/folder101/file02.txt
rm -f /mnt/folder101/file07.txt
rm -f /mnt/folder101/file006.xls
- Copy the script to ECA node 1, change to executable and run it to remove file from PowerScale.
- Verify that file is no longer there.
© Superna Inc