Eyeglass Search & Recover - Archive Solution with Dell EMC ECS from Windows Host
Eyeglass Search & Recover - Scripted Archive Solution with Dell EMC ECS (s3cmd and SMB)
- Overview
- Solution Test Environment
- ECS Configuration
- s3cmd installation
- Create PowerScale SMB Share
- Create Scripts from Eyeglass Search & Recover for Push to ECS and Delete From PowerScale
Overview
This solution allows simple scripted upload of data into ECA using Search’s powerful searching and script creation feature to simplify the process of bulk copy or move operations into object storage. The tested solution uses SMB to read the files and send over the S3 protocol to ECS storage bucket for archive.
Solution Test Environment
-
Eyeglass Search & Recover
-
Dell EMC PowerScale (Source)
-
Dell EMC ECS v3.3 (Archive)
-
s3cmd (Windows platform)
ECS Configuration
-
Create a new test user: test1
(Manage → Users → New Object Users)
-
Next to Add password → Generate & Add Secret Key (S3)
Show Secret Key to get the key
-
Create Bucket
Manage → Buckets → New Bucket
Set the Bucket Name, e.g. Bucket2
And set the owner → test1 user
s3cmd installation
- On a Windows machine, install python (python 2.7 or higher version)
- Set the environment variable for the path to this python installation directory (e.g. c:\Python27)
- Download s3cmd from http://s3tools.org/download
- Extract to c:\s3cmd directory (e.g. c:\s3cmd-2.0.2). Change directory to this directory cd c:\s3cmd-2.0.2
- Configure s3cmd by using this command:
- python s3cmd --configure
- (it may prompt a warning about missing python-dateutil module. To install this python-dateutil module, change directory to python script directory (e.g cd c:\Python27\Scripts\ and then run pip install python-dateutil command to install that module. Once installed, re-run the s3cmd configuration command from c:\s3cmd-2.0.2 directory)
- Specify the following settings:
- Access Key: test1 (the S3 user that was created before)
- Secret Key: a4TMiLkD2S1CRu/vhmVnMrZv87vlOlakYWdq1Do7 (secret key for that s3 user)
- Default Region: US (accept default)
- S3 Endpoint: ecs25.ad1.test:9020 (Set the endpoint - DNS name /IP address of our ECS node)
- DNS-style bucket+hostname:port template for accessing a bucket:
- ecs25.ad1.test:9020 (set the bucket setting)
- Save the setting
- Example:
- c:\s3cmd-2.0.2>python s3cmd --configure
- ERROR: Option --preserve is not yet supported on MS Windows platform. Assuming --no-preserve.
- ERROR: Option --progress is not yet supported on MS Windows platform. Assuming --no-progress.
- Enter new values or accept defaults in brackets with Enter.
- Refer to user manual for detailed description of all options.
- Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
- Access Key: test1
- Secret Key: a4TMiLkD2S1CRu/vhmVnMrZv87vlOlakYWdq1Do7
- Default Region [US]:
- Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
- S3 Endpoint [s3.amazonaws.com]: ecs25.ad1.test:9020
- Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
- if the target S3 system supports dns based buckets.
- DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: ecs25.ad1.test:9020
- Encryption password is used to protect your files from reading
- by unauthorized persons while in transfer to S3
- Encryption password:
- Path to GPG program:
- When using secure HTTPS protocol all communication with Amazon S3
- servers is protected from 3rd party eavesdropping. This method is
- slower than plain HTTP, and can only be proxied with Python 2.7 or newer
- Use HTTPS protocol [Yes]: No
- On some networks all internet access must go through a HTTP proxy.
- Try setting it here if you can't connect to S3 directly
- HTTP Proxy server name:
- New settings:
- Access Key: test1
- Secret Key: a4TMiLkD2S1CRu/vhmVnMrZv87vlOlakYWdq1Do7
- Default Region: US
- S3 Endpoint: ecs25.ad1.test:9020
- DNS-style bucket+hostname:port template for accessing a bucket: ecs25.ad1.test:9020
- Encryption password:
- Path to GPG program: None
- Use HTTPS protocol: False
- HTTP Proxy server name:
- HTTP Proxy server port: 0
- Test access with supplied credentials? [Y/n] Y
- Please wait, attempting to list all buckets...
- Success. Your access key and secret key worked fine :-)
- Now verifying that encryption works...
- Not configured. Never mind.
- Save settings? [y/N] y
- Configuration saved to 'C:\Users\administrator\AppData\Roaming\s3cmd.ini'
- Verify s3cmd, use this command to list bucket that belongs to that user
- c:\s3cmd-2.0.2>python c:\s3cmd-2.0.2\s3cmd ls
- 2019-06-14 09:33 s3://bucket2
Create PowerScale SMB Share
-
Create PowerScale SMB sharefor the folder’s contents will be archived to ECS.
-
Example of the path: /ifs/data/search2/
-
And assign the user that will run the s3cmd command to have the permission to read (for upload) and for deleting file after copy to cloud, require read and write permission
Create Scripts from Eyeglass Search & Recover for Push to ECS and Delete From PowerScale
-
The search used to locate files can be any search including content aware searches or age based search using last accessed or last modified data stamps on files or any combination.
- Login to Eyeglass Search & Recover
- Search the folder that the contents will be archived. Example: folder1
- Click the CMD Writer icon
-
- Define the Script content as follows:
-
cmd
Select
cmd
Script Format
Surround file location with quotes
python c:\s3cmd-2.0.2\s3cmd put
File Location
s3://<bucket-name>
Plain
Checked
cmd
Select
cmd
Script Format
Surround file location with quotes
python c:\s3cmd-2.0.2\s3cmd put
File Location
s3://bucket2
Plain
Checked
- Example:
- Click “Create For All” button
- Open the script use text editor tools and save it as a batch file
- Example:
- # -Content: folder1
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\10.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\1.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\2.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\3.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\9.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\5.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\8.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\4.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\6.txt" s3://bucket2
- python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\7.txt" s3://bucket2
- Example of the output:
- c:\s3cmd-2.0.2>python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\3.txt" s3://bucket2
- WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
- upload: '\\rnsm04-c08.ad1.test\searchfolder2\folder1\3.txt' -> 's3://bucket2/3.txt' (15 bytes in 0.6 seconds, 23.40 B/s) [1 of 1]
- c:\s3cmd-2.0.2>python c:\s3cmd-2.0.2\s3cmd put "\\rnsm04-c08.ad1.test\searchfolder2\folder1\4.txt" s3://bucket2
- WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
- upload: '\\rnsm04-c08.ad1.test\searchfolder2\folder1\4.txt' -> 's3://bucket2/4.txt' (15 bytes in 1.2 seconds, 12.47 B/s) [1 of 1]
- To verify the content of the bucket, after file copy has been completed, run this command:
- Python c:\s3cmd-2.0.2\s3cmd ls s3:\\bucket2
- Example of the output:
- c:\s3cmd-2.0.2>python c:\s3cmd-2.0.2\s3cmd ls s3://bucket2
- 2019-06-27 06:34 15 s3://bucket2/1.txt
- 2019-06-27 07:34 15 s3://bucket2/2.txt
- 2019-06-27 07:54 15 s3://bucket2/3.txt
- 2019-06-27 07:54 15 s3://bucket2/4.txt
- Verify that the files are copied to ECS bucket
(Example:. dir \\rnsm04-c08.ad1.test\searchfolder2\folder1)
- Create another script to remove the files from PowerScale SMB share
- Repeat step#3, and define the command as follows:
-
cmd
Select
cmd
Script Format
Surround file location with quotes
del
File Location
Plain
Checked
- Click “Create For All” button
- Open the script use text editor tools and save it as a batch file
- Example:
- # -Content: folder1
- del "\\rnsm04-c08.ad1.test\searchfolder2\folder1\10.txt"
- del "\\rnsm04-c08.ad1.test\searchfolder2\folder1\1.txt"
- del "\\rnsm04-c08.ad1.test\searchfolder2\folder1\2.txt"
- del "\\rnsm04-c08.ad1.test\searchfolder2\folder1\9.txt"
- del "\\rnsm04-c08.ad1.test\searchfolder2\folder1\3.txt"
- del "\\rnsm04-c08.ad1.test\searchfolder2\folder1\5.txt"
- Run that batch file script to remove files from PowerScale
- Verify that files are no longer on PowerScale