Administration Guides

Golden Copy Back Bundle & Adv License Configuration Steps

Home


Overview

This topic covers installations licensed with the backup bundle or advanced license key features.  These features require the license key installed.


Golden Copy Advanced License And Backup Bundle Feature Configuration

Overview

These features require the Advanced license addon for Golden Copy to use the Backup use case features.  The backup features make managing backup workflows easier monitor and automated reporting.  New features to protect data allow several new workflows.   Data integrity features , API access for automation with external tools and new reporting options.  

Requirements

  1. Advanced Golden Copy license applied
  2. Golden Copy Backup Bundle license

How to Assign an Advanced License to a Cluster

  1. A Golden Copy Advanced license enables the features in this guide for a cluster managed by Golden Copy.  Use the steps below.
  2. searchctl isilons license --name <clusterName> --applications GCA
    1. NOTE:  An available advanced license must exist for this command to succeed and the cluster also requires a base Golden Copy license before an Advanced license key can be applied.



Cloud Storage Tier Aware Copy and Sync (AWS, Azure)

  1. This feature allows copying or syncing data directly into an archive tier for AWS S3 and Azure targets.  Google Cloud storage requires creating the bucket with a storage class and does not support setting the tier of individual objects.
    1. This feature avoids life cycle policies to move objects to different tiers.
  2. The add folder and modify folder CLI command allows specifying the tier that objects should be copied into.
    1. [--tier] default is standard (AWS), Cool (Azure)
      1. Requires Golden Copy Advanced license or Backup Bundle license
      2. Requires 1.1.6
      3. Azure
        1. flag to specify the tier the API calls are sent to, this should match the container tier configuration options are Access tier for Azure e.g. hot, cool, archive)  Without this flag the default is cold.
      4. AWS
        1. specify AWS tier using (STANDARD (default), STANDARD_IA, GLACIER, DEEP_ARCHIVE, INTELLIGENT_TIERING, ONEZONE_IA, OUTPOSTS, REDUCED_REDUNDANCY) Use upper case tier name.  
        2. NOTE: Not all tier options are valid for all use cases.  Consult AWS documentation.
  3. Example command
    1. searchctl archivedfolders add --isilon gcsource --folder /ifs/archive --secretkey  xxx --endpoint blob.core.windows.net --container gc1 --accesskey yyyy --cloudtype azure --tier STANDARD_IA 


Version Aware Recall / Restore

Overview
  1. Requires:
    1. Release 1.1.6
  2. Full and incremental with S3 bucket versioning allows multiple versions of files to be protected using S3 policies configured on the target storage. The Storage bucket must have versioning enabled and the folder should be configured in sync mode, or have run a copy job multiple times to detect file changes and update objects with a new version.  NOTE: Storage bucket version configuration is external to Golden Copy consult your S3 storage device documentation.
    1. This allows recall jobs to select files based on a date range using older than x date or newer than Y date.  This allows selecting files based object creation date (the date the backup ran) using the older newer than flags on the recall job. 
    2. NOTE: The date range is evaluated against the object creation date of the object in the version history of an object.  This date is when the object was backed up.
    3. NOTE: If you run multiple recall jobs with the same path the files in the recall staging area under /ifs/goldencopy/recall  will be overwritten if they already exist.
  3. This feature also adds the ability to scan the metadata in the objects properties to recall files based on created or modified data stamps of the files that existed on the file system at the time they were backed up.    
  4. The recall command adds to new options with the following date syntax. Use double quotes.
    1. --newer-than  "<date and time>" (yyyy-mm-dd HH:MM:SS e.g 2020-09-14 14:01:00)
    2. --older--than "<date and time>"  (yyyy-mm-dd HH:MM:SS e.g 2020-09-14 14:01:00)
  5. Use Case #1: Recall "hot data" first
    1. This solution is when a large recall of data is needed to be recalled /restored but you want the most recent data recalled first.  This would use the newer than flag to select a date example 2 weeks in the past.
    2. Example
      1. searchctl archivedfolders recall --id 3fd5f459aab4f84e --subdir /ifs/xxx --newer--than "2020-09-14 14:01:00" --apply-metadata
  6. Use Case #2: Recall "cold data" last
    1. This solution would be used to recall data after "hot data" is recalled since it has not been recently updated.  This would use a recall job and the older than flag , using the example above using the same date 2 weeks in the past with the older than flag would start a recall job to locate and recall data unmodified that is at least 2 weeks or older.
    2. Example
      1. searchctl archivedfolders recall --id 3fd5f459aab4f84e --subdir /ifs/xxx --older--than "2020-09-14 14:01:00" --apply-metadata 
  7. Use Case #3: Recall files with a specific created or modified time stamp
    1. This use case allows scanning the metadata that Golden Copy encodes into the properties of the objects as criteria to select data to recall the data based on the created date stamp of the files or modified time time stamp.
    2. [--start-time STARTTIME]  (yyyy-mm-dd HH:MM:SS e.g 2020-09-14 14:01:00)
    3. [--end-time ENDTIME] (yyyy-mm-dd HH:MM:SS e.g 2020-09-14 14:01:00) 
    4. [--timestamps-type {modified, created}]  The default is modified date stamp.  The files are backed up with created and modified time stamps, this flag allows selecting which time stamp to use when evaluating the older than or newer than dates.
    5. Example to scan for files with a last modified date stamp between Sept 14, 2020 and Sept 30 2020 under the /ifs/xxx folder.
      1. searchctl archivedfolders recall --id 3fd5f459aab4f84e --subdir /ifs/xxx --start-time "2020-09-14 14:01:00"  --end-time "2020-09-30 14:01:00" --apply-metadata 
    6. Example to scan for files with a created date date stamp between Sept 14, 2020 and Sept 30 2020 under the /ifs/xxx folder.
      1. searchctl archivedfolders recall --id 3fd5f459aab4f84e --subdir /ifs/xxx --start-time "2020-09-14 14:01:00"  --end-time "2020-09-30 14:01:00" --timestamps-type created --apply-metadata 

How to redirect recall object data to a different target cluster 

Overview
  1. Use this option to add a cluster to Golden copy that does not require a license, since it will be used as a recall target only.  This option requires the Advanced or Backup Bundle license key.
  1. Requires
    1. Release 1.1.6 
    2. Advanced license key or backup bundle license 
  2. This process requires adding the target cluster to Golden Copy, this cluster does not require a license when using the --goldencopy-recall-only flag. 
  3. searchctl isilons add --host IP address --user EyeglassSR [--isilon-ips x.x.x.x, y.y.y.y] --goldencopy-recall-only 
  4. Follow steps below:
    1. searchctl archivedfolders recall  --id ID  [--subdir SUBDIR] --target-cluster TARGETCLUSTER  --apply-metadata 
    2. NOTE:  The recall NFS mount must be created on the target cluster on the /ifs/goldencopy/recall path before a recall can be started.
    3. Replace TARGETCLUSTER with the redirected cluster name added above to redirect the restore to the new target cluster.


Target Object Store Stats Job to Monitor the backup object store

Overview

This job type will scan all the objects protected by a folder definition and will provide a count of objects, the sum total of data along with the the age of the objects. 

In addition it will summarize:

  1. File count
  2. Total data stored
  3. oldest file found
  4. newest file found
How to Configure Full Backup Mode Overview

Overview

This feature allows specifying  the number of full backup copies to maintain.  This feature must integrate with the target storage data retention feature that will automatically delete full backup copies after X days.   The feature will be applied to a folder and accepts the number of copies to maintain.  Each copy will be placed into a folder in the target storage with a date stamp with all data placed under this folder.   The folder definition includes a scheduled interval to make each full copy example weekly or monthly.     The data retention is configured on the target bucket to 2 x the retention setting to have a previous copy and current copy available for recall.

Example: full copies interval is weekly (every 7 days) with 2 full copies maintained at all times,  the bucket retention would be set to 15 days so that 2 full weekly copies will be retained at all times.  On day 15 the first copy will be deleted by the storage target.

How to configure full backup mode CLI commands
  1. When adding a folder a new parameter --backup-num  x  is added to indicate how many full copies this folder is going to be retained.   This flag will create a folder with a date stamp in the target storage device each time a full archive job is executed against this folder.
    1. Example --backup-num of 3  with a full archive schedule of weekly and S3 storage bucket retention set to 22 days (7 x 3 = 21 + 1 day).   The folder will need a schedule applied using the full archive schedule to weekly.    On the 22nd day the first backup taken will be deleted by the storage target.    On the 21st day 3 full copies of the data exist in 3 different folders with date stamps when each full backup job was started.
  2. The full archive schedule can be configured with the command below.  This example is every 7 days on Sunday at midnight.
    1. searchctl archivedfolders add (other parameters) --full-archive-schedule "0 0 * * 0"
  3. Make sure to set the object retention on the storage bucket used on the folder definition.  This is what will handle the deletes of object data.   Follow your vendors S3 documentation to set object retention features.
How to Report on the object count and quantity of data protected on a folder Definition
  1. searchctl archivedfolders s3stat --id <folderID>
    1. NOTE:  Usage charges for cloud provider storage will be assessed based on API list and get requests to objects.   The job can be canceled at any time using searchctl jobs cancel command.
    2. Use the searchctl jobs view --follow --id xxxxx (job ID) to view the results
  2. Sample output
    1.  



How to Run a Data Integrity Job

  1. Overview
    1. This feature provides a data integrity audit of folders by leveraging the metadata checksum custom property. Random files are selected for audit on the target device ,   downloaded the file computes the checksum and compares to the checksum stored in the metadata.   If any files fail the audit the job report will summarize failures and successful audit passing.   This verifies your target device stored the data correctly.
  2. Requirements
    1. Release 1.1.6 update 2 or later
    2. NOTE: If data did not have a checksum applied during a copy the job will return  100% error rate.
    3. This feature will require data is copied with the --checksum ON global flag enabled.   See the Global configuration settings.
      1. searchctl archivedfolders getConfig
      2. searchctl archivedfolders configure -h
  3. How to run a data integrity job
    1. searchctl archivedfolders audit --folderid yy --size  x  (note x is GB total of  data that will be randomly audited, and yy is the folder ID that will be audited)
    2. Using the job id to monitor the success of the job
      1. searchctl jobs view --id   xxxxx 
      2.   


How to Enable Ransomware Defender Smart Airgap

  1. This feature integrates with Ransomware Defender to enable Smart Airgap.  This blocks full or incremental jobs if an active alarm is raised in Ransomware Defender or Easy Auditor Active Auditor triggers.
  2. login to node 1 of Golden Copy as ecaadmin
  3. nano /opt/superna/eca/eca-env-common.conf
  4. Add these variables and generate a new API token in eyeglass to authenticate Golden Copy API calls.  This can be completed from Eyeglass GUI, main action menu, Eyeglass Rest API, API tokens and create a new new token named for Golden Copy.
  5. Add these variables and enter the eyeglass ip address and api token.
  6. export EYEGLASS_LOCATION=x.x.x.x
  7. export EYEGLASS_API_TOKEN=yyyy
    1. copy api token created in eyeglass api menu to replace yyyy
  8. export ARCHIVE_RSW_CHECK_THRESHOLD=WARNING
    1. Options are WARNING, MAJOR, CRITICAL to determine the severity of alarm that will block the backup process.  If set to warning then all severities will block,  if set to Major then Warnings will be ignored, if set to critical then warning and major will be ignored
  9. export ARCHIVE_RSW_CHECK_INTERVAL_MINS=1
    1. How often to poll eyeglass for ransomware events,  recommended to use 1 minute
  10. export ARCHIVE_RSW_CHECK_ENABLED=TRUE
    1. True / False to enable the functionality.  True required to activate the feature.
  11. control + x to save the file
  12. ecactl cluster down
  13. ecactl cluster up


Golden Copy Pipeline Workflow License Overview

Overview

This feature license allows S3 to file workflows and S3 to S3 work flows in addition to the File to Object, Object to File workflow available in Golden Copy.    The S3 to File direction require incremental detection feature that will leverage the date stamps on the file system set to match the object time stamp.  This will allow incremental sync from S3 to file or S3 to S3.

Use Cases

  1. Media workflows to pickup media contribution from a 3rd party from an Cloud S3 bucket and transfer the data to an on premise Powerscale for editing workflows
  2. Media workflow to download S3 output from a rendering farm that produces output that is needed on premise for video editing workflows.
  3. HPC cloud analysis for AI/ML that requires on premise data to be copied to an S3 Cloud bucket for analysis input to AI/ML that produces an output in a different bucket that needs to be copied back on premise.

These workflows are file to object and object to file with different source and destinations along with scheduled copies or incremental sync in both directions example on premise to cloud and cloud back on premise.

The solution is designed to allow scheduled incremental in both directions to pickup new or modified files only from the S3 bucket and copy to the cluster.

Requirements

  1. Pipeline license key applied to Golden Copy

Configuration Examples

  1. Use this command to add a folder to receive data from an S3 bucket
    1. searchctl archivedfolders add/modify --isilon HOST --folder PATH [--source-path SOURCEPATH] [--recall-schedule RECALLCRON]
      1. --source-path "SOURCEPATH" (enter the path with double quotes ")
      2. the path on s3 that contains the data we want to copy. Defaults to /<clustername>/<foldername>.
      3. --recall-schedule RECALLCRON Recall job Cron expression e.g. "*/1 * * * *"
  2. searchctl archivedfolders recall --id ID --source-path SOURCEPATH
    1. --id ID Id of archived folder
    2. --source-path SOURCEPATH the path on s3 that contains the data to copy.
  3. Example to create a pipeline configuration from an S3 bucket and path to a file system path on the cluster
    1. searchctl archivedfolders add --folder /ifs/notUploadedByGC --isilon ofs3830 --source-path "/notUploadedByGC" --recall-schedule "*/30 * * *" --cloudtype aws --bucket gcsoak2 --secretkey <> --accesskey xxxxx
      1.  --folder /ifs/notUploadedByGC   path where the data will be copied to on the cluster
      2.  --source-path "/notUploadedByGC"    - The S3 path to start the copy from in the bucket
      3. --recall-schedule "*/30 * * *"  - The schedule to scan the S3 bucket to copy new data or modified data found in the bucket.   This example is scanning the S3 bucket every 30 minutes
    2. After adding the folder you can run the job to scan the bucket and copy data
      1. searchctl archivedfolders recall --id <folderID> --sourcePath "/notUploadedByGC"



Automation with Golden Copy Rest API

Overview

The rest API can be used to automate copy jobs, monitoring of jobs to allow integration to application work flows that require data movement tasks from a file system to S3 or from S3 back to a file system.  Examples include media work flows, machine learning and AI training where Cloud computing is used with data and the results are returned to on premise file systems.


How to use the Golden Copy API








© Superna LLC