Software Releases

Current Release - Release Notes Golden Copy

Home



What’s New in Superna Eyeglass Golden Copy Release 1.1.4


What’s New! In Superna Eyeglass Golden Copy can be found here.


Supported OneFS releases

8.1.x.x

8.2.x.x

9.1.x.x


Supported S3 Protocol Targets

AWS S3 version 4 of the authentication is supported (details here)

Dell ECS version 2

Azure blob services using S3 version of the authenticated protocol

Cohesity 6.3.1e AWS version 4 signature (ask about other versions) See vendor documentation for versioning support and object retention poicy support.

OpenIO - versioning not tested. Requires --meta-prefix when adding folders and value of oo-

Ceph version 15 or later Octopus (aws v4 signature only)


End of Life Notifications


Description

End of Life Date

No End of Life Notification at this time 

New/Fixed in 1.1.4-20178

New: T15170 Option to add prefix to directory objects in S3 target

New option is available to specify a prefix to directory object in S3. This may be required for some targets where it is not supported to have an object which a name which is the same as a part of another object key.

Contact support.superna.net for assistance and recommendations prior to implementing this option.
—————————————————–

T16110 searchctl archivedfolders errors '--head' option is not working

The searchctl archivedfolders errors '--head' option results in error and cannot be used.

Resolution: The searchctl archivedfolders errors '--head' option is now working. 

—————————————————–

T16378 Redirected recall option not working

The option --golden-recall-only option to redirect recall to a different Isilon cluster is not working and not available in current release.

Resolution: The --golden-recall-only option is now available to redirect recall of files from S3 to a cluster that is different from the cluster where it was originally uploaded. 

—————————————————–

T16548 searchctl jobs view missing some stats for recall job

The searchctl jobs view command is missing progress information for Uploaded, Count and Errors when using to monitor a recall job.

Resolution: Upload, Count and Errors stats are now provided for recall job.

—————————————————–

T16559 Recall from AWS downloads files correctly but reports may show as failed

Recall from AWS and all folders are created successfully and files are recalled successfully but reports may report as failed.

Resolution: Reports no longer show failed for successful recall from AWS.

—————————————————–

T16928 Throughput Performance patch

File upload is single threaded which reduces performance.

Resolution: File upload now occurs in parallel.


New in 1.1.4-20178 - Testing Pending

New: T15450 Option to set Incremental and Full Archive schedule per folder

Option available to set or disable Incremental or Full Archive scheduled per archive folder. Related documentation can be found here.

—————————————————–

New: T15671 Version aware recall

This enhancement allows you to recall versions of files that are newer than or older than a specified date and time. Related documentation can be found here.

Pre-requisite: Folder must have been archived multiple times and S3 storage bucket must have versioning configured. Note that configuration of versioning on S3 storage bucket is external to Golden Copy.

—————————————————–

New: T16090 Recall restores original meta-data stored in S3 object properties

On recall of object from S3, file meta-data stored in the S3 object properties is applied to the downloaded file/directory.

—————————————————–

New: T16242 Authenticated login log

A log that makes available a record of authenticated login to Golden Copy.

—————————————————–

New: T16268 Adding archivedfolder for Azure provide accountid instead of bucket

When adding an archivedfolder for Azure, enter an accountid instead of a bucket.

—————————————————–

New: T16600 New stat RETRY_PENDING

New statistic to track files that errored on initial upload and an automatic re-run attempt is pending. If the upload succeeds on re-run, the success statistic for the uploaded file will be incremented.

—————————————————–

New: T16829 Option to skip upload of ACL list to S3 object properties

New option when starting an archive job to skp uploading of ACL list into S3 object properties.


Fixed in 1.1.4-20178 - Testing Pending

T15810 Archive job affected by external issues

Upload archive job under certain conditions external to Golden Copy such as persistent sustained networking issues, permission issues or DNS resolution issues will continue to process files for upload that result in errored uploads for those files. The job automatically resumes once the external condition has been resolved. In this case stats may not properly reflect failed files.

Resolution: External issue handling has been improved.

—————————————————–

T16129 Recall from Cohesity may fail where folder or file contain special characters

Recall of files or folders from Cohesity which contain special characters may fail. Job is started successfully but no files are recalled.

Resolution: Able to recall files or folders from Cohesity where name contains special characters.

—————————————————–

T16161 HTML Report issues for running jobs

HTML report is to be used to report on jobs that have completed and should be blocked for any job that is in running state. This has the following issues:

- if a folder has a running archive job, reporting on a previously completed job is incorrectly blocked

- reporting is not blocked for incremental, recall or rerun jobs - if run content will be inaccurate

Resolution: HTML report for archivefolder completed job while a job is active is not blocked. Report for running incremental, recall and rerun jobs is blocked.

—————————————————–

T16342 Stats for folder re-archive not accurate

If a archive job is restarted, the stats reported by the searchctl stats view command and the HTML report are not accurate.

Resolution: Statistics for folder re-archive are now accurate.

—————————————————–

T16369 ACL metadata with Unicode special characters not stored against S3 object for files or folders

File or folder which has ACL metadata such as group or owner that contains Unicode special characters will not be stored properly in the S3 object properties. Unicode characters are removed.

Resolution: ACL metadata wtih Unicode special characters now stored in S3 object properties.

—————————————————–

T16374, T16371 Cluster Down/Up loses running job ID

If you do a cluster down/up on Golden Copy while you have a running job, files remaining to be archived are archived but the related job is no longer active. Build 1.1.4-20121 job ID is lost and job cannot be tracked or reported on using Golden Copy reports or CLI commands. Later builds the job appears in the job history as cancelled (even though remaining files continue to be archived).

Resolution: Running jobs can now be tracked after cluster down/up of Golden Copy.

—————————————————–

T16420 Rerun option not available for recall job

If a recall job were to encounter errors, the rerun option is not available to only attempt recall for the errored files.

Resolution: Rerun option is now available for recall job.

—————————————————–

T16366 Able to start rerun job even if there are no errored files

You are allowed to start a rerun job when there are no files. Job runs and can be tracked in jobs view and jobs history despite no files being uploaded. Attempting to report on this rerun job will result in a stuck reporting job which has to be cancelled manually.

Resolution: Provide correct reporting and feedback when rerun job is attempted with no errored files.

—————————————————–

T16413 Recall to different folder requires overwrite option to be specified

If running a recall job to a folder which is different than the original folder using the --pathtarget option you must also specify --overwrite true in order for the recall to work.

IMPORTANT: It is not possible to recall without the overwrite option to a different folder currently. If there are existing files in the folder they will be overwritten with the version of the files from the S3 target.

Resolution: Able to now recall to a different folder without the overwrite option without overwriting existing files.

—————————————————–

T16432 Issues with Incremental Schedule after Upgrade to 1.1.4-20132 or higher

After upgrade from a release previous to 1.1.4-20132 there are the following issues related to incremental archiving:

  • existing incremental schedule is lost
  • Eyeglass Search & Recover incremental schedule is incorrectly configured to run hourly

Resolution: Upgrade now preserves existing schedule.

Eyeglass Search & Recover incremental schedule is incorrectly configured to run hourly: Disable Search & Recover incremental by running the command searchctl schedule disable --INCREMENTAL-INDEX

—————————————————–

T16517 Golden Copy backup includes encrypted Powerscale password and S3 accessKey and secretKey

Golden Copy backup includes files with encrypted Powerscale password or S3 accessKey and secretKey

Resolution: Backup no longer includes encrypted Powerscale password or S3 accessKey or secretKey.

—————————————————–

T16546 Statistics not updated for rerun job

The searchctl jobs view and searchctl stats view commands are not updated with counts for files uploaded as part of an auto-rerun or manually initiated rerun job.

Resolution: The searchctl jobs view and searchctl stats view commands are now updated with counts for files uploaded as part of an auto-rerun or manually initiated rerun job. 

—————————————————–

T16793 Rerun job can cause restarts

While a Rerun job is running this may cause restarts of some Golden Copy processes. The processes recover on their own and there are no missed files.

Resolution: ssh session management improvement to avoid restarts of any archive related job.

—————————————————–


New/Fixed in 1.1.4-20133

New: Manage incremental schedule using CLI command

New CLI command searchctl schedule now used manage incremental schedule. Related documentation can be found here.

Known Issue: T16432 Issues with Incremental Schedule after Upgrade to 1.1.4-20132 or higher

 —————————————————–

New: Increase snapshot expiry to 25 days

Powerscale snapshot that is created for an archive job now has a 25 day expiry.

 —————————————————– 

New: Enhanced scalability for rerun jobs

Scalability improvements for rerun job to re-upload errored files.

—————————————————–

T15372 Archive stats using searchctl archivedfolder stats command is missing stat for failed file

The searchctl archivedfolder stats command lists a failed file upload as an archived file upload.

Resolultion: searchctl archivedfolder stats command now lists failed file upload as ERRORED stat. 

—————————————————–

T15023 Recall start and end time option are based on upload time instead of object modified time

When using the start and end time options for recall, the time is compared against the object upload time instead of the object meta-data property for modified time.

Resolution: File modified date is now used to find files where start or end time parameter specified for recall.

—————————————————–

T16387 Error on recall deletes existing file when overwrite = true

If overwrite option enabled when recalling files, if there is an error on the recall, the existing file on the Powerscale will be deleted.

Resolution: Recall with overwrite enabled does not delete existing files if download fails.

—————————————————–

T16341 Folder list missing 'lastFullArchiveDate' and 'lastArchiveDate' information

The searchctl archivedfolders list output for 'lastFullArchiveDate' (full archive job) and 'lastArchiveDate' (incremental archive job) is not populated for the archive folders listed.

Resolution: searchctl archivedfolders list output now has 'lastFullArchiveDate' (full archive job) and 'lastArchiveDate' (incremental archive job) populated. 

—————————————————–

T16347 HTML Report for large file upload has incorrect information

The following fields in the HTML report do not report correct information for an archive job with large files:

Job Type

Average Upload Rate (files/min)

Average Upload Rate (MB/s)

End Time

Duration

Resolution: Report now shows correct values for above fields.

—————————————————–

T16349 List of Errored Files

List of errored files must be determined by examining multiple files generated after running a report. No identification of which file contains errored file list.

Resolution: In hte downloades folder the json file which contain errored files now contain the word Errors in the filename - example 2020_08_27_UTC_19_37_58-1-Errors.json.

—————————————————–

T16356, T16113 Stats for recall may not be accurate

The searchctl archivedfolder stats output for following counters for recalled files may not be accurate:

FULL/FILES_ARCHIVE_RECALLED

FULL/BYTES_ARCHIVE_RECALLED

FULL/FOLDERS_RECALL_ACCEPTED

FULL/FOLDERS_RECALL_METADATA

Resolution: Above statistics are now accurate.

—————————————————–

T16414 Cancelled job cannot be rerun

After cancelling a running job for archive or recall, resubmitting the job to run it again results in error.

Resolution: Cancelled job can now be resubmitted.

—————————————————–


New/Fixed in 1.1.4-20121

New: Recall from S3 - ability to recall all or some of the data from S3

Available in 1.1.4 is the ability to download archive folder or sub folder to Powerscale from Golden Copy.  In this release we provide the ability to download files in the archive folder or a subfolder thereof. 

Documentation can be found here.  

Known Issues can be found here.

Not available in this release:

  • Restore of original meta-data stored in S3 object properties. 
    • At this time the files themselves are recalled but original metadata stored against S3 object is not restored reliably and should not be used.  
  • Single File Recall
    • At this time individual files cannot be recalled. Recall must be done at the folder level.
  • Recall to a different directory
    • At this time recall of files to a different directory requires that the overwrite flag is specified and that parent folder exists - please refer to T16413
    • IMPORTANT: If there are existing files in the folder they will be overwritten with the version of the file from S3 target.
  • Recall to a different cluster
    • Not working in this release. Note no license is required for an alternate cluster for recall
  • Recall using a date range of data
    • Not fully working in this release.

Operational Notes

  • The recall option --skip-metadata-restore must be used when recalling files. If recall to an empty directory root permissions are applied thus securing files until appropriate permissions can be applied manually.


—————————————————–

New: beta GUI for Golden Copy Archive Monitoring

New GUI is available as beta in 1.1.4 for monitoring Golden Copy Archiving. 

Documentation can be found here.

Known Issues can be found here.

Operational Notes:

  • Isilon Management available for viewing only what has been configured using CLI - do not use to manage clusters
  • Archived Folder  available for viewing only what has been configured using CLI - do not use to manage archive folders

—————————————————–

T15401 Modify folder settings without specifing other values picks up defaults

For example if you were to modify --skip-s3-file-exists setting from false to true without specifying any other options in the modify command

searchctl archivedfolders modify --skip-s3-file-exists true --id <folder id>

settings for other options such as checksum would return to their default setting and you would lost customized settings for those options.

Resolution: Modifying folder settings now maintains customized settings.

—————————————————–

T15402 Files that failed to upload due to connectivity issue with PowerScale are not tracked

Summary reports do not contain a count or list of files that failed to upload.

Impact: Using re-run option will not be able to selectively upload those files.

Resolution: Report now includes failed files.

—————————————————–

T15827 Folder objects not storing expected Metadata for AD owned folders (ECS, Cohesity)

Folder objects may be missing metadata for Group, ACL, Mode (permission), Owner for ECS and Cohesity.

Resolution: Metadata now stored for folders.

—————————————————–

T15317, T15363 Delay in publishing reports

There may be a delay in publishing either interim or final reports so that they are available for review as html files. While the report data is not being updated, the html report for viewing progress will either be out of date or not available.

Resolution: Not applicable in this release which has a new architecture for generating reports.

—————————————————–


Technical Advisories

Technical Advisories for all products are available here.

Known Issues

Known Issues Archiving

T14014 Incremental upload requires 2 cycles before picking up changes

For incremental upload, changes are detected by comparing 2 snapshots. After enabling incremental or incremental on a newly added folder will required 2 incremental upload cycles to run and create 2 different point in time snapshots before changes will be detected.

Workaround: none required

—————————————————–

T14703 Loss of connectivity to one PowerScale node stops archiving on other available PowerScale nodes

If Golden Copy loses connectivity to one of the PowerScale nodes, all upload on nodes with connectivity also stops until the connectivity restored.

Workaround: None required. Once connectivity is restored upload continues after approximately 10 minutes.

—————————————————–

T15201 Upload jobs not marked as complete

Reporting under following scenarios does not recognize upload job as being completed:

- incremental upload jobs that have connectivity errors to PowerScale or S3 target 

- full archive jobs that have connectivity errors to PowerScale or S3 target

Impact:

-The job report naming includes "checkpoint" string which is used to identify interim reports and it may take 30 to 60 minutes for final summary reports to appear.

- Summary reports show job as running and still have the checkpoint string

- If Golden Copy archiveworker is restarted it will reupload files based on content of checkpoint files that may already have been successfully uploaded.

- checkpoint files continue to be updated

Workaround: File upload information in summary report is accurate. To determine whether a job is really still running ssh to the Golden Copy VM and execute the command:

searchctl jobs running

and check whether the Job ID from the report is listed as a running job.

—————————————————–

T15301 Archive jobs cannot run in parallel

Multiple archive jobs cannot be run in parallel.

Workaround: Run archive jobs in series. Do not start a new archive job while there is a running archive job.

—————————————————–

T15312 Archive job incorrectly presented as completed

Condition may occur where upload is still in progress but the searchctl jobs running command will not longer show the job as running even though uploading is still in progress.

Workaround: None required. The progress can be viewed in the logs. Final summary.html file once completed is correct.

—————————————————–

T16138, T16636 Job may incorrectly report as running

Jobs for some job types may incorrectly report as running when in fact they are completed

Workaround: For archive or rerun jobs, check S3 file count to confirm all files have been uploaded. For recall jobs check Isilon file count to confirm all files have been recalled.

Note that by design no incremental archive will run while you have a recall job in running state. You may cancel the job using the command searchctl jobs cancel --id <job id> . If you cannot cancel the job please contact support.superna.net for assistance.

—————————————————–

T16427 Incremental archive does not run with multiple Powerscale clusters added

When multiple Powerscale clusters are added to Golden Copy, incremental archive is blocked and does not run.

Workaround: None available.

—————————————————–

T16425 Archive incremental upload does not handle changelist in waiting state and incremental fails

A changelist on the Powerscale which is in waiting state is not handled by Golden Copy incremental archiving which fails the incremental job instead of waiting and applying a timeout.

Workaround: None required - following incremental interval will pick up the changes.

—————————————————–

T16629 Azure upload error where name contains %

Upload of file or folder with name that contains % character to Azure is not handled and will fail.

Workaround: None available.

—————————————————–

T16795 Full Archive scheduled job does not run

Full Archive scheduled jobs do not run on the configured schedule.

Workaround: Manually initiate a full archive job.

—————————————————–

T16849, T16852 Archive job stops unexpectedly

Running archive job stops uploading unexpectedly even though all associated processes are running.

Workaround: Contact support.superna.net for assistance.

—————————————————–

T16941 Running archive job stuck in running state after upgrade to 1.1.4-20178

If you have an archive job that is in running state when you upgrade to 1.1.4-20178, after running the upgrade the archive completes but is reported to be in running state. This running job cannot be cancelled and prevents a subsequent archive job for that folder. Statistics for the folder or job may report incorrectly. This running job also cannot be removed by deleting the folder or factory reset.

Workaround: Cancel running archive jobs before upgrade to 1.1.4-20178 and then restart the archive job after the upgrade is completed. Any files that were already uploaded will be skipped. Verify upload independently in S3 target.

—————————————————–

T16946 Incremental archive job does not run for all archive folders

With multiple archived folders added to Golden Copy, the global incremental schedule does not run against all folders and may run for 1 folder.

Workaround: None available




Known Issues Reporting


T15389 Summary html report shows incorrect endtime and duration when S3 target is not reachable

The Summary html report shows invalid endtime and duration when S3 target is not reachable for an archive job.

Workaround: None required. Duration, start and end time can be tracked using manual process.

—————————————————–

T16073, T16300 Beta GUI Known Issues

  • Archive Success (Bytes, All Time) always shows 100%
  • Isilon Management available for viewing only what has been configured using CLI - do not use to manage clusters
  • Archived Folder  available for viewing only what has been configured using CLI - do not use to manage archive folders

—————————————————–

T16088 HTML Report for file recall has incorrect information

The following fields in the HTML report do not report correct information for a recall job:

Total size archived

Average Upload Rate (MB/s)

Also related summary json files have similar issue with incorrect values for sizeGB and total size archived in GB and snapshot.

Workaround: The following commands can be used from the Golden Copy CLI to view job stats and job summary:

searchctl archivedfolders stats --id <folder id>

searchctl jobs view --id <job id>

—————————————————–

T15244 HTML Report may show incorrect information for completed job

In the HTML Report for a completed upload job, the Job Type may show as "undefined" and in the associated json files the job state incorrectly shows as running.

Workaround: None required.

—————————————————–

T16143 Foldername with Unicode special language characters display issue

Foldername with Unicode special characters can be added (with exception of !). The following have display issue for the folder name:

  • The HTML Report landing page with links to reports for archive folders does not display folder name correctly with Unicode special language characters.
  • searchctl archivedfolders archive response does not display foldername correctly

Workaround: None required. For HTML report, the link can be selected and the report when opened show the archive path correctly. Archive job is successfully started.

—————————————————–

T16450 searchctl stats view may not correctly report errored files

The searchctl stats view command may not correct report on errored files.

Workaround: The searchctl jobs view command can be used while the job is running or once the job is completed the exported HTML report can be used.

—————————————————–

T16631 HTML report export job status reported is incorrect

When job is completed to create the HTML summary report for upload or archive, the searchctl jobs view and searchctl jobs history commands show the job as failed even though the report has successfully been created. A report job that has been cancelled shows as success.

Workaround: Verify from downloads folder that report has been created.



Known Issues Recall

T16089 Sub-folder owner and group overwritten on recall when overwrite flag is set to false

This issue affects recall of a folder that contains sub-folders. If recall is done with overwrite flag set to false (it is false by default) file content and metadata is preserved, but sub-folder owner and group metadata is not preserved. Existing sub-folder owner and group are overwritten with the owner and group that was stored against the sub-folder S3 object. Sub-folder mode is not overwritten.

Workaround: None available

—————————————————–

T16550 Empty folder is recalled as a file

If you recall a folder that contains empty sub-folders, those sub-folders will be recalled as files instead of directories.

Workaround: If the empty directory is required on the file system is will need to be recreated manually.




Known Issues General & Administration

T13683 Global Rate Limit not visible from CLI

If a global rate limit is set using searchctl archivedfolders configure the limit that is set is not visible using CLI commands or as a parameter of the archived folder.

Workaround: ssh to Golden Copy and execute these 2 commands:

1. ecactl zk shell

2. get /ecasettings/PowerScale-clusters.json

The global rate limit will be shown at the top of the output as "limitRate"

—————————————————–

T14025 Changing PowerScale user requires a cluster down/up

If the user that was used when adding PowerScale to Golden Copy is changed, sessions still established with PowerScale using previous user.

Workaround: A cluster down/up is required to refresh user being used to connect to PowerScale. Contact support.superna.net for assistance.

—————————————————–

T15758 Unable to restore Golden Copy configuration to a new VM

The ecactl cluster restore command runs but does not actually restore any of the Golden Copy configuration to the new VM.

Workaround: Contact support.superna.net for assistance.

—————————————————–

T16466 Command for testing data copy configuration does not work for Cohesity target

The command searcharchivedfolders test used to do a test data copy and verify configuration does not work for the Cohesity target.

Workaround: On regular upload verify directly in S3 target the upload.

—————————————————–

T16640 searchctl schedules uses UTC time

When configuring schedule using searchctl schedules command time must be entered as UTC time.

Workaround: None required

—————————————————–

T16794 searchctl schedules enable command cannot be used to set schedules

Executing the command using options as per help completes without error but schedule is not set.

Workaround: Contact support.superna.net for assistance.

—————————————————–

T16824 Powerscale directory usage logging does not specify cluster

Logging of the space consumed in the /ifs/goldencopy/temp directory on the Powerscale cluster does not include the cluster name. For deployments with multiple clusters you do not know which cluster the logging is referring to.

Workaround: Quota should be applied to the Golden Copy directory on the Powerscale cluster as documented here. Use Powerscale tools to determine space consumed by the /ifs/goldencopy/temp directory.

—————————————————–

T16855 Archived Folders for Powerscale cluster added with the --goldencopy-recall-only option does not appear in the archivedfolders list command

The searchctl archivedfolders list command does not list folders for Powerscale clusters that were added to Golden Copy using the --goldencopy-recall-only option.

Workaround: Keep a record of the folder id after adding the folder and then it can be referenced in other commands such as searchctl archived folders remove .


Known Limitations

T15251 Upload from snapshot requires snapshot to be in Golden Copy Inventory

Golden Copy upload from an PowerScale snapshot requires snapshot to be in Golden Copy Inventory. Inventory task is run once a day. If you attempt to start archive without snapshot in inventory you will get error message "Incorrect snapshot provided".

Workaround: Wait for scheduled inventory to run or run inventory manually using command: searchctl PowerScales runinventory

—————————————————–

T15752 Cancel Job does not clear cached files for processing

Any files that were already cached for archive will still be archived once a job has been cancelled.

Workaround: None required. Once cached files are processed there is no further processing.

—————————————————–

T16429 Golden Copy Archiving rate fluctuates

Golden Copy archiving rates may fluctuate over the course of an upload or recall job.

Workaround: None required.

—————————————————–

T16628 Upgrade to 1.1.3 may result in second copy of files uploaded for Azure

In the Golden Copy 1.1.3 release, upload to Azure replaced any special characters in the cluster, file or folder names with "_". In 1.1.4 release the special characters are handled so that a subsequent upload in 1.1.4 wil re-upload any files/folders because the names are not identical in S3 to what was uploaded in 1.1.3. If the cluster name contained a special character - for example Isilon-1 - then all files will be re-uploaded.

Workaround: None

—————————————————–

HTML report cannot be exported twice for the same job

The HTML report cannot be run again after having been previously executed.

Workaround: None required. Use the previoTusly exported report.

—————————————————–

T16250 AWS accelerated mode is not supported

Golden Copy does not support adding AWS with accelerated mode as an S3 target.

—————————————————–

T16646 Golden Copy Job status

When viewing the status of a Golden Copy Job it is possible that a job whch has a status of SUCCESS contains errors in processing files. The job status is used to indicate the whether the job was able to run successfully. Then the searchctl jobs view or searchctl stats view or HTML report should be used to determine the details related to the execution of the job including errors and successes.

Copyright Superna LLC