Administration Guides

Limitations, Requirements and Supported Target List

Home


Firewall Port Requirements:

  1. SSH and HTTPS 8080 REST API from Golden Copy to PowerScale IP pool in the System Zone.
  2. S3 port 443 https to target S3 storage.
  3. NFS access from Golden Copy and Virtual Accelerator Nodes to Systems zone IP pool
  4. Full firewall port table and directions can be found here.

Product Limitations and S3 Target Requirements

  1. Data copied to the bucket using tools other than Golden Copy will be recalled but metadata will not be recalled for this data.   The data must exist under the Golden Copy path in the bucket  <cluster name>/ifs/xxxxx   if the data is copied under this object path it will be recalled without metadata.  (Requires 1.1.6 or later)
  2. Symlink files are skipped and will not be copied
  3. hardlink files are copied
  4. Fast Incremental performance requires OneFs 8.2.1 or later releases.  The change list API includes metadata and removes the requirement to query the file system for metadata of the files in the change list.
    1. Releases before 8.2.1 do not support change list + metadata, this requires additional query to the file system for each file in the change list this will impact performance of processing a large change list of files.
    2. Solution is to upgrade to 8.2.1 of Onefs
  5. Copying paths longer than 1001 characters due to REST API limitation on characters per request. This means files at folders below this level will not be copied to the S3 bucket the path is calculated /ifs/.snapshot/.... path to file and file name.   NOTE:  Windows SMB path length limit is 260 characters.
  6. Concurrent Running jobs - Releases above 1.1.4 21073 or later
    1. NOTE:  More concurrent jobs does not mean increased throughput.  Copy resources are shared across all running jobs.
    2. Number of folders configured for Incremental always scheduled concurrently is a maximum 80.  It is Not supported to exceed this concurrent folder incremental limit, contact support.
    3. In order to exceed 80 folders with incremental it is possible to offset additional folders to start on different hours to stay within the 80 concurrent limit.  example every 4 hours start 5 folder incremental and the next 4 hour interval start 5 different folders.
    4. Full copy concurrent testing limit of 80 jobs but only 10 jobs execute at a time with resources shared across all jobs that are running
  7. Report creation is limited to 15 M success files but can be extended by adding disk to the vm to support exporting copy job reports in json format to any file count required by allocating more disk space for logging successful file copies.
  8. Concurrent running jobs display is limited to 10 active running jobs in the CLI any job submitted after 10 will not be displayed but it will be queued once one of the first 10 jobs completes,  queued jobs will appear as running.
  9. Largest Tested  single file multipart upload is 1TB  file.   No architectural limit blocks larger files other than the S3 target max object size.
  10. Directories with no files stored in them will not be copied to the object store as an empty directory.  The directory will not be visible in the object store.
  11. ADS or Alternate data stream files will not have ADS streams copied
  12. Characters supported in file and directory names as per Amazon S3 specification listed here.
    1. Tested special character list for files and folders
      1. Safe Characters:  ~ ` ! @ # $ % ^ & () _ + - = [] {} : ; ' , . \ : * ? " < > | 
      2. Windows does not support these characters in file or folder names \ / : * ? " < > | 
      3. Azure does not support:   \ %  
      4. Language characters have been tested successfully for non ascii characters in file names or folders.  Not all languages or characters have been tested.
    2. Not supported
      1. All other special characters not listed above
  13. Connecting to S3 targets on the Internet requires Golden Copy IP addresses to be NATed to the Internet. This applies to AWS and Azure or any other Internet S3 target.
  14. S3 Targets must support custom object properties for protecting meta data of the files and folders.
  15. Isilon or Powerscale any folder can be copied including snapshots.  In addition the DR cluster that is read-only due to synciq can also be used as a source of a folder copy.
  16. Recall  Cluster Targets
    1. Golden Copy Basic license 1 recall target configured at a one time.
    2. Golden Copy Advanced or Backup Bundle License recall target limit of 5 with the ability to select a target during a recall job creation

S3 Target Device Specific Feature Support

  1. Dell ECS

    1. Compliance mode is supported
    2. Bucket Versions are supported
    3. Retention policies and versions at the bucket level in a combined policy is not supported by ECS
    4. Application Aware meta data content type encoding
    5. Checksum support
    6. Supports a load balance option with health check of ECS nodes + load balancing copies evenly to ECS nodes provided.  Typically has higher performance than using a load balancer.
  2. Amazon Amazon S3

    1. Snowball Edge
    2. AWS auth v4
    3. Transfer Acceleration Buckets. (planned future release)
    4. Client Side KMS keys - not supported - TLS inflight encryption for inflight security only.
    5. Retention policies
    6. Access and Security authentication only
    7. Application Aware meta data content type encoding
    8. Checksum support
    9. Multipart upload and download 
    10. Object Lock support
  3. Microsoft Azure Blob 

    1. Blob storage REST API
    2. Data Box
    3. Retention policies
    4. checksum support
    5. Multipart uploads
    6. Does Not support multipart downloads
    7. No content type encoding support binary/octet used for all files
    8. Object lock supported release 1.1.6 or later.
  4. Google Cloud Storage 

    1. Native SDK API
    2. Checksum support
    3. Multipart upload and download
    4. all metadata options supported
  5. Cohesity

    1. See vendor documentation for versioning support, and object retention policy support.
    2. NOTE: Does not support load balancing feature in Golden Copy due to no support for multi part copy of chunks load balanced across Cohesity nodes.
    3. checksum support
    4. Application Aware meta data content type encoding
    5. Checksum support
  6. OpenIO

    1. Versioning not tested
    2. Requires --meta-prefix when adding folders and value of oo- .
    3. checksum support
    4. Application Aware meta data content type encoding
    5. Checksum support
  7. Ceph version 15 or later Octopus 

    1. aws v4 signature only
    2. checksum support
    3. Application Aware meta data content type encoding
    4. Checksum support
  8. MinIO https://min.io/

    1. aws v4 signature only
    2. Requires --meta-prefix xxx (where xxx is a number to identify folder objects separately from object key paths)
    3. cloud type other must be used
    4. checksum support
    5. Application Aware meta data content type encoding
    6. Checksum support
  9. Scality Open Source 

    1. Versioning not tested
    2. cloud type other must be used
    3. checksum supported
    4. example add command
      1. searchctl archivedfolders add --isilon gcsource --folder /ifs/archive --accesskey accessKey1 --secretkey verySecretKey1 --endpoint http://x.x.x.x:8000 --region us-east-1 --bucket bucket --cloudtype other
        create a folder which will become the bucket name when created at the root of the target device.
    5. Application Aware meta data content type encoding
    6. Checksum support
  10. Wasabi

    1. Versioning tested
    2. Checksum support
    3. cloud type other
    4. full permissions to the bucket are required
    5. example command
      1. searchctl archivedfolders add --isilon gcsource --folder /ifs/archive --accesskey --secretkey --endpoint https://s3.wasabisys.com --bucket gctest --cloudtype other
  11. IBM Cloud Object Storage

    1. Versioning tested
    2. Checksum support
    3. cloud type other
    4. example command
      1. searchctl archivedfolders add --isilon gcsource --folder /ifs/archive --accesskey --secretkey --endpoint https://<endpoint> --bucket gctest --cloudtype other
  12. BackBlaze S3

    1. Versioning tested
    2. Checksum support
    3. cloud type other
    4. example command
      1. searchctl archivedfolders add --isilon gcsource --folder /ifs/archive --accesskey --secretkey --endpoint https://<endpoint> --bucket gctest --cloudtype other
    1. Versioning tested
    2. Checksum support
    3. Multi part upload
    4. cloud type AWS
    5. note: bucket url must resolve to the end point ip address using url access method.  example gctest.s3-us-east.internal.superna.net  must resolve in DNS on the Golden Copy VM's
    6. Example Command
      1. searchctl archivedfolders add --isilon gcsource --folder /ifs/archive --accesskey --secretkey  --endpoint https://s3-us-east.internal.superna.net --bucket gctest --region us-east --cloudtype aws
© Superna LLC