Administration Guides

Understanding Copy Performance Configurations

Home



Overview

This section should be reviewed when deciding how to configure which Golden Copy for copying files, and which IP pool interfaces will be used to copy files.   In addition, decisions on parallel copy influences the speed of copies and the bandwidth consumed for a copy job.  Golden Copy also supports multipart uploads and downloads. 

How to scale out and scale up copy performance

  1. Deploy Virtual Accelerator nodes to increase copy performance.  The configuration is 1 VM or 6 VM's.  The main Golden Copy VM is the control VM and the remaining 5 VM's are only used for copying.    All VM's must have an NFS mount to the source cluster.
    1. See the install guide to deploy VAN nodes here.
  2. Parallel threads can be increased to use more threads on multi part uploads.  The default is 10 threads.
    1. This can be changed following steps below.
    2. ssh to node 1 as ecaadmin
    3. nano /opt/superna/eca/eca-env-common.conf
    4. Add this tag to double the threads on large files.  This should only be used if the majority of files are over 1GB in size.
    5. export ARCHIVE_PARALLEL_THREAD_SDK=20
    6. control+x , answer yes to save
    7. ecactl cluster down 
    8. ecactl cluster up

How to Increase VM memory and Parallel copy threads on a Single Golden Copy VM

  1. Before power on, modify RAM and CPU to match above settings
    1. NOTE: > 4 folder definitions requires additional disk space to store file copy history for each folder. Additional 110 GB for 10 folders added
    2. disk latency read and write latency < 20 ms (test with command iostat -xyz -d 3)
  2. Modify the following file to expand the parallel file copies per VM
  3. nano /opt/superna/eca/eca-env-common.conf
  4. Add a line
    1. export ARCHIVE_PARALLEL_THREAD_COUNT=400
  5. control+x to save and exit
  6. Change memory configuration (note the the spacing must be Exactly as shown below, 2 spaces for the service name and parameters should be 4 spaces)
  7. nano /opt/superna/eca/docker-compose.overrides
version: '2.4'
services:
  indexworker:
    mem_limit: 8GB
    mem_reservation: 8GB
    memswap_limit: 8GB

  archiveworker:
    mem_limit: 8GB
    mem_reservation: 8GB
    memswap_limit: 8GB

  kafka:
    mem_limit: 4GB
    mem_reservation: 4GB
    memswap_limit: 4GB


How to create a Golden Copy Cluster > 6 nodes

  1. Boot all the Golden Copy OVA's and allow first boot to complete.  Any number of VM's can be configured up to 99 nodes​
  2. Login to node 1 and update the eca-env-common.conf
    1. nano /opt/superna/eca/eca-env-common.conf
    2. Set the value x to 1 minus 2 times the number of Golden copy nodes  , example if 12 nodes enter 22 (12-1 x 2=22)
    3. export ECA_GOLDENCOPY_ARCHIVEWORKER_PARTITIONS=x
    4. export ECA_INDEXWORKER_PARTITIONS=x
    5. add the node entries and ip addresses of each of the Golden Copy VM's
    6. export ECA_LOCATION_NODE_1=x.x.x.x
    7. export ECA_LOCATION_NODE_N=x.x.x.x   (enter an entry for all node in the cluster and increment N value)
    8. control+x to save and exit
  3. Bring up the cluster 
    1. ecactl components configure-nodes    (sets up keyless ssh on all nodes)
    2. ecactl cluster up
  4. Expand an existing cluster (impacts current copy jobs)
  5. Login to kafkahq
    1. https://x.x.x.x/kafkahq  (enter ecaadmin and default password 3y3gl4ss)
    2. click on topics
    3. Click trashcan icon for all topics (NOTE: This will delete all back log files queued for copying and new full job will be required on the expanded cluster)
    4. ecactl cluster down
    5. Boot additional Golden Copy OVA's to get the final cluster count
      1. Follow the steps from step #2 above
      2. Restart a full archive job after the cluster has been restarted.


Dell ECS Performance Copy Mode

  1. Golden Copy supports a load balance option with health check of ECS nodes + load balancing copies evenly to ECS nodes provided. 
  2. This will maximize performance and eliminates the requirement to use an external load balancer that adds cost to the overall solution.
  3. Use the --
© Superna Inc