Administration Guides

Advanced Cluster Configuration

Home

The clusters configuration can be customized from defaults.   The cluster can be shutdown, the main configuration file updated, and restart the cluster for the changes to take effect.


Compressed File Content Ingestion Handling

  1. Shutdown the cluster first.
    1. ecactl cluster down 
  2. Edit the file "/opt/superna/eca/eca-env-common.conf " 
    1. nano /opt/superna/eca/eca-env-common.conf  
  3. Add variable to set the size of compressed files that should be processed for full content indexing by decompressing the file, and then indexing the files inside the compressed file.
  4. Add the line below to change the size of the compressed files:
    1. export INGESTION_WORKER_MAX_COMPRESSED_FILE_SIZE_MB=x  (where x is a number in MB's,  (default value is 0.4 * INGESTION_WORKER_MAX_FILE_SIZE_MB)).
  5. Add the line below to control the compressed file types that will be processed:
    1. export INGESTION_WORKER_COMPRESSED_FILE_EXTENSIONS=.zip,.tar.gz .
  6. Save the file with changes control+x
  7. Start the cluster:
    1. ecactl cluster up 

How to configure faster Content indexing with Parallel Content Ingestion per Worker

  1. Follow these step to enable more CPU consumption to indexing more files per working in parallel .
  2. Shut down the cluster
    1. ecactl cluster down 
  3. Edit the file "/opt/superna/eca/eca-env-common.conf"
    1. nano /opt/superna/eca/eca-env-common.conf
  4. Add the line below to change the size of the compressed files:
    1. export INGESTION_WORKER_PARALLEL_LIMIT=10 (where x is a number of files to process at a time, default is 5 files))
  5. Save the file with changes control+x
  6. Start the cluster
    1. ecactl cluster up

How to Analyze Content Parsing Latency Distribution

  1. These steps will use a hidden attribute in the schema to return latency of parsing documents.  This will indicate if content indexing is processing longer or shorter time based on the reading and parsing of text in various document types.   Support can use this data to optimize indexing speed for content.
  2. Login to solr UI
    1. https://x.x.x.x/solr  (enter ecaadmin user and password)
    2. Click on the collection named IGLS on the left hand side
    3. Select the query menu option on the left hand side
    4. Fill in the query parameters as follows. See Screenshot below.
    5. In the q filed enter -0> contentindexedat:[* TO *] 
    6. in the start , rows enter 0 and 0 
    7. In the Raw Query Parameters --facet.range=contentparsetimer&facet.range.start=0&facet.range.gap=100&facet.range.end=30000&stats=true&stats.field=contentparsetimer
    8. Click the Facet check box to enable it.
    9. In the Facet Query enter --facet.range=contentparsetimer&facet.range.start=0&facet.range.gap=100&facet.range.end=30000&stats=true&stats.field=contentparsetimer
    10. Click the Execute Query button
    11. Results should display similar to the screen shot.    Copy this text response completely and post it in to a support case for analysis. 
    12.  
    13.  

Advanced Cluster Configuration 

Suppress PowerScale changelist mode - Use only if directed by support

  1. Search creates snapshots with a 5 day expiry. When we've disabled incremental ingestion, but are processing a long-running full ingestion, this snapshot can be deleted and we're left without one on disk. To avoid the issue, add an optional setting to "eca-env-common.conf" to disable incremental that will stop processing after the snapshot has been created, but before the changelist is created. This will keep creating snaps on the PowerScale, but will not process any changelist data.
  2. add to eca-env-common.conf  and requires a restart of the taskmaster container on node 1 .
  3. export SUPPRESS_CHANGELIST_TASK_CRON="0 0 * * *"  

Disable Query Index before saving new records to the Index

  1. This option disables the index check to verify if a file is already in the index.  This check will increment the skipped counter folder stat.  In some scenarios, this extra processing can be undesirable when you want to force index all data on a path into the index.  Use this variable to disable the index check before saving records to the database.
  2. on node 1 add this variable below and save the file
  3. nano /opt/superna/eca/eca-env-common.conf
  4. export SOLR_QUERY_BEFORE_SAVE=true
  5. control+x
  6. ecactl cluster down 
  7. ecactl cluster up


How to Increase HTTP API Timeout

  1. Use these steps when the cluster cannot return API requests under the default 30 seconds.  It is abnormal for a cluster to take longer than 30 seconds.  Only apply if directed by support.
  2. on node 1 add this variable below and save the file.   The value is in ms and this adds timeout of 5 minutes
  3. nano /opt/superna/eca/eca-env-common.conf
  4. export HTTP_TIMEOUT_MS=300000
  5. control+x  (save the changes)
  6. ecactl cluster push-config
  7. ecactl cluster services restart isilongateway
     


How to Increase Changelist job timeout for clusters with Performance issues and need more time to process a changelist job

  1. Login to node 1 as ecaadmin
  2. nano /opt/superna/eca/eca-env-common.conf
  3. add this variable . NOTE Default is 4 hours or 14400000 ms 
    1. export ISILON_CHANGELIST_REQUEST_TIMEOUT_MS=14400000
  4. control + x to save and exit
  5. ecactl cluster push-config
  6. ecactl cluster services restart isilongateway

How to enable fast incremental mode for high change rate file systems

  1. This feature will skip collection of metadata from the file system for each file in the change list which takes additional api latency to retrieve.  The change list contains the data stamps of the file but does not contain owner, group information.  This means any reports that depend on owner of the file will no longer work for incremental data in the index.  This feature will increase the performance of incremental indexing and still allows age based reporting to function normally.
  2. Requires a patch build of 1.1.8 , please open a case to request it.
  3. Login to node 1 as ecaadmin
  4. nano /opt/superna/eca/eca-env-common.conf
  5. add this variable.  
    1. export INGESTION_SMART_INCREMENTAL=true
  6. control + x to save and exit
  7. ecactl cluster push-config
  8. ecactl cluster services restart isilongateway



© Superna Inc