Administration Guides

Advanced Cluster Configuration

Home

The clusters configuration can be customized from defaults.   The cluster can be shutdown, the main configuration file updated, and restart the cluster for the changes to take effect.


Compressed File Content Ingestion Handling

  1. Shutdown the cluster first.
    1. ecactl cluster down .
  2. Edit the file "/opt/superna/eca/eca-env-common.conf ". 
    1. vim /opt/superna/eca/eca-env-common.conf  .
  3. Add variable to set the size of compressed files that should be processed for full content indexing by decompressing the file, and then indexing the files inside the compressed file.
  4. Add the line below to change the size of the compressed files:
    1. export INGESTION_WORKER_MAX_COMPRESSED_FILE_SIZE_MB=x  (where x is a number in MB's,  (default value is 0.4 * INGESTION_WORKER_MAX_FILE_SIZE_MB)).
  5. Add the line below to control the compressed file types that will be processed:
    1. export INGESTION_WORKER_COMPRESSED_FILE_EXTENSIONS=.zip,.tar.gz .
  6. Save the file with changes :wq .
  7. Start the cluster:
    1. ecactl cluster up .

How to configure faster Content indexing with Parallel Content Ingestion per Worker

  1. Follow these step to enable more CPU consumption to indexing more files per working in parallel .
  2. Shut down the cluster
    1. ecactl cluster down .
  3. Edit the file "/opt/superna/eca/eca-env-common.conf"
    1. vim /opt/superna/eca/eca-env-common.conf
  4. Add variable to set the size of compressed files that should be processed for full content indexing by decompressing the file, and then indexing the files inside the compressed file.
  5. Add the line below to change the size of the compressed files:
    1. export INGESTION_WORKER_PARALLEL_LIMIT=x (where x is a number of files to process at a time, default is 5 files))
  6. Save the file with changes :wq
  7. Start the cluster
    1. ecactl cluster up

How to Analyze Content Parsing Latency Distribution

  1. These steps will use a hidden attribute in the schema to return latency of parsing documents.  This will indicate if content indexing is processing longer or shorter time based on the reading and parsing of text in various document types.   Support can use this data to optimize indexing speed for content.
  2. Login to solr UI
    1. https://x.x.x.x/solr  (enter ecaadmin user and password)
    2. Click on the collection named IGLS on the left hand side
    3. Select the query meny option on the left hand side
    4. Fill in the query paramters as follows. See Screenshot below.
    5. In the q filed enter -0> contentindexedat:[* TO *] 
    6. in the start , rows enter 0 and 0 
    7. In the Raw Query Parameters --facet.range=contentparsetimer&facet.range.start=0&facet.range.gap=100&facet.range.end=30000&stats=true&stats.field=contentparsetimer
    8. Click the Facet check box to enable it.
    9. In the Facet Query enter --facet.range=contentparsetimer&facet.range.start=0&facet.range.gap=100&facet.range.end=30000&stats=true&stats.field=contentparsetimer
    10. Click the Execute Query button
    11. Results should display similar to the screen shot.    Copy this text response completely and post it in to a support case for analysis. 
    12.  
    13.  

Advanced Cluster Configuration 

  1. Suppress PowerScale changelist mode - Use only if directed by support
    1. Search creates snapshots with a 5 day expiry. When we've disabled incremental ingestion, but are processing a long-running full ingestion, this snapshot can be deleted and we're left without one on disk. To avoid the issue, add an optional setting to "eca-env-common.conf" to disable incremental that will stop processing after the snapshot has been created, but before the changelist is created. This will keep creating snaps on the PowerScale, but will not process any changelist data.
    2. add to eca-env-common.conf  and requires a restart of the taskmaster container on node 1 .
    3. export SUPPRESS_CHANGELIST_TASK_CRON="0 0 * * *"  



© Superna LLC