Administration Guides

Product Requirements, Cluster Sizing and Tested Scaling Limits

Home

Sizing the Search Cluster


File Count

Full Content Index or Meta data only IndexECA Cluster Node Count Sustained Disk Throughput over 1 minute Average IO Disk Latency (iostat -xyz)

Disk size 

> 250 Million (files content indexed) Or number of clusters added to search appliance great than 1Full Content ****7 ECA nodes with 20 minimum GB RAM per VM **

200 MB/s Read

100 MB/s Write

Avg Read < 10 ms

Avg write < 10 ms ***

430 GB x 7 VM's (starting size and will require more disk space to be added over time) 
< 250 Million (files content indexed) Or number of clusters added to search appliance great than 1 Full Content ****4 ECA nodes with 20 minimum RAM per VM **

200 MB/s Read

100 MB/s Write


Avg Read < 10 ms

Avg write < 10 ms

***


 430 GB x 4 VM's (starting size and will require more disk space to be added over time) 
> 1 Billion * or number of clusters greater than 1Meta data7 ECA nodes nodes with 20G RAM per VM** 

200 MB/s Read

100 MB/s Write


Avg Read < 10 ms

Avg write < 10 ms

***


 430 GB x 7 VM's
< 1 Billion * Or number of clusters added to search appliance great than 1Meta data4 ECA nodes nodes with 20G RAM per VM** 

200 MB/s Read

100 MB/s Write


Avg Read < 10 ms

Avg write < 10 ms

***


430 GB x 4 VM's 

* Note this is starting disk size, content indexing and metadata indexing will require adding more storage as more files are indexed.  If disk space utilization reaches 70% all indexing will stop automatically.  The above are only starting disk space requirements.  Additional RAM is required as per below

** Content indexing or high file count  or quick reports that run against a high file count requires more RAM per VM 20G is the minimum and file count will determine total ram per node.  Heap usage per node must be below 75% used.  If Heap usage rises additional RAM will be required for each node.  Heap usage can be viewed on node one https://x.x.x.x/solr  then select solr cloud

*** Indexing rate is directly dependent on read latency first and then write latency 2nd.  If read latency is above specified values, indexing rate will drop as a factor of the read latency to the disks in the VM.

**** Content indexing - It is expected to add disk space to the index as different content types, and numbers of files that require content ingestion will increase the index size.  This is normal, expected, and supported to add disk space online during indexing operations.  Content type and volume of content indexing differences does not allow prediction of disk space required.  Rough estimates for planning purposes should assume 10%-20% of the original data size. File formats vary greatly in how much text vs formatting or images a file contains.

Tested Limits

The following are tested limits.  These do not represent actual limits.  These numbers will be updated with future releases.


Scaling Limit ItemTested Value
Number of files or directories meta data only index (requires 7-9 VM configuration)12 billion
Maximum file size for content Indexing500 MB
Number of files in a directory1 Million
Number of subdirectories in a path1 Million
Number of paths added for indexing25
Number of clusters added to single appliance for indexing2
Number Snapshots to Monitor (1.1.5)25

© Superna LLC