Administration Guides

Search & Recover Cluster Configuration Steps

Home


        Quick Start Steps 

        This quick setup guide provides exact steps to get up and running, and a link to learn more if needed.

        Note:

        • All searchctl commands must be run as the ecaadmin user from Search node 1
        1. License Keys:
          1. Copy license zip file to Search node /home/ecaadmin directory and change permissions chmod 777.
          2. searchctl licenses add --path /home/ecaadmin/<name of zip>.zip.
          3. Learn more about - license key CLI commands .
        2. Add a cluster to Inventory:
          1. searchctl isilons add --host x.x.x.x --user root   (x.x.x.x Subnet Service IP of the system zone DO NOT USE FQDN. PowerScale does not support Session cluster wide session authentication.  Use root user for quick setup (Create eyeglass service account user for production use).
          2. Learn more about - Add a cluster CLI commands.
        3. Add a folder to be indexed:
          1. searchctl folders add --isilon <name of cluster> --folder /ifs/data/testsearch (NOTE: The name of the cluster is returned from set #2, record the folder ID returned from this command, default add command is metadata only indexing).
          2. Learn More About - Adding folders and index option CLI commands.
        4. Start Index Job for a folder that has been added:
          1. searchctl folders index --id <folder ID>  (NOTE: Replace <folder ID> with ID from step #3. Example: only 3fe1c53bdaa2eedd).
          2. Learn more About - Starting index jobs for folders CLI commands.
        5.  (Optional step if  indexed data is not present in the System Access zone) Add SmartConnect UNC to be added to user search results for Access Zones with Indexed Data:
          1. This feature allows users to see a UNC path with SMB shares inserted into the results to enable simple click to copy and open end ensures results in none system zone are displayed to users.
          2. searchctl settings zoneunc add --isilon prod-cluster-8 --zone  system --fqdn <smartconnect name>  (NOTE: Repeat for each Access zone that has indexed data, SmartConnect name should be from each Access Zone added).
          3. Learn more about - Managing Search Dynamic UNC path CLI commands.
        6. Monitor Indexing Job Progress
          1. This command will show progress of files as they are indexed, with real-time updates every few seconds.
          2. searchctl folders stats --id <folder ID>  (NOTE: replace <folder ID> with ID from step #3, example only 3fe1c53bdaa2eedd).
          3. Lean more about - Monitoring Index Job with Stats CLI commands.
          4. Learn more about - Index Update Intervals.
          5. WARNING:  Incremental indexing for changed files runs every hour after a folder is added, and runs on the hour.  Full index scan starts after the index job is started, but files are "committed" to the index every 30 minutes, which means files will NOT be returned until 30 minutes after starting the index job.  See below.  
          6. For testing use the commit command to force files that are indexed to be visible in the search results.
            1. searchctl solr commit.
        7. Start Searching
          1. Open a browser to https://<ip of node 1>.
          2. Login as a user with SMB share permissions to the folder path added for indexing (Userid syntax user@domain.com) .
          3. Type enter in the search bar this will return all files indexed so far.
          4. Refer to the user search guide or administrator search guide and Advanced Searching Guide.
          5. NOTE You will need to monitor the index job stats to see if any files have been indexed BEFORE trying to search for files.

        Search CLI Basics

        The Search & Recover CLI uses ecactl CLI syntax, or a new shortcut to search only commands called searchctl.  This limits what needs to be typed.

        1. searchctl supports -h  for help on a command.
        2. seachctl supports -v for verbose and easier to read output for some commands.
        3. searchctl supports tab completion.
        4. searchctl supports tab tab to show list of available commands.
        5. ecactl does not support tab or tab tab features.

        Adding and viewing License keys

        1. Licensing is per PowerScale node or per PowerScale cluster.   The license allows a cluster to be added to the configuration for indexing.    
          1. When a cluster is added to the configuration for indexing the node count is detected and reduced from node count licenses, when no more node licenses are available you will be able to add clusters to the configuration.
          2. If cluster based licensing is used, each cluster added to the configuration will reduce the count of clusters from the license count.
          3. NOTE: License keys are locked to the cluster GUID when the license key is installed and a cluster is added .  License keys cannot be moved to another cluster.  License keys cannot be reset if the wrong cluster was added.
          4. NOTE: Unlicensed clusters will not be indexed, and a license error will display in the UI to end users.
          5. NOTE: Search results for unlicensed clusters in the index will not be returned in the results list.

          Licensing CLI Commands 

        1. searchctl licenses add   (Uploads a new license zip file)  
          1. searchctl licenses add --path <full path to zip file>.
          2. NOTE: license zip file permissions   chmod 777 filename.zip.
        2. searchctl licenses uninstall  (Removes all of the licenses on the system).
        3. searchctl licenses list (Lists the currently installed licenses).
        4. searchtl licenses applications list (list all applications on a unified deployment with Golden Copy).


        Adding, Viewing Clusters 


        1. To Add a cluster:
          1. NEW feature supports automatic load balancing and HA features for maximum performance to index the file system.  This operates like SmartConnect but 100% supports session authentication and CRSF secured clusters.  
            1. Select an IP address in the system zone IP pool but do NOT use the SSIP.  This will turn on Load Balance mode and will:
              1. inventory all the IP's in the pool.
              2. Use round robin API calls to each node to increase performance.
              3. Supports failover if a node fails.
            2. NOTE: Not recommended. but adding via SSIP will disable load balance mode and will only send api calls to the SSIP node.
          2. searchctl isilons add --host x.x.x.x (pool ip) --user yyy  [--applications APPLICATIONS]
          3. [--applications APPLICATIONS]  is used on the unified deployment with Golden Copy.  Use GC for Golden Copy cluster and ES for a Search & Recover cluster
          4. NOTE:  xxxx is  the a pool IP address of a management in the system zone.
          5. NOTE: yyy is the local user created on the cluster.  This service account can be created by following minimum permissions documented in this guide.  
        2. To list clusters and license status:
          1. searchctl isilons list   
        3. To remove a cluster:
          1. NOTE: Do NOT remove a cluster and try to add a different cluster, this will be blocked.  Licenses are bound to the cluster when its added.  Support will be unable to assist.  Sales will be required to assist with the purchase of a new license for a 2nd cluster.  You will be able to add the same cluster back to the configuration.
          2. NOTE:  The snapshots created to ingest content are not deleted and must be manually deleted from the cluster
          3. searchctl isilons remove --name    (use searchctl isilons list to get the exact name of the cluster)


        How to Change a PowerScale cluster's IP address , enable Snapshot Recovery root password or change service account user name 

        1. List the current ip address:
          1. searchctl isilons list.
        2. How to change ip address for an PowerScale cluster in inventory:
          1. searchctl isilons modify --name <PowerScale_Name> --ip x.x.x.x --user --update-password   (Get the PowerScale name from the list command,  x.x.x.x is the ip address used to add the cluster,   --user is the service account name normally eyeglassadminSR.
          2. Example to add password for root user and snapshot monitor feature:
            1. searchctl isilons modify --name xxxx --ip y.y.y.y  {--root-pw} ( hit enter and a prompt will ask for password, xxxx is the name of the cluster displayed from the list command and y.y.y.y is new ip address to use for REST API calls to the cluster).
            2. The --root-pw  requires the root user password to be used with recovery portal with snapshot monitor mode.   Only enter root user password if you plan to enable snapshot mode feature. 
          3. Example to change the user and password to connect to the cluster:
            1. searchctl isilons modify --name SC-8120A --ip 172.25.27.32 --user  (you will be prompted to enter the user name and then the password).
          4. Example to change the service account password only:
            1. searchctl isilons modify --name SC-8120A --ip 172.25.27.32 --update-password  (you will be prompted to enter the new password)>

        Adding, Viewing, and Starting Full Indexing Jobs Section

        Content Ingestion Overview

        Full content ingestion uses a 2 stage approach when processing a path configured for ingestion.  The means files are added to a queue for content ingestion are first indexed for metadata and second stage index for matching files targetted for full content.  This approach allows search results to  appear to users based on path, name and extension quickly while allowing content ingestion to be processed by the full content ingestion queue.

        A parallel process monitors incremental ingestion by detecting changes in the file system for any configured paths.  The changed files are processed in a separate queue from full content ingestion processing and will process metadata and content ingestion at the same time.  This allows file  content that is changed to appear with full content, and allows users to find active content in the file system.  These two queues operate until the full ingestion is completed and incremental ingestion runs continuously.

        The objective of this solution is to allow active content to appear in the results faster than stale content. 

        List of supported file types for full content ingestion and indexing

        This is the list of file types that support content ingestion in this release.  This list may change over time.

        1. This page lists all formats supported by tika, all file types may not by fully supported.   No metadata indexing is performed on files and only text within the document is indexed.   Example author field is metadata in Microsoft Office Word document is not indexed.
        2. https://www.digipres.org/formats/sources/tika/formats/

        How to Enable Security Mode for Search Results on Indexed Data

        This section is important to understand before adding folders to be indexed.  If multiple modes are used on different folders the security of the results is processed for each folder and all results are returned from all indexed folders.

        1.   The flag --auth-type {SHARE_ACCESS,FILE_OWNER,SHARE_OWNER, SHARE_ACL} is used when adding a folder to be indexed, this flag is used when adding a folder path to be indexed.  NOTE: If the flag is not used the default is share access mode
          1. Share Access mode - Means a users SMB share paths are used to restrict results to data that is at or below SMB share paths they have access to mount, AND the data is indexed with Share_Access mode at or below the share paths they have permissions.  
          2. File Owner mode - Will only return results to the user on the folder or below if the user owns the file in the file system.  NOTE: Use this mode for the home directory folder.
          3. Share Owner Mode - This mode combines share and file ownership filters on results.  This should be used on group share paths if a group share  is secured using ACL's in the file system, versus share level permissions.  Combining the security mode on a path means the user MUST be a member of the share to see the results, AND must be owner on the file in the results. 

        2. SHARE_ACL mode configuration:

          1. This mode is designed for group share space with a share and ACL's applied to folders directly below the share.
            1. See detailed explanation of this security mode here.
            2. NOTE: Release 1.1.2 or later is required for this security mode
            3. NOTE: Release 1.1.5 removes the requirement for proxy root user to read ACL's and uses service account backup and restore role permissions on the cluster.
            4. This feature allows ACL's in the file system to determine if a user should see results from a given indexed path.
              1. This feature will not evaluate all folder ACL's under a folder due to performance reasons.
              2. A cluster wide setting controls how many sub folders below each SMB share path will be scanned for ACL's to build the user filters on search results.
          2. SHARE_ACL mode defaults directory depth to 1 and means only 1 folder below the indexed folder will be scanned for ACL permissions to determine the users access.   The cluster configuration to change this requires the following steps:
            1. vim /opt/superna/eca/eca-env-defaults.conf
            2. find the export ECA_AUTH_ACL_DEPTH=1  (change to a depth value up to 10)
            3. save the file
            4. Then restart the cluster
            5. ecactl cluster down
            6. followed by below to ensure the change takes effect
            7. ecactl cluster up   

        How to add a folder path to be Indexed

        NOTE: Default mode is metadata only indexing.  See below  for an example of how to enable full content .

        1. To add a folder to be indexed with metadata ONLY:
          1. searchctl folders add --isilon <name of PowerScale> --folder /ifs/something  
          2. [--metadata-only] [--includes INCLUDES]
            [--excludes EXCLUDES]
            [--metaIncludes META_INCLUDES]
            [--fullIncludes FULL_INCLUDES]
            [--snapshotMode TYPE]
            [--auth-type {SHARE_ACCESS,FILE_OWNER,SHARE_OWNER,SHARE_ACL}]
        2. To add a folder with full content AND metadata: 
          1. searchctl folders add --isilon <name of PowerScale> --folder /ifs/something --metadata-only false.
          2. NOTE: Name of PowerScale is the PowerScale cluster name of cluster added to Search & Recover.
        3. To list folders that are indexed (returns folder id used for other commands):
          1. searchctl folders list.
          2. searchctl folder list --verbose (provides more details on the configuration of the folder configuration).
        4. To remove an indexed folder:
          1. searchctl folders remove  --id ID  (get the folder id with searchctl folders list).
          2. NOTE:  The snapshots created to ingest content are not deleted and must be manually deleted from the cluster.
        5. (Advanced Option) To modify an indexed folder, and change includes or excluded file types:
          1. NOTE: Modify command will require all settings needed and will replace previous settings with the new settings.  If adding extensions or paths for content indexing, all required paths or extensions need to be added when modifying a folder configuration.
          2. searchctl folders modify  --id ID  (add new flag values below to update the folders settings).
          3. [--metadata-only {true or false}  [--includes INCLUDES]
            [--excludes EXCLUDES]
            [--metaIncludes META_INCLUDES]
            [--fullIncludes FULL_INCLUDES]
            [--snapshotMode TYPE]
            [--auth-type {SHARE_ACCESS,FILE_OWNER,SHARE_OWNER,SHARE_ACL}]


        (Optional Advanced Configuration) Folder Ingestion processing of include, exclude patterns and metadata or full content overrides 

        Content ingestion configuration allows for includes and excludes to overide default ingestion rules which will ingest all file types all paths under the configured path.  In addition, a folder configured for metadata can only have an override to full content index paths, or even specific file types.  The reverse is also supported on a full content ingestion folder to apply an override to metadata index certain paths or specific files.   Uses cases below explain the use cases.   

        Order of processing

        1. Includes patterns are processed first. 
        2. Then excludes patterns are processed 2nd.
        3. Then folder override for metadata or full content is processed 3rd.

        How to Configure Common Use Cases to include or exclude a path or file type Best Practice

        1. Home Directory or Group share space:
          1. Best Practice:  Index the home directory for metadata only and include the file types you want to index as full content.  This reduces the index size to focus on high value content only.
          2. How to Configure Content Indexing by file extension:
            1. This command will index contents of files matching the above extensions in the home directory.
              1. searchctl folders modify --id <ID> --fullIncludes="*.ppt,*.docx,*.xls,*.pdf"
            2. This command will exclude all the roaming profile registry data in the home directory and full  content index files by extension.  This will also reduce low value content to be indexed and searchable.
              1. searchctl folders modify --id <ID> --exclude "**/AppData/**" --fullIncludes="*.ppt,*.docx,*.xls,*.pdf"
        2. Full Content Folder added with file types that cannot be indexed:
          1. Best Practice: A directory path with a lot of image formats and some content types that can be indexed, should be optimized to avoid processing file types that do not have content to index.
          2. Note: The folder was added for full content indexing
          3. How to Configure: 
            1. searchctl folders modify --id <ID> --metaIncludes="*.png,*.jpeg,*.tiff"     (NOTE: To add new extensions, you must apply all previous and new to modify the folder)
            2. This command will skip an attempt to process these file types for content ingestion, and only process them for metadata on this ingestion folder.


        (Optional Advanced Configuration) Overview and Examples of include and exclude

        For the searchctl folders [add|modify] commands, add new arguments:

        A glob is a pattern match syntax to match files or folders using examples shown below.

        FlagDescription
        --includeFile paths matching this glob will be included in the indexing operation. If not specified, all files will be included.
        --excludeFile paths matching this glob will be excluded from indexation. This flag only applies to those files that are included by the --include flag. If not specified, no files will be excluded.
        --metaIncludesFile paths matching this glob will be indexed with metadata only. This argument only applies to files that are included by the --include and --exclude flags. It will have no effect if applied to folders that have the --metadata-only flag set to true.
        --fullIncludesOnly file paths matching this glob will be full content indexed. This argument only applies to files that are included by the --include and --excludeflags. It will have no effect if applied to folders that have the --metadata-only flag set to false.
        --auth-type {SHARE_ACCESS,FILE_OWNER,SHARE_OWNER}]
        		Default security is Share level access results filtering

        Examples:

        Exclude everything in the user’s appdata profile:
        --exclude ‘/ifs/home/*/AppData/**’

        Only index docx and pdf files, and exclude everything in a tmp directory:
        --include ‘*.pdf,*.docx’ --exclude ‘/ifs/data/home/tmp/**’

        Only index docx, pdf and bmp files, and but treat bmp files as metadata only. 
        --include ‘*.pdf,*.docx,*.bmp’ --metaIncludes ‘*.bmp’

        Index all files except those in AppData, but only do full content for pdf and docx
        --exclude ‘‘/ifs/home/*/AppData/**’ --fullIncludes ‘*.pdf,*.docx’

        Index all files with full content, except for those with a .png suffix which should be metadata only: 
        --metadata-only=false --metaIncludes="*.png"

        Index all files as metadata only, except for docx, which should be included for full content. 
        --metadata-only=true --fullIncludes="*.docx"


        How to Index S3 Storage Buckets

        This 1.1.5 feature allows S3 storage to be added as a target for indexing object names.  Results will return the https url to the object.

        1. login to node 1 as ecaadmin
        2. nano /opt/superna/eca/eca-env-common.conf and add this variable export ARCHIVEWORKER_ENABLE=true  and save the file with control+x answer Y to save
        3. S3 commands
          1. searchctl s3 [-h] {add,list,index,remove}
          2. usage: searchctl s3 add [-h] --endpoint ENDPOINT --secretkey SECRETKEY
            [--accesskey ACCESSKEY] [--region REGION] --bucket
            BUCKET [--container CONTAINER] --cloudtype
            CLOUDTYPE
          3. [--includes INCLUDES]  - Use this to include or exclude object keys in the object store.   
            [--excludes EXCLUDES]
          4. Cloud types are: aws, ecs, other
          5. Example command for ECS
            1. searchctl s3 add --accesskey username --secretkey Q5EY6abpUdbNRC7t --endpoint https://172.25.24.53:9021 --bucket test --cloudtype ecs 
          6. Example command for AWS
            1. searchctl s3 add  --accesskey AKIAIsdf45LN3GQ --secretkey AGV7tMlPOmlpSVsctyoqaP7k6Oxv --endpoint s3.ca-central-1.amazonaws.com --region ca-central-1 --bucket mybucketname --cloudtype aws
          7. searchctl s3 list - Use to show configured s3 endpoints
          8. searchctl s3 remove --id xx - Use list command to get the S3 ID to remove, where xx is the configuration id.
          9. searchctl s3 index --id xx (Use this command to start the S3 bucket walk and object index task, where xx is the configuration id of the s3 target)


        How to Index Snapshots for User Self Serve File Recovery 


        Video Overview

        Requirements

        1. Release 1.1.4 adds snapshot only mode that provides recover portal for many snapshots for a recover solution.  This is a separate mode from snapshot monitor that integrates with file system indexing.   See this section.
        2. Release 1.1.5 adds multiple snapshots under an indexed path and snapshots that match the indexed path, and snapshots schedules that overlap on a single path.
        3. NOTE: Add Disk space before enabling this feature.  Adding more data to the index will require disk space.  For every 5 snapshots that are monitored 200 GB per VM is required (only nodes 2 - N need disk space added).  See the guide.

        Limitations: 

        1. Limitations Snapshot folder only mode release 1.1.5:
          1. Total of 25 monitored snapshots
          2. Snapshots at the same level as the indexed folder will be supported.
          3. Multiple snapshots under an indexed path will be supported.
        2. NOTE: Snapshot monitor mode will start monitoring and differencing snapshots after the CLI command is applied.  This means no files will be indexed until a new snapshot is created by the snapshot schedule before new files will be indexed in the snapshot path.   To test this feature you will need to wait or increase the frequency of the snapshots to get changed, new files added to the snapshot.
        3. NOTE: Expired snapshots will orphan files in the snapshot index in this release and will still be returned to users in search results but will fail to restore the file.  A later release will add purge expired snapshot data from the Snapshot index which will remove expired snapshot data from the index automatically.
        4. See the User Guide on user procedures to search and recover files. 


        Understanding Snapshot Index Modes


        1. Snapshot Monitor Mode integrates with indexed folder paths, and allows a snapshot below the index folder path to be monitored and included in incremental indexing of the existing snapshots scheduled on the PowerScale.  This has the limits above with a single snapshot below the indexed folder path.
          1. Use Case:  Use this mode with a small quantity of snapshots < 3 and primary goal is file system indexing with some snapshots.
        2. Snapshot Folder Only Mode allows adding a folder and specify the snapshot only mode that will only index the snapshot data, and will not index the file system itself.  This is the recovery only solution and multiple snapshot only folders can be added.  See limitations above. This mode is recommended for backup administrator use cases that search snapshots for recovery requests. 
          1. Use Case: Use this mode to indexing snapshots only  This mode would be used when the file system indexing is not the primary objective, but recovery of snapshot data by backup administrators or end users is the objective.


        Use Cases:

        1. User Self Service Restore - Allow users to see versions of files that exist to restore from snapshots. 
        2. Backup Admin  -  A backup admin can find files for users in snapshots.
        3. Find deleted files in snapshots - deleted files in the file system often exist in the snapshots. This allows users and administrators to recover files easily from snapshots when they are not present in the active file system.

        How to Configure - Snapshot Monitor Mode Integrated with Folder Indexing

        This mode is added to an indexed folder and will index metadata within snapshots at this path.  The feature will monitor new snapshots that appear from a schedule and will difference the changes and index new files in the snapshot.  This provides users or administrators with the ability to search snapshot metadata for files. NOTE: The path must  be added for indexing before the Monitor Mode can be enabled on a path at or below the indexed path configured in the appliance.

        1. User interface  will allow searching the snapshots in the advanced options window.
        2. CLI commands to configure monitoring snapshots below indexed folders that are already configured:
          1. Get the folder id of the target folder:
            1. searchctl folders list
          2. searchctl snapshotmonitor add --folderid <folderid> --path <snapshot path>.
            1. the folder id is the parent path folder id that is already configured for indexing, to list all folder id's: searchctl folders list.
          3. searchctl snapshotmonitor remove --folderid <folderid> --path <snapshot path> (snapshot path is the path entered that has snapshots configured in Onefs and added for monitoring of new snapshots)
          4. searchctl snapshotmonitor list --folderid <folderid>


        How to Configure - Snapshot Only Mode 

        Requirements:

        1.1.5 or later release

        Overview:

        This mode is used to only index snapshots on a cluster, to use Search & Recover as a recovery tool for backup administrators, or allow end users to search through all snapshots and perform self server file recovery.  See next section on configuring global file collision settings.

        How to Configure Snapshot Only Mode:

        Two steps:

        • Add the snapshot folder in snapshot only mode and run the baseline deep scan index on the snapshot folder
        • Then you must enable snapshot monitor to index new snapshots created by the Cluster's scheduled snapshots 

        Step 1:

        1. add
          1. searchctl folders --isilon <cluster name> --folder <FOLDER_PATH> --snapshotMode "SNAPSHOT_ONLY"  (the folder path is the path where a snapshot schedule is configured to protect data.
        2. How create the baseline Index of the snapshot folder
          1. searchctl folders index --id <FOLDER_ID> --snapshot <SNAPSHOT_NAME>
          2. The snapshot name is the name of the snapshot in the OneFs gui or CLI and specifies which snapshot creates the baseline index of the snapshot.  NOTE:  It should be a recent snapshot not a snapshot from the past, because the snapshots between the baseline and new snapshots are NOT indexed.
          3. NOTE: The index folder command supports a flag to deep index the snapshot only folders, and create a baseline in the index that includes all files in the snapshot. This is a full scan of this path, but all data will be added to the snapshot index and searchable with the search previous versions option in the GUI.  See below: How to search for Snapshot only files.

        Step 2

        1. searchctl snapshotmonitor add --folderid <folderid> --path <snapshot path> 
        2. The folder ID is the from Step 1,  snapshot path is the same used in Step 1.  This command will monitor for new snapshots that appear and index the differences between the baseline deep scan, executed in in Step 1, and the new snapshot that appears on this path based on the cluster schedule. 

        How to search for Snapshot only files

        NOTE:  This mode will allow searches that select the search backups flag in the user interface to locate files within snapshots.  See screenshot.

         



        How to Enable File Global Recovery Modes to control snapshot Restore file Collisions

        This sets the user or administrator defaults for file recovery from snapshots feature.   This restore from snapshot is a cluster side operation that copies the user  selected file back into the file system, and secures the file to the user logged into the Search & Recover user interface. NOTE: The owner of the file will be the Search & Recover service account EyeglassAdminSR. 

        Requirements:

        1. Snapshot monitor mode must be enabled to index snapshots at or under a path configured for indexing.
        2. Release 1.1.5 or later releases.

        Configuration:

        1. searchctl settings filerecovery mode {OVERWRITE,OVERWRITE_AND_BACKUP_ORIGINAL,NO_OVERWRITE_AND_RESTORE}
        2. Usage: ecactl search settings filerecovery mode --isilon HOST OVERWRITE,OVERWRITE_AND_BACKUP_ORIGINAL,NO_OVERWRITE_AND_RESTORE} (NOTE: HOST is the cluster name)
          1. OVERWRITE - This will overwrite the file with the same name in the file system if it exists, or simply create the file in the file system from the snapshot.
          2. OVERWRITE_AND_BACKUP_ORIGINAL - This will overwrite the file with the same name in the file system if it exists, and will backup the existing file in the file system with igls-original-<current_date>-<filename>.
          3. NO_OVERWRITE_AND_RESTORE - This mode will never overwrite the file if it exists in the file system and will create a restore file as follows igls-restored-<date>-<filename>.


        How to start a full index or incremental job on a folder path


        Requirements:
        1. Full index jobs supported with < 1.1.5
        2. Incremental index on demand > 1.1.5
        How to start a full index job on a path that has already been added:
        1. searchctl folders index --id   (where id is the folder id, list ID's with searchctl folders list)  
          1. Option flags include:
            1. --incremental (requires 1.1.5 or later)  This option allows running a snapshot based compare incremental job before the next scheduled incremental job.
            2. --content (requires 1.1.5 patch) searches index for matching file extensions configured with --full-includes statement and places matching files directly into the queue for content indexing.  This allows metadata indexing go locate file types first and then target file types of interest with a rescan for content only after adding extensions to a folder definition with --fill-includes statement.
        2. Example: searchctl folders index --id 3fc3613a0fe814b8 
          1. NOTE:  This will start a file and directory scan to index all files at /ifs/data and below.  
        3. New commands have been added to allow targeted full re-ingestion of a single folder, or all folders below the target path.
        4. --subdir <path>  this is required to enter the path to rescan all files in the folder, but it will not walk any child paths found within this folder.
        5. --recursive (optional, default is true)  this is not required if the folder and children folders are expected to be full indexed.  If only a single folder needs to be index this should be set to false.
          1. Examples: 
            1. searchctl folders index --id xxxxxx --subdir /ifs/data/toindex/somesubfolder (index's this path and all children folders)
            2. searchctl folders index --id xxxxxxxx --subdir /ifs/data/toindex/somesubfolder --recursive false (will only index the subdir folder) 
        6. --solrUpdate  - (release 1.1.2) over time some types of file system actions can leave orphaned directories, for example renaming a directory can leave the old directory and path of old files.  This index option will fix the index and remove orphaned folders and files.  This is a result of the PowerScale change list not supporting rename directory events.
        7. --content (release 1.1.5 or later) This allows running an index job on a folder when the fullincludes flag was used to add additional file extensions for content indexing.  This comment can be used with the --subdir command to specify where to start the scan.   The index job will not tree walk the file system but will instead query the index for files that match the --fullincludes flag and place these files in the queue for content indexing.  If a file is already content indexed and has not been updated the file will be skipped.   This will speed up a content scan update on a large path of data when new file types are added.   
          1. Any content indexing configuration on the folder will be used when searching the file system for files to be queued for content indexing assessment.

        How to Manage Scheduled Jobs (Global and Folder full and incremental)

        Requirements:

        1. Release 1.1.5 or > 

        Schedule Job Definitions

        1. INVENTORY -  collects shares, acl's and user information for security - must be enabled
        2. INCREMENTAL_INGESTION - enables incremental changelist scheduled to run against all defined folders. Default disabled
        3. FULL_INGESTION - enables full index job on all folders, this will skip files already in the index with date stamp compare to the index.  Default disabled
        4. DAILY_REPORT_SCHEDULE  - Sends daily reports at this time. Default enabled once per day 
        5. SOLR_HEALTH_WATCHDOG - Enables health check on the index process for support purposes. 

        Commands to Manage Schedules (enable, disable, set schedule)

        1. Schedule modify syntax
          1. searchctl schedules modify [-h] --id ID (--schedule SCHEDULE | --disabled) 
          2. SCHEDULE is a cron string with double quotes
          3. ID values can be listed with searchctl schedules list 
        2. List Schedules
          1. searchctl schedules list  (list schedules)
        3. disable a schedule
          1. searchctl schedules modify  --id xxxx  --disable 


        Example Full and incremental index job Schedule Configuration

        1. Enable incremental on all folders with 6 hour or 1 hour  interval or daily at 8 am
          1. searchctl schedules modify --id INCREMENTAL_INGESTION --schedule "0 */6 * * *"​
          2. searchctl schedules modify --id INCREMENTAL_INGESTION --schedule "0 * * * *" 
          3. searchctl schedules modify --id INCREMENTAL_INGESTION --schedule "0 8 * * *"
        2. Enable full index job on all folders with 1 hour interval (note this will skip files that are already in the index automatically)
          1. searchctl schedules modify --id FULL_INGESTION --schedule "0 * * * *" 


        How to Monitor Index Job Status 

        NOTE: Execute commands on node 1 of the cluster. 

        1. searchctl jobs running.
          1. This command will show all running jobs full and incremental, and the current state of the job along with the date and time it started.

          2. job id                      folder id      type                          started at

            --------------------------  -------------  ----------------------------  ------------

            job-1550880760575311032660  FullIngestion  Sat Feb 23 00:12:40 UTC 2019  SCANNING


        2. searchctl jobs history.
          1. Use this command to see the start and stop times for previous full and incremental jobs, as well as the status of the job.
        3. searchctl jobs view --id  <job-xxxxxxxxxxxx>  (Use this command to monitor the status on a running job).
        4. OR searchctl jobs view --id <job-xxxxxxxxxxxx>  --follow (Use this to monitor an active running job progress through steps, with real-time updates).
          1. Use this command to view details of the running see example below:

          2. ecaadmin@demosearch-1:~> searchctl jobs view --id job-1550880760575311032660

             

            Folder ID: 3fe4b6a5d4b3c899

             

            FullIngestion (  Running ...   )

            ----Take snapshot of /ifs/data ( SUCCESS : 0.17 seconds )

            ----update snapshot alias ( SUCCESS : 0.45 seconds )

            ----Walking File System at /ifs/data (  Running ...   )


          3. See example of a completed job:

          4. FullIngestion ( SUCCESS : 2 minutes, 13.12 seconds )

            ----Take snapshot of /ifs/data ( SUCCESS : 0.17 seconds )

            ----update snapshot alias ( SUCCESS : 0.45 seconds )

            ----Walking File System at /ifs/data ( SUCCESS : 2 minutes, 12.37 seconds )

            ----Collect settings ( SUCCESS : 0.13 seconds )

            Status: SUCCESS


        How to Monitor Ingestion with the stats command 

        1. This command only shows stats that have values default.  add --all to see all stats available. 
        2. searchctl folders stats --id <id of job>  <--no-stream>  <--all>
        1. To get the job id of a folder index, run the index command searchctl folders list
        2. "name": "PowerScale-1",
          "indexedFolders": [
          {
          "id": "",
        3. searchctl folders stats --id <folder ID here>  (optional flag --no-stream  to get stats without auto refresh )   
        4. Sample stats
        5. NOTE: The rates columns is a rate per second average over the time period.
        Per Node stats command allows monitoring statistics for a single node or for all nodes
        1. ecactl search stats view  (--folder <folder_id> | --node <node_id>) [--all] [--no-stream]
        2. If entering a node the stats will be specific to the nodes processing of indexed data.


        Statistics for folder: 3fe3631c41a7e74a

        name                                    total_alltime    total_min    total_hr    total_day    rate_min    rate_hr    rate_day

        ------------------------------------  ---------------  -----------  ----------  -----------  ----------  ---------  ----------

        FULL/FILES_ACCEPTED                                 0            0           0            0           0          0           0

        FULL/FILES_CONTENT_ERRORED                          0            0           0            0           0          0           0

        FULL/FILES_CONTENT_INDEXED                          0            0           0            0           0          0           0

        FULL/FILES_IGNORED                                  0            0           0            0           0          0           0

        FULL/FILES_METADATA_ERRORED                         0            0           0            0           0          0           0

        FULL/FILES_METADATA_INDEXED                         0            0           0            0           0          0           0

        FULL/FOLDERS_ACCEPTED                               0            0           0            0           0          0           0

        FULL/FOLDERS_IGNORED                                0            0           0            0           0          0           0

        FULL/FOLDERS_METADATA_ERRORED                       0            0           0            0           0          0           0

        FULL/FOLDERS_METADATA_INDEXED                       0            0           0            0           0          0           0

        INCREMENTAL/FILES_ACCEPTED                          2            0           0            0           0          0           0

        INCREMENTAL/FILES_CONTENT_ERRORED                   0            0           0            0           0          0           0

        INCREMENTAL/FILES_CONTENT_INDEXED                   0            0           0            0           0          0           0

        INCREMENTAL/FILES_IGNORED                           0            0           0            0           0          0           0

        INCREMENTAL/FILES_METADATA_ERRORED                  1            0           0            0           0          0           0

        INCREMENTAL/FILES_METADATA_INDEXED                  0            0           0            0           0          0           0

        INCREMENTAL/FOLDERS_ACCEPTED                        0            0           0            0           0          0           0

        INCREMENTAL/FOLDERS_IGNORED                         0            0           0            0           0          0           0

        INCREMENTAL/FOLDERS_METADATA_ERRORED                0            0           0            0           0          0           0

        INCREMENTAL/FOLDERS_METADATA_INDEXED                0            0           0            0           0          0           0



        Running Inventory Scans and Viewing users and SMB Share Access

        1. To run inventory command and collect cluster information:
          1. searchctl isilons runinventory.
        2. To display AD users collected from inventory:
          1. searchctl isilons list --users.
        3. To display SMB shares collected from inventory:
          1. searchctl isilons list --shares.
        4. To display details about a users SMB share path access.  Use this command to identify the filters applied to search results for a given user. It will list the Access zone, the path and the cluster:
          1. searchctl users view --name user@domain.com.
          2. searchctl users view --name 'DOMAIN\\user' . (NOTE: the domain must be upper case and double backslash must be used to seperate the user from the domain) .

          3. Attribute    Value

            -----------  ---------------------------------------------  -------------  -----------  --------------

            Name:        AD01\dfs1

            SID:         S-1-5-21-1825440792-1775492485-428706412-1157

            DLLN:        AD01\dfs1

            UPN:         dfs1@AD1.TEST

            Shares:

                         Path                                           Share Name     Access Zone  PowerScale

                         ----                                           ----------     -----------  ------

                         /ifs/data/userdata/dfs1                        igls-dfs-dfs1  data         prod-cluster-8

                         /ifs/data/userdata/share2                      share2         data         prod-cluster-8

                         /ifs/data/userdata/share1                      share1         data         prod-cluster-8

                         /ifs/data/policy1                              SMB2           System       prod-cluster-8


        How to Enable User Authentication to Data within Access Zones and Return Search Results with Smartconnect UNC's to Files

        This section is required to enable authentication to the WebUI for users, allows users to see Smartconnect UNC path to files for copying to the clipboard, and opening files from Windows Explorer or Mac Finder.   If this is not configured, users will see a full path to the file from /ifs which will not be accessible without a UNC path to the file.  

        Authentication Requirements for User Data in Access Zones

        The configured FQDN per Access Zone setting is required for all Access Zones that will have users authenticating on the WebUI login page.  Each user that logs in will have the userID and password checked against each Access Zone FQDN configured, to verify the users has access to data.   The first Access Zone that validates the user credentials will exit the authentication process and proceed to identify all SMB shares in all Access Zones.  Review the authentication data flow below.

        NOTE: Each Access Zone used to authenticate users MUST have a configured FQDN entered in to the configuration, AND MUST have at least 1 SMB shared within the Access Zone to be used for authentication and password validation.


        Authentication Data Flow

        1. Access Zone system  - FQDN UNC authentication request with Userid and password against an SMB share discovered in System Zone.
          1. If successful exit and identify data access to SMB shares in ALL Access Zones.
          2. If authentication fails check next Access Zone UNC FQDN that was configured.
        2. Access Zone Data1 - FQDN UNC authentication request with userID and password against an SMB share discovered in Data1 Zone. 
          1. If successful exit and identify data access to SMB shares in ALL Access Zones. 
          2. if authentication fails check next Access Zone.
        3. Repeat until user is authenticated or denied access to search login page.
        4. At the end of this process all SMB shares in ALL zones are used to filter login results to the user.


        CLI commands to add Zone FQDN to Authentication and Search Results Display 

        1. searchctl settings zoneunc add --isilon clusterA --zone ZoneA --fqdn mycluster.example.com  
          1. This command will return all search results on ClusterA for all files in ZoneA with \\mycluster.example.com\<sharename>
        2. searchctl settings zoneunc list
        3. searchctl settings zoneunc remove --isilon clusterA --zone ZoneA 

          

        How to Enable Administrator Search Security Override

        By default results are secured by SMB permissions, file ownership or both SMB access and then file ownership.   File owner security will block results for administrator use cases.  This feature is used to disable all security for administrator users to execute file system searches that are required to analyze the file system and use the automation script feature or eDiscovery use cases even when their AD account does not have access to the data on the cluster.

        Compliance, File System Antlaytics Administrators

        A list of AD user ID's are added to an approved list to override all security on search results.  Local user accounts on the applicance are also supported when AD users are not required, and local login to the search UI is required.  

        Use cases for this feature

        1. Compliance officer content search
        2. eDiscovery administrator 

        How to configure a List of Search administrators

        New in 1.1.1 build AD group support for administrator groups.

        usage: ecactl search settings admins add [-h] [--name NAME] [--group GROUP] [--local]
        1. searchctl settings admins [add | list | remove ] searchctl settings admins add --name <userid> [--local ]
          1. searchctl settings admins add --name username@domain.com
            1. To Use local user on the appliance without using AD accounts
              1. searchctl settings admins add --name ecaadmin --local   (local default OS account on the cluster)
              2. NOTE: This uses local users and AD is always the best practice.  A single default OS account already exists on the appliance that can be used.
              3. NOTE: Authentication is against the password in the /opt/superna/eca/conf/nginx/.htpasswd file instead of the PowerScale AD provider.  Additional users must manually be added to the .htpasswd file AND via searchctl admins add.  
              4. To add additional local users
                1. ssh ecaadmin@xxxx  (ip of node 1)
                2. sudo -s
                3. useradd xxxx (xxxx is the name of the user)
                4. exit
                5. Set the web login password (Note OS login is not required) change the user name for xxxx and yyyy for the password.
                  1. ecactl cluster exec "htpasswd -b /opt/superna/eca/conf/nginx/.htpasswd xxxx  yyyy"
                6. Add the local admin with:
                  1. searchctl settings admins add --name xxxx --local 
            2. To add AD group of administrators
              1. searchctl settings admins add --group "groupname@domain.com"   (use groupname case senstive with @ AD DNS name syntax)
          2. searchtl settings admins remove --sid SID  (the SID is required to remove a user)
          3. To remove group admin use domain syntax with 3 \\\ to escape the \ character
            1. searchctl settings admins remove --sid "AD01\\\mlrsw1"


        How to add Data Owner Search Administrators

        This feature allows a data search administator that has span of control over one or more paths, regardless of SMB or ACL permissions modes set on the indexed folder.  The paths added to these data owners is added to the users existing security profile and allows them to search file metadata and content within documents on the allowed path.  

        For example if the users SMB share permissions grants them access to a subset of the data, based on AD groups and SMB share permissions, the paths added with this Data Owner admin will be added to the existing auto detected permissions and allow analytics and searching at this path and below.

        This feature also allows the data owner to be restricted to metadata only. It will block content searching to protect data in the index and allow the data owner admin to report on data, but not identify by content.   This is available in 1.1.1 or later releases.

        Use cases for this feature

        1. Department admin for reporting on business unit data
        2. Project administrator for reporting on project data

          

        NOTE: If the Data Owner Admin does not have access to the files in the search results or reports, they will not be able to open the files, and no additional file access is possible from the Search results.

        NOTE: The PowerScale must be added to the appliance inventory and the path must be at or below an indexed folder configurd in the system.

        Data Owner Admin Configuration

        1. Add Data Owner Admin to a path:
          1. searchctl settings adminaccesslist add --user <user@domain.com> --isilon <PowerScale Name> --path </ifs/path/to/folder> [--metadata-only] .
        2. Remove Data Owner Admin from a path:
          1. searchctl settings adminaccesslist remove --user <user@domain.com> --isilon <PowerScale Name> --path </ifs/path/to/folder> .
        3. List Data Owner Administrator assigned paths:
          1. searchctl settings adminaccesslist list --user <user@domain.com> --isilon <PowerScale Name> .

        How to Configure admin only login mode and block user login

        This is for administrator only mode, where end users do not need to login to the UI, and allows an administrator listed on the admin list to login while all other users are blocked if not on the list. NOTE:  This includes the local ecaadmin account, that must be added to the admin list in order to login.


        1. Login to node 1 of the Search cluster as admin over ssh .
        2. edit conf file and make the change below.
        3. nano /opt/superna/eca/eca-env-common.conf 
          1. Add this line export SEARCHMW_ADMIN_MODE_ONLY=true 
        4. Save the file control+x
        5. Shut down the cluster and start up again to take effect
          1. ecactl cluster down 
        6. wait until down completes.
          1. ecactcl cluster up 
        7. Now only users listed on the admin list will be allowed to login to execute searches. 


        © Superna Inc