Administration Guides

Scripted API Searches with Search & Recover GraphQL API

Home


Overview

The API can be used to automate searches and apply logic to the results.   Several examples are available below.

  1. Search for credit card or SSN on newly added files and send an alert email on any hits.
  2. Monitor a folder for a threshold number of files and send an alert email if the threshold is crossed.


Scheduled Content Search with Email Alerts Example 

Description:

  • To search file with specific path that contain specific Social Security Number (SSN) (i.e: filePath:"/ifs/data/search3/folder1",content:"333-44-5555")

  • If found that file,  send e-mail by utilizing mailx function on Eyeglass Search node.


Script:

  1. Login to any Eyeglass Search node as ecaadmin user.

  2. Create a script (Example: /home/ecadmin/contentsearchtrigger.sh).


Bash Script:

======================



#!/bin/bash 

## declare mail variables

##email subject 

subject="Eyeglass Search Found file with requested SSN"

##sending mail as

from="eyeglassSR@exampledomain.com"

## sending mail to

to="admin1@exampledomain.com"

## send carbon copy to

also_to="admin2@exampledomain.com"


## token

token=$(curl -s -G -k https://<eyelgassSR-node1-IPaddress>/graphql --data-urlencode 'query={

   login(id:"searchuser@exampledomain.com", pass:"NotReal!") {

     token

   }

 }'| awk -F'"' '{print $8}')

 

## Search

found=$(curl -s -Gk -H Authorization:"Bearer $token" https://<eyelgassSR-node1-IPaddress>/graphql --data-urlencode 'query={

  fileInfo(filePath:"/ifs/data/search3/folder1",content:"333-44-5555") {

        pageInfo {

            totalNum

        }

  }

}'| tr -dc '0-9')


## check if found

if [[ "$found" > 0  ]]; then

        

        ## send email if file with requested SSN found

        echo -e "Found\n\nthe requested Social Security Number" | mailx -s "$subject" -r "$from" -c "$to" "$also_to"

fi


exit 0



  1. Change mode to executable:

chmod +x contentsearchtrigger.sh

  1. Run the script:

./contentsearchtrigger.sh

  1. Check e-mail.

  2. Then follow setup on a schedule instructions.


Scheduled File Count Monitoring in a Folder with email alerts Example

Description:

This example will count the number of files in a folder (Example: Path:"/ifs/data/search3/folder1"). If the number of files is greater than threshold, send e-mail by utilizing mailx function on Eyeglass Search node.

Script:

  1. Login to any Eyeglass Search node as ecaadmin user
  2. Create a script (Example: /home/ecadmin/foldermonitor.sh)
  3. Requires modifications for emails and password to authenticate to the index
  4. Can use eccaadmin user
  5. The depth property is how many folder levels the search should use.  This example /ifs/data/path1/path2 is a depth 5 to limit the search to files in path2 folder, level 5 depth means the folders equal a depth and the file name is also included in the depth value.  Adjust the script to use a depth value equal to the path used in the search. See bolded yellow highlight in the script to set the depth value of the search.  Note:  This is not a recursive search and will not count files below the path in the search.
Bash Script:
======================


#!/bin/bash
## declare mail variables
##email subject
subject="Eyeglass Search Found the number of files is greater than threshold"
##sending mail as
from="eyeglassSR@exampledomain.com"
## sending mail to
to="admin1@exampledomain.com"
## send carbon copy to
also_to="admin2@exampledomain.com"

## token
token=$(curl -s -G -k https://<eyelgassSR-node1-IPaddress>/graphql --data-urlencode 'query={
login(id:"searchuser@exampledomain.com", pass:"NotReal!") {
token
}
}'| awk -F'"' '{print $8}')
## Search. Set the depth of the search where each folder equals 1 level of depth in the example 5 means 5 folders in the path
found=$(curl -s -Gk -H Authorization:"Bearer $token" https://<eyelgassSR-node1-IPaddress>/graphql --data-urlencode 'query={
fileInfo(filePath:"/ifs/data/search3/folder1",content:'depth:5'") {
pageInfo {
totalNum
}
}
}'| tr -dc '0-9')

## check if greater than threshold (Replace the <Threshold with the actual threshold number>
if [[ "$found" > <Threshold> ]]; then
## send email if found greater than threshold
echo -e "Found\n\nthe number of files greater than threshold" | mailx -s "$subject" -r "$from" -c "$to" "$also_to"
fi

exit 0

============

  1. Change mode to executable
  2. chmod +x foldermonitor.sh
  3. Run the script
  4. ./foldermonitor.sh
  5. Check e-mail
  6. Then follow setup on a schedule instructions.

Monitor a Directory for files being created in the last hour

Description:

This solution can assist with application workflows that expect files to be created in a directory on a regular basis.  If no new files are created it may indicate the application process has crashed or has an issue writing data to the directory.    This solution can use an hourly search to check for new files created in the last hour and runs on a schedule, to monitor for files > 0 or some other threshold and send an email alert.

  1. To count the number of files created for the last 1 hour in a folder (i.e.: Path:"/ifs/data/search3/folder1") .
  2. If the number of files is zero, send e-mail by utilizing mailx function on Eyeglass Search node.
  3. Time format: Epoch .

Script:

  1. Login to any Eyeglass Search node as ecaadmin user .
  2. Create a script (Example: /home/ecadmin/monitornewfileslasthour.sh).
Bash Script:
======================


#!/bin/bash
## declare mail variables
##email subject
subject="Eyeglass Search detect 0 file was created during last 1 hour"
##sending mail as
from="eyeglassSR@exampledomain.com"
## sending mail to
to="admin1@exampledomain.com"
## send carbon copy to
also_to="admin2@exampledomain.com"

## token
token=$(curl -s -G -k https://<eyelgassSR-node1-IPaddress>/graphql --data-urlencode 'query={
login(id:"searchuser@exampledomain.com", pass:"NotReal!") {
token
}
}'| awk -F'"' '{print $8}')

## Stat Time and End Time for search last 1 hour
starttime=$(date +%s%N -d "1 hour ago" | cut -b1-13)
endtime=$(date +%s%N | cut -b1-13)

## Search. Set the folder and depth of the search
found=$(curl -s -Gk -H Authorization:"Bearer ${token}" https://<eyelgassSR-node1-IPaddress>/graphql --data-urlencode "query={
fileInfo(filePath:\"/ifs/data/search3/folder1\",content:\"depth:5\",creationStart:\"${starttime}\", creationEnd:\"${endtime}\") {
pageInfo {
totalNum
}
}
}"| tr -dc '0-9')

## check if number of file equal to zero
if [[ "$found" = 0 ]]; then
## send email if 0 file was created during last 1 hour
echo -e "Found\n\n0 file was created during last 1 hour" | mailx -s "$subject" -r "$from" -c "$to" "$also_to"
fi

exit 0


============

  1. Change mode to executable.
  2. chmod +x monitornewfileslasthour.sh
  3. Run the script:
  4. ./monitornewfileslasthour.sh
  5. Check e-mail.
  6. Then follow setup on a schedule instructions.

Setup a script to run on a Schedule with Cron

  1. Login to node 1 as ecaadmin use examples below for hourly, daily or use a different cron tab string. Find examples here.
  2. Type crontab -e  . 
  3. Press letter i (to insert).
  4. Copy and paste the job information into the editor.
  5. Then press esc key.
  6. Then press :
  7. Followed by wq (write and quit).

Example cron entries:

  1. Runs hourly:
    1.  0 * * * * $HOME/myscript.sh     (note $HOME = /home/ecaadmin path)
  2. Runs Daily at midnight:
  1. 0 0 * * * $HOME/myscript.sh

How to Search for last modified date files and save the results to a file (sample code provided)

  1. Use this sample code in python to search for modified files between 2 dates and save the results to a file.
  2. Download the sample here.
  3. If using python3
    1. pip3 install requests
    2. or python2
    3. pip install requests
  4. How to customize the script for your environment
    1. the script uses ecaadmin and default password change values based on your installation
    2. The url must be changed to use node 1 IP address on the Search & Recover appliance
      1. Search for url in the file and edit the ip address to match your installation
    3. Search for modStart  and modEnd and set the date and time using epoch time 
      1.  Use an epoch conversion tool  to get the numeric value of the dates.  tool here.
    4. change the file name by searching for data_file.csv  and change the file name
  5. How to execute the script (python3)
    1. python3  <file name>.py





© Superna Inc