Eyeglass Solutions Publication

ADAF - Administration Guide

Home

Autonomous Storage Failover

Application Server

Storage Configuration

Application Server reads and writes data from and to the PowerScale cluster. Data is stored on the PowerScale and accessed through SMB or NFS protocols.

For configuring SMB share or NFS export on the PowerScale, refer to OneFS configuration documentation.

DFS Namespace

For the case that data is stored on the PowerScale as SMB service and accessed through Microsoft DFS namespace, we can integrate with Eyeglass DFS Mode Failover.

Configure DFS Folder Target for accessing the data by using DFS namespace UNC format. Notes for Eyeglass DFS Mode Failover, we configure the DFS Folder target as Dual Folder Targets. Refer to this document for detail procedure Example

Map network drive to Windows Server by using DFS namespace UNC. Example

Eyeglass DR

Eyeglass DR supports 4 different types of Failover:

SyncIQ Policy Failover
DFS Mode Failover
Access Zone Failover
IP Pool Failover

The most common types of Failover for integration with autonomous storage failover are DFS Mode and Access Zone Failover.

Eyeglass DFS Mode Failover

Refer to Eyeglass DFS Mode Failover configuration document link

Eyeglass Access Zone Failover

Refer to Eyeglass Access Zone Failover configuration document link

Creating Eyeglass REST API Token

Reference:

Eyeglass API Guide

We need to generate REST API token on the Eyeglass appliance to be used by external component to access Eyeglass.

Open the Superna Eyeglass REST API settings from the Eyeglass Main Menu

Click on Create New Token button and enter a name to reference the token

Click OK and the new token will be shown in Tokens list
Click the Token from the list to copy to clipboard and we can paste to our code for connection to Eyeglass

Using Curl Command to Retrieve IDs Related to Failover

Run the following curl command from Eyeglass appliance

To retrieve PowerScale ID (Source Cluster ID and Target Cluster ID)

curl -X GET --header 'Accept: application/json' --header 'api_key: igls-xxxxxxxxxx' -k 'https://xx.xx.xx.xx/sera/v1/nodes'

Replace igls-xxxxxxx with the API token

Replace xx.xx.xx.xx with the IP Address of Eyeglas Appliance

Example of the output

[{"id":"dg-isi34_005056a46190d0a99a634405e3a211305c49","ip":"dg-isi34-aa.bb.cc.dd.igls.local","name":"dg-isi34"},{"id":"dg-isi38_005056a4885736b19a6389001550ffc12935","ip":"dg-isi38-ee.ff.gg.hh.igls.local","name":"dg-isi38"}]

Source Cluster ID = dg-isi34_005056a46190d0a99a634405e3a211305c49

Target Cluster ID = dg-isi38_005056a4885736b19a6389001550ffc12935

To retrieve Failover Target ID (ID for the policy to be failed over (for the case of DFS Mode failover))

curl -X GET --header 'Accept: application/json' --header 'api_key: igls-xxxxxxxxxx' -k 'https://xx.xx.xx.xx/sera/v1/nodes/<source-cluster-id>/policies/<name-of-synciq-policy>'

Replace igls-xxxxxxx with the API token

Replace xx.xx.xx.xx with the IP Address of Eyeglas Appliance

Replace <source-cluster-id> with Source Cluster ID

Replace <name-of-synciq-policy> with the name of SyncIQ Policy that needs to be failed over.

Example of the output

{"failoverReadiness":{"status":"INFO"},"id":"policy-dg-isi34_azdata-policy1","name":"azdata-policy1","readinessDetail":[{"name":"OneFS SyncIQ Readiness","status":{"status":"INFO"},...

Failover Target ID: policy-dg-isi34_azdata-policy1

To retrieve Failover Target ID (ID for the policy to be failed over (for the case of Access Zone failover))

curl -X GET --header 'Accept: application/json' --header 'api_key: igls-xxxxxxxxxx' -k 'https://xx.xx.xx.xx/sera/v1/nodes/<source-cluster-id>/zones/<name-of-access-zone>'

Replace igls-xxxxxxx with the API token

Replace xx.xx.xx.xx with the IP Address of Eyeglas Appliance

Replace <source-cluster-id> with Source Cluster ID

Replace <name-of-access-zone> with the name of Access Zone that needs to be failed over.

Example of the output

{"failoverReadiness":{"status":"INFO"},"id":"zone-dg-isi34_az-data2","name":"az-data2","readinessDetail":[{"name":"Zone Path Validation","status":{"status":"OK"}, …

Access Zone ID = zone-dg-isi34_az-data2

Autonomous Failover Code

This autonomous Failover Code consists of the following functions:

Monitoring condition
Detecting event that matches Failover Requirement Condition
Send Email Alert
Trigger Failover from Eyeglass
Pre and Post Failover step

Depending on the use case, type of Failover and our requirements, we place these autonomous Failover Code modules on the Eyeglass appliance and / or Application server.

Failover Requirement Condition - Monitoring and Fault Detection

We specify the condition that must be met as a trigger for Failover. One way to determine this requirement is based on the accessibility of the PowerScale cluster from the Application Server through SMB / NFS protocols.

Define a simple test (text) file and place it under SMB / NFS path to let the code to continuously perform read / write tests against that test file.

Upon failing to do so indicates Primary Storage inaccessibility and determine to trigger the failover through Eyeglass.

E-mail Alert Notification Configuration

As the commonly used SMTP for E-mail alert notifications is through gmail SMTP, the following configuration is based on the Gmail App Password Configuration.

An App Password is a 16-digit passcode that gives a less secure app or device permission to access Google Account for sending e-mail. App Passwords can only be used with Google accounts that have 2-Step Verification turned on

Go to the Google Account Settings.
Select Security.
Under "Signing in to Google," select App Passwords. You may need to sign in.
At the bottom, choose Select app and choose the app you are using and then Select device and choose the device you’re using and then Generate.
Follow the instructions to enter the App Password. The App Password is the 16-character code in the yellow bar on your device.
Tap Done.

For further information refer to gmail app password document.

Trigger Failover from Eyeglass

Upon detecting an event condition that matches Failover requirement condition, the Autonomous Failover code will run curl command for executing Eyeglass REST API failover command.

For creating Eyeglass API Token refer this section

For getting IDs related to failover refer to this section

Code Variable for Failover

We need to have the following information to set some variables in this code.

Variable	Description
IGLS_SERVER_IP	Eyeglass Appliance IP Address
IGLS_API_KEY	Eyeglass API Token - Refer to this step for creating Eyeglass API Token
SOURCE_ID	Specify PowerScale Cluster Source ID - refer to this step
TARGET_ID	Specify PowerScale Cluster Target ID - refer to this step
FAILOVER_TARGET	Specify PowerScale Failover Target - refer to this step
PRIMARY_SERVER	Specify Primary Application Server
SECONDARY_SERVER	Specify Secondary Application Server
SMTP_SERVER	Specify SMTP Server
SMTP_PORT	Specify SMTP Server Port
FROM_ADDR	Specify Sender E-mail Address
TO_ADDR	Specify Recipient E-mail Address
APP_PASS	Specify E-mail App password For gmail, refer to this section for creating app password
TIMER	Specify the interval of checking status (in seconds).
Outfile	Specify the path of the test file for storage accessibility check. Set the test file on the network drive mapped from PowerScale SMB Share (through DFS Namespace UNC) or mounted NFS export

Autonomous Failover Code in PowerShell

For the case if the Autonomous Failover Code needs to be run from Windows Server through PowerShell (Example integration with Windows Cluster Network Load Balancing), we can use Autonomous Failover code in PowerShell.

PowerShell - Eyeglass Security Certificate handling

We need to run the following PowerShell Command to allow certificate handling for executing Eyeglass failover curl command from the Windows Server PowerShell

===

if (-not("dummy" -as [type])) {

add-type -TypeDefinition @"

using System;

using System.Net;

using System.Net.Security;

using System.Security.Cryptography.X509Certificates;

public static class Dummy {

public static bool ReturnTrue(object sender,

X509Certificate certificate,

X509Chain chain,

SslPolicyErrors sslPolicyErrors) { return true; }

public static RemoteCertificateValidationCallback GetDelegate() {

return new RemoteCertificateValidationCallback(Dummy.ReturnTrue);

}

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = [dummy]::GetDelegate()

===

Code Example

cls

#IP Address & API Key of Eyeglass

$IGLS_SERVER_IP = "<#Specify Eyeglass IP Address>" #Specify Eyeglass IP Address

$IGLS_API_KEY = "<#Specify Eyeglass API Token>" #Specify Eyeglass API Token

#Failover Source & Target

$SOURCE_ID = "<#Specify PowerScale Cluster Source ID>" #Specify PowerScale Cluster Source ID

$TARGET_ID = "<#Specify PowerScale Cluster Target ID>" #Specify PowerScale Cluster Target ID

$FAILOVER_TARGET = "<#Specify PowerScale Failover Target>" #Specify PowerScale Failover Target

#Primary Server & Secondary Server

$PRIMARY_SERVER = "<#Specify Primary Windows Server>" #Specify Primary Server

$SECONDARY_SERVER = "<#Specify Secondary Windows Server>" #Specify Secondary Server

#CURL Command

$URI1 = "https://$($IGLS_SERVER_IP)/sera/v2/jobs/failover?sourceid=$($SOURCE_ID)&targetid=$($TARGET_ID)&failovertarget=$($FAILOVER_TARGET)&controlled=true&datasync=true&configsync=false&resyncprep=true&disablemirror=false&quotasync=true&blockonwarnings=true&rollbackrenameshares=true&smbdataintegrity=false"

#E-mail Settings

$SMTP_SERVER = "<#Specify SMTP Server>" #Specify SMTP Server

$SMTP_PORT = "<#Specify SMTP Server Port>" #Specify SMTP Server Port

$FROM_ADDR = "<#Specify Sender E-mail Address>" #Specify Sender E-mail Address

$TO_ADDR = "<#Specify Recipient E-mail Address>" #Specify Recipient E-mail Address

# E-mail App Password

$APP_PASS = "<#Specify Password>" | ConvertTo-SecureString -AsPlainText -Force

#Message & Subject for Primary Storage Fault Detection

$message2 = "Detect Primary Storage is not Accessible.Storage Failover to Secondary PowerScale is started"

$Subject = "Detect Primary Storage is not Accessible.Storage Failover to Secondary PowerScale is started"

#Timer in seconds

$TIMER = "30"

#Test File

$Outfile = "z:\test\test-write-file.txt"

function FailoverPriToSec {

#Initiate Failover from Primary to Secondary PowerScale

$failovercmd = Invoke-RestMethod -Method POST -Uri $URI1 -Verbose:$false -Headers @{

'Accept' = 'application/json'

'api_key' = $IGLS_API_KEY

'Content-Type' = 'application/json'

}

$failoverstatus = $failovercmd.id.ToString()

$URI2 = "https://$($IGLS_SERVER_IP)/sera/v2/jobs/failover/$($failoverstatus)"

Invoke-RestMethod -Method GET -Uri $URI2 -Verbose:$false -Headers @{

'Accept' = 'application/json'

'api_key' = $IGLS_API_KEY

}

Start-Sleep -Seconds $TIMER

Do {

$fostatus = Invoke-RestMethod -Method GET -Uri $URI2 -Verbose:$false -Headers @{

'Accept' = 'application/json'

'api_key' = $IGLS_API_KEY

}

$fostatus.jobStatusDetails

$fostatus.jobStatusDetails.state

$fostatus.jobStatusDetails.status

$jobstatus = $fostatus.jobStatusDetails.status

$jobstate = $fostatus.jobStatusDetails.state.ToString()

Write-Host "Failover is $jobstate ! with Status $jobstatus"

$message2 = "Failover is $jobstate ! with Status $jobstatus . Please check Eyeglass Failover Log for further information"

$Subject = "Failover is $jobstate ! with Status $jobstatus"

if ($jobstate -eq "FINISHED") {

SendEmail

$response = "F"

}

else {

SendEmail

Start-Sleep -Seconds $TIMER

}

until ($response -eq "F")

}

function SendEmail {

$Body = $message2

$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $FROM_ADDR, $APP_PASS

Send-MailMessage -From $FROM_ADDR -To $TO_ADDR -Subject $Subject -Body $Body -SmtpServer $SMTP_SERVER -port $SMTP_PORT -UseSsl -Credential $Credential

}

do {

Try {

[io.file]::OpenWrite($outfile).close()

Write-Host "Primary Storage is accessible"

Start-Sleep -Seconds $TIMER

}

Catch {

Write-Warning "Primary Storage is NOT accessible. Failover to Secondary Storage is started!"

SendEmail

FailoverPriToSec

$response = "A"

}

until ($response -eq "A")

====

Autonomous Failover Code in Python

The following example is for the case if the Autonomous Failover Code needs to be run from the Application Server as Python code.

Code Example

===

#!/usr/bin/python3

import sys, time, requests, urllib3, json, smtplib, socket

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

#Specify the following Eyeglass parameters accordingly

IGLS_API_KEY="<replace with Eyeglass REST API Token>"

IGLS_SERVER_IP="<replace with Eyeglass IP address"

#Specify Source Cluster ID, Target Cluster ID, Access Zone ID for Failover

SOURCE_ID="<replace with Source Cluster ID>"

TARGET_ID="<replace with Target Cluster ID>"

FAILOVER_TARGET="<replace with Access Zone ID>"

#TEST_TIME in seconds

TEST_TIME= 1

MAX_FAULTS= 2

#Specify checker test file path and file name

TESTFILEPATH= "/mnt/autofail/www/root"

TESTFILENAME= "demotest.html"

filepath=TESTFILEPATH+"/"+TESTFILENAME

faults = 0

#Specify SMTP Server, SMTP Port number,Sender and Recipient e-mail address, App password, Interval

SMTP_SERVER="<replace with SMTP Server IP Address / FQDN>"

SMTP_PORT="<replace with SMTP Port number>"

FROM_ADDR="<replace with Sender e-mail address for notification>"

TO_ADDR="<replace with Recipient e-mail address that will receive notification>"

APP_PASS="<replace with APP Password for emailing>"

#E-mail Notification during Failover time interval in seconds

MAIL_INTVL=60

STATUS="CHECK"

STATE="CHECK"

print("Monitor file is:", filepath)

print("Test Time=",TEST_TIME,"s")

print("Max Faults=",MAX_FAULTS)

def testFile():

try:

file = open(filepath, mode='r', encoding='utf-8')

file.close()

print ("File is readable")

clearFaults()

except OSError:

print ("File is not readable")

faultDetected()

def clearFaults():

global faults

faults=0

def faultDetected():

global faults

faults +=1

print (faults)

def checkForAction(totalFaults, maxFaults):

# Are we in an evaluated fault state?

# Simple, single dimensional test to check the number

# of accumulated faults vs the number of tolerable faults.

# return True or False

if (totalFaults > maxFaults):

return True

return False

def failOver():

#issues curl command to execute a failover.

global faults

faults=0

print ("Failing Over from "+SOURCE_ID+" to "+TARGET_ID+"!")

url = "https://"+IGLS_SERVER_IP+"/sera/v2/jobs/failover?sourceid="+SOURCE_ID+"&targetid="+TARGET_ID+"&failovertarget="+FAILOVER_TARGET+"&controlled=true&datasync=true&configsync=false&resyncprep=true&disablemirror=false&quotasync=true&blockonwarnings=true&rollbackrenameshares=true&smbdataintegrity=false"

headers = {

"api_key":IGLS_API_KEY,

"accept":"application/json"

}

res = requests.post(url, headers=headers, verify=False)

json_data = json.loads(res.text)

myid = json_data['id']

url = "https://"+IGLS_SERVER_IP+"/sera/v2/jobs/failover/"+myid

headers = {

"api_key":IGLS_API_KEY,

"accept":"application/json"

}

myid2 = 'START'

myid3 = 'START'

try:

while myid2 != 'FINISHED':

print ("Failover is In progress!")

time.sleep(MAIL_INTVL)

res = requests.get(url, headers=headers, verify=False)

json_data2 = json.loads(res.text)

myid2 = json_data2['jobStatusDetails']['state']

if myid2 == 'RUNNING':

print ("State : RUNNING")

STATE=myid2

sendEmail3(STATE)

myid3 = json_data2['jobStatusDetails']['status']

if myid3 == 'OK':

print ("Current Status : OK!")

elif myid3 == 'ERROR':

print ("Current Status : ERROR. Please check Eyeglass Failover log for details")

else:

print ("Current Status : WARNING. Please check Eyeglass Failover log for details")

print ("Failover from "+SOURCE_ID+" to "+TARGET_ID+" is FINISHED! and the result: "+myid3)

STATUS=myid3

sendEmail2(STATUS)

except:

print ('\nException occurs! Please check Eyeglass Failover log for details')

#Defined SendMail

def sendMail(message):

try:

server = smtplib.SMTP(SMTP_SERVER,SMTP_PORT,timeout=10)

server.ehlo()

server.starttls()

server.login(FROM_ADDR,APP_PASS )

server.sendmail(FROM_ADDR,TO_ADDR,message)

server.quit()

except socket.error as e:

print ("Error: unable to send email")

#Send Mail 1 for alert Fault Detection

def sendEmail():

message = f"""From: {FROM_ADDR}

To: {TO_ADDR}

Subject: Faults Detected! : Failover from {SOURCE_ID} to {TARGET_ID} will be started!

Faults Detected : Failover from {SOURCE_ID} to {TARGET_ID} will be started!.

"""

sendMail(message)

#Send Mail 2 for Finished status

def sendEmail2(status):

message = f"""From: {FROM_ADDR}

To: {TO_ADDR}

Subject: Failover from {SOURCE_ID} to {TARGET_ID} is FINISHED with Status {status} !

Failover from {SOURCE_ID} to {TARGET_ID} is FINISHED with Status {status} !. Please refer to Eyeglass Failover Log for further information.

"""

sendMail(message)

#Send Mail 3 for Running State update

def sendEmail3(state):

message = f"""From: {FROM_ADDR}

To: {TO_ADDR}

Subject: Failover from {SOURCE_ID} to {TARGET_ID} is IN PROGRESS with State {state} !

Failover from {SOURCE_ID} to {TARGET_ID} is still in PROGRESS with State : {state} !

"""

sendMail(message)

def main():

# simple loop to issue a failover command if a set number of faults are

# incurred within a given interval

while(True):

testFile()

if checkForAction(faults,MAX_FAULTS):

# our only action here - modify and modularize (function)

# to have more allowable actions

sendEmail()

failOver()

time.sleep(TEST_TIME)

main()

===

Pre and Post Failover Step

Many failover scenarios depend on extra steps performed on devices, software, and infrastructure external to the NAS cluster. These tasks can now be automated with output captured and appended to Eyeglass failover logic logs. Example: NFS host mount and remount automation.

Pre and Post Failover Steps are integrated to Eyeglass Failover logic by creating the scripting code and placing it as Pre / Post / Unified Failover Steps and enable it as part of the Failover.

The pre/post failover step will execute remote script that is stored on the remote machine (e.g. application server)

Example for Pre and Post Failover for unmount and remount NFS Export on Application Server

Refer to Eyeglass Pre Post Failover Scripting Guide document for further detailed information.

Pre Failover Script (Unmount) Example

#!/bin/bash

echo starting unmount remotely called script on remote hosts

echo source-cluster: $SOURCE

echo zone data: $ZONE_DATA

me=$(whoami)

echo name of user that runs the script : $me

if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-src-01"'); then

echo found autofail failover zonename

# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)

rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./umount.sh)

rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./umount.sh)

# remote script runs and returns output, can be output below to be captured in the failover log

echo result of host script was: $rc

echo result of host script was: $rc2

else

echo did not find autofail failover zonename to process

if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-dr-01"'); then

echo found autofail failback zonename

# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)

rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./umount.sh)

rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./umount.sh)

# remote script runs and returns output, can be output below to be captured in the failover log

echo result of host script was: $rc

echo result of host script was: $rc2

else

echo did not find autofail failback zonename to process

Post Failover Script (mount) Example

#!/bin/bash

echo starting renmount remotely called script on remote hosts

echo source-cluster: $SOURCE

echo zone data: $ZONE_DATA

me=$(whoami)

echo name of user that runs the script : $me

if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-src-01"'); then

echo found autofail failover zonename

# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)

rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./mount.sh)

rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./mount.sh)

# remote script runs and returns output, can be output below to be captured in the failover log

echo result of host script was: $rc

echo result of host script was: $rc2

else

echo did not find autofail failover zonename to process

if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-dr-01"'); then

echo found autofail failback zonename

# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)

rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./mount.sh)

rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./mount.sh)

# remote script runs and returns output, can be output below to be captured in the failover log

echo result of host script was: $rc

echo result of host script was: $rc2

else

echo did not find autofail failback zonename to process

umount.sh - Example script

#unmount script

echo "unmounting filesystem pre failover"

sudo umount -fl /mnt/autofail/www/root

mount | grep "/mnt/autofail/www/root"

mount.sh - Example script

#mount script

echo "remounting filesystem post failover"

sudo mount -a

mount | grep "/mnt/autofail/www/root"

System Information Gathering Module

System Information gathering module is configured by allowing Eyeglass appliance to establish passwordless SSH connections to various remote machines.

Best Practices

Use the following user accounts for the setup

Machine	User account
Eyeglass	sca (default eyeglass user for running services)
PowerScale	eyeglass - refer to this Eyeglass minimum permission configuration document to create this service account on PowerScale
Linux	root - or other user account with sufficient permission to run the administrative commands
Windows	administrator - or other user account with sufficient permission to run the administrative commands

Passwordless SSH Configuration

ssh to Eyeglass appliance as admin user.
sudo -s (to switch to root). Enter admin password.
cd /opt/superna (this is the home directory for the sca user used by Eyeglass processes).
Create directory /opt/superna/.ssh
Type ‘ssh-keygen -t rsa’ do not set a password and accept all default prompts but enter a path of /opt/superna/.ssh/id_rsa.
su sca
ssh user@remotehost (creates known_hosts file for target host, answer yes to accept ssh ID).
Exit (you are now root again).
cd /opt/superna/.ssh
chown sca *
chgrp users *
ssh User@remotehost mkdir -p .ssh (User is the user that has access to the script that must execute, remotehost is dns or host name of remote linuxhost ) - this will create .ssh if it does not already exist.
cat /opt/superna/.ssh/id_rsa.pub | ssh User@remotehost 'cat >> .ssh/authorized_keys' (this places pub ssh keys into the remote users .ssh authorized keys file to allow passwordless login from a script).
Enter password for User on remote host.
Test SCA remote ssh:
su sca
ssh user@remotehost (if no pwd requested the setup is complete) .
Done.

For Windows to copy public ssh keys:

cat /opt/superna/.ssh/id_rsa.pub (to get the content of the id_rsa.pub file)
Copy the content of id_rsa.pub file to .ssh/authorized_keys file on windows server (this places pub ssh keys into the remote users .ssh/authorized keys file to allow passwordless login from eyeglass). We can use text editor to copy the content of id_rsa.pub file to the authorized_keys file

OpenSSH for Windows : Allowing SSH connection using public ssh key

Edit sshd_config file (default path: C:\ProgramData\ssh\sshd_config) and set this parameter to yes

PubkeyAuthentication yes

Module Codes

Main Python Code

This main python code is stored and executed from Eyeglass.

As the user on Eyeglass used for establishing passwordless SSH to remote machines is sca, we can place the main python code under /opt/superna folder that is owned by sca user.

Example

Path	Purpose
/opt/superna/mycode	Place the main python code in this directory
/opt/superna/mycode/mylogs	Create the sub-directory for placing the gathered logs

===

#!/usr/bin/python3

import urllib3, subprocess

urllib3.disable_warnings()

import sys

#Linux Commands

LINCMD1 = "cat"

LINCMD2= "/etc/os-release"

LINCMD3 = "python3 -V"

LIN_SCRIPT_DIR = "/<direcgtory-path>" # Specify the directory path where we store the script on a remote linux machine. Example: /home/myadmin/

LIN_SC1 = "script1.sh" # Specify the script file name

LIN_SC2 = "script2.sh" # Specify the script file name

LINCMD4 = LIN_SCRIPT_DIR + LIN_SC1

LINCMD5 = LIN_SCRIPT_DIR + LIN_SC2

#Windows Commands

WIN_SCRIPT_DIR = "/mytest1/" # Specify the directory path where we store the powershell script on remote windows machine

WIN_PS1 = "gather.ps1" # Specify the powershell script file name

WIN_PS2 = "compress.ps1" # Specify the powershell script file name

PS = "powershell.exe "

WINCMD1 = PS + WIN_SCRIPT_DIR + WIN_PS1

WINCMD2 = PS + WIN_SCRIPT_DIR + WIN_PS2

#PowerScale OneFS command

OFS1 = "isi version"

OFS2 = "isi status -v"

#ZIP Path

REMOTE_ZIP_PATH = ":/var/mytest*.zip" # Specify the remote zip path from linux machine

LOCAL_ZIP_PATH = "/opt/superna/mytest/testlogs" # Specify the local path in Eyeglass to store copy zip file

WIN_ZIP_PATH = ":/Users/Administrator/*_diagnostics.zip" # Specify the remote zip path from windows machine

#List of Machines refer to .ssh/config file

host = ['linserver1','linserver2' ] # linux servers

winhost = ['winserver1','winserver2'] # windows servers

isilon = ['isilon1','isilon2'] # PowerScale /Isilon clusters

def onefs_gather():

for i in range(len(isilon)):

result1 = subprocess.check_output(['ssh', isilon[i], OFS1])

print ("OneFS version : ", isilon[i], result1.decode("utf-8"), sep = "\n")

result2 = subprocess.check_output(['ssh', isilon[i], OFS2])

print ("OneFS Status : ", isilon[i], result2.decode("utf-8"), sep = "\n")

def lin_gather():

for i in range(len(host)):

result = subprocess.check_output(['ssh', host[i], LINCMD1 , LINCMD2])

print ("Host : ", host[i], result.decode("utf-8"), sep = "\n")

result2 = subprocess.check_output(['ssh', host[i], LINCMD3])

print ("Python version : ", host[i], result2.decode("utf-8"), sep = "\n")

result3 = subprocess.check_output(['ssh', host[i], LINCMD4])

print ("Network Info : ", host[i], result3.decode("utf-8"), sep = "\n")

result4 = subprocess.check_output(['ssh', host[i], LINCMD5])

print ("System Info : ", host[i], result4.decode("utf-8"), sep = "\n")

result5 = subprocess.run(['scp', host[i] + REMOTE_ZIP_PATH , LOCAL_ZIP_PATH])

def win_gather():

for i in range(len(winhost)):

result = subprocess.check_output(['ssh', winhost[i], WINCMD1])

print ("WinServer : ", winhost[i], result.decode("utf-8"), sep = "\n")

result2 = subprocess.check_output(['ssh', winhost[i], WINCMD2])

print ("WinServer : ", winhost[i], result2.decode("utf-8"), sep = "\n")

result3 = subprocess.run(['scp', winhost[i] + WIN_ZIP_PATH , LOCAL_ZIP_PATH])

def main():

onefs_gather()

lin_gather()

win_gather()

main()

Locally Stored Commands

On Linux Server

The following scripts are placed on the remote Linux servers. The main code on the Eyeglass appliance will call and execute these scripts on Linux servers.

The path location and the name of the scripts should also be reflected in the main code

LIN_SCRIPT_DIR = "/<direcgtory-path>" # Specify the directory path where we store the script on a remote linux machine. Example: /home/myadmin/

LIN_SC1 = "script1.sh" # Specify the script file name

LIN_SC2 = "script2.sh" # Specify the script file name

Example:

Folder location: /home/myadmin/

Script #1 : /home/myadmin/script1.sh

Script #2 : /home/yadmin/script2.sh

script1.sh

This script is stored in the Linux server to be run remotely to gather the network information and also to zip the /var/log directory

===

ifconfig

cd /var

zip -r "mytest-$(hostname)-$(date +"%Y-%m-%d-%T").zip" log/

===

script2.sh

This script is stored in the Linux server to be run remotely to gather some other system information.

===

#!/bin/bash

echo -e "-------------------------------System Information----------------------------"

echo -e "Hostname:\t\t"`hostname`

echo -e "uptime:\t\t\t"`uptime | awk '{print $3,$4}' | sed 's/,//'`

echo -e "Manufacturer:\t\t"`cat /sys/class/dmi/id/chassis_vendor`

echo -e "Product Name:\t\t"`cat /sys/class/dmi/id/product_name`

echo -e "Version:\t\t"`cat /sys/class/dmi/id/product_version`

echo -e "Serial Number:\t\t"`cat /sys/class/dmi/id/product_serial`

echo -e "Machine Type:\t\t"`vserver=$(lscpu | grep Hypervisor | wc -l); if [ $vserver -gt 0 ]; then echo "VM"; else echo "Physical"; fi`

echo -e "Operating System:\t"`hostnamectl | grep "Operating System" | cut -d ' ' -f5-`

echo -e "Kernel:\t\t\t"`uname -r`

echo -e "Architecture:\t\t"`arch`

echo -e "Processor Name:\t\t"`awk -F':' '/^model name/ {print $2}' /proc/cpuinfo | uniq | sed -e 's/^[ \t]*//'`

echo -e "Active User:\t\t"`w | cut -d ' ' -f1 | grep -v USER | xargs -n1`

echo -e "System Main IP:\t\t"`hostname -I`

echo ""

echo -e "-------------------------------CPU/Memory Usage------------------------------"

echo -e "Memory Usage:\t"`free | awk '/Mem/{printf("%.2f%"), $3/$2*100}'`

echo -e "Swap Usage:\t"`free | awk '/Swap/{printf("%.2f%"), $3/$2*100}'`

echo -e "CPU Usage:\t"`cat /proc/stat | awk '/cpu/{printf("%.2f%\n"), ($2+$4)*100/($2+$4+$5)}' | awk '{print $0}' | head -1`

echo ""

echo -e "-------------------------------Disk Usage >80%-------------------------------"

df -Ph | sed s/%//g | awk '{ if($5 > 80) print $0;}'

echo ""

echo -e "-------------------------------For WWN Details-------------------------------"

vserver=$(lscpu | grep Hypervisor | wc -l)

if [ $vserver -gt 0 ]

then

echo "$(hostname) is a VM"

else

cat /sys/class/fc_host/host?/port_name

echo ""

echo -e "-------------------------------Oracle DB Instances---------------------------"

if id oracle >/dev/null 2>&1; then

/bin/ps -ef|grep pmon

#then

else

echo "oracle user does not exist on $(hostname)"

echo ""

if (( $(cat /etc/*-release | grep -w "Oracle|Red Hat|CentOS|Fedora" | wc -l) > 0 ))

then

echo -e "-------------------------------Package Updates-------------------------------"

yum updateinfo summary | grep 'Security|Bugfix|Enhancement'

echo -e "-----------------------------------------------------------------------------"

else

echo -e "-------------------------------Package Updates-------------------------------"

cat /var/lib/update-notifier/updates-available

echo -e "-----------------------------------------------------------------------------"

===

On Windows Servers

The following scripts are placed on the remote Windows servers. The main code on the Eyeglass appliance will call and execute these scripts on Windows servers.

The path location and the name of the scripts should also be reflected in the main code

WIN_SCRIPT_DIR = "/mytest1/" # Specify the directory path where we store the powershell script on remote windows machine

WIN_PS1 = "gather.ps1" # Specify the powershell script file name

WIN_PS2 = "compress.ps1" # Specify the powershell script file name

Example:

Folder location: C:/mytest1, we specify the variable as WIN_SCRIPT_DIR = "/mytest1/"

Script #1 : C:/mytest1/gather.ps1

Script #2 : C:/mytest1/compress.ps1

gather.ps1

This powershell script is to gather event logs, system information and network information of Windows Server.

===

# clean previous outfile

$folder = Test-Path -PathType Container -Path "c:\$env:computername"

if ($folder -eq $true) {

Remove-Item -Path "c:\$env:computername" -Force -Recurse | Out-Null

}

Write-Host ""

# Event Logs in evtx format

New-Item -Path "c:\$env:computername\eventlogs" -ItemType "directory" | Out-Null

Set-Location -Path "c:\$env:computername\eventlogs"

Write-Host "Retrieving Event Logs unfiltered." -ForegroundColor Yellow

wevtutil epl System "systemlog.evtx"

wevtutil epl Setup "setuplog.evtx"

wevtutil epl Security "securitylog.evtx"

wevtutil epl Application "applicationlog.evtx"

Write-Host " Completed .evtx log files." -ForegroundColor Green

# system information

Set-Location -Path "c:\$env:computername"

Write-Host "Retrieving MSInfo32 information. This will take some time to complete. Please wait..." -ForegroundColor Yellow

msinfo32 /report msinfo32.txt | Out-Null

Write-Host "Completed MSInfo32 information." -ForegroundColor Green

Write-Host ""

# Network information

New-Item -Path "c:\$env:computername\network" -ItemType "directory" | Out-Null

Set-Location -Path "c:\$env:computername\network"

Write-Host "Retrieving Network information..." -ForegroundColor Yellow

Get-NetAdapter | Format-Table Name,ifIndex,Status,MacAddress,LinkSpeed,InterfaceDescription -AutoSize | Out-File "Get-NetAdapter.txt"

Get-NetAdapterAdvancedProperty | Format-Table DisplayName, DisplayValue, ValidDisplayValues | Out-File "Get-NetAdapterAdvancedProperty.txt" -Width 160

Write-Host "Completed Network information." -ForegroundColor Green

Write-Host ""

===

compress.ps1

This powershell script is to compress the collected data on the Windows Server into a zip file.

===

Get-Module

$ProgressPreference = "Ignore"

$compress = @{

Path = "c:\$env:computername"

CompressionLevel = "Optimal"

DestinationPath = $env:computername + (get-date -Format -yyyyMMdd-hhmmss) + "_diagnostics.zip"

}

Compress-Archive @compress -Force

Write-Host "Completed folder compression." -ForegroundColor Green

===

Eyeglass SSH config File

SSH config file that contains the list of remote machines is stored in the Eyeglass appliance with the path: /opt/superna/.ssh/config

Populate this config file with the list of PowerScale cluster, Linux Server, Windows Server for passwordless SSH connections. Follow this procedure to configure passwordless SSH connections to those machines

The following example is based on this list

Machine Type	Server	Configured User for passwordless SSH
Linux	linserver1	root
Linux	linserver2	root
Windows	winserver1	administrator
Windows	winserver2	administrator
PowerScale	isilon1	eyeglass
PowerScale	isilon2	eyeglass

===

Host linserver1

HostName <IP address of Linux Svr1>

User root

Host linserver2

HostName <IP address of Linux Svr1>

User root

Host winserver1

HostName <IP address of win Svr1>

User administrator

Host winserver2

HostName <IP address of win Svr2>

User administrator

Host isilon1

HostName <IP address of Isilon 1>

User eyeglass

Host isilon2

HostName <IP address of Isilon 2>

User eyeglass

===