- Autonomous Storage Failover
- Application Server
- Storage Configuration
- DFS Namespace
- Eyeglass DR
- Eyeglass DFS Mode Failover
- Eyeglass Access Zone Failover
- Creating Eyeglass REST API Token
- Using Curl Command to Retrieve IDs Related to Failover
- Autonomous Failover Code
- Failover Requirement Condition - Monitoring and Fault Detection
- E-mail Alert Notification Configuration
- Trigger Failover from Eyeglass
- Code Variable for Failover
- Autonomous Failover Code in PowerShell
- Autonomous Failover Code in Python
- Pre and Post Failover Step
- Pre Failover Script (Unmount) Example
- Post Failover Script (mount) Example
- umount.sh - Example script
- mount.sh - Example script
- Best Practices
- Passwordless SSH Configuration
- OpenSSH for Windows : Allowing SSH connection using public ssh key
- Module Codes
- Main Python Code
- Locally Stored Commands
- On Linux Server
- On Windows Servers
- Eyeglass SSH config File
Autonomous Storage Failover
Application Server
Storage Configuration
Application Server reads and writes data from and to the PowerScale cluster. Data is stored on the PowerScale and accessed through SMB or NFS protocols.
For configuring SMB share or NFS export on the PowerScale, refer to OneFS configuration documentation.
DFS Namespace
For the case that data is stored on the PowerScale as SMB service and accessed through Microsoft DFS namespace, we can integrate with Eyeglass DFS Mode Failover.
- Configure DFS Folder Target for accessing the data by using DFS namespace UNC format. Notes for Eyeglass DFS Mode Failover, we configure the DFS Folder target as Dual Folder Targets. Refer to this document for detail procedure Example
- Map network drive to Windows Server by using DFS namespace UNC. Example
Eyeglass DR
Eyeglass DR supports 4 different types of Failover:
- SyncIQ Policy Failover
- DFS Mode Failover
- Access Zone Failover
- IP Pool Failover
The most common types of Failover for integration with autonomous storage failover are DFS Mode and Access Zone Failover.
Eyeglass DFS Mode Failover
Refer to Eyeglass DFS Mode Failover configuration document link
Eyeglass Access Zone Failover
Refer to Eyeglass Access Zone Failover configuration document link
Creating Eyeglass REST API Token
Reference:
We need to generate REST API token on the Eyeglass appliance to be used by external component to access Eyeglass.
- Open the Superna Eyeglass REST API settings from the Eyeglass Main Menu
- Click on Create New Token button and enter a name to reference the token
- Click OK and the new token will be shown in Tokens list
- Click the Token from the list to copy to clipboard and we can paste to our code for connection to Eyeglass
Using Curl Command to Retrieve IDs Related to Failover
Run the following curl command from Eyeglass appliance
- To retrieve PowerScale ID (Source Cluster ID and Target Cluster ID)
curl -X GET --header 'Accept: application/json' --header 'api_key: igls-xxxxxxxxxx' -k 'https://xx.xx.xx.xx/sera/v1/nodes'
Replace igls-xxxxxxx with the API token
Replace xx.xx.xx.xx with the IP Address of Eyeglas Appliance
Example of the output
[{"id":"dg-isi34_005056a46190d0a99a634405e3a211305c49","ip":"dg-isi34-aa.bb.cc.dd.igls.local","name":"dg-isi34"},{"id":"dg-isi38_005056a4885736b19a6389001550ffc12935","ip":"dg-isi38-ee.ff.gg.hh.igls.local","name":"dg-isi38"}]
Source Cluster ID = dg-isi34_005056a46190d0a99a634405e3a211305c49
Target Cluster ID = dg-isi38_005056a4885736b19a6389001550ffc12935
- To retrieve Failover Target ID (ID for the policy to be failed over (for the case of DFS Mode failover))
curl -X GET --header 'Accept: application/json' --header 'api_key: igls-xxxxxxxxxx' -k 'https://xx.xx.xx.xx/sera/v1/nodes/<source-cluster-id>/policies/<name-of-synciq-policy>'
Replace igls-xxxxxxx with the API token
Replace xx.xx.xx.xx with the IP Address of Eyeglas Appliance
Replace <source-cluster-id> with Source Cluster ID
Replace <name-of-synciq-policy> with the name of SyncIQ Policy that needs to be failed over.
Example of the output
{"failoverReadiness":{"status":"INFO"},"id":"policy-dg-isi34_azdata-policy1","name":"azdata-policy1","readinessDetail":[{"name":"OneFS SyncIQ Readiness","status":{"status":"INFO"},...
Failover Target ID: policy-dg-isi34_azdata-policy1
- To retrieve Failover Target ID (ID for the policy to be failed over (for the case of Access Zone failover))
curl -X GET --header 'Accept: application/json' --header 'api_key: igls-xxxxxxxxxx' -k 'https://xx.xx.xx.xx/sera/v1/nodes/<source-cluster-id>/zones/<name-of-access-zone>'
Replace igls-xxxxxxx with the API token
Replace xx.xx.xx.xx with the IP Address of Eyeglas Appliance
Replace <source-cluster-id> with Source Cluster ID
Replace <name-of-access-zone> with the name of Access Zone that needs to be failed over.
Example of the output
{"failoverReadiness":{"status":"INFO"},"id":"zone-dg-isi34_az-data2","name":"az-data2","readinessDetail":[{"name":"Zone Path Validation","status":{"status":"OK"}, …
Access Zone ID = zone-dg-isi34_az-data2
Autonomous Failover Code
This autonomous Failover Code consists of the following functions:
- Monitoring condition
- Detecting event that matches Failover Requirement Condition
- Send Email Alert
- Trigger Failover from Eyeglass
- Pre and Post Failover step
Depending on the use case, type of Failover and our requirements, we place these autonomous Failover Code modules on the Eyeglass appliance and / or Application server.
Failover Requirement Condition - Monitoring and Fault Detection
We specify the condition that must be met as a trigger for Failover. One way to determine this requirement is based on the accessibility of the PowerScale cluster from the Application Server through SMB / NFS protocols.
Define a simple test (text) file and place it under SMB / NFS path to let the code to continuously perform read / write tests against that test file.
Upon failing to do so indicates Primary Storage inaccessibility and determine to trigger the failover through Eyeglass.
E-mail Alert Notification Configuration
As the commonly used SMTP for E-mail alert notifications is through gmail SMTP, the following configuration is based on the Gmail App Password Configuration.
An App Password is a 16-digit passcode that gives a less secure app or device permission to access Google Account for sending e-mail. App Passwords can only be used with Google accounts that have 2-Step Verification turned on
- Go to the Google Account Settings.
- Select Security.
- Under "Signing in to Google," select App Passwords. You may need to sign in.
- At the bottom, choose Select app and choose the app you are using and then Select device and choose the device you’re using and then Generate.
- Follow the instructions to enter the App Password. The App Password is the 16-character code in the yellow bar on your device.
- Tap Done.
For further information refer to gmail app password document.
Trigger Failover from Eyeglass
Upon detecting an event condition that matches Failover requirement condition, the Autonomous Failover code will run curl command for executing Eyeglass REST API failover command.
For creating Eyeglass API Token refer this section
For getting IDs related to failover refer to this section
Code Variable for Failover
We need to have the following information to set some variables in this code.
Variable | Description |
IGLS_SERVER_IP | Eyeglass Appliance IP Address |
IGLS_API_KEY | Eyeglass API Token - Refer to this step for creating Eyeglass API Token |
SOURCE_ID | Specify PowerScale Cluster Source ID - refer to this step |
TARGET_ID | Specify PowerScale Cluster Target ID - refer to this step |
FAILOVER_TARGET | Specify PowerScale Failover Target - refer to this step |
PRIMARY_SERVER | Specify Primary Application Server |
SECONDARY_SERVER | Specify Secondary Application Server |
SMTP_SERVER | Specify SMTP Server |
SMTP_PORT | Specify SMTP Server Port |
FROM_ADDR | Specify Sender E-mail Address |
TO_ADDR | Specify Recipient E-mail Address |
APP_PASS | Specify E-mail App password For gmail, refer to this section for creating app password |
TIMER | Specify the interval of checking status (in seconds). |
Outfile | Specify the path of the test file for storage accessibility check. Set the test file on the network drive mapped from PowerScale SMB Share (through DFS Namespace UNC) or mounted NFS export
|
Autonomous Failover Code in PowerShell
For the case if the Autonomous Failover Code needs to be run from Windows Server through PowerShell (Example integration with Windows Cluster Network Load Balancing), we can use Autonomous Failover code in PowerShell.
PowerShell - Eyeglass Security Certificate handling
We need to run the following PowerShell Command to allow certificate handling for executing Eyeglass failover curl command from the Windows Server PowerShell
===
if (-not("dummy" -as [type])) {
add-type -TypeDefinition @"
using System;
using System.Net;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
public static class Dummy {
public static bool ReturnTrue(object sender,
X509Certificate certificate,
X509Chain chain,
SslPolicyErrors sslPolicyErrors) { return true; }
public static RemoteCertificateValidationCallback GetDelegate() {
return new RemoteCertificateValidationCallback(Dummy.ReturnTrue);
}
}
"@
}
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = [dummy]::GetDelegate()
===
Code Example
cls
#IP Address & API Key of Eyeglass
$IGLS_SERVER_IP = "<#Specify Eyeglass IP Address>" #Specify Eyeglass IP Address
$IGLS_API_KEY = "<#Specify Eyeglass API Token>" #Specify Eyeglass API Token
#Failover Source & Target
$SOURCE_ID = "<#Specify PowerScale Cluster Source ID>" #Specify PowerScale Cluster Source ID
$TARGET_ID = "<#Specify PowerScale Cluster Target ID>" #Specify PowerScale Cluster Target ID
$FAILOVER_TARGET = "<#Specify PowerScale Failover Target>" #Specify PowerScale Failover Target
#Primary Server & Secondary Server
$PRIMARY_SERVER = "<#Specify Primary Windows Server>" #Specify Primary Server
$SECONDARY_SERVER = "<#Specify Secondary Windows Server>" #Specify Secondary Server
#CURL Command
$URI1 = "https://$($IGLS_SERVER_IP)/sera/v2/jobs/failover?sourceid=$($SOURCE_ID)&targetid=$($TARGET_ID)&failovertarget=$($FAILOVER_TARGET)&controlled=true&datasync=true&configsync=false&resyncprep=true&disablemirror=false"async=true&blockonwarnings=true&rollbackrenameshares=true&smbdataintegrity=false"
#E-mail Settings
$SMTP_SERVER = "<#Specify SMTP Server>" #Specify SMTP Server
$SMTP_PORT = "<#Specify SMTP Server Port>" #Specify SMTP Server Port
$FROM_ADDR = "<#Specify Sender E-mail Address>" #Specify Sender E-mail Address
$TO_ADDR = "<#Specify Recipient E-mail Address>" #Specify Recipient E-mail Address
# E-mail App Password
$APP_PASS = "<#Specify Password>" | ConvertTo-SecureString -AsPlainText -Force
#Message & Subject for Primary Storage Fault Detection
$message2 = "Detect Primary Storage is not Accessible.Storage Failover to Secondary PowerScale is started"
$Subject = "Detect Primary Storage is not Accessible.Storage Failover to Secondary PowerScale is started"
#Timer in seconds
$TIMER = "30"
#Test File
$Outfile = "z:\test\test-write-file.txt"
function FailoverPriToSec {
#Initiate Failover from Primary to Secondary PowerScale
$failovercmd = Invoke-RestMethod -Method POST -Uri $URI1 -Verbose:$false -Headers @{
'Accept' = 'application/json'
'api_key' = $IGLS_API_KEY
'Content-Type' = 'application/json'
}
$failoverstatus = $failovercmd.id.ToString()
$URI2 = "https://$($IGLS_SERVER_IP)/sera/v2/jobs/failover/$($failoverstatus)"
Invoke-RestMethod -Method GET -Uri $URI2 -Verbose:$false -Headers @{
'Accept' = 'application/json'
'api_key' = $IGLS_API_KEY
}
Start-Sleep -Seconds $TIMER
Do {
$fostatus = Invoke-RestMethod -Method GET -Uri $URI2 -Verbose:$false -Headers @{
'Accept' = 'application/json'
'api_key' = $IGLS_API_KEY
}
$fostatus.jobStatusDetails
$fostatus.jobStatusDetails.state
$fostatus.jobStatusDetails.status
$jobstatus = $fostatus.jobStatusDetails.status
$jobstate = $fostatus.jobStatusDetails.state.ToString()
Write-Host "Failover is $jobstate ! with Status $jobstatus"
$message2 = "Failover is $jobstate ! with Status $jobstatus . Please check Eyeglass Failover Log for further information"
$Subject = "Failover is $jobstate ! with Status $jobstatus"
if ($jobstate -eq "FINISHED") {
SendEmail
$response = "F"
}
else {
SendEmail
Start-Sleep -Seconds $TIMER
}
}
until ($response -eq "F")
}
function SendEmail {
$Body = $message2
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $FROM_ADDR, $APP_PASS
Send-MailMessage -From $FROM_ADDR -To $TO_ADDR -Subject $Subject -Body $Body -SmtpServer $SMTP_SERVER -port $SMTP_PORT -UseSsl -Credential $Credential
}
do {
Try {
[io.file]::OpenWrite($outfile).close()
Write-Host "Primary Storage is accessible"
Start-Sleep -Seconds $TIMER
}
Catch {
Write-Warning "Primary Storage is NOT accessible. Failover to Secondary Storage is started!"
SendEmail
FailoverPriToSec
$response = "A"
}
}
until ($response -eq "A")
====
Autonomous Failover Code in Python
The following example is for the case if the Autonomous Failover Code needs to be run from the Application Server as Python code.
Code Example
===
#!/usr/bin/python3
import sys, time, requests, urllib3, json, smtplib, socket
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
#Specify the following Eyeglass parameters accordingly
IGLS_API_KEY="<replace with Eyeglass REST API Token>"
IGLS_SERVER_IP="<replace with Eyeglass IP address"
#Specify Source Cluster ID, Target Cluster ID, Access Zone ID for Failover
SOURCE_ID="<replace with Source Cluster ID>"
TARGET_ID="<replace with Target Cluster ID>"
FAILOVER_TARGET="<replace with Access Zone ID>"
#TEST_TIME in seconds
TEST_TIME= 1
MAX_FAULTS= 2
#Specify checker test file path and file name
TESTFILEPATH= "/mnt/autofail/www/root"
TESTFILENAME= "demotest.html"
filepath=TESTFILEPATH+"/"+TESTFILENAME
faults = 0
#Specify SMTP Server, SMTP Port number,Sender and Recipient e-mail address, App password, Interval
SMTP_SERVER="<replace with SMTP Server IP Address / FQDN>"
SMTP_PORT="<replace with SMTP Port number>"
FROM_ADDR="<replace with Sender e-mail address for notification>"
TO_ADDR="<replace with Recipient e-mail address that will receive notification>"
APP_PASS="<replace with APP Password for emailing>"
#E-mail Notification during Failover time interval in seconds
MAIL_INTVL=60
STATUS="CHECK"
STATE="CHECK"
print("Monitor file is:", filepath)
print("Test Time=",TEST_TIME,"s")
print("Max Faults=",MAX_FAULTS)
def testFile():
try:
file = open(filepath, mode='r', encoding='utf-8')
file.close()
print ("File is readable")
clearFaults()
except OSError:
print ("File is not readable")
faultDetected()
def clearFaults():
global faults
faults=0
def faultDetected():
global faults
faults +=1
print (faults)
def checkForAction(totalFaults, maxFaults):
# Are we in an evaluated fault state?
# Simple, single dimensional test to check the number
# of accumulated faults vs the number of tolerable faults.
# return True or False
if (totalFaults > maxFaults):
return True
return False
def failOver():
#issues curl command to execute a failover.
global faults
faults=0
print ("Failing Over from "+SOURCE_ID+" to "+TARGET_ID+"!")
url = "https://"+IGLS_SERVER_IP+"/sera/v2/jobs/failover?sourceid="+SOURCE_ID+"&targetid="+TARGET_ID+"&failovertarget="+FAILOVER_TARGET+"&controlled=true&datasync=true&configsync=false&resyncprep=true&disablemirror=false"async=true&blockonwarnings=true&rollbackrenameshares=true&smbdataintegrity=false"
headers = {
"api_key":IGLS_API_KEY,
"accept":"application/json"
}
res = requests.post(url, headers=headers, verify=False)
json_data = json.loads(res.text)
myid = json_data['id']
url = "https://"+IGLS_SERVER_IP+"/sera/v2/jobs/failover/"+myid
headers = {
"api_key":IGLS_API_KEY,
"accept":"application/json"
}
myid2 = 'START'
myid3 = 'START'
try:
while myid2 != 'FINISHED':
print ("Failover is In progress!")
time.sleep(MAIL_INTVL)
res = requests.get(url, headers=headers, verify=False)
json_data2 = json.loads(res.text)
myid2 = json_data2['jobStatusDetails']['state']
if myid2 == 'RUNNING':
print ("State : RUNNING")
STATE=myid2
sendEmail3(STATE)
myid3 = json_data2['jobStatusDetails']['status']
if myid3 == 'OK':
print ("Current Status : OK!")
elif myid3 == 'ERROR':
print ("Current Status : ERROR. Please check Eyeglass Failover log for details")
else:
print ("Current Status : WARNING. Please check Eyeglass Failover log for details")
print ("Failover from "+SOURCE_ID+" to "+TARGET_ID+" is FINISHED! and the result: "+myid3)
STATUS=myid3
sendEmail2(STATUS)
except:
print ('\nException occurs! Please check Eyeglass Failover log for details')
#Defined SendMail
def sendMail(message):
try:
server = smtplib.SMTP(SMTP_SERVER,SMTP_PORT,timeout=10)
server.ehlo()
server.starttls()
server.login(FROM_ADDR,APP_PASS )
server.sendmail(FROM_ADDR,TO_ADDR,message)
server.quit()
except socket.error as e:
print ("Error: unable to send email")
#Send Mail 1 for alert Fault Detection
def sendEmail():
message = f"""From: {FROM_ADDR}
To: {TO_ADDR}
Subject: Faults Detected! : Failover from {SOURCE_ID} to {TARGET_ID} will be started!
Faults Detected : Failover from {SOURCE_ID} to {TARGET_ID} will be started!.
"""
sendMail(message)
#Send Mail 2 for Finished status
def sendEmail2(status):
message = f"""From: {FROM_ADDR}
To: {TO_ADDR}
Subject: Failover from {SOURCE_ID} to {TARGET_ID} is FINISHED with Status {status} !
Failover from {SOURCE_ID} to {TARGET_ID} is FINISHED with Status {status} !. Please refer to Eyeglass Failover Log for further information.
"""
sendMail(message)
#Send Mail 3 for Running State update
def sendEmail3(state):
message = f"""From: {FROM_ADDR}
To: {TO_ADDR}
Subject: Failover from {SOURCE_ID} to {TARGET_ID} is IN PROGRESS with State {state} !
Failover from {SOURCE_ID} to {TARGET_ID} is still in PROGRESS with State : {state} !
"""
sendMail(message)
def main():
# simple loop to issue a failover command if a set number of faults are
# incurred within a given interval
while(True):
testFile()
if checkForAction(faults,MAX_FAULTS):
# our only action here - modify and modularize (function)
# to have more allowable actions
sendEmail()
failOver()
time.sleep(TEST_TIME)
main()
===
Pre and Post Failover Step
Many failover scenarios depend on extra steps performed on devices, software, and infrastructure external to the NAS cluster. These tasks can now be automated with output captured and appended to Eyeglass failover logic logs. Example: NFS host mount and remount automation.
Pre and Post Failover Steps are integrated to Eyeglass Failover logic by creating the scripting code and placing it as Pre / Post / Unified Failover Steps and enable it as part of the Failover.
The pre/post failover step will execute remote script that is stored on the remote machine (e.g. application server)
Example for Pre and Post Failover for unmount and remount NFS Export on Application Server
Refer to Eyeglass Pre Post Failover Scripting Guide document for further detailed information.
Pre Failover Script (Unmount) Example
#!/bin/bash
echo starting unmount remotely called script on remote hosts
echo source-cluster: $SOURCE
echo zone data: $ZONE_DATA
me=$(whoami)
echo name of user that runs the script : $me
if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-src-01"'); then
echo found autofail failover zonename
# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)
rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./umount.sh)
rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./umount.sh)
# remote script runs and returns output, can be output below to be captured in the failover log
echo result of host script was: $rc
echo result of host script was: $rc2
else
echo did not find autofail failover zonename to process
fi
if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-dr-01"'); then
echo found autofail failback zonename
# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)
rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./umount.sh)
rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./umount.sh)
# remote script runs and returns output, can be output below to be captured in the failover log
echo result of host script was: $rc
echo result of host script was: $rc2
else
echo did not find autofail failback zonename to process
fi
Post Failover Script (mount) Example
#!/bin/bash
echo starting renmount remotely called script on remote hosts
echo source-cluster: $SOURCE
echo zone data: $ZONE_DATA
me=$(whoami)
echo name of user that runs the script : $me
if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-src-01"'); then
echo found autofail failover zonename
# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)
rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./mount.sh)
rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./mount.sh)
# remote script runs and returns output, can be output below to be captured in the failover log
echo result of host script was: $rc
echo result of host script was: $rc2
else
echo did not find autofail failover zonename to process
fi
if (echo "$ZONE_DATA" | grep -q '"source":{"name":"autoFail"') && (echo "$SOURCE" | grep -q '"name":"dem-dr-01"'); then
echo found autofail failback zonename
# remotely execute the remount.sh script on the remote host (NOTE: requires ssh pub keys from eyeglass on the remote host)
rc=$(ssh admin@<PRI_SVR_IP_ADDR> ./mount.sh)
rc2=$(ssh admin@<SEC_SVR_IP_ADDR> ./mount.sh)
# remote script runs and returns output, can be output below to be captured in the failover log
echo result of host script was: $rc
echo result of host script was: $rc2
else
echo did not find autofail failback zonename to process
Fi
umount.sh - Example script
#unmount script
echo "unmounting filesystem pre failover"
sudo umount -fl /mnt/autofail/www/root
mount | grep "/mnt/autofail/www/root"
mount.sh - Example script
#mount script
echo "remounting filesystem post failover"
sudo mount -a
mount | grep "/mnt/autofail/www/root"
System Information Gathering Module
System Information gathering module is configured by allowing Eyeglass appliance to establish passwordless SSH connections to various remote machines.
Best Practices
Use the following user accounts for the setup
Machine | User account |
Eyeglass | sca (default eyeglass user for running services) |
PowerScale | eyeglass - refer to this Eyeglass minimum permission configuration document to create this service account on PowerScale |
Linux | root - or other user account with sufficient permission to run the administrative commands |
Windows | administrator - or other user account with sufficient permission to run the administrative commands |
Passwordless SSH Configuration
- ssh to Eyeglass appliance as admin user.
- sudo -s (to switch to root). Enter admin password.
- cd /opt/superna (this is the home directory for the sca user used by Eyeglass processes).
- Create directory /opt/superna/.ssh
- Type ‘ssh-keygen -t rsa’ do not set a password and accept all default prompts but enter a path of /opt/superna/.ssh/id_rsa.
- su sca
- ssh user@remotehost (creates known_hosts file for target host, answer yes to accept ssh ID).
- Exit (you are now root again).
- cd /opt/superna/.ssh
- chown sca *
- chgrp users *
- ssh User@remotehost mkdir -p .ssh (User is the user that has access to the script that must execute, remotehost is dns or host name of remote linuxhost ) - this will create .ssh if it does not already exist.
- cat /opt/superna/.ssh/id_rsa.pub | ssh User@remotehost 'cat >> .ssh/authorized_keys' (this places pub ssh keys into the remote users .ssh authorized keys file to allow passwordless login from a script).
- Enter password for User on remote host.
- Test SCA remote ssh:
- su sca
- ssh user@remotehost (if no pwd requested the setup is complete) .
- Done.
For Windows to copy public ssh keys:
- cat /opt/superna/.ssh/id_rsa.pub (to get the content of the id_rsa.pub file)
- Copy the content of id_rsa.pub file to .ssh/authorized_keys file on windows server (this places pub ssh keys into the remote users .ssh/authorized keys file to allow passwordless login from eyeglass). We can use text editor to copy the content of id_rsa.pub file to the authorized_keys file
OpenSSH for Windows : Allowing SSH connection using public ssh key
Edit sshd_config file (default path: C:\ProgramData\ssh\sshd_config) and set this parameter to yes
PubkeyAuthentication yes
Module Codes
Main Python Code
This main python code is stored and executed from Eyeglass.
As the user on Eyeglass used for establishing passwordless SSH to remote machines is sca, we can place the main python code under /opt/superna folder that is owned by sca user.
Example
Path | Purpose |
/opt/superna/mycode | Place the main python code in this directory |
/opt/superna/mycode/mylogs | Create the sub-directory for placing the gathered logs |
===
#!/usr/bin/python3
import urllib3, subprocess
urllib3.disable_warnings()
import sys
#Linux Commands
LINCMD1 = "cat"
LINCMD2= "/etc/os-release"
LINCMD3 = "python3 -V"
LIN_SCRIPT_DIR = "/<direcgtory-path>" # Specify the directory path where we store the script on a remote linux machine. Example: /home/myadmin/
LIN_SC1 = "script1.sh" # Specify the script file name
LIN_SC2 = "script2.sh" # Specify the script file name
LINCMD4 = LIN_SCRIPT_DIR + LIN_SC1
LINCMD5 = LIN_SCRIPT_DIR + LIN_SC2
#Windows Commands
WIN_SCRIPT_DIR = "/mytest1/" # Specify the directory path where we store the powershell script on remote windows machine
WIN_PS1 = "gather.ps1" # Specify the powershell script file name
WIN_PS2 = "compress.ps1" # Specify the powershell script file name
PS = "powershell.exe "
WINCMD1 = PS + WIN_SCRIPT_DIR + WIN_PS1
WINCMD2 = PS + WIN_SCRIPT_DIR + WIN_PS2
#PowerScale OneFS command
OFS1 = "isi version"
OFS2 = "isi status -v"
#ZIP Path
REMOTE_ZIP_PATH = ":/var/mytest*.zip" # Specify the remote zip path from linux machine
LOCAL_ZIP_PATH = "/opt/superna/mytest/testlogs" # Specify the local path in Eyeglass to store copy zip file
WIN_ZIP_PATH = ":/Users/Administrator/*_diagnostics.zip" # Specify the remote zip path from windows machine
#List of Machines refer to .ssh/config file
host = ['linserver1','linserver2' ] # linux servers
winhost = ['winserver1','winserver2'] # windows servers
isilon = ['isilon1','isilon2'] # PowerScale /Isilon clusters
def onefs_gather():
for i in range(len(isilon)):
result1 = subprocess.check_output(['ssh', isilon[i], OFS1])
print ("OneFS version : ", isilon[i], result1.decode("utf-8"), sep = "\n")
result2 = subprocess.check_output(['ssh', isilon[i], OFS2])
print ("OneFS Status : ", isilon[i], result2.decode("utf-8"), sep = "\n")
def lin_gather():
for i in range(len(host)):
result = subprocess.check_output(['ssh', host[i], LINCMD1 , LINCMD2])
print ("Host : ", host[i], result.decode("utf-8"), sep = "\n")
result2 = subprocess.check_output(['ssh', host[i], LINCMD3])
print ("Python version : ", host[i], result2.decode("utf-8"), sep = "\n")
result3 = subprocess.check_output(['ssh', host[i], LINCMD4])
print ("Network Info : ", host[i], result3.decode("utf-8"), sep = "\n")
result4 = subprocess.check_output(['ssh', host[i], LINCMD5])
print ("System Info : ", host[i], result4.decode("utf-8"), sep = "\n")
result5 = subprocess.run(['scp', host[i] + REMOTE_ZIP_PATH , LOCAL_ZIP_PATH])
def win_gather():
for i in range(len(winhost)):
result = subprocess.check_output(['ssh', winhost[i], WINCMD1])
print ("WinServer : ", winhost[i], result.decode("utf-8"), sep = "\n")
result2 = subprocess.check_output(['ssh', winhost[i], WINCMD2])
print ("WinServer : ", winhost[i], result2.decode("utf-8"), sep = "\n")
result3 = subprocess.run(['scp', winhost[i] + WIN_ZIP_PATH , LOCAL_ZIP_PATH])
def main():
onefs_gather()
lin_gather()
win_gather()
main()
Locally Stored Commands
On Linux Server
The following scripts are placed on the remote Linux servers. The main code on the Eyeglass appliance will call and execute these scripts on Linux servers.
The path location and the name of the scripts should also be reflected in the main code
LIN_SCRIPT_DIR = "/<direcgtory-path>" # Specify the directory path where we store the script on a remote linux machine. Example: /home/myadmin/
LIN_SC1 = "script1.sh" # Specify the script file name
LIN_SC2 = "script2.sh" # Specify the script file name
Example:
Folder location: /home/myadmin/
Script #1 : /home/myadmin/script1.sh
Script #2 : /home/yadmin/script2.sh
script1.sh
This script is stored in the Linux server to be run remotely to gather the network information and also to zip the /var/log directory
===
ifconfig
cd /var
zip -r "mytest-$(hostname)-$(date +"%Y-%m-%d-%T").zip" log/
===
script2.sh
This script is stored in the Linux server to be run remotely to gather some other system information.
===
#!/bin/bash
echo -e "-------------------------------System Information----------------------------"
echo -e "Hostname:\t\t"`hostname`
echo -e "uptime:\t\t\t"`uptime | awk '{print $3,$4}' | sed 's/,//'`
echo -e "Manufacturer:\t\t"`cat /sys/class/dmi/id/chassis_vendor`
echo -e "Product Name:\t\t"`cat /sys/class/dmi/id/product_name`
echo -e "Version:\t\t"`cat /sys/class/dmi/id/product_version`
echo -e "Serial Number:\t\t"`cat /sys/class/dmi/id/product_serial`
echo -e "Machine Type:\t\t"`vserver=$(lscpu | grep Hypervisor | wc -l); if [ $vserver -gt 0 ]; then echo "VM"; else echo "Physical"; fi`
echo -e "Operating System:\t"`hostnamectl | grep "Operating System" | cut -d ' ' -f5-`
echo -e "Kernel:\t\t\t"`uname -r`
echo -e "Architecture:\t\t"`arch`
echo -e "Processor Name:\t\t"`awk -F':' '/^model name/ {print $2}' /proc/cpuinfo | uniq | sed -e 's/^[ \t]*//'`
echo -e "Active User:\t\t"`w | cut -d ' ' -f1 | grep -v USER | xargs -n1`
echo -e "System Main IP:\t\t"`hostname -I`
echo ""
echo -e "-------------------------------CPU/Memory Usage------------------------------"
echo -e "Memory Usage:\t"`free | awk '/Mem/{printf("%.2f%"), $3/$2*100}'`
echo -e "Swap Usage:\t"`free | awk '/Swap/{printf("%.2f%"), $3/$2*100}'`
echo -e "CPU Usage:\t"`cat /proc/stat | awk '/cpu/{printf("%.2f%\n"), ($2+$4)*100/($2+$4+$5)}' | awk '{print $0}' | head -1`
echo ""
echo -e "-------------------------------Disk Usage >80%-------------------------------"
df -Ph | sed s/%//g | awk '{ if($5 > 80) print $0;}'
echo ""
echo -e "-------------------------------For WWN Details-------------------------------"
vserver=$(lscpu | grep Hypervisor | wc -l)
if [ $vserver -gt 0 ]
then
echo "$(hostname) is a VM"
else
cat /sys/class/fc_host/host?/port_name
fi
echo ""
echo -e "-------------------------------Oracle DB Instances---------------------------"
if id oracle >/dev/null 2>&1; then
/bin/ps -ef|grep pmon
#then
else
echo "oracle user does not exist on $(hostname)"
fi
echo ""
if (( $(cat /etc/*-release | grep -w "Oracle|Red Hat|CentOS|Fedora" | wc -l) > 0 ))
then
echo -e "-------------------------------Package Updates-------------------------------"
yum updateinfo summary | grep 'Security|Bugfix|Enhancement'
echo -e "-----------------------------------------------------------------------------"
else
echo -e "-------------------------------Package Updates-------------------------------"
cat /var/lib/update-notifier/updates-available
echo -e "-----------------------------------------------------------------------------"
fi
===
On Windows Servers
The following scripts are placed on the remote Windows servers. The main code on the Eyeglass appliance will call and execute these scripts on Windows servers.
The path location and the name of the scripts should also be reflected in the main code
WIN_SCRIPT_DIR = "/mytest1/" # Specify the directory path where we store the powershell script on remote windows machine
WIN_PS1 = "gather.ps1" # Specify the powershell script file name
WIN_PS2 = "compress.ps1" # Specify the powershell script file name
Example:
Folder location: C:/mytest1, we specify the variable as WIN_SCRIPT_DIR = "/mytest1/"
Script #1 : C:/mytest1/gather.ps1
Script #2 : C:/mytest1/compress.ps1
gather.ps1
This powershell script is to gather event logs, system information and network information of Windows Server.
===
# clean previous outfile
$folder = Test-Path -PathType Container -Path "c:\$env:computername"
if ($folder -eq $true) {
Remove-Item -Path "c:\$env:computername" -Force -Recurse | Out-Null
}
Write-Host ""
# Event Logs in evtx format
New-Item -Path "c:\$env:computername\eventlogs" -ItemType "directory" | Out-Null
Set-Location -Path "c:\$env:computername\eventlogs"
Write-Host "Retrieving Event Logs unfiltered." -ForegroundColor Yellow
wevtutil epl System "systemlog.evtx"
wevtutil epl Setup "setuplog.evtx"
wevtutil epl Security "securitylog.evtx"
wevtutil epl Application "applicationlog.evtx"
Write-Host " Completed .evtx log files." -ForegroundColor Green
# system information
Set-Location -Path "c:\$env:computername"
Write-Host "Retrieving MSInfo32 information. This will take some time to complete. Please wait..." -ForegroundColor Yellow
msinfo32 /report msinfo32.txt | Out-Null
Write-Host "Completed MSInfo32 information." -ForegroundColor Green
Write-Host ""
# Network information
New-Item -Path "c:\$env:computername\network" -ItemType "directory" | Out-Null
Set-Location -Path "c:\$env:computername\network"
Write-Host "Retrieving Network information..." -ForegroundColor Yellow
Get-NetAdapter | Format-Table Name,ifIndex,Status,MacAddress,LinkSpeed,InterfaceDescription -AutoSize | Out-File "Get-NetAdapter.txt"
Get-NetAdapterAdvancedProperty | Format-Table DisplayName, DisplayValue, ValidDisplayValues | Out-File "Get-NetAdapterAdvancedProperty.txt" -Width 160
Write-Host "Completed Network information." -ForegroundColor Green
Write-Host ""
===
compress.ps1
This powershell script is to compress the collected data on the Windows Server into a zip file.
===
Get-Module
$ProgressPreference = "Ignore"
$compress = @{
Path = "c:\$env:computername"
CompressionLevel = "Optimal"
DestinationPath = $env:computername + (get-date -Format -yyyyMMdd-hhmmss) + "_diagnostics.zip"
}
Compress-Archive @compress -Force
Write-Host "Completed folder compression." -ForegroundColor Green
===
Eyeglass SSH config File
SSH config file that contains the list of remote machines is stored in the Eyeglass appliance with the path: /opt/superna/.ssh/config
Populate this config file with the list of PowerScale cluster, Linux Server, Windows Server for passwordless SSH connections. Follow this procedure to configure passwordless SSH connections to those machines
The following example is based on this list
Machine Type | Server | Configured User for passwordless SSH |
Linux | linserver1 | root |
linserver2 | root | |
Windows | winserver1 | administrator |
winserver2 | administrator | |
PowerScale | isilon1 | eyeglass |
isilon2 | eyeglass |
===
Host linserver1
HostName <IP address of Linux Svr1>
User root
Host linserver2
HostName <IP address of Linux Svr1>
User root
Host winserver1
HostName <IP address of win Svr1>
User administrator
Host winserver2
HostName <IP address of win Svr2>
User administrator
Host isilon1
HostName <IP address of Isilon 1>
User eyeglass
Host isilon2
HostName <IP address of Isilon 2>
User eyeglass
===
- Autonomous Storage Failover
- Application Server
- Storage Configuration
- DFS Namespace
- Eyeglass DR
- Eyeglass DFS Mode Failover
- Eyeglass Access Zone Failover
- Creating Eyeglass REST API Token
- Using Curl Command to Retrieve IDs Related to Failover
- Autonomous Failover Code
- Failover Requirement Condition - Monitoring and Fault Detection
- E-mail Alert Notification Configuration
- Trigger Failover from Eyeglass
- Code Variable for Failover
- Autonomous Failover Code in PowerShell
- Autonomous Failover Code in Python
- Pre and Post Failover Step
- Pre Failover Script (Unmount) Example
- Post Failover Script (mount) Example
- umount.sh - Example script
- mount.sh - Example script
- Best Practices
- Passwordless SSH Configuration
- OpenSSH for Windows : Allowing SSH connection using public ssh key
- Module Codes
- Main Python Code
- Locally Stored Commands
- On Linux Server
- On Windows Servers
- Eyeglass SSH config File