Operations Guides

All Products Hardening Guide

Home



How to use this Guide

This guide provides additional security hardening steps that are optionally applied to one or more products as indicated below.   Not all sections apply to each product.   OS Customizations are provided as is without support under the support contract.  All OS customizations are not backed up and will not be migrated to a new appliance.

Securing Eyeglass, ECA, Search & Recover and Golden Copy by Applying OS Patches

  1. Before scanning the appliance with security tools the following steps must be taken:
    1. Upgrade to the latest OVA operating system using backup and restore to get web server configured with default hardening.  Upgrade guide.  Follow the backup and restore steps.  NOTE:  Os patching is not covered by the support contract and is customer responsibility.
    2. Patch the operating system (Requires internet access to the appliance to reach OS internet repositories)
      1. login to eyeglass as admin user
      2. sudo -s  (enter admin password)
      3. zypper refresh  (updates repositories)
      4. zypper update  (applies patches)
      5. Review any messages that indicate a reboot is required to have the update take effect
    3. Use Eyeglass service account and review all information to make sure permissions are up to date
      1. Reference: PowerScale Cluster User Minimum Privileges for Eyeglass
    4. Subscribe to OS updates and change patching to use https see this link.


General Purpose OS Advanced Hardening (All products)

  1. Grub password -- It is recommended to secure VM console access.
    1. Follow opensuse instructions if want to set a password
    2. sudo -s
    3. yast
    4. System --> boot loader --> boot loader options tab  (use arrow keys to select )
    5. enter a password
    6. NOTE: This will require password to make any changes to the boot loader it will not require a password to boot the OS.
    7. To enable boot password uncheck the option below "Protect Entry Modifications Only"
  2. Disable ICMP Redirects
    1. sudo -s
    2. nano /etc/sysctl.conf
    3. add these entries to the file and save the file, then control+x to save.   Then reboot the OS with reboot command.
    4. NET.IPV4.CONF.ALL.ACCEPT_REDIRECTS = 0
    5. NET.IPV4.CONF.ALL.SEND_REDIRECTS = 0
    6. NET.IPV6.CONF.ALL.ACCEPT_REDIRECTS = 0
    7. NET.IPV6.CONF.ALL.SEND_REDIRECTS = 0
  3. User home directory Hardening 
    1. sudo -s
    2. cd /home
    3. chmod 750 admin ecaadmin screenshots


How to add a Signed Certificate to the WebUI's (Eyeglass,  ECA Cluster, Golden Copy, Search & Recover)

  1. Eyeglass VM Steps
    1. Follow the TLS cert steps in the admin guide here.
  2. For ECA, Golden Copy, Search & Recover follow these steps.
    1. Access the WebUI from node 1 and create a DNS entry for node to create a FQDN to create a signed cert.  The objective is to install the signed cert for nginx ECA Node-1
    2. Create A record in DNS name for ECA Node-1 and verify with nslookup. Example eca1.domain.com
    3. SSH to ECA Node-1 as ecaadmin
      1. cd /opt/superna/eca/conf/nginx
      2. Verify that the nginx.key is there with ls -la
    4. Create csr with that key file
      1. Command: openssl req -key nginx.key -new -out nginx.csr
      2. SCP the nginx.csr file for signing
      3. Or type cat nginx.csr and copy and paste the text to submit for signing.
      4. When it is asked about the Common Name: provide the fqdn of ECA Node-1 (the name registered in DNS e.g. search.domain.com)
    5. With that CSR certificate submit the request to Certificate Authority at your enterprise
    6. NOTE: These steps are CA specific, consult with your security team
    7. Once received the signed certificate encoded in PEM format
    8. scp (use WinSCP for Windows) and copy this file to ECA-1 under /opt/superna/eca/conf/nginx with name nginx.crt
    9. NOTE: if not in PEM format, convert to PEM format or ask your Security team for pem format
    10. Replace existing nginx.crt certificate with this new signed CA certificate.
    11. mv nginx.crt nginx.crt.bak (backup old file)
    12. cp /pathtonewfile/nginx.crt to /opt/superna/eca/conf/nginx/nginx.crt
    13. Restart nginx
    14. Bring down and up the ECA cluster to push the config to all the other ECA nodes
      1. ecactl cluster down
      2. ecactl cluster up
    15. Verify the certificate when accessing the UI (e.g. https://FQDN)


Web Server HTTP Hardening Directives for Eyeglass and Search & Recover and Golden Copy

  1. This section has specific web server directives that address specific hardening http header responses and setting the Search & Recover TLS protocol requirements.  NOTE:  Not required for Eyeglass 2.5.7 or later as these are set by default.
  2. Eyeglass WebUI 

    1. For Eyeglass lighttpd https and http HEADER fix
    2. login as admin
    3. sudo -s (enter admin password)
    4. nano /etc/lighttpd/lighttpd.conf 
    5. Add the following inside SERVER 443 block
    6. control+w and type ":443"  [enter key]
    7. Add the text below and replace the previous section for this between the ( )  for this section setenv.add-response-header 
       
             setenv.add-response-header = (
                      "Strict-Transport-Security" => "max-age=15768000",
                       "Content-Security-Policy" => "frame-ancestors 'self';",
                       "X-Content-Type-Options" => "nosniff",
                        "X-Frame-Options" => "DENY",
                        "X-XSS-Protection" => "1; mode=block"
                )
  1. See example 
  2. Now locate the section for http (only used to redirect to 443 port)
    1. control+w  and type ":80" [enter key]
    2. Add the text below and replace the previous section for this between the ( )  setenv.add-response-header
    3. setenv.add-response-header = (
                            "Strict-Transport-Security" => "max-age=15768000",
                            "Content-Security-Policy" => "frame-ancestors 'self';",
                            "X-Content-Type-Options" => "nosniff",
                            "X-Frame-Options" => "DENY",
                            "X-XSS-Protection" => "1; mode=block"
                       )
  1. save the file with control+x answer yes to save and exit
  2. Restart the web server
    1. systemctl restart lighttpd.service
  3. Verify with Google Chrome Developer tools (Press F12 ).
    1. Login to eyeglass, select Network tab, select eyeglass web page on left side, click headers tab , expand response headers,  verify "content-security-policy"  and "strict transport security"   Use the screen shot as per below. 

HTTPS Security Algorithm Hardening Eyeglass DR < 2.5.7


  1. Update Java HTTPS algorithms and certificate settings
  2. nano /opt/superna/java/jre/lib/security/java.security
  3. press control+w
  4. type jdk.tls.disabledAlgorithms  and the press enter
  5. remove the #  comment from this line see image below
    1.    
  6. Then press control+W 
  7. type jdk.certpath.disabledAlgorithms   [enter]
  8. repeat control+w [enter] 3 times until you locate the line with the # comment on the lines below
  9. remove the # from both lines , refer to the image below.
    1.       
  10. press control+x  
  11. answer yes to save the file and exit
  12. For changes to take effect restart the sca
  13. systemctl restart sca
  14. done 

HTTPS Security Algorithm Hardening Eyeglass DR 2.5.7 >

  1. ssh as admin to eyeglass 
  2. backup existing file
    1. mv /opt/superna/java/jre/lib/security/java.security  /opt/superna/java/jre/lib/security/java.security.bak 
  3. Copy enhanced security file 
    1. cp /opt/superna/java/java.security.enhanced /opt/superna/java/jre/lib/security/java.security
    2. sudo chown sca:users java.security
  4. systemctl restart sca
  5. done

Overview:   Port 80 is only used to redirect to to port 443, it is not used for anything else.  To block port 80 follow these steps

  1. Login to Eyeglass appliance as admin user. Elevate to root using command: sudo su -
  2. Create Eyeglass port 80 firewall script:
    1. nano /opt/superna/bin/firewall-rules.sh
  3. Type 'i' to enter insert mode. Copy and paste the following to the file:
    1. #!/bin/bash
    2. iptables -I IN_public_deny -p tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable
  4. Type ESC and :wq! to exit when file contents match above.
  5. Change ownership and modify the file:
    1. chown sca:users /opt/superna/bin/firewall-rules.sh ; if [ "$?" == 0 ]; then echo Success; fi
    2. chmod u+x /opt/superna/bin/firewall-rules.sh ; if [ "$?" == 0 ]; then echo Success; fi
  6. Create a service file to be run at boot after Network service is registered as RUNNING:
    1. nano /etc/systemd/system/boot-firewall-rules.service
    2. [Unit]
    3. After=network.target
    4. [Service]
    5. ExecStart=/opt/superna/bin/firewall-rules.sh
    6. [Install]
    7. WantedBy=default.target
  7. Type ESC and :wq! to exit when file contents match above.
  8. Run the following commands for the changes to take effect (note: NO Reboot required. No impact):
  9. systemctl daemon-reload ; if [ "$?" == 0 ]; then echo Success; fi
  10. systemctl enable boot-firewall-rules.service
  11. systemctl start boot-firewall-rules.service ; if [ "$?" == 0 ]; then echo Success; fi
  12. systemctl status boot-firewall-rules.service
  13. done

HTTPs Web Server Hardening (Search & Recover and  Golden Copy)

  1. Login to Search & Recover over ssh as ecaadmin
    1. nano /opt/superna/eca/conf/nginx/eca.conf.template
  2. Login to Golden Copy
    1. nano /opt/superna/eca/conf/nginx/eca.conf.simpletemplate 
  3. Add the following inside server 443 block
    1. add_header Strict-Transport-Security "max-age=15768000";
      add_header Content-Security-Policy "frame-ancestors 'self';";
      add_header X-Content-Type-Options "nosniff";
      add_header X-Frame-Options "DENY";
      add_header X-XSS-Protection "1; mode=block";
      ssl_protocols TLSv1.2 TLSv1.3;


  4. Push the config to all nodes
    1. ecactl cluster push-config
  5. Restart containers to read the new configuration
    1. ecactl cluster exec "ecactl containers restart nginx"
  6. Verify with Google Chrome developer tools

  7. Done

How to turn off bash history (Eyeglass, Golden Copy, ECA, Search & Recover)

  1. Bash history can contain access key commands.   Disabling bash history disables command history.
  2. Login as ecaadmin  (Golden copy)  or admin (eyeglass)
  3. history -c   (cleans current history)
  4. echo 'set +o history' >> ~/.bashrc
  5. logout 
  6. done 


Hardening Password Complexity (All products)

Follow these steps to enable local password complexity of the builtin users admin, auditor and rwdefend. NOTE:  These settings only apply to the local OS users, if using RBAC proxy login to PowerScale or AD use the password features  of the PowerScale or AD to setup password complexity.

To set these password rules the - (minus number) means MUST have in the password.  Use the definitions below to customize the example provided.

  • Minimum password length should be x characters 
    • value minlen 
  • Password should have one UPPERCASE Character
    • value ucredit 
  • Password should have one LOWERCASE Character
    • value lcredit 
  • Password should have one Numeric Character
    • value dcredit 
  • Password should have Special characters
    • value ocredit 
  • Minimum Passwords to Remember or Password History 
    • value pwhistory-remember 
  • Accounts should be lockout after bad login attempts, see next section that blocks the source ip of the machine after failed local logins using fail 2 ban and firewall rules.
  1. Verify pam modules are installed (may not be required on all appliances depending on OS version, it may return no module found on 15.1 or later OS version which can be ignored and continue the steps)
  2. login as admin
  3. sudo -s
  4. enter admin password
  5. zypper install pam-modules  (this requires internet access to install additional pam modules)
  6. Answer yes to install new modules
  7. cd /etc/pam.d/
  8. cp common-password common-password.bak  (backup old password file rules)
  9. pam-config -a --cracklib --cracklib-minlen=6 --cracklib-lcredit=-1 --cracklib-ucredit=-1 --cracklib-dcredit=-1 --cracklib-ocredit=-1 --pwhistory --pwhistory-use_authtok --pwhistory-remember=3
    1. See definitions above for each value to customize
    2. This will generate a new common-password file
    3. When users try to change passwords they will require a password that matches these rules. NOTE the root user can set a password for a user account that does not match these rules.



Banning local user accounts after repeated failed login attempts

The appliance has several local users admin, auditor, and rwdefend used for builtin roles for different products. NOTE: the root user password is randomized and sudo access to root should be used and leave the password randomized.

To ban users that attempt brute force login attempts the following appliance enhancement allows control of lockouts and timed locked outs.  This will setup firewall rules to block the ip of the user.   The blocked login will cover ssh access and https to the WebUI.  NOTE:  If proxy login is used to AD or PowerScale local users, using the RBAC features, these users will also be banned as well.


  1. Login as admin 
  2. sudo -s
  3. enter admin password
  4. zypper install fail2ban  (requires Internet access to the appliance)
  5. systemctl start fail2ban


Configuration Steps for Eyeglass

Highlevel:

  • modified /etc/fail2ban/jail.conf [added 'eyeglass' section]
  • enabled eyeglass filtering from /etc/fail2ban/jail.local
  • added 'eyeglass' custom filter file in /etc/fail2ban/filter.d/ directory
  1. nano /etc/fail2ban/filter.d/eyeglass.conf     (add the contents below to the file and save the file with :wq)
    1. # Fail2ban filter for Superna Eyeglass
      #
      #

      [INCLUDES]

      before = common.conf

      [Definition]

      failregex = <HOST> \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b - \[.* "POST /RestClient/login/login HTTP/1.1" 500
      datepattern = %%d/%%b/%%Y:%%H:%%M:%%S
      ignoreregex =
  2. nano /etc/fail2ban/jail.local
    1. add The following to this file
      1. [DEFAULT]
        ignoreip = 127.0.0.1/8
        bantime = 300
        findtime = 300
        maxretry = 3

        [sshd]
        enabled = true

        [eyeglass]
        enabled = true

    2. Modify /etc/fail2ban/filter.d/sshd.conf file
      1. sed -e /'spam_unix/s/^/#/g' -i /etc/fail2ban/filter.d/sshd.conf
    3. Modify /etc/fail2ban/jail.conf to add eyeglass jail rule
      1. sed -i "/HTTP servers/a[eyeglass]\n \nport = http,https\nlogpath = /var/log/lighttpd/access.log" /etc/fail2ban/jail.conf
    4. restart the service
      1. systemctl restart fail2ban
      2. check status 
      3. systemctl status fail2ban
    5. Optional - Find bantime and change default from 300 seconds to a value that meets your requirements
    6. Optional - Find findtime and change default from 600 to a value that meets your requirements (A host is banned if it has generated "maxretry" during the last "findtime")
    7. Optional -  Find maxretry and change default from 3 to a value that meets your requirements
  3. Save the file after changes control+x  answer yes to save
  4. done.


Configuration Steps for Golden Copy & Search & Recover

There is a fail2ban folder under /opt/superna/eca/conf/fail2ban in builds 1.1.4 > 20300.  There is also a default jail.local, the following is its default content below. After modifying any conf under fail2ban folder the fail2ban container must be restarted to activate the new configuration.

How to edit the configuration defaults

  1. ssh to the node as ecaadmin
  2. nano /opt/superna/eca/conf/fail2ban/jail.local
  3. edit the bold settings shown below to adjust ban time and retries for webui and ssh
  4. control + x to save and exit
  5. ecactl container restart fail2ban (for changes to take effect) 

Default configuration:

  1. banned logins will be 300 seconds or 5 minutes
  2. 5 retries of the password will be allowed before banning from the webui login
  3. 5 retries of the ssh password will be allowed before banning.
  4. nano /opt/superna/eca/conf/fail2ban/jail.local 


[nginx-http-auth]

enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log
maxretry = 5
bantime = 300
chain = DOCKER-USER

[sshd]

enabled = true
backend = systemd
filter = sshd
maxretry = 5
bantime = 300

[searchmw]
enabled = true
filter = searchmw
port = http,https
logpath = /var/log/searchmw/loginError.log
maxretry = 5
bantime = 300
chain = DOCKER-USER


Eyeglass WebUI Security API Auditing

This feature is available in 2.5.7 or later releases.  This allows a user GUI audit log of which UI functions are accessed by logged in users.  Combined with the web server access log the user name and ip address can be located for any UI actions taken by any user including proxy login users.


How to monitor user UI actions and Authentication Login

  1. The audit log can be monitored from an ssh session on the eyeglass appliance.   
  2. Login as admin over ssh
  3. Run this command to monitor user interface login in real-time
  4. tail -n 100 -f /opt/superna/sca/logs/apiaudit.log
  5. You can also search through this log with grep example
  6. grep "rsw" /opt/superna/sca/logs/apiaudit.log  (this will locate all the api calls sent to the Ransomware Defender application UI icon.  Each application has a name in the log that can be used to look for set or get or view functions.)
  7. Example log output
    1.  
  8. How to view the web server access log to locate IP address information
    1. The web server access log can be used to locate the IP address of a logged user.  This requires matching the time stamps in the api access log to the web server time stamps to find the source ip address of the user session.
    2. Using ssh to the eyeglass appliance as admin
    3. sudo -s  (enter admin password to become root)
    4. cd /var/log/lighttpd
    5. ls  (list the files and view the file with recent date stamp)
    6. Then use cat or grep or tail to view the log file 
    7. Example log below
      1.  

How to view the GUI API Log

  1. Login as admin
  2. sudo -s (enter admin password)
  3. journalctl -u scagateway
    1. This command will return all client browser api calls


ECA VM Hardened Virtual Secured Network (Ransomware Defender, Easy Auditor, Performance Auditor)

  1. The Eyeglass and ECA installation and admin guides list firewall ports and directions required including management PC access to UI's.  This feature will automatically secure the communications between Eyeglass and the ECA vm's.    No open ports will be returned from ECA vm's with the exception of SSH and HTTPS.  This creates a virtual secure network between Eyeglass and the ECA vm's with no external access to any ports.   This feature is automatically enabled and configured.
  2. Requirements:
    1. 2.5.7 update 1
  3. This release adds 2 new security features.
    1. Automatic Firewall for Superna VM's: The ECA VM's need to be accessed by eyeglass over various ports.  The installation of the ECA and cluster up process will apply IP tables firewall rules to only allow access to ECA ports from Eyeglass VM and between ECA VM's.  This provides a secure network between Superna VM's without requiring customer infrastructure.  This will be applied automatically.
    2. Authenticated Management UI's - Various management UI's on ECA nodes will be accessed through a HTTPS proxy built into the ECA nodes that will require authentication.  None of the ECA UI's will be directly accessible.  



2 Factor SSH Authentication for Eyeglass, Golden copy, Search & Recover or ECA VM's

  1. This procedure only secures SSH access to VMs.   
  2. Requirements:
    1. Google Authenticator application on a mobile phone
      1. IOS
      2. Android
  3. Installation
    1. ssh to the VM 
    2. sudo -s  (enter password)
    3. Install the pam module
    4. zypper in google-authenticator-libpam
    5. answer yes
    6. To run the initialization app
      1. google-authenticator
      2. Do you want authentication tokens to be time-based (y/n)   y
      3. NOTE: Very important step to complete
        1. You will be presented with a secret key used in the step below and multiple scratch codes. We strongly suggest saving these emergency scratch codes in a safe place, like a password manager. These codes are the only way to regain access if you lose your phone or lose access to your authenticator application, and each one can only be used once, so they really are in case of emergency. 
      4. Activate Google Authenticator application with output from the step above that shows the secret key.
        1. Click the plus to add a new profile and select add setup key
        2.   
        3. Now enter the name of this VM for account name example Eyeglass and the secret key number output to the console  from the step above.
        4.  
        5. Your client is now configured
      5. Do you want me to update your "~/.google_authenticator" file (y/n)  y
      6. Do you want to disallow multiple uses of the same authentication
        token? This restricts you to one login about every 30s, but it
        increases your chances to notice or even prevent
        man-in-the-middle attacks (y/n)   (This part is a time-based login. We suggest answering ‘yes’ (y) here since this will prevent a replay attack, allowing you 30 seconds from the point of getting the code on your mobile to typing in your login prompt)
      7. By default, tokens are good for 30 seconds and in order to
        compensate for possible time-skew between the client and the server,
        we allow an extra token before and after the current time. If you
        experience problems with poor time synchronization, you can increase
        the window from its default size of 1:30min to about 4min.
        Do you want to do so (y/n)  (Answer yes for more secure, answer No to allow 8 valid codes in a 4:00-minute rolling window)
      8.  If the computer that you are logging into isn't hardened against
        brute-force login attempts, you can enable rate-limiting for the
        authentication module. By default, this limits attackers to no more
        than 3 login attempts every 30s.
        Do you want to enable rate-limiting (y/n)  (yes is more secure)
    7. Configuring OpenSSH
      1. nano /etc/pam.d/sshd
      2. add this line to the file 
        1. auth required pam_google_authenticator.so
      3. control + x to save
      4. nano /etc/ssh/sshd_config
      5. find this line and remove the comment at the front of the line
        1. ChallengeResponseAuthentication yes
      6. control + x to save
    8. Activate
      1. systemctl restart sshd
    9. Test login
    10. ssh to the vm and enter the admin user password.  You will now be prompted to enter a one time number called the verification code.
    11.   
    12. Use the Google Authenticator application and type in the code displayed in application.   You have 30 seconds to enter the code to login successfully.
    13. NOTE:  The settings above determine what happens if you enter a bad or out of sync verification code.  You may get rate limited to login if you have failed login attempts
    14. done.




© Superna LLC