Technical Advisories

Technical Advisory #1

Technical Advisory #2

Technical Advisory #3 - reissued April 20

Technical Advisory #4 - End of Support for 1.2, 1.3, 1.4 and 1.4.x Releases

Technical Advisory #5 - Incomplete response from PowerScale PAPI may result in deletion of Configuration Objects

Technical Advisory #6 - Set SyncIQ policies to manual schedule or longer schedule with 1.6 release or risk scheduled jobs running during failover and failing overall Eyeglass failover steps

Technical Advisory #7 - OneFS 8 BMC firmware bug API call

Technical Advisory #8 - End of Support for 1.5.4 Release

Technical Advisory #9 - Open files Detection on PowerScale

Technical Advisory #10 - Uncontrolled Failover Issue when PowerScale Cluster added to Eyeglass with FQDN

Technical Advisory #11 - End of Support for 1.6.x Release Notice as of May 31, 2017

Technical Advisory #12 - DR Dashboard/Zone Readiness display issue for Failed Over Status - no loss of failover functionality

Technical Advisory #13 - Spectra/Meltdown Available in Appliance 2.5.x

Technical Advisory #14 - EOS 1.9.x Releases

EOS for Releases 1.9.x March 20, 2018 

Technical Advisory #15 - cross site scripting CVE on PowerScale 

cross site scripting vulnerabilities in CVE-2017-8024 requires a patch from PowerScale available for certain PowerScale release streams.  This patch also requires disabling HTTP authentication which will affect Eyeglass directly.   Eyeglass will not able to login to perform any DR functions.   An updated version 2.5.3 will include alternate authentication solution for PowerScale to allow this complete CVE procedures to be applied.   Partial implementation is possible by applying the path to PowerScale without disabling the HTTP authentication on the cluster 

---

Technical Advisory #16 - Config Sync may skip steps on Error

This issue is present in > 1.9.4 and can result in SMB shares and export not synced to DR if an object cannot be synced remaining objects are not attempted.  

Resolution:  Clear the error and all unsync changes will be synced successfully.  A work around exists and requires a support case opened for instructions.   This has been fixed in release 2.5.3 once it is GA.  We will also be releasing a patch to 2.5.2 that corrects this issue. The patch can be downloaded from the download page on located on the support site and will require download of the 2.5.2 offline installer and upgrading from 2.5.2 to latest build of 2.5.2 for an existing installation.

---

Technical Advisory #17 - PowerScale CSRF Authentication is not compatible with Smartconnect and API services Affects Eyeglass releases 2.5.3 and later  

NOTE: Only 2.5.3 and later supports a cluster with CSRF enabled.

Issue: PowerScale does NOT support multi node cluster aware CSRF sessions for authentication and is NOT compatible with Smartconnect FQDN method #1.  This is known limitation of PowerScale CSRF implementation.

Impact: None. SSIP is fully supported with many deployments using this method.  The load balancing of FQDN has some potential to expose issues on various nodes in the cluster and a minor performance improvement for large object customers.

Resolution:  To support CSRF on PowerScale requires settings that disable basic HTTPS authentication and uses session tokens for authentication.  This requires Eyeglass to use the SSIP in the management access zone due to above PowerScale limitation.

1. If you have added clusters to Eyeglass with FQDN (How to check:  Open Inventory Icon, right click the cluster and select Edit to determine how Eyeglass was added).
2. Open Jobs icon, record all policy states in all sections of this window. You will need to re-enable these policies and enable DFS mode based on your records from this step.
3. Delete the cluster (right click menu on the cluster name, chose the delete option)
   1. Repeat the delete for each cluster listed in the inventory icon
   2. Re-add the cluster with the SSIP in the system access zone, enter eyeglass service account name used before and password
   3. Repeat for each cluster Note:  more than one cluster can be added before submitting the inventory job
   4. Open Jobs Icon, then running jobs tab,  wait for initial inventory to complete
   5. Once complete click on job definition tab and enable all jobs based on the recorded configurations from the step above.
   6. You may also need to enable DFS mode (How to enable DFS mode
   7. Use the bulk actions menu to enable the jobs (How to enable jobs)
   8. Then wait 5 minutes and verify all jobs are green
   9. If any errors, please open a support case.

---

Technical Advisory #18 - Ransomware Defender ECA cluster without Internet access potential for false positives

Issue: In release 2.5.3 of Ransomware Defender a threat detector will sometimes match files that should not be considered Ransomware only when the ECA cluster does not have Internet access.

Impact: False positive detection of some users depending on the IO pattern.

Resolution:  A new build of 2.5.3 with number 18257 is available for download now that addresses this issue, without any requirement to connect ECA clusters to the Internet. 

1. Instructions to apply patch
2. Open a support case and request assistance to apply the patch.

 

---

Technical Advisory #19 - SMB Data Integrity corner case can leave Deny permission on some or all shares

Issue:  Normal configuration sync runs every 5 minutes and under some conditions may detect the Deny everyone permission used by the Data Integrity DR Assistant failover feature and sync it to the DR cluster before the deny is removed by the failover process.  This is a corner case that has rare. 

Impact:  After a failover some SMB shares may be left with a deny everyone permission blocking access to users.

Solution: Manually remove the Deny Everyone permission on the affected shares.  

Solution to Use SMB Data Integrity Feature:   

1. Disable the config Sync Job before the failover
   1. using SSH to the Eyeglass appliance run:
   2. igls admin schedules set --id Replication --enabled false
      
2. After the failover is complete
   1. igls admin schedules set --id Replication --enabled true
      

Resolution:  A new build of 2.5.4 with number 18275 has been released to address this issue. 

---

Technical Advisory #20 - PowerScale Auditing incorrectly records audit events for paths that do not exist on the cluster when SMB share is mounted with subfolders with a case that does not match the file system

Issue:  Mounting below the SMB share of a file system path of \ifs\data\temp and the mount example \\dnsname\sharename\TEMP  where temp is a subfolder will be audited incorrectly and recoreded as an audit event that does not exist in the file system.  The audit event will be incorrectly created as /ifs/data/TEMP even though this path does not exist in the file system.   PowerScale file system is case senstive.   

NOTE:  If all SMB share mounts mount the share only and does not include subfolders this issue will not be seen, or if the subfolders of the mount matches the case of the actual file system.  NOTE: Onefs is a case senstive file system,  NFS does not have this issue and denies mounts if the case does not match the actual file system case. NOTE: Mounting the SMB share only, will allow Windows OS to auto correct the case when browsing the file system with explorer or command prompt and will not have this issue.

Best Practise:  If sub share mounts are needed , the case should match the file system path.

Work Around: With Easy Auditor start searches from the share path, not the sub path, only if you know you have mismatched mounts with sub folders.

PowerScale fix for this bug:  No known fix planned  

---

Technical Advisory #21 - OneFS CLI command bug fails to exit ram 

Description of issue:  Onefs CLI command with a bug fails to exit after execution in some scenarios; the conditions for this to occur are unknown at this time.

A Onefs CLI command isi status has a bug that can cause the command to fail to exit memory.  A patch exists from PowerScale support for this bug.  This command is used by Eyeglass to collect node usage data and runs on a schedule.  Due to this bug, it is possible that this command may not exit ram and each new execution of the command will consume more ram on PowerScale nodes.  

As a proactive measure the following steps should be followed to avoid the potential of ram usage consumption due to the Onefs CLI bug.

Impact To Eyeglass DR:  No impact by following these steps, no loss of DR functionality.

Side Effects:  An alarm may be raised about CLI data incomplete which can be ignored.

Steps to follow:

Remove these lines from your sudoer file or place a # at the front of the line and save it.  

1. SSh to PowerScale
2. Edit the sudoer file using the PowerScale isi_visudo command.
3. Sudo file opens in vi editor.  place # in front of these lines
4. Save the file with :wq
5. eyeglass ALL=(ALL) NOPASSWD: /usr/bin/isi_for_array isi status*
6. eyeglass ALL=(ALL) NOPASSWD: /usr/bin/isi status*

---

Technical Advisory #22 - Onefs 8.2 Does not support REST API or SSH SSIP connections from Eyeglass

1. Onefs releases 8.2 and later no longer support API connections or SSH connections needed by Eyeglass.  If you are planning an upgrade to 8.2 the following steps should be completed before your Onefs Upgrade to 8.2.
   1. Upgrade to 2.5.5 latest release number - Upgrade guide.
   2. Change the ip address used to add the cluster to Eyeglass to use a node IP in a system zone pool.  The pool should be dynamic mode to ensure the IP will failover to other nodes.
   3. This is done with the Inventory Icon, right click , edit , change the ip address, re-enter password and save.
   4. Verify by opening the jobs icon and selecting a job with a check box, bulk actions , run now.  Then monitor in the running jobs icon.
   5. For more detailed steps open a support case with support to assist or provide health check after the change has been made.

Technical Advisory #23 - 2.5.6 Patch 1 Addresses failover Scenario's that can delay some steps from starting immediately

1. Some scenario's can cause delays in failover jobs starting to execute steps due to timeouts. An updated 2.5.6 patch addresses this issue.
2. Recommendation:  Upgrade to patch one following the offline upgrade steps here or open a support case for assistance.
3. Full release notes on patch 1 that includes additional fixes is available here 

Technical Advisory #24 Config Only Migration Job deletes shares and exports on the Destination Access Zone

In 2.5.6 build 84 or earlier this feature is a mirror mode and will remove any overlapping or non matching configuration data found on the target path or below. This will be changed in a patch release to default to merge mode and will leave all target configuration data as is and copy only new configuration to the target path.  Note that a config only migration job will run on regular configuration replication cycle and if you are in this situation it should be deleted as it will delete any share or export you recreated to solve this problem on the next run.

Technical Advisory #25 OneFS fails to create Quotas with corrupt configuration

During failover Quotas with corrupt / invalid configuration on the failover source cluster will be blocked from being created on the target cluster by OneFS.  In the Superna Eyeglass failover log this will display as a quota error.  For example a Quota on the source cluster with both Advisory and Advisory Threshold percentage configuration will be blocked by OneFS from being created on the target cluster with the message that only on advisory threshold option is supported.  It is unknown how a Quota would enter this state on the source cluster and Dell EMC should be contacted for troubleshooting/resolution of the corrupt quota configuration issue.

Work Around Patch Available from Dell

Technical Advisory #26 Eyeglass VM Log4j Hardening 

Eyeglass DR Edition Release with patched log4j version (click the About Eyeglass Icon to check your version)

1. 2.5.8 build 21330 -  Only this exact build number includes the log4j 2.17, all other version can use the remediation steps below.

Please Read the below Exposure Statement:   Nessus Scanner using the profile below scanning the following releases returns no exposure to the log4J.  This means the exploit would not succeed based on the scan results that attempt to use the exploit on all open IP ports.  The Eyeglass vm does not expose ports that can be used to exploit this log4j CVE using a port scan and attack attempt.

     2. Eyeglass VM  2.5.7.1 and 2.5.8 releases - with no modifications made to these versions - Pass a scan of the VM for the vulnerability

If you want to apply remediation to these product versions you can follow these steps below, read statement above.

To remove the CVE present in log4j the following steps can be done.

1. Login to eyeglass
2. sudo -s
3. Run this command
   1. find /opt/superna -name '*.jar' | xargs -I {} zip -q -d {} org/apache/logging/log4j/core/lookup/JndiLookup.class
   2. CVE-2021-4104
      1. find /opt/superna -name '*.jar' | xargs -I {} zip -q -d {} org/apache/log4j/net/JMSAppender.class 
4. Run the command a 2nd time
   1. find /opt/superna -name '*.jar' | xargs -I {} zip -q -d {} org/apache/logging/log4j/core/lookup/JndiLookup.class
   2. CVE-2021-4104
      1. find /opt/superna -name '*.jar' | xargs -I {} zip -q -d {} org/apache/log4j/net/JMSAppender.class
         
   3. If on the second execution all of the output states "zip error: Nothing to do!" this means the patch was applied.
5. Then run this command
   1. systemctl restart sca
6. done. 

Technical Advisory #27 - ECA VM Log4j Hardening - (Ransomware Defender, Easy Auditor, Performance Auditor)

Eyeglass ECA VM's patched log4j version (click the About Eyeglass Icon to check your version and the managed services icon to see the ECA version)

1. 2.5.8 build 21330 -  Only this exact build number includes the log4j 2.17, all other version can use the remediation steps below.

Please Read below Exposure Statement:   Nessus Scanner using the profile below scanning the following releases returns no exposure to the log4J.  This means the exploit would not succeed based on the scan results that attempts to use the exploit on all open IP ports.  The ECA does not expose ports that can be used to exploit this log4j CVE.

1. ECA (Ransomware Defender, Easy Auditor, Performance Auditor)  2.5.7.x releases - no modifications made to this version - Pass
2. ECA 2.5.8 - has updated log4j - Pass

If you want to apply remediation to these product versions you can follow these steps below, read statement above.

Supported versions:  

1. 2.5.7 and 2.5.8
2. All other versions will need upgrade.
3. NOTE: Make sure you have completed steps on technical advisory #26 above first on the eyeglass VM

1. Login to ECA node 1 as ecaadmin user
2. Run this command to get the product version to determine which file to download
   1. ecactl version
   2. If 2.5.7  download this file(Right click the link and select "Save link as" option, to avoid your browser opening the file)
   3. if 2.5.8 download this file  (Right click the link and select "Save link as" option, to avoid your browser opening the file) 
3. Using winscp tool (https://winscp.net/eng/download.php)
   1. Login to node 1 of the eca vm as the ecaadmin user with winscp and copy the correct version of the file downloaded in step #2 to this path /opt/superna/eca
4. On the ssh login to eca node 1 as the ecaadmin user
   1. cat /opt/superna/eca/docker-compose.overrides.yml
   2. If your file looks like screenshot then continue to step #5, Only if it does NOT look like below image, open a case to get additional information on how to merge override files.    When opening the case attach the text of the docker-compose-overrides.yml file using output above.
   3.   
5. Update the docker override file with the steps below.
   1. 2.5.8 release command (use the correct command for the release you are running) 
      1. cp /opt/superna/eca/docker-compose.CVE-2021-44228.2.5.8.yml  /opt/superna/eca/docker-compose.overrides.yml 
   2. 2.5.7 release command (use the correct command for the release you are running)
      1. cp /opt/superna/eca/docker-compose.CVE-2021-44228.2.5.7.yml /opt/superna/eca/docker-compose.overrides.yml 
6. Restart the ECA cluster
   1. ecactl cluster down
   2. ecactl cluster exec 'sudo rm -rf /opt/superna/mnt/zk-ramdisk/*'
      1. NOTE:  you will be asked to enter the ecaadmin password for each ECA VM to complete this command. 
   3. ecactl cluster up
   4. Watch the cluster up log for any errors until it completes
7. This will delete the docker containers and launch them all with the fix applied.
8. Verify with security guard and robo audit features normal operations
9. Done

Technical Advisory #28 - Search & Recover, Golden  Copy VM Log4j Hardening 

These products use docker containers and isolate code in containers with minimal OS inside the container. The containers are ephemeral and are destroyed when the shut down commands are run. The steps below provide a method to disable the java class inside the containers that is part of the log4j CVE.     The steps below adds additional hardening to the product VM's recommended for all customers.  

Please Read below Exposure Statement:   Nessus Scanner using the profile below scanning the following releases returns no exposure to the log4J.  This means the exploit would not succeed based on the scan results that attempts to use the exploit on all open IP ports.

1. Search & Recover 1.1.2, 1.1.5 release - no modifications made to this version - Pass
2. Golden Copy 1.1.6 no modifications made to this version - Pass 

This plugin is used in the scan template 'Log4Shell Vulnerability Ecosystem' as a way to include other plugins related to the Log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046.

- 155998 Apache Log4j Message Lookup Substitution RCE (Log4Shell) (Direct Check)

- 155999 Apache Log4j < 2.15.0 Remote Code Execution

- 156000 Apache Log4j Installed (Unix)

- 156001 Apache Log4j JAR Detection (Windows)

- 156002 Apache Log4j < 2.15.0 Remote Code Execution

- 156014 Apache Log4Shell RCE detection via callback correlation (Direct Check HTTP)

- 156017 SIP Script Remote Command Execution via log4shell

- 156016 Apache Log4Shell RCE detection via Path Enumeration (Direct Check HTTP)

- 156035 VMware vCenter Log4Shell Direct Check (CVE-2021-44228) (VMSA-2021-0028)

- 156032 Log4j EOL / Unsupported Apache Log4j Unsupported Version Detection

- 156056 Apache Log4Shell RCE detection via Raw Socket Logging (Direct Check)

- 156057 Apache Log4j 2.x < 2.16.0 DoS

If you want to apply remediation to these product versions you can follow these steps below, read statement above.

Supported versions:  

1. Search & Recover release 1.1.5
2. Golden Copy releases 1.1.6

1. Login to ECA node 1 as ecaadmin user
2. Run this command to get the product version to determine which file to download
   1. ecactl version
   2. Search & Recover
      1. If 1.1.2  download this file(Right click the link and select "Save link as" option, to avoid your browser opening the file)
      2. if 1.1.5 download this file  (Right click the link and select "Save link as" option, to avoid your browser opening the file) 
   3. Golden Copy
      1. if 1.1.4 or 1.1.6 download this file(Right click the link and select "Save link as" option, to avoid your browser opening the file)
3. Using winscp tool (https://winscp.net/eng/download.php)
   1. Login to node 1 of the eca vm as the ecaadmin user with winscp and copy the correct version of the file downloaded in step #2 to this path /opt/superna/eca
4. On the ssh login to eca node 1 as the ecaadmin user
   1. cat /opt/superna/eca/docker-compose.overrides.yml
   2. If your file looks like screenshot then continue to step #5,   Only if it does NOT look like below image, open a case to get additional information on how to merge override files.    When opening the case attach the TEXT of the docker-compose-overrides.yml file using output above. 
   3.   
5. Update the docker override file with the steps below.
   1. Search & Recover 1.1.2 release command (use the correct command for the release you are running)
      1. cp /opt/superna/eca/docker-compose.CVE-2021-44228.1.1.2.yml  /opt/superna/eca/docker-compose.overrides.yml 
   2. Search & Recover 1.1.5 release command  (use the correct command for the release you are running)
      1. cp /opt/superna/eca/docker-compose.CVE-2021-44228.1.1.5.yml /opt/superna/eca/docker-compose.overrides.yml 
   3. Golden Copy 1.1.4, or 1.1.6 release command (use the correct command for the release you are running)
      1. cp /opt/superna/eca/docker-compose.CVE-2021-44228.gc.yml /opt/superna/eca/docker-compose.overrides.yml 
6. Update configuration file ONLY for Search & Recover (both versions)
   1. nano /opt/superna/eca/eca-env-common.conf 
   2. add this line to the file (location in the file does not matter) copy and paste.
   3. export SOLR_JAVA_OPTS="-XX:MaxDirectMemorySize=8g -DformatMsgNoLookups=true"
   4. save the file with control+x  (answer yes to save the file) 
7. Restart the product cluster (Applies to Search & Recover and Golden Copy)d
   1. ecactl cluster down
   2. ecactl cluster up
   3. Watch the cluster up log for any errors until it completes
8. This will delete the docker containers and launch them all with the fix applied.
9. Verify Search & Recover by logging into the gui and run a search
10. Verify Golden Copy by running a copy command or login to the GUI
11. Done

Technical Advisory #29 CVE-2021-45046 log4j (2.16)  - Does not apply to any Superna product 

This CVE-2021-45046 (log4j 2.16)  does not apply to any Superna products.  This vulnerability does not apply since the MDC function is not used and this CVE requires that function to be used to be exposed.    

Nessus Scanner using this profile all pass 2.5.7.x with remediation applied and 2.5.8 with CVE patch all pass scans with no vulnerability.

Technical Advisory #30 How to Mitigate Vulnerabilities in JRE 1.8.0_291 in Eyeglass Appliance

These opensuse vulnerabilities exist in the embedded Java JRE

Applies to 2.5.7.x and 2.5.8.0 and 2.5.8.1

1. https://www.tenable.com/plugins/nessus/154345  CVE-2021-3517
   1. Login as admin
   2. sudo -s
   3. mv /opt/superna/java/jre/lib/ext/jfxrt.jar /opt/superna/java/jre/lib/ext/jfxrt.bak
   4. systemctl restart sca
2. Does not apply to Eyeglass (This vulnerability does not apply to Java deployments, typically in servers, that load and run only trusted code (e.g., code installed by an administrator)
   1. https://www.tenable.com/plugins/nessus/154345 
   2. https://www.tenable.com/plugins/nessus/152021
   3. https://www.tenable.com/plugins/nessus/148961
      

Technical Advisory #31 Ransomware Defender NFS Event Detection Always Enabled in 2.5.8.1-22080/22100

In Ransomware Defender 2.5.8.1-22080 and 2.5.8.1-22100, NFS Event Detection is always enabled and cannot be turned off.  For environments with NFS workload, even if NFS Event Detection was not previously enabled this can lead to new detections and possible unexpected lockout for NFS events if Ransomware Defender is in Enforcement mode.

NOTE: if you use only SMB, this advisory does NOT apply to you, you can still make the change below. NOTE: If you are in monitor mode,  no impact but the change below should be applied.

Workaround:

1. Login to node 1 of the eca cluster as ecaadmin
2. ecactl cluster down
3. nano /opt/superna/eca/eca-env-common.conf
4. add this variable
5. export TURBOAUDIT_PROTCOL_FILTER_ENABLD=true
6. Save and exit with control+x  (answer yes)
7. ecactl cluster up
8. Done.  The NFS lockout will be disabled. 

Resolution: This issue has been resolved as of 2.5.8.1-22116.  If you want NFS event detection enabled you must manually enable it as described here.

Technical Advisory #32 Onefs version 9.3 and later - papi session API bug affecting Golden Copy and Search & Recover

Overview

The session authentication behavior was changed in 9.3 and later Onefs releases and the change is not backwards compatible with previous session authentication.  This bug causes authentication sessions to fail which causes API calls to the cluster to fail.

Work Around

Set the session timeout values using the root user to a longer value

Onefs Patch Availability 

Technical Advisory #33 - Eyeglass REST API allows unauthenticated access

Using an API token that does not match a token in the authorized list, an unauthenticated user can access routes on the Eyeglass REST API by including a made-up token. All API routes are affected.

Affected Products:

All Eyeglass DR, Ransomware Defender, Easy Auditor, and Performance Auditor versions from 2.5.9-22219 through 2.5.11-23110

Impact: No risk to data on Powerscale, no risk to any configuration information within any Eyeglass product

Remediation

To temporarily block access to all but the heartbeat routes on the REST API:

1. edit the /etc/lighttpd/lighttpd.conf file

2. add the following block of text on line 419

$HTTP["url"] =~ "/sera/.*" {

  $HTTP["url"] !~ "/sera/v1/(healthcheck|ransomware/heartbeat)" {

    url.access-deny = ("")

  }

}

 3. Save the file

 4. Restart the lighttpd service: systemctl restart lighttpd

Patching

A new build of Eyeglass Release 2.5.11 with version number 23111 or higher is expected to be released in the coming 24 hours, that re-enables the authentication on the necessary routes.