Administration Guides

How to Configure Syslog Forwarding of Formatted Audit messages to an External Syslog Server

Home


Overview

This configuration is optional and only needed when events should be forwarded another logging system link Splunk or other logging tools.  The ECA can run an additional dock container that consumes events and formats for syslog forwarding.  This container can run on all nodes and allows for parallel forwarding of events.


How to configure syslog forwarding

  1. Login to each node that you want to enable syslog forwarding over ssh as ecaadmin.  NOTE: each node needs the file edited to configure the forwarding.  The instructions assume all nodes except node 1 will forward to syslog server.
  2. vim /opt/superna/eca/conf/syslogpublisher/log4j2.xml
  3. Add the ip address of the syslog server and the UDP port for your syslog server.  NOTE: You must edit the default port in the file 5140 and change this to the port used by your syslog server.  NOTE: Default syslog servers use port 514
    1.  
  4. Save the file
  5. The docker container does not start by default to start this container, Add an entry to the docker overrides file to start the container on nodes 2-N. 
    1. vim /opt/superna/eca/docker-compose.overrides.yml  
    2. add a section as per image below make sure to space the entries exactly as shown

  6. save the file
  7. To update all nodes with the new syslog configuration run the following command:
    1. ecactl cluster push-config  
  8. To start the container now and follow the steps below:
    1. Now create the container on all nodes
      1. ecactl cluster exec "ecactl containers up -d syslogpublisher" 
      2. NOTE: This will start the container
      3. This will start the container on node 1 and should be removed in production environments
      4. On node 1 run this commands to stop and remove the container,  Answer yes to the prompt to remove.
        1. ecactl containers stop syslogpublisher
        2. ecactl containers rm syslogpublisher 
  9. Verify your syslog server is now receiving  events sample syslog format below.
  10. To monitor the forwarding function and events received and sent  use this command to monitor the syslog container on one eca node (note all ECA nodes are forwarding events).
    1. ecactl logs --tail 200 --follow syslogpublisher  (this command will show stats every minute for events received by the container and sent to your syslog server).   
  11. done


Example syslog message format sent by the ECA


  1. 2019-07-07T20:49:22.328Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x8","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"node 172.31.1.131 07/07 16:49:21.964
  2. 2019-07-07T20:49:22.332Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
  3. 2019-07-07T20:49:22.333Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x8000","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"n 172.31.1.131 07/07 16:49:21.964
  4. 2019-07-07T20:49:22.334Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40000","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":" 172.31.1.131 07/07 16:49:21.964
  5. 2019-07-07T20:49:22.334Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
  6. 2019-07-07T20:49:22.335Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
  7. 2019-07-07T20:49:22.335Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x20","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
  8. 2019-07-07T20:59:40.816Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.430
  9. 2019-07-07T20:59:40.817Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
  10. 2019-07-07T20:59:40.817Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x2","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"17 172.31.1.131 07/07 16:59:40.446
  11. 2019-07-07T20:59:40.821Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
  12. 2019-07-07T20:59:40.823Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
  13. 2019-07-07T20:59:40.829Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
  14. 2019-07-07T20:59:40.831Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.461

How to view Syslog forwarding Statistics

  1. NOTE:  No log exists to see events within the ECA.  The forwarding feature uses programmatic access to an internal message bus that is not exposed to viewable.
  2. To view statistics of each ECA nodes forwarding function run this command on node 1 of the eca cluster after logging as the ecaadmin user.
  3. ecactl cluster exec "ecactl logs --tail 20 syslogpublisher"
  4. Each node will output the events it received for forwarding and the number of files it sent to the configured syslog server
  5. example
    1. In the example below the Sent events shows the total all time and the rate per second over the last minute. Stats are updated each minute on each node.    The example below shows a rate of 794 audit events forwarded per second over the last minute.     The last event ts date is the date of the time stamp indicating when the event itself was created on the cluster.  This gives you an idea of how current relative to the current time versus the forwarding functions progress.  
    2. syslogpublisher | 2020-09-21 12:58:28,269 AnalysisModule:146 INFO : Events Sent: | total 15040319 | rate 794.77 | last event ts: Mon Sep 21 12:58:28 UTC 2020 


How to debug syslog forwarding when you syslog server does not receive messages

  1. These steps assume you have check firewalls and verified this is not the issue and that the correct forwarding port has been used on the ECA configuration.  
  2. login to eca node 2, 3, 4, 5 etc.. since each node forwards syslog messages.  You should use the stats command in the above section to determine which ECA node is showing sent audit events.  Then use tcpdump on that node to capture UDP messages.
  3. sudo -s (to become root user)
  4. zypper in tcpdump (requires Internet connection to ECA)
  5. Monitor UDP syslog on the ECA node 
    1. tcpdump -i eth0 udp port 514 (this command will display all UDP packets on port 514 to the console)
  6. The messages will look like this once a packet is captured with the destination host name or IP shown.  Yellow highlight in the example.
    1. 09:54:43.379985 IP 172.31.1.135.45750 > syslog.internal.superna.net.syslog: SYSLOG local0.info, length: 664

How to Configure event filtering before forwarding

Use this configuration to select events from a specific path or below and forward only these events to the syslog server. This avoids a large volume of syslog data being sent when only a subset is needed.   This same concept can be used to pattern match on a SID, event type in the raw syslog message.   You will need to experiment with the pattern match for specific events.    


The example below covers path based matching, to match against other fields setup forwarding first to syslog, then review the fields in the event messages to build matching filters for other fields such as user or event type.

  1. Review the syntax below and edit the log4j2.xml file to add your filter 
    1. Example to match all syslog events for the path /ifs/data/smb01/test123
    2. login to eca node 1 as ecaadmin over ssh
    3. vim /opt/superna/eca/conf/syslogpublisher/log4j2.xml
    4. Insert a line using yellow highlighted example below and adjust the filter for your matching criteria.
    5. Save the file
    6. :wq
    7. done
    8. Push the configruation to all nodes
    9. ecactl cluster push-config
    10. Restart the syslog publishing container on all ECA nodes to reload the configuration
    11. ecactl cluster services restart --container sysylogpublisher --all 
    12. done

Example filter for a path called /ifs/data/smb01/test123

After making a change to this file you must

  1. ecactl cluster push-config 
  2. ecactl cluster services restart --container sysylogpublisher --all


<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%highlight{%d %C{1}:%L %-5level: %msg%n%throwable}"/>
</Console>
<Syslog name="SupernaSyslog" format="RFC5424" facility="LOCAL0"
host="172.22.4.19" port="514" protocol="UDP" appName="ECA"
messageId="AuditLogs" id="Event" connectTimeoutMillis="10000"
newLine="true" mdcId="mdc" includeMDC="true" enterpriseNumber="18060">
<RegexFilter regex=".*ifs.*data.*smb01.*test123.*" useRawMsg="true" onMatch="ACCEPT" onMismatch="DENY"/>
</Syslog>
</Appenders>
<Loggers>
<Root level="ALL">
<AppenderRef ref="STDOUT"/>
</Root>
<Logger name="org.apache.log4j.xml" level="info"/>
<Logger name="SYSLOG" level="ALL">
<AppenderRef ref="SupernaSyslog"/>
</Logger>
</Loggers>
</Configuration>

Example of how to exclude audit records from /ifs/.ifsvar


After making a change to this file you must

  1. ecactl cluster push-config 
  2. ecactl cluster services restart --container sysylogpublisher --all
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%highlight{%d %C{1}:%L %-5level: %msg%n%throwable}"/>
</Console>
<Syslog name="SupernaSyslog" format="RFC5424" facility="LOCAL0"
host="172.22.4.19" port="514" protocol="UDP" appName="ECA"
messageId="AuditLogs" id="Event" connectTimeoutMillis="10000"
newLine="true" mdcId="mdc" includeMDC="true" enterpriseNumber="18060">
<RegexFilter regex="^((?!.*ifsvar.*).)*$" useRawMsg="true" onMatch="ACCEPT" onMismatch="DENY"/>
</Syslog>
</Appenders>
<Loggers>
<Root level="ALL">
<AppenderRef ref="STDOUT"/>
</Root>
<Logger name="org.apache.log4j.xml" level="info"/>
<Logger name="SYSLOG" level="ALL">
<AppenderRef ref="SupernaSyslog"/>
</Logger>
</Loggers>
</Configuration>


Syslog Configuration Forwarding Parameters 


Adavanced options for forwarding.


Parameters:
host - The name of the host to connect to.
port - The port to connect to on the target host.
protocolStr - The Protocol to use.
sslConfiguration - TODO
connectTimeoutMillis - the connect timeout in milliseconds.
reconnectDelayMillis - The interval in which failed writes should be retried.
immediateFail - True if the write should fail if no socket is immediately available.
name - The name of the Appender.
immediateFlush - "true" if data should be flushed on each write.
ignoreExceptions - If "true" (default) exceptions encountered when appending events are logged; otherwise they are propagated to the caller.
facility - The Facility is used to try to classify the message.
id - The default structured data id to use when formatting according to RFC 5424.
enterpriseNumber - The IANA enterprise number.
includeMdc - Indicates whether data from the ThreadContextMap will be included in the RFC 5424 Syslog record. Defaults to "true:.
mdcId - The id to use for the MDC Structured Data Element.
mdcPrefix - The prefix to add to MDC key names.
eventPrefix - The prefix to add to event key names.
newLine - If true, a newline will be appended to the end of the syslog record. The default is false.
escapeNL - String that should be used to replace newlines within the message text.
appName - The value to use as the APP-NAME in the RFC 5424 syslog record.
msgId - The default value to be used in the MSGID field of RFC 5424 syslog records.
excludes - A comma separated list of mdc keys that should be excluded from the LogEvent.
includes - A comma separated list of mdc keys that should be included in the FlumeEvent.
required - A comma separated list of mdc keys that must be present in the MDC.
format - If set to "RFC5424" the data will be formatted in accordance with RFC 5424. Otherwise, it will be formatted as a BSD Syslog record.
filter - A Filter to determine if the event should be handled by this Appender.
configuration - The Configuration.
charset - The character set to use when converting the syslog String to a byte array.
exceptionPattern - The converter pattern to use for formatting exceptions.
loggerFields - The logger fields
advertise - Whether to advertise



© Superna LLC