Administration Guides

Syslog forwarding Configuration

Home


Overview

This configuration is optional and only needed when events should be forwarded another logging system link Splunk or other logging tools.  The ECA can run an additional dock container that consumes events and formats for syslog forarding.  This container can run on all nodes and allows for parallel forwarding of events.


How to configure syslog forwarding

  1. Login to each node that you want to enable syslog forwarding over ssh as ecaadmin.  NOTE: each node needs the file edited to configure the forwarding.  The instructions assume all nodes except node 1 will forward to syslog server.
  2. vim /opt/superna/eca/conf/syslogpublisher/log4j2.xml
  3. add the ip address of the syslog server and port for syslog. NOTE: Default syslog servers use port 514
    1.  
  4. Save the file
  5. The docker container does not start by default to start this container, Add an entry to the docker overrrides file to start the container on nodes 2-N. 
    1. vim /opt/superna/eca/docker-compose.overrides.yml  
    2. add a section as per image below make sure to space the entries exactly as shown

  6. save the file
  7. To update all nodes with the new syslog configuration run the following command:
    1. ecactl components configure-nodes 
  8. To start the container now and follow the steps below:
    1. Now create the container on all nodes
      1. ecactl cluster exec "ecactl containers up -d syslogpublisher" 
      2. NOTE: This will start the container
      3. This will start the container on node 1 and should be removed in production environemnts
      4. On node 1 run this commands to stop and remove the container,  Answer yes to the prompt to remove.
        1. ecactl containers stop syslogpublisher
        2. ecactl containers rm syslogpublisher 
  9. Verify syslog server is now receiving  events sample syslog format below
  10. <134>1 2019-07-07T20:49:22.328Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x8","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"node 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:49:22.332Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:49:22.333Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x8000","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"n 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:49:22.334Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40000","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":" 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:49:22.334Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:49:22.335Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:49:22.335Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x20","path":"\\\\00505699a9f1aecd965b770a3472e43955d2\\System\\ifs\\spark-logs\\.f6b02da1-6f42-43a3-b02a-ae87564c255b","protocol":"HDFS","server":"nod 172.31.1.131 07/07 16:49:21.964
    <134>1 2019-07-07T20:59:40.816Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.430
    <134>1 2019-07-07T20:59:40.817Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
    <134>1 2019-07-07T20:59:40.817Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x2","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"17 172.31.1.131 07/07 16:59:40.446
    <134>1 2019-07-07T20:59:40.821Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
    <134>1 2019-07-07T20:59:40.823Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
    <134>1 2019-07-07T20:59:40.829Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.446
    <134>1 2019-07-07T20:59:40.831Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.461
    <134>1 2019-07-07T20:59:40.834Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:40.461
    <134>1 2019-07-07T20:59:43.847Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:43.470
    <134>1 2019-07-07T20:59:43.850Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x4","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"17 172.31.1.131 07/07 16:59:43.470
    <134>1 2019-07-07T20:59:43.852Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x40","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:43.470
    <134>1 2019-07-07T20:59:43.853Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x80","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\data\\policy1\\search\\cow.txt","protocol":"SMB2","server":"node001","clientIP":"1 172.31.1.131 07/07 16:59:43.470
    <134>1 2019-07-07T20:59:53.186Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x4","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\email\\mynasmainfile.pst","protocol":"SMB2","server":"node001","clientIP":"172.31.1 172.31.1.131 07/07 16:59:52.809
    <134>1 2019-07-07T20:59:53.187Z syslogpublisher.node1.demoeca.eca.local ECA 1 AuditLogs - {"eventCode":"0x8","path":"\\\\0050569960fcd70161594d21dd22a3c10cbe\\System\\ifs\\email\\~mynasmainfile.pst.tmp","protocol":"SMB2","server":"node001","clientIP":"172 172.31.1.131 07/07 16:59:52.809
  11. done


How to Configure event filtering before forwarding

Use this configuration to select events from a specific path or below and forward only these events to the syslog server. This avoids a large volume of syslog data being sent when only a subset is needed.   This same concept can be used to pattern match on a SID, event type in the raw syslog message.   You will need to experment with the pattern match for specific events.    


The example below covers path based matching, to match against other fields setup forwarding first to syslog, then reivew the fields in the event messages to build matching filters for other fields such as user or event type.

  1. Review the syntax below and edit the log4j2.xml file to add your filter 
    1. Example to match all syslog events for the path /ifs/data/smb01/test123
    2. login to eca node 1 as ecaadmin over ssh
    3. vim /opt/superna/eca/conf/syslogpublisher/log4j2.xml
    4. Insert a line using yellow highlighted example below and adjust the filter for your matching criteria.
    5. Save the file
    6. :wq
    7. done
    8. Restart the syslog publishing container on all ECA nodes to reload the configuration
    9. ecactl cluster exec "ecactl containers restart syslogpublisher" 
    10. done
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%highlight{%d %C{1}:%L %-5level: %msg%n%throwable}"/>
</Console>
<Syslog name="SupernaSyslog" format="RFC5424" facility="LOCAL0"
host="172.22.4.19" port="514" protocol="UDP" appName="ECA"
messageId="AuditLogs" id="Event" connectTimeoutMillis="10000"
newLine="true" mdcId="mdc" includeMDC="true" enterpriseNumber="18060">
<RegexFilter regex=".*ifs.*data.*smb01.*test123.*" useRawMsg="true" onMatch="ACCEPT" onMismatch="DENY"/>
</Syslog>
</Appenders>
<Loggers>
<Root level="ALL">
<AppenderRef ref="STDOUT"/>
</Root>
<Logger name="org.apache.log4j.xml" level="info"/>
<Logger name="SYSLOG" level="ALL">
<AppenderRef ref="SupernaSyslog"/>
</Logger>
</Loggers>
</Configuration>



Syslog Configuration Forwarding Parameters 


Adavanced options for forwarding.


Parameters:
host - The name of the host to connect to.
port - The port to connect to on the target host.
protocolStr - The Protocol to use.
sslConfiguration - TODO
connectTimeoutMillis - the connect timeout in milliseconds.
reconnectDelayMillis - The interval in which failed writes should be retried.
immediateFail - True if the write should fail if no socket is immediately available.
name - The name of the Appender.
immediateFlush - "true" if data should be flushed on each write.
ignoreExceptions - If "true" (default) exceptions encountered when appending events are logged; otherwise they are propagated to the caller.
facility - The Facility is used to try to classify the message.
id - The default structured data id to use when formatting according to RFC 5424.
enterpriseNumber - The IANA enterprise number.
includeMdc - Indicates whether data from the ThreadContextMap will be included in the RFC 5424 Syslog record. Defaults to "true:.
mdcId - The id to use for the MDC Structured Data Element.
mdcPrefix - The prefix to add to MDC key names.
eventPrefix - The prefix to add to event key names.
newLine - If true, a newline will be appended to the end of the syslog record. The default is false.
escapeNL - String that should be used to replace newlines within the message text.
appName - The value to use as the APP-NAME in the RFC 5424 syslog record.
msgId - The default value to be used in the MSGID field of RFC 5424 syslog records.
excludes - A comma separated list of mdc keys that should be excluded from the LogEvent.
includes - A comma separated list of mdc keys that should be included in the FlumeEvent.
required - A comma separated list of mdc keys that must be present in the MDC.
format - If set to "RFC5424" the data will be formatted in accordance with RFC 5424. Otherwise, it will be formatted as a BSD Syslog record.
filter - A Filter to determine if the event should be handled by this Appender.
configuration - The Configuration.
charset - The character set to use when converting the syslog String to a byte array.
exceptionPattern - The converter pattern to use for formatting exceptions.
loggerFields - The logger fields
advertise - Whether to advertise



Copyright Superna LLC