How to Configure Syslog Forwarding of Formatted Audit messages to an External Syslog Server

• Overview
• How to configure syslog forwarding
• Example syslog message format sent by the ECA
• How to view Syslog forwarding Statistics
• How to debug syslog forwarding when you syslog server does not receive messages
• How to Configure event filtering before forwarding
• Example filter for a path called /ifs/data/smb01/test123
• Example of how to exclude audit records from /ifs/.ifsvar
• Syslog Configuration Forwarding Parameters

Overview

This configuration is optional and only needed when events should be forwarded another logging system link Splunk or other logging tools.  The ECA can run an additional dock container that consumes events and formats for syslog forwarding.  This container can run on all nodes and allows for parallel forwarding of events.

How to configure syslog forwarding

1. Login to each node that you want to enable syslog forwarding over ssh as ecaadmin.  NOTE: each node needs the file edited to configure the forwarding.  The instructions assume all nodes except node 1 will forward to syslog server.
2. vim /opt/superna/eca/conf/syslogpublisher/log4j2.xml
3. Add the ip address of the syslog server and the UDP port for your syslog server.  NOTE: You must edit the default port in the file 5140 and change this to the port used by your syslog server.  NOTE: Default syslog servers use port 514
   1.  
4. Save the file
5. The docker container does not start by default to start this container, Add an entry to the docker overrides file to start the container on nodes 2-N. 
   1. vim /opt/superna/eca/docker-compose.overrides.yml  
   2. add a section as per image below make sure to space the entries exactly as shown
   3. 
      
6. save the file
7. To update all nodes with the new syslog configuration run the following command:
   1. ecactl cluster push-config  
8. To start the container now and follow the steps below:
   1. Now create the container on all nodes
      1. ecactl cluster exec "ecactl containers up -d syslogpublisher" 
      2. NOTE: This will start the container
      3. This will start the container on node 1 and should be removed in production environments
      4. On node 1 run this commands to stop and remove the container,  Answer yes to the prompt to remove.
         1. ecactl containers stop syslogpublisher
         2. ecactl containers rm syslogpublisher 
9. Verify your syslog server is now receiving  events sample syslog format below.
10. To monitor the forwarding function and events received and sent  use this command to monitor the syslog container on one eca node (note all ECA nodes are forwarding events).
    1. ecactl logs --tail 200 --follow syslogpublisher  (this command will show stats every minute for events received by the container and sent to your syslog server).   
11. done

Example syslog message format sent by the ECA

How to view Syslog forwarding Statistics

1. NOTE:  No log exists to see events within the ECA.  The forwarding feature uses programmatic access to an internal message bus that is not exposed to viewable.
2. To view statistics of each ECA nodes forwarding function run this command on node 1 of the eca cluster after logging as the ecaadmin user.
3. ecactl cluster exec "ecactl logs --tail 20 syslogpublisher"
4. Each node will output the events it received for forwarding and the number of files it sent to the configured syslog server
5. example
   1. In the example below the Sent events shows the total all time and the rate per second over the last minute. Stats are updated each minute on each node.    The example below shows a rate of 794 audit events forwarded per second over the last minute.     The last event ts date is the date of the time stamp indicating when the event itself was created on the cluster.  This gives you an idea of how current relative to the current time versus the forwarding functions progress.  
   2. syslogpublisher | 2020-09-21 12:58:28,269 AnalysisModule:146 INFO : Events Sent: | total 15040319 | rate 794.77 | last event ts: Mon Sep 21 12:58:28 UTC 2020 

How to debug syslog forwarding when you syslog server does not receive messages

1. These steps assume you have check firewalls and verified this is not the issue and that the correct forwarding port has been used on the ECA configuration.  
2. login to eca node 2, 3, 4, 5 etc.. since each node forwards syslog messages.  You should use the stats command in the above section to determine which ECA node is showing sent audit events.  Then use tcpdump on that node to capture UDP messages.
3. sudo -s (to become root user)
4. zypper in tcpdump (requires Internet connection to ECA)
5. Monitor UDP syslog on the ECA node 
   1. tcpdump -i eth0 udp port 514 (this command will display all UDP packets on port 514 to the console)
6. The messages will look like this once a packet is captured with the destination host name or IP shown.  Yellow highlight in the example.
   1. 09:54:43.379985 IP 172.31.1.135.45750 > syslog.internal.superna.net.syslog: SYSLOG local0.info, length: 664

How to Configure event filtering before forwarding

Use this configuration to select events from a specific path or below and forward only these events to the syslog server. This avoids a large volume of syslog data being sent when only a subset is needed.   This same concept can be used to pattern match on a SID, event type in the raw syslog message.   You will need to experiment with the pattern match for specific events.    

The example below covers path based matching, to match against other fields setup forwarding first to syslog, then review the fields in the event messages to build matching filters for other fields such as user or event type.

1. Review the syntax below and edit the log4j2.xml file to add your filter 
   1. Example to match all syslog events for the path /ifs/data/smb01/test123
   2. login to eca node 1 as ecaadmin over ssh
   3. vim /opt/superna/eca/conf/syslogpublisher/log4j2.xml
      
   4. Insert a line using yellow highlighted example below and adjust the filter for your matching criteria.
   5. Save the file
   6. :wq
   7. done
   8. Push the configruation to all nodes
   9. ecactl cluster push-config
   10. Restart the syslog publishing container on all ECA nodes to reload the configuration
   11. ecactl cluster services restart --container sysylogpublisher --all 
   12. done

Example filter for a path called /ifs/data/smb01/test123

After making a change to this file you must

1. ecactl cluster push-config 
2. ecactl cluster services restart --container sysylogpublisher --all
   

Example of how to exclude audit records from /ifs/.ifsvar

After making a change to this file you must

1. ecactl cluster push-config 
2. ecactl cluster services restart --container sysylogpublisher --all

Syslog Configuration Forwarding Parameters 

Adavanced options for forwarding.