Configure a centralized logging server

This chapter describes how to use the rsyslog daemon of a nevisAppliance to receive syslog messages from Nevis product or components. See the chapter Filebeat if you prefer to use Filebeat instead of syslog to forward log data.

Every Nevis instance forwards its log messages to the central nevisAppliance which stores the data in individual files. For this to work, every message must include a "tag" identifying its source using the following syntax: <instance type> '/' <file source> '/' <instance name>.

Example: nevisProxy/navajo/prod

The length of the tag must not exceed 32 characters.The facility used by Nevis product and components is local3.

Configure rsyslog on the loghost

Ensure that you have enabled rsyslog to receive log messages from remote servers and to store the data within the /var/log/nevis directory. These settings are made within the /etc/rsyslog.conf file. You can use the /etc/rsyslog.conf.server.tmpl as a template for your configuration.

rsyslog.conf

# loghost config start
$ModLoad imudp
$UDPServerRun 514

$template TmplNevis, "/var/log/nevis/%HOSTNAME%/%SYSLOGTAG:F,58:1%.log"
:SYSLOGTAG,!startswith, "nevis" ~
local3.* ?TmplNevis
# loghost config end

You can restart the rsyslog daemon now:

# systemctl restart rsyslog.service

All log data, received from any Nevis product or component, is now stored within the directory /var/log/nevis.

Log rotation

The file /etc/logrotate.d/nevissyslog defines when files are rotated (size) and how many copies are stored.

nevissyslog configuration

/var/log/nevis/*/*/*/*.log {
 missingok
 compress
 size 500M
 rotate 30
 sharedscripts
 postrotate
 /usr/bin/killall -HUP rsyslogd
 endscript
}

Edit this file according to your requirements.

• Ensure that you not only have enough disk space to store all files but also that you choose a server with a very good disk I/O performance.

Configure the Nevis product and components

Each Nevis instance needs to know where to forward the log data to. You can configure this within the instance's log configuration files.

• bc.properties

  Log forwarding of the nevisProxy's navajo.log is configured within the bc.properties file. The qslogger tool forwards log data to the local rsyslog daemon which in turn forwards the log data to the log server.

  Example for the navajo log of a nevisProxy instance:

  BC.Tracer.LogFile=pipe:///opt/nevisproxy/bin/qslogger -p -t   nevisProxy/navajo/default -f
  local3 -l INFO -d debug -r "^[a-zA-Z0-9\.: -]+ [0-9]-([A-Z_]\{4,  9\}).*"
  |/opt/nevisproxy/bin/bclogmgr size=52428800 archives=1
  /var/opt/nevisproxy/default/logs/navajo.log

• navajo.xml

  The navajo.xml file defines the ErrorLog and TransferLog attributes.

  Example:

  ErrorLog=""|/opt/nevisproxy/bin/qslogger -p -t nevisProxy/error/  default -f local3 -
  l INFO |/opt/nevisproxy/bin/bclogmgr size=52428800 archives=1
  /var/opt/nevisproxy/default/logs/apache.log""
  TransferLog=""|/opt/nevisproxy/bin/qssign -s 12345 -e |/opt/  nevisproxy/bin/qslogger
  -p -t nevisProxy/access/default -f local3 |/opt/nevisproxy/bin/ bclogmgr size=52428800
  archives=1 /var/opt/nevisproxy/default/logs/access.log""

• rsyslog.conf

  As nevisProxy uses the local rsyslog deamon to forward the log messages, you need to specify the target log server's host name within this file.

  Example:

  local3.* @ loghost.zh.adnovum.ch

• log4j.xml

  With log4j you can configure an additional appender to forward log data to your log server using syslog UDP.

  Example for the audit log of a nevisAuth instance:

  <appender class="ch.nevis.commons.log4j.net.NevisSyslogAppender"
     name="SYSLOGS_SERVER_audit">
   <param name="SyslogHost" value="loghost.zh.adnovum.ch:514" />
   <param name="Facility" value="LOCAL3" />
   <param name="Threshold" value="INFO" />
   <layout class="org.apache.log4j.PatternLayout">
   <param name="ConversionPattern"
     value="nevisAuth/audit/default: %d{ISO8601} %15.15t %12X {conversationId}%X{transferId} %-20.20c %-5.5p %X{indent}  %m%n" />
   </layout>
  </appender>
  <category additivity="false" name="ch.nevis.esauth.util.audit">
   <priority value="TRACE" />
   <appender-ref ref="SYSLOGS_SERVER_audit" />
  </category>

  See the nevisAdmin reference guide chapter "Syslog" on how to enable syslog forwarding using the web front end of nevisAdmin.

Encrypting syslog traffic

The nevisAppliance supports encryption of syslog traffic in different ways:

• You can configure IPsec between the servers.
• Enable TLS within the rsyslog daemons. How to configure TLS is described here: http://www.rsyslog.com/doc/rsyslog_tls.html. A sample configuration is available within the rsyslog configuration templates /etc/rsyslog.conf.tmpl (for the sender) and /etc/rsyslog.conf.server.tmpl (for the receiver).

*While nevisProxy uses the local syslog facility by default, you need to configure the Java-based Nevis product and components (nevisAuth, nevisIDM, etc.) manually to use the local rsyslog daemon, similar as shown in the [figure] below.

Highly reliable syslog forwarding

If you want to ensure that no log data is ever lost, even when you are restarting or rebooting your log server, you should configure a redundant setup by the following syslog settings:

• Use TCP/TLS instead of UDP to ensure that no data package gets lost.
• Set up two redundant log servers receiving log data.
• Configure a buffer within your syslog configuration to ensure no data is lost if a server loses the connection to both log hosts.

Sample rsyslog client configuration:

local3.* @@(o)loghostprimary.zh.adnovum.ch:10514
$ActionExecOnlyWhenPreviousIsSuspended on
&@@(o) loghostsecondary.zh.adnovum.ch:10514
& /var/log/syslogbuffer
$ActionExecOnlyWhenPreviousIsSuspended off