Skip to main content
Version: 2.202111.x.x LTS

Data Shippers - Elastic Beats

Beats is a platform from Elastic for single-purpose data shippers. In Nevis, Beats are used to send data from machine and Nevis installations to Logstash, nevisReports and nevisAdmin v4.

On all nevisAppliance types, the following Beats are included by default:

  • Filebeat, to forward log files.
  • Experimental: Heartbeat, to periodically check the status of services.
  • Experimental: Journalbeat, to forward log files from journald,the system that collects the logs from systemd processes.

This chapter explains how to enable and start these services.

For more information about the configuration, read the official Filebeat, Heartbeat and Journalbeat documentation. For Nevis specific integration documentation see nevisReports Reference Guide or the nevisAdmin v4 documentation.

Filebeat​

Alternatively to syslog (see also the chapter Configure a centralized logging server), you can use Filebeat to transmit log messages with the Beats protocol for the centralized log server.

Forwarding logs using Filebeat

Filebeat has to be configured on every server. It appends the log files of the Nevis product or components. On the receiving log server, a Logstash must run using a Beats input filter plug-in. Logstash stores the received data into local files and/or writes all data to an Elasticsearch cluster.

Sample Filebeat configuration in /etc/filebeat/filebeat.yml​

filebeat:
  prospectors:
    -
      paths: /var/opt/nevisproxy/proxy01/logs/navajo.log*
      exclude_files:
      fields_under_root: true
      fields:
        host: srv01
        type: navajo
        comp: nevisproxy
        instance: proxy01

output:
  logstash:
    hosts: "loghost.zh.adnovum.ch:5044"
    ssl.certificate_authorities:
    - "/var/opt/neviskeybox/default/default/truststore/node1.pem"

logging:
  to_files: true
  files:
    path: "/var/log/filebeat"
    name: "filebeat.log"
    rotateeverybytes: 10485760
    keepfiles: 7
  level: error

Within the output configuration, you set the local server’s node truststore as well as the hostname and port of the remote log host.

Sample logstash configuration​

input {
  beats {
    host => "loghost.zh.adnovum.ch"
    port => 5044
    ssl => true
    ssl_certificate => "/var/opt/neviskeybox/default/default/node_certificate.pem"
    ssl_key => "/var/opt/neviskeybox/default/default/node_key.pem"
    # ssl_key_passphrase => "<password>"
}

output{
  file {
    path => "/var/opt/log/nevis/%{host}/%{comp}/%{type}/%{instance}.%{+YYYY-MM-dd-HH}
.log.gz"
    message_format => "%{message}"
    gzip => true
  }
}

Within the input.beats configuration, you set the log hosts certificate and the key file. Note that the key file must contain a plaintext key.

The above output configuration example writes the log data to local files on the disk (where they are compressed and rotated every hour).

Use the systemctl command from systemd to manage, start or stop the process.

  • Configuration: /etc/filebeat/filebeat.yml
  • Start/stop: systemctl start|stop filebeat
  • Enable: systemctl enable filebeat

Heartbeat (Experimental)​

Use the systemctl command from systemd to manage, start or stop the process.

  • Configuration: /etc/heartbeat/heartbeat.yml
  • Start/stop: systemctl start|stop heartbeat-elastic
  • Enable: systemctl enable heartbeat-elastic

Journalbeat (Experimental)​

Use the systemctl command from systemd to manage, start or stop the process.

  • Configuration: /etc/journalbeat/journalbeat.yml
  • Start/stop: systemctl start|stop journalbeat
  • Enable: systemctl enable journalbeat