OpenTelemetry

This chapter describes how to configure Nevis components to integrate with a self-hosted observability stack or with an OpenTelemetry compatible cloud provider.

OpenTelemetry is used to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

An example cloud based setup of the observability stack can be seen here: OpenTelemetry setup.

For production use, a managed service is recommended.

The monitoring nevisAppliance flavor comes with Prometheus and OpenTelemetry collector which can be used to collect metrics from all Nevis components.

Configuration

Monitoring components

For Kubernetes deployments the monitoring components can be installed with the cloud based OpenTelemetry setup, and for classic deployments it is provided by the monitoring nevisAppliance flavor.

nevisAdmin 4 based configuration

In nevisAdmin 4 based setups, the Java agent and the minimal settings for OpTrace is configured by default.

To configure the forwarding to OpenTelemetry see Java Observability Settings for Java based components and nevisProxy Observability Settings for nevisProxy.

Manual configuration

Java Agent

For Java based components, a custom OpenTelemetry agent is used.

To use the agent, extend the JAVA_OPS of the relevant component with:

-javaagent:<opentelemetry-javaagent.jar location>
-Dotel.javaagent.logging=application
-Dotel.javaagent.configuration-file=<otel.properties location>
-Dotel.instrumentation.jdbc.enabled=false

And create the following otel.properties file:

otel.service.name=<service.name>
otel.resource.attributes=service.version=<version>,service.instance.id=<intanceId>
otel.exporter.otlp.protocol = http/protobuf
otel.exporter.otlp.traces.protocol = http/protobuf
otel.exporter.otlp.traces.endpoint = <tracesEndpoint>
otel.exporter.otlp.metrics.protocol = http/protobuf
otel.exporter.otlp.metrics.endpoint = <metricsEndpoint>
otel.exporter.otlp.metrics.temporality.preference = cumulative
otel.exporter.otlp.logs.protocol = http/protobuf
otel.exporter.otlp.logs.endpoint = <logsEndpoint>

where

• service.name: Name of the service, e.g.: auth
• version: Version of the service, e.g.: 1.0
• intanceId: Name of the instance, such as auth-dev
• tracesEndpoint: Endpoint of the OpenTelemetry traces collector, e.g.: https://otel-collector:4318/v1/traces
• metricsEndpoint: Endpoint of the OpenTelemetry metrics collector, e.g.: https://otel-collector:4318/v1/metrics
• logsEndpoint: Endpoint of the OpenTelemetry logs collector, e.g.: https://otel-collector:4318/v1/logs

The OpenTelemetry jar can be downloaded on the Nevis portal at: https://portal.nevis.net/portal/secure/releases/rolling by searching for opentelemetry.

See Agent config for more configuration options.

OpTrace

The Nevis OpenTelemetry agent also serves as a replacement for OpTrace. It can be enabled by setting the OpTrace log category to INFO:

  Loggers:
    Logger:
    - name: "OpTrace"
      level: "INFO"

If the agent is only used for logging purposes, the forwarding to an OpenTelemetry collector can be disabled with the following otel.properties file:

otel.service.name=<service.name>
otel.resource.attributes=service.version=<version>,service.instance.id=<intanceId>
otel.traces.exporter=none
otel.metrics.exporter=none
otel.logs.exporter=none

Additionally, change the log format to include span_id and trace_id:

      PatternLayout:
        pattern: "[esauth4sv.log] %d{ISO8601} %-15.15t %mdc{trace_id} %mdc{span_id} %-20.20c %-5.5p %m%n"

nevisProxy

By default, nevisProxy uses OpenTelemetry for unique ID tracing, as a replacement for the TransferID, see ch.nevis.navajo.tracing.TraceId.Format for more information. For further configuration options, see Telemetry Configuration, or the example below. In navajo.xml:

        <OpenTelemetry>
            <Trace
                ExporterType="http"
                ExporterAddress="localhost:4318"
                PropagateContextExtract="true"
                PropagateContextInject="true"
                ResourceServiceName="myService_001"
            />
            <Metrics
                ExporterType="http"
                ExporterAddress="localhost:4318"
                Interval="10"
                Timeout="500"
            />
        </OpenTelemetry>

And in the web.xml, if the traceparent header is required on the frontend side:

    <filter>
        <filter-name>Observability_ResponseTraceId_Default</filter-name>
        <filter-class>ch::nevis::isiweb4::filter::lua::LuaFilter</filter-class>
        <init-param>
            <param-name>Script</param-name>
            <param-value>
                function outputHeader(req, resp)
                   local traceid = req:getHeader("traceparent")
                   if traceid then
                       resp:setHeader("traceparent", traceid)
                   end
                end
            </param-value>
        </init-param>
        <init-param>
            <param-name>Script.OutputHeaderFunctionName</param-name>
            <param-value>outputHeader</param-value>
        </init-param>
    </filter>

    <filter-mapping>
        <filter-name>Observability_ResponseTraceId_Default</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>