OpenTelemetry

This chapter describes how to configure Nevis components to integrate with a self-hosted observability stack or with an OpenTelemetry compatible cloud provider.

OpenTelemetry is used to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

Configuration

Choose the section that applies to your environment

NevisAdmin 4 based configuration

In nevisAdmin 4 based setups, the Java agent and the minimal settings for OpTrace is configured by default.

To configure the forwarding to OpenTelemetry simply add the OpenTelemetry endpoints to the inventory using the __otel variable:

vars:
  __otel:
    metricsEndpoint: https://otel-otel-collector.observability:4318/v1/metrics
    logsEndpoint: https://otel-otel-collector.observability:4318/v1/logs
    tracesEndpoint: https://otel-otel-collector.observability:4318/v1/traces

To change the default settings Java Observability Settings and nevisProxy Observability Settings can be used.

Manual configuration without nevisAdmin 4

Java Agent

For Java based components, a custom OpenTelemetry agent is used.

To use the agent, extend the JAVA_OPS of the relevant component with:

-javaagent:/opt/agent/opentelemetry-javaagent.jar
-Dotel.javaagent.logging=application
-Dotel.javaagent.configuration-file=<otel.properties location>
-Dotel.instrumentation.jdbc.enabled=false
-Dotel.instrumentation.metro.enabled=false

And create the following otel.properties file:

otel.service.name=<service.name>
otel.resource.attributes=service.version=<version>,service.instance.id=<intanceId>
otel.exporter.otlp.protocol = http/protobuf
otel.exporter.otlp.traces.protocol = http/protobuf
otel.exporter.otlp.traces.endpoint = <tracesEndpoint>
otel.exporter.otlp.metrics.protocol = http/protobuf
otel.exporter.otlp.metrics.endpoint = <metricsEndpoint>
otel.exporter.otlp.metrics.temporality.preference = cumulative
otel.exporter.otlp.logs.protocol = http/protobuf
otel.exporter.otlp.logs.endpoint = <logsEndpoint>

where

• service.name: Name of the service, e.g.: auth
• version: Version of the service, e.g.: 1.0
• intanceId: Name of the instance, such as auth-dev
• tracesEndpoint: Endpoint of the OpenTelemetry traces collector, e.g.: https://otel-collector:4318/v1/traces
• metricsEndpoint: Endpoint of the OpenTelemetry metrics collector, e.g.: https://otel-collector:4318/v1/metrics
• logsEndpoint: Endpoint of the OpenTelemetry logs collector, e.g.: https://otel-collector:4318/v1/logs

The OpenTelemetry jar can be downloaded on the Nevis portal at: https://portal.nevis.net/portal/secure/releases/rolling by searching for opentelemetry.

See Agent config for more configuration options.

nevisProxy

By default, nevisProxy uses OpenTelemetry for unique ID tracing, as a replacement for the TransferID, see ch.nevis.navajo.tracing.TraceId.Format for more information. For further configuration options, see Telemetry Configuration, or the example below. In navajo.xml:

        <OpenTelemetry>
            <Trace
                ExporterType="http"
                ExporterAddress="localhost:4318"
                PropagateContextExtract="true"
                PropagateContextInject="true"
                ResourceServiceName="myService_001"
            />
            <Metrics
                ExporterType="http"
                ExporterAddress="localhost:4318"
                Interval="10"
                Timeout="500"
            />
        </OpenTelemetry>

And in the web.xml, if the traceparent header is required on the frontend side:

    <filter>
        <filter-name>Observability_ResponseTraceId_Default</filter-name>
        <filter-class>ch::nevis::isiweb4::filter::lua::LuaFilter</filter-class>
        <init-param>
            <param-name>Script</param-name>
            <param-value>
                function outputHeader(req, resp)
                   local traceid = req:getHeader("traceparent")
                   if traceid then
                       resp:setHeader("traceparent", traceid)
                   end
                end
            </param-value>
        </init-param>
        <init-param>
            <param-name>Script.OutputHeaderFunctionName</param-name>
            <param-value>outputHeader</param-value>
        </init-param>
    </filter>

    <filter-mapping>
        <filter-name>Observability_ResponseTraceId_Default</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

Known Issues

nevisAuth 7.2311+ - java.lang.NullPointerException: Cannot invoke "javax.xml.namespace.QName.getLocalPart()"

The following exception can occur for the SecurityTokenService or ArtifactResolutionService or any other service using JAX-WS webservice stack:

java.lang.NullPointerException: Cannot invoke "javax.xml.namespace.QName.getLocalPart()" because the return value of "com.sun.xml.ws.api.message.Packet.getWSDLOperation()" is null
at io.opentelemetry.javaagent.instrumentation.metro.MetroRequest.getSpanName(MetroRequest.java:30) ~[opentelemetry-javaagent.jar:?]
at io.opentelemetry.javaagent.instrumentation.metro.MetroRequest.[init](MetroRequest.java:17) ~[opentelemetry-javaagent.jar:?]
at io.opentelemetry.javaagent.instrumentation.metro.MetroHelper.start(MetroHelper.java:29) ~[opentelemetry-javaagent.jar:?]
at io.opentelemetry.javaagent.instrumentation.metro.TracingTube.processRequest(TracingTube.java:37) ~[opentelemetry-javaagent.jar:?]

The issue here is that OpenTelemetry Metro extension does not yet support JAX-WS version 3 or newer. In order to mitigate the issue you can disable the Metro OpenTelemetry extension by the following configuration option in the JAVA_OPTS in the env.conf: -Dotel.instrumentation.metro.enabled=false