Frequently Asked Questions
How do trace levels work in nevisProxy?
In nevisProxy, the verbosity of the logs depends on two things:
- The base threshold set by
BC.Tracer.ThresholdBase. - The individual LogLevels set by the filters/servlets.
For example, if we set BC.Tracer.ThresholdBase=5, and set the LogLevel of the filters/servlets to 0, the logs contain all messages with a severity level of 0-5. If we take the previous ThresholdBase, and set for example, BC.Tracer.DebugProfile.IW4IdentCreaFlt=4, beside the already logged messages with severity 0-5, nevisProxy traces all log messages for the IdentityCreationFilter with a severity level of 6-9. This happens because the ThresholdBase is set to 5 and the individual LogLevel of 4 set for the filter results in a LogLevel of 9 (= 5 + 4) for the IdentityCreationFilter.
The mapping between the LogLevels and severity is shown in the following table:
| Level | Severity | Description |
|---|---|---|
| 0 | EMERGENCY | Nothing works anymore. |
| 1 | ALERT | Something should be corrected immediately. |
| 2 | CRITICAL | A critical situation has occurred. |
| 3 | ERROR | Other error messages. |
| 5 | NOTICE | Things you should know. |
| 6 | INFO | Things of interest. |
| 7 | DEBUG_LOW | Low debug level. |
| 8 | DEBUG_MED | Medium debug level. |
| 9 | DEBUG_HIGH | High debug level (enter/leave). |
| 10 | DEBUG_MAX | Highest debug level. |
Frequently used trace groups for debugging
To make debugging and opening support tickets more efficient, here are a few relevant trace groups, you should consider raising or enabling before reproducing an issue, and sending the logs to the nevisProxy team:
General trace groups
BC.Tracer.DebugProfile.NavajoOp=4
BC.Tracer.DebugProfile.IsiwebOp=4
Trace groups regarding backend connectivity
BC.Tracer.DebugProfile.IW4WebSocketSrv=4
BC.Tracer.DebugProfile.IW4HttpConnSrv=4
BC.Tracer.DebugProfile.IW4HttpsConnSrv=4
BC.Tracer.DebugProfile.NPBackendSrv=4
Frontend connection
BC.Tracer.DebugProfile.NavajoRequest=4
Trace groups relevant to the authentication process
BC.Tracer.DebugProfile.IW4IdentCreaFlt=4
BC.Tracer.DebugProfile.IW4Esau4ConnSrv=4
BC.Tracer.DebugProfile.IW4SecRoleFlt=4
SSL/TLS-related trace groups
For frontend connections
BC.Tracer.DebugProfile.NavajoSSL=4
BC.Tracer.DebugProfile.Apache=4
Also set the property Loglevel in the file navajo.xml, as follows:
Loglevel="debug ssl:trace5"
For backend connections
BC.Tracer.DebugProfile.OpenSSLHandshk=4
BC.Tracer.DebugProfile.OpenSSLRecord=4
BC.Tracer.DebugProfile.OpenSSLStream=4
bcx.net.debug.host=all
bcx.net.debug=handshake,record,ssl, ssl-key
Configuring logging in nevisAdmin 4
In nevisAdmin 4, you can use the pattern nevisProxy Log Settings to configure the severity of the logs. The LogLevel, which is the ThresholdBase, is set to NOTICE by default. This can be overridden on a filter/servlet basis, by defining a value for the filter/servlet under the LogLevel Overrides.
In some cases, nevisAdmin 4 uses a different name than nevisProxy for the different severity levels.
| Severity Level | nevisProxy | nevisAdmin4 |
|---|---|---|
| 3 | ERROR | ERROR |
| 5 | NOTICE | NOTICE |
| 6 | INFO | INFO |
| 7 | DEBUG_LOW | DEBUG |
| 9 | DEBUG_HIGH | DEBUG_HIGH |
| 10 | DEBUG_MAX | TRACE |
Use only the suffix of the name of the described trace group combined with the LogLevel as string. Do not use the numbers of the Reference Guide, as nevisAdmin 4 calculates the numbers automatically.
How to solve frontend or backend connection issues in nevisProxy?
Solving frontend connection issues
The most frequent issue on the frontend side is when a request gets blocked but there is no corresponding log entry in navajo.log. This can happen when the request fails on the Apache level, before reaching the navajo module, due to some limitation or restriction, for example, it has too many request headers. To debug the issue, check the apache log which should have a proper error message describing the exact error. You may need to increase the log level of the apache log, which can be done by setting the Loglevel directive in navajo.xml. Possible causes can be:
- The mod_qos directives are too strict.
- The various LimitRequest* directives are too strict.
- Client certificate is required but the client did not provided one.
- Make sure the MaxClientsPerIpAddr configuration is not used as it may lead to cores.
- Verify the server certificates are correct using
openssl verify. - Verify the server is reachable using
openssl s_client.
If even the Apache log does not have any entry about the failed request, it means it failed before reaching nevisProxy, and the error comes from a previous network element (e.g. firewall, load balancer).
Solving backend connection issues
If nevisProxy runs into a backend connection issue, usually an error is logged in navajo.log. If that is not descriptive enough, it is worth to increase the log level of the servlet experiencing the connection issues, and check the traced error code in the appendix for details and hints. Other possible steps can be:
- Check the health of the backend.
- Check that during the nevisProxy startup, there is no error traced in the initialization step.
- Verify that if a hostname is used instead of IP address, the name can be resolved.
- Verify that the backend is reachable from the nevisProxy host. Example command:
openssl s_client -connect <hostname>:<port> -showcerts [-CAfile ca.pem -verify_hostname <hostname>] - Verify that the used certificates are correct and valid if TLS is enabled. Example command:
openssl verify -verbose -CAfile ca.pem certificate.pem - Check that if the backend sends a valid and complete HTTP response message. Consider using network tools such as Wireshark or tcpdump for this.
- If you are connecting to a backend with multiple virtual hosts, make sure SNI is enabled.
Why does the user session get lost during or after authentication?
When using cookie-based identification with the dynamic session management, for security reasons nevisProxy regenerates the cookie after a successful authentication. In certain contexts (parallel AJAX requests), the new cookie may get lost if there are parallel requests with the old cookie before the new cookie could reach the client, which means the session may get lost on the user side.
To prevent this, the best solution is to prohibit parallel requests during the authentication phase on the client side, so the new cookie is registered properly, and follow-up requests use the valid cookie.
Another workaround can be setting the Cookie.OnInvalid parameter of the SessionManagementFilter to a blocking status code, which would mean that the parallel requests using the old cookie would get blocked and when retried, they should succeed with the new cookie.
The parameter Cookie.KeepOldCookieTimeout can also help in this use-case. It configures the number of seconds that an existing cookie is valid after a new one is renegotiated. Increasing the value can help minimize the chance of session loss during a login or step-up with parallel request.
Finally, the cookie regeneration can be disabled with the RenewIdentification parameter of the IdentityCreationFilter, but this is not recommended due to security concerns.
This does not work as expected - how can I get support for nevisProxy issues?
General issues
- First try to isolate, and reproduce the issue locally.
- Enable the relevant trace groups, with the correct trace levels.
- From the collected logs, identify, and mark out the issue.
- Contact Nevis support with the collected navajo.log, steps to reproduce the issue, and the necessary configuration files: web.xml and navajo.xml for the reproduction.
Connectivity issues
- Check the FAQ on connectivity issues.
- If the issue persists, enable the relevant trace groups, and try to reproduce the issue.
- Contact support with the collected navajo.log and apache.log, steps to reproduce the issue, and necessary configuration files: web.xml and navajo.xml.
HSM-related issues
- Try to isolate, and reproduce the issue locally.
- Get in touch with your HSM support vendor to verify the issue is not related to their libraries/engines, HSM configuration, or both.
- Try to replicate the issue with a plain Apache httpd server, setting numservers=1 and serverlimit=1.
- If all the above do not lead to a solution, and the HSM works as expected with a plain Apache, then follow the steps described in the general issues part of the FAQ.
Memory related issues
If the host runs out of memory after some days there may be a memory leak in one of the nevisproxy components. We regularly run memory leak tests on our own components but we can't exclude any leak on external software.
The following third-party libraries had some memory leak issues in the past:
- mariadb-connector-c version 3.1.10 or older