Skip to main content
Version: 4.6.x.x LTS

Troubleshooting

This chapter describes some known issues and how to solve them.

RedHat 7.2 issue

The following error may occur if you leave the shell where you start nevisproxy: "Invalid argument: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully." This is due to a bug in RedHat 7.2. RedHat proposes the following workaround:

  • Edit /etc/systemd/logind.conf to add
  • RemoveIPC=no
  • Restart the application (nevisproxy) so that the changes take effect.

The following workaround works as well:

  • use the option '-t' if you login with ssh on the host where you start nevisproxy.

Performance optimization notice

  • To keep a high-performance functioning of regexp, remove irrelevant wildcard patterns with ".". For example, instead of ".someValue.**", you can use directly "someValue*".
  • Enable "keep-alive" between the proxy and the application.

Authentication with a client certificate and OpenSSL 1.1.1

As of release 3.14.3.0, authentication with a client certificate possibly does not work as expected anymore. This is due to the upgrade of nevisProxy to OpenSSL version 1.1.1. The problems may occur if you have set the parameter ClientCert of the IdentityCreationFilter to "optional" or "optional_no_ca".

There are two possible workarounds to fix this problem:

Via the IdentityCreationFilter​

If you want to fix this problem via the IdentityCreationFilter, set the filter parameter StoreClientCert to "true".

This fix works without any performance loss, but is not so secure: A user using a smart card will not be logged out automatically if he removes the smart card from the slot. To avoid this, use the crypto API of the browsers to notify the application of card removal events. Make the application act accordingly.

Via the ApacheConfigFilter​

If you want to fix this problem via the ApacheConfigFilter, configure the filter as follows:

 <filter>
 <filter-name>ClientCertApacheConfigFilter</filter-name>
 <filter-class>ch:nevis::navajo::apglue::httpd_${HTTPD_LIB_VERSION}_x::servlet::ApacheConfigFilter</filter-class>
 <filter-lib>/opt/nevisproxy/webapp/WEB-INF/lib/libApache${HTTPD_LIB_VERSION}_Servlet.so.1</filter-lib>
 <init-param>
 <param-name>
 LocationConfig
 </param-name>
 <param-value>
 SSLVerifyClient optional_no_ca
 SSLVerifyDepth 2
 </param-value>
 </init-param>
 </filter>

Pay attention to the following points:

  • Set the value of the parameter SSLVerifyClient to "optional", if the parameter ClientCert of the IdentityCreationFilteris also set to "optional" (<SSLVerifyClient> field).
  • Give the parameter SSLVerifyDepth the same value as the parameter ClientCert of the IdentityCreationFilter (<SSLVerifyDepth> field).
  • Map the filter to the same URL as the one set in the IdentityCreationFilter.

This is the most secure workaround. But it can slow down performance, especially if you use a smart card. To optimize performance, increase the value of the attribute MaxKeepAliveRequests and/or KeepAliveTimeout in the navajo.xml configuration file.

Depending on the smart card used, the user may be forced to enter its password after the keep-alive timeout expires.

Lost POST request body in LuaFilter in case of chunked transfer-encoding

The LuaFilter sometimes omits the body of a chunked POST request. This may happen if the LuaFilter comes before the IdentityCreationFilter in the filter chain.

To solve this problem, add the modifier flag RESET_BUFFERED_BODY to the configuration of the respective LuaFilter:

   <init-param>
    <param-name>ModifierFlags</param-name>
    <param-value>+RESET_BUFFERED_BODY</param-value>
   </init-param>

Increase the number of open file descriptors

Under heavy load, nevisProxy may log the following error message:

2021 04 28 12:48:31.306 4.5.0.0rc2 Apache 06027.140088593544960 3-ERROR : AH02179: apr_socket_accept: (client socket)
2021 04 28 12:48:31.769 4.5.0.0rc2 Apache 125211.140091297666880 3-ERROR : AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.

This could happen if the number of open file descriptors is too low.

To increase it only for the nevisProxy service, adapt the /usr/lib/systemd/system/[email protected] file by setting the LimitNOFILE parameter in the section, for example:

...
ExecStop=/opt/nevisproxy/bin/nevisproxy %i stop
KillMode=control-group

LimitNOFILE=10000

[Install]
WantedBy=multi-user.target

Then reload the new configuration by running the command systemctl daemon-reload.

Another solution is to increase it for the whole system by setting the parameter DefaultLimitNOFILE in the /etc/systemd/system.conf file:

...
#DefaultLimitRSS
DefaultLimitNOFILE=10000
#DefaultLimitAS
...

This modification requires reboot to take effect.

The new value can be verified after restarting the nevisProxy instance, it should log the resource limitations:

2021 05 03 16:12:42.114 5.5.5.5 NavajoStar 29674.140620893026112 5-NOTICE: using 'RLIMIT_FSIZE' hardlimit='-1' softlimit='-1'
2021 05 03 16:12:42.114 5.5.5.5 NavajoStar 29674.140620893026112 5-NOTICE: using 'RLIMIT_NOFILE' hardlimit='10000' softlimit='10000'
2021 05 03 16:12:42.114 5.5.5.5 NavajoStar 29674.140620893026112 5-NOTICE: using 'RLIMIT_STACK' hardlimit='-1' softlimit='8388608'

Or by calling the cat /proc/<pid>/limits command with the parent or child process of nevisProxy, for example:

10:55 $ cat /proc/38619/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63335 63335 processes
Max open files 10000 10000 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63335 63335 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us