Server scaling

Every server has limitations on how many parallel requests may be served, that is, processes and threads are limited; and on the amount of operating system resources such as file descriptors or memory.

Configuration options in nevisAuth

Java heap size

Configuration via JAVA_OPTS in the env.conf configuration file.

Recommendation	JAVA_OPTS
Default value	-Xms256m -Xmx1024m
Production system	-Xms2048m -Xmx2048m
Production system with substantial workload	-Xms4096m -Xmx4096m

info

• The authentication flows in nevisAuth are highly customizable; therefore, depending on your configuration, memory requirements can be very different.
• It is recommended to do performance tests in a test environment to determine the exact needs of your production environment.
• In Kubernetes Cluster setups, the sizing is not handled with JVM arguments but with Kubernetes Limits.

Worker threads

By default, nevisAuth uses a maximum of 200 worker threads to process incoming requests. This can be changed in the server configuration with the property max-threads.

The most common reason for the number of threads to reach maximum is:

• Threads blocking each other
• Bottlenecks in certain components
• Number of incoming requests require more parallel processing

caution

The worker thread value should not be changed. When modified, only in small incremental changes should be applied.

Using a value above 500 worker threads is not recommended.

Detecting number of used threads

You can also check the number of currently used threads on the OS level. Use the following command:

ps -o nlwp <PID> | tail -1 | xarg

caution

• Java has different mechanisms which use threads, garbage collection for example, so the number of threads overall for the process will be higher than the max number of worker threads.
• The worker threads are in a pool therefore they are ramped up only once. The currently running ones are occupied with processing previous requests.
  • Once the load subsides, the pool slowly ramps down.
  • Ramping up threads is CPU and memory intensive

Searching for bottlenecks

To identify slow parts inside nevisAuth, use the following loggers on INFO:

AuthPerf will display the processing time of each in DbPerformance will display response times for Remote session store and Out of context data store.

Detecting blocked threads

Bad nevisAuth response times together with low CPU and memory usage indicate blocking between working threads.

To investigate blocked threads, do the following:

1. Run the following command to create a thread dump:

   sudo -u nvauser jstack -l <PID> > /tmp/threaddump.txt
   caution

   jstack requires JDK, which is not installed on all systems by default. It is delivered in the nevisAppliance package.

2. Open and analyze /tmp/threaddump.txt.

   info

   Contact nevisAuth support if you are having trouble analyzing the thread dump.