Troubleshooting

This section provides help when you run into trouble while using nevisAdmin 4.

Log files

Application Logs

See nevisAdmin 4 Log Files.

Deployment Command Logs (Classic Mode)

Run the following command on a target host to find out which commands have been executed using sudo during deployment. This includes the restart commands of the Nevis components.

journalctl -xe | grep COMMAND

Upload of Large Files via the GUI

Problem: The upload of small files via the GUI or REST API works, but for larger files, it fails.

• nevisAdmin 4.4.0 and higher: Max file size is 100MB. For performance reasons, it is not recommended to upload larger files.

  • If you are hosting large files to be downloaded by end users, put them on a separate, non-Nevis web-server and configure it as a Web Application to reverse proxy requests for the large files.

  • If the large file is part of the Nevis configuration, set the 'nevisadmin.multipart.max-file-size' property to increase the upload size beyond 100MB:

    nevisadmin:
      multipart:
        max-file-size: 150MB

    Increasing the upload size beyond 100MB may cause severe performance issues. Contact Nevis Support for an alternative solution.

• nevisAdmin 4.2.3 and 4.3.0: Max file size was 10MB. See the above for instructions to change this.

• If the above settings are correct, but you still run into upload problems, check the following:

  Cause error in nevisadmin4.log

  2019-02-11 16:32:51,926 [qtp725832346-16] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - (conn=215) query size (1048576) is >= to max_allowed_packet (1048576

  or

  REST error for file upload

  {
    "error" : {
      "type" : "about:blank",
      "title" : "could not execute statement; nested exception is org.hibernate.exception.GenericJDBCException: could not execute statement",
      "sources" : [ ],
      "status" : 500
    }
  }

In case you encounter an exception similar to this then you need adjust the database settings. Proceed as follows:

• Locate your mariadb config file. Possible my.cnf locations:

  /etc/my.cnf
  /etc/mysql/my.cnf
  $MYSQL_HOME/my.cnf
  [datadir]/my.cnf
  ~/.my.cnf

• Add this line to the config (can be set bigger if you want to use files bigger than 10Mb):

  max_allowed_packet=10M

Deployment Connection (Classic Mode)

Consider the following points when there are troubles with the SSH deployment connection:

• Check the nevisAdmin 4 log file first. For more information about the log files, see nevisAdmin 4 Log Files.
• Set the following host-level variables in the inventory if nevisAdmin 4 cannot resolve the hostname:
  • __connection_host
  • __connection_port

You can also define __connection_user and __connection_password in the inventory. See Defining Classic Inventory Files for details.

• If no user and password are configured, nevisAdmin 4 will try to use a private key instead. The default path to the private key file is /home/nvbuser/.ssh/id_rsa. Check if the nevisadmin4.yml file contains the correct path. Also verify if the key is accessible for the runtime usernvbuser. If the private key is encrypted, set the passphrase in nevisadmin4.yml as well.

• Test the SSH connection on the shell with the ssh command:

  ssh <connection-user>@<connection-host> -i <path>/id_rsa

Reset the Password of the Local admin User

Only follow the procedure below if you lost the password of the admin user.

If you still have the password but want to change it, log in with the admin user and change the password via the drop-down menu in the top right corner of the screen.

1. Log in to the DB with the SQL client of your choice.

2. Run the following SQL statements:

   UPDATE user
   SET password='$shiro1$SHA-256$500000$PCclk+JLwTFwxYkyVcJHbQ==$ZdsfddJRblU2Zh1PQTAmqQyemuOZEEFpZHvMYqqMWkI='
   WHERE user_id='admin';

   This sets the admin user's password to "admin". It also makes the backup master key inaccessible.

3. Log in to nevisAdmin 4 with the credentials "admin" - "admin". During login, the backup master key will be recreated automatically.

4. Change the password via the drop-down menu in the top right corner of the screen.

The admin user's password also protects the encryption key (the backup master key) that allows you to recover access to secrets. In case you lost the admin password, also recreate the backup master key. The procedure above takes care of this.

For background information, see Encryption and Storage of Secrets (this chapter is only available in the documentation of nevisAdmin 4.7.0 and higher).

Clean Up a Target Host

After deploying a nevisAdmin 4 project on a target host, the Nevis components are up and running.

Now let's assume that you want to deploy a different project on the same host for testing purposes. This can cause issues for various reasons:

• Deployment may be prevented because the ports are in use.
• The configuration may be overwritten if the same instance names are used.
• Resources (e.g., CPU, RAM, file descriptors) may not be sufficient to run all instances.

It is therefore recommended that you clean up the target host. This consists of several steps:

1. Disable the Nevis instances.
2. Stop the running Nevis instances.
3. Remove the Nevis instances.

Disable Nevis Instances

Run the following commands to disable all Nevis instances that are currently enabled in systemd:

systemctl list-units -t service --full --plain --no-legend | cut -d' ' -f1 | grep nevis | grep -v nevisadmin | while read s; do systemctl disable $s; done

Disabling the instances ensures that they are not started when you reboot the host.

Stop Nevis Instances

Nevis components have a command line interface, which allows you to run the following command:

<component> all stop

However, if you do not know which Nevis components have been deployed on the host, it is easier to stop them all via systemd:

systemctl list-units -t service --full --plain --no-legend | cut -d' ' -f1 | grep nevis | grep -v nevisadmin | while read s; do systemctl stop $s; done

Monitoring-Related Instances

If you use Monitoring Patterns, then you can disable and stop everything related to Elastic Stack as follows:

systemctl list-units -t service --full --plain --no-legend | cut -d' ' -f1 | grep -E '(elastic|logstash|kibana|beat)' | while read s; do systemctl disable $s; systemctl stop $s; done

See the section below for the command to remove Nevis instances. Note that it also removes all files related to Elastic Stack.

Remove Nevis Instances

To remove a Nevis component, you can use the inst remove command provided by the component. However, if you do not know which Nevis components have been deployed, it is easier to remove all instances.

The following command removes all Nevis instances (except nevisAdmin 4) deployed on the target host:

# remove the Nevis instances, including Elastic Stack-related instances and all their data
find /var/opt/*/* | grep -v nevisadmin4 | grep -v edu | grep -v keybox | xargs rm -rf
# remove automatic key material deployed by nevisAdmin 4
rm -rf /var/opt/keys/*
# remove instance description files
rm -f /etc/nevis/*