Configuration Project Directory Structure and YAML File Formats using Git

Introduction

The nevisAdmin 4 GUI stores the current application state in a database. To interface with other systems, such as Git, nevisAdmin 4 can export the project configuration data into text file formats. This page describes the formats themselves and the directory structure in which they are stored.

There are three main features relying on this:

• Version control with Git. Git is designed for working with text files.
  • When a user has modified a project in the GUI and wants to publish the changes, the first step of Publishing local changes is to export the application state to YAML files and run the git commit and git push operations on them.
  • When importing data from the repository the same files are processed, and their content is added to the current application state.
• Exporting and importing a project (Zip).
  • The user can export the project directory structure in a Zip package and import it again. These operations are available from the GUI.
  • In this case the directory structure will also contain the project libraries as JAR files.
• The Generation Engine (aka DevOps mode): This is a stand-alone tool for generating and deploying the component configuration files. It works without a running nevisAdmin 4 application and does not rely on a database. Instead, it reads the configuration data from YAML files, which may be created by hand or can be checked out from the Git repository.

Working with Project Files Directly

When working with the GUI, you mostly do not see the files. But still there are some situations where you will encounter them:

• When using Git tools to perform merge / branch operations, the files and their contents will be shown.
• The file-level interface is useful to do bulk operations, such as:
  • creating many patterns in one go
  • adding many files
  • doing search/replace in a big project

The workflow for doing changes at the file level and then re-importing the changes into the GUI is as follows:

• Publish your project to Git
• Check out locally
• Work with the files directly. Refer to the sections below for information about the file structure to use and format conventions to adhere to.
• Push your changes to Git
• Update your project to see the changes in the GUI

At the same time changing the project in the GUI and remotely is not supported. The reason is that this could cause conflicts. The GUI is not designed to handle these.

Common Conventions

nevisAdmin 4 uses the following keywords in the project files

expression	Description	Permitted characters	Example
id	Identifier that is unique to an object (collision free). It is a sequence of hex encoded random bytes. Once generated and assigned, an ID may never change.	a-z A-Z 0-9	patternId = edd4e30213107e0941af28d8
key	A key is an identifier of an object or a field of an object that has recognition value to a user and may not be randomly generated. There are technical systems that rely on the key having a specific syntax or the key is used to identify objects toward a user. Keys may not change!	A-Z _ -	tenantKey = SIVEN; projectKey= SIVEN-FAB-MASTER
name	The user-given (or 'smart-generated') name of an object for display to users. This may change at any time and is not used for referencing objects. On principle all text characters are allowed, including punctuation, quotes, exotic character sets etc.	any character except single and double quotes and reverse ticks	patternName = "Mobile Banking Login"

File Format Versioning

Most YAML files start with the schemaVersion property. This versions the data format, including the possible properties and their semantics. When the format is changed in a major way, the schemaVersion will be increased. Currently the version is always "1.0".

Note, some files allow the use of complex data structures not defined in the file format: an example is the properties property in pattern YAML files. The schemaVersion does not change when such embedded structures are changed.

The export.yml file contains an exportVersion property instead of schemaVersion. This property contains the version of the export used to help detect format changes.

Project Key Syntax

Note: the project key is not used within the export / file system interface directly. It is provided by the context in which a project is imported or managed, this section is only here for clarification.

• Project keys MUST start with the tenant key
• Project keys MUST only use capital letters
• Project keys should list repository and branch for clarity, i.e. <TENANT>-<REPOSITORY>-<BRANCH>. This is the default suggested to users on creating a new project.
• The allowed characters are A-Z, 0-9 and underscore _ and dash -

Sanitation Rules for Using Object Names in File Names

Some names (ie: patternName) is used in the generated yaml file names. These values are provided by users with very little restrictions, and as such they are sanitized before they are used in file or directory names. To be able to do this, nevisAdmin 4

• first strips all diacritical marks
• replaces any non-alphanumeric characters a-zA-Z0-9 except - and _ by underscores _
• as a final step any sequence of multiple underscores __ is replaced by exactly one underscore.

This means that a generated file or directory name might be slightly different from the name entered in the GUI.

Project Directory Structure and File Formats

<project directory>/

The root directory of the project is given by the use case. This can be a sub-directory of a Git repository, a directory used by the Generation Engine, or a directory exported in a Zip package.

<project directory>/bundles.yml

The plugin bundles file lists all libraries and versions that the configuration project uses.

The project's pattern libraries are stored in the file bundles.yml.

bundles.yml

schemaVersion: "1.0"
bundles:
- "nevisadmin-plugin-nevisproxy:1.1.0-20181112T085833Z"
- "nevisadmin-plugin-monitoring:1.1.0-20181109T103230Z"
- "nevisadmin-plugin-base-generation:4.1.0-20181112T161418Z"
- "nevisadmin-plugin-nevisidm:1.1.0-20181112T095117Z"
- "nevisadmin-plugin-nevisauth:1.1.0-20181112T085934Z"

This is just a sequence of bundle keys with versions. Quotation marks are necessary otherwise YAML will parse as sequence of maps instead of as plain sequence.

<project directory>/bundles/

The plugin bundles directory contains the libraries used by the configuration project as JAR files. It is only present in Zip exports of projects.

<project directory>/bundles/<plugin-name>-<version>-<time-stamp>.jar

The file names are constructed from the name, version, and time-stamp of the library.

<project directory>/variables.yml

The variables file that defines all variables of the project and their default values.

All variables defined in a project are stored in the variables.yml file.

variables.yml

schemaVersion: "1.0"
variables:
  proxy-bind-address:    # Set by the user. Variable name must be strictly alphanumeric!
    className: "ch.nevis.admin.v4.plugin.base.generation.property.URLProperty" # defines the class representing the variable type
    parameters:          # configures the property type, mainly to be able to display on variables screen
      minRequired: 0
      portInputMode: "REQUIRED"
      hostNameInputMode: "REQUIRED"
      schemeInputMode: "REQUIRED"
      pathInputMode: "NONE"
      allowedSchemes: "http,https"
    value: "http://dummy-proxy-bind-address:8000"  # the sample value for this variable
    requireOverloading: true # true: during deployment, the target inventory must specify a value for this variable
  proxy-host-name:
    className: "ch.nevis.admin.v4.plugin.base.generation.property.URLProperty"
    parameters:
      minRequired: 1
      maxAllowed: 1
      portInputMode: "OPTIONAL"
      hostNameInputMode: "REQUIRED"
      schemeInputMode: "OPTIONAL"
      pathInputMode: "NONE"
      allowedSchemes: "http,https"
    value:
    - "http://dummy-proxy-host-name"
    requireOverloading: true # true: during deployment, the target inventory must specify a value for this variable

<project directory>/export.yml

The export file contains meta data about the exported project. It will be only present in Zip exports of projects.

The time of the Zip export can be found in the export.yml file.

export.yml

exportVersion: 1
timestamp: 2019-02-19T10:04:16.512Z

<project directory>/patterns/

The directory containing the patterns. Each configured pattern is stored in a separate file.

<project directory>/patterns/<pattern-name>_<pattern-id>.yml

Pattern file names are constructed out of the sanitized pattern name and its ID.

<project directory>/patterns/<pattern-id>_<property-name>/

Resources are stored in directories dedicated for pattern properties.

<project directory>/patterns/<pattern-id>_<property-name>/<file-name>.<file-extension>

The files keep their original names.

Patterns file structure

schemaVersion: "1.0"
pattern:
  id: "bd73c0740ae895fc8ae722f9"    # unique ID identifying the pattern, do not change after initial deployment
  className: "ch.nevis.admin.v4.plugin.nevisproxy.patterns.NevisProxyDeployable" # the type (class) of the pattern
  name: "Webshop Proxy"
  deploymentHosts: "proxy"          # deployment target of the pattern (in case of instance patterns)
  label: "webshop"                  # custom group the pattern is assigned to
  link:                             # If the pattern was copied from another project, the link holds a reference to the original pattern
    sourceProjectKey: "DEFAULT-WEBSHOP-MASTER"
    sourcePatternId: "bd73c0740ae895fc8ae722f9"
    author: "admin"
    lastCopied: "2020-08-13T09:32:56Z"
  properties:                       # properties are defined via their keys. All further data about properties is defined by the pattern type
    host:
    - "pattern://6abafff4a15cd1c122d1e4f6"              # referencing another pattern
    path: "/webshop/"
    backends: "var://webshop-backend"                   # referencing a variable

Notes:

• For a list of available patterns/properties, see the reports Pattern library and Pattern Library - Properties Summary.
• Variables are referenced by setting a scalar property value with the prefix var:// followed by the variable name. Any property can be set to a var:// value instead of the in-line value.
• Patterns are referenced by setting a scalar property value with the prefix pattern:// followed by the pattern ID. These can appear only for properties of type PatternReferenceProperty.
• See also the section File Format Versioning above.

Pattern Resources (Attached Files)

For various patterns, files can be uploaded to further customize the behavior of Nevis. For example:

• Defining custom error pages
• Uploading JARs with custom AuthStates
• Managing mod_security rules files
• Configuring login screen templates
• Adding resource files such as images and CSS definitions that are required by HTML pages or templates

Resources must belong to a pattern property. They are stored in the same folder as the patterns, in sub-folders called <pattern-id>_<property-name>. The files inside this directory keep their original names and extensions. To add a file manually it's enough to copy it to this folder.

If the folder does not exist yet, it means that the property is not yet set for this pattern. In this case first set the property on the pattern like <property-name>: "res://<pattern-id>#<property-name>", then create the resource's sub-directory and copy the file(s) in there.