Setting up periodic update of IP geolocation and reputation mappings

There are two ways to set up database updates. The first option is to use built-in schedulers. The schedulers are either configured via the nevisAdapt Instance pattern in nevisAdmin 4 or directly defined in the nevisadapt.properties file. The other option is to use update templates. This chapter explains both options.

Setup with built-in schedulers

General scheduler configuration

You can configure the scheduler either via the nevisAdapt Instance pattern in nevisAdmin 4 or directly in the nevisadapt.properties file. Both methods need the following details to work:

• For the schedule definition: cron expression.

• For the download source: URL or URL parameters, depending on the provider. More details follow below.

  If these details are missing, no update will occur.

  As the database has to be downloaded (and potentially unzipped) first, the nevisAdapt service may start with some delay on new instances.

IP-to-location mapping

You can configure the update of the IP-to-location mapping in two ways, depending on the provider.

All options are mutually exclusive. Apply only one of them.

• The IP2LOCATION Update URL is known already. The following additional parameters are required:
  • The download token generated and distributed by IP2LOCATION.
  • File code
    • Free options: DB1LITEBIN, DB5LITEBIN, see https://lite.ip2location.com.
    • Commercial options (larger files with higher accuracy): DB1BIN, DB5BIN see https://www.ip2location.com.

The update has two phases: download and replace. The downloaded file is a .zip archive, which will be extracted before the actual update.

nevisAdapt does not provide any free or commercial licence for IP2LOCATION.

To set up an automated update, the access token must be generated once the registration/purchase is complete. One database can be downloaded up to 5 times per hour with the same token.

• The MaxMind Update URL is known already. The following additional parameters are required:

  • The license key is generated and distributed by MaxMind.
  • File code
    • Free options: GeoLite2-City, see http://dev.maxmind.com/geoip/geolite2-free-geolocation-data?lang=en.
    • Commercial options (larger files with higher accuracy): Geo2-City, see http://www.maxmind.com/en/geoip2-databases.

• Custom Upload a CSV file that matches the schema of either DB1 or DB5. This file will never be updated automatically. Also, the file must not exceed 20 MB in size. This option is recommended for testing purposes only.

IP Reputation mapping

The IP reputation only requires the URL to specify the source. By default, this is a free blacklist that is available at http://iplists.firehol.org/files/firehol_level1.netset

The update has two phases: download and replace.

Configuring schedulers with nevisAdmin 4 patterns

Find all details in the nevisAdmin 4 reference guide.

Configuring schedulers directly in the properties file

To configure the schedulers directly, open the nevisadapt.properties file and set the following entries (see the following table).

More Details

For more details, see the chapter nevisAdapt service - Description.

	IP-to-location mapping	IP Reputation mapping
Schedule	nevisadapt.ipToLocation.update.cron	nevisadapt.ipReputation.update.cron
Source	nevisadapt.ipToLocation.update.token nevisadapt.ipToLocation.update.fileSelector	nevisadapt.ipReputation.update.url
Download directory	nevisadapt.ipToLocation.update.directory	nevisadapt.ipReputation.update.directory
Target	nevisadapt.ipToLocationFilePath	nevisadapt.ipReputationFilePath
	Forward proxy host	nevisadapt.proxy.host
	Forward proxy port	nevisadapt.proxy.port

Setup with update templates

The Geolocation, IP Velocity and IP Reputation analyzers use external databases to get the required data for the risk score calculation.

• IP-to-location mapping:
  • IP2LOCATION DB1 (4 fields): IP start (decimal), IP end (decimal), 2-letter country code, country name OR
  • IP2LOCATION DB5 (8 fields): those of DB1 + county name, city name, latitude, longitude
• MaxMind mapping:
  • GeoIP/GeoLite City: similar to IP2LOCATION DB5
• IP blocklist:
  • FireHOL level 1 - minimum false positives (1 field): a single column of CIDR values

Maximum size of the geolocation database

If you use the extended version of the Geolocation CSV database (IP2LOCATION DB5 CSV) allocate at least 5 GB of memory for the nevisAdapt service. The size of this geolocation database file should not exceed 500 MB!

The above databases contain dynamically allocated IP ranges. Update these DBs regularly:

• IP2LOCATION: Updated on a monthly basis.
• MaxMind: Updated twice a week.
• FireHOL: This is a collection of other databases. Its content is generally updated on a daily basis.

To set up a periodic update, two .sh scripts are provided:

• IP2LOCATION / download-ip-location.sh The setup requires the DB code (for example, DB5LITE) and a user token that has access to the DB (even for the free version). The script:

  • downloads the .zip file,
  • looks for the new CSV,
  • checks it against the active one,
  • if different replaces it, and
  • cleans up the ZIP and the extracted content when done.

• FireHOL / download-ip-blacklist.sh FireHOL is a free service; the URL is optional (you can set a custom value if you use a different provider). The script:

  • downloads the new CSV,
  • checks it against the active one,
  • and if different, replaces it.

• MaxMind provide their own installer for Linux/Unix, details can be found here.

For both scripts, there is a crontab template prepared: crontab-ip-location_template and crontab-ip-blacklist_template respectively. Both templates have the same structure, with environment variables on top followed by the actual crontab entry.

1. Collect the following information:

• The desired target path for the CSV (should be the same as configured in the file nevisadapt.properties).
• The actual installation of the above shell scripts.
• The download URL for the provider
  • Also consider potential parameters, such as token and DB code for the location mapping.
• The forward proxy address (if needed).

2. Configure the specific template as needed. See also the following sample configuration for IP2LOCATION:

Template - Sample configuration for IP2LOCATION

# Default target path for the download - uncomment to set a custom value
IP2L_MAP_PATH="/var/opt/nevisadapt/conf/IP2LOCATION-LITE-DB5.CSV"
# Forward proxy definition - uncomment if needed
#IP2L_PROXY="[protocol://][user:password@]proxyhost[:port]"
# Set download URL - uncomment and define token and DB code
IP2L_MAP_URL="https://www.ip2location.com/download/?token=<IP2L_DOWNLOAD_TOKEN>&file=<IP2L_DATABASE_CODE>"


# *----------- minute (0 - 59)
# | *--------- hour (0 - 23)
# | | *------- day of month (1 - 31)
# | | | *----- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | *--- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | | -- avoid high download concurrency between clients with a random offset
  0 3 4 * * sleep $(( RANDOM % 3600 )); /var/opt/nevisadapt/conf/download-ip-location.sh

3. Add the result configuration as a new crontab entry:

$> crontab -e