Skip to main content
Version: 2.18.x.x LTS

High-Level Overview

Introduction​

This chapter provides a high-level, feature-oriented overview of the nevisDataPorter component and provides information about the context in which nevisDataPorter can be used.

nevisDataPorter can be used to export data from a data source to a data sink keeping source and sink in sync. It can also be used to do batch data transformations as part of data migration tasks. The following picture gives a coarse overview of nevisDataPorter's architecture.

nevisDataPorter overview

A DataSource is used to retrieve objects (e.g., from a database). Each object retrieved by a DataSource is then passed down a chain of DataFilters. Each DataFilter may create Attributes for the object to be exported. The resulting object is then passed to a DataSink which stores the object (e.g., in LDAP directory).

Design Goals​

nevisDataPorter was designed in line with the following design goals:

  • Flexible and extensible data migration framework.
  • May run within a Java EE application server or standalone as a server or command-line application.
  • Flexible configuration language.
  • Detection of configuration errors during initialization.

Feature Overview​

Data Transformations​

When exporting data from a source to a sink, the characteristics of data entities may vary on the data source and data sink. To address this mismatch, data transformations need to be applied. The following transformation-demands need to be addressed:

Entity transformation​

  • An entity is a piece of data to be provisioned from a data source to a data sink.
  • An entity on the source corresponds to exactly one entity on the sink.
  • Multiple entities on the source correspond to one entity on the sink.
  • An entity on the source corresponds to multiple entities on the sink.

Attribute transformations​

  • A named subset of the data of an entity as denoted as an attribute.
  • An attribute on the source corresponds to exactly one attribute on the sink.
  • Multiple attributes on the source correspond to one attribute on the sink.
  • An attribute on the source corresponds to multiple attributes on the sink.
  • An attribute on the source needs a different encoding on the sink.

Data Source Implementations​

The following data source implementations are available:

  • The BeanDataSource uses Java objects as data sources.
  • The CSVDataSource uses CSV-files as its input.
  • The JDBCDataSource uses JDBC to access a database.
  • The LDAPDataSource uses JNDI to access an LDAP directory.
  • The NevisIDMSource uses a web service to access nevisIDM.
  • The NevisIDMHistorySource uses the history web service to access nevisIDM.
  • The SoapDataSource uses SOAP web services to request data.
  • The RestDataSource uses REST-ful web services to request data.
  • The XmlDataSource uses XML file as its input.

Data Sink Implementations​

The following data sink implementations are available:

  • The BeanDataSink adds all objects into a pre-existing list.
  • The CSVDataSink appends objects to a CVS-file.
  • The EmailDataSink uses SMTP to send e-mail messages.
  • The JDBCDataSink uses JDBC to update a database.
  • The JmsDataSink uses JMS to send messages.
  • The LDAPDataSink keeps an LDAP directory in sync with some data source.
  • The ActiveDirectoryUser sink is a derivation of the LDAPDataSink. It implements the special constraints Microsoft Active Directory enforces for user creation.
  • The NevisIDMSink uses a web service to update data stored in nevisIDM.
  • The NevisAgentDataSink sends scripts to NevisAgent and commands the agent to execute the scripts.
  • The SoapDataSink uses SOAP web services to send data.
  • The RestDataSink uses REST-ful web services to request data.
  • The TextDataSink is used to write plain text files.