Skip to main content
Version: 8.2411.x.x RR

Configuration of batch jobs

The following chapters describe how batch jobs are configured. For a more detailed description of the facility, refer to the Spring reference documentation and review the chapter on Scheduling/Quartz integration.

Batch context​

The batch context is an XML file containing a Spring application context. A typical configuration contains at least:

  • A scheduler factory (Spring SchedulerFactoryBean).
  • One or more batch job definitions (Spring JobDetailBeans)
  • One or more triggers referencing the jobs (Spring SimpleTriggerBeans or CronTriggerBeans)

Specify the path to the batch context in the nevisIDM config properties (see the chapter nevisidm-prod.properties.

By default, if you do not specify the job store type in the definition of scheduler factory, the RAMJobStore will be used. This means if you have more than one IDM instances, the scheduler parts work independently. All the instances will schedule the configured jobs and it is possible that more than one IDM instance schedules the same job. If your environment contains only one IDM instance or you have not configured jobs, failure in parallel execution would not impact your system. See the sample configuration with RAM job store: Example of a batch job configuration file with RAM job store.

In the batch context, you can use a org.springframework.scheduling.quartz.LocalDataSourceJobStore job store as well. The required tables are already created and are supported by the database with the TIDMQ_ prefix. The mentioned example contains the database level configuration. If your environment contains more than one IDM instances, use the database level JDBC job store. With the same configuration (with exactly the same trigger names) the clustered configuration will fire a trigger on one of IDM instances and there will no parallel executions. If you want parallel execution on more than one IDM instance, you need to configure more than one trigger for the jobs, as many as you want to fire in parallel. This mode of operation has not been tested fully: not for all job types and with special emphasis on the individually developed jobs. Consequently, proceed with caution when configuring such a use case. If there is more than one trigger for a job, it is possible that they trigger execution on the same IDM instance but there is a load balancing mechanism in Quartz in the background. The clustered scheduler configuration provides the opportunity to stop and start IDM instances. Quartz recognizes when a cluster member stopped and the trigger will fire on an other available IDM instance. If you have only one trigger for a single job, Quartz usually schedules the job on the same instance until the instance stops. See the sample configuration with JDBC job store: Example of a batch job configuration file with JDBC job store.

If you use the LocalDataSourceJobStore job store, the cluster flag org.quartz.jobStore.isClustered needs to be set to true and org.quartz.jobStore.useProperties needs to be set to false. These parameters guarantee the correct clustering operation. Without setting the isClustered property, if you have more than one IDM instance, you will face database-level primary key violation problems on the starting of a second instance. It is also very important to set the dataSource property of the Quartz scheduler to bean reference with the name dataSource. This passes the already configured DataSource instance to the JDBC job store.

You can see the mentioned example, all the job definitions and triggers have name and group properties. These values are stored in the database as job keys. When an IDM instance starts, Quartz checks the existence of a job or trigger record. If the record is missing, the scheduler inserts it, otherwise the scheduler lines up into the available instances where the triggers can fire jobs. This could cause a primary key violation in the first startup of new configuration on multiple IDM instance environment because more than one instance tries to insert the missing job or trigger details into the database tables.

The database level clustering merges configuration when you start your instances, or when you modify the batchContext.xml file. That is, nevisIDM incorporates settings from batchContext.xml into database tables. In the nevisIDM admin console, the list of jobs and trigger contain the possible newly defined elements.

Batch job​

Batch jobs are registered by defining a JobDetailBean for each job. See the chapter Example of a batch job configuration file with RAM job store and Example of a batch job configuration file with JDBC job store for examples. Job configuration parameters are supplied via the JobDataMap of the job. See Configuration of batch jobs.

Triggers​

Triggers are registered by defining trigger beans, such as the SimpleTriggerFactoryBean or the CronTriggerFactoryBean. The triggers registered with the scheduler factory will be displayed in the nevisIDM GUI and can also be triggered manually. The GUI shows the trigger description, the description of the associated job, and the next fire time.

Example of a batch job configuration file with RAM job store​

The following configuration fragment shows a sample batch context featuring a scheduler with the RAMJobStore (by defult), a single client data consistency job, and two triggers (simple and cron) that fire every 24 hours.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="
 http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

 <!-- scheduler factory -->
 <bean id="batchScheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
  <property name="jobDetails">
   <list>
    <ref bean="singleClientDataConsistencyJob"/>
   </list>
  </property>
  <property name="triggers">
   <list>
 <ref bean="dailySingleClientDataConsistencytTrigger"/>
 <ref bean="everyDayAtOneAmSingleClientDataConsistency"/>
 </list>
  </property>
  <property name="quartzProperties">
   <props>
    <prop key="org.quartz.threadPool.threadCount">8</prop>
   </props>
  </property>
 </bean>

 <!-- jobs -->
 <bean id="singleClientDataConsistencyJob" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
  <property name="description" value="Single client data consistency job"/>
  <property name="name" value="SingleClientDataConsistencyJob"/>
  <property name="group" value="BatchGroup"/>
  <property name="jobClass" value="ch.nevis.idm.batch.jobs.SingleClientDataConsistencyJob"/>
  <property name="durability" value="true"/>
 </bean>

 <!-- triggers -->
 <bean id="dailySingleClientDataConsistencytTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerFactoryBean">
       <property name="description" value="Daily single client data consistency"/>
        <property name="name" value="DailySingleClientDataConsistencytTrigger"/>
 <property name="group" value="BatchGroup"/>
 <property name="jobDetail" ref="singleClientDataConsistencyJob"/>
 <property name="repeatInterval" value="86400000"/>
 <property name="misfireInstructionName" value="MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT"/>
 </bean>

 <bean id="everyDayAtOneAmSingleClientDataConsistency" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
 <property name="description" value="Every day at 1am single client data consistency"/>
       <property name="name" value="EveryDayAtOneAmSingleClientDataConsistency"/>
 <property name="group" value="BatchGroup"/>
 <property name="jobDetail" ref="singleClientDataConsistencyJob" />
 <property name="cronExpression" value="0 0 1 * * ?" />
 </bean>
</beans>

Example of a batch job configuration file with JDBC job store​

The following configuration fragment shows a sample batch context featuring a scheduler with JDBCJobStore (with data source settings), a single client data consistency job, and two triggers (simple and cron) that fire every 24 hours.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="
 http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

 <!-- scheduler factory -->
 <bean id="batchScheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
  <property name="jobDetails">
   <list>
    <ref bean="singleClientDataConsistencyJob"/>
   </list>
  </property>
  <property name="triggers">
   <list>
 <ref bean="dailySingleClientDataConsistencytTrigger"/>
 <ref bean="everyDayAtOneAmSingleClientDataConsistency"/>
 </list>
  </property>
  <property name="dataSource">
   <ref bean="dataSource"/>
  </property>
  <property name="quartzProperties">
   <props>
    <prop key="org.quartz.scheduler.instanceId">AUTO</prop>
    <prop key="org.quartz.scheduler.instanceName">exportScheduler</prop>
    <prop key="org.quartz.scheduler.makeSchedulerThreadDaemon">true</prop>
    <prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>
    <prop key="org.quartz.threadPool.makeThreadsDaemons">true</prop>
    <prop key="org.quartz.threadPool.threadCount">8</prop>
    <prop key="org.quartz.jobStore.tablePrefix">TIDMQ_</prop>
    <prop key="org.quartz.jobStore.class">org.springframework.scheduling.quartz.LocalDataSourceJobStore</prop>
    <prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.StdJDBCDelegate</prop>
    <prop key="org.quartz.jobStore.isClustered">true</prop>
    <prop key="org.quartz.jobStore.useProperties">false</prop>
   </props>
  </property>
 </bean>

 <!-- jobs -->
 <bean id="singleClientDataConsistencyJob" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
  <property name="description" value="Single client data consistency job"/>
  <property name="name" value="SingleClientDataConsistencyJob"/>
  <property name="group" value="BatchGroup"/>
  <property name="jobClass" value="ch.nevis.idm.batch.jobs.SingleClientDataConsistencyJob"/>
  <property name="durability" value="true"/>
 </bean>

 <!-- triggers -->
 <bean id="dailySingleClientDataConsistencytTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerFactoryBean">
       <property name="description" value="Daily single client data consistency"/>
        <property name="name" value="DailySingleClientDataConsistencytTrigger"/>
 <property name="group" value="BatchGroup"/>
 <property name="jobDetail" ref="singleClientDataConsistencyJob"/>
 <property name="repeatInterval" value="86400000"/>
 <property name="misfireInstructionName" value="MISFIRE_INSTRUCTION_RESCHEDULE_NEXT_WITH_EXISTING_COUNT"/>
 </bean>

 <bean id="everyDayAtOneAmSingleClientDataConsistency" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
 <property name="description" value="Every day at 1am single client data consistency"/>
       <property name="name" value="EveryDayAtOneAmSingleClientDataConsistency"/>
 <property name="group" value="BatchGroup"/>
 <property name="jobDetail" ref="singleClientDataConsistencyJob" />
 <property name="cronExpression" value="0 0 1 * * ?" />
 </bean>
</beans>

Batch job configuration of JDBC job store for multiple type of databases​

Quartz supports multiple database systems with driver delegates, providing further specification instead of the general org.quartz.impl.jdbcjobstore.StdJDBCDelegate class. For more details, refer to the database specific configuration documents of Quartz.

One such example for specified driver delegate is the org.quartz.impl.jdbcjobstore.PostgreSQLDelegate for PostgreSQL databases.

nevisIDM utilizes this class automatically if the system runs with a PostgreSQL database configuration.