Scheduler

Upon startup, the Aspire Scheduler reads its configuration from the application.xml file and sets schedules within Quartz to represent each of the configured schedules. Each configuration will be given an id, known as the scheduleId. Optionally, the component will also attempt to read schedule information from a relational database.

The configuration of the schedules will include a “cron” style definition of the job execution time. The format of this definition is described later.

The Quartz scheduler is then started. If the scheduler and schedule are enabled, then when the scheduled time is reached, Quartz runs a java method. This method checks that the current schedule does not already have a job outstanding and publishes a job onto the configured Aspire pipeline. Each job published by the Aspire scheduler is given a unique job number known as the jobNumber.

Note:  The data in the Aspire job is configurable.

Cron Style Execution Schedule

The configuration stored in the system.xml file defines the execution time using a “cron” style format. The format used by Quartz differs from some “cron” implementations and is described below.

Cron expressions provide the ability to specify complex time combinations such as "At 8:00am every Monday through Friday" or "At 1:30am every last Friday of the month".

Cron expressions are comprised of 6 required fields and one optional field separated by white space. The fields respectively are described as follows:

The special characters are described below:

Notes

• Support for specifying both a day-of-week and a day-of-month value is not complete (you'll need to use the '?' character in one of these fields).
• Overflowing ranges is supported - that is, having a larger number on the left hand side than the right. You might do 22-2 to catch 10 o'clock at night until 2 o'clock in the morning, or you might have NOV-FEB. It is very important to note that overuse of overflowing ranges creates ranges that don't make sense and no effort has been made to determine which interpretation CronExpression chooses. An example would be "0 0 14-6 ? * FRI-MON".

Examples

• 0 10 20 * * ? This combination is legal and would fireup at 8:10pm on every day. Here, the star stands for Every month and every date and the question mark for Any day of the week.
• 0 10 20 ? * 1 This combination is legal and would fireup at 8:10pm on every Sunday. Here, the star stands for Every month and question mark for Any date.
• 0 10 20 * * 1 This combination IS NOT legal. Combination of All dates and Specific day is not accepted by Quartz.
• 0 10 20 ? ? SUN This combination IS NOT legal. Month can be specified as specific, a range, or All, but not Any.
• * 10 20 * * ? This combination is legal, but DANGEROUS as it would fire up 60 times, once for every second of the 10th minute after 8pm of every day.

Published Jobs

The basic configuration taken from the system.xml file allows the user to optionally specify in XML or JSON the data for the job that will be published to the pipeline. If specified, then the published job will be as configured, but the path to the scheduler, the sourceName, scheduleId and jobNumber will be added as attributes to the root tag (normally <doc>). The source id, action (start/stop/pause/resume), event type (scheduled/manual) and properties are also added.

If the job data is not specified, then an empty document is published onto the configured pipeline:

 <doc scheduler="/path/schedulerName" scheduleId="1" jobNumber="1" sourceName="myJob" sourceId="XXXX" actionProperties="full" actionType="manual" crawlId="123" action="start"/>

Note

• sourceId is only available when the schedule has come from an RDB.
• crawlId is only available when the schedule has come from an RDB and the rdb/sql/getCrawlId SQL is configured.

This method may be used to trigger sub job processing where the contents of the job is irrelevant, but something is needed to start processing at a scheduled time.

Once jobs are published they will run to completion. Jobs that error will be logged to the scheduler log file. Jobs could run indefinitely as they will not timeout.

User interface

The user interface allows the administrator to view and update the schedules via the normal Aspire web interface.

On browsing to the Aspire Schedulers status page, the administrator is able to see the current schedules and their status. This includes the schedule, event, whether the schedule is currently enabled, its last and next execution time and whether it is currently running (i.e. has submitted a job which has not yet completed). Clicking on this schedule provides further information about the schedule, such as the job data, pipeline and last error response.

From the status page, the administrator is able to enable or disable individual schedules and enable or disable the scheduler.

The administrator is also able to add a new schedule, specifying the schedule, event, and optionally whether the schedule is enabled and is a singleton.

The administrator may also manually "fire" events, causing jobs to be published on to the Aspire pipeline. The administrator may send "start", "stop", "pause" and "resume" jobs. These jobs will specify the action in the action attribute and show "manual" as the actionType attribute.

Configuration

The scheduler recognizes the following configuration tags.

The scheduler can read its schedules from a database. To configure this, the following configuration can be used:

Database Schedule Selection SQL

The SQL should return the mandatory columns and may return the optional columns from the following:

The format of the columns follows the formats given in the Basic Configuration section above. Column names can be enforced by use of the SQL “AS” keyword.

Database Job Control SQL

SQL contained in the jobRunningCheck, jobStarted, jobFinished and jobFailed may contain variables for substitution. Variables are surrounded with { } (see Simple Templates for more details). The following variables my be specified:

Branch Configuration

The Aspire Scheduler publishes jobs using the branch manager. Thus it requires the standard Branch Handler configuration detailed below:

Example Configuration

   <component name="myScheduler" subType="default" factoryName="aspire-scheduler">
     <schedules>
       <schedule name="myFirstSchedule" enabled="false">
         <cron>1/10 * * * * ?</cron>
         <event>onPublish</event>
         <job>
           <![CDATA[
           <doc>
             <fetchUrl>support.searchtechnologies.com</fetchUrl>
           </doc>
           ]]>
         </job>
       </schedule>
       <schedule enabled="false">
         <cron>2/10 * * * * ?</cron>
         <event>onPublish2</event>
       </schedule>
       <schedule enabled="false">
         <cron>3/10 * * * * ?</cron>
         <event>onPublish3</event>
         <job type="json">
           <![CDATA[
           {
             "doc" : {
               "fetchUrl" : "www.searchtechnologies.com"
             }
           }
           ]]>
         </job>
       </schedule>
       <schedule enabled="false">
         <cron>4/10 * * * * ?</cron>
         <event>onPublish4</event>
         <job type="json">
           <![CDATA[
           {
             "doc" : {
               "fetchUrl" : "repositories.searchtechnologies.com"
             }
           }
           ]]>
         </job>
       </schedule>
     </schedules>
     <branches>
       <branch event="onPublish" pipelineManager="PipelineManager" />
       <branch event="onPublish2" pipelineManager="PipelineManager" pipeline="myPipeline" />
       <branch event="onPublish3" pipelineManager="PipelineManager" pipeline="myPipeline" stage="myStage" />
       <branch event="onPublish4" pipelineManager="PipelineManager-not-exist" />
     </branches>
   </component>

Servlet Commands

The following servlet commands are available via the scheduler (via http://server:port/scheduler?cmd=XXXX&param=value):

Services Interface