Jira Scanner (Aspire 2)

The Jira Scanner component performs full and incremental scans over a Jira repository, maintaining a snapshot of the repository and comparing it with the current content to establish what content has been updated. Updated content is then submitted to the configured pipeline in AspireObjects attached to Jobs. As well as the URL of the changed item, the AspireObject will also contain metadata extracted from the repository. Updated content is split in to three types: add, update, and delete. Each type of content is published on a different event so that it may be handled by different Aspire pipelines

The scanner reacts to an incoming job. This job may instruct the scanner to start, stop, pause or resume. Typically the start job will contain all information required by the job to perform the crawl. However, the scanner can be configured with default values via application.xml file. When pausing or stopping, the scanner will wait until all the jobs it published have completed before itself completing.

Jira Scanner (Aspire 2)
Factory Name	com.searchtechnologies.aspire:aspire-jira-connector
subType	default
Inputs	AspireObject from a content source submitter holding all the information required for a crawl
Outputs	Jobs from the crawl

Input Job Parameters

The scanner recognizes the following parameters on the control Job:

Element	Type	Description
id	int	The database id from the aspire_content_sources table.
@crawlId	int	The crawl id from the content database.
@action	String	The control job action - start, stop, pause or resume.
@actionProperties	String	Properties for the crawl - full or incremental.
connectorSource/displayName	String	The name of the crawl.
connectorSource/URL	String	The URL to Jira instance.
connectorSource/User	String	Login of the Jira account.
connectorSource/Password	string	Password for Jira accound.
connectorSource/PageSize	integer	Indicates the amount of documents or folders that will be returned by the API each call.
connectorSource/excludeExtensions	string	Indicates the list of extensions (separeted by comma) you don't want to extract the text, for instance dll or exe.
connectorSource/ExcludeSubFolders/include	string	Optional. A list of folders that can be excluded from the crawling.
connectorSource/fileNamePatterns/include	regex	Optional. A regular expression pattern to evaluate file urls against; if the file name matches the pattern, the file is included by the scanner. Multiple include nodes can be added.
connectorSource/fileNamePatterns/exclude	regex	Optional. A regular expression pattern to evaluate file urls against; if the file name matches the pattern, the file is excluded by the scanner. Multiple exclude nodes can be added.

Configuration

The scanner recognizes the following configuration parameters:

Element	Type	Default	Description
url	String		The url to the Jira instance.
username	String		The username to use when accessing Jira instance.
password	String		The password to use when accessing Jira instance.
pageSize	int	100	The number of elements per page.
snapshotDir	String	.	The directory for snapshot files.

Branch Configuration

This component publishes to the onAdd, onDelete and onUpdate, so a branch must be configured for each of these three events.

Element	Type	Description
branches/branch/@event	string	The event to configure - onAdd, onDelete or onUpdate.
branches/branch/@pipelineManager	string	The name of the pipeline manager to publish to. Can be relative.
branches/branch/@pipeline	string	The name of the pipeline to publish to. If missing, publishes to the default pipeline for the pipeline manager.

Example Configuration

        <component name="Scanner" subType="scanner"
          factoryName="aspire-jira-scanner">
          <debug>${debug}</debug>
          <fullRecovery>${fullRecovery}</fullRecovery>
          <incrementalRecovery>${incrementalRecovery}</incrementalRecovery>
          <metadataMap>
            <map from="action" to="action" />
            <map from="doc-type" to="docType" />
            <map from="last-modified-date" to="lastModified" />
            <map from="content-length-bytes" to="dataSize" />
            <map from="owner" to="owner" />
          </metadataMap>
          <snapshotDir>${snapshotDir}</snapshotDir>
          <fileNamePatterns>
            <include pattern=".*" />
            <exclude pattern=".*tmp$" />
          </fileNamePatterns>
          <emitCrawlStartJobs>${emitStartJob}</emitCrawlStartJobs>
          <emitCrawlEndJobs>${emitEndJob}</emitCrawlEndJobs>
          <waitForSubJobsTimeout>${waitForSubJobs}</waitForSubJobsTimeout>
          <enableAuditing>${enableAuditing}</enableAuditing>
          <!-- Group cache -->
          <useExternalGroups>${useExternalGroups}</useExternalGroups>
          <externalGroupServerPath>${externalGroupServerPath}</externalGroupServerPath>
          <geCache lowercase="${geLowerCase}">
            <domain strip="${geStripDomain}" add="${geAddDomain}"/>
          </geCache>
            <failedDocumentsService>${fdServiceUrl}</failedDocumentsService>
          <branches>
            <branch event="onAdd" pipelineManager="../ProcessPipelineManager"
              pipeline="addUpdatePipeline" allowRemote="true" batching="true"
              batchSize="${batchSize}" batchTimeout="${batchTimeout}" simultaneousBatches="2" />
            <branch event="onUpdate" pipelineManager="../ProcessPipelineManager"
              pipeline="addUpdatePipeline" allowRemote="true" batching="true"
              batchSize="${batchSize}" batchTimeout="${batchTimeout}" simultaneousBatches="2" />
            <branch event="onDelete" pipelineManager="../ProcessPipelineManager"
              pipeline="deletePipeline" allowRemote="true" batching="true"
              batchSize="${batchSize}" batchTimeout="${batchTimeout}" simultaneousBatches="2" />
            <branch event="onCrawlStart" pipelineManager="../ProcessPipelineManager"
              pipeline="crawlStartEndPipeline" allowRemote="true"/>
            <branch event="onCrawlEnd" pipelineManager="../ProcessPipelineManager"
              pipeline="crawlStartEndPipeline" allowRemote="true"/>
          </branches>
        </component>
      </components>
    </component>

Source Configuration

Scanner Control Configuration

The following table describes the list of attributes that the AspireObject of the incoming scanner job requires to correctly execute and control the flow of a scan process.

Element	Type	Options	Description
@action	string	start, stop, pause, resume, abort	Control command to tell the scanner which operation to perform. Use start option to launch a new crawl.
@actionProperties	string	full, incremental	When a start @action is received, it will tell the scanner to either run a full or an incremental crawl.
@normalizedCSName	string		Unique identifier name for the content source that will be crawled.
displayName	string		Display or friendly name for the content source that will be crawled.

Header Example

  <doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="0" jobNumber="0" normalizedCSName="FeedOne_Connector"
   scheduleId="0" scheduler="##AspireSystemScheduler##" sourceName="ContentSourceName">
    ...
    <displayName>testSource</displayName>
    ...
  </doc>

Page tree