Page History
Table of Contents |
---|
Jive Socialcast Scanner | |
---|---|
Factory Name | com.searchtechnologies.aspire:aspire-Socialcast-connector |
subType | default |
Inputs | AspireObject from a content source submitter holding all the information required for a crawl |
Outputs | Jobs from the crawl |
The Socialcast Scanner component performs full and incremental scans over a Jive Community, maintaining a snapshot of the community and comparing it with the current content to establish what content has been updated. Updated content is then submitted to the configured pipeline in AspireObjects attached to Jobs. As well as the URL of the changed item, the AspireObject will also contain metadata extracted from the repository. Updated content is split into three types -add, update and delete-. Each type of content is published as a different event so that it may be handled by different Aspire pipelines.
The scanner reacts to an incoming job. This job may instruct the scanner to start, stop, pause or resume. Typically the start job will contain all information required by the job to perform the crawl. However, the scanner can be configured with default values via application.xml file. When pausing or stopping, the scanner will wait until all the jobs it published have completed before completing itself.
Input Job Parameters
The scanner recognizes the following parameters on the control Job:
Element | Type | Description |
---|---|---|
id | int | The database id from the aspire_content_sources table. |
@crawlId | int | The crawl id from the content database. |
@action | String | The control job action - start, stop, pause or resume. |
@actionProperties | String | Properties for the crawl - full or incremental. |
connectorSource/displayName | String | The name of the crawl. |
connectorSource/url | String | The URL to scan. |
connectorSource/username | String | The username to connect to Socialcast Community with. |
connectorSource/password | String | The password to connect to Socialcast Community with. |
connectorSource/pageSize | String | Number of elements per page. |
Configuration
The scanner recognizes the following configuration parameters:
Element | Type | Default | Description |
---|---|---|---|
url | String | The url to the Socialcast Community. | |
username | String | The username to use when accessing Socialcast Community.. | |
password | String | The password to use when accessing Socialcast Community. | |
pageSize | int | 25 | The number of elements per page. |
maxRetries | int | 5 | The number of maximum retries for the connection before it throws error. |
timeOut | int | 10000 (10sec) | The number of milliseconds for timeout of the connection. |
snapshotDir | String | . | The directory for snapshot files. |
Branch Configuration
This component publishes to the onAdd, onDelete and onUpdate, so a branch must be configured for each of these three events.
Element | Type | Description |
---|---|---|
branches/branch/@event | string | The event to configure - onAdd, onDelete or onUpdate. |
branches/branch/@pipelineManager | string | The name of the pipeline manager to publish to. Can be relative. |
branches/branch/@pipeline | string | The name of the pipeline to publish to. If missing, publishes to the default pipeline for the pipeline manager. |
Example Configuration
Panel | ||
---|---|---|
| ||
<component name="SocialcastScanner" subType="scanner" factoryName="aspire-socialcast-scanner">
<debug>${debug}</debug>
<snapshotDir>${SnapshotDir}</snapshotDir>
<timeOut>15000</timeOut>
<maxRetries>3</maxRetries>
<branches>
<branch event="onAdd" pipelineManager="../ProcessPipelineManager" pipeline="add-update-pipeline" allowRemote="true"
batching="true"
batchSize="50"
batchTimeout="60000"
simultaneousBatches="2"/>
<branch event="onUpdate" pipelineManager="../ProcessPipelineManager" pipeline="add-update-pipeline" allowRemote="true"
batching="true"
batchSize="50"
batchTimeout="60000"
simultaneousBatches="2"/>
<branch event="onDelete" pipelineManager="../ProcessPipelineManager" pipeline="post-to-search-engine-pipeline" allowRemote="true"
batching="true"
batchSize="50"
batchTimeout="60000"
simultaneousBatches="2"/>
</branches>
</component> |
Source Configuration
Scanner Control Configuration
The following table describes the list of attributes that the AspireObject of the incoming scanner job requires to correctly execute and control the flow of a scan process.
Element | Type | Options | Description |
---|---|---|---|
@action | string | start, stop, pause, resume, abort | Control command to tell the scanner which operation to perform. Use start option to launch a new crawl. |
@actionProperties | string | full, incremental | When a start @action is received, it will tell the scanner to either run a full or an incremental crawl. |
@normalizedCSName | string | Unique identifier name for the content source that will be crawled. | |
displayName | string | Display or friendly name for the content source that will be crawled. |
Header Example
Panel | ||
---|---|---|
| ||
<doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="0" jobNumber="0" normalizedCSName="FeedOne_Connector"
scheduleId="0" scheduler="##AspireSystemScheduler##" sourceName="ContentSourceName">
...
<displayName>testSource</displayName>
...
</doc> |