he SharePoint Scanner component performs full and incremental scans over a SharePoint repository, maintaining a snapshot of the repository and comparing it with the current content to establish what content has been updated. Updated content is then submitted to the configured pipeline in AspireObjects attached to Jobs. As well as the URL of the changed item, the AspireObject will also contain metadata extracted from the repository. Updated content is split in to three types: add, update, and delete. Each type of content is published on a different event so that it may be handled by different Aspire pipelines
The scanner reacts to an incoming job. This job may instruct the scanner to start, stop, pause, resume or cacheGroups. Typically the start job will contain all information required by the job to perform the crawl. However, the scanner can be configured with default values via application.xml file. When pausing or stopping, the scanner will wait until all the jobs it published have completed before itself completing.
SharePoint 2010 Scanner | |
---|---|
Factory Name | com.searchtechnologies.aspire:aspire-sharepoint-connector |
subType | default |
Inputs | AspireObject from a content source submitter holding all the information required for a crawl |
Outputs | Jobs from the crawl |
Configuration
This section lists all configuration parameters available to configure the SharePoint 2010 Scanner component.
General Scanner Component Configuration
Element | Type | Default | Description |
---|---|---|---|
snapshotDir | String | snapshots | The directory for snapshot files. |
numOfSnapshotBackups | int | 2 | The number of snapshots to keep after processing. |
waitForSubJobsTimeout | long | 600000 (=10 mins) | Scanner timeout while waiting for published jobs to complete. |
maxOutstandingTimeStatistics | long | 1m | The max about of time to wait before updating the statistics file. Whichever happens first between this property and maxOutstandingUpdatesStatistics will trigger an update to the statistics file. |
Branch Handler Configuration
Element | Type | Default | Description |
---|---|---|---|
branches/branch/@event | string | The event to configure - onAdd, onDelete or onUpdate. | |
branches/branch/@pipelineManager | string | The name of the pipeline manager to publish to. Can be relative. | |
branches/branch/@pipeline | string | The name of the pipeline to publish to. If missing, publishes to the default pipeline for the pipeline manager. | |
branches/branch/@allowRemote | boolean | Indicates if this pipeline can be found on remote servers (see Distributed Processing for details). | |
branches/branch/@batching | boolean | Indicates if the jobs processed by this pipeline should be marked for batch processing (useful for publishers or other components that support batch processing). | |
branches/branch/@batchSize | int | The max size of the batches that the branch handler will created. | |
branches/branch/@batchTimeout | long | Time to wait before the batch is closed if the batchSize hasn't been reached. | |
branches/branch/@simultaneousBatches | int | The max number of simultanous batches that will be handled by the branch handler. |
SharePoint 2010 Specific Configuration
Element | Type | Default | Description |
---|---|---|---|
userName | String | username | The user name to connect to SharePoint with, if one is not given in the control job. |
password | String | secretpassword | The password to connect to SharePoint with, if one is not given in the control job. |
domain | string | Domain used to authenticate against SharePoint. | |
defaultDisplayName | String | SharePoint | The name of the crawl, if one is not given in the control job. |
groupPrefixSeparator | String | @ | The separator inserted between the site URL and group name when extracting groups from sites. |
snapshotDir | String | . | The directory for snapshot files. |
standardWebService | boolean | false | Use standard SharePoint web services only. |
waitForSubJobsTimeout | long | 600000 (=10 mins) | Scanner time out while waiting for published jobs to complete. |
allowLimitedAccess | boolean | false | Indicates if an ACL should be set as allow or deny on an item where the user has Limited Access. |
scanRecursively | boolean | false | Indicates whether the child containers should be scanned or not. |
indexContainers | boolean | false | Indicates whether the container items should be indexed or not. |
claimsAuthentication | boolean | false | Indicates whether the group expansion should check for claims authentication or not |
useLDAPCache | boolean | false | Check for an installed "Aspire LDAP Cache" component for group expansion. |
externalGroupServerPath | string | empty | List of installed "Aspire LDAP Cache" components. |
Example Configuration
<component name="Scanner" subType="default" factoryName="aspire-sharepoint-connector"> <debug>${debug}</debug> <domain>${DefaultDomain}</domain> <username>${DefaultUserName}</username> <password>${DefaultPassword}</password> <standardWebService>${standardWebService}</standardWebService> <groupPrefixSeparator>@</groupPrefixSeparator> <allowLimitedAccess>${allowLimitedAccess}</allowLimitedAccess> <claimsAuthentication>${claimsAuthentication}</claimsAuthentication> <snapshotDir>${snapshotDir}</snapshotDir> <scanRecursively>${scanRecursively}</scanRecursively> <indexContainers>${indexContainers}</indexContainers> <useLDAPCache>${useLDAPCache}</useLDAPCache> <externalGroupServerPath>${externalGroupServerPath}</externalGroupServerPath> <branches> <branch event="onAdd" pipelineManager="../ProcessPipelineManager" pipeline="addUpdatePipeline" allowRemote="true" batching="true" batchSize="50" batchTimeout="60000" simultaneousBatches="2" /> <branch event="onUpdate" pipelineManager="../ProcessPipelineManager" pipeline="addUpdatePipeline" allowRemote="true" batching="true" batchSize="50" batchTimeout="60000" simultaneousBatches="2" /> <branch event="onDelete" pipelineManager="../ProcessPipelineManager" pipeline="deletePipeline" allowRemote="true" batching="true" batchSize="50" batchTimeout="60000" simultaneousBatches="2" /> </branches> </component>
Source Configuration
Scanner Control Configuration
The following table describes the list of attributes that the AspireObject of the incoming scanner job requires to correctly execute and control the flow of a scan process.
Element | Type | Options | Description |
---|---|---|---|
@action | string | start, stop, pause, resume, abort | Control command to tell the scanner which operation to perform. Use start option to launch a new crawl. |
@actionProperties | string | full, incremental | When a start @action is received, it will tell the scanner to either run a full or an incremental crawl. |
@normalizedCSName | string | Unique identifier name for the content source that will be crawled. | |
displayName | string | Display or friendly name for the content source that will be crawled. |
Header Example
<doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="0" jobNumber="0" normalizedCSName="FeedOne_Connector" scheduleId="0" scheduler="##AspireSystemScheduler##" sourceName="ContentSourceName"> ... <displayName>testSource</displayName> ... </doc>
@action | string | start, stop, pause, resume, abort | Control command to tell the scanner which operation to perform. Use start option to launch a new crawl. |
@actionProperties | string | full, incremental | When a start @action is received, it will tell the scanner to either run a full or an incremental crawl. |
@normalizedCSName | string | Unique identifier name for the content source that will be crawled. | |
displayName | string | Display or friendly name for the content source that will be crawled. |