he SharePoint 2013 Scanner component performs full and incremental scans over a SharePoint 2013 repository, maintaining the last SharePoint change token of the repository to get updates next time an incremental crawl is executed. Updated content is then submitted to the configured pipeline in AspireObjects attached to Jobs. As well as the URL of the changed item, the AspireObject will also contain metadata extracted from the repository. Updated content is split in to three types: add, update, and delete. Each type of content is published on a different event so that it may be handled by different Aspire pipelines.

The scanner reacts to an incoming job. This job may instruct the scanner to startstoppauseresume or cacheGroups. Typically the start job will contain all information required by the job to perform the crawl. However, the scanner can be configured with default values via application.xml file. When pausing or stopping, the scanner will wait until all the jobs it published have completed before itself completing.

SharePoint 2013 Scanner
Factory Namecom.searchtechnologies.aspire:aspire-sharepoint2013-connector
subTypedefault
InputsAspireObject from a content source submitter holding all the information required for a crawl
OutputsJobs from the crawl

Configuration


This section lists all configuration parameters available to configure the SharePoint 2013 Scanner component.

General Scanner Component Configuration

Basic Configuration

ElementTypeDefaultDescription
snapshotDirStringsnapshotsThe directory for snapshot files.
numOfSnapshotBackupsint2The number of snapshots to keep after processing.
waitForSubJobsTimeoutlong600000
(=10 mins)
Scanner timeout while waiting for published jobs to complete.
maxOutstandingTimeStatisticslong1mThe max about of time to wait before updating the statistics file. Whichever happens first between this property and maxOutstandingUpdatesStatistics will trigger an update to the statistics file.
maxOutstandingUpdatesStatisticslong1000The max number of files to process before updating the statistics file. Whichever happens first between this property and maxOutstandingTimeStatistics will trigger an update to the statistics file.
usesDomainbooleantrueIndicates if the group expansion request will use a domain\user format (useful for connectors that does not support domain in the group expander).

 

Branch Handler Configuration

This component publishes to the onAddonDelete and onUpdate, so a branch must be configured for each of these three events.

ElementTypeDescription
branches/branch/@eventstringThe event to configure - onAddonDelete or onUpdate.
branches/branch/@pipelineManagerstringThe name of the pipeline manager to publish to. Can be relative.
branches/branch/@pipelinestringThe name of the pipeline to publish to. If missing, publishes to the default pipeline for the pipeline manager.
branches/branch/@allowRemotebooleanIndicates if this pipeline can be found on remote servers (see Distributed Processing for details).
branches/branch/@batchingbooleanIndicates if the jobs processed by this pipeline should be marked for batch processing (useful for publishers or other components that support batch processing).
branches/branch/@batchSizeintThe max size of the batches that the branch handler will created.
branches/branch/@batchTimeoutlongTime to wait before the batch is closed if the batchSize hasn't been reached.
branches/branch/@simultaneousBatchesintThe max number of simultanous batches that will be handled by the branch handler.

 

SharePoint Scanner Configuration

ElementTypeDefaultDescription
userNameStringusernameThe user name to connect to SharePoint with, if one is not given in the control job.
passwordStringsecretpasswordThe password to connect to SharePoint with, if one is not given in the control job.
domainstring Domain used to authenticate against SharePoint.
defaultDisplayNameStringSharePoint2013The name of the crawl, if one is not given in the control job.
groupPrefixSeparatorString|The separator inserted between the site URL and group name when extracting groups from sites.
snapshotDirString.The directory for snapshot files.
waitForSubJobsTimeoutlong600000
(=10 mins)
Scanner time out while waiting for published jobs to complete.
scanRecursivelybooleanfalseIndicates whether the child containers should be scanned or not.
indexContainersbooleanfalseIndicates whether the container items should be indexed or not.
crawlAttachmentsbooleanfalseCrawl attachments from list items. E.g. documents attached to an Event.
crawlExtraSiteCollectionsbooleanfalseIndicates if the user will crawl more than one site collection.
subSiteCollections/siteCollectionUrlstringemptyList of sub site collections to crawl. More than one allowed.
useLDAPCachebooleanfalseCheck for an installed "Aspire LDAP Cache" component for group expansion.
externalGroupServerPathstringemptyList of installed "Aspire LDAP Cache" components.

 

Example Configuration

 <component name="Scanner" subType="default" factoryName="aspire-sharepoint2013-connector">
  <debug>${debug}</debug>
  <groupPrefixSeparator>${groupPrefixSeparator}</groupPrefixSeparator>
  <snapshotDir>${snapshotDir}</snapshotDir>
  <scanRecursively>${scanRecursively}</scanRecursively>
  <indexContainers>${indexContainers}</indexContainers>
  <crawlAttachments>${crawlAttachments}</crawlAttachments>
  <useLDAPCache>${useLDAPCache}</useLDAPCache>
  <externalGroupServerPath>${externalGroupServerPath}</externalGroupServerPath>
  <crawlExtraSiteCollections>${crawlExtraSiteCollections}</crawlExtraSiteCollections>
  <subSiteCollections>
    <siteCollectionUrl>${siteCollectionUrl}</siteCollectionUrl>
  </subSiteCollections>
  <branches>
    <branch event="onAdd" pipelineManager="../ProcessPipelineManager"
     pipeline="addUpdatePipeline" allowRemote="true" batching="true"
     batchSize="50" batchTimeout="60000" simultaneousBatches="2" />
    <branch event="onUpdate" pipelineManager="../ProcessPipelineManager"
     pipeline="addUpdatePipeline" allowRemote="true" batching="true"
     batchSize="50" batchTimeout="60000" simultaneousBatches="2" />
    <branch event="onDelete" pipelineManager="../ProcessPipelineManager"
     pipeline="deletePipeline" allowRemote="true" batching="true"
     batchSize="50" batchTimeout="60000" simultaneousBatches="2" />
  </branches>
</component>

 

Source Configuration

Scanner Control Configuration

The following table describes the list of attributes that the AspireObject of the incoming scanner job requires to correctly execute and control the flow of a scan process.

ElementTypeOptionsDescription
@actionstringstart, stop, pause, resume, abortControl command to tell the scanner which operation to perform. Use start option to launch a new crawl.
@actionPropertiesstringfull, incrementalWhen a start @action is received, it will tell the scanner to either run a full or an incremental crawl.
@normalizedCSNamestring Unique identifier name for the content source that will be crawled.
displayNamestring Display or friendly name for the content source that will be crawled.
Header Example
  <doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="0" jobNumber="0" normalizedCSName="FeedOne_Connector"
   scheduleId="0" scheduler="##AspireSystemScheduler##" sourceName="ContentSourceName">
    ...
    <displayName>testSource</displayName>
    ...
  </doc>

 

All configuration properties described in this section are relative to /doc/connectorSource of the AspireObject of the incoming Job.

ElementTypeDefaultDescription
urlstring The URL to scan (allowed http or https).
usernamestring The username to connect to SharePoint with.
passwordstring The password to connect to SharePoint with.
domainstring Domain used to authenticate against SharePoint.
indexContainersbooleanfalsetrue if folders (as well as files) should be indexed.
scanRecursivelybooleanfalsetrue if subfolders of the given URL should be scanned.
indexContainersbooleanfalseIndicates whether the container items should be indexed or not.
crawlAttachmentsbooleanfalseCrawl attachments from list items. E.g. documents attached to an Event.
crawlExtraSiteCollectionsbooleanfalseIndicates if the user will crawl more than one site collection.
subSiteCollections/siteCollectionUrlstringemptyList of sub site collections to crawl. More than one allowed.
fileNamePatterns/include/@patternregexnoneOptional. A regular expression pattern to evaluate file urls against; if the file name matches the pattern, the file is included by the scanner. Multiple include nodes can be added.
fileNamePatterns/include/@patternregexnoneOptional. A regular expression pattern to evaluate file urls against; if the file name matches the pattern, the file is excluded by the scanner. Multiple exclude nodes can be added.
Content Source Configuration Example
<doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="2" jobNumber="5" normalizedCSName="SharePoint2013" scheduleId="2" scheduler="AspireScheduler" sourceName="SharePoint2013">
  <connectorSource>
    <url>http://10.10.21.127/sites/aspire</url>
    <crawlExtraSiteCollections>true</crawlExtraSiteCollections>
    <subSiteCollections>
      <siteCollectionUrl>http://10.10.21.127/sites/mysite</siteCollectionUrl>
    </subSiteCollections>
    <domain>qa</domain>
    <username>sp_farm</username>
    <password>encrypted:562E81591F85B858E5A5D3876F9C9FDB</password>
    <scanRecursively>true</scanRecursively>
    <indexContainers>true</indexContainers>
    <crawlAttachments>true</crawlAttachments>
    <fileNamePatterns/>
  </connectorSource>
  <displayName>SharePoint2013</displayName>
</doc>

 

Output


<doc>
  <url>http://10.10.21.127/sites/aspire/_api/Web</url>
  <snapshotUrl>001 http://10.10.21.127/sites/aspire/_api/Web</snapshotUrl>
  <repItemType>aspire/sharePoint</repItemType>
  <docType>container</docType>
  <GUID>0f7bc97f-ac37-40b7-89be-5c009d79173b</GUID>
  <description/>
  <title>Aspire</title>
  <lastModified>2014-08-22T15:30:01Z</lastModified>
  <dateCreated>2014-01-15T23:38:39Z</dateCreated>
  <dataSize>0</dataSize>
  <displayUrl>http://10.10.21.127/sites/aspire</displayUrl>
  <id>http://10.10.21.127/sites/aspire/_api/Web</id>
  <fetchUrl>http://10.10.21.127/sites/aspire</fetchUrl>
  <sourceName>SharePoint2013</sourceName>
  <sourceType>sp2013</sourceType>
  <connectorSpecific type="sp2013">
    <field name="AllowRssFeeds">true</field>
    <field name="AppInstanceId">00000000-0000-0000-0000-000000000000</field>
    <field name="Configuration">0</field>
    <field name="Created">2014-01-15T23:38:39</field>
    <field name="CustomMasterUrl">/sites/aspire/_catalogs/masterpage/seattle.master</field>
    <field name="DocumentLibraryCalloutOfficeWebAppPreviewersDisabled">false</field>
    <field name="EnableMinimalDownload">true</field>
    <field name="Id">0f7bc97f-ac37-40b7-89be-5c009d79173b</field>
    <field name="Language">1033</field>
    <field name="LastItemModifiedDate">2014-08-22T15:30:01Z</field>
    <field name="MasterUrl">/sites/aspire/_catalogs/masterpage/seattle.master</field>
    <field name="QuickLaunchEnabled">true</field>
    <field name="RecycleBinEnabled">true</field>
    <field name="ServerRelativeUrl">/sites/aspire</field>
    <field name="SyndicationEnabled">true</field>
    <field name="Title">Aspire</field>
    <field name="TreeViewEnabled">false</field>
    <field name="UIVersion">15</field>
    <field name="UIVersionConfigurationEnabled">false</field>
    <field name="Url">http://10.10.21.127/sites/aspire</field>
    <field name="WebTemplate">STS</field>
  </connectorSpecific>
  <acls>
    <acl Permissions="Read, Limited Access, " Sid="s-1-5-21-3023650700-3092893521-1383129343-1112" access="allow" domain="qa" entity="user" fullname="qa\spadmin" name="spadmin" scope="global"/>
    <acl Permissions="Full Control, " access="allow" domain="" entity="group" fullname="902A40CF40D9CD503EE3199DA5D7F113|Aspire Owners" name="Aspire Owners" scope="machine"/>
    <acl Permissions="Read, Limited Access, " Sid="s-1-5-21-3023650700-3092893521-1383129343-1107" access="allow" domain="qa" entity="user" fullname="qa\sp_farm" name="sp_farm" scope="global"/>
    <acl Permissions="Edit, " access="allow" domain="" entity="group" fullname="902A40CF40D9CD503EE3199DA5D7F113|Aspire Members" name="Aspire Members" scope="machine"/>
    <acl Permissions="Read, " access="allow" domain="" entity="group" fullname="902A40CF40D9CD503EE3199DA5D7F113|Aspire Visitors" name="Aspire Visitors" scope="machine"/>
    <acl Permissions="View Only, " access="deny" domain="" entity="group" fullname="902A40CF40D9CD503EE3199DA5D7F113|Excel Services Viewers" name="Excel Services Viewers" scope="machine"/>
    <acl Permissions="Limited Access, " Sid="S-1-0-0" access="deny" domain="" entity="user" fullname="SHAREPOINT\system" name="System Account" scope="global"/>
    <acl Permissions="Limited Access, View Only, " access="deny" domain="" entity="group" fullname="Everyone" name="Everyone" scope="global"/>
  </acls>
  <hierarchy>
    <item id="73886DD2E0982FC87918CD9006F4456A" level="1" name="Aspire" type="aspire/sharePoint" url="http://10.10.21.127/sites/aspire"/>
  </hierarchy>
  <connectorSource>
    <url>http://10.10.21.127/sites/aspire</url>
    <crawlExtraSiteCollections>true</crawlExtraSiteCollections>
    <subSiteCollections>
      <siteCollectionUrl>http://10.10.21.127/sites/mysite</siteCollectionUrl>
    </subSiteCollections>
    <domain>qa</domain>
    <username>sp_farm</username>
    <password>encrypted:562E81591F85B858E5A5D3876F9C9FDB</password>
    <scanRecursively>true</scanRecursively>
    <indexContainers>true</indexContainers>
    <crawlAttachments>true</crawlAttachments>
    <fileNamePatterns/>
    <displayName>SharePoint2013</displayName>
  </connectorSource>
  <action>add</action>
  <content/>
</doc>