Page History

Configuration

This section lists all configuration parameters available to configure the RDB via Snapshots Scanner component.

General Scanner Component Configuration

Basic Scanner Configuration

Branch Handler Configuration

This component publishes to the onAdd, onDelete and onUpdate, so a branch must be configured for each of these three events.

File System Specific Configuration

Configuration Example

<component name="RDBMSRDB" subType="default" factoryName="aspire-multiple-rdb">
  <debug>true</debug>
  <timeout>30m</timeout>
  <purgePoll>5m</purgePoll>
</component>

<component name="Scanner" subType="default" factoryName="aspire-rdb-connector">
  <debug>true</debug>
  <rdb>./RDBMSRDB</rdb>
  <branches>
    <branch event="onAdd" pipelineManager="../ProcessPipelineManager"
      pipeline="addUpdatePipeline" allowRemote="true" batching="true"
      batchSize="50" batchTimeout="60000" simultaneousBatches="2" />
    <branch event="onUpdate" pipelineManager="../ProcessPipelineManager"
      pipeline="addUpdatePipeline" allowRemote="true" batching="true"
      batchSize="50" batchTimeout="60000" simultaneousBatches="2" />
    <branch event="onDelete" pipelineManager="../ProcessPipelineManager"
      pipeline="deletePipeline" allowRemote="true" batching="true"
      batchSize="50" batchTimeout="60000" simultaneousBatches="2" />
  </branches>
</component>

Source Configuration

Scanner Control Configuration

The following table describes the list of attributes that the AspireObject of the incoming scanner job requires to correctly execute and control the flow of a scan process.

Header Example

  <doc action="start" actionProperties="full" actionType="manual" crawlId="0" dbId="0" jobNumber="0" normalizedCSName="FeedOne_Connector"
   scheduleId="0" scheduler="##AspireSystemScheduler##" sourceName="ContentSourceName">
    ...
    <displayName>testSource</displayName>
    ...
  </doc>

All configuration properties described in this section are relative to /doc/connectorSource of the Aspire Object of the incoming Job.

Scanner Configuration Example

<doc action="start" actionProperties="full" normalizedCSName="My_RDB_Source">
  <connectorSource>
    <useSlices>false</useSlices>
    <retrievePerBatch>true</retrievePerBatch>
    <discoverySQL>Select ID from main_data</discoverySQL>
    <extractionSQL>Select ID from main_data WHERE ID IN {IDS}</extractionSQL>
    <idColumn>ID</idColumn>
    <idColumnString>false</idColumnString>
    <indexContainers>false</indexContainers>
    <waitForSubJobs>600000</waitForSubJobs>
    <aclMethod>SQL</aclMethod>
    <aclSQL>Select 'mydomain\\user@NT' as ACL from main_data</aclSQL>
    <displayName>RBMS snapshots</displayName>
    <dbUrl>jdbc:mysql://127.0.0.1:3306/test</dbUrl>
    <dbUser>root</dbUser>
    <dbPassword>encrypted:771FA372D6F46E9F0C058B359683E8CB</dbPassword>
    <dbDriverJar>lib/mysql-connector-java-5.1.27-bin.jar</dbDriverJar>
    <dbProps/>
    <dbDriverClass/>
  </connectorSource>
  <displayName>My RDB Source</displayName>
</doc>

Output

<doc>
  <url>5</url>
  <id>5</id>
  <displayUrl>5</displayUrl>
  <snapshotUrl>001 5</snapshotUrl>
  <repItemType>aspire/row</repItemType>
  <fetchId>5</fetchId>
  <fetchUrl>5</fetchUrl>
  <rdb>../MultiConnectionPool</rdb>
  <docType>item</docType>
  <sourceName>RBMS_snapshots</sourceName>
  <sourceType>database</sourceType>
  <connectorSource>
    <useSlices>false</useSlices>
    <retrievePerBatch>true</retrievePerBatch>
    <discoverySQL>Select ID from main_data</discoverySQL>
    <extractionSQL>Select ID from main_data WHERE ID IN {IDS}</extractionSQL>
    <idColumn>ID</idColumn>
    <idColumnString>false</idColumnString>
    <indexContainers>false</indexContainers>
    <waitForSubJobs>600000</waitForSubJobs>
    <aclMethod>SQL</aclMethod>
    <aclSQL>Select 'mydomain\\user@NT' as ACL from main_data</aclSQL>
    <displayName>RBMS snapshots</displayName>
    <dbUrl>jdbc:mysql://127.0.0.1:3306/test</dbUrl>
    <dbUser>root</dbUser>
    <dbPassword>encrypted:771FA372D6F46E9F0C058B359683E8CB</dbPassword>
    <dbDriverJar>lib/mysql-connector-java-5.1.27-bin.jar</dbDriverJar>
    <dbProps/>
    <dbDriverClass/>
    <contentSourceId>RBMS_snapshots</contentSourceId>
  </connectorSource>
  <action>add</action>
  <connectorSpecific>
    <field name="id">5</field>
  </connectorSpecific>
  <acls>
    <acl access="allow" domain="mydomain" entity="group" fullname="mydomain\user@NT" name="user@NT" scope="global"/>
  </acls>
</doc>

Scanner Operation

Retrieve data per batch

This mode uses SQL taken from the job (<connectorSource/discoverySQL>, <connectorSource/extractSQL> or configuration) and execute them against the database configured via a Multi RDBMS Connection Pool stage. Each resulting row is formed into an AspireObject using the column names as document elements, and this document is submitted to a pipeline manager using the event configured for inserts. As the document is created, the value of the column identified in the job (<connectorSource/idColumn>) is noted as the primary key of the document. The value insert will be placed in the action attribute of the document.

Column names from the extractSQL query are added to the AspireObject inside the "connectorSpecific" field. If the column names are standard AspireObject fields, they are added to the root level. See Connector Metadata for further details on which are standard fields.

Example AspireObject from a retrieve data per batch

<doc>
  <url>5</url>
  <id>5</id>
  <displayUrl>5</displayUrl>
  <snapshotUrl>001 5</snapshotUrl>
  <repItemType>aspire/row</repItemType>
  <fetchId>5</fetchId>
  <fetchUrl>5</fetchUrl>
  <rdb>../MultiConnectionPool</rdb>
  <docType>item</docType>
  <sourceName>RBMS_snapshots</sourceName>
  <sourceType>database</sourceType>
  <connectorSource>
    <useSlices>false</useSlices>
    <retrievePerBatch>true</retrievePerBatch>
    <discoverySQL>Select ID from main_data</discoverySQL>
    <extractionSQL>Select ID from main_data WHERE ID IN {IDS}</extractionSQL>
    <idColumn>ID</idColumn>
    <idColumnString>false</idColumnString>
    <indexContainers>false</indexContainers>
    <waitForSubJobs>600000</waitForSubJobs>
    <aclMethod>SQL</aclMethod>
    <aclSQL>Select 'mydomain\\user@NT' as ACL from main_data</aclSQL>
    <displayName>RBMS snapshots</displayName>
    <dbUrl>jdbc:mysql://127.0.0.1:3306/test</dbUrl>
    <dbUser>root</dbUser>
    <dbPassword>encrypted:771FA372D6F46E9F0C058B359683E8CB</dbPassword>
    <dbDriverJar>lib/mysql-connector-java-5.1.27-bin.jar</dbDriverJar>
    <dbProps/>
    <dbDriverClass/>
    <contentSourceId>RBMS_snapshots</contentSourceId>
  </connectorSource>
  <action>add</action>
  <connectorSpecific>
    <field name="id">5</field>
  </connectorSpecific>
  <acls>
    <acl access="allow" domain="mydomain" entity="group" fullname="mydomain\user@NT" name="user@NT" scope="global"/>
  </acls>
</doc>

Retrieve everything

This mode uses SQL taken from the job (<connectorSource/fullSQL> or configuration) and execute them against the database configured via a Multi RDBMS Connection Pool stage. Each resulting row is formed into an AspireObject using the column names as document elements, and this document is submitted to a pipeline manager using the event configured for inserts. As the document is created, the value of the column identified in the job (<connectorSource/idColumn>) is noted as the primary key of the document. The value insert will be placed in the action attribute of the document.

Column names from SQL queries are added to the AspireObject inside the "connectorSpecific" field. If the column names are standard AspireObject fields, they are added to the root level. See Connector AspireObject Metadata for further details on which are standard fields.

<doc>
  <url>5</url>
  <id>5</id>
  <displayUrl>5</displayUrl>
  <snapshotUrl>001 5</snapshotUrl>
  <repItemType>aspire/row</repItemType>
  <fetchId>5</fetchId>
  <fetchUrl>5</fetchUrl>
  <rdb>../MultiConnectionPool</rdb>
  <docType>item</docType>
  <sourceName>RBMS_snapshots</sourceName>
  <sourceType>database</sourceType>
  <connectorSource>
    <useSlices>false</useSlices>
    <retrievePerBatch>false</retrievePerBatch>
    <fullSQL>Select * from main_data</fullSQL>
    <idColumn>ID</idColumn>
    <idColumnString>false</idColumnString>
    <indexContainers>false</indexContainers>
    <waitForSubJobs>600000</waitForSubJobs>
    <aclMethod>SQL</aclMethod>
    <aclSQL>Select 'mydomain\\user@NT' as ACL from main_data</aclSQL>
    <displayName>RBMS snapshots</displayName>
    <dbUrl>jdbc:mysql://127.0.0.1:3306/test</dbUrl>
    <dbUser>root</dbUser>
    <dbPassword>encrypted:771FA372D6F46E9F0C058B359683E8CB</dbPassword>
    <dbDriverJar>lib/mysql-connector-java-5.1.27-bin.jar</dbDriverJar>
    <dbProps/>
    <dbDriverClass/>
    <contentSourceId>RBMS_snapshots</contentSourceId>
  </connectorSource>
  <action>add</action>
  <connectorSpecific>
    <field name="id">5</field>
  </connectorSpecific>
  <acls>
    <acl access="allow" domain="mydomain" entity="group" fullname="mydomain\user@NT" name="user@NT" scope="global"/>
  </acls>
</doc>

Slicing Full SQL

This feature consists in splitting the Full crawl SQL into multiple queries, for the scanning process to be faster. Slicing the fullSQL should significantly improve the performance when scanning large databases. Only works when idColumn is an integer.

Full crawl SQL Example (with slicing)

Having the following fullSQL:

  SELECT C.page_id, C.modified_date, 
    FROM mw_content C, mw_published P, mw_old T
    WHERE P.rev_id = C.page_latest AND T.old_id = P.rev_text_id;

For the connector to support slices you should add the {SLICES} tag into your WHERE clause in your fullSQL as shown in the next example:

  SELECT C.page_id, C.modified_date,
    FROM mw_content C, mw_published P, mw_old T
    WHERE P.rev_id = C.page_latest AND T.old_id = P.rev_text_id AND {SLICES};

In this case, there already was a WHERE clause, so we added "AND {SLICES}". In the case you don't need any condition, your query must contain WHERE {SLICES} for slices to work.

Slicing Retrieve data per batch SQL

This feature consists in splitting the discoverySQL into multiple queries, for the scanning process to be faster. Slicing the extractionSQL should significantly improve the performance when scanning large databases. Only works when idColumn is an integer.

Explore SQL Example (with slicing)

Having the following discoverySQL:

  SELECT C.page_id, C.modified_date, 
    FROM mw_content C, mw_published P, mw_old T
    WHERE P.rev_id = C.page_latest AND T.old_id = P.rev_text_id AND P.rev_id IN {IDS};

For the connector to support slices you should add the {SLICES} tag into your WHERE clause in your extractionSQL as shown in the next example:

  SELECT C.page_id, C.modified_date,
    FROM mw_content C, mw_published P, mw_old T
    WHERE P.rev_id = C.page_latest AND T.old_id = P.rev_text_id AND P.rev_id IN {IDS} AND {SLICES};

In this case, there already was a WHERE clause, so we added "AND {SLICES}". In the case you don't need any condition, your query must contain WHERE {SLICES} for slices to work.