Introduction

RDB connector via Snapshots features include the following:

• Connects to database server using JDBC drivers (these must be downloaded separately)
• Performs full crawling
• Performs incremental crawling, so that only new/updated documents are indexed, using snapshot files
• Fetches data from the database using SQL statements
• Is search engine independent
• Runs from any machine with access to the given database

Environment and Access Requirements

Repository Support

JDBC Drivers

The RDB via Snapshots connects to databases via JDBC, so you'll need the appropriate JDBC client (driver) JAR file for the database you want to connect to. These are available for most (if not all) major database vendors, and your first port of call for the driver should be the vendor's website.

Account Privileges

A prerequisite for crawling any RDBMS is to have an RDBMS account. The recommended name for this account is "aspire_crawl_account" or something similar. The username and password for this account will be required below.

The "aspire_crawl_account" will need to have sufficient access rights to read all of the documents in the RDBMS that you wish to crawl.

To set the rights for your "aspire_crawl_account", do the following:

1. Log into the RDBMS as an Administrator.
2. Make the role of the "aspire_crawl_account" either administrator or superuser (so that it has access to all RDBMS content).

You will need this login information later in these procedures, when entering properties for your RDB Connector via Table.

Environment Requirements

No special requirements here

Framework and Connector Features

Framework Features

Connector Features

Retrieve Data per Batch

This mode uses SQL taken from the seed configuration (<discoverySQL>, <extractSQL>) and execute them against the database configured. Each resulting row is formed into a result object using the column names as document elements, and this document is submitted to a pipeline manager using the event configured for inserts. As the document is created, the value of the column identified in the seed configuration (<idColumn>) is noted as the primary key of the document. The value insert will be placed in the action attribute of the document.

Column names from the extractSQL query are added to the result object inside the "connectorSpecific" field. If the column names are standard Aspire fields, they are added to the root level.

Any change detected in the query set in discoverySQL field will be compare with the snapshot file and report the change if required.

Retrieve Everything

This mode uses SQL taken from the seed configuration (<fullSQL> or configuration) and execute them against the database configured. Each resulting row is formed into a  result object using the column names as document elements, and this document is submitted to a pipeline manager using the event configured for inserts. As the document is created, the value of the column identified in the seed configuration (<idColumn>) is noted as the primary key of the document. The value insert will be placed in the action attribute of the document.

Column names from SQL queries are added to the result object inside the "connectorSpecific" field. If the column names are standard Aspire fields, they are added to the root level.

Any change detected in the query set in fullSQL field will be compare with the snapshot file and report the change if required.

Content Crawled

The content retrieved by the connector is entirely defined using SQL statements, so you can select all or subsets of columns from one or more tables. Initially, the data is inserted in to Aspire using the returned column names, but this may be changed by further Aspire processing.

The RDB via Snapshots connector is able to crawl the following objects:

Limitations

No limitations defined