You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Full crawls

Full crawls use the SQL from the Full crawl sql configuration and execute this against the database configured. Each selected row is formed into a document using the column names as document elements, and this document is sent to the Aspire workflow as an add. The document is creating with an id whose value is taken from the column identified in the Id column configuration. All columns selected will appear in the document sent to Aspire, although their position in the document will depend if they are recognised as standard Aspire Fields

Example SQL for a full crawls

Any valid sql that performs a select will allow you to perform a full crawl.

Simple

select * from main_table

Complex

select mt.id as ID, mt.first_name, mt.last_name, mt.first_name + " " + mt.last_name AS full_name, , mt.col3, jt.col1, jt.col2, jt.col3 from main_table mt, joined_table jt where mt.id = jt.id

Using slices

To increase performance of full crawls, you can opt to "slice" data. When selected the full crawl sql will be modified with a where clause based on the chosen number of slices and the row id. This means that rather than trying to select all rows from the data base in a single select, the connector will perform multiple smaller selects in parallel. 

Example

If you had full crawl sql of

select * from main_table

and chose to use 10 slices with an id column of id, the sql executed at the server would be

select * from main_table where id mod 10 = 0
select * from main_table where id mod 10 = 1
select * from main_table where id mod 10 = 2
select * from main_table where id mod 10 = 3
select * from main_table where id mod 10 = 4
select * from main_table where id mod 10 = 5
select * from main_table where id mod 10 = 6
select * from main_table where id mod 10 = 7
select * from main_table where id mod 10 = 8
select * from main_table where id mod 10 = 9



  • No labels