Page History

Full crawls

Full crawls use the SQL from the Full crawl sql configuration and execute this against the database configured. Each selected row is formed into a document using the column names as document elements, and this document is sent to the Aspire workflow as an add. The document is creating with an id whose value is taken from the column identified in the Id column configuration. All columns selected will appear in the document sent to Aspire, although their position in the document will depend if they are recognised as standard Aspire Fields

Example SQL for a full crawls

Any valid sql that performs a select will allow you to perform a full crawl.

...

and chose to use 10 slices with an id column of id, you should modify the full crawl sql, adding {slices} to the condition (or adding the condition where there is none):

select * from main_table where {slices}

The sql executed at the server would then be

select * from main_table where id mod 10 = 0
select * from main_table where id mod 10 = 1
select * from main_table where id mod 10 = 2
select * from main_table where id mod 10 = 3
select * from main_table where id mod 10 = 4
select * from main_table where id mod 10 = 5
select * from main_table where id mod 10 = 6
select * from main_table where id mod 10 = 7
select * from main_table where id mod 10 = 8
select * from main_table where id mod 10 = 9

Each of these would return (on average) one tenth of the data