Page History
...
When running an incremental crawl, the connector will first check for any remaining failed documents and retry those before starting the crawl. Any document or batch that fails during this check and during the incremental crawl, will be retried at the end of the crawl after the deletes are sent. The number of incremental crawls that a document should be reprocessed is configured using the "Maximum crawls to retry" option.
| Info |
|---|
Reprocessing documents is not done in batches, which means that if a batch fails because of a single document error, the reprocessing should be able to process all documents and keep retrying the failing ones. |
Failure Detection
To detect document or batch failures, the component receives a list of regex patterns to match with the errors thrown by the documents or batches. If any of those patterns matches an error message, the document will be marked as failed and retried on a later stage. In the case of a pattern matching a batch error, all documents that are part of the batch will be marked as failed. These patterns are set up in the "Exception Patterns to retry" section.
...
Overview
Content Tools