Page History

...

Is incremental indexing supported by the connector?

Yes. At the moment, this is done by accessing SharePoint Online changes database directlyusing Aspire Snapshots, so after a full scan the connector will only fetch objects that changes in a given date range (since the last scan and the current scan).SharePoint Online doesn't keep track of item updates/deletes for BCS external list items on it's change log. To be able to check for changes on BCS external lists, all external list items are crawled and checked against the information on the current snapshot file. If any of those items has any changes, then it is sent to the pipeline to be updated or deletedboth full crawls and incremental crawls will traverse the site or list depending on the root url, with the difference that during incremental crawls only adds, updates or deletes will be indexed.

For a discussion on crawling, see here.

Why am I getting Http 429 errors on my crawls?

SharePoint Online has a secret threshold that once reached by our connector will start throwing HTTP 429 errors.  We have implemented the practices recommended here by Microsoft but it's still possible to reach the threshold.

General