429 Errors for O365

HTTP 429 errors occurs when too many requests have been sent to a content source in a given amount of time.

These errors may occur when requesting data from certain content sources such as Microsoft Cloud Services (Sharepoint, OneDrive, O365, Azure AD), Box, Elasticsearch and others.

The errors are retried automatically by Aspire and what Aspire does is waiting a set amount of time before retrying, based on the error response header or if this is not returned, on the configured parameters.

This is a mitigation strategy and not a final fix since, in the end, the underlying problem is that the content source is not able to respond to the requests fast enough and as a result, sometimes a new crawl will be necessary to pick up items that failed completely.

If HTTP 429 errors are happening too frequently it might be worth it to place a throttling policy for the crawl as described here.