...
- Multi-threading processing
- Distribute the crawl processing
- Store and fetch documents from the database.
- Maintain a snapshot for incremental crawling (adding, updating or deleting documents)
- Handle statistics
- Start, Pause, Stop, Resume the crawl
- Send the documents to the respective workflows for processing and search engine indexing
...
{"serverDuration": 62, "requestCorrelationId": "7f1bc64d439e54a0"}