The biggest change in Aspire 3.1 is related to the way the connectors work, they now use an external database to hold all the crawling information such as document urls, status, statistics, snapshots (for incrementals), logs, etc. The idea behind this change is allow the connectors to work distributed from its very architectural design.
Now all the connectors run under the same principles, using the same logic, so each connector is more like a Repository Access Provider so we keep them as simple as possible, rather than a complex (multi-threaded) crawling application; so the complexity of distributed crawling and multi-threading relies on the Connector Framework.
Among the responsibilities of the Connector Implementations we have:
- Scan the repository document containers to discover new documents to process
- Populate document metadata
- Fetch document content
If you want to learn more about the Connector Framework check out NoSQL Connector Framework.
Overview
Content Tools