Since the 3.1 release, Aspire connectors are able to crawl in distributed mode automatically. Since all the crawl control data is stored in MongoDB, by just adding more Aspire servers configured to use the same MongoDB, the common connectors are going to crawl distributively.
Each connector is responsible for talking to the repositories, scanning through all the items to fetch and store its IDs to MongoDB for being processed later by any other server or itself.
On this page:
Table of Contents |
---|
In order to setup an Aspire Cluster for Distributed Processing, you need to configure each Aspire server do the following steps:
You need to configure all Aspire servers to use the same MongoDB
Installation, configure all the Aspire Servers config/settings.xml file
Excerpt Include | ||||
---|---|---|---|---|
|
Once you have configured each instance, you need to