Since the 3.1 release, Aspire connectors are able to crawl in distributed mode automatically. Since all the crawl control data is stored in MongoDB, by just adding more Aspire servers configured to use the same MongoDB, the common connectors are going to crawl distributively.
On this page:
In order to setup an Aspire Cluster for Distributed Processing, you need to configure each Aspire server to use the same MongoDB instance:
<!-- noSql database provider for the 3.1 connector framework --> <noSQLConnectionProvider connectionsPerHost="10" sslEnabled="false" sslInvalidHostNameAllowed="false"> <implementation>com.searchtechnologies.aspire:aspire-mongodb-provider</implementation> <dropOnClear>false</dropOnClear> <servers>mongodb-host:27017</servers> </noSQLConnecitonProvider>
If you need to connect to a multi node MongoDB installation, check: Connect to a Multi-node MongoDB Installation
Once you have configured each instance, you need to