You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Aspire Distributed Processing

Since the 3.1 release, Aspire connectors are able to crawl in distributed mode automatically. Since all the crawl  control data is stored in MongoDB, by just adding more Aspire servers configured to use the same MongoDB, the common connectors are going to crawl distributively.

On this page:

In order to setup an Aspire Cluster for Distributed Processing, you need to configure each Aspire server to use the same MongoDB instance:

MongoDB Settings
  <!-- noSql database provider for the 3.1 connector framework -->
  <noSQLConnectionProvider connectionsPerHost="10" sslEnabled="false" sslInvalidHostNameAllowed="false">
    <implementation>com.searchtechnologies.aspire:aspire-mongodb-provider</implementation>
    <dropOnClear>false</dropOnClear>
    <servers>mongodb-host:27017</servers>
  </noSQLConnecitonProvider>

If you need to connect to a multi node MongoDB installation, check: Connect to a Multi-node MongoDB Installation

Once you have configured each instance, you need to

  • No labels