What is the Crawl State?

The Crawl State is the representation of the progress made by an in progress crawl. By using the crawl state Aspire is able to distribute the processing tasks between multiple Aspire nodes, as well as pausing/resuming crawls. In order for it to be synchronized between multiple Aspire nodes, as well as preserve the state on restarts, it needs to be stored in an external database.

What Databases does Aspire use?

As of Aspire 4.0, we support two different databases:

• MongoDB 3.X, Open Source, document-oriented database.
  • Installing and Configuring MongoDB 3.X
• Apache HBase, Open Source, distributed, scalable, big data store.
  
  • Configuring Aspire to use HBase