Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Aspire Manager/Worker Architecture

Aspire 5.0 introduces two types of nodes: Manager and Worker.

Image Added

A Manager is responsible for coordinating the execution of "jobs" from any given crawl and the crawl state, it prepares batches of jobs for eventual assigning to Worker nodes.

  • There is an automatically elected main Manager who coordinates which manager will get to manage each crawl
    • it also takes appropriate actions when either a Manager or Worker node is detected to be down.

A Worker is responsible for processing batch of "jobs" obtained from the Manager nodes.

  • Also executes all rules inside any workflows configured for the associated job crawls.
  • Fetching of content from repositories
  • Content and metadata modification/extraction
  • Indexing of documents with Publishers


Crawl Configuration

Crawls are now configured in separate entities which allows for maximum re-usability.