All Aspire Logs are now stored in the aspire-log NoSQL Database. This provides a centralized log that will contain information from all components loaded on all nodes.

These are the fields stored on each log entry:

timestampLong valued representing the date and time the log was generated.
severitySeverity of the log entry. Possible values: DEBUG, INFO, WARN, ERROR.1
componentThe Aspire component this log entry comes from.
messageLog message.
stackIn case the log is an ERROR, the stack trace of the exception causing the error is stored here.
serverJson object containing all information related to the server this log comes from.
server.ipThe server ip address.

The server port.

server.nameHostname of the server.
server.startLong representing the Aspire node start time.
server.idUnique id assigned to the server when the Aspire node is started.

Error Log

The Connector Framework provides an extra NoSQL database that will store all crawl related errors. This error entry will contain the corresponding log as well as information related to the error.

seedIdSeed that logged the error.
crawlIdCrawl where this error was generated.
typeError type. Possible types: process, scan, connector or identity.
timestampLong valued representing the date and time the error was generated.
messageError log message. It can contain the exception and a stack trace.

Error Types

Error TypeDescription
processAny error that comes from processing an item.
scanAny error generated while scanning a container.
identityAny error created when extracting an identity.
connectorAny error generated while processing a connector control item.  See Crawl Control to learn more about control items.

Log Monitoring

See How to Enable Aspire Logs in Kibana guide.

Audit Log

Instead of trying to compute statistics based on the actions triggered during a crawl, Aspire 5 provides a NoSQL database, called aspire-audit, where all events are logged.

The following table describes each field on an audit log entry:

seedIdSeed this event is part of.
connectionIdConnection this event is part of.
crawlIdCrawl this event is part of.
itemIdThe item id.
crawlTypeType of the crawl where this event was triggered. Can be full, incremental or identity
typeThe event type. See the Event Types section for a detailed description of possible values.
timestampLong valued representing the date and time the event was generated.
statusWhether this event was success or failure. 
retryWhether this event was caused due to a failed document retry.
isGroupWhen the item is an identity, this flag determines whether the identity is a group or a user.
workerThe node id of the worker that processed the event.

Event Types

addAn item add.
updateAn item update.
deleteAn item delete.
unchangedTriggered when no changes were detected for a given item.
batchEvent triggered when a batch is done.
crawlBeginTriggered when the start control item is done.
crawlEndTriggered when the end control item is done.
scannedTriggered when a container is successfully scanned. This type won't have a status, since it is logged by the final action (i.e. add or update).
scannedNotIndexedA container that was scanned but not indexed. This type will have status since there is no further action on it.
enqueuedAn item that was discovered during a scan and sent to the queue. This type won't have a status, since it is logged by a final action (i.e. add, update or delete)
terminatedAn item that was terminated during its processing. 
excludedAn item that was excluded by an Include or Exclude pattern.
retrySkippedAn item that was skipped during a retry.
controlItemTriggered when any of the control items is done processing. See Crawl Control for more information on control items.
discoveredIdentityAn identity discovered during an identity crawl.
processedIdentityTriggered when an identity is done.
terminatedIdentityTriggered when an identity is terminated.


Aspire 5 metrics can be computed and displayed using Kibana Dashboards. See Install Kibana Dashboards for more information.

  • No labels