Page History
| Easy Heading Free | ||||
|---|---|---|---|---|
|
Logging
All Aspire Logs are now stored in the aspire-log NoSQL Database. This provides a centralized log that will contain information from all components loaded on all nodes.
These are the fields stored on each log entry:
| Field | Description |
|---|---|
| timestamp | Long valued representing the date and time the log was generated. |
| severity | Severity of the log entry. Possible values: DEBUG, INFO, WARN, ERROR.1 |
| component | The Aspire component this log entry comes from. |
| message | Log message. |
| stack | In case the log is an ERROR, the stack trace of the exception causing the error is stored here. |
| server | Json object containing all information related to the server this log comes from. |
| server.ip | The server ip address. |
| server.port | The server port. |
| server.name | Hostname of the server. |
| server.start | Long representing the Aspire node start time. |
| server.id | Unique id assigned to the server when the Aspire node is started. |
Error Log
The Connector Framework provides an extra NoSQL database that will store all crawl related errors. This error entry will contain the corresponding log as well as information related to the error.
| Field | Description |
|---|---|
| seedId | Seed that logged the error. |
| crawlId | Crawl where this error was generated. |
| type | Error type. Possible types: process, scan, connector or identity. |
| timestamp | Long valued representing the date and time the error was generated. |
| message | Error log message. It can contain the exception and a stack trace. |
Error Types
| Error Type | Description |
|---|---|
| process | Any error that comes from processing an item. |
| scan | Any error generated while scanning a container. |
| identity | Any error created when extracting an identity. |
| connector | Any error generated while processing a connector control item. See Crawl Control to learn more about control items. |
Log Monitoring
See How to Enable Aspire Logs in Kibana guide.
Audit Log
Instead of trying to compute statistics based on the actions triggered during a crawl, Aspire 5 provides a NoSQL database, called aspire-audit, where all events are logged.
The following table describes each field on an audit log entry:
| Field | Description |
|---|---|
| seedId | Seed this event is part of. |
| connectionId | Connection this event is part of. |
| crawlId | Crawl this event is part of. |
| itemId | The item id. |
| crawlType | Type of the crawl where this event was triggered. Can be full, incremental or identity |
| type | The event type. See the Event Types section for a detailed description of possible values. |
| timestamp | Long valued representing the date and time the event was generated. |
| status | Whether this event was success or failure. |
| retry | Whether this event was caused due to a failed document retry. |
| isGroup | When the item is an identity, this flag determines whether the identity is a group or a user. |
| worker | The node id of the worker that processed the event. |
Event Types
| Event | Description |
|---|---|
| add | An item add. |
| update | An item update. |
| delete | An item delete. |
| unchanged | Triggered when no changes were detected for a given item. |
| batch | Event triggered when a batch is done. |
| crawlBegin | Triggered when the start control item is done. |
| crawlEnd | Triggered when the end control item is done. |
| scanned | Triggered when a container is successfully scanned. This type won't have a status, since it is logged by the final action (i.e. add or update). |
| scannedNotIndexed | A container that was scanned but not indexed. This type will have status since there is no further action on it. |
| enqueued | An item that was discovered during a scan and sent to the queue. This type won't have a status, since it is logged by a final action (i.e. add, update or delete) |
| terminated | An item that was terminated during its processing. |
| excluded | An item that was excluded by an Include or Exclude pattern. |
| retrySkipped | An item that was skipped during a retry. |
| controlItem | Triggered when any of the control items is done processing. See Crawl Control for more information on control items. |
| discoveredIdentity | An identity discovered during an identity crawl. |
| processedIdentity | Triggered when an identity is done. |
| terminatedIdentity | Triggered when an identity is terminated. |
Metrics
Aspire 5 metrics can be computed and displayed using Kibana Dashboards. See Install Kibana Dashboards for more information.
Overview
Content Tools