Aspire Index Auditing

Available from  (2.1 Release)   The Aspire Index Auditing is a feature aimed to help the Administrators to keep track of all content source actions and search engine indexes in order to identify possible differences and problems between both of them. The Index Auditing consists in three new additions:

• Content Source auditing
• Search Engine index auditing (publisher index dumps)
• Reconciliation file between the content source and search engine auditing files

Training Material

If you're interested in learning more, here's a recording of the Tech Talk on the Performance and Auditing Tech Talk along with the Performance and Auditing Tech Talk presentation.

Content sources auditing

By default every content source will log every action done for each of the documents crawled, to disable it go to the Advanced Properties in the content source configuration and uncheck the Enable Auditing option.

The actions that are being logged are:

• Add

The document was sent to be processed to the workflow as an add

• Update

The document was sent to be processed to the workflow as an update

• Delete

The document was sent to be processed to the workflow as a delete

• No Change

The document was found as not changed, no further processing was done for it

• Excluded

The document was excluded by index pattern configuration, no further processing was done for it

• Workflow completed

The document finished the workflow successfully

• Workflow error

The document finished the workflow with an error

• Workflow terminated

The document was terminated by a workflow rule

• Batch completed

The processing batch finished successfully

• Batch error

The processing batch finished with error

To see the audit logs of any crawls from the Aspire UI, open the content source statistics, and click View Audit Logs:

Image Added

Once you clicked on View Audit Logs the following page will be displayed:

Image Added

You can also filter the audit logs by action:

Image Added

Search Engine index auditing

The auditing log files for a search engine are generated via index dumps. At the moment only these publishers are able to create index dumps:

• Solr Publisher

  See tutorial on how to configure the index dump at Solr Tutorial

• ElasticSearch Publisher

  See tutorial on how to configure the index dump at ElasticSearch Tutorial

• Google Search Appliance Publisher

  See tutorial on how to configure the index dump at GSA Tutorial

To dump an index and compare it you have to:

1. Go to any auditing page of a content source, and click on Index Compare, select the publisher (should be configured to create index dumps) and click on Compare to:

    

2. It will open a pop-up window for generating both index dumps and reconciliation files

    

3. Select New Index Dump and then click on Start Dump to start an index dump. Take into account that the index dump will be executed agains the documents indexed by this content source only.
4. You will see the dump state getting refresh by clicking on the refresh button, and once it finish you will see it in the dumps dropdown list:

    

5. Select the index dump you have just created (notice the button action changes when you select an index dump) and click on Start Comparison
6. When the comparison finishes you are going to be able to see the reconciliation file:

Auditing files

All publisher and content source audit files are located at {aspire-distribution-home}/audit.

The reconciliation audit files, once created, will be located under the folder named: {content-source-name}_{publisher-name}_diff

Each content source, publisher or reconciliation folder under the audit folder can contain multiple audit files each identified by a timestamp:

Image Added

For content sources audit files, the timestamp is the crawl start time. For publishers and reconciliation the timestamp is the time of the audit file creation.