Aspire 5.1

NoSQL DB provider supported by the Aspire 5.1 release:

• New Elasticsearch/Kibana versions supported (7.17.7).
• OpenSearch v. 1.3.7.

Java version supported

• OpenJDK 17. 

New Features 

• Aspire Framework
  • Cluster-wide resource management (Files).
  • New workflow events (onCrawlStart, onCrawlEnd, and onSubJob).
  • Advanced Groovy component.
  • Bulk action improvements (Seeds and Schedules).
  • Single Sign-on authentication method now supported (UI and API).
  • Support for hierarchical ACLs (Intersection ACLs) supported for Confluence crawls, Confluence Identity crawls, and Group Expansion.
• Connectors. Newly migrated.
  • Confluence. Now supporting On-Premise and Cloud Servers.
  • OneDrive.
• New Python Migration Scripts for Aspire 4.0 configurations. Available for the following connectors:
  
  • File System.
  • SMB.
  • Sharepoint Online.
•  Publishers. Newly migrated.
  • Solr v. 8.9. 
• Services. Newly migrated.
  • HTTP Listener.
• Workflow. Newly migrated.
  •  Components
    • Archive Extractor. 
    • Tabular Files Extractor. 

Improvements

• Aspire Framework
  •  General
    • JRE v. 17 now embedded on the distribution. 
    • Improved queue management, allowing timeouts.
    • Improved requests to nodes in case of communication failure.
    • Clarified the escape characters for specific URLs on Scope fields (Connections).
    • Changed MavenRepository implementation to use Apache Maven Resolver instead of Maven Aether Provider.
  • NOSQL Providers
    • Implemented index rollover management.
  • Scan Reports
    • Security Assessments: Critical, high, medium, and low tickets fixed (Dynamic and Static Application Assessments).
    • Vulnerabilities: Critical vulnerabilities fixed.
  • Resource Manager
    • Configuration item manager instead of using file paths.
•  Connectors
  • Azure Identity
    • Allows multiple threads to speed up crawling.
  • DB Server
    
    • Support for the SAP HANA and IRIS database ports. 
    • Separated fields for Table and View Metadata query.
  • REST
    
    • 'Stop on Scan Error' option added. 
    • Pagination by next page link.
    • Support for expiration date format for Bearer authentication.
    • Include children metadata info inside customizable fields.
  • S3
    • MinIO support.
  •  ServiceNow
    • Inherit Knowledge Base User Criteria for Knowledge Article items. Known issue on 5.0.4.
    • Custom Search Query option for Knowledge Articles.
•  Workflow
  • Summarizers
    • Job Summarizer Executor
      • Row sampling options added.
      • 'Fetch row using slices' option added.
    • Saga Summarizer Executor
      • Allows to detect Saga tags in columns of tabular files.
      • 'Python server manager & cache' option added.
      • 'Limit Entity Samples' option added to matched tags.

Bugs

• Aspire Framework
  
  • API
    • NullPointerException executing a schedule with no seeds.
  •  General
    • Workers nodes not closing RAP Connections after completion.
    • Pausing a crawl closing in use connections immediately.
    • AspireObject not allowing valid characters in JSON format. Known issue on 5.0.4.
    • Items marked as excluded being processed in the Scan phase.
    • Non released items for dead workers.
    • Aspire not recovering after errors on standalone mode.
• Connectors
  
  • Aspider:
    • Crawling a URL including the meta tag "robots" with content="noindex,nofollow".
    • Credentials not being encrypted correctly.
  • Group Expansion Manager
    • Issue merging entities.
    • No option to map groups based on attributes.
    • Some duplicated entries.
    • Some other minor issues.
  •  ServiceNow
    • Missing metadata fields for Knowledge Articles and their attachments.
  • Sharepoint Online
    • Crawl failing due to folder with special characters.
    • Hotfixes migrated to this version.
    • Some minor issues.
• Publishers
  • Elasticsearch
    • Publisher failing having ES server configured to use self-signed certificates.
• Schedules
  • Schedules triggered when it was not time for execution.
  • Duplicated entries for schedules on the Debug Console.
  • Crawl executed for disabled Sequence Schedule.
• UI
  • Improved redirect after creating configuration items.
  • Some columns case-sensitive on the Seeds page when filtering.
• Workflow
  
  • Summarizers
    
    • Saga Summarizer Executor
      • 'Entity Metadata' option removed.

Known Issues

• Aspire Framework
  • Throttling is considering control items during the crawl.
  • Resource Manager
    • Resources (drivers) loaded not working for the RDB connections.
• Connectors
  
  • REST
    • Bearer authentication failing with unknown expiring time format.
  • ServiceNow
    • Include pattern option not getting items by using the Short Description pattern field.
• UI
  
  • Edit page for Seeds loading empty rare times.
  • UI actions are limited for users with both Admin and Operator roles.
• Workflow
  • Extract Text
    • The Custom mappings option is not working.

Migrating from 5.0.2 to 5.1

Update the aspire-mappings index to refresh mappings:

PUT aspire-audit/_mapping
{
  "properties": {
    "audit.timestamp": {
      "type": "date"
    },
    "audit.seedId": {
      "type": "keyword"
    },
    "audit.crawlId": {
      "type": "keyword"
    },
    "audit.type": {
      "type": "keyword"
    },
    "audit.worker": {
      "type": "keyword"
    },
    "audit.statistics.seedId": {
      "type": "keyword"
    },
    "audit.statistics.connectionId": {
      "type": "keyword"
    },
    "audit.statistics.connectorId": {
      "type": "keyword"
    },
    "audit.statistics.id": {
      "type": "keyword"
    }
  }
}