Anchor
enhancements enhancements

New and Enhanced Features

Anchor
AspireCoreEnhance AspireCoreEnhance

Aspire Core and Framework Components

• The Request log now contains the remote IP addresses for successful and failed logins
• The following fields have been added for every record: 
  • “ingestionStart” (date time when a record was first created by the connector)
  • “ingestionEnd” (date time when a record was ingested into the search engine) 

• • “crawlStart” (date time when the crawl responsible for a record was started)

   For debugging, we can now identify which records were ingested when and what batches of records exist.  We can also more easily identify and remove orphans

• An option has been added to the Aspire Parcel and Service to configure the Aspire log directory.
• Connectors now have a concept of a “domain”
  • When creating or indexing ACLs, the search engine will include the domain, which defaults to the connector name (or ID)
  • The domain can be overridden
• Improvements have been made to failed document handling
• Aspire now works with Java 11
• A message appears when adding a custom connector with ‘com.searchtechnologies.aspire’ Group ID
• Aspire Parcel and Service – An option has been added to configure the Aspire log directory
• The Aspire wiki has been updated to indicate that system properties can be configured in the settings.xml file for use in the Aspire UI.
  This is useful for connectors, services, workflow and can be applied to content sources and services to configure from the UI

• A Publisher framework has been added that uses a provider for the actual calls to the repository, handles batches, allows extensions, records errors and supports clear/commit jobs

• SMBJ
• You can now disable the "last accessed" update from the SMB2 client for Java open source connector

• The ability to show Provider Information has been added

• Aspider
  • Added functionality to cleanse the ID on incremental crawls
  •  Multiqueue (based on item status and grouping) has been added in HBase
• Elastic
  • The Elastic connector code would not compile in 4.0
• ServiceNow
  • ACL security has been added
• SharePoint Online
  • You can now crawl SharePoint content sources using either the Change Tokens or Snapshots
• Staging Repository
  • Field level help has been added for the special scope $record
• Yammer
  • A new Yammer connector retrieves Yammer messages including some metadata

• Elasticsearch
  • In order to support Elastic 6.x, the mapping script for Elastic has been changed to use only one "_type" value regardless of the situation
  • Elasticsearch publisher now forces a lowercase index name
  • DeleteByQuery was not working on 6.1.1

• Unable to see errors for failed services
• The LDAP Cache Service can now validate users from multiple domains.  Similarly, LDAP group caching can now cache groups from multiple domains

• Connector framework now supports Jive connector hierarchy
• Validation has been added for application input at the server-side for length, type, and business rules before accepting any data to be stored or processed
• For enhanced security, use HTTP POST (without parameters in the URL string) to avoid the disclosure of URLs and parameter values.
• A password placeholder has been added for developer users.
• If the HTML contained multiple meta tags with the same name but different values, the extract-text component was only extracting one value
• Group Expansion is no longer rendered for the connectors that do not use it:
• File System, CIFS, Amazon S3
• RDB Tables, RDB Snapshots
• Jira Issues, FTP, Feed One, Hbase, RSS,
• IBM Connections, Kinesis, ServiceNow
• Connector Framework
  • AspireObject was casting an incorrect numeric type with createFromJSON
  • Aspire Distributed Mode - Services in master node were not starting automatically after saving changes
  • The EncryptPassword script was deleting comments in the settings.xml file
  • ProcessDeletes was missing a status page
• Aspire Application
• SMB2 - An  error could occur when loading the connector for the first time

• It was not possible to edit a custom application in the Admin UI
• The link that points to the Wiki needed to be updated
  

• Aspider
  • Aspider to crawl dynamic rendering page
• Confluence
  • Incremental crawls when using a list were not working
  • When executing runs, no audit logs were being shown for jobs being "Excluded" nor "Not Changed"
  • Parent ACL's are now taken from the immediate ancestor
• IBM Connections
  • The connector was unable to look up ACL information in LDAP-Cache
• SharePoint Online
  • Added attachment was not crawled until the second incremental crawl
• SharePoint 2013
  • The connector could request more list items in one request than allowed
  • The content was not extracted (401 UNAUTHORIZED)
  • The connector could not process change item type SP.ChangeSite
  • When crawling a specific list, incremental reports could show deletes that never happened

• Elasticsearch
  • DeleteByQuery was not working on 6.1.1
• SolrCloud
  
  • Unable to add the component

• LDAP Cache Service
• Refresh was taking too long
• The service was refusing to start

• New Elasticsearch as a NoSQL provider.
• New Publisher Framework.
• Updates Listener service (push updates).
• Cluster Mode improvements (Zookeeper stability).
• Implemented the FIFOQueue for the MongoDB Provider.
• New options for the Extract Text configuration and new Throttling section in Advanced Configuration for connectors.
• Background processing.
• Saga integration.

Anchor
UIenhance UIenhance

Aspire UI

• Import/Export System configuration.

• Log Browser.

Anchor
connectorEnhance connectorEnhance

Connectors

• Confluence User & Groups.
• OneDrive connector.

Anchor
publisherEnhance publisherEnhance

Publishers

• Amazon S3.
• Google Cloud Search Identity Publisher.

Anchor
plugEnhance plugEnhance

Plugins

• Stager BDC Plugin.
• Confluence Real Time Plugin.

Anchor
bugfixes bugfixes

Bug Fixes

Anchor
AspireCorebugs AspireCorebugs

Aspire Core and Framework Components

• Missing headers on OAuth classes.
• Wrong URL info for the Aspire UI Authentication documentation in settings.xml file.
• Master password ssh file not working on Centos OS.
• Errors processing failed documents with the Exception Patterns option.
• Components on Workflow not saved if the content source was not saved first.
• Invalid entitlements host caused missing workflow applications.
• Double click ignored on disabled workflow item.
• NPE  after shutting down 2 Aspire instances in distributed mode.
• Java 1.8 Error when name of the Application and name of Publish was the same.
• Aspire not starting in shell mode on Centos OS.
• Publisher added to the workflow not being unpacked into cache folder so they were unavailable and not working.
• Error installing Aspire as a service in Windows.
• Status not displayed in Aspire after a crawl was aborted.
• NPE scheduling the "cacheGroups" option without the GEM configured.
• Two different entries in status collection being generated for the same crawl ID.
• NPE having Artifactory user with not entitlements assigned.
• Mongo database name limit exceeded by the Aspire Database name.
• NPE with the Non Text Document Filter and Open Data Stream options enabled.
• NPE using encrypted password at the SSL settings in settings.xml file.
• NPE pausing a crawl with MongoDB and Zookeeper in distributed mode.
• Error trying to import a Service since some services do not have a workflow associated.
• Previous crawl errors displayed when current crawl was running.
• Crawls on distributed mode not populating correctly ancestor ID and ACLs.
• Error uninstalling Aspire as a service.
• Aspire not getting alert if Elasticsearch provider is not running.
• Crawl statistics not reflecting the deletes if there were adds/updates.
• NPE after an authentication method configured in the settings.xml file.
• NPE displayed while stopping a crawl after it just started.
• Some Aspire UI settings configured in settings.xml file being ignored.
• Out of Memory error using a very big number in Hierarchy Cache Size option.
• Every time a groovy script was updated, a blank line was added at the beginning of the script.
• Invalid characters validation in the Extension List option of the Non Text Document filter.

Anchor
appbugs appbugs

Applications

• Archive Extractor
  
  • Using Select/Deselect All option closed the Configuration window.
• AVRO Extractor
  • ASPIRE-8112/ASPIRE-8113  Routing section options not displaying correctly.
• Hierarchy Extractor
  • The User/Group field on ACLs section is now required.

Anchor
UIbugs UIbugs

Aspire UI

• Typos on Accenture license information.
• UI refreshing stacks over and over while changing between the Cards View and the List View.
• Aspire DXF not accepting Windows relative paths.
  
• The word "content sources" displayed in the Service Group control.
  
• Navigation controls at the bottom overlapping the footer.
  
• Fixed special characters allowed in the connector's name.
  

Anchor
connectorsbugs connectorsbugs

Connectors

• Adobe Experience Manager
  
  • Use scheduled (de)activation item settings not working without include/exclude properties.
  • Fetch ACLs option not working.
  • Updates on pages not crawled on incremental.
  • Wrong credentials threw unclear message on Basic Authentication.
  • Malformed URLs not validated.
  • More user friendly exception for non-existent pages/assets.
  • Normalized date format for the "lastModified" field.
    
    
• Amazon S3
  
  • Crawl failing for items published with the S3 Publisher.
  • Some exceptions using the connector, the Archive Extractor application and the Elasticsearch publisher.
  • Crawl failing if directory URL not ending in  the "/" character.
  • Using bad Include Pattern prevented crawl to start.
    
    
• Aspider
  
  • Crawls not finishing in distributed mode.
  • Updates processed as Add instead of Updates.
  • Missed some options on the Extract Text section.
  • Hierarchy info appearing having the Hierarchy option disabled.
  • NPE displayed while content cleanup is selected but nothing is configured.
  • A [NO-NAME] value displayed in the hierarchy section.
  • Extract Text & Hierarchy options took out of the Advanced Configuration section.
  • Content cleanup of web pages not working in Aspire 3.3.0.4.
  • Images not being crawled using the Extract Text option enabled.
    
    
• Azure Blob
  
  • Seed file option is not working.
  • <Non text document> tag not published using open data stream.
  • Split Words per XML/HTML Tag is not working.
  • HTML Output not producing any document output.
  • Crawl errors displayed in the UI.
  • Incremental crawl not detecting updates.
  • Storage Connection String set as a placeholder.
  • Problem crawling folders.
    
    
• Azure EventHub 
  
  • Valid tooltips for the field in the Credentials section.
    
    
• Database Server
  • Scan errors crawling all tables in the RDBMS.
    
    
• Elasticsearch
  
  • 429 Error Management.
    
    
• File System
  • Hierarchy information incomplete.
  • No error using invalid filename specified in the "Path to Root directories file" field.
  • NPE using Multiple starting points option.
    
    
• RDB Snapshot
  • Problems crawling delete actions.
• RDB Tables
  • Exception using the Slices option not reported on the UI.
• Sharepoint 2013
  • Connector processes same document with different ID between crawls.
  • NPE pausing a crawl.
  • Problem running incremental using Lists option. All content being crawled.
• Sharepoint 2016
  •  Issue on incremental when an External List was included in a pattern.
  • Problem on incremental using Tokens with Crawl Attachments option enabled.
  • Connector not crawling folders created under site collection.
• Sharepoint Online
  • List threshold: not all items on a big list are being crawled.
  •  Group Expansion not working.
  • Error generating FetchUrl and Display URL for link list items inside a folder.
• StageR 
  • No error message on console or UI indicating wrong storage/scope used.
• Yammer
  • No error message on Aspire Web UI when Yammer token is invalid.

Anchor
publishersbugs publishersbugs

Publishers

• Elasticsearch
  • Updated items published as a new item.
  • NPE when using an incorrect ES port/host.
  • Added validation for malformed index name.
  • Groovy Transform is not validated with absolute/relative path.
  • Minor UI changes (tooltips and validations)
• Google Cloud Search
  • NPE when hierarchy info not coming from the connector. 
  • $superSearcherAcl being added as part of the ACLs when setting is empty.
  • Content type Raw not extracting the content for binary files.
  • Option to populate the gcsUniqueId field.
  • SocketTimeout exception.
  • Date fields using a month range from 0 to 11 instead of 1 to 12.
    
    
• Solr
  
  •  Option to set multiple URLs not working.
  •  XSL Transform is not validated with absolute/relative path.
  •  Solr URL field required Malformed URL validation.
  • Removed info from tooltip about 'default core'. Core field now is required.
    
    
• StageR
  • Delete All Action is not always executed first.

Anchor
servicesbugs servicesbugs

Services

• NPE while using services with workflows.
• Group Expansion not loading after Aspire was restarted.
• Error while adding Services with no workflows.
• Broken images/icons on Services UI.
• Encryption issue with authentication using LDAP Cache Service.
• LDAP Cache Service: Unavailable Critical Extension error querying IBM Domino.
• LDAP Cache error after importing the Service and run it.
• LDAP Cache authentication problem using service account.
• Discovery by Regex will throw error for non-pst files.

Anchor
knownissues knownissues

Known Issues

Anchor
connectorknown connectorknown

Aspire Core and Framework Components

• Completed items not being removed from the process queue.
• Crawl time execution still running after pause it. 
• Felix startup warning using Java version 11.
• Connectors/publishers saved twice when Aspire components are still downloading.
• HttpFeeder - Servlet added with the same name of another servlet is not notified in the UI.
• Error validating field Maximum size on Extract Text

Anchor
techLimitations techLimitations

External Technical Limitations

• Aspire Core and Framework Components

  • Elasticsearch Provider - "FATAL: Flushing-Error" can happen in some connectors.

• Servicename