New and Enhanced Features

Aspire Core and Framework Components

• Salted Challenge Response Authentication Mechanism (SCRAM) support has been added to the MongoDB used with Aspire.
• The ability to dynamically load jar files has been added to Aspire with Java 9.
• When starting Aspire either normally or in debug mode, the debug line in the settings.xml file is handled appropriately. 

• A section has been added to the settings.xml file for HBase information.

• Logging of remote IP addresses for successful or failed logins will now occur.

• The Mongo Provider provider now encrypts/hashes IDs.
• Record fields have been improved.

• Entitlements checking now no longer checks missing components at every restart.

• Time zones have been normalized for Aspire, including logs and statistics.

• The documentation has been updated for Keytab/Kerberos.
• Improvements have been added to Job usage.
• Updates have been made to the ExtractText default configuration limit for text extracted from a stream.
• Sharepoint Commons
  

• List page retrieval and metadata extraction have been improved in SharePoint Commons.

Aspire UI

• To re-fetch entitled components (after deleting the Resources folder), an "Allow Refresh" button has been added.

• The ability to show Provider Information has been added.

Connectors

• Aspider
  • Added a A Headless browser has been added for rendering dynamically generated pages (client-side JavaScript pages).
• IBM Connections
• Sharepoint Online
• Sharepoint 2010

• Elastic

• SharePoint 2010

  • On the Multiple URls drop-down, when the 'Site Discovery' option is set, the 'Set List View' option is removed. 

• SharePoint Online
  • An NPE at crawl end error could occur if bad credentials were used.
  • Incremental crawls no longer detect containers as updated items.

  • Scan recursively was not working as expected.

• SMB

  • Added DFS support and override last access date of documents
• Twitter

Publishers

• Elasticsearch
  • Case sensitive index names can be handled properly now.
    
• Google Cloud Search
  • A new Google Cloud Search (GCS) publisher receives content from Aspire connectors and uses the Java Client library to index the content into Cloud Search.
• HBase
  • Content can now be deleted.
  • During a full crawl, the publisher now defaults to clean.
    

  • When not in file configuration mode, the publisher can now be used without security. 

• Publish to StageR

• Field level help has been added for the special scope $record.

Applications

• The Entitlements Admin application has been updated.

Bug Fixes

AspireCorebugs

---

Aspire Core and Framework Components

• Admin UI
  • The ability to configure a weekly schedule could cause an error when saving
• Aspire Application

  • ConfigManager could log a debug message into {aspire.home}/logs/configmanager.log

  • A problem could occur when editing a custom application in the Admin UI

  • Startup problems could occur using the Staging Publisher
• Connector Framework

  • When stopping and restarting Aspire while the GroupDownload process was running, the group download did not start again

• MongoDB Provider
  • Sharepoint Commons

  • • The LDAP Cache could report a MongoDB Duplicate key error

    • Aspider could stop with a MongoDB Duplicate key error

  • SharePoint Commons

    • An out of memory (OOM) exception could occur during large crawls

    • Added support for incrementals using Aspire Snapshots on SP
  • The Aspire Archetype had "http" rather than "https" repository and entitlement URLs.
  • Failed to connect to Artifactory with custom keystore. Artifactory certificates were added to the distribution. See: https://contentanalytics.digital.accenture.com/pages/viewpage.action?spaceKey=aspire33&title=Crawling+via+HTTPs
  • AspireObject was casting an incorrect numeric type when created from JSON.

  • The AspireObject isEmpty method returned true even if the object had children.

  • The processDeletes (String) was missing a Status page.
  • The Aspire Connector Framework was not using shouldScan during incremental crawls.
  • When running a full crawl, a "Provider 'encrypted' not installed" message could occur.

  • The Mongo Provider provider generated an invalid JSON object during document conversion.

  • Audit logs were incomplete.

  • For AIP integration, the logout action was not being logged.

  • Publisher framework retryDelay, retryDelayMultiplier and maxRetryDelay properties were not supported by Dynamic XML Forms (DFX).

  • The Aspire-Services jar file was missing a noSQL package.

  • The "Loading Application" message could display whether a connector was loading or not.

  • Extract Text
    • Use the Apache Tika SAX Parser for Microsoft documents.
  • Scheduler
    • The option to create a Cache Groups scheduler was not being displayed.
  AnchorUIbugsUIbugs

  Aspire UI

  • A Connector component might not show the actual state of a crawl.
  • The link that points to the Confluence wiki has been updated.
  Anchorconnectorsbugs
  connectorsbugs

  Connectors

  • Aspider
    • An authentication form error could occur indicating "Target host is not specified while crawling".
    • Neither NTLM nor ADFS authentication was occurring when a host was specified in the Credentials.

    • On any port, the Port field was not working correctly with any value except "-1".

    • A crawl could cause a warning about duplicate IDs in MongoDB.

    • To indicate that the Gateway was not working, the exception message in ADFS needed updating. updating 

  • Confluence

    • ACLs info appeared inside the hierarchy section

    • A batch error could display while publishing to Elasticsearch 6.3.0
      
  • Documentum
    • Exception was being thrown during Group Expansion.
  • File System
    • Starting Directories in the File option was not working as expected.

  • IBM ConnectionsJive

    • The connector needed to use the Aspire GroupExpansion instead of SharePoint Integrated security with an optimized IBM Connections Group Downloader
    • Memory leaks could occur
    • During an incremental crawl, the deletes of Blogs, Wikis and Files were not working
    • The Content crawled from IBM Connection did not contain a last-modified date. The problem was with the date format
  • Kafka
    

    • A "NO-NAME" field could occur

  • SharePoint 2010

    • A problem could occur when identifying the site-collections for a WEB-Application
    • When adding a link on a site collection to crawl, [NO-NAME] should not be part of the name attribute in the hierarchy section
    • No error should occur during the incremental crawl for the Blog site collection
    • No errors should occur when crawling a specified list (views included)

  • SharePoint 2013

    • When crawling incrementals for an External list, the connector was not picking up the changes
    • When crawling SP2013, errors such as "HTTP Error 400.
  • One Drive

  • Sharepoint 2010

  • Sharepoint 2013

  • Sharepoint 2016

  • Sharepoint Online

  • • The size of the request headers is too Long" might occur

    • KeyNotFoundException while trying to check attachments for list with lookup references deleted

    • Crawl a list and the name in the hierarchy of the documents will be displayed as NO-NAME even though the items have  title field.

    • The placeholder needed to be changed for the 'Seeds file' field

    • The connector was unable to crawl large lists

  • SharePoint 2016

    • An error could occur while crawling site

  • SharePoint Online

    • NPE crawling on distributed mode. Random NPE in the item complete callback

    • String index out of range while getting a List display url

    • Error while crawling after a crawl was stopped: Item parent wasn't assigned during crawl

    SMB

  • Standalone Mode

    • When a user added a custom connector, feedback needed to be provided by the Aspire UI. 
  • Staging Repository
    Anchorpublishersbugs

    • • A global variable was not working when configuring the server in the Staging Repository connector

      • When crawling over multiple documents and publishing at two different scopes, the items published could be duplicated

      • The Stager connection could be broken when running a full crawl

    Publishers

    • Stager BDC Plugin could randomly fail during the crawls after setup

    publishersbugsPublishers

    • Azure Blob Publisher

    • DBC Plugin 

    • Elasticsearch
      • DeleteByQuery was not being used with Elasticsearch 6.1.1

    • GCS Publisher

      • A resource config/application.xml was missing on the jar file

      GCS Publisher
      • A relative path was not working in the Credentials Key File field

      • An error could occur when crawling and publishing to GCS

    • Kafka

      • An error could be masked when running a non-batched job.

    • Publish to Avro

      • Validation needed to be added to the Time Rollover Threshold field.

    • Solr Publisher

    • TLS 1.2 support was needed for the SharePoint Sharepoint Security Pre Trimmer

    • StageR Publisher

    Anchorservicesbugs

    Services

    • Azure Group Expander could refuse to start.

    servicesbugsServices

    • Group Expansion failed if user data exceeded the Mongo Max Document Limit (16MB).

    • For Aspire Distributed mode, Services in the master node were not starting automatically after saving changes.
    • Errors to reflect failed Services were not being generated.

    • Services that were set up in an Aspire cluster were not synced up correctly.

    • Azure Active Directory Group Expander

      • Users were not being removed.

    • Group Expansion Service
      • The userGroupCache map was accessed when the Group Expansion Service was running.

    • LDAP Cache Service

      • The controls did not display and the Schedule was set to Advanced even if Minutes or Hourly were set

      • Problems with  LDAP-Cache component could include: reporting a duplicate key error twice, stopping with a duplicate error, taking too long, and refresh refusing to start.

    Anchorappbugs

    • • The connector could not look up ACL information in the LDAP-Cache component

    • You are now able to check the user’s cache for the Azure Group Expander via the Debug console

    appbugs

    Applications

    • Archive Extractor

      • The "Send delete by query first" option could throw an exception.

      • Deleting files inside of an archive file was not handled properly for incremental crawls.

    • AVRO Extractor

      • During an incremental crawl, a "duplicate key error" message could display.

    Anchorknownissuesknownissues

Known Issues

Connectors 

• FTP 

  • FTP connector is only working with Unix systems and not in Windows.
• Twitter
  • Full/Incremental crawls for retweets are not working

Publishers

• Google Cloud Search
  • Bundle location error loading the publisher for the first time
  • NullPointerException publishing with Batch and Content Type Raw options
    
  • ItemUploadRequest exception
    

  • Pending required field validation for the 'Indexer Type' field