The following are the NoSQL DB providers supported by Aspire 3.2 release:

• MongoDB version 3.4.10
• HBase version 1.2.4

Below you can find the list

of the updates for

this version.

Anchor
newfeatures newfeatures

New Features

• NoSQL DB Provider
  • HBase supported.
• AIP Integration related features (Aspire Cloudera Parcel mode)
  • Audit Logs
    
    • Log Aspire actions via Jetty.
  • Cloudera Parcel updated.
  • Licensing
    
    • Licensing to build entitlements for AIP
    • Licensing to Access Aspire Application: Annual, Perpetual and Trial.
  • Log Tika errors as warnings.
  • Logging improvements .
    
    • Added processes and scripts to delete old logs.
  • Security Access Control Configuration.
    
    • User Role Authentication with LDAP.
  • Security
    
    • Access to file system/APIs via groovy.

• Applications
  
  • HDFS Binary File Writer.
  • Connectors
    
    • HBase
    • Jive
      
      • Simplified way to bring Security ACLs.
      • Support version 9.
      • New option to crawl specific places.
    • Kafka
    • Salesforce
      
      • Chatter Feeds Endpoint updated to use the SOAP API and Accounts management.
    • Service Now
      • New option to choose what to use for the Include/Exclude patterns. Either URLs or Knowledge Article Id.
    • SharePoint 2016
    • SMB
    • Framework
      
      • Updated start scripts to fail with versions of Java less than 1.8.
      • Normalize time zone for Aspire including logs and statistics.
      • Remove Recovery Policy options from the UI, since they are no longer used.
      • Aspire Archetype - Add required Felix properties for Kerberos authentication in the Aspire Distribution.
      • Jetty web server modifications for Aspire: HTTP TRACE method and server header info.
      • Option enable to change Aspire's port when distribution is created with maven command.
      • LDAP Cache collection names changed for something more meaningful.
    • Services
      
      • HAR Compactor
    • Publishers
      
      • HBase
      • Kafka
      • Solr Cloud (SolrJ library)
    • UI
      
      • New grid/list view on Content Source page.
  • Publishers
    • Azure Blobs

Anchor
ConnectorsReleased ConnectorsReleased

Items Released

• Applications
  • Archive Extractor
  • AVRO Extractor
  • Parquet Extractor
  • PST Extractor
• Connectors
  • Aspider
  • Amazon S3
  • Box
  • File System 
  • CIFS
  • Confluence
  • Documentum 
  • Documentum DQL 
  • eRoom
  • HBase
  • HDFS
  • IBM Connections
  • Jira Issues
  • Jive
  • Jive (support ver. 9)
  • Kafka (Security pending)
  • Kinesis
  • RDB Snapshot 
  • RDB via Table 
  • Salesforce
  • Service Now
  • SharePoint 2013
  • SharePoint 2016
  • SharePoint Online
  • SMB
  • SMTP 
  • SocialCast
  • StageR 
  • FTP
  • RSS
•  Publishers
  • Azure Blob
  • Elasticsearch 
  • HBase 
  • Kafka 
  • Kinesis
  • SolrCloud 
  • Solr 
  • StageR
• Services
  • SharePoint 2013 CEWS Listener

Anchor
bugfixes bugfixes

Bug Fixes

Anchor

AspireCorebugs

AspireCorebugs

Aspire Core

• Incorrect Historical Statistics for a new connector after another was executed.
• Negative DPS appeared on the Statistics.
• Link related to Aspire Authentication was updated in the config/settings.xml file.
• Crawl Begin/End Auditing actions displayed different elements on Auditing.
• NPE when a crawl stopped without pausing.
• NPE when a crawl stopped in ReleaseController after pausing.
• UI not showing correctly the error for an invalid groovy script.
• Content source name could be updated with blank spaces.
• Historical Crawl Statistics for one connector appearing in another one.
• Incorrect Logs showed in QueueLoader component.
• Exception loading workflow.xml file when an empty custom groovy script was added.
• Advanced scheduler option was not working.
• Server error was encountered when displaying Audit Log.
• Group Expansion and Advance properties check boxes were misplaced.
• Page was stuck when adding a Custom Publisher.
• Service start and stop buttons displayed wrong tooltips.
• Crawl stalling in Linux after publishing end job.
• Couldn't save content source when using multiple check box selectors with one option checked.
• OpenDXF was not escaping characters in JSON inputs.
• Any exception thrown produced a NPE in the app-rap-connector.
• Failover: Dual instance full test (interrupted) - Recovery Option Full - Not all items Crawled
• Failover: Dual instance full test (interrupted) - Recovery Option Incremental - Never ended crawling
• Allowed laxing of deletes policy in connector framework.
• Aspire not detecting changes in the connector settings and not asking to save them.
• Stop crawl option not working correctly.
• Crawl showing wrong time when crawling in Linux.
• Statistics showing items In Progress when paused.
• Docs not crawled were reported as Adds on Auditing.
• Server error was encountered when displaying Audit Log.
• Aspire.sh -create_master option not working properly on Linux.
• Hierarchy extractor needed to have default values selected on Workflow jobs to work properly.
• Minor improvements and fixes in Aspire UI.
• Validations improvements and fixes for several components.

• Archive Extractor
  • Incremental on archives files was not working using the Lotus connector.
  • Nested archive threw an "Archive not recognized" error.
    

• CIFS
  • Malformed URL was not being validated.
  • Removed slash character at the end of name attribute on Hierarchy.
• Confluence
  • Name attribute for the level 1 hierarchy showed the name of the content source.
• Documentum DQL
  • Fixed NPE when running an incremental crawl.
  • DisplayUrl field was not separating webtop from document id
  • Include/Exclude fields appeared as part of the configurations.
• eRoom
  • Updates and Deletes were not picked up by Incremental crawl over certain items (Comments and Votes for polls.)
  • UI validation when using wrong URL.
  • No error was reported when setting a wrong username/password.
• FileSystem
  • Improved the wording in some tooltips.
• Heritrix
  • Implemented deletes handling feature from Heritrix in the connector framework.
• Jive
  • Changes on Document ACL's were reflected incorrectly for both Activity Incremental and Normal Incremental.
  • Non-text Document filtering reported Add instead of Update for the documents filtered.
  • Page Size value was not using the UI parameter.
• Lotus
  • Exclude pattern was not working as expected for items that were not attachments.
  • Incremental on archive files was not working.
  • Incremental crawl with index containers was not working.
  • No error showed if the database and view were the same.

• RDB Snapshot
  • Crawl was not finishing with the Use Slices option and set bad Extract SQL.
  • No error was reported when setting a wrong ACL SQL.
  • Wrong sql statement in Full crawl was not showing errors.
• RDB Tables
  • Action column was ignored for the incremental crawl.
• Service Now
  • Displayed incorrect URL field in Knowledge Articles (XML representation).
  • Inclusion\Exclusion pattern was not working for attachments.
  • Aspire error when two images files were attached and a full crawl was run.
• Social Cast
  • Tag nonTextDocument was missed in the Aspire Object.
• SharePoint 2007
  • Error on console and UI while crawling an item updated on root using both Index Containers and Scan Recursively disabled.
  • NPE processed container after changing ACL on an Incremental crawl.
  • ACLs showed the same item as group and user.
• SharePoint 2010
  • Minor fixes to the tooltips.
• SharePoint 2013
  • Incremental reported duplicate jobs when adding a subsite.
  • Delete job had the incorrect displayUrl and fetchUrl after renaming a file.
• SharePoint Online
  • Adding specific site collections made incremental crawl everything.
  • Renaming an item returned an add, update and delete on the same crawl.
  • Error when crawling site URL with encoded blank spaces.

• Publish to Solr
  • Deletes were not working correctly.

• Add Service button was not working.
• Azure Group Expander
  • Azure GE and SharePoint Online GE were not deleting users.
• CEWS Listener
  • PropertyOflong and PropertyOfArrayOflong were not working.
• Fast Content API
  • Missing validations.
• Group Expansion Manager
  • Fixed 'Missing version number' error when service was loaded.
  • Some validations were missed.
• LDAP Cache
  • Some validations were missed.
  • Problems with tooltips for LDAP Attribute in Cache user and Cache group options.

• NPE killing Aspire distribution
• Error when second server joins a distributed crawl.

Anchor
auditingbugs auditingbugs

Auditing

• Page Navigation not showing all the jobs.

Anchor
connectorsbugs connectorsbugs

Connectors

• Aspider

  • Crawl getting stuck (Error claiming the entry from the queue).

  • Decoded characters in URLs when no needed.

  • Case-sensitive option not considering robots.txt file.

  • NPE when running an incremental crawl using an invalid URL in the seed file.

  • NPE when running a full crawl using default options.

  • Full Crawl - NPE while running a Full Crawl using default options.

  • Minor UI issues.

• CIFS
  
  • Connector not allowing special characters as paths to crawl.
  • Missing placeholder on "Path to URLs field.
• Jive
  • ACLs not getting updated when the privacy of a group changed.
  • Hierarchy for documents with attachments not constructed correctly.
  • Extract Text stage is not being executed with default settings.
  • Activity Incremental + Creation Date Filter not catching updates after the first one.
  • Activity Incremental crawl generating duplicate key error collection.
  • Unicode characters not handled at all in several Jive contents.
  • Hierarchy for Social Group's blogs being generated incorrectly.
  • Error not being displayed retrieving ACLs from Entitlement API using on-premise Jive.
  • No UI error when Aspire Security Plugin is disabled.
  • The first level of the hierarchy populating the content source name in "name" field.
• RDB Snapshot
  
  • Minor UI validations.
• Salesforce
  
  • ACLs error with default connector configuration.
  • ACLs for Groups/Users not reflected correctly after updates.
  • Error crawling Feed deleted.
  • Error on fetching Attachments.
  • Error getting deletes on Incremental for Tasks and Profiles.
  • Error on invalid session.
  • Error ending crawl.
• Service Now
  •  Deletes being reported over attachments not updated.
• StageR
  
  • Connector doing reprocessing for incremental crawl.

Anchor
publishersbugs publishersbugs

Publishers

• StageR
  
  • Validation for Document Key field when field does not exist.

Anchor
UIbugs UIbugs

UI

• Importing connector with special characters in the path fields not loading correctly.

Anchor
ToRelease ToRelease

To Be Released

• Connectors
  
  • Lotus
  • RightNow
  • TeamForge
• Publishers
• CDH Hadoop
• Elasticsearch in Azure
• Web HDFS
• Services
  • Azure Group Expansion
  • HTTP Listener
  • HTTP Service

Anchor
ExtTechLimit ExtTechLimit

External Technical Limitations

• HBase: When running Aspire with long-term, large ingestion (with HBASE as the underlying HBase libraries may eventually stop returning results without throwing any error back, degrading the crawl performance down until it stalls completely. When this happens the only solution is to restart the affected aspire servers so the underlying HBase library threads get to connect from scratch.
• SalesForce connector: Due to SalesForce API limitations, the connector has the following limitations:
  • For incremental crawls, the  getUpdated and getDeleted methods are used, but when an attachment is updated from any item, that action will not be processed by the methods mentioned.
  • Security and incremental related limitations:
    • In security, we are only supporting 'Supported elements'.
    • For sharing related incremental crawling, unsharing of Salesforce item is not working.
    • For incremental crawling of Salesforce task items, we are only supporting tasks based on accounts.
    • If removal of sharing occurs for a item (e.g., removing sharing of an account), it is not reflect in the incremental crawl.
    • Pricebook sharing ACLs are not supported.
    • We are only supporting Tasks that are based on accounts for incremental crawling.
  • Chatter security
    • Chatter ACLs only will be retrieved if the “Filter TrackedChange feeds” option is checked.
    • Chatter ACLs are only supported for items that were created by a User or a Group, otherwise no ACL will be generated for the item.
    • The public chatter groups will have two ACLs, one for the public group and a PUBLIC:ALL ACL.
    • Private and Unlisted chatter groups will have one ACL for the group.
    • The followers of a chatter user will be treated as a private group called “<username>’s followers”, all the feed items created by a user to their followers will have this ACL.
    • The chatter item attachments will inherit the parent item ACLs.
    • Reducing the users retrieval scope might lead to a loss of ACLs, since no ACLs won’t be generated for followers of users outside the scope of the user retrieval.
  • Salesforce Compatibility limitation
    • Every 3 months Salesforce releases a new version of their API and, sometimes makes changes to the data structures, after each update there is a possibility that the compatibility between the connector and Salesforce will break.

Anchor
knownissues knownissues

Known Issues

Anchor
ItemDeprecate ItemDeprecate

Items Deprecated on Aspire 3.2

Anchor
AspireCore1 AspireCore1

Applications

• Archive Extractor
  • Deletes are not handled properly for incremental crawls.
  • Delete by query option is not working for ElasticSearch versions greater than 5.0. Ask to the Core Engineering team for a workaround if you need it.
    

Anchor
AspireCore2 AspireCore2

Aspire Core 

• Loading Application message trying to add connector but it does not load.
• Intermittent black screen showing in the UI for less than a second (flickering screen).
• Failover:
• Triple instance full crawl (double-interrupted): Having missing jobs.
• Dual Test Full Crawl interrupted: after aspire shutdown in one instance the other instance continue the crawl but never ends, Not all Docs are published on Solr.
•  Full test interrupted: after aspire shutdown and restarted docs are not published on Solr.
• NoSQL provider:
• Configurations - Encrypted fields is not working.
• Missing 'NoSQL provider unavailable' message when provider is down.

Anchor
AspireCore3 AspireCore3

Connectors

• Aspider

  • Crawl statistics are shown more items than actually processed.

• Documentum

  • Connector fails during group expansion with large number of users and groups. 

• Jive
  • Resume button

The following items are deprecated on this Aspire version:

• Zip files are not crawled with the Activity Incrementals when they are created inside Jive Documents.

• Amazon S3
• Box
• CEWS Listener
• FTP
• GSA Publisher
• IBM Connections
• PST Extractor
• Publish to HDFS
• Publish to SharePoint 2013
• Publish to SharePoint 2013 (Install & Setup)
• Salesforce
• Subversion
• Teamforge

The following items are marked to be deprecated on the next Aspire version: 

• Elasticsearch bootloader
  • aspire-elastic-bootloader
• DCMaspire-dcm-enterprise
  
  • aspire-amazonec2-dm
  • aspire-zk-dm
  The old Admin UI(s)
  • Parts of aspire-application
• Big Data
  • app-semantic-co-occurrence-hadoop
  • app-semantic-co-occurrence-hadoop-soln
  • aspire-hadoop-job-launcher
  • aspire-hadoop-hdfs
  • aspire-hadoop-wiki-dict-generator
  • aspire-load-hdfs
• Connectors
  • Staging Repo Connector (File System)
  • SVN
• Services:
  • Fast Components
  • Fast Content API
  • Fast Query Completion Listener
  • Fast Query Listener
•  SolutionsSolutions
  • OCR
  • Semantic Co-ocurrence
• Publishers
  • Cloudsearch
  • SharePoint 2013
  • Staging Repo Publisher

• Importing connector with special characters in the path fields not loading correctly.
• Auditing
  • Dump option not working with Solr 6.2.0 & 6.3.0
  • Dump option not working with ElasticSearch 5.0.2
  • Incremental Crawl - Unchanged documents not displayed in Audit Log.
• Aspire Shell
  • The option load-content-sources not working.
  • Relative paths were not working for the commands that create jobs.
  • Sometimes it was possible to delete the Aspire Shell prompt.
• Failed Documents
  • FailedDocuments - Connector getting stuck when stopping the crawl.
• Failover
  • Single instance, full test, interrupted, incremental recovery: Error Processing some files after resuming crawl.
  • During full crawls, some documents were left out if an instance was killed.
  • File System connector resumed crawling after restart.

• Archive Extractor
  • Routing options not working with OnError.
  • Delete by Query not working as expected Using ElasticSearch 5.0.1

• Include and Exclude pattern trimming empty spaces.
• Aspider
  • Aspider - Crawl statistics not displayed on the UI when version was used.
• Documentum
  • GroupExpansion marking groups as users.
  • Error on console was displayed while crawling a folder: "Stream handler unavailable due to: null"
• Heritrix
  • Reject Images/Videos/Javascript/CSS not working for external site out of domain.
• Jira Issues
  • Multiple connection timeouts occurring.
• Jive
  • Crawl generated unnecessary deletes on Normal Incremental with the Use Progressive Retries option.
• RDB Snapshot
  • bigINT SQL Server database type not supported for SQL Slices.
  • Inconsistency when crawling ACL information using ACL fetching options.
  • Crawl not working with a specific column when the Use column from Extraction SQL option was specified.
• RDB Tables
  • Wrong value in "sequence column" parameter not showing UI error.
  • Inconsistent crawling ACL information when using ACL fetching options.
• SharePoint Connectors
  • Renaming one document included by a pattern not generating a deletion.
  • No descriptive error with non-existent URL. 
• SharePoint 2007
  • Connector generating wrong hierarchy on updates.
  • Custom headers not being added to the request.
• SharePoint 2010
  • Documents inside a folder not being picked up by incrementals if the parent ACL changed.

• Import Service option not working when the service under the same name already existed.
• Group Expansion Service
  • Removing GE collection from mongo causing unusable GE for connectors.
• Group Expansion Manager
  • GE Manager - More than one GE Service using the same servlet name generating an error.

• Add Source List not showing until clicking refresh sources.
• Update service name to chain of spaces was not validated all the time.
  • (File System)