Page History

...

Elasticsearch provider can claim items from queues by larger units and after changing queue items status they are sent back to Elasticsearch as a bulk unit. By this technique we can achieve better performance.  We can configure claim unit size parameters based of on the current Aspire installation (e.g. standalone/ distributed mode, etc.). This is how it works:

• Pre-fetch "claimPrefetch" number of queue items of status "available". The default value is 10 000.
• Take randomly "claim" number of items from the above pre-fetched set. The default value is 10 000.
• Change the status of all "claimed" items from "available" to "inProgress" and sent this as a bulk update request to Elasticsearch.
• Examine the result of the bulk request. All successfully updated items with no version conflicts can be sent one by one as a result of bulk claim to the connector framework.

Keep search context alive

<!-- noSql database provider for the 4.0 connector framework -->
<noSQLConnectionProvider>
    <implementation>com.searchtechnologies.aspire:aspire-elasticsearch-provider</implementation>
    <url>http://localhost:9200</url>
    <keepSearchContextAlive>5m</keepSearchContextAlive>
</noSQLConnectionProvider>

Elasticsearch provider iterators uses scroll technique. The scrolls are resources which should be deleted after their use. This is done explicitly whenever possible by calling iterators close method or when the iteration ends. There are cases when the iteration cannot be completed and in that case "scrolls" stay and can potentially reach the limit of resources. The parameter "keepSearchContextAlive" controls how long the scrolls should stay before deletion. The default value is "5m".  The format of this parameter and other information is described here.

Full configuration

<!-- noSql database provider for the 4.0 connector framework -->
<noSQLConnectionProvider>
    <implementation>com.searchtechnologies.aspire:aspire-elasticsearch-provider</implementation>
    <url>http://localhost:9200</url>
    <claimPrefetch>300</claimPrefetch>
    <claim>100</claim>
    <keepSearchContextAlive>5m</keepSearchContextAlive>
    <authentication type="basic">
        <username>admin</username>
        <password>encrypted:password</password>
    </authentication>
    <debugOutFile>/tmp/aspire/profile.txt</debugOutFile>
    <maxRetries>3</maxRetries>
</noSQLConnectionProvider>