To specify exactly which shared folder to crawl, we will need to create a new "Content Source".
To create a new content source:
In the "General" tab in the Content Source Configuration window, specify basic information for the content source:
|You can add more schedules by clicking in the Add New option, and rearranging the order of the schedules.|
|If you want to disable the content source, clear the Enable check box. This is useful if the folder will be under maintenance and no crawls are wanted during that period of time.|
|Real Time and Cache Groups crawl will be available depending of the connector.|
In the Connector tab, specify the connection information to crawl a Azure Data Lake folder.
For Windows: D:\folder\folder1\paths.txt
Index Containers: Select if folders are to be indexed
Scan Recursively: Select if sub-folder are to be scanned
Scan Excluded Items: If selected, the scanner will scan sub items of container items that have been excluded by a pattern (because it matches an exclude pattern or because it doesn't match an include pattern)
In the "Workflow" tab, specify the workflow steps for the jobs that come out of the crawl. Drag and drop rules to determine which steps should an item follow after being crawled. This rules could be where to publish the document or transformations needed on the data before sending it to a search engine. See See Workflow for more information.
Now that the content source is set up, the crawl can be initiated.
If there are errors, a clickable "Error" flag will take you to a detailed error message page.
If you only want to process content updates from the Azure Data Lake (documents which are added, modified, or removed), then click on the "Incremental" button instead of the "Full" button. The Azure Data Lake connector will automatically identify only changes which have occurred since the last crawl.
If this is the first time that the connector has crawled, the action of the "Incremental" button depends on the exact method of change discovery. It may perform the same action as a "Full" crawl crawling everything, or it may not crawl anything. Thereafter, the Incremental button will only crawl updates.
|Statistics are reset for every crawl.|
An Azure Data Lake Store as part of the Azure ecosystem relies on the Azure Active Directory to delimit permissions against Users and Groups. Aspire provides a separate connector to pull those records via service.
Group expansion configuration is performed on the Azure Active Directory Group Expander.
Please refer to Azure Active Directory Group Expander for more information.