To specify exactly which shared folder to crawl, we will need to create a new "Content Source".
To create a new content source:
In the "General" tab in the Content Source Configuration window, specify basic information for the content source:
In the Connector tab, specify the connection information to crawl a Azure Data Lake folder.
For Windows: D:\folder\folder1\paths.txt
Index Containers: Select if folders are to be indexed
Scan Recursively: Select if sub-folder are to be scanned
Scan Excluded Items: If selected, the scanner will scan sub items of container items that have been excluded by a pattern (because it matches an exclude pattern or because it doesn't match an include pattern)
In the "Workflow" tab, specify the workflow steps for the jobs that come out of the crawl. Drag and drop rules to determine which steps should an item follow after being crawled. This rules could be where to publish the document or transformations needed on the data before sending it to a search engine. See Workflow for more information.
Now that the content source is set up, the crawl can be initiated.
If there are errors, a clickable "Error" flag will take you to a detailed error message page.
If you only want to process content updates from the Azure Data Lake (documents which are added, modified, or removed), then click on the "Incremental" button instead of the "Full" button. The Azure Data Lake connector will automatically identify only changes which have occurred since the last crawl.
If this is the first time that the connector has crawled, the action of the "Incremental" button depends on the exact method of change discovery. It may perform the same action as a "Full" crawl crawling everything, or it may not crawl anything. Thereafter, the Incremental button will only crawl updates.
An Azure Data Lake Store as part of the Azure ecosystem relies on the Azure Active Directory to delimit permissions against Users and Groups. Aspire provides a separate connector to pull those records via service.
Group expansion configuration is performed on the Azure Active Directory Group Expander.
Please refer to Azure Active Directory Group Expander for more information.