The Azure Data Lake Connector will The Azure Data Lake Connector will crawl content from a an Azure Data Lake Store repositoryStorage Gen2 for either all file systems or specified file system and paths.
Easy Heading Free | ||||||
---|---|---|---|---|---|---|
|
Azure Data Lake Storage is a comprehensive, scalable, and efficient data lake solution designed for big data analysis, and it provides a hierarchical file system. It brings the capabilities of Azure Data Lake Storage Gen1 together with the Azure Blob storage.
For more information about the Azure Data Lake Storage Gen 2, see the official Microsoft Overview of Azure Data Lake Storage Gen2 documentation.
The Azure Data Lake connector supports crawling Lake Connector supports crawling the following the repositories:
Repository | Version | Connector Version |
---|
For the Azure Data Lake connector to be able to crawl content, the Aspire Worker nodes must be run with an account with .
Azure Data Lake Storage | Gen 2 | 5.1 |
Before installing the Azure Data Lake connector, make sure that:
To access the Azure Data Lake, an Application Account with sufficient privileges must be supplied. The following fields must be configured to set up a new Data Lake connection:
Following are the steps on how to get the required credentials:
Name | Supported |
---|---|
Content Crawling | Yes |
Identity Crawling | Use Azure Identity Connector |
Snapshot-based Incrementals | Yes |
Non-snapshot-based Incrementals | No |
Document Hierarchy | Yes |
The Azure Data Lake connector has the following features:
The Azure The Azure Data Lake connector is able to can crawl the following objects:
Name | Type | Relevant Metadata | Content Fetch and Extraction | Description |
---|
The Azure Data Lake Connector has the following limitations:
<Limitation 1>File System | container | N/A | Contains Folders and files | |
Folders | container | N/A | The directories of the files. Each directory will be scanned to retrieve more subfolders or documents. | |
Files | document | Yes | Files stored in folders/subfolders |