Version of Azure Data Lake required

The Aspire Azure Data Lake connector was created and tested using version 3.2.

Before installing the Azure Data Lake connector, make sure that:

• You have necessary Service to Service Application account created with pertinent access to your Data Lake
• Azure Data Lake is up and running
• You have admin rights to allow Read and Execute permissions on folders to Crawl

User Account Requirements 

In order to access Azure Data Lake an Application Account with sufficient privileges must be supplied. The following fields are mandatory to be configured in order to setup a new Data Lake connection:

• Authorization Token End Point: p.e. https://login.microsoftonline.com/[yourkey]/oauth2/token
• Application ID
  
• Application Secret (Application Key)
  

• Fully Qualified Domain Name (FQDN): p.e [yourdomain].azuredatalakestore.net. No HTTP prefix is required
  

Get Application Account

This link will guide you through the steps on how to properly create an Application ID and it's key. Make sure to write down your Application Key at time of creation as it will not be shown again once you exit portal. It's also important to grant necessary reader access to your application, these steps are included.

This connector uses a OAuth 2.0 authorization via Token End Point. Azure will supply this authorization, please follow this link and see Step 4. 

Once these steps are completed, and you have a valid Application created, make sure to grant Read and Execute access - at least - to folders and files to crawl. Steps are included on this link (Step 3). Kindly take recommendation to use Advance Features of Data Lake file explorer to recursively apply same Parent folder permissions to sub-folders folder via "Apply folder permissions to sub-folders" option. In any case Application does not have access to an specific folder. Aspire will log this warning during crawling process

Windows or Linux

The Azure Data Lake connector run on either Windows or Linux instance of Aspire.