Azure Data Lake Store

Azure Configuration

Dremio connects to Azure Data Lake Store using a pre-generated authentication key.

Registering application with Azure Active Directory

  1. Go to Azure Portal and select Azure Active Directory from the left navigation bar. Then select App Registrations.

  2. Click on Endpoints and note down OAUTH 2.0 TOKEN ENDPOINT. This will be the OAuth 2.0 Token Endpoint for your Azure Data Lake Store source in Dremio.

  3. Click New application registration. Select a Name, choose Web app / API as type and enter a Sign-on URL. This URL can be a valid arbitrary URL.

  4. Once created, click on the registered app and note down Application ID. This will be the Application ID for your Azure Data Lake Store source in Dremio.

  5. While in the registered app, select Keys under API Access. Then enter a Description and select an expiration. Click Save and note down Value. This will be the Password for your Azure Data Lake Store source in Dremio.

Granting Azure Data Lake Store access

  1. Go to your desired Data Lake Store resource. Note down the name of the Data Lake Store. This will be the Resource Name for your Azure Data Lake Store source in Dremio.

  2. Click on Access control (IAM) and click Add. Select Owner as the Role and find the name of the application you registered above. Click Save.

  3. For the same Data Lake Store resource, go to Data explorer, then navigate to the top level directory that'll be used by Dremio. Click Access and add Read, Execute and Write (if using as distributed store) permissions to the application registered above.

Dremio Configuration

Here are all available source specific options:

Dremio Field Azure Property
Resource Name Name of the Azure Data Lake Store resource
Application ID Application ID of the registered application under Azure Active Directory
OAuth 2.0 Token Endpoint Azure Active Directory OAuth 2.0 Token Endpoint for registered applications
Password Generated password value for the registered application

REQUIRED: After successful first connection in Dremio

Once the source is successfully created and tested, go back to the source configuration and add the following Connection Property:

Property Name Value
fs.adl.impl.disable.cache false

Restart your cluster after saving the source with this property.

results matching ""

    No results matching ""