Azure Private Link
Outbound connectivity from your Azure virtual network (VNet) and subnets is required to allow query engines to communicate with Dremio Cloud. To establish a more secure and private connection, you can use Azure Private Link, which provides a way for Dremio-managed virtual machines in your Dremio Cloud account to connect to Azure Storage through a private IP address. This connection avoids the public internet and prevents exposure of your traffic outside your VNet and its associated services.
Configuring the Virtual Network
The Azure private endpoints must be created in your existing Dremio subnet.
The following steps walk through preparing your VNet's private subnet for private endpoints.
Log in to the Azure portal.
Search for "Virtual networks" and select the Virtual networks service in the search results.
Click on the name of the VNet that is to be used.
On the VNet page, select Subnets from the left sidebar.
Click on the name of the appropriate subnet.
In the subnet properties panel on the right, for Private endpoint network policy, check Network security groups.
Click Save.
For additional information, see Manage network policies for private endpoints in the Microsoft Azure documentation.
To create or edit a VNet and subnet and add both inbound and outbound rules, perform the steps outlined in Creating a Virtual Network.
Creating a Private Endpoint
Dremio connects with VNets using clouds, which must be configured to your Azure account and private endpoint. Configuring Azure Private Link effectively creates a network interface card (NIC) within your subnet where each endpoint's private IP address serves as an authorized entry point for traffic bound to a specific service, such as Azure Storage. For additional information, see an overview of private endpoints.
To create a private endpoint, complete the following steps:
Log in to the Azure portal.
Search for "Private endpoints" and select the Private endpoints service in the search results.
On the Private endpoints page, click Create in the top left.
On the Basics tab, complete the following:
a. For Subscription, select the appropriate subscription from the dropdown.
b. For Resource group, select the resource group name from the dropdown.
c. For Name, enter a name for the private endpoint.
noteThe private endpoint name must be unique. If you delete this private endpoint (which has been approved for connection by Dremio) and recreate it with the same name, the private endpoint will remain in a pending state because it will not be approved again. In such cases, recreate a private endpoint with different name and update your Dremio Cloud configuration.
d. (Optional) For Network Interface Name, the name should have been created automatically, but you can make edits if needed.
e. For Region, select the VNet region from the dropdown.
f. Click Next : Resource.
On the Resource tab, complete the following:
a. For Connection method, select Connect to an Azure resource by resource ID or alias.
b. For Resource ID or alias, enter the regional resource ID received from Dremio.
c. For Request message, enter a message and include your customer name and organization ID in the message. For example, "Dremio; Organization ID: 10a2a123-0000-0000-9a52-000e70269a32; We'd like to use this endpoint to connect to Dremio Cloud Service through Azure Private Link."
d. Click Next : Virtual Network.
On the Virtual Network tab, complete the following:
a. For Virtual network, select your VNet.
b. For Subnet, select your subnet.
c. For Network policy for private endpoints, select Disabled.
d. For Private IP configuration, select Dynamically allocate IP address.
noteEnsure the VNet and subnet information match according to where you need Dremio to manage the executor compute resources, and save in a location that you can retrieve it from after your required networking resources are set up.
e. Click Next : DNS.
On the DNS tab, accept the default values and click Next : Tags.
On the Tags tab, complete the following:
a. (Optional) For Name, enter a tag name for your resource group.
b. (Optional) For Value, enter a value to help search and filter your resources or track your Azure costs.
c. Click Next : Review + create.
On the Review + create tab, click Create.
After the private endpoint is created, click Go to resource.
On the Private endpoint resource page, copy the private endpoint name at the top of the page or click JSON View on the right and copy the name in the panel. Save the private endpoint name in a location where it can be retrieved, because you'll need the name when connecting your Azure account to Dremio Cloud.
Initially, the status for a newly created private endpoint will display as Pending. When you provide the private endpoint name as part of creating a compute cloud in your Dremio Cloud organization, Dremio will approve the connection and it will display as Approved.
Obtaining the Private Endpoint Name
If you already have an Azure private endpoint, you'll need to obtain the private endpoint name for connecting your Azure account to Dremio Cloud. The private endpoint name is used within the Dremio Cloud service to fully configure the route that the traffic will follow. To locate your private endpoint name and use for the cloud setup process, perform the following steps:
Log in to the Azure portal.
Search for "Private endpoints", and select the Private endpoints in the search results.
On the Private endpoints page, locate the private endpoint that you created.
noteThe status for the private endpoint may initially display as Pending. Upon completion of the Dremio compute cloud creation, the status will display as Approved.
Copy the Private endpoint name.
Return to the cloud setup process for connecting your Azure account to Dremio Cloud and use the private endpoint name in the 3 - Set Up Network Access section.
Next Steps
Dremio is now fully configured to work with your Azure virtual network via Azure Private Link. If you have not already, you should add a source with which to begin managing data.
Additional Information
Private Link: Azure Private Link enables you to access Azure PaaS Services and Azure hosted customer-owned/partner services, such as Dremio, over a private endpoint in your virtual network. Traffic between your virtual network and the service travels the Microsoft backbone network.
Private endpoints: A private endpoint is a network interface that uses a private IP address from your VNet to connect privately and securely to a service, like Dremio, that's powered by Azure Private Link. By enabling a private endpoint, you're bringing the service into your VNet.