Deployment Dremio on AKS
This topic provides information about setting up and deploying Dremio on Azure AKS.
Requirements
- Azure AKS version 1.12.7 or later.
- Kubernetes cluster on Azure AKS. See Setting up an AKS Cluster for instructions.
- Worker node instance type (minimum): E16s_v3
(16 core, 128 GiB memory, 400 GiB SSD, and 32 GB disk size)
Setting up an AKS Cluster
To set up a Kubernetes cluster on Azure Kubernetes Service (AKS), use the Azure portal.
- Sign in to the Azure portal at https://portal.azure.com.
- Create an AKS cluster. See Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal for more information.
- The number of allocated nodes in the AKS cluster must be equivalent to the number of Dremio executors plus one (1) for the Dremio master-coordinator.
- If you plan on using AKS logging, you can do so when creating the cluster by enabling container monitoring. Otherwise, you can enable logging after creating the AKS cluster (or to an already existing cluster). See Monitoring Logs for more information.
- Connect to the cluster. See Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal for more information.
- Install helm. See Install Apps with Helm in AKS for more information.
- If you plan on using a local shell for access, install helm.
- If you plan on using the cloud shell for access, initialize helm and install tiller.
- Begin Deploying Dremio.
Deploying Dremio
Deployment of Dremio with Azure AKS involves the following:
- Cloning the Dremio container tools on Github.
- Configuring Dremio.
- Installing Dremio on the Kubernetes pods.
- Connecting to Dremio.
Step 1: Clone the Dremio Containers Tools
To clone the Dremio's Cloud Tools
repo into your local environment:git clone https://github.com/dremio/dremio-cloud-tools.git
Step 2: Configure Dremio
This section provides information on deploying with Dremio Community Edition. To customize the Dremio configuration, see Customizing Deployment for more information.
- Go to the Dremio website and obtain the most current Dremio CE image name.
To deploy with Dremio Enterprise Edition, see Dremio Enterprise Edition. - Modify the following properties in values.yaml file:
image
: Dremio CE image name. For example:image: dremio/dremio-oss:3.1.0
(The default for theimage
property points to the latest Dremio CE version and if you perform an update, it may unintentionally upgrade the Dremio image.)
[info] Out-of-the-Box
The default properties in the the values.yaml file can be used out-of-the-box.
Step 3: Install Dremio on the Kubernetes Pods
Dremio provides Helm charts for installing Dremio on a Kubernetes cluster.
- Go to the local directory where you cloned the Dremio's Cloud Tools repository.
- Go to the dremio charts directory:
cd charts/dremio_v2
- Run
helm install
from the charts directory.helm install --wait --timeout 900 .
[info] Tip
If it takes longer than a couple of minutes to complete, check the status of the pods via
kubectl get pods
.If the pods are pending scheduling due to limited memory or cpu, either adjust the values in the values.yaml file (see Changing your Configuration) or add more resources to your Kubernetes cluster.
Step 4: Connect to Dremio
Connect to Dremio using one of the following ports along with the Load Balancer external IP address.
- Port 9047 -- Used for Dremio UI and REST connections
- Port 31010 -- Used for ODBC and JDBC connections
Example: Dremio UI connection
The Dremio client port 9047 and the Load Balancer's external IP address is used for Dremio UI and REST connections.
In the following example, the Dremio client service (kubectl get services dremio-client
) shows
that the serviceType is LoadBalancer and the EXTERNAL-IP is 35.226.31.211.
You use 35.226.31.211:9047
to access the Dremio UI.
kubectl get services dremio-client
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dremio-client LoadBalancer 10.99.227.180 35.226.31.211 31010:32260/TCP,9047:30620/TCP 2d
Example: ODBC or JDBC connection
The Dremio client port 31010 and the Load Balancer's external IP address is used for ODBC and JDBC connections.
In the following example, the Dremio client service (kubectl get services dremio-client
) shows
that the serviceType is LoadBalancer and the EXTERNAL-IP is 35.226.31.211.
You use 35.226.31.211:31010
in your ODBC or JDBC connections to access Dremio.
kubectl get services dremio-client
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dremio-client LoadBalancer 10.99.227.180 35.226.31.211 31010:32260/TCP,9047:30620/TCP 2d