Dremio Community

Deployment Dremio on AKS

This topic provides information about setting up and deploying Dremio on Azure AKS.

Requirements

  • Azure AKS version 1.12.7 or later.
  • Kubernetes cluster on Azure AKS. See Setting up an AKS Cluster for instructions.
  • Worker node instance type (minimum): E16s_v3
    (16 core, 128 GiB memory, 400 GiB SSD, and 32 GB disk size)

Setting up an AKS Cluster

To set up a Kubernetes cluster on Azure Kubernetes Service (AKS), use the Azure portal.

  1. Sign in to the Azure portal at https://portal.azure.com.
  2. Create an AKS cluster. See Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal for more information.
    • The number of allocated nodes in the AKS cluster must be equivalent to the number of Dremio executors plus one (1) for the Dremio master-coordinator.
    • If you plan on using AKS logging, you can do so when creating the cluster by enabling container monitoring. Otherwise, you can enable logging after creating the AKS cluster (or to an already existing cluster). See Monitoring Logs for more information.
  3. Connect to the cluster. See Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using the Azure portal for more information.
  4. Install helm. See Install Apps with Helm in AKS for more information.
    • If you plan on using a local shell for access, install helm.
    • If you plan on using the cloud shell for access, initialize helm and install tiller.
  5. Begin Deploying Dremio.

Deploying Dremio

Deployment of Dremio with Azure AKS involves the following:

  1. Cloning the Dremio container tools on Github.
  2. Configuring Dremio.
  3. Installing Dremio on the Kubernetes pods.
  4. Connecting to Dremio.

Step 1: Clone the Dremio Containers Tools

To clone the Dremio’s Cloud Tools repo into your local environment:
git clone https://github.com/dremio/dremio-cloud-tools.git

Step 2: Configure Dremio

This section provides information on deploying with Dremio Community Edition. To customize the Dremio configuration, see Customizing Deployment for more information.

  1. Go to the Dremio website and obtain the most current Dremio CE image name.
    To deploy with Dremio Enterprise Edition, see Dremio Enterprise Edition.
  2. Modify the following properties in values.yaml file:
    • image: Dremio CE image name. For example: image: dremio/dremio-oss:3.1.0
      (The default for the image property points to the latest Dremio CE version and if you perform an update, it may unintentionally upgrade the Dremio image.)

Out-of-the-Box

The default properties in the the values.yaml file can be used out-of-the-box.

Step 3: Install Dremio on the Kubernetes Pods

Dremio provides Helm charts for installing Dremio on a Kubernetes cluster.

  1. Go to the local directory where you cloned the Dremio’s Cloud Tools repository.
  2. Go to the dremio charts directory: cd charts/dremio_v2
  3. Run helm install from the charts directory.
helm install --wait --timeout 900 .

Tip

If it takes longer than a couple of minutes to complete, check the status of the pods via kubectl get pods.

If the pods are pending scheduling due to limited memory or cpu, either adjust the values in the values.yaml file (see Changing your Configuration) or add more resources to your Kubernetes cluster.

Step 4: Connect to Dremio

Connect to Dremio using one of the following ports along with the Load Balancer external IP address.

  • Port 9047 – Used for Dremio UI and REST connections
  • Port 31010 – Used for ODBC and JDBC connections

Example: Dremio UI connection

The Dremio client port 9047 and the Load Balancer’s external IP address is used for Dremio UI and REST connections.

In the following example, the Dremio client service (kubectl get services dremio-client) shows that the serviceType is LoadBalancer and the EXTERNAL-IP is 35.226.31.211.

You use 35.226.31.211:9047 to access the Dremio UI.

kubectl get services dremio-client
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                          AGE
dremio-client   LoadBalancer   10.99.227.180   35.226.31.211     31010:32260/TCP,9047:30620/TCP   2d

Example: ODBC or JDBC connection

The Dremio client port 31010 and the Load Balancer’s external IP address is used for ODBC and JDBC connections.

In the following example, the Dremio client service (kubectl get services dremio-client) shows that the serviceType is LoadBalancer and the EXTERNAL-IP is 35.226.31.211.

You use 35.226.31.211:31010 in your ODBC or JDBC connections to access Dremio.

kubectl get services dremio-client
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                          AGE
dremio-client   LoadBalancer   10.99.227.180   35.226.31.211     31010:32260/TCP,9047:30620/TCP   2d

For More Information