Deployment Dremio on EKS

This topic provides information about setting up and deploying Dremio on Amazon Elastic Container Service for Kubernetes (EKS).

Requirements

  • Amazon EKS version 1.11 or later.
  • Kubernetes cluster on Amazon EKS. See Setting up an EKS Cluster for instructions.
  • Worker node instance type (minimum): r5d.4xlarge
    (16 core, 128 GiB memory, and 2 x 300 GB NVMe SSD)

Setting up an EKS Cluster

Before you begin, refer to Getting Started with Amazon EKS for more information.

To set up a Kubernetes cluster on Amazon Elastic Container Service for Kubernetes (EKS):

  1. Create an EKS cluster with the console. See Getting Started with Amazon EKS: Create Your Amazon EKS Cluster for more information.
  2. Configure kubectl for Amazon EKS. See Getting Started with Amazon EKS: Configure kubectl for Amazon EKS
  3. Launch and Configure Amazon EKS Worker Nodes. See Getting Started with Amazon EKS: Launch and Configure Amazon EKS Worker Nodes for more information.
  4. Install helm. See Using Helm with Amzaon EKS.
  5. Begin Deploying Dremio.

Deploying Dremio

Deployment of Dremio with Amazon EKS involves the following:

  1. Cloning the Dremio container tools on Github.
  2. Configuring Dremio.
  3. Installing Dremio on the Kubernetes pods.
  4. Connecting to Dremio.

Step 1: Clone the Dremio Containers Tools

To clone the Dremio's Cloud Tools repo into your local environment:
git clone https://github.com/dremio/dremio-cloud-tools.git

Step 2: Configure Dremio

This section provides information on deploying with Dremio Community Edition. To customize the Dremio configuration, see Customizing Deployment for more information.

  1. Go to the Dremio website and obtain the most current Dremio CE image name.
    To deploy with Dremio Enterprise Edition, see Dremio Enterprise Edition.
  2. Modify the following properties in values.yaml file:
    • image: Dremio CE image name. For example: image: dremio/dremio-oss:3.1.0
      (The default for the image property points to the latest Dremio CE version and if you perform an update, it may unintentionally upgrade the Dremio image.)

[info] Out-of-the-Box

The default properties in the the values.yaml file can be used out-of-the-box.

Step 3: Install Dremio on the Kubernetes Pods

Dremio provides Helm charts for installing Dremio on a Kubernetes cluster.

  1. Go to the local directory where you cloned the Dremio's Cloud Tools repository.
  2. Go to the dremio charts directory: cd charts/dremio
  3. Run helm install from the charts directory.
    helm install --wait .
    

[info] Tip

If it takes longer than a couple of minutes to complete, check the status of the pods via kubectl get pods.

If the pods are pending scheduling due to limited memory or cpu, either adjust the values in the values.yaml file (see Changing your Configuration) or add more resources to your Kubernetes cluster.

Step 4: Connect to Dremio

Connect to Dremio with a local shell using one of the following ports along with the Load Balancer external IP address.

  • Port 9047 -- Used for Dremio UI and REST connections
  • Port 31010 -- Used for ODBC and JDBC connections

Example: Dremio UI connection

The Dremio client port 9047 and the Load Balancer's external IP address is used for Dremio UI and REST connections.

In the following example, the Dremio client service (kubectl get services dremio-client) shows that the serviceType is LoadBalancer and the EXTERNAL-IP is 35.226.31.211.

You use 35.226.31.211:9047 to access the Dremio UI.

kubectl get services dremio-client
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                          AGE
dremio-client   LoadBalancer   10.99.227.180   35.226.31.211     31010:32260/TCP,9047:30620/TCP   2d

Example: ODBC or JDBC connection

The Dremio client port 31010 and the Load Balancer's external IP address is used for ODBC and JDBC connections.

In the following example, the Dremio client service (kubectl get services dremio-client) shows that the serviceType is LoadBalancer and the EXTERNAL-IP is 35.226.31.211.

You use 35.226.31.211:31010 in your ODBC or JDBC connections to access Dremio.

kubectl get services dremio-client
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                          AGE
dremio-client   LoadBalancer   10.99.227.180   35.226.31.211     31010:32260/TCP,9047:30620/TCP   2d

For More Information


results matching ""

    No results matching ""