Red Hat OpenShift
Learn more about the deployment architecture, requirements, and recommendations for installing Dremio in an OpenShift cluster on Red Hat.
Architecture
High Availability
High availability is dependent on the Red Hat OpenShift infrastructure. If any of the Kubernetes pods go down for any reason, Kubernetes brings up another pod to replace the pod that is out of commission.
-
The Dremio master-coordinator and secondary-coordinator pods are each StatefulSet. If the master-coordinator pod goes down, it recovers with the associated persistent volume and Dremio metadata preserved.
-
The Dremio executor pods are a StatefulSet with an associated persistent volume. secondary-coordinator pods do not have a persistent volume. If an executor pod goes down, it recovers with the associated persistent volume and data preserved.
Requirements
Kubernetes Version
To ensure compatibility and support, Dremio requires regular updates to your Kubernetes version in OpenShift to stay on an officially supported version. For details on supported versions, refer to the Red Hat OpenShift documentation on their release notes of each version. For example, for version 4.16, go to OpenShift Container Platform 4.16 release notes. You can use the dropdown in the breadcrumb to change to other versions.
Helm Charts
Dremio requires regular updates to the latest version of the official Dremio Helm charts, which are available in the dremio-cloud-tools repository on GitHub. When using the official Dremio Helm charts (Dremio does not support the use of Helm charts published by third parties), it is strongly recommended to commit the Helm configuration file (i.e. values.yaml file) along with the rest of the Helm chart to a version control system such as Git.
Dremio Docker Image
Dremio requires using the official Dremio Docker image. Any modifications to this image must be preapproved by Dremio before use, and Dremio does not support the inclusion or execution of other applications within the Dremio image. Separate applications must be run in their own containers to avoid potential interference with the Dremio application.
Distributed Storage
Dremio requires object storage to be configured as distributed storage, where the type of object storage to use will depend on the location and edition of the OpenShift cluster.
For self-hosted Container Platform Edition clusters, we recommend using object storage from a cloud provider for hybrid architectures or, if fully on-premises, the usage of MinIO. For further details, see the documentation on Configuring Dremio for Minio.
ZooKeeper Pods
Three ZooKeeper pods run on the default system pool. Each ZooKeeper pod requires an additional node and should have 1 CPU and 2 GB memory. For resiliency against individual node failures, these pods are expected to run on separate nodes.
Disk Storage Class
Dremio requires the use of storage classes that guarantee 100 MB/s throughput for the following volumes:
- Coordinator volume: 512 GB
- Executor volume #1 (results & spilling): 256 GB
- Executor volume #2 (Columnar Cloud Cache): 256 GB
- ZooKeeper volume: 16 GB
For executors, if the underlying node provides at least 500 GB of local SSD/NVMe storage included, we recommend using these local disks instead of disks attached over the network. Since the executor data is cached or consists of result sets, these disks do not need to be reattachable in case of a node failure. Local NVMes storage provides significantly faster I/O compared with disks attached over the network.
Recommendations
Container Registry
Required Docker images (e.g., Dremio Enterprise Edition, ZooKeeper, Nessie, etc.) should be pushed to a private container registry (e.g., Azure Container Registry), because Dremio does not provide any service-level agreements (SLAs) for Quay.io and docker.io repositories. Depending on the type of OpenShift edition in use, you should use the container registry specific to your hosting provider or Red Hat Quay for self managed clusters.
Node Sizes
When determining node sizes, the amount of CPU and memory available for Dremio will always be less than the total node capacity because Kubernetes, the system pods, and the operating system also consume resources. Typically, 2 CPUs and 10-20 GB of memory are subtracted from the node's theoretical maximum when allocating resources to the Dremio pods.
The many different OpenShift editions available mean that there is no unique hardware recommendation, but we will provide comparable recommendations for AWS, Azure, and IBM Cloud editions. For self-hosted Container Platform edition clusters, please refer to the service-specific paragraphs to choose the resources to allocate to your nodes.
For Azure, Dlsv5 (no local storage but Premium disk support for coordinators) or Ddsv5 (with local storage support and Premium disk support for executors) VM series are recommended. Both are part of the same family as Ddv5, which is used in Dremio Cloud.
For AWS, m5d VM series are recommended, which have NVMe SSD support. They are part of the same family as those used in Dremio Cloud.
For IBM, bx2d VM series are recommended, as they support NVMe SSD as secondary local storage.
Coordinators
For coordinators, Dremio recommends using VMs (or equivalent) with at least 32 CPUs and 64 GB of memory, offering a CPU-to-memory ratio of 1:2. In Helm charts, this configuration results in approximately 30 CPUs and 54 GB of memory allocated to the pod.
For more information on JVM Garbage Collection (GC) parameters, see G1GC settings for the Dremio JVM.
If workload demands exceed the limits of a single coordinator, additional scale-out coordinators should be added. For more information on when and how to scale coordinators, see this PDF guide on Evaluating Coordinator Scaling.
To enable flexible scaling, it is recommended to set the coordinator node pool to autoscale.
Executors
For executors, Dremio recommends using VMs to achieve a node size of 32 CPUs and 128 GB of memory, maintaining a CPU-to-memory ratio of 1:4. In Helm charts, this results in approximately 30 CPUs and 108 GB of memory allocated to the pod.
To allow elastic resources scaling, it is required to set the executor node pool to autoscale, if such an option is available. This will allow new VMs to be provisioned automatically, when Dremio intends to scale out its executors.
Load Balancing
Although the provided Helm charts include a basic load balancer server configuration, it is strongly recommended to operate the load balancer independently (e.g., via Nginx Helm charts or a load balancer provided by the OpenShift platform).
Network
The network bandwidth should be at least 10 Gbps.