Dremio utilizes Apache ZooKeeper behind the scenes for cluster coordination. Dremio automatically runs an embedded ZooKeeper instance on every coordinator node, and this should be entirely transparent to you. You can alternatively configure Dremio to use an external ZooKeeper cluster.
[info] Standalone vs Multi-Node Cluster Deployment
The only time you do not need to add the zookeeper property to the
dremio.conffile is with a single node cluster. Otherwise, this property must be added to the
dremio.conffile on each master-coordinator, coordinator, and executor node in the Dremio cluster.
By default, Dremio runs an embedded ZooKeeper with port 2181.
In a Dremio cluster, other than a standalone node, you must add the Zookeeper property
dremio.conf file on each master-coordinator, coordinator, and executor node in the Dremio cluster.
If you have one master node (masterA) with an embedded Zookeeper and multiple coordinator and executor nodes, then you specify that master node with the Zookeeper property in each master-coordinator, coordinator, and executor node in the
If you are configuring for HA and you have an embedded Zookeeper on each master node, then the Zookeeper property specifies those master nodes.
For example, if masterA is the primary master node and masterB is the secondary master node, then your zookeeper setting is:
If you want to change the default port used by the embedded ZooKeeper, then you add the following property:
If you are configuring for HA, you have an embedded Zookeeper on each master node, and you are using a different port number, then you specify the following settings on each Dremio coordinator and executor node in the cluster.
For example, if masterA is the primary master node, masterB is the secondary master node, and the zookeeper port is 1234, you specify the following:
services.coordinator.master.embedded-zookeeper.port: 1234 zookeeper: "masterA:1234,masterB:1234"
To use your own external ZooKeeper cluster rather than Dremio's embedded ZooKeeper,
you add the following configuration to the
dremio.conf on every Dremio
master-coordinator, coordinator, and executor nodes:
services.coordinator.master.embedded-zookeeper.enabled: false zookeeper: "<host1>:2181,<host2>:2181"
When no ZooKeeper path is specified, Dremio defaults to
If you are configuring for HA, then you set the Zookeeper property for every node that Zookeeper is on.
For example, if zooA is the primary Zookeeper node and zooB is the secondary Zookeeper node, then your Zookeeper setting is
Multiple Dremio Clusters Using a Single Zookeeper Cluster
Dremio clusters can be configured to run using the same Zookeeper quorum by pointing to a different root.
The Zookeeper property must be added to the
dremio.conf file on each of the master-coordinator,
coordinator, and executor nodes in a Dremio cluster.
Sample zookeeper entries in