Dremio utilizes Apache ZooKeeper behind the scenes for cluster coordination. Dremio automatically runs an embedded ZooKeeper instance on every coordinator node, and this should be entirely transparent to you. You can alternatively configure Dremio to use an external ZooKeeper cluster.
|Standalone cluster||The Zookeeper property is not added to the dremio.conf file with a single node cluster.|
|Multi-node cluster||The Zookeeper property must be added to the dremio.conf file on each master-coordinator, coordinator, and executor node in the Dremio cluster.|
[info] Zookeeper Node List Note that there are no spaces between the comma-separated list of Zookerper nodes. If there are extra spaces, you may receive an error message similar to the following:
Error: java.net.UnknownHostException: 18.104.22.168: Name or service not known
By default, Dremio runs an embedded ZooKeeper with port 2181. In a Dremio cluster, other than a standalone node, you must add the Zookeeper property to the dremio.conf file on each master-coordinator, coordinator, and executor node in the Dremio cluster.
If you have one master node (masterA) with an embedded Zookeeper and multiple coordinator and executor nodes, then you specify that master node with the Zookeeper property in each master-coordinator, coordinator, and executor node in the dremio.conf file.
json zookeeper: "masterA:2181"it sat
If you are configuring for HA and you have an embedded Zookeeper on each master node, then the Zookeeper property specifies those master nodes.
For example, if masterA is the primary master node and masterB is the secondary master node, then your zookeeper setting is:
If you want to change the default port used by the embedded ZooKeeper, then you add the following property:
If you are configuring for HA, you have an embedded Zookeeper on each master node, and you are using a different port number, then you specify the following settings on each Dremio coordinator and executor node in the cluster.
For example, if masterA is the primary master node, masterB is the secondary master node, and the zookeeper port is 1234, you specify the following:
services.coordinator.master.embedded-zookeeper.port: 1234 zookeeper: "masterA:1234,masterB:1234"
To use your own external ZooKeeper cluster rather than Dremio's embedded ZooKeeper, you add the following configuration to the dremio.conf on every Dremio master-coordinator, coordinator, and executor nodes:
services.coordinator.master.embedded-zookeeper.enabled: false zookeeper: "<host1>:2181,<host2>:2181"
When no ZooKeeper path is specified, Dremio defaults to
If you are configuring for HA, then you set the Zookeeper property for every node that Zookeeper is on.
For example, if zooA is the primary Zookeeper node and zooB is the secondary Zookeeper node, then your Zookeeper setting is
Multiple Dremio Clusters Using a Single Zookeeper Cluster
Dremio clusters can be configured to run using the same Zookeeper quorum by pointing to a different root. The Zookeeper property must be added to the dremio.conf file on each of the master-coordinator, coordinator, and executor nodes in a Dremio cluster.
Sample zookeeper entries in dremio.conf: