Configuring Zookeeper

This topic describes how to configure Zookeeper.

To configure Zookeeper, the zookeeper property with the hostname and port must be added to the dremio.conf file on all nodes in the Dremio cluster. This is particularly important when Zookeeper is on an external node. Default port: 2181

Deployment Zookeeper property
Single-Node cluster The Zookeeper property is not added to the dremio.conf file with a single node cluster.
Multi-node cluster The Zookeeper property must be added to the dremio.conf file on each coordinator and executor node in the Dremio cluster.

[info] Zookeeper Node List

Note that there are no spaces between the comma-separated list of Zookeeper nodes. If there are extra spaces, you may receive an error message similar to the following: Error: java.net.UnknownHostException: 11.22.33.142: Name or service not known

Property Syntax

zookeeper: "<host1>:2181,<host2>:2181"

[info]

The zookeeper host is the hostname (or ID address) where Zookeeper is located. See for more information.

  • If Zookeeper is an embedded Zookeeper on the master node, then the Zookeeper hostname is the hostname of the master node.
    zookeeper: "<master-coordinator-host1>:2181,<master-coordinator-host2>:2181"
  • If Zookeeper is on an external node(s), then it is the hostname of the node(s) where it is located.
    zookeeper: "<zookeeper-host1>:2181,<zookeeper-host2>:2181"

Embedded Zookeeper Examples

The following examples show how to configure an embedded Zookeeper depending on different deployment variables.

  • If you have one coordinator node (masterA) with an embedded Zookeeper and multiple executor nodes, then you specify that master node with the Zookeeper property in the dremio.conf file on every node in the Dremio cluster. json zookeeper: "masterA:2181"it sat

  • If you are configuring for HA and you have an embedded Zookeeper on each coordinator node, then the Zookeeper property specifies those master nodes.

    For example, if masterA is the primary master node and masterB is the secondary master node, then your zookeeper setting is:

      zookeeper: "masterA:2181,masterB:2181"
    
  • If you want to change the default port used by the embedded ZooKeeper, then you add the following property:

      services.coordinator.master.embedded-zookeeper.port: <port>
    
  • If you are configuring for HA, you have an embedded Zookeeper on each master node, and you are using a different port number, then you specify the following settings on each Dremio coordinator and executor node in the cluster.

    For example, if masterA is the primary master node, masterB is the secondary master node, and the zookeeper port is 1234, you specify the following:

      services.coordinator.master.embedded-zookeeper.port: 1234
      zookeeper: "masterA:1234,masterB:1234"
    

External Zookeeper Examples

To use your own external ZooKeeper cluster rather than Dremio's embedded ZooKeeper, modify the zookeeper property in the dremio.conf file. If no ZooKeeper path is specified, Dremio defaults to /dremio.

[info] The dremio.conf must be updated on every Dremio coordinator and executor node in the cluster.

The following example shows the default property and syntax:

services.coordinator.master.embedded-zookeeper.enabled: false
zookeeper: "<host1>:2181,<host2>:2181"
  • If you are configuring for HA, then you set the Zookeeper property for every node that Zookeeper is on.

    For example, if zooA is the primary Zookeeper node and zooB is the secondary Zookeeper node, then your Zookeeper setting is zookeeper: "zooA:2181,zooB:2181".

  • If you are configuring multiple Dremio cluster to run using the same Zookeeper quorum, then you set the property to point to a different root.

    The following example shows sample zookeeper entries in dremio.conf for cluster A and cluster B:

    Cluster A

      zookeeper: "<host1>:2181/path/to/clusterA,<host2>:2181/path/to/clusterA,<host3>:2181/path/to/clusterA"
    

    Cluster B

      zookeeper: "<host1>:2181/path/to/clusterB,<host2>:2181/path/to/clusterB,<host3>:2181/path/to/clusterB"
    

For More Information


results matching ""

    No results matching ""