System Requirements

This topic covers system requirements for standalone clusters including general requirements for Hadoop on YARN and MapR on YARN deployments.

Hadoop Distributions

Dremio 4.3+ supports the following Hadoop distributions:

• Apache Hadoop 2.7.2+ and 3+
• Hortonworks HDP 2.6.5 to HDP 3.0.x
• Cloudera CDH 5 and 6
• MapR 5.2.x and 6.x

Operating Systems

Dremio supports the following distributions and versions of Linux:

• RHEL and CentOS 6.7+ and 7.3+ (RPM and Tarball)
• SLES 12 SP2+ (Tarball)
• Ubuntu 14.04+ (Tarball)
• Debian 7+ (Tarball)

Java Development Kit

Dremio requires that Java SE 8 (also known as JDK 1.8) be installed. Supported distributions are OpenJDK and Oracle JDK. Other versions are currently not supported.

Dremio utilizes the Java compiler (javac) for runtime code generation. You can check to see if your operating system has Java installed (and which version) with this command:

$ java -version

You can download the latest version of Java SE from the Oracle website.

Server or Instance Hardware

The following hardware are minimum recommendations basedon the Dremio service enabled on the node.

Network

There should be a low-latency, high-bandwidth network connection between Dremio and the data sources.

The following ports must be open:

Configuring the Conduit Port

Dremio uses an ephemeral network port allocated by the operating system for inter-node communication between coordinators and executors. To assign a static port number to the conduit port, configure services.conduit.port in the Dremio configuration file. If TLS is enabled on your deployment, Dremio applies the same configuration to communications using the conduit port. To use a different configuration or to enable TLS for only the new conduit port, specify all values for services.conduit.ssl in dremio.conf.

Web Sockets

Dremio uses Web Sockets. If you encounter the following error message: "Your Internet connection may be offline, or WebSockets to Dremio are being blocked.", ensure that your environment allows WebSocket communication.

Performance

A 10 GbE network is recommended when connecting to large data sources that hold terabytes or petabytes of data.

In particular, for maximum performance, it is recommended to use a 10 GbE network between coordinators and executors, executors and executors, and executors and data sources.

Privileges

To install Dremio, the following access privileges are required:

• ssh and scp access
• root or sudo privileges

Best Practices

• For Unix/Linux operating systems, increase your open file limit for users (this impacts the Dremio processes) to 65536.

• If you have a machine with a large amount of RAM (for example, 64GB), increase Dremio's default settings for either the direct memory RAM setting or the HEAP setting. A recommended HEAP value is 8GB.

  To increase Dremio's RAM setting:

  1. In the dremio-env file, modify either the DREMIO_MAX_DIRECT_MEMORY_SIZE_MB or DREMIO_MAX_HEAP_MEMORY_SIZE_MB property.
     See Configuring Memory for more information.
  2. Restart the executor node(s).

[warning] Warning: For the DREMIO_MAX_DIRECT_MEMORY_SIZE_MB allocation, be sure to leave at least 1-2 GB of memory for the OS.

• To verify the memory assigned to the nodes (heap and direct), run the following query:
  SELECT * FROM sys.memory