Managing Elastic Engines

Multiple AWS Engines

The AWS Edition of Dremio supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, dynamically schedule execution engines to run independently at different times and automatically start and stop based on workload requirements at runtime. This provides several benefits, including:

  • Workloads are isolated within their own set of CPU, Memory & C3 resources and not impacted by other workloads
  • Time-sensitive yet resource intensive workloads (nightly jobs, reflections refreshes, etc.) can be provisioned with the appropriate amount of resources to complete on time, but remain cost effective by only running when required
  • Track cost by team by running workloads on their own resources
  • Right size execution resources for each distinct workload, instead of implementing a one sized fits model
  • Easily experiment with different execution resource sizes, at any scale

Engines Overview

Creating AWS Engines

New execution Engines on AWS are configured by navigating to the Provisioning page Admin -> Engine -> Elastic Engines. From the Provisioning page select Add New.

Engines Overview

The "Set Up Engine" popup page by default automatically selects EC2 configuration options used to launch the engine (Security Group, EC2 Key Pair, etc). Assuming the engine will use these defaults the main configuration options to specify are:

  • Name - The name for the engine
  • Instance Type - The AWS Instance Type to use for each executor in the engine, different engines may be configured with different instance types.
    • m5d.8xlarge (32c/128gb) - General Purpose (default)
    • m5ad.8xlarge (32c/128gb) - General Purpose
    • r5d.4xlarge (16c/128gb) - Higher memory per unit of compute
    • c5d.18xlarge (72c/144gb) - Higher compute
    • i3.4xlarge (16c/122gb) - Higher local storage for caching
  • Node Count - The number of executors for this engine
  • Advanced Options - for advanced options, see "Advanced Configuration" section below

Engines Overview

After specifying the Engine Name, Instance Type and Node Count, create and start the execution engine by selecting Save & Launch, which:

  1. Saves the execution engines's configuration
  2. Starts the execution engine and provisions EC2 nodes

Typically execution engines complete AWS EC2 provisioning and startup in less than one minute, however timeframes can vary based on instance availability in EC2.

After launching an execution engine, new EC2 instances are provisioned as executor nodes. The state for each node can be monitored on the Provisioning page, nodes start in the Pending state while waiting for AWS EC2 to identify resources, transition to the Provisioning state during startup, and finally transition to Active once the EC2 node and Dremio Software are fully available on the node.

drawing

The execution engine is available for queries once at least 70% of the nodes are available. If that percentage is not available after 5 minutes Dremio cancels the operation and terminates the engine, including any launched nodes.

Monitoring Engines

Engines can be viewed and monitored from two administration pages.

The Elastic Engine admin page shows each engine, the summary status for each node in the engine and controls to start/stop the engine, change the engines's configuration or delete the engine.

drawing

The Node Activity admin page shows each coordinator and executor node along with which engine a node is associated with

drawing

Stopping Engines

Running engines are stopped by selecting Stop on the Provisioning page. Stopping an engine immediately cancels all active queries on the engine and deletes all resources provisioned for the engine, this includes EC2 instances, EBS volumes, etc.

drawing

Starting Engines

Stopped engines are started by selecting Start on the Elastic Engines page. Starting an engine initiates the same process to launch the engine as when the engine was first created and started with Save & Launch described above.

drawing

Routing Queries

There are two methods available to control which engine queries run on.

  1. Workload Management based Routing
  2. Direct Routing

Workload Management based Routing

Workload Management can control the engine each query runs on by defining Rules processed during runtime that determine the Queue for each individual query. In deployments with only one engine, all Queues share the same execution resources and route to the same single engine. However, when provisioning multiple engines, each Queue can be scheduled on a different engine.

To route queries to a specific engine:

  1. Configure WLM Rules to route a set of queries to a specific Queue
  2. Goto Edit Queue and change the Queue's properties and set the Engine Name to desired engine

The following example demonstrates routing high cost reflection rebuild queries to a specific engine just for large reflection rebuilds.

drawing

The default Engine Name for a queue is Any which allows queries in that WLM Queue to run on any engine available at the time of execution. Once a specific Engine Name is selected, queries in that Queue will only run on the selected engine. If the engine is not available the query will fail with the error message Error: No executors are available even if other engines are currently active.

Note: Configurations that either have more than one engine or use the Auto Start feature should specify an Engine Name for all WLM Queues and make sure than no WLM Queues route to 'Any'. 'Any' is intended only for configurations that do not make use of multiple engines and for backward compatibility. Queues that route to 'Any' will not automatically start an engine and can also route queries to engines that are stopped.

Direct Routing

Direct Routing is used to specify the exact Queue and engine to run queries on for a given ODBC or JDBC session. With Direct Routing WLM Rules are not considered and instead queries are routed directly to the specified Queue. Clients can be configured so that all queries run on a specific engines or queries run on different engines on a per-session basis.

To use Direct Routing add the Connection property ROUTING_QUEUE = <WLM Queue Name> to the ODBC or JDBC session parameters when connecting to Dremio. When set all queries for the session are automatically routed to the specified WLM Queue and the engine selected for that Queue.

To disable Tag Routing set the Dremio support key dremio.wlm.direct_routing to false. By default Direct Routing is enabled.

Workload management provides details on how to configure ODBC and JDBC connections for Direct Routing

Automatic Engine Management

Dremio supports automatic management of engines and can automatically both start or stop engines based on workload requirements. Automatic Engine Management eliminates the need to manually provision and terminate engines and helps users cost optimize Drmeio by only running resources when workloads require them.

Auto Start

Auto Start instructs Dremio to automatically provision and start an engine when new queries are issued to that engine. When Enable Auto Start is selected in the engine's Properties page under Advanced Properties, Dremio automatically starts the engine under the following conditions:

  1. The engine is selected as the queue's Engine Name
  2. The execution engine is currently stopped

Engines are only automatically started if they are selected as the Engine Name for a queue. If the Engine Name for a queue is Any and no engines are running, the query will fail with Error: No executors are available instead of selecting a random engine to start.

Auto Stop

Auto Stop instructs Dremio to automatically stop and delete an engine after a period of inactivity. When Enable Auto Stop is selected the engine will automatically stop after no queries are issued to that engine for 5 minutes by default.

The Auto Stop time period can be configured by setting the support key dremio.wlm.shutdown_ms to the desired time period in milliseconds. This setting affects all engines with Auto Stop enabled.

Advanced Configuration

Engine Properties

Additional engine properties to configure are:

Option Description
Name Name of the engine
Instance Type AWS Instance Type used for engine nodes, supported types are:
  • m5d.8xlarge (32c/128gb) - General Purpose (default)
  • m5ad.8xlarge (32c/128gb) - General Purpose
  • r5d.4xlarge (16c/128gb) - Higher memory per unit of compute
  • c5d.18xlarge (72c/144gb) - Higher compute
  • i3.4xlarge (16c/122gb) - Higher local storage for caching
Instance Count Number of execution nodes
EC2 Key Pair Name AWS Key Pair used to log onto server instances
Security Group ID The Group ID of the Security Group to use
IAM Role for S3 Access IAM Role used to access S3 buckets for Distributed Storage (optional)
Use Clustered Placement Whether or not to use placement groups which locates nodes closer together. It is recommended to enable this option but can take longer for AWS to identify resources with larger engines
Enable Auto Start Automatically start the engine when new queries are submitted to the engine
Enable Auto Stop Automatically stop the engine after 5 minutes of inactivity
AWS API Authentication Mode The authentication mode used to provision EC2 nodes and stop EC2 nodes. Auto uses an IAM Role to management execution nodes. Alternatively an AWS Access and Secret Key can be used
Access Key The AWS Access Key if using Key/Secret authentication
Secret The AWS Secret Key if using Key/Secret authentication
IAM Role for API Operations The IAM Role to assume to provision and manage the engine nodes, by default the IAM Role of the coordinator node is used.
Extra Dremio Configuration Properties Additional configuration options

Additional

Additional behaviors to be aware of:

  • When an engine stops the EC2 nodes are terminated and deleted. As a result there are no additional expenses incurred when an engine is stopped, however logs on the executor nodes are not maintained after the engine is stopped.
  • Auto Stop only stops an engine if there is at least one WLM Queue that connects to the engine and routes queries to the engine. Engines not configured within WLM will not automatically stop.
  • Refreshing the node activity page could cause a engine to start
  • If an engine is in the process of stopping, a new query will not cause it to auto start and instead the query will be canceled with an error
  • Stopping the coordinator node does not stop execution engines, execution engines should be stopped prior to shutting down the coordinator node

results matching ""

    No results matching ""