Managing Elastic Engines
Multiple AWS Engines
The AWS Edition of Dremio supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, dynamically schedule execution engines to run independently at different times and automatically start and stop based on workload requirements at runtime. This provides several benefits, including:
- Workloads are isolated within their own set of CPU, Memory & C3 resources and not impacted by other workloads
- Time-sensitive yet resource intensive workloads (nightly jobs, reflections refreshes, etc.) can be provisioned with the appropriate amount of resources to complete on time, but remain cost effective by only running when required
- Track cost by team by running workloads on their own resources
- Right size execution resources for each distinct workload, instead of implementing a one sized fits model
- Easily experiment with different execution resource sizes, at any scale
Creating AWS Engines
New execution Engines on AWS are configured by navigating to the Provisioning page
Admin -> Engine -> Elastic Engines. From the Provisioning page select
The "Set Up Engine" popup page by default automatically selects EC2 configuration options used to launch the engine (Security Group, EC2 Key Pair, etc). Assuming the engine will use these defaults the main configuration options to specify are:
- Name - The name for the engine
- Instance Type - The AWS Instance Type to use for each executor in the engine, different engines may be configured with different instance types.
- m5d.8xlarge (32c/128gb) - General Purpose (default)
- m5ad.8xlarge (32c/128gb) - General Purpose
- r5d.4xlarge (16c/128gb) - Higher memory per unit of compute
- c5d.18xlarge (72c/144gb) - Higher compute
- i3.4xlarge (16c/122gb) - Higher local storage for caching
- Node Count - The number of executors for this engine
- Advanced Options - for advanced options, see "Advanced Configuration" section below
After specifying the Engine Name, Instance Type and Node Count, create and start the execution engine by selecting
Save & Launch, which:
- Saves the execution engines's configuration
- Starts the execution engine and provisions EC2 nodes
Typically execution engines complete AWS EC2 provisioning and startup in less than one minute, however timeframes can vary based on instance availability in EC2.
After launching an execution engine, new EC2 instances are provisioned as executor nodes. The state for each node can be monitored on the Provisioning page, nodes start in the
Pending state while waiting for AWS EC2 to identify resources, transition to the
Provisioning state during startup, and finally transition to
Active once the EC2 node and Dremio Software are fully available on the node.
The execution engine is available for queries once at least 70% of the nodes are available. If that percentage is not available after 5 minutes Dremio cancels the operation and terminates the engine, including any launched nodes.
Engines can be viewed and monitored from two administration pages.
The Elastic Engine admin page shows each engine, the summary status for each node in the engine and controls to start/stop the engine, change the engines's configuration or delete the engine.
The Node Activity admin page shows each coordinator and executor node along with which engine a node is associated with
Running engines are stopped by selecting
Stop on the Provisioning page. Stopping an engine immediately cancels all active queries on the engine and deletes all resources provisioned for the engine, this includes EC2 instances, EBS volumes, etc.
Stopped engines are started by selecting
Start on the Elastic Engines page. Starting an engine initiates the same process to launch the engine as when the engine was first created and started with
Save & Launch described above.
There are two methods available to control which engine queries run on.
- Workload Management based Routing
- Direct Routing
Workload Management based Routing
Workload Management can control the engine each query runs on by defining Rules processed during runtime that determine the Queue for each individual query. In deployments with only one engine, all Queues share the same execution resources and route to the same single engine. However, when provisioning multiple engines, each Queue can be scheduled on a different engine.
To route queries to a specific engine:
- Configure WLM Rules to route a set of queries to a specific Queue
Edit Queueand change the Queue's properties and set the Engine Name to desired engine
The following example demonstrates routing high cost reflection rebuild queries to a specific engine just for large reflection rebuilds.
The default Engine Name for a queue is
Any which allows queries in that WLM Queue to run on any engine available at the time of execution. Once a specific Engine Name is selected, queries in that Queue will only run on the selected engine. If the engine is not available the query will fail with the error message
Error: No executors are available even if other engines are currently active.
Note: Configurations that either have more than one engine or use the Auto Start feature should specify an Engine Name for all WLM Queues and make sure than no WLM Queues route to 'Any'. 'Any' is intended only for configurations that do not make use of multiple engines and for backward compatibility. Queues that route to 'Any' will not automatically start an engine and can also route queries to engines that are stopped.
Direct Routing is used to specify the exact Queue and engine to run queries on for a given ODBC or JDBC session. With Direct Routing WLM Rules are not considered and instead queries are routed directly to the specified Queue. Clients can be configured so that all queries run on a specific engines or queries run on different engines on a per-session basis.
To use Direct Routing add the Connection property
ROUTING_QUEUE = <WLM Queue Name> to the ODBC or JDBC session parameters when connecting to Dremio. When set all queries for the session are automatically routed to the specified WLM Queue and the engine selected for that Queue.
To disable Tag Routing set the Dremio support key
dremio.wlm.direct_routing to false. By default Direct Routing is enabled.
Workload management provides details on how to configure ODBC and JDBC connections for Direct Routing
Automatic Engine Management
Dremio supports automatic management of engines and can automatically both start or stop engines based on workload requirements. Automatic Engine Management eliminates the need to manually provision and terminate engines and helps users cost optimize Drmeio by only running resources when workloads require them.
Auto Start instructs Dremio to automatically provision and start an engine when new queries are issued to that engine. When
Enable Auto Start is selected in the engine's Properties page under Advanced Properties, Dremio automatically starts the engine under the following conditions:
- The engine is selected as the queue's Engine Name
- The execution engine is currently stopped
Engines are only automatically started if they are selected as the Engine Name for a queue. If the Engine Name for a queue is
Any and no engines are running, the query will fail with
Error: No executors are available instead of selecting a random engine to start.
Auto Stop instructs Dremio to automatically stop and delete an engine after a period of inactivity. When
Enable Auto Stop is selected the engine will automatically stop after no queries are issued to that engine for 5 minutes by default.
The Auto Stop time period can be configured by setting the support key
dremio.wlm.shutdown_ms to the desired time period in milliseconds. This setting affects all engines with Auto Stop enabled.
Additional engine properties to configure are:
|Name||Name of the engine|
|Instance Type||AWS Instance Type used for engine nodes, supported types are:
|Instance Count||Number of execution nodes|
|EC2 Key Pair Name||AWS Key Pair used to log onto server instances|
|Security Group ID||The Group ID of the Security Group to use|
|IAM Role for S3 Access||IAM Role used to access S3 buckets for Distributed Storage (optional)|
|Use Clustered Placement||Whether or not to use placement groups which locates nodes closer together. It is recommended to enable this option but can take longer for AWS to identify resources with larger engines|
|Enable Auto Start||Automatically start the engine when new queries are submitted to the engine|
|Enable Auto Stop||Automatically stop the engine after 5 minutes of inactivity|
|AWS API Authentication Mode||The authentication mode used to provision EC2 nodes and stop EC2 nodes. Auto uses an IAM Role to management execution nodes. Alternatively an AWS Access and Secret Key can be used|
|Access Key||The AWS Access Key if using Key/Secret authentication|
|Secret||The AWS Secret Key if using Key/Secret authentication|
|IAM Role for API Operations||The IAM Role to assume to provision and manage the engine nodes, by default the IAM Role of the coordinator node is used.|
|Extra Dremio Configuration Properties||Additional configuration options|
Additional behaviors to be aware of:
- When an engine stops the EC2 nodes are terminated and deleted. As a result there are no additional expenses incurred when an engine is stopped, however logs on the executor nodes are not maintained after the engine is stopped.
- Auto Stop only stops an engine if there is at least one WLM Queue that connects to the engine and routes queries to the engine. Engines not configured within WLM will not automatically stop.
- Refreshing the node activity page could cause a engine to start
- If an engine is in the process of stopping, a new query will not cause it to auto start and instead the query will be canceled with an error
- Stopping the coordinator node does not stop execution engines, execution engines should be stopped prior to shutting down the coordinator node