Managing Elastic Engines
Multiple AWS Engines
The AWS Edition of Dremio supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, dynamically schedule execution engines to run independently at different times and automatically start and stop based on workload requirements at runtime. This provides several benefits, including:
- Workloads are isolated within their own set of CPU, Memory & C3 resources and not impacted by other workloads
- Time-sensitive yet resource intensive workloads (nightly jobs, reflections refreshes, etc.) can be provisioned with the appropriate amount of resources to complete on time, but remain cost effective by only running when required
- Track cost by team by running workloads on their own resources
- Right size execution resources for each distinct workload, instead of implementing a one sized fits model
- Easily experiment with different execution resource sizes, at any scale
Creating AWS Engines
Create and configure new execution engines for AWS Edition by navigating to the Elastic Engines page and clicking New Engine.
- Engine Name - The name for the engine
- Engine Size (Nodes) - The number of nodes in the Dremio cluster. Choose
Customto override the default instance type for for each selection. - Number of Nodes - The number of nodes in the engine. It only appears when
Customis selected forEngine Size (Nodes). - Engine Node Type - The EC2 instance type of the nodes in the engine. It only appears when
Customis selected forEngine Size (Nodes). The instance types are:- m5d.2xlarge (8c/32gb) - Evaluation
- m5d.8xlarge (32c/128gb) - Standard (default)
- r5d.4xlarge (16c/128gb) - High memory per unit of compute
- c5d.18xlarge (72c/144gb) - High CPU
- i3.4xlarge (16c/122gb) - Higher local storage for caching
- Advanced Options - See “Advanced Configuration” below
After specifying the Engine Name, Instance Type and Node Count, create and start the execution engine by selecting Save & Launch, which:
- Saves the execution engines’s configuration
- Starts the execution engine and provisions EC2 nodes
Typically execution engines complete AWS EC2 provisioning and startup in less than one minute, however timeframes can vary based on instance availability in EC2.
After launching an execution engine, new EC2 instances are provisioned as executor nodes. The state for each node can be monitored on the Provisioning page, nodes start in the Pending state while waiting for AWS EC2 to identify resources, transition to the Provisioning state during startup, and finally transition to Active once the EC2 node and Dremio Software are fully available on the node.
The execution engine is available for queries once at least 70% of the nodes are available. If that percentage is not available after 5 minutes Dremio cancels the operation and terminates the engine, including any launched nodes.
Monitoring Engines
Click an engine on the Engines page to display a detailed status of the engine or to stop the engine. Click Edit Settings to reconfigure the engine.
The Node Activity page shows each coordinator and executor node, the engine associated with the node, and the Stop Project button.
Stopping Engines
Running engines are stopped by selecting Stop on the Elastic Engines page. Stopping an engine immediately cancels all active queries on the engine and deletes all resources provisioned for the engine, this includes EC2 instances, EBS volumes, etc.
Starting Engines
Start a stopped engines by clicking Start on the Elastic Engines page. Starting an engine initiates the same process to launch the engine as when the engine was first created and started with Save & Launch described above.
Routing Queries
There are two methods available to control which engine queries run on.
- Workload Management based Routing
- Direct Routing
Workload Management based Routing
Workload Management can control the engine each query runs on by defining Rules processed during runtime that determine the Queue for each individual query. In deployments with only one engine, all Queues share the same execution resources and route to the same single engine. However, when provisioning multiple engines, each Queue can be scheduled on a different engine.
To route queries to a specific engine:
- Configure WLM Rules to route a set of queries to a specific Queue
- Goto
Edit Queueand change the Queue’s properties and set the Engine Name to desired engine
The following example demonstrates routing high cost reflection rebuild queries to a specific engine just for large reflection rebuilds.
The default Engine Name for a queue is Any which allows queries in that WLM Queue to run on any engine available at the time of execution. Once a specific Engine Name is selected, queries in that Queue will only run on the selected engine. If the engine is not available the query will fail with the error message Error: No executors are available even if other engines are currently active.
Note: Configurations that either have more than one engine or use the Auto Start feature should specify an Engine Name for all WLM Queues and make sure than no WLM Queues route to ‘Any’. ‘Any’ is intended only for configurations that do not make use of multiple engines and for backward compatibility. Queues that route to ‘Any’ will not automatically start an engine and can also route queries to engines that are stopped.
Direct Routing
Direct Routing is used to specify the exact Queue and engine to run queries on for a given ODBC or JDBC session. With Direct Routing WLM Rules are not considered and instead queries are routed directly to the specified Queue. Clients can be configured so that all queries run on a specific engines or queries run on different engines on a per-session basis.
To use Direct Routing add the Connection property ROUTING_QUEUE = <WLM Queue Name> to the ODBC or JDBC session parameters when connecting to Dremio. When set all queries for the session are automatically routed to the specified WLM Queue and the engine selected for that Queue.
To disable Tag Routing set the Dremio support key dremio.wlm.direct_routing to false. By default Direct Routing is enabled.
Workload management provides details on how to configure ODBC and JDBC connections for Direct Routing
Automatic Engine Management
Dremio supports automatic management of engines and can automatically both start or stop engines based on workload requirements. Automatic Engine Management eliminates the need to manually provision and terminate engines and helps users cost optimize Drmeio by only running resources when workloads require them.
Auto Start
Auto Start instructs Dremio to automatically provision and start an engine when new queries are issued to that engine. When Enable Auto Start is selected in the engine’s Properties page under Advanced Properties, Dremio automatically starts the engine under the following conditions:
- The engine is selected as the queue’s Engine Name
- The execution engine is currently stopped
Engines are only automatically started if they are selected as the Engine Name for a queue. If the Engine Name for a queue is Any and no engines are running, the query will fail with Error: No executors are available instead of selecting a random engine to start.
Auto Stop
Auto Stop instructs Dremio to automatically stop and delete an engine after a period of inactivity. When Enable Auto Stop is selected the engine will automatically stop after no queries are issued to that engine for 5 minutes by default.
The Auto Stop time period can be configured by setting the support key dremio.wlm.shutdown_ms to the desired time period in milliseconds. This setting affects all engines with Auto Stop enabled.
Advanced Configuration
By default, the Set Up Elastic Engine dialog automatically selects the security group ID and EC2 key pair used to launch the existing engine.
Engine Properties
Additional engine properties to configure are:
| Option | Description |
|---|---|
| Name | Name of the engine |
| Instance Type | AWS Instance Type used for engine nodes. The supported types are:
|
| Instance Count | Number of execution nodes |
| EC2 Key Pair Name | AWS Key Pair used to log onto server instances |
| Security Group ID | The Group ID of the Security Group to use |
| IAM Role for S3 Access | IAM Role used to access S3 buckets for Distributed Storage (optional) |
| Use Clustered Placement | Whether or not to use placement groups which locates nodes closer together. It is recommended to enable this option but can take longer for AWS to identify resources with larger engines |
| Enable Auto Start | Automatically start the engine when new queries are submitted to the engine |
| Enable Auto Stop | Automatically stop the engine after 5 minutes of inactivity |
| AWS API Authentication Mode | The authentication mode used to provision EC2 nodes and stop EC2 nodes. Auto uses an IAM Role to management execution nodes. Alternatively an AWS Access and Secret Key can be used |
| Access Key | The AWS Access Key if using Key/Secret authentication |
| Secret | The AWS Secret Key if using Key/Secret authentication |
| IAM Role for API Operations | The IAM Role to assume to provision and manage the engine nodes, by default the IAM Role of the coordinator node is used. |
| Extra Dremio Configuration Properties | Additional configuration options |
Additional
Additional behaviors to be aware of:
- When an engine stops the EC2 nodes are terminated and deleted. As a result there are no additional expenses incurred when an engine is stopped, however logs on the executor nodes are not maintained after the engine is stopped.
- Auto Stop only stops an engine if there is at least one WLM Queue that connects to the engine and routes queries to the engine. Engines not configured within WLM will not automatically stop.
- Refreshing the node activity page could cause a engine to start
- If an engine is in the process of stopping, a new query will not cause it to auto start and instead the query will be canceled with an error
- Stopping the coordinator node does not stop execution engines, execution engines should be stopped prior to shutting down the coordinator node