The AWS Edition of Dremio supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, dynamically schedule execution engines to run independently at different times and automatically start and stop based on workload requirements at runtime. This provides several benefits, including:
Create and configure new execution engines for AWS Edition by navigating to the
Elastic Engines page and clicking
Customto override the default instance type for for each selection.
Customis selected for
Engine Size (Nodes).
Customis selected for
Engine Size (Nodes). The instance types are:
After specifying the Engine Name, Instance Type and Node Count, create and start the execution engine by selecting
Save & Launch, which:
Typically execution engines complete AWS EC2 provisioning and startup in less than one minute, however timeframes can vary based on instance availability in EC2.
After launching an execution engine, new EC2 instances are provisioned as executor nodes. The state for each node can be monitored on the Provisioning page, nodes start in the
Pending state while waiting for AWS EC2 to identify resources, transition to the
Provisioning state during startup, and finally transition to
Active once the EC2 node and Dremio Software are fully available on the node.
The execution engine is available for queries once at least 70% of the nodes are available. If that percentage is not available after 5 minutes Dremio cancels the operation and terminates the engine, including any launched nodes.
Click an engine on the
Engines page to display a detailed status of the engine or to stop the engine. Click
Edit Settings to reconfigure the engine.
Node Activity page shows each coordinator and executor node, the engine associated with the node, and the
Stop Project button.
Running engines are stopped by selecting
Stop on the
Elastic Engines page. Stopping an engine immediately cancels all active queries on the engine and deletes all resources provisioned for the engine, this includes EC2 instances, EBS volumes, etc.
Start a stopped engines by clicking
Start on the
Elastic Engines page. Starting an engine initiates the same process to launch the engine as when the engine was first created and started with
Save & Launch described above.
There are two methods available to control which engine queries run on.
Workload Management can control the engine each query runs on by defining Rules processed during runtime that determine the Queue for each individual query. In deployments with only one engine, all Queues share the same execution resources and route to the same single engine. However, when provisioning multiple engines, each Queue can be scheduled on a different engine.
To route queries to a specific engine:
Edit Queueand change the Queue’s properties and set the Engine Name to desired engine
The following example demonstrates routing high cost reflection rebuild queries to a specific engine just for large reflection rebuilds.
The default Engine Name for a queue is
Any which allows queries in that WLM Queue to run on any engine available at the time of execution. Once a specific Engine Name is selected, queries in that Queue will only run on the selected engine. If the engine is not available the query will fail with the error message
Error: No executors are available even if other engines are currently active.
Note: Configurations that either have more than one engine or use the Auto Start feature should specify an Engine Name for all WLM Queues and make sure than no WLM Queues route to ‘Any’. ‘Any’ is intended only for configurations that do not make use of multiple engines and for backward compatibility. Queues that route to ‘Any’ will not automatically start an engine and can also route queries to engines that are stopped.
Direct Routing is used to specify the exact Queue and engine to run queries on for a given ODBC or JDBC session. With Direct Routing WLM Rules are not considered and instead queries are routed directly to the specified Queue. Clients can be configured so that all queries run on a specific engines or queries run on different engines on a per-session basis.
To use Direct Routing add the Connection property
ROUTING_QUEUE = <WLM Queue Name> to the ODBC or JDBC session parameters when connecting to Dremio. When set all queries for the session are automatically routed to the specified WLM Queue and the engine selected for that Queue.
To disable Tag Routing set the Dremio support key
dremio.wlm.direct_routing to false. By default Direct Routing is enabled.
Workload management provides details on how to configure ODBC and JDBC connections for Direct Routing
Dremio supports automatic management of engines and can automatically both start or stop engines based on workload requirements. Automatic Engine Management eliminates the need to manually provision and terminate engines and helps users cost optimize Drmeio by only running resources when workloads require them.
Auto Start instructs Dremio to automatically provision and start an engine when new queries are issued to that engine. When
Enable Auto Start is selected in the engine’s Properties page under Advanced Properties, Dremio automatically starts the engine under the following conditions:
Engines are only automatically started if they are selected as the Engine Name for a queue. If the Engine Name for a queue is
Any and no engines are running, the query will fail with
Error: No executors are available instead of selecting a random engine to start.
Auto Stop instructs Dremio to automatically stop and delete an engine after a period of inactivity. When
Enable Auto Stop is selected the engine will automatically stop after no queries are issued to that engine for 5 minutes by default.
The Auto Stop time period can be configured by setting the support key
dremio.wlm.shutdown_ms to the desired time period in milliseconds. This setting affects all engines with Auto Stop enabled.
By default, the
Set Up Elastic Engine dialog automatically selects the security group ID and EC2 key pair used to launch the existing engine.
Additional engine properties to configure are:
|Name||Name of the engine|
|Instance Type||AWS Instance Type used for engine nodes. The supported types are:
|Instance Count||Number of execution nodes|
|EC2 Key Pair Name||AWS Key Pair used to log onto server instances|
|Security Group ID||The Group ID of the Security Group to use|
|IAM Role for S3 Access||IAM Role used to access S3 buckets for Distributed Storage (optional)|
|Use Clustered Placement||Whether or not to use placement groups which locates nodes closer together. It is recommended to enable this option but can take longer for AWS to identify resources with larger engines|
|Enable Auto Start||Automatically start the engine when new queries are submitted to the engine|
|Enable Auto Stop||Automatically stop the engine after 5 minutes of inactivity|
|AWS API Authentication Mode||The authentication mode used to provision EC2 nodes and stop EC2 nodes. Auto uses an IAM Role to management execution nodes. Alternatively an AWS Access and Secret Key can be used|
|Access Key||The AWS Access Key if using Key/Secret authentication|
|Secret||The AWS Secret Key if using Key/Secret authentication|
|IAM Role for API Operations||The IAM Role to assume to provision and manage the engine nodes, by default the IAM Role of the coordinator node is used.|
|Extra Dremio Configuration Properties||Additional configuration options|
Additional behaviors to be aware of: