Managing Engines in Kubernetes

Dremio supports the ability to provision multiple separate execution engines in Kubernetes from a Dremio master-coordinator node, and automatically start and stop based on workload requirements at runtime. This provides several benefits, including:

• Creating a new engine does not require restarting Dremio, which enables administrators to achieve workload isolation efficiently.
• When creating a new engine, you can use Kubernetes metadata to label engines to keep track of resources.
• Right-size execution resources for each distinct workload, instead of implementing a one-size-fits-all model.
• Easily experiment with different execution resource sizes at any scale.

To manage your engines, open the Engines page as follows:

1. Open your Dremio console.
2. Click in the side navigation bar to open the Settings sidebar.
3. Select Engines.

Monitoring Engines

You can monitor the status and properties of your engines on the Engines page.

Each engine has the following information available:

• Name - The name of the engine, which you can click to see its details. See the section about Viewing Engine Details.
• Size - The size configured for the engine.
• Status - The engine status. For more information, see the section in this topic about Engine Statuses.
• Auto start/stop - Whether the engine has auto start/stop enabled for autoscaling.
• Idle period - If auto start/stop is enabled, the idle time to auto stop.
• Queues - Query queues routed to the engine.
• Labels - Labels associated with the engine.

Performing Actions on Engines

While monitoring engines, you have actions you can perform on each engine through the icons displayed on the right-hand side when hovering over the engine row.

Stopping/Starting an Engine

You can click / to stop/start an engine manually at any time. Stopping an engine will cause running queries to fail while new queries will remain queued, which can also fail by timeout if the engine is not started. To prevent query failures, reroute queries to another engine, and stop the engine only when no queries are running or queued for the engine.

note

You can enable autoscaling on an engine to make it stop automatically after an idle time without queries and start again automatically when new queries are issued, all without any human intervention.

Autoscaling is configured when you add an engine or edit an engine:

Editing the Engine Settings

You can click to edit the engine settings. After saving the new settings, the engine may restart, causing running queries to fail and new queries to be queued.

note

The name of the engine must follow these rules:

• Must start with a lowercase alphanumeric character ([a-z0-9]).
• Must end with a lowercase alphanumeric character ([a-z0-9]).
• Must contain only lowercase alphanumeric characters or a hyphen ([\-a-z0-9]).
• The maximum length is 30 characters.

Deleting an Engine

You can click to delete an engine. Deleting an engine will cause running, queued, and new queries to fail. To prevent query failures, you can reroute queries to another engine, and only delete when no more queries are running or queued for the engine.

Viewing Engine Details

While monitoring engines, if you need to know more details about engines, click the engine's name to view all the information about it.

On this page, you will also find a set of buttons at the top to delete the engine, stop/start the engine, and edit the engine settings.

Adding an Engine

You can create more engines by clicking Add Engine at the top-right corner of the Engines page.

In the New engine dialog, do the following:

1. Fill out the General section:

   1. Name - Type the name of the engine. Use a meaningful name that helps you to identify the engine better. For example, low-cost-query.

      note

      The name of the engine must follow these rules:

      • Must start with a lowercase alphanumeric character ([a-z0-9]).
      • Must end with a lowercase alphanumeric character ([a-z0-9]).
      • Must contain only lowercase alphanumeric characters or a hyphen ([\-a-z0-9]).
      • The maximum length is 30 characters.

   2. CPU - Select the number of CPU cores for the node.

   3. Size - Select the size as the number of executor pods per replica and memory per pod.

   4. Automatically start/stop - If checked, the engine automatically stops after the specified idle time and automatically starts when new queries are issued to the engine. If not checked, the engine only stops and starts through manual intervention. By default, this setting is checked and the engine stops automatically after 2 hours of idle time. For more information, see the section about Customizing Engine Autoscaling.

   5. (Optional) Expand Advanced Options for further settings.

      

      Fill out the advanced options as follows:

      1. Cloud cache volume (c3) - Specify the amount of local storage for caching data.
      2. Spill volume - Specify the disk size allocated for temporary storage when operations exceed memory limits.

2. (Optional) Select Kubernetes pod metadata to define pod metadata for the engine, such as labels, annotations, node selectors, and tolerations. Define those values with care and foreknowledge of expected entries because any misconfiguration may result in Kubernetes being unable to start the executors that make up the engine.

   

   Fill out the pod's metadata with:

   1. Labels - Add labels as key/value pairs to identify and organize pods. Use them to group, filter, and select subsets of resources efficiently.

      note

      The engine label must follow these rules:

      • Must start with an alphanumeric character ([a-z0-9]).
      • Must end with an alphanumeric character ([a-z0-9]).
      • Must contain only lowercase alphanumeric characters, a hyphen, or a underscore ([-_a-z0-9]).
      • The maximum length is 63 characters.

   2. Annotations - Add annotations as key/value pairs to store non-identifying metadata, such as build information or pointers to logging services. Unlike labels, they are not used for selection or grouping.

      note

      The engine annotation must follow these rules:

      • Must be UTF-8 encoded and can include any valid UTF-8 character.
      • Can be in plain text, JSON, or any other UTF-8 compatible format.
      • The maximum size is 256KB.
      • The maximum size of all engine annotations is 1MB.

   3. Node selectors - Add node selectors as key/value pairs for node-specific constraints to schedule pods on nodes matching specified labels. Use this to target nodes with specific configurations or roles.

   4. Tolerations - Add tolerations to allow pods to be scheduled on nodes with matching taints, but they don’t restrict scheduling to only those nodes; the pod can still land on a node without the taint.

3. Click Add to add the engine.

The newly added engine will be displayed in the listed engines.

Engine Statuses

The following table describes each engine status:

Status	Icon	Description
Starting		The engine is starting. This is the initial state of an engine after being created. New queries are queued to be processed.
Running		The engine is running. New queries are queued and processed.
Stopping		The engine is stopping. Running queries will fail. New queries will remain queued, which can also fail by timeout if the engine is not started.
Stopped		The engine is stopped. New queries will remain queued, which can fail by timeout if the engine is not started.
Recovering		The engine is recovering. New queries will remain queued, which can fail by timeout if the engine does not recover.
Failed		The engine failed. New queries will remain queued, which can fail by timeout if the engine does not start.

See Also

• Engine Management API - Manage your engines using Dremio's Engine Management API.