On this page

    Managing Engines

    An engine represents a Dremio Cloud entity that manages compute resources. All the engines in a project run in the cloud account associated with the project. Each engine has one or more replicas that are created for executing queries. An engine replica is a logical group of executors defined by the engine capacity. Executors are AWS EC2 instances.

    An engine can be in one of the following states.

    StateIconDescription
    RunningRepresents an enabled engine (replicas are provisioned automatically or running as per the minimum number of replicas configured). You can use this engine for running queries.
    Adding ReplicaRepresents an engine that is scaling up (adding a replica).
    Removing ReplicaRepresents an engine that is scaling down (removing a replica).
    DisablingRepresents an engine that is being disabled.
    DisabledRepresents a disabled engine (no engine replicas have been provisioned dynamically or there are no active replicas). You cannot use this engine for running queries.
    Starting EngineRepresents an engine that is starting (transitioning from the disabled state to the enabled state).
    Stopping EngineRepresents an engine that is stopping (transitioning from the enabled state to the disabled state).
    StoppedRepresents an enabled engine that has been stopped (zero replicas running).
    DeletingRepresents an engine that is being deleted.

    When you sign up as a user, Dremio Cloud provides a organization with a project and cloud for a given Dremio Cloud account. A newly created project has a default engine and a preview engine enabled. Every engine created in a project will be created in the cloud account associated with that project. The following figure illustrates the Dremio Cloud control plane, the default engine’s replica, and the preview engine.

    warning:

    The default engine gets created when your project is created but it does not start up until the first query is executed. The default engine requires active AWS instances to start up. The default configuration of the default engine does not have active AWS instances. So, when you run a query, active AWS instances run and simultaneously, the engine starts up.

    The default engine starts and stops based on the query load. The preview engine continuously runs, and provides query previews. Preview queries that the user and the system triggers are routed to the preview engine, which is created by default when a project is created and runs continuously.

    Each engine has a size that is associated with a fixed number of nodes. These nodes are the AWS instances and are also called executors. An engine has a concurrency limit that is the number of parallel queries that it can execute.

    The different configurable engine sizes mapped to AWS instance types are as listed below.

    SizeNumber of NodesAWS Instance TypeDCUs per Hour of Engine Uptime
    XX Small (default size)1m5d.4xlarge4
    XSmall1m5d.8xlarge8
    Small2m5d.8xlarge16
    Medium4m5d.8xlarge32
    Large8m5d.8xlarge64
    X Large16m5d.8xlarge128
    2X Large32m5d.8xlarge256
    3X Large64m5d.8xlarge512

    Viewing All Engines

    To view engines:

    1. In the Dremio Cloud application, click the Settings (gear) icon that is towards the bottom of the left sidebar. Click Project Settings from the menu.
    2. Click Engines from the sidebar menu to see the list of engines in the project. On the Engines page, you can also see engines as per the status. Click the Status dropdown list to see the different statuses.

    Adding an Engine

    Perform the following steps to add a new engine:

    1. In the Project Settings page, click Engines in the sidebar menu. The Engines page lists the engines created for the project. Every engine created in a project is created in the cloud account associated with that project.
    2. Click the Add Engine button on the top-right of the Engines page to create a new engine.
    3. In the Add Engine dialog, for Engine, enter a name.
    4. (Optional) For Description, enter a description.
    5. (Optional) For Size, select the size of the engine. The size designates the number and type of AWS instances that make up the engine. The default size is XXSmall (1 m5d.4xlarge node).
    6. (Optional) For Max Concurrency per Replica, enter the maximum number of jobs that can be run concurrently on this engine. The default maximum concurrency is 2 per replica for XXSmall (1 m5d.4xlarge node). Each engine size has a default maximum concurrency per replica.

    The following parameters are for Engine Replicas:

    1. For Min Replicas, enter the minimum number of engine replicas that Dremio Cloud has running at any given time. For auto-stop, set it to 0. To guarantee low-latency query execution, set it to 1 or higher. The default number of minimum replicas is 0.
    2. For Max Replicas, enter the maximum number of engine replicas that Dremio Cloud scales up to. The default number of maximum replicas is 1.

    note:

    You can use this setting to control costs and ensure that excessive replicas are not spun up.

    1. Click Advanced Configuration.
    2. For Last Replica Auto-Stop (secs), enter the number of seconds to wait before deleting the last replica if engine is not in use. The default value is 7200 seconds, and the minimum value is 60 seconds.

    warning:

    The last replica auto stop is not valid when the minimum number of engine replicas is 1 or higher. No replicas are run when an engine is not in use.

    The following parameters are for Time Limit (secs):

    1. For Enable Enqueued Time Limit, check the box.
    2. For Enqueued Time Limit, enter the time (in seconds) a query waits in the engine’s queue before being cancelled. The default value is 5 minutes.

    warning:

    You should not set it to less than two minutes (120 seconds), which is the typical time to start a new replica. Changing this setting does not affect queries that are currently running or in the queue.

    1. (Optional) For Enable Query Time Limit, check the box to enable the query time limit for making a query run before it is canceled.

    2. (Optional) For Query Runtime Limit, enter the time (in seconds) a query can run before it is canceled. The default query runtime limit is 5 minutes.

    3. For Drain Time Limit, enter the time (in seconds) that an engine replica continues to run after the engine is resized, disabled, or deleted before it is terminated and the running queries fail. The default value is 1800 seconds. If there are no queries running on a replica, the engine is terminated without waiting for the drain time limit.

    4. (Optional) For AWS Tags, enter the AWS tags for this engine. The parameters are key-value pairs. You can use these tags to identify or query EC2 instances associated with the engine in AWS.

      Click Add Tag. The Name and Value text boxes appear. Add a tag name and a corresponding value. For example, For example, add Engine1 as the key and Europe as the value.

    5. Click Save and Launch. This action saves the configuration, enables this engine, and provisions AWS EC2 instances.

    Editing an Engine

    Perform the steps below to edit an engine:

    1. In the Project Settings page, click the Engines from the sidebar menu.

    2. On the Engines page, hover on the row of the engine that you want to edit and click on the edit (pencil) button. The Edit Engine dialog is opened.

      Alternatively, you can click the engine to go to the engine’s page. Click the Edit Engine button on the top-right of the page.

    note:

    You cannot edit the Engine name parameter.

    1. For Description, enter a description.
    2. For Size, select the size based on the number and type of instances that you want the engine to have.
    3. For Max Concurrency per Replica, enter the maximum number of jobs that can be run concurrently on this engine. The default maximum concurrency is 2 per replica for XX Small (1 m5d.4xlarge node). Each engine size has a default maximum concurrency per replica.

    The following parameters are for Engine Replicas:

    1. For Min Replicas, enter the number of engine replicas that Dremio Cloud has running at any given time. For auto-stop, set it to 0. To guarantee low-latency query execution, set it to 1 or higher. The default number of minimum replicas is 0.
    2. For Max Replicas, enter the maximum number of engine replicas that Dremio Cloud scales up to. The default number of maximum replicas is 1.

    note:

    You can use this setting to control costs and ensure that excessive replicas are not spun up.

    1. Click Advanced Configuration.
    2. For Last Replica Auto-Stop (secs), enter the number of seconds to wait before deleting the last replica if the engine is not in use. The default value is 7200 seconds.

    warning:

    The last replica auto stop is not valid when the minimum number of engine replicas is 1 or higher. No replicas are run when an engine is not in use.

    The following parameters are for Time Limit (secs):

    1. For Enable Enqueued Time Limit, check the box.
    2. For Enqueued Time Limit, enter the time (in seconds) a query waits in the engine’s queue before being canceled. The default value is 5 minutes.

    warning:

    You should not set it to less than two minutes (120 seconds), which is the typical time to start a new replica. Changing this setting does not affect queries that are currently running or in the queue.

    1. (Optional) For Enable Query Time Limit, check the box to enable the query time limit for making a query run before it is canceled.

    2. (Optional) For Query Runtime Limit, enter the time (in seconds) a query can run before it is canceled. The default query runtime limit is 5 minutes.

    3. For Drain Time Limit, enter the time (in seconds) that an engine replica continues to run after the engine is resized, disabled, or deleted before it is terminated and the running queries fail. The default value is 1800 seconds. If there are no queries running on a replica, the engine is terminated without waiting for the drain time limit.

    4. For AWS Tags, enter the AWS tags for this engine (the parameters are key-value pairs). You can use these tags to identify or query EC2 instances associated with the engine in AWS. You can change the existing tags and values.

      To add a new tag, click Add Tag. The Name and Value text boxes appear. Add a tag name and a corresponding value.

    5. Click Save.

    Disabling an Engine

    You can disable an engine that is not being used:

    Perform the steps below to disable the engine:

    1. On the Project Settings page, click Engines in the sidebar menu. The list of engines in this project are displayed.
    2. Disable the engine by using the toggle in the Enabled column.
    3. Confirm that you want to disable the engine.

    Enabling an Engine

    Perform the steps below to enable a disabled engine:

    1. On the Project Settings page, click Engines in the sidebar menu. The list of engines in this project are displayed.
    2. Enable the engine by using the toggle in the Enabled column.
    3. Confirm that you want to enable the engine.

    Deleting an Engine

    You can permanently delete an engine if it is not in use (this action is irreversible). If queries are running on the engine, then Dremio Cloud waits for the drain-time-limit seconds for the running queries to complete before deleting the engine.

    warning:

    An engine that has a routing rule associated with it cannot be deleted. Delete the rules before deleting the engine.

    Perform the steps below to delete an engine:

    1. On the Project Settings page, click Engines in the sidebar menu. The list of engines in this project are displayed.
    2. On the Engines page, hover on the row of the engine that you want to delete and click on the delete (trash can) button.
    3. Confirm that you want to delete the engine.