AWS Edition Engine REST API
Dremio administrators use the AWS Edition REST API to create, update, and delete engines for Dremio AWS Edition. Before performing any of the following procedures, first generate an API token as described in Authentication and open a Dremio project.
The engine endpoints described here use Dremio's v2 API, so apiv2 replaces api/v3 in their URLs.
Listing Engines for a Project
Use the GET http://{DREMIO_ORIGIN}/apiv2/provision/clusters endpoint to retrieve a list of all engines in a Dremio project, as shown in the following example:
curl -X GET 'http://18.237.234.109:9047/apiv2/provision/clusters' -H 'Accept: application/json' -H 'Content-Type:application/json' -H 'Authorization:_dremioapikey'
Dremio returns a json document in the HTTP response body after a successful request:
{
"clusterList":[
{
"id":"82a37f06-ec34-4eaf-b30c-354837795b0c",
"tag":"ryFMjQ2KjjA=",
"currentState":"STOPPED",
"stateChangeTime":1611183365149,
"clusterType":"EC2",
"name":"default",
"dynamicConfig":{
"containerCount":2
},
"containers":{
"runningList":[
],
"decommissioningCount":0,
"provisioningCount":0,
"pendingCount":0,
"disconnectedList":[
],
"pendingList":[
],
"decommissioningList":[
]
},
"desiredState":"STOPPED",
"awsProps":{
"vpc":"vpc-007bd9db0278fdf7f",
"subnetId":"my-subnet",
"useClusterPlacementGroup":true,
"disablePublicIp":false,
"sshKeyName":"mykey",
"instanceType":"m5d.8xlarge",
"connectionProps":{
"authMode":"AUTO",
"region":"us-west-2"
},
"awsTags":[
]
},
"shutdownInterval":7200000,
"isAllowAutoStart":true,
"isAllowAutoStop":true
}
]
}
Retrieving Information about an Engine
Use the GET http://{DREMIO_ORIGIN}/apiv2/provision/cluster/{CLUSTER-ID} endpoint to retrieve detailed information about a specific engine, as shown in the following example:
curl -X GET 'http://18.237.234.109:9047/apiv2/provision/cluster/82a37f06-ec34-4eaf-b30c-354837795b0c'
-H 'Accept: application/json' -H 'Content-Type:application/json' -H 'Authorization:_dremioapikey'
Dremio returns a json document in the HTTP response body after a successful request:
{
"id":"82a37f06-ec34-4eaf-b30c-354837795b0c",
"tag":"ryFMjQ2KjjA=",
"currentState":"STOPPED",
"stateChangeTime":1611183365149,
"clusterType":"EC2",
"name":"default",
"dynamicConfig":{
"containerCount":2
},
"containers":{
"runningList":[
],
"decommissioningCount":0,
"provisioningCount":0,
"pendingCount":0,
"disconnectedList":[
],
"pendingList":[
],
"decommissioningList":[
]
},
"desiredState":"STOPPED",
"awsProps":{
"vpc":"vpc-007bd9db0278fdf7f",
"subnetId":"subnet-09daa36b480c4d9b7",
"useClusterPlacementGroup":true,
"disablePublicIp":false,
"sshKeyName":"my-key",
"instanceType":"m5d.8xlarge",
"connectionProps":{
"authMode":"AUTO",
"region":"us-west-2"
},
"awsTags":[
]
},
"shutdownInterval":7200000,
"isAllowAutoStart":true,
"isAllowAutoStop":true
}
Retrieving AWS Engine Defaults
Use the GET http://{DREMIO_ORIGIN}:9047/apiv2/provision/aws/defaults endpoint to retreive the default values for an engine. Many of these values are required in the HTTP request body for creating and updating engines.
curl -X GET 'http://18.237.234.109:9047/apiv2/provision/aws/defaults
-H Accept: application/json' -H 'Authorization:_dremioapikey'
Dremio returns a json document such as the following in the HTTP response body:
{
"vpc": "my-vpc",
"subnetId": "my-subnet",
"useClusterPlacementGroup": true,
"disablePublicIp": false,
"sshKeyName": "my-key",
"instanceType": "m5d.8xlarge",
"connectionProps": {
"authMode": "AUTO",
"region": "us-west-2"
},
"awsTags": []
Creating an Engine
Use the POST http://{DREMIO_ORIGIN}/apiv2/provision/cluster/{CLUSTER-ID} endpoint to create an engine.
curl -X POST 'http://18.237.234.109:9047/apiv2/provision/cluster'
-H 'Accept: application/json' -H 'Content-Type:application/json'
-H 'Authorization:_dremioapikey' --data '
{
"clusterType": "EC2",
"name": "name",
"isAllowAutoStop": true,
"isAllowAutoStart": true,
"shutdownInterval": 600000,
"dynamicConfig": {
"containerCount": 1
},
"awsProps": {
"vpc": "myvpc",
"subnetId": "mysubnet",
"useClusterPlacementGroup": true,
"disablePublicIp": false,
"sshKeyName": "my-key",
"instanceType": "m5d.2xlarge",
"connectionProps": {
"authMode": "AUTO",
"region": "us-west-2"
}
}
}'
Dremio returns a json document in the HTTP response body after a successful request:
{
"id": "8872b8b2-1f68-44b9-84ce-9c50e06431e0",
"tag": "VBuslFKCxtg=",
"currentState": "STARTING",
"stateChangeTime": 1611968046413,
"clusterType": "EC2",
"name": "name",
"dynamicConfig": {
"containerCount": 1
},
"containers": {
"runningList": [],
"decommissioningCount": 0,
"provisioningCount": 0,
"pendingCount": 1,
"disconnectedList": [],
"pendingList": [
{
"containerId": "i-02d94fb6b2cde3cf8",
"containerPropertyList": [
{
"key": "instanceId",
"value": "i-02d94fb6b2cde3cf8"
}
]
}
],
"decommissioningList": []
},
"desiredState": "RUNNING",
"awsProps": {
"vpc": "vpc-849000e1",
"subnetId": "subnet-f8d476a1",
"useClusterPlacementGroup": true,
"disablePublicIp": false,
"sshKeyName": "my-key",
"instanceType": "m5d.2xlarge",
"connectionProps": {
"authMode": "AUTO",
"region": "us-west-2"
},
"awsTags": []
},
"shutdownInterval": 600000,
"isAllowAutoStart": true,
"isAllowAutoStop": true
}
Updating Engine Configuration
Use the PUT http://{DREMIO_ORIGIN}:9047/apiv2/provision/cluster/{CLUSTER-ID} endpoint to update the configuration of an engine in a cluster. The request must contain the following HTTP request body, as shown in the following example. The request body must contain all of the following parameters.
{
"id":"{CLUSTER-ID}",
"tag":"TAG",
"desiredState":"[RUNNING,STOPPED]",
"clusterType":"EC2",
"dynamicConfig":{
"containerCount":"{CONTAINER-COUNT}"
},
"awsProps":{
"vpc":"{VPC-ID}",
"subnetId":"{SUBNET-ID}",
"sshKeyName":"SSH-KEYNAME",
"connectionProps":{
"authMode":"AUTO"
}
}
}
The following example updates the shutdown interval from 600000 to 120000:
curl -X PUT 'http://{DREMIO_ORIGIN:9047/apiv2/provision/cluster/{CLUSTER-ID}' -H 'Accept: application/json' -H 'Content-Type:application/json' -H 'Authorization:_dremio97g0qje3cgjt8vmi77ri6lwtk8' --data '
{
"id": "cc76d7ae-5476-44b8-8316-26743327662g",
"tag": "ih6o2hxO2AI=",
"currentState": "STOPPED",
"stateChangeTime": 1611691702661,
"clusterType": "EC2",
"name": "js",
"dynamicConfig": {
"containerCount": 1
},
"containers": {
"runningList": [],
"decommissioningCount": 0,
"provisioningCount": 0,
"pendingCount": 0,
"disconnectedList": [],
"pendingList": [],
"decommissioningList": []
},
"desiredState": "STOPPED",
"awsProps": {
"vpc": "my-vpc",
"subnetId": "my-subnet",
"useClusterPlacementGroup": true,
"disablePublicIp": false,
"sshKeyName": "my-key",
"instanceType": "m5d.8xlarge",
"connectionProps": {
"authMode": "AUTO",
"region": "us-west-2"
},
"awsTags": []
},
"shutdownInterval": 120000,
"isAllowAutoStart": true,
"isAllowAutoStop": true
}'
Dremio returns a json document in the HTTP response body after a successful request:
{
"id": "cc76d7ae-5476-44b8-8316-267433276562g",
"tag": "7IjCt36qTgo=",
"currentState": "STARTING",
"stateChangeTime": 1611711836819,
"clusterType": "EC2",
"name": "js",
"dynamicConfig": {
"containerCount": 1
},
"containers": {
"runningList": [],
"decommissioningCount": 0,
"provisioningCount": 0,
"pendingCount": 1,
"disconnectedList": [],
"pendingList": [
{
"containerId": "i-0df3e31ca87cc31b6",
"containerPropertyList": [
{
"key": "instanceId",
"value": "i-0df3e31ca87cc31b6"
}
]
}
],
"decommissioningList": []
},
"desiredState": "RUNNING",
"awsProps": {
"vpc": "vpc-007bd9db0278fdf7",
"subnetId": "my-subnet",
"useClusterPlacementGroup": true,
"disablePublicIp": false,
"sshKeyName": "my-key",
"instanceType": "m5d.8xlarge",
"connectionProps": {
"authMode": "AUTO",
"region": "us-west-2"
},
"awsTags": []
},
"shutdownInterval": 120000,
"isAllowAutoStart": true,
"isAllowAutoStop": true
}
Starting and Stopping an Engine
Use the PUT http://{DREMIO_ORIGIN}:9047/apiv2/provision/cluster/{CLUSTER-ID} endpoint to stop and restart an engine. To stop an engine, the HTTP request body must set the value of desiredState to STOPPED. To restart the engine, set the value of desiredState to RUNNING, as shown in the following example:
curl -X PUT 'http://{DREMIO_ORIGIN}/apiv2/provision/cluster/93a37f06-ec34-4eaf-b30c-354837795b0c'
-H 'Accept: application/json" -H 'Content-Type:application/json'
-H 'Authorization:_dremio{API-TOKEN}' --data '
{
"id":"93a37f06-ec34-4eaf-b30c-354837795c1d",
"tag":"mytag",
"desiredState":"RUNNING",
"clusterType":"EC2",
"dynamicConfig":{
"containerCount":2
},
"awsProps":{
"vpc":"my-vpc",
"subnetId":"my-subnet",
"sshKeyName":"my-key",
"connectionProps":{
"authMode":"AUTO"
}
}
}
'
Dremio returns a json document with the following structure after a successful request:
{
"id": "02cc318f-7807-4466-88c6-0e3355db848f",
"tag": "mcB9qWZPiuc=",
"currentState": "STOPPING",
"stateChangeTime": 1611709681032,
"clusterType": "EC2",
"name": "default",
"dynamicConfig": {
"containerCount": 1
},
"containers": {
"runningList": [
{
"containerId": "i-0172dd52ce7c9db95",
"containerPropertyList": [
{
"key": "instanceId",
"value": "i-0172dd52ce7c9db95"
},
{
"key": "host",
"value": "ip-174-57-188-194.us-west-2.compute.internal"
},
{
"key": "privateIp",
"value": "174-57-188-194"
},
{
"key": "publicIp",
"value": "43.322.63.57"
},
{
"key": "launchTime",
"value": "2021-01-27T00:08:16Z"
}
]
}
],
"decommissioningCount": 0,
"provisioningCount": 0,
"pendingCount": 0,
"disconnectedList": [],
"pendingList": [],
"decommissioningList": []
},
"desiredState": "RUNNING",
"awsProps": {
"vpc": "my-vpc",
"subnetId": "subnet-09daa36b480c4d9b7",
"useClusterPlacementGroup": true,
"disablePublicIp": false,
"sshKeyName": "my-key",
"instanceType": "m5d.2xlarge",
"connectionProps": {
"authMode": "AUTO",
"region": "us-west-2"
},
"awsTags": []
},
"shutdownInterval": 300000,
"isAllowAutoStart": true,
"isAllowAutoStop": true
}
Deleting an Engine
Use the DELETE 'http://{DREMIO_ORIGIN}/apiv2/provision/cluster/{CLUSTER-ID} endpoint with no HTTP request body to delete an engine, as shown in the following example:
curl -X DELETE 'http://{DREMIO_ORIGIN}/apiv2/provision/cluster/93a37f06-ec34-4eaf-b30c-354837795b0c'
Dremio returns 1 after a successful request.