Skip to main content

Schedules

Dremio Arctic enables you to schedule optimization jobs to help you manage the accumulation of the data files that occurs through DML operations. This API allows you to add, modify, retrieve, list, and delete schedules.

Schedules Object
{
  "type": "OPTIMIZE",
  "catalogId": "1a234bc5-6789-46d7-b9cb-d236e195b34a",
  "id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
  "schedule": "@daily",
  "username": "dremio_user@company.com",
  "config": {
    "tableId": "sample-nyc-taxi-data",
    "reference": "main",
    "maxFileSize": "460 MB",
    "minFileSize": "192 MB",
    "minFiles": 5,
    "targetFileSize": "256 MB"
  }
}

Schedule Attributes​

type String

The type of job that is run by the schedule.

Enum: OPTIMIZE, VACUUM

Example: OPTIMIZE

catalogId String (UUID)

Unique identifier for the catalog where the schedule ran.

Example: 1a234bc5-6789-46d7-b9cb-d236e195b34a

id String (UUID)

Unique identifier for the schedule.

Example: 123ab456-cdef-78ab-c901-23de4f5ab6c7

schedule String

The interval and time at which Dremio runs the optimization job. Can be a preset option or a supported cron expression. The cron expressions that are supported include:

  • Dash-separated ranges
  • Comma-separated lists
  • Quotients
  • Individual field values for minutes, hours, day of month, month, and day of week
  • JAN-DEC, SUN-SAT, and their equivalent numeric values

The enum lists the preset options. The example shows a cron expression to run the job on the first day of every month. For information about cron expressions, see cron.

Enum: @once, @hourly, @daily, @weekly, @monthly

Example: 0 0 1 * *

username String

The user who created the schedule.

Example: dremio_user@company.com

config Object

For OPTIMIZE job types, the configuration options for the specified schedule. Not included for VACUUM job types.

Attributes of the config Object​

reference String

Identifies the branch, tag, or commit where the table containing the optimization schedule is located.

Example: main

tableId String

The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator.

Example: sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips

maxFileSize String

The maximum file size threshold for type OPTIMIZE. Files that are larger in size than the specified maxFileSize qualify for optimization. The integer must be non-negative. The default value is 460 MB.

Example: 460 MB

minFileSize String

The minimum file size threshold for type OPTIMIZE. Files that are smaller in size than the specified minFileSize qualify for optimization. The integer must be non-negative. The default value is 192 MB.

Example: 192 MB

minFiles String

The minimum number of qualified files required to run a table optimization job. Supports only type OPTIMIZE. The default is 5 files.

Example: 5

targetFileSize String

Controls the target size of the files that are generated for type OPTIMIZE. The integer must be non-negative. The default file size is 256 MB.

Example: 256 MB

Listing All Schedules​

List all the job schedules available for the specified Arctic catalog.

Method and URL
GET /v0/arctic/catalogs/{catalogId}/schedules

Parameters​

catalogId Path   String (UUID)

Unique identifier of the catalog you want to list all job schedules for.

Example: 1a234bc5-6789-46d7-b9cb-d236e195b34a

filter Query   Object   Optional

A common expression language (CEL) expression that filters responses so that they include only results with the specified attributes and values. Value is a URL-encoded string that represents a JSON object. The JSON object specifies the attributes to filter on and the values to match for each attribute. Read filter Query Parameter for usage examples.

maxResults Query   Integer   Optional

The maximum number of results to to return in the response. The server may return fewer results, but not more. The default is 10. Read maxResults Query Parameter for usage examples.

pageToken Query   String   Optional

Token for retrieving the next page of jobs. If the Dremio instance has more jobs than the maximum per page (default 10), the response will include a nextPageToken after the data array. Use the nextPageToken value in your request URL as the pageToken value. Do not change any other query parameters included in the request URL when you use pageToken. Read pageToken Query Parameter for usage examples.

Attributes of the filter Object​

type Body   String

The type of job to be run by the schedule.

Enum: OPTIMIZE, VACUUM

Example: OPTIMIZE

tableId Body   String

The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator. If a tableId is used, then the type is OPTIMIZE.

Example: sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips

reference Body   String

Identifies the branch, tag, or commit where the table containing the optimization schedule is located. If a reference is used, then the type is OPTIMIZE.

Example: main

user Body   String

The user who created the schedule.

Example: dremio_user@company.com

Example Request
curl -X GET 'https://api.dremio.cloud/v0/arctic/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
Example Response
[
  {
    "data": [
    {
      "type": "OPTIMIZE",
      "id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
      "schedule": "@weekly",
      "username": "dremio_user@company.com",
      "config": {
        "maxFileSize": "460 MB",
        "minFileSize": "192 MB",
        "minFiles": 5,
        "reference": "etl",
        "tableId": "sample-NYC-taxi-trips",
        "targetFileSize": "256 MB"
      }
    },
    {
      "type": "VACUUM",
      "id": "d970e95e-49fe-4324-5748-7d153e25ad85",
      "schedule": "0 0 * * *",
      "username": "dremio_user@company.com"
    },
    {
      "type": "OPTIMIZE",
      "id": "abc123d4-ab12-345cd-1234-abc123def456",
      "schedule": "@monthly",      
      "username": "dremio_user@company.com",
      "config": {
        "maxFileSize": "920 MB",
        "minFileSize": "384 MB",
        "minFiles": 5,
        "reference": "dev",
        "tableId": "sample-SF-taxi-trips",
        "targetFileSize": "512 MB"
      }
    }
   ]
  }
]

Responses​

200   OK

400   Bad Request

404   Not Found

415   Unauthorized

500   Internal Server Error

Creating a Schedule​

Creates a table optimization (OPTIMIZE-type) or table cleanup (VACUUM-type) job schedule for the specified Arctic catalog.

Method and URL
POST /v0/arctic/catalogs/{catalogId}/schedules

Parameters​

catalogId Path   String (UUID)

Unique identifier of the catalog you want to add a job schedule for.

Example: 1a234bc5-6789-46d7-b9cb-d236e195b34a

type Body   String

The type of job that is run by the schedule.

Enum: OPTIMIZE, VACUUM

Example: OPTIMIZE

schedule Body   String

The interval and time at which Dremio should run the optimization job. Can be a preset option or a supported cron expression. The cron expressions that are supported include:

  • Dash-separated ranges
  • Comma-separated lists
  • Quotients
  • Individual field values for minutes, hours, day of month, month, and day of week
  • JAN-DEC, SUN-SAT, and their equivalent numeric values

For Vacuum-type jobs, you can use the schedule parameter to automatically run vacuum jobs at a different interval and time than the defaults, which are once each day and 00:00 UTC. If you disable table cleanup in the Dremio console, Dremio does not retain the customized schedule. If you re-enable table cleanup, you must recreate the customized vacuum schedule.

The enum lists the preset options. The example shows a cron expression to run the job on the first day of every month. For information about cron expressions, see cron.

Enum: @once, @hourly, @daily, @weekly, @monthly

Example: 0 0 1 * *

config Body   Object

Configuration options for the specified schedule. Not used for VACUUM job types.

Parameters of the config Object​

reference Body   String

Identifies the branch, tag, or commit where the table containing the optimization schedule should be created.

Example: main

tableId Body   String

The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator.

Example: sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips

maxFileSize Body   String

The maximum file size threshold for type OPTIMIZE. Files that are larger in size than the specified maxFileSize qualify for optimization. The integer must be non-negative. The default value is 460 MB.

Example: 460 MB

minFileSize Body   String

The minimum file size threshold for type OPTIMIZE. Files that are smaller in size than the specified minFileSize qualify for optimization. The integer must be non-negative. The default value is 192 MB.

Example: 192 MB

minFiles Body   String

The minimum number of qualified files required to run a table optimization job. Supports only the type OPTIMIZE. The default is 5 files.

Example: 5

targetFileSize Body   String

Controls the target size of the files that are generated for type OPTIMIZE. The integer must be non-negative. The default file size is 256 MB.

Example: 70 MB

The following example request creates a weekly schedule for OPTIMIZE jobs.

Example Request for OPTIMIZE-Type Jobs
curl -X POST 'https://api.dremio.cloud/v0/arctic/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
    "type": "OPTIMIZE",
    "schedule": "@weekly",
    "config": {
       "reference": "etl",
       "tableId": "sample-NYC-taxi-trips",
       "maxFileSize": "460 MB",
       "minFileSize": "192 MB",
       "minFiles": 5,
       "targetFileSize": "70 MB"
      }
    }'
Example Response for OPTIMIZE-Type Jobs
{
  "type": "OPTIMIZE",
  "id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
  "schedule": "@weekly",
  "username": "dremio_user@company.com",
  "config": {
     "maxFileSize": "460 MB",
     "minFileSize": "192 MB",
     "minFiles": 5,
     "targetFileSize": "70 MB"
  }
}

The following example creates a custom schedule to automatically run VACUUM CATALOG on the Arctic catalog every 7 days at 06:00 UTC.

Example Request to Create a Custom Schedule for VACUUM-Type Jobs
curl -X POST 'https://api.dremio.cloud/v0/arctic/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
    "type": "VACUUM",
    "schedule": "0 6 1,8,16,24,31 * *"
}
Example Response for VACUUM-Type Jobs
{
  "type": "VACUUM",
  "id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
  "schedule": "0 6 1,8,16,24,31 * *",
  "username": "dremio_user@company.com"
}

Responses​

200   OK

400   Bad Request

404   Not Found

415   Unauthorized

500   Internal Server Error

Retrieving a Schedule​

Retrieve the specified schedule.

Method and URL
GET /v0/arctic/catalogs/{catalogId}/schedules/{scheduleId}

Parameters​

catalogId Path   String (UUID)

Unique identifier of the catalog you want to list all job schedules for.

Example: 1a234bc5-6789-46d7-b9cb-d236e195b34a

scheduleId Path   String (UUID)

Unique identifier of the schedule you want to retrieve.

Example: 9876a54b-681b-49fe-afff-0d753ed5ad85

Example Request
curl -X GET 'https://api.dremio.cloud/v0/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules/9876a54b-681b-49fe-afff-0d753ed5ad85' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
Example Response
{
  "type": "OPTIMIZE",
  "id": "673cd876-abcd-1234-b478-91ca5e9cc8f6",
  "schedule": "@daily",
  "username": "analyst@company.com",
  "config": {
    "reference": "dev",
    "tableId": "sales_2022",
    "maxFileSize": "2048 MB",
    "minFileSize": "768 MB",
    "minFiles": 10,
    "targetFileSize": "1024 MB"
    },
}

Responses​

200   OK

404   Not Found

415   Unauthorized

500   Internal Server Error

Modifying a Schedule​

Modify an Arctic optimization or vacuum job schedule.

Method and URL
PUT /v0/arctic/catalogs/{catalogId}/schedules/{scheduleId}

Parameters​

catalogId Path   String (UUID)

Unique identifier of the catalog you want to modify a schedule for.

Example: 1a234bc5-6789-46d7-b9cb-d236e195b34a

scheduleId Path   String (UUID)

Unique identifier of the schedule you want to modify.

Example: 9876a54b-681b-49fe-afff-0d753ed5ad85

type Body   String

The type of job that is run by the schedule.

Enum: OPTIMIZE, VACUUM

Example: OPTIMIZE

schedule Body   String

The interval and time at which Dremio should run the optimization job. Can be a preset option or a supported cron expression. The cron expressions that are supported include:

  • Dash-separated ranges
  • Comma-separated lists
  • Quotients
  • Individual field values for minutes, hours, day of month, month, and day of week
  • JAN-DEC, SUN-SAT, and their equivalent numeric values

The enum lists the preset options. The example shows a cron expression to run the job on the first day of every month. For information about cron expressions, see cron.

Enum: @once, @hourly, @daily, @weekly, @monthly

Example: 0 0 1 * *

config Body   Object

Configuration options for the specified schedule. Not used for VACUUM job types.

Parameters of the config Object​

reference Body   String

Identifies the branch, tag, or commit where the table containing the optimization schedule is located.

Example: main

tableId Body   String

The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator.

Example: sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips

maxFileSize Body   String

The maximum file size threshold for type OPTIMIZE. Files that are larger in size than the specified maxFileSize qualify for optimization. The integer must be non-negative. The default value is 460 MB.

Example: 460 MB

minFileSize Body   String

The minimum file size threshold for type OPTIMIZE. Files that are smaller in size than the specified minFileSize qualify for optimization. The integer must be non-negative. The default value is 192 MB.

Example: 192 MB

minFiles Body   String

The minimum number of qualified files required to run a table optimization job. Supports only the type OPTIMIZE. The default is 5 files.

Example: 5

targetFileSize Body   String

Controls the target size of the files that are generated for type OPTIMIZE. The integer must be non-negative. The default file size is 256 MB.

Example: 70 MB

The following example modifies a weekly schedule to a daily one for OPTIMIZE jobs.

Example Request to Modify a Schedule for OPTIMIZE Jobs
curl -X PUT 'https://api.dremio.cloud/v0/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules/9876a54b-681b-49fe-afff-0d753ed5ad85' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
    "type": "OPTIMIZE",
    "schedule": "@daily",
    "config": {
       "reference": "compare",
       "tableId": "customer-sales",
       "maxFileSize": "460 MB",
       "minFileSize": "192 MB",
       "minFiles": 3,
       "targetFileSize": "70 MB"
      }
    }'
Example Response
{
  "type": "OPTIMIZE",
  "id": "9876a54b-681b-49fe-afff-0d753ed5ad85",
  "schedule": "@daily",
  "username": "marketing-admin@company.com",
  "config": {
     "reference": "compare",
     "tableId": "customer-sales",
     "maxFileSize": "460 MB",
     "minFileSize": "192 MB",
     "minFiles": 3,
     "targetFileSize": "70 MB"
  },
}

The following example modifies a custom schedule for VACUUM jobs so that they run every 7 days at 06:00 UTC to daily at 01:00 UTC.

Example Request to Modify a Custom Schedule for VACUUM Jobs
curl -X POST 'https://api.dremio.cloud/v0/arctic/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
    "type": "VACUUM",
    "schedule": "0 1 * * *"
}
Example Response
{
  "type": "VACUUM",
  "id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
  "schedule": "0 1 * * *",
  "username": "dremio_user@company.com"
}

Responses​

200   OK

400   Bad Request

404   Not Found

415   Unsupported Media Type

500   Internal Server Error

Deleting a Schedule​

Delete an Arctic job schedule.

Method and URL
DELETE /v0/arctic/catalogs/{catalogId}/schedules/{scheduleId}

Parameters​

catalogId Path   String (UUID)

Unique identifier of the catalog you want to delete.

Example: 1a234bc5-6789-46d7-b9cb-d236e195b34a

scheduleId Path   String (UUID)

Unique identifier of the schedule you want to delete.

Example: 9876a54b-681b-49fe-afff-0d753ed5ad85

Example Request
curl -X DELETE 'https://api.dremio.cloud/v0/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules/9876a54b-681b-49fe-afff-0d753ed5ad85' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
Example Response
No response

Responses​

204   No Content

404   Not Found

415   Unsupported Media Type

500   Internal Server Error