Schedules
Dremio Arctic enables you to schedule optimization jobs to help you manage the accumulation of the data files that occurs through DML operations. This API allows you to add, modify, retrieve, list, and delete schedules.
Schedules Object{
"type": "OPTIMIZE",
"catalogId": "1a234bc5-6789-46d7-b9cb-d236e195b34a",
"id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
"schedule": "@daily",
"username": "dremio_user@company.com",
"config": {
"tableId": "sample-nyc-taxi-data",
"reference": "main",
"maxFileSize": "460 MB",
"minFileSize": "192 MB",
"minFiles": 5,
"targetFileSize": "256 MB"
}
}
Schedule Attributes
type
String
The type of job that is run by the schedule. For Arctic optimization jobs, the type is OPTIMIZE
.
Example OPTIMIZE
catalogId
String (UUID)
Unique identifier for the catalog where the schedule ran.
Example 1a234bc5-6789-46d7-b9cb-d236e195b34a
id
String (UUID)
Unique identifier for the schedule.
Example 123ab456-cdef-78ab-c901-23de4f5ab6c7
schedule
String
Identifies a schedule's frequency. Can be a preset option or a supported cron expression. The cron expressions that are supported include:
• Dash-separated ranges
• Comma-separated lists
• Quotients
• Individual field values for minutes, hours, day of month, month, and day of week
• JAN-DEC, SUN-SAT, and their equivalent numeric values
The enum lists the preset options. The example shows a cron expression to run the job on the first day of every month. For information about cron expressions, read cron.
@once, @hourly, @daily, @weekly, @monthly
Example 0 0 1 * *
username
String
The user who created the schedule.
Example dremio_user@company.com
Object
Configuration options for the specified schedule.
config
Object Attributes
reference
String
Identifies the branch, tag, or commit where the table containing the optimization schedule is located.
Example main
tableId
String
The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator.
Example sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips
maxFileSize
String
The maximum file size threshold for type OPTIMIZE
. Files that are larger in size than the specified maxFileSize
qualify for optimization. The integer must be non-negative. The default value is 460 MB.
Example 460 MB
minFileSize
String
The minimum file size threshold for type OPTIMIZE
. Files that are smaller in size than the specified minFileSize
qualify for optimization. The integer must be non-negative. The default value is 192 MB.
Example 192 MB
minFiles
String
The minimum number of qualified files required to run an optimization job. Supports only type OPTIMIZE
. The default is 5 files.
Example 5
targetFileSize
String
Controls the target size of the files that are generated for type OPTIMIZE
. The integer must be non-negative. The default file size is 256 MB.
Example 256 MB
Listing All Schedules
List all the job schedules available for the specified Arctic catalog.
Method and URLGET /v0/arctic/catalogs/{catalogId}/schedules
Parameters
catalogId
path
String (UUID)
Unique identifier of the catalog you want to list all job schedules for.
Example 1a234bc5-6789-46d7-b9cb-d236e195b34a
query
Object
Optional
A common expression language (CEL) expression that filters responses so that they include only results with the specified attributes and values. Value is a URL-encoded string that represents a JSON object. The JSON object specifies the attributes to filter on and the values to match for each attribute. Read filter Query Parameter for usage examples.
maxResults
query
Integer
Optional
The maximum number of results to to return in the response. The server may return fewer results, but not more. The default is 10
. Read maxResults Query Parameter for usage examples.
pageToken
query
String
Optional
Token for retrieving the next page of jobs. If the Dremio instance has more jobs than the maximum per page (default 10), the response will include a nextPageToken after the data array. Use the nextPageToken value in your request URL as the pageToken value. Do not change any other query parameters included in the request URL when you use pageToken. Read pageToken Query Parameter for usage examples.
filter
Object Attributes
type
String
The type of job to be run by the schedule. For Arctic optimization jobs, the type is OPTIMIZE
.
Example OPTIMIZE
tableId
String
The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator. If a tableID
is used, then the type is OPTIMIZE
.
Example sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips
reference
String
Identifies the branch, tag, or commit where the table containing the optimization schedule is located. If a reference is used, then the type is OPTIMIZE
.
Example main
user
String
The user who created the schedule.
Example dremio_user@company.com
Example Request
curl -X GET 'https://api.dremio.cloud/v0/arctic/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
[
{
"data": [
{
"type": "OPTIMIZE",
"id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
"schedule": "@weekly",
"username": "dremio_user@company.com",
"config": {
"maxFileSize": "460 MB",
"minFileSize": "192 MB",
"minFiles": 5,
"reference": "etl",
"tableId": "sample-NYC-taxi-trips",
"targetFileSize": "256 MB"
}
},
{
"type": "OPTIMIZE",
"id": "abc123d4-ab12-345cd-1234-abc123def456",
"schedule": "@monthly",
"username": "dremio_user@company.com",
"config": {
"maxFileSize": "920 MB",
"minFileSize": "384 MB",
"minFiles": 5,
"reference": "dev",
"tableId": "sample-SF-taxi-trips",
"targetFileSize": "512 MB"
}
}
]
}
]
Responses
200
Returned schedules on the specified catalog
400
Invalid filter parameter
404
Catalog not found
415
Content type(s) requested by client are not available on the server
500
Internal server error
Creating a Schedule
Creates a job schedule for the specified Arctic catalog.
Method and URLPOST /v0/arctic/catalogs/{catalogId}/schedules
Parameters
catalogId
path
String (UUID)
Unique identifier of the catalog you want to add a job schedules for.
Example 1a234bc5-6789-46d7-b9cb-d236e195b34a
type
body
String
The type of job that is run by the schedule. For Arctic optimization jobs, the type is OPTIMIZE
.
Example OPTIMIZE
schedule
body
String
Identifies a schedule's frequency. Can be a preset option or a supported cron expression. The cron expressions that are supported include:
• Dash-separated ranges
• Comma-separated lists
• Quotients
• Individual field values for minutes, hours, day of month, month, and day of week
• JAN-DEC, SUN-SAT, and their equivalent numeric values
The enum lists the preset options. The example shows a cron expression to run the job on the first day of every month. For information about cron expressions, read cron.
@once, @hourly, @daily, @weekly, @monthly
Example 0 0 1 * *
body
Object
Configuration options for the specified schedule.
config
Object Attributes
reference
body
String
Identifies the branch, tag, or commit where the table containing the optimization schedule should be created.
Example main
tableId
body
String
The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator.
Example sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips
maxFileSize
body
String
The maximum file size threshold for type OPTIMIZE
. Files that are larger in size than the specified maxFileSize
qualify for optimization. The integer must be non-negative. The default value is 460 MB.
Example 460 MB
minFileSize
body
String
The minimum file size threshold for type OPTIMIZE
. Files that are smaller in size than the specified minFileSize
qualify for optimization. The integer must be non-negative. The default value is 192 MB.
Example 192 MB
minFiles
body
String
The minimum number of qualified files required to run an optimization job. Supports only the type OPTIMIZE
. The default is 5 files.
Example 5
targetFileSize
body
String
Controls the target size of the files that are generated for type OPTIMIZE
. The integer must be non-negative. The default file size is 256 MB.
Example 70 MB
Example Request to Create a Weekly Schedule
curl -X POST 'https://api.dremio.cloud/v0/arctic/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
"type": "OPTIMIZE",
"schedule": "@weekly",
"config": {
"reference": "etl",
"tableID": "sample-NYC-taxi-trips",
"maxFileSize": "460 MB",
"minFileSize": "192 MB",
"minFiles": 5,
"targetFileSize": "70 MB"
}
}'
{
"type": "OPTIMIZE",
"id": "123ab456-cdef-78ab-c901-23de4f5ab6c7",
"schedule": "@weekly",
"username": "dremio_user@company.com",
"config": {
"maxFileSize": "460 MB",
"minFileSize": "192 MB",
"minFiles": 5,
"targetFileSize": "70 MB"
}
}
Responses
200
Successfully created a schedule
400
Invalid schedule definition
404
Catalog not found
415
Content type(s) requested by client are not available on the server
500
Internal server error
Retrieving a Schedule
Retrieve the specified schedule.
Method and URLGET /v0/arctic/catalogs/{catalogId}/schedules/{scheduleId}
Parameters
catalogId
path
String (UUID)
Unique identifier of the catalog you want to list all job schedules for.
Example 1a234bc5-6789-46d7-b9cb-d236e195b34a
scheduleId
path
String (UUID)
Unique identifier of the schedule you want to retrieve.
Example 9876a54b-681b-49fe-afff-0d753ed5ad85
curl -X GET 'https://api.dremio.cloud/v0/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules/9876a54b-681b-49fe-afff-0d753ed5ad85' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
{
"type": "OPTIMIZE",
"id": "673cd876-abcd-1234-b478-91ca5e9cc8f6",
"schedule": "@daily",
"username": "analyst@company.com",
"config": {
"reference": "dev",
"tableId": "sales_2022",
"maxFileSize": "2048 MB",
"minFileSize": "768 MB",
"minFiles": 10,
"targetFileSize": "1024 MB"
},
}
Responses
200
Found and returned a schedule
404
Catalog or schedule not found
415
Content type(s) requested by client are not available on the server
500
Internal server error
Modifying a Schedule
Modify an Arctic job schedule.
Method and URLPUT /v0/arctic/catalogs/{catalogId}/schedules/{scheduleId}
Parameters
catalogId
path
String (UUID)
Unique identifier of the catalog you want to modify a schedule for.
Example 1a234bc5-6789-46d7-b9cb-d236e195b34a
scheduleId
path
String (UUID)
Unique identifier of the schedule you want to modify.
Example 9876a54b-681b-49fe-afff-0d753ed5ad85
type
body
String
The type of job that is run by the schedule. For Arctic optimization jobs, the type is OPTIMIZE
.
Example OPTIMIZE
schedule
body
String
Identifies a schedule's frequency. Can be a preset option or a supported cron expression. The cron expressions that are supported include:
• Dash-separated ranges
• Comma-separated lists
• Quotients
• Individual field values for minutes, hours, day of month, month, and day of week
• JAN-DEC, SUN-SAT, and their equivalent numeric values
The enum lists the preset options. The example shows a cron expression to run the job on the first day of every month. For information about cron expressions, read cron.
@once, @hourly, @daily, @weekly, @monthly
Example 0 0 1 * *
body
Object
Configuration options for the specified schedule.
config
Object Attributes
reference
body
String
Identifies the branch, tag, or commit where the table containing the optimization schedule is located.
Example main
tableId
body
String
The name of the table that the optimization schedule applies to. The table name is preceded by zero or more namespaces with a period (.) as the separator.
Example sample-NYC-taxi-trips, tripdata.sample-NYC-taxi-trips
maxFileSize
body
String
The maximum file size threshold for type OPTIMIZE
. Files that are larger in size than the specified maxFileSize
qualify for optimization. The integer must be non-negative. The default value is 460 MB.
Example 460 MB
minFileSize
body
String
The minimum file size threshold for type OPTIMIZE
. Files that are smaller in size than the specified minFileSize
qualify for optimization. The integer must be non-negative. The default value is 192 MB.
Example 192 MB
minFiles
body
String
The minimum number of qualified files required to run an optimization job. Supports only the type OPTIMIZE
. The default is 5 files.
Example 5
targetFileSize
body
String
Controls the target size of the files that are generated for type OPTIMIZE
. The integer must be non-negative. The default file size is 256 MB.
Example 70 MB
Example Request to Modify a Weekly Schedule
curl -X GET 'https://api.dremio.cloud/v0/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules/9876a54b-681b-49fe-afff-0d753ed5ad85' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
"type": "OPTIMIZE",
"schedule": "@weekly",
"config": {
"reference": "compare",
"tableID": "customer-sales",
"maxFileSize": "460 MB",
"minFileSize": "192 MB",
"minFiles": 3,
"targetFileSize": "70 MB"
}
}'
{
"type": "OPTIMIZE",
"id": "9876a54b-681b-49fe-afff-0d753ed5ad85",
"schedule": "@weekly",
"username": "marketing-admin@company.com",
"config": {
"reference": "compare",
"tableID": "customer-sales",
"maxFileSize": "460 MB",
"minFileSize": "192 MB",
"minFiles": 3,
"targetFileSize": "70 MB"
},
}
Responses
200
Successfully updated the schedule
400
Invalid schedule definition
404
Catalog or schedule not found
415
Content type(s) requested by client are not available on the server
500
Internal server error
Deleting a Schedule
Delete an Arctic job schedule.
Method and URLDELETE /v0/arctic/catalogs/{catalogId}/schedules/{scheduleId}
Parameters
catalogId
path
String (UUID)
Unique identifier of the catalog you want to delete.
Example 1a234bc5-6789-46d7-b9cb-d236e195b34a
scheduleId
path
String (UUID)
Unique identifier of the schedule you want to delete.
Example 9876a54b-681b-49fe-afff-0d753ed5ad85
curl -X DELETE 'https://api.dremio.cloud/v0/catalogs/1a234bc5-6789-46d7-b9cb-d236e195b34a/schedules/9876a54b-681b-49fe-afff-0d753ed5ad85' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
{
"code": 200,
"id": "e04bd884-fa37-4d10-bb4e-153443b8234c",
"message": "Successfully found and deleted the schedule"
}
Responses
200
Successfully found and deleted the schedule
404
Catalog, job, or schedule not found
415
Content type(s) requested by client are not available on the server
500
Internal server error