Table
Use the Catalog API to retrieve tables, format files and folders as tables, update and refresh tables, and revert tables to files and folders.
Table Object{
"entityType": "dataset",
"id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
"type": "PHYSICAL_DATASET",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"createdAt": "2023-01-13T19:52:01.894Z",
"tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
"accelerationRefreshPolicy": {
"refreshPeriodMs": 3600000,
"gracePeriodMs": 10800000,
"method": "FULL",
"neverExpire": false,
"neverRefresh": false
},
"format": {
"type": "Parquet",
"name": "restaurant_reviews.parquet",
"fullPath": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"ctime": 0,
"isFolder": false,
"location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
"autoCorrectCorruptDates": true
},
"accessControlList": {
"users": [
{
"id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
"permissions": [
"SELECT",
"ALTER"
]
},
{
"id": "30fca499-4abc-4469-7142-fc8dd29acac8",
"permissions": [
"SELECT",
"ALTER",
"MANAGE_GRANTS"
]
}
],
"roles": [
{
"id": "76a9884b-aea5-46d5-a73a-000edf23f390",
"permissions": [
"SELECT",
"ALTER"
]
}
]
},
"owner": {
"ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
"ownerType": "USER"
},
"fields": [
{
"name": "_id",
"type": {
"name": "VARCHAR"
}
},
{
"name": "name",
"type": {
"name": "VARCHAR"
}
},
{
"name": "city",
"type": {
"name": "VARCHAR"
}
},
{
"name": "state",
"type": {
"name": "VARCHAR"
}
},
{
"name": "categories",
"type": {
"name": "LIST",
"subSchema": [
{
"type": {
"name": "VARCHAR"
}
}
]
}
},
{
"name": "review_count",
"type": {
"name": "BIGINT"
}
},
{
"name": "stars",
"type": {
"name": "DOUBLE"
}
},
{
"name": "attributes",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "Parking",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "garage",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "street",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "lot",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "valet",
"type": {
"name": "BOOLEAN"
}
}
]
}
},
{
"name": "Accepts Credit Cards",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Wheelchair Accessible",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Price Range",
"type": {
"name": "BIGINT"
}
}
]
}
},
{
"name": "date",
"type": {
"name": "VARCHAR"
}
}
],
"approximateStatisticsAllowed": false
}
Table Attributes
entityType
String
Type of catalog entity. For tables, the entityType is dataset
.
Example dataset
id
String
Unique identifier for the table. For tables in non-Arctic sources, the id may be a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72
or a text path. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c9c11d32-0576-4200-5a5b-8c7229cb3d72
type
String
Type of dataset. For tables, the type is PHYSICAL_DATASET
.
Example PHYSICAL_DATASET
path
[String]
Path to the table within Dremio, expressed in an array. The path lists each level of hierarchy in order, from outer to inner: source first, then folder and subfolders, then the table itself as the last item in the array.
Example [ "Samples", "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]
createdAt
String
Date and time that the table was created. In UTC format.
Example 2023-01-13T19:52:01.894Z
tag
String (UUID)
Unique identifier for the table instance. Dremio changes the tag whenever the table changes and uses the tag value to ensure that PUT requests apply to the most recent version of the table.
Example cb2905bb-39c0-497f-ae74-4c310d534f25
Object
Attributes that define the acceleration refresh policy for the table.
Object
Table format attributes.
Object
Information about users and roles with privileges on the table and the specific privileges each user or role has. May include a users array, a roles array, or both, depending on the configured access and privileges. The accessControlList object is empty if table-specific privileges are not set.
Note: For tables in Arctic sources and the primary Arctic catalog, the table object does not include the accessControlList object.
For more information, read Inheritance, Scope, and Ownership.
Object
Information about the table's owner.
Note: For tables in Arctic sources and the primary Arctic catalog, the table object does not include the owner object.
Object
Attributes that represent the table schema.
approximateStatisticsAllowed
Boolean
If true, COUNT DISTINCT
queries run on the table will return approximate results. Otherwise, false.
Example false
accelerationRefreshPolicy
Object Attributes
refreshPeriodMs
Integer
Refresh period for the data in all reflections for the table. In milliseconds.
Example 3600000
gracePeriodMs
Integer
Maximum age allowed for reflection data used to accelerate queries. In milliseconds.
Example 10800000
method
String
Approach used for refreshing the data in reflections defined on tables that are not in the Apache Iceberg format.
For more information, read Refreshing Reflections.
EnumFULL, INCREMENTAL
Example FULL
refreshField
String
For the INCREMENTAL
refresh method, the field to refresh for the table. Used only if method is INCREMENTAL
. This parameter applies only to tables that are not in the Apache Iceberg format.
Example business_id
neverExpire
Boolean
If the reflection will never expire, true
. Otherwise, false
.
Example false
neverRefresh
Boolean
If the reflection will never refresh, true
. Otherwise, false
.
Example false
format
Object Attributes
type
String
Type of data in the table.
EnumDelta, Excel, Iceberg, JSON, Parquet, Text, Unknown, XLS
Example Parquet
name
String
Table name. Dremio automatically duplicates the name of the origin file or folder to populate this value.
Example restaurant_reviews.parquet
fullPath
[String]
Path to the table within Dremio, expressed in an array. The path lists each level of hierarchy in order, from outer to inner: source first, then folder and subfolders, then the table itself as the last item in the array.
Example [ "Samples", "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]
ctime
Integer
Not used. Has the value 0
.
Example 0
isFolder
Boolean
If true
, the table was created from a folder. If false
, the table was created from a file.
Example false
location
String
Location where the table's metadata is stored within a Dremio source, expressed in a string.
Example /samples.dremio.com/Dremio University/restaurant_reviews.parquet
metaStoreType
String
Not used. Has the value HDFS
.
Example HDFS
Object
Information about data format for Parquet tables.
dataFormatTypeList
[String]
List of data format types in the table. Included only for Iceberg tables, and PARQUET
is the only valid value.
Example [ "PARQUET" ]
sheetName
String
For tables created from files with multiple sheets, name of the sheet used to create the table.
Example location_1
extractHeader
Boolean
For tables created from files, true
if Dremio extracted the table's column names from the first line of the file. Otherwise, false
.
Example false
hasMergedCells
Boolean
For tables created from files, true
if Dremio expanded merged cells in the file when creating the table. Otherwise, false
.
Example true
fieldDelimiter
String
Character used to indicate separate fields in the table. May be ,
for a comma (default), \t
for a tab, |
for a pipe, or a custom character.
quote
String
Character used for quotes in the table. May be \"
for a double quote (default), '
for a single quote, or a custom character.
comment
String
Character used to indicate comments in the table. May be #
for a number sign (default) or a custom character.
escape
String
Character used to indicate an escape in the table. May be "
for a double quote (default), `
for a back quote, \\
for a backward slash, or a custom character.
lineDelimiter
String
Character used to indicate separate lines in the table. May be \r\n
for a carriage return plus new line (default), \n
for a new line, or a custom character.
skipFirstLine
Boolean
If Dremio skipped the first line in the file or folder when creating the table, true
. Otherwise, false
.
Example false
autoGenerateColumnNames
Boolean
If Dremio used the existing columnn names in the file or folder for the table columns, true
. Otherwise, false
.
Example true
trimHeader
Boolean
If Dremio trimmed column names to a specific number of characters when creating the table, true
. Otherwise, false
.
Example true
autoCorrectCorruptDates
Boolean
If Dremio Cloud automatically corrects corrupted date fields in the table, true
. Otherwise, false
.
Example true
accessControlList
Object Attributes
[Object]
Information about the users that have been granted privileges on the table and the privileges each user has.
Example [ { "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3", "permissions": [ "SELECT", "ALTER" ] }, { "id": "30fca499-4abc-4469-7142-fc8dd29acac8", "permissions": [ "SELECT", "ALTER", "MANAGE_GRANTS" ] } ]
[Object]
Information about the roles that have been granted privileges on the table and the privileges each role has.
Example [ { "id": "76a9884b-aea5-46d5-a73a-000edf23f390", "permissions": [ "SELECT", "ALTER" ] } ]
user
and role
Object Attributes
id
String
The ID of the role or user.
Example c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3
permissions
[String]
The privileges that the role or user has on the table.
For more information, read Table Privileges.
Example [ "SELECT', "ALTER" ]
owner
Object Attributes
ownerId
String (UUID)
Unique identifier for the table's owner.
Example 30fca499-4abc-4469-7142-fc8dd29acac8
ownerType
String
Type of owner for the table.
EnumUSER, ROLE
Example USER
fields
Object Attributes
type
Object Attributes
name
String
Name of the table field's type.
EnumSTRUCT, LIST, UNION, INTEGER, BIGINT, FLOAT, DOUBLE, VARCHAR, VARBINARY, BOOLEAN, DECIMAL, TIME, DATE, TIMESTAMP, INTERVAL DAY TO SECOND, INTERVAL YEAR TO MONTH
Example BIGINT
precision
Integer
Total number of digits in the number. Included only for the DECIMAL
type.
Example 38
scale
Integer
Number of digits to the right of the decimal point. Included only for the DECIMAL
type.
Example 0
[Object]
List of objects that represent the field's composition. For example, a field composed of data about a restaurant might have a subSchema with an object for parking options, another for payment methods, and so on. subSchemas may be nested within other subSchemas. subSchema appears only for the STRUCT
, LIST
, and UNION
types.
subSchema
Array Attributes
name
String
Name for the subSchema object.
Example Parking
type
Object
Object that contains a name
attribute that provides the field's type.
Example { "name": "BOOLEAN" }
parquetDataFormat
Object Attributes
type
String
Type of data in the table. Within the parquetDataFormat object, the only valid type is Parquet
.
Example Parquet
ctime
Integer
Not used. Has the value 0
.
Example 0
isFolder
Boolean
If true
, the table was created from a folder. If false
, the table was created from a file.
Example true
autoCorrectCorruptDates
Boolean
If true
, Dremio Cloud automatically corrects corrupted date fields in the table. Otherwise, false
.
Example true
Format a File or Folder as a Table
Format a file or folder as a table so that you can query the data in Dremio.
To format a folder, all files in the folder must be the same format.
POST /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
Unique identifier for the project that contains the file or folder you want to format.
Example 1df71752-69b7-47d9-9e6c-990e6b194aa4
id
path
String
Unique identifier for the file or folder you want to format. For files and folders in non-Arctic sources, the id may be a UUID or a text path. If the id is a text path, use URL-encoded format to replace special characters with their UTF-8-equivalent characters: %3A
for a colon; %2F
for a forward slash, and %20
for a space. For example, if the id value is dremio:/Samples/samples.dremio.com/Dremio University
, the encoded id is dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University
. For files and folders in Arctic sources, the id is a JSON string like {"tableKey":["myArcticSource","myFolder"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c590ed7f-7142-4e1f-ba7d-94173afdc9a3
entityType
body
String
Type of catalog entity. To format a file or folder as a table, the entityType is dataset
.
path
body
[String]
Path to the file or folder you want to format, expressed in an array. List each level of hierarchy in order, from outer to inner: source first, then any folder and subfolders, then the file or folder itself as the last item in the array. Get the path from the file or folder's children object in the response to a Folder request.
Example [ "Samples', "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]
type
body
String
Type of dataset. For tables, the type is PHYSICAL_DATASET
.
body
String
Formatting parameters for the file or folder.
format
Object Parameters
type
body
String
Type of data in the file or folder. To format a folder, all files in the folder must be the same format.
EnumDelta, Excel, Iceberg, JSON, Parquet, Text, Unknown, XLS
skipFirstLine
body
Boolean
Optional
If Dremio should skip the first line in the file or folder when creating the table, true
. Otherwise, false
(default). Optional for Excel and Text types.
Example true
extractHeader
body
Boolean
Optional
If Dremio should extract the table's column names from the first line of the file, true
. Otherwise, false
(default). Optional for Excel and Text types.
Example true
hasMergedCells
body
Boolean
Optional
If Dremio should expand merged cells in the file when creating the table, true
. Otherwise, false
(default). Optional for Excel types.
Example true
sheetName
body
String
Optional
For tables created from Excel files with multiple sheets, name of the sheet to use to create the table. Default is the first sheet in the file (for files with multiple sheets).
Example location_1
fieldDelimiter
body
String
Optional
Character to use to indicate separate fields in the table. May be ,
for a comma (default), \t
for a tab, |
for a pipe, or a custom character. Optional for Text type.
quote
body
String
Optional
Character to use for quotes in the table. May be \"
for a double quote (default), '
for a single quote, or a custom character. Optional for Text type.
comment
body
String
Optional
Character to use to indicate comments in the table. May be #
for a number sign (default) or a custom character. Optional for Text type.
escape
body
String
Optional
Character used to indicate an escape in the table. May be "
for a double quote (default), `
for a back quote, \\
for a backward slash, or a custom character. Optional for Text type.
lineDelimiter
body
String
Optional
Character used to indicate separate lines in the table. May be \r\n
for a carriage return plus new line (default), \n
for a new line, or a custom character. Optional for Text type.
autoGenerateColumnNames
body
Boolean
Optional
If Dremio should use the existing columnn names in the file or folder for the table columns, true
(default). Otherwise, false
. Optional for Text type.
Example true
trimHeader
body
Boolean
Optional
If Dremio should trim column names to a specific number of characters when creating the table, true
. Otherwise, false
(default). Optional for Text type.
Example true
Example Request
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"entityType": "dataset",
"path": [
"Samples",
"Dremio University",
"restaurant_reviews.parquet"
],
"type": "PHYSICAL_DATASET",
"format": {
"type": "Parquet"
},
"accessControlList": {
"users": [
{
"id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
"permissions": [
"SELECT",
"ALTER"
]
},
{
"id": "30fca499-4abc-4469-7142-fc8dd29acac8",
"permissions": [
"SELECT",
"ALTER",
"MANAGE_GRANTS"
]
}
],
"roles": [
{
"id": "76a9884b-aea5-46d5-a73a-000edf23f390",
"permissions": [
"SELECT",
"ALTER"
]
}
]
}
}'
{
"entityType": "dataset",
"id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
"type": "PHYSICAL_DATASET",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"createdAt": "2023-01-13T19:52:01.894Z",
"tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
"format": {
"type": "Parquet",
"name": "restaurant_reviews.parquet",
"fullPath": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"ctime": 0,
"isFolder": false,
"location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
"autoCorrectCorruptDates": true
},
"accessControlList": {
"users": [
{
"id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
"permissions": [
"SELECT",
"ALTER"
]
},
{
"id": "30fca499-4abc-4469-7142-fc8dd29acac8",
"permissions": [
"SELECT",
"ALTER",
"MANAGE_GRANTS"
]
}
],
"roles": [
{
"id": "76a9884b-aea5-46d5-a73a-000edf23f390",
"permissions": [
"SELECT",
"ALTER"
]
}
]
},
"owner": {
"ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
"ownerType": "USER"
},
"fields": [
{
"name": "_id",
"type": {
"name": "VARCHAR"
}
},
{
"name": "name",
"type": {
"name": "VARCHAR"
}
},
{
"name": "city",
"type": {
"name": "VARCHAR"
}
},
{
"name": "state",
"type": {
"name": "VARCHAR"
}
},
{
"name": "categories",
"type": {
"name": "LIST",
"subSchema": [
{
"type": {
"name": "VARCHAR"
}
}
]
}
},
{
"name": "review_count",
"type": {
"name": "BIGINT"
}
},
{
"name": "stars",
"type": {
"name": "DOUBLE"
}
},
{
"name": "attributes",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "Parking",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "garage",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "street",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "lot",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "valet",
"type": {
"name": "BOOLEAN"
}
}
]
}
},
{
"name": "Accepts Credit Cards",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Wheelchair Accessible",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Price Range",
"type": {
"name": "BIGINT"
}
}
]
}
},
{
"name": "date",
"type": {
"name": "VARCHAR"
}
}
],
"approximateStatisticsAllowed": false
}
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University%2Foracle-departments.xlsx' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"entityType": "dataset",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"oracle-departments.xlsx"
],
"type": "PHYSICAL_DATASET",
"format": {
"type": "Excel",
"extractHeader": true,
"hasMergedCells": true,
"sheetName": "Sheet1"
}
}'
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/6ba3bd6e-fd27-4572-a535-77e1548283b3' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"entityType": "dataset",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"airbnb_listings.csv"
],
"type": "PHYSICAL_DATASET",
"format": {
"type": "Text",
"fieldDelimiter": ",",
"skipFirstLine": false,
"extractHeader": true,
"quote": "\"",
"comment": "#",
"escape": "\"",
"lineDelimiter": "\r\n",
"autoGenerateColumnNames": true,
"trimHeader": false
}
}'
Response Status Codes
200
OK
400
Bad Request
401
Unauthorized
404
Not Found
500
Internal Server Error
Retrieve a Table by ID
Retrieve a table by specifying the table's id
value.
GET /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
Unique identifier for the project that contains the table you want to retrieve.
Example 1df71752-69b7-47d9-9e6c-990e6b194aa4
id
path
String
Unique identifier for the table you want to retrieve. For tables in non-Arctic sources, the id may be a UUID or a text path. If the id is a text path, use URL-encoded format to replace special characters with their UTF-8-equivalent characters: %3A
for a colon; %2F
for a forward slash, and %20
for a space. For example, if the id value is dremio:/Samples/samples.dremio.com/Dremio University
, the encoded id is dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University
. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c9c11d32-0576-4200-5a5b-8c7229cb3d72
Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
{
"entityType": "dataset",
"id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
"type": "PHYSICAL_DATASET",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"createdAt": "2023-01-13T19:52:01.894Z",
"tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
"format": {
"type": "Parquet",
"name": "restaurant_reviews.parquet",
"fullPath": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"ctime": 0,
"isFolder": false,
"location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
"autoCorrectCorruptDates": true
},
"accessControlList": {
"users": [
{
"id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
"permissions": [
"SELECT",
"ALTER"
]
},
{
"id": "30fca499-4abc-4469-7142-fc8dd29acac8",
"permissions": [
"SELECT",
"ALTER",
"MANAGE_GRANTS"
]
}
],
"roles": [
{
"id": "76a9884b-aea5-46d5-a73a-000edf23f390",
"permissions": [
"SELECT",
"ALTER"
]
}
]
},
"owner": {
"ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
"ownerType": "USER"
},
"fields": [
{
"name": "_id",
"type": {
"name": "VARCHAR"
}
},
{
"name": "name",
"type": {
"name": "VARCHAR"
}
},
{
"name": "city",
"type": {
"name": "VARCHAR"
}
},
{
"name": "state",
"type": {
"name": "VARCHAR"
}
},
{
"name": "categories",
"type": {
"name": "LIST",
"subSchema": [
{
"type": {
"name": "VARCHAR"
}
}
]
}
},
{
"name": "review_count",
"type": {
"name": "BIGINT"
}
},
{
"name": "stars",
"type": {
"name": "DOUBLE"
}
},
{
"name": "attributes",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "Parking",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "garage",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "street",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "lot",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "valet",
"type": {
"name": "BOOLEAN"
}
}
]
}
},
{
"name": "Accepts Credit Cards",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Wheelchair Accessible",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Price Range",
"type": {
"name": "BIGINT"
}
}
]
}
},
{
"name": "date",
"type": {
"name": "VARCHAR"
}
}
],
"approximateStatisticsAllowed": false
}
Response Status Codes
200
OK
400
Bad Request
401
Unauthorized
404
Not Found
Retrieve a Table by Path
Retrieve a table by specifying the table's path.
Method and URLGET /v0/projects/{project-id}/catalog/by-path/{path}
Parameters
project-id
path
String (UUID)
Unique identifier for the project that contains the table you want to retrieve.
Example 1df71752-69b7-47d9-9e6c-990e6b194aa4
path
path
String
Table's location within Dremio, using forward slashes as separators. For example, for the "NYC-taxi-trips" table in the "samples.dremio.com" folder within the source "Samples," the path is Samples/samples.dremio.com/NYC-taxi-trips
. If the name of any component in the path includes special characters for URLs, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University
in the URL path.
Example Samples/samples.dremio.com/Dremio%20University/restaurant_reviews.parquet
Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/by-path/Samples/samples.dremio.com/Dremio%20University/restaurant_reviews.parquet' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
{
"entityType": "dataset",
"id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
"type": "PHYSICAL_DATASET",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"createdAt": "2023-01-13T19:52:01.894Z",
"tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
"format": {
"type": "Parquet",
"name": "restaurant_reviews.parquet",
"fullPath": [
"Samples",
"samples.dremio.com",
"Dremio University",
"restaurant_reviews.parquet"
],
"ctime": 0,
"isFolder": false,
"location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
"autoCorrectCorruptDates": true
},
"accessControlList": {
"users": [
{
"id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
"permissions": [
"SELECT",
"ALTER"
]
},
{
"id": "30fca499-4abc-4469-7142-fc8dd29acac8",
"permissions": [
"SELECT",
"ALTER",
"MANAGE_GRANTS"
]
}
],
"roles": [
{
"id": "76a9884b-aea5-46d5-a73a-000edf23f390",
"permissions": [
"SELECT",
"ALTER"
]
}
]
},
"owner": {
"ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
"ownerType": "USER"
},
"fields": [
{
"name": "_id",
"type": {
"name": "VARCHAR"
}
},
{
"name": "name",
"type": {
"name": "VARCHAR"
}
},
{
"name": "city",
"type": {
"name": "VARCHAR"
}
},
{
"name": "state",
"type": {
"name": "VARCHAR"
}
},
{
"name": "categories",
"type": {
"name": "LIST",
"subSchema": [
{
"type": {
"name": "VARCHAR"
}
}
]
}
},
{
"name": "review_count",
"type": {
"name": "BIGINT"
}
},
{
"name": "stars",
"type": {
"name": "DOUBLE"
}
},
{
"name": "attributes",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "Parking",
"type": {
"name": "STRUCT",
"subSchema": [
{
"name": "garage",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "street",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "lot",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "valet",
"type": {
"name": "BOOLEAN"
}
}
]
}
},
{
"name": "Accepts Credit Cards",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Wheelchair Accessible",
"type": {
"name": "BOOLEAN"
}
},
{
"name": "Price Range",
"type": {
"name": "BIGINT"
}
}
]
}
},
{
"name": "date",
"type": {
"name": "VARCHAR"
}
}
],
"approximateStatisticsAllowed": false
}
Response Status Codes
200
OK
400
Bad Request
401
Unauthorized
404
Not Found
Update a Table
Update the specified table in Dremio.
Method and URLPUT /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
Unique identifier for the project that contains the table you want to update.
Example 1df71752-69b7-47d9-9e6c-990e6b194aa4
id
path
String
Unique identifier for the table you want to update. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72
. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c9c11d32-0576-4200-5a5b-8c7229cb3d72
entityType
body
String
Type of catalog entity. For tables, the entityType is dataset
.
id
body
String (UUID)
Unique identifier for the table you want to update. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72
. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c9c11d32-0576-4200-5a5b-8c7229cb3d72
path
body
[String]
Path to the table you want to update, expressed in an array. List each level of hierarchy in order, from outer to inner: source first, then any folder and subfolders, then the table itself as the last item in the array. Get the path from the table's children object in the response to a Folder request.
Example [ "Samples', "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]
tag
body
String (UUID)
Optional
Unique identifier for the version of the table you want to update. If you provide a tag in the request body, Dremio uses the tag value to ensure that you are requesting to update the most recent version of the table. If you do not provide a tag, Dremio automatically updates the most recent version of the table.
Example cb2905bb-39c0-497f-ae74-4c310d534f25
type
body
String
Type of dataset. For tables, the type is PHYSICAL_DATASET
.
body
String
Formatting parameters for the table.
body
Object
Optional
Object used to specify which users and roles should have privileges on the table and the specific privileges each user and role has. May include a users array, a roles array, or both. Omit if you want the table to inherit access and privileges. Not supported for tables in Arctic sources and the primary Arctic catalog.
For more information, read Inheritance, Scope, and Ownership.
format
Object Parameters
type
body
String
Type of data in the table.
EnumDelta, Excel, Iceberg, JSON, Parquet, Text, Unknown, XLS
skipFirstLine
body
Boolean
Optional
If Dremio should skip the first line in the table, true
. Otherwise, false
(default). Optional for Excel and Text types.
Example true
extractHeader
body
Boolean
Optional
If Dremio should extract the table's column names from the first line of the file, true
. Otherwise, false
(default). Optional for Excel and Text types.
Example true
hasMergedCells
body
Boolean
Optional
If Dremio should expand merged cells in the table, true
. Otherwise, false
(default). Optional for Excel types.
Example true
fieldDelimiter
body
String
Optional
Character to use to indicate separate fields in the table. May be ,
for a comma (default), \t
for a tab, |
for a pipe, or a custom character. Optional for Text type.
quote
body
String
Optional
Character to use for quotes in the table. May be \"
for a double quote (default), '
for a single quote, or a custom character. Optional for Text type.
comment
body
String
Optional
Character to use to indicate comments for the table. May be #
for a number sign (default) or a custom character. Optional for Text type.
escape
body
String
Optional
Character to use to indicate an escape for the table. May be "
for a double quote (default), `
for a back quote, \\
for a backward slash, or a custom character. Optional for Text type.
lineDelimiter
body
String
Optional
Character to use to indicate separate lines for the table. May be \r\n
for a carriage return plus new line (default), \n
for a new line, or a custom character. Optional for Text type.
autoGenerateColumnNames
body
Boolean
Optional
If Dremio should use the existing columnn names for the table columns, true
(default). Otherwise, false
. Optional for Text type.
Example true
trimHeader
body
Boolean
Optional
If Dremio should trim column names to a specific number of characters when updating the table, true
. Otherwise, false
(default). Optional for Text type.
Example true
accessControlList
Object Parameters
body
[Object]
Optional
The users that should have privileges on the table and the privileges each user should have.
Example [ { "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3", "permissions": [ "SELECT", "ALTER" ] }, { "id": "30fca499-4abc-4469-7142-fc8dd29acac8", "permissions": [ "SELECT", "ALTER", "MANAGE_GRANTS" ] } ]
body
[Object]
Optional
The roles that should have privileges on the table and the privileges each role should have.
Example [ { "id": "76a9884b-aea5-46d5-a73a-000edf23f390", "permissions": [ "SELECT", "ALTER" ] } ]
user
and role
Object Parameters
id
body
String
Optional
The ID of the role or user.
Example c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3
permissions
body
[String]
Optional
The privileges that the role or user should have on the table.
For more information, read Table Privileges.
Example [ "SELECT', "ALTER" ]
Example Request
curl -X PUT 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/dba1e4fe-6351-44d2-a3e0-7aa20e782bf3' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
"entityType": "dataset",
"id": "dba1e4fe-6351-44d2-a3e0-7aa20e782bf3",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"airbnb_listings.csv"
],
"type": "PHYSICAL_DATASET",
"format": {
"type": "Text",
"fieldDelimiter": ",",
"skipFirstLine": false,
"extractHeader": true,
"quote": "\"",
"comment": "#",
"escape": "\"",
"lineDelimiter": "\r\n",
"autoGenerateColumnNames": true,
"trimHeader": true
}
}'
{
"entityType": "dataset",
"id": "dba1e4fe-6351-44d2-a3e0-7aa20e782bf3",
"type": "PHYSICAL_DATASET",
"path": [
"Samples",
"samples.dremio.com",
"Dremio University",
"airbnb_listings.csv"
],
"createdAt": "2023-01-23T21:26:59.568Z",
"tag": "fc1707df-35a1-45c1-87d7-5f66fb11a729",
"format": {
"type": "Text",
"ctime": 0,
"isFolder": false,
"location": "/samples.dremio.com/Dremio University/airbnb_listings.csv",
"fieldDelimiter": ",",
"skipFirstLine": false,
"extractHeader": true,
"quote": "\"",
"comment": "#",
"escape": "\"",
"lineDelimiter": "\r\n",
"autoGenerateColumnNames": true,
"trimHeader": true
},
"accessControlList": {},
"owner": {
"ownerId": "c590ed7f-7142-4e1f-ba7d-94173afdc9a3",
"ownerType": "USER"
},
"fields": [
{
"name": "id",
"type": {
"name": "VARCHAR"
}
},
{
"name": "listing_url",
"type": {
"name": "VARCHAR"
}
},
{
"name": "scrape_id",
"type": {
"name": "VARCHAR"
}
},
{
"name": "last_scraped",
"type": {
"name": "VARCHAR"
}
},
{
"name": "name",
"type": {
"name": "VARCHAR"
}
},
{
"name": "summary",
"type": {
"name": "VARCHAR"
}
},
{...},
{
"name": "reviews_per_month",
"type": {
"name": "VARCHAR"
}
}
],
"approximateStatisticsAllowed": false
}
Response Status Codes
200
OK
400
Bad Request
401
Unauthorized
404
Not Found
500
Internal Server Error
Refresh a Table
Refresh the data stored in the specified table and the reflections associated with the table.
Read Refreshing Reflections to learn how refreshing works.
Method and URLPOST /v0/projects/{project-id}/catalog/{id}/refresh
Parameters
project-id
path
String (UUID)
Unique identifier for the project that contains the table you want to refresh.
Example 1df71752-69b7-47d9-9e6c-990e6b194aa4
id
path
String (UUID)
Unique identifier for the table you want to refresh. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72
. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c9c11d32-0576-4200-5a5b-8c7229cb3d72
Example Request
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72/refresh' \
-H 'Authorization: Bearer <PersonalAccessToken>' \
-H 'Content-Type: application/json'
No response
Response Status Codes
204
No Content
400
Bad Request
401
Unauthorized
404
Not Found
Revert a Table to a File or Folder
Revert a table in a source to change the data in the table back to its original format, file or folder.
Tables in primary Arctic catalogs and Arctic Catalog sources are deleted rather than reverted.
DELETE /v0/projects/{project-id}/catalog/{id}
Parameters
project-id
path
String (UUID)
Unique identifier for the project that contains the table you want to revert to a file or folder.
Example 1df71752-69b7-47d9-9e6c-990e6b194aa4
id
path
String
Unique identifier for the table you want to revert to a file or folder. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72
. For tables in Arctic sources, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}
.
Example c9c11d32-0576-4200-5a5b-8c7229cb3d72
Example Request
curl -X DELETE 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72' \
-H 'Authorization: Bearer <PersonalAccessToken>' \
-H 'Content-Type: application/json'
No response
Response Status Codes
204
No Content
400
Bad Request
401
Unauthorized
404
Not Found