Skip to main content

Table

Use the Catalog API to retrieve tables, format files and folders as tables, update and refresh tables, and revert tables to files and folders.

Table Object
{
  "entityType": "dataset",
  "id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
  "type": "PHYSICAL_DATASET",
  "path": [
    "Samples",
    "samples.dremio.com",
    "Dremio University",
    "restaurant_reviews.parquet"
  ],
  "createdAt": "2023-01-13T19:52:01.894Z",
  "tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
  "accelerationRefreshPolicy": {
    "refreshPeriodMs": 3600000,
    "gracePeriodMs": 10800000,
    "method": "FULL",
    "neverExpire": false,
    "neverRefresh": false
  },
  "format": {
    "type": "Parquet",
    "name": "restaurant_reviews.parquet",
    "fullPath": [
      "Samples",
      "samples.dremio.com",
      "Dremio University",
      "restaurant_reviews.parquet"
    ],
    "ctime": 0,
    "isFolder": false,
    "location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
    "autoCorrectCorruptDates": true
  },
  "accessControlList": {
    "users": [
      {
        "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
        "permissions": [
          "SELECT",
          "ALTER"
        ]
      },
      {
        "id": "30fca499-4abc-4469-7142-fc8dd29acac8",
        "permissions": [
          "SELECT",
          "ALTER",
          "MANAGE_GRANTS"
        ]
      }
    ],
    "roles": [
      {
        "id": "76a9884b-aea5-46d5-a73a-000edf23f390",
        "permissions": [
          "SELECT",
          "ALTER"
        ]
      }
    ]
  },
  "owner": {
    "ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
    "ownerType": "USER"
  },
  "fields": [
    {
      "name": "_id",
      "type": {
        "name": "VARCHAR"
      }
    },
    {
      "name": "name",
      "type": {
        "name": "VARCHAR"
      }
    },
    {
      "name": "city",
      "type": {
        "name": "VARCHAR"
      }
    },
    {
      "name": "state",
      "type": {
        "name": "VARCHAR"
      }
    },
    {
      "name": "categories",
      "type": {
        "name": "LIST",
        "subSchema": [
          {
            "type": {
              "name": "VARCHAR"
            }
          }
        ]
      }
    },
    {
      "name": "review_count",
      "type": {
        "name": "BIGINT"
      }
    },
    {
      "name": "stars",
      "type": {
        "name": "DOUBLE"
      }
    },
    {
      "name": "attributes",
      "type": {
        "name": "STRUCT",
        "subSchema": [
          {
            "name": "Parking",
            "type": {
              "name": "STRUCT",
              "subSchema": [
                {
                  "name": "garage",
                  "type": {
                    "name": "BOOLEAN"
                  }
                },
                {
                  "name": "street",
                  "type": {
                    "name": "BOOLEAN"
                  }
                },
                {
                  "name": "lot",
                  "type": {
                    "name": "BOOLEAN"
                  }
                },
                {
                  "name": "valet",
                  "type": {
                    "name": "BOOLEAN"
                  }
                }
              ]
            }
          },
          {
            "name": "Accepts Credit Cards",
            "type": {
              "name": "BOOLEAN"
            }
          },
          {
            "name": "Wheelchair Accessible",
            "type": {
              "name": "BOOLEAN"
            }
          },
          {
            "name": "Price Range",
            "type": {
              "name": "BIGINT"
            }
          }
        ]
      }
    },
    {
      "name": "date",
      "type": {
        "name": "VARCHAR"
      }
    }
  ],
  "approximateStatisticsAllowed": false
}

Table Attributes​

entityType

String

Type of catalog entity. For tables, the entityType is dataset.

Example dataset

id

String

Unique identifier for the table. For tables in non-Arctic sources, the id may be a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72 or a text path. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c9c11d32-0576-4200-5a5b-8c7229cb3d72

type

String

Type of dataset. For tables, the type is PHYSICAL_DATASET.

Example PHYSICAL_DATASET

path

[String]

Path to the table within Dremio, expressed in an array. The path lists each level of hierarchy in order, from outer to inner: source first, then folder and subfolders, then the table itself as the last item in the array.

Example [ "Samples", "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]

createdAt

String

Date and time that the table was created. In UTC format.

Example 2023-01-13T19:52:01.894Z

tag

String (UUID)

Unique identifier for the table instance. Dremio changes the tag whenever the table changes and uses the tag value to ensure that PUT requests apply to the most recent version of the table.

Example cb2905bb-39c0-497f-ae74-4c310d534f25

accelerationRefreshPolicy

Object

Attributes that define the acceleration refresh policy for the table.

format

Object

Table format attributes.

accessControlList

Object

Information about users and roles with privileges on the table and the specific privileges each user or role has. May include a users array, a roles array, or both, depending on the configured access and privileges. The accessControlList object is empty if table-specific privileges are not set.

Note: For tables in Arctic sources and the primary Arctic catalog, the table object does not include the accessControlList object.

For more information, read Inheritance, Scope, and Ownership.

owner

Object

Information about the table's owner.

Note: For tables in Arctic sources and the primary Arctic catalog, the table object does not include the owner object.

fields

Object

Attributes that represent the table schema.

approximateStatisticsAllowed

Boolean

If true, COUNT DISTINCT queries run on the table will return approximate results. Otherwise, false.

Example false

accelerationRefreshPolicy Object Attributes​

refreshPeriodMs

Integer

Refresh period for the data in all reflections for the table. In milliseconds.

Example 3600000

gracePeriodMs

Integer

Maximum age allowed for reflection data used to accelerate queries. In milliseconds.

Example 10800000

method

String

Approach used for refreshing the data in reflections defined on tables that are not in the Apache Iceberg format.

For more information, read Refreshing Reflections.

Enum FULL, INCREMENTAL

Example FULL

refreshField

String

For the INCREMENTAL refresh method, the field to refresh for the table. Used only if method is INCREMENTAL. This parameter applies only to tables that are not in the Apache Iceberg format.

Example business_id

neverExpire

Boolean

If the reflection will never expire, true. Otherwise, false.

Example false

neverRefresh

Boolean

If the reflection will never refresh, true. Otherwise, false.

Example false

format Object Attributes​

type

String

Type of data in the table.

Enum Delta, Excel, Iceberg, JSON, Parquet, Text, Unknown, XLS

Example Parquet

name

String

Table name. Dremio automatically duplicates the name of the origin file or folder to populate this value.

Example restaurant_reviews.parquet

fullPath

[String]

Path to the table within Dremio, expressed in an array. The path lists each level of hierarchy in order, from outer to inner: source first, then folder and subfolders, then the table itself as the last item in the array.

Example [ "Samples", "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]

ctime

Integer

Not used. Has the value 0.

Example 0

isFolder

Boolean

If true, the table was created from a folder. If false, the table was created from a file.

Example false

location

String

Location where the table's metadata is stored within a Dremio source, expressed in a string.

Example /samples.dremio.com/Dremio University/restaurant_reviews.parquet

metaStoreType

String

Not used. Has the value HDFS.

Example HDFS

parquetDataFormat

Object

Information about data format for Parquet tables.

dataFormatTypeList

[String]

List of data format types in the table. Included only for Iceberg tables, and PARQUET is the only valid value.

Example [ "PARQUET" ]

sheetName

String

For tables created from files with multiple sheets, name of the sheet used to create the table.

Example location_1

extractHeader

Boolean

For tables created from files, true if Dremio extracted the table's column names from the first line of the file. Otherwise, false.

Example false

hasMergedCells

Boolean

For tables created from files, true if Dremio expanded merged cells in the file when creating the table. Otherwise, false.

Example true

fieldDelimiter

String

Character used to indicate separate fields in the table. May be , for a comma (default), \t for a tab, | for a pipe, or a custom character.

quote

String

Character used for quotes in the table. May be \" for a double quote (default), ' for a single quote, or a custom character.

comment

String

Character used to indicate comments in the table. May be # for a number sign (default) or a custom character.

escape

String

Character used to indicate an escape in the table. May be " for a double quote (default), ` for a back quote, \\ for a backward slash, or a custom character.

lineDelimiter

String

Character used to indicate separate lines in the table. May be \r\n for a carriage return plus new line (default), \n for a new line, or a custom character.

skipFirstLine

Boolean

If Dremio skipped the first line in the file or folder when creating the table, true. Otherwise, false.

Example false

autoGenerateColumnNames

Boolean

If Dremio used the existing columnn names in the file or folder for the table columns, true. Otherwise, false.

Example true

trimHeader

Boolean

If Dremio trimmed column names to a specific number of characters when creating the table, true. Otherwise, false.

Example true

autoCorrectCorruptDates

Boolean

If Dremio Cloud automatically corrects corrupted date fields in the table, true. Otherwise, false.

Example true

accessControlList Object Attributes​

user

[Object]

Information about the users that have been granted privileges on the table and the privileges each user has.

Example [ { "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3", "permissions": [ "SELECT", "ALTER" ] }, { "id": "30fca499-4abc-4469-7142-fc8dd29acac8", "permissions": [ "SELECT", "ALTER", "MANAGE_GRANTS" ] } ]

role

[Object]

Information about the roles that have been granted privileges on the table and the privileges each role has.

Example [ { "id": "76a9884b-aea5-46d5-a73a-000edf23f390", "permissions": [ "SELECT", "ALTER" ] } ]

user and role Object Attributes​

id

String

The ID of the role or user.

Example c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3

permissions

[String]

The privileges that the role or user has on the table.

For more information, read Table Privileges.

Example [ "SELECT', "ALTER" ]

owner Object Attributes​

ownerId

String (UUID)

Unique identifier for the table's owner.

Example 30fca499-4abc-4469-7142-fc8dd29acac8

ownerType

String

Type of owner for the table.

Enum USER, ROLE

Example USER

fields Object Attributes​

name

String

Name of the table field.

Example review_count

type

Object

Information about the table field.

type Object Attributes​

name

String

Name of the table field's type.

Enum STRUCT, LIST, UNION, INTEGER, BIGINT, FLOAT, DOUBLE, VARCHAR, VARBINARY, BOOLEAN, DECIMAL, TIME, DATE, TIMESTAMP, INTERVAL DAY TO SECOND, INTERVAL YEAR TO MONTH

Example BIGINT

precision

Integer

Total number of digits in the number. Included only for the DECIMAL type.

Example 38

scale

Integer

Number of digits to the right of the decimal point. Included only for the DECIMAL type.

Example 0

subSchema

[Object]

List of objects that represent the field's composition. For example, a field composed of data about a restaurant might have a subSchema with an object for parking options, another for payment methods, and so on. subSchemas may be nested within other subSchemas. subSchema appears only for the STRUCT, LIST, and UNION types.

subSchema Array Attributes​

name

String

Name for the subSchema object.

Example Parking

type

Object

Object that contains a name attribute that provides the field's type.

Example { "name": "BOOLEAN" }

parquetDataFormat Object Attributes​

type

String

Type of data in the table. Within the parquetDataFormat object, the only valid type is Parquet.

Example Parquet

ctime

Integer

Not used. Has the value 0.

Example 0

isFolder

Boolean

If true, the table was created from a folder. If false, the table was created from a file.

Example true

autoCorrectCorruptDates

Boolean

If true, Dremio Cloud automatically corrects corrupted date fields in the table. Otherwise, false.

Example true

Format a File or Folder as a Table​

Format a file or folder as a table so that you can query the data in Dremio.

note

To format a folder, all files in the folder must be the same format.

Method and URL
POST /v0/projects/{project-id}/catalog/{id}

Parameters​

project-id

path

String (UUID)

Unique identifier for the project that contains the file or folder you want to format.

Example 1df71752-69b7-47d9-9e6c-990e6b194aa4

id

path

String

Unique identifier for the file or folder you want to format. For files and folders in non-Arctic sources, the id may be a UUID or a text path. If the id is a text path, use URL-encoded format to replace special characters with their UTF-8-equivalent characters: %3A for a colon; %2F for a forward slash, and %20 for a space. For example, if the id value is dremio:/Samples/samples.dremio.com/Dremio University, the encoded id is dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University. For files and folders in Arctic sources, the id is a JSON string like {"tableKey":["myArcticSource","myFolder"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c590ed7f-7142-4e1f-ba7d-94173afdc9a3

entityType

body

String

Type of catalog entity. To format a file or folder as a table, the entityType is dataset.

path

body

[String]

Path to the file or folder you want to format, expressed in an array. List each level of hierarchy in order, from outer to inner: source first, then any folder and subfolders, then the file or folder itself as the last item in the array. Get the path from the file or folder's children object in the response to a Folder request.

Example [ "Samples', "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]

type

body

String

Type of dataset. For tables, the type is PHYSICAL_DATASET.

format

body

String

Formatting parameters for the file or folder.

format Object Parameters​

type

body

String

Type of data in the file or folder. To format a folder, all files in the folder must be the same format.

Enum Delta, Excel, Iceberg, JSON, Parquet, Text, Unknown, XLS

skipFirstLine

body

Boolean

Optional

If Dremio should skip the first line in the file or folder when creating the table, true. Otherwise, false (default). Optional for Excel and Text types.

Example true

extractHeader

body

Boolean

Optional

If Dremio should extract the table's column names from the first line of the file, true. Otherwise, false (default). Optional for Excel and Text types.

Example true

hasMergedCells

body

Boolean

Optional

If Dremio should expand merged cells in the file when creating the table, true. Otherwise, false (default). Optional for Excel types.

Example true

sheetName

body

String

Optional

For tables created from Excel files with multiple sheets, name of the sheet to use to create the table. Default is the first sheet in the file (for files with multiple sheets).

Example location_1

fieldDelimiter

body

String

Optional

Character to use to indicate separate fields in the table. May be , for a comma (default), \t for a tab, | for a pipe, or a custom character. Optional for Text type.

quote

body

String

Optional

Character to use for quotes in the table. May be \" for a double quote (default), ' for a single quote, or a custom character. Optional for Text type.

comment

body

String

Optional

Character to use to indicate comments in the table. May be # for a number sign (default) or a custom character. Optional for Text type.

escape

body

String

Optional

Character used to indicate an escape in the table. May be " for a double quote (default), ` for a back quote, \\ for a backward slash, or a custom character. Optional for Text type.

lineDelimiter

body

String

Optional

Character used to indicate separate lines in the table. May be \r\n for a carriage return plus new line (default), \n for a new line, or a custom character. Optional for Text type.

autoGenerateColumnNames

body

Boolean

Optional

If Dremio should use the existing columnn names in the file or folder for the table columns, true (default). Otherwise, false. Optional for Text type.

Example true

trimHeader

body

Boolean

Optional

If Dremio should trim column names to a specific number of characters when creating the table, true. Otherwise, false (default). Optional for Text type.

Example true


Example Request
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "entityType": "dataset",
  "path": [
    "Samples",
    "Dremio University",
    "restaurant_reviews.parquet"
  ],
  "type": "PHYSICAL_DATASET",
  "format": {
    "type": "Parquet"
  },
  "accessControlList": {
    "users": [
      {
        "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
        "permissions": [
          "SELECT",
          "ALTER"
        ]
      },
      {
        "id": "30fca499-4abc-4469-7142-fc8dd29acac8",
        "permissions": [
          "SELECT",
          "ALTER",
          "MANAGE_GRANTS"
        ]
      }
    ],
    "roles": [
      {
        "id": "76a9884b-aea5-46d5-a73a-000edf23f390",
        "permissions": [
          "SELECT",
          "ALTER"
        ]
      }
    ]
  }
}'
Example Response
{
    "entityType": "dataset",
    "id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
    "type": "PHYSICAL_DATASET",
    "path": [
      "Samples",
      "samples.dremio.com",
      "Dremio University",
      "restaurant_reviews.parquet"
    ],
    "createdAt": "2023-01-13T19:52:01.894Z",
    "tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
    "format": {
      "type": "Parquet",
      "name": "restaurant_reviews.parquet",
      "fullPath": [
        "Samples",
        "samples.dremio.com",
        "Dremio University",
        "restaurant_reviews.parquet"
      ],
      "ctime": 0,
      "isFolder": false,
      "location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
      "autoCorrectCorruptDates": true
    },
    "accessControlList": {
      "users": [
        {
          "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
          "permissions": [
            "SELECT",
            "ALTER"
          ]
        },
        {
          "id": "30fca499-4abc-4469-7142-fc8dd29acac8",
          "permissions": [
            "SELECT",
            "ALTER",
            "MANAGE_GRANTS"
          ]
        }
      ],
      "roles": [
        {
          "id": "76a9884b-aea5-46d5-a73a-000edf23f390",
          "permissions": [
            "SELECT",
            "ALTER"
          ]
        }
      ]
    },
    "owner": {
      "ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
      "ownerType": "USER"
    },
    "fields": [
      {
        "name": "_id",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "name",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "city",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "state",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "categories",
        "type": {
          "name": "LIST",
          "subSchema": [
            {
              "type": {
                "name": "VARCHAR"
              }
            }
          ]
        }
      },
      {
        "name": "review_count",
        "type": {
          "name": "BIGINT"
        }
      },
      {
        "name": "stars",
        "type": {
          "name": "DOUBLE"
        }
      },
      {
        "name": "attributes",
        "type": {
          "name": "STRUCT",
          "subSchema": [
            {
              "name": "Parking",
              "type": {
                "name": "STRUCT",
                "subSchema": [
                  {
                    "name": "garage",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "street",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "lot",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "valet",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  }
                ]
              }
            },
            {
              "name": "Accepts Credit Cards",
              "type": {
                "name": "BOOLEAN"
              }
            },
            {
              "name": "Wheelchair Accessible",
              "type": {
                "name": "BOOLEAN"
              }
            },
            {
              "name": "Price Range",
              "type": {
                "name": "BIGINT"
              }
            }
          ]
        }
      },
      {
        "name": "date",
        "type": {
          "name": "VARCHAR"
        }
      }
    ],
    "approximateStatisticsAllowed": false
  }
Example Request for Excel format type
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University%2Foracle-departments.xlsx' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "entityType": "dataset",
  "path": [
      "Samples",
      "samples.dremio.com",
      "Dremio University",
      "oracle-departments.xlsx"
    ],
  "type": "PHYSICAL_DATASET",
  "format": {
    "type": "Excel",
    "extractHeader": true,
    "hasMergedCells": true,
    "sheetName": "Sheet1"
    }
}'
Example Request for Text format type
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/6ba3bd6e-fd27-4572-a535-77e1548283b3' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "entityType": "dataset",
  "path": [
    "Samples",
    "samples.dremio.com",
    "Dremio University",
    "airbnb_listings.csv"
  ],
  "type": "PHYSICAL_DATASET",
  "format": {
    "type": "Text",
    "fieldDelimiter": ",",
    "skipFirstLine": false,
    "extractHeader": true,
    "quote": "\"",
    "comment": "#",
    "escape": "\"",
    "lineDelimiter": "\r\n",
    "autoGenerateColumnNames": true,
    "trimHeader": false
  }
}'

Response Status Codes​

200

OK

400

Bad Request

401

Unauthorized

404

Not Found

500

Internal Server Error

Retrieve a Table by ID​

Retrieve a table by specifying the table's id value.

Method and URL
GET /v0/projects/{project-id}/catalog/{id}

Parameters​

project-id

path

String (UUID)

Unique identifier for the project that contains the table you want to retrieve.

Example 1df71752-69b7-47d9-9e6c-990e6b194aa4

id

path

String

Unique identifier for the table you want to retrieve. For tables in non-Arctic sources, the id may be a UUID or a text path. If the id is a text path, use URL-encoded format to replace special characters with their UTF-8-equivalent characters: %3A for a colon; %2F for a forward slash, and %20 for a space. For example, if the id value is dremio:/Samples/samples.dremio.com/Dremio University, the encoded id is dremio%3A%2FSamples%2Fsamples.dremio.com%2FDremio%20University. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c9c11d32-0576-4200-5a5b-8c7229cb3d72


Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
Example Response
{
    "entityType": "dataset",
    "id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
    "type": "PHYSICAL_DATASET",
    "path": [
      "Samples",
      "samples.dremio.com",
      "Dremio University",
      "restaurant_reviews.parquet"
    ],
    "createdAt": "2023-01-13T19:52:01.894Z",
    "tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
    "format": {
      "type": "Parquet",
      "name": "restaurant_reviews.parquet",
      "fullPath": [
        "Samples",
        "samples.dremio.com",
        "Dremio University",
        "restaurant_reviews.parquet"
      ],
      "ctime": 0,
      "isFolder": false,
      "location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
      "autoCorrectCorruptDates": true
    },
    "accessControlList": {
      "users": [
        {
          "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
          "permissions": [
            "SELECT",
            "ALTER"
          ]
        },
        {
          "id": "30fca499-4abc-4469-7142-fc8dd29acac8",
          "permissions": [
            "SELECT",
            "ALTER",
            "MANAGE_GRANTS"
          ]
        }
      ],
      "roles": [
        {
          "id": "76a9884b-aea5-46d5-a73a-000edf23f390",
          "permissions": [
            "SELECT",
            "ALTER"
          ]
        }
      ]
    },
    "owner": {
      "ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
      "ownerType": "USER"
    },
    "fields": [
      {
        "name": "_id",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "name",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "city",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "state",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "categories",
        "type": {
          "name": "LIST",
          "subSchema": [
            {
              "type": {
                "name": "VARCHAR"
              }
            }
          ]
        }
      },
      {
        "name": "review_count",
        "type": {
          "name": "BIGINT"
        }
      },
      {
        "name": "stars",
        "type": {
          "name": "DOUBLE"
        }
      },
      {
        "name": "attributes",
        "type": {
          "name": "STRUCT",
          "subSchema": [
            {
              "name": "Parking",
              "type": {
                "name": "STRUCT",
                "subSchema": [
                  {
                    "name": "garage",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "street",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "lot",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "valet",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  }
                ]
              }
            },
            {
              "name": "Accepts Credit Cards",
              "type": {
                "name": "BOOLEAN"
              }
            },
            {
              "name": "Wheelchair Accessible",
              "type": {
                "name": "BOOLEAN"
              }
            },
            {
              "name": "Price Range",
              "type": {
                "name": "BIGINT"
              }
            }
          ]
        }
      },
      {
        "name": "date",
        "type": {
          "name": "VARCHAR"
        }
      }
    ],
    "approximateStatisticsAllowed": false
  }

Response Status Codes​

200

OK

400

Bad Request

401

Unauthorized

404

Not Found

Retrieve a Table by Path​

Retrieve a table by specifying the table's path.

Method and URL
GET /v0/projects/{project-id}/catalog/by-path/{path}

Parameters​

project-id

path

String (UUID)

Unique identifier for the project that contains the table you want to retrieve.

Example 1df71752-69b7-47d9-9e6c-990e6b194aa4

path

path

String

Table's location within Dremio, using forward slashes as separators. For example, for the "NYC-taxi-trips" table in the "samples.dremio.com" folder within the source "Samples," the path is Samples/samples.dremio.com/NYC-taxi-trips. If the name of any component in the path includes special characters for URLs, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University in the URL path.

Example Samples/samples.dremio.com/Dremio%20University/restaurant_reviews.parquet


Example Request
curl -X GET 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/by-path/Samples/samples.dremio.com/Dremio%20University/restaurant_reviews.parquet' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json'
Example Response
{
    "entityType": "dataset",
    "id": "c9c11d32-0576-4200-5a5b-8c7229cb3d72",
    "type": "PHYSICAL_DATASET",
    "path": [
      "Samples",
      "samples.dremio.com",
      "Dremio University",
      "restaurant_reviews.parquet"
    ],
    "createdAt": "2023-01-13T19:52:01.894Z",
    "tag": "cb2905bb-39c0-497f-ae74-4c310d534f25",
    "format": {
      "type": "Parquet",
      "name": "restaurant_reviews.parquet",
      "fullPath": [
        "Samples",
        "samples.dremio.com",
        "Dremio University",
        "restaurant_reviews.parquet"
      ],
      "ctime": 0,
      "isFolder": false,
      "location": "/samples.dremio.com/Dremio University/restaurant_reviews.parquet",
      "autoCorrectCorruptDates": true
    },
    "accessControlList": {
      "users": [
        {
          "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3",
          "permissions": [
            "SELECT",
            "ALTER"
          ]
        },
        {
          "id": "30fca499-4abc-4469-7142-fc8dd29acac8",
          "permissions": [
            "SELECT",
            "ALTER",
            "MANAGE_GRANTS"
          ]
        }
      ],
      "roles": [
        {
          "id": "76a9884b-aea5-46d5-a73a-000edf23f390",
          "permissions": [
            "SELECT",
            "ALTER"
          ]
        }
      ]
    },
    "owner": {
      "ownerId": "30fca499-4abc-4469-7142-fc8dd29acac8",
      "ownerType": "USER"
    },
    "fields": [
      {
        "name": "_id",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "name",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "city",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "state",
        "type": {
          "name": "VARCHAR"
        }
      },
      {
        "name": "categories",
        "type": {
          "name": "LIST",
          "subSchema": [
            {
              "type": {
                "name": "VARCHAR"
              }
            }
          ]
        }
      },
      {
        "name": "review_count",
        "type": {
          "name": "BIGINT"
        }
      },
      {
        "name": "stars",
        "type": {
          "name": "DOUBLE"
        }
      },
      {
        "name": "attributes",
        "type": {
          "name": "STRUCT",
          "subSchema": [
            {
              "name": "Parking",
              "type": {
                "name": "STRUCT",
                "subSchema": [
                  {
                    "name": "garage",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "street",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "lot",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  },
                  {
                    "name": "valet",
                    "type": {
                      "name": "BOOLEAN"
                    }
                  }
                ]
              }
            },
            {
              "name": "Accepts Credit Cards",
              "type": {
                "name": "BOOLEAN"
              }
            },
            {
              "name": "Wheelchair Accessible",
              "type": {
                "name": "BOOLEAN"
              }
            },
            {
              "name": "Price Range",
              "type": {
                "name": "BIGINT"
              }
            }
          ]
        }
      },
      {
        "name": "date",
        "type": {
          "name": "VARCHAR"
        }
      }
    ],
    "approximateStatisticsAllowed": false
  }

Response Status Codes​

200

OK

400

Bad Request

401

Unauthorized

404

Not Found

Update a Table​

Update the specified table in Dremio.

Method and URL
PUT /v0/projects/{project-id}/catalog/{id}

Parameters​

project-id

path

String (UUID)

Unique identifier for the project that contains the table you want to update.

Example 1df71752-69b7-47d9-9e6c-990e6b194aa4

id

path

String

Unique identifier for the table you want to update. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c9c11d32-0576-4200-5a5b-8c7229cb3d72

entityType

body

String

Type of catalog entity. For tables, the entityType is dataset.

id

body

String (UUID)

Unique identifier for the table you want to update. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c9c11d32-0576-4200-5a5b-8c7229cb3d72

path

body

[String]

Path to the table you want to update, expressed in an array. List each level of hierarchy in order, from outer to inner: source first, then any folder and subfolders, then the table itself as the last item in the array. Get the path from the table's children object in the response to a Folder request.

Example [ "Samples', "samples.dremio.com", "Dremio University", "restaurant_reviews.parquet" ]

tag

body

String (UUID)

Optional

Unique identifier for the version of the table you want to update. If you provide a tag in the request body, Dremio uses the tag value to ensure that you are requesting to update the most recent version of the table. If you do not provide a tag, Dremio automatically updates the most recent version of the table.

Example cb2905bb-39c0-497f-ae74-4c310d534f25

type

body

String

Type of dataset. For tables, the type is PHYSICAL_DATASET.

format

body

String

Formatting parameters for the table.

accessControlList

body

Object

Optional

Object used to specify which users and roles should have privileges on the table and the specific privileges each user and role has. May include a users array, a roles array, or both. Omit if you want the table to inherit access and privileges. Not supported for tables in Arctic sources and the primary Arctic catalog.

For more information, read Inheritance, Scope, and Ownership.

format Object Parameters​

type

body

String

Type of data in the table.

Enum Delta, Excel, Iceberg, JSON, Parquet, Text, Unknown, XLS

skipFirstLine

body

Boolean

Optional

If Dremio should skip the first line in the table, true. Otherwise, false (default). Optional for Excel and Text types.

Example true

extractHeader

body

Boolean

Optional

If Dremio should extract the table's column names from the first line of the file, true. Otherwise, false (default). Optional for Excel and Text types.

Example true

hasMergedCells

body

Boolean

Optional

If Dremio should expand merged cells in the table, true. Otherwise, false (default). Optional for Excel types.

Example true

fieldDelimiter

body

String

Optional

Character to use to indicate separate fields in the table. May be , for a comma (default), \t for a tab, | for a pipe, or a custom character. Optional for Text type.

quote

body

String

Optional

Character to use for quotes in the table. May be \" for a double quote (default), ' for a single quote, or a custom character. Optional for Text type.

comment

body

String

Optional

Character to use to indicate comments for the table. May be # for a number sign (default) or a custom character. Optional for Text type.

escape

body

String

Optional

Character to use to indicate an escape for the table. May be " for a double quote (default), ` for a back quote, \\ for a backward slash, or a custom character. Optional for Text type.

lineDelimiter

body

String

Optional

Character to use to indicate separate lines for the table. May be \r\n for a carriage return plus new line (default), \n for a new line, or a custom character. Optional for Text type.

autoGenerateColumnNames

body

Boolean

Optional

If Dremio should use the existing columnn names for the table columns, true (default). Otherwise, false. Optional for Text type.

Example true

trimHeader

body

Boolean

Optional

If Dremio should trim column names to a specific number of characters when updating the table, true. Otherwise, false (default). Optional for Text type.

Example true

accessControlList Object Parameters​

user

body

[Object]

Optional

The users that should have privileges on the table and the privileges each user should have.

Example [ { "id": "c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3", "permissions": [ "SELECT", "ALTER" ] }, { "id": "30fca499-4abc-4469-7142-fc8dd29acac8", "permissions": [ "SELECT", "ALTER", "MANAGE_GRANTS" ] } ]

role

body

[Object]

Optional

The roles that should have privileges on the table and the privileges each role should have.

Example [ { "id": "76a9884b-aea5-46d5-a73a-000edf23f390", "permissions": [ "SELECT", "ALTER" ] } ]

user and role Object Parameters​

id

body

String

Optional

The ID of the role or user.

Example c590ed7f-b2b4-4e1f-ba7d-94173afdc9a3

permissions

body

[String]

Optional

The privileges that the role or user should have on the table.

For more information, read Table Privileges.

Example [ "SELECT', "ALTER" ]


Example Request
curl -X PUT 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/dba1e4fe-6351-44d2-a3e0-7aa20e782bf3' \
--header 'Authorization: Bearer <PersonalAccessToken>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "entityType": "dataset",
  "id": "dba1e4fe-6351-44d2-a3e0-7aa20e782bf3",
  "path": [
    "Samples",
    "samples.dremio.com",
    "Dremio University",
    "airbnb_listings.csv"
  ],
  "type": "PHYSICAL_DATASET",
  "format": {
    "type": "Text",
    "fieldDelimiter": ",",
    "skipFirstLine": false,
    "extractHeader": true,
    "quote": "\"",
    "comment": "#",
    "escape": "\"",
    "lineDelimiter": "\r\n",
    "autoGenerateColumnNames": true,
    "trimHeader": true
  }
}'
Example Response
{
    "entityType": "dataset",
    "id": "dba1e4fe-6351-44d2-a3e0-7aa20e782bf3",
    "type": "PHYSICAL_DATASET",
    "path": [
        "Samples",
        "samples.dremio.com",
        "Dremio University",
        "airbnb_listings.csv"
    ],
    "createdAt": "2023-01-23T21:26:59.568Z",
    "tag": "fc1707df-35a1-45c1-87d7-5f66fb11a729",
    "format": {
        "type": "Text",
        "ctime": 0,
        "isFolder": false,
        "location": "/samples.dremio.com/Dremio University/airbnb_listings.csv",
        "fieldDelimiter": ",",
        "skipFirstLine": false,
        "extractHeader": true,
        "quote": "\"",
        "comment": "#",
        "escape": "\"",
        "lineDelimiter": "\r\n",
        "autoGenerateColumnNames": true,
        "trimHeader": true
    },
    "accessControlList": {},
    "owner": {
        "ownerId": "c590ed7f-7142-4e1f-ba7d-94173afdc9a3",
        "ownerType": "USER"
    },
    "fields": [
        {
            "name": "id",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "listing_url",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "scrape_id",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "last_scraped",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "name",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "summary",
            "type": {
                "name": "VARCHAR"
            }
        },
        {...},
        {
            "name": "reviews_per_month",
            "type": {
                "name": "VARCHAR"
            }
        }
    ],
    "approximateStatisticsAllowed": false
}

Response Status Codes​

200

OK

400

Bad Request

401

Unauthorized

404

Not Found

500

Internal Server Error

Refresh a Table​

Refresh the data stored in the specified table and the reflections associated with the table.

Read Refreshing Reflections to learn how refreshing works.

Method and URL
POST /v0/projects/{project-id}/catalog/{id}/refresh

Parameters​

project-id

path

String (UUID)

Unique identifier for the project that contains the table you want to refresh.

Example 1df71752-69b7-47d9-9e6c-990e6b194aa4

id

path

String (UUID)

Unique identifier for the table you want to refresh. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72. For tables in Arctic sources and the primary Arctic catalog, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c9c11d32-0576-4200-5a5b-8c7229cb3d72


Example Request
curl -X POST 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72/refresh' \
-H 'Authorization: Bearer <PersonalAccessToken>' \
-H 'Content-Type: application/json'
Example Response
No response

Response Status Codes​

204

No Content

400

Bad Request

401

Unauthorized

404

Not Found

Revert a Table to a File or Folder​

Revert a table in a source to change the data in the table back to its original format, file or folder.

caution

Tables in primary Arctic catalogs and Arctic Catalog sources are deleted rather than reverted.

Method and URL
DELETE /v0/projects/{project-id}/catalog/{id}

Parameters​

project-id

path

String (UUID)

Unique identifier for the project that contains the table you want to revert to a file or folder.

Example 1df71752-69b7-47d9-9e6c-990e6b194aa4

id

path

String

Unique identifier for the table you want to revert to a file or folder. For tables that are not in an Arctic source, the id is a UUID like c9c11d32-0576-4200-5a5b-8c7229cb3d72. For tables in Arctic sources, the id is a JSON string like {"tableKey":["myArcticSource","myFolder","myTable"],"contentId":"65b24a53-c56e-4037-8ffc-f25297614c3c","versionContext":{"type":"BRANCH","value":"main"}}.

Example c9c11d32-0576-4200-5a5b-8c7229cb3d72


Example Request
curl -X DELETE 'https://api.dremio.cloud/v0/projects/1df71752-69b7-47d9-9e6c-990e6b194aa4/catalog/c9c11d32-0576-4200-5a5b-8c7229cb3d72' \
-H 'Authorization: Bearer <PersonalAccessToken>' \
-H 'Content-Type: application/json'
Example Response
No response

Response Status Codes​

204

No Content

400

Bad Request

401

Unauthorized

404

Not Found