Skip to main content

Source

The Source API enables you to manage the sources that are supported in Dremio. You can add, retrieve, update, and delete sources.

The source object contains the same attributes for all source types except for the config object, which is different for each source type. The examples on this page use an Amazon S3 source to demonstrate the available endpoints. For configuration information for each source, see Source Configuration.

Source Object for Amazon S3 
{
  "entityType": "source",
  "config": {
    "accessKey": "AKIAIOSFODNN7EXAMPLE",
    "accessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "secure": true,
    "whitelistedBuckets": [
      "sales",
      "marketing"
    ],
    "rootPath": "/",
    "enableAsync": true,
    "compatibilityMode": false,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "requesterPays": false,
    "enableFileStatusCheck": true,
    "defaultCtasFormat": "ICEBERG",
    "isPartitionInferenceEnabled": false,
    "credentialType": "ACCESS_KEY"
  },  
  "state": {
    "status": "good", 
    "suggestedUserAction": "",
    "messages": []
  }, 
  "id": "4abb1367-5282-43e8-a0f6-54a8ae6428ed",
  "tag": "676e73e0-8b51-44d0-972d-ef80c29ac0c6",
  "type": "S3",
  "name": "source-name",
  "createdAt": "2025-07-26T19:56:28.826Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 3600000,
    "datasetRefreshAfterMs": 3600000,
    "datasetExpireAfterMs": 10800000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true,
    "autoPromoteDatasets": false
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "accelerationActivePolicyType": "NEVER",
  "accelerationRefreshSchedule": "",
  "children": [
    {
      "id": "dremio:/customer-records/sales",
      "path": [
          "customer-records",
          "sales"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    },
    {
      "id": "dremio:/customer-records/marketing",
      "path": [
          "customer-records",
          "marketing"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "sourceChangeState": "NONE", 
  "accessControlList": {
     "roles": [
       {
         "id": "1e5f38e4-8209-46dc-96f0-cfbd3276dbd8",
         "permissions": [
             "ALTER",
             "CREATE_TABLE",
             "DROP",
             "INSERT",
             "DELETE",
             "UPDATE",
             "TRUNCATE",
             "VIEW_REFLECTION",
             "ALTER_REFLECTION",
             "MODIFY",
             "MANAGE_GRANTS",
             "SELECT"
           ]
        }
    ],
    "users": [
      {
        "id": "4deaf0e4-f05c-441c-bb37-d4d6e43fdec7",
        "permissions": [
            "VIEW_REFLECTION",
            "SELECT"
        ]
      }
    ]
  },
  "permissions": [
    "READ",
    "WRITE",
    "ALTER_REFLECTION",
    "SELECT",
    "ALTER",
    "VIEW_REFLECTION",
    "MODIFY",
    "MANAGE_GRANTS",
    "CREATE_TABLE",
    "DROP",
    "EXTERNAL_QUERY",
    "INSERT",
    "TRUNCATE",
    "DELETE",
    "UPDATE",
    "EXECUTE",
    "CREATE_SOURCE",
    "ALL"
  ],
  "checkTableAuthorizer": false, 
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  },
  "accelerationRefreshOnDataChanges": false,
  "isPrimaryCatalog": false
}

Source Attributes​

entityType String

Specifies the type of container. For sources, the type is source.

config Object

Configuration options for the specified data source.

Example:

{
  "accessKey": "EXAMPLE78HT89VS4YJEL",
  "accessSecret": "$DREMIO_EXISTING_VALUE$",
  "secure": true,
  "rootPath": "/",
  "enableAsync": true,
  "compatibilityMode": false,
  "isCachingEnabled": true,
  "maxCacheSpacePct": 100,
  "requesterPays": false,
  "enableFileStatusCheck": true,
  "defaultCtasFormat": "ICEBERG",
  "isPartitionInferenceEnabled": false,
  "credentialType": "ACCESS_KEY"
}

id String (UUID)

UUID of the source.

tag String (UUID)

Unique identifier of a tag on the source. Dremio changes this tag whenever a change is made to the source. Immutable by the user.

Example: 676e73e0-8b51-44d0-972d-ef80c29ac0c6

type String

Identifies the source type. The source type determines the config parameters. See Attributes of the config Object for more information.

Valid Values: AWSGLUE, AZURESYNAPSE, BIGQUERY, DCS_DREMIO_CATALOG_V1, DB2, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, RESTCATALOG, S3, SAPHANA, SNOWFLAKE, SNOWFLAKEOPENCATALOG, UNITY, VERTICA

name String

The user-defined name of the source.

Example: customer-records

createdAt String

Timestamp when source was created, in ISO 8601 UTC format.

metadataPolicy Object

The policies covering the update of a source's metadata.

Example:

{
  "authTTLMs": 86400000,
  "namesRefreshMs": 3600000,
  "datasetRefreshAfterMs": 3600000,
  "datasetExpireAfterMs": 10800000,
  "datasetUpdateMode": "PREFETCH_QUERIED",
  "deleteUnavailableDatasets": true,
  "autoPromoteDatasets": false
}

accelerationGracePeriodMs Integer

Identifies the length of time to keep Reflections for all the datasets in a source before these Reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour. See Setting the Expiration Policy for Reflections for more information.

Example: 10800000

accelerationRefreshPeriodMs Integer

Identifies the refresh frequency for a dataset's Reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example: 3600000

accelerationNeverExpire Boolean

Option to set an expiration for Reflections. Default setting is false. Set to true to prevent Reflections from expiring and to override the accelerationGracePeriodMs setting.

accelerationNeverRefresh Boolean

Option to set a refresh for Reflections. Default setting is false. Set to true to prevent Reflections from refreshing and to override the accelerationRefreshPeriodMs setting.

accelerationActivePolicyType String

Option to set the policy for refreshing Reflections that are defined on the source. For this option to take effect, the accelerationNeverRefresh parameter must be set to false.

The possible values are:

  • NEVER: The Reflections are never refreshed.
  • PERIOD: The Reflections are refreshed at the end of every period that is defined by accelerationRefreshPeriodMs.
  • SCHEDULE: The Reflections are refreshed according to the schedule that is set by accelerationRefreshSchedule.

accelerationRefreshSchedule String

A cron expression that sets the schedule, in UTC time, according to which the Reflections that are defined on the source are refreshed.

Cron expressions consist of six fields that specify when a job should run:

Second  Minute  Hour  Day-of-Month  Month  Day-of-Week
  0      0-59   0-23      1-31       1-12    1-7 or SUN-SAT

The Second field is always set to 0 and cannot be modified.

Cron expressions support several special characters:

  • * (asterisk) – Matches all values in that field
    • Example: * in the Month field means "every month"
  • ? (question mark) – Wildcard for day fields (use in either Day-of-Month OR Day-of-Week, but not both)
    • Equivalent to "any day"
  • , (comma) – Specifies multiple values
    • Example: MON,WED,FRI means "Monday, Wednesday, and Friday"
  • - (hyphen) – Defines a range of values
    • Example: 2-6 in Day-of-Week means "Monday through Friday"

Examples:

  • 0 0 0 * * ?: Every day at midnight UTC
  • 0 45 15 * * 1,4,7: Sundays, Wednesdays, and Saturdays at 3:45 PM UTC
  • 0 15 7 ? * 2-6: Monday through Friday at 7:15 AM UTC
  • 0 0 */6 * * ?: Every 6 hours
  • 0 30 9 1 * ?: First day of every month at 9:30 AM UTC

Tips:

  • Remember that schedules run in UTC, so convert your local time accordingly
  • Use ? in Day-of-Month when specifying Day-of-Week, and vice versa
  • Day-of-Week uses 1=Sunday through 7=Saturday (or SUN-SAT)

children Array of Object

The catalog entities contained in the source, such as objects of type FILE, CONTAINER, and DATASET.

Example:

[ 
  { 
    "id": "dremio:/AWS-S3_testgroup/archive.dremio.com",
    "path": 
      [
        "AWS-S3_testgroup",
        "archive.dremio.com"
      ],
    "type": "CONTAINER",
    "containerType": "FOLDER"
  },
  {
    "id": "dremio:/AWS-S3_testgroup/logs_east-1",
    "path": 
      [
        "AWS-S3_testgroup",
        "logs_east-1"
      ],
    "type": "CONTAINER",
    "containerType": "FOLDER"
  }
]

allowCrossSourceSelection Boolean

For queries that can select from multiple sources, this option enables a source to be available in that query. Default setting is false. Set to true to enable this option.

disableMetadataValidityCheck Boolean

Option to set a validity check for the source's metadata. Default setting is false. Set to true to disable the validity check. This attribute is not supported by default. Contact Dremio Support to enable this option.

accessControlList Object

Information about users and roles with privileges on the source and the specific privileges each user or role has. May include a users array, a roles array, or both, depending on the configured access and privileges. The accessControlList object is empty if source-specific access control privileges are not set.

permissions Array of String

The permissions that the user submitting the API call has to a source. This will be an empty array unless the query parameter include=permissions is set. When set, the permission values that are available will be based on the source you are connecting to. See Privileges for more information.

checkTableAuthorizer Boolean

Not used. Has the value false.

owner Object

The owner of the source.

Example:

{
  "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
  "ownerType": "USER"
}

accelerationRefreshOnDataChanges Boolean

If Reflections automatically refresh for underlying tables that are in Iceberg format when new snapshots are created after an update, true. Otherwise, false.

isPrimaryCatalog Boolean

Not used. Has the value false.

Attributes of the config Object​

accessKey String

A unique identifier from your AWS account used in conjunction with a secret access key to sign API requests made to your data source.

Example: AKIAIOSFODNN7EXAMPLE

accessSecret String

A secret access key from your AWS account that is used in conjunction with an access key to cryptographically sign programmatic requests. Signing a request identifies the sender and prevents the request from being altered.

Example: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

secure Boolean

Option to enable SSL encryption for the source. Default setting is true. Set to false to disable SSL encryption.

whitelistedBuckets Array of String

A list of allowed Amazon S3 buckets.

Example:

[
  "sales",
  "marketing"
]

rootPath String

Identifies the root path of the S3 bucket.

Example: /

enableAsync Boolean

Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Default setting is true. Set to false to disable asynchronous access.

compatibilityMode Boolean

Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Default setting is false. Set to true to enable compatibility mode.

isCachingEnabled Boolean

Option to enable local caching. The default setting is true. Set to false to disable local caching.

maxCacheSpacePct Integer

Sets the maximum percent of total available cache space to use when this space is available. The minimum value is 1 and the maximum value is 100. The default value is 100.

Example: 100

requesterPays Boolean

Option to apply S3 requests to the requester. Default setting is false. Set to true to enable this option.

enableFileStatusCheck Boolean

Option to enable file status check. The default setting is true. Set to false to disable file status check.

defaultCtasFormat String

Sets the default format for tables that are created in Dremio.

Valid Values: ICEBERG or PARQUET

isPartitionInferenceEnabled Boolean

Option to enable partition column inference. The default setting is false. Set to true to enable partition column inference.

credentialType String

Specifies the type of credential to enable Dremio to connect to the source. For Amazon S3, the options are ACCESS_KEY or PROJECT_DATA_CRED.

Attributes of the metadataPolicy Object​

authTTLMs Integer

Sets the length of time, in milliseconds, that source permissions are cached. For example, if the default 24 hours (86400000 milliseconds) is used, then for each file you query, your permission status is checked once every 24 hours. The minimum supported time period is one minute (60000 milliseconds).

Example: 86400000

namesRefreshMs Integer

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 3600000

datasetRefreshAfterMs Integer

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example: 3600000

datasetExpireAfterMs Integer

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 10800000

datasetUpdateMode String

Sets the metadata policy for when a dataset is updated. Must be set to PREFETCH_QUERIED to update the details for previously queried objects in a source.

deleteUnavailableDatasets Boolean

Option to remove dataset definitions if the underlying data is unavailable to Dremio. The default setting is true. Set to false to keep the dataset definitions.

autoPromoteDatasets Boolean

Option to automatically format files into tables when a query is issued. The default setting is false. Set to true to enable this capability. This attribute applies only to metastore and object storage sources.

Attributes of Objects in the children Array​

id String

Unique identifier of the child catalog entity. May be a UUID or a fully-qualified path that uniquely identifies a folder within Dremio.

path Array of String

An array containing the path to the child object.

Example:

[
  "customer-records",
  "sales"
]

type String

The type of the child object:

  • CONTAINER: The id within the object is the unique identifier of a folder or user-defined function object.
  • DATASET: The id within the object is the unique identifier of a table or view object.
  • FILE: Unpromoted Parquet, CSV, JSON file object.

containerType String

This attribute is displayed only if type is set to CONTAINER. This attribute is automatically generated by Dremio and is immutable.

  • FOLDER: The object contains the unique identifier of a folder.
  • FUNCTION: The object contains the unique identifier for a user-defined function.

datasetType String

Identifies the dataset type. This attribute is displayed only if type is set to DATASET.

  • DIRECT: A table that is from an external source (such as PostgreSQL).
  • PROMOTED: A table that is promoted from a source.
  • VIRTUAL: A view object.

Attributes of the accessControlList Object​

roles Array of Object

The roles with privileges on the source and their privileges.

users Array of Object

The users with privileges on the source and their privileges.

Attributes of Objects in the roles Array​

id String (UUID)

UUID of the role.

permissions Array of String

The privileges that the role has on the source. See Privileges for more information.

Attributes of Objects in the users Array​

id String (UUID)

UUID of the user.

permissions Array of String

The privileges that the user has on the source. See Privileges for more information.

Attributes of the owner Object​

ownerId String (UUID)

UUID for the owner (user or role) of the source. This identifier corresponds to the user_id or role_id in the users or roles system table. See the Users system table for more information.

ownerType String

The type of the owner. Options are USER and ROLE.

Create a Source​

When you create a source that supports dataset discovery, metadata is initially populated in the catalog for each table that is accessible in the source. If the source contains a very large number of tables, this process may take longer than the request timeout that is configured by the Dremio gateway, and the response will contain a 502 or 504 response code. In this case, instead of retrying the request, we recommend that you poll the Retrieve a Source by Path endpoint to await the source's creation.

Method and URL 
POST  /v0/projects/{project_id}/catalog

Parameters​

project_id Path   String (UUID)

entityType Body   String

Specifies the type of container. For sources, the type is source.

config Body   Object

Configuration options for the specified data source.

type Body   String

Identifies the source type that you are configuring. The source type determines the config parameters that display. See Parameters of the config Object for more information.

Valid Values: AWSGLUE, AZURESYNAPSE, BIGQUERY, DCS_DREMIO_CATALOG_V1, DB2, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, RESTCATALOG, S3, SAPHANA, SNOWFLAKE, SNOWFLAKEOPENCATALOG, UNITY, VERTICA

name Body   String

The user-defined name of the source. The name cannot include the following special characters: /, :, [, or ].

Example: customer-records

metadataPolicy Body   Object

The policies covering the update of a source's metadata.

accelerationGracePeriodMs Body   Integer

Identifies the length of time to keep Reflections for all the datasets in a source before these Reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour. See Setting the Expiration Policy for Reflections for more information.

Example: 10800000

accelerationRefreshPeriodMs Body   Integer

Identifies the refresh frequency for a dataset's Reflections. Optional if you set accelerationActivePolicyType to PERIOD. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example: 3600000

accelerationActivePolicyType Body   String

Option to set the policy for refreshing Reflections that are defined on the source. For this option to take effect, the accelerationNeverRefresh parameter must be set to false.

The possible values are:

  • NEVER: The Reflections are never refreshed.
  • PERIOD: Default. The Reflections are refreshed at the end of every period that is defined by accelerationRefreshPeriodMs.
  • SCHEDULE: The Reflections are refreshed according to the schedule that is set by accelerationRefreshSchedule.

accelerationRefreshSchedule Body   String

A cron expression that sets the schedule, in UTC time, according to which the Reflections that are defined on the source are refreshed.

Cron expressions consist of six fields that specify when a job should run:

Second  Minute  Hour  Day-of-Month  Month  Day-of-Week
  0      0-59   0-23      1-31       1-12    1-7 or SUN-SAT

The Second field is always set to 0 and cannot be modified.

Cron expressions support several special characters:

  • * (asterisk) – Matches all values in that field
    • Example: * in the Month field means "every month"
  • ? (question mark) – Wildcard for day fields (use in either Day-of-Month OR Day-of-Week, but not both)
    • Equivalent to "any day"
  • , (comma) – Specifies multiple values
    • Example: MON,WED,FRI means "Monday, Wednesday, and Friday"
  • - (hyphen) – Defines a range of values
    • Example: 2-6 in Day-of-Week means "Monday through Friday"

Examples:

  • 0 0 0 * * ?: Every day at midnight UTC
  • 0 45 15 * * 1,4,7: Sundays, Wednesdays, and Saturdays at 3:45 PM UTC
  • 0 15 7 ? * 2-6: Monday through Friday at 7:15 AM UTC
  • 0 0 */6 * * ?: Every 6 hours
  • 0 30 9 1 * ?: First day of every month at 9:30 AM UTC

Tips:

  • Remember that schedules run in UTC, so convert your local time accordingly
  • Use ? in Day-of-Month when specifying Day-of-Week, and vice versa
  • Day-of-Week uses 1=Sunday through 7=Saturday (or SUN-SAT)

accelerationRefreshOnDataChanges Body   Boolean

To refresh Reflections on underlying tables that are in Iceberg format in the source when new snapshots are created after an update, true. Otherwise, false. Reflections that are automatically updated based on Iceberg source table changes also update according to the source-level policy as the minimum refresh frequency. For this option to take effect, the source must support Iceberg table format, the accelerationNeverRefresh parameter must be set to false, and the accelerationActivePolicyType parameter must be set to either PERIOD or SCHEDULE.

accessControlList Body   Object

Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both.

Parameters of the config Object​

hostname Body   String

The name of the server you are connecting to.

Example: 177.15.0.112

port Body   String

The port number of the server you are connecting to.

Example: 3306

authenticationType Body   String

The type of authentication needed to connect to the server.

  • ANONYMOUS: No authentication required
  • MASTER: Master credentials used in the connection.

netWriteTimeout Body   Integer

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example: 60

fetchSize Body   Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body   Integer

Sets the maximum number of idle connections that you want to keep.

Example: 8

idleTimeSec Body   Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

propertyList Body   Array of Object;Optional

An optional array of name/value pairs defining optional connection properties that are used by the source. Use a comma-separated list to specify multiple name/value pairs.

Example:

{
  "name": "fs.s3a.proxy.host",
  "value": "proxyHost.example.com"
}

Parameters of the metadataPolicy Object​

authTTLMs Body   Integer

Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 86400000

namesRefreshMs Body   Integer

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 86400000

datasetRefreshAfterMs Body   Integer

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example: 86400000

datasetExpireAfterMs Body   Integer

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 259200000

datasetUpdateMode Body   String

Sets the metadata policy for when a dataset is updated. Must be set to PREFETCH_QUERIED to update the details for previously queried objects in a source.

deleteUnavailableDatasets Body   Boolean

Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

Parameters of the accessControlList Object​

roles Body   Array of Object

The roles with privileges on the source and their privileges.

users Body   Array of Object

The users with privileges on the source and their privileges.

Parameters of Objects in the roles Array​

id Body   String (UUID)

UUID of a role.

permissions Body   Array of String

The privileges that the role should have on the source. See Privileges for more information.

Parameters of Objects in the users Array​

id Body   String (UUID)

UUID of a user.

permissions Body   Array of String

The privileges that the user should have on the source. See Privileges for more information.

Example​

Request 
curl -X POST "https://api.dremio.cloud/v0/projects/$PROJECT_ID/catalog" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "type": "MYSQL",
      "name": "recruitingdb",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accelerationActivePolicyType": "PERIOD",
      "accessControlList": {
        "roles": [
          {
            "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
            "permissions": []
          }
        ],
        "users": []
      }
    }'
Response 
{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2025-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationActivePolicyType": "PERIOD",
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
        "recruitingdb",
        "contacts"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {
    "roles": [
      {
        "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
        "permissions": []
      }
    ],
    "users": []
  },
  "permissions": [],
  "checkTableAuthorizer": false,
  "accelerationRefreshOnDataChanges": false,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Response Status Codes​

200   OK

400   Bad Request

401   Unauthorized

403   Forbidden

502   Bad Gateway

504   Gateway Timeout

Retrieve a Source by ID​

Method and URL 
GET /v0/projects/{project_id}/catalog/{id}

Parameters​

project_id Path   String (UUID)

id Path   String (UUID)

UUID for the source that you want to retrieve.

include Query   String   Optional

When using include, the name of a non-default response field is returned. You can retrieve either a list of access control list names (aclNames) or a list of permissions. See the include query parameter for usage examples.

Valid Values: aclNames, permissions

Example: ?include=permissions

exclude Query   String   Optional

When using exclude, the children field is excluded from the response. See the exclude query parameter for usage examples.

Example: ?exclude=children

maxChildren Query   Integer   Optional

Specify the maximum number of child objects to include in each page of results. Use in concert with the pageToken parameter to split large sets of results into multiple pages. For more information, see the maxChildren query parameter.

NOTE: The maxChildren query parameter is not supported for filesystem sources.

Example: ?maxChildren=25

pageToken Query   String   Optional

Specify the token for retrieving the next page of results. Must be used in concert with the maxChildren parameter – the first request URL includes maxChildren set to the maximum number of child objects to include in each page of results. If the source has more child objects than the specified maxChildren value, the response includes a nextPageToken attribute. Add the pageToken query parameter with the nextPageToken value to the request URL to retrieve the next page of results. Do not remove or change the maxChildren query parameter when you add pageToken to the request URL. See pageToken query parameter for more information.

NOTE: Dremio ignores the pageToken query parameter for filesystem sources.

Example: ?pageToken=cHAAFLceQCKsTVpwaEVisqgjDntZJUCuTqVNghPdkyBDUNoJvwrEXAMPLE

Example​

Request 
curl -X GET "https://api.dremio.cloud/v0/projects/$PROJECT_ID/catalog/$SOURCE_ID" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json'
Response 
{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2025-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationActivePolicyType": "PERIOD",
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
        "recruitingdb",
        "contacts"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {},
  "permissions": [],
  "checkTableAuthorizer": false,
  "accelerationRefreshOnDataChanges": false,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Response Status Codes​

200   OK

400   Bad Request

401   Unauthorized

403   Forbidden

404   Not Found

Retrieve a Source by Path​

Method and URL 
GET /v0/projects/{project_id}/catalog/by-path/{path}

Parameters​

project_id Path   String (UUID)

path Path   String

Source's location within Dremio, using forward slashes as separators. For example, for the source "Samples," the path is Samples. If the name of any component in the path includes special characters for URLs, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University in the URL path.

Example: recruitingdb

maxChildren Query   Integer   Optional

Specify the maximum number of child objects to include in each page of results. Use in concert with the pageToken parameter to split large sets of results into multiple pages. For more information, see the maxChildren query parameter.

NOTE: The maxChildren query parameter is not supported for filesystem sources.

Example: ?maxChildren=25

pageToken Query   String   Optional

Specify the token for retrieving the next page of results. Must be used in concert with the maxChildren parameter – the first request URL includes maxChildren set to the maximum number of child objects to include in each page of results. If the source has more child objects than the specified maxChildren value, the response includes a nextPageToken attribute. Add the pageToken query parameter with the nextPageToken value to the request URL to retrieve the next page of results. Do not remove or change the maxChildren query parameter when you add pageToken to the request URL. See the pageToken query parameter for more information.

NOTE: Dremio ignores the pageToken query parameter for filesystem sources.

Example: ?pageToken=cHAAFLceQCKsTVpwaEVisqgjDntZJUCuTqVNghPdkyBDUNoJvwrEXAMPLE

Example​

Request 
curl -X GET "https://api.dremio.cloud/v0/projects/$PROJECT_ID/catalog/by-path/recruitingdb" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json'
Response 
{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2025-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationActivePolicyType": "PERIOD",
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
        "recruitingdb",
        "contacts"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {},
  "permissions": [],
  "checkTableAuthorizer": false,
  "accelerationRefreshOnDataChanges": false,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Response Status Codes​

200   OK

400   Bad Request

401   Unauthorized

403   Forbidden

404   Not Found

Update a Source by ID​

Method and URL 
PUT /v0/projects/{project_id}/catalog/{id}

When you update a source that supports dataset discovery, metadata is initially populated in the catalog for each table that is accessible in the source. If the source contains a very large number of tables, this process may take longer than the request timeout that is configured by the Dremio gateway, and the response will contain a 502 or 504 response code. In this case, instead of retrying the request, we recommend that you poll the Retrieve a Source by Path endpoint to await the source's update. The source is updated when the value for the tag attribute in the response changes.

Parameters​

project_id Path   String (UUID)

id Path   String (UUID)

UUID of the source.

entityType Body   String

Specifies the type of container. For sources, the type is source.

config Body   Object

Configuration options for the specified data source.

type Body   String

Identifies the source type that you are configuring. The source type determines the config parameters that display. See Parameters of the config Object for more information.

Valid Values: AWSGLUE, BIGQUERY, DB2, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, RESTCATALOG, S3, SAPHANA, SNOWFLAKE, SNOWFLAKEOPENCATALOG, UNITY, VERTICA

name Body   String

The user-defined name of the source.

Example: recruitingdb

metadataPolicy Body   Object

The policies covering the update of a source's metadata.

accelerationGracePeriodMs Body   Integer

Identifies the length of time to keep Reflections for all the datasets in a source before these Reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour. See Setting the Expiration Policy for Reflections for more information.

Example: 10800000

accelerationRefreshPeriodMs Body   Integer

Identifies the refresh frequency for a dataset's Reflections. Optional if you set accelerationActivePolicyType to PERIOD. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example: 3600000

accelerationActivePolicyType Body   String

Option to set the policy for refreshing Reflections that are defined on the source. For this option to take effect, the accelerationNeverRefresh parameter must be set to false.

The possible values are:

  • NEVER: The Reflections are never refreshed.
  • PERIOD: The Reflections are refreshed at the end of every period that is defined by accelerationRefreshPeriodMs.
  • SCHEDULE: The Reflections are refreshed according to the schedule that is set by accelerationRefreshSchedule.

accelerationRefreshSchedule Body   String

A cron expression that sets the schedule, in UTC time, according to which the Reflections that are defined on the source are refreshed.

Cron expressions consist of six fields that specify when a job should run:

Second  Minute  Hour  Day-of-Month  Month  Day-of-Week
  0      0-59   0-23      1-31       1-12    1-7 or SUN-SAT

The Second field is always set to 0 and cannot be modified.

Cron expressions support several special characters:

  • * (asterisk) – Matches all values in that field
    • Example: * in the Month field means "every month"
  • ? (question mark) – Wildcard for day fields (use in either Day-of-Month OR Day-of-Week, but not both)
    • Equivalent to "any day"
  • , (comma) – Specifies multiple values
    • Example: MON,WED,FRI means "Monday, Wednesday, and Friday"
  • - (hyphen) – Defines a range of values
    • Example: 2-6 in Day-of-Week means "Monday through Friday"

Examples:

  • 0 0 0 * * ?: Every day at midnight UTC
  • 0 45 15 * * 1,4,7: Sundays, Wednesdays, and Saturdays at 3:45 PM UTC
  • 0 15 7 ? * 2-6: Monday through Friday at 7:15 AM UTC
  • 0 0 */6 * * ?: Every 6 hours
  • 0 30 9 1 * ?: First day of every month at 9:30 AM UTC

Tips:

  • Remember that schedules run in UTC, so convert your local time accordingly
  • Use ? in Day-of-Month when specifying Day-of-Week, and vice versa
  • Day-of-Week uses 1=Sunday through 7=Saturday (or SUN-SAT)

accelerationRefreshOnDataChanges Body   Boolean

To refresh Reflections on underlying tables that are in Iceberg format in the source when new snapshots are created after an update, true. Otherwise, false. Reflections that are automatically updated based on Iceberg source table changes also update according to the source-level policy as the minimum refresh frequency. For this option to take effect, the source must support Iceberg table format, the accelerationNeverRefresh parameter must be set to false, and the accelerationActivePolicyType parameter must be set to either PERIOD or SCHEDULE.

accessControlList Body   Object   Optional

Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both.

Parameters of the config Object​

hostname Body   String

The name of the server you are connecting to.

Example: 177.15.0.112

port Body   String

The port number of the server you are connecting to.

Example: 3306

authenticationType Body   String

The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user are required.

  • ANONYMOUS: No authentication required.
  • MASTER: Master credentials used in the connection.

netWriteTimeout Body   Integer

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example: 60

fetchSize Body   Integer

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example: 200

maxIdleConns Body   Integer

Sets the maximum number of idle connections that you want to keep.

Example: 8

idleTimeSec Body   Integer

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example: 60

propertyList Body   Array of Object   Optional

An array of name/value pairs defining optional connection properties that are used by the source. Use a comma-separated list to specify multiple name/value pairs.

Example:

{
  "name": "fs.s3a.proxy.host",
  "value": "proxyHost.example.com"
}

Parameters of the metadataPolicy Object​

authTTLMs Body   Integer

Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 86400000

namesRefreshMs Body   Integer

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 86400000

datasetRefreshAfterMs Body   Integer

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example: 86400000

datasetExpireAfterMs Body   Integer

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example: 259200000

datasetUpdateMode Body   String

Sets the metadata policy for when a dataset is updated. Must be set to PREFETCH_QUERIED to update the details for previously queried objects in a source.

deleteUnavailableDatasets Body   Boolean

Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

Parameters of the accessControlList Object​

roles Body   Array of Object

The roles that should have privileges on the source and the privileges each role should have.

users Body   Array of Object

The users that should have privileges on the source and the privileges each user should have.

Parameters of Objects in the roles Array​

id Body   String (UUID)

UUID of the role.

permissions Body   Array of String

The privileges that the role should have on the source. See Privileges for more information.

Parameters of Objects in the users Array​

id Body   String (UUID)

UUID of the user.

permissions Body   Array of String

The privileges that the user should have on the source. See Privileges for more information.

Example​

Request 
curl -X PUT "https://api.dremio.cloud/v0/projects/$PROJECT_ID/catalog/$SOURCE_ID" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "type": "MYSQL",
      "name": "recruitingdb",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accelerationActivePolicyType": "PERIOD",
      "accelerationRefreshOnDataChanges": true,
      "accessControlList": {
        "roles": [
          {
            "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
            "permissions": []
          }
        ],
        "users": []
      }
    }'
Response 
{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2025-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationActivePolicyType": "PERIOD",
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
        "recruitingdb",
        "contacts"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {
    "roles": [
      {
        "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
        "permissions": []
      }
    ],
    "users": []
  },
  "permissions": [],
  "checkTableAuthorizer": false,
  "accelerationRefreshOnDataChanges": true,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Response Status Codes​

200   OK

400   Bad Request

401   Unauthorized

403   Forbidden

404   Not Found

502   Bad Gateway

504   Gateway Timeout

Delete a Source​

Method and URL 
DELETE /v0/projects/{project_id}/catalog/{id}

Parameters​

project_id Path   String (UUID)

id Path   String (UUID)

UUID for the source.

Example​

Example Request 
curl -X DELETE "https://api.dremio.cloud/v0/projects/$PROJECT_ID/catalog/$SOURCE_ID" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json'

A successful request returns an empty response body with HTTP status 204 No Content.

Response Status Codes​

204   No Content

400   Bad Request

401   Unauthorized

403   Forbidden

404   Not Found