Source

The Source API enables you to manage the sources that are supported in Dremio. You can add, retrieve, update, and delete sources. Sources are of entityType: source.

The primary Arctic catalog is created when you add a Sonar project and provides data management capabilities for your project. The primary Arctic catalog is considered a source entity within the API.

note

The source object contains the same attributes for all source types except for the config object, which is different for each source type. The examples on this page use an Amazon S3 source to demonstrate the available endpoints. For configuration information for each source, see Source Configuration.

Source Object for Amazon S3

{
  "entityType": "source",
  "config": {
    "accessKey": "AKIAIOSFODNN7EXAMPLE",
    "accessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "secure": true,
    "whitelistedBuckets": [
        "sales",
        "marketing"
    ],
    "rootPath": "/",
    "enableAsync": true,
    "compatibilityMode": false,
    "isCachingEnabled": true,
    "maxCacheSpacePct": 100,
    "requesterPays": false,
    "enableFileStatusCheck": true,
    "defaultCtasFormat": "ICEBERG",
    "isPartitionInferenceEnabled": false,
    "credentialType": "ACCESS_KEY"
  },  
  "id": "4abb1367-5282-43e8-a0f6-54a8ae6428ed",
  "tag": "676e73e0-8b51-44d0-972d-ef80c29ac0c6",
  "type": "S3",
  "name": "source-name",
  "createdAt": "2022-07-26T19:56:28.826Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 3600000,
    "datasetRefreshAfterMs": 3600000,
    "datasetExpireAfterMs": 10800000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true,
    "autoPromoteDatasets": false
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "dremio:/customer-records/sales",
      "path": [
          "customer-records",
          "sales"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
  },
  {
      "id": "dremio:/customer-records/marketing",
      "path": [
          "customer-records",
          "marketing"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
  },
],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {
     "roles": [
       {
         "id": "1e5f38e4-8209-46dc-96f0-cfbd3276dbd8",
         "permissions": [
             "ALTER",
             "CREATE_TABLE",
             "DROP",
             "INSERT",
             "DELETE",
             "UPDATE",
             "TRUNCATE",
             "VIEW_REFLECTION",
             "ALTER_REFLECTION",
             "MODIFY",
             "MANAGE_GRANTS",
             "SELECT"
           ]
        }
    ],
    "users": [
      {
        "id": "4deaf0e4-f05c-441c-bb37-d4d6e43fdec7",
        "permissions": [
            "VIEW_REFLECTION",
            "SELECT"
        ]
      }
    ]
  },
  "permissions": [
    "READ",
    "WRITE",
    "ALTER_REFLECTION",
    "SELECT",
    "ALTER",
    "VIEW_REFLECTION",
    "MODIFY",
    "MANAGE_GRANTS",
    "CREATE_TABLE",
    "DROP",
    "EXTERNAL_QUERY",
    "INSERT",
    "TRUNCATE",
    "DELETE",
    "UPDATE",
    "EXECUTE",
    "CREATE_SOURCE",
    "ALL"
  ],
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Source Attributes

Specifies the type of container. For sources, the type is source.

Configuration options for the specified data source.

The unique identifier that is generated to identify a source.

Example 4abb1367-5282-43e8-a0f6-54a8ae6428ed

Identifies the instance of the source. Dremio changes this tag whenever a change is made to the source. Immutable by the user.

Example 676e73e0-8b51-44d0-972d-ef80c29ac0c6

Identifies the source type that you are configuring. The source type determines the config parameters that display.

For more information, read Configuration Attributes.

Enum ARCTIC, AWSGLUE, AZURE_STORAGE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE

The user-defined name of the source.

Example customer-records

The date and time that the source was created.

Example 2022-07-26T19:56:28.826Z

The policies covering the update of a source’s metadata.

Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour.

For more information, read Setting the Expiration Policy for Reflections.

Example 10800000

Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example 3600000

Option to set an expiration for accelerations. Default setting is false. Set to true to prevent accelerations from expiring and to override the accelerationGracePeriodMs setting.

Option to set a refresh for accelerations. Default setting is false. Set to true to prevent accelerations from refreshing and to override the accelerationRefreshPeriodMs setting.

The catalog entities (including datasets, files, and folders) contained in the source.

For queries that can select from multiple sources, this option enables a source to be available in that query. Default setting is false. Set to true to enable this option.

Option to set a validity check for the source's metadata. Default setting is false. Set to true to disable the validity check.

Note: This attribute is not supported by default. Contact Dremio Support to enable the option.

Information about users and roles with privileges on the source and the specific privileges each user or role has. May include a users array, a roles array, or both, depending on the configured access and privileges. The accessControlList object is empty if source-specific access control privileges are not set.

Note: For Arctic sources and the primary Arctic catalog, the source object does not include the accessControlList object.

The permissions that the user submitting the API call has to a source. This will be an empty array, unless the query parameter include=permissions is set. When set, the permission values that are available will be based on the source you are connecting to.

For more information, read Privileges.

The owner of the source.

config

A unique identifier from your AWS account used in conjunction with a secret access key to sign API requests made to your data source.

Example AKIAIOSFODNN7EXAMPLE

A secret access key from your AWS account that is used in conjunction with an access key to cryptographically sign programmatic requests. Signing a request identifies the sender and prevents the request from being altered.

Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Option to enable SSL encryption for the source. Default setting is true. Set to false to disable SSL encryption.

A list of allowed Amazon S3 buckets.

Example sales, marketing

Identifies the root path of the S3 bucket.

Example /

Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Default setting is true. Set to false to disable asynchronous access.

Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Default setting is false. Set to true to enable compatibility mode.

Option to enable local caching. Default setting is true. Set to false to disable local caching.

Sets the maximum percent of total available cache space to use, when this space is available. The minimum value is 1 and the maximum value is 100. The default value is 100.

Example 100

Option to apply S3 requests to the requestor. Default setting is false. Set to true to enable this option.

Option to enable file status check. Default setting is true. Set to false to disable file status check.

Sets the default format for tables that are created in Dremio. The default format that is used is ICEBERG.

Enum ICEBERG, PARQUET

Option to enable partition column inference. The default setting is false. Set to true to enable partition column inference.

Specifies the type of credential to enable Dremio to connect to the source.

Enum ACCESS_KEY, PROJECT_DATA_CRED

metadataPolicy

Sets the length of time, in milliseconds, that source permissions are cached. For example, if the default 24 hours (86400000 milliseconds) is used, then for each file you query, your permission status is checked once every 24 hours. The minimum supported time period is one minute (60000 milliseconds).

Example 86400000

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 3600000

Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

Example 3600000

Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 10800000

Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

Enum PREFETCH_QUERIED

Option to remove dataset definitions if the underlying data is unavailable to Dremio. The default setting is true. Set to false to keep the dataset definitions.

Option to automatically format files into tables when a query is issued. The default setting is false. Set to true to enable this capability.

Note: This attribute applies only to metastore and object storage sources.

children

The catalog entity ID, which is generated by Dremio and is immutable.

Example dremio:/customer-records/sales

The catalog entity path, which is immutable.

Example customer-records

The type of the object that is contained in the source. This typing is generated by Dremio and is immutable. For folders, the type is CONTAINER. For datasets, the type is DATASET. For Parquet and other file formats, the type is FILE.

Enum CONTAINER, DATASET, FILE

Identifies the container type. This attribute is displayed only if type is set to CONTAINER. This attribute is automatically generated by Dremio and is immutable.

Enum FOLDER

Identifies the dataset type. This attribute is displayed only if type is set to DATASET. The dataset can be one of the following types:
• DIRECT: A dataset that is from an external source (such as PostgreSQL).
• PROMOTED: A table that is promoted from a source.
• VIRTUAL: A view that is created on another dataset.

This attribute is automatically generated by Dremio and is immutable.

Enum DIRECT, PROMOTED, VIRTUAL

accessControlList

Information about the roles that have been granted privileges on the source and the privileges each role has.

Information about the users that have been granted privileges on the source and the privileges each user has.

roles

The ID of the role.

Example 1e5f38e4-8209-46dc-96f0-cfbd3276dbd8

The privileges that the role has on the source.

For more information, read Privileges.

users

The ID of the user.

Example 4deaf0e4-f05c-441c-bb37-d4d6e43fdec7

The privileges that the user has on the source.

For more information, read Privileges.

owner

The unique identifier for the owner (user or role) of the source. This identifier corresponds to the user_id or role_id in the users or roles system table.

For more information, read the Users system table.

Example 5c9c394d-7395-4687-97dd-beac4384c359

The type of the owner.

Enum USER, ROLE

Adding a Source

Add a source.

Adding a Source

POST  /v0/projects/{project-id}/catalog

Parameters

The unique identifier for the project that contains the source that you want to add.

Example 02d36975-73eb-47ed-9bb5-de73060380f6

Specifies the type of container. For sources, the type is source.

Configuration options for the specified data source.

Identifies the source type that you are configuring. The source type determines the config parameters that display.

For more information, read Configuration Attributes.

Enum ARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE

The user-defined name of the source.

Example customer-records

The policies covering the update of a source’s metadata.

For more information, read Setting the Expiration Policy for Reflections.

Example 10800000

Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example 3600000

Object used to specify which users and roles should have privileges on the source and the specific privileges each user and role has. May include a users array, a roles array, or both. Not supported for Arctic sources and the primary Arctic catalog.

config

The name of the server you are connecting to.

Example 177.15.0.112

The port number of the server you are connecting to.

Example 3306

The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

Enum ANONYMOUS, MASTER

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example 60

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.

metadataPolicy

Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000

Example 259200000

Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

Enum PREFETCH_QUERIED

Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

accessControlList

The roles that should have privileges on the source and the privileges each role should have.

The users that should have privileges on the source and the privileges each user should have.

roles

The ID of the role.

Example 9ab42853-bdef-465f-b9bb-381a13a9bf78

The privileges that the role should have on the source.

For more information, read Privileges.

users

The ID of the user.

The privileges that the user should have on the source.

For more information, read Privileges.

Example Request

curl -X POST 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json' \
-d '{
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "type": "MYSQL",
      "name": "recruitingdb",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accessControlList": {
        "roles": [
          {
            "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
            "permissions": []
          }
        ],
        "users": []
      }
    }'

Example Response

{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2022-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
      {
        "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
        "path": [
            "recruitingdb",
            "contacts"
          ],
          "type": "CONTAINER",
          "containerType": "FOLDER"
      }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {
      "roles": [
        {
            "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
            "permissions": []
        }
      ],
      "users": []
  },
  "permissions": [],
  "checkTableAuthorizer": false,
  "owner": {
      "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
      "ownerType": "USER"
  }
}

Responses

Retrieving a Source by ID

Get information about the source using the source ID.

Retrieving a Source by ID

GET /v0/projects/{project-id}/catalog/{id}

Parameters

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6

The unique identifier for the source that you want to retrieve.

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0

When using include, the name of a non-default response field is returned. You can retrieve either a list of access control list names - aclNames - or a list of permissions. Read Include and Exclude Query Parameters for usage examples.

Enum aclNames, permissions

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?include=permissions

When using exclude, the children field is excluded from the response. Read Include and Exclude Query Parameters for usage examples.

Enum children

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?exclude=children

Example Request

curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2022-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
        "recruitingdb",
        "contacts"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {},
  "permissions": [],
  "checkTableAuthorizer": false,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Responses

Retrieving a Source by Path

Get information about the source using the source's path.

Retrieving a Source by Path

GET /v0/projects/{project-id}/catalog/by-path/{path}

Parameters

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6

Source's location within Dremio, using forward slashes as separators. For example, for the source "Samples," the path is Samples. If the name of any component in the path includes special characters for URLs, such as spaces, use URL encoding to replace the special characters with their UTF-8-equivalent characters. For example, "Dremio University" should be Dremio%20University in the URL path.

Example recruitingdb

Example Request

curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/by-path/recruitingdb' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2022-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
        "recruitingdb",
        "contacts"
      ],
      "type": "CONTAINER",
      "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {},
  "permissions": [],
  "checkTableAuthorizer": false,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Responses

Updating a Source

Edit the source information using the source ID.

Updating a Source

PUT /v0/projects/{project-id}/catalog/{id}

Parameters

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6

The unique identifier for the source that you want to update.

Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0

Specifies the type of container. For sources, the type is source.

Configuration options for the specified data source.

Identifies the source type that you are configuring. The source type determines the config parameters that display.

For more information, read Configuration Attributes.

Enum ARCTIC, AWSGLUE, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, S3, SNOWFLAKE

The user-defined name of the source.

Example recruitingdb

The policies covering the update of a source’s metadata.

For more information, read Setting the Expiration Policy for Reflections.

Example 10800000

Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

Example 3600000

config

The name of the server you are connecting to.

Example 177.15.0.112

The port number of the server you are connecting to.

Example 3306

The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

Enum ANONYMOUS, MASTER

The amount of time, in seconds, to wait for data from the server before aborting the connection.

Example 60

The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

Example 200

Sets the maximum number of idle connections that you want to keep.

Example 8

Sets the idle time, in seconds, before a connection is evaluated for closure.

Example 60

An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.

metadataPolicy

Example 86400000

Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

Example 86400000

Example 259200000

Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

Enum PREFETCH_QUERIED

accessControlList

The roles that should have privileges on the source and the privileges each role should have.

The users that should have privileges on the source and the privileges each user should have.

roles

The ID of the role.

Example 9ab42853-bdef-465f-b9bb-381a13a9bf78

The privileges that the role should have on the source.

For more information, read Privileges.

users

The ID of the user.

The privileges that the user should have on the source.

For more information, read Privileges.

Example Request

curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'
-d '{
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "type": "MYSQL",
      "name": "recruitingdb",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accessControlList": {
        "roles": [
          {
            "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
            "permissions": []
          }
        ],
        "users": []
      }
    }'

Example Response

{
  "entityType": "source",
  "config": {
    "hostname": "177.15.0.112",
    "port": "3306",
    "authenticationType": "ANONYMOUS",
    "netWriteTimeout": 60,
    "fetchSize": 200,
    "maxIdleConns": 8,
    "idleTimeSec": 60,
    "propertyList": []
  },
  "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
  "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
  "type": "MYSQL",
  "name": "recruitingdb",
  "createdAt": "2022-09-12T07:22:19.334Z",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 86400000,
    "datasetRefreshAfterMs": 86400000,
    "datasetExpireAfterMs": 259200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": true
  },
  "accelerationGracePeriodMs": 10800000,
  "accelerationRefreshPeriodMs": 3600000,
  "accelerationNeverExpire": false,
  "accelerationNeverRefresh": false,
  "children": [
    {
      "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
      "path": [
          "recruitingdb",
          "contacts"
        ],
        "type": "CONTAINER",
        "containerType": "FOLDER"
    }
  ],
  "allowCrossSourceSelection": false,
  "disableMetadataValidityCheck": false,
  "accessControlList": {
    "roles": [
      {
          "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
          "permissions": []
      }
    ],
    "users": []
  },
  "permissions": [],
  "checkTableAuthorizer": false,
  "owner": {
    "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
    "ownerType": "USER"
  }
}

Responses

Deleting a Source

Delete a source.

note

Dremio does not support deleting the primary Arctic catalog.

Deleting a Source

DELETE /v0/projects/{project-id}/catalog/{id}

Parameters

The unique identifier for the project that contains the source.

Example 02d36975-73eb-47ed-9bb5-de73060380f6

The unique identifier for the source that you want to delete. Deletion is not supported for the primary Arctic catalog.

Example 21a74cb1-3fcf-48c6-8f69-94c257ec3da3

Example Request

curl -X DELETE 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/21a74cb1-3fcf-48c6-8f69-94c257ec3da3' \
-H 'Authorization: Bearer <personal access token>' \
-H 'Content-Type: application/json'

Example Response

No response

Source

Source Attributes​

config​

metadataPolicy​

children​

accessControlList​

roles​

users​

owner​

Adding a Source​

Parameters​

config​

metadataPolicy​

accessControlList​

roles​

users​

Responses​

Retrieving a Source by ID​

Parameters​

Responses​

Retrieving a Source by Path​

Parameters​

Responses​

Updating a Source​

Parameters​

config​

metadataPolicy​

accessControlList​

roles​

users​

Responses​

Deleting a Source​

Parameters​

Responses​

Source Attributes

config

metadataPolicy

children

accessControlList

roles

users

owner

Adding a Source

Parameters

config

metadataPolicy

accessControlList

roles

users

Responses

Retrieving a Source by ID

Parameters

Responses

Retrieving a Source by Path

Parameters

Responses

Updating a Source

Parameters

config

metadataPolicy

accessControlList

roles

users

Responses

Deleting a Source

Parameters

Responses