On this page

    Source

    The Source API enables you to manage the sources that are supported in Dremio. You can add, retrieve, update, and delete sources. Sources are of entityType: source.

    note:

    The source object contains the same attributes for all source types except for the config object, which is different for each source type. The examples on this page use an Amazon S3 source to demonstrate the available endpoints. For configuration information for each source, see Configuration Attributes.

    Source Object for Amazon S3
    {
      "entityType": "source",
      "config": {
        "accessKey": "AKIAIOSFODNN7EXAMPLE",
        "accessSecret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
        "secure": true,
        "whitelistedBuckets": [
            "sales",
            "marketing"
        ],
        "rootPath": "/",
        "enableAsync": true,
        "compatibilityMode": false,
        "isCachingEnabled": true,
        "maxCacheSpacePct": 100,
        "requesterPays": false,
        "enableFileStatusCheck": true,
        "defaultCtasFormat": "ICEBERG",
        "isPartitionInferenceEnabled": false,
        "credentialType": "ACCESS_KEY"
      },  
      "id": "4abb1367-5282-43e8-a0f6-54a8ae6428ed",
      "tag": "676e73e0-8b51-44d0-972d-ef80c29ac0c6",
      "type": "S3",
      "name": "source-name",
      "createdAt": "2022-07-26T19:56:28.826Z",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 3600000,
        "datasetRefreshAfterMs": 3600000,
        "datasetExpireAfterMs": 10800000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true,
        "autoPromoteDatasets": false
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accelerationNeverExpire": false,
      "accelerationNeverRefresh": false,
      "children": [
        {
          "id": "dremio:/customer-records/sales",
          "path": [
              "customer-records",
              "sales"
          ],
          "type": "CONTAINER",
          "containerType": "FOLDER"
      },
      {
          "id": "dremio:/customer-records/marketing",
          "path": [
              "customer-records",
              "marketing"
          ],
          "type": "CONTAINER",
          "containerType": "FOLDER"
      },
    ],
      "allowCrossSourceSelection": false,
      "disableMetadataValidityCheck": false,
      "accessControlList": {
         "roles": [
           {
             "id": "1e5f38e4-8209-46dc-96f0-cfbd3276dbd8",
             "permissions": [
                 "ALTER",
                 "CREATE_TABLE",
                 "DROP",
                 "INSERT",
                 "DELETE",
                 "UPDATE",
                 "TRUNCATE",
                 "VIEW_REFLECTION",
                 "ALTER_REFLECTION",
                 "MODIFY",
                 "MANAGE_GRANTS",
                 "SELECT"
               ]
            }
        ],
        "users": [
          {
            "id": "4deaf0e4-f05c-441c-bb37-d4d6e43fdec7",
            "permissions": [
                "VIEW_REFLECTION",
                "SELECT"
            ]
          }
        ]
      },
      "permissions": [
        "READ",
        "WRITE",
        "ALTER_REFLECTION",
        "SELECT",
        "ALTER",
        "VIEW_REFLECTION",
        "MODIFY",
        "MANAGE_GRANTS",
        "CREATE_TABLE",
        "DROP",
        "EXTERNAL_QUERY",
        "INSERT",
        "TRUNCATE",
        "DELETE",
        "UPDATE",
        "EXECUTE",
        "CREATE_SOURCE",
        "ALL"
      ],
      "owner": {
        "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
        "ownerType": "USER"
      }
    }
    

    Source Attributes

    entityType

    String

    Specifies the type of container. For sources, the type is source.


    config

    Object

    Configuration options for the specified data source.


    id

    String (UUID)

    The unique identifier that is generated to identify a source.

    Example 4abb1367-5282-43e8-a0f6-54a8ae6428ed


    tag

    String (UUID)

    Identifies the instance of the source. Dremio changes this tag whenever a change is made to the source. Immutable by the user.

    Example 676e73e0-8b51-44d0-972d-ef80c29ac0c6


    type

    String

    Identifies the source type that you are configuring. The source type determines the config parameters that display. For more information, read Configuration Attributes.

    Enum ARCTIC , AWSGLUE , MSSQL , MYSQL , ORACLE , POSTGRES , REDSHIFT , S3 , SNOWFLAKE


    name

    String

    The user-defined name of the source.

    Example customer-records


    createdAt

    String

    The date and time that the source was created.

    Example 2022-07-26T19:56:28.826Z


    metadataPolicy

    Object

    The policies covering the update of a source’s metadata.


    accelerationGracePeriodMs

    Number

    Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour. For more information, read Setting the Expiration Policy for Reflections.

    Example 10800000


    accelerationRefreshPeriodMs

    Number

    Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

    Example 3600000


    accelerationNeverExpire

    Boolean

    Option to set an expiration for accelerations. Default setting is false. Set to true to prevent accelerations from expiring and to override the accelerationGracePeriodMs setting.


    accelerationNeverRefresh

    Boolean

    Option to set a refresh for accelerations. Default setting is false. Set to true to prevent accelerations from refreshing and to override the accelerationRefreshPeriodMs setting.


    children

    Object

    The catalog entities (including datasets, files, and folders) contained in the source.


    allowCrossSourceSelection

    Boolean

    For queries that can select from multiple sources, this option enables a source to be available in that query. Default setting is false. Set to true to enable this option.


    disableMetadataValidityCheck

    Boolean

    Option to set a validity check for the source's metadata. Default setting is false. Set to true to disable the validity check.

    Note:

    This attribute is not supported by default. Contact Dremio Support to enable the option.


    accessControlList

    Object

    List of permissions that is set for a source.


    permissions

    Array

    The permissions that the user submitting the API call has to a source. This will be an empty array, unless the query parameter include=permissions is set. When set, the permission values that are available will be based on the source you are connecting to. For more information, read All Supported Privileges.


    owner

    Object

    The owner of the source.

    config

    accessKey

    String

    A unique identifier from your AWS account used in conjunction with a secret access key to sign API requests made to your data source.

    Example AKIAIOSFODNN7EXAMPLE


    accessSecret

    String

    A secret access key from your AWS account that is used in conjunction with an access key to cryptographically sign programmatic requests. Signing a request identifies the sender and prevents the request from being altered.

    Example wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY


    secure

    Boolean

    Option to enable SSL encryption for the source. Default setting is true. Set to false to disable SSL encryption.


    whitelistedBuckets

    Array

    A list of allowed Amazon S3 buckets.

    Example sales, marketing


    rootPath

    String

    Identifies the root path of the S3 bucket.

    Example /


    enableAsync

    Boolean

    Option to enable asynchronous access, which allows cloud caching for the source to support simultaneous actions such as adding and editing. Default setting is true. Set to false to disable asynchronous access.


    compatibilityMode

    Boolean

    Option to enable compatibility mode for an Amazon S3 source, which allows you to use S3-compatible storage. Default setting is false. Set to true to enable compatibility mode.


    isCachingEnabled

    Boolean

    Option to enable local caching. Default setting is true. Set to false to disable local caching.


    maxCacheSpacePct

    Number

    Sets the maximum percent of total available cache space to use, when this space is available. The minimum value is 1 and the maximum value is 100. The default value is 100.

    Example 100


    requesterPays

    Boolean

    Option to apply S3 requests to the requestor. Default setting is false. Set to true to enable this option.


    enableFileStatusCheck

    Boolean

    Option to enable file status check. Default setting is true. Set to false to disable file status check.


    defaultCtasFormat

    String

    Sets the default format for tables that are created in Dremio. The default format that is used is ICEBERG.

    Enum ICEBERG , PARQUET


    isPartitionInferenceEnabled

    Boolean

    Option to enable partition column inference. The default setting is false. Set to true to enable partition column inference.


    credentialType

    String

    Specifies the type of credential to enable Dremio to connect to the source.

    Enum ACCESS_KEY , PROJECT_DATA_CRED

    metadataPolicy

    authTTLMs

    Number

    Sets the length of time, in milliseconds, that source permissions are cached. For example, if the default 24 hours (86400000 milliseconds) is used, then for each file you query, your permission status is checked once every 24 hours. The minimum supported time period is one minute (60000 milliseconds).

    Example 86400000


    namesRefreshMs

    Number

    Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 3600000


    datasetRefreshAfterMs

    Number

    Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

    Example 3600000


    datasetExpireAfterMs

    Number

    Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 10800000


    datasetUpdateMode

    String

    Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

    Enum PREFETCH_QUERIED


    deleteUnavailableDatasets

    Boolean

    Option to remove dataset definitions if the underlying data is unavailable to Dremio. The default setting is true. Set to false to keep the dataset definitions.


    autoPromoteDatasets

    Boolean

    Option to automatically format files into tables when a query is issued. The default setting is false. Set to true to enable this capability.

    Note:

    This attribute applies only to Metastores and Object Storage sources.

    children

    id

    String

    The catalog entity ID, which is generated by Dremio and is immutable.

    Example dremio:/customer-records/sales


    path

    String

    The catalog entity path, which is immutable.

    Example customer-records


    type

    String

    The type of the object that is contained in the source. This typing is generated by Dremio and is immutable. For folders, the type is CONTAINER. For datasets, the type is DATASET. For Parquet and other file formats, the type is FILE.

    Enum CONTAINER , DATASET , FILE


    containerType

    String

    Identifies the container type. This attribute is displayed only if type is set to CONTAINER. This attribute is automatically generated by Dremio and is immutable.

    Enum FOLDER


    datasetType

    String

    Identifies the dataset type. This attribute is displayed only if type is set to DATASET. The dataset can be one of the following types:

    • DIRECT: A dataset that is from an external source (such as PostgreSQL).

    • PROMOTED: A table that is promoted from a source.

    • VIRTUAL: A view that is created on another dataset.

    This attribute is automatically generated by Dremio and is immutable.

    Enum DIRECT , PROMOTED , VIRTUAL

    accessControlList

    roles

    Array

    The roles that have been granted privileges to a source.


    users

    Array

    The users that have been granted privileges to a source.

    roles

    id

    String

    The ID of the role.

    Example 1e5f38e4-8209-46dc-96f0-cfbd3276dbd8


    permissions

    Array

    The permissions that the role has to a source. For more information, read All Supported Privileges.

    users

    id

    String

    The ID of the user.

    Example 4deaf0e4-f05c-441c-bb37-d4d6e43fdec7


    permissions

    Array

    The permissions that the user has to a source. For more information, read All Supported Privileges.

    owner

    ownerId

    String (UUID)

    The unique identifier for the owner (user or role) of the source. This identifier corresponds to the user_id or role_id in the users or roles system table. For more information, read the Users system table.

    Example 5c9c394d-7395-4687-97dd-beac4384c359


    ownerType

    String

    The type of the owner.

    Enum USER , ROLE

    Adding a Source

    Add a source.

    Adding a Source
    POST  /v0/projects/{project-id}/catalog
    

    Parameters

    project-id

    path

    String (UUID)

    The unique identifier for the project that contains the source that you want to add.

    Example 02d36975-73eb-47ed-9bb5-de73060380f6


    entityType

    String

    Specifies the type of container. For sources, the type is source.


    config

    Object

    Configuration options for the specified data source.


    type

    String

    Identifies the source type that you are configuring. The source type determines the config parameters that display. For more information, read Configuration Attributes.

    Enum ARCTIC , AWSGLUE , MSSQL , MYSQL , ORACLE , POSTGRES , REDSHIFT , S3 , SNOWFLAKE


    name

    String

    The user-defined name of the source.

    Example customer-records


    metadataPolicy

    Source Metadata

    The policies covering the update of a source’s metadata.


    accelerationGracePeriodMs

    Number

    Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour. For more information, read Setting the Expiration Policy for Reflections.

    Example 10800000


    accelerationRefreshPeriodMs

    Number

    Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

    Example 3600000


    accessControlList

    Object

    List of permissions that is set for a source.

    config

    hostname

    String

    The name of the server you are connecting to.

    Example 177.15.0.112


    port

    String

    The port number of the server you are connecting to.

    Example 3306


    authenticationType

    String

    The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

    Enum ANONYMOUS , MASTER


    netWriteTimeout

    Number

    The amount of time, in seconds, to wait for data from the server before aborting the connection.

    Example 60


    fetchSize

    Number

    The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

    Example 200


    maxIdleConns

    Number

    Sets the maximum number of idle connections that you want to keep.

    Example 8


    idleTimeSec

    Number

    Sets the idle time, in seconds, before a connection is evaluated for closure.

    Example 60


    propertyList

    Array

    An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.

    metadataPolicy

    authTTLMs

    Number

    Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 86400000


    namesRefreshMs

    Number

    Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 86400000


    datasetRefreshAfterMs

    Number

    Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

    Example 86400000


    datasetExpireAfterMs

    Number

    Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 259200000


    datasetUpdateMode

    String

    Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

    Enum PREFETCH_QUERIED


    deleteUnavailableDatasets

    Boolean

    Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

    accessControlList

    roles

    Array

    The roles that have been granted privileges to a source.


    users

    Array

    The users that have been granted privileges to a source.

    roles

    id

    String

    The ID of the role.

    Example 9ab42853-bdef-465f-b9bb-381a13a9bf78


    permissions

    Array

    The permissions that the role has to a source. For more information, read All Supported Privileges.

    users

    id

    String

    The ID of the user.


    permissions

    Array

    The permissions that the user has to a source. For more information, read All Supported Privileges.

    Example Request
    curl -X POST 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog' \
    -H 'Authorization: Bearer <personal access token>' \
    -H 'Content-Type: application/json' \
    -d '{
          "entityType": "source",
          "config": {
            "hostname": "177.15.0.112",
            "port": "3306",
            "authenticationType": "ANONYMOUS",
            "netWriteTimeout": 60,
            "fetchSize": 200,
            "maxIdleConns": 8,
            "idleTimeSec": 60,
            "propertyList": []
          },
          "type": "MYSQL",
          "name": "recruitingdb",
          "metadataPolicy": {
            "authTTLMs": 86400000,
            "namesRefreshMs": 86400000,
            "datasetRefreshAfterMs": 86400000,
            "datasetExpireAfterMs": 259200000,
            "datasetUpdateMode": "PREFETCH_QUERIED",
            "deleteUnavailableDatasets": true
          },
          "accelerationGracePeriodMs": 10800000,
          "accelerationRefreshPeriodMs": 3600000,
          "accessControlList": {
            "roles": [
              {
                "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
                "permissions": []
              }
            ],
            "users": []
          }
        }'
    
    Example Response
    {
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
      "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
      "type": "MYSQL",
      "name": "recruitingdb",
      "createdAt": "2022-09-12T07:22:19.334Z",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accelerationNeverExpire": false,
      "accelerationNeverRefresh": false,
      "children": [
          {
            "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
            "path": [
                "recruitingdb",
                "contacts"
              ],
              "type": "CONTAINER",
              "containerType": "FOLDER"
          }
      ],
      "allowCrossSourceSelection": false,
      "disableMetadataValidityCheck": false,
      "accessControlList": {
          "roles": [
            {
                "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
                "permissions": []
            }
          ],
          "users": []
      },
      "permissions": [],
      "checkTableAuthorizer": false,
      "owner": {
          "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
          "ownerType": "USER"
      }
    }
    

    Responses

    200

    Source created successfully

    400

    Bad input parameter

    401

    Unauthorized access


    Retrieving a Source

    Get information about the source using the source ID.

    Retrieving a Source
    GET /v0/projects/{project-id}/catalog/{id}
    

    Parameters

    project-id

    path

    String (UUID)

    The unique identifier for the project that contains the source.

    Example 02d36975-73eb-47ed-9bb5-de73060380f6


    id

    path

    String (UUID)

    The unique identifier for the source that you want to retrieve.

    Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0


    include

    query

    String

    Optional

    When using include, the name of a non-default response field is returned. You can retrieve either a list of access control list names - aclNames - or a list of permissions. Read Include and Exclude Query Parameters for usage examples.

    Enum aclNames , permissions

    Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?include=permissions


    exclude

    query

    String

    Optional

    When using exclude, the children field is excluded from the response. Read Include and Exclude Query Parameters for usage examples.

    Enum children

    Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0?exclude=children

    Example Request
    curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
    -H 'Authorization: Bearer <personal access token>' \
    -H 'Content-Type: application/json'
    
    Example Response
    {
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
      "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
      "type": "MYSQL",
      "name": "recruitingdb",
      "createdAt": "2022-09-12T07:22:19.334Z",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accelerationNeverExpire": false,
      "accelerationNeverRefresh": false,
      "children": [
        {
          "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
          "path": [
            "recruitingdb",
            "contacts"
          ],
          "type": "CONTAINER",
          "containerType": "FOLDER"
        }
      ],
      "allowCrossSourceSelection": false,
      "disableMetadataValidityCheck": false,
      "accessControlList": {},
      "permissions": [],
      "checkTableAuthorizer": false,
      "owner": {
        "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
        "ownerType": "USER"
      }
    }
    

    Responses

    200

    Source returned successfully

    400

    Bad input parameter

    401

    Unauthorized access

    404

    Not found


    Updating a Source

    Edit the source information using the source ID.

    Updating a Source
    PUT /v0/projects/{project-id}/catalog/{id}
    

    Parameters

    project-id

    path

    String (UUID)

    The unique identifier for the project that contains the source.

    Example 02d36975-73eb-47ed-9bb5-de73060380f6


    id

    path

    String (UUID)

    The unique identifier for the source that you want to update.

    Example ffbe8c1d-1db7-48d1-9c58-f452838fedc0


    entityType

    String

    Specifies the type of container. For sources, the type is source.


    config

    Object

    Configuration options for the specified data source.


    type

    String

    Identifies the source type that you are configuring. The source type determines the config parameters that display. For more information, read Configuration Attributes.

    Enum ARCTIC , AWSGLUE , MSSQL , MYSQL , ORACLE , POSTGRES , REDSHIFT , S3 , SNOWFLAKE


    name

    String

    The user-defined name of the source.

    Example recruitingdb


    metadataPolicy

    Source Metadata

    The policies covering the update of a source’s metadata.


    accelerationGracePeriodMs

    Number

    Identifies the length of time to keep reflections for all the datasets in a source before these reflections expire. The default setting is 10800000 milliseconds or three hours. The minimum amount of time that is supported is 3600000 milliseconds or one hour. For more information, read Setting the Expiration Policy for Reflections.

    Example 10800000


    accelerationRefreshPeriodMs

    Number

    Identifies the refresh frequency for a dataset's reflections. The default setting is 3600000 milliseconds or one hour, which is also the minimum amount of time that is supported.

    Example 3600000


    accessControlList

    Object

    List of permissions that is set for a source.

    config

    hostname

    String

    The name of the server you are connecting to.

    Example 177.15.0.112


    port

    String

    The port number of the server you are connecting to.

    Example 3306


    authenticationType

    String

    The type of authentication needed to connect to the server. Use ANONYMOUS when no authentication is needed and MASTER when credentials from a master database user is required.

    Enum ANONYMOUS , MASTER


    netWriteTimeout

    Number

    The amount of time, in seconds, to wait for data from the server before aborting the connection.

    Example 60


    fetchSize

    Number

    The number of records to fetch per request. Set to 0 to enable Dremio to automatically set the size.

    Example 200


    maxIdleConns

    Number

    Sets the maximum number of idle connections that you want to keep.

    Example 8


    idleTimeSec

    Number

    Sets the idle time, in seconds, before a connection is evaluated for closure.

    Example 60


    propertyList

    Array

    An array of name/value pairs defining optional connection properties that is used by the source. Use a comma separated list to specify multiple name value pairs.

    metadataPolicy

    authTTLMs

    Number

    Sets the expiration for an authorization to a metadata policy, in milliseconds. The default time period is 24 hours (86400000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 86400000


    namesRefreshMs

    Number

    Sets when to run a refresh of a source, in milliseconds. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 86400000


    datasetRefreshAfterMs

    Number

    Determines how often the metadata in the dataset is refreshed, in milliseconds. The default refresh rate is one hour (3600000 milliseconds). The minimum supported refresh rate is one minute (60000 milliseconds).

    Example 86400000


    datasetExpireAfterMs

    Number

    Sets the amount of time, in milliseconds, to keep the metadata before it expires. The default time period is one hour (3600000 milliseconds). The minimum supported time period is one minute (60000 milliseconds).

    Example 259200000


    datasetUpdateMode

    String

    Sets the metadata policy for when a dataset is updated. Use PREFETCH_QUERIED to update the details for previously queried objects in a source.

    Enum PREFETCH_QUERIED


    deleteUnavailableDatasets

    Boolean

    Option to remove dataset definitions if the underlying data is unavailable to Dremio, for example, at the time the data is queried. The default setting is true. Set to false to keep the dataset definitions.

    accessControlList

    roles

    Array

    The roles that have been granted privileges to a source.


    users

    Array

    The users that have been granted privileges to a source.

    roles

    id

    String

    The ID of the role.

    Example 9ab42853-bdef-465f-b9bb-381a13a9bf78


    permissions

    Array

    The permissions that the role has to a source. For more information, read All Supported Privileges.

    Example Request
    curl -X GET 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/ffbe8c1d-1db7-48d1-9c58-f452838fedc0' \
    -H 'Authorization: Bearer <personal access token>' \
    -H 'Content-Type: application/json'
    -d '{
          "entityType": "source",
          "config": {
            "hostname": "177.15.0.112",
            "port": "3306",
            "authenticationType": "ANONYMOUS",
            "netWriteTimeout": 60,
            "fetchSize": 200,
            "maxIdleConns": 8,
            "idleTimeSec": 60,
            "propertyList": []
          },
          "type": "MYSQL",
          "name": "recruitingdb",
          "metadataPolicy": {
            "authTTLMs": 86400000,
            "namesRefreshMs": 86400000,
            "datasetRefreshAfterMs": 86400000,
            "datasetExpireAfterMs": 259200000,
            "datasetUpdateMode": "PREFETCH_QUERIED",
            "deleteUnavailableDatasets": true
          },
          "accelerationGracePeriodMs": 10800000,
          "accelerationRefreshPeriodMs": 3600000,
          "accessControlList": {
            "roles": [
              {
                "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
                "permissions": []
              }
            ],
            "users": []
          }
        }'
    
    Example Response
    {
      "entityType": "source",
      "config": {
        "hostname": "177.15.0.112",
        "port": "3306",
        "authenticationType": "ANONYMOUS",
        "netWriteTimeout": 60,
        "fetchSize": 200,
        "maxIdleConns": 8,
        "idleTimeSec": 60,
        "propertyList": []
      },
      "id": "629904cf-9c06-4ae6-8cc1-faf1d2f1ab5f",
      "tag": "a927dd14-2c7d-4684-9de2-4000e05c0d07",
      "type": "MYSQL",
      "name": "recruitingdb",
      "createdAt": "2022-09-12T07:22:19.334Z",
      "metadataPolicy": {
        "authTTLMs": 86400000,
        "namesRefreshMs": 86400000,
        "datasetRefreshAfterMs": 86400000,
        "datasetExpireAfterMs": 259200000,
        "datasetUpdateMode": "PREFETCH_QUERIED",
        "deleteUnavailableDatasets": true
      },
      "accelerationGracePeriodMs": 10800000,
      "accelerationRefreshPeriodMs": 3600000,
      "accelerationNeverExpire": false,
      "accelerationNeverRefresh": false,
      "children": [
        {
          "id": "42312a90-1723-4bcb-b62c-bf9abee371b1",
          "path": [
              "recruitingdb",
              "contacts"
            ],
            "type": "CONTAINER",
            "containerType": "FOLDER"
        }
      ],
      "allowCrossSourceSelection": false,
      "disableMetadataValidityCheck": false,
      "accessControlList": {
        "roles": [
          {
              "id": "9ab42853-bdef-465f-b9bb-381a13a9bf78",
              "permissions": []
          }
        ],
        "users": []
      },
      "permissions": [],
      "checkTableAuthorizer": false,
      "owner": {
        "ownerId": "5c9c394d-7395-4687-97dd-beac4384c359",
        "ownerType": "USER"
      }
    }
    

    Responses

    204

    No content

    400

    Bad input parameter

    401

    Unauthorized access

    404

    Not found


    Deleting a Source

    Delete a source.

    Deleting a Source
    DELETE /v0/projects/{project-id}/catalog/{id}
    

    Parameters

    project-id

    path

    String (UUID)

    The unique identifier for the project that contains the source.

    Example 02d36975-73eb-47ed-9bb5-de73060380f6


    id

    path

    String (UUID)

    The unique identifier for the source that you want to delete.

    Example 21a74cb1-3fcf-48c6-8f69-94c257ec3da3

    Example Request
    curl -X DELETE 'https://api.dremio.cloud/v0/projects/02d36975-73eb-47ed-9bb5-de73060380f6/catalog/21a74cb1-3fcf-48c6-8f69-94c257ec3da3' \
    -H 'Authorization: Bearer <personal access token>' \
    -H 'Content-Type: application/json'
    
    Example Response
    No response
    

    Responses

    204

    No content

    400

    Bad input parameter

    401

    Unauthorized access

    404

    Not found