Dremio Cloud
Dremio Software
 
On this page

    Promote a File/Folder to a Physical Dataset

    This API promotes a file or folder in a file-based source to a physical dataset (PDS). The supplied path is used to determine what entity is promoted.

    Files or folders inside a source can be promoted to physical datasets. This converts the folder/file to a dataset; the dataset then has a new ID since it is a new entity.

    Note:
    To unpromote a physical dataset (PDS), you delete the dataset. This reverts the PDS back to its original form (a folder or file).

    To promote a file or folder, you need the following:

    • ID (encoded) of the folder or file. Use GET /catalog/by-path to obtain information about the file or folder.
    • Request input. As a minimum: entityType, id, path, type, and format.
    • Dataset format.

    Syntax

    POST /api/v3/catalog/{id}

    Request Input

    See Dataset for more information.

    {
  "entityType": "dataset" [immutable, generated by Dremio],
  "id": String [immutable, generated by Dremio],
  "path": [String] [immutable after creation],
  "tag": String [immutable, generated by Dremio],
  "type": String ["PHYSICAL_DATASET", "VIRTUAL_DATASET"] [immutable],
  "fields": [DatasetField] [immutable],
  "createdAt": String (RFC3339 date) [immutable, generated by Dremio],
  "accelerationRefreshPolicy": DatasetAccelerationRefreshPolicy [optional, only for physical datasets in a source],
  "sql": String [optional, required for virtual datasets],
  "sqlContext": [String] [optional, only for virtual datasets],
  "format": DatasetFormat [optional, required for promoted datasets],
  "approximateStatisticsAllowed": Boolean [optional, introduced in Dremio 2.1.0]
}

    Response Output

    See Dataset for more information.

    {
  "entityType": "dataset" [immutable, generated by Dremio],
  "id": String [immutable, generated by Dremio],
  "path": [String] [immutable after creation],
  "tag": String [immutable, generated by Dremio],
  "type": String ["PHYSICAL_DATASET", "VIRTUAL_DATASET"] [immutable],
  "fields": [DatasetField] [immutable],
  "createdAt": String (RFC3339 date) [immutable, generated by Dremio],
  "accelerationRefreshPolicy": DatasetAccelerationRefreshPolicy [optional, only for physical datasets in a source],
  "sql": String [optional, required for virtual datasets],
  "sqlContext": [String] [optional, only for virtual datasets],
  "format": DatasetFormat [optional, required for promoted datasets],
  "approximateStatisticsAllowed": Boolean [optional, introduced in Dremio 2.1.0]
}

    Response Codes

    400 - The supplied CatalogEntity object is invalid.
    403 - User does not have permission to create the catalog entity.

    Example: Promote a Folder

    In this example, a folder, acquisition-mini, in a HDFS source, my_hdfs_2, is being promoted to a PDS. We have the following information about the folder:

    • Encoded ID:
      dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini
    • Path:
      [
    "my_hdfs_2",
    "data",
    "loans",
    "acquisition-mini"
    ]
    • Dataset format:
    {
        "type": "Text",
        "fieldDelimiter": "|",
        "lineDelimiter": "\n",
        "escape": "\"",
        "skipFirstLine": false,
        "extractHeader": false,
        "trimHeader": false,
        "autoGenerateColumnNames": true
    }

    Note:
    Postman was used to generate samples.

    HTTP

    POST localhost:9047/api/v3/catalog/dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini

    Raw Body Input

    {
    "entityType": "dataset",
    "id": "dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini",
    "path": [
    	"my_hdfs_2",
    	"data",
    	"loans",
    	"acquisition-mini"
    	],
    	
    "type": "PHYSICAL_DATASET",
    "format": {
        "type": "Text",
        "fieldDelimiter": "|",
        "lineDelimiter": "\n",
        "escape": "\"",
        "skipFirstLine": false,
        "extractHeader": false,
        "trimHeader": false,
        "autoGenerateColumnNames": true
    }
}

    Curl

    curl -X POST \
  http://localhost:9047/api/v3/catalog/dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini \
  -H 'Authorization: _dremioo8opojj6vn4ughkvcpalpr46d6' \
  -H 'Content-Type: application/json' \
  -d '{
    "entityType": "dataset",
    "id": "dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini",
    "path": [
    	"my_hdfs_2",
    	"data",
    	"loans",
    	"acquisition-mini"
    	],
    	
    "type": "PHYSICAL_DATASET",
    "format": {
        "type": "Text",
        "fieldDelimiter": "|",
        "lineDelimiter": "\n",
        "escape": "\"",
        "skipFirstLine": false,
        "extractHeader": false,
        "trimHeader": false,
        "autoGenerateColumnNames": true
    }
}'

    Python

    import requests

url = "http://localhost:9047/api/v3/catalog/dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini"

payload = "{\n    \"entityType\": \"dataset\",\n    \"id\": \"dremio%3A%2Fmy_hdfs_2%2Fdata%2Floans%2Facquisition-mini\",\n    \"path\": [\n    \t\"my_hdfs_2\",\n    \t\"data\",\n    \t\"loans\",\n    \t\"acquisition-mini\"\n    \t],\n    \t\n    \"type\": \"PHYSICAL_DATASET\",\n    \"format\": {\n        \"type\": \"Text\",\n        \"fieldDelimiter\": \"|\",\n        \"lineDelimiter\": \"\\n\",\n        \"escape\": \"\\\"\",\n        \"skipFirstLine\": false,\n        \"extractHeader\": false,\n        \"trimHeader\": false,\n        \"autoGenerateColumnNames\": true\n    }\n}"
headers = {
    'Authorization': "_dremioo8opojj6vn4ughkvcpalpr46d6",
    'Content-Type': "application/json"
    }

response = requests.request("POST", url, data=payload, headers=headers)

print(response.text)

    Response

    {
    "entityType": "dataset",
    "id": "cf771ed4-8ffc-49c6-b75c-b6ce4a518289",
    "type": "PHYSICAL_DATASET",
    "path": [
        "my_hdfs_2",
        "data",
        "loans",
        "acquisition-mini"
    ],
    "createdAt": "2019-03-26T18:56:57.085Z",
    "tag": "0",
    "format": {
        "type": "Text",
        "ctime": 0,
        "isFolder": true,
        "location": "/data/loans/acquisition-mini",
        "fieldDelimiter": "|",
        "skipFirstLine": false,
        "extractHeader": false,
        "quote": "\"",
        "comment": "#",
        "escape": "\"",
        "lineDelimiter": "\n",
        "autoGenerateColumnNames": true,
        "trimHeader": false
    },
    "accessControlList": {
        "version": "0"
    },
    "fields": [
        {
            "name": "A",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "B",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "C",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "D",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "E",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "F",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "G",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "H",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "I",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "J",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "K",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "L",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "M",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "N",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "O",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "P",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "Q",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "R",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "S",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "T",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "U",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "V",
            "type": {
                "name": "VARCHAR"
            }
        },
        {
            "name": "W",
            "type": {
                "name": "VARCHAR"
            }
        }
    ],
    "approximateStatisticsAllowed": false
}