Skip to main content

Lineage

Use the Catalog API to retrieve lineage information about datasets (tables and views). The lineage object includes information about the dataset's sources, parent objects, and child objects.

Lineage Object 
{
  "sources": [
    {
      "id": "21077e5d-fe6f-4a29-843f-58fa3acb17c2",
      "path": [
        "Samples"
      ],
      "tag": "1a87a4b2-82df-4de1-853a-35fa727197e6",
      "type": "CONTAINER",
      "containerType": "SOURCE",
      "createdAt": "2022-02-14T21:57:48.794Z"
    }
  ],
  "parents": [
    {
      "id": "3419fa3a-b5b3-4438-b864-a27ec4e18752",
      "path": [
        "Samples",
        "samples.dremio.com",
        "zips.json"
      ],
      "tag": "882a7dcf-adbb-405b-a496-0cc591f885d7",
      "type": "DATASET",
      "datasetType": "PROMOTED",
      "createdAt": "2023-01-18T18:49:09.669Z"
    }
  ],
  "children": [
    {
      "id": "170e211e-4235-4d8d-acb5-3d4dbfe99c75",
      "path": [
        "@dremio",
        "NYC_zip"
      ],
      "tag": "46778141-d515-4c7e-bbf4-78d88a749913",
      "type": "DATASET",
      "datasetType": "VIRTUAL",
      "createdAt": "2023-01-25T02:11:46.344Z"
    },
    {
      "id": "7f79c068-a3c3-4af7-8cd4-35896ef0a0e0",
      "path": [
        "@dremio",
        "Chicago_zip"
      ],
      "tag": "dbbf1a68-7da5-4504-87d5-c66cbbc187e0",
      "type": "DATASET",
      "datasetType": "VIRTUAL",
      "createdAt": "2023-01-25T00:09:12.461Z"
    }
  ]
}

Lineage Attributes​

sources Array of Object

Information about the sources the dataset uses. Each object in the sources array represents one source.

parents Array of Object

Information about the parent objects for the dataset. Each object in the parents array represents one parent object. For example, if a view represents a join of two other datasets, the parents array includes the two joined datasets. The parents array is empty if the dataset does not have parent objects.

children Array of Object

Information about other catalog objects that reference the dataset. Each object in the children array represents one child object. The children array is empty if the dataset does not have child objects.

Attributes of Objects in the sources Array​

id String (UUID)

UUID of the source.

path Array of String

Path of the source within Dremio, expressed as an array.

Example:

[
  "Samples"
]

tag String (UUID)

UUID of the version of the source. Dremio changes the tag whenever the source changes.

type String

Type of source. For sources in lineage responses, the only valid type is CONTAINER.

containerType String

Type of container for the source. The only valid value is SOURCE.

createdAt String

Timestamp of when the source was created.

Attributes of Objects in the parents Array​

id String (UUID)

UUID of the parent object.

path Array of String

Path of the parent object within Dremio, expressed as an array. The path consists of the source, followed by any folders and subfolders, followed by the parent object itself as the last item in the array.

Example:

[
  "Samples",
  "samples.dremio.com",
  "zips.json"
]

tag String (UUID)

UUID of the version of the parent object. Dremio changes the tag whenever the parent object changes.

type String

Type of parent object. For parent objects in lineage responses, the only valid type is DATASET.

datasetType String

Dataset type for the parent object. If the parent object is a table, the value is PROMOTED. If the parent object is a view, the value is VIRTUAL.

Example: PROMOTED

createdAt String

Timestamp of when the parent was created.

Attributes of Objects in the children Array​

id String (UUID)

UUID of the child object.

path Array of String

Path of the child object within Dremio, expressed as an array. The path consists of the source, followed by any folders and subfolders, followed by the child object itself as the last item in the array.

Example:

[
  "@dremio",
  "NYC_zip"
]

tag String (UUID)

UUID of the version of the child object. Dremio changes the tag whenever the child object changes.

type String

Type of child object. For child objects in lineage responses, the only valid type is DATASET.

datasetType String

Dataset type for the child object. For child objects in lineage responses, the only valid value is VIRTUAL.

createdAt String

Timestamp of when the child was created.

Retrieve Lineage Information About a Dataset​

Method and URL 
GET /v0/projects/{project_id}/catalog/{id}/graph

Parameters​

project_id Path   String (UUID)

id Path   String (UUID)

UUID of the dataset.

Example​

Request 
curl -X GET "https://api.dremio.cloud/v0/projects/$PROJECT_ID/catalog/$DATASET_ID/graph" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json'
Response 
{
  "sources": [
    {
      "id": "21077e5d-fe6f-4a29-843f-58fa3acb17c2",
      "path": [
        "Samples"
      ],
      "tag": "1a87a4b2-82df-4de1-853a-35fa727197e6",
      "type": "CONTAINER",
      "containerType": "SOURCE",
      "createdAt": "2022-02-14T21:57:48.794Z"
    }
  ],
  "parents": [
    {
      "id": "3419fa3a-b5b3-4438-b864-a27ec4e18752",
      "path": [
        "Samples",
        "samples.dremio.com",
        "zips.json"
      ],
      "tag": "882a7dcf-adbb-405b-a496-0cc591f885d7",
      "type": "DATASET",
      "datasetType": "PROMOTED",
      "createdAt": "2023-01-18T18:49:09.669Z"
    }
  ],
  "children": [
    {
      "id": "170e211e-4235-4d8d-acb5-3d4dbfe99c75",
      "path": [
        "@dremio",
        "NYC_zip"
      ],
      "tag": "46778141-d515-4c7e-bbf4-78d88a749913",
      "type": "DATASET",
      "datasetType": "VIRTUAL",
      "createdAt": "2023-01-25T02:11:46.344Z"
    },
    {
      "id": "7f79c068-a3c3-4af7-8cd4-35896ef0a0e0",
      "path": [
        "@dremio",
        "Chicago_zip"
      ],
      "tag": "dbbf1a68-7da5-4504-87d5-c66cbbc187e0",
      "type": "DATASET",
      "datasetType": "VIRTUAL",
      "createdAt": "2023-01-25T00:09:12.461Z"
    }
  ]
}

Response Status Codes​

200   OK

400   Bad Request

401   Unauthorized

403   Forbidden

404   Not Found