Skip to main content

Pipes Preview

Use the Pipes API to trigger data ingestion with an existing pipe.

The response contains a request ID associated with the batch of files you submitted in the body of your request. Use the request ID to monitor the batch's ingestion progress with the SYS.PROJECT.COPY_FILE_HISTORY system table.

Loading Files Using an Existing Pipe​

Load files using an existing pipe and retrieve the associated request ID for the batch.

Use the request ID in the response to monitor the batch's ingestion progress with the SYS.PROJECT.COPY_FILE_HISTORY system table.

Method and URL 
POST /v0/projects/{project-id}/pipes/{id}/loadfiles

Parameters​

project-id Path   String (UUID)

Unique identifier of the project for which you want to load files.

Example: 319fbada-4857-4d52-8901-7a857416ded6

id Path   String (UUID)

Unique identifier of the data ingestion pipe you want to use to load files.

Example: a7472a55-6858-4b44-a070-ed79d1c4dd43

files Body   Array of Object

The paths and sizes of the files to load using the data ingestion pipe specified in the request URL.

Example: [{"path": "Samples/samples.dremio.com/NYC-weather.csv","size": "80 MB"},{"path": "prod/NYC/NYC_incidents2016.json","size": "800 KB"},{"path": "Samples/samples.dremio.com/NYC-taxi-trips.csv","size": "2 GB"},{"path": "newCatalog/testFolder/Test_view","size": "50 KB"}]

Parameters of the files Array​

path Body   String

Path to the file to load. The path lists each level of hierarchy in order, from outer to inner: source first, then folder and subfolders, then the file itself.

Example: Samples/samples.dremio.com/NYC-weather.csv

size Body   String

The estimated size of the file, expressed as an integer followed by a single space and the unit. Supported units are B, KB, MB, and GB. The estimated size improves performance by helping Dremio batch files more efficiently.

Example: 80 MB

Example Request 
curl -X POST 'https://api.dremio.cloud/v0/projects/319fbada-4857-4d52-8901-7a857416ded6/pipes/a7472a55-6858-4b44-a070-ed79d1c4dd43/loadfiles' \
--header 'Authorization: Bearer <personal access token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "files": [
    {
      "path": "Samples/samples.dremio.com/NYC-weather.csv",
      "size": "80 MB"
    },
    {
      "path": "prod/NYC/NYC_incidents2016.json",
      "size": "800 KB"
    },
    {
      "path": "Samples/samples.dremio.com/NYC-taxi-trips.csv",
      "size": "2 GB"
    },
    {
      "path": "newCatalog/testFolder/Test_view",
      "size": "50 KB"
    }
  ]
}'
Example Response 
{
  "requestId":"4f7b12c1-adcc-40c3-ad0f-097bc7911a56"
}

Response Status Codes​

200   OK

400   Bad Request

404   Not Found

415   Unsupported Media Type

500   Internal Server Error