Skip to main content

Pipes

Use the Pipes API to trigger data ingestion with an existing pipe.

The response contains a request ID associated with the batch of files you submitted in the body of your request. Use the request ID to monitor the batch's ingestion progress with the SYS.PROJECT.COPY_FILE_HISTORY system table.

Load Files Using an Existing Pipe​

Use the request ID in the response to monitor the batch's ingestion progress with the SYS.PROJECT.COPY_FILE_HISTORY system table.

Method and URL 
POST /v0/projects/{project_id}/pipes/{id}/loadfiles

Parameters​

project_id Path   String (UUID)

id Path   String (UUID)

UUID of the data ingestion pipe you want to use to load files.

files Body   Array of Objects

The paths and sizes of the files to load using the data ingestion pipe specified in the request URL.

Example:

[
  {
    "path": "Samples/samples.dremio.com/NYC-weather.csv",
    "size": "80 MB"
  },
  {
    "path": "prod/NYC/NYC_incidents2016.json",
    "size": "800 KB"
  },
  {
    "path": "Samples/samples.dremio.com/NYC-taxi-trips.csv",
    "size": "2 GB"
  },
  {
    "path": "newCatalog/testFolder/Test_view",
    "size": "50 KB"
  }
]

Parameters of the files Array​

path Body   String

Path to the file to load. The path lists each level of hierarchy in order, from outer to inner: source first, then folders and subfolders, then the file itself.

Example: Samples/samples.dremio.com/NYC-weather.csv

size Body   String

The estimated size of the file, expressed as an integer followed by a single space and the unit. Supported units are B, KB, MB, and GB. The estimated size improves performance by helping Dremio batch files more efficiently.

Example: 80 MB

Example​

Request 
curl -X POST "https://api.dremio.cloud/v0/projects/$PROJECT_ID/pipes/$PIPE_ID/loadfiles" \
-H "Authorization: Bearer $DREMIO_TOKEN" \
-H 'Content-Type: application/json' \
--data-raw '{
  "files": [
    {
      "path": "Samples/samples.dremio.com/NYC-weather.csv",
      "size": "80 MB"
    },
    {
      "path": "prod/NYC/NYC_incidents2016.json",
      "size": "800 KB"
    },
    {
      "path": "Samples/samples.dremio.com/NYC-taxi-trips.csv",
      "size": "2 GB"
    },
    {
      "path": "newCatalog/testFolder/Test_view",
      "size": "50 KB"
    }
  ]
}'
Response 
{
  "requestId": "4f7b12c1-adcc-40c3-ad0f-097bc7911a56"
}

Response Status Codes​

200   OK

400   Bad Request

404   Not Found

415   Unsupported Media Type