Pipes Preview
Use the Pipes API to trigger data ingestion with an existing pipe.
The response contains a request ID associated with the batch of files you submitted in the body of your request. Use the request ID to monitor the batch's ingestion progress with the SYS.PROJECT.COPY_FILE_HISTORY system table.
Loading Files Using an Existing Pipe
Load files using an existing pipe and retrieve the associated request ID for the batch.
Use the request ID in the response to monitor the batch's ingestion progress with the SYS.PROJECT.COPY_FILE_HISTORY system table.
Method and URLPOST /v0/projects/{project-id}/pipes/{id}/loadfiles
Parameters
project-id Path String (UUID)
Unique identifier of the project for which you want to load files.
Example: 319fbada-4857-4d52-8901-7a857416ded6
id Path String (UUID)
Unique identifier of the data ingestion pipe you want to use to load files.
Example: a7472a55-6858-4b44-a070-ed79d1c4dd43
files Body Array of Object
The paths and sizes of the files to load using the data ingestion pipe specified in the request URL.
Example: [{"path": "Samples/samples.dremio.com/NYC-weather.csv","size": "80 MB"},{"path": "prod/NYC/NYC_incidents2016.json","size": "800 KB"},{"path": "Samples/samples.dremio.com/NYC-taxi-trips.csv","size": "2 GB"},{"path": "newCatalog/testFolder/Test_view","size": "50 KB"}]
Parameters of the files
Array
path Body String
Path to the file to load. The path lists each level of hierarchy in order, from outer to inner: source first, then folder and subfolders, then the file itself.
Example: Samples/samples.dremio.com/NYC-weather.csv
size Body String
The estimated size of the file, expressed as an integer followed by a single space and the unit. Supported units are B
, KB
, MB
, and GB
. The estimated size improves performance by helping Dremio batch files more efficiently.
Example: 80 MB
Example Requestcurl -X POST 'https://api.dremio.cloud/v0/projects/319fbada-4857-4d52-8901-7a857416ded6/pipes/a7472a55-6858-4b44-a070-ed79d1c4dd43/loadfiles' \
--header 'Authorization: Bearer <personal access token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"files": [
{
"path": "Samples/samples.dremio.com/NYC-weather.csv",
"size": "80 MB"
},
{
"path": "prod/NYC/NYC_incidents2016.json",
"size": "800 KB"
},
{
"path": "Samples/samples.dremio.com/NYC-taxi-trips.csv",
"size": "2 GB"
},
{
"path": "newCatalog/testFolder/Test_view",
"size": "50 KB"
}
]
}'
{
"requestId":"4f7b12c1-adcc-40c3-ad0f-097bc7911a56"
}
Response Status Codes
200 OK400 Bad Request
404 Not Found
415 Unsupported Media Type
500 Internal Server Error