SYS.PIPES Enterprise

The sys.pipes table contains the metadata for autoingest pipes.

Syntax

SELECT *
FROM sys.pipes

Example Output

| pipe_id | pipe_name | source_path | target_table | target_branch | dedup_lookback_period_days | file_format | format_options | pipe_status | error_message | notification_provider | notification_queue_ref | created_at | last_updated_at | created_by | last_updated_by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 525ffe04-97bb-473e-8e76-fa559faa7d0d | test_pipe | @s3/ingestion-test | s3."ingestion-test".pipe_sink_table | main | 14 | CSV | {"DATE_FORMAT":"YYYY-MM-DD\"T\"HH24:MI:SS.FFF","RECORD_DELIMITER":"\n","FIELD_DELIMITER":","} | STOPPED_STORAGE_LOCATION_ALTERED | Resource not found at path @s3/ingestion-test | AWS_SQS | arn:aws:sqs:us-west-2:123243142826:data_ingestion_queue | 2024-08-05T13:39:41.167Z | 2024-08-05T13:39:41.167Z | dremio | dremio |

Columns

Column	Data Type	Description
pipe_id	varchar	The unique identifier of the autoingest pipe.
pipe_name	varchar	The name of the autoingest pipe.
source_path	varchar	The preconfigured source path specified in the CREATE PIPE command.
target_table	varchar	The name of the target Iceberg table.
target_branch	varchar	The name of the target branch if Nessie was used as a catalog.
dedup_lookback_period_days	integer	The number of days that Dremio looks back when comparing newly uploaded files to previously loaded files for file deduplication.
file_format	varchar	The file format associated with the pipe.
format_options	varchar	The specific file format options that are used in the COPY INTO statement when the pipe is triggered.
pipe_status	varchar	The state of the pipe. Enum: RUNNING: The pipe is running as expected. PAUSED: The pipe exists but loads are not triggered when files are uploaded to the source. STOPPED_MISSING_TABLE_OR_BRANCH: Dremio has stopped the pipe because either the target table or target branch does not exist. STOPPED_STORAGE_LOCATION_ALTERED: Dremio has stopped the pipe because a user has altered the source root path. STOPPED_ACCESS_DENIED: Dremio has stopped the pipe because Dremio cannot access the source location. STOPPED_MISSING_DREMIO_SOURCE: Dremio has stopped the pipe because the source referenced in the pipe no longer exists. UNHEALTHY: Autoingestion is failing due to an unknown root cause. Often a transient issue, and the system usually self-corrects and catches up. STOPPED_INTERNAL_ERROR: The pipe is stopped because all pipe jobs failed within 5 days due to an unknown cause.
error_message	varchar	The last error message for a pipe job that did not complete successfully.
notification_provider	varchar	The provider of the event notification queue. Enum: AWS_SQS.
notification_queue_ref	varchar	The unique identifier of the event notification queue.
created_at	timestamp	The date and time at which the pipe was created.
last_updated_at	timestamp	The date and time of the last modification to the pipe definition.
created_by	varchar	The user who created the pipe.
last_updated_by	varchar	The user who last updated the pipe definition.