Skip to main content
Version: current [26.x]

SYS.PIPES Enterprise

The sys.pipes table contains the metadata for autoingest pipes.

Syntax
SELECT *
FROM sys.pipes

Example Output

pipe_idpipe_namesource_pathtarget_tabletarget_branchdedup_lookback_period_daysfile_formatformat_optionspipe_statuserror_messagenotification_providernotification_queue_refcreated_atlast_updated_atcreated_bylast_updated_by
525ffe04-97bb-473e-8e76-fa559faa7d0dtest_pipe@s3/ingestion-tests3."ingestion-test".pipe_sink_tablemain14CSV{"DATE_FORMAT":"YYYY-MM-DD\"T\"HH24:MI:SS.FFF","RECORD_DELIMITER":"\n","FIELD_DELIMITER":","}STOPPED_STORAGE_LOCATION_ALTEREDResource not found at path @s3/ingestion-testAWS_SQSarn:aws:sqs:us-west-2:123243142826:data_ingestion_queue2024-08-05T13:39:41.167Z2024-08-05T13:39:41.167Zdremiodremio

Columns

ColumnData TypeDescription
pipe_idvarcharThe unique identifier of the autoingest pipe.
pipe_namevarcharThe name of the autoingest pipe.
source_pathvarcharThe preconfigured source path specified in the CREATE PIPE command.
target_tablevarcharThe name of the target Iceberg table.
target_branchvarcharThe name of the target branch if Nessie was used as a catalog.
dedup_lookback_period_daysintegerThe number of days that Dremio looks back when comparing newly uploaded files to previously loaded files for file deduplication.
file_formatvarcharThe file format associated with the pipe.
format_optionsvarcharThe specific file format options that are used in the COPY INTO statement when the pipe is triggered.
pipe_statusvarcharThe state of the pipe. Enum:
  • RUNNING: The pipe is running as expected.
  • PAUSED: The pipe exists but loads are not triggered when files are uploaded to the source.
  • STOPPED_MISSING_TABLE_OR_BRANCH: Dremio has stopped the pipe because either the target table or target branch does not exist.
  • STOPPED_STORAGE_LOCATION_ALTERED: Dremio has stopped the pipe because a user has altered the source root path.
  • STOPPED_ACCESS_DENIED: Dremio has stopped the pipe because Dremio cannot access the source location.
  • STOPPED_MISSING_DREMIO_SOURCE: Dremio has stopped the pipe because the source referenced in the pipe no longer exists.
  • UNHEALTHY: Autoingestion is failing due to an unknown root cause. Often a transient issue, and the system usually self-corrects and catches up.
  • STOPPED_INTERNAL_ERROR: The pipe is stopped because all pipe jobs failed within 5 days due to an unknown cause.
error_messagevarcharThe last error message for a pipe job that did not complete successfully.
notification_providervarcharThe provider of the event notification queue. Enum: AWS_SQS.
notification_queue_refvarcharThe unique identifier of the event notification queue.
created_attimestampThe date and time at which the pipe was created.
last_updated_attimestampThe date and time of the last modification to the pipe definition.
created_byvarcharThe user who created the pipe.
last_updated_byvarcharThe user who last updated the pipe definition.