Skip to main content

SYS.PROJECT.PIPES Preview

The sys.project.pipes table contains the metadata for autoingest pipes in a project.

Syntax
SELECT *
FROM sys.project.pipes

Example Output

org_idproject_idpipe_namepipe_idsource_pathtarget_tabletarget_branchdedup_lookback_periodfile_formatformat_optionspipe_statuserror_messagecloud_settingscloud_cli_command_settingscreated_atlast_updated_atcreated_bylast_updated_bycreated_by_id
c47082e0-830b-4ca8-8d52-b0dd794a59335b4b8cdb-a308-4919-aa42-486d4bbcd879test_pipe525ffe04-97bb-473e-8e76-fa559faa7d0d@pipetest/folderpipe_sink_tablemain20json{"DATE_FORMAT":"DD-MM-YYYY","TIME_FORMAT":"HH24:MI:SS"}Runningempty text{"eventName":"pipe_5b4b8cdb-a308-4919-ac42-486d4bbcd879_525ffe04-97bb-473e-8e76-fa559faa7d0d","prefix":"ingestionE2E/232","eventTypes":["s3:ObjectCreated:Put","s3:ObjectCreated:Post"],"destination":"arn:aws:sns:us-west-2:1234asad:dremio-69505c78-17f8-456e-ba37-ee72a151fec8"}aws s3api put-bucket-notification-configuration --bucket tlelek-test-bucket --notification-configuration "$(notification=$(aws s3api get-bucket-notification-configuration --bucket pipetest); if [ -z "$notification" ]; then echo '{"TopicConfigurations": []}'; else echo "$notification"; fi | jq --argjson new_config '{"Id":"pipe_5b4b8cdb-a308-4919-ac42-486d4bbcd879_525ffe04-97bb-473e-8e76-fa559faa7d0d","TopicArn":"arn:aws:sns:us-west-2:1234abc:dremio-69505c78-17f8-456e-ba37-aa72a151fec8","Events":["s3:ObjectCreated:Put","s3:ObjectCreated:Post"],"Filter":{"Key":{"FilterRules":[{"Name":"prefix","Value":"ingestionE2E/232"}]}}}' '.TopicConfigurations += [$new_config]')"2024-05-20 15:34:182024-05-25 5:46:52user@customer.comuser@customer.com09ff36d0-0d53-4f87-8a05-0211d9fab76c

Columns

ColumnData TypeDescription
org_idvarcharThe unique identifier of the organization in which the autoingest pipe was created.
project_idvarcharThe unique identifier of the project in which the autoingest pipe was created.
pipe_namevarcharThe name of the autoingest pipe.
pipe_idvarcharThe unique identifier of the autoingest pipe.
source_pathvarcharThe preconfigured source path specified in the CREATE PIPE statement.
target_tablevarcharThe name of the target Iceberg table.
target_branchvarcharThe name of the target branch if Arctic was used as a catalog.
dedup_lookback_periodintegerThe number of days that Dremio looks back when comparing newly uploaded files to previously loaded files for file deduplication.
file_formatvarcharThe file format associated with the pipe.
format_optionsvarcharThe specific file format options that are used in the COPY INTO statement when the pipe is triggered.
pipe_statusvarcharThe state of the pipe. Enum:
  • Running: The pipe is running as expected.
  • Paused: The pipe exists but loads are not triggered when files are uploaded to the source.
  • Stopped_Missing_Table_or_Branch: Dremio has stopped the pipe because either the target table or target branch does not exist.
  • Stopped_Storage_Location_Altered: Dremio has stopped the pipe because a user has altered the source root path.
  • Stopped_Access_Denied: Dremio has stopped the pipe because Dremio cannot access the source location.
  • Stopped_Missing_Dremio_Source: Dremio has stopped the pipe because the source referenced in the pipe no longer exists.
  • Unhealthy: Autoingestion is failing due to an unknown root cause. Often a transient issue, and the system usually self-corrects and catches up.
  • Stopped_Internal_Error: The pipe is stopped because all pipe jobs failed within 5 days due to an unknown cause.
error_messagevarcharThe last error message for a pipe job that did not complete successfully.
cloud_settingstimestampThe configuration settings for connecting to the cloud environment.
cloud_cli_command_settingsvarcharThe command that uses the cloud_settings to set up autoingest pipes in the user's cloud storage location.
created_attimestampThe date and time at which the pipe was created.
last_updated_attimestampThe date and time of the last modification to the pipe definition.
created_byvarcharThe user who created the pipe.
last_updated_byvarcharThe user who last updated the pipe definition.
created_by_idvarcharThe ID of the user who created the pipe.