Skip to main content
Version: current [25.x]

SYS.PIPE_SUMMARY Enterprise

The sys.pipe_summary table summarizes high-level statistics for autoingest pipes. Each row in the table represents a single pipe and its historical metadata.

Syntax
SELECT * FROM sys.pipe_summary

Example Output

pipe_namepipe_idjobs_countfiles_loaded_countfiles_skipped_countfiles_partially_loaded_countpipe_statuserror_messagelast_updated_attotal_records_count
test_pipef0336bf3-f082-4c89-9753-7dcf66ce24721010020RUNNINGempty text2024-07-08 22:07:42.995100150

Columns

Column NameData TypeDescription
pipe_namevarcharThe name of the pipe.
pipe_idvarcharThe unique identifier of the pipe.
jobs_countintegerThe number of jobs completed for autoingestion related to this pipe.
files_loaded_countintegerThe number of files loaded successfully.
files_skipped_countintegerThe number of files skipped completely due to validation errors.
files_partially_loaded_countintegerThe number of files partially loaded.
pipe_statusvarcharThe state of the pipe. Enum:
  • Running: The pipe is running as expected.
  • Paused: The pipe exists but loads are not triggered when files are uploaded to the source. The user initiates this state by running ALTER PIPE and setting the pipe execution.
  • Stopped_Missing_Table_or_Branch: Dremio has stopped the pipe because either the target table or target branch does not exist.
  • Stopped_Storage_Location_Altered: Dremio has stopped the pipe because a user has altered the source root path.
  • Stopped_Access_Denied: Dremio has stopped the pipe because Dremio cannot access the source location.
  • Stopped_Missing_Dremio_Source: Dremio has stopped the pipe because the source referenced in the pipe no longer exists.
  • Unhealthy: Autoingestion is failing due to an unknown root cause. Often a transient issue, and the system usually self-corrects and catches up.
  • Stopped_Internal_Error: The pipe is stopped because all pipe jobs failed within 5 days due to an unknown cause.
error_messagevarcharThe last error message for a pipe job that did not complete successfully. Reasons include:
  • Missing (Target) Table
  • Missing Branch
  • Storage Location Altered
  • Notification Dropped
  • Access Denied
  • All Files Failed to Load
last_updated_attimestampThe date and time at which the pipe state changed, which should default to the pipe creation time.
total_records_countintegerThe number of records processed by the pipe.