pipe_id | varchar | The unique identifier of the autoingest pipe. |
pipe_name | varchar | The name of the autoingest pipe. |
source_path | varchar | The preconfigured source path specified in the CREATE PIPE command. |
target_table | varchar | The name of the target Iceberg table. |
target_branch | varchar | The name of the target branch if Nessie was used as a catalog. |
dedup_lookback_period_days | integer | The number of days that Dremio looks back when comparing newly uploaded files to previously loaded files for file deduplication. |
file_format | varchar | The file format associated with the pipe. |
format_options | varchar | The specific file format options that are used in the COPY INTO statement when the pipe is triggered. |
pipe_status | varchar | The state of the pipe. Enum: RUNNING : The pipe is running as expected.PAUSED : The pipe exists but loads are not triggered when files are uploaded to the source.STOPPED_MISSING_TABLE_OR_BRANCH : Dremio has stopped the pipe because either the target table or target branch does not exist.STOPPED_STORAGE_LOCATION_ALTERED : Dremio has stopped the pipe because a user has altered the source root path.STOPPED_ACCESS_DENIED : Dremio has stopped the pipe because Dremio cannot access the source location.STOPPED_MISSING_DREMIO_SOURCE : Dremio has stopped the pipe because the source referenced in the pipe no longer exists.UNHEALTHY : Autoingestion is failing due to an unknown root cause. Often a transient issue, and the system usually self-corrects and catches up.STOPPED_INTERNAL_ERROR : The pipe is stopped because all pipe jobs failed within 5 days due to an unknown cause.
|
error_message | varchar | The last error message for a pipe job that did not complete successfully. |
notification_provider | varchar | The provider of the event notification queue. Enum: AWS_SQS . |
notification_queue_ref | varchar | The unique identifier of the event notification queue. |
created_at | timestamp | The date and time at which the pipe was created. |
last_updated_at | timestamp | The date and time of the last modification to the pipe definition. |
created_by | varchar | The user who created the pipe. |
last_updated_by | varchar | The user who last updated the pipe definition. |