Skip to main content
Version: current [25.x]

SYS.COPY_FILE_HISTORY Enterprise

The sys.copy_file_history table contains the history of every file in a COPY INTO operation, including files loaded by an autoingest pipe. Each row in the table represents a single file and its load metadata.

Syntax
SELECT *
FROM sys.copy_file_history

Example Output

event_timestampjob_idfile_pathfile_staterecords_loaded_countrecords_rejected_countpipe_idfile_sizefirst_error_messagefile_notification_timestampingestion_source_typerequest_id
2024-05-22T10:47:44.84619b232b3-362b-f366-1d4a-772da8a70d00tlelek-test-bucket/ingestionE2E/217/s3:/tlelek-test-bucket/ingestion/1k_jsons/generated_json_f488.jsonSKIPPED000244f174-ce25-4876-98fc-8ac9ae53b29411000No such file or directory: s3a://tlelek-test-bucket/ingestionE2E/217/s3:/tlelek-test-bucket/ingestion/1k_jsons/generated_json_f488.json2024-05-20T15:41:33.873RESTb2c7c734-187e-4c32-a100-6f6529fdec74

Columns

Column NameData TypeDescription
event_timestamptimestampThe date and time at which the file was loaded in a COPY INTO operation. In cases where the COPY INTO command was initiated by an autoingest pipe, the event_timestamp value with the file_notification_timestamp value to determine the lag time between when the file is uploaded and when Dremio is notified about the file.
job_idvarcharThe unique identifier for the job associated with the file being loaded in the COPY INTO operation.
file_pathvarcharThe path for the file loaded by the autoingest pipe.
file_statevarcharThe state of the file. Enum:
  • IN_PROGRESS: Load is in progress. Applies only to files that are loaded by the autoingest pipe.
  • FULLY_LOADED: The entire file was loaded into a table with no errors. Applies only to files that are loaded by the autoingest pipe.
  • PARTIALLY_LOADED: At least one row of the file was loaded into a table, and at least one row of the file had an error. Does not apply to files that are loaded by the autoingest pipe.
  • SKIPPED: No rows from the file were loaded. Occurs when a file does not contain any rows or every row in the file had an error.
records_loaded_countintegerThe number of records loaded from the file.
records_rejected_countintegerThe number of records rejected from the file.
pipe_idvarcharThe unique identifier for the autoingest pipe.
file_sizeintegerThe file size in bytes.
first_error_messagevarcharThe first error message returned while the file was being loaded in the COPY INTO operation.
file_notification_timestamptimestampThe date and time at which Dremio received a notification that the file was loaded into cloud storage in cases where the COPY INTO command was initiated by an autoingest pipe.
ingestion_source_typevarcharThe type of source used for the autoingest pipe. Enum: AWS, REST.
request_idvarcharThe unique identifier of the request associated with REST API-based ingestion. The request_id value is NULL if you used the COPY INTO command to load the file.