Limits

This page details limits for a variety of Dremio resources on a per-deployment basis.

note

System limits are hard limits enforced by Dremio. Recommended limits are not enforced, but surpassing these limits can lead to cluster instability.

Limits are grouped according to Dremio components, as follows:

• Catalog
• Execution
• Metadata
• Planner
• Workload Management

Catalog

Item	Limit	Limit Type	Additional Notes
Autoingest Pipes	100	System	Maintains coordinator stability. If you would like to increase the limit, please contact Dremio.
Field size in records	32 KB	System	Prevents excessive heap usage.
Leaf columns in a table	6,400	System	See Creating and Querying Wide Tables.
Sources	250	Recommended	Adding too many sources can impact performance in the Dremio console and the size of the KV store on disk.
Spaces	500	Recommended	Exceeding this limit can cause degraded response times in the Dremio console.
Folders in a space	250	Recommended	Adding too many folders can impact performance in the Dremio console and the size of the KV store on disk.
Views in a folder	500	Recommended	Adding too many views can impact performance in the Dremio console and the size of the KV store on disk.
Size of uploaded Excel file (.xlsx)	10 MB	System	Maintains coordinator stability.
Size of uploaded files (other formats)	50 MB	Recommended	Maintains coordinator stability.
JSON, CSV, or TXT files	300,000	Recommended	Maintains coordinator stability (metadata collection).
Scripts per user	1,000	System	Guardrail to prevent excessive space usage in the KV store.
Starred datasets	25	System	Guardrail to prevent excessive space usage in the KV store.
Wikis and Labels	5,000	Recommended	Guardrail to prevent excessive space usage in the KV store.
Local roles	1,000	Recommended	Guardrail to prevent excessive space usage in the KV store.
Local users	1,000	Recommended	Guardrail to prevent excessive space usage in the KV store.
Catalog name length	256 characters	Recommended	256 characters is an industry standard (prevents excessive space usage in the KV store).
Script size	250,000 characters	System	Prevents excessive heap usage.

Execution

Item	Limit	Limit Type	Additional Notes
Leaf columns that can be queried	800	System	See Creating and Querying Wide Tables.
Array size (Parquet)	128	System	Too many elements in a LIST or ARRAY can exceed direct memory allocation.
Query records returned via console or REST API	1,000,000	System	Can lead to excessive disk usage. Use ODBC or JDBC to see full result set.

Metadata

Item	Limit	Limit Type	Additional Notes
Number of CSV/JSON/TXT datasets promoted to tables	1,000	Recommended	Prevents stale data due to long-running metadata collection on the coordinator (full metadata collection).
Number of Parquet datasets promoted to tables	5,000	Recommended	Prevents stale data due to long-running metadata collection on executors (incremental metadata collection).
Minimum background refresh frequency	Duration of one refresh	Recommended	Background refresh frequency should be greater than the time it takes for a refresh to complete, otherwise refresh may run constantly.
Parquet footer	16 MB	System	Parquet footers are read on heap. If a Parquet file is mostly footer you should adjust ETL to create larger files.
Partitions per HDFS and Hive source	300,000	System	HDFS and Hive sources only — to prevent excessive heap usage.
Promoted Delta or Iceberg tables	5,000	Recommended	Prevents stale data due to long-running metadata collection (incremental metadata collection).

Planner

Item	Limit	Limit Type	Additional Notes
Nodes in a query plan	25,000	System	Large plans can impact planning times, holding up incoming queries and impacting heap usage.
Number Limits	Varies by data type	System	See Data Types.
Varchar and Varbinary Limits	32,000	System	See Data Types.

Workload Management

Item	Limit	Limit Type	Additional Notes
Total active queries (queued and executing)	1,000	System	If more than 1,000 total active queries, consider adding another cluster with dedicated engines.