Manage Sources

Dremio supports a variety of data sources, including lakehouse catalogs, object storage, and databases.

As the Dremio Shared Responsibility Models outline, metadata is a shared responsibility between Dremio and you. The Shared Responsibility Models lay out Dremio's responsibilities for enabling data source configurations and your responsibilities for managing metadata.

Dremio Catalog

Dremio comes with a built-in lakehouse catalog, built on Apache Polaris (incubating). The catalog enables centralized, secure read and write access to your Iceberg tables across various REST-compatible query engines and automates data maintenance operations to maximize query performance.

• Dremio Catalog

Lakehouse Catalogs

Lakehouse catalogs provide you the ability to connect to centralized catalogs. Dremio Catalog, Snowflake Open Catalog, Unity Catalog, and Iceberg REST Catalog all connect to the destination sources over the Apache Iceberg REST API.

• AWS Glue Data Catalog
• Dremio Catalog (External)
• Hive
• Iceberg REST Catalog
• Nessie
• Snowflake Open Catalog
• Unity Catalog

Object Storage

• Amazon S3
• Azure Storage
• Google Cloud Storage
• HDFS
• NAS

Databases

• Amazon OpenSearch Service
• Amazon Redshift
• Apache Druid
• Dremio Cluster (you can connect to one or more other Dremio Software clusters and run queries on the data sources to which they are connected, and you can run queries that federate data across connected clusters)
• Google BigQuery
• Elasticsearch
• IBM Db2
• Microsoft Azure Data Explorer
• Microsoft Azure Synapse Analytics
• Microsoft SQL Server
• MongoDB
• MySQL
• Oracle
• PostgreSQL
• SAP HANA
• Snowflake
• Teradata
• Vertica

Dremio enables users to run external queries, queries that use the native syntax of the relational database, to process SQL statements that are not yet supported by Dremio or are too complex to convert. Dremio administrators enable the feature for each data source and specify which Dremio users can edit that source. See Querying Relational-Database Sources Directly for more information.

Dremio improves query performance for relational database datasets with Runtime Filtering, which applies dimension table filters to joined fact tables at runtime.

note

• Decimal Support: Decimal-to-decimal mappings are supported for relational database sources.
• Collation: Relational database sources must have a collation equivalent to LATIN1_GENERAL_BIN2 to ensure consistent results when operations are pushed down. For non-equivalent collations, create a view that coerces the collation to one that is equivalent to LATIN1_GENERAL_BIN2 and access that view.
• For all sources, case-sensitive source data file/table names are not supported. In Dremio, case is ignored in the names of data files. file1.parquet, File1.parquet, and FILE1.parquet are considered to be equivalent names. Therefore, searching on one of these names can result in unanticipated results.
  In addition, columns in a table that have the same name with different cases are not supported. For example, if two columns named Trip_Pickup_DateTime and trip_pickup_datetime exist in the same table, one of the columns may disappear when the header is extracted.

Files and Folders

• Formatting Data to a Table

• Uploading Files

  note Case-sensitive source data file/table names are not supported. In Dremio, data filenames in your data source are "seen" in a case-insensitive manner. So, if you have three file names with difference cases (for example, JOE Joe, and joe), Dremio "sees" the files as having the same name. Thus, searching on Joe, JOE, or joe, can result in unanticipated data results.
  
  In addition, columns in a table that have the same name with different cases are not supported. For example, if two columns named Trip_Pickup_DateTime and trip_pickup_datetime exist in the same table, one of the columns may disappear when the header is extracted.