Skip to main content

Prepare Your Data

You can bring curated data into Dremio by:

  • Landing it in Dremio's Open Catalog as Iceberg tables using your tool of choice, such as dbt, Fivetran, Confluent, or others.
  • Connecting your existing data sources to Dremio

Once this data is in Dremio, you can further cleanse, combine, transform, and aggregate your data using SQL functions in Dremio and create virtual representations of your data in the form of views.

Unified Data Access​

You can query and combine data across multiple sources and formats in Dremio. Dremio's Query Engine can federate queries across sources in real time without requiring ETL. This allows you to pull together Iceberg tables from Dremio's Open Catalog with existing data in other catalogs, object stores, and databases. For more information on supported data sources, see Connect to Your Data. You can then create views from the SQL that queries across your sources into your Open Catalog.

Process Unstructured Data​

Dremio allows you to process and combine structured, semi-structured, and unstructured data. Examples of unstructured data include PDFs, images, and videos that are stored in object storage.

Dremio natively offers AI functions to enable you to extract and process unstructured data:

  • AI_GENERATE – Process unstructured data, primarily for complex data extraction requiring multiple fields from source files.
  • AI_CLASSIFY – Categorize documents or analyze sentiment as VARCHAR values, using a provided classification list.
  • AI_COMPLETE – Generate text or create summaries as VARCHAR values.

These functions are processed using Dremio's Query Engine and the AI model provider of your choice. For more information on how to configure your model provider, see Configure Model Providers.

Create a View​

To create a virtual representation of your transformed and aggregated data, you can create a view by following these steps in the Dremio console:

  1. Click The SQL Runner icon in the side navigation bar to open the SQL Runner.

  2. Write a SQL query that transforms your data and click Run to validate the query.

  3. After the query has finished running, click the arrow next to Save as View in the top right of the SQL editor, and select Save as View... from the dropdown menu.

  4. In the Save View As dialog, name the new view and select from a list of folders where it will be stored.

You can also run the CREATE VIEW from the SQL Runner or your tool of choice to achieve the same results.