Concepts

Physical and Virtual Datasets

Physical Datasets Physical datasets are stored within data sources . Physical datasets cannot be modified by Dremio.

Virtual Datasets Virtual datasets are derrived from physical datasets or other virtual datasets. Virtual datasets are defined by the steps necessary for their creation, including transformations, filters, joins, and other modifications. Because virtual datasets do not make a copy of the data, they use very little space, and they always reflect the current state of the physical datasets they are derrived from.

[success] Example

Suppose we have a collection on a MongoDB source called 'sales.' Inside Dremio, this collection will be represented as a physical dataset under 'sales.' We can open this dataset within Dremio, and then save it as a virtual dataset called 'salesRaw.' Later on we can derive another virtual dataset called 'salesNY' from 'salesRaw' by excluding all data that doesn't originate from the state of New York. 'salesRaw' and 'salesNY' can each be queried and will return different results, but they are both based on the same underlying physical dataset.

Spaces and Folders

Spaces Spaces are where virtual datasets are saved. Spaces provide a way to group datasets by a common theme such as a project, department, or geographic region. Each user also has a default Home Space.

For instance, a list of spaces for an online retailer might look like:

  • Users
  • Transactions
  • Products
  • Sales Analysis
  • Web Traffic

Folders You can use folders to provide a deeper layer of organization to spaces. Folders can contain other folders.

Paths in Dremio

Paths are a dot-separated list that indicates the location of a dataset, starting with the name of the source or space in which that dataset resides, followed by any folders or data source structures, and ending in the name of the dataset. Here are a few examples of what dataset paths look like in Dremio:

  • Transactions.regions.salesNY
  • Web Traffic.october.visits

Transactions is a space, regions is a folder, and salesNY is a virtual dataset. In the second example, Web Traffic is a file system data source, october is a directory on that file system, and visits is a sub-directory with a group of files in a common structure.

[info] TIP

SQL queries always reference data sets using their full path, for example SELECT * FROM "web traffic".october.visits.


results matching ""

    No results matching ""