Skip to main content

Scratch Directory

Dremio Sonar provides a scratch directory in which tables can be saved. The scratch directory provides a way to work with tables and do ad hoc analysis that you do not want to share with other users in the project. This scratch directory is created in the project store (that is, the Amazon S3 bucket) that was set up when you initially set up your Dremio account.

There are no security protocols or permissions associated with the scratch location; it is readable and writable by all users.

Identifying the Scratch Location​

When creating a table, you can specify the scratch directory as the location to store this new table. When the table is created, the output of the operation includes the path to the scratch directory. For more information, see the following topic Working in the Scratch Directory.

You can also identify the path to the scratch directory by using the Catalog API. Specifically, you use the get catalog by path call to return a list of tables that is in the directory. For information about using this API, see Retrieving a Catalog Entity By Path.

Working in the Scratch Directory​

When you create a table in the scratch directory, it is created as a Parquet file.

To create a table in the scratch directory, include $scratch as part of the <table_path> parameter. For example:

Create a table in the scratch directory
CREATE TABLE \$scratch.example_table AS
  SELECT *
  FROM Samples."samples.dremio.com"."NYC-taxi-trips"
  LIMIT 3
note

The example uses Dremio’s Samples Data Lake, which includes a number of sample datasets that you can practice with.

To locate your newly created table in the scratch directory, review the output of the operation and note the Path column (as identified by the pin in the following screenshot).

Tables are created as Parquet files. To work with these tables, you must format the Parquet file to a table. For more information, see Formatting Data. After formatting the dataset, you can query the data using SQL commands.

Tables created in the scratch directory will persist until you manually drop them.

Drop a table from the scratch directory
DROP TABLE \$scratch.example_table