Dremio Cloud
Dremio Software
 
On this page

    Dremio Arctic

    Dremio Arctic is an intelligent metastore for Apache Iceberg, powered by Nessie. It provides a modern, cloud-native alternative to Hive Metastore.

    Adding a Dremio Arctic Source

    To add an Arctic source to your project:

    1. From the Datasets page, in the Sources section, click the plus sign (+).

    note:

    Alternatively, click Add Source at the bottom of the data panel on the Datasets page.

    1. In the Add Data Source dialog, Data-as-code, click Arctic (Preview).

    The New Arctic (Preview) Source dialog box appears, which contains the following sections:

    • General: Select an existing Arctic catalog or create a new one.

    • Storage: Use the default settings or, optionally, edit the authentication method, AWS root path, and other connection properties.

    • Advanced Options: Use the default settings or, optionally, configure access preferences and cache options.

    • Privileges: Add privileges for users or roles.

    Refer to the following for guidance on how to edit each section.

    General

    To configure an Arctic source:

    1. From the Choose a catalog menu, select an existing Arctic Catalog or select + New Arctic Catalog.

    2. If you are adding a new catalog, provide a name for the new catalog, then click Add.

    Storage

    Authentication Using Project Data Credentials

    Use project data credentials to enable Dremio to access Amazon S3 using the IAM role that is associated with your Dremio project. This IAM role was created when you signed up for Dremio and is the default credential that is used to access all the sources in your project.

    For this option, you will attach the necessary IAM policies to your Dremio project’s IAM role. The following policies are available:

    For instructions to set up these policies and attach them to an IAM role, see Set up AWS IAM Permissions.

    Authentication Using Data Source Credentials

    Use data source credentials to enable Dremio to access Amazon S3 using either a source-specific access key or an IAM role. This method provides you the flexibility to assign either an access key or an IAM role to each source that is in your Amazon S3 account.

    The following IAM policies are available:

    Choose one of the following authentication methods to access the data source. During this set up, you will attach the IAM policies to provide Dremio read and/or write access to the data source.

    • Use an access key: Create an IAM user in your AWS account. The access key is generated during the set up process. For the steps, see Creating an IAM role.

    • Use a new IAM role: For the steps to create a new role, see Creating an IAM user.

    • Use the Dremio project IAM role: To attach these policy templates to the Dremio project’s IAM role, see Set up AWS IAM Permissions.

    AWS Root Path

    The root path to the Amazon S3 bucket for the catalog is already filled in.

    Connection Properties (Optional)

    Provide the custom key value pairs for the connection relevant to the source.

    1. Click Add Property.

    2. For Name, enter a connection property.

    3. For Value, enter the corresponding connection property value.

    Advanced Options

    Review each option provided in the following table to set up the advanced options to meet your needs.

    Advanced Option Description
    Enable asynchronous access when possible Activated by default, uncheck the box to deactivate. Enables cloud caching for the S3 bucket to support simultaneous actions such as adding and editing a new source.

    Under Cache Options, review the following table and edit the options to meet your needs.

    Cache Options Description
    Enable local caching when possible Selected by default, along with asynchronous access for cloud caching. Uncheck the checkbox to disable this option. For more information about local caching, see Columnar Cloud Cache.
    Max percent of total available cache space to use when possible Specifies the disk quota, as a percentage, that a source can use on any single executor node only when local caching is enabled. The default is 100 percent of the total disk space available on the mount point provided for caching. You can either manually enter in a percentage in the value field or use the arrows to the far right to adjust the percentage.

    Privileges

    Click Privileges in the left menu sidebar. This section lets you grant privileges to specific users or roles. To learn more about how Dremio allows for the implementation of granular-level privileges, see Privileges.

    note:

    All privileges parameters are optional.

    To add a privilege to a user or role:

    • In the Add User/Role field, enter the user or role name that you want to apply privileges to and then click Add to Privileges. The user or role is added to the Users table.

    To set privileges for a user or role:

    1. In the Users table, identify the user to set privileges for and click under the appropriate column (Select, Alter, Create Table, etc.) to either enable or disable that privilege. A green checkmark indicates that the privilege is enabled.
    2. Click Save.

    Editing an Arctic Source

    Once an Arctic source has been created, you can make edits to it, when needed. To edit an Arctic source:

    1. From the Datasets page, click Data-as-code. A list of Arctic sources displays.

    2. On the All Data-as-code page, hover over an Arctic source to display the hidden Settings (gear) icon, then click the icon.

    note:

    Alternatively, you can hover over an Arctic source, click the More (…) option, and then click Settings.

    1. In the Source Settings dialog box, you can make changes to the settings for Storage, Advanced Options, or Privileges.

    note:

    You cannot change the catalog selected for a configured Arctic source.

    1. Click Save.

    Removing an Arctic Source

    To remove an Arctic source:

    1. From the Datasets page, click Data-as-code. A list of Arctic sources displays.

    2. On the All Data-as-code page, hover over an Arctic source, click the More (…) option, and then click Remove Source.

    3. Click Remove to confirm removal of the source, or click Cancel to return to the All Data-as-code page.