Managing Projects

A project isolates compute, data, and other resources needed by a team for data analysis. An organization may contain multiple projects. A project must be linked to a single cloud account. Your first project was created and linked to your cloud account as part of the sign-up process.

Each project requires access to an Amazon S3 bucket or Azure Storage container for storing various types of data, including:

• The data for reflections that are created in the project
• The default path for new tables that are used for data and manifests for datasets
• All of the tables that store records of events and other historical data

Each project in your organization contains a preview engine. The preview engine is always on by default, which incurs compute costs. You can manually disable this engine, or any other engine, under Project Settings > Engines.

Project Visibility and Access

New projects are private by default. In the Sonar Projects page, users can see only the projects for which they have USAGE or OWNERSHIP privileges. The Sonar Projects page is empty for users who do not have USAGE or OWNERSHIP privileges on any projects. The project dropdown list also includes only projects for which the current user has USAGE and OWNERSHIP privileges.

Similarly, the Projects API returns an HTTP 403 Forbidden error for requests from users who do not have USAGE or OWNERSHIP privileges on the project. Also, users must have USAGE or OWNERSHIP privileges on a project before they can make API requests or run SQL queries on any objects in the project, even if they have object-level privileges on sources, folders, or other objects in the project.

To allow users to access a project, use the GRANT TO ROLE or GRANT TO USER SQL command or the Grants API to grant them the USAGE privilege on the project. For users who do not own the project, USAGE is the minimum privilege required to perform any operation on the project and the objects the project contains. For example, if you are using GRANT TO USER, you can run GRANT USAGE ON PROJECT TO USER <username>.

Inactive Projects (Automatically Deactivated)

Dremio automatically deactivates any project that has not been accessed in the last 15 days. Dremio sends a courtesy email to project owners three days prior to deactivation. Inactive projects are displayed in the project selector in the side navigation bar and on the Projects page.

• An inactive project will be activated automatically when any user tries to access it via the Dremio console, an ODBC or JDBC connection, or an API call.

• You can activate a project manually from the Projects page.

• It takes approximately five minutes to activate a project.

note

Inactive projects do not consume any compute resources and do not incur any costs in your cloud account.

Manually Archived Projects

Users with OWNERSHIP privileges or users assigned to the ADMIN role can archive a project. Archived projects are displayed only on the Projects page.

• An archived project will not be restored automatically if a user tries to access it.

• An archived project can only be restored manually by a user with OWNERSHIP privileges on the project or users assigned to the ADMIN role.

• It takes approximately five minutes to restore an archived project.

note

Archived projects do not consume any compute resources and do not incur any costs in your cloud account.

Viewing All Projects

To view the list of all projects, click in the side navigation bar and select Sonar Projects.

The Sonar Projects page displays the status of all projects in your organization, as follows:

• Creating
• Active
• Inactive
• Deactivating
• Activating
• Archiving
• Archived
• Restoring

Adding a Project

Before you add a project, ensure there is a project store in your cloud provider for storing metadata and Dremio reflections.

• For AWS, if you need to create an S3 bucket, follow the steps in AWS Resources: Creating a Project Store.
• For Azure, if you need to create a container in Azure storage, follow the steps in Azure Resources: Creating a Project Store.

note

For AWS, the CloudFormation template (CFT) is the recommended method to configure cloud resources. For a breakdown of the resources that will be created and the permissions that will be granted, see the annotated CloudFormation template.

If you would like to create the cloud resources and Sonar project manually, see Adding a Project Manually.

caution

Once you create the project, you cannot move it to a different cloud.

To add a project:

1. In the top-right corner of the Sonar Projects page, click Add Project.

2. For Project name, specify a name that is unique within the organization.

3. For Arctic catalog name, enter a name for your catalog.

4. For Cloud, select the cloud in which to create the project store and compute engines for running queries against datasets in the project.

To continue the configuration, follow the steps for your cloud provider.

AWS

5. Click Launch CFT, which will navigate you to the AWS Console. For next steps, see Adding a Project Using the CFT.


6. On the "Quick create stack" page, for Stack name, enter a name to identify the stack. This name must be unique to your account and cannot include underscores.


7. (Optional) For Project Store, a unique name has been generated for the S3 bucket that will be created to store metadata. You can specify a different name here but it must be a unique name across AWS.


8. (Optional) For Encryption Type, SSE-S3 is selected by default to encrypt the project store S3 bucket. You can select SSE-KMS (AWS Managed Key) or SSE-KMS (Customer Managed Key) if additional encryption is required.


9. (Optional) If you select SSE-KMS (Customer Managed Key), specify the KMS Key ARN.


10. Select I acknowledge that AWS CloudFormation might create IAM resources.


11. Click Create stack.

Stack creation can take up to four minutes. When stack creation is complete, the required cloud resources have been configured successfully.

Azure

5. For Tenant ID, paste the ID of the Azure tenant.


6. For Project Store, paste the name of the storage container that you want to use for the project store.


7. For Account Name, paste the name of the storage account that you want to use for the project store.


8. For Application (client) ID, paste the application (client) ID of the registered application.


9. For Client Secret, paste the value of the client secret that you created in the registered application.


10. (Optional) If you want to use a tailored set of permissions, copy the policy template to create a custom role for Dremio Cloud to manage storage resources.


11. Click Add.

It will take 4-5 minutes for your project to complete the setup process.

Adding an AWS Project Manually

To add an AWS project manually:

1. In the top-right corner of the Sonar Projects page, click the Add Project button.

2. For Project Name, specify a name that is unique within the organization.

3. For Cloud, select the cloud in which to create the project store and compute engines for running queries against datasets in the project.

4. Click the link at the bottom of the Add Project dialog. A Storage Access section will appear in the Add Project dialog.

5. For Project Store, paste the path of the Amazon S3 bucket. Doing so sets the path in the Resource section of the policy JSON. The rest of the policy JSON sets the minimum policy requirement to allow Dremio Cloud read and write access to the project store.

6. Copy the content of the Policy JSON text box.

7. To grant Dremio Cloud read and write permissions to the project store, follow either of these two sets of steps:

   • To grant the permissions by means of an IAM user:

     a. For Project Data Credentials, select Access Key.

     b. Create an IAM user with an access key.

     c. For AWS Access Key ID, paste the Access Key ID from the .csv file you downloaded when you created the user.

     d. For AWS Secret Access Key, paste the AWS Secret Access Key from the .csv file you downloaded when you created the user.

   • To grant the permissions by means of a cross-account IAM role:

     a. For Project Data Credentials, select IAM Role.

     b. Copy this JSON snippet and paste it into a location that you can access for the rest of this procedure.

     Policy JSON

         {
           "Effect": "Allow",
           "Principal": {
             "AWS": "arn:aws:iam::<trust account ID>:root"
           },
           "Action": "sts:AssumeRole",
           "Condition": {
             "StringEquals": {
               "sts:ExternalId": "<external ID>"
             }
           }
         }

     c. Copy the trust account ID and the external ID and paste them into the JSON that you copied in the previous step.

     d. Create a cross-account IAM role in the AWS Console.

     e. For AWS Cross-Account Role ARN, paste the cross-account role ARN that you copied when you created the role.

     f. For AWS Instance Profile ARN, paste the instance profile ARN that you copied when you created the role.

8. Click Add. A new project has been manually added to your organization.

Activating a Project

You can activate an inactive project on the Projects page, or by clicking the project in the project selector.

To activate a project from the Projects page:

1. Click the Dremio Services icon in the side navigation bar and select Sonar Projects.

2. Click the ellipsis menu to the far right of the inactive project, and then click Activate Project.

   The project status will change to Activating while the project is activated. You can access the project after the status changes to Active.

To activate a project from the project selector:

1. Hover over the project selector in the side navigation bar.

2. Click the name of an inactive project.

   Dremio displays a page that tells you the project had been deactivated and it will be activated in a few minutes. When the project has been activated, Dremio displays the project home page in the Datasets view.

Archiving a Project (Project Owners and Admin Users Only)

note

You cannot archive an Arctic project or the default project.

1. Click the Dremio Services icon in the side navigation bar and select Sonar Projects.

2. Click the ellipsis menu to the far right of an active or inactive project, and then click Archive Project.

   The project status will change to Archiving while the project is archived. When archiving is complete, the status changes to Archived.

Restoring an Archived Project (Project Owners and Admin Users Only)

1. Click the Dremio Services icon in the side navigation bar and select Sonar Projects.

2. Click the ellipsis menu to the far right of an archived project and select Restore Project.

   The project status will change to Restoring while the project is restored. You can access the project after the status changes to Active.

Setting the Default Project

When your data consumers connect to Dremio Cloud from BI tools, they need to connect to the projects in which their datasets are located. They can either connect to the default project or they can select a different project to connect to.

If an organization administrator does not set this value, Dremio Cloud automatically sets the default project to the oldest project in your organization.

You can change the default project at any time.

note

Data consumers who do not have access to the default project must select a project ID for an alternative project when they connect to Dremio Cloud from their BI tools.

To specify the default project for your organization:

1. Click the Organization icon in the side navigation bar.

2. Click the Settings icon at the top of the Organization page.

3. Select General Information in the organization settings sidebar.

4. In the Default Project field, select the project that you want data consumers to connect to by default through their BI tools.

5. Click Save.

Obtaining the ID of a Project

A BI client application might require the ID of a project as part of the information for creating a connection to Dremio Cloud. You can obtain the ID from the General Information page of a project's settings.

To obtain a project ID:

1. Click the Project Settings icon in the side navigation bar.

2. Select General Information in the project settings sidebar.

3. Copy the value in the Project ID field.

Editing a Project

The name is the only editable attribute of a project.

To edit the name of a project:

1. Click the Project Settings icon in the side navigation bar.

2. Select General Information in the project settings sidebar.

3. In the Project Name field, change the name and click Save.

Deleting a Project

If you delete the default project for your organization, Dremio Cloud automatically selects the oldest project to be the new default. You can override this default. See Setting the Default Project for an Organization.

note

Every organization must contain at least one project. If your organization contains only the project that you want to delete, then you cannot delete the project until another project is created in the organization.

To delete a project:

1. Click the Project Settings icon in the side navigation bar.

2. Select General Information in the project settings sidebar.

3. Click Delete Project.

4. Confirm that you want to delete the project.