On this page

    Connecting Apache Spark to Arctic preview

    To connect Apache Spark to Arctic, use the configuration instructions provided in Project Nessie for Spark via Iceberg and edit the following information:

    • The full url of the Nessie API Endpoint to Nessie: Replace the example URL in the configuration with the Nessie API Endpoint.
    • The authentication type: Set the Nessie authentication type to BEARER.
    • The authentication token: Provide the personal access token you generated in your Dremio Cloud organization. If you have not created a personal access token, see Personal Access Tokens for information about how Dremio Cloud uses these tokens and how to generate one.

    Refer to the following table for the configuration settings required for Arctic, which are based on the Project Nessie documentation examples that provide the language-specific syntax (Java, Python).

    Replace the following:

    • Replace <catalog_name> with a unique value that is valid in Spark SQL (for example, nessie).

    note:

    Project Nessie uses the example catalog name of nessie.

    • Replace <project-id> with the Project ID of the Arctic project.
      For example: 06eg3b55-cc99-4c96-9acb-6f0ce010a81e

    To obtain the Arctic project ID:

    1. Sign in to Dremio Cloud.

    2. From the side navigation bar, select an Arctic project.

    3. Select the Settings (gear) icon.

    4. From the General Information tab, copy the Project ID and replace 〈project-id〉.

    • Replace <personal-access-token> with your personal access token. See Creating a Token to learn how to generate one.
      For example: RDViJJHrS/u+JAwrzQVV2+kAuLxiNkbTgdWQKQhAUS72o2BMKuRWDnjuPEjACw==

      Configuration property nameValue
      spark.sql.catalog.<catalog_name>.urihttps://nessie.dremio.cloud/v1/projects/<project-id>
      spark.sql.catalog.<catalog_name>.authentication.typeBEARER
      spark.sql.catalog.<catalog_name>.authentication.token<personal-access-token>

    Write and Read Tables in Apache Spark

    To learn about the available support for writing tables in the different versions of Spark, see Writing in the Project Nessie website.

    To read tables in Apache Spark, see Reading in the Project Nessie website.