Skip to main content

Apache Flink

To connect Apache Flink to Arctic, use the configuration instructions provided in Project Nessie for Flink via Iceberg and edit the following information:

  • The full url of the Nessie API Endpoint to Nessie: Replace the example URL in the configuration with the Arctic catalog API Endpoint.

  • The authentication type: Set the Nessie authentication type to BEARER.

  • The authentication token: Provide the personal access token you generated in your Dremio Cloud organization. If you have not created a personal access token, see Personal Access Tokens for information about how Dremio Cloud uses these tokens and how to generate one.

    note

    The personal access token identifies the Dremio user who can read and write data as well as the user's role-based access control (RBAC) permissions on the Arctic catalog. Dremio enforces the user's existing Arctic RBAC permissions when they read and write data through the Apache Flink engine.

Replace the following properties in the Project Nessie configuration example:

  • Set the location of the Nessie server using the uri property of the catalog with the Arctic catalog API Endpoint.
    For example: https://nessie.dremio.cloud/repositories/80d313e4-fa73-4252-a757-e50174f86000/api/v1

    To obtain the Arctic catalog API Endpoint:

    1. Sign in to Dremio Cloud.

    2. Click This is the icon that represents the organization. in the side navigation bar.

    3. On the Organization page, select the option to see the Arctic Catalogs.

    4. On the Arctic Catalogs page, hover over the Arctic Catalog and click Catalog Settings This is the icon that represents the Catalog settings. next to the Arctic Catalog.

    5. Under General Information in the sidebar, copy the Catalog Endpoint.

  • Replace <arctic-catalog-api-endpoint> with the actual Catalog Endpoint that you copied earlier.

  • Replace <personal-access-token> with your personal access token. See Creating a Token to learn how to generate one.
    For example: RDViJJHrS/u+JAwrzQVV2+kAuLxiNkbTgdWQKQhAUS72o2BMKuRWDnjuPEjACw==

Flink Configuration 
CREATE CATALOG <catalog_name> WITH (
   'type'='iceberg',
   'catalog-impl'='org.apache.iceberg.nessie.NessieCatalog',
   'uri'='<arctic-catalog-api-endpoint>',
   'ref'='main',
   'warehouse' = '/path/to/flink/warehouse',
   'authentication.type' = 'BEARER',
   'authentication.token' = '<personal-access-token>'
    )

To create tables in Apache Flink that are managed by Arctic, see Create tables in the Project Nessie website.

To read tables in Apache Flink, see Reading tables in the Project Nessie website.

To learn more about the available Data Definition Language (DDL) commands in Apache Flink and the SQL queries you can run, see DDL commands in the Apache Iceberg documentation.