This topic provides information on deploying Dremio on AWS.
[info] Prerequisite: AWS Account
AWS Edition Overview
Dremio AWS Edition is a separate Dremio edition that is provisioned via the AWS Marketplace. It comes in both Community (free) and Enterprise versions.
Dremio AWS Edition is made up of the following components, all of which exist within your existing AWS account:
Projects - Projects are persistent objects that contain all customer-specific definitions and metadata, to include user definitions, VDS and PDS definitions, reflection definitions, logs, and admin settings. An account can have one or more Projects, all of which can be considered as separate “instances” of Dremio. No data or resources are shared between Projects. Projects can be opened or closed depending on need, and by default are automatically backed up once a day. AWS services used: EBS, EFS, S3.
Coordinator - The coordinator node contains the core Dremio application and is the “brains” of the cluster. It issues commands to one or more engines. It is responsible for serving the Dremio’s UI, query planning, handling client connections, and managing metadata. The coordinator node is ephemeral, and can be shut down if it is not in use (to save money, for example), and can be restarted as needed. A coordinator node needs to be running in order to open and use a Project. AWS services used: EC2
Engines - Engines are the workers of the cluster and they execute and process query jobs issued from the coordinator node. Engines are “elastic” in the sense that they are made up of one or more nodes (EC2 instances) and can be automatically started and stopped. Engines can be managed to process specific workloads. AWS services used: EC2
Data Sources - Data sources are where physical data resides. Data sources can either be Data Lakes (like S3), or External Sources like NoSQL databases, relational databases, and local filesystems. In Dremio, the physical data remains in the Data Sources.
AWS Marketplace Deployment
The AWS Edition provides the ability to quickly and easily deploy Dremio through the AWS Marketplace.
In order to deploy Dremio through the AWS Marketplace, the following need to be done from within your Amazon AWS account:
The following instructions can be used to configure a Dremio instance through the AWS Marketplace:
Managing Dremio AWS Edition
- Create and Open a Project
- Open an Existing Project
- Close an Open Project
- Delete a Project
- Managing Engines
- Upgrade to Enterprise version
- Configuring LDAP in AWS Edition