Version: current [25.x]

Migrate AWSE to Kubernetes Enterprise

This page explains migrating from Dremio AWS Edition (AWSE) to Dremio Enterprise Edition on Kubernetes using the dremio-admin command line interface (CLI).

To migrate from AWSE to a Kubernetes deployment:

Back up your AWSE instance to /tmp/backup using dremio-admin backup. Replace the <DREMIO_ADMIN_USERNAME> and <DREMIO_ADMIN_PASSWORD> placeholders with the correct information for your instance before you run the command.
Back up AWSE instance
```
/opt/dremio/bin/dremio-admin backup -u <DREMIO_ADMIN_USERNAME> -p <DREMIO_ADMIN_PASSWORD> -d /tmp/backup
```
Compress the backup to a tarball and copy it to /tmp/backup:
Compress AWSE backup to tarball and copy
```
tar czvf backup.tar /tmp/backup/
```
Spawn the dremio-admin pod in the cluster using Helm. Replace the <NAMESPACE> placeholder with the correct information for your Kubernetes cluster before you run the command.
Spawn dremio-admin pod
```
helm upgrade --wait dremio . --set DremioAdmin=true -n <NAMESPACE>
```
Copy the AWSE backup to the /tmp/ directory on the dremio-admin pod.
Copy backup to dremio-admin pod
```
kubectl cp backup.tar <NAMESPACE>/dremio-admin:/tmp/
```
Open and access the shell of the dremio-admin pod. Replace the <NAMESPACE> placeholder with the correct information for your Kubernetes cluster before you run the command.
Spawn dremio-admin pod
```
kubectl exec -it --namespace <NAMESPACE> dremio-admin bash
```

Update the paths.dist property in the dremio.conf file for your Kubernetes deployment so that the value matches the distributed path of your AWSE instance. For example:

Distributed path value in dremio.conf

paths: {

  # The local path for dremio to store data.
  local: "/data/dremio",

  # The distributed path of Dremio data including job results, downloads, uploads, etc
  # dist: "dremiogcs:///k8s-dac-dist/jenkins-k8s-dial-a-cluster-1111"
  dist: "dremioS3:///dremio-me-2e1f8e3b-1a88-4a67-b419-a72f25907b39-11c3900b7924d03a/dremio"

  # Storage area for the accelerator cache.
  accelerator: ${paths.dist}/accelerator

  # Staging area for json and csv ui downloads.
  downloads: ${paths.dist}/downloads

  # Stores uploaded data associated with user home directories.
  uploads: ${paths.dist}/uploads

  # Stores data associated with the job results cache.
  results: ${paths.dist}/results
}

services: {
  coordinator: {
    master: {
      enabled: true
    },
    enabled: true
  }
  executor: {
    enabled: false
  }
}

zookeeper: "dremiononyarn-0:"${services.coordinator.master.embedded-zookeeper.port}

In the /opt/dremio/conf/ directory in your Kubernetes deployment, create a file named core-site.xml that contains the following XML. This ensures that Dremio can access the S3 path specified in dremio.conf.

core-site.xml

<?xml version="1.0"?>
<configuration>
<property>
  <name>fs.dremioS3.impl</name>
  <description>The FileSystem implementation. Must be set to com.dremio.plugins.s3.store.S3FileSystem</description>
  <value>com.dremio.plugins.s3.store.S3FileSystem</value>
</property>
<property>
  <name>fs.s3a.access.key</name>
  <description>AWS access key ID.</description>
  <value>accessKey</value>
</property>
<property>
  <name>fs.s3a.secret.key</name>
  <description>AWS secret key.</description>
  <value>secretKey</value>
</property>
<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <description>The credential provider type.</description>
  <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
</configuration>

Empty the /opt/dremio/data/db directory.
Empty db directory
```
cd /opt/dremio/data/db
rm -rf *
```
note
The db directory must exist in /opt/data/dremio for the restore to work, so make sure to delete only the contents of the db directory and not the directory itself.
Unzip the backup tarball and restore the backup to your Kubernetes deployment using dremio-admin restore-awse. Replace the <BACKUP_FILENAME> placeholder with the correct information for your instance before you run the command.
Spawn dremio-admin pod
```
untar /tmp/backup.tar
cd /opt/dremio/bin
./dremio-admin restore-awse -d /tmp/tmp/backup/<BACKUP_FILENAME>/
```
Remove the dremio-admin pod. Replace the <NAMESPACE> placeholder with the correct information for your Kubernetes cluster before you run the command.
Remove dremio-admin pod
```
helm upgrade --wait dremio . --set DremioAdmin=false -n <NAMESPACE>
```
Wait for the Kubernetes deployment to start and confirm that it contains the data from your AWSE instance.
Log in to the Dremio console for your Kubernetes deployment.
Click in the left panel to open the settings.
Click the Queues tab.
Hover over a queue name and click at the right side of the page.
In the dropdown menu under Engine Name, name the new engine.
Click Save.
Repeat steps 15 through 17 for each queue.