Migrate AWSE to Kubernetes Enterprise
This page explains migrating from Dremio AWS Edition (AWSE) to Dremio Enterprise Edition on Kubernetes using the dremio-admin
command line interface (CLI).
To migrate from AWSE to a Kubernetes deployment:
Back up your AWSE instance to
Back up AWSE instance/tmp/backup
usingdremio-admin backup
. Replace the<DREMIO_ADMIN_USERNAME>
and<DREMIO_ADMIN_PASSWORD>
placeholders with the correct information for your instance before you run the command./opt/dremio/bin/dremio-admin backup -u <DREMIO_ADMIN_USERNAME> -p <DREMIO_ADMIN_PASSWORD> -d /tmp/backup
Compress the backup to a tarball and copy it to
Compress AWSE backup to tarball and copy/tmp/backup
:tar czvf backup.tar /tmp/backup/
Spawn the
Spawn dremio-admin poddremio-admin
pod in the cluster using Helm. Replace the<NAMESPACE>
placeholder with the correct information for your Kubernetes cluster before you run the command.helm upgrade --wait dremio . --set DremioAdmin=true -n <NAMESPACE>
Copy the AWSE backup to the
Copy backup to dremio-admin pod/tmp/
directory on thedremio-admin
pod.kubectl cp backup.tar <NAMESPACE>/dremio-admin:/tmp/
Open and access the shell of the
Spawn dremio-admin poddremio-admin
pod. Replace the<NAMESPACE>
placeholder with the correct information for your Kubernetes cluster before you run the command.kubectl exec -it --namespace <NAMESPACE> dremio-admin bash
Update the
Distributed path value in dremio.confpaths.dist
property in the dremio.conf file for your Kubernetes deployment so that the value matches the distributed path of your AWSE instance. For example:paths: {
# The local path for dremio to store data.
local: "/data/dremio",
# The distributed path of Dremio data including job results, downloads, uploads, etc
# dist: "dremiogcs:///k8s-dac-dist/jenkins-k8s-dial-a-cluster-1111"
dist: "dremioS3:///dremio-me-2e1f8e3b-1a88-4a67-b419-a72f25907b39-11c3900b7924d03a/dremio"
# Storage area for the accelerator cache.
accelerator: ${paths.dist}/accelerator
# Staging area for json and csv ui downloads.
downloads: ${paths.dist}/downloads
# Stores uploaded data associated with user home directories.
uploads: ${paths.dist}/uploads
# Stores data associated with the job results cache.
results: ${paths.dist}/results
}
services: {
coordinator: {
master: {
enabled: true
},
enabled: true
}
executor: {
enabled: false
}
}
zookeeper: "dremiononyarn-0:"${services.coordinator.master.embedded-zookeeper.port}In the
core-site.xml/opt/dremio/conf/
directory in your Kubernetes deployment, create a file namedcore-site.xml
that contains the following XML. This ensures that Dremio can access the S3 path specified in dremio.conf.<?xml version="1.0"?>
<configuration>
<property>
<name>fs.dremioS3.impl</name>
<description>The FileSystem implementation. Must be set to com.dremio.plugins.s3.store.S3FileSystem</description>
<value>com.dremio.plugins.s3.store.S3FileSystem</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<description>AWS access key ID.</description>
<value>accessKey</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>AWS secret key.</description>
<value>secretKey</value>
</property>
<property>
<name>fs.s3a.aws.credentials.provider</name>
<description>The credential provider type.</description>
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
</configuration>Empty the
Empty db directory/opt/dremio/data/db
directory.cd /opt/dremio/data/db
rm -rf *noteThe
db
directory must exist in/opt/data/dremio
for the restore to work, so make sure to delete only the contents of thedb
directory and not the directory itself.Unzip the backup tarball and restore the backup to your Kubernetes deployment using
Spawn dremio-admin poddremio-admin restore-awse
. Replace the<BACKUP_FILENAME>
placeholder with the correct information for your instance before you run the command.untar /tmp/backup.tar
cd /opt/dremio/bin
./dremio-admin restore-awse -d /tmp/tmp/backup/<BACKUP_FILENAME>/Remove the
Remove dremio-admin poddremio-admin
pod. Replace the<NAMESPACE>
placeholder with the correct information for your Kubernetes cluster before you run the command.helm upgrade --wait dremio . --set DremioAdmin=false -n <NAMESPACE>
Wait for the Kubernetes deployment to start and confirm that it contains the data from your AWSE instance.
Log in to the Dremio console for your Kubernetes deployment.
Click in the left panel to open the settings.
Click the Queues tab.
Hover over a queue name and click at the right side of the page.
In the dropdown menu under Engine Name, name the new engine.
Click Save.
Repeat steps 15 through 16 for each queue.