Backup Dremio

Dremio metadata and user uploaded files can be backed up. The backup does not include the contents of the distributed cache such as acceleration cache, downloaded files and query results.

Requirements

  • The Dremio daemon is running.
  • The backup command is run on the coordinator node.
  • The backup path is writable by the user running Dremio daemon.

Restoring Dremio

A backup can only be restored using the same version of Dremio that the backup was created on.

Syntax

<dremio_home>/bin/dremio-admin backup
    -u <DREMIO_ADMIN_USER> \
    -p <DREMIO_ADMIN_PASS> \
    -d <BACKUP_PATH>

Options

To obtain a list of backup options on the command line:

./dremio-admin backup -h

Backup options:

    -a, --accept-all
      accept all ssl certificates
      Default: false
  * -d, --backupdir
      backup directory path. for example, /mnt/dremio/backups or 
      hdfs://$namenode:8020/dremio/backups 
    -h, --help
      show usage
    -i, --include-profiles
      include profiles in backup. Available as of Dremio 4.1.4
      Default: false
    -j, --json
      Use a json formatted backup (default binary). Available as of Dremio 4.1.4
      Default: false
    -l, --local-attach
        Attach locally to Dremio JVM to authenticate user. Available as of Dremio 3.3
        Not compatible with user/password options. 
    -p, --password
      password
    -u, --user
      username (admin)

Note

If the --password flag is specified without a value, the utility prompts for one interactively.

Example: Basic Backup

This following example backs up Dremio to the /tmp/backup directory:

/dremio-admin backup -u dremio -p dremio123 -d /tmp/backup

A confirmation message is displayed when the backup is complete.

Example: Backup with local-attach mode

The following example backs up Dremio to the /tmp/backup directory in local attach mode.

The --local-attach option is available as of Dremio 3.3

usage

./dremio-admin backup -d /tmp/backup -l

output message
The output message in server.out.

Backup created at /tmp/dremio_backup_2017-02-23_18.25, dremio tables 1400, uploaded files 150

Backing Up Dremio Step-by-Step

  1. Ensure that the Dremio daemon is running, on the master node:
  2. On the master node, run the following command:
     <dremio_home>/bin/dremio-admin backup \
      -u <DREMIO_ADMIN_USER> \
      -p <DREMIO_ADMIN_PASS> \
      -d <BACKUP_PATH>
    
  3. Backup is complete! Look for the confirmation message:
Backup created at /tmp/dremio_backup_2017-02-23_18.25, dremio tables 1400, uploaded files 150

Backing Up Dremio with AWS

For administrators with configurations using AWS, you have the option to automatically back up your Dremio project on a daily basis. With the option for this enabled, Dremio will regularly initiate snapshots of the EBS volume, which stores all of the project metadata.

However, in some scenarios, you may need to perform a manual backup of the project, such as when you’re preparing to upgrade Dremio and a arecent snapshot of the project is advised in case of a rollback.

Locating the EBS Volume

  1. From the AWS console, open the EC2 service and from the left-hand menu navigate to Elastic Block Store > Volumes.
  2. From the volumes listing, filter by the tag dremio_project_name : <your_project_name>, replacing the text after the colon with your project’s name. You should find one volume for your project.
  1. Select the volume and open the Tags tab. Copy of all the tags and their values onto a note, as this will be needed in the next section.

Creating a Snapshot of the Volume

  1. Select the checkbox to the left of the volume name.
  2. Click the Actions button above the filter bar.
  3. Click Create Snapshot.
  1. In the snapshot’s configuration window, add a useful description as well as the tags and values you noted in the last step of the previous section. You may also want to add a tag identifying the volume as a manual (rather than automatic) backup. While the tags may be added later, we recommend including them now for tracking purposes.

Note:

Including these tags is required, as these are what new Dremio deployments use to discover the correct snapshot.

  1. Click Create Snapshot.

Success is indicated by a message stating Create Snapshot Request Succeeded.

If you need to restore from this backup, see the section Restoring Dremio from an AWS Backup. If you want to create a clone from this backup, see the section Cloning a Dremio Stack using Backup/Restore.