Administering Dremio on GKE

This topic discusses administration activities such as log monitoring, pod scaling, configuration changes, basic administrative tasks (backup, restore, clean, and so on), and Dremio upgrading.

Monitoring Logs

Logs are written to the container's console (stdout). These logs can be monitored using the kubectl command. All the logs (server.log, server.out, server.gc and access.log) are written to the console simultaneously. You can view the logs using the kubectl logs or kubectl logs -f command.

View logs

kubectl logs <container-name>
kubectl logs -f <container-name>

If you want to persist logs in the PVC, follow the instructions here

Scaling Up or Down

Scaling up or down refers to increasing or decreasing the number of Dremio's pods (master-coordinator, executors, or secondary-coordinators). All scaling values remain in affect until you run another helm upgrade command.

note

Performance: If you decrease the number of pods (either temporarily or permanently), existing reflections might need to be recreated rather than refreshed. Recreating them could take longer than refreshing them.

1. Obtain the name of the helm chart release by using the helm list command. For example:

   Helm list command example

   helm list
   NAME                REVISION    UPDATED                     STATUS      CHART           NAMESPACE
   dremio  1           Wed Jul 18 09:36:14 2018    DEPLOYED    dremio-24.0.0   default

2. Run the helm upgrade --wait <chart release name> . --set <dremio pod=value> command.

   For example, the number of Dremio executor pods could be changed with the following command, where dremio is the chart release name and the new number of pods is 5:

   Helm upgrade command example

   helm upgrade --wait dremio . --set executor.count=5

Resetting to Defaults

After you scale up or down the number of Dremio pods, if helm upgrade is run again (either for scaling, changing your configuration, or upgrading) the configuration resets to the defaults specified in the values.yaml file.

All scaling values remain in affect until you run the helm upgrade command. When a subsequent helm upgrade command is run, values are reset to the default in the value.yaml file. For example, if you scale up the secondary-coordinators to 3 and then scale up the executors to 5, the secondary-coordinator is reset to 0 (default) after the executor is scaled up to 5.

note

If you scale the cluster down to zero pods, you are effectively shutting down the Dremio cluster.

To change your default values, update the values.yaml file. See Changing your Configuration for more information.

Changing your Configuration

If you need to update your configuration, you can do so after the installation by editing the configuration files and then upgrading using the helm upgrade <chart release name> . command. The upgrade process pushes out your changes to all of the pods in your Kubernetes cluster and restarts the pods.

For example, if you want to permanently increase the number of Dremio executor pods to five (5):

1. Edit the values.yaml file and change the number of executor pods via executor.count. In this example, executor.count is set to 5 and the other executor defaults remain unchanged.

   <CodeHeader>Executor properties in values.yaml file</CodeHeader>
   
   ```properties
   executor:
     memory: 12280
     cpu: 15
     count: 5
     volumeSize: 120Gi
   ```

2. Run the helm upgrade --wait <chart release name> . command. In this example, dremio is the chart release name:

   <CodeHeader>Helm upgrade command example</CodeHeader>
   
   ```properties
   helm upgrade --wait dremio .
   ```

note

Tip: If it takes longer than a couple of minutes to complete, check the status of the pods via kubectl get pods. If the pods are pending scheduling due to limited memory or cpu, either adjust the values in the values.yaml file or add more resources to your Kubernetes cluster.

Dremio Admin Commands

These are the Dremio administration commands that can be run on the Dremio+Kubernetes cluster. All commands except for the Dremio backup command require that Dremio be shutdown or offline.

Command	Offline/Online	Notes
backup	online	/opt/dremio/bin/dremio-admin backup See Backup Dremio for more information.
clean	offline	/opt/dremio/bin/dremio-admin clean See Metadata Cleanup for more information.
restore	offline	/opt/dremio/bin/dremio-admin restore See Restore Dremio for more information.
set-password	offline	/opt/dremio/bin/dremio-admin set-password See Reset Password for more information.

Backup

The backup command is run when Dremio is online. Run it from a bash shell on the master-coordinator pod.

To run the backup command:

1. Connect to the master-coordinator pod by using the exec command.

   Connect to master coordinator pod

   kubectl exec -it dremio-master-0 -- bash

2. Run the command from the bash shell. See Backup Dremio for more information.

   Run bash shell command

    /opt/dremio/bin/dremio-admin backup \
     -u <DREMIO_ADMIN_USER> \
     -p <DREMIO_ADMIN_PASS> \
     -d <BACKUP_PATH>

3. Store the backup files in some persistent volume or copy the files out of the local pod.

Clean, Restore, and Set-Password

The following Dremio commands are offline commands, that is, Dremio must not be running.

• clean
• restore
• set-password

note

To temporarily shut down Dremio, scale down the master-coordinator to zero (0).

These offline commands are run by creating a Dremio Admin pod with the Dremio image and mounting the master-coordinator pod's persistent volume.

To run offline dremio-admin commands:

1. Create a Dremio Admin pod by running this command:

   Create Dremio Admin pod

   helm install --wait dremio --set DremioAdmin=true

2. Run the dremio-admin command from the bash shell on the Dremio Admin pod. See Advanced Administration for more information on each command.
   For example, the two following commands connect you to the pod and allow you to perform an offline command:

   <CodeHeader>Connect to pod and perform offline command</CodeHeader>
   
   ```
   kubectl exec -it dremio-admin -- bash
   bin/dremio-admin <offline command>
   ```

3. Delete the pod.

   Delete pod

   kubectl delete pod dremio-admin

After deleting the pod, you can restart your Dremio cluster.

Upgrading Dremio

You upgrade Dremio by updating the image value in the values.yaml file to the new Dremio version and running the helm upgrade command.

During the upgrade process, existing pods are terminated and new pods are created with the new image. Once all the newly created pods are restarted and running, your Dremio cluster is upgraded.

To upgrade Dremio:

1. Ensure that your Dremio+Kubernetes cluster is backed up. See Backup for more information.

2. Ensure that there are no queries are running on the cluster.

3. Update the Dremio image tag in your values.yaml file.
   For example, to change the Dremio CE image:

   Change Dremio CE image

   image: dremio/dremio-oss
   imageTag: 23.0.0
   ...

   Note:
   

   If you are changing the Dremio Enterprise Edition image, you do not need to change the imagePullSecrets property.

4. Get the name of the chart release by using helm list command. In the example below, the name is dremio.

   Get chart release name

   helm list
   NAME                REVISION    UPDATED                     STATUS      CHART           NAMESPACE
   dremio  1           Wed Jul 18 09:36:14 2018    DEPLOYED    dremio-24.0.0   default

5. Run the helm upgrade --wait <chart release name> . command to upgrade the deployment.
   For example, if the name of the chart release is dremio, then the command is helm upgrade dremio .

note

Tip: The pods are restarted automatically after upgrading.If it takes longer than a couple of minutes to complete, check the status of the pods via kubectl get pods. If the pods are pending scheduling due to limited memory or cpu, either adjust the values in the values.yaml file (see Changing your Configuration) or add more resources to your Kubernetes cluster.