Administering Dremio on EKS
This topic discusses administration activities such as scaling pods, changing configurations, performing basic administrative tasks (such as backing up, restoring, and cleaning), and upgrading Dremio.
Logs are written to the container's console (stdout).
These logs can be monitored using the
All the logs (server.log, server.out, server.gc and access.log) are written to the console simultaneously.
You can view the logs using the
kubectl logs or
kubectl logs -f command.
kubectl logs <container-name>
kubectl logs -f <container-name>
Scaling Up or Down
Scaling up or down refers to increasing or decreasing the number of Dremio's pods (master-coordinator, executors, or secondary-coordinators). All scaling values remain in affect until you run another
helm upgrade command.
Performance: If you scale down the number of pods (either temporarily or permanently), already created reflections may need to be re-created during the next refresh. This could result in a delay for up-to-date data reflections.
Obtain the name of the helm chart release with theHelm list command example
helm listcommand. For example:
NAME REVISION UPDATED STATUS CHART NAMESPACE
plundering-alpaca 1 Wed Jul 18 09:36:14 2018 DEPLOYED dremio-0.0.5 default
helm upgrade --wait <chart release name> . --set <dremio pod=value>command.
For example, Dremio executor pods could be scaled up or down with the following commands where plundering-alpaca is the chart release name and the pod count is 5:Helm upgrade command example
helm upgrade --wait plundering-alpaca . --set executor.count=5
Resetting to Defaults
After you scale up or down the number of Dremio pods, if
helm upgrade is run again
(either for scaling, changing your configuration, or upgrading)
the configuration resets to the defaults specified in the values.yaml file.
All scaling values remain in affect until you run the
helm upgrade command.
When a subsequent
helm upgrade command is run, values are reset to the default in the value.yaml file.
For example, if you scale up the secondary-coordinators to 3 and then scale up the executors to 5,
the secondary-coordinator is reset to 0 (default) after the executor is scaled up to 5.
Scale to zero: If you scale all of the Dremio pods down to zero (0), you are effectively shutting down the Dremio cluster.
To permanently change your default values, update the values.yaml file. See Changing your Configuration for more information.
Changing your Configuration
If you need to update your configuration, you can do so after the installation by
editing the configuration files and then upgrading using the
helm upgrade <chart release name> . command.
The upgrade process pushes out your changes to all of the pods in your Kubernetes cluster and restarts the pods.
For example, if you want to permanently increase the number of Dremio executor pods to five (5):
Edit the values.yaml file and change the number of executor pods via
executor.count. In this example,
executor.countis 5 and the other executor defaults remain unchanged.
<CodeHeader>Example executor property values</CodeHeader>
helm upgrade --wait <chart release name> .command. In this example, plundering-alpaca is the chart release name:
<CodeHeader>Helm upgrade command example</CodeHeader>
helm upgrade --wait plundering-alpaca .
If the command takes longer than a few minutes to finish,
check the status of the pods via
kubectl get pods. If the pods are pending scheduling due to limited memory or cpu,
either re-adjust the values in the values.yaml file
or add more resources to your Kubernetes cluster.
Dremio Admin Commands
You can run these Dremio administration commands on the Dremio+Kubernetes cluster.
All commands except for the Dremio
backup command required that Dremio be shutdown/offline.
See Backup Dremio for more information.
See Metadata Cleanup for more information.
See Restore Dremio for more information.
See Reset Password for more information.
backup command is run when Dremio is online.
It is run on the master-coordinator pod from a bash shell.
To run the backup command:
Connect to the master-coordinator pod using the exec command.Exec command
kubectl exec -it dremio-master-0 -- bash
Run the command from the bash shell. See Backup Dremio for more information.Shell command example
/opt/dremio/bin/dremio-admin backup \
-u <DREMIO_ADMIN_USER> \
-p <DREMIO_ADMIN_PASS> \
Store the backup files in some persistent volume or copy the files out of the local pod.
Clean, Restore, and Set-Password
The following Dremio commands are offline commands, that is, Dremio must not be running.
To temporarily shut down Dremio, scale down the master-coordinator to zero (0).
These offline commands are run by creating a Dremio Admin pod with the Dremio image and mounting the master-coordinator pod's persistent volume.
To run offline dremio-admin commands:
Create a Dremio Admin pod to run the dremio-admin commands. Use the following command to create a Dremio Admin pod:
<CodeHeader>Create Dremio Admin pod</CodeHeader>
helm install --wait dremio --set DremioAdmin=true
Run the dremio-admin commands from the bash shell on the Dremio Admin pod. See Advanced Administration for more information on each command.
The following commands connect you to the pod and allow you to perform the offline command:
<CodeHeader>Connect to pod and perform offline command</CodeHeader>
kubectl exec -it dremio-admin -- bash
bin/dremio-admin <offline command>
Delete the pod.Delete pod
kubectl delete pod dremio-admin
After performing the offline command, you can restart your Dremio cluster.