This topic describes how to use log monitoring for a Kubernetes cluster in Azure (AKS).
In Kubernetes, Dremio processes writes logs to the console (stdout). These logs can be monitored using kubectl. But monitoring logs across lots of pods can be a pain. Also, when a pod, e.g. Dremio executor, crashes and restarts, only the logs from the last pod is available. So, if a pod is crashing regularly, logs will be lost and it will be difficult to analyze the reasons for crash. So, in container world, it is standard practice to put infrastructure in place to aggregate logs from containers into a central log store and analyze them.
Azure provides a nice and a tight integration with AKS clusters and Azure Log Analytics to monitor container logs. This document covers how to enable container monitoring, including logs from containers and manage them.
While creating a AKS cluster, enable container monitoring. You can use can existing Log Analytics workspace or create a new one. That’s it!
In an existing AKS cluster where monitoring was not enabled during creation, it can be enabled equally easily. Go to Logs on the AKS cluster and enable it!
All your container stdout and stderr will now be collected by the infrastructure for you to analyze.
Go to the Logs and you can view all the container logs. You can filter down to see the logs from the containers you are interested in and other filter options.