Service Telemetry for Kubernetes Deployments Enterprise

As of Dremio 26.0, enterprise customers deploying Dremio Enterprise on Kubernetes automatically transmit telemetry data to Dremio's corporate endpoint. This telemetry provides valuable insights into system performance and health, and is also used in the calculations for usage-based billing measured in Dremio Consumption Units (DCUs).

Telemetry must remain enabled. Disabling telemetry breaks the data path that backs DCU billing and reduces Dremio's ability to provide stability and timely support.

Service telemetry is available only on Kubernetes deployments of Dremio Enterprise. Air-gapped clusters do not transmit telemetry. For these deployments, usage data is exported through a separate offline billing process.

Telemetry Data Collection

Dremio's telemetry data collection is strictly limited to operational and performance metrics. These metrics provide visibility into various components and services, ensuring optimal performance and reliability. Importantly, no customer content (e.g., business data) or user-entered information is transmitted. If you would like to develop a deeper understanding of the metrics transmitted and their contents, you can set up your own internal monitoring of your Dremio cluster by following the steps in Monitoring Dremio Nodes.

The following data is never transmitted:

• SQL query text or query results.
• Query profiles.
• The queries.json audit log.
• Kubernetes pod or container logs.
• Any user-entered data, including credentials, source configurations, and view definitions.

The collected telemetry data is categorized as follows:

Category	Description
Application Metrics	These metrics provide insights into the usage and performance of objects within a Dremio deployment, including: Number of queries, Reflections, sources, and views. Success and failure rates of queries. Success and failure rates of Reflection and source refresh operations.
Java Metrics	These metrics capture internal Java Virtual Machine (JVM) performance indicators from containers running the Dremio application, such as: Number of active threads. Memory allocation and usage. Garbage collection activity and pauses.
Service Metrics	These metrics measure the health of core components supporting Dremio's execution and coordination services, including: KVstore performance. Zookeeper availability and network health.
Kubernetes Metrics	These metrics provide insight into container and pod behavior for all containers in a Dremio deployment, including: CPU, memory, and storage requests. Container restarts and readiness. Horizontal Pod Autoscaler (HPA) current and desired replica count. StatefulSet desired and current pod count.

Billing Metrics

The following Kubernetes metrics back DCU billing calculations:

Metric	Description
k8s.container.cpu_request	CPU requested per container.
k8s.container.memory_request	Memory requested per container.
k8s.container.storage_request	Storage requested per container.
k8s.container.restarts	Container restart count.
k8s.container.ready	Container readiness state.
k8s.hpa.current_replicas	Current Horizontal Pod Autoscaler (HPA) replica count.
k8s.hpa.desired_replicas	Desired Horizontal Pod Autoscaler (HPA) replica count.
k8s.statefulset.desired_pods	Desired StatefulSet pod count.
k8s.statefulset.current_pods	Current StatefulSet pod count.

Operational Metrics by Component

The OTEL collector also scrapes Prometheus metrics from each Dremio component for operational monitoring and troubleshooting. These metrics are not used for billing.

Component	Key Metrics	What Is Monitored
dremio-master-coordinator	block_map_builder_seconds, credentials_service_*, exec_to_coord_screen_complete_time_seconds, execution_complete_to_query_complete_time_seconds, fabric_send_durations_ms, dremio_memory_*, dremio_metadata_io_executor_*	JVM heap and direct memory, query lifecycle, coordinator fabric latency, metadata I/O pool statistics.
dremio-coordinator	buffer_pool_direct/mapped_*, dremio_memory_*, fabric_send_durations_ms, dremio_metadata_io_executor_*	Buffer pool capacity and usage, JVM and Netty/Arrow memory, metadata I/O thread pools.
dremio-executor	c3_eviction_allocator_size, c3_request_allocator_size, c3_valid_buffer_map_count, exec_fragments_active, exec_started/ended_queries_total, exec_query_memory_dist_bytes, exec_query_peak_memory_non_spillable_operators_bytes, fragment_stats_*, fabric_send_durations_ms, dremio_memory_*	C3 cache allocator state, active fragments, per-query memory distribution, fragment setup and evaluation heap, executor fabric latency.
zookeeper	ack_latency, avg_latency, bytes_received_count, ephemerals_count, fsynctime, election_time, follower_sync_time, dead_watchers_*, diff_count, auth_failed_count, ensemble_auth_*	ZooKeeper cluster health: latency, election time, fsync, watcher state.
dremio-catalog-server, dremio-catalog-services-server, engine-operator, otelcollector	count:up1	Liveness and up-status.

Telemetry Transmission Requirements

Telemetry transmission to Dremio follows the Dremio Subscription Agreement.

Transport

An OpenTelemetry (OTEL) collector runs as a sidecar pod in the Dremio Kubernetes deployment. The collector transmits telemetry to Dremio using the OpenTelemetry Protocol (OTLP) over HTTPS.

Network Requirements for Telemetry Transmission

To ensure successful telemetry transmission, the following network configurations must be in place:

• Your network must allow traffic egress to Dremio's endpoint observability.dremio.com.
• Dremio's OpenTelemetry collectors use port 443 for secure data transmission via TLS.

Setting Up a Proxy

If traffic egresses to the endpoint and the port is restricted, a proxy can be configured to enable telemetry transmission:

1. Edit your .yaml configuration file to deploy Dremio to Kubernetes. For more information, refer to Configuring Your Values.

2. Add your proxy configuration values to the .yaml file using the following syntax:

   HTTPS Proxy (Recommended)

   telemetry:
     extraEnvs: []
       - name: HTTPS_PROXY
         value: https://proxy.example.com:443

   HTTP Proxy

   telemetry:
     extraEnvs: []
       - name: HTTP_PROXY
         value: http://proxy.example.com:3128

Tagging Production and Non-Production Clusters

You can tag each cluster as production or non-production so that telemetry records are labeled accordingly for downstream filtering and billing. Set the cluster.type field in your values-overrides.yaml file when deploying Dremio with the Helm chart.

The field accepts two values:

• prod: The default value. Use this for production clusters. If cluster.type is not set, the cluster is treated as a production cluster.
• non-prod: Use this for development, test, staging, or any other non-production cluster.

To tag a cluster as non-production, add the following to your values-overrides.yaml file:

Configuration to tag a cluster as non-production

cluster:
  type: non-prod

Telemetry continues to egress in both cases. The cluster.type value is included with each telemetry record.

Troubleshooting or Support

For troubleshooting or support, please contact your account representative or Dremio Support.

Related Topics

• Licensing - Learn more about Dremio's licensing and telemetry.
• Billing - Learn more about Dremio's billing and usage data.