Integrating with Datadog

You can use the Agent to collect and send logs to Datadog for data monitoring and analysis.

To ingest logs to Datadog, complete these steps:

1. Install the Agent on each node of your Dremio cluster.

2. Change logs_enabled: false to logs_enabled: true in the datadog.yaml.

3. Create a dremio.d/ folder in the conf.d/ Agent configuration directory.

4. Create a conf.yaml file in java.d/ directory using the following example:

Example for conf.yaml

# This is an example inputs.conf for uploading Dremio logs to Datadog.
#
# To learn more about Dremio log files please see the documentation located at
# https://docs.dremio.com/current/sonar/monitoring/#logs

logs:

  # Audit log.
  - type: file
    path: <DREMIO_LOG_PATH>/audit.json
    service: dremio
    source: audit
    sourcecategory: audit

  # HTTP access log for the Dremio web server. This log will be generated by coordinator nodes only.
  - type: file
    path: <DREMIO_LOG_PATH>/access.log
    service: dremio
    source: access
    sourcecategory: system

  # Garbage collection log.
  - type: file
    path: <DREMIO_LOG_PATH>/server.gc
    service: dremio
    source: server.gc
    sourcecategory: system

  # Server log.
  - type: file
    path: <DREMIO_LOG_PATH>/json/server.json
    service: dremio
    source: server
    sourcecategory: system

  # Log for Dremio daemon standard out.
  - type: file
    path: <DREMIO_LOG_PATH>/server.out
    service: dremio
    source: server.out
    sourcecategory: system

  # Metadata refresh log.
  - type: file
    path: <DREMIO_LOG_PATH>/metadata_refresh.log
    service: dremio
    source: metadata_refresh
    sourcecategory: system

  # Tracker log.
  - type: file
    path: <DREMIO_LOG_PATH>/tracker.json
    service: dremio
    source: tracker
    sourcecategory: system

  # Query Log.
  - type: file
    path: <DREMIO_LOG_PATH>/queries.json
    service: dremio
    source: query
    sourcecategory: query

1. Restart the Agent.