Logs

This topic describes default log files locations as well as the types of available logs.

Default Log File Locations

By default, Dremio uses the following locations to write logs:

  • Tarball - <DREMIO_HOME>/log
  • RPM - /var/log/dremio

Audit Logs

For organizations subject to compliance and regulation where auditing is regularly required, Dremio offers full audit logging, wherein all user activities performed within Dremio are tracked and traceable via the audit.json file. Each a user performs an action within Dremio, such as editing an object or running a query, the audit log captures the user’s ID and username, object(s) affected, action performed and event type, SQL statements used, and more.

By default, audit logging is enabled and only available to users with administrative rights at the System level.

Audit Log Location

  • For non-AWSE organizations, the log file location may be configured the same way as all other current log files, via the dremio.log.path property on the dremio-env file, may have their location, size, and rotation specified.
  • For organizations using AWSE, the log file is located within the same folder as queries.json.

EKS/AKS, YARN, AWSE

Events & Actions Tracked

Dremio supports audit logging for the following objects (event types) and actions:

Event Actions
ACL (privileges)
  • UPDATE
AUTHENTICATION
  • LOGIN
FOLDER
  • CREATE
  • DELETE
  • UPDATE
PHYSICAL_DATASET
  • CREATE
  • DELETE
  • UPDATE
REFLECTION
  • CREATE
  • DELETE
  • UPDATE
SOURCE
  • CREATE
  • DELETE
  • UPDATE
SPACE
  • CREATE
  • DELETE
  • UPDATE
VIRTUAL_DATASET
  • CREATE
  • DELETE
  • UPDATE
  • RENAME

Audit Log Format

Audit logs include the following information:

  • timestamp - The time when the event was recorded.
  • userId - The ID value associated with the user’s account.
  • userName - The username associated with the user account (which is typically used to log in).
  • status - The status of the action, typically used to indicate whether the event was approved or allowed.
  • eventType - The object or scope of the interaction that occurred.
  • action - The actual activity performed within the specified scope. This varies based on the eventType, but most often would be CREATE, DELETE, and UPDATE.
  • details - The data altered or created. This varies based on the eventType.

The following are examples of audit records Dremio would capture, and the information an audit entry would include depending on the event type.

Example: Authentication

The user dremio logged in on the Dremio application, so the following information would be added to the audit log:

{
  "timestamp": "2021-11-23 16:30:53,400",
  "userContext": {
    "userId": "1",
    "userName": "$dremio$"
  },
  "status": "OK",
  "eventType": "AUTHENTICATION",
  "action": "LOGIN",
  "details": {
    "userName": "dremio",
    "userId": "",
    "source": "FLIGHT"
  }
}

Example: Reflection

The user dremio created a reflection called Raw Reflection (1) via Dremio’s UI, so the following information would be added to the audit log:

{
  "timestamp": "2021-11-22 10:06:38,432",
  "userContext": {
    "userId": "6ab04602-410b-4031-87ae-2d3d5f7dc",
    "userName": "dremio"
  },
  "status": "OK",
  "eventType": "REFLECTION",
  "action": "CREATE",
  "details": {
    "reflectionId": "a5251b05-4873-4a9d-a008-303eeeeed",
    "name": "Raw Reflection (1)",
    "dataset": "7e3d4a8a-b92d-41ab-96dc-6a76a6248",
    "type": "RAW",
    "sortColumns": [
      {
        "name": "fare_amount"
      }
    ],
    "partitionColumns": [
      {
        "name": "passenger_count"
      }
    ],
    "distributionColumns": [],
    "dimensions": [],
    "measures": [],
    "displayColumns": [
      {
        "name": "passenger_count"
      },
      {
        "name": "pickup_datetime"
      },
      {
        "name": "trip_distance_mi"
      },
      {
        "name": "fare_amount"
      },
      {
        "name": "tip_amount"
      },
      {
        "name": "total_amount"
      }
    ],
    "partitiondistributionstrategy": "CONSOLIDATED",
    "arrowCachingEnabled": false,
    "targetDataset": ""
  }
}

Example: Virtual Dataset

The user dremio performed an SQL command to CREATE a virtual dataset (VDS) from Dremio’s SQL Editor, so the following information would be added to the audit log:

{
  "timestamp": "2021-11-17 14:31:43,594",
  "userContext": {
    "userId": "4a3ea2fa-a3f6-4adb-8852-041a28cac",
    "userName": "dremio"
  },
  "status": "OK",
  "eventType": "VIRTUAL_DATASET",
  "action": "CREATE",
  "details": {
    "id": "da08848d-d80d-4414-aaf4-40ce866ea",
    "name": "a_employees",
    "tag": "gwN/p7E5E1Q=",
    "path": "myVDSes.a_employees",
    "sql": "SELECT \"firstname\", \"zipcode\"\nFROM \"local-msql\".\"dremio_db1\".\"Employees\"\nWHERE \"zipcode\" LIKE 'a%'",
    "sqlContext": "myVDS",
    "fields": [
      {
        "name": "firstname",
        "dataType": "TEXT"
      },
      {
        "name": "zipcode",
        "dataType": "TEXT"
      }
    ],
    "oldName": "",
    "oldPath": ""
  }
}

System Logs

The following logs are enabled by default:

  • access.log - HTTP access log for the Dremio web server. This log will be generated by coordinator nodes only.
  • server.gc - Garbage collection log.
  • server.log - Server log.
  • server.out - Log for Dremio daemon standard out.

Query Logs

Query logs are located in the queries.json file. This file contains a log of completed queries; it does not include queries currently in planning or execution.

Query logging is enabled by default.

Query logs can be queried by Dremio itself or another tool for monitoring and analytics.

Format

Query logs include the following information:

  • queryId - Unique ID of the executed query.
  • queryText- SQL query text.
  • start - Start time of the query.
  • finish - End time of the query.
  • outcome - Whether the query was completed or failed.
  • username - User that executed the query.
  • commandDescription - Type of the command. This maybe a reqular SQL query execution job or another SQL command.

Additional information may be found depending on your Dremio configuration.