Example: Amount of Consumed Memory

This example determines the amount of consumed memory.

To find out how much memory each executor is consuming during the query process:

  1. Upload your profile_attempt_0.json. Alternatively, you can use the sample profile_attempt_0.json file.
  2. Create a file called operator_type.json and add the sample field values in Example: Operator Type Mapping to the file.
  3. Upload the operator type JSON file.
  4. Using a SQL query, obtain the amount of memory consumed. Group by and order by nested_2.endpoint.address.

The following is a sample SQL query snippet for obtaining the amount of memory each executor is consuming:

SELECT nested_2.endpoint.address AS address, sum(nested_2.minorFragmentProfile.maxMemoryUsed)/1024/1024 AS maxMemoryUsed_MB
-- nested_2.minorFragmentProfile.memoryUsed AS memoryUsed, nested_2.minorFragmentProfile.maxMemoryUsed AS maxMemoryUsed, nested_2.endpoint.address AS address, nested_2.endpoint.maxDirectMemory AS maxDirectMemory, nested_2.endpoint.provisionId AS provisionId, majorFragmentId
 SELECT fragmentProfile, minorFragmentProfile, nested_1.minorFragmentProfile.endpoint AS endpoint, majorFragmentId
   SELECT fragmentProfile, flatten(nested_0.fragmentProfile.minorFragmentProfile) AS minorFragmentProfile, nested_0.fragmentProfile.majorFragmentId AS majorFragmentId
   FROM (
     SELECT flatten(fragmentProfile) AS fragmentProfile
     FROM "@dremio"."profile_attempt_0" AS "profile_attempt_0"
   ) nested_0
 ) nested_1
) nested_2
group by nested_2.endpoint.address
order by nested_2.endpoint.address

The following screenshot shows the output of this query. This screenshot uses the provided sample profile data.

Alternative SQL code:

  • Substitute sum(nested_2.minorFragmentProfile.memoryUsed AS memoryUsed) for sum(nested_2.minorFragmentProfile.maxMemoryUsed)/1024/1024 AS maxMemoryUsed_MB.
  • Substitute nested_2.endpoint.provisionId instead of nested_2.endpoint.address to obtain the yarn container name instead of the executor name.

For More Information