Dremio Cloud
Dremio Software
 
On this page

    Example: Amount of Consumed Memory

    This example determines the amount of consumed memory.

    To find out how much memory each executor is consuming during the query process:

    1. Upload your profile_attempt_0.json. Alternatively, you can use the sample profile_attempt_0.json file.
    2. Create a file called operator_type.json and add the sample field values in Example: Operator Type Mapping to the file.
    3. Upload the operator type JSON file.
    4. Using a SQL query, obtain the amount of memory consumed. Group by and order by nested_2.endpoint.address.

    The following is a sample SQL query snippet for obtaining the amount of memory each executor is consuming:

    SELECT nested_2.endpoint.address AS address, sum(nested_2.minorFragmentProfile.maxMemoryUsed)/1024/1024 AS maxMemoryUsed_MB
-- nested_2.minorFragmentProfile.memoryUsed AS memoryUsed, nested_2.minorFragmentProfile.maxMemoryUsed AS maxMemoryUsed, nested_2.endpoint.address AS address, nested_2.endpoint.maxDirectMemory AS maxDirectMemory, nested_2.endpoint.provisionId AS provisionId, majorFragmentId
FROM (
 SELECT fragmentProfile, minorFragmentProfile, nested_1.minorFragmentProfile.endpoint AS endpoint, majorFragmentId
 FROM (
   SELECT fragmentProfile, flatten(nested_0.fragmentProfile.minorFragmentProfile) AS minorFragmentProfile, nested_0.fragmentProfile.majorFragmentId AS majorFragmentId
   FROM (
     SELECT flatten(fragmentProfile) AS fragmentProfile
     FROM "@dremio"."profile_attempt_0" AS "profile_attempt_0"
   ) nested_0
 ) nested_1
) nested_2
group by nested_2.endpoint.address
order by nested_2.endpoint.address

    Alternative SQL code:

    • Substitute sum(nested_2.minorFragmentProfile.memoryUsed AS memoryUsed) for sum(nested_2.minorFragmentProfile.maxMemoryUsed)/1024/1024 AS maxMemoryUsed_MB.
    • Substitute nested_2.endpoint.provisionId instead of nested_2.endpoint.address to obtain the yarn container name instead of the executor name.

    For More Information