Gandiva-based Execution

[info] Preview only

The Gandiva feature is available for PREVIEW ONLY and is disabled by default.

The Gandiva feature supports efficient evaluation of arbitrary SQL expressions on Arrow buffers using runtime code generation in LLVM. It uses LLVM tools to generate and compile code that makes optimal use of underlying CPU architecture. By combining LLVM with Apache Arrow libraries, Gandiva can perform low-level operations on Arrow in-memory buffers that are highly optimized for specific runtime environments, thus:

  • Improving resource utilization
  • Providing faster, lower-cost operations of analytical workloads

[info] Note

LLVM tools are a set of modular compiler tools that deal with code generation. They are used to compile and execute arbitrary expressions efficiently (instead of interpreting them). In the Dremio context, this is useful for generating code at runtime for two SQL operators that deal with arbitrary user expressions such as Project and Filter.

Enabling Gandiva

To request access and enable Gandiva, please contact Dremio at preview@dremio.com.

Supported Functions

[info]If you are using a combination of supported and non-supported functions, your code generation defaults to Java.

Function Type Supported Functions
Operators Arithmetic : +, -, *, /,
Relational : <, >, <=, >=, ==
Misc isnull
isnotnull
hash32
hash64
is_distinct_from
Is_not_distinct_from
not
Math cbrt
exp
log
Log10
power
Date/Time date_add
date_diff
date_sub
date_trunc_Century
date_trunc_Day
date_trunc_Decade
date_trunc_Hour
date_trunc_Millennium
date_trunc_Minute
date_trunc_Month
date_trunc_Quarter
date_trunc_Second
date_trunc_Week
date_trunc_Year
date_add
date_diff
date_sub
extractCentury
extractDay
extractDecade
extractDow
extractDoy
extractEpoch
extractHour
extractMillennium
extractMinute
extractMonth
extractQuarter
extractSecond
extractWeek
timestampaddDay
timestampaddHour
timestampaddMinute
timestampaddMonth
timestampaddQuarter
timestampaddSecond
timestampaddWeek
timestampaddYear
timestampdiffDay
timestampdiffHour
timestampdiffMinute
timestampdiffMonth
timestampdiffQuarter
timestampdiffSecond
timestampdiffWeek
timestampdiffYear
todate
String length
octet_length
like
Conditional Case statements support
Cast castBIGINT
castDATE
castFLOAT4
castFLOAT8
Boolean AND, OR

Limitations

Gandiva is not supported on Windows, Mac, and CentOS 6. If you enable Gandiva (set exec.preferred.codegenerator to gandiva) on one of these operating systems, the following occurs:

  • Windows and Mac
    The default, Java, is used for all operations and the following error is displayed:
    2018-09-27 22:23:59,071 [e0 - 2452451d-7b0b-a561-9a9e-270714fee800:frag:0:0] ERROR c.d.s.o.l.expr.GandivaPushdownSieve - Exception in running sieve org.apache.arrow.gandiva.exceptions.GandivaException: gandiva_jni.dll was not found inside JAR.

  • CentOS
    The query fails and the following error is displayed:
    SYSTEM ERROR: UnsatisfiedLinkError: /tmp/libgandiva_jni.sod52c7dfb-4edc-4e72-941b-7ba7e0eed6bc: /lib64/libc.so.6: version `GLIBC_2.15' not found ‘


results matching ""

    No results matching ""