Hive

Dremio and Hive

Dremio supports the following:

  • Hive table access using Hive's out-of-the-box SerDes interface, as well as custom SerDes or InputFormat/OutputFormat.
  • Hive-supported reading file format using Hive's own readers -- even if Dremio does not support them natively.

[info] Note

Dremio does not support Hive views. However, you can create and query virtual datasets instead.

Table Statistics

By default, Dremio utilizes its own estimates for Hive table statistics when planning queries.

However, if you want to use Hive's own statistics, do the following:

  1. Set the store.hive.use_stats_in_metastore parameter to true.
    Example: true: store.hive.use_stats_in_metastore

  2. Run the ANALYZE TABLE COMPUTE STATISTICS command for relevant Hive tables in Hive. This step is required so that all of the tables (that Dremio interacts with), have up-to-date statistics.

    ANALYZE TABLE <Table1> [PARTITION(col1,...)] COMPUTE STATISTICS;
    

Dremio Configuration

Configuration is primarily accomplished through either the General or Advanced Options.

The General configuration activities include setting or selecting the following:

  • Name -- Hive source name
  • Connection -- Hive connection and security
  • Authorization -- Authorization type for the client

Connection

The connection parameters include:

  • Hive Metastore Host -- IP address. Example: 123.123.123.123
  • Port -- Port number. Default: 9083
  • SASL -- Box to enable SALS. If you enable SASL, specify the Hive Kerberos Principal.

Authorization

When adding a new Hive source, you have the following client options for Hive authorization:

  • Storage Based with User Impersonation -- A storage-based authorization in the Metastore Server which is commonly used to add authorization to metastore server API calls. Dremio utilizes user impersonation to implement Storage Based authorization.
  • SQL Based -- A SQL standards-based authorization option that allows Hive to be fully SQL compliant in its authorization model without causing backward compatibility issues.
  • Ranger Based -- An Apache Ranger plug-in that provides a security framework for authorization.
    • Ranger Service Name - This field corresponds to the security profile in Ranger. Example: hivedev
    • Ranger Host URL - This field is the path to the actual Ranger server. Example: http://yourhostname.com:6080

Advance Options

The Advanced Options configuration provides:

  • Connection Properties -- Name and value of each Hive connection property.

results matching ""

    No results matching ""