Kerberos Setup and Troubleshooting

This section provides a Knowledge Base article on Kerberos setup and troubleshooting.

Dremio Configuration for a Kerbero-ized Cluster

When customer runs a kerbero-ized cluster (like CDH, HDP), the following configuration must be done on the Dremio-side. This configuration is required to add a Kerbero-ized HDFS or Hive data source.

  1. Create a Kerberos principal for the Dremio service user. This is the user that runs Dremio on the Hadoop cluster.
  2. Generate a Keytab file for the Dremio Kerberos principal.
  3. Depending on the deployment, either symlink the core-site.xml file or add a local copy of the core-site.xml file under Dremio's configuration directory (same directory as dremio.conf file).
  4. Add the following properties to the core-site.xml file.
    <property>
     <name>hadoop.security.authentication</name>
     <value>kerberos</value>
    </property>
    <property>
    <name>hadoop.proxyuser.dremio.hosts</name>
    <value>*</value>
    </property>
    <property>
    <name>hadoop.proxyuser.dremio.groups</name>
    <value>*</value>
    </property>
    

[info] IMPORTANT

Even if impersonation is turned off and the service user Dremio is running and present on all the Hadoop nodes, Kerberos by default turns on impersonation so proxy settings have to be defined.

Issue #1

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

Resolution: The Keytab either has wrong permissions or ownership. Even the keytab has the right permission, check all directories above the file recursively and make sure the owner (usually Dremio service user) can read the file.

2018-04-04 22:47:33,574 [qtp527426932-247] ERROR o.a.thrift.transport.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[na:1.8.0_161]
    at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) [libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [libthrift-0.9.2.jar:0.9.2]
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) [dremio-hive-exec-shaded-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) [dremio-hive-exec-shaded-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_161]
    at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_161]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) [hadoop-common-2.8.0.jar:na]
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) [dremio-hive-exec-shaded-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420) [hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236) [hive-metastore-1.2.1.jar:1.2.1]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) [hive-metastore-1.2.1.jar:1.2.1]
    at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:140) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.HiveClient$1.run(HiveClient.java:137) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.HiveClient$9.run(HiveClient.java:306) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_161]
    at javax.security.auth.Subject.doAs(Subject.java:422) [na:1.8.0_161]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) [hadoop-common-2.8.0.jar:na]
    at com.dremio.exec.store.hive.HiveClient.doAsCommand(HiveClient.java:303) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.HiveClient.connect(HiveClient.java:136) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.HiveClient.createClient(HiveClient.java:124) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.HiveStoragePlugin.<init>(HiveStoragePlugin.java:125) [dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_161]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_161]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_161]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_161]
    at com.dremio.exec.store.StoragePluginRegistryImpl.create(StoragePluginRegistryImpl.java:459) [dremio-sabot-kernel-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.StoragePluginRegistryImpl.createOrUpdate(StoragePluginRegistryImpl.java:271) [dremio-sabot-kernel-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:125) [dremio-dac-backend-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:117) [dremio-dac-backend-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.dac.resource.EnterprisePutSourceResource.putSource(EnterprisePutSourceResource.java:98) [dremio-extra-dac-enterprise-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.8.0_161]
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[na:1.8.0_161]
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.8.0_161]
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[na:1.8.0_161]
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.8.0_161]
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.8.0_161]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[na:1.8.0_161]

Issue #2

java.io.IOException: Can't get Master Kerberos principal for use as renewer

Resolution: Symlink/Copy the following xml files from the Hadoop nodes to the Dremio configuration directory:
core-site.xml, yarn-site.xml, hdfs-site.xml, mapred-site.xml.

Caused by: java.io.IOException: Can't get Master Kerberos principal for use as renewer
    at     (TokenCache.java:132) ~[hadoop-mapreduce-client-core-2.8.0.jar:na]
    at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100) ~[hadoop-mapreduce-client-core-2.8.0.jar:na]
    at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) ~[hadoop-mapreduce-client-core-2.8.0.jar:na]
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:213) ~[hadoop-mapreduce-client-core-2.8.0.jar:na]
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) ~[hadoop-mapreduce-client-core-2.8.0.jar:na]
    at com.dremio.exec.store.hive.DatasetBuilder$HiveSplitsGenerator.runInner(DatasetBuilder.java:377) ~[dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.DatasetBuilder.buildSplits(DatasetBuilder.java:444) ~[dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.DatasetBuilder.buildIfNecessary(DatasetBuilder.java:285) ~[dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.hive.DatasetBuilder.getDataset(DatasetBuilder.java:204) ~[dremio-hive-plugin-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]
    at com.dremio.exec.store.SimpleSchema.getTableFromDataset(SimpleSchema.java:319) [dremio-sabot-kernel-1.4.6-201801281620070890-2771674.jar:1.4.6-201801281620070890-2771674]

Issue #3

(java.io.IOException) org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection():551

Resolution: From the stack, it is evident that the in addition to the Kerberized HDFS cluster, it is also encrypted. To access an encrypted HDFS cluster proxy settings need to be entered into kms-site.xml

<property>
    <name>hadoop.kms.proxyuser.dremio.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.kms.proxyuser.dremio.hosts</name>
    <value>*</value>
</property>
SYSTEM ERROR: AuthenticationException: Authentication failed, status: 403, message: Forbidden

Fragment 0:0

[Error Id: 37f62446-2324-4b06-8bab-58773d119962 on z9awsstiku1w115.celgene.com:-1]

  (java.lang.RuntimeException) Failed to read parquet footer for file hdfs://nameservice1/user/hive/warehouse/.../fa461609e54c759e-28ff333100000000_1976891529_data.0.parq
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.getFooter():60
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1.run():125
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1.run():117
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():360
    org.apache.hadoop.security.UserGroupInformation.doAs():1787
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1.apply():117
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1.apply():113
    com.google.common.collect.Iterators$8.transform():799
    com.google.common.collect.TransformedIterator.next():48
    com.dremio.sabot.op.scan.ScanOperator.<init>():135
    com.dremio.sabot.op.scan.ScanOperator.<init>():114
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.createNativeParquet():144
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():257
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():65
    com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():94
    com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():89
    com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():86
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    com.dremio.sabot.driver.UserDelegatingOperatorCreator.getProducerOperator():86
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():208
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():115
    com.dremio.exec.physical.base.AbstractSubScan.accept():43
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
    com.dremio.exec.physical.config.Limit.accept():55
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
    com.dremio.exec.physical.config.Limit.accept():55
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter():69
    com.dremio.exec.physical.base.AbstractWriter.accept():37
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter():59
    com.dremio.exec.physical.config.WriterCommitterPOP.accept():75
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():234
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():115
    com.dremio.exec.physical.config.Screen.accept():43
    com.dremio.sabot.driver.PipelineCreator.get():107
    com.dremio.sabot.driver.PipelineCreator.get():101
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():320
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():227
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
    com.dremio.sabot.task.AsyncTaskWrapper.run():92
    com.dremio.sabot.task.slicing.SlicingThread.run():71
  Caused By (java.io.IOException) org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden
    org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection():551
    org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey():833
    org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey():388
    org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey():975
    org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream():1043
    org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall():326
    org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall():321
    org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
    org.apache.hadoop.hdfs.DistributedFileSystem.open():321
    org.apache.hadoop.fs.FileSystem.open():786
    com.dremio.exec.store.dfs.FileSystemWrapper.open():174
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.readFooter():99
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.readFooter():84
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.getFooter():58
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1.run():125
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1.run():117
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():360
    org.apache.hadoop.security.UserGroupInformation.doAs():1787
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1.apply():117
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1.apply():113
    com.google.common.collect.Iterators$8.transform():799
    com.google.common.collect.TransformedIterator.next():48
    com.dremio.sabot.op.scan.ScanOperator.<init>():135
    com.dremio.sabot.op.scan.ScanOperator.<init>():114
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.createNativeParquet():144
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():257
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():65
    com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():94
    com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():89
    com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():86
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    com.dremio.sabot.driver.UserDelegatingOperatorCreator.getProducerOperator():86
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():208
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():115
    com.dremio.exec.physical.base.AbstractSubScan.accept():43
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
    com.dremio.exec.physical.config.Limit.accept():55
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
    com.dremio.exec.physical.config.Limit.accept():55
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter():69
    com.dremio.exec.physical.base.AbstractWriter.accept():37
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter():59
    com.dremio.exec.physical.config.WriterCommitterPOP.accept():75
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():234
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():115
    com.dremio.exec.physical.config.Screen.accept():43
    com.dremio.sabot.driver.PipelineCreator.get():107
    com.dremio.sabot.driver.PipelineCreator.get():101
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():320
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():227
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
    com.dremio.sabot.task.AsyncTaskWrapper.run():92
    com.dremio.sabot.task.slicing.SlicingThread.run():71
  Caused By (org.apache.hadoop.security.authentication.client.AuthenticationException) Authentication failed, status: 403, message: Forbidden
    org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken():274
    org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate():77
    org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate():132
    org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate():214
    org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate():132
    org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection():215
    org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection():322
    org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run():542
    org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run():537
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection():536
    org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey():833
    org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey():388
    org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey():975
    org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream():1043
    org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall():326
    org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall():321
    org.apache.hadoop.fs.FileSystemLinkResolver.resolve():81
    org.apache.hadoop.hdfs.DistributedFileSystem.open():321
    org.apache.hadoop.fs.FileSystem.open():786
    com.dremio.exec.store.dfs.FileSystemWrapper.open():174
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.readFooter():99
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.readFooter():84
    com.dremio.exec.store.parquet.SingletonParquetFooterCache.getFooter():58
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1.run():125
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1.run():117
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():360
    org.apache.hadoop.security.UserGroupInformation.doAs():1787
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1.apply():117
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1.apply():113
    com.google.common.collect.Iterators$8.transform():799
    com.google.common.collect.TransformedIterator.next():48
    com.dremio.sabot.op.scan.ScanOperator.<init>():135
    com.dremio.sabot.op.scan.ScanOperator.<init>():114
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.createNativeParquet():144
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():257
    com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():65
    com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():94
    com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():89
    com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():86
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1807
    com.dremio.sabot.driver.UserDelegatingOperatorCreator.getProducerOperator():86
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():208
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():115
    com.dremio.exec.physical.base.AbstractSubScan.accept():43
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
    com.dremio.exec.physical.config.Limit.accept():55
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
    com.dremio.exec.physical.config.Limit.accept():55
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter():69
    com.dremio.exec.physical.base.AbstractWriter.accept():37
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter():59
    com.dremio.exec.physical.config.WriterCommitterPOP.accept():75
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
    com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
    com.dremio.exec.physical.config.Project.accept():53
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():234
    com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():115
    com.dremio.exec.physical.config.Screen.accept():43
    com.dremio.sabot.driver.PipelineCreator.get():107
    com.dremio.sabot.driver.PipelineCreator.get():101
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():320
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():227
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
    com.dremio.sabot.task.AsyncTaskWrapper.run():92
    com.dremio.sabot.task.slicing.SlicingThread.run():71


Fragment 0:0

com.dremio.exec.store.parquet.SingletonParquetFooterCache(SingletonParquetFooterCache.java:60)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1(HiveScanBatchCreator.java:125)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1$1(HiveScanBatchCreator.java:117)
...(:0)
org.apache.hadoop.security.UserGroupInformation(UserGroupInformation.java:1787)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1(HiveScanBatchCreator.java:117)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator$1(HiveScanBatchCreator.java:113)
com.google.common.collect.Iterators$8(Iterators.java:799)
com.google.common.collect.TransformedIterator(TransformedIterator.java:48)
com.dremio.sabot.op.scan.ScanOperator(ScanOperator.java:135)
com.dremio.sabot.op.scan.ScanOperator(ScanOperator.java:114)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator(HiveScanBatchCreator.java:144)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator(HiveScanBatchCreator.java:257)
com.dremio.exec.store.hive.exec.HiveScanBatchCreator(HiveScanBatchCreator.java:65)
com.dremio.sabot.driver.OperatorCreatorRegistry(OperatorCreatorRegistry.java:94)
com.dremio.sabot.driver.UserDelegatingOperatorCreator$4(UserDelegatingOperatorCreator.java:89)
com.dremio.sabot.driver.UserDelegatingOperatorCreator$4(UserDelegatingOperatorCreator.java:86)
...(:0)
org.apache.hadoop.security.UserGroupInformation(UserGroupInformation.java:1807)
com.dremio.sabot.driver.UserDelegatingOperatorCreator(UserDelegatingOperatorCreator.java:86)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:208)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractSubScan(AbstractSubScan.java:43)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:109)
com.dremio.exec.physical.config.Limit(Limit.java:55)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:109)
com.dremio.exec.physical.config.Limit(Limit.java:55)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:84)
com.dremio.exec.physical.config.Project(Project.java:53)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:84)
com.dremio.exec.physical.config.Project(Project.java:53)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:69)
com.dremio.exec.physical.base.AbstractWriter(AbstractWriter.java:37)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:59)
com.dremio.exec.physical.config.WriterCommitterPOP(WriterCommitterPOP.java:75)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:247)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.base.AbstractPhysicalVisitor(AbstractPhysicalVisitor.java:84)
com.dremio.exec.physical.config.Project(Project.java:53)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:234)
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor(PipelineCreator.java:115)
com.dremio.exec.physical.config.Screen(Screen.java:43)
com.dremio.sabot.driver.PipelineCreator(PipelineCreator.java:107)
com.dremio.sabot.driver.PipelineCreator(PipelineCreator.java:101)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:320)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:227)
com.dremio.sabot.exec.fragment.FragmentExecutor(FragmentExecutor.java:83)
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl(FragmentExecutor.java:577)
com.dremio.sabot.task.AsyncTaskWrapper(AsyncTaskWrapper.java:92)
com.dremio.sabot.task.slicing.SlicingThread(SlicingThread.java:71)

Issue #4

(java.io.IOException) org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab: Login failure for dremio@DOMAIN.COM from keytab /PATH/TO/KEYTAB/dremio.keytab: Cannot locate KDC

Resolution: The KDC location is stored in the krb5.conf folder. Java, by default, looks for this file in the locations shown in the table below dependent upon the OS.

Operating System Default Location
Windows c:\winnt\krb5.ini
Linux /etc/krb5.conf
Other UNIX-based /etc/krb5/krb5.conf

If the krb5.conf file has been renamed or moved then it is required for us to add it to the DREMIO_JAVA_EXTRA_OPTS giving the location and name of the krb5.conf file. An example of this is:
DREMIO_JAVA_EXTRA_OPTS=”-Djava.security.krb5.conf=/home/Test-krb5.conf”

java.io.IOException: Login failure for dldev@IBCORE-DL-DEV.SWISSBANK.COM from keytab /home/dldev/security/dldev.keytab: javax.security.auth.login.LoginException: Cannot locate KDC
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1062) ~[hadoop-common-2.8.3.jar:na]
    at com.dremio.dac.daemon.DACDaemon.setupHadoopUserUsingKerberosKeytab(DACDaemon.java:223) ~[dremio-dac-backend-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
    at com.dremio.dac.daemon.DACDaemon.<init>(DACDaemon.java:109) ~[dremio-dac-backend-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
    at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:286) ~[dremio-dac-backend-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
    at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:101) ~[dremio-dac-daemon-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
Caused by: javax.security.auth.login.LoginException: Cannot locate KDC
    at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804) ~[na:1.8.0_31]
    at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[na:1.8.0_31]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_31]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_31]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_31]
    at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_31]
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) ~[na:1.8.0_31]
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ~[na:1.8.0_31]
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ~[na:1.8.0_31]
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ~[na:1.8.0_31]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_31]
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ~[na:1.8.0_31]
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ~[na:1.8.0_31]
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1053) ~[hadoop-common-2.8.3.jar:na]
    ... 4 common frames omitted
Caused by: sun.security.krb5.KrbException: Cannot locate KDC
    at sun.security.krb5.Config.getKDCList(Config.java:1061) ~[na:1.8.0_31]
    at sun.security.krb5.KdcComm.send(KdcComm.java:218) ~[na:1.8.0_31]
    at sun.security.krb5.KdcComm.send(KdcComm.java:200) ~[na:1.8.0_31]
    at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316) ~[na:1.8.0_31]
    at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361) ~[na:1.8.0_31]
    at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776) ~[na:1.8.0_31]
    ... 17 common frames omitted
Caused by: sun.security.krb5.KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm IBCORE-DL-DEV.SWISSBANK.COM
    at sun.security.krb5.Config.getKDCFromDNS(Config.java:1158) ~[na:1.8.0_31]
    at sun.security.krb5.Config.getKDCList(Config.java:1034) ~[na:1.8.0_31]
    ... 22 common frames omitted

Issue #5

Connecting to a secured Hive cluster and an unsecured Hive cluster at the same time does not work

This might not be possible because of the security settings in core-site.xml. Details will be completed once tested.


results matching ""

    No results matching ""