作者:mobiledu2502909783 | 来源:互联网 | 2023-09-12 13:11
执行sparksql,插入hive时候提示以下警告:2021-04-0300:22:02WARN[org.apache.hadoop.hive.shims.HadoopShimsS
执行sparksql,插入hive 时候提示以下警告:
2021-04-03 00:22:02 WARN [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00000
from file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00000
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174)
at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:3825)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6784)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setAcl(FSNamesystem.java:9296)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setAcl(NameNodeRpcServer.java:1642)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setAcl(AuthorizationProviderProxyClientProtocol.java:902)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setAcl(ClientNamenodeProtocolServerSideTranslatorPB.java:1347)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
2021-04-03 00:22:02 WARN [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00004 from file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00004
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174)
at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)
处理方式:
设置hive参数 hive.warehouse.subdir.inherit.perms:false,关闭hive的文件权限继承来规避该问题
SparkConf sparkCOnf= new SparkConf();
SparkSession session= SparkSession
.builder()
.config(sparkConf)
.config("hive.warehouse.subdir.inherit.perms","false")
.enableHiveSupport()
.getOrCreate();