Kudu遇到的问题
时间:2022-07-27
本文章向大家介绍Kudu遇到的问题，主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项，具有一定的参考价值，需要的朋友可以参考一下。
安装Kudu
kudu已经集成在了CDP runtime中，安装比较简单，选择master和tablet之后，配置数据目录即可
安装完成后，我们要在impala中显式集成kudu
为了省去每次建表都需要在TBLPROPERTIES中添加kudumasteraddresses属性，我们还要在Impala的高级配置KuduMaster的地址 --kudu_master_hosts=192.168.0.207:7051
在impala-shell中建kudu表
[root@cdh2 ~]# impala-shell
Starting Impala Shell without Kerberos authentication
Opened TCP connection to cdh2.macro.com:21000
Connected to cdh2.macro.com:21000
Server version: impalad version 3.4.0-SNAPSHOT RELEASE (build 25402784335c39cc24076d71dab7a3ccbd562094)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v3.4.0-SNAPSHOT (2540278) built on Wed Aug 5 11:07:32 UTC 2020)
You can change the Impala daemon that you're connected to by using the CONNECT
command.To see how Impala will plan to run your query without actually executing
it, use the EXPLAIN command. You can change the level of detail in the EXPLAIN
output by setting the EXPLAIN_LEVEL query option.
***********************************************************************************
[cdh2.macro.com:21000] default> CREATE TABLE my_first_table
> (
> id BIGINT,
> name STRING,
> PRIMARY KEY(id)
> )
> PARTITION BY HASH PARTITIONS 16
> STORED AS KUDU
> TBLPROPERTIES (
> 'kudu.master_addresses' = 'cdh2.macro.com:7051'
> );
Query: CREATE TABLE my_first_table
(
id BIGINT,
name STRING,
PRIMARY KEY(id)
)
PARTITION BY HASH PARTITIONS 16
STORED AS KUDU
TBLPROPERTIES (
'kudu.master_addresses' = 'cdh2.macro.com:7051'
)
+-------------------------+
| summary |
+-------------------------+
| Table has been created. |
+-------------------------+
Fetched 1 row(s) in 2.35s
[cdh2.macro.com:21000] default> desc formatted my_first_table;
Query: describe formatted my_first_table
+------------------------------+------------------------------------------------------------------------------+------------------------------------------------+
| name | type | comment |
+------------------------------+------------------------------------------------------------------------------+------------------------------------------------+
| # col_name | data_type | comment |
| | NULL | NULL |
| id | bigint | NULL |
| name | string | NULL |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| OwnerType: | USER | NULL |
| Owner: | root | NULL |
| CreateTime: | Sat Sep 12 16:50:11 CST 2020 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | hdfs://cdh2.macro.com:8020/warehouse/tablespace/external/hive/my_first_table | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | EXTERNAL | TRUE |
| | TRANSLATED_TO_EXTERNAL | TRUE |
| | external.table.purge | TRUE |
| | kudu.master_addresses | cdh2.macro.com:7051 |
| | kudu.table_name | impala::default.my_first_table |
| | storage_handler | org.apache.hadoop.hive.kudu.KuduStorageHandler |
| | transient_lastDdlTime | 1599900611 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.kudu.KuduSerDe | NULL |
| InputFormat: | org.apache.hadoop.hive.kudu.KuduInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.kudu.KuduOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | 0 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
+------------------------------+------------------------------------------------------------------------------+------------------------------------------------+
Fetched 31 row(s) in 0.83s我们可以看到kudu表创建成功。
遇到的问题
1、启动过程中报错
Check failed: _s.ok() Bad status:
Invalid argument: Unable to initialize catalog manager: Failed to initialize sys tables
async: on-disk master list解决办法
停掉master和tserver 删掉之前残余的 /kudu_master/fswal_dir/kudu_master/fsdata_dir/kudu_tablet/fswal_dir/kudu_tablet/fsdata_dir几个数据目录
2.建表一直卡住，然后报错:
CreateTablet RPC failed for tablet :not authorized: client connection negotiation failed: client connection to 192.168.0.207:7050: FATAL_UNAUTHORIZED: not authorized: unencrypted connections from publicly routable IPs are prohibited.我们需要在全局gflagfile中配置如下
--rpc_encryption=disabled
--rpc_authentication=disabled
--rusted_subnets=0.0.0.0/03.impala-shell中直接报错
FATAL_UNAUTHORIZED: not authorized: unencrypted connections from publicly routable IPs are prohibited解决办法：
同样在impala中配置
--rpc_encryption=disabled
--rpc_authentication=disabled
--rusted_subnets=0.0.0.0/04.建表报错
ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::default.my_first_table'
CAUSED BY: NonRecoverableException: not enough live tablet servers to create a table with the requested replication factor 3; 2 tablet servers are alive这是因为kudu默认的存储副本是3，由于我这是单节点，副本只有1份（这与HDFS不同，HDFS没作此强制限制，1份副本也可以），于是到CM里面修改副本设置，如下图：
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- 使用 HTTP/2 提升性能的几个建议
- 玩树莓派(raspberry pi) 2/3 raspbian的遇到的一些问题
- Dnsmasq加速本地DNS请求
- swoole深入学习 2. tcp Server和tcp Client
- 一切皆Socket
- Go代码重构：23倍的性能爆增
- 分布式系统的一致性协议之 2PC 和 3PC
- Docker快速搭建WordPress博客网站
- Elasticsearch入门，这一篇就够了
- 玩转 lua in Redis
- 用docker快速搭建wordpress博客
- 生产环境优雅的重启基于Nginx、Tornado的Web服务进程
- nginx记录post body/payload数据 日志用打印出结果
- nginx的location、rewrite玩法详解
- Redis Lua脚本调试