编译及使用hive-testbench生成Hive基准测试数据
TPC-DS采用星型、雪花型等多维数据模式。它包含7张事实表,17张纬度表平均每张表含有18列。其工作负载包含99个SQL查询,覆盖SQL99和2003的核心部分以及OLAP。这个测试集包含对大数据集的统计、报表生成、联机查询、数据挖掘等复杂应用,测试用的数据和值是有倾斜的,与真实数据一致。TPC-DS是与真实场景非常接近的一个测试集,也是难度较大的一个测试集。
TPC-DS成为客观衡量多个不同Hadoop版本以及SQL on Hadoop技术的最佳测试集。这个基准测试有以下几个主要特点:
- 一共99个测试案例,遵循SQL’99和SQL 2003的语法标准,SQL案例比较复杂
- 分析的数据量大,并且测试案例是在回答真实的商业问题
- 测试案例中包含各种业务模型(如分析报告型,迭代式的联机分析型,数据挖掘型等)
- 几乎所有的测试案例都有很高的IO负载和CPU计算需求
hive-testbench提供了基于TPC-DS和TPC-H基准测试的数据生成器和示例查询。这里我们使用TPC-DS。
环境准备
从GitHub官网clone hive-testbench源码,Git地址如下:
https://github.com/hortonworks/hive-testbench.git
安装hive-testbench编译依赖环境
yum -y install gcc gcc-c++
编译并打包hive-testbench
在hive-testbench目录下执行如下脚本进行编译打包
./tpcds-build.sh
等待编译成功即可。
生成并加载数据
在hive-testbench目录下执行如下脚本生成并加载测试数据,生成数据的方式是向集群提交一个MapReduce作业
./tpcds-setup.sh 5
5表示生成的数据量大小GB单位,我们的测试集群规模比较小,这里先生成5G数据
后面可以跟一个数据生成的目录,目录不存在则自动生成,如果不指定数据目录则默认生成到tpcds-generate目录下。
有如上图显示则表示脚本执行成功
Hive查看生成的两个数据库tpcds_bin_partitioned_orc_5,tpcds_text_5
命令行查看HDFS上的数据是否与我们指定的量一致,各个表的大小
数据总量与指定5GB数据量一致
通过Hue验证生成的测试数据
使用Impala命令创建Parquet格式表
使用Impala命令将Hive 库中Text格式的表转换给Parquet格式的表,将tpcds_text_5库中所有表数据插入到对应Parquet格式的表中并对表执行分析
在cdp2.soundhearer.cn节点执行命令:
impala-shell -f ddl_impala_parquet.sql
SQL脚本如下
drop database if exists tpcds_parquet_5 cascade;
create database tpcds_parquet_5;
use tpcds_parquet_5;
set parquet_file_size=512M;
set COMPRESSION_CODEC=snappy;
create table call_center like tpcds_text_5.call_center stored as parquetfile;
create table catalog_page like tpcds_text_5.catalog_page stored as parquetfile;
create table catalog_returns like tpcds_text_5.catalog_returns stored as parquetfile;
create table catalog_sales like tpcds_text_5.catalog_sales stored as parquetfile;
create table customer_address like tpcds_text_5.customer_address stored as parquetfile;
create table customer_demographics like tpcds_text_5.customer_demographics stored as parquetfile;
create table customer like tpcds_text_5.customer stored as parquetfile;
create table date_dim like tpcds_text_5.date_dim stored as parquetfile;
create table household_demographics like tpcds_text_5.household_demographics stored as parquetfile;
create table income_band like tpcds_text_5.income_band stored as parquetfile;
create table inventory like tpcds_text_5.inventory stored as parquetfile;
create table item like tpcds_text_5.item stored as parquetfile;
create table promotion like tpcds_text_5.promotion stored as parquetfile;
create table reason like tpcds_text_5.reason stored as parquetfile;
create table ship_mode like tpcds_text_5.ship_mode stored as parquetfile;
create table store_returns like tpcds_text_5.store_returns stored as parquetfile;
create table store_sales like tpcds_text_5.store_sales stored as parquetfile;
create table store like tpcds_text_5.store stored as parquetfile;
create table time_dim like tpcds_text_5.time_dim stored as parquetfile;
create table warehouse like tpcds_text_5.warehouse stored as parquetfile;
create table web_page like tpcds_text_5.web_page stored as parquetfile;
create table web_returns like tpcds_text_5.web_returns stored as parquetfile;
create table web_sales like tpcds_text_5.web_sales stored as parquetfile;
create table web_site like tpcds_text_5.web_site stored as parquetfile;
insert overwrite table call_center select * from tpcds_text_5.call_center;
insert overwrite table catalog_page select * from tpcds_text_5.catalog_page;
insert overwrite table catalog_returns select * from tpcds_text_5.catalog_returns;
insert overwrite table catalog_sales select * from tpcds_text_5.catalog_sales;
insert overwrite table customer_address select * from tpcds_text_5.customer_address;
insert overwrite table customer_demographics select * from tpcds_text_5.customer_demographics;
insert overwrite table customer select * from tpcds_text_5.customer;
insert overwrite table date_dim select * from tpcds_text_5.date_dim;
insert overwrite table household_demographics select * from tpcds_text_5.household_demographics;
insert overwrite table income_band select * from tpcds_text_5.income_band;
insert overwrite table inventory select * from tpcds_text_5.inventory;
insert overwrite table item select * from tpcds_text_5.item;
insert overwrite table promotion select * from tpcds_text_5.promotion;
insert overwrite table reason select * from tpcds_text_5.reason;
insert overwrite table ship_mode select * from tpcds_text_5.ship_mode;
insert overwrite table store_returns select * from tpcds_text_5.store_returns;
insert overwrite table store_sales select * from tpcds_text_5.store_sales;
insert overwrite table store select * from tpcds_text_5.store;
insert overwrite table time_dim select * from tpcds_text_5.time_dim;
insert overwrite table warehouse select * from tpcds_text_5.warehouse;
insert overwrite table web_page select * from tpcds_text_5.web_page;
insert overwrite table web_returns select * from tpcds_text_5.web_returns;
insert overwrite table web_sales select * from tpcds_text_5.web_sales;
insert overwrite table web_site select * from tpcds_text_5.web_site;
compute stats call_center ;
compute stats catalog_page ;
compute stats catalog_returns ;
compute stats catalog_sales ;
compute stats customer_address ;
compute stats customer_demographics ;
compute stats customer ;
compute stats date_dim ;
compute stats household_demographics ;
compute stats income_band ;
compute stats inventory ;
compute stats item ;
compute stats promotion ;
compute stats reason ;
compute stats ship_mode ;
compute stats store_returns ;
compute stats store_sales ;
compute stats store ;
compute stats time_dim ;
compute stats warehouse ;
compute stats web_page ;
compute stats web_returns ;
compute stats web_sales ;
compute stats web_site ;
查看Impala表信息
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法