HDFS的Shell操作(开发重点)
时间:2022-07-24
本文章向大家介绍HDFS的Shell操作(开发重点),主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项,具有一定的参考价值,需要的朋友可以参考一下。
图丨pixabay
1.基本语法
bin/hadoop fs
具体命令 OR bin/hdfs dfs
具体命令
dfs
是fs
的实现类。
2.命令大全
[hadoop@hadoop103 hadoop-2.7.2]$ bin/hadoop fs
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] <path> ...]
[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-getfacl [-R] <path>]
[-getfattr [-R] {-n name | -d} [-e en] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-d] [-h] [-R] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {-n name [-v value] | -x name} <path>]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touchz <path> ...]
[-truncate [-w] <length> <path> ...]
[-usage [cmd ...]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|resourcemanager:port> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
3.常用命令实操
先启动集群:
102
[hadoop@hadoop102 ~]$ cd /opt/module/hadoop-2.7.2/
[hadoop@hadoop102 hadoop-2.7.2]$ sbin/start-dfs.sh
在这里插入图片描述
103
[hadoop@hadoop103 ~]$ cd /opt/module/hadoop-2.7.2/
[hadoop@hadoop103 hadoop-2.7.2]$ sbin/start-yarn.sh
在这里插入图片描述
(1)-help:输出这个命令参数
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -help rm
(2)-ls: 显示目录信息
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -ls /
(3)-mkdir:在HDFS上创建目录
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -mkdir -p /chongqing/chengkou
(4)-moveFromLocal:从本地剪切粘贴到HDFS
[hadoop@hadoop103 hadoop-2.7.2]$ touch bossxiang.txt
[hadoop@hadoop103 hadoop-2.7.2]$ vi bossxiang.txt
woshibossxiang
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -moveFromLocal ./bossxiang.txt /chongqing/chengkou
(5)-appendToFile:追加一个文件到已经存在的文件末尾
[hadoop@hadoop103 hadoop-2.7.2]$ touch yuan.txt
[hadoop@hadoop103 hadoop-2.7.2]$ vi yuan.txt
wo shi chen yuan
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -appendToFile ./yuan.txt /chongqing/chengkou/bossxiang.txt
(6)-cat:显示文件内容
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -cat /chongqing/chengkou/bossxiang.txt
[hadoop@hadoop103 hadoop-2.7.2]$ hadoop fs -chgrp hadoop /chongqing/chengkou/bossxiang.txt
在这里插入图片描述
(8)-copyFromLocal:从本地文件系统中拷贝文件到HDFS路径去
[hadoop@hadoop102 hadoop-2.7.2]$ touch xinyue.txt
[hadoop@hadoop102 hadoop-2.7.2]$ vi xinyue.txt
wo shi xinyue
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -copyFromLocal ./xinyue.txt /chongqing/chengkou/
(9)-copyToLocal:从HDFS拷贝到本地
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -copyToLocal /chongqing/chengkou/bossxiang.txt ./
(10)-cp :从HDFS的一个路径拷贝到HDFS的另一个路径
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -cp /chongqing/chengkou/xinyue.txt /chongqing/
在这里插入图片描述
(11)-mv:在HDFS目录中移动文件
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -mv /chongqing/xinyue.txt /
在这里插入图片描述
(12)-get:等同于copyToLocal
,就是从HDFS下载文件到本地
[hadoop@hadoop102 hadoop-2.7.2]$ rm xinyue.txt
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -get /xinyue.txt
(13)-getmerge:合并下载多个文件,比如HDFS的目录 /user/atguigu/test
下有多个文件:log.1, log.2,log.3,…
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -getmerge /chongqing/chengkou/* ./zaiyiqi.txt
[hadoop@hadoop102 hadoop-2.7.2]$ cat zaiyiqi.txt
woshibossxiang
wo shi xinyue
在这里插入图片描述
(14)-put:等同于copyFromLocal
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -put ./LICENSE.txt /chongqing/chengkou
在这里插入图片描述
(15)-tail:显示一个文件的末尾
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -tail /chongqing/chengkou/LICENSE.txt
mentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
(16)-rm:删除文件或文件夹
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -rm /chongqing/chengkou/LICENSE.txt
20/08/08 04:21:42 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted /chongqing/chengkou/LICENSE.txt
(17)-rmdir:删除空目录
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -mkdir /test
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -rmdir /test
(18)-du统计文件夹的大小信息
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -du /
29 /chongqing
212046774 /hadoop-2.7.2.tar.gz
13 /sanguo
37 /wc.input
14 /xinyue.txt
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -du -h /
29 /chongqing
202.2 M /hadoop-2.7.2.tar.gz
13 /sanguo
37 /wc.input
14 /xinyue.txt
[hadoop@hadoop102 hadoop-2.7.2]$ hadoop fs -du -h -s /
202.2 M /
(19)-setrep:设置HDFS中文件的副本数量
[hadoop@hadoop102 hadoop-2.7.2]$ cd data/tmp/dfs/data/current/BP-1454198558-117.59.224.141-1595323236787/current/finalized/subdir0/subdir0/
[hadoop@hadoop102 subdir0]$ hadoop fs -rm -R /chongqing
[hadoop@hadoop102 subdir0]$ hadoop fs -rm /hadoop-2.7.2.tar.gz
[hadoop@hadoop102 subdir0]$ hadoop fs -rm -R /sanguo
[hadoop@hadoop102 subdir0]$ ll
total 207096
-rw-rw-r--. 1 hadoop hadoop 37 Jul 21 09:10 blk_1073741825
-rw-rw-r--. 1 hadoop hadoop 11 Jul 21 09:10 blk_1073741825_1001.meta
-rw-rw-r--. 1 hadoop hadoop 14 Aug 8 02:57 blk_1073741831
-rw-rw-r--. 1 hadoop hadoop 11 Aug 8 02:57 blk_1073741831_1011.meta
drwxr-xr-x. 9 hadoop hadoop 149 Jan 25 2016 hadoop-2.7.2
-rw-rw-r--. 1 hadoop hadoop 212046774 Jul 21 09:26 tmp.txt
[hadoop@hadoop102 subdir0]$ cat blk_1073741825
xiang
xiang
lin
lin yuan chen yuan
[hadoop@hadoop102 subdir0]$ cat blk_1073741831
wo shi xinyue
[hadoop@hadoop102 subdir0]$
[hadoop@hadoop102 finalized]$ hadoop fs -setrep 2 /xinyue.txt
所有巧合的是要么是上天注定要么是一个人偷偷的在努力。
结束!
参考链接:尚硅谷hadoop教程
- 手把手教 | 深度学习库PyTorch(附代码)
- 如何定位Obj-C野指针随机Crash(一):先提高野指针Crash率
- 如何定位Obj-C野指针随机Crash(二):让非必现Crash变成必现
- 六种开发环境部署大全:基于Openshift
- 手把手教你由TensorFlow上手PyTorch(附代码)
- 如何定位Obj-C野指针随机Crash(三):如何让Crash自报家门
- 【MIG专项测试组】腾讯手机管家实战分析:内存突增是为神马?
- 全系统栈崩溃是什么鬼?手机管家高级工程师 jaylin,教你如何抓鬼!
- 【团队分享】苍翼之刃:论File Descriptor泄漏如何导致Crash?
- 【沙龙干货分享】你要知道的N个Android适配问题
- 优化安卓应用内存的神秘方法以及背后的原理,一般人我不告诉他
- 安全快速下载Xcode的姿势
- Android之实现妙趣横生的粘连布局
- 萝莉有话说:你的App真正适配了iOS 9吗?
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- 前端黑魔法之远程控制地址栏
- 攻击LNMP架构Web应用的几个小Tricks
- Hbase-2.0.0_02_常用操作
- phpjiami 数种解密方法
- Hbase-2.0.0_03_Hbase数据模型
- ELK-elasticsearch-6.3.2部署
- Real World CTF 2018 bookhub 总结
- python http.server open redirect vulnerability
- ELK-elasticsearch-6.3.2插件【head,bigdesk,cerebro[kopf]】安装
- ELK-kibana-6.3.2部署
- node.js + postgres 从注入到Getshell
- ELK-logstash-6.3.2部署
- K8S节点异常怎么办?TKE"节点健康检查和自愈"来帮忙
- ELK-logstash-6.3.2-常用配置
- ELK-elkstack-使用消息队列