【s3cmd】给s3cmd加点debug日志再编一个

时间:2022-07-27
本文章向大家介绍【s3cmd】给s3cmd加点debug日志再编一个,主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项,具有一定的参考价值,需要的朋友可以参考一下。

最近有同学反映 s3cmd 上传的数据比原文件小…WTF,居然有这种事?我不信,然后看了下同学的需求,上传一个1G的文件,按照分段上传的默认配置,就是以15M为一段,分段上传,基本公式就是 1024M/15M=68.2。

通过给 s3cmd -d put 加下 debug 日志,查看一下分段上传的过程,发现段数和最后一段的尾巴 Bytes 相加确实就是1024M也就是1G,应该不存在丢数据的问题。 业务场景下,这个文件是通过其他进程写,一般来说他们会先写然后 mv到某个目录下再去上传,那到底是哪个环节有问题呢? 先不管是哪个环节的问题,我先要确认不是 s3cmd 的问题,看了下分段上传的代码,决定分段上传的段数的因素就一个,就是文件的总大小,参考下图,这个 size 是通过 python 的 os.stat() 方法获得的。

直接使用的 debug 信息里没有和这个文件的总大小,debug 日志里只有关于文件分了几段15M地上传以及最后的剩余段,所以这次我 debug 的方法就是把这个 size 打印出来,确认 s3cmd put 里获得文件的大小是没问题的,加入的代码如下,编译完后就会生成 rpm 包和相应的 tar 包,解压出来的目录名是以 commit 名来做后缀的,直接使用就可以了。

然后就是重新编一个 s3cmd 出来了,幸好 s3cmd 有提供 Makefile,一个命令就解决了。下面是 s3cmd 编译的日志。

➜  s3cmd git:(master) make git-rpm
rm -rf s3cmd-*.tar.gz *.rpm *~ s3cmd.spec
find . -name *.pyc -exec rm {} ;
find . -name *.pyo -exec rm {} ;
git archive --format tar --prefix s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16/ HEAD S3/ s3cmd NEWS README.md LICENSE INSTALL.md setup.cfg s3cmd.1 setup.py| gzip -c > s3cmd-2.1.0+-e2bf7668.tar.gz
sed -e 's/##VERSION##/2.1.0+/' 
            -e 's/##COMMIT##/e2bf76682d405d88b7c6beb1cd319b811ce30c16/' 
            -e 's/##SHORTCOMMIT##/e2bf7668/' 
            s3cmd.spec.in > s3cmd.spec
tmp_dir=`mktemp -d` ; 
mkdir -p ${tmp_dir}/{BUILD,RPMS,SRPMS,SPECS,SOURCES} ; 
cp s3cmd-2.1.0+-e2bf7668.tar.gz ${tmp_dir}/SOURCES ; 
cp s3cmd.spec ${tmp_dir}/SPECS ; 
cd ${tmp_dir} > /dev/null 2>&1; 
rpmbuild -ba --define "_topdir ${tmp_dir}" 
  --define "_source_filedigest_algorithm 0" 
  --define "_binary_filedigest_algorithm 0" 
  --define "dist %{nil}" 
          SPECS/s3cmd.spec ; 
cd - > /dev/null 2>&1; 
cp ${tmp_dir}/RPMS/noarch/* ${tmp_dir}/SRPMS/* . ; 
rm -rf ${tmp_dir} ; 
rpmlint *.rpm *.spec
执行(%prep): /bin/sh -e /var/tmp/rpm-tmp.U9O8Qx
+ umask 022
+ cd /tmp/tmp.6mpg1lGxXd/BUILD
+ cd /tmp/tmp.6mpg1lGxXd/BUILD
+ rm -rf s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
+ /usr/bin/gzip -dc /tmp/tmp.6mpg1lGxXd/SOURCES/s3cmd-2.1.0+-e2bf7668.tar.gz
+ /usr/bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ exit 0
执行(%build): /bin/sh -e /var/tmp/rpm-tmp.iX0qIl
+ umask 022
+ cd /tmp/tmp.6mpg1lGxXd/BUILD
+ cd s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
+ exit 0
执行(%install): /bin/sh -e /var/tmp/rpm-tmp.QLQ6z9
+ umask 022
+ cd /tmp/tmp.6mpg1lGxXd/BUILD
+ '[' /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64 '!=' / ']'
+ rm -rf /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
++ dirname /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
+ mkdir -p /tmp/tmp.6mpg1lGxXd/BUILDROOT
+ mkdir /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
+ cd s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
+ rm -rf /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
+ S3CMD_PACKAGING=Yes
+ python setup.py install --prefix=/usr --root=/tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
Using xml.etree.ElementTree for XML processing
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/S3
copying S3/Custom_httplib27.py -> build/lib/S3
copying S3/ExitCodes.py -> build/lib/S3
copying S3/HashCache.py -> build/lib/S3
copying S3/SortedDict.py -> build/lib/S3
copying S3/CloudFront.py -> build/lib/S3
copying S3/FileLists.py -> build/lib/S3
copying S3/S3Uri.py -> build/lib/S3
copying S3/AccessLog.py -> build/lib/S3
copying S3/Custom_httplib3x.py -> build/lib/S3
copying S3/Utils.py -> build/lib/S3
copying S3/Config.py -> build/lib/S3
copying S3/Crypto.py -> build/lib/S3
copying S3/Exceptions.py -> build/lib/S3
copying S3/MultiPart.py -> build/lib/S3
copying S3/__init__.py -> build/lib/S3
copying S3/ACL.py -> build/lib/S3
copying S3/PkgInfo.py -> build/lib/S3
copying S3/ConnMan.py -> build/lib/S3
copying S3/FileDict.py -> build/lib/S3
copying S3/Progress.py -> build/lib/S3
copying S3/S3.py -> build/lib/S3
copying S3/BidirMap.py -> build/lib/S3
running build_scripts
creating build/scripts-2.7
copying and adjusting s3cmd -> build/scripts-2.7
changing mode of build/scripts-2.7/s3cmd from 644 to 755
running install_lib
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Custom_httplib27.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/ExitCodes.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/HashCache.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/SortedDict.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/CloudFront.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/FileLists.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/S3Uri.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/AccessLog.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Custom_httplib3x.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Utils.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Config.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Crypto.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Exceptions.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/MultiPart.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/__init__.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/ACL.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/PkgInfo.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/ConnMan.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/FileDict.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/Progress.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/S3.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
copying build/lib/S3/BidirMap.py -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Custom_httplib27.py to Custom_httplib27.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/ExitCodes.py to ExitCodes.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/HashCache.py to HashCache.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/SortedDict.py to SortedDict.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/CloudFront.py to CloudFront.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/FileLists.py to FileLists.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/S3Uri.py to S3Uri.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/AccessLog.py to AccessLog.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Custom_httplib3x.py to Custom_httplib3x.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Utils.py to Utils.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Config.py to Config.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Crypto.py to Crypto.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Exceptions.py to Exceptions.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/MultiPart.py to MultiPart.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/__init__.py to __init__.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/ACL.py to ACL.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/PkgInfo.py to PkgInfo.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/ConnMan.py to ConnMan.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/FileDict.py to FileDict.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/Progress.py to Progress.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/S3.py to S3.pyc
byte-compiling /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/S3/BidirMap.py to BidirMap.pyc
running install_egg_info
running egg_info
creating s3cmd.egg-info
writing requirements to s3cmd.egg-info/requires.txt
writing s3cmd.egg-info/PKG-INFO
writing top-level names to s3cmd.egg-info/top_level.txt
writing dependency_links to s3cmd.egg-info/dependency_links.txt
writing manifest file 's3cmd.egg-info/SOURCES.txt'
reading manifest file 's3cmd.egg-info/SOURCES.txt'
writing manifest file 's3cmd.egg-info/SOURCES.txt'
Copying s3cmd.egg-info to /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7/site-packages/s3cmd-2.1.0_-py2.7.egg-info
running install_scripts
creating /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/bin
copying build/scripts-2.7/s3cmd -> /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/bin
changing mode of /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/bin/s3cmd to 755
+ install -d /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/man/man1
+ install -m 644 s3cmd.1 /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/man/man1
+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id -m --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 /tmp/tmp.6mpg1lGxXd/BUILD/s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
/usr/lib/rpm/sepdebugcrcfix: Updated 0 CRC32s, 0 CRC32s did match.
find: ‘debug’: No such file or directory
+ /usr/lib/rpm/check-buildroot
+ /usr/lib/rpm/redhat/brp-compress
+ /usr/lib/rpm/redhat/brp-strip-static-archive /usr/bin/strip
+ /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
Bytecompiling .py files below /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/lib/python2.7 using /usr/bin/python2.7
+ /usr/lib/rpm/redhat/brp-python-hardlink
+ /usr/lib/rpm/redhat/brp-java-repack-jars
处理文件:s3cmd-2.1.0+-1.noarch
执行(%doc): /bin/sh -e /var/tmp/rpm-tmp.4nqsHY
+ umask 022
+ cd /tmp/tmp.6mpg1lGxXd/BUILD
+ cd s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
+ DOCDIR=/tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/doc/s3cmd-2.1.0+
+ export DOCDIR
+ /usr/bin/mkdir -p /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/doc/s3cmd-2.1.0+
+ cp -pr NEWS /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/doc/s3cmd-2.1.0+
+ cp -pr README.md /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/doc/s3cmd-2.1.0+
+ cp -pr LICENSE /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64/usr/share/doc/s3cmd-2.1.0+
+ exit 0
Provides: s3cmd = 2.1.0+-1
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PartialHardlinkSets) <= 4.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: /usr/bin/python python(abi) = 2.7
检查未打包文件:/usr/lib/rpm/check-files /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
写道:/tmp/tmp.6mpg1lGxXd/SRPMS/s3cmd-2.1.0+-1.src.rpm
写道:/tmp/tmp.6mpg1lGxXd/RPMS/noarch/s3cmd-2.1.0+-1.noarch.rpm
执行(%clean): /bin/sh -e /var/tmp/rpm-tmp.AR3EKw
+ umask 022
+ cd /tmp/tmp.6mpg1lGxXd/BUILD
+ cd s3cmd-e2bf76682d405d88b7c6beb1cd319b811ce30c16
+ rm -rf /tmp/tmp.6mpg1lGxXd/BUILDROOT/s3cmd-2.1.0+-1.x86_64
+ exit 0
s3cmd.noarch: W: incoherent-version-in-changelog 1.5.1.2-5 ['2.1.0+-1', '2.1.0+-1']

s3cmd.noarch: W: invalid-url URL: http://s3tools.com <urlopen error [Errno -3] Temporary failure in name resolution>
s3cmd.noarch: W: manual-page-warning /usr/share/man/man1/s3cmd.1.gz 271: warning: macro `restore'' not defined
s3cmd.src: W: invalid-url URL: http://s3tools.com <urlopen error [Errno -3] Temporary failure in name resolution>
s3cmd.src: W: invalid-url Source0: https://github.com/s3tools/s3cmd/archive/e2bf76682d405d88b7c6beb1cd319b811ce30c16/s3cmd-2.1.0+-e2bf7668.tar.gz HTTP Error 404: Not Found
s3cmd.spec: W: invalid-url Source0: https://github.com/s3tools/s3cmd/archive/e2bf76682d405d88b7c6beb1cd319b811ce30c16/s3cmd-2.1.0+-e2bf7668.tar.gz HTTP Error 404: Not Found
2 packages and 1 specfiles checked; 0 errors, 6 warnings.