compression功能测试

compression功能测试

1.启动compression

功能描述: 1.目前从k版本开始支持compression，目前支持zlib、snappy、zstd，注意从ceph version 12.0.0开始才加入了zstd插件。 2.注意配置的时候如果设置的插件不存在或者名称错误，则不启用压缩特性。 3.目前压缩特性只对data_pool这个pool的数据有效。 4. 可以实时变更插件类型而不影响之前写入的数据访问。（写入object的时候每个object都有一个内置变量标记当前object使用的压缩插件类型。） 5.compression特性对用户透明，用户是无法感知压缩功能是否存在的。

root@demo:/home/user# radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zstd
{
    "id": "eab5746a-26fa-42b4-a5f3-5d887ba846f9",
    "name": "default",
    "domain_root": "default.rgw.data.root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.gc",
    "lc_pool": "default.rgw.lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.intent-log",
    "usage_log_pool": "default.rgw.usage",
    "user_keys_pool": "default.rgw.users.keys",
    "user_email_pool": "default.rgw.users.email",
    "user_swift_pool": "default.rgw.users.swift",
    "user_uid_pool": "default.rgw.users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": "zstd" #插件类型
            }
        },
        {
            "key": "new",
            "val": {
                "index_pool": "new.buckets.index",
                "data_pool": "new.buckets.data",
                "data_extra_pool": "new.buckets.extra",
                "index_type": 0,
                "compression": "zlib" #插件类型
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": ""
}

2.查看compression配置

root@demo:/home/user# radosgw-admin zone placement list --rgw-zone=default
[
    {
        "key": "default-placement",
        "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 0,
            "compression": "zstd" #插件类型
        }
    },
    {
        "key": "new",
        "val": {
            "index_pool": "new.buckets.index",
            "data_pool": "new.buckets.data",
            "data_extra_pool": "new.buckets.extra",
            "index_type": 0,
            "compression": "zlib" #插件类型
        }
    }
]

3.压缩效果对比

测试使用一个iso镜像文件，对比压缩前后的体检差异

root@demo:/home/user# radosgw-admin bucket stats --bucket=snappy-test
{
    "bucket": "snappy-test",
    "pool": "default.rgw.buckets.data",
    "index_pool": "default.rgw.buckets.index",
    "id": "0fef9464-bfe0-428b-86b5-b8d51876ff81.4274.1",
    "marker": "0fef9464-bfe0-428b-86b5-b8d51876ff81.4274.1",
    "owner": "u-gfn2636",
    "ver": "0#41",
    "master_ver": "0#0",
    "mtime": "2017-03-21 09:44:13.698936",
    "max_marker": "0#",
    "usage": {
        "rgw.main": {
            "size": 258998272,
            "size_actual": 248091783 ，#实际文件大小
            "size_utilized": 248091783, #压缩后的大小
            "size_kb": 252928,
            "size_kb_actual": 252928,
            "size_kb_utilized": 242278,
            "num_objects": 1
        },
        "rgw.multimeta": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 0
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

4.总结

不考虑性能和时间的情况下，zstd效果最佳，由于每个插件还有一些压缩测试可以微调，所以测试的结果仅供参考。

附 zstd介绍 https://github.com/facebook/zstd snappy介绍 https://github.com/google/snappy