聊聊claudb的exportRDB
时间:2022-07-23
本文章向大家介绍聊聊claudb的exportRDB,主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项,具有一定的参考价值,需要的朋友可以参考一下。
序
本文主要研究一下claudb的exportRDB
exportRDB
claudb-1.7.1/src/main/java/com/github/tonivade/claudb/DBServerState.java
public class DBServerState {
//......
public void exportRDB(OutputStream output) throws IOException {
RDBOutputStream rdb = new RDBOutputStream(output);
rdb.preamble(RDB_VERSION);
for (int i = 0; i < databases.size(); i++) {
Database db = databases.get(i);
if (!db.isEmpty()) {
rdb.select(i);
rdb.dabatase(db);
}
}
rdb.end();
}
//......
}
- exportRDB方法先通过rdb.preamble(RDB_VERSION)写入redis魔数及版本;然后遍历databases,挨个执行rdb.select(i)写入SELECT及db的长度,在执行rdb.dabatase(db),遍历entry,挨个按expiredAt、type、key、value写入数据;end方法写入END_OF_STREAM,然后再写入checksum
RDBOutputStream
claudb-1.7.1/src/main/java/com/github/tonivade/claudb/persistence/RDBOutputStream.java
public class RDBOutputStream {
private static final byte[] REDIS = safeString("REDIS").getBytes();
private static final int TTL_MILLISECONDS = 0xFC;
private static final int END_OF_STREAM = 0xFF;
private static final int SELECT = 0xFE;
private final CheckedOutputStream out;
public RDBOutputStream(OutputStream out) {
super();
this.out = new CheckedOutputStream(out, new CRC64());
}
public void preamble(int version) throws IOException {
out.write(REDIS);
out.write(version(version));
}
private byte[] version(int version) {
StringBuilder sb = new StringBuilder(String.valueOf(version));
for (int i = sb.length(); i < Integer.BYTES; i++) {
sb.insert(0, '0');
}
return sb.toString().getBytes(StandardCharsets.UTF_8);
}
public void select(int db) throws IOException {
out.write(SELECT);
length(db);
}
public void dabatase(Database db) throws IOException {
for (Tuple2<DatabaseKey, DatabaseValue> entry : db.entrySet()) {
value(entry.get1(), entry.get2());
}
}
private void value(DatabaseKey key, DatabaseValue value) throws IOException {
expiredAt(value.getExpiredAt());
type(value.getType());
key(key);
value(value);
}
private void expiredAt(Instant expiredAt) throws IOException {
if (expiredAt != null) {
out.write(TTL_MILLISECONDS);
out.write(ByteUtils.toByteArray(expiredAt.toEpochMilli()));
}
}
private void type(DataType type) throws IOException {
out.write(type.ordinal());
}
private void key(DatabaseKey key) throws IOException {
string(key.getValue());
}
private void value(DatabaseValue value) throws IOException {
switch (value.getType()) {
case STRING:
string(value.getString());
break;
case LIST:
list(value.getList());
break;
case HASH:
hash(value.getHash());
break;
case SET:
set(value.getSet());
break;
case ZSET:
zset(value.getSortedSet());
break;
default:
break;
}
}
private void length(int length) throws IOException {
if (length < 0x40) {
// 1 byte: 00XXXXXX
out.write(length);
} else if (length < 0x4000) {
// 2 bytes: 01XXXXXX XXXXXXXX
int b1 = length >> 8;
int b2 = length & 0xFF;
out.write(0x40 | b1);
out.write(b2);
} else {
// 5 bytes: 10...... XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
out.write(0x80);
out.write(toByteArray(length));
}
}
private void string(String value) throws IOException {
string(safeString(value));
}
private void string(SafeString value) throws IOException {
byte[] bytes = value.getBytes();
length(bytes.length);
out.write(bytes);
}
private void string(double value) throws IOException {
string(String.valueOf(value));
}
private void list(ImmutableList<SafeString> value) throws IOException {
length(value.size());
for (SafeString item : value) {
string(item);
}
}
private void hash(ImmutableMap<SafeString, SafeString> value) throws IOException {
length(value.size());
for (Tuple2<SafeString, SafeString> entry : value.entries()) {
string(entry.get1());
string(entry.get2());
}
}
private void set(ImmutableSet<SafeString> value) throws IOException {
length(value.size());
for (SafeString item : value) {
string(item);
}
}
private void zset(NavigableSet<Entry<Double, SafeString>> value) throws IOException {
length(value.size());
for (Entry<Double, SafeString> item : value) {
string(item.getValue());
string(item.getKey());
}
}
public void end() throws IOException {
out.write(END_OF_STREAM);
out.write(toByteArray(out.getChecksum().getValue()));
out.flush();
}
}
- RDBOutputStream的构造器用CheckedOutputStream包装了OutputStream;其dabatase方法遍历db.entrySet(),挨个执行value方法;value方法先分别执行expiredAt、type、key、value方法;value方法针对STRING、LIST、HASH、SET、ZSET这几种value类型做了不同的处理;string方法直接写入string;list方法先写入list大小,再挨个写入list元素;hash方法先写入hash大小,再挨个写入key和value;set先写入set大小,在挨个写入set元素;zset先写入zset大小,再挨个写入value和score
CheckedOutputStream
java.base/java/util/zip/CheckedOutputStream.java
public
class CheckedOutputStream extends FilterOutputStream {
private Checksum cksum;
/**
* Creates an output stream with the specified Checksum.
* @param out the output stream
* @param cksum the checksum
*/
public CheckedOutputStream(OutputStream out, Checksum cksum) {
super(out);
this.cksum = cksum;
}
/**
* Writes a byte. Will block until the byte is actually written.
* @param b the byte to be written
* @exception IOException if an I/O error has occurred
*/
public void write(int b) throws IOException {
out.write(b);
cksum.update(b);
}
/**
* Writes an array of bytes. Will block until the bytes are
* actually written.
* @param b the data to be written
* @param off the start offset of the data
* @param len the number of bytes to be written
* @exception IOException if an I/O error has occurred
*/
public void write(byte[] b, int off, int len) throws IOException {
out.write(b, off, len);
cksum.update(b, off, len);
}
/**
* Returns the Checksum for this output stream.
* @return the Checksum
*/
public Checksum getChecksum() {
return cksum;
}
}
- CheckedOutputStream继承了FilterOutputStream,其构造器要求输入OutputStream及Checksum,每次write的时候都会执行cksum.update;其getChecksum方法直接返回cksum
CRC64
public class CRC64 implements Checksum {
private static final int LOOKUPTABLE_SIZE = 256;
private static final long POLY64REV = 0xC96C5795D7870F42L;
private static final long LOOKUPTABLE[] = new long[LOOKUPTABLE_SIZE];
private long crc = -1;
static {
for (int b = 0; b < LOOKUPTABLE.length; ++b) {
long r = b;
for (int i = 0; i < Long.BYTES; ++i) {
if ((r & 1) == 1) {
r = (r >>> 1) ^ POLY64REV;
} else {
r >>>= 1;
}
}
LOOKUPTABLE[b] = r;
}
}
@Override
public void update(int b) {
crc = LOOKUPTABLE[((b & 0xFF) ^ (int) crc) & 0xFF] ^ (crc >>> 8);
}
@Override
public void update(byte[] buf, int off, int len) {
int end = off + len;
while (off < end) {
crc = LOOKUPTABLE[(buf[off++] ^ (int) crc) & 0xFF] ^ (crc >>> 8);
}
}
@Override
public long getValue() {
return ~crc;
}
@Override
public void reset() {
crc = -1;
}
}
- CRC64实现了Checksum接口,其update方法会借用LOOKUPTABLE更新crc
小结
exportRDB方法先通过rdb.preamble(RDB_VERSION)写入redis魔数及版本;然后遍历databases,挨个执行rdb.select(i)写入SELECT及db的长度,在执行rdb.dabatase(db),遍历entry,挨个按expiredAt、type、key、value写入数据;end方法写入END_OF_STREAM,然后再写入checksum
doc
- TensorFlow从0到1丨 第六篇:解锁梯度下降算法
- .Net多线程编程—误用点分析
- Web开发常见的几个漏洞解决方法
- .Net多线程编程—同步机制
- .Net多线程编程—Parallel LINQ、线程池
- 没有自己的服务器如何学习生物数据分析(下篇)
- .Net多线程编程—并发集合
- .Net多线程编程—任务Task
- 学会WCF之试错法——安全配置报错分析
- 生物信息学技能面试题(第5题)-根据GTF画基因的多个转录本结构
- 学会WCF之试错法——超时
- 学会WCF之试错法——客户端调用基础
- 【直播】我的基因组58:用R包SNPRelate来对我的基因型跟hapmap计划数据比较
- 生物信息学技能面试题(第4题)-多个同样的行列式文件合并起来
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- Python 简单实现贪吃蛇小游戏
- Python数据可视化 词云图 绘制词云的方法总结
- Python 利用OpenCV给照片换底色
- Python爬虫实战 批量下载高清美女图片
- Python matplotlib数据可视化 绘制柱形图、堆叠图、折线图、饼图和环图
- Python Matplotlib数据可视化 绘制箱形图、散点图和直方图
- 【Lighthouse教程】scrapy爬虫初探
- Python爬虫 爬取B站视频弹幕 + 绘制词云
- Python爬虫学习笔记 asyncio+aiohttp 异步爬虫原理和解析
- Python数据分析 利用NetworkX绘制网络图
- Python爬虫练习 爬取网络小说保存到txt
- Python爬取链家成都二手房源信息 asyncio + aiohttp 异步爬虫实战
- Python matplotlib数据可视化 subplot绘制多个子图
- python pathlib模块的基本使用和总结
- Python 爬取前程无忧最新招聘数据 matplotlib数据分析与可视化