AutoTikv是一个用于对TiKV数据库进行自动调优的工具。它的设计灵感来自于SIGMOD 2017的一篇paper:Automatic Database Management System Tuning Through Large-scale Machine Learning,使用机器学习模型对数据库参数进行自动调优。
项目地址:https://github.com/pentium3/AutoTiKV
整个调优过程大致如下图:
整个过程会循环跑200个round(可以用户自定义),或者也可以定义成到结果收敛为止。
AutoTiKV支持在修改参数之后重启tikv(如果不需要也可以选择不重启)。需要调节的参数和需要查看的metric可以在controller.py里声明。
以下是一个knob的声明样板:
"rocksdb.defaultcf.write-buffer-size": # name of the knob 用点分隔不同session名和knob名
{
"changebyyml": True, # True表示通过修改tikv-ansible/conf/tikv.yml来调节
"set_func": None, # 若changebyyml==False,则在此指定修改参数的函数名(函数也定义在controller.py里,一般就用tikv-ctl命令行来调节)
"minval": 64, # if type!=enum, indicate min possible value
"maxval": 1024, # if type!=enum, indicate max possible value
"enumval": [], # if type==enum, list all valid values
"type": "int", # int / enum / real
"default": 64 # default value
},
以下是一个metric的声明样板:
"write_latency":
{
"read_func": read_write_latency, # 声明查看该指标的函数(函数也定义在controller.py里)
"lessisbetter": 1, # whether less value of this metric is better(1: yes)
"calc": "ins", # ins表示该参数的值就是benchmark之后查看的结果。inc表示该参数是incremental的,需要把benchmark之后和之前的值相减作为结果。
},
一开始的10轮(具体大小可以调节)是用随机生成的knob去benchmark,之后的都是用ML模型推荐的参数去benchmark。
AutoTikv 使用了和 OtterTune 一样的高斯过程回归(Gaussian Process Regression,以下简称 GP)来推荐新的 knob,它是基于高斯分布的一种非参数模型。高斯过程回归的好处是:
但 GP 本身其实只能估计样本的分布,为了得到最终的预测值,我们需要把它应用到贝叶斯优化(Bayesian Optimization)中。贝叶斯优化算法大致可分为两步:
采集函数(Acquisition Function)的作用是:在寻找新的推荐值的时候,平衡探索(exploration)和利用(exploitation)两个性质:
在推荐的过程中,需要平衡上述两种指标。exploitation 过多会导致结果陷入局部最优值(重复推荐目前已知的最好的点,但可能还有更好的点没被发现),而 exploration 过多又会导致搜索效率太低(一直在探索新区域,而没有对当前比较好的区域进行深入尝试)。而平衡二者的核心思想是:当数据足够多时,利用现有的数据推荐;当缺少数据时,我们在点最少的区域进行探索,探索最未知的区域能给我们最大的信息量。
贝叶斯优化的第二步就可以帮我们实现这一思想。前面提到 GP 可以帮我们估计 X 的均值 m(X) 和标准差 s(X),其中均值 m(x) 可以作为 exploitation 的表征值,而标准差 s(x) 可以作为 exploration 的表征值。这样就可以用贝叶斯优化方法来求解了。
使用置信区间上界(Upper Confidence Bound)作为采集函数。假设我们需要找 X 使 Y 值尽可能大,则 U(X) = m(X) + k*s(X),其中 k > 0 是可调的系数。我们只要找 X 使 U(X) 尽可能大即可。
在具体实现中,一开始随机生成若干个 candidate knobs,然后用上述模型计算出它们的 U(X),找出 U(X) 最大的那一个作为本次推荐的结果。
Ref:https://mp.weixin.qq.com/s/y8VIieK0LO37SjRRyPhtrw
workload
我们定义了writeheavy、longscan、shortscan、point-lookup四种workload。数据库大小都是80GB。
knobs
我们试验了如下参数:
Options
Expected Behavior
valid range/value set
how to set/view knob
write-buffer-size
point-lookup, range-scan: larger is better
[64MB, 1GB]
tidb-ansible/conf/tikv.yml
max-bytes-for-level-base
point-lookup, range-scan: larger is better
[512MB, 4GB]
tidb-ansible/conf/tikv.yml
target-file-size-base
point-lookup, range-scan: larger is better
{8M, 16M, 32M, 64M, 128M}
tidb-ansible/conf/tikv.yml
disable-auto-compactions
write-heavy: 1 is better
point-lookup, range-scan: 0 is better
{1, 0}
tidb-ansible/conf/tikv.yml
or
tikv-ctl
block-size
point-lookup: smaller the better
range-scan: larger the better
{4k,8k,16k,32k,64k}
tidb-ansible/conf/tikv.yml
bloom-filter-bits-per-key
point-lookup, range-scan: larger is better
{5,10,15,20}
tidb-ansible/conf/tikv.yml
optimize-filters-for-hits
point-lookup, range-scan: 0 is better
{1, 0}
tidb-ansible/conf/tikv.yml
这些参数的含义如下:
几个试验过但最终放弃了的参数:
metrics
我们选择了如下几个metrics作为优化指标。
其中throughput和latency通过go-ycsb的输出结果获得,store_size和compaction_cpu通过tikv-ctl获得。
Ref:
https://rdrr.io/github/richfitz/rleveldb/man/leveldb_open.html
http://mysql.taobao.org/monthly/2016/08/03/
https://www.jianshu.com/p/8e0018b6a8b6
https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
测试平台
AMD Ryzen5-2600(6C12T), 32GB RAM, 512GB NVME SSD, Ubuntu 18.04, tidb-ansible用的master版本
workload=writeheavy knobs={disable-auto-compactions, block-size} metric=write_latency
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
recordcount=80000000
operationcount=5000000
workload=core
readallfields=true
readproportion=0
updateproportion=1
scanproportion=0
insertproportion=0
requestdistribution=zipfian
ycsb workload定义
实验效果如下:
################## data ##################
------------------------------previous:------------------------------
knobs:
[[0. 4.]
[1. 3.]
[0. 0.]
[0. 3.]
[0. 1.]
[1. 4.]
[0. 1.]
[1. 0.]
[1. 4.]
[0. 4.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]]
metrics:
[[1.01428000e+04 5.04230000e+04 8.86174709e+10 1.84750000e+02]
[1.01703000e+04 5.02510000e+04 8.98934985e+10 2.50000000e+00]
[1.24102000e+04 4.10920000e+04 8.95223916e+10 2.18850000e+02]
[1.09910000e+04 4.64880000e+04 8.86518967e+10 1.89610000e+02]
[1.20731000e+04 4.21960000e+04 8.90833010e+10 1.88950000e+02]
[9.42460000e+03 5.42690000e+04 8.98143324e+10 3.32000000e+00]
[1.19275000e+04 4.28240000e+04 8.90753594e+10 1.94820000e+02]
[1.18271000e+04 4.32470000e+04 9.11159380e+10 3.08000000e+00]
[9.34830000e+03 5.47160000e+04 8.98211663e+10 3.27000000e+00]
[1.02665000e+04 4.97860000e+04 8.86331145e+10 1.87730000e+02]
[1.25193000e+04 4.08050000e+04 8.94974748e+10 2.19960000e+02]
[1.24805000e+04 4.07670000e+04 8.95419805e+10 2.20190000e+02]
[1.24086000e+04 4.11510000e+04 8.94650026e+10 2.24280000e+02]
[1.21789000e+04 4.18830000e+04 8.95860725e+10 2.18360000e+02]
[1.21835000e+04 4.19280000e+04 8.95094852e+10 2.25200000e+02]
[1.21365000e+04 4.20690000e+04 8.94701087e+10 2.18990000e+02]]
rowlabels: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
num: 16
------------------------------new:------------------------------
knobs: [[0. 0.]]
metrics: [[1.23137000e+04 4.14700000e+04 8.95614611e+10 2.17990000e+02]]
rowlabels: [1]
------------------------------TARGET:------------------------------
knob: ['disable-auto-compactions' 'block-size']
metric: write_latency
num of knobs == 2
knobs: ['disable-auto-compactions' 'block-size']
num of metrics == 4
################## data ##################
这个实验中推荐结果是启用compaction、同时block size设为4KB。
一开始还挺惊讶的(毕竟按理说写入时关闭 compaction 肯定是提升性能的)。后来分析因为TiKV 里面使用了 Percolator 进行分布式事务,写流程也涉及读操作(写冲突检测),所以关闭 compaction 也导致写入性能下降。同理更小的 block size 提高点查性能,对 TiKV 的写流程性能也有提升。
为了排除这一干扰因素,接下来用point lookup这一纯读取的workload进行了试验:
workload=pntlookup80 knobs={'bloom-filter-bits-per-key', 'optimize-filters-for-hits', 'block-size', 'disable-auto-compactions'} metric=get_latency
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
recordcount=80000000
operationcount=4000000
workload=core
readallfields=true
readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0
requestdistribution=zipfian
ycsb workload定义
实验效果如下:
------------------------------previous:------------------------------
rowlabels, finish_time, knobs, metrics
1 , 2019-08-15 20:12:21 , [2. 0. 2. 0.] , [3.66446000e+04 1.39670000e+04 8.62385543e+10 2.36200000e+01]
2 , 2019-08-15 21:01:30 , [2. 0. 2. 1.] , [2.00085000e+04 2.55740000e+04 8.65226052e+10 0.00000000e+00]
3 , 2019-08-15 22:06:48 , [3. 1. 0. 0.] , [4.18042000e+04 1.22580000e+04 8.68646096e+10 4.99000000e+01]
4 , 2019-08-15 23:12:15 , [0. 1. 1. 0.] , [3.97759000e+04 1.28700000e+04 8.64727843e+10 4.36500000e+01]
5 , 2019-08-16 00:18:39 , [3. 1. 1. 0.] , [4.0698500e+04 1.2577000e+04 8.6412687e+10 4.2540000e+01]
6 , 2019-08-16 01:08:15 , [3. 0. 4. 1.] , [1.75872000e+04 2.90890000e+04 8.63167881e+10 1.80000000e-01]
7 , 2019-08-16 02:13:59 , [2. 1. 0. 0.] , [4.14569000e+04 1.23490000e+04 8.68367156e+10 4.94200000e+01]
8 , 2019-08-16 03:20:14 , [0. 1. 3. 0.] , [3.2892000e+04 1.5563000e+04 8.6045883e+10 4.1360000e+01]
9 , 2019-08-16 04:26:29 , [2. 1. 2. 0.] , [3.56923000e+04 1.43400000e+04 8.61031652e+10 3.95600000e+01]
10 , 2019-08-16 05:32:04 , [1. 0. 0. 0.] , [4.09599000e+04 1.25000000e+04 8.69347684e+10 4.80500000e+01]
11 , 2019-08-16 06:38:25 , [3. 0. 0. 0.] , [4.11105000e+04 1.24550000e+04 8.70293207e+10 4.88900000e+01]
12 , 2019-08-16 07:44:29 , [1. 1. 0. 0.] , [4.18002000e+04 1.22470000e+04 8.68315762e+10 4.95400000e+01]
13 , 2019-08-16 08:50:32 , [2. 0. 0. 0.] , [4.21299000e+04 1.21530000e+04 8.69322719e+10 3.92500000e+01]
14 , 2019-08-16 09:56:32 , [0. 0. 0. 0.] , [3.96365000e+04 1.29120000e+04 8.68696194e+10 5.50400000e+01]
15 , 2019-08-16 11:02:19 , [2. 1. 0. 0.] , [4.13551000e+04 1.23780000e+04 8.68479242e+10 5.01600000e+01]
16 , 2019-08-16 12:08:19 , [0. 1. 0. 0.] , [3.98915000e+04 1.28310000e+04 8.68413685e+10 4.53700000e+01]
17 , 2019-08-16 13:14:13 , [2. 1. 0. 0.] , [4.1778800e+04 1.2253000e+04 8.6845963e+10 4.8780000e+01]
18 , 2019-08-16 14:05:52 , [0. 1. 0. 1.] , [1.37462000e+04 3.72160000e+04 8.74961963e+10 0.00000000e+00]
19 , 2019-08-16 15:11:48 , [2. 1. 1. 0.] , [4.03858000e+04 1.26740000e+04 8.64025255e+10 3.95100000e+01]
20 , 2019-08-16 16:18:06 , [0. 0. 2. 0.] , [3.49978000e+04 1.46240000e+04 8.61336679e+10 2.37300000e+01]
21 , 2019-08-16 17:24:02 , [2. 0. 1. 0.] , [4.13509000e+04 1.23770000e+04 8.65494483e+10 2.70600000e+01]
22 , 2019-08-16 18:29:36 , [3. 1. 0. 0.] , [4.18111000e+04 1.22440000e+04 8.68484968e+10 4.96900000e+01]
23 , 2019-08-16 19:36:16 , [1. 0. 1. 0.] , [4.03078000e+04 1.27000000e+04 8.64872698e+10 3.91300000e+01]
24 , 2019-08-16 20:41:55 , [3. 1. 0. 0.] , [4.26687000e+04 1.19980000e+04 8.68488277e+10 3.38800000e+01]
25 , 2019-08-16 21:47:55 , [2. 0. 0. 0.] , [4.19810000e+04 1.21900000e+04 8.69691844e+10 4.00500000e+01]
26 , 2019-08-16 22:54:13 , [3. 1. 0. 0.] , [4.18609000e+04 1.22290000e+04 8.68388398e+10 5.11200000e+01]
27 , 2019-08-17 00:01:29 , [3. 1. 4. 0.] , [2.9123000e+04 1.7575000e+04 8.6027491e+10 4.3860000e+01]
28 , 2019-08-17 01:07:53 , [2. 0. 0. 0.] , [4.12169000e+04 1.24210000e+04 8.69920328e+10 4.67300000e+01]
29 , 2019-08-17 02:13:38 , [3. 1. 0. 0.] , [4.18402000e+04 1.22350000e+04 8.68513516e+10 4.57200000e+01]
30 , 2019-08-17 03:19:31 , [2. 0. 0. 0.] , [4.20812000e+04 1.21640000e+04 8.69824656e+10 4.01500000e+01]
31 , 2019-08-17 04:25:12 , [3. 1. 0. 0.] , [4.16913000e+04 1.22760000e+04 8.68498155e+10 4.98100000e+01]
32 , 2019-08-17 05:31:00 , [3. 0. 0. 0.] , [4.15515000e+04 1.23180000e+04 8.70275493e+10 4.94400000e+01]
33 , 2019-08-17 06:37:15 , [3. 1. 0. 0.] , [4.16460000e+04 1.22920000e+04 8.68442154e+10 4.66100000e+01]
34 , 2019-08-17 07:43:24 , [3. 0. 0. 0.] , [4.22696000e+04 1.21100000e+04 8.70264613e+10 3.65300000e+01]
35 , 2019-08-17 08:49:24 , [3. 1. 0. 0.] , [4.18575000e+04 1.22280000e+04 8.68419002e+10 4.99000000e+01]
36 , 2019-08-17 09:55:36 , [3. 0. 0. 0.] , [4.07931000e+04 1.25500000e+04 8.70300743e+10 4.98500000e+01]
37 , 2019-08-17 11:00:54 , [3. 1. 0. 0.] , [4.19244000e+04 1.22080000e+04 8.68508093e+10 4.98500000e+01]
38 , 2019-08-17 12:06:37 , [3. 0. 0. 0.] , [4.1197800e+04 1.2425000e+04 8.7020173e+10 4.6780000e+01]
39 , 2019-08-17 13:12:35 , [3. 1. 0. 0.] , [4.19859000e+04 1.21920000e+04 8.68462752e+10 4.20200000e+01]
40 , 2019-08-17 14:18:12 , [3. 0. 0. 0.] , [4.09505000e+04 1.25020000e+04 8.70206609e+10 5.18800000e+01]
41 , 2019-08-17 15:23:32 , [3. 1. 0. 0.] , [4.19558000e+04 1.22030000e+04 8.68409963e+10 4.25600000e+01]
42 , 2019-08-17 16:29:22 , [3. 0. 0. 0.] , [4.15804000e+04 1.23100000e+04 8.70172108e+10 4.56500000e+01]
43 , 2019-08-17 17:35:13 , [3. 1. 0. 0.] , [4.16524000e+04 1.22890000e+04 8.68602952e+10 4.62100000e+01]
44 , 2019-08-17 18:41:04 , [3. 0. 0. 0.] , [4.09697000e+04 1.24950000e+04 8.70105798e+10 4.56000000e+01]
45 , 2019-08-17 19:46:55 , [3. 1. 0. 0.] , [4.16999000e+04 1.22770000e+04 8.68411373e+10 4.83400000e+01]
46 , 2019-08-17 20:52:48 , [3. 0. 0. 0.] , [4.11311000e+04 1.24450000e+04 8.70303738e+10 4.90000000e+01]
47 , 2019-08-17 21:58:48 , [3. 1. 0. 0.] , [4.23772000e+04 1.20780000e+04 8.68478265e+10 3.74500000e+01]
48 , 2019-08-17 23:04:49 , [3. 0. 0. 0.] , [4.12347000e+04 1.24120000e+04 8.70284529e+10 3.89000000e+01]
49 , 2019-08-18 00:10:42 , [3. 1. 0. 0.] , [4.29264000e+04 1.19250000e+04 8.68530475e+10 3.23300000e+01]
50 , 2019-08-18 01:16:15 , [3. 0. 0. 0.] , [4.15186000e+04 1.23290000e+04 8.70386584e+10 3.65400000e+01]
51 , 2019-08-18 02:21:36 , [3. 1. 0. 0.] , [4.26975000e+04 1.19900000e+04 8.68521299e+10 4.03900000e+01]
52 , 2019-08-18 03:27:19 , [3. 0. 0. 0.] , [4.08752000e+04 1.25230000e+04 8.70437235e+10 4.79600000e+01]
------------------------------new:------------------------------
knobs: [[3. 1. 0. 0.]]
metrics: [[4.21738000e+04 1.21390000e+04 8.68461987e+10 4.58900000e+01]]
rowlabels: [1]
timestamp: 2019-08-18 04:33:07
------------------------------TARGET:------------------------------
knob: ['bloom-filter-bits-per-key' 'optimize-filters-for-hits' 'block-size' 'disable-auto-compactions']
metric: get_latency
num of knobs == 4
knobs: ['bloom-filter-bits-per-key' 'optimize-filters-for-hits' 'block-size' 'disable-auto-compactions']
num of metrics == 4
推荐结果为:bloom-filter-bits-per-key=20,block-size=4K,不disable auto compaction。而optimize-filters-for-hits是否启用影响不大(所以会出现这一项的推荐结果一直在摇摆的情况)。
推荐的结果都挺符合预期的。关于 optimize-filter 这一项,应该是试验里面 block cache 足够大,所以 bloom filter 大小对 cache 性能影响不大;而且我们是设置 default CF 相应的选项,而对于 TiKV 来说查询 default CF 之前我们已经确定相应的 key 肯定存在,所以是否有 filter 并没有影响。之后的试验中我们会设置 writeCF 中的 optimize-filters-for-hits(defaultCF的这一项默认就是0了);然后分别设置 defaultCF 和 writeCF 中的 bloom-filter-bits-per-key,把它们作为两个 knob。
workload=pntlookup80 knobs={rocksdb.writecf.bloom-filter-bits-per-key, rocksdb.defaultcf.bloom-filter-bits-per-key, rocksdb.writecf.optimize-filters-for-hits, rocksdb.defaultcf.block-size, rocksdb.defaultcf.disable-auto-compactions} metric=get_throughput
为了能尽量测出来 bloom filter 的效果,除了上述改动之外,我们把 workload 也改了一下:把 run phase 的 recordcount 设成 load phase 的两倍大,这样强制有一半的查找对应的 key 不存在,这样应该会测出来 write CF 的 optimize-filters-for-hits 必须关闭。改完之后的 workload 如下:
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
recordcount=80000000
operationcount=5000000
workload=core
readallfields=true
readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0
requestdistribution=zipfian
Load, db大小是80GB
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
recordcount=160000000
operationcount=5000000
workload=core
readallfields=true
readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0
requestdistribution=zipfian
Run, db大小是160GB
这次的实验效果如下(发现一个很出乎意料的现象喔):
################## data ##################
------------------------------previous:------------------------------
rowlabels, finish_time, knobs, metrics
1 , 2019-08-21 20:11:14 , [3. 0. 0. 1. 0.] , [4.17141000e+04 1.22700000e+04 8.63951981e+10 3.22600000e+01]
2 , 2019-08-21 21:01:50 , [1. 2. 1. 0. 1.] , [1.84393000e+04 2.77530000e+04 8.76023557e+10 0.00000000e+00]
3 , 2019-08-21 21:52:36 , [3. 2. 1. 2. 1.] , [1.7525800e+04 2.9193000e+04 8.6489484e+10 0.0000000e+00]
4 , 2019-08-21 22:57:46 , [3. 2. 1. 4. 0.] , [3.1377400e+04 1.6311000e+04 8.6011209e+10 3.3480000e+01]
5 , 2019-08-22 00:03:06 , [2. 2. 1. 3. 0.] , [3.57383000e+04 1.43210000e+04 8.60386882e+10 3.97000000e+01]
6 , 2019-08-22 00:53:29 , [2. 3. 1. 3. 1.] , [1.78001000e+04 2.87420000e+04 8.64059038e+10 0.00000000e+00]
7 , 2019-08-22 01:58:09 , [2. 3. 1. 1. 0.] , [4.29348000e+04 1.19210000e+04 8.64341341e+10 3.46900000e+01]
8 , 2019-08-22 03:03:35 , [1. 1. 1. 2. 0.] , [3.66121000e+04 1.39810000e+04 8.61325773e+10 4.39700000e+01]
9 , 2019-08-22 04:09:05 , [2. 2. 0. 3. 0.] , [3.59254000e+04 1.42450000e+04 8.60441991e+10 4.09100000e+01]
10 , 2019-08-22 05:14:04 , [2. 2. 1. 0. 0.] , [4.3455900e+04 1.1779000e+04 8.6827261e+10 4.7180000e+01]
11 , 2019-08-22 06:19:28 , [2. 1. 0. 1. 0.] , [4.32743000e+04 1.18270000e+04 8.64087651e+10 2.76900000e+01]
12 , 2019-08-22 07:25:29 , [2. 2. 1. 0. 0.] , [4.37505000e+04 1.16980000e+04 8.68377817e+10 3.66300000e+01]
13 , 2019-08-22 08:30:47 , [2. 0. 0. 1. 0.] , [4.09163000e+04 1.25110000e+04 8.63941229e+10 4.32500000e+01]
14 , 2019-08-22 09:36:14 , [2. 2. 0. 0. 0.] , [4.36414000e+04 1.17270000e+04 8.68246281e+10 4.70600000e+01]
15 , 2019-08-22 10:41:49 , [2. 1. 1. 0. 0.] , [4.29599000e+04 1.19140000e+04 8.68228424e+10 4.91600000e+01]
16 , 2019-08-22 11:47:07 , [2. 1. 0. 0. 0.] , [4.28121000e+04 1.19560000e+04 8.68404965e+10 4.76900000e+01]
17 , 2019-08-22 12:53:03 , [2. 2. 1. 0. 0.] , [4.33225000e+04 1.18150000e+04 8.68531672e+10 4.67000000e+01]
18 , 2019-08-22 13:58:13 , [3. 1. 0. 0. 0.] , [4.42762000e+04 1.15600000e+04 8.68428438e+10 3.25600000e+01]
19 , 2019-08-22 15:03:52 , [2. 2. 1. 0. 0.] , [4.43796000e+04 1.15330000e+04 8.68332426e+10 3.37300000e+01]
20 , 2019-08-22 16:09:26 , [3. 1. 0. 0. 0.] , [4.24397000e+04 1.20590000e+04 8.68403016e+10 5.07300000e+01]
21 , 2019-08-22 17:14:34 , [2. 2. 1. 0. 0.] , [4.35737000e+04 1.17460000e+04 8.68471932e+10 4.73400000e+01]
22 , 2019-08-22 18:19:47 , [3. 1. 0. 0. 0.] , [4.28986000e+04 1.19310000e+04 8.68300705e+10 4.80600000e+01]
23 , 2019-08-22 19:25:22 , [2. 2. 1. 0. 0.] , [4.34617000e+04 1.17780000e+04 8.68395239e+10 4.80400000e+01]
24 , 2019-08-22 20:31:11 , [2. 1. 0. 0. 0.] , [4.32535000e+04 1.18330000e+04 8.68426298e+10 4.46100000e+01]
25 , 2019-08-22 21:36:29 , [3. 2. 1. 0. 0.] , [4.30494000e+04 1.18900000e+04 8.68364294e+10 4.78600000e+01]
26 , 2019-08-22 22:42:20 , [2. 1. 0. 0. 0.] , [4.27872000e+04 1.19630000e+04 8.68309331e+10 4.76100000e+01]
27 , 2019-08-22 23:47:42 , [3. 2. 0. 0. 0.] , [4.32865000e+04 1.18250000e+04 8.68361102e+10 4.83400000e+01]
28 , 2019-08-23 00:53:08 , [2. 1. 1. 0. 0.] , [4.29929000e+04 1.19080000e+04 8.68338814e+10 5.06200000e+01]
29 , 2019-08-23 01:58:37 , [2. 2. 0. 0. 0.] , [4.36637000e+04 1.17220000e+04 8.67981041e+10 4.49300000e+01]
30 , 2019-08-23 03:03:42 , [3. 1. 1. 0. 0.] , [4.30542000e+04 1.18890000e+04 8.68628124e+10 5.10200000e+01]
31 , 2019-08-23 04:09:01 , [2. 2. 0. 0. 0.] , [4.31552000e+04 1.18600000e+04 8.68568929e+10 5.26200000e+01]
32 , 2019-08-23 05:13:59 , [3. 1. 1. 0. 0.] , [4.29512000e+04 1.19180000e+04 8.68360587e+10 5.17800000e+01]
33 , 2019-08-23 06:19:15 , [2. 2. 0. 0. 0.] , [4.34998000e+04 1.17670000e+04 8.68505644e+10 4.75000000e+01]
34 , 2019-08-23 07:24:36 , [3. 1. 1. 0. 0.] , [4.29066000e+04 1.19310000e+04 8.68417278e+10 4.94600000e+01]
35 , 2019-08-23 08:30:13 , [2. 2. 0. 0. 0.] , [4.37385000e+04 1.17030000e+04 8.68307716e+10 4.26100000e+01]
36 , 2019-08-23 09:34:58 , [3. 1. 1. 0. 0.] , [4.29117000e+04 1.19300000e+04 8.68479672e+10 4.71600000e+01]
37 , 2019-08-23 10:40:21 , [2. 2. 0. 0. 0.] , [4.30777000e+04 1.18810000e+04 8.68356132e+10 4.95800000e+01]
38 , 2019-08-23 11:45:43 , [3. 1. 1. 0. 0.] , [4.36291000e+04 1.17310000e+04 8.68428416e+10 4.08700000e+01]
39 , 2019-08-23 12:51:25 , [2. 2. 0. 0. 0.] , [4.36237000e+04 1.17360000e+04 8.68353864e+10 4.00500000e+01]
40 , 2019-08-23 13:57:10 , [3. 1. 1. 0. 0.] , [4.39189000e+04 1.16570000e+04 8.68385229e+10 3.60400000e+01]
------------------------------new:------------------------------
knobs: [[2. 2. 0. 0. 0.]]
metrics: [[4.36609000e+04 1.17230000e+04 8.68364011e+10 4.77100000e+01]]
rowlabels: [1]
timestamp: 2019-08-23 15:02:11
------------------------------TARGET:------------------------------
knob: ['rocksdb.writecf.bloom-filter-bits-per-key'
'rocksdb.defaultcf.bloom-filter-bits-per-key'
'rocksdb.writecf.optimize-filters-for-hits'
'rocksdb.defaultcf.block-size'
'rocksdb.defaultcf.disable-auto-compactions']
metric: get_throughput
num of knobs == 5
knobs: ['rocksdb.writecf.bloom-filter-bits-per-key'
'rocksdb.defaultcf.bloom-filter-bits-per-key'
'rocksdb.writecf.optimize-filters-for-hits'
'rocksdb.defaultcf.block-size'
'rocksdb.defaultcf.disable-auto-compactions']
num of metrics == 4
################## data ##################
测出来发现推荐配置基本集中在以下两种:
{3,1,1,0,0}
{2,2,0,0,0}
分析了一下,感觉是因为 write CF 比较小,当 block cache size 足够大时,bloom filter 的效果可能就不很明显了。
如果仔细看一下结果,比较如下两个sample,会有个很神奇的发现:
30 , 2019-08-23 03:03:42 , [3. 1. 1. 0. 0.] , [4.30542000e+04 1.18890000e+04 8.68628124e+10 5.10200000e+01]
20 , 2019-08-22 16:09:26 , [3. 1. 0. 0. 0.] , [4.24397000e+04 1.20590000e+04 8.68403016e+10 5.07300000e+01]
它们 knob 的唯一区别就是 30 号关闭了底层 bloom filter(optimize-filters-for-hits==True),20 号启用了底层 bloom filter(optimize-filters-for-hits==False)。结果 20 号的 throughput 比 30 还低了一点,和预期完全不一样。于是我们打开 grafana 琢磨了一下,分别截取了这两个 sample 运行时段的图表:
(两种场景run时候的block-cache-size都是12.8GB,篇幅有限就不截那部分的图了)
图中粉色竖线左边是 load 阶段,右边是 run 阶段。可以看出来这俩情况下 cache hit 其实相差不大,而且 20 号还稍微低一点点。这种情况是因为 bloom filter 本身也是占空间的,如果本来 block cache size 够用,但 bloom filter 占空间又比较大,就会影响 cache hit。这个一开始确实没有预料到。其实这是一个好事情,说明 ML 模型确实可以帮我们发现一些人工想不到的东西。
接下来再试验一下short range scan。这次要优化的metric改成scan latency
workload=shortscan knobs={'bloom-filter-bits-per-key', 'optimize-filters-for-hits', 'block-size', 'disable-auto-compactions'} metric=scan_latency
# Copyright (c) 2010 Yahoo! Inc. All rights reserved.
recordcount=80000000
operationcount=200000
workload=core
readallfields=true
readproportion=0
updateproportion=0
scanproportion=1
insertproportion=0
requestdistribution=uniform
minscanlength=100
maxscanlength=100
shortscan workload定义
实验结果如下:
################## data ##################
------------------------------previous:------------------------------
rowlabels, finish_time, knobs, metrics
1 , 2019-08-24 18:29:05 , [1. 1. 1. 2. 1.] , [6.72800000e+02 7.53744000e+05 8.64420017e+10 2.40000000e-01]
2 , 2019-08-24 19:20:03 , [0. 3. 1. 3. 1.] , [6.1490000e+02 8.2401700e+05 8.6410917e+10 0.0000000e+00]
3 , 2019-08-24 20:10:54 , [3. 0. 0. 0. 1.] , [6.64200000e+02 7.62370000e+05 8.74716093e+10 0.00000000e+00]
4 , 2019-08-24 21:14:30 , [0. 1. 0. 1. 0.] , [4.05440000e+03 1.25855000e+05 8.64184132e+10 2.80100000e+01]
5 , 2019-08-24 22:18:31 , [2. 1. 0. 3. 0.] , [4.23970000e+03 1.20196000e+05 8.60256954e+10 3.74100000e+01]
6 , 2019-08-24 23:08:55 , [0. 0. 0. 1. 1.] , [7.07000000e+02 7.16597000e+05 8.68539722e+10 0.00000000e+00]
7 , 2019-08-25 00:12:24 , [2. 1. 0. 3. 0.] , [4.5478000e+03 1.1218900e+05 8.6033236e+10 2.5120000e+01]
8 , 2019-08-25 01:04:55 , [1. 3. 1. 4. 1.] , [4.96200000e+02 1.02048400e+06 8.63227618e+10 0.00000000e+00]
9 , 2019-08-25 01:56:06 , [3. 3. 1. 0. 1.] , [6.6310000e+02 7.6451400e+05 8.7654137e+10 0.0000000e+00]
10 , 2019-08-25 02:47:01 , [3. 3. 1. 2. 1.] , [6.66900000e+02 7.60646000e+05 8.65341307e+10 0.00000000e+00]
11 , 2019-08-25 03:51:18 , [1. 1. 0. 2. 0.] , [4.19610000e+03 1.21614000e+05 8.60931486e+10 2.51200000e+01]
12 , 2019-08-25 04:55:47 , [2. 0. 0. 3. 0.] , [4.3978000e+03 1.1592900e+05 8.6036505e+10 3.6290000e+01]
13 , 2019-08-25 05:59:51 , [1. 1. 0. 3. 0.] , [4.35150000e+03 1.17180000e+05 8.60368063e+10 3.63800000e+01]
14 , 2019-08-25 07:03:58 , [2. 0. 0. 2. 0.] , [3.77810000e+03 1.35018000e+05 8.60859856e+10 3.57900000e+01]
15 , 2019-08-25 08:07:51 , [1. 0. 0. 3. 0.] , [4.66590000e+03 1.09339000e+05 8.60241768e+10 2.76200000e+01]
16 , 2019-08-25 09:11:58 , [2. 1. 0. 2. 0.] , [4.09160000e+03 1.24662000e+05 8.60801061e+10 2.85700000e+01]
17 , 2019-08-25 10:16:10 , [0. 0. 0. 2. 0.] , [4.05350000e+03 1.25774000e+05 8.60802488e+10 2.62900000e+01]
18 , 2019-08-25 11:20:09 , [1. 0. 0. 3. 0.] , [4.68850000e+03 1.08877000e+05 8.59966196e+10 2.37400000e+01]
19 , 2019-08-25 12:24:28 , [0. 2. 0. 2. 0.] , [4.25840000e+03 1.19757000e+05 8.60873241e+10 2.42100000e+01]
20 , 2019-08-25 13:29:06 , [1. 0. 0. 2. 0.] , [3.77300000e+03 1.35303000e+05 8.60943509e+10 3.77800000e+01]
21 , 2019-08-25 14:33:43 , [0. 1. 0. 3. 0.] , [4.67830000e+03 1.09096000e+05 8.60373353e+10 2.58500000e+01]
22 , 2019-08-25 15:37:49 , [1. 0. 0. 3. 0.] , [4.72760000e+03 1.07929000e+05 8.60229122e+10 2.41700000e+01]
23 , 2019-08-25 16:42:13 , [0. 1. 0. 2. 0.] , [3.83190000e+03 1.33200000e+05 8.61015852e+10 3.75200000e+01]
24 , 2019-08-25 17:46:31 , [0. 0. 0. 4. 0.] , [4.80830000e+03 1.06059000e+05 8.59515848e+10 3.18500000e+01]
25 , 2019-08-25 18:50:39 , [1. 0. 0. 3. 0.] , [4.51200000e+03 1.13177000e+05 8.60177759e+10 3.22500000e+01]
26 , 2019-08-25 19:54:26 , [0. 2. 0. 4. 0.] , [4.86770000e+03 1.04802000e+05 8.59837067e+10 3.25800000e+01]
27 , 2019-08-25 20:58:22 , [1. 0. 0. 4. 0.] , [4.9614000e+03 1.0285500e+05 8.5950186e+10 3.1870000e+01]
28 , 2019-08-25 22:02:31 , [0. 0. 0. 3. 0.] , [4.37540000e+03 1.16648000e+05 8.60301063e+10 3.36500000e+01]
29 , 2019-08-25 23:06:31 , [1. 2. 0. 4. 0.] , [4.95800000e+03 1.03017000e+05 8.60147679e+10 3.06400000e+01]
30 , 2019-08-26 00:10:15 , [1. 0. 0. 4. 0.] , [5.20820000e+03 9.80490000e+04 8.59992036e+10 3.10200000e+01]
31 , 2019-08-26 01:09:36 , [1. 3. 0. 3. 0.] , [4.63750000e+03 1.10141000e+05 8.60371023e+10 3.01500000e+01]
32 , 2019-08-26 02:10:54 , [1. 1. 0. 4. 0.] , [4.89860000e+03 1.04158000e+05 8.59848252e+10 3.12700000e+01]
33 , 2019-08-26 03:12:48 , [1. 0. 0. 3. 0.] , [4.54700000e+03 1.12233000e+05 8.60197859e+10 3.15300000e+01]
34 , 2019-08-26 04:15:28 , [2. 2. 0. 4. 0.] , [4.95670000e+03 1.02892000e+05 8.60205523e+10 3.21900000e+01]
35 , 2019-08-26 05:18:03 , [1. 0. 0. 4. 0.] , [4.82490000e+03 1.05684000e+05 8.59840325e+10 3.27900000e+01]
36 , 2019-08-26 06:20:38 , [3. 1. 0. 4. 0.] , [4.98140000e+03 1.02350000e+05 8.59992772e+10 3.16700000e+01]
37 , 2019-08-26 07:23:21 , [1. 0. 0. 4. 0.] , [4.97320000e+03 1.02554000e+05 8.59940724e+10 3.17100000e+01]
38 , 2019-08-26 08:26:04 , [3. 3. 0. 3. 0.] , [4.59460000e+03 1.11100000e+05 8.60488145e+10 2.85000000e+01]
39 , 2019-08-26 09:28:30 , [2. 0. 0. 4. 0.] , [4.85840000e+03 1.05104000e+05 8.59982211e+10 3.17800000e+01]
40 , 2019-08-26 10:31:31 , [2. 3. 0. 2. 0.] , [4.13200000e+03 1.23462000e+05 8.61029034e+10 2.78400000e+01]
41 , 2019-08-26 11:35:06 , [1. 0. 0. 4. 0.] , [5.00720000e+03 1.01956000e+05 8.60064623e+10 3.17800000e+01]
42 , 2019-08-26 12:38:18 , [3. 0. 0. 4. 0.] , [4.87100000e+03 1.04930000e+05 8.59962461e+10 3.14800000e+01]
43 , 2019-08-26 13:41:29 , [1. 0. 0. 4. 0.] , [4.9381000e+03 1.0334100e+05 8.6066299e+10 3.2380000e+01]
44 , 2019-08-26 14:44:25 , [2. 1. 0. 4. 0.] , [5.01210000e+03 1.01852000e+05 8.59967147e+10 3.18600000e+01]
45 , 2019-08-26 15:47:21 , [1. 0. 0. 4. 0.] , [4.86200000e+03 1.04912000e+05 8.60001832e+10 3.25000000e+01]
------------------------------new:------------------------------
knobs: [[3. 0. 1. 4. 0.]]
metrics: [[5.02470000e+03 1.01642000e+05 8.59832276e+10 3.08800000e+01]]
rowlabels: [1]
timestamp: 2019-08-26 16:50:32
------------------------------TARGET:------------------------------
knob: ['rocksdb.writecf.bloom-filter-bits-per-key'
'rocksdb.defaultcf.bloom-filter-bits-per-key'
'rocksdb.writecf.optimize-filters-for-hits'
'rocksdb.defaultcf.block-size'
'rocksdb.defaultcf.disable-auto-compactions']
metric: scan_latency
num of knobs == 5
knobs: ['rocksdb.writecf.bloom-filter-bits-per-key'
'rocksdb.defaultcf.bloom-filter-bits-per-key'
'rocksdb.writecf.optimize-filters-for-hits'
'rocksdb.defaultcf.block-size'
'rocksdb.defaultcf.disable-auto-compactions']
num of metrics == 4
################## data ##################
由于时间有限我们先看前 45 轮的结果。这个推荐结果还没有完全收敛,但基本上满足optimize-filters-for-hits==False,block-size==32KB 或者 64KB,disable-auto-compactions==False,这三个也是对结果影响最明显的参数了。根据 Intel 的 SSD 白皮书,SSD 对 32KB 和 64KB 大小的随机读性能其实差不多。bloom filter 的位数对 scan 操作的影响也不大。这个实验结果也是符合预期了。
之后我们还会测试long scan的结果
Ref:
我们的试验场景和 OtterTune 还是有一些区别的,主要集中在以下几点:
Ref:正式开始编码之前对OtterTune的一些详细解析
由于时间有限,这里只实现了一些很基础的功能。围绕这个模型还有很多可以扩展的地方。这里记录几个扩展思路:
一个复杂的系统需要很多环节的取舍和平衡,才能使得总体运行效果达到最好。这需要对整个系统各个环节都有很深入的理解。调试 AutoTikv 的时候也发现过很多参数设置的结果并不符合预期的情况,后来仔细分析了 grafana 中的图表才发现其中的一些门路:
Ref:
贝叶斯优化
https://blog.csdn.net/Leon_winter/article/details/86604553
手机扫一扫
移动阅读更方便
你可能感兴趣的文章