剖析生产系统的I/O模式

阅读原文时间：2023年07月08日阅读：2

剖析生产系统的I/O模式

2019/02/13 vmunix

了解I/O的特点对于优化系统性能非常重要，I/O是顺序的还是随机的，是读操作还是写操作，读写的比例是多少，I/O数据块的大小，这些都是影响性能的关键因素。很多存储设备都基于特定的I/O模式做过调校，通用的测试工具跑分都相当漂亮，然而一到实际环境区别就来了，同样的应用环境下，不同的设备表现可能天差地别。我就见过不同厂商的设备，档次差不多，测试跑分高的那个在生产环境下的IO响应速度却慢了十倍。所以跑分高的设备真的不一定适合你的应用。

如果能够模拟出应用的I/O模式，那么在问题复现、乃至设备选型等方面都会有很大帮助。在此之前，了解I/O模式是第一步，这并不容易，像iostat之类的工具只能看到平均值，然而应用系统的I/O请求有可能是波浪式的，一秒之内也可以时高时低，I/O延迟可能平均值不高但是波动很大，而且I/O块大小也可以是变化的，尤其现在大数据应用的块大小可能在很大的范围内变化，与过去常见的交易型数据库有所不同，它们的块大小基本是固定的。

要剖析生产系统的I/O模式，好像没有现成的工具。但是我们可以利用blktrace自己做一个，blktrace在内核的block layer记录每一个I/O，提供了分析的素材。它记录的格式如下：

下面是一个简化版的示例，主要利用了Event “Q”和”C”，分别表示IO开始和IO完成，两者之间的耗时就相当于iostat看到的await，但blktrace可以精确到单个IO：

#!/bin/bash

if [ $# -ne 1 ]; then
echo "Usage: $0 "
exit
fi
if [ ! -b $1 ]; then
echo "could not find block device $1"
exit
fi

duration=10
echo "running blktrace for $duration seconds to collect data…"
timeout $duration blktrace -d $1 >/dev/null 2>&1

DEVNAME=`basename $1`

echo "parsing blktrace data…"
blkparse -i $DEVNAME |sort -g -k8 -k10 -k4 |awk '
BEGIN {
total_read=0;
total_write=0;
maxwait_read=0;
maxwait_write=0;
}
{
if ($6=="Q") {
queue_ts=$4;
block=$8;
nblock=$10;
rw=$7;
};
if ($6=="C" && $8==block && $10==nblock && $7==rw) {
await=$4-queue_ts;
if (rw=="R") {
if (await>maxwait_read) maxwait_read=await;
total_read++;
read_count_block[nblock]++;
if (await>0.001) read_count1++;
if (await>0.01) read_count10++;
if (await>0.02) read_count20++;
if (await>0.03) read_count30++;
}
if (rw=="W") {
if (await>maxwait_write) maxwait_write=await;
total_write++;
write_count_block[nblock]++;
if (await>0.001) write_count1++;
if (await>0.01) write_count10++;
if (await>0.02) write_count20++;
if (await>0.03) write_count30++;
}
}
} END {
printf("========\nsummary:\n========\n");
printf("total number of reads: %d\n", total_read);
printf("total number of writes: %d\n", total_write);
printf("slowest read : %.6f second\n", maxwait_read);
printf("slowest write: %.6f second\n", maxwait_write);
printf("reads\n> 1ms: %d\n>10ms: %d\n>20ms: %d\n>30ms: %d\n", read_count1, read_count10, read_count20, read_count30);
printf("writes\n> 1ms: %d\n>10ms: %d\n>20ms: %d\n>30ms: %d\n", write_count1, write_count10, write_count20, write_count30);
printf("\nblock size:%16s\n","Read Count");
for (i in read_count_block)
printf("%10d:%16d\n", i, read_count_block[i]);
printf("\nblock size:%16s\n","Write Count");
for (i in write_count_block)
printf("%10d:%16d\n", i, write_count_block[i]);
}'

#!/bin/bash

if [ $# -ne 1 ]; then

echo "Usage: $0 "

exit

if [ ! -b $1 ]; then

echo "could not find block device $1"

exit

duration=10

echo "running blktrace for $duration seconds to collect data…"

timeout $duration blktrace -d $1 >/dev/null 2>&1

DEVNAME=`basename $1`

echo "parsing blktrace data…"

blkparse -i $DEVNAME |sort -g -k8 -k10 -k4 |awk '

BEGIN {

total_read=0;

total_write=0;

maxwait_read=0;

maxwait_write=0;

}

{

if ($6=="Q") {

queue_ts=$4;

block=$8;

nblock=$10;

rw=$7;

};

if ($6=="C" && $8==block && $10==nblock && $7==rw) {

await=$4-queue_ts;

if (rw=="R") {

if (await>maxwait_read) maxwait_read=await;

total_read++;

read_count_block[nblock]++;

if (await>0.001) read_count1++;

if (await>0.01) read_count10++;

if (await>0.02) read_count20++;

if (await>0.03) read_count30++;

}

if (rw=="W") {

if (await>maxwait_write) maxwait_write=await;

total_write++;

write_count_block[nblock]++;

if (await>0.001) write_count1++;

if (await>0.01) write_count10++;

if (await>0.02) write_count20++;

if (await>0.03) write_count30++;

}

} END {

printf("========\nsummary:\n========\n");

printf("total number of reads: %d\n", total_read);

printf("total number of writes: %d\n", total_write);

printf("slowest read : %.6f second\n", maxwait_read);

printf("slowest write: %.6f second\n", maxwait_write);

printf("reads\n> 1ms: %d\n>10ms: %d\n>20ms: %d\n>30ms: %d\n", read_count1, read_count10, read_count20, read_count30);

printf("writes\n> 1ms: %d\n>10ms: %d\n>20ms: %d\n>30ms: %d\n", write_count1, write_count10, write_count20, write_count30);

printf("\nblock size:%16s\n","Read Count");

for (i in read_count_block)

printf("%10d:%16d\n", i, read_count_block[i]);

printf("\nblock size:%16s\n","Write Count");

for (i in write_count_block)

printf("%10d:%16d\n", i, write_count_block[i]);

输出示例：

========
summary:
========
total number of reads: 1081513
total number of writes: 0
slowest read : 0.032560 second
slowest write: 0.000000 second
reads
> 1ms: 18253
>10ms: 17058
>20ms: 17045
>30ms: 780
writes
> 1ms: 0
>10ms: 0
>20ms: 0
>30ms: 0

block size: Read Count
256: 93756
248: 1538
64: 98084
56: 7475
8: 101218
48: 15889
240: 1637
232: 1651
224: 1942
40: 21693
216: 1811
32: 197893
208: 1907
24: 37787
128: 97382
16: 399850