原文链接：https://blog.csdn.net/weixin_44791964/article/details/106014717

睿智的目标检测29——Keras搭建YoloV4目标检测平台

• 学习前言
• 什么是YOLOV4
• 代码下载
• YOLOV4改进的部分（不完全）
• YOLOV4结构解析
• * 1、主干特征提取网络Backbone
  • 2、特征金字塔
  • 3、YoloHead利用获得到的特征进行预测
  • 4、预测结果的解码
  • 5、在原图上进行绘制
• YOLOV4的训练
• * 1、YOLOV4的改进训练技巧
  • * a)、Mosaic数据增强
    • b)、Label Smoothing平滑
    • c)、CIOU
    • d)、学习率余弦退火衰减
  • 2、loss组成
  • * a)、计算loss所需参数
    • b)、y_pre是什么
    • c)、y_true是什么。
    • d)、loss的计算过程
• 训练自己的YOLOV4模型

学习前言

哈哈哈我最喜欢的YOLO更新了！

什么是YOLOV4

YOLOV4是YOLOV3的改进版，在YOLOV3的基础上结合了非常多的小Tricks。
尽管没有目标检测上革命性的改变，但是YOLOV4依然很好的结合了速度与精度。
根据上图也可以看出来，YOLOV4在YOLOV3的基础上，在FPS不下降的情况下，mAP达到了44，提高非常明显。

YOLOV4整体上的检测思路和YOLOV3相比相差并不大，都是使用三个特征层进行分类与回归预测。

请注意！

强烈建议在学习YOLOV4之前学习YOLOV3，因为YOLOV4确实可以看作是YOLOV3结合一系列改进的版本！

强烈建议在学习YOLOV4之前学习YOLOV3，因为YOLOV4确实可以看作是YOLOV3结合一系列改进的版本！

强烈建议在学习YOLOV4之前学习YOLOV3，因为YOLOV4确实可以看作是YOLOV3结合一系列改进的版本！

（重要的事情说三遍！）

YOLOV3可参考该博客：
https://blog.csdn.net/weixin_44791964/article/details/103276106

代码下载

https://github.com/bubbliiiing/yolov4-keras
喜欢的可以给个star噢！

哔哩哔哩视频地址：
https://www.bilibili.com/video/BV1kt4y117G6

YOLOV4改进的部分（不完全）

1、主干特征提取网络：DarkNet53 => CSPDarkNet53

2、特征金字塔：SPP，PAN

3、分类回归层：YOLOv3（未改变）

4、训练用到的小技巧：Mosaic数据增强、Label Smoothing平滑、CIOU、学习率余弦退火衰减

5、激活函数：使用Mish激活函数

以上并非全部的改进部分，还存在一些其它的改进，由于YOLOV4使用的改进实在太多了，很难完全实现与列出来，这里只列出来了一些我比较感兴趣，而且非常有效的改进。

整篇BLOG会结合YOLOV3与YOLOV4的差别进行解析

YOLOV4结构解析

1、主干特征提取网络Backbone

当输入是416x416时，特征结构如下：

当输入是608x608时，特征结构如下：

主干特征提取网络Backbone的改进点有两个：
a).主干特征提取网络：DarkNet53 => CSPDarkNet53 b).激活函数：使用Mish激活函数

如果大家对YOLOV3比较熟悉的话，应该知道Darknet53的结构，其由一系列残差网络结构构成。在Darknet53中，其存在如下resblock_body模块，其由一次下采样和多次残差结构的堆叠构成，Darknet53便是由resblock_body模块组合而成。

def resblock_body(x, num_filters, num_blocks):
    x = ZeroPadding2D(((1,0),(1,0)))(x)
    x = DarknetConv2D_BN_Leaky(num_filters, (3,3), strides=(2,2))(x)
    for i in range(num_blocks):
        y = DarknetConv2D_BN_Leaky(num_filters//2, (1,1))(x)
        y = DarknetConv2D_BN_Leaky(num_filters, (3,3))(y)
        x = Add()([x,y])
    return x

而在YOLOV4中，其对该部分进行了一定的修改。
1、其一是将DarknetConv2D的激活函数由LeakyReLU修改成了Mish，卷积块由DarknetConv2D_BN_Leaky变成了DarknetConv2D_BN_Mish。
Mish函数的公式与图像如下：
M i s h = x × t a n h ( l n ( 1 + e x ) ) M i s h = x × t a n h ( l n ( 1 + e x ) ) M i s h = x × t a n h ( l n ( 1 + e x ) ) Mish=x×tanh(ln(1+ex))Mish=x×tanh(ln(1+ex)) Mish=x \times tanh(ln(1+e^x)) Mish=x×tanh(ln(1+ex))Mish=x×tanh(ln(1+ex))Mish=x×tanh(ln(1+ex))Mish=x×tanh(ln(1+ex))

2、其二是将resblock_body的结构进行修改，使用了CSPnet结构。此时YOLOV4当中的Darknet53被修改成了CSPDarknet53。

CSPnet结构并不算复杂，就是将原来的残差块的堆叠进行了一个拆分，拆成左右两部分：
主干部分继续进行原来的残差块的堆叠；
另一部分则像一个残差边一样，经过少量处理直接连接到最后。
因此可以认为CSP中存在一个大的残差边。

#---------------------------------------------------#
#   CSPdarknet的结构块
#   存在一个大残差边
#   这个大残差边绕过了很多的残差结构
#---------------------------------------------------#
def resblock_body(x, num_filters, num_blocks, all_narrow=True):
    # 进行长和宽的压缩
    preconv1 = ZeroPadding2D(((1,0),(1,0)))(x)
    preconv1 = DarknetConv2D_BN_Mish(num_filters, (3,3), strides=(2,2))(preconv1)

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27

全部实现代码为：

from functools import wraps
from keras import backend as K
from keras.layers import Conv2D, Add, ZeroPadding2D, UpSampling2D, Concatenate, MaxPooling2D, Layer
from keras.layers.advanced_activations import LeakyReLU
from keras.layers.normalization import BatchNormalization
from keras.regularizers import l2
from utils.utils import compose

class Mish(Layer):
def init(self, kwargs):
super(Mish, self).init(kwargs)
self.supports_masking = True

<span class="token keyword">def</span> <span class="token function">call</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> inputs<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">return</span> inputs <span class="token operator">*</span> K<span class="token punctuation">.</span>tanh<span class="token punctuation">(</span>K<span class="token punctuation">.</span>softplus<span class="token punctuation">(</span>inputs<span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token keyword">def</span> <span class="token function">get_config</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    config <span class="token operator">=</span> <span class="token builtin">super</span><span class="token punctuation">(</span>Mish<span class="token punctuation">,</span> self<span class="token punctuation">)</span><span class="token punctuation">.</span>get_config<span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token keyword">return</span> config

<span class="token keyword">def</span> <span class="token function">compute_output_shape</span><span class="token punctuation">(</span>self<span class="token punctuation">,</span> input_shape<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">return</span> input_shape

#--------------------------------------------------#
# 单次卷积
#--------------------------------------------------#
@wraps(Conv2D)
def DarknetConv2D(args, **kwargs):
darknet_conv_kwargs = {‘kernel_regularizer’: l2(5e-4)}
darknet_conv_kwargs[‘padding’] = ‘valid’ if kwargs.get(‘strides’)==(2,2) else ‘same’
darknet_conv_kwargs.update(kwargs)
return Conv2D(args, **darknet_conv_kwargs)

#---------------------------------------------------#
# 卷积块
# DarknetConv2D + BatchNormalization + Mish
#---------------------------------------------------#
def DarknetConv2D_BN_Mish(args, **kwargs):
no_bias_kwargs = {‘use_bias’: False}
no_bias_kwargs.update(kwargs)
return compose(
DarknetConv2D(args, **no_bias_kwargs),
BatchNormalization(),
Mish())

#---------------------------------------------------#
# CSPdarknet的结构块
# 存在一个大残差边
# 这个大残差边绕过了很多的残差结构
#---------------------------------------------------#
def resblock_body(x, num_filters, num_blocks, all_narrow=True):
# 进行长和宽的压缩
preconv1 = ZeroPadding2D(((1,0),(1,0)))(x)
preconv1 = DarknetConv2D_BN_Mish(num_filters, (3,3), strides=(2,2))(preconv1)

<span class="token comment"># 生成一个大的残差边 </span>
shortconv <span class="token operator">=</span> DarknetConv2D_BN_Mish<span class="token punctuation">(</span>num_filters<span class="token operator">//</span><span class="token number">2</span> <span class="token keyword">if</span> all_narrow <span class="token keyword">else</span> num_filters<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">(</span>preconv1<span class="token punctuation">)</span>

<span class="token comment"># 主干部分的卷积</span>
mainconv <span class="token operator">=</span> DarknetConv2D_BN_Mish<span class="token punctuation">(</span>num_filters<span class="token operator">//</span><span class="token number">2</span> <span class="token keyword">if</span> all_narrow <span class="token keyword">else</span> num_filters<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">(</span>preconv1<span class="token punctuation">)</span>
<span class="token comment"># 1x1卷积对通道数进行整合-&gt;3x3卷积提取特征，使用残差结构</span>
<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>num_blocks<span class="token punctuation">)</span><span class="token punctuation">:</span>
    y <span class="token operator">=</span> compose<span class="token punctuation">(</span>
            DarknetConv2D_BN_Mish<span class="token punctuation">(</span>num_filters<span class="token operator">//</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span>
            DarknetConv2D_BN_Mish<span class="token punctuation">(</span>num_filters<span class="token operator">//</span><span class="token number">2</span> <span class="token keyword">if</span> all_narrow <span class="token keyword">else</span> num_filters<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">(</span>mainconv<span class="token punctuation">)</span>
    mainconv <span class="token operator">=</span> Add<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">[</span>mainconv<span class="token punctuation">,</span>y<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token comment"># 1x1卷积后和残差边堆叠</span>
postconv <span class="token operator">=</span> DarknetConv2D_BN_Mish<span class="token punctuation">(</span>num_filters<span class="token operator">//</span><span class="token number">2</span> <span class="token keyword">if</span> all_narrow <span class="token keyword">else</span> num_filters<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">(</span>mainconv<span class="token punctuation">)</span>
route <span class="token operator">=</span> Concatenate<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token punctuation">[</span>postconv<span class="token punctuation">,</span> shortconv<span class="token punctuation">]</span><span class="token punctuation">)</span>

<span class="token comment"># 最后对通道数进行整合</span>
<span class="token keyword">return</span> DarknetConv2D_BN_Mish<span class="token punctuation">(</span>num_filters<span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">(</span>route<span class="token punctuation">)</span>

#---------------------------------------------------#
# CSPdarknet53 的主体部分
#---------------------------------------------------#
def darknet_body(x):
x = DarknetConv2D_BN_Mish(32, (3,3))(x)
x = resblock_body(x, 64, 1, False)
x = resblock_body(x, 128, 2)
x = resblock_body(x, 256, 8)
feat1 = x
x = resblock_body(x, 512, 8)
feat2 = x
x = resblock_body(x, 1024, 4)
feat3 = x
return feat1,feat2,feat3

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72
• 73
• 74
• 75
• 76
• 77
• 78
• 79
• 80
• 81
• 82
• 83
• 84
• 85
• 86
• 87

2、特征金字塔

当输入是416x416时，特征结构如下：

当输入是608x608时，特征结构如下：

在特征金字塔部分，YOLOV4结合了两种改进:
a).使用了SPP结构。 b).使用了PANet结构。
如上图所示，除去CSPDarknet53和Yolo Head的结构外，都是特征金字塔的结构。
1、SPP结构参杂在对CSPdarknet53的最后一个特征层的卷积里，在对CSPdarknet53的最后一个特征层进行三次DarknetConv2D_BN_Leaky卷积后，分别利用四个不同尺度的最大池化进行处理，最大池化的池化核大小分别为13x13、9x9、5x5、1x1（1x1即无处理）

# 使用了SPP结构，即不同尺度的最大池化后堆叠。
maxpool1 = MaxPooling2D(pool_size=(13,13), strides=(1,1), padding='same')(P5)
maxpool2 = MaxPooling2D(pool_size=(9,9), strides=(1,1), padding='same')(P5)
maxpool3 = MaxPooling2D(pool_size=(5,5), strides=(1,1), padding='same')(P5)
P5 = Concatenate()([maxpool1, maxpool2, maxpool3, P5])

其可以它能够极大地增加感受野，分离出最显著的上下文特征。

2、PANet是2018的一种实例分割算法，其具体结构由反复提升特征的意思。

上图为原始的PANet的结构，可以看出来其具有一个非常重要的特点就是特征的反复提取。
在（a）里面是传统的特征金字塔结构，在完成特征金字塔从下到上的特征提取后，还需要实现（b）中从上到下的特征提取。

而在YOLOV4当中，其主要是在三个有效特征层上使用了PANet结构。

实现代码如下：

#---------------------------------------------------#
#   特征层->最后的输出
#---------------------------------------------------#
def yolo_body(inputs, num_anchors, num_classes):
    # 生成darknet53的主干模型
    feat1,feat2,feat3 = darknet_body(inputs)

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55

3、YoloHead利用获得到的特征进行预测

当输入是416x416时，特征结构如下：

当输入是608x608时，特征结构如下：

1、在特征利用部分，YoloV4提取多特征层进行目标检测，一共提取三个特征层，分别位于中间层，中下层，底层，三个特征层的shape分别为(76,76,256)、(38,38,512)、(19,19,1024)。

2、输出层的shape分别为(19,19,75)，(38,38,75)，(76,76,75)，最后一个维度为75是因为该图是基于voc数据集的，它的类为20种，YoloV4只有针对每一个特征层存在3个先验框，所以最后维度为3x25； 如果使用的是coco训练集，类则为80种，最后的维度应该为255 = 3x85，三个特征层的shape为(19,19,255)，(38,38,255)，(76,76,255)

实现代码如下：

#---------------------------------------------------#
#   特征层->最后的输出
#---------------------------------------------------#
def yolo_body(inputs, num_anchors, num_classes):
# 省略了一部分，只看最后的head部分
    P3_output = DarknetConv2D_BN_Leaky(256, (3,3))(P3)
    P3_output = DarknetConv2D(num_anchors*(num_classes+5), (1,1))(P3_output)

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13

4、预测结果的解码

由第二步我们可以获得三个特征层的预测结果，shape分别为(N,19,19,255)，(N,38,38,255)，(N,76,76,255)的数据，对应每个图分为19x19、38x38、76x76的网格上3个预测框的位置。

但是这个预测结果并不对应着最终的预测框在图片上的位置，还需要解码才可以完成。

此处要讲一下yolo3的预测原理，yolo3的3个特征层分别将整幅图分为19x19、38x38、76x76的网格，每个网络点负责一个区域的检测。

我们知道特征层的预测结果对应着三个预测框的位置，我们先将其reshape一下，其结果为(N,19,19,3,85)，(N,38,38,3,85)，(N,76,76,3,85)。

最后一个维度中的85包含了4+1+80，分别代表x_offset、y_offset、h和w、置信度、分类结果。

yolo3的解码过程就是将每个网格点加上它对应的x_offset和y_offset，加完后的结果就是预测框的中心，然后再利用 先验框和h、w结合 计算出预测框的长和宽。这样就能得到整个预测框的位置了。

当然得到最终的预测结构后还要进行得分排序与非极大抑制筛选
这一部分基本上是所有目标检测通用的部分。不过该项目的处理方式与其它项目不同。其对于每一个类进行判别。
1、取出每一类得分大于self.obj_threshold的框和得分。 2、利用框的位置和得分进行非极大抑制。

实现代码如下，当调用yolo_eval时，就会对每个特征层进行解码：

#---------------------------------------------------#
#   将预测值的每个特征层调成真实值
#---------------------------------------------------#
def yolo_head(feats, anchors, num_classes, input_shape, calc_loss=False):
    num_anchors = len(anchors)
    # [1, 1, 1, num_anchors, 2]
    anchors_tensor = K.reshape(K.constant(anchors), [1, 1, 1, num_anchors, 2])

#---------------------------------------------------#
# 对box进行调整，使其符合真实图片的样子
#---------------------------------------------------#
def yolo_correct_boxes(box_xy, box_wh, input_shape, image_shape):
box_yx = box_xy[…, ::-1]
box_hw = box_wh[…, ::-1]

input_shape <span class="token operator">=</span> K<span class="token punctuation">.</span>cast<span class="token punctuation">(</span>input_shape<span class="token punctuation">,</span> K<span class="token punctuation">.</span>dtype<span class="token punctuation">(</span>box_yx<span class="token punctuation">)</span><span class="token punctuation">)</span>
image_shape <span class="token operator">=</span> K<span class="token punctuation">.</span>cast<span class="token punctuation">(</span>image_shape<span class="token punctuation">,</span> K<span class="token punctuation">.</span>dtype<span class="token punctuation">(</span>box_yx<span class="token punctuation">)</span><span class="token punctuation">)</span>

new_shape <span class="token operator">=</span> K<span class="token punctuation">.</span><span class="token builtin">round</span><span class="token punctuation">(</span>image_shape <span class="token operator">*</span> K<span class="token punctuation">.</span><span class="token builtin">min</span><span class="token punctuation">(</span>input_shape<span class="token operator">/</span>image_shape<span class="token punctuation">)</span><span class="token punctuation">)</span>
offset <span class="token operator">=</span> <span class="token punctuation">(</span>input_shape<span class="token operator">-</span>new_shape<span class="token punctuation">)</span><span class="token operator">/</span><span class="token number">2</span><span class="token punctuation">.</span><span class="token operator">/</span>input_shape
scale <span class="token operator">=</span> input_shape<span class="token operator">/</span>new_shape

box_yx <span class="token operator">=</span> <span class="token punctuation">(</span>box_yx <span class="token operator">-</span> offset<span class="token punctuation">)</span> <span class="token operator">*</span> scale
box_hw <span class="token operator">*=</span> scale

box_mins <span class="token operator">=</span> box_yx <span class="token operator">-</span> <span class="token punctuation">(</span>box_hw <span class="token operator">/</span> <span class="token number">2</span><span class="token punctuation">.</span><span class="token punctuation">)</span>
box_maxes <span class="token operator">=</span> box_yx <span class="token operator">+</span> <span class="token punctuation">(</span>box_hw <span class="token operator">/</span> <span class="token number">2</span><span class="token punctuation">.</span><span class="token punctuation">)</span>
boxes <span class="token operator">=</span>  K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span><span class="token punctuation">[</span>
    box_mins<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">:</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span>  <span class="token comment"># y_min</span>
    box_mins<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">:</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">,</span>  <span class="token comment"># x_min</span>
    box_maxes<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">:</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span>  <span class="token comment"># y_max</span>
    box_maxes<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">:</span><span class="token number">2</span><span class="token punctuation">]</span>  <span class="token comment"># x_max</span>
<span class="token punctuation">]</span><span class="token punctuation">)</span>

boxes <span class="token operator">*=</span> K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span><span class="token punctuation">[</span>image_shape<span class="token punctuation">,</span> image_shape<span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> boxes

#---------------------------------------------------#
# 获取每个box和它的得分
#---------------------------------------------------#
def yolo_boxes_and_scores(feats, anchors, num_classes, input_shape, image_shape):
# 将预测值调成真实值
# box_xy对应框的中心点
# box_wh对应框的宽和高
# -1,19,19,3,2; -1,19,19,3,2; -1,19,19,3,1; -1,19,19,3,80
box_xy, box_wh, box_confidence, box_class_probs = yolo_head(feats, anchors, num_classes, input_shape)
# 将box_xy、和box_wh调节成y_min,y_max,xmin,xmax
boxes = yolo_correct_boxes(box_xy, box_wh, input_shape, image_shape)
# 获得得分和box
boxes = K.reshape(boxes, [-1, 4])
box_scores = box_confidence * box_class_probs
box_scores = K.reshape(box_scores, [-1, num_classes])
return boxes, box_scores

#---------------------------------------------------#
# 图片预测
#---------------------------------------------------#
def yolo_eval(yolo_outputs,
anchors,
num_classes,
image_shape,
max_boxes=20,
score_threshold=.6,
iou_threshold=.5):
# 获得特征层的数量
num_layers = len(yolo_outputs)
# 特征层1对应的anchor是678
# 特征层2对应的anchor是345
# 特征层3对应的anchor是012
anchor_mask = [[6,7,8], [3,4,5], [0,1,2]]

input_shape <span class="token operator">=</span> K<span class="token punctuation">.</span>shape<span class="token punctuation">(</span>yolo_outputs<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token number">3</span><span class="token punctuation">]</span> <span class="token operator">*</span> <span class="token number">32</span>
boxes <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
box_scores <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token comment"># 对每个特征层进行处理</span>
<span class="token keyword">for</span> l <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>num_layers<span class="token punctuation">)</span><span class="token punctuation">:</span>
    _boxes<span class="token punctuation">,</span> _box_scores <span class="token operator">=</span> yolo_boxes_and_scores<span class="token punctuation">(</span>yolo_outputs<span class="token punctuation">[</span>l<span class="token punctuation">]</span><span class="token punctuation">,</span> anchors<span class="token punctuation">[</span>anchor_mask<span class="token punctuation">[</span>l<span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">,</span> num_classes<span class="token punctuation">,</span> input_shape<span class="token punctuation">,</span> image_shape<span class="token punctuation">)</span>
    boxes<span class="token punctuation">.</span>append<span class="token punctuation">(</span>_boxes<span class="token punctuation">)</span>
    box_scores<span class="token punctuation">.</span>append<span class="token punctuation">(</span>_box_scores<span class="token punctuation">)</span>
<span class="token comment"># 将每个特征层的结果进行堆叠</span>
boxes <span class="token operator">=</span> K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span>boxes<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>
box_scores <span class="token operator">=</span> K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span>box_scores<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>

mask <span class="token operator">=</span> box_scores <span class="token operator">&gt;=</span> score_threshold
max_boxes_tensor <span class="token operator">=</span> K<span class="token punctuation">.</span>constant<span class="token punctuation">(</span>max_boxes<span class="token punctuation">,</span> dtype<span class="token operator">=</span><span class="token string">'int32'</span><span class="token punctuation">)</span>
boxes_ <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
scores_ <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
classes_ <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> c <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>num_classes<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token comment"># 取出所有box_scores &gt;= score_threshold的框，和成绩</span>
    class_boxes <span class="token operator">=</span> tf<span class="token punctuation">.</span>boolean_mask<span class="token punctuation">(</span>boxes<span class="token punctuation">,</span> mask<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> c<span class="token punctuation">]</span><span class="token punctuation">)</span>
    class_box_scores <span class="token operator">=</span> tf<span class="token punctuation">.</span>boolean_mask<span class="token punctuation">(</span>box_scores<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> c<span class="token punctuation">]</span><span class="token punctuation">,</span> mask<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> c<span class="token punctuation">]</span><span class="token punctuation">)</span>

    <span class="token comment"># 非极大抑制，去掉box重合程度高的那一些</span>
    nms_index <span class="token operator">=</span> tf<span class="token punctuation">.</span>image<span class="token punctuation">.</span>non_max_suppression<span class="token punctuation">(</span>
        class_boxes<span class="token punctuation">,</span> class_box_scores<span class="token punctuation">,</span> max_boxes_tensor<span class="token punctuation">,</span> iou_threshold<span class="token operator">=</span>iou_threshold<span class="token punctuation">)</span>

    <span class="token comment"># 获取非极大抑制后的结果</span>
    <span class="token comment"># 下列三个分别是</span>
    <span class="token comment"># 框的位置，得分与种类</span>
    class_boxes <span class="token operator">=</span> K<span class="token punctuation">.</span>gather<span class="token punctuation">(</span>class_boxes<span class="token punctuation">,</span> nms_index<span class="token punctuation">)</span>
    class_box_scores <span class="token operator">=</span> K<span class="token punctuation">.</span>gather<span class="token punctuation">(</span>class_box_scores<span class="token punctuation">,</span> nms_index<span class="token punctuation">)</span>
    classes <span class="token operator">=</span> K<span class="token punctuation">.</span>ones_like<span class="token punctuation">(</span>class_box_scores<span class="token punctuation">,</span> <span class="token string">'int32'</span><span class="token punctuation">)</span> <span class="token operator">*</span> c
    boxes_<span class="token punctuation">.</span>append<span class="token punctuation">(</span>class_boxes<span class="token punctuation">)</span>
    scores_<span class="token punctuation">.</span>append<span class="token punctuation">(</span>class_box_scores<span class="token punctuation">)</span>
    classes_<span class="token punctuation">.</span>append<span class="token punctuation">(</span>classes<span class="token punctuation">)</span>
boxes_ <span class="token operator">=</span> K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span>boxes_<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>
scores_ <span class="token operator">=</span> K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span>scores_<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>
classes_ <span class="token operator">=</span> K<span class="token punctuation">.</span>concatenate<span class="token punctuation">(</span>classes_<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span>

<span class="token keyword">return</span> boxes_<span class="token punctuation">,</span> scores_<span class="token punctuation">,</span> classes_

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72
• 73
• 74
• 75
• 76
• 77
• 78
• 79
• 80
• 81
• 82
• 83
• 84
• 85
• 86
• 87
• 88
• 89
• 90
• 91
• 92
• 93
• 94
• 95
• 96
• 97
• 98
• 99
• 100
• 101
• 102
• 103
• 104
• 105
• 106
• 107
• 108
• 109
• 110
• 111
• 112
• 113
• 114
• 115
• 116
• 117
• 118
• 119
• 120
• 121
• 122
• 123
• 124
• 125
• 126
• 127
• 128
• 129
• 130
• 131
• 132
• 133
• 134
• 135
• 136
• 137

5、在原图上进行绘制

通过第四步，我们可以获得预测框在原图上的位置，而且这些预测框都是经过筛选的。这些筛选后的框可以直接绘制在图片上，就可以获得结果了。

YOLOV4的训练

1、YOLOV4的改进训练技巧

a)、Mosaic数据增强

Yolov4的mosaic数据增强参考了CutMix数据增强方式，理论上具有一定的相似性！
CutMix数据增强方式利用两张图片进行拼接。

但是mosaic利用了四张图片，根据论文所说其拥有一个巨大的优点是丰富检测物体的背景！且在BN计算的时候一下子会计算四张图片的数据！
就像下图这样：

实现思路如下：
1、每次读取四张图片。

2、分别对四张图片进行翻转、缩放、色域变化等，并且按照四个方向位置摆好。




3、进行图片的组合和框的组合

def rand(a=0, b=1):
    return np.random.rand()*(b-a) + a

def merge_bboxes(bboxes, cutx, cuty):

merge_bbox <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>bboxes<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">for</span> box <span class="token keyword">in</span> bboxes<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">:</span>
        tmp_box <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
        x1<span class="token punctuation">,</span>y1<span class="token punctuation">,</span>x2<span class="token punctuation">,</span>y2 <span class="token operator">=</span> box<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> box<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">,</span> box<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">,</span> box<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span>

        <span class="token keyword">if</span> i <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">:</span>
            <span class="token keyword">if</span> y1 <span class="token operator">&gt;</span> cuty <span class="token operator">or</span> x1 <span class="token operator">&gt;</span> cutx<span class="token punctuation">:</span>
                <span class="token keyword">continue</span>
            <span class="token keyword">if</span> y2 <span class="token operator">&gt;=</span> cuty <span class="token operator">and</span> y1 <span class="token operator">&lt;=</span> cuty<span class="token punctuation">:</span>
                y2 <span class="token operator">=</span> cuty
                <span class="token keyword">if</span> y2<span class="token operator">-</span>y1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>
            <span class="token keyword">if</span> x2 <span class="token operator">&gt;=</span> cutx <span class="token operator">and</span> x1 <span class="token operator">&lt;=</span> cutx<span class="token punctuation">:</span>
                x2 <span class="token operator">=</span> cutx
                <span class="token keyword">if</span> x2<span class="token operator">-</span>x1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

        <span class="token keyword">if</span> i <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">:</span>
            <span class="token keyword">if</span> y2 <span class="token operator">&lt;</span> cuty <span class="token operator">or</span> x1 <span class="token operator">&gt;</span> cutx<span class="token punctuation">:</span>
                <span class="token keyword">continue</span>

            <span class="token keyword">if</span> y2 <span class="token operator">&gt;=</span> cuty <span class="token operator">and</span> y1 <span class="token operator">&lt;=</span> cuty<span class="token punctuation">:</span>
                y1 <span class="token operator">=</span> cuty
                <span class="token keyword">if</span> y2<span class="token operator">-</span>y1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

            <span class="token keyword">if</span> x2 <span class="token operator">&gt;=</span> cutx <span class="token operator">and</span> x1 <span class="token operator">&lt;=</span> cutx<span class="token punctuation">:</span>
                x2 <span class="token operator">=</span> cutx
                <span class="token keyword">if</span> x2<span class="token operator">-</span>x1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

        <span class="token keyword">if</span> i <span class="token operator">==</span> <span class="token number">2</span><span class="token punctuation">:</span>
            <span class="token keyword">if</span> y2 <span class="token operator">&lt;</span> cuty <span class="token operator">or</span> x2 <span class="token operator">&lt;</span> cutx<span class="token punctuation">:</span>
                <span class="token keyword">continue</span>

            <span class="token keyword">if</span> y2 <span class="token operator">&gt;=</span> cuty <span class="token operator">and</span> y1 <span class="token operator">&lt;=</span> cuty<span class="token punctuation">:</span>
                y1 <span class="token operator">=</span> cuty
                <span class="token keyword">if</span> y2<span class="token operator">-</span>y1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

            <span class="token keyword">if</span> x2 <span class="token operator">&gt;=</span> cutx <span class="token operator">and</span> x1 <span class="token operator">&lt;=</span> cutx<span class="token punctuation">:</span>
                x1 <span class="token operator">=</span> cutx
                <span class="token keyword">if</span> x2<span class="token operator">-</span>x1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

        <span class="token keyword">if</span> i <span class="token operator">==</span> <span class="token number">3</span><span class="token punctuation">:</span>
            <span class="token keyword">if</span> y1 <span class="token operator">&gt;</span> cuty <span class="token operator">or</span> x2 <span class="token operator">&lt;</span> cutx<span class="token punctuation">:</span>
                <span class="token keyword">continue</span>

            <span class="token keyword">if</span> y2 <span class="token operator">&gt;=</span> cuty <span class="token operator">and</span> y1 <span class="token operator">&lt;=</span> cuty<span class="token punctuation">:</span>
                y2 <span class="token operator">=</span> cuty
                <span class="token keyword">if</span> y2<span class="token operator">-</span>y1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

            <span class="token keyword">if</span> x2 <span class="token operator">&gt;=</span> cutx <span class="token operator">and</span> x1 <span class="token operator">&lt;=</span> cutx<span class="token punctuation">:</span>
                x1 <span class="token operator">=</span> cutx
                <span class="token keyword">if</span> x2<span class="token operator">-</span>x1 <span class="token operator">&lt;</span> <span class="token number">5</span><span class="token punctuation">:</span>
                    <span class="token keyword">continue</span>

        tmp_box<span class="token punctuation">.</span>append<span class="token punctuation">(</span>x1<span class="token punctuation">)</span>
        tmp_box<span class="token punctuation">.</span>append<span class="token punctuation">(</span>y1<span class="token punctuation">)</span>
        tmp_box<span class="token punctuation">.</span>append<span class="token punctuation">(</span>x2<span class="token punctuation">)</span>
        tmp_box<span class="token punctuation">.</span>append<span class="token punctuation">(</span>y2<span class="token punctuation">)</span>
        tmp_box<span class="token punctuation">.</span>append<span class="token punctuation">(</span>box<span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
        merge_bbox<span class="token punctuation">.</span>append<span class="token punctuation">(</span>tmp_box<span class="token punctuation">)</span>
<span class="token keyword">return</span> merge_bbox

def get_random_data(annotation_line, input_shape, random=True, hue=.1, sat=1.5, val=1.5, proc_img=True):
‘’‘random preprocessing for real-time data augmentation’’’
h, w = input_shape
min_offset_x = 0.4
min_offset_y = 0.4
scale_low = 1-min(min_offset_x,min_offset_y)
scale_high = scale_low+0.2

image_datas <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span> 
box_datas <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
index <span class="token operator">=</span> <span class="token number">0</span>

place_x <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span><span class="token number">0</span><span class="token punctuation">,</span><span class="token builtin">int</span><span class="token punctuation">(</span>w<span class="token operator">*</span>min_offset_x<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token builtin">int</span><span class="token punctuation">(</span>w<span class="token operator">*</span>min_offset_x<span class="token punctuation">)</span><span class="token punctuation">]</span>
place_y <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span><span class="token builtin">int</span><span class="token punctuation">(</span>h<span class="token operator">*</span>min_offset_y<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token builtin">int</span><span class="token punctuation">(</span>w<span class="token operator">*</span>min_offset_y<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> line <span class="token keyword">in</span> annotation_line<span class="token punctuation">:</span>
    <span class="token comment"># 每一行进行分割</span>
    line_content <span class="token operator">=</span> line<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token comment"># 打开图片</span>
    image <span class="token operator">=</span> Image<span class="token punctuation">.</span><span class="token builtin">open</span><span class="token punctuation">(</span>line_content<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
    image <span class="token operator">=</span> image<span class="token punctuation">.</span>convert<span class="token punctuation">(</span><span class="token string">"RGB"</span><span class="token punctuation">)</span> 
    <span class="token comment"># 图片的大小</span>
    iw<span class="token punctuation">,</span> ih <span class="token operator">=</span> image<span class="token punctuation">.</span>size
    <span class="token comment"># 保存框的位置</span>
    box <span class="token operator">=</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span><span class="token punctuation">[</span>np<span class="token punctuation">.</span>array<span class="token punctuation">(</span><span class="token builtin">list</span><span class="token punctuation">(</span><span class="token builtin">map</span><span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">,</span>box<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">','</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token keyword">for</span> box <span class="token keyword">in</span> line_content<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span>

    <span class="token comment"># image.save(str(index)+".jpg")</span>
    <span class="token comment"># 是否翻转图片</span>
    flip <span class="token operator">=</span> rand<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">&lt;</span><span class="token number">.5</span>
    <span class="token keyword">if</span> flip <span class="token operator">and</span> <span class="token builtin">len</span><span class="token punctuation">(</span>box<span class="token punctuation">)</span><span class="token operator">&gt;</span><span class="token number">0</span><span class="token punctuation">:</span>
        image <span class="token operator">=</span> image<span class="token punctuation">.</span>transpose<span class="token punctuation">(</span>Image<span class="token punctuation">.</span>FLIP_LEFT_RIGHT<span class="token punctuation">)</span>
        box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token operator">=</span> iw <span class="token operator">-</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">,</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">]</span>

    <span class="token comment"># 对输入进来的图片进行缩放</span>
    new_ar <span class="token operator">=</span> w<span class="token operator">/</span>h
    scale <span class="token operator">=</span> rand<span class="token punctuation">(</span>scale_low<span class="token punctuation">,</span> scale_high<span class="token punctuation">)</span>
    <span class="token keyword">if</span> new_ar <span class="token operator">&lt;</span> <span class="token number">1</span><span class="token punctuation">:</span>
        nh <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>scale<span class="token operator">*</span>h<span class="token punctuation">)</span>
        nw <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>nh<span class="token operator">*</span>new_ar<span class="token punctuation">)</span>
    <span class="token keyword">else</span><span class="token punctuation">:</span>
        nw <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>scale<span class="token operator">*</span>w<span class="token punctuation">)</span>
        nh <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span>nw<span class="token operator">/</span>new_ar<span class="token punctuation">)</span>
    image <span class="token operator">=</span> image<span class="token punctuation">.</span>resize<span class="token punctuation">(</span><span class="token punctuation">(</span>nw<span class="token punctuation">,</span>nh<span class="token punctuation">)</span><span class="token punctuation">,</span> Image<span class="token punctuation">.</span>BICUBIC<span class="token punctuation">)</span>

    <span class="token comment"># 进行色域变换</span>
    hue <span class="token operator">=</span> rand<span class="token punctuation">(</span><span class="token operator">-</span>hue<span class="token punctuation">,</span> hue<span class="token punctuation">)</span>
    sat <span class="token operator">=</span> rand<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> sat<span class="token punctuation">)</span> <span class="token keyword">if</span> rand<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">&lt;</span><span class="token number">.5</span> <span class="token keyword">else</span> <span class="token number">1</span><span class="token operator">/</span>rand<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> sat<span class="token punctuation">)</span>
    val <span class="token operator">=</span> rand<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> val<span class="token punctuation">)</span> <span class="token keyword">if</span> rand<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token operator">&lt;</span><span class="token number">.5</span> <span class="token keyword">else</span> <span class="token number">1</span><span class="token operator">/</span>rand<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> val<span class="token punctuation">)</span>
    x <span class="token operator">=</span> rgb_to_hsv<span class="token punctuation">(</span>np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>image<span class="token punctuation">)</span><span class="token operator">/</span><span class="token number">255</span><span class="token punctuation">.</span><span class="token punctuation">)</span>
    x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">+=</span> hue
    x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span>x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token operator">&gt;</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">-=</span> <span class="token number">1</span>
    x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span>x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token operator">&lt;</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">+=</span> <span class="token number">1</span>
    x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">*=</span> sat
    x<span class="token punctuation">[</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span> <span class="token operator">*=</span> val
    x<span class="token punctuation">[</span>x<span class="token operator">&gt;</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">1</span>
    x<span class="token punctuation">[</span>x<span class="token operator">&lt;</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">0</span>
    image <span class="token operator">=</span> hsv_to_rgb<span class="token punctuation">(</span>x<span class="token punctuation">)</span>

    image <span class="token operator">=</span> Image<span class="token punctuation">.</span>fromarray<span class="token punctuation">(</span><span class="token punctuation">(</span>image<span class="token operator">*</span><span class="token number">255</span><span class="token punctuation">)</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span>np<span class="token punctuation">.</span>uint8<span class="token punctuation">)</span><span class="token punctuation">)</span>
    <span class="token comment"># 将图片进行放置，分别对应四张分割图片的位置</span>
    dx <span class="token operator">=</span> place_x<span class="token punctuation">[</span>index<span class="token punctuation">]</span>
    dy <span class="token operator">=</span> place_y<span class="token punctuation">[</span>index<span class="token punctuation">]</span>
    new_image <span class="token operator">=</span> Image<span class="token punctuation">.</span>new<span class="token punctuation">(</span><span class="token string">'RGB'</span><span class="token punctuation">,</span> <span class="token punctuation">(</span>w<span class="token punctuation">,</span>h<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token number">128</span><span class="token punctuation">,</span><span class="token number">128</span><span class="token punctuation">,</span><span class="token number">128</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    new_image<span class="token punctuation">.</span>paste<span class="token punctuation">(</span>image<span class="token punctuation">,</span> <span class="token punctuation">(</span>dx<span class="token punctuation">,</span> dy<span class="token punctuation">)</span><span class="token punctuation">)</span>
    image_data <span class="token operator">=</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>new_image<span class="token punctuation">)</span><span class="token operator">/</span><span class="token number">255</span>

    <span class="token comment"># Image.fromarray((image_data*255).astype(np.uint8)).save(str(index)+"distort.jpg")</span>

    index <span class="token operator">=</span> index <span class="token operator">+</span> <span class="token number">1</span>
    box_data <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token comment"># 对box进行重新处理</span>
    <span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>box<span class="token punctuation">)</span><span class="token operator">&gt;</span><span class="token number">0</span><span class="token punctuation">:</span>
        np<span class="token punctuation">.</span>random<span class="token punctuation">.</span>shuffle<span class="token punctuation">(</span>box<span class="token punctuation">)</span>
        box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token operator">=</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token operator">*</span>nw<span class="token operator">/</span>iw <span class="token operator">+</span> dx
        box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">]</span> <span class="token operator">=</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token operator">*</span>nh<span class="token operator">/</span>ih <span class="token operator">+</span> dy
        box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">:</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">[</span>box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">:</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token operator">&lt;</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">0</span>
        box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">[</span>box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span><span class="token operator">&gt;</span>w<span class="token punctuation">]</span> <span class="token operator">=</span> w
        box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">[</span>box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">]</span><span class="token operator">&gt;</span>h<span class="token punctuation">]</span> <span class="token operator">=</span> h
        box_w <span class="token operator">=</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">]</span> <span class="token operator">-</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span>
        box_h <span class="token operator">=</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">]</span> <span class="token operator">-</span> box<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">]</span>
        box <span class="token operator">=</span> box<span class="token punctuation">[</span>np<span class="token punctuation">.</span>logical_and<span class="token punctuation">(</span>box_w<span class="token operator">&gt;</span><span class="token number">1</span><span class="token punctuation">,</span> box_h<span class="token operator">&gt;</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">]</span>
        box_data <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>box<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token number">5</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
        box_data<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token builtin">len</span><span class="token punctuation">(</span>box<span class="token punctuation">)</span><span class="token punctuation">]</span> <span class="token operator">=</span> box

    image_datas<span class="token punctuation">.</span>append<span class="token punctuation">(</span>image_data<span class="token punctuation">)</span>
    box_datas<span class="token punctuation">.</span>append<span class="token punctuation">(</span>box_data<span class="token punctuation">)</span>

    img <span class="token operator">=</span> Image<span class="token punctuation">.</span>fromarray<span class="token punctuation">(</span><span class="token punctuation">(</span>image_data<span class="token operator">*</span><span class="token number">255</span><span class="token punctuation">)</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span>np<span class="token punctuation">.</span>uint8<span class="token punctuation">)</span><span class="token punctuation">)</span>
    <span class="token keyword">for</span> j <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>box_data<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        thickness <span class="token operator">=</span> <span class="token number">3</span>
        left<span class="token punctuation">,</span> top<span class="token punctuation">,</span> right<span class="token punctuation">,</span> bottom  <span class="token operator">=</span> box_data<span class="token punctuation">[</span>j<span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">:</span><span class="token number">4</span><span class="token punctuation">]</span>
        draw <span class="token operator">=</span> ImageDraw<span class="token punctuation">.</span>Draw<span class="token punctuation">(</span>img<span class="token punctuation">)</span>
        <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>thickness<span class="token punctuation">)</span><span class="token punctuation">:</span>
            draw<span class="token punctuation">.</span>rectangle<span class="token punctuation">(</span><span class="token punctuation">[</span>left <span class="token operator">+</span> i<span class="token punctuation">,</span> top <span class="token operator">+</span> i<span class="token punctuation">,</span> right <span class="token operator">-</span> i<span class="token punctuation">,</span> bottom <span class="token operator">-</span> i<span class="token punctuation">]</span><span class="token punctuation">,</span>outline<span class="token operator">=</span><span class="token punctuation">(</span><span class="token number">255</span><span class="token punctuation">,</span><span class="token number">255</span><span class="token punctuation">,</span><span class="token number">255</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    img<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span>


<span class="token comment"># 将图片分割，放在一起</span>
cutx <span class="token operator">=</span> np<span class="token punctuation">.</span>random<span class="token punctuation">.</span>randint<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">(</span>w<span class="token operator">*</span>min_offset_x<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span>w<span class="token operator">*</span><span class="token punctuation">(</span><span class="token number">1</span> <span class="token operator">-</span> min_offset_x<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
cuty <span class="token operator">=</span> np<span class="token punctuation">.</span>random<span class="token punctuation">.</span>randint<span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">(</span>h<span class="token operator">*</span>min_offset_y<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span>h<span class="token operator">*</span><span class="token punctuation">(</span><span class="token number">1</span> <span class="token operator">-</span> min_offset_y<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

new_image <span class="token operator">=</span> np<span class="token punctuation">.</span>zeros<span class="token punctuation">(</span><span class="token punctuation">[</span>h<span class="token punctuation">,</span>w<span class="token punctuation">,</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
new_image<span class="token punctuation">[</span><span class="token punctuation">:</span>cuty<span class="token punctuation">,</span> <span class="token punctuation">:</span>cutx<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span> <span class="token operator">=</span> image_datas<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token punctuation">:</span>cuty<span class="token punctuation">,</span> <span class="token punctuation">:</span>cutx<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
new_image<span class="token punctuation">[</span>cuty<span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">:</span>cutx<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span> <span class="token operator">=</span> image_datas<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">[</span>cuty<span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">:</span>cutx<span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
new_image<span class="token punctuation">[</span>cuty<span class="token punctuation">:</span><span class="token punctuation">,</span> cutx<span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span> <span class="token operator">=</span> image_datas<span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">[</span>cuty<span class="token punctuation">:</span><span class="token punctuation">,</span> cutx<span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>
new_image<span class="token punctuation">[</span><span class="token punctuation">:</span>cuty<span class="token punctuation">,</span> cutx<span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span> <span class="token operator">=</span> image_datas<span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token punctuation">:</span>cuty<span class="token punctuation">,</span> cutx<span class="token punctuation">:</span><span class="token punctuation">,</span> <span class="token punctuation">:</span><span class="token punctuation">]</span>

<span class="token comment"># 对框进行进一步的处理</span>
new_boxes <span class="token operator">=</span> merge_bboxes<span class="token punctuation">(</span>box_datas<span class="token punctuation">,</span> cutx<span class="token punctuation">,</span> cuty<span class="token punctuation">)</span>

<span class="token keyword">return</span> new_image<span class="token punctuation">,</span> new_boxes

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72
• 73
• 74
• 75
• 76
• 77
• 78
• 79
• 80
• 81
• 82
• 83
• 84
• 85
• 86
• 87
• 88
• 89
• 90
• 91
• 92
• 93
• 94
• 95
• 96
• 97
• 98
• 99
• 100
• 101
• 102
• 103
• 104
• 105
• 106
• 107
• 108
• 109
• 110
• 111
• 112
• 113
• 114
• 115
• 116
• 117
• 118
• 119
• 120
• 121
• 122
• 123
• 124
• 125
• 126
• 127
• 128
• 129
• 130
• 131
• 132
• 133
• 134
• 135
• 136
• 137
• 138
• 139
• 140
• 141
• 142
• 143
• 144
• 145
• 146
• 147
• 148
• 149
• 150
• 151
• 152
• 153
• 154
• 155
• 156
• 157
• 158
• 159
• 160
• 161
• 162
• 163
• 164
• 165
• 166
• 167
• 168
• 169
• 170
• 171
• 172
• 173
• 174
• 175
• 176
• 177
• 178
• 179
• 180
• 181
• 182
• 183

b)、Label Smoothing平滑

标签平滑的思想很简单，具体公式如下：

new_onehot_labels = onehot_labels * (1 - label_smoothing) + label_smoothing / num_classes

当label_smoothing的值为0.01得时候，公式变成如下所示：

new_onehot_labels = y * (1 - 0.01) + 0.01 / num_classes

其实Label Smoothing平滑就是将标签进行一个平滑，原始的标签是0、1，在平滑后变成0.005(如果是二分类)、0.995，也就是说对分类准确做了一点惩罚，让模型不可以分类的太准确，太准确容易过拟合。

实现代码如下：

#---------------------------------------------------#
#   平滑标签
#---------------------------------------------------#
def _smooth_labels(y_true, label_smoothing):
    num_classes = K.shape(y_true)[-1],
    label_smoothing = K.constant(label_smoothing, dtype=K.floatx())
    return y_true * (1.0 - label_smoothing) + label_smoothing / num_classes

c)、CIOU

IoU是比值的概念，对目标物体的scale是不敏感的。然而常用的BBox的回归损失优化和IoU优化不是完全等价的，寻常的IoU无法直接优化没有重叠的部分。

于是有人提出直接使用IOU作为回归优化loss，CIOU是其中非常优秀的一种想法。

CIOU将目标与anchor之间的距离，重叠率、尺度以及惩罚项都考虑进去，使得目标框回归变得更加稳定，不会像IoU和GIoU一样出现训练过程中发散等问题。而惩罚因子把预测框长宽比拟合目标框的长宽比考虑进去。

CIOU公式如下
C I O U = I O U − ρ 2 ( b , b g t ) c 2 − α v C I O U = I O U − ρ 2 ( b , b g t ) c 2 − α v C I O U = I O U − ρ 2 ( b , b g t ) c 2 − α v CIOU=IOU−ρ2(b,bgt)c2−αvCIOU=IOU−ρ2(b,bgt)c2−αv CIOU = IOU - \frac{\rho^2(b,b^{gt})}{c^2} - \alpha v CIOU=IOU−ρ2(b,bgt)c2−αvCIOU=IOU−ρ2(b,bgt)c2−αvCIOU=IOU−c2ρ2(b,bgt)​−αvLOSSCIOU​=1−IOU+c2ρ2(b,bgt)​+αv

def box_ciou(b1, b2):
    """
    输入为：
    ----------
    b1: tensor, shape=(batch, feat_w, feat_h, anchor_num, 4), xywh
    b2: tensor, shape=(batch, feat_w, feat_h, anchor_num, 4), xywh