1、openmi 下载安装

　　下载连接：

　　　　https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.gz

　　安装命令

1
2
3
4
5

shell$ gunzip -c openmpi-4.0.1.tar.gz | tar xf -
shell$ cd openmpi-4.0.1
shell$ ./configure --prefix=/usr/local
<…lots of output…>
shell$ make all install

sudo ldconfig

2、horovod安装

官方文档： https://github.com/horovod/horovod#install

[sudo] pip3 install horovod

安装支持NCCL的版本的horovod

HOROVOD_GPU_ALLREDUCE=NCCL pip3 install --no-cache-dir horovod

3、horovod 使用

3.1 tensorFLow 修改

import tensorflow as tf
import horovod.tensorflow as hvd

Initialize Horovod

hvd.init()

Pin GPU to be used to process local rank (one GPU per process)

config = tf.ConfigProto()
config.gpu_options.visible_device_list = str(hvd.local_rank())

Build model…

loss = …
opt = tf.train.AdagradOptimizer(0.01 * hvd.size())

Add Horovod Distributed Optimizer

opt = hvd.DistributedOptimizer(opt)

Add hook to broadcast variables from rank 0 to all other processes during

initialization.

hooks = [hvd.BroadcastGlobalVariablesHook(0)]

Make training operation

train_op = opt.minimize(loss)

Save checkpoints only on worker 0 to prevent other workers from corrupting them.

checkpoint_dir = '/tmp/train_logs' if hvd.rank() == 0 else None

The MonitoredTrainingSession takes care of session initialization,

restoring from a checkpoint, saving to a checkpoint, and closing when done

or an error occurs.

with tf.train.MonitoredTrainingSession(checkpoint_dir=checkpoint_dir,
config=config,
hooks=hooks) as mon_sess:
while not mon_sess.should_stop():
# Perform synchronous training.
mon_sess.run(train_op)

3.2 tensorflow 运行

mpi 指定mca通讯端口

mpirun --allow-run-as-root --oversubscribe \
-np 8-H ubuntu1:4,ubuntu2:4 \
-bind-to none -map-by slot \
-mca plm_rsh_args "-p 22" \
-x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \
-mca pml ob1 -mca btl ^openib \
python3 -u train.py