XStar's Libvirt+KVM部署记录

Create: 2013-12-11
Update: 2014-01-03
KVM网站
- http://sourceforge.net/projects/kvm/files/
- http://www.linux-kvm.org/
CPU支持检查

如果CPU支持，还需要在BIOS打开虚拟化支持。通常，主板默认关闭这个选项。

kvm-ok

或

grep -E "vmx|svm" /proc/cpuinfo

Intel CPU:vmx

AMD CPU:svm

libvirt: http://libvirt.org/

使用libvirt负责虚拟化的创建及管理。

网络结构
- NAT：用libvirt原有的default网络，通过virbr0连接、Dnsmasq做dhcp、iptables提供NAT；
- Bridge：手工在系统中配置网桥virbr1，在libvirt配置虚拟机直接连接该该网桥即可。
相关软件包
- qemu-kvm - Full virtualization on i386 and amd64 hardware
- qemu-system - QEMU full system emulation binaries
- qemu-utils - qemu utilities
- libvirt-bin - programs for the libvirt library
- pm-utils - utilities and scripts for power management
- virt-manager - desktop application For managing virtual machines
- virt-viewer - Displaying the graphical console of a virtual machine
- virtinst - Programs to create and clone virtual machines
  - virt-install: provision new virtual machines
  - virt-clone: clone existing virtual machines
  - virt-image: create virtual machines from an image description
  - virt-convert: convert virtual machines between formats
从源安装

# Ubuntu
sudo apt-get -y install qemu-kvm qemu-utils libvirt-bin pm-utils virtinst
sudo apt-get -y install virt-viewer virt-manager

CentOS

yum install -y qemu-kvm qemu-kvm-tools libvirt libvirt-devel libvirt-client pm-utils
yum install -y virt-viewer virt-manager
编译安装

# Ubuntu中解依编译信赖关系
sudo apt-get build-dep qemu-kvm

tar -xzvf qemu-kvm-1.2.0.tar.gz
cd qemu-kvm-1.2.0
/configure --prefix=/usr/local/kvm
make
sudo make install

安装后加载KVM模块

注意:系统重启后要重新加载kvm内核模块，从源安装时会有qemu-kvm服务来自动加载该模块。

sudo modprobe kvm
sudo modprobe kvm-intel #INTEL处理器就用这个
sudo modprobe kvm-amd #AMD处理器就用这个
/etc/default/libvirt-bin

start_libvirtd="yes"
libvirtd_opts="-d -l"
开启TCP侦听
- /etc/libvirt-bin/libvir-bin.conf
  
  listen_tls = 0 # 禁用tls
  listen_tcp = 1 # 启用tcp侦听
  tcp_port = "16509" # tcp端口
  listen_addr = "0.0.0.0" # tcp侦听IP
  auth_unix_ro = "none"
  auth_unix_rw = "none"
  auth_tcp = "sasl" # tcp登录认证方式,"none"为不使用认证
  log_level = 3
  log_outputs="3:syslog:libvirtd"
  
  max_clients = 1024 # 最大总的连接客户数1024
  min_workers = 50 # libvirtd启动时，初始的工作线程数目
  max_workers = 200 # 同上，最大数目
  max_requests = 1000 # 最大同时支持的RPC调用，必须大于等于max_workers
  max_client_requests = 200 # 每个客户端支持的最大连接数
- 安装sasl2、设置自启动
  
  apt-get install sasl2-bin
  sed -i 's/$^START$=.*$/\1=yes/g' /etc/default/saslauthd
- sasl用户设置(http://libvirt.org/auth.html))
  
  # 添加用户
  saslpasswd2 -a libvirt yuanxing
  #列出用户
  sasldblistusers2 -f /etc/libvirt/passwd.db
  
  yuanxing@kvmsrv: userPassword，注意：登录时用户名要包含@kvmsrv
  
  #删除用户
  saslpasswd2 -a libvirt -d yuanxing
设置VNC侦听地址:/etc/libvirt/qemu.conf

sed -i -e '/^vnc_listen/d' i-e '/^vnc_password/d' /etc/libvirt/qemu.conf
echo 'vnc_listen = "0.0.0.0"' >> /etc/libvirt/qemu.conf
echo 'vnc_password = "XYZ12345"' >> /etc/libvirt/qemu.conf
服务管理

service libvirt-bin start|restart|stop|status|reload|force-reload
service qemu-kvm start|restart|stop|status|reload|force-reload

软件安装

相关软件包
- bridge-utils - Utilities for configuring the Linux Ethernet bridge
- vde2 - Virtual Distributed Ethernet
- ifplugd - configuration daemon for ethernet devices
- uml-utilities - User-mode Linux (utility programs)(tunctl工具)
安装bridge工具

sudo apt-get -y install bridge-utils

使用ifplugd方便向桌面发送状态通知、通过异步处理加快启动速度(dhcp浪费时间)。

使用vde2使用可能是方便以普通用户身份来管理(UserModeLinux，涉及到uml工具)？

配置主机网络

配置/etc/network/interfaces

auto lo
iface lo inet loopback

auto virbr1
iface virbr1 inet static
address 172.17.20.20
network 172.17.20.0
netmask 255.255.255.0
broadcast 172.17.20.255
gateway 172.17.20.1
dns-nameservers 8.8.8.8 4.4.4.4
bridge_ports eth0
bridge_stp on
bridge_maxwait 0
禁用NetworkManager

sudo stop network-manager

Create an override file for the upstart job:

echo "manual" | sudo tee /etc/init/network-manager.override
网桥关闭Netfilter、开启ip_forward
- /etc/sysctl.conf
  
  net.ipv4.ip_forward = 1
  net.bridge.bridge-nf-call-ip6tables = 0
  net.bridge.bridge-nf-call-iptables = 0
  net.bridge.bridge-nf-call-arptables = 0
- sysctl -p /etc/sysctl.conf
- 注释: 在网桥上关闭Netfilter是出于安全考虑，具体参考:http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaai/kvmsec/kvmsecstart.htm

libvirt网络配置

NAT网络:default.xml(libvirt缺省)

default 3be6d4fa-72c4-667b-64ae-85f954f1b2bf
桥接网络:network-host-bridge.xml

network-host-bridge 3b8bf95c-6971-8794-1c63-bc2b8c551979
导入配置

virsh net-list --all
#virsh net-define default.xml
virsh net-define network-host-bridge.xml
virsh net-start network-host-bridge
virsh net-autostart network-host-bridge

virsh net-dumpxml network-host-bridge > /etc/libvirt/qemu/network/network-host-bridge.xml

也可不在libvirt中设置主机网桥，配置虚拟机时，直接指向bridge，运行时传递给kvm的参数相同。

指向network-host-bridge

<interface type='network'>  
  <mac address='52:54:00:6c:07:87'/>  
  <source network='network-host-bridge'/>  
  <target dev='vnet0'/>  
  <model type='virtio'/>  
  <alias name='net0'/>  
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>  
</interface>

直接指向bridge

<interface type='bridge'>  
  <mac address='52:54:00:6c:07:87'/>  
  <source bridge='virbr1'/>  
  <target dev='vnet0'/>  
  <model type='virtio'/>  
  <alias name='net0'/>  
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>  
</interface>

libvirt生成的iptables NAT策略

# Generated by iptables-save v1.4.12 on Sat Dec 14 01:24:40 2013
*nat
:PREROUTING ACCEPT [310:34187]
:INPUT ACCEPT [41:4149]
:OUTPUT ACCEPT [471:31957]
:POSTROUTING ACCEPT [711:59731]
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
COMMIT

Completed on Sat Dec 14 01:24:40 2013

Generated by iptables-save v1.4.12 on Sat Dec 14 01:24:40 2013

*mangle
:PREROUTING ACCEPT [789078:428240757]
:INPUT ACCEPT [381196:51105050]
:FORWARD ACCEPT [407928:377150708]
:OUTPUT ACCEPT [399811:110095092]
:POSTROUTING ACCEPT [807739:487245800]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT

Completed on Sat Dec 14 01:24:40 2013

Generated by iptables-save v1.4.12 on Sat Dec 14 01:24:40 2013

*filter
:INPUT ACCEPT [381157:51098101]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [399811:110095092]
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
-A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
COMMIT

Completed on Sat Dec 14 01:24:40 2013
libvirt为default网络启动的dnsmasq进程

/usr/sbin/dnsmasq \
-u libvirt-dnsmasq \
--strict-order \
--bind-interfaces \
--pid-file=/var/run/libvirt/network/default.pid \
--conf-file= \
--except-interface lo \
--listen-address 192.168.122.1 \
--dhcp-range 192.168.122.2,192.168.122.254 \
--dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases \
--dhcp-lease-max=253 \
--dhcp-no-override

常用工具

virsh
virt-install
virt-manager
virt-clone
virt-convert
virt-image
qeum-img

连接libvirtd

virsh -c qemu:///system # 本地登录libvirt控制台
virsh -c qemu+ssh://root@172.17.20.20/system # ssh远程登录libvirt控制台
virsh -c qemu+tcp://172.17.20.20:16509/system # tcp远程登录libvirt控制台
virt-manager -c qemu+tcp://172.17.20.20:16509/system # 运行libvirt图形管理界面
virt-manager -c qemu+ssh://root@172.17.20.20/system

管理存储池
管理接口(bridge)
管理网络(Networking)
管理虚拟机(Domain)
- 创建/修改
- 调整内存
- 起停、悬挂、恢复
- 迁移
- console
- ttyconsole
- attach/detach
监控虚拟机
Host and Hypervisor
管理虚拟卷
快照管理
图形界面:VNC

配置存储池

data.xml

data 117251c7-3444-a5e3-3174-1947c4933321 446342758400 7004667904 439338090496 /data/libvirt/images 0700 -1 -1
ISOs.xml

ISOs e77d9811-5497-521d-0d50-e5104c7e66b1 446342758400 7004667904 439338090496 /data/ISOs 0700 -1 -1
存储池名称

virsh pool-list

Name State Autostart

data active yes
ISOs active yes

使用 virt-manager -c qemu+ssh:///system 远程连接libvirtd后，在图形界面中创建了第一个虚拟机Ubuntu，并安装了操作系统。

Ubuntu.xml

Ubuntu d2a37ea8-d343-c4bd-c6d0-fb6bd1943f91 1048576 1048576 2 hvm ¨C34C ¨C35C ¨C75C ¨C76C ¨C77C ¨C78C ¨C79C ¨C80C
虚拟机进程

# 进程以libvirt-qemu用户身份运行
/usr/bin/kvm \
-S \
-M pc-1.0 \
-enable-kvm \
-m 1024 \
-smp 2,sockets=2,cores=1,threads=1 \
-name Ubuntu \
-uuid d2a37ea8-d343-c4bd-c6d0-fb6bd1943f91 \
-nodefconfig \
-nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Ubuntu.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-reboot \
-no-shutdown \
-drive file=/data/libvirt/images/Ubuntu.img,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 \
-drive file=/data/ISOs/ubuntu-12.04.3-server-amd64.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw \
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 \
-netdev tap,fd=19,id=hostnet0,vhost=on,vhostfd=20 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6c:07:87,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-usb \
-vnc 127.0.0.1:0 \
-vga cirrus \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
libvirt通过nc对外服务(自动)

sh -c if nc -q 2>&1 | grep "requires an argument" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;nc $ARG -U /var/run/libvirt/libvirt-sock
nc -q0 -U /var/run/libvirt/libvirt-sock

nc -q 0 127.0.0.1 5900

使用vmbuilder创建Ubuntu JeOS虚拟服务器，包括虚拟机创建、系统安装、包安装，以及系统设置定制均可一步完成。

关于vmbuilder未启用磁盘的vistio特性问题: http://www.sysadminworld.com/2012/automated-installs-with-vmbuilder/
安装vmbuilder

apt-get instal python-vm-builder
使用参数
- vmbuilder --help
- vmbuilder kvm ubuntu --help
优化设置

Vmbuilder缺省对网卡启用了virtio特性，但虚拟磁盘未启用virtio特性，可修改其配置档解决：

vim /etc/vmbuilder/libvirt/libvirtxml.tmpl

to

Since my virtual machines require a pointopoint entry in /etc/network/interfaces I also edit /etc/vmbuilder/ubuntu/interfaces.tmpl and add an additional line pointopoint underneath gateway. This will set pointopoint to the gateway address that I specify with vmbuilder.

pointopoint $gw

As a last ugly rather hack I edit /usr/share/pyshared/VMBuilder/plugins/kvm/vm.py and change the default file image type from qcow2 to raw to improve the disk throughput.

filetype = 'raw'

Instead of editing vm.py you could also use the qemu-img command after running vmbuilder to convert the qcow2 image to raw (or just keep qcow2 if you are fine with it).

默认模板文件：/etc/vmbuilder/libvirt/
自定义配置文件:MyJeOS.cfg

[DEFAULT]
arch=amd64
mem=512
cpus=2
ip=172.17.20.31
gw=172.17.20.1
mask=255.255.255.0
dns=8.8.8.8
user=yuanxing
name=YuanXing
pass=123456
tmpfs=-
firstboot=/data/libvirt/JeOStemplate/firstboot.sh
#firstlogin=/data/libvirt/JeOStemplate/firstlogin.sh
#bridge=virbr1
network=network-host-bridge

[ubuntu]
mirror=http://mirrors.163.com/ubuntu
iso=/data/ISOs/ubuntu-12.04.3-server-amd64.iso
suite=precise
flavour=virtual
hostname=JeOS
addpkg=openssh-server,acpid,vim,at,bc,byobu,tmux,screen,iptables
[kvm]
libvirt = qemu:///system
首次运行firstboot.sh

# This script will run the first time the virtual machine boots

It is ran as root.

echo "Expire the user account"
passwd -e user

echo "Regenerate ssh keys"
rm -f /etc/ssh/ssh_host*key*
dpkg-reconfigure -fnoninteractive -pcritical openssh-server
首次登录firstlogin.sh

# This script is ran the first time a user logs in
echo "Your appliance is about to be finished to be set up."
echo "In order to do it, we'll need to ask you a few questions,"
echo "starting by setting you keyboard and other console informations."
分区配置:vmbuilder.partition

# ubuntu 12.04中vmbuilder的cli.py中对part的处理存在bug，不能正确按man中的格式处理设备名和文件名，因为只能使用如下简单格式
root 8000

swap 4000

/data 20000

vim +391 /usr/lib/python2.7/dist-packages/VMBuilder/contrib/cli.py

创建虚拟机

sudo vmbuilder kvm ubuntu -c MyJeOS.cfg --part vmbuilder.partition --timezone=CST -d /data/libvirt/images/JeOS

或者直接在命令行中指定参数

sudo vmbuilder kvm ubuntu --suite precise --flavour virtual --arch amd64 --libvirt qemu:///system \
--hostname gonzo --ip 192.0.32.9 --net 192.0.32.9 --mask 255.255.255.255 --bcast 192.0.32.9 --gw 213.103.105.1 \
--domain example.com --user john --name john --pass 12345 \
--verbose --rootsize=20480 --swapsize=2048 --mem 1024 --cpus 1 \
--addpkg=openssh-server --addpkg=vim --addpkg=acpid \
--dest=/var/lib/libvirt/images/gonzo/

virt-install --name=Ubuntu2 --ram 1024 --vcpus=2 \
--disk pool=data,size=8,bus=virtio,format=qcow2,sparse=false,cache=writeback \
--cdrom /data/ISOs/ubuntu-12.04.3-server-amd64.iso \
--vnc --network bridge=virbr1,model=virtio --noautoconsole \
--connect=qemu:///system

或用网络名称：

--network network=network-host-bridge

说明：在linux系统安装开始就要注意添加提高性能的一些参数，后面就不需要做一些调整了。

参数说明:

--name 指定虚拟机名称
--ram 分配内存大小
--vcpus 分配CPU核心数，最大与实体机CPU核心数相同
--disk 指定虚拟机镜像，size 指定分配大小单位为G。
--network 网络类型，此处用的是默认，一般用的应该是 bridge 桥接。
--cdrom 指定安装镜像iso
--vnc 启用VNC远程管理，一般安装系统都要启用。
--vncport 指定VNC 监控端口，默认端口为5900，端口不能重复。
--vnclisten 指定VNC 绑定IP，默认绑定127.0.0.1，这里改为 0.0.0.0。
--noautoconsole Don't automatically try to connect to the guest console

clone卷

virsh -c qemu:///system vol-clone --pool data tmpiEMBDQ.qcow2 JeOS_2.qcow2
virt-clone:虚拟机克隆

--connect=CONNECT
-o ORIGINAL_GUEST , --original=ORIGINAL_GUEST 被克隆的原始Guest名称
-m MAC , --mac=MAC MAC地址
--auto-clone, Generate a new guest name, and paths for new storage.
-n NAME , --name=NAME 新虚拟机Guest的名称
-u UUID , --uuid=UUID 指定UUID
-f DISKFILE , --file=DISKFILE

# 将JeOS克隆为指定NAME，按--file指定路径Clone虚拟磁盘。
NAME=JeOS1;
mkdir /data/libvirt/images/$NAME \
&& virt-clone \
--connect qemu:///system \
--original JeOS \
--name $NAME \
--file /data/libvirt/images/$NAME/$NAME.qcow2 \
--file /data/libvirt/images/$NAME/${NAME}_data.qcow2