Since I started working on big companies I’ve been becoming a little interested in distributed systems. There are some distributed technologies I’ve been wanting to play with, but I don’t have a bunch of machines I can use to test how they work. To avoid having to buy multiple machines I decided to learn how to do it in a single machine using virtualization.

In this post I’m going to try to explain the basics of virtualization so we can build a few virtual machines that can talk to each other.

Types of virtualization

There are a few types of virtualization:

  • Hardware emulation – This is generally very slow because the hardware is emulated with software.
  • Full virtualization – Uses an hypervisor to share hardware with the host machine.
  • Para-virtualization – Shares the process with the guest operating system.
  • Operating System-level virtualization – Partitions a host into insulated guests. This is kind of chroot but with much stronger resource isolation.

Getting the host ready

I’m going to be using Fedora as my host system since that is what I’m currently running. The first thing you want to check is if you have Full Virtualization available:

1
egrep '^flags.*(vmx|svm)' /proc/cpuinfo

If no lines were printed it means you will have to use hardware emulation which is much slower.

To install QEMU, KVEM and other virtualization tools we can run this command:

1
su -c "yum install @virtualization"

Once the necessary packages are installed, you can check that the KVM modules are properly loaded by running:

1
lsmod | grep kvm

You should see kvm_intel or kvm_amd in the output if everything is fine.

Creating guests

Before we create a guest we have to decide what kind of virtual disk we are going use for it. I’m going to use an LVM2 volume with 16384MB. You can find more information about the types of disks in the virt-tools documentation.

To create an LVM2 volume we need to create a Logical Volume. To start we need to understand what a physical volume and a logical volume are. A physical volume is something that from the OS point of view looks like a physical storage unit. You can display the physical volumes on your machine using this command:

1
sudo pvdisplay

Output looks like this:

1
2
3
4
5
6
7
8
9
10
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               fedora
  PV Size               930.83 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              238291
  Free PE               16
  Allocated PE          238275
  PV UUID               4I0Ygj-Nbh4-t5kn-6Hk7-SoM5-C2uJ-ILrXp8

The output is telling us that we have a single physical volume called /dev/sda3 which is part of a volume group called fedora. A volume group is a group of physical volumes. A physical volume can only be part of one group but there can be multiple physical volumes in one group. Logical volumes are virtual volumes created on top of volume groups. We can see all the current logical volumes on the fedora group with this command:

1
sudo lvdisplay -v fedora

And the output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
...
  --- Logical volume ---
  LV Path                /dev/fedora/home
  LV Name                home
  VG Name                fedora
  LV UUID                zDJx1P-ZzLT-PAjW-WImL-LaFN-RTOR-86QKAU
  LV Write Access        read/write
  LV Creation host, time localhost, 2014-12-18 17:35:24 -0600
  LV Status              available
  # open                 1
  LV Size                873.01 GiB
  Current LE             223491
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
...

The logical volume I’m most interested in right now is the home volume because it has a size of 873.01GB from where I can steal some space for a new logical volume.

Follow the instructions here to safely shrink an LVM volume. The commands I used are:

1
2
3
4
5
umount /dev/fedora/home
e2fsck -f /dev/fedora/home
resize2fs /dev/fedora/home 720G
lvreduce -L 800G /dev/fedora/home
resize2fs /dev/fedora/home

Running lvdisplay after, shows that the changes correctly took effect:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
  --- Logical volume ---
  LV Path                /dev/fedora/home
  LV Name                home
  VG Name                fedora
  LV UUID                zDJx1P-ZzLT-PAjW-WImL-LaFN-RTOR-86QKAU
  LV Write Access        read/write
  LV Creation host, time localhost, 2014-12-18 17:35:24 -0600
  LV Status              available
  # open                 1
  LV Size                800.00 GiB
  Current LE             204800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

The output of pvdisplay has also changed to show the available PE:

1
2
3
4
5
6
7
8
9
  PV Name               /dev/sda3
  VG Name               fedora
  PV Size               930.83 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              238291
  Free PE               18707
  Allocated PE          219584
  PV UUID               4I0Ygj-Nbh2-t5kn-6Hk7-SoM5-C2uJ-ILrXp8

Now I can go ahead and create an LVM2 logical volume for my guest:

1
sudo lvcreate -n vm1 -L 16384M fedora

Where vm1 is the name of the logical volume and fedora is the group where I want to add it.

I’m going to install fedora on my virtual machine, so I’m going to use this command:

1
sudo virt-install -r 1024 --accelerate -n VirtualFedora -f /dev/fedora/vm1 --cdrom /tmp/Fedora-Live-Workstation-x86_64-21-5.iso

Where VirtualFedora is the name of the virtual machine, /dev/fedora/vm1 is the full path to the logical volume and /tmp/Fedora-Live-Workstation-x86_64-21-5.iso is the path to the ISO I want to install. The installation process is the same as with a real machine.

Starting and stopping

We now have a virtual machine ready and we need to know how to use it. If you want to know which virtual machines you have available in the current host you can use:

1
sudo virsh list --all

I got this output:

1
2
3
 Id    Name                           State
----------------------------------------------------
 -     VirtualFedora                  shut off

You can start the virtual machine using this command:

1
sudo virsh start VirtualFedora

This command basically acts as pressing the on button on a machine, but it doesn’t do anything else. You can see that it actually worked using “sudo virsh list –all”. This is an example output:

1
2
3
 Id    Name                           State
----------------------------------------------------
 13    VirtualFedora                  running

An Id has been assigned and now the state is “running”. To get an actual screen where you can interact with the machine, use:

1
sudo virt-viewer VirtualFedora

You can shut off the computer from the GUI the way you would normally do it(And that is the recommended way to shut down the machine) but if for some reason the GUI is not responsive you can use this command to pull the virtual plug of your machine:

1
sudo virsh destroy VirtualFedora

Networking

By default libvirt will create a private network for your guests on the host machine. This makes it possible for the guests and the host to talk among them. To allow computers in the host’s network to talk to guests a little configuration is necessary.

Currently I’m running two guests on my machine. If I run ifconfig on the host I get an output like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::fc54:ff:fe41:b869  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:12:33:75  txqueuelen 0  (Ethernet)
        RX packets 201940  bytes 11524447 (10.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 349889  bytes 513753400 (489.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc54:ff:fe12:3375  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:12:33:75  txqueuelen 500  (Ethernet)
        RX packets 48060  bytes 3582652 (3.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 81083  bytes 115864011 (110.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vnet1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fc54:ff:febc:3817  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:bc:38:17  txqueuelen 500  (Ethernet)
        RX packets 43588  bytes 3055909 (2.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 76266  bytes 112107093 (106.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I have one virtual bridge and two virtual networks. From here there is no way to know which IP addresses belong to my guests. To get a guest IP address you can run ifconfig from within the guest:

1
2
3
4
5
6
7
8
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.48  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:fe12:3375  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:12:33:75  txqueuelen 1000  (Ethernet)
        RX packets 80952  bytes 115855356 (110.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 48013  bytes 3578503 (3.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Enabling communication to the private network from the outside world has to be done using iptables rules and is usually done in a per-need basis.

As an example I will have a web server running on port 1337 on my guest that lives on IP 192.168.122.48 and I will make it accessible to the outside from port 8765 from my host machine. We will also need the name of the guest, which is VirtualFedora in my example.

Create this file: /etc/libvirt/hooks/qemu and make it executable using:

1
sudo chmod +x /etc/libvirt/hooks/qemu

The content of the file will be:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/bin/sh
# used some from advanced script to have multiple ports: use an equal number of guest and host ports

Guest_name=VirtualFedora
Guest_ipaddr=192.168.122.48
Host_port=( '8765' )
Guest_port=( '1337' )
length=$(( ${#Host_port[@]} - 1 ))
if [ "${1}" = "${Guest_name}" ]; then
   if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then
       for i in `seq 0 $length`; do
               iptables -t nat -D PREROUTING -p tcp --dport ${Host_port[$i]} -j DNAT \
                       --to ${Guest_ipaddr}:${Guest_port[$i]}
               iptables -D FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW \
                       -m tcp --dport ${Guest_port[$i]} -j ACCEPT
       done
   fi
   if [ "${2}" = "start" -o "${2}" = "reconnect" ]; then
       for i in `seq 0 $length`; do
               iptables -t nat -A PREROUTING -p tcp --dport ${Host_port[$i]} -j DNAT \
                        --to ${Guest_ipaddr}:${Guest_port[$i]}
               iptables -I FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW \
                        -m tcp --dport ${Guest_port[$i]} -j ACCEPT
       done
   fi
fi

Once the script is ready we need to close all our guests and restart libvirt using this command:

1
service libvirtd restart

Once we start our guest and our service in port 1337, machines inside the host network will be able to access the service using the host IP address and port 8765 as per our configuration.

[ linux  ]