[libvirt] cgroup limits only affect kvm guest under certain conditions

[libvirt] cgroup limits only affect kvm guest under certain conditions

[Dominik Klein]

Hi

I am playing with cgroups and try to limit block io for guests.

The proof of concept is:

# mkdir /dev/cgroup/blkio
# mount -t cgroup -o blkio blkio /dev/cgroup/blkio/
# cd blkio/
# mkdir test
# cd test/
# ls -l /dev/vdisks/kirk
lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5
# ls -l /dev/dm-5
brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5
# echo "253:5  1048576" > blkio.throttle.write_bps_device
# echo $$ > tasks
# dd if=/dev/zero of=/dev/dm-5 bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s

So limit applies to the dd child of my shell.

Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm
pid into tasks. Limits are not applied, the guest can happily use max io
bandwidth.

However, if I start the guest manually like

# qemu-kvm <options like libvirt creates> & echo $! >
/dev/cgroup/blkio/test/tasks

The limits _are_ applied.

So, this looks like some sort of race condition to me.

I tried to get information on this on the kernel mailing list [1], but
either noone read it or at least no one replied. Maybe someone here can
shed some light and maybe even fix the issue, if it is an issue.

Need more information? Please ask for it. I don't know what else to
supply at this point.

My current lab is built by:
OpenSuSE 11.3 64bit
Vanilla Kernel 2.6.37
libvirt 0.8.7
qemu-kvm 0.13.0
vm is started like this:
/usr/bin/qemu-kvm -M pc-0.12 -enable-kvm -m 2048 -smp
2,sockets=2,cores=1,threads=1 -name cliff -uuid
a8247e1e-e3d2-d0fc-c5e5-47a173c3e460 -nodefconfig -nodefaults -chardev
socket,id=monitor,path=/var/lib/libvirt/qemu/cliff.monitor,server,nowait
-mon chardev=monitor,mode=readline -rtc base=utc -boot c -device
lsi,id=scsi0,bus=pci.0,addr=0x6 -drive
file=/dev/vdisks/cliff,if=none,id=drive-virtio-disk0,boot=on,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-drive
file=/root/openSUSE-11.3-NET-x86_64.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-drive file=/dev/vdisks/jason,if=none,id=drive-virtio-disk1,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1
-drive file=/dev/vdisks/rob,if=none,id=drive-ide0-0-0,format=raw -device
ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive
file=/dev/vdisks/james,if=none,id=drive-scsi0-0-0,format=raw -device
scsi-disk,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0
-netdev tap,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:37:84:e0,bus=pci.0,addr=0x5
-usb -vnc 127.0.0.1:0 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

Regards
Dominik

[1]
http://help.lockergnome.com/linux/race-condition-net_cls-found-qemu-kvm-environment--ftopict529787.html

Re: [libvirt] cgroup limits only affect kvm guest under certain conditions

[Daniel P. Berrange]

On Thu, Jan 06, 2011 at 02:15:37PM +0100, Dominik Klein wrote:
> Hi
> 
> I am playing with cgroups and try to limit block io for guests.
> 
> The proof of concept is:
> 
> # mkdir /dev/cgroup/blkio
> # mount -t cgroup -o blkio blkio /dev/cgroup/blkio/
> # cd blkio/
> # mkdir test
> # cd test/
> # ls -l /dev/vdisks/kirk
> lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5
> # ls -l /dev/dm-5
> brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5
> # echo "253:5  1048576" > blkio.throttle.write_bps_device
> # echo $$ > tasks
> # dd if=/dev/zero of=/dev/dm-5 bs=1M count=20
> 20+0 records in
> 20+0 records out
> 20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
> 
> So limit applies to the dd child of my shell.
> 
> Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm
> pid into tasks. Limits are not applied, the guest can happily use max io
> bandwidth.

Did you just echo the main qemu-kvm PID, or did you also
add the PIDs of every thread too ? From this description
of the problem, I'd guess you've only confined the main
process thread and thus the I/O & VCPU threads are not
confined.

Daniel

Re: [libvirt] cgroup limits only affect kvm guest under certain conditions

[Dominik Klein]

> Did you just echo the main qemu-kvm PID, or did you also
> add the PIDs of every thread too ? From this description
> of the problem, I'd guess you've only confined the main
> process thread and thus the I/O & VCPU threads are not
> confined.

That was indeed correct. I was mislead by the fact that no child
_processes_ were shown in ps.

Once I added /proc/<qemu-pid>/tasks/* to tasks, it worked even when
libvirt started the process.

Thank you very much!
Dominik

Re: cgroup limits only affect kvm guest under certain conditions

[KAMEZAWA Hiroyuki]

On Thu, 06 Jan 2011 14:15:37 +0100
Dominik Klein <dk@in-telegence.net> wrote:

> Hi
> 
> I am playing with cgroups and try to limit block io for guests.
> 
> The proof of concept is:
> 
> # mkdir /dev/cgroup/blkio
> # mount -t cgroup -o blkio blkio /dev/cgroup/blkio/
> # cd blkio/
> # mkdir test
> # cd test/
> # ls -l /dev/vdisks/kirk
> lrwxrwxrwx 1 root root 7 2011-01-06 13:46 /dev/vdisks/kirk -> ../dm-5
> # ls -l /dev/dm-5
> brw-rw---- 1 root disk 253, 5 2011-01-06 13:36 /dev/dm-5
> # echo "253:5  1048576" > blkio.throttle.write_bps_device
> # echo $$ > tasks
> # dd if=/dev/zero of=/dev/dm-5 bs=1M count=20
> 20+0 records in
> 20+0 records out
> 20971520 bytes (21 MB) copied, 20.0223 s, 1.0 MB/s
> 
> So limit applies to the dd child of my shell.
> 
> Now I assign /dev/dm-5 (/dev/vdisks/kirk) to a vm and echo the qemu-kvm
> pid into tasks. Limits are not applied, the guest can happily use max io
> bandwidth.
> 

qemu consists of several threads. Cgroup works per thread now.
Could you double check "all threads for qemu" are in a cgroup ?
I think you have to write all thread-ID to "tasks" file when you
move qemu after starting it.

Thanks,
-Kame

Re: [libvirt] cgroup limits only affect kvm guest under certain conditions

[Dominik Klein]

> qemu consists of several threads. Cgroup works per thread now.
> Could you double check "all threads for qemu" are in a cgroup ?
> I think you have to write all thread-ID to "tasks" file when you
> move qemu after starting it.

As stated yesterday: Yes, that was the problem. Works as expected now,
thanks.

Dominik