rbd caching issue

["Jens Rehpöhler" <jens.rehpoehler@filoo.de>]

[-- Attachment #1: Type: text/plain, Size: 2797 bytes --]

Hi folks,

i just received a mail of a customer. He reclaimed that the "ping
latency" of his VM rises, if he does a lot of IO
inside of the VM. I have done the same with a test VM.

I could reproduce this behavior. If i disable the rbd cache the VM IO is
slower but the latency is ok. Even SSH und other
programs are affected .... so its not a problem of slow ICMP.

Device Setting:

virtio0:
rbd:9997/vm-1171-disk-1.rbd:rbd_cache=true:rbd_cache_size=16777216:rbd_cache_max_dirty=8388608:rbd_cache_target_dirty=4194304,cache=none

normal ping:

64 bytes from 109.75.x.x: icmp_seq=38 ttl=56 time=29.2 ms
64 bytes from 109.75.x.x: icmp_seq=39 ttl=56 time=20.8 ms
64 bytes from 109.75.x.x: icmp_seq=40 ttl=56 time=22.4 ms

with lots of IO:

64 bytes from 109.75.x.x: icmp_seq=87 ttl=56 time=28.1 ms
64 bytes from 109.75.x.x: icmp_seq=88 ttl=56 time=665 ms
64 bytes from 109.75.x.x: icmp_seq=89 ttl=56 time=226 ms
64 bytes from 109.75.x.x: icmp_seq=90 ttl=56 time=179 ms
64 bytes from 109.75.x.x: icmp_seq=91 ttl=56 time=140 ms
64 bytes from 109.75.x.x: icmp_seq=92 ttl=56 time=25.6 ms
64 bytes from 109.75.x.x: icmp_seq=93 ttl=56 time=568 ms
64 bytes from 109.75.x.x: icmp_seq=94 ttl=56 time=405 ms
64 bytes from 109.75.x.x: icmp_seq=95 ttl=56 time=223 ms
64 bytes from 109.75.x.x: icmp_seq=96 ttl=56 time=24.5 ms
64 bytes from 109.75.x.x: icmp_seq=97 ttl=56 time=321 ms
64 bytes from 109.75.x.x: icmp_seq=98 ttl=56 time=391 ms
64 bytes from 109.75.x.x: icmp_seq=99 ttl=56 time=4200 ms
64 bytes from 109.75.x.x: icmp_seq=101 ttl=56 time=2194 ms

But if i disable caching:

virtio0: rbd:9997/vm-1171-disk-1.rbd:rbd_cache=false,cache=writeback

with lots of IO:

64 bytes from 109.75.x.x: icmp_seq=62 ttl=56 time=22.1 ms
64 bytes from 109.75.x.x: icmp_seq=63 ttl=56 time=26.5 ms
64 bytes from 109.75.x.x: icmp_seq=64 ttl=56 time=30.7 ms
64 bytes from 109.75.x.x: icmp_seq=65 ttl=56 time=24.8 ms
64 bytes from 109.75.x.x: icmp_seq=66 ttl=56 time=21.9 ms

Can someone please explain me this behavior ? Why is the latency of the
VM spiky if i enable rbd caching ? I've played around with the caching
parameters
but with caching enabled its always the same.

KVM Version: 1.2.1
Ceph Version: ceph version 0.48.2argonaut
(commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)

Thanks a lot !!

-- 
mit freundlichen Grüssen

Jens Rehpöhler

----------------------------------------------------------------------
filoo GmbH
Moltkestr. 25a
33330 Gütersloh
HRB4355 AG Gütersloh

Geschäftsführer: S.Grewing | J.Rehpöhler | Dr. C.Kunz
Telefon: +49 5241 8673012 | Mobil: +49 151 54645798
Hotline: +49 5241 8673026| Fax: +49 5241 8673020

Folgen Sie uns auf:
Twitter: http://twitter.com/filoogmbh
Facebook: http://facebook.com/filoogmbh



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]