[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Michael Tokarev]

12.07.2010 09:24, Giangiacomo Mariotti wrote:
> Hi, is it a known problem how much slow is Btrfs with kvm/qemu(meaning
> that the image kvm/qemu uses as the hd is on a partition formatted
> with Btrfs, not that the fs used by the hd inside the kvm environment
> is Btrfs, in fact inside kvm the / partition is formatted with ext3)?
> I haven't written down the exact numbers, because I forgot, but while
> I was trying to make it work, after I noticed how much longer than
> usual it was taking to just install the system, I took a look at iotop
> and it was reporting a write speed of the kvm process of approximately
> 3M/s, while the Btrfs kernel thread had an approximately write speed
> of 7K/s! Just formatting the partitions during the debian installation
> took minutes. When the actual installation of the distro started I had
> to stop it, because it was taking hours! The iotop results made me
> think that the problem could be Btrfs, but, to be sure that it wasn't
> instead a kvm/qemu problem, I cut/pasted the same virtual hd on an
> ext3 fs and started kvm with the same parameters as before. The
> installation of debian inside kvm this time went smoothly and fast,
> like normally it does. I've been using Btrfs for some time now and
> while it has never been a speed champion(and I guess it's not supposed
> to be one and I don't even really care that much about it), I've never
> had any noticeable performance problem before and it has always been
> quite stable. In this test case though, it seems to be doing very bad.

This looks quite similar to a problem with ext4 and O_SYNC which I
reported earlier but no one cared to answer (or read?) - there:
http://permalink.gmane.org/gmane.linux.file-systems/42758
(sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
try a few other options, esp. cache=none and re-writing some guest
files to verify.

/mjt

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Justin P. Mattock]

On 07/12/2010 12:09 AM, Michael Tokarev wrote:
> 12.07.2010 09:24, Giangiacomo Mariotti wrote:
>> Hi, is it a known problem how much slow is Btrfs with kvm/qemu(meaning
>> that the image kvm/qemu uses as the hd is on a partition formatted
>> with Btrfs, not that the fs used by the hd inside the kvm environment
>> is Btrfs, in fact inside kvm the / partition is formatted with ext3)?
>> I haven't written down the exact numbers, because I forgot, but while
>> I was trying to make it work, after I noticed how much longer than
>> usual it was taking to just install the system, I took a look at iotop
>> and it was reporting a write speed of the kvm process of approximately
>> 3M/s, while the Btrfs kernel thread had an approximately write speed
>> of 7K/s! Just formatting the partitions during the debian installation
>> took minutes. When the actual installation of the distro started I had
>> to stop it, because it was taking hours! The iotop results made me
>> think that the problem could be Btrfs, but, to be sure that it wasn't
>> instead a kvm/qemu problem, I cut/pasted the same virtual hd on an
>> ext3 fs and started kvm with the same parameters as before. The
>> installation of debian inside kvm this time went smoothly and fast,
>> like normally it does. I've been using Btrfs for some time now and
>> while it has never been a speed champion(and I guess it's not supposed
>> to be one and I don't even really care that much about it), I've never
>> had any noticeable performance problem before and it has always been
>> quite stable. In this test case though, it seems to be doing very bad.
>
> This looks quite similar to a problem with ext4 and O_SYNC which I
> reported earlier but no one cared to answer (or read?) - there:
> http://permalink.gmane.org/gmane.linux.file-systems/42758
> (sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
> try a few other options, esp. cache=none and re-writing some guest
> files to verify.
>
> /mjt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

cool a solution... glad to see... no chance at a bisect with this?
(getting this down too a commit or two makes things easier)

Justin P. Mattock

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Giangiacomo Mariotti]

On Mon, Jul 12, 2010 at 9:17 AM, Justin P. Mattock
<justinmattock@gmail.com> wrote:
> On 07/12/2010 12:09 AM, Michael Tokarev wrote:
>>
>> This looks quite similar to a problem with ext4 and O_SYNC which I
>> reported earlier but no one cared to answer (or read?) - there:
>> http://permalink.gmane.org/gmane.linux.file-systems/42758
>> (sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
>> try a few other options, esp. cache=none and re-writing some guest
>> files to verify.
>>
>> /mjt
>
> cool a solution... glad to see... no chance at a bisect with this?
> (getting this down too a commit or two makes things easier)
>
> Justin P. Mattock
>
I didn't even say what kernel version I was using, sorry! Kernel
2.6.34.1+"patches in stable queue for next stable release". I tried
this some time ago with 2.6.33.x(don't remember which version exactly)
and it had the same problem, but at the time I stopped trying thinking
that it was a kvm problem. So basically there's no known(to me) good
version and no, I can't bisect this because this is my production
system. Anyway, I suspect this is reproducible. Am I the only one who
created a virtual hd file on a Btrfs and then used it with kvm/qemu? I
mean, it's not a particularly exotic test-case!


-- 
Giangiacomo

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Giangiacomo Mariotti]

On Mon, Jul 12, 2010 at 9:09 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
>
> This looks quite similar to a problem with ext4 and O_SYNC which I
> reported earlier but no one cared to answer (or read?) - there:
> http://permalink.gmane.org/gmane.linux.file-systems/42758
> (sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
> try a few other options, esp. cache=none and re-writing some guest
> files to verify.
>
> /mjt
>
Either way, changing to cache=none I suspect wouldn't tell me much,
because if it's as slow as before, it's still unusable and if instead
it's even slower, well it'd be even more unusable, so I wouldn't be
able to tell the difference. What I can say for certain is that with
the exact same virtual hd file, same options, same system, but on an
ext3 fs there's no problem at all, on a Btrfs is not just slower, it
takes ages.

-- 
Giangiacomo

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Michael Tokarev]

Giangiacomo Mariotti wrote:
> On Mon, Jul 12, 2010 at 9:09 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
>> This looks quite similar to a problem with ext4 and O_SYNC which I
>> reported earlier but no one cared to answer (or read?) - there:
>> http://permalink.gmane.org/gmane.linux.file-systems/42758
>> (sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
>> try a few other options, esp. cache=none and re-writing some guest
>> files to verify.
>>
>> /mjt
>>
> Either way, changing to cache=none I suspect wouldn't tell me much,
> because if it's as slow as before, it's still unusable and if instead
> it's even slower, well it'd be even more unusable, so I wouldn't be
> able to tell the difference.

Actually it's not that simple.

>     What I can say for certain is that with
> the exact same virtual hd file, same options, same system, but on an
> ext3 fs there's no problem at all, on a Btrfs is not just slower, it
> takes ages.

It is exactly the same with ext4 vs ext3.  But only on metadata-intensitive
operations (for qcow2 image).  Once you allocate space, it becomes fast,
and _especially_ fast with cache=none.  Actually, it looks like O_SYNC
(default cache mode) is _slower_ on ext4 than O_DIRECT (cache=none).

(And yes, I know O_DIRECT does NOT imply O_SYNC and vise versa).

/mjt

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Josef Bacik]

On Mon, Jul 12, 2010 at 03:34:44PM +0200, Giangiacomo Mariotti wrote:
> On Mon, Jul 12, 2010 at 9:09 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> >
> > This looks quite similar to a problem with ext4 and O_SYNC which I
> > reported earlier but no one cared to answer (or read?) - there:
> > http://permalink.gmane.org/gmane.linux.file-systems/42758
> > (sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
> > try a few other options, esp. cache=none and re-writing some guest
> > files to verify.
> >
> > /mjt
> >
> Either way, changing to cache=none I suspect wouldn't tell me much,
> because if it's as slow as before, it's still unusable and if instead
> it's even slower, well it'd be even more unusable, so I wouldn't be
> able to tell the difference. What I can say for certain is that with
> the exact same virtual hd file, same options, same system, but on an
> ext3 fs there's no problem at all, on a Btrfs is not just slower, it
> takes ages.
>

O_DIRECT support was just introduced recently, please try on the latest kernel
with the normal settings (which IIRC uses O_DIRECT), that should make things
suck alot less.  Thanks,

Josef 

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Michael Tokarev]

Josef Bacik wrote:
[]
> O_DIRECT support was just introduced recently, please try on the latest kernel
> with the normal settings (which IIRC uses O_DIRECT), that should make things
> suck alot less.  Thanks,

Um.  Do you mean it were introduced in BTRFS or general? :)

Because, wel, O_DIRECT is here and supported since some 2.2 times... ;)

/mjt

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Josef Bacik]

On Mon, Jul 12, 2010 at 05:42:04PM +0400, Michael Tokarev wrote:
> Josef Bacik wrote:
> []
> > O_DIRECT support was just introduced recently, please try on the latest kernel
> > with the normal settings (which IIRC uses O_DIRECT), that should make things
> > suck alot less.  Thanks,
> 
> Um.  Do you mean it were introduced in BTRFS or general? :)
> 
> Because, wel, O_DIRECT is here and supported since some 2.2 times... ;)

Btrfs obviously.

Josef

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Giangiacomo Mariotti]

On Mon, Jul 12, 2010 at 3:43 PM, Josef Bacik <josef@redhat.com> wrote:
>
> O_DIRECT support was just introduced recently, please try on the latest kernel
> with the normal settings (which IIRC uses O_DIRECT), that should make things
> suck alot less.  Thanks,
>
> Josef
>
With latest kernel do you mean the current Linus' git tree? Because if
instead you're talking about the current stable kernel, that's the one
I used on my test.

-- 
Giangiacomo

[Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Josef Bacik]

On Mon, Jul 12, 2010 at 10:23:14PM +0200, Giangiacomo Mariotti wrote:
> On Mon, Jul 12, 2010 at 3:43 PM, Josef Bacik <josef@redhat.com> wrote:
> >
> > O_DIRECT support was just introduced recently, please try on the latest kernel
> > with the normal settings (which IIRC uses O_DIRECT), that should make things
> > suck alot less.  Thanks,
> >
> > Josef
> >
> With latest kernel do you mean the current Linus' git tree? Because if
> instead you're talking about the current stable kernel, that's the one
> I used on my test.
> 

Yes Linus' git tree.  Thanks,

Josef

Re: [Qemu-devel] Re: BTRFS: Unbelievably slow with kvm/qemu

[Kevin Wolf]

Am 12.07.2010 15:43, schrieb Josef Bacik:
> On Mon, Jul 12, 2010 at 03:34:44PM +0200, Giangiacomo Mariotti wrote:
>> On Mon, Jul 12, 2010 at 9:09 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
>>>
>>> This looks quite similar to a problem with ext4 and O_SYNC which I
>>> reported earlier but no one cared to answer (or read?) - there:
>>> http://permalink.gmane.org/gmane.linux.file-systems/42758
>>> (sent to qemu-devel and linux-fsdevel lists - Cc'd too).  You can
>>> try a few other options, esp. cache=none and re-writing some guest
>>> files to verify.
>>>
>>> /mjt
>>>
>> Either way, changing to cache=none I suspect wouldn't tell me much,
>> because if it's as slow as before, it's still unusable and if instead
>> it's even slower, well it'd be even more unusable, so I wouldn't be
>> able to tell the difference. What I can say for certain is that with
>> the exact same virtual hd file, same options, same system, but on an
>> ext3 fs there's no problem at all, on a Btrfs is not just slower, it
>> takes ages.
>>
> 
> O_DIRECT support was just introduced recently, please try on the latest kernel
> with the normal settings (which IIRC uses O_DIRECT), that should make things
> suck alot less. 

IIUC, he uses the default cache option of qemu, which is
cache=writethrough and maps to O_DSYNC without O_DIRECT. O_DIRECT would
only be used for cache=none.

Kevin