[Qemu-devel] EXT3-fs error on Git HEAD

[Qemu-devel] EXT3-fs error on Git HEAD

[Erik de Castro Lopo]

Hi all,

I have a number of Qemu VMs. Over the weekend I updated to the latest
Git head and now two of my VMs (one i386 and one x86-64) are having
disk errors. Strange thing is, my arm VM is working flawlessly.

In the x86-64 VM I get errors like this:

EXT3-fs error (device sda1): ext3_xattr_block_get: inode 405992: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 405989: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 405986: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 405998: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 428975: bad block 244347
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 389303: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 290747: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 290762: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 239599: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 389850: bad block 64
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 389282: bad block 8
EXT3-fs error (device sda1): ext3_xattr_block_get: inode 230699: bad block 8

Any clues?

Cheers,
Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

Re: [Qemu-devel] EXT3-fs error on Git HEAD

[Erik de Castro Lopo]

Erik de Castro Lopo wrote:

> I have a number of Qemu VMs. Over the weekend I updated to the latest
> Git head and now two of my VMs (one i386 and one x86-64) are having
> disk errors. Strange thing is, my arm VM is working flawlessly.

These VM disk images are all qcow. The i386 and x86-64 ones I converted
to qcow2 and they now work fine. The arm one is still qcow and is still
working fine.

Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

Re: [Qemu-devel] EXT3-fs error on Git HEAD

[Filip Navara]

On Mon, Jun 29, 2009 at 5:42 AM, Erik de Castro
Lopo<mle+tools@mega-nerd.com> wrote:
> Erik de Castro Lopo wrote:
>
>> I have a number of Qemu VMs. Over the weekend I updated to the latest
>> Git head and now two of my VMs (one i386 and one x86-64) are having
>> disk errors. Strange thing is, my arm VM is working flawlessly.
>
> These VM disk images are all qcow. The i386 and x86-64 ones I converted
> to qcow2 and they now work fine. The arm one is still qcow and is still
> working fine.

Weird. I thought it could be related to the bug fixed by "[PATCH]
block: Clean up after deleting BHs" which is not in GIT yet afaik.

http://lists.gnu.org/archive/html/qemu-devel/2009-06/msg01661.html

Best regards,
Filip Navara

Re: [Qemu-devel] EXT3-fs error on Git HEAD

[Frank Arnold]

Am Montag, den 29.06.2009, 13:42 +1000 schrieb Erik de Castro Lopo:
> Erik de Castro Lopo wrote:
> 
> > I have a number of Qemu VMs. Over the weekend I updated to the latest
> > Git head and now two of my VMs (one i386 and one x86-64) are having
> > disk errors. Strange thing is, my arm VM is working flawlessly.
> 
> These VM disk images are all qcow. The i386 and x86-64 ones I converted
> to qcow2 and they now work fine. The arm one is still qcow and is still
> working fine.

We're seeing the same on qemu-kvm.git. I tracked it down to following
commits:

commit c16b5a2ca0b186de618654a576bdad9cdd2d1ab2
Author: Christoph Hellwig <hch@lst.de>
Date:   Mon May 25 12:37:32 2009 +0200

    fully split aio_pool from BlockDriver

commit ad53089b0d0b4bc0731d978e5713365e1a91ba74
Author: Christoph Hellwig <hch@lst.de>
Date:   Mon May 25 15:45:37 2009 +0200

    qcow: add qcow_aio_setup helper


Steps to reproduce:

1. Install some Linux distro with EXT3 file system in a qcow image.
Everything up to kvm-86-178-g54c5a33 should be working if qemu-kvm.git
is used for it.
2. Start a VM with the image on kvm-86-235-g0936eb0 (merge containing
the two commits above) or newer.
3. Run CTCS [1] or some other file system stress test on it. Usually,
after 5-10 minutes the file system on the image is dead.

[1] http://sourceforge.net/projects/va-ctcs/

Quick howto for CTCS:
 $ tar xzf ctcs-1.3.1-pre1.tgz
 $ cd ctcs-1.3.1pre1
 $ make install
 $ ./newburn 10m

-- 
Frank Arnold <frank.arnold@amd.com>
System Design Technician, Software Test
AMD Operating System Research Center
Dresden, Germany
Tel: +49 351 448 356702


Legal Information:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34
85609 Dornach b. München

Geschäftsführer: Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis München
Registergericht München, HRB Nr. 43632

Re: [Qemu-devel] EXT3-fs error on Git HEAD

[Kevin Wolf]

Frank Arnold schrieb:
> Am Montag, den 29.06.2009, 13:42 +1000 schrieb Erik de Castro Lopo:
>> Erik de Castro Lopo wrote:
>>
>>> I have a number of Qemu VMs. Over the weekend I updated to the latest
>>> Git head and now two of my VMs (one i386 and one x86-64) are having
>>> disk errors. Strange thing is, my arm VM is working flawlessly.
>> These VM disk images are all qcow. The i386 and x86-64 ones I converted
>> to qcow2 and they now work fine. The arm one is still qcow and is still
>> working fine.
> 
> We're seeing the same on qemu-kvm.git. I tracked it down to following
> commits:
> 
> commit c16b5a2ca0b186de618654a576bdad9cdd2d1ab2
> Author: Christoph Hellwig <hch@lst.de>
> Date:   Mon May 25 12:37:32 2009 +0200
> 
>     fully split aio_pool from BlockDriver
> 
> commit ad53089b0d0b4bc0731d978e5713365e1a91ba74
> Author: Christoph Hellwig <hch@lst.de>
> Date:   Mon May 25 15:45:37 2009 +0200
> 
>     qcow: add qcow_aio_setup helper

This one has an obvious bug. Does the following change help?

diff --git a/block/qcow.c b/block/qcow.c
index 55a68a6..7fc85ae 100644
--- a/block/qcow.c
+++ b/block/qcow.c
@@ -723,7 +723,7 @@ static BlockDriverAIOCB
*qcow_aio_writev(BlockDriverState *bs,

     s->cluster_cache_offset = -1; /* disable compressed cache */

-    acb = qcow_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 0);
+    acb = qcow_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 1);
     if (!acb)
         return NULL;

Kevin

Re: [Qemu-devel] EXT3-fs error on Git HEAD

[Frank Arnold]

Am Dienstag, den 30.06.2009, 11:07 +0200 schrieb Kevin Wolf:
> Frank Arnold schrieb:
> > commit ad53089b0d0b4bc0731d978e5713365e1a91ba74
> > Author: Christoph Hellwig <hch@lst.de>
> > Date:   Mon May 25 15:45:37 2009 +0200
> > 
> >     qcow: add qcow_aio_setup helper
> 
> This one has an obvious bug. Does the following change help?
> 
> diff --git a/block/qcow.c b/block/qcow.c
> index 55a68a6..7fc85ae 100644
> --- a/block/qcow.c
> +++ b/block/qcow.c
> @@ -723,7 +723,7 @@ static BlockDriverAIOCB
> *qcow_aio_writev(BlockDriverState *bs,
> 
>      s->cluster_cache_offset = -1; /* disable compressed cache */
> 
> -    acb = qcow_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 0);
> +    acb = qcow_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 1);
>      if (!acb)
>          return NULL;

Yes, this change seems to fix it. The guest survived 3 hours of stress
testing and the image was still intact.
Tested on top of kvm-87-71-g6179556 (qemu-kvm.git).

Thanks for looking into it.
-- 
Frank Arnold <frank.arnold@amd.com>
System Design Technician, Software Test
AMD Operating System Research Center
Dresden, Germany
Tel: +49 351 448 356702


Legal Information:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34
85609 Dornach b. München

Geschäftsführer: Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis München
Registergericht München, HRB Nr. 43632

Re: [Qemu-devel] EXT3-fs error on Git HEAD

[Andre Przywara]

Hi,

can someone please apply the patch below from Kevin?
I know that QCOW1 is kind of deprecated, but this should not mean that 
such serious errors stay unfixed. I have this patch in all, well: almost 
  :-( of my branches and would love to see it upstream.
And yes, we are about to convert our images to QCOW2...

Thanks!

Regards,
Andre.


Kevin Wolf wrote:
> Frank Arnold schrieb:
>> Am Montag, den 29.06.2009, 13:42 +1000 schrieb Erik de Castro Lopo:
>>> Erik de Castro Lopo wrote:
>>>
>>>> I have a number of Qemu VMs. Over the weekend I updated to the latest
>>>> Git head and now two of my VMs (one i386 and one x86-64) are having
>>>> disk errors. Strange thing is, my arm VM is working flawlessly.
>>> These VM disk images are all qcow. The i386 and x86-64 ones I converted
>>> to qcow2 and they now work fine. The arm one is still qcow and is still
>>> working fine.
>> We're seeing the same on qemu-kvm.git. I tracked it down to following
>> commits:
>>
>> commit c16b5a2ca0b186de618654a576bdad9cdd2d1ab2
>> Author: Christoph Hellwig <hch@lst.de>
>> Date:   Mon May 25 12:37:32 2009 +0200
>>
>>     fully split aio_pool from BlockDriver
>>
>> commit ad53089b0d0b4bc0731d978e5713365e1a91ba74
>> Author: Christoph Hellwig <hch@lst.de>
>> Date:   Mon May 25 15:45:37 2009 +0200
>>
>>     qcow: add qcow_aio_setup helper
> 
> This one has an obvious bug. Does the following change help?
> 
> diff --git a/block/qcow.c b/block/qcow.c
> index 55a68a6..7fc85ae 100644
> --- a/block/qcow.c
> +++ b/block/qcow.c
> @@ -723,7 +723,7 @@ static BlockDriverAIOCB
> *qcow_aio_writev(BlockDriverState *bs,
> 
>      s->cluster_cache_offset = -1; /* disable compressed cache */
> 
> -    acb = qcow_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 0);
> +    acb = qcow_aio_setup(bs, sector_num, qiov, nb_sectors, cb, opaque, 1);
>      if (!acb)
>          return NULL;
> 
> Kevin
> 
> 
> 

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448 3567 12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632