virtio-blk XFS corruption

virtio-blk XFS corruption

[Peter Lieven]

Hi all,

we experience filesystem corruption using virtio-blk on some guest systems togehter with XFS. We still use qemu-kvm 0.12.4.
Does someone remember if there has been a fix submitted meanwhile?

It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for months.

The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is the following:

[19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 0xffffffffa013091d
[19001.346897] 
[19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server #43-Ubuntu
[19002.174492] Call Trace:
[19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
[19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
[19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
[19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]
[19002.174492]  [<ffffffffa013c2e7>] xfs_vn_mknod+0xa7/0x1c0 [xfs]
[19002.174492]  [<ffffffffa013c430>] xfs_vn_create+0x10/0x20 [xfs]
[19002.174492]  [<ffffffff8114fb74>] vfs_create+0xb4/0xe0
[19002.174492]  [<ffffffff8114fc64>] __open_namei_create+0xc4/0x110
[19002.174492]  [<ffffffff8115340b>] do_filp_open+0xa6b/0xba0
[19002.174492]  [<ffffffff8115e6ea>] ? alloc_fd+0x10a/0x150
[19002.174492]  [<ffffffff81142439>] do_sys_open+0x69/0x170
[19002.174492]  [<ffffffff81142580>] sys_open+0x20/0x30
[19002.174492]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
[19002.174492] xfs_force_shutdown(vda1,0x8) called from line 1163 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Return address = 0xffffffffa012bb4e
[19002.211293] Filesystem "vda1": Corruption of in-memory data detected.  Shutting down filesystem: vda1

I will pull:
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=618fbb84299780af96e3d4c4b6f2148656fe3708
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=20a81e4d178379381fbd522eda5f664ba2ecdaaa

and see if it helps here.

Any other ideas?

Thanks,
Peter

[Qemu-devel] virtio-blk XFS corruption

[Peter Lieven]

Hi all,

we experience filesystem corruption using virtio-blk on some guest systems togehter with XFS. We still use qemu-kvm 0.12.4.
Does someone remember if there has been a fix submitted meanwhile?

It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for months.

The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is the following:

[19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 0xffffffffa013091d
[19001.346897] 
[19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server #43-Ubuntu
[19002.174492] Call Trace:
[19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
[19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
[19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
[19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]
[19002.174492]  [<ffffffffa013c2e7>] xfs_vn_mknod+0xa7/0x1c0 [xfs]
[19002.174492]  [<ffffffffa013c430>] xfs_vn_create+0x10/0x20 [xfs]
[19002.174492]  [<ffffffff8114fb74>] vfs_create+0xb4/0xe0
[19002.174492]  [<ffffffff8114fc64>] __open_namei_create+0xc4/0x110
[19002.174492]  [<ffffffff8115340b>] do_filp_open+0xa6b/0xba0
[19002.174492]  [<ffffffff8115e6ea>] ? alloc_fd+0x10a/0x150
[19002.174492]  [<ffffffff81142439>] do_sys_open+0x69/0x170
[19002.174492]  [<ffffffff81142580>] sys_open+0x20/0x30
[19002.174492]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
[19002.174492] xfs_force_shutdown(vda1,0x8) called from line 1163 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Return address = 0xffffffffa012bb4e
[19002.211293] Filesystem "vda1": Corruption of in-memory data detected.  Shutting down filesystem: vda1

I will pull:
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=618fbb84299780af96e3d4c4b6f2148656fe3708
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=20a81e4d178379381fbd522eda5f664ba2ecdaaa

and see if it helps here.

Any other ideas?

Thanks,
Peter

Re: virtio-blk XFS corruption

[Stefan Hajnoczi]

On Sat, Sep 25, 2010 at 2:43 PM, Peter Lieven <pl@dlh.net> wrote:
> we experience filesystem corruption using virtio-blk on some guest systems togehter with XFS. We still use qemu-kvm 0.12.4.
[...]
> It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
> an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for months.

Affected guests: 64-bit Ubuntu LTS 10.04.1, openSuse 11.1 2.6.27.45-0.1-pae
Unaffected guests: openSuse 11.1 2.6.27.45-0.1-default
qemu-kvm version: 0.12.4
qemu-kvm command-line: ?
Disk image format: ?
Steps to reproduce: ?

Can you please provide information on the unknown items above?

Does this happen with IDE (-drive file=myimage.img,if=ide) or only
with virtio-blk?

> The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is the following:
>
> [19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 0xffffffffa013091d
> [19001.346897]
> [19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server #43-Ubuntu
> [19002.174492] Call Trace:
> [19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
> [19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
> [19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
> [19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]

XFS has given up because of an error in xfs_create() but I don't think
this output gives enough information to determine what the error was.

Stefan

[Qemu-devel] Re: virtio-blk XFS corruption

[Stefan Hajnoczi]

On Sat, Sep 25, 2010 at 2:43 PM, Peter Lieven <pl@dlh.net> wrote:
> we experience filesystem corruption using virtio-blk on some guest systems togehter with XFS. We still use qemu-kvm 0.12.4.
[...]
> It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
> an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for months.

Affected guests: 64-bit Ubuntu LTS 10.04.1, openSuse 11.1 2.6.27.45-0.1-pae
Unaffected guests: openSuse 11.1 2.6.27.45-0.1-default
qemu-kvm version: 0.12.4
qemu-kvm command-line: ?
Disk image format: ?
Steps to reproduce: ?

Can you please provide information on the unknown items above?

Does this happen with IDE (-drive file=myimage.img,if=ide) or only
with virtio-blk?

> The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is the following:
>
> [19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 0xffffffffa013091d
> [19001.346897]
> [19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server #43-Ubuntu
> [19002.174492] Call Trace:
> [19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
> [19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
> [19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
> [19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]

XFS has given up because of an error in xfs_create() but I don't think
this output gives enough information to determine what the error was.

Stefan

Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 16:44 schrieb Stefan Hajnoczi:

> On Sat, Sep 25, 2010 at 2:43 PM, Peter Lieven <pl@dlh.net> wrote:
>> we experience filesystem corruption using virtio-blk on some guest systems togehter with XFS. We still use qemu-kvm 0.12.4.
> [...]
>> It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
>> an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for months.
> 
> Affected guests: 64-bit Ubuntu LTS 10.04.1, openSuse 11.1 2.6.27.45-0.1-pae
> Unaffected guests: openSuse 11.1 2.6.27.45-0.1-default
> qemu-kvm version: 0.12.4
> qemu-kvm command-line: ?
/usr/bin/qemu-kvm-0.12.4  -net tap,vlan=140,script=no,downscript=no,ifname=tap6 -net nic,vlan=140,model=e1000,macaddr=52:54:00:fe:00:ee   -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-7c5e1ab04-c6f7a1269cc4c99d-ubuntu-newsfeed-system,if=ide,boot=on,cache=none,aio=native  -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-a1e4ce107-9c6000e78ec4c988-ubuntu-newsfeed-data,if=scsi,boot=off,cache=none,aio=native  -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-fb34ce107-9ff000e78ef4c98a-ubuntu-newsfeed-data2,if=virtio,boot=off,cache=none,aio=native  -m 4096 -smp 4 -monitor tcp:0:4012,server,nowait -vnc :12 -name 'ubuntu-newsfeed'  -boot order=dc,menu=off  -k de  -pidfile /var/run/qemu/vm-244.pid  -mem-pat
 h /hugepages -mem-prealloc  -cpu qemu64,model_id='Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz',-tsc,-nx  -rtc base=utc -no-hpet -no-kvm-clock 

> Disk image format: ?

raw host_device (iscsi san via open-iscsi and multipathd)

> Steps to reproduce: ?

we encounter this problem only when running a diablo newsserver with heavy i/o. however, i will try to construct a test scenario that can
be reproduced.

might this bug related to the barrier fix for virtio_blk that has been introduced in 0.12.5. ?

> 
> Can you please provide information on the unknown items above?
> 
> Does this happen with IDE (-drive file=myimage.img,if=ide) or only
> with virtio-blk?

if i take the same vm and change the emulation to scsi the system works without any problems.

> 
>> The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is the following:
>> 
>> [19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 0xffffffffa013091d
>> [19001.346897]
>> [19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server #43-Ubuntu
>> [19002.174492] Call Trace:
>> [19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
>> [19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
>> [19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
>> [19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]
> 
> XFS has given up because of an error in xfs_create() but I don't think
> this output gives enough information to determine what the error was.

ok, thank you. here is the log output from the opensuse system:

Sep 20 15:44:04 newsfeed2-neu kernel: ------------[ cut here ]------------
Sep 20 15:44:04 newsfeed2-neu kernel: WARNING: at fs/inode.c:602 unlock_new_inode+0x29/0x49()
Sep 20 15:44:04 newsfeed2-neu kernel: Modules linked in: af_packet nls_utf8 joydev st ide_disk ide_cd_mod binfmt_misc ipv6 fuse xfs loop dm_mod virtio_blk i2c_piix4 virtio_pci sr_mod ppdev e1000 rtc_cmos pcspkr i2c_core virtio_ring cdrom button rtc_core virtio parport_pc rtc_lib parport floppy sg sd_mod crc_t10dif edd ext3 mbcache jbd fan ide_pci_generic piix ide_core ata_generic ata_piix libata scsi_mod dock thermal processor thermal_sys hwmon [last unloaded: speedstep_lib]
Sep 20 15:44:04 newsfeed2-neu kernel: Supported: No, Unsupported modules are loaded
Sep 20 15:44:04 newsfeed2-neu kernel: Pid: 14732, comm: diablo Tainted: G        W 2.6.27.45-0.1-pae #1
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c010622c>] dump_trace+0x6b/0x249
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0106d03>] show_trace+0x20/0x39
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0349303>] dump_stack+0x71/0x76
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0129d50>] warn_on_slowpath+0x4d/0x70
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c01a324d>] unlock_new_inode+0x29/0x49
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f75148>] xfs_ialloc+0x516/0x52c [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f88519>] xfs_dir_ialloc+0x7f/0x27a [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f8b501>] xfs_create+0x2be/0x549 [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f9545c>] xfs_vn_mknod+0x138/0x205 [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0199c9c>] vfs_create+0x12e/0x1a2
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c019c102>] do_filp_open+0x1b8/0x6ee
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0190c17>] do_sys_open+0x46/0xcc
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0190ceb>] sys_open+0x23/0x28
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0104ad2>] syscall_call+0x7/0xb
Sep 20 15:44:04 newsfeed2-neu kernel:  [<ffffe422>] 0xffffe422
Sep 20 15:44:04 newsfeed2-neu kernel:  =======================
Sep 20 15:44:04 newsfeed2-neu kernel: ---[ end trace b8b179fad244f381 ]---


> 
> Stefan

thanks,
peter

[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 16:44 schrieb Stefan Hajnoczi:

> On Sat, Sep 25, 2010 at 2:43 PM, Peter Lieven <pl@dlh.net> wrote:
>> we experience filesystem corruption using virtio-blk on some guest systems togehter with XFS. We still use qemu-kvm 0.12.4.
> [...]
>> It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
>> an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for months.
> 
> Affected guests: 64-bit Ubuntu LTS 10.04.1, openSuse 11.1 2.6.27.45-0.1-pae
> Unaffected guests: openSuse 11.1 2.6.27.45-0.1-default
> qemu-kvm version: 0.12.4
> qemu-kvm command-line: ?
/usr/bin/qemu-kvm-0.12.4  -net tap,vlan=140,script=no,downscript=no,ifname=tap6 -net nic,vlan=140,model=e1000,macaddr=52:54:00:fe:00:ee   -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-7c5e1ab04-c6f7a1269cc4c99d-ubuntu-newsfeed-system,if=ide,boot=on,cache=none,aio=native  -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-a1e4ce107-9c6000e78ec4c988-ubuntu-newsfeed-data,if=scsi,boot=off,cache=none,aio=native  -drive format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-fb34ce107-9ff000e78ef4c98a-ubuntu-newsfeed-data2,if=virtio,boot=off,cache=none,aio=native  -m 4096 -smp 4 -monitor tcp:0:4012,server,nowait -vnc :12 -name 'ubuntu-newsfeed'  -boot order=dc,menu=off  -k de  -pidfile /var/run/qemu/vm-244.pid  -mem-path /hugepages -mem-prealloc  -cpu qemu64,model_id='Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz',-tsc,-nx  -rtc base=utc -no-hpet -no-kvm-clock 

> Disk image format: ?

raw host_device (iscsi san via open-iscsi and multipathd)

> Steps to reproduce: ?

we encounter this problem only when running a diablo newsserver with heavy i/o. however, i will try to construct a test scenario that can
be reproduced.

might this bug related to the barrier fix for virtio_blk that has been introduced in 0.12.5. ?

> 
> Can you please provide information on the unknown items above?
> 
> Does this happen with IDE (-drive file=myimage.img,if=ide) or only
> with virtio-blk?

if i take the same vm and change the emulation to scsi the system works without any problems.

> 
>> The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is the following:
>> 
>> [19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 0xffffffffa013091d
>> [19001.346897]
>> [19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server #43-Ubuntu
>> [19002.174492] Call Trace:
>> [19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
>> [19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
>> [19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
>> [19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]
> 
> XFS has given up because of an error in xfs_create() but I don't think
> this output gives enough information to determine what the error was.

ok, thank you. here is the log output from the opensuse system:

Sep 20 15:44:04 newsfeed2-neu kernel: ------------[ cut here ]------------
Sep 20 15:44:04 newsfeed2-neu kernel: WARNING: at fs/inode.c:602 unlock_new_inode+0x29/0x49()
Sep 20 15:44:04 newsfeed2-neu kernel: Modules linked in: af_packet nls_utf8 joydev st ide_disk ide_cd_mod binfmt_misc ipv6 fuse xfs loop dm_mod virtio_blk i2c_piix4 virtio_pci sr_mod ppdev e1000 rtc_cmos pcspkr i2c_core virtio_ring cdrom button rtc_core virtio parport_pc rtc_lib parport floppy sg sd_mod crc_t10dif edd ext3 mbcache jbd fan ide_pci_generic piix ide_core ata_generic ata_piix libata scsi_mod dock thermal processor thermal_sys hwmon [last unloaded: speedstep_lib]
Sep 20 15:44:04 newsfeed2-neu kernel: Supported: No, Unsupported modules are loaded
Sep 20 15:44:04 newsfeed2-neu kernel: Pid: 14732, comm: diablo Tainted: G        W 2.6.27.45-0.1-pae #1
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c010622c>] dump_trace+0x6b/0x249
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0106d03>] show_trace+0x20/0x39
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0349303>] dump_stack+0x71/0x76
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0129d50>] warn_on_slowpath+0x4d/0x70
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c01a324d>] unlock_new_inode+0x29/0x49
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f75148>] xfs_ialloc+0x516/0x52c [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f88519>] xfs_dir_ialloc+0x7f/0x27a [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f8b501>] xfs_create+0x2be/0x549 [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f9545c>] xfs_vn_mknod+0x138/0x205 [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0199c9c>] vfs_create+0x12e/0x1a2
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c019c102>] do_filp_open+0x1b8/0x6ee
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0190c17>] do_sys_open+0x46/0xcc
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0190ceb>] sys_open+0x23/0x28
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0104ad2>] syscall_call+0x7/0xb
Sep 20 15:44:04 newsfeed2-neu kernel:  [<ffffe422>] 0xffffe422
Sep 20 15:44:04 newsfeed2-neu kernel:  =======================
Sep 20 15:44:04 newsfeed2-neu kernel: ---[ end trace b8b179fad244f381 ]---


> 
> Stefan

thanks,
peter

Re: virtio-blk XFS corruption

[Christoph Hellwig]

FYI, qemu 0.12.2 is missing:

	block: fix sector comparism in multiwrite_req_compare

which in the past was very good at triggering XFS guest corruption.
Please try with the patch applied or even better latests qemu from git.

[Qemu-devel] Re: virtio-blk XFS corruption

[Christoph Hellwig]

FYI, qemu 0.12.2 is missing:

	block: fix sector comparism in multiwrite_req_compare

which in the past was very good at triggering XFS guest corruption.
Please try with the patch applied or even better latests qemu from git.

Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:

> FYI, qemu 0.12.2 is missing:

you mean 0.12.4 not 0.12.2, don't you?

> 
> 	block: fix sector comparism in multiwrite_req_compare
> 
> which in the past was very good at triggering XFS guest corruption.
> Please try with the patch applied or even better latests qemu from git.
> 

i'm just trying with 0.12.5. 

i'm not so familiar with git. is there a command to pull only patches
that are marked as stable and will be in the next official release?

thanks,
peter

[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:

> FYI, qemu 0.12.2 is missing:

you mean 0.12.4 not 0.12.2, don't you?

> 
> 	block: fix sector comparism in multiwrite_req_compare
> 
> which in the past was very good at triggering XFS guest corruption.
> Please try with the patch applied or even better latests qemu from git.
> 

i'm just trying with 0.12.5. 

i'm not so familiar with git. is there a command to pull only patches
that are marked as stable and will be in the next official release?

thanks,
peter

Re: virtio-blk XFS corruption

[Christoph Hellwig]

On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
> 
> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
> 
> > FYI, qemu 0.12.2 is missing:
> 
> you mean 0.12.4 not 0.12.2, don't you?

Yes, sorry. (but 0.12.2 is of course missing it, too..)

> > which in the past was very good at triggering XFS guest corruption.
> > Please try with the patch applied or even better latests qemu from git.
> > 
> 
> i'm just trying with 0.12.5. 
> 
> i'm not so familiar with git. is there a command to pull only patches
> that are marked as stable and will be in the next official release?

All the qemu stable releases are tagged and you can check do

	git-checkout v0.12.5

but that's not the main git HEAD which would also be interesting.

[Qemu-devel] Re: virtio-blk XFS corruption

[Christoph Hellwig]

On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
> 
> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
> 
> > FYI, qemu 0.12.2 is missing:
> 
> you mean 0.12.4 not 0.12.2, don't you?

Yes, sorry. (but 0.12.2 is of course missing it, too..)

> > which in the past was very good at triggering XFS guest corruption.
> > Please try with the patch applied or even better latests qemu from git.
> > 
> 
> i'm just trying with 0.12.5. 
> 
> i'm not so familiar with git. is there a command to pull only patches
> that are marked as stable and will be in the next official release?

All the qemu stable releases are tagged and you can check do

	git-checkout v0.12.5

but that's not the main git HEAD which would also be interesting.

Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:

> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>> 
>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>> 
>>> FYI, qemu 0.12.2 is missing:
>> 
>> you mean 0.12.4 not 0.12.2, don't you?
> 
> Yes, sorry. (but 0.12.2 is of course missing it, too..)

sure ;-)

> 
>>> which in the past was very good at triggering XFS guest corruption.
>>> Please try with the patch applied or even better latests qemu from git.
>>> 
>> 
>> i'm just trying with 0.12.5. 
>> 
>> i'm not so familiar with git. is there a command to pull only patches
>> that are marked as stable and will be in the next official release?
> 
> All the qemu stable releases are tagged and you can check do
> 
> 	git-checkout v0.12.5
> 
> but that's not the main git HEAD which would also be interesting.
> 

thank you, i will first check with v0.12.5 and then later try git HEAD if it does not help.

while trying to start the vm with a ubuntu 10.04 lts desktop cd to repair the corrupted xfs i got the following emulation error with
v0.12.5. is this something i should worry about?

KVM internal error. Suberror: 1
rax 0000000000000100 rbx 0000000000000286 rcx 0000000000000000 rdx 0000000000000000
rsi 0000000000000286 rdi ffff88011926a6ec rsp ffff880116dc5e98 rbp ffff880116dc5e98
r8  00007f0261f2b253 r9  0000000000000000 r10 00007f025fad8e90 r11 0000000000000206
r12 ffff88011926a6e8 r13 00007f025fb0a480 r14 00007fff7428ea10 r15 ffff880116df16f0
rip ffffffff81039719 rflags 00010086
cs 0010 (00000000/ffffffff p 1 dpl 0 db 0 s 1 type b l 1 g 1 avl 0)
ds 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
es 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
ss 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
fs 0000 (7f02602047a0/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gs 0000 (ffff880028280000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
tr 0040 (ffff880028293780/00002087 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gdt ffff880028284000/7f
idt ffffffff81941000/fff
cr0 80050033 cr2 7f025fb0a480 cr3 11aebb000 cr4 6e0 cr8 0 efer 501
emulation failure, check dmesg for details

This reproducible happens if I try to boot this VM via CD. dmesg has no relevant entries...

[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:

> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>> 
>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>> 
>>> FYI, qemu 0.12.2 is missing:
>> 
>> you mean 0.12.4 not 0.12.2, don't you?
> 
> Yes, sorry. (but 0.12.2 is of course missing it, too..)

sure ;-)

> 
>>> which in the past was very good at triggering XFS guest corruption.
>>> Please try with the patch applied or even better latests qemu from git.
>>> 
>> 
>> i'm just trying with 0.12.5. 
>> 
>> i'm not so familiar with git. is there a command to pull only patches
>> that are marked as stable and will be in the next official release?
> 
> All the qemu stable releases are tagged and you can check do
> 
> 	git-checkout v0.12.5
> 
> but that's not the main git HEAD which would also be interesting.
> 

thank you, i will first check with v0.12.5 and then later try git HEAD if it does not help.

while trying to start the vm with a ubuntu 10.04 lts desktop cd to repair the corrupted xfs i got the following emulation error with
v0.12.5. is this something i should worry about?

KVM internal error. Suberror: 1
rax 0000000000000100 rbx 0000000000000286 rcx 0000000000000000 rdx 0000000000000000
rsi 0000000000000286 rdi ffff88011926a6ec rsp ffff880116dc5e98 rbp ffff880116dc5e98
r8  00007f0261f2b253 r9  0000000000000000 r10 00007f025fad8e90 r11 0000000000000206
r12 ffff88011926a6e8 r13 00007f025fb0a480 r14 00007fff7428ea10 r15 ffff880116df16f0
rip ffffffff81039719 rflags 00010086
cs 0010 (00000000/ffffffff p 1 dpl 0 db 0 s 1 type b l 1 g 1 avl 0)
ds 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
es 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
ss 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
fs 0000 (7f02602047a0/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gs 0000 (ffff880028280000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
tr 0040 (ffff880028293780/00002087 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gdt ffff880028284000/7f
idt ffffffff81941000/fff
cr0 80050033 cr2 7f025fb0a480 cr3 11aebb000 cr4 6e0 cr8 0 efer 501
emulation failure, check dmesg for details

This reproducible happens if I try to boot this VM via CD. dmesg has no relevant entries...

Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:

> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>> 
>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>> 
>>> FYI, qemu 0.12.2 is missing:
>> 
>> you mean 0.12.4 not 0.12.2, don't you?
> 
> Yes, sorry. (but 0.12.2 is of course missing it, too..)
> 
>>> which in the past was very good at triggering XFS guest corruption.
>>> Please try with the patch applied or even better latests qemu from git.
>>> 
>> 
>> i'm just trying with 0.12.5. 
>> 
>> i'm not so familiar with git. is there a command to pull only patches
>> that are marked as stable and will be in the next official release?
> 
> All the qemu stable releases are tagged and you can check do
> 
> 	git-checkout v0.12.5
> 
> but that's not the main git HEAD which would also be interesting.
> 

with v0.12.5 no xfs error, but the machine hangs after a few minutes...

(gdb)  thread apply all backtrace full

Thread 5 (Thread 0x7f15131c7950 (LWP 3579)):
#0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
No symbol table info available.
#1  0x000000000042b9f1 in kvm_run (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
	r = 0
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c6f000
	fd = 16
#2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
#4  0x000000000042d819 in ap_main_loop (_env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d4cef0
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (Thread 0x7f15129c6950 (LWP 3580)):
#0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
No symbol table info available.
#1  0x000000000042b9f1 in kvm_run (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
	r = 0
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c6c000
	fd = 17
#2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
#4  0x000000000042d819 in ap_main_loop (_env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d67010
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x7f15121c5950 (LWP 3581)):
#0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
No locals.
#4  0x000000000042ba68 in kvm_run (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
	r = -1
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c69000
	fd = 18
#5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
#7  0x000000000042d819 in ap_main_loop (_env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d74d90
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (Thread 0x7f15119c4950 (LWP 3582)):
#0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
No locals.
#4  0x000000000042ba68 in kvm_run (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
	r = -1
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c66000
	fd = 19
#5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
---Type <return> to continue, or q <return> to quit---
#7  0x000000000042d819 in ap_main_loop (_env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d82b10
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (Thread 0x7f1514c766f0 (LWP 3576)):
#0  0x00007f15136c2742 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x000000000040c25a in main_loop_wait (timeout=1000) at /usr/src/qemu-kvm-0.12.5/vl.c:3994
	ioh = (IOHandlerRecord *) 0x0
	rfds = {fds_bits = {83886728, 0 <repeats 15 times>}}
	wfds = {fds_bits = {0 <repeats 16 times>}}
	xfds = {fds_bits = {0 <repeats 16 times>}}
	ret = 3
	nfds = 26
	tv = {tv_sec = 0, tv_usec = 999994}
#2  0x000000000042dd9d in kvm_main_loop () at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:2126
	fds = {24, 25}
	mask = {__val = {268443712, 0 <repeats 15 times>}}
	sigfd = 26
#3  0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.5/vl.c:4212
	r = 0
#4  0x000000000041055a in main (argc=39, argv=0x7fffe23b2838, envp=0x7fffe23b2978) at /usr/src/qemu-kvm-0.12.5/vl.c:6256
	gdbstub_dev = 0x0
	boot_devices_bitmap = 12
	i = 0
	snapshot = 0
	linux_boot = 0
	initrd_filename = 0x0
	kernel_filename = 0x0
	kernel_cmdline = 0x58958c ""
	boot_devices = "dc", '\0' <repeats 30 times>
	ds = (DisplayState *) 0x1d9ff00
	dcl = (DisplayChangeListener *) 0x0
	cyls = 0
	heads = 0
	secs = 0
	translation = 0
	hda_opts = (QemuOpts *) 0x0
	opts = (QemuOpts *) 0x1d2adc0
	optind = 39
---Type <return> to continue, or q <return> to quit---
	r = 0x7fffe23b4bf7 "-no-kvm-clock"
	optarg = 0x0
	loadvm = 0x0
	machine = (QEMUMachine *) 0x862720
	cpu_model = 0x7fffe23b4b80 "qemu64,model_id=Intel(R) Xeon(R) CPU", ' ' <repeats 11 times>, "E5430  @ 2.66GHz,-tsc,-nx"
	fds = {-499439560, 32767}
	tb_size = 0
	pid_file = 0x7fffe23b4b3f "/var/run/qemu/vm-244.pid"
	incoming = 0x0
	fd = 0
	pwd = (struct passwd *) 0x0
	chroot_dir = 0x0
	run_as = 0x0
	env = (struct CPUX86State *) 0x0
	show_vnc_port = 0
	params = {0x58d2a6 "order", 0x58d2ac "once", 0x58d2b1 "menu", 0x0}

[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:

> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>> 
>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>> 
>>> FYI, qemu 0.12.2 is missing:
>> 
>> you mean 0.12.4 not 0.12.2, don't you?
> 
> Yes, sorry. (but 0.12.2 is of course missing it, too..)
> 
>>> which in the past was very good at triggering XFS guest corruption.
>>> Please try with the patch applied or even better latests qemu from git.
>>> 
>> 
>> i'm just trying with 0.12.5. 
>> 
>> i'm not so familiar with git. is there a command to pull only patches
>> that are marked as stable and will be in the next official release?
> 
> All the qemu stable releases are tagged and you can check do
> 
> 	git-checkout v0.12.5
> 
> but that's not the main git HEAD which would also be interesting.
> 

with v0.12.5 no xfs error, but the machine hangs after a few minutes...

(gdb)  thread apply all backtrace full

Thread 5 (Thread 0x7f15131c7950 (LWP 3579)):
#0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
No symbol table info available.
#1  0x000000000042b9f1 in kvm_run (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
	r = 0
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c6f000
	fd = 16
#2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
#4  0x000000000042d819 in ap_main_loop (_env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d4cef0
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (Thread 0x7f15129c6950 (LWP 3580)):
#0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
No symbol table info available.
#1  0x000000000042b9f1 in kvm_run (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
	r = 0
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c6c000
	fd = 17
#2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
#4  0x000000000042d819 in ap_main_loop (_env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d67010
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x7f15121c5950 (LWP 3581)):
#0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
No locals.
#4  0x000000000042ba68 in kvm_run (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
	r = -1
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c69000
	fd = 18
#5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
#7  0x000000000042d819 in ap_main_loop (_env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d74d90
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (Thread 0x7f15119c4950 (LWP 3582)):
#0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
No symbol table info available.
#1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
No symbol table info available.
#2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
No locals.
#4  0x000000000042ba68 in kvm_run (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
	r = -1
	kvm = (kvm_context_t) 0x1d2b7b0
	run = (struct kvm_run *) 0x7f1514c66000
	fd = 19
#5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
	r = 0
#6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
	run_cpu = 1
---Type <return> to continue, or q <return> to quit---
#7  0x000000000042d819 in ap_main_loop (_env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
	env = (struct CPUX86State *) 0x1d82b10
	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
	data = (struct ioperm_data *) 0x0
#8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (Thread 0x7f1514c766f0 (LWP 3576)):
#0  0x00007f15136c2742 in select () from /lib/libc.so.6
No symbol table info available.
#1  0x000000000040c25a in main_loop_wait (timeout=1000) at /usr/src/qemu-kvm-0.12.5/vl.c:3994
	ioh = (IOHandlerRecord *) 0x0
	rfds = {fds_bits = {83886728, 0 <repeats 15 times>}}
	wfds = {fds_bits = {0 <repeats 16 times>}}
	xfds = {fds_bits = {0 <repeats 16 times>}}
	ret = 3
	nfds = 26
	tv = {tv_sec = 0, tv_usec = 999994}
#2  0x000000000042dd9d in kvm_main_loop () at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:2126
	fds = {24, 25}
	mask = {__val = {268443712, 0 <repeats 15 times>}}
	sigfd = 26
#3  0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.5/vl.c:4212
	r = 0
#4  0x000000000041055a in main (argc=39, argv=0x7fffe23b2838, envp=0x7fffe23b2978) at /usr/src/qemu-kvm-0.12.5/vl.c:6256
	gdbstub_dev = 0x0
	boot_devices_bitmap = 12
	i = 0
	snapshot = 0
	linux_boot = 0
	initrd_filename = 0x0
	kernel_filename = 0x0
	kernel_cmdline = 0x58958c ""
	boot_devices = "dc", '\0' <repeats 30 times>
	ds = (DisplayState *) 0x1d9ff00
	dcl = (DisplayChangeListener *) 0x0
	cyls = 0
	heads = 0
	secs = 0
	translation = 0
	hda_opts = (QemuOpts *) 0x0
	opts = (QemuOpts *) 0x1d2adc0
	optind = 39
---Type <return> to continue, or q <return> to quit---
	r = 0x7fffe23b4bf7 "-no-kvm-clock"
	optarg = 0x0
	loadvm = 0x0
	machine = (QEMUMachine *) 0x862720
	cpu_model = 0x7fffe23b4b80 "qemu64,model_id=Intel(R) Xeon(R) CPU", ' ' <repeats 11 times>, "E5430  @ 2.66GHz,-tsc,-nx"
	fds = {-499439560, 32767}
	tb_size = 0
	pid_file = 0x7fffe23b4b3f "/var/run/qemu/vm-244.pid"
	incoming = 0x0
	fd = 0
	pwd = (struct passwd *) 0x0
	chroot_dir = 0x0
	run_as = 0x0
	env = (struct CPUX86State *) 0x0
	show_vnc_port = 0
	params = {0x58d2a6 "order", 0x58d2ac "once", 0x58d2b1 "menu", 0x0}

Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 20:11 schrieb Peter Lieven:

> 
> Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:
> 
>> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>>> 
>>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>>> 
>>>> FYI, qemu 0.12.2 is missing:
>>> 
>>> you mean 0.12.4 not 0.12.2, don't you?
>> 
>> Yes, sorry. (but 0.12.2 is of course missing it, too..)
>> 
>>>> which in the past was very good at triggering XFS guest corruption.
>>>> Please try with the patch applied or even better latests qemu from git.
>>>> 
>>> 
>>> i'm just trying with 0.12.5. 
>>> 
>>> i'm not so familiar with git. is there a command to pull only patches
>>> that are marked as stable and will be in the next official release?
>> 
>> All the qemu stable releases are tagged and you can check do
>> 
>> 	git-checkout v0.12.5
>> 
>> but that's not the main git HEAD which would also be interesting.
>> 
> 
> with v0.12.5 no xfs error, but the machine hangs after a few minutes...
> 
> (gdb)  thread apply all backtrace full
> 
> Thread 5 (Thread 0x7f15131c7950 (LWP 3579)):
> #0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000042b9f1 in kvm_run (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
> 	r = 0
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c6f000
> 	fd = 16
> #2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> #4  0x000000000042d819 in ap_main_loop (_env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d4cef0
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #7  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 4 (Thread 0x7f15129c6950 (LWP 3580)):
> #0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000042b9f1 in kvm_run (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
> 	r = 0
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c6c000
> 	fd = 17
> #2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> #4  0x000000000042d819 in ap_main_loop (_env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d67010
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #7  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> ---Type <return> to continue, or q <return> to quit---
> Thread 3 (Thread 0x7f15121c5950 (LWP 3581)):
> #0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
> No symbol table info available.
> #1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
> No symbol table info available.
> #2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
> No symbol table info available.
> #3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
> No locals.
> #4  0x000000000042ba68 in kvm_run (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
> 	r = -1
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c69000
> 	fd = 18
> #5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> #7  0x000000000042d819 in ap_main_loop (_env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d74d90
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #10 0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 2 (Thread 0x7f15119c4950 (LWP 3582)):
> #0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
> No symbol table info available.
> #1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
> No symbol table info available.
> #2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
> No symbol table info available.
> #3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
> No locals.
> #4  0x000000000042ba68 in kvm_run (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
> 	r = -1
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c66000
> 	fd = 19
> #5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> ---Type <return> to continue, or q <return> to quit---
> #7  0x000000000042d819 in ap_main_loop (_env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d82b10
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #10 0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 1 (Thread 0x7f1514c766f0 (LWP 3576)):
> #0  0x00007f15136c2742 in select () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000040c25a in main_loop_wait (timeout=1000) at /usr/src/qemu-kvm-0.12.5/vl.c:3994
> 	ioh = (IOHandlerRecord *) 0x0
> 	rfds = {fds_bits = {83886728, 0 <repeats 15 times>}}
> 	wfds = {fds_bits = {0 <repeats 16 times>}}
> 	xfds = {fds_bits = {0 <repeats 16 times>}}
> 	ret = 3
> 	nfds = 26
> 	tv = {tv_sec = 0, tv_usec = 999994}
> #2  0x000000000042dd9d in kvm_main_loop () at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:2126
> 	fds = {24, 25}
> 	mask = {__val = {268443712, 0 <repeats 15 times>}}
> 	sigfd = 26
> #3  0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.5/vl.c:4212
> 	r = 0
> #4  0x000000000041055a in main (argc=39, argv=0x7fffe23b2838, envp=0x7fffe23b2978) at /usr/src/qemu-kvm-0.12.5/vl.c:6256
> 	gdbstub_dev = 0x0
> 	boot_devices_bitmap = 12
> 	i = 0
> 	snapshot = 0
> 	linux_boot = 0
> 	initrd_filename = 0x0
> 	kernel_filename = 0x0
> 	kernel_cmdline = 0x58958c ""
> 	boot_devices = "dc", '\0' <repeats 30 times>
> 	ds = (DisplayState *) 0x1d9ff00
> 	dcl = (DisplayChangeListener *) 0x0
> 	cyls = 0
> 	heads = 0
> 	secs = 0
> 	translation = 0
> 	hda_opts = (QemuOpts *) 0x0
> 	opts = (QemuOpts *) 0x1d2adc0
> 	optind = 39
> ---Type <return> to continue, or q <return> to quit---
> 	r = 0x7fffe23b4bf7 "-no-kvm-clock"
> 	optarg = 0x0
> 	loadvm = 0x0
> 	machine = (QEMUMachine *) 0x862720
> 	cpu_model = 0x7fffe23b4b80 "qemu64,model_id=Intel(R) Xeon(R) CPU", ' ' <repeats 11 times>, "E5430  @ 2.66GHz,-tsc,-nx"
> 	fds = {-499439560, 32767}
> 	tb_size = 0
> 	pid_file = 0x7fffe23b4b3f "/var/run/qemu/vm-244.pid"
> 	incoming = 0x0
> 	fd = 0
> 	pwd = (struct passwd *) 0x0
> 	chroot_dir = 0x0
> 	run_as = 0x0
> 	env = (struct CPUX86State *) 0x0
> 	show_vnc_port = 0
> 	params = {0x58d2a6 "order", 0x58d2ac "once", 0x58d2b1 "menu", 0x0}
> 

sorry people, i have to apologize. i was testing 0.12.5 on an old dell with a xeon e5430. obviously, something is broken on these systems.
if i run the same tests on a recent xeon L5640 everything runs flawlessly. no hangs and no kvm emulation errors.

my stress tests with xfs using a diablo newsserver have also not generated an xfs corruption within the last 24 hours with
0.12.5. with 0.12.4 it appeared always after a few hours.

i will keep you posted if anything changes.

thanks,
peter

[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 20:11 schrieb Peter Lieven:

> 
> Am 25.09.2010 um 17:58 schrieb Christoph Hellwig:
> 
>> On Sat, Sep 25, 2010 at 05:40:34PM +0200, Peter Lieven wrote:
>>> 
>>> Am 25.09.2010 um 17:37 schrieb Christoph Hellwig:
>>> 
>>>> FYI, qemu 0.12.2 is missing:
>>> 
>>> you mean 0.12.4 not 0.12.2, don't you?
>> 
>> Yes, sorry. (but 0.12.2 is of course missing it, too..)
>> 
>>>> which in the past was very good at triggering XFS guest corruption.
>>>> Please try with the patch applied or even better latests qemu from git.
>>>> 
>>> 
>>> i'm just trying with 0.12.5. 
>>> 
>>> i'm not so familiar with git. is there a command to pull only patches
>>> that are marked as stable and will be in the next official release?
>> 
>> All the qemu stable releases are tagged and you can check do
>> 
>> 	git-checkout v0.12.5
>> 
>> but that's not the main git HEAD which would also be interesting.
>> 
> 
> with v0.12.5 no xfs error, but the machine hangs after a few minutes...
> 
> (gdb)  thread apply all backtrace full
> 
> Thread 5 (Thread 0x7f15131c7950 (LWP 3579)):
> #0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000042b9f1 in kvm_run (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
> 	r = 0
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c6f000
> 	fd = 16
> #2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> #4  0x000000000042d819 in ap_main_loop (_env=0x1d4cef0) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d4cef0
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #7  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 4 (Thread 0x7f15129c6950 (LWP 3580)):
> #0  0x00007f15136c1cd7 in ioctl () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000042b9f1 in kvm_run (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:921
> 	r = 0
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c6c000
> 	fd = 17
> #2  0x000000000042cf4e in kvm_cpu_exec (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #3  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> #4  0x000000000042d819 in ap_main_loop (_env=0x1d67010) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d67010
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #5  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #7  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> ---Type <return> to continue, or q <return> to quit---
> Thread 3 (Thread 0x7f15121c5950 (LWP 3581)):
> #0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
> No symbol table info available.
> #1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
> No symbol table info available.
> #2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
> No symbol table info available.
> #3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
> No locals.
> #4  0x000000000042ba68 in kvm_run (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
> 	r = -1
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c69000
> 	fd = 18
> #5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> #7  0x000000000042d819 in ap_main_loop (_env=0x1d74d90) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d74d90
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #10 0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 2 (Thread 0x7f15119c4950 (LWP 3582)):
> #0  0x00007f151464da94 in __lll_lock_wait () from /lib/libpthread.so.0
> No symbol table info available.
> #1  0x00007f1514649190 in _L_lock_102 () from /lib/libpthread.so.0
> No symbol table info available.
> #2  0x00007f1514648a7e in pthread_mutex_lock () from /lib/libpthread.so.0
> No symbol table info available.
> #3  0x000000000042b7dc in post_kvm_run (kvm=0x1d2b7b0, env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:858
> No locals.
> #4  0x000000000042ba68 in kvm_run (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:930
> 	r = -1
> 	kvm = (kvm_context_t) 0x1d2b7b0
> 	run = (struct kvm_run *) 0x7f1514c66000
> 	fd = 19
> #5  0x000000000042cf4e in kvm_cpu_exec (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1651
> 	r = 0
> #6  0x000000000042d6d8 in kvm_main_loop_cpu (env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1893
> 	run_cpu = 1
> ---Type <return> to continue, or q <return> to quit---
> #7  0x000000000042d819 in ap_main_loop (_env=0x1d82b10) at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:1943
> 	env = (struct CPUX86State *) 0x1d82b10
> 	signals = {__val = {18446744067267100671, 18446744073709551615 <repeats 15 times>}}
> 	data = (struct ioperm_data *) 0x0
> #8  0x00007f15146473ba in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #9  0x00007f15136c9fcd in clone () from /lib/libc.so.6
> No symbol table info available.
> #10 0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Thread 1 (Thread 0x7f1514c766f0 (LWP 3576)):
> #0  0x00007f15136c2742 in select () from /lib/libc.so.6
> No symbol table info available.
> #1  0x000000000040c25a in main_loop_wait (timeout=1000) at /usr/src/qemu-kvm-0.12.5/vl.c:3994
> 	ioh = (IOHandlerRecord *) 0x0
> 	rfds = {fds_bits = {83886728, 0 <repeats 15 times>}}
> 	wfds = {fds_bits = {0 <repeats 16 times>}}
> 	xfds = {fds_bits = {0 <repeats 16 times>}}
> 	ret = 3
> 	nfds = 26
> 	tv = {tv_sec = 0, tv_usec = 999994}
> #2  0x000000000042dd9d in kvm_main_loop () at /usr/src/qemu-kvm-0.12.5/qemu-kvm.c:2126
> 	fds = {24, 25}
> 	mask = {__val = {268443712, 0 <repeats 15 times>}}
> 	sigfd = 26
> #3  0x000000000040c98c in main_loop () at /usr/src/qemu-kvm-0.12.5/vl.c:4212
> 	r = 0
> #4  0x000000000041055a in main (argc=39, argv=0x7fffe23b2838, envp=0x7fffe23b2978) at /usr/src/qemu-kvm-0.12.5/vl.c:6256
> 	gdbstub_dev = 0x0
> 	boot_devices_bitmap = 12
> 	i = 0
> 	snapshot = 0
> 	linux_boot = 0
> 	initrd_filename = 0x0
> 	kernel_filename = 0x0
> 	kernel_cmdline = 0x58958c ""
> 	boot_devices = "dc", '\0' <repeats 30 times>
> 	ds = (DisplayState *) 0x1d9ff00
> 	dcl = (DisplayChangeListener *) 0x0
> 	cyls = 0
> 	heads = 0
> 	secs = 0
> 	translation = 0
> 	hda_opts = (QemuOpts *) 0x0
> 	opts = (QemuOpts *) 0x1d2adc0
> 	optind = 39
> ---Type <return> to continue, or q <return> to quit---
> 	r = 0x7fffe23b4bf7 "-no-kvm-clock"
> 	optarg = 0x0
> 	loadvm = 0x0
> 	machine = (QEMUMachine *) 0x862720
> 	cpu_model = 0x7fffe23b4b80 "qemu64,model_id=Intel(R) Xeon(R) CPU", ' ' <repeats 11 times>, "E5430  @ 2.66GHz,-tsc,-nx"
> 	fds = {-499439560, 32767}
> 	tb_size = 0
> 	pid_file = 0x7fffe23b4b3f "/var/run/qemu/vm-244.pid"
> 	incoming = 0x0
> 	fd = 0
> 	pwd = (struct passwd *) 0x0
> 	chroot_dir = 0x0
> 	run_as = 0x0
> 	env = (struct CPUX86State *) 0x0
> 	show_vnc_port = 0
> 	params = {0x58d2a6 "order", 0x58d2ac "once", 0x58d2b1 "menu", 0x0}
> 

sorry people, i have to apologize. i was testing 0.12.5 on an old dell with a xeon e5430. obviously, something is broken on these systems.
if i run the same tests on a recent xeon L5640 everything runs flawlessly. no hangs and no kvm emulation errors.

my stress tests with xfs using a diablo newsserver have also not generated an xfs corruption within the last 24 hours with
0.12.5. with 0.12.4 it appeared always after a few hours.

i will keep you posted if anything changes.

thanks,
peter