Re: [Qemu-devel] Qemu-KVM 0.12.3 and Multipath -> Assertion

[Peter Lieven <pl@dlh.net>]

hi kevin,

i set a breakpint at bmdma_active_if. the first 2 breaks encountered 
when the last path in the multipath
failed, but the assertion was not true.
when i kicked one path back in the breakpoint was reached again, this 
time leading to an assert.
the stacktrace is from the point shortly before.

hope this helps.

br,
peter
--

(gdb) b bmdma_active_if
Breakpoint 2 at 0x43f2e0: file 
/usr/src/qemu-kvm-0.12.3/hw/ide/internal.h, line 507.
(gdb) c
Continuing.
[Switching to Thread 0x7f7b3300d950 (LWP 21171)]

Breakpoint 2, bmdma_active_if (bmdma=0xe31fd8) at 
/usr/src/qemu-kvm-0.12.3/hw/ide/internal.h:507
507        assert(bmdma->unit != (uint8_t)-1);
(gdb) c
Continuing.

Breakpoint 2, bmdma_active_if (bmdma=0xe31fd8) at 
/usr/src/qemu-kvm-0.12.3/hw/ide/internal.h:507
507        assert(bmdma->unit != (uint8_t)-1);
(gdb) c
Continuing.

Breakpoint 2, bmdma_active_if (bmdma=0xe31fd8) at 
/usr/src/qemu-kvm-0.12.3/hw/ide/internal.h:507
507        assert(bmdma->unit != (uint8_t)-1);
(gdb) bt full
#0  bmdma_active_if (bmdma=0xe31fd8) at 
/usr/src/qemu-kvm-0.12.3/hw/ide/internal.h:507
    __PRETTY_FUNCTION__ = "bmdma_active_if"
#1  0x000000000043f6ba in ide_read_dma_cb (opaque=0xe31fd8, ret=0) at 
/usr/src/qemu-kvm-0.12.3/hw/ide/core.c:554
    bm = (BMDMAState *) 0xe31fd8
    s = (IDEState *) 0xe17940
    n = 0
    sector_num = 0
#2  0x000000000058730c in dma_bdrv_cb (opaque=0xe17940, ret=0) at 
/usr/src/qemu-kvm-0.12.3/dma-helpers.c:94
    dbs = (DMAAIOCB *) 0xe17940
    cur_addr = 0
    cur_len = 0
    mem = (void *) 0x0
#3  0x000000000049e510 in qemu_laio_process_completion (s=0xe119c0, 
laiocb=0xe179c0) at linux-aio.c:68
    ret = 0
#4  0x000000000049e611 in qemu_laio_enqueue_completed (s=0xe119c0, 
laiocb=0xe179c0) at linux-aio.c:107
No locals.
#5  0x000000000049e787 in qemu_laio_completion_cb (opaque=0xe119c0) at 
linux-aio.c:144
    iocb = (struct iocb *) 0xe179f0
    laiocb = (struct qemu_laiocb *) 0xe179c0
    val = 1
    ret = 8
    nevents = 1
    i = 0
    events = {{data = 0x0, obj = 0xe179f0, res = 4096, res2 = 0}, {data 
= 0x0, obj = 0x0, res = 0, res2 = 0} <repeats 46 times>, {data = 0x0, 
obj = 0x0, res = 0,
    res2 = 4365191}, {data = 0x429abf, obj = 0x7f7b3300c410, res = 
4614129721674825936, res2 = 14777248}, {data = 0x3000000018, obj = 
0x7f7b3300c4c0, res = 140167113393152,
    res2 = 47259417504}, {data = 0xe17740, obj = 0xa3300c4e0, res = 
140167113393184, res2 = 0}, {data = 0xe17740, obj = 0x0, res = 0, res2 = 
17}, {data = 0x7f7b3300ccf0,
    obj = 0x92, res = 32, res2 = 168}, {data = 0x7f7b33797a00, obj = 
0x801000, res = 0, res2 = 140167141433408}, {data = 0x7f7b34496e00, obj 
= 0x7f7b33797a00,
    res = 140167113393392, res2 = 8392704}, {data = 0x0, obj = 
0x7f7b34aca040, res = 140167134932480, res2 = 140167118209654}, {data = 
0x7f7b3300d950, obj = 0x42603d, res = 0,
    res2 = 42949672960}, {data = 0x7f7b3300c510, obj = 0xe17ba0, res = 
14776128, res2 = 43805361568}, {data = 0x7f7b3300ced0, obj = 0x42797e, 
res = 0, res2 = 14777248}, {
    data = 0x174, obj = 0x0, res = 373, res2 = 0}, {data = 0x176, obj = 
0x0, res = 3221225601, res2 = 0}, {data = 0x4008ae89c0000083, obj = 0x0, 
res = 209379655938, res2 = 0}, {
    data = 0x7f7bc0000084, obj = 0x0, res = 3221225602, res2 = 0}, {data 
= 0x7f7b00000012, obj = 0x0, res = 17, res2 = 0}, {data = 0x0, obj = 
0x11, res = 140167113395840,
    res2 = 146}, {data = 0x20, obj = 0xa8, res = 140167121304064, res2 = 
8392704}, {data = 0x0, obj = 0x7f7b34aca040, res = 140167134932480, res2 
= 140167121304064}, {
    data = 0x7f7b3300c680, obj = 0x801000, res = 0, res2 = 
140167141433408}, {data = 0x7f7b34496e00, obj = 0x7f7b334a4276, res = 
140167113398608, res2 = 4350013}, {data = 0x0,
    obj = 0xa00000000, res = 140167113393824, res2 = 14777248}, {data = 
0xe2c010, obj = 0xa3300c730, res = 140167113396320, res2 = 4356478}, 
{data = 0x0, obj = 0xe17ba0,
    res = 372, res2 = 0}, {data = 0x175, obj = 0x0, res = 374, res2 = 
0}, {data = 0xc0000081, obj = 0x0, res = 3221225603, res2 = 0}, {data = 
0xc0000102, obj = 0x0,
    res = 3221225604, res2 = 0}, {data = 0xc0000082, obj = 0x0, res = 
18, res2 = 0}, {data = 0x11, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, 
obj = 0x0, res = 0, res2 = 0}, {
    data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 0x0, obj = 0x0, 
res = 0, res2 = 0}, {data = 0x0, obj = 0x0, res = 0, res2 = 0}, {data = 
0x0, obj = 0x0, res = 0,
    res2 = 140167139245116}, {data = 0x0, obj = 0x7f7b34abe118, res = 9, 
res2 = 13}, {data = 0x25bf5fc6, obj = 0x7f7b348b40f0, res = 
140167117719264, res2 = 6}, {
    data = 0x96fd7f, obj = 0x7f7b3300c850, res = 140167113394680, res2 = 
140167117724520}, {data = 0x0, obj = 0x7f7b34abe168, res = 
140167141388288, res2 = 4206037}, {
    data = 0x7f7b3343a210, obj = 0x402058, res = 21474836480, res2 = 
4294968102}, {data = 0x0, obj = 0x7f7b34ac8358, res = 140167113394736, 
res2 = 140167113394680}, {
    data = 0x25bf5fc6, obj = 0x7f7b3300c9e0, res = 0, res2 = 
140167139246910}, {data = 0x0, obj = 0x7f7b34abe168, res = 5, res2 = 
140167139245116}, {data = 0x1,
    obj = 0x7f7b34abe120, res = 10, res2 = 13}, {data = 0x9fd7b9dd, obj 
= 0x7f7b348b40f0, res = 140167139205904, res2 = 140166257704989}, {data 
= 0x27f5ee7,
    obj = 0x7f7b3300c950, res = 140167113394936, res2 = 
140167139205920}, {data = 0x0, obj = 0x7f7b34abe510, res = 
140167141434664, res2 = 140167134874710}, {
    data = 0x7f7b348aa5a8, obj = 0x7f7b34486e30, res = 21474836480, res2 
= 4294967319}, {data = 0x7f7b3428427c, obj = 0x7f7b34ac7cc8, res = 
140167113394992,
    res2 = 140167113394936}, {data = 0x9fd7b9dd, obj = 0x7f7b3300cae0, 
res = 0, res2 = 140167139246910}, {data = 0x0, obj = 0x7f7b34abe510, res 
= 140166257704965, res2 = 0}, {
    data = 0x500000001, obj = 0xffffffff, res = 0, res2 = 8627912}, 
{data = 0x801000, obj = 0x100000000, res = 140167141385488, res2 = 
140167141424328}, {data = 0x7f7b3300cb60,
    obj = 0x7f7b34ac7970, res = 140167134874710, res2 = 0}, {data = 0x5, 
obj = 0x0, res = 140167117743864, res2 = 140167113398608}, {data = 
0x7f7b3300cb00, obj = 0x7f7b348bf592,
    res = 14665264, res2 = 90}, {data = 0x2, obj = 0x7f7b33508255, res = 
140167113398608, res2 = 140167113394944}, {data = 0x801000, obj = 0x0, 
res = 140167141433408,
    res2 = 4966659}, {data = 0x50, obj = 0x23300cb2f, res = 
140167139206472, res2 = 140167141434664}, {data = 0x98, obj = 
0xffffffff, res = 140167113395072, res2 = 2191368}, {
    data = 0x801000, obj = 0x3, res = 140167141433408, res2 = 
140167139245116}, {data = 0x7f7b34486000, obj = 0x7f7b34abe0e8, res = 3, 
res2 = 13}, {data = 0xa896c0a2,
    obj = 0x7f7b348b40f0, res = 140167132758616, res2 = 34}, {data = 
0x2a25b02, obj = 0x7f7b3300cb90, res = 140167113395512, res2 = 
140167132758672}, {data = 0x0,
    obj = 0x7f7b34abe1b0, res = 140167141396480, res2 = 4204852}, {data 
= 0x7f7b34284458, obj = 0x400d38, res = 21474836480, res2 = 4294967302}, 
{data = 0xc008ae67325339e0,
---Type <return> to continue, or q <return> to quit---
    obj = 0x7f7b34ac8358, res = 140167113395568, res2 = 
140167113395512}, {data = 0xa896c0a2, obj = 0x7f7b3300cd20, res = 0, 
res2 = 140167139246910}, {data = 0x0,
    obj = 0x7f7b34abe1b0, res = 140166257704965, res2 = 0}, {data = 0x1, 
obj = 0x101010101010101, res = 140167113395504, res2 = 14776768}, {data 
= 0x7f7b3300cd20,
    obj = 0x13300cd7c, res = 140167141384624, res2 = 140167141426008}, 
{data = 0x7f7b3300cda0, obj = 0x7f7b34ac8000, res = 4204852, res2 = 
4364819}, {data = 0x42994b,
    obj = 0x7f7b3300ccf0, res = 13837501612500713496, res2 = 14658720}, 
{data = 0x3000000018, obj = 0x7f7b3300cda0, res = 140167113395424, res2 
= 4371109}, {
    data = 0x7f7b3300cdc0, obj = 0x3300cd4c, res = 64424509441, res2 = 
4364819}, {data = 0x42994b, obj = 0x7f7b3300cd50, res = 
13837501612500713473, res2 = 14658720}, {
    data = 0x3000000018, obj = 0x7f7b3300ce00, res = 140167132758816, 
res2 = 140167141396480}, {data = 0x7f7b3300ce20, obj = 0xffffffff, res = 
140167113395580, res2 = 8626296}, {
    data = 0x801000, obj = 0x0, res = 140167141433408, res2 = 
140167134932480}, {data = 0x0, obj = 0x7f7b348b871a, res = 
140166257704965, res2 = 0}, {data = 0x7f7b3300cdd0,
    obj = 0x42b2a5, res = 140167132758816, res2 = 140167113398608}}
    ts = {tv_sec = 0, tv_nsec = 0}
    s = (struct qemu_laio_state *) 0xe119c0
#6  0x000000000049e841 in laio_cancel (blockacb=0xe179c0) at linux-aio.c:184
    laiocb = (struct qemu_laiocb *) 0xe179c0
    event = {data = 0x1, obj = 0x4c7fb1, res = 140167113395792, res2 = 
4384262}
    ret = -22
#7  0x000000000049a29b in bdrv_aio_cancel (acb=0xe179c0) at block.c:1792
No locals.
#8  0x000000000058755a in dma_aio_cancel (acb=0xe17940) at 
/usr/src/qemu-kvm-0.12.3/dma-helpers.c:138
    dbs = (DMAAIOCB *) 0xe17940
#9  0x000000000049a29b in bdrv_aio_cancel (acb=0xe17940) at block.c:1792
No locals.
#10 0x0000000000444a0c in ide_dma_cancel (bm=0xe31fd8) at 
/usr/src/qemu-kvm-0.12.3/hw/ide/core.c:2838
No locals.
#11 0x0000000000444f39 in bmdma_cmd_writeb (opaque=0xe31fd8, addr=49152, 
val=8) at /usr/src/qemu-kvm-0.12.3/hw/ide/pci.c:44
    bm = (BMDMAState *) 0xe31fd8
#12 0x00000000004c81bc in ioport_write (index=0, address=49152, data=8) 
at ioport.c:80
    func = (IOPortWriteFunc *) 0x444f0c <bmdma_cmd_writeb>
    default_func = {0x4c81d0 <default_ioport_writeb>, 0x4c8225 
<default_ioport_writew>, 0x4c8282 <default_ioport_writel>}
#13 0x00000000004c8543 in cpu_outb (addr=49152, val=8 '\b') at ioport.c:198
No locals.
#14 0x0000000000429689 in kvm_handle_io (port=49152, 
data=0x7f7b34ab7000, direction=1, size=1, count=1) at 
/usr/src/qemu-kvm-0.12.3/kvm-all.c:535
    i = 0
    ptr = (uint8_t *) 0x7f7b34ab7000 "\b"
#15 0x000000000042bac3 in kvm_run (env=0xe17ba0) at 
/usr/src/qemu-kvm-0.12.3/qemu-kvm.c:964
    r = 0
    kvm = (kvm_context_t) 0xdfb0d0
    run = (struct kvm_run *) 0x7f7b34ab6000
    fd = 15
#16 0x000000000042cdda in kvm_cpu_exec (env=0xe17ba0) at 
/usr/src/qemu-kvm-0.12.3/qemu-kvm.c:1647
    r = 0
#17 0x000000000042d564 in kvm_main_loop_cpu (env=0xe17ba0) at 
/usr/src/qemu-kvm-0.12.3/qemu-kvm.c:1889
    run_cpu = 1
#18 0x000000000042d6a5 in ap_main_loop (_env=0xe17ba0) at 
/usr/src/qemu-kvm-0.12.3/qemu-kvm.c:1939
    env = (struct CPUX86State *) 0xe17ba0
    signals = {__val = {18446744067267100671, 18446744073709551615 
<repeats 15 times>}}
    data = (struct ioperm_data *) 0x0
#19 0x00007f7b3448d3ba in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#20 0x00007f7b3350ffcd in clone () from /lib/libc.so.6
No symbol table info available.
#21 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) c
Continuing.
kvm: /usr/src/qemu-kvm-0.12.3/hw/ide/internal.h:507: bmdma_active_if: 
Assertion `bmdma->unit != (uint8_t)-1' failed.

Program received signal SIGABRT, Aborted.
0x00007f7b3345cfb5 in raise () from /lib/libc.so.6


Kevin Wolf wrote:
> Am 03.05.2010 23:26, schrieb Peter Lieven:
>   
>> Hi Qemu/KVM Devel Team,
>>
>> i'm using qemu-kvm 0.12.3 with latest Kernel 2.6.33.3.
>> As backend we use open-iSCSI with dm-multipath.
>>
>> Multipath is configured to queue i/o if no path is available.
>>
>> If we create a failure on all paths, qemu starts to consume 100%
>> CPU due to i/o waits which is ok so far.
>>
>> 1 odd thing: The Monitor Interface is not responding any more ...
>>
>> What es a really blocker is that KVM crashes with:
>> kvm: /usr/src/qemu-kvm-0.12.3/hw/ide/internal.h:507: bmdma_active_if: 
>> Assertion `bmdma->unit != (uint8_t)-1' failed.
>>
>> after the multipath has reestablisched at least one path.
>>     
>
> Can you get a stack backtrace with gdb?
>
>   
>> Any ideas? I remember this was working with earlier kernel/kvm/qemu 
>> versions.
>>     
>
> If it works in the same setup with an older qemu version, bisecting
> might help.
>
> Kevin
>
>