[Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

[-- Attachment #1: Type: text/plain, Size: 4848 bytes --]

Hi,

If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio storage 
backend that supports iSCSI
qemu-kvm crashes reliably with the following error:

Bad ram pointer 0x3039303620008000

This happens directly after the confirmation of the Timezone before the 
Disk is partitioned.

If I specify  -global virtio-blk-pci.scsi=off in the cmdline this does 
not happen.

Here is a stack trace:

Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
#0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 <https://github.com/sahlberg/libiscsi/issues/1> 0x00005555557b751d in 
qemu_ram_addr_from_host_nofail (
ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
ram_addr = 0
#2 <https://github.com/sahlberg/libiscsi/issues/2> 0x00005555557b9177 in 
cpu_physical_memory_unmap (
buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
addr1 = 93825009559312
#3 <https://github.com/sahlberg/libiscsi/issues/3> 0x000055555580a9ca in 
virtqueue_fill (vq=0x5555565da710,
elem=0x555556722238, len=1, idx=0)
at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:240
size = 1
offset = 0
i = 0
#4 <https://github.com/sahlberg/libiscsi/issues/4> 0x000055555580abf0 in 
virtqueue_push (vq=0x5555565da710,
elem=0x555556722238, len=1) at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:276
No locals.
#5 <https://github.com/sahlberg/libiscsi/issues/5> 0x0000555555800952 in 
virtio_blk_req_complete (req=0x555556722230,
status=0) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:62
s = 0x5555565da640
#6 <https://github.com/sahlberg/libiscsi/issues/6> 0x00005555558010bf in 
virtio_blk_handle_scsi (req=0x555556722230)
at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:261
ret = 0
i = 1
status = 0
hdr = {interface_id = 83, dxfer_direction = -3, cmd_len = 6 '\006',
mx_sb_len = 96 '`', iovec_count = 1, dxfer_len = 56,
dxferp = 0x555556726248, cmdp = 0x2aab24b6c838 "\022\001\200",
sbp = 0x2aab1d677c30 "", timeout = 0, flags = 0, pack_id = 0,
usr_ptr = 0x0, status = 0 '\000', masked_status = 0 '\000',
msg_status = 0 '\000', sb_len_wr = 0 '\000', host_status = 0,
driver_status = 0, resid = 0, duration = 0, info = 0}
#7 <https://github.com/sahlberg/libiscsi/issues/7> 0x0000555555801724 in 
virtio_blk_handle_request (req=0x555556722230,
mrb=0x7fffffffd9f0) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:393
type = 2
#8 <https://github.com/sahlberg/libiscsi/issues/8> 0x00005555558018c3 in 
virtio_blk_handle_output (vdev=0x5555565da640,
vq=0x5555565da710) at /usr/src/qemu-kvm-1.2.0/hw/virtio-blk.c:426
s = 0x5555565da640
req = 0x555556722230
mrb = {blkreq = {{sector = 0, nb_sectors = 0, qiov = 0x0, cb = 0,
opaque = 0x0, error = 0} }, num_writes = 0}
#9 <https://github.com/sahlberg/libiscsi/issues/9> 0x000055555580bd81 in 
virtio_queue_notify_vq (vq=0x5555565da710)
at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:648
vdev = 0x5555565da640
#10 <https://github.com/sahlberg/libiscsi/issues/10> 0x000055555580d2ff 
in virtio_queue_host_notifier_read (n=0x5555565da75c)
at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:1020
vq = 0x5555565da710
#11 <https://github.com/sahlberg/libiscsi/issues/11> 0x000055555565a47e 
in qemu_iohandler_poll (readfds=0x555556073160,
writefds=0x5555560731e0, xfds=0x555556073260, ret=1) at iohandler.c:122
pioh = 0x555556541290
ioh = 0x7ffff0000e70
#12 <https://github.com/sahlberg/libiscsi/issues/12> 0x000055555572b742 
in main_loop_wait (nonblocking=0) at main-loop.c:497
ret = 1
timeout = 4294967295
#13 <https://github.com/sahlberg/libiscsi/issues/13> 0x00005555557235e2 
in main_loop () at /usr/src/qemu-kvm-1.2.0/vl.c:1643
nonblocking = false
last_io = 1
#14 <https://github.com/sahlberg/libiscsi/issues/14> 0x000055555572a21c 
in main (argc=42, argv=0x7fffffffe548,
envp=0x7fffffffe6a0) at /usr/src/qemu-kvm-1.2.0/vl.c:3790
i = 64
snapshot = 0
linux_boot = 0
icount_option = 0x0
initrd_filename = 0x0
kernel_filename = 0x0
kernel_cmdline = 0x5555558d442a ""
boot_devices = "dc", '\000'
ds = 0x5555565465a0
dcl = 0x0
cyls = 0
heads = 0
secs = 0
translation = 0
hda_opts = 0x0
opts = 0x55555650f4b0
machine_opts = 0x55555650fcb0
olist = 0x5780f638f2e0
optind = 42
optarg = 0x7fffffffebd9 "cirrus"
loadvm = 0x0
machine = 0x555555c66780
cpu_model = 0x7fffffffeb5b "host,+x2apic,model_id=Intel(R) Xeon(R) CPU", 
' ' , "L5640 @ 2.27GHz,-tsc"
vga_model = 0x7fffffffebd9 "cirrus"
pid_file = 0x7fffffffeb1a "/var/run/qemu/vm-279.pid"
incoming = 0x0
show_vnc_port = 0
defconfig = true
userconfig = true
log_mask = 0x0
log_file = 0x0
mem_trace = {malloc = 0x55555572683e ,
realloc = 0x555555726896 ,
free = 0x5555557268fd , calloc = 0, try_malloc = 0,
try_realloc = 0}
trace_events = 0x0
trace_file = 0x0

Is this a regression in qemu-kvm. I remember there where some 
modifications regarding SCSI
passthru lately. Maybe there was a problem introduced with this.

BR,
Peter


[-- Attachment #2: Type: text/html, Size: 7159 bytes --]

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Stefan Hajnoczi]

On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
> Hi,

Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
different virtio device type from virtoi-blk and is not present in the
backtrace you posted.

Sounds pedantic but I want to make sure this gets chalked up against the
right device :).

> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
> storage backend that supports iSCSI
> qemu-kvm crashes reliably with the following error:

Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?

Have you tried qemu-kvm.git/master?

Have you tried a local raw disk image to check whether libiscsi is
involved?

> Bad ram pointer 0x3039303620008000
> 
> This happens directly after the confirmation of the Timezone before
> the Disk is partitioned.
> 
> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
> does not happen.
> 
> Here is a stack trace:
> 
> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #1 <https://github.com/sahlberg/libiscsi/issues/1>
> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
> ram_addr = 0
> #2 <https://github.com/sahlberg/libiscsi/issues/2>
> 0x00005555557b9177 in cpu_physical_memory_unmap (
> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645

buffer and len are ASCII junk.  It appears to be hex digits and it's not
clear where they come from.

It would be interesting to print *elem one stack frame up in #3
virtqueue_fill() to show the iovecs and in/out counts.

> addr1 = 93825009559312
> #3 <https://github.com/sahlberg/libiscsi/issues/3>
> 0x000055555580a9ca in virtqueue_fill (vq=0x5555565da710,
> elem=0x555556722238, len=1, idx=0)
> at /usr/src/qemu-kvm-1.2.0/hw/virtio.c:240
> size = 1
> offset = 0
> i = 0

Stefan

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-BLK -> Bad ram pointer

[Peter Lieven]

On 30.10.2012 09:32, Stefan Hajnoczi wrote:
> On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
>> Hi,
> Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
> different virtio device type from virtoi-blk and is not present in the
> backtrace you posted.
you are right, sorry for that.
>
> Sounds pedantic but I want to make sure this gets chalked up against the
> right device :).
>
>> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
>> storage backend that supports iSCSI
>> qemu-kvm crashes reliably with the following error:
> Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
I use vanilla qemu-kvm 1.2.0 except for one virtio-blk related patch 
(CVE-2011-4127):
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=1ba1f2e319afdcb485963cd3f426fdffd1b725f2
that for some reason did not made it into qemu-kvm 1.2.0 and two aio 
related patchs:
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=00f78533326c5ba2e62fafada16655aa558a5520
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=2db2bfc0ccac5fd68dbf0ceb70fbc372c5d8a8c7

this is why I can circumvent the issue with scsi=off i guess.
>
> Have you tried qemu-kvm.git/master?
not yet.
>
> Have you tried a local raw disk image to check whether libiscsi is
> involved?
I have, here it does not happen. For a raw device scsi is scsi=off, 
isn't it?
>
>> Bad ram pointer 0x3039303620008000
>>
>> This happens directly after the confirmation of the Timezone before
>> the Disk is partitioned.
>>
>> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
>> does not happen.
>>
>> Here is a stack trace:
>>
>> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
>> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> No symbol table info available.
>> #1 <https://github.com/sahlberg/libiscsi/issues/1>
>> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
>> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
>> ram_addr = 0
>> #2 <https://github.com/sahlberg/libiscsi/issues/2>
>> 0x00005555557b9177 in cpu_physical_memory_unmap (
>> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
>> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
> buffer and len are ASCII junk.  It appears to be hex digits and it's not
> clear where they come from.
>
> It would be interesting to print *elem one stack frame up in #3
> virtqueue_fill() to show the iovecs and in/out counts.
I will collect that info for you.

Peter

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

On 30.10.2012 09:32, Stefan Hajnoczi wrote:
> On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
>> Hi,
> Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
> different virtio device type from virtoi-blk and is not present in the
> backtrace you posted.
>
> Sounds pedantic but I want to make sure this gets chalked up against the
> right device :).
>
>> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
>> storage backend that supports iSCSI
>> qemu-kvm crashes reliably with the following error:
> Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
>
> Have you tried qemu-kvm.git/master?
>
> Have you tried a local raw disk image to check whether libiscsi is
> involved?
>
>> Bad ram pointer 0x3039303620008000
>>
>> This happens directly after the confirmation of the Timezone before
>> the Disk is partitioned.
>>
>> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
>> does not happen.
>>
>> Here is a stack trace:
>>
>> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
>> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> No symbol table info available.
>> #1 <https://github.com/sahlberg/libiscsi/issues/1>
>> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
>> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
>> ram_addr = 0
>> #2 <https://github.com/sahlberg/libiscsi/issues/2>
>> 0x00005555557b9177 in cpu_physical_memory_unmap (
>> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
>> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
> buffer and len are ASCII junk.  It appears to be hex digits and it's not
> clear where they come from.
>
> It would be interesting to print *elem one stack frame up in #3
> virtqueue_fill() to show the iovecs and in/out counts.

(gdb) print *elem
$6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
     2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
     140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
     2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
     12385, 93825009696000, 140737328183160, 0 <repeats 501 times>}, 
in_sg = {{
       iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
       iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
       iov_base = 0x2aab32443039, iov_len = 16}, {iov_base = 0x2aab2365c628,
       iov_len = 1}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0,
       iov_len = 0}, {iov_base = 0x2041, iov_len = 93825010788016}, {
       iov_base = 0x7ffff673f778, iov_len = 0}, {iov_base = 0x0,
       iov_len = 0} <repeats 256 times>, {iov_base = 0x1021,
       iov_len = 93825010788016}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 0} <repeats 255 times>, {iov_base = 0x0,
       iov_len = 24768}, {iov_base = 0x1020, iov_len = 93824999311936}, {
       iov_base = 0x0, iov_len = 2}, {iov_base = 0x0,
       iov_len = 0} <repeats 256 times>, {iov_base = 0x1021,
       iov_len = 93825009696000}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 0} <repeats 242 times>}, out_sg = {{
       iov_base = 0x2aab2365c608, iov_len = 16}, {iov_base = 0x2aab243fbae8,
       iov_len = 6}, {iov_base = 0x0, iov_len = 0} <repeats 11 times>, {
       iov_base = 0x0, iov_len = 33024}, {iov_base = 0x30,
       iov_len = 93825010821424}, {iov_base = 0x55555670d7a0, iov_len = 
0}, {
       iov_base = 0x55555670cbb0, iov_len = 0}, {iov_base = 0x71,
       iov_len = 93825008729792}, {iov_base = 0x55555670e960, iov_len = 
0}, {
       iov_base = 0x31, iov_len = 140737328183192}, {iov_base = 
0x7ffff673f798,
       iov_len = 0}, {iov_base = 0x555556711e20, iov_len = 80}, {
       iov_base = 0x20, iov_len = 93825010821584}, {iov_base = 0x0,
       iov_len = 33184}, {iov_base = 0x30, iov_len = 93825010821536}, {
       iov_base = 0x55555670e840, iov_len = 0}, {iov_base = 0x55555670e1b0,
       iov_len = 0}, {iov_base = 0x41, iov_len = 93825010821584}, {
       iov_base = 0x55555670eb20, iov_len = 32}, {iov_base = 0x20,
       iov_len = 93825010821920}, {iov_base = 0x0, iov_len = 33296}, {
       iov_base = 0x30, iov_len = 93825010821872}, {iov_base = 
0x55555670e8b0,
       iov_len = 0}, {iov_base = 0x55555670dc68, iov_len = 0}, {
       iov_base = 0x191, iov_len = 93825009696736}, {iov_base = 
0x55555670eb20,
       iov_len = 0}, {iov_base = 0x21, iov_len = 93825010826352}, {
       iov_base = 0x55555670e880, iov_len = 64}, {iov_base = 0x30,
       iov_len = 93825010821200}, {iov_base = 0x55555670e920, iov_len = 
0}, {
       iov_base = 0x55555670e5c8, iov_len = 0}, {iov_base = 0x41,
       iov_len = 93825008729792}, {iov_base = 0x55555670e9d0, iov_len = 
32}, {
       iov_base = 0x20, iov_len = 93825010821696}, {iov_base = 0x0,
       iov_len = 176}, {iov_base = 0x30, iov_len = 93825010821648}, {
       iov_base = 0x55555670e990, iov_len = 0}, {iov_base = 0x55555670e080,
       iov_len = 0}, {iov_base = 0x41, iov_len = 93825008729792}, {
       iov_base = 0x55555670eb20, iov_len = 32}, {iov_base = 0x20,
       iov_len = 93825010822032}, {iov_base = 0x0, iov_len = 288}, {
       iov_base = 0x30, iov_len = 93825010821984}, {iov_base = 
0x55555670ea00,
       iov_len = 0}, {iov_base = 0x55555670e000, iov_len = 0}, {
       iov_base = 0x41, iov_len = 93825010821808}, {iov_base = 
0x555556602590,
       iov_len = 32}, {iov_base = 0x20, iov_len = 93825009373648}, {
       iov_base = 0x0, iov_len = 33744}, {iov_base = 0x30,
       iov_len = 93825009373680}, {iov_base = 0x55555670ea70, iov_len = 
0}, {
       iov_base = 0x55555670da18, iov_len = 0}, {iov_base = 0x17271,
       iov_len = 93825009696736}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 93825010821920}, {iov_base = 
0x55555670ec00,
       iov_len = 64}, {iov_base = 0x30, iov_len = 93825010822096}, {
       iov_base = 0x55555670eae0, iov_len = 0}, {iov_base = 0x55555670da48,
       iov_len = 0}, {iov_base = 0xb1, iov_len = 93825009696736}, {
       iov_base = 0x5555567066b0, iov_len = 0}, {iov_base = 0x21,
       iov_len = 93825010822032}, {iov_base = 0x55555670e8f0, iov_len = 
64}, {
       iov_base = 0x30, iov_len = 93825010821312}, {iov_base = 
0x55555670eb50,
       iov_len = 0}, {iov_base = 0x55555670df90, iov_len = 0}, {
       iov_base = 0x41, iov_len = 93825008729792}, {iov_base = 
0x55555670eab0,
       iov_len = 32}, {iov_base = 0x20, iov_len = 93825010821808}, {
       iov_base = 0x0, iov_len = 288}, {iov_base = 0x30,
       iov_len = 93825010821760}, {iov_base = 0x55555670ebc0, iov_len = 
0}, {
       iov_base = 0x55555670da40, iov_len = 0}, {iov_base = 0x17121,
       iov_len = 93825009372400}, {iov_base = 0x5555565fbfe0, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 140737328184728}, {iov_base = 
0x5555565f9ea0,
       iov_len = 0}, {iov_base = 0x0, iov_len = 0} <repeats 255 times>, {
       iov_base = 0x0, iov_len = 4160}, {iov_base = 0x30,
       iov_len = 93825009721504}, {iov_base = 0x0, iov_len = 
93825010826368}, {
       iov_base = 0x3, iov_len = 3}, {iov_base = 0x160b1,
       iov_len = 93825009372400}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 93825010826304}, {iov_base = 
0x55555670fc50,
       iov_len = 0}, {iov_base = 0x55555670edd0, iov_len = 0}, {
       iov_base = 0x16061, iov_len = 93825009372400}, {
       iov_base = 0x7ffff673f778, iov_len = 0}, {iov_base = 0x0,
       iov_len = 93825009688224}, {iov_base = 0x7ffff673fd98,
       iov_len = 93825010826464}, {iov_base = 0x55555670fce0, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 0} <repeats 254 times>, {iov_base = 0x0,
       iov_len = 4160}, {iov_base = 0x30, iov_len = 93825010830976}, {
       iov_base = 0x55555670fcf0, iov_len = 0}, {iov_base = 0x55555670f060,
       iov_len = 0}, {iov_base = 0x71, iov_len = 93825008729792}, {
       iov_base = 0x555556710eb0, iov_len = 0}, {iov_base = 0x31,
       iov_len = 93825010821120}, {iov_base = 0x7ffff673f798,
       iov_len = 93825010821136}, {iov_base = 0x5555565fbbe8, iov_len = 
80}, {
       iov_base = 0x20, iov_len = 93825010831136}, {iov_base = 0x0,
       iov_len = 4320}, {iov_base = 0x30, iov_len = 93825010831088}, {
       iov_base = 0x555556710d90, iov_len = 0}, {iov_base = 0x555556710700,
       iov_len = 0}, {iov_base = 0x41, iov_len = 93825010831136}, {
       iov_base = 0x555556711070, iov_len = 32}, {iov_base = 0x20,
       iov_len = 93825010831472}, {iov_base = 0x0, iov_len = 4432}, {
       iov_base = 0x30, iov_len = 93825010831424}, {iov_base = 
0x555556710e00,
       iov_len = 0}, {iov_base = 0x5555567101b8, iov_len = 0}, {
       iov_base = 0x191, iov_len = 93825009372400}, {iov_base = 
0x555556711070,
       iov_len = 0}, {iov_base = 0x21, iov_len = 93825009374992}, {
       iov_base = 0x555556710dd0, iov_len = 64}, {iov_base = 0x30,
       iov_len = 93825010830752}, {iov_base = 0x555556710e70, iov_len = 
0}, {
       iov_base = 0x555556710b18, iov_len = 0}, {iov_base = 0x41,
       iov_len = 93825008729792}, {iov_base = 0x555556710f20, iov_len = 
32}, {
       iov_base = 0x20, iov_len = 93825010831248}, {iov_base = 0x0,
       iov_len = 176}, {iov_base = 0x30, iov_len = 93825010831200}, {
       iov_base = 0x555556710ee0, iov_len = 0}, {iov_base = 0x5555567105d0,
       iov_len = 0}, {iov_base = 0x41, iov_len = 93825008729792}, {
       iov_base = 0x555556711070, iov_len = 32}, {iov_base = 0x20,
       iov_len = 93825010831584}, {iov_base = 0x0, iov_len = 288}, {
       iov_base = 0x30, iov_len = 93825010831536}, {iov_base = 
0x555556710f50,
       iov_len = 0}, {iov_base = 0x555556710550, iov_len = 0}, {
       iov_base = 0x41, iov_len = 93825010831360}, {iov_base = 
0x55555670fc70,
       iov_len = 32}, {iov_base = 0x20, iov_len = 93825010826432}, {
       iov_base = 0x0, iov_len = 4880}, {iov_base = 0x30,
       iov_len = 93825010826384}, {iov_base = 0x555556710fc0, iov_len = 
0}, {
       iov_base = 0x55555670ff68, iov_len = 0}, {iov_base = 0x14d21,
       iov_len = 93825009372400}, {iov_base = 0x7ffff673f778, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 93825010831472}, {iov_base = 
0x555556711150,
       iov_len = 64}, {iov_base = 0x30, iov_len = 93825010831648}, {
       iov_base = 0x555556711030, iov_len = 0}, {iov_base = 0x55555670ff98,
       iov_len = 0}, {iov_base = 0xb1, iov_len = 93825009372400}, {
       iov_base = 0x55555670fcc0, iov_len = 0}, {iov_base = 0x21,
       iov_len = 93825010831584}, {iov_base = 0x555556710e40, iov_len = 
64}, {
       iov_base = 0x30, iov_len = 93825010830864}, {iov_base = 
0x5555567110a0,
       iov_len = 0}, {iov_base = 0x5555567104e0, iov_len = 0}, {
       iov_base = 0x41, iov_len = 93825010831696}, {iov_base = 
0x555556711000,
       iov_len = 32}, {iov_base = 0x20, iov_len = 93825010831360}, {
       iov_base = 0x0, iov_len = 288}, {iov_base = 0x30,
       iov_len = 93825010831312}, {iov_base = 0x555556711110, iov_len = 
0}, {
       iov_base = 0x55555670ff90, iov_len = 0}, {iov_base = 0x14bd1,
       iov_len = 93825008729792}, {iov_base = 0x55555670fc70, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 140737328185080}, {iov_base = 
0x7ffff673fef8,
       iov_len = 93825010831728}, {iov_base = 0x555556711170, iov_len = 
0}, {
       iov_base = 0x0, iov_len = 0} <repeats 255 times>...}}

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Stefan Hajnoczi]

On Tue, Oct 30, 2012 at 4:56 PM, Peter Lieven <pl@dlhnet.de> wrote:
> On 30.10.2012 09:32, Stefan Hajnoczi wrote:
>>
>> On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
>>>
>>> Hi,
>>
>> Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
>> different virtio device type from virtoi-blk and is not present in the
>> backtrace you posted.
>>
>> Sounds pedantic but I want to make sure this gets chalked up against the
>> right device :).
>>
>>> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
>>> storage backend that supports iSCSI
>>> qemu-kvm crashes reliably with the following error:
>>
>> Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
>>
>> Have you tried qemu-kvm.git/master?
>>
>> Have you tried a local raw disk image to check whether libiscsi is
>> involved?
>>
>>> Bad ram pointer 0x3039303620008000
>>>
>>> This happens directly after the confirmation of the Timezone before
>>> the Disk is partitioned.
>>>
>>> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
>>> does not happen.
>>>
>>> Here is a stack trace:
>>>
>>> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
>>> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>> No symbol table info available.
>>> #1 <https://github.com/sahlberg/libiscsi/issues/1>
>>> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
>>> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
>>> ram_addr = 0
>>> #2 <https://github.com/sahlberg/libiscsi/issues/2>
>>> 0x00005555557b9177 in cpu_physical_memory_unmap (
>>> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
>>> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
>>
>> buffer and len are ASCII junk.  It appears to be hex digits and it's not
>> clear where they come from.
>>
>> It would be interesting to print *elem one stack frame up in #3
>> virtqueue_fill() to show the iovecs and in/out counts.
>
>
> (gdb) print *elem

Great, thanks for providing this info:

> $6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
>     2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
>     140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
>     2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
>     12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},

Up to here everything is fine.

>in_sg =
> {{
>       iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
>       iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {

The fields are bogus, in_sg has been overwritten with ASCII data.
Unfortunately I don't see any hint of where this ASCII data came from
yet.

The hdr fields you provided in stack frame #6 show that in_sg was
overwritten during or after the bdrv_ioctl() call.  We pulled valid
data out of the vring and mapped buffers correctly.  But something is
overwriting in_sg and when we complete the request we blow up due to
the bogus values.

Please post your full qemu-kvm command-line.

Please also post the exact qemu-kvm version you are using.  I can see
it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g.
distro packages may carry patches so the full package version
information would be useful)?

Thanks,
Stefan

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:
> On Tue, Oct 30, 2012 at 4:56 PM, Peter Lieven <pl@dlhnet.de> wrote:
>> On 30.10.2012 09:32, Stefan Hajnoczi wrote:
>>> On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
>>>> Hi,
>>> Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
>>> different virtio device type from virtoi-blk and is not present in the
>>> backtrace you posted.
>>>
>>> Sounds pedantic but I want to make sure this gets chalked up against the
>>> right device :).
>>>
>>>> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
>>>> storage backend that supports iSCSI
>>>> qemu-kvm crashes reliably with the following error:
>>> Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
>>>
>>> Have you tried qemu-kvm.git/master?
>>>
>>> Have you tried a local raw disk image to check whether libiscsi is
>>> involved?
>>>
>>>> Bad ram pointer 0x3039303620008000
>>>>
>>>> This happens directly after the confirmation of the Timezone before
>>>> the Disk is partitioned.
>>>>
>>>> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
>>>> does not happen.
>>>>
>>>> Here is a stack trace:
>>>>
>>>> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
>>>> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>> No symbol table info available.
>>>> #1 <https://github.com/sahlberg/libiscsi/issues/1>
>>>> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
>>>> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
>>>> ram_addr = 0
>>>> #2 <https://github.com/sahlberg/libiscsi/issues/2>
>>>> 0x00005555557b9177 in cpu_physical_memory_unmap (
>>>> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
>>>> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
>>> buffer and len are ASCII junk.  It appears to be hex digits and it's not
>>> clear where they come from.
>>>
>>> It would be interesting to print *elem one stack frame up in #3
>>> virtqueue_fill() to show the iovecs and in/out counts.
>>
>> (gdb) print *elem
> Great, thanks for providing this info:
>
>> $6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
>>      2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
>>      140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
>>      2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
>>      12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},
> Up to here everything is fine.
>
>> in_sg =
>> {{
>>        iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
>>        iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
> The fields are bogus, in_sg has been overwritten with ASCII data.
> Unfortunately I don't see any hint of where this ASCII data came from
> yet.
>
> The hdr fields you provided in stack frame #6 show that in_sg was
> overwritten during or after the bdrv_ioctl() call.  We pulled valid
> data out of the vring and mapped buffers correctly.  But something is
> overwriting in_sg and when we complete the request we blow up due to
> the bogus values.
Ok. What I have to mention. I've been testing with qemu-kvm 1.2.0
and libiscsi for a few weeks now. Its been very stable. The only thing
it blows up is during the debian/ubuntu installer. Ubuntu itself for
instance is running flawlessly. My guess is that the installer is probing
for something. The installer itself also runs flawlessly when I disable
scsi passthru with scsi=off.
>
> Please post your full qemu-kvm command-line.
/usr/bin/qemu-kvm-1.2.0  -net 
tap,vlan=164,script=no,downscript=no,ifname=tap0  -net 
nic,vlan=164,model=e1000,macaddr=52:54:00:ff:01:35   -iscsi 
initiator-name=iqn.2005-03.org.virtual-core:0025b51f001c  -drive 
format=iscsi,file=iscsi://172.21.200.56/iqn.2001-05.com.equallogic:0-8a0906-335f4e007-d29001a3355508e8-libiscsi-test-hd0/0,if=virtio,cache=none,aio=native 
-m 2048 -smp 2,sockets=1,cores=2,threads=1  -monitor 
tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name 
'libiscsi-debug'  -boot order=dc,menu=off  -k de  -pidfile 
/var/run/qemu/vm-280.pid  -mem-path /hugepages  -mem-prealloc  -cpu 
host,+x2apic,model_id='Intel(R) Xeon(R) CPU           L5640  @ 
2.27GHz',-tsc  -rtc base=utc -usb -usbdevice tablet -no-hpet -vga cirrus
>
> Please also post the exact qemu-kvm version you are using.  I can see
> it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g.
> distro packages may carry patches so the full package version
> information would be useful)?
I use vanilly qemu-kvm 1.2.0 with some cherry picked patches. I will
retry with untouched qemu-kvm 1.2.0 and latest git tomorrow at latest.
>
> Thanks,
> Stefan
Thank you, too
Peter

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[ronnie sahlberg]

But older distros/kernels work fine?

Can you take a network trace?

About half a year there was an issue where recent kernels had added
support to start using new scsi opcodes,  but the qemu functions that
determine "which transfer direction is used for this opcode" had not
yet been updated, so that the opcode was sent with the wrong transfer
direction.

That caused the guests memory to be overwritten and crash.

I dont have (easy) access to the git tree right now, but it was a
patch for the ATA_PASSTHROUGH command that fixed that.

Could you take a network trace and check maybe this is something
similar ?  I.e. does the guest send an "unusual" scsi opcode just
before the crash ?


regards
ronnie sahlberg


On Tue, Oct 30, 2012 at 12:37 PM, Peter Lieven <pl@dlhnet.de> wrote:
> Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:
>
>> On Tue, Oct 30, 2012 at 4:56 PM, Peter Lieven <pl@dlhnet.de> wrote:
>>>
>>> On 30.10.2012 09:32, Stefan Hajnoczi wrote:
>>>>
>>>> On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
>>>>>
>>>>> Hi,
>>>>
>>>> Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
>>>> different virtio device type from virtoi-blk and is not present in the
>>>> backtrace you posted.
>>>>
>>>> Sounds pedantic but I want to make sure this gets chalked up against the
>>>> right device :).
>>>>
>>>>> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
>>>>> storage backend that supports iSCSI
>>>>> qemu-kvm crashes reliably with the following error:
>>>>
>>>> Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
>>>>
>>>> Have you tried qemu-kvm.git/master?
>>>>
>>>> Have you tried a local raw disk image to check whether libiscsi is
>>>> involved?
>>>>
>>>>> Bad ram pointer 0x3039303620008000
>>>>>
>>>>> This happens directly after the confirmation of the Timezone before
>>>>> the Disk is partitioned.
>>>>>
>>>>> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
>>>>> does not happen.
>>>>>
>>>>> Here is a stack trace:
>>>>>
>>>>> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
>>>>> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>>> No symbol table info available.
>>>>> #1 <https://github.com/sahlberg/libiscsi/issues/1>
>>>>> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
>>>>> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
>>>>> ram_addr = 0
>>>>> #2 <https://github.com/sahlberg/libiscsi/issues/2>
>>>>> 0x00005555557b9177 in cpu_physical_memory_unmap (
>>>>> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
>>>>> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
>>>>
>>>> buffer and len are ASCII junk.  It appears to be hex digits and it's not
>>>> clear where they come from.
>>>>
>>>> It would be interesting to print *elem one stack frame up in #3
>>>> virtqueue_fill() to show the iovecs and in/out counts.
>>>
>>>
>>> (gdb) print *elem
>>
>> Great, thanks for providing this info:
>>
>>> $6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960,
>>> 1916656688,
>>>      2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
>>>      140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
>>>      2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512
>>> times>,
>>>      12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},
>>
>> Up to here everything is fine.
>>
>>> in_sg =
>>> {{
>>>        iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
>>>        iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
>>
>> The fields are bogus, in_sg has been overwritten with ASCII data.
>> Unfortunately I don't see any hint of where this ASCII data came from
>> yet.
>>
>> The hdr fields you provided in stack frame #6 show that in_sg was
>> overwritten during or after the bdrv_ioctl() call.  We pulled valid
>> data out of the vring and mapped buffers correctly.  But something is
>> overwriting in_sg and when we complete the request we blow up due to
>> the bogus values.
>
> Ok. What I have to mention. I've been testing with qemu-kvm 1.2.0
> and libiscsi for a few weeks now. Its been very stable. The only thing
> it blows up is during the debian/ubuntu installer. Ubuntu itself for
> instance is running flawlessly. My guess is that the installer is probing
> for something. The installer itself also runs flawlessly when I disable
> scsi passthru with scsi=off.
>
>>
>> Please post your full qemu-kvm command-line.
>
> /usr/bin/qemu-kvm-1.2.0  -net
> tap,vlan=164,script=no,downscript=no,ifname=tap0  -net
> nic,vlan=164,model=e1000,macaddr=52:54:00:ff:01:35   -iscsi
> initiator-name=iqn.2005-03.org.virtual-core:0025b51f001c  -drive
> format=iscsi,file=iscsi://172.21.200.56/iqn.2001-05.com.equallogic:0-8a0906-335f4e007-d29001a3355508e8-libiscsi-test-hd0/0,if=virtio,cache=none,aio=native
> -m 2048 -smp 2,sockets=1,cores=2,threads=1  -monitor
> tcp:0:4002,server,nowait -vnc :2 -qmp tcp:0:3002,server,nowait -name
> 'libiscsi-debug'  -boot order=dc,menu=off  -k de  -pidfile
> /var/run/qemu/vm-280.pid  -mem-path /hugepages  -mem-prealloc  -cpu
> host,+x2apic,model_id='Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz',-tsc
> -rtc base=utc -usb -usbdevice tablet -no-hpet -vga cirrus
>
>>
>> Please also post the exact qemu-kvm version you are using.  I can see
>> it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g.
>> distro packages may carry patches so the full package version
>> information would be useful)?
>
> I use vanilly qemu-kvm 1.2.0 with some cherry picked patches. I will
> retry with untouched qemu-kvm 1.2.0 and latest git tomorrow at latest.
>>
>>
>> Thanks,
>> Stefan
>
> Thank you, too
> Peter

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Stefan Hajnoczi]

On Tue, Oct 30, 2012 at 10:09 PM, ronnie sahlberg
<ronniesahlberg@gmail.com> wrote:
> About half a year there was an issue where recent kernels had added
> support to start using new scsi opcodes,  but the qemu functions that
> determine "which transfer direction is used for this opcode" had not
> yet been updated, so that the opcode was sent with the wrong transfer
> direction.
>
> That caused the guests memory to be overwritten and crash.
>
> I dont have (easy) access to the git tree right now, but it was a
> patch for the ATA_PASSTHROUGH command that fixed that.

This patch?

http://patchwork.ozlabs.org/patch/174946/

Stefan

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[ronnie sahlberg]

On Tue, Oct 30, 2012 at 10:48 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Tue, Oct 30, 2012 at 10:09 PM, ronnie sahlberg
> <ronniesahlberg@gmail.com> wrote:
>> About half a year there was an issue where recent kernels had added
>> support to start using new scsi opcodes,  but the qemu functions that
>> determine "which transfer direction is used for this opcode" had not
>> yet been updated, so that the opcode was sent with the wrong transfer
>> direction.
>>
>> That caused the guests memory to be overwritten and crash.
>>
>> I dont have (easy) access to the git tree right now, but it was a
>> patch for the ATA_PASSTHROUGH command that fixed that.
>
> This patch?
>
> http://patchwork.ozlabs.org/patch/174946/
>
> Stefan

This is the one I was thinking about :
381b634c275ca1a2806e97392527bbfc01bcb333

But that also crashed when using local /dev/sg* devices.

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

Am 31.10.2012 um 15:08 schrieb ronnie sahlberg:

> On Tue, Oct 30, 2012 at 10:48 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Tue, Oct 30, 2012 at 10:09 PM, ronnie sahlberg
>> <ronniesahlberg@gmail.com> wrote:
>>> About half a year there was an issue where recent kernels had added
>>> support to start using new scsi opcodes,  but the qemu functions that
>>> determine "which transfer direction is used for this opcode" had not
>>> yet been updated, so that the opcode was sent with the wrong transfer
>>> direction.
>>> 
>>> That caused the guests memory to be overwritten and crash.
>>> 
>>> I dont have (easy) access to the git tree right now, but it was a
>>> patch for the ATA_PASSTHROUGH command that fixed that.
>> 
>> This patch?
>> 
>> http://patchwork.ozlabs.org/patch/174946/
>> 
>> Stefan
> 
> This is the one I was thinking about :
> 381b634c275ca1a2806e97392527bbfc01bcb333
> 
> But that also crashed when using local /dev/sg* devices.

I was using a local LVM Volume not an iSCSI disk. 

I added debugging output and breakpoints to scsi_cmd_xfer_mode().
The function is not called.

Peter

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

Has anyone any other idea what the cause could be or where to start?

Peter

Am 31.10.2012 um 15:08 schrieb ronnie sahlberg:

> On Tue, Oct 30, 2012 at 10:48 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Tue, Oct 30, 2012 at 10:09 PM, ronnie sahlberg
>> <ronniesahlberg@gmail.com> wrote:
>>> About half a year there was an issue where recent kernels had added
>>> support to start using new scsi opcodes,  but the qemu functions that
>>> determine "which transfer direction is used for this opcode" had not
>>> yet been updated, so that the opcode was sent with the wrong transfer
>>> direction.
>>> 
>>> That caused the guests memory to be overwritten and crash.
>>> 
>>> I dont have (easy) access to the git tree right now, but it was a
>>> patch for the ATA_PASSTHROUGH command that fixed that.
>> 
>> This patch?
>> 
>> http://patchwork.ozlabs.org/patch/174946/
>> 
>> Stefan
> 
> This is the one I was thinking about :
> 381b634c275ca1a2806e97392527bbfc01bcb333
> 
> But that also crashed when using local /dev/sg* devices.

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Stefan Hajnoczi]

On Thu, Nov 8, 2012 at 4:26 PM, Peter Lieven <pl@dlhnet.de> wrote:
> Has anyone any other idea what the cause could be or where to start?

Hi Peter,
I suggested posting the source tree you are building.  Since you have
applied patches yourself no one else is able to follow along with the
gdb output or reproduce the issue accurately.

Stefan

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

On 19.11.2012 18:20, Stefan Hajnoczi wrote:
> On Thu, Nov 8, 2012 at 4:26 PM, Peter Lieven <pl@dlhnet.de> wrote:
>> Has anyone any other idea what the cause could be or where to start?
>
> Hi Peter,
> I suggested posting the source tree you are building.  Since you have
> applied patches yourself no one else is able to follow along with the
> gdb output or reproduce the issue accurately.

Sorry for the late reply, I used qemu git at e24dc9feb0d68142d54dc3c097f57588836d1338
and libiscsi git at 3b3036b9dae55f0c3eef9d75db89c7b78f637a12.

The cmdline:
qemu-system-x86_64 -enable-kvm -m 1024 -drive if=virtio,file=iscsi://172.21.200.56/iqn.2001-05.com.equallogic:0-8a0906-62ff4e007-e4a3c8908af50839-test-3000g/0 -cdrom ubuntu-12.04.1-server-amd64.iso -vnc :1

The vm crashes with:

Bad ram pointer 0x7fd220008000

after the user settings and timezone config when loading the module
libdmraid1.0.0.rc16-udeb

I hope this helps to reproduce.

Peter

>
> Stefan
>

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Stefan Hajnoczi]

On Tue, Oct 30, 2012 at 8:37 PM, Peter Lieven <pl@dlhnet.de> wrote:
> Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:
>> Please also post the exact qemu-kvm version you are using.  I can see
>> it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g.
>> distro packages may carry patches so the full package version
>> information would be useful)?
>
> I use vanilly qemu-kvm 1.2.0 with some cherry picked patches. I will
> retry with untouched qemu-kvm 1.2.0 and latest git tomorrow at latest.

Please post the source tree including applied patches.  For example,
you could push to a public git repo.  This is necessary so that we can
dig into the source tree to figure out where the bug lies.

There are some other interesting questions that Ronnie has raised, so
perhaps they will lead to the answer if you cannot publish the tree at
the moment.

My next question is: have you tested qemu.git/master?

Stefan

Re: [Qemu-devel] Ubuntu/Debian Installer + Virtio-SCSI -> Bad ram pointer

[Peter Lieven]

Am 30.10.2012 19:27, schrieb Stefan Hajnoczi:
> On Tue, Oct 30, 2012 at 4:56 PM, Peter Lieven <pl@dlhnet.de> wrote:
>> On 30.10.2012 09:32, Stefan Hajnoczi wrote:
>>> On Mon, Oct 29, 2012 at 03:09:37PM +0100, Peter Lieven wrote:
>>>> Hi,
>>> Bug subject should be virtio-blk, not virtio-scsi.  virtio-scsi is a
>>> different virtio device type from virtoi-blk and is not present in the
>>> backtrace you posted.
>>>
>>> Sounds pedantic but I want to make sure this gets chalked up against the
>>> right device :).
>>>
>>>> If I try to Install Ubuntu 12.04 LTS / 12.10 64-bit on a virtio
>>>> storage backend that supports iSCSI
>>>> qemu-kvm crashes reliably with the following error:
>>> Are you using vanilla qemu-kvm-1.2.0 or are there patches applied?
>>>
>>> Have you tried qemu-kvm.git/master?
>>>
>>> Have you tried a local raw disk image to check whether libiscsi is
>>> involved?
>>>
>>>> Bad ram pointer 0x3039303620008000
>>>>
>>>> This happens directly after the confirmation of the Timezone before
>>>> the Disk is partitioned.
>>>>
>>>> If I specify  -global virtio-blk-pci.scsi=off in the cmdline this
>>>> does not happen.
>>>>
>>>> Here is a stack trace:
>>>>
>>>> Thread 1 (Thread 0x7ffff7fee700 (LWP 8226)):
>>>> #0 0x00007ffff63c0a10 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>>>> No symbol table info available.
>>>> #1 <https://github.com/sahlberg/libiscsi/issues/1>
>>>> 0x00005555557b751d in qemu_ram_addr_from_host_nofail (
>>>> ptr=0x3039303620008000) at /usr/src/qemu-kvm-1.2.0/exec.c:2835
>>>> ram_addr = 0
>>>> #2 <https://github.com/sahlberg/libiscsi/issues/2>
>>>> 0x00005555557b9177 in cpu_physical_memory_unmap (
>>>> buffer=0x3039303620008000, len=4986663671065686081, is_write=1,
>>>> access_len=1) at /usr/src/qemu-kvm-1.2.0/exec.c:3645
>>> buffer and len are ASCII junk.  It appears to be hex digits and it's not
>>> clear where they come from.
>>>
>>> It would be interesting to print *elem one stack frame up in #3
>>> virtqueue_fill() to show the iovecs and in/out counts.
>>
>> (gdb) print *elem
> Great, thanks for providing this info:
>
>> $6 = {index = 3, out_num = 2, in_num = 4, in_addr = {1914920960, 1916656688,
>>      2024130072, 2024130088, 0 <repeats 508 times>, 4129, 93825009696000,
>>      140737328183160, 0 <repeats 509 times>}, out_addr = {2024130056,
>>      2038414056, 0, 8256, 4128, 93824999311936, 0, 3, 0 <repeats 512 times>,
>>      12385, 93825009696000, 140737328183160, 0 <repeats 501 times>},
> Up to here everything is fine.
>
>> in_sg =
>> {{
>>        iov_base = 0x3039303620008000, iov_len = 4986663671065686081}, {
>>        iov_base = 0x3830384533334635, iov_len = 3544389261899019573}, {
> The fields are bogus, in_sg has been overwritten with ASCII data.
> Unfortunately I don't see any hint of where this ASCII data came from
> yet.
>
> The hdr fields you provided in stack frame #6 show that in_sg was
> overwritten during or after the bdrv_ioctl() call.  We pulled valid
> data out of the vring and mapped buffers correctly.  But something is
> overwriting in_sg and when we complete the request we blow up due to
> the bogus values.
>
> Please post your full qemu-kvm command-line.
>
> Please also post the exact qemu-kvm version you are using.  I can see
> it's based on qemu-kvm-1.2.0 but are there any patches applied (e.g.
> distro packages may carry patches so the full package version
> information would be useful)?
Stefan, Ronnie, if I do remove the following patch from my cherry-picked 
patches its
working again:

iSCSI: We need to support SG_IO also from iscsi_ioctl()

Peter