loop in copy_user_generic_string

loop in copy_user_generic_string

[Zdenek Kabelac]

Hello


I'm having weird problem and being a bit puzzled about where to look
for this bug.

I'm using T61 - C2D  2GB

So I'll describe symptoms:

When I run inside my 0.5G smp  qemu-kvm guest with Debian these two
loops in parallel:

'while : ; do dmsetup status  ; done'

and

'while : ; do cat /dev/zero >/dev/mapper/any_free_to_use_lvm_partition ; done'

after a while dmsetup start to loop in this place:

[  356.257323] dmsetup       R  running task        0  3385   2342
[  356.257323]  ffff81001c9fbc58 ffffffff812ee85e 0000000000000000
0000000000000001
[  356.257323]  0000000000070617 0000000000000001 ffff81001be94000
ffff81001f864000
[  356.257323]  ffff81001be94360 000000018100cddc ffff81001be94000
ffffffff812f13b1
[  356.257323] Call Trace:
[  356.257323]  [<ffffffff812ee85e>] ? thread_return+0x99/0x51b
[  356.257323]  [<ffffffff812f13b1>] ? trace_hardirqs_on_thunk+0x35/0x3a
[  356.257323]  [<ffffffff81065749>] ? trace_hardirqs_on+0x139/0x1a0
[  356.257323]  [<ffffffff812f13b1>] ? trace_hardirqs_on_thunk+0x35/0x3a
[  356.257323]  [<ffffffff8100cddc>] ? restore_args+0x0/0x30
[  356.257323]  [<ffffffff8117c017>] ? copy_user_generic_string+0x17/0x40
[  356.257323]  [<ffffffff8804c527>] ? :dm_mod:copy_params+0x87/0xb0
[  356.257323]  [<ffffffff81047d11>] ? __capable+0x11/0x30
[  356.257323]  [<ffffffff8804c6b9>] ? :dm_mod:ctl_ioctl+0x169/0x260
[  356.257323]  [<ffffffff8804c7dd>] ? :dm_mod:dm_compat_ctl_ioctl+0xd/0x20
[  356.257323]  [<ffffffff810f6c12>] ? compat_sys_ioctl+0x182/0x3d0
[  356.257323]  [<ffffffff810bbba0>] ? vfs_write+0x130/0x170
[  356.257323]  [<ffffffff812f13eb>] ? trace_hardirqs_off_thunk+0x35/0x37
[  356.257323]  [<ffffffff8102aec2>] ? sysenter_do_call+0x1b/0x70
[  356.257323]  [<ffffffff812f13b1>] ? trace_hardirqs_on_thunk+0x35/0x3a


I'm using preemptible kernel and the code will stay in the
copy_user_generic_string call forever eating 100%cpu - without
preemption the kernel gets dead.

With preemption when I run at this moment second dmsetup status in
paralllel the busy-looped dmsetup gets finished and while loop starts
to continue agains until next dmsetup busy-loop.

I've noticed that if I change inside  drivers/md/dm-ioctl.c
copy_params  the parameter tmp.data_size in the copy_from_user call to
just page size (4kB) - or when I replace vmalloc to kmalloc - the busy
loop will not happen.

So it seems to be related to page jump somehow

Anyway might have any idea - what is going on here ?

Thanks

Zdenek

Cc me please I'm not a litst member.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Avi Kivity]

Zdenek Kabelac wrote:
> Hello
>
>
> I'm having weird problem and being a bit puzzled about where to look
> for this bug.
>
> I'm using T61 - C2D  2GB
>
> So I'll describe symptoms:
>
> When I run inside my 0.5G smp  qemu-kvm guest with Debian these two
> loops in parallel:
>
> 'while : ; do dmsetup status  ; done'
>
> and
>
> 'while : ; do cat /dev/zero >/dev/mapper/any_free_to_use_lvm_partition ; done'
>
> after a while dmsetup start to loop in this place:
>
> [  356.257323]  [<ffffffff8117c017>] ? copy_user_generic_string+0x17/0x40
>
>
> I'm using preemptible kernel and the code will stay in the
> copy_user_generic_string call forever eating 100%cpu - without
> preemption the kernel gets dead.
>
> With preemption when I run at this moment second dmsetup status in
> paralllel the busy-looped dmsetup gets finished and while loop starts
> to continue agains until next dmsetup busy-loop.
>
> I've noticed that if I change inside  drivers/md/dm-ioctl.c
> copy_params  the parameter tmp.data_size in the copy_from_user call to
> just page size (4kB) - or when I replace vmalloc to kmalloc - the busy
> loop will not happen.
>
> So it seems to be related to page jump somehow
>
> Anyway might have any idea - what is going on here ?
>   

Most likely movs emulation is broken for long counts.  Please post a 
disassembly of copy_user_generic_string to make sure we're looking at 
the same code.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

2008/3/4, Avi Kivity <avi@qumranet.com>:
> Zdenek Kabelac wrote:
>  > Hello
>  >
>  >
>  > I'm having weird problem and being a bit puzzled about where to look
>  > for this bug.
>  >
>  > I'm using T61 - C2D  2GB
>  >
>  > So I'll describe symptoms:
>  >
>  > When I run inside my 0.5G smp  qemu-kvm guest with Debian these two
>  > loops in parallel:
>  >
>  > 'while : ; do dmsetup status  ; done'
>  >
>  > and
>  >
>  > 'while : ; do cat /dev/zero >/dev/mapper/any_free_to_use_lvm_partition ; done'
>  >
>  > after a while dmsetup start to loop in this place:
>  >
>
> > [  356.257323]  [<ffffffff8117c017>] ? copy_user_generic_string+0x17/0x40
>  >
>  >
>
> > I'm using preemptible kernel and the code will stay in the
>  > copy_user_generic_string call forever eating 100%cpu - without
>  > preemption the kernel gets dead.
>  >
>  > With preemption when I run at this moment second dmsetup status in
>  > paralllel the busy-looped dmsetup gets finished and while loop starts
>  > to continue agains until next dmsetup busy-loop.
>  >
>  > I've noticed that if I change inside  drivers/md/dm-ioctl.c
>  > copy_params  the parameter tmp.data_size in the copy_from_user call to
>  > just page size (4kB) - or when I replace vmalloc to kmalloc - the busy
>  > loop will not happen.
>  >
>  > So it seems to be related to page jump somehow
>  >
>  > Anyway might have any idea - what is going on here ?
>  >
>
>
> Most likely movs emulation is broken for long counts.  Please post a
>  disassembly of copy_user_generic_string to make sure we're looking at
>  the same code.
>

Is it emulated ? I've thought it's running natively with vmx?

Anyway here is  objdump of copy_user_64.o
(loops on 0x1d7)

00000000000001c0 <copy_user_generic_string>:
 1c0:	41 89 c8             	mov    %ecx,%r8d
 1c3:	89 d1                	mov    %edx,%ecx
 1c5:	c1 e9 03             	shr    $0x3,%ecx
 1c8:	83 e2 07             	and    $0x7,%edx
 1cb:	74 0a                	je     1d7 <copy_user_generic_string+0x17>
 1cd:	f3 48 a5             	rep movsq %ds:(%rsi),%es:(%rdi)
 1d0:	89 d1                	mov    %edx,%ecx
 1d2:	f3 a4                	rep movsb %ds:(%rsi),%es:(%rdi)
 1d4:	89 c8                	mov    %ecx,%eax
 1d6:	c3                   	retq
 1d7:	f3 48 a5             	rep movsq %ds:(%rsi),%es:(%rdi)
 1da:	31 c0                	xor    %eax,%eax
 1dc:	c3                   	retq
 1dd:	48 8d 04 ca          	lea    (%rdx,%rcx,8),%rax
 1e1:	eb 02                	jmp    1e5 <copy_user_generic_string+0x25>
 1e3:	89 c8                	mov    %ecx,%eax
 1e5:	45 85 c0             	test   %r8d,%r8d
 1e8:	74 08                	je     1f2 <copy_user_generic_string+0x32>
 1ea:	89 c1                	mov    %eax,%ecx
 1ec:	50                   	push   %rax
 1ed:	31 c0                	xor    %eax,%eax
 1ef:	f3 aa                	rep stos %al,%es:(%rdi)
 1f1:	58                   	pop    %rax
 1f2:	c3                   	retq

Zdenek

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Avi Kivity]

Zdenek Kabelac wrote:
> Is it emulated ? I've thought it's running natively with vmx?
>
>   

In some cases (memory mapped I/O, writes to page tables) some 
instructions are emulated.  Usually they run natively.

Please post the output of 'kvm_stat -1' to ensure the problem is with 
the emulator.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Andi Kleen]

Avi Kivity <avi@qumranet.com> writes:
>
> Most likely movs emulation is broken for long counts.  Please post a 
> disassembly of copy_user_generic_string to make sure we're looking at 
> the same code.

Be careful -- this code is patched at runtime and what you 
see in the vmlinux is not necessarily the same that is executed

Incidentially that might cause problems.

-Andi

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Avi Kivity]

Andi Kleen wrote:
> Avi Kivity <avi@qumranet.com> writes:
>   
>> Most likely movs emulation is broken for long counts.  Please post a 
>> disassembly of copy_user_generic_string to make sure we're looking at 
>> the same code.
>>     
>
> Be careful -- this code is patched at runtime and what you 
> see in the vmlinux is not necessarily the same that is executed
>
>   

If the disassembled instruction isn't marked as an alternative in the 
source, then it can't be patched, right?


> Incidentially that might cause problems.

Specific to kvm?  how?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

2008/3/5, Avi Kivity <avi@qumranet.com>:
> Andi Kleen wrote:
>  > Avi Kivity <avi@qumranet.com> writes:
>  >
>  >> Most likely movs emulation is broken for long counts.  Please post a
>  >> disassembly of copy_user_generic_string to make sure we're looking at
>  >> the same code.
>  >>
>  >
>  > Be careful -- this code is patched at runtime and what you
>  > see in the vmlinux is not necessarily the same that is executed
>  >
>  >
>
>
> If the disassembled instruction isn't marked as an alternative in the
>  source, then it can't be patched, right?
>
>
>
>  > Incidentially that might cause problems.
>
>
> Specific to kvm?  how?
>

As for me - I'm note sure were this bug come from - I just can easily
reproduce it on my box with Qemu-kvm - the problem could be also
directly in kernel - (either MMU or dm) - I just know the bug is not
reproducible with vmware nor natively running code.
On the other hand Qemu-kvm easily catches racing bugs compared with
native execution - so maybe it's exposing some MMU problem.

I've traced the problem to the instruction place - but I'm not sure
how to help more with this issue - so if anyone has some idea what
else should I check - let me know.

I've got an idea to replace rep movqs with plain  asm loop - does
anyone thinks it might be worth to check this ??

Zdenek

PS: In the attachment there is my config file - thought there is
probably nothing special

[-- Attachment #2: config.bz2 --]
[-- Type: application/x-bzip2, Size: 12984 bytes --]

[-- Attachment #3: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #4: Type: text/plain, Size: 158 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

2008/3/5, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
> 2008/3/5, Avi Kivity <avi@qumranet.com>:
>
> > Andi Kleen wrote:
>  >  > Avi Kivity <avi@qumranet.com> writes:
>  >  >
>  >  >> Most likely movs emulation is broken for long counts.  Please post a
>  >  >> disassembly of copy_user_generic_string to make sure we're looking at
>  >  >> the same code.
>  >  >>
>  >  >
>  >  > Be careful -- this code is patched at runtime and what you
>  >  > see in the vmlinux is not necessarily the same that is executed
>  >  >
>  >  >
>  >
>  >
>  > If the disassembled instruction isn't marked as an alternative in the
>  >  source, then it can't be patched, right?
>  >

Hello

Any progress on this - It looks like I get this bug quite often when I test
device-mapper code.

Should I test something special ?

Also I'm seeing some problems with nfs - not yet tracked down, but I'd
like to get bugs fixed after another. Also about two times qemu-kvm
coredumped - unfortunately it's compiled without debugs thus the
traceback was not really useful to make any report...

Zdenek

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

2008/3/7, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
> 2008/3/5, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
>
> > 2008/3/5, Avi Kivity <avi@qumranet.com>:
>  >
>  > > Andi Kleen wrote:
>  >  >  > Avi Kivity <avi@qumranet.com> writes:
>  >  >  >
>  >  >  >> Most likely movs emulation is broken for long counts.  Please post a
>  >  >  >> disassembly of copy_user_generic_string to make sure we're looking at
>  >  >  >> the same code.
>  >  >  >>
>  >  >  >
>  >  >  > Be careful -- this code is patched at runtime and what you
>  >  >  > see in the vmlinux is not necessarily the same that is executed
>  >  >  >
>  >  >  >
>  >  >
>  >  >
>  >  > If the disassembled instruction isn't marked as an alternative in the
>  >  >  source, then it can't be patched, right?
>  >  >
>
>
> Hello
>
>  Any progress on this - It looks like I get this bug quite often when I test
>  device-mapper code.
>


Hello

I've made some more experiments and noticed few more things:

a) - it is just enough to run parallel loop with cat LVM partition
>/dev/null and dmsetup status

b) when I insert for() loop for zeroing allocated memory in the
dm-ioctl copy_params function my loop start once the memory crosses
exactly 4KB boundary (visible from register content)

c) in my trace log I could usually always see this pattern:
[  160.634897]  [<ffffffff812ee5ba>] preempt_schedule_irq+0x5a/0xa0
[  160.634897]  [<ffffffff8100cf46>] retint_kernel+0x26/0x30

from the look in the arch/x86/kernel/entry64.s I could really see
there is some potentiality for internal loop that may call
preempt_schedule_irq in upon some check in  exit_intr - but having
actually now idea what's this all about...

I've put there just some extra dump_stack trace in the
preempt_schedule_irq - and it's really being printed - but quite
slowly actually considering process eats 100% CPU

So anyone has any idea what might be wrong ?

Zdenek

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

2008/3/9, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
> 2008/3/7, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
>
> > 2008/3/5, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
>  >
>  > > 2008/3/5, Avi Kivity <avi@qumranet.com>:
>  >  >
>  >  > > Andi Kleen wrote:
>  >  >  >  > Avi Kivity <avi@qumranet.com> writes:
>  >  >  >  >
>  >  >  >  >> Most likely movs emulation is broken for long counts.  Please post a
>  >  >  >  >> disassembly of copy_user_generic_string to make sure we're looking at
>  >  >  >  >> the same code.
>  >  >  >  >>
>  >  >  >  >
>  >  >  >  > Be careful -- this code is patched at runtime and what you
>  >  >  >  > see in the vmlinux is not necessarily the same that is executed
>  >  >  >  >
>  >  >  >  >
>  >  >  >
>  >  >  >
>  >  >  > If the disassembled instruction isn't marked as an alternative in the
>  >  >  >  source, then it can't be patched, right?
>  >  >  >
>  >
>  >
>  > Hello
>  >
>  >  Any progress on this - It looks like I get this bug quite often when I test
>  >  device-mapper code.
>  >
>
>
>
> Hello
>
>  I've made some more experiments and noticed few more things:
>
>  a) - it is just enough to run parallel loop with cat LVM partition
>  >/dev/null and dmsetup status
>
>  b) when I insert for() loop for zeroing allocated memory in the
>  dm-ioctl copy_params function my loop start once the memory crosses
>  exactly 4KB boundary (visible from register content)
>
>  c) in my trace log I could usually always see this pattern:
>  [  160.634897]  [<ffffffff812ee5ba>] preempt_schedule_irq+0x5a/0xa0
>  [  160.634897]  [<ffffffff8100cf46>] retint_kernel+0x26/0x30
>
>  from the look in the arch/x86/kernel/entry64.s I could really see
>  there is some potentiality for internal loop that may call
>  preempt_schedule_irq in upon some check in  exit_intr - but having
>  actually now idea what's this all about...
>
>  I've put there just some extra dump_stack trace in the
>  preempt_schedule_irq - and it's really being printed - but quite
>  slowly actually considering process eats 100% CPU
>
>  So anyone has any idea what might be wrong ?


Hello

I've some more news here - it looks I've found working setup on my C2D.

All I need to do is compile my 64bit kernel with optimization for space.
This will magical start to work - at least in this case.

I'll now probably slowly try to figure out which directory with -Os
compilation makes the difference.

Also I've noticed that standard Debian 2.6.24-4-686 kernel loops in
Qemu, but 486 version doesn't.

So if anyone starts to get idea what could be wrong...

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

2008/3/11, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
> 2008/3/9, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
>
> > 2008/3/7, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
>  >
>  > > 2008/3/5, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
>  >  >
>  >  > > 2008/3/5, Avi Kivity <avi@qumranet.com>:
>  >  >  >
>  >  >  > > Andi Kleen wrote:
>  >  >  >  >  > Avi Kivity <avi@qumranet.com> writes:
>  >  >  >  >  >
>  >  >  >  >  >> Most likely movs emulation is broken for long counts.  Please post a
>  >  >  >  >  >> disassembly of copy_user_generic_string to make sure we're looking at
>  >  >  >  >  >> the same code.
>  >  >  >  >  >>
>  >  >  >  >  >
>  >  >  >  >  > Be careful -- this code is patched at runtime and what you
>  >  >  >  >  > see in the vmlinux is not necessarily the same that is executed
>  >  >  >  >  >
>  >  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  > If the disassembled instruction isn't marked as an alternative in the
>  >  >  >  >  source, then it can't be patched, right?
>  >  >  >  >
>  >  >
>  >  >
>  >  > Hello
>  >  >
>  >  >  Any progress on this - It looks like I get this bug quite often when I test
>  >  >  device-mapper code.
>  >  >
>  >
>  >
>  >
>  > Hello
>  >
>  >  I've made some more experiments and noticed few more things:
>  >
>  >  a) - it is just enough to run parallel loop with cat LVM partition
>  >  >/dev/null and dmsetup status
>  >
>  >  b) when I insert for() loop for zeroing allocated memory in the
>  >  dm-ioctl copy_params function my loop start once the memory crosses
>  >  exactly 4KB boundary (visible from register content)
>  >
>  >  c) in my trace log I could usually always see this pattern:
>  >  [  160.634897]  [<ffffffff812ee5ba>] preempt_schedule_irq+0x5a/0xa0
>  >  [  160.634897]  [<ffffffff8100cf46>] retint_kernel+0x26/0x30
>  >
>  >  from the look in the arch/x86/kernel/entry64.s I could really see
>  >  there is some potentiality for internal loop that may call
>  >  preempt_schedule_irq in upon some check in  exit_intr - but having
>  >  actually now idea what's this all about...
>  >
>  >  I've put there just some extra dump_stack trace in the
>  >  preempt_schedule_irq - and it's really being printed - but quite
>  >  slowly actually considering process eats 100% CPU
>  >
>  >  So anyone has any idea what might be wrong ?
>
>
>
> Hello
>
>  I've some more news here - it looks I've found working setup on my C2D.
>
>  All I need to do is compile my 64bit kernel with optimization for space.
>  This will magical start to work - at least in this case.
>
>  I'll now probably slowly try to figure out which directory with -Os
>  compilation makes the difference.
>
>  Also I've noticed that standard Debian 2.6.24-4-686 kernel loops in
>  Qemu, but 486 version doesn't.

Argh - being stupid here - it looks like these 'working' kernels were
not SMP actually.
As long as the SMP is used - I still get the busy loop :(
Now being clueless....

Zdenek

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Andi Kleen]

> Argh - being stupid here - it looks like these 'working' kernels were
> not SMP actually.
> As long as the SMP is used - I still get the busy loop :(
> Now being clueless....

Sorry don't have the cycles to look into your problem, but the 
standard procedure for hard problems that can be reproduced
is to git bisect them down to the change set that introduced the
problem originally and then complain to whoever authored that.

-Andi

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

2008/3/12, Andi Kleen <andi@firstfloor.org>:
> > Argh - being stupid here - it looks like these 'working' kernels were
>  > not SMP actually.
>  > As long as the SMP is used - I still get the busy loop :(
>  > Now being clueless....
>
>
> Sorry don't have the cycles to look into your problem, but the
>  standard procedure for hard problems that can be reproduced
>  is to git bisect them down to the change set that introduced the
>  problem originally and then complain to whoever authored that.

The problem is - I don't know about any working SMP kernel which would
survive this test - thought haven't got into a really big history -
tried something like 2.6.22 kernels - no luck - also many kernel seems
to be unbootable in SMP mode on my machine giving many oopses - in
fact just 2.6.24 series starts to be at least reliable in booting in
my Qemu setup without failing during disk mounting or in some other
place...

Will try to find probably some 2.6.18 kernel and will check what happens.

On the other hand - I've tried to replicate my bug on few other
machines with no luck actually - so it's something which might not be
easy to trace :(

Zdenek

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: loop in copy_user_generic_string

[Zdenek Kabelac]

[-- Attachment #1: Type: text/plain, Size: 3509 bytes --]

2008/3/12, Zdenek Kabelac <zdenek.kabelac@gmail.com>:
> 2008/3/12, Andi Kleen <andi@firstfloor.org>:
>
> > > Argh - being stupid here - it looks like these 'working' kernels were
>  >  > not SMP actually.
>  >  > As long as the SMP is used - I still get the busy loop :(
>  >  > Now being clueless....
>  >
>  >
>  > Sorry don't have the cycles to look into your problem, but the
>  >  standard procedure for hard problems that can be reproduced
>  >  is to git bisect them down to the change set that introduced the
>  >  problem originally and then complain to whoever authored that.
>
>
> The problem is - I don't know about any working SMP kernel which would
>  survive this test - thought haven't got into a really big history -
>  tried something like 2.6.22 kernels - no luck - also many kernel seems
>  to be unbootable in SMP mode on my machine giving many oopses - in
>  fact just 2.6.24 series starts to be at least reliable in booting in
>  my Qemu setup without failing during disk mounting or in some other
>  place...
>
>  Will try to find probably some 2.6.18 kernel and will check what happens.
>
>  On the other hand - I've tried to replicate my bug on few other
>  machines with no luck actually - so it's something which might not be
>  easy to trace :(
>

Btw - just for testing purposes - I've taken current fedora rawhide kernel.
Started machine with this kernel and installed it into qemu guest as well.

And this is what I get when running in SMP mode:

BUG: soft lockup - CPU#1 stuck for 61s! [udevd:583]
CPU 1:
Modules linked in: floppy ata_piix ata_generic pata_acpi pcnet32 mii
libata scsi_m
od
Pid: 583, comm: udevd Not tainted 2.6.25-0.105.rc5.fc9 #1
RIP: 0010:[<ffffffff8113b907>]  [<ffffffff8113b907>] clear_page_c+0x7/0x10
RSP: 0000:ffff810015455b20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810015455be8 RCX: 0000000000000200
RDX: 00000000000006a0 RSI: ffff810015455a74 RDI: ffff810015001000
RBP: ffff810000000000 R08: 0000000015562000 R09: ffff810000000000
R10: 0000000000000292 R11: 0000000000000001 R12: 00003ffffffff000
R13: ffff810000009540 R14: ffff810015454000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff810017509320(0063) knlGS:00000000f7f1d720
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff810015001000 CR3: 00000000159e9000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff810833ac>] ? get_page_from_freelist+0x51f/0x6b6
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810f0d4b>] ? show_stat+0x2a/0x4af
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810a621d>] ? __kmalloc+0x3e/0xf0
 [<ffffffff810c555f>] ? seq_read+0x143/0x2a2
 [<ffffffff810c5532>] ? seq_read+0x116/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810e9df5>] ? proc_reg_read+0x8a/0xa7
 [<ffffffff810ab489>] ? vfs_read+0xab/0x154
 [<ffffffff810ab5f6>] ? sys_read+0x47/0x70
 [<ffffffff81023f32>] ? ia32_sysret+0x0/0xa

(Full trace attached)
So I guess I'm kind of lucky that my own kernels actually boot in smp
mode properly.
Guest was started with 384MB - host has 2GB - around 1GB was free when started.
Kernel boots with nosmp flag.

Zdenek

[-- Attachment #2: qemu-debian.tty --]
[-- Type: application/octet-stream, Size: 14586 bytes --]

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.25-0.105.rc5.fc9 (mockbuild@) (gcc version 4.3.0 20080307 (Red Hat 4.3.0-2) (GCC) ) #1 SMP Mon Mar 10 20:59:23 EDT 2008
Command line: root=/dev/sda1 ro console=ttyS0 console=tty0 selinux=off
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 0000000017ff0000 (usable)
 BIOS-e820: 0000000017ff0000 - 0000000018000000 (ACPI data)
 BIOS-e820: 00000000fffbd000 - 0000000100000000 (reserved)
end_pfn_map = 1048576
DMI 2.4 present.
ACPI: RSDP 000FB9F0, 0014 (r0 QEMU  )
ACPI: RSDT 17FF0000, 002C (r1 QEMU   QEMURSDT        1 QEMU        1)
ACPI: FACP 17FF002C, 0074 (r1 QEMU   QEMUFACP        1 QEMU        1)
ACPI: DSDT 17FF0100, 1632 (r1   BXPC   BXDSDT        1 INTL 20061109)
ACPI: FACS 17FF00C0, 0040
ACPI: APIC 17FF1738, 00E0 (r1 QEMU   QEMUAPIC        1 QEMU        1)
No NUMA configuration found
Faking a node at 0000000000000000-0000000017ff0000
Bootmem setup node 0 0000000000000000-0000000017ff0000
  NODE_DATA [000000000000b000 - 0000000000012fff]
  bootmap [0000000000013000 -  0000000000015fff] pages 3
early res: 0 [0-fff] BIOS data page
early res: 1 [6000-7fff] SMP_TRAMPOLINE
early res: 2 [200000-bf5b57] TEXT DATA BSS
early res: 3 [17a7f000-17fdfb5e] RAMDISK
early res: 4 [9fc00-a0bff] EBDA
early res: 5 [8000-afff] PGTABLE
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1048576
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->      159
    0:      256 ->    98288
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e8000
PM: Registered nosave memory: 00000000000e8000 - 0000000000100000
Allocating PCI resources starting at 20000000 (gap: 18000000:e7fbd000)
SMP: Allowing 16 CPUs, 14 hotplug CPUs
PERCPU: Allocating 436896 bytes of per cpu data
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 93330
Policy zone: DMA32
Kernel command line: root=/dev/sda1 ro console=ttyS0 console=tty0 selinux=off
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 16384 bytes)
TSC calibrated against PM_TIMER
Marking TSC unstable due to TSCs unsynchronized
time.c: Detected 2194.538 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
console [ttyS0] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:    8
... MAX_LOCK_DEPTH:          48
... MAX_LOCKDEP_KEYS:        2048
... CLASSHASH_SIZE:           1024
... MAX_LOCKDEP_ENTRIES:     8192
... MAX_LOCKDEP_CHAINS:      16384
... CHAINHASH_SIZE:          8192
 memory used by lock dependency info: 1712 kB
 per task-struct memory footprint: 3456 bytes
Checking aperture...
Memory: 359744k/393152k available (2718k kernel code, 33020k reserved, 1460k data, 740k init)
SLUB: Genslabs=13, HWalign=64, Order=0-1, MinObjects=4, CPUs=16, Nodes=1
Calibrating delay using timer specific routine.. 4410.71 BogoMIPS (lpj=2205359)
Security Framework initialized
SELinux:  Disabled at boot.
Capability LSM initialized
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU 0/0 -> Node 0
checking if image is initramfs... it is
debug: unmapping init memory ffff810017a7f000..ffff810017fe0000
ACPI: Core revision 20070126
ACPI: Checking initramfs for custom DSDT
Using local APIC timer interrupts.
Detected 62.501 MHz APIC timer.
lockdep: fixing up alternatives.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 4389.10 BogoMIPS (lpj=2194550)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
CPU 1/1 -> Node 0
QEMU Virtual CPU version 0.9.1 stepping 03
Brought up 2 CPUs
khelper used greatest stack depth: 6264 bytes left
net_namespace: 1152 bytes
Time: 16:00:57  Date: 03/12/08
NET: Registered protocol family 16
No dock devices found.
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 7 devices
ACPI: ACPI bus type pnp unregistered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
DMAR:parse DMAR table failure.
PCI-GART: No AMD northbridge found.
Time: acpi_pm clocksource has been installed.
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
audit: initializing netlink socket (disabled)
type=2000 audit(1205337655.109:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: PIIX3: Enabling Passive Release
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: ACPI0007:00 is registered as cooling_device0
ACPI: ACPI0007:01 is registered as cooling_device1
khelper used greatest stack depth: 6240 bytes left
Non-volatile memory driver v1.2
Linux agpgart interface v0.103
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
ÿserial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16450
00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16450
isa bounce pool size: 16 pages
brd: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
Clocksource tsc unstable (delta = 4386122119919 ns)
usbcore: registered new interface driver usbhid
drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
registered taskstats version 1
  Magic number: 0:540:34
debug: unmapping init memory ffffffff814ca000..ffffffff81583000
Write protecting the kernel read-only data: 1144k
busybox used greatest stack depth: 5776 bytes left
busybox used greatest stack depth: 5456 bytes left
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
SCSI subsystem initialized
pcnet32.c:v1.34-NAPI 14.Aug.2007 tsbogend@alpha.franken.de
ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:03.0[A] -> Link [LNKC] -> GSI 11 (level, high) -> IRQ 11
pcnet32: PCnet/PCI II 79C970A at 0xc020, 52:54:00:12:34:56 assigned IRQ 11.
eth0: registered as PCnet/PCI II 79C970A
pcnet32: 1 cards_found.
modprobe used greatest stack depth: 4832 bytes left
exe used greatest stack depth: 4752 bytes left
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
ata1.00: ATA-7: QEMU HARDDISK, 0.9.1, max UDMA/100
ata1.00: 4194304 sectors, multi 16: LBA48 
ata1.01: ATA-7: QEMU HARDDISK, 0.9.1, max UDMA/100
ata1.01: 16777216 sectors, multi 16: LBA48 
ata1.00: configured for MWDMA2
ata1.01: configured for MWDMA2
FDC 0 is a S82078B
ata2.00: ATA-7: QEMU HARDDISK, 0.9.1, max UDMA/100
ata2.00: 1433600 sectors, multi 16: LBA48 
ata2.00: configured for MWDMA2
scsi 0:0:0:0: Direct-Access     ATA      QEMU HARDDISK    0.9. PQ: 0 ANSI: 5
scsi 0:0:1:0: Direct-Access     ATA      QEMU HARDDISK    0.9. PQ: 0 ANSI: 5
scsi 1:0:0:0: Direct-Access     ATA      QEMU HARDDISK    0.9. PQ: 0 ANSI: 5
modprobe used greatest stack depth: 3904 bytes left
BUG: soft lockup - CPU#1 stuck for 61s! [udevd:583]
CPU 1:
Modules linked in: floppy ata_piix ata_generic pata_acpi pcnet32 mii libata scsi_mod
Pid: 583, comm: udevd Not tainted 2.6.25-0.105.rc5.fc9 #1
RIP: 0010:[<ffffffff8113b907>]  [<ffffffff8113b907>] clear_page_c+0x7/0x10
RSP: 0000:ffff810015455b20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810015455be8 RCX: 0000000000000200
RDX: 00000000000006a0 RSI: ffff810015455a74 RDI: ffff810015001000
RBP: ffff810000000000 R08: 0000000015562000 R09: ffff810000000000
R10: 0000000000000292 R11: 0000000000000001 R12: 00003ffffffff000
R13: ffff810000009540 R14: ffff810015454000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff810017509320(0063) knlGS:00000000f7f1d720
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff810015001000 CR3: 00000000159e9000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff810833ac>] ? get_page_from_freelist+0x51f/0x6b6
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810f0d4b>] ? show_stat+0x2a/0x4af
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810a621d>] ? __kmalloc+0x3e/0xf0
 [<ffffffff810c555f>] ? seq_read+0x143/0x2a2
 [<ffffffff810c5532>] ? seq_read+0x116/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810e9df5>] ? proc_reg_read+0x8a/0xa7
 [<ffffffff810ab489>] ? vfs_read+0xab/0x154
 [<ffffffff810ab5f6>] ? sys_read+0x47/0x70
 [<ffffffff81023f32>] ? ia32_sysret+0x0/0xa

BUG: soft lockup - CPU#1 stuck for 61s! [udevd:583]
CPU 1:
Modules linked in: floppy ata_piix ata_generic pata_acpi pcnet32 mii libata scsi_mod
Pid: 583, comm: udevd Not tainted 2.6.25-0.105.rc5.fc9 #1
RIP: 0010:[<ffffffff8113b907>]  [<ffffffff8113b907>] clear_page_c+0x7/0x10
RSP: 0000:ffff810015455b20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810015455be8 RCX: 0000000000000200
RDX: 00000000000006a0 RSI: ffff810015455a74 RDI: ffff810015001000
RBP: ffff810000000000 R08: 0000000015562000 R09: ffff810000000000
R10: 0000000000000292 R11: 0000000000000001 R12: 00003ffffffff000
R13: ffff810000009540 R14: ffff810015454000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff810017509320(0063) knlGS:00000000f7f1d720
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffff810015001000 CR3: 00000000159e9000 CR4: 00000000000006a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 [<ffffffff810833ac>] ? get_page_from_freelist+0x51f/0x6b6
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810f0d4b>] ? show_stat+0x2a/0x4af
 [<ffffffff810838ae>] ? __alloc_pages+0xed/0x3c3
 [<ffffffff8109d5d8>] ? alloc_pages_current+0x100/0x109
 [<ffffffff81082e4e>] ? __get_free_pages+0xe/0x4d
 [<ffffffff810a621d>] ? __kmalloc+0x3e/0xf0
 [<ffffffff810c555f>] ? seq_read+0x143/0x2a2
 [<ffffffff810c5532>] ? seq_read+0x116/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810c541c>] ? seq_read+0x0/0x2a2
 [<ffffffff810e9df5>] ? proc_reg_read+0x8a/0xa7
 [<ffffffff810ab489>] ? vfs_read+0xab/0x154
 [<ffffffff810ab5f6>] ? sys_read+0x47/0x70
 [<ffffffff81023f32>] ? ia32_sysret+0x0/0xa


[-- Attachment #3: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #4: Type: text/plain, Size: 158 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel