Re: Xenlinux oops and crashes with nbd, Xen no longer boots

Re: Xenlinux oops and crashes with nbd, Xen no longer boots

[Keir Fraser]

Okay, detangling the backtrace....

 Entry EIP    Function                  Caller's return EIP
 ---------    --------                  -------------------
 0xc00c1220   alloc_skb(1656,GFP_NOIO)  0xc00ddd58
 0xc00dd4b0   tcp_sendmsg               0xc00fa9a1
 0xc00fa960   inet_sendmsg              0xc00bde23
 0xc00bddb0   sock_sendmsg              0xc3cad25f
 unknown      nbd_xmit                  0xc3cad511
 unknown      nbd_send_req              0xc3cad91c
 unknown      do_nbd_request            0xc008fc46
 0xc008fbe0   generic_unplug_device     0xc000c342
 0xc000c2e0   __run_task_queue          n/a (tail call)
 0xc00b8f20   io_schedule               0xc000c0a8
 0xc000c050   tasklet_action            0xc000be9e
 0xc000bdc0   do_softirq                0xc006f2c0
 0xc006f220   do_IRQ(133,-)             0xc007395e
 0xc00738c0   evtchn_do_upcall          0xc006dcd7
 0xc006dca4   hypervisor_callback       n/a (async callback)

I see the problem. Well hidden because the key function (io_schedule)
ends in a tail call. The problem is that we are scheduling I/O
requests in interrupt context rather than deferring to a process
context --- a known issue that hasn't bitten until now. I'll have to
think about a fix.

 -- Keir

> Down not across
> ksymoops 2.4.9 on i686 2.4.26-xen0.  Options used
>      -v ./vmlinux (specified)
>      -k /proc/ksyms (default)
>      -l /proc/modules (default)
>      -o /lib/modules/2.4.26-xen0/ (default)
>      -M (specified)
> 
> invalid operand: 0000
> CPU:    0
> EIP:    0819:[<c00c133b>]    Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00213286
> eax: 0000003a   ebx: c048f040   ecx: c0178000   edx: 00000000
> esi: 00000030   edi: c048f160   ebp: c0179cd0   esp: c0179cc0
> ds: 0821   es: 0821   ss: 0821
> Process swapper (pid: 0, stackpage=c0179000)<1>
> Stack: c012c920 c00ddd58 c048f040 c048f040 c0179d48 c00ddd58 00000678 00000030 
>        00000004 00000008 0000000e c27f9b24 c0179d18 c00cc1cc c05dac00 c05dac00 
>        c05d000e c0179d28 c002097c c048f098 c0179ea8 0000001c 00000000 000005a8 
>        00004000 00000000 c048f160 c0179ddc c37fe4e0 c05dac00 7fffffff c048f040 
>        c0179e5c c0179d78 c0179d60 c00fa9a1 c048f040 c0179e5c 0000001c 00000000 
>        c0179da4 c00bde23 c0d7bf0c c0179e5c 0000001c c0179d78 00000000 00000000 
>        00000000 00000000 00000000 c37fe3d8 c218e000 c218ede8 c0179ea8 c0d7bf0c 
>        0000001c c0179e88 c3cad25f c0d7bf0c c0179e5c 0000001c c1eaa538 c35581dc 
>        c1d13960 00000000 00000000 00000000 ffffffff c0179ea8 0000001c c1eaa538
>        00000000 00000030 c35581dc c0179e10 c00bbe62 c01c5444 c35581dc c250e000
>        00000000 c1d13800 c1eaa538 10000000 c0179e28 c00c550f c1eaa538 c1d13800
>        00000040 c1eaa538 c0179e6c c3c0fa28 c1eaa538 00000010 00000000 00000006
>        c0179e90 0000003a c22997ac 00000002 c0179e6c 00000000 c0179e6c 00000000
>        00000000 c0179dd4 00000001 00000000 00000000 00004000 00000002 c3caf440
>        c0d7bf0c c3caf420 c0179ed4 c3cad511 00000001 c0d7bf0c c0179ea8 0000001c
>        00000000 c0d7bf0c 13956025 00000000 c2478c64 c0109f58 00000000 00040000
>        00040000 c010b1d4 c2478c64 c3caf420 00000000 c0179ef0 c3cad91c c3caf420
>        c2478c64 00000000 c0179f0c fffffff7 c0179f00 c008fc46 c01ab6a4 c0179f0c
>        c0179f1c c000c342 c01ab6a4 c01ab704 c01ab704 00000000 c01881d8 c0179f2c
>        c000c0a8 c0147510 00000001 c0179f48 c000be9e c01881d8 00000001 c018a400
>        c1d4bbec 00002140 c0179f68 c006f2c0 00000085 c0179f9c c1d4bbec 00000000
>        00000000 fbff9000 c0179f90 c007395e 00000085 c0179f9c 00000001 00000000
>        c0179f9c c0178000 c0178000 00000006 c0179fe4 c006dcd7 c0179f9c 00000001
>        39854980 38ecb300 c0178000 00000006 c0179fe4 00000006 00000821 00000821
>        00000006 c006bcc2 00000819 00203246 c0179fe4 c0178000 00000000 c01e2200
>        c01983a0 c0179ffc c017a57c c0116f40 c01986a0 00000000 c01985a0 00000000
> Call Trace: [<c00ddd58>] [<c00ddd58>] [<c00cc1cc>] [<c002097c>] [<c00fa9a1>]
>    [<c00bde23>] [<c3cad25f>] [<c00bbe62>] [<c00c550f>] [<c3c0fa28>] [<c3caf440>]
>    [<c3caf420>] [<c3cad511>] [<c0109f58>] [<c010b1d4>] [<c3caf420>] [<c3cad91c>]
>    [<c3caf420>] [<c008fc46>] [<c000c342>] [<c000c0a8>] [<c000be9e>] [<c006f2c0>]
>    [<c007395e>] [<c006dcd7>] [<c006bcc2>]
>  <0>Kernel panic: Aiee, killing interrupt handler!
> Warning (Oops_read): Code line not seen, dumping what data is available
> 
> 
> >>EIP; c00c133b <alloc_skb+11b/130>   <=====
> 
> >>ebx; c048f040 <_end+2bd130/3a2e170>
> >>ecx; c0178000 <init_task_union+0/2000>
> >>edi; c048f160 <_end+2bd250/3a2e170>
> >>ebp; c0179cd0 <init_task_union+1cd0/2000>
> >>esp; c0179cc0 <init_task_union+1cc0/2000>
> 
> Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
> Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
> Trace; c00cc1cc <rtnetlink_put_metrics+9c/150>
> Trace; c002097c <kmem_cache_free_one+ec/200>
> Trace; c00fa9a1 <inet_sendmsg+41/50>
> Trace; c00bde23 <sock_sendmsg+73/b0>
> Trace; c3cad25f <[nbd]nbd_xmit+11f/2e0>
> Trace; c00bbe62 <netif_be_start_xmit+122/180>
> Trace; c00c550f <dev_queue_xmit+9f/270>
> Trace; c3c0fa28 <[ipv6]ndisc_send_rs+1a8/230>
> Trace; c3caf440 <[nbd].data.end+a59/3699>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c3cad511 <[nbd]nbd_send_req+f1/180>
> Trace; c0109f58 <br_config_bpdu_generation+38/40>
> Trace; c010b1d4 <br_hello_timer_expired+14/30>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c3cad91c <[nbd]do_nbd_request+fc/1d0>
> Trace; c3caf420 <[nbd].data.end+a39/3699>
> Trace; c008fc46 <generic_unplug_device+66/70>
> Trace; c000c342 <__run_task_queue+62/80>
> Trace; c000c0a8 <tasklet_action+58/b0>
> Trace; c000be9e <do_softirq+de/100>
> Trace; c006f2c0 <do_IRQ+a0/b0>
> Trace; c007395e <evtchn_do_upcall+9e/100>
> Trace; c006dcd7 <hypervisor_callback+33/49>
> Trace; c006bcc2 <cpu_idle+62/c0>
> 
> 
> 1 warning issued.  Results may not be reliable.
\x1f -=- MIME -=- \x1f\f


--u3/rZRmxL6MmkK24
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Mon, Jul 12, 2004 at 10:06:55PM +0100, Keir Fraser wrote:
> Even better, recompile your DOM0 kernel with arch/xen/kernel/traps.c
> modified with 'kstack_depth_to_print' changed to a larger value like
> 240. This will give a fuller stack backtrace (it's currently less
> useful because it's truncated). Then provide the backtrace and the
> vmlinux file.

OK I've done that the vmlinux file is at
http://www.yuri.org.uk/~murble/vmlinux-xen

New Oops file should be attached.


cheers

bill
-- 
Bill Boughton		<murble(at)@yuri.org.uk>
 			<bill@(at)xencat.murble.org.uk>
Down not across

--u3/rZRmxL6MmkK24
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=xen0-oops

ksymoops 2.4.9 on i686 2.4.26-xen0.  Options used
     -v ./vmlinux (specified)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.26-xen0/ (default)
     -M (specified)

invalid operand: 0000
CPU:    0
EIP:    0819:[<c00c133b>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00213286
eax: 0000003a   ebx: c048f040   ecx: c0178000   edx: 00000000
esi: 00000030   edi: c048f160   ebp: c0179cd0   esp: c0179cc0
ds: 0821   es: 0821   ss: 0821
Process swapper (pid: 0, stackpage=c0179000)<1>
Stack: c012c920 c00ddd58 c048f040 c048f040 c0179d48 c00ddd58 00000678 00000030 
       00000004 00000008 0000000e c27f9b24 c0179d18 c00cc1cc c05dac00 c05dac00 
       c05d000e c0179d28 c002097c c048f098 c0179ea8 0000001c 00000000 000005a8 
       00004000 00000000 c048f160 c0179ddc c37fe4e0 c05dac00 7fffffff c048f040 
       c0179e5c c0179d78 c0179d60 c00fa9a1 c048f040 c0179e5c 0000001c 00000000 
       c0179da4 c00bde23 c0d7bf0c c0179e5c 0000001c c0179d78 00000000 00000000 
       00000000 00000000 00000000 c37fe3d8 c218e000 c218ede8 c0179ea8 c0d7bf0c 
       0000001c c0179e88 c3cad25f c0d7bf0c c0179e5c 0000001c c1eaa538 c35581dc 
       c1d13960 00000000 00000000 00000000 ffffffff c0179ea8 0000001c c1eaa538
       00000000 00000030 c35581dc c0179e10 c00bbe62 c01c5444 c35581dc c250e000
       00000000 c1d13800 c1eaa538 10000000 c0179e28 c00c550f c1eaa538 c1d13800
       00000040 c1eaa538 c0179e6c c3c0fa28 c1eaa538 00000010 00000000 00000006
       c0179e90 0000003a c22997ac 00000002 c0179e6c 00000000 c0179e6c 00000000
       00000000 c0179dd4 00000001 00000000 00000000 00004000 00000002 c3caf440
       c0d7bf0c c3caf420 c0179ed4 c3cad511 00000001 c0d7bf0c c0179ea8 0000001c
       00000000 c0d7bf0c 13956025 00000000 c2478c64 c0109f58 00000000 00040000
       00040000 c010b1d4 c2478c64 c3caf420 00000000 c0179ef0 c3cad91c c3caf420
       c2478c64 00000000 c0179f0c fffffff7 c0179f00 c008fc46 c01ab6a4 c0179f0c
       c0179f1c c000c342 c01ab6a4 c01ab704 c01ab704 00000000 c01881d8 c0179f2c
       c000c0a8 c0147510 00000001 c0179f48 c000be9e c01881d8 00000001 c018a400
       c1d4bbec 00002140 c0179f68 c006f2c0 00000085 c0179f9c c1d4bbec 00000000
       00000000 fbff9000 c0179f90 c007395e 00000085 c0179f9c 00000001 00000000
       c0179f9c c0178000 c0178000 00000006 c0179fe4 c006dcd7 c0179f9c 00000001
       39854980 38ecb300 c0178000 00000006 c0179fe4 00000006 00000821 00000821
       00000006 c006bcc2 00000819 00203246 c0179fe4 c0178000 00000000 c01e2200
       c01983a0 c0179ffc c017a57c c0116f40 c01986a0 00000000 c01985a0 00000000
Call Trace: [<c00ddd58>] [<c00ddd58>] [<c00cc1cc>] [<c002097c>] [<c00fa9a1>]
   [<c00bde23>] [<c3cad25f>] [<c00bbe62>] [<c00c550f>] [<c3c0fa28>] [<c3caf440>]
   [<c3caf420>] [<c3cad511>] [<c0109f58>] [<c010b1d4>] [<c3caf420>] [<c3cad91c>]
   [<c3caf420>] [<c008fc46>] [<c000c342>] [<c000c0a8>] [<c000be9e>] [<c006f2c0>]
   [<c007395e>] [<c006dcd7>] [<c006bcc2>]
 <0>Kernel panic: Aiee, killing interrupt handler!
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; c00c133b <alloc_skb+11b/130>   <=====

>>ebx; c048f040 <_end+2bd130/3a2e170>
>>ecx; c0178000 <init_task_union+0/2000>
>>edi; c048f160 <_end+2bd250/3a2e170>
>>ebp; c0179cd0 <init_task_union+1cd0/2000>
>>esp; c0179cc0 <init_task_union+1cc0/2000>

Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
Trace; c00ddd58 <tcp_sendmsg+8a8/f40>
Trace; c00cc1cc <rtnetlink_put_metrics+9c/150>
Trace; c002097c <kmem_cache_free_one+ec/200>
Trace; c00fa9a1 <inet_sendmsg+41/50>
Trace; c00bde23 <sock_sendmsg+73/b0>
Trace; c3cad25f <[nbd]nbd_xmit+11f/2e0>
Trace; c00bbe62 <netif_be_start_xmit+122/180>
Trace; c00c550f <dev_queue_xmit+9f/270>
Trace; c3c0fa28 <[ipv6]ndisc_send_rs+1a8/230>
Trace; c3caf440 <[nbd].data.end+a59/3699>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c3cad511 <[nbd]nbd_send_req+f1/180>
Trace; c0109f58 <br_config_bpdu_generation+38/40>
Trace; c010b1d4 <br_hello_timer_expired+14/30>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c3cad91c <[nbd]do_nbd_request+fc/1d0>
Trace; c3caf420 <[nbd].data.end+a39/3699>
Trace; c008fc46 <generic_unplug_device+66/70>
Trace; c000c342 <__run_task_queue+62/80>
Trace; c000c0a8 <tasklet_action+58/b0>
Trace; c000be9e <do_softirq+de/100>
Trace; c006f2c0 <do_IRQ+a0/b0>
Trace; c007395e <evtchn_do_upcall+9e/100>
Trace; c006dcd7 <hypervisor_callback+33/49>
Trace; c006bcc2 <cpu_idle+62/c0>


1 warning issued.  Results may not be reliable.

--u3/rZRmxL6MmkK24--



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Xenlinux oops and crashes with nbd, Xen no longer boots

[Keir Fraser]

Now fixed. You've been able to hit this more reliably than anyone
else, so I'll be interested to hear confirmation that the problem
really has now gone away.

 -- Keir

> 
> Okay, detangling the backtrace....
> 
>  Entry EIP    Function                  Caller's return EIP
>  ---------    --------                  -------------------
>  0xc00c1220   alloc_skb(1656,GFP_NOIO)  0xc00ddd58
>  0xc00dd4b0   tcp_sendmsg               0xc00fa9a1
>  0xc00fa960   inet_sendmsg              0xc00bde23
>  0xc00bddb0   sock_sendmsg              0xc3cad25f
>  unknown      nbd_xmit                  0xc3cad511
>  unknown      nbd_send_req              0xc3cad91c
>  unknown      do_nbd_request            0xc008fc46
>  0xc008fbe0   generic_unplug_device     0xc000c342
>  0xc000c2e0   __run_task_queue          n/a (tail call)
>  0xc00b8f20   io_schedule               0xc000c0a8
>  0xc000c050   tasklet_action            0xc000be9e
>  0xc000bdc0   do_softirq                0xc006f2c0
>  0xc006f220   do_IRQ(133,-)             0xc007395e
>  0xc00738c0   evtchn_do_upcall          0xc006dcd7
>  0xc006dca4   hypervisor_callback       n/a (async callback)
> 
> I see the problem. Well hidden because the key function (io_schedule)
> ends in a tail call. The problem is that we are scheduling I/O
> requests in interrupt context rather than deferring to a process
> context --- a known issue that hasn't bitten until now. I'll have to
> think about a fix.
> 
>  -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Xenlinux oops and crashes with nbd, Xen no longer boots

[Wm Boughton]

On Tue, Jul 13, 2004 at 04:29:00PM +0100, Keir Fraser wrote:
> 
> Now fixed. You've been able to hit this more reliably than anyone
> else, so I'll be interested to hear confirmation that the problem
> really has now gone away.

I've not been abel to hit it and am now happily booting
XenLinux instances over nbd.


thanks

bill


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Xenlinux oops and crashes with nbd, Xen no longer boots

[Steve Traugott]

Hi Bill,

On Tue, Jul 13, 2004 at 05:36:38PM +0100, Wm Boughton wrote:
> I've not been abel to hit it and am now happily booting
> XenLinux instances over nbd.

Which NBD are you using?  The kernel NBD, ENBD, or GNBD?

Steve
-- 
Stephen G. Traugott  (KG6HDQ)
UNIX/Linux Infrastructure Architect, TerraLuna LLC
stevegt@TerraLuna.Org 
http://www.stevegt.com -- http://Infrastructures.Org 


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click

how to debug xen?

[Yan Li]

hi there,

How to debug xen? just use the nsplitd as XenDebugger-HOWTO said? It seems
quite tricky. I believe lots of people are doing development on xen, so
usually how to debug it?

Thanks
Yan


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click

Re: how to debug xen?

[David Becker]

" How to debug xen? just use the nsplitd as XenDebugger-HOWTO said? It seems
" quite tricky. I believe lots of people are doing development on xen, so
" usually how to debug it?

This is how I debug xen.  The starting point is reading
    xeno-unstable.bk/docs/HOWTOs/XenDebugger-HOWTO
I do not use nsplitd.

First, build debug kernels.
For xen, in the xeno-unstable.bk/xen directory add this to arch/x86/Rules.mk
        CFLAGS += -g
    and build xen saying
        make debugger=y
For xenolinux,  in xeno-unstable.bk/linux-2.4.26-xen0
    turn on CONFIG_FRAME_POINTER (in the Kernel Hacking section of
        menuconfig or xconfig)
    in Makefile put -g into CFLAGS_KERNEL (line 52)
        CFLAGS_KERNEL = -g
    and build xenolinux with
        make ARCH=xen bzImage


Second, find a target machine with some serial lines.
I prefer virtual machines and use VMware Workstation 4.5 on Linux (4.0 didn't
work, btw).  In the vmware Machine Settings, config both com1 and com2
to be named pipes with "This end is the client" and "The other end is a
virtual machine".   Run two socat jobs in the background
to convert the pipes to ptys:
    socat UNIX-LISTEN:/tmp/com1pipe,unlink-early,fork pty,link=/tmp/com1 &
    socat UNIX-LISTEN:/tmp/com2pipe,unlink-early,fork pty,link=/tmp/com2 &


Third, boot the target machine and connect the debugger.
Put the debug xen and xenolinux on the machine.  In the grub menu.lst
file, add the serial lines and pdb to the xen command line:
   kernel /boot/xen.gz.debug dom0_mem=100000 ifname=eth0 com1=9600,8n1 com2=9600,8n1 pdb=com2
Boot it up and connect to the console.  In my vmware setup, I connect
to com1 from the machine hosting VMware with
    kermit -c -l /tmp/com1
Then on com1 type ^A^A^A to make xen listen to console cmds.
At this point type 'h' and see if the 'D' command is listed.  If not,
pdb is missing so make sure the xen being run was built with debugger=y.
Hit 'D' on the console to make xen drop into the debugger.

>From the machine listening to the serial lines, connect to the target machine
with gdb and the xen-syms file and check if was built with -g by trying to
list a function:
    $ gdb xeno-unstable.bk/xen/xen-syms
    (gdb) list do_dom0_op 
    26      extern unsigned int alloc_new_dom_mem(struct domain *,
    unsigned int);
    27      extern long arch_do_dom0_op(dom0_op_t *op, dom0_op_t
    *u_dom0_op);
    28      extern void arch_getdomaininfo_ctxt(struct domain *,
    full_execution_context_t *);
    29      
    30      long do_dom0_op(dom0_op_t *u_dom0_op)
    31      {
    32          long ret = 0;
    33          dom0_op_t curop, *op = &curop;
    34      
    35          if ( !IS_PRIV(current) )
    (gdb) target remote /tmp/com2
Now you can debug xen itself.  Note that the breakpoint is in xenolinux
so gdb will be cryptic about the current function and stacktrace.  Put a
breakpoint in xen somewhere and continue to get gdb to a function is can
decode.

To debug xenolinux, drop to the debugger:
   (gdb) set pdb_ctx.domain=0
   (gdb) set pdb_ctx.valid=1
   (gdb) add-symbol-file xeno-unstable.bk/linux-2.4.26-xen0/vmlinux
Now you can debug vmlinux itself (except where the xen and linux
symbols clash).



-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click

Re: how to debug xen?

[Kip Macy]

One non-obvious thing that I didn't see in the docs is that if you use
multiple nsplitds they have to be spaced 4 ports apart not 2.

			-Kip



On Fri, 23 Jul 2004, David Becker wrote:

>
> " How to debug xen? just use the nsplitd as XenDebugger-HOWTO said? It seems
> " quite tricky. I believe lots of people are doing development on xen, so
> " usually how to debug it?
>
> This is how I debug xen.  The starting point is reading
>     xeno-unstable.bk/docs/HOWTOs/XenDebugger-HOWTO
> I do not use nsplitd.
>
> First, build debug kernels.
> For xen, in the xeno-unstable.bk/xen directory add this to arch/x86/Rules.mk
>         CFLAGS += -g
>     and build xen saying
>         make debugger=y
> For xenolinux,  in xeno-unstable.bk/linux-2.4.26-xen0
>     turn on CONFIG_FRAME_POINTER (in the Kernel Hacking section of
>         menuconfig or xconfig)
>     in Makefile put -g into CFLAGS_KERNEL (line 52)
>         CFLAGS_KERNEL = -g
>     and build xenolinux with
>         make ARCH=xen bzImage
>
>
> Second, find a target machine with some serial lines.
> I prefer virtual machines and use VMware Workstation 4.5 on Linux (4.0 didn't
> work, btw).  In the vmware Machine Settings, config both com1 and com2
> to be named pipes with "This end is the client" and "The other end is a
> virtual machine".   Run two socat jobs in the background
> to convert the pipes to ptys:
>     socat UNIX-LISTEN:/tmp/com1pipe,unlink-early,fork pty,link=/tmp/com1 &
>     socat UNIX-LISTEN:/tmp/com2pipe,unlink-early,fork pty,link=/tmp/com2 &
>
>
> Third, boot the target machine and connect the debugger.
> Put the debug xen and xenolinux on the machine.  In the grub menu.lst
> file, add the serial lines and pdb to the xen command line:
>    kernel /boot/xen.gz.debug dom0_mem=100000 ifname=eth0 com1=9600,8n1 com2=9600,8n1 pdb=com2
> Boot it up and connect to the console.  In my vmware setup, I connect
> to com1 from the machine hosting VMware with
>     kermit -c -l /tmp/com1
> Then on com1 type ^A^A^A to make xen listen to console cmds.
> At this point type 'h' and see if the 'D' command is listed.  If not,
> pdb is missing so make sure the xen being run was built with debugger=y.
> Hit 'D' on the console to make xen drop into the debugger.
>
> >From the machine listening to the serial lines, connect to the target machine
> with gdb and the xen-syms file and check if was built with -g by trying to
> list a function:
>     $ gdb xeno-unstable.bk/xen/xen-syms
>     (gdb) list do_dom0_op
>     26      extern unsigned int alloc_new_dom_mem(struct domain *,
>     unsigned int);
>     27      extern long arch_do_dom0_op(dom0_op_t *op, dom0_op_t
>     *u_dom0_op);
>     28      extern void arch_getdomaininfo_ctxt(struct domain *,
>     full_execution_context_t *);
>     29
>     30      long do_dom0_op(dom0_op_t *u_dom0_op)
>     31      {
>     32          long ret = 0;
>     33          dom0_op_t curop, *op = &curop;
>     34
>     35          if ( !IS_PRIV(current) )
>     (gdb) target remote /tmp/com2
> Now you can debug xen itself.  Note that the breakpoint is in xenolinux
> so gdb will be cryptic about the current function and stacktrace.  Put a
> breakpoint in xen somewhere and continue to get gdb to a function is can
> decode.
>
> To debug xenolinux, drop to the debugger:
>    (gdb) set pdb_ctx.domain=0
>    (gdb) set pdb_ctx.valid=1
>    (gdb) add-symbol-file xeno-unstable.bk/linux-2.4.26-xen0/vmlinux
> Now you can debug vmlinux itself (except where the xen and linux
> symbols clash).
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by BEA Weblogic Workshop
> FREE Java Enterprise J2EE developer tools!
> Get your free copy of BEA WebLogic Workshop 8.1 today.
> http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click

Re: how to debug xen?

[Yan Li]

hi

Thanks,but now I'm stuck. I wanna try nsplitd first. so I do it according to
the Debugger-howto. I built xen and xeno with -g added and
fomit-frame-pointer
disabled.

The problem is I'm really confused by the configuration of nsplitd.

1.it said set 'pdb=xxx'. How can I tell which port I'm connecting to?

2.in nsplitd configuration.
  what are the /etc/xinetd.d/nsplit and all the 'wcons00', 'wanda remote
console,
  stuff? I'm asuming it's in configuration in their working machine. Then
can
  anyone tell me what does this mean?
  The whole nsplitd configuration section,please let me know the details
about it.
  I totally have no idea what's going on there.

  For example, I have two machines, A.x.x.x, B.x.x.x, I'm running xen on
machine A,
  using machine B to connect to the serial line. What should all the
settings look
  like?

btw, David, is the socat just this one: http://www.socat.org/?

Thanks a lot!

Q
----- Original Message ----- 
From: "Kip Macy" <kmacy@eventdriven.org>
To: "David Becker" <becker@cs.duke.edu>
Cc: "Yan Li" <yan_li00@hotmail.com>; <xen-devel@lists.sourceforge.net>
Sent: Friday, July 23, 2004 8:08 AM
Subject: Re: [Xen-devel] how to debug xen?


> One non-obvious thing that I didn't see in the docs is that if you use
> multiple nsplitds they have to be spaced 4 ports apart not 2.
>
> -Kip
>
>
>
> On Fri, 23 Jul 2004, David Becker wrote:
>
> >
> > " How to debug xen? just use the nsplitd as XenDebugger-HOWTO said? It
seems
> > " quite tricky. I believe lots of people are doing development on xen,
so
> > " usually how to debug it?
> >
> > This is how I debug xen.  The starting point is reading
> >     xeno-unstable.bk/docs/HOWTOs/XenDebugger-HOWTO
> > I do not use nsplitd.
> >
> > First, build debug kernels.
> > For xen, in the xeno-unstable.bk/xen directory add this to
arch/x86/Rules.mk
> >         CFLAGS += -g
> >     and build xen saying
> >         make debugger=y
> > For xenolinux,  in xeno-unstable.bk/linux-2.4.26-xen0
> >     turn on CONFIG_FRAME_POINTER (in the Kernel Hacking section of
> >         menuconfig or xconfig)
> >     in Makefile put -g into CFLAGS_KERNEL (line 52)
> >         CFLAGS_KERNEL = -g
> >     and build xenolinux with
> >         make ARCH=xen bzImage
> >
> >
> > Second, find a target machine with some serial lines.
> > I prefer virtual machines and use VMware Workstation 4.5 on Linux (4.0
didn't
> > work, btw).  In the vmware Machine Settings, config both com1 and com2
> > to be named pipes with "This end is the client" and "The other end is a
> > virtual machine".   Run two socat jobs in the background
> > to convert the pipes to ptys:
> >     socat UNIX-LISTEN:/tmp/com1pipe,unlink-early,fork pty,link=/tmp/com1
&
> >     socat UNIX-LISTEN:/tmp/com2pipe,unlink-early,fork pty,link=/tmp/com2
&
> >
> >
> > Third, boot the target machine and connect the debugger.
> > Put the debug xen and xenolinux on the machine.  In the grub menu.lst
> > file, add the serial lines and pdb to the xen command line:
> >    kernel /boot/xen.gz.debug dom0_mem=100000 ifname=eth0 com1=9600,8n1
com2=9600,8n1 pdb=com2
> > Boot it up and connect to the console.  In my vmware setup, I connect
> > to com1 from the machine hosting VMware with
> >     kermit -c -l /tmp/com1
> > Then on com1 type ^A^A^A to make xen listen to console cmds.
> > At this point type 'h' and see if the 'D' command is listed.  If not,
> > pdb is missing so make sure the xen being run was built with debugger=y.
> > Hit 'D' on the console to make xen drop into the debugger.
> >
> > >From the machine listening to the serial lines, connect to the target
machine
> > with gdb and the xen-syms file and check if was built with -g by trying
to
> > list a function:
> >     $ gdb xeno-unstable.bk/xen/xen-syms
> >     (gdb) list do_dom0_op
> >     26      extern unsigned int alloc_new_dom_mem(struct domain *,
> >     unsigned int);
> >     27      extern long arch_do_dom0_op(dom0_op_t *op, dom0_op_t
> >     *u_dom0_op);
> >     28      extern void arch_getdomaininfo_ctxt(struct domain *,
> >     full_execution_context_t *);
> >     29
> >     30      long do_dom0_op(dom0_op_t *u_dom0_op)
> >     31      {
> >     32          long ret = 0;
> >     33          dom0_op_t curop, *op = &curop;
> >     34
> >     35          if ( !IS_PRIV(current) )
> >     (gdb) target remote /tmp/com2
> > Now you can debug xen itself.  Note that the breakpoint is in xenolinux
> > so gdb will be cryptic about the current function and stacktrace.  Put a
> > breakpoint in xen somewhere and continue to get gdb to a function is can
> > decode.
> >
> > To debug xenolinux, drop to the debugger:
> >    (gdb) set pdb_ctx.domain=0
> >    (gdb) set pdb_ctx.valid=1
> >    (gdb) add-symbol-file xeno-unstable.bk/linux-2.4.26-xen0/vmlinux
> > Now you can debug vmlinux itself (except where the xen and linux
> > symbols clash).
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by BEA Weblogic Workshop
> > FREE Java Enterprise J2EE developer tools!
> > Get your free copy of BEA WebLogic Workshop 8.1 today.
> > http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
>


-------------------------------------------------------
This SF.Net email is sponsored by OSTG. Have you noticed the changes on
Linux.com, ITManagersJournal and NewsForge in the past few weeks? Now,
one more big change to announce. We are now OSTG- Open Source Technology
Group. Come see the changes on the new OSTG site. www.ostg.com

Xenlinux oops and crashes with nbd, Xen no longer boots

[murble-xendev]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]


Hi,


When I start a new domain using an nbd exported device into domain0,
xenlinux Oops and crashes.  Nbd otherwise works fine in domain0 I
can mount filesystems on it, dd them all, export them from domain0
to a new domain via nfs from domain0 also works.

This is easily reproducable, with somthing like

disk = [ 'phy:/dev/xen-uml0,sda1,w' ]

Where xen-uml0 is the nbd exported device.

With Changeset 1.1063  Xen crashes at start up only displaying the
figlet banner + (Xen) Unknown Interrupt.

Hardware is 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 3
model name      : AMD Duron(tm) Processor
stepping        : 1
cpu MHz         : 697.541
cache size      : 64 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips        : 6973.03


-- 
bill boughton

[-- Attachment #2: nbd-crash-ksymoops --]
[-- Type: text/plain, Size: 2454 bytes --]

ksymoops 2.4.9 on i686 2.4.26-xen0.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.26-xen0/ (default)
     -M (specified)

invalid operand: 0000
CPU:    0
EIP:    0819:[<c00b8abb>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00213286
eax: 0000003a   ebx: c46e4040   ecx: c016a000   edx: 00000000
esi: 00000030   edi: c46e4160   ebp: c016bcd0   esp: c016bcc0
ds: 0821   es: 0821   ss: 0821
Process swapper (pid: 0, stackpage=c016b000)<1>
Stack: c011eba0 c00d5258 c46e4040 c46e4040 c016bd48 c00d5258 00000678 00000030
       00000004 00000008 0000000e c5d0aca0 c016bd18 c00c394c c349c800 00000008
       0000000e c016bd40 0000000c c46e4098 c016bea8 0000001c 00000000 000005a8
Call Trace: [<c00d5258>] [<c00d5258>] [<c00c394c>] [<c001fc7e>] [<c00f1ea1>]
   [<c00b55a3>] [<c7c800e0>] [<c009209f>] [<c00b3622>] [<c00bcc8f>] [<c7c82a28>]
   [<c0092351>] [<c7ccd118>] [<c7cce394>] [<c009275c>] [<c008f546>] [<c000c342>]
   [<c000c0a8>] [<c000be9e>] [<c006ebc0>] [<c007325e>] [<c006d5d7>] [<c006b5c2>]
 <0>Kernel panic: Aiee, killing interrupt handler!
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; c00b8abb <alloc_skb+11b/250>   <=====

>>ebx; c46e4040 <_end+4525cc0/7a61d00>
>>ecx; c016a000 <init_task_union+0/a430>
>>edi; c46e4160 <_end+4525de0/7a61d00>
>>ebp; c016bcd0 <init_task_union+1cd0/a430>
>>esp; c016bcc0 <init_task_union+1cc0/a430>

Trace; c00d5258 <tcp_sendmsg+8a8/1340>
Trace; c00d5258 <tcp_sendmsg+8a8/1340>
Trace; c00c394c <rtnetlink_put_metrics+9c/580>
Trace; c001fc7e <kfree+3e/70>
Trace; c00f1ea1 <inet_sendmsg+41/50>
Trace; c00b55a3 <sock_sendmsg+73/b0>
Trace; c7c800e0 <[ipv6]fib6_clean_tree+30/40>
Trace; c009209f <get_gendisk+94f/16b0>
Trace; c00b3622 <pci_assign_resource+6b62/85c0>
Trace; c00bcc8f <dev_queue_xmit+9f/2e0>
Trace; c7c82a28 <[ipv6]ndisc_send_rs+1a8/230>
Trace; c0092351 <get_gendisk+c01/16b0>
Trace; c7ccd118 <.data.end+4b99/????>
Trace; c7cce394 <.data.end+5e15/????>
Trace; c009275c <get_gendisk+100c/16b0>
Trace; c008f546 <generic_unplug_device+66/70>
Trace; c000c342 <__run_task_queue+62/140>
Trace; c000c0a8 <__tasklet_hi_schedule+d8/1e0>
Trace; c000be9e <do_softirq+de/100>
Trace; c006ebc0 <enable_irq+160/170>
Trace; c007325e <evtchn_do_upcall+9e/1310>
Trace; c006d5d7 <__up_wakeup+13e7/27b0>
Trace; c006b5c2 <enable_hlt+72/d0>


1 warning issued.  Results may not be reliable.

[-- Attachment #3: lspci-vv.log --]
[-- Type: text/plain, Size: 4921 bytes --]

0000:00:00.0 Host bridge: Silicon Integrated Systems [SiS] 730 Host (rev 02)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
	Latency: 32
	Region 0: Memory at d0000000 (32-bit, non-prefetchable) [size=64M]
	Expansion ROM at <unassigned> [disabled] [size=2K]
	Capabilities: <available only to root>

0000:00:00.1 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE] (rev d0) (prog-if 80 [Master])
	Subsystem: Silicon Integrated Systems [SiS] SiS5513 EIDE Controller (A,B step)
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 16
	Region 4: I/O ports at ff00 [size=16]
	Expansion ROM at <unassigned> [disabled] [size=2K]

0000:00:01.0 ISA bridge: Silicon Integrated Systems [SiS] SiS85C503/5513 (LPC Bridge)
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Expansion ROM at <unassigned> [disabled] [size=2K]

0000:00:01.1 Ethernet controller: Silicon Integrated Systems [SiS] SiS900 PCI Fast Ethernet (rev 82)
	Subsystem: Silicon Integrated Systems [SiS] SiS900 10/100 Ethernet Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64
	Interrupt: pin ? routed to IRQ 5
	Region 0: I/O ports at cc00 [size=256]
	Region 1: Memory at cfff7000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at cffc0000 [disabled] [size=128K]
	Capabilities: <available only to root>

0000:00:01.2 USB Controller: Silicon Integrated Systems [SiS] USB 1.0 Controller (rev 07) (prog-if 10 [OHCI])
	Subsystem: Silicon Integrated Systems [SiS] USB 1.0 Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64, Cache Line Size: 0x08 (32 bytes)
	Interrupt: pin ? routed to IRQ 11
	Region 0: Memory at cfffc000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at <unassigned> [disabled] [size=2K]

0000:00:01.3 USB Controller: Silicon Integrated Systems [SiS] USB 1.0 Controller (rev 07) (prog-if 10 [OHCI])
	Subsystem: Silicon Integrated Systems [SiS] Onboard USB Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64, Cache Line Size: 0x08 (32 bytes)
	Interrupt: pin ? routed to IRQ 11
	Region 0: Memory at cfffd000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at <unassigned> [disabled] [size=2K]

0000:00:01.4 Multimedia audio controller: Silicon Integrated Systems [SiS] SiS PCI Audio Accelerator (rev 02)
	Subsystem: Silicon Integrated Systems [SiS] SiS PCI Audio Accelerator
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64
	Interrupt: pin ? routed to IRQ 11
	Region 0: I/O ports at d000 [size=256]
	Region 1: Memory at cfffe000 (32-bit, non-prefetchable) [size=4K]
	Expansion ROM at <unassigned> [disabled] [size=2K]
	Capabilities: <available only to root>

0000:00:01.6 Modem: Silicon Integrated Systems [SiS] AC'97 Modem Controller (rev a0) (prog-if 00 [Generic])
	Subsystem: Silicon Integrated Systems [SiS] AC'97 Modem Controller
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64
	Interrupt: pin ? routed to IRQ 5
	Region 0: I/O ports at d800 [size=256]
	Region 1: I/O ports at d400 [size=128]
	Expansion ROM at <unassigned> [disabled] [size=2K]
	Capabilities: <available only to root>

0000:01:00.0 VGA compatible controller: Silicon Integrated Systems [SiS] 630/730 PCI/AGP VGA Display Adapter (rev 31) (prog-if 00 [VGA])
	Subsystem: Silicon Integrated Systems [SiS] 630/730 PCI/AGP VGA Display Adapter
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	BIST result: 00
	Region 0: Memory at c0000000 (32-bit, prefetchable) [size=128M]
	Region 1: Memory at cfee0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at ac00 [size=128]
	Expansion ROM at <unassigned> [disabled] [size=2K]
	Capabilities: <available only to root>

Re: Xenlinux oops and crashes with nbd, Xen no longer boots

[Ian Pratt]

> When I start a new domain using an nbd exported device into domain0,
> xenlinux Oops and crashes.  Nbd otherwise works fine in domain0 I
> can mount filesystems on it, dd them all, export them from domain0
> to a new domain via nfs from domain0 also works.

You mean the domain0 oops'es, right?

There was a fix checked in earlier today that might be relevant
to this. Please can you try it out.

> With Changeset 1.1063  Xen crashes at start up only displaying the
> figlet banner + (Xen) Unknown Interrupt.

This bug was introduced as part on an x86_64 checkin, and was due
to a gcc version incompatibility, and is believed fixed. 

Ian



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Xenlinux oops and crashes with nbd, Xen no longer boots

[Keir Fraser]

> 
> Hi,
> 
> 
> When I start a new domain using an nbd exported device into domain0,
> xenlinux Oops and crashes.  Nbd otherwise works fine in domain0 I
> can mount filesystems on it, dd them all, export them from domain0
> to a new domain via nfs from domain0 also works.
> 
> This is easily reproducable, with somthing like
> 
> disk = [ 'phy:/dev/xen-uml0,sda1,w' ]
> 
> Where xen-uml0 is the nbd exported device.

>From what I can figure from the backtrace, it looks like the problem
is an assertion failure due to a non-atomic memory allocation request
while in interrupt context.

Unfortunately the backtrace is a real mess. I might have better luck
deciphering it if you can provide a link to your 'vmlinux' file.

Even better, recompile your DOM0 kernel with arch/xen/kernel/traps.c
modified with 'kstack_depth_to_print' changed to a larger value like
240. This will give a fuller stack backtrace (it's currently less
useful because it's truncated). Then provide the backtrace and the
vmlinux file.

> With Changeset 1.1063  Xen crashes at start up only displaying the
> figlet banner + (Xen) Unknown Interrupt.

I fixed this earlier today.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com