Another OOPS while creating domains

Another OOPS while creating domains

[Niraj Tolia]

After a bk pull today morning, I am getting reproducible OOPes when
trying to create a new domain. Happens when the new domain kernel is
trying to DHCP.

There are the kernel messages before the OOPS

Sending DHCP requests ...... timed out!
IP-Config: Reopening network devices...
Sending DHCP requests .

(there might have been a few "?" signs after the "." following the
"Sending DHCP requests message that got eaten up by the oops)

The oops passed through ksymoops is as follows. My kernel binaries can
be found at <http://www.cs.cmu.edu/~ntolia/xen>. Let me know if you
need anything else from me.

Niraj

ksymoops 2.4.9 on i686 2.4.22-1.2194.nptl.  Options used
     -v vmlinux-syms-2.4.26-xen0 (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m ./System.map (specified)
 
Unable to handle kernel NULL pointer dereference at virtual address 00000076
c0167996
Oops: 0000
CPU:    0
EIP:    0819:[<c0167996>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246
eax: 00000000   ebx: c164b160   ecx: 00000002   edx: 00000001
esi: 00000001   edi: 00000006   ebp: c164b000   esp: c159ded4
ds: 0821   es: 0821   ss: 0821
Process swapper (pid: 1, stackpage=c159d000)<1>
Stack: dfdfbc80 000016e1 c1642054 00000006 c164b160 c0167cce c164b000 00000000
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
       00000000 00000000 c164b000 dfdfbc80 c164b07c c017b9ef dfdfbc80 c164b000
Call Trace: [<c0167cce>] [<c017b9ef>] [<c017b4b0>] [<c0172a5f>] [<c0003880>]
   [<c00aab9e>] [<c0003870>]
 <0>Kernel panic: Aiee, killing interrupt handler!
Warning (Oops_read): Code line not seen, dumping what data is available
 
 
>>EIP; c0167996 <network_tx_buf_gc+66/100>   <=====
 
Trace; c0167cce <network_start_xmit+fe/270>
Trace; c017b9ef <qdisc_restart+4f/d0>
Trace; c017b4b0 <eth_header+0/110>
Trace; c0172a5f <dev_queue_xmit+1af/280>
Trace; c0003880 <init+10/100>
Trace; c00aab9e <arch_kernel_thread+2e/40>
Trace; c0003870 <init+0/100>
 
 
1 warning issued.  Results may not be reliable.


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Another OOPS while creating domains

[Ian Pratt]

> After a bk pull today morning, I am getting reproducible OOPes when
> trying to create a new domain. Happens when the new domain kernel is
> trying to DHCP.
> 
> Sending DHCP requests ...... timed out!
> IP-Config: Reopening network devices...
> Sending DHCP requests .

DHCP seems to work fine if you have the DHCP server set up
properly.

What's happening here is that the kernel isn't getting replies,
and in desperation decides to blip the interface down and then up
and try again.

The rapid interface up/down trips a race condition, in the
frontend driver, which causes it to explode, but you were going
to die anyway as you weren't getting a DHCP reply. We just
sped up your reboot cycle ;-)

Shouldn't be too hard to fix (shutdown just needs to wait for
outstanding buffers to be returned.)

Ian


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Another OOPS while creating domains

[Keir Fraser]

> DHCP seems to work fine if you have the DHCP server set up
> properly.
> 
> What's happening here is that the kernel isn't getting replies,
> and in desperation decides to blip the interface down and then up
> and try again.
> 
> The rapid interface up/down trips a race condition, in the
> frontend driver, which causes it to explode, but you were going
> to die anyway as you weren't getting a DHCP reply. We just
> sped up your reboot cycle ;-)
> 
> Shouldn't be too hard to fix (shutdown just needs to wait for
> outstanding buffers to be returned.)

Turned out teh problem was somewhat messier than it first appeared,
but I've checked in a cleanup of the network driver so that closing
and reopening it will now not crash the virtual machine. :-)

That means the DHCP problem should be gone.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: Another OOPS while creating domains

[Niraj Tolia]

> Turned out teh problem was somewhat messier than it first appeared,
> but I've checked in a cleanup of the network driver so that closing
> and reopening it will now not crash the virtual machine. :-)
> 

Keir, Ian, and everyone else... Thanks for all your hard work. I
really do appreciate it :).

Niraj


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com