Re: Kernel crash in 2.6.0-test9-mm3

Re: Kernel crash in 2.6.0-test9-mm3

[Andrew Morton]

It's one for the networking guys.

The mm kernels have a patch which detects when atomic_dec_and_test
takes an atomic_t negative - it is assumed that this is a bug so
a warning is generated.


Reuben Farrelly <reuben-linux@reub.net> wrote:
>
> Hi Andrew,
> 
> Have started to see this problem occur in -test9 and I think, -test8 (but 
> not before then).  At the moment I'm running -test9-mm3 which has the same 
> problem.  Kernel is not tainted.
> 
> Fortunately when this box died a few mins ago (twice in a row) it was able 
> to dump a heap of stuff to syslog, so I got something useful out of the 
> crashes.  This first one triggered when I started to load the ethernet card 
> up with a "debug ip packet detail" from a Cisco router ;-)
> 
> Reuben
> 
> Crash 1:
> 
> 
> Nov 18 23:09:00 tornado kernel: Badness in atomic_dec_and_test at 
> include/asm/atomic.h:150
> Nov 18 23:09:00 tornado kernel: Call Trace:
> Nov 18 23:09:00 tornado kernel:  [<c029203c>] skb_release_data+0x14c/0x160
> Nov 18 23:09:00 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> Nov 18 23:09:00 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> Nov 18 23:09:00 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
> Nov 18 23:09:00 tornado kernel:  [<c0296e9a>] net_tx_action+0x4a/0xf0
> Nov 18 23:09:00 tornado kernel:  [<c01234a5>] do_softirq+0x95/0xa0
> Nov 18 23:09:00 tornado kernel:  [<c010cd9b>] do_IRQ+0xfb/0x130
> Nov 18 23:09:00 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:00 tornado kernel:  [<c02f56fc>] common_interrupt+0x18/0x20
> Nov 18 23:09:00 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:00 tornado kernel:  [<e08cd23d>] 
> acpi_processor_idle+0xd4/0x1c5 [processor]
> Nov 18 23:09:00 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:00 tornado kernel:  [<c0109044>] cpu_idle+0x34/0x40
> Nov 18 23:09:00 tornado kernel:  [<c0398795>] start_kernel+0x185/0x1c0
> Nov 18 23:09:00 tornado kernel:  [<c03984b0>] unknown_bootoption+0x0/0x120
> Nov 18 23:09:00 tornado kernel:
> Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef
> Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket dfd4c780
> Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef
> Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket dfd4c780
> Nov 18 23:09:01 tornado kernel: BUG: dst underflow 0: c02921ef
> Nov 18 23:09:01 tornado kernel: Badness in atomic_dec_and_test at 
> include/asm/atomic.h:150
> Nov 18 23:09:01 tornado kernel: Call Trace:
> Nov 18 23:09:01 tornado kernel:  [<c029203c>] skb_release_data+0x14c/0x160
> Nov 18 23:09:01 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> Nov 18 23:09:01 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> Nov 18 23:09:01 tornado kernel:  [<c02921ef>] __kfree_skb+0x16f/0x1b0
> Nov 18 23:09:01 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
> Nov 18 23:09:01 tornado kernel:  [<c0296e9a>] net_tx_action+0x4a/0xf0
> Nov 18 23:09:01 tornado kernel:  [<c01234a5>] do_softirq+0x95/0xa0
> Nov 18 23:09:01 tornado kernel:  [<c010cd9b>] do_IRQ+0xfb/0x130
> Nov 18 23:09:01 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:01 tornado kernel:  [<c02f56fc>] common_interrupt+0x18/0x20
> Nov 18 23:09:01 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:01 tornado kernel:  [<e08cd21c>] 
> acpi_processor_idle+0xb3/0x1c5 [processor]
> Nov 18 23:09:01 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:01 tornado kernel:  [<c0109044>] cpu_idle+0x34/0x40
> Nov 18 23:09:01 tornado kernel:  [<c0398795>] start_kernel+0x185/0x1c0
> Nov 18 23:09:01 tornado kernel:  [<c03984b0>] unknown_bootoption+0x0/0x120
> Nov 18 23:09:01 tornado kernel:
> Nov 18 23:09:01 tornado kernel: BUG: dst underflow -1: c02921ef
> Nov 18 23:09:01 tornado kernel: Badness in atomic_dec_and_test at 
> include/asm/atomic.h:150
> Nov 18 23:09:01 tornado kernel: Call Trace:
> Nov 18 23:09:01 tornado kernel:  [<c0291136>] sock_wfree+0x86/0xa0
> Nov 18 23:09:01 tornado kernel:  [<c02920fe>] __kfree_skb+0x7e/0x1b0
> Nov 18 23:09:01 tornado kernel:  [<c02921ef>] __kfree_skb+0x16f/0x1b0
> Nov 18 23:09:01 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
> Nov 18 23:09:01 tornado kernel:  [<c0296e9a>] net_tx_action+0x4a/0xf0
> Nov 18 23:09:01 tornado kernel:  [<c01234a5>] do_softirq+0x95/0xa0
> Nov 18 23:09:01 tornado kernel:  [<c010cd9b>] do_IRQ+0xfb/0x130
> Nov 18 23:09:01 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:01 tornado kernel:  [<c02f56fc>] common_interrupt+0x18/0x20
> Nov 18 23:09:01 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:01 tornado kernel:  [<e08cd21c>] 
> acpi_processor_idle+0xb3/0x1c5 [processor]
> Nov 18 23:09:01 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:09:01 tornado kernel:  [<c0109044>] cpu_idle+0x34/0x40
> Nov 18 23:09:01 tornado kernel:  [<c0398795>] start_kernel+0x185/0x1c0
> Nov 18 23:09:01 tornado kernel:  [<c03984b0>] unknown_bootoption+0x0/0x120
> Nov 18 23:09:01 tornado kernel:
> Nov 18 23:09:02 tornado kernel: Attempt to release alive inet socket dfd4c780
> Nov 18 23:09:02 tornado kernel: Attempt to release alive inet socket dfd4c780
> Nov 18 23:09:02 tornado kernel: BUG: dst underflow 0: c02d5c1b
> Nov 18 23:09:02 tornado kernel: BUG: dst underflow 0: c02921ef
> Nov 18 23:09:02 tornado kernel: Attempt to release alive inet socket dfd4c780
> Nov 18 23:09:02 tornado kernel: BUG: dst underflow -1: c02921ef
> Nov 18 23:09:02 tornado kernel: Badness in atomic_dec_and_test at 
> include/asm/atomic.h:150
> Nov 18 23:09:02 tornado kernel: Call Trace:
> Nov 18 23:09:02 tornado kernel:  [<c0291136>] sock_wfree+0x86/0xa0
> Nov 18 23:09:02 tornado kernel:  [<c02920fe>] __kfree_skb+0x7e/0x1b0
> Nov 18 23:09:02 tornado kernel:  [<c02921ef>] __kfree_skb+0x16f/0x1b0
> Nov 18 23:09:02 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
> Nov 18 23:09:02 tornado kernel:  [<c0296e9a>] net_tx_action+0x4a/0xf0
> Nov 18 23:09:02 tornado kernel:  [<c01234a5>] do_softirq+0x95/0xa0
> Nov 18 23:09:02 tornado kernel:  [<c010cd9b>] do_IRQ+0xfb/0x130
> Nov 18 23:09:02 tornado kernel:  [<c02f56fc>] common_interrupt+0x18/0x20
> Nov 18 23:09:02 tornado kernel:  [<c02ee87e>] unix_dgram_sendmsg+0x3e/0x700
> Nov 18 23:09:02 tornado kernel:  [<c01387bc>] find_get_page+0x2c/0x60
> Nov 18 23:09:02 tornado kernel:  [<c028e3d3>] sock_aio_write+0xc3/0xf0
> Nov 18 23:09:02 tornado kernel:  [<c0156841>] do_sync_write+0xb1/0xe0
> Nov 18 23:09:02 tornado kernel:  [<c01385c5>] unlock_page+0x15/0x60
> Nov 18 23:09:02 tornado kernel:  [<c011b670>] schedule+0x350/0x680
> Nov 18 23:09:02 tornado kernel:  [<c011d2a0>] autoremove_wake_function+0x0/0x50
> Nov 18 23:09:02 tornado kernel:  [<c011b9f0>] default_wake_function+0x0/0x20
> Nov 18 23:09:02 tornado kernel:  [<c0127be0>] do_timer+0xe0/0xf0
> Nov 18 23:09:02 tornado kernel:  [<c015696f>] vfs_write+0xff/0x130
> Nov 18 23:09:02 tornado kernel:  [<c0156a52>] sys_write+0x42/0x70
> Nov 18 23:09:02 tornado kernel:  [<c02f4d3a>] sysenter_past_esp+0x43/0x65
> Nov 18 23:09:02 tornado kernel:
> Nov 18 23:09:02 tornado kernel: BUG: dst underflow -2: c02921ef
> 
> 
> Crash 2:
> 
> Nov 18 23:23:56 tornado kernel: Badness in atomic_dec_and_test at 
> include/asm/atomic.h:150
> Nov 18 23:23:56 tornado kernel: Call Trace:
> Nov 18 23:23:56 tornado kernel:  [<c029203c>] skb_release_data+0x14c/0x160
> Nov 18 23:23:56 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> Nov 18 23:23:56 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> Nov 18 23:23:56 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
> Nov 18 23:23:56 tornado kernel:  [<c0296e9a>] net_tx_action+0x4a/0xf0
> Nov 18 23:23:56 tornado kernel:  [<c01234a5>] do_softirq+0x95/0xa0
> Nov 18 23:23:56 tornado kernel:  [<c010cd9b>] do_IRQ+0xfb/0x130
> Nov 18 23:23:56 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:23:56 tornado kernel:  [<c02f56fc>] common_interrupt+0x18/0x20
> Nov 18 23:23:56 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:23:56 tornado kernel:  [<e08cd23d>] 
> acpi_processor_idle+0xd4/0x1c5 [processor]
> Nov 18 23:23:56 tornado kernel:  [<c0107000>] rest_init+0x0/0x60
> Nov 18 23:23:56 tornado kernel:  [<c0109044>] cpu_idle+0x34/0x40
> Nov 18 23:23:56 tornado kernel:  [<c0398795>] start_kernel+0x185/0x1c0
> Nov 18 23:23:56 tornado kernel:  [<c03984b0>] unknown_bootoption+0x0/0x120
> Nov 18 23:23:56 tornado kernel:
> Nov 18 23:23:56 tornado kernel: BUG: dst underflow 0: c02921ef
> Nov 18 23:23:56 tornado kernel: Attempt to release alive inet socket dfd4c780
> Nov 18 23:23:56 tornado kernel: BUG: dst underflow 0: c02921ef
> Nov 18 23:23:56 tornado kernel: Attempt to release alive inet socket dfd4c780
> 

Re: Kernel crash in 2.6.0-test9-mm3

[David S. Miller]

On Tue, 18 Nov 2003 11:01:39 -0800
Andrew Morton <akpm@osdl.org> wrote:

> It's one for the networking guys.
> 
> The mm kernels have a patch which detects when atomic_dec_and_test
> takes an atomic_t negative - it is assumed that this is a bug so
> a warning is generated.

It is a bug especially for the backtrace cases shown here, I'll take a
look.

Re: Kernel crash in 2.6.0-test9-mm3

[David S. Miller]

On Tue, 18 Nov 2003 11:01:39 -0800
Andrew Morton <akpm@osdl.org> wrote:

> It's one for the networking guys.
> 
> The mm kernels have a patch which detects when atomic_dec_and_test
> takes an atomic_t negative - it is assumed that this is a bug so
> a warning is generated.

Andrew I've analyzed this a bit.  This is incredible evidence in
these dumps that either there is a bug in Linus's atomic_dec_and_test()
debugging hack or GCC is miscompiling it in certain cases with certain
versions of the compiler.

Look at this:

> > Nov 18 23:09:00 tornado kernel:  [<c029203c>] skb_release_data+0x14c/0x160
> > Nov 18 23:09:00 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> > Nov 18 23:09:00 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> > Nov 18 23:09:00 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290

Ok, releasing an SKB data area twice.

> > Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef

Freeing a 'dst' entry one too many times.

> > Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket dfd4c780

A socket refcount dropping to zero too early, before it's marked dead.

These last two problems are very serious errors, and would have
printed out debugging messages before the atomic_dec_and_test() patch.
If these last two messages don't show up without the
atomic_dec_and_test() debugging patch applied, well there you
go... :-)

In that debugging patch, I'm wondering something about x86.
When one goes "sete %reg; sets %reg" does the first 'sete' modify
the condition codes by chance?  Probably not...

Re: Kernel crash in 2.6.0-test9-mm3

[Reuben Farrelly]

FWIW I'm compiling with:

[root@tornado log]# gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.3.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --enable-shared --enable-threads=posix 
--disable-checking --with-system-zlib --enable-__cxa_atexit 
--host=i386-redhat-linux
Thread model: posix
gcc version 3.3.2 20031107 (Red Hat Linux 3.3.2-2)
[root@tornado log]#

Reuben


At 13:49 19/11/2003, David S. Miller wrote:
>On Tue, 18 Nov 2003 11:01:39 -0800
>Andrew Morton <akpm@osdl.org> wrote:
>
> > It's one for the networking guys.
> >
> > The mm kernels have a patch which detects when atomic_dec_and_test
> > takes an atomic_t negative - it is assumed that this is a bug so
> > a warning is generated.
>
>Andrew I've analyzed this a bit.  This is incredible evidence in
>these dumps that either there is a bug in Linus's atomic_dec_and_test()
>debugging hack or GCC is miscompiling it in certain cases with certain
>versions of the compiler.
>
>Look at this:
>
> > > Nov 18 23:09:00 tornado kernel:  [<c029203c>] 
> skb_release_data+0x14c/0x160
> > > Nov 18 23:09:00 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> > > Nov 18 23:09:00 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> > > Nov 18 23:09:00 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
>
>Ok, releasing an SKB data area twice.
>
> > > Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef
>
>Freeing a 'dst' entry one too many times.
>
> > > Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket 
> dfd4c780
>
>A socket refcount dropping to zero too early, before it's marked dead.
>
>These last two problems are very serious errors, and would have
>printed out debugging messages before the atomic_dec_and_test() patch.
>If these last two messages don't show up without the
>atomic_dec_and_test() debugging patch applied, well there you
>go... :-)
>
>In that debugging patch, I'm wondering something about x86.
>When one goes "sete %reg; sets %reg" does the first 'sete' modify
>the condition codes by chance?  Probably not...

Re: Kernel crash in 2.6.0-test9-mm3

[Andrew Morton]

"David S. Miller" <davem@redhat.com> wrote:
>
> On Tue, 18 Nov 2003 11:01:39 -0800
> Andrew Morton <akpm@osdl.org> wrote:
> 
> > It's one for the networking guys.
> > 
> > The mm kernels have a patch which detects when atomic_dec_and_test
> > takes an atomic_t negative - it is assumed that this is a bug so
> > a warning is generated.
> 
> Andrew I've analyzed this a bit.  This is incredible evidence in
> these dumps that either there is a bug in Linus's atomic_dec_and_test()
> debugging hack or GCC is miscompiling it in certain cases with certain
> versions of the compiler.
> 
> Look at this:
> 
> > > Nov 18 23:09:00 tornado kernel:  [<c029203c>] skb_release_data+0x14c/0x160
> > > Nov 18 23:09:00 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> > > Nov 18 23:09:00 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> > > Nov 18 23:09:00 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
> 
> Ok, releasing an SKB data area twice.
> 
> > > Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef
> 
> Freeing a 'dst' entry one too many times.
> 
> > > Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket dfd4c780
> 
> A socket refcount dropping to zero too early, before it's marked dead.
> 
> These last two problems are very serious errors, and would have
> printed out debugging messages before the atomic_dec_and_test() patch.
> If these last two messages don't show up without the
> atomic_dec_and_test() debugging patch applied, well there you
> go... :-)
> 
> In that debugging patch, I'm wondering something about x86.
> When one goes "sete %reg; sets %reg" does the first 'sete' modify
> the condition codes by chance?  Probably not...

Beats me David.  This is the only time where the correctness of that patch
has been questioned.

Reuben, can you please do a patch -R of

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.0-test9/2.6.0-test9-mm3/broken-out/atomic_dec-debug.patch

and see if the problem goes away?

Re: Kernel crash in 2.6.0-test9-mm3

[Krishna Kumar]

Could this be happening on an SMP system only ? If so, e100intr routine
services tx queues (e100_tx_srv)
without holding a lock. Can't multiple rx interrupts be scheduled on
different cpus at the same time, and
each execute dev_kfree_skb_irq() which decrements the ref count too many
times ? But the softirq handler
(net_tx_action) seems to clean up the skb once as the dec_test returns 1
only if count is zero, so I don't see
where the dst ref is being decremented wrongly in this case.

Can someone explain why the intr handler doesn't need locks to stop other
intr on different cpu's from going
through the same devices memory at the same time ?

Thanks,

- KK



|---------+---------------------------->
|         |           Reuben Farrelly  |
|         |           <reuben-linux@reu|
|         |           b.net>           |
|         |           Sent by:         |
|         |           netdev-bounce@oss|
|         |           .sgi.com         |
|         |                            |
|         |                            |
|         |           11/18/2003 05:22 |
|         |           PM               |
|         |                            |
|---------+---------------------------->
  >-----------------------------------------------------------------------------------------------------------------|
  |                                                                                                                 |
  |       To:       "David S. Miller" <davem@redhat.com>, Andrew Morton <akpm@osdl.org>                             |
  |       cc:       netdev@oss.sgi.com                                                                              |
  |       Subject:  Re: Kernel crash in 2.6.0-test9-mm3                                                             |
  |                                                                                                                 |
  >-----------------------------------------------------------------------------------------------------------------|




FWIW I'm compiling with:

[root@tornado log]# gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.3.2/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--disable-checking --with-system-zlib --enable-__cxa_atexit
--host=i386-redhat-linux
Thread model: posix
gcc version 3.3.2 20031107 (Red Hat Linux 3.3.2-2)
[root@tornado log]#

Reuben


At 13:49 19/11/2003, David S. Miller wrote:
>On Tue, 18 Nov 2003 11:01:39 -0800
>Andrew Morton <akpm@osdl.org> wrote:
>
> > It's one for the networking guys.
> >
> > The mm kernels have a patch which detects when atomic_dec_and_test
> > takes an atomic_t negative - it is assumed that this is a bug so
> > a warning is generated.
>
>Andrew I've analyzed this a bit.  This is incredible evidence in
>these dumps that either there is a bug in Linus's atomic_dec_and_test()
>debugging hack or GCC is miscompiling it in certain cases with certain
>versions of the compiler.
>
>Look at this:
>
> > > Nov 18 23:09:00 tornado kernel:  [<c029203c>]
> skb_release_data+0x14c/0x160
> > > Nov 18 23:09:00 tornado kernel:  [<c0292063>] kfree_skbmem+0x13/0x30
> > > Nov 18 23:09:00 tornado kernel:  [<c0292138>] __kfree_skb+0xb8/0x1b0
> > > Nov 18 23:09:00 tornado kernel:  [<c0218815>] e100intr+0x1e5/0x290
>
>Ok, releasing an SKB data area twice.
>
> > > Nov 18 23:09:00 tornado kernel: BUG: dst underflow 0: c02921ef
>
>Freeing a 'dst' entry one too many times.
>
> > > Nov 18 23:09:00 tornado kernel: Attempt to release alive inet socket
> dfd4c780
>
>A socket refcount dropping to zero too early, before it's marked dead.
>
>These last two problems are very serious errors, and would have
>printed out debugging messages before the atomic_dec_and_test() patch.
>If these last two messages don't show up without the
>atomic_dec_and_test() debugging patch applied, well there you
>go... :-)
>
>In that debugging patch, I'm wondering something about x86.
>When one goes "sete %reg; sets %reg" does the first 'sete' modify
>the condition codes by chance?  Probably not...

Re: Kernel crash in 2.6.0-test9-mm3

[David S. Miller]

On Tue, 18 Nov 2003 18:22:42 -0800
Krishna Kumar <kumarkr@us.ibm.com> wrote:

> Could this be happening on an SMP system only ? If so, e100intr routine
> services tx queues (e100_tx_srv) without holding a lock.

[ Scott we have a bug report, and we're trying to determine if the
  cause is that the e100 driver frees a TX SKB multiple times due
  to some race or other problem in current 2.6.x ]

That's a very good point, and I looked a bit in this area.

When e100intr() is doing it's work, it disables and clears
the interrupt, only after doing RX and TX processing does
it reenable chip interrupts via e100_set_intr_mask().

That is my analysis of the situation.

However, with things like IOAPIC and such, it might be possible
for two cpus to enter e100intr() simultaneously, both read
the same status, both see that the interrupt is pending, and
both thus process the interrupt and race with each other.

Scott, what prevents the above from happening?

Re: Kernel crash in 2.6.0-test9-mm3

[Reuben Farrelly]

The system is a UP system..

# CONFIG_SMP is not set
CONFIG_PREEMPT=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y

I've put the full .config up at http://www.reub.net/linux/

Reuben



At 15:22 19/11/2003, Krishna Kumar wrote:
>Could this be happening on an SMP system only ? If so, e100intr routine
>services tx queues (e100_tx_srv)
>without holding a lock. Can't multiple rx interrupts be scheduled on
>different cpus at the same time, and
>each execute dev_kfree_skb_irq() which decrements the ref count too many
>times ? But the softirq handler
>(net_tx_action) seems to clean up the skb once as the dec_test returns 1
>only if count is zero, so I don't see
>where the dst ref is being decremented wrongly in this case.

RE: Kernel crash in 2.6.0-test9-mm3

[Feldman, Scott]

> However, with things like IOAPIC and such, it might be 
> possible for two cpus to enter e100intr() simultaneously, 
> both read the same status, both see that the interrupt is 
> pending, and both thus process the interrupt and race with each other.
> 
> Scott, what prevents the above from happening?

Whoa, this question is freaking me out just a little bit: my assumption
is that the device's interrupt line has been masked off at the CPU/PIC
before e100intr() is ever called, so 1) there really isn't any need to
disable device's interrupts from the driver (see eepro100.c), 2) or even
hold a lock unless we shared something critical on the queuing side (see
e1000), and 3) only one e100intr is running.  [public spanking in
order?]

I'm not sure what's behind the rest of the bug report, but if you're
saying e100intr() can be running simultaneously on two different CPUs,
then there is a problem because the test for device interrupt and the
acking of device interrupt are two steps that need to be protected with
a lock.

-scott

Re: Kernel crash in 2.6.0-test9-mm3

[Reuben Farrelly]

Yes - with the patch backed out, the system has been up for 20 hours 
without a crash and nothing has been logged.  I was just holding off a bit 
longer before reporting back in case it just needs a bit more time to 
trigger ;-)

Reuben


At 15:51 20/11/2003, David S. Miller wrote:
>On Wed, 19 Nov 2003 15:58:18 +1300
>Reuben Farrelly <reuben-linux@reub.net> wrote:
>
> > The system is a UP system..
>
>Have you done ask Andrew asked, backing out the atomic_dec_and_test()
>debugging patch to see if all the weird kernel log messages go away
>as a result?

Re: Kernel crash in 2.6.0-test9-mm3

[Reuben Farrelly]

Ok I'll take this up with jakub@redhat, the gcc package maintainer at Redhat.

What is interesting (or scary) is that the version of gcc that I'm using 
(3.3.2-2) is only one rebuild out from the one shipped in Fedora Core 1 
(3.3.2-1), so I may not be the last person to hit this problem..

Reuben


At 04:02 p.m. 20/11/2003, David S. Miller wrote:
>On Thu, 20 Nov 2003 16:05:27 +1300
>Reuben Farrelly <reuben-linux@reub.net> wrote:
>
> > Yes - with the patch backed out, the system has been up for 20 hours
> > without a crash and nothing has been logged.  I was just holding off a bit
> > longer before reporting back in case it just needs a bit more time to
> > trigger ;-)
>
>Thanks a lot for this feedback.
>
>I really think gcc-3.x you're using is miscompiling some parts of
>the kernel with the patch applied.  Someone should look at the
>x86 output assembler for things like the atomic_dec_and_test() call
>in net/core/skbuff.c:kfree_skbmem()

Re: Kernel crash in 2.6.0-test9-mm3

[Rask Ingemann Lambertsen]

On Wed, Nov 19, 2003 at 06:40:04PM -0800, Feldman, Scott wrote:
> > However, with things like IOAPIC and such, it might be 
> > possible for two cpus to enter e100intr() simultaneously, 
> > both read the same status, both see that the interrupt is 
> > pending, and both thus process the interrupt and race with each other.
> > 
> > Scott, what prevents the above from happening?
> 
> Whoa, this question is freaking me out just a little bit: my assumption
> is that the device's interrupt line has been masked off at the CPU/PIC
> before e100intr() is ever called, so 1) there really isn't any need to
> disable device's interrupts from the driver (see eepro100.c), 2) or even
> hold a lock unless we shared something critical on the queuing side (see
> e1000), and 3) only one e100intr is running.  [public spanking in
> order?]

Me too. My impression is also that the kernel guarantees that the interrupt
handler is single threaded, regardless of IOAPIC, SMP and such.

-- 
Regards,
Rask Ingemann Lambertsen