Re: Linux 2.6.21-rc5

[Ayaz Abdulla <aabdulla@nvidia.com>]

This issue might be resolved with the patch provided in the following 
bug report: http://bugzilla.kernel.org/show_bug.cgi?id=8058

Please try out the patch in the bug report without your patch and see if 
the issue reproduces.

Ayaz


Ingo Molnar wrote:
> * Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
> 
>>There's various fixes here, ranging from some architecture updates 
>>(ia64, ARM, MIPS, SH, Sparc64) to KVM, networking and network drivers.
> 
> 
> here's a new v2.6.20 -> v2.6.21 forcedeth.c regression:
> 
> in the last week or so i've been seeing sporadic under-load forcedeth.c 
> crashes (see the full oops further below):
> 
>  eth1: too many iterations (6) in nv_nic_irq.
>  Unable to handle kernel NULL pointer dereference at 0000000000000088 RIP: 
>  [<ffffffff80404587>] nv_tx_done+0xf4/0x1cf
> 
> this is line 1906 of drivers/net/forcedeth.c:
> 
>     np->stats.tx_bytes += np->get_tx_ctx->skb->len;
> 
> struct sk_buff's len field is at offset 88, so np->get_tx_ctx->skb is 
> NULL. That is an 'impossible' scenario for tx descriptors here - the tx 
> ring descriptors are always set up with a valid skb (and a valid dma 
> address), and their completion is serialized via np->lock.
> 
> these crashes are almost instant on the .21-rc5-rt kernel, but extremely 
> sporadic on the upstream kernel and needed very high networking loads to 
> trigger. Today i found a good way to trigger it almost instantly on 
> upstream kernels too: apply the debug patch attached further below and 
> do:
> 
> 	echo 100 > /proc/sys/kernel/panic
> 
> that will inject 100 artificial 'too many iterations' failures and 
> provokes a TX timeout - which TX timeout will crash. (i've used a 
> dual-core Athlon64 system in this test)
> 
> my first quick guess was to extend np->priv locking to the whole of 
> nv_start_xmit/nv_start_xmit_optimized - while that appeared to make the 
> crash a bit less likely, it did not prevent it. So there must be some 
> other, more fundamental problem be left as well. At first glance the SMP 
> locking looks OK, so maybe the ring indices are messed up somehow and we 
> got into a 'ring head bites the tail' scenario?
> 
> i can provide more info if needed.
> 
> 	Ingo
> 
> -------------->
> eth1: too many iterations (6) in nv_nic_irq.
> Unable to handle kernel NULL pointer dereference at 0000000000000088 RIP: 
>  [<ffffffff80404587>] nv_tx_done+0xf4/0x1cf
> PGD 34d03067 PUD 34d02067 PMD 0 
> Oops: 0000 [1] PREEMPT SMP 
> CPU 1 
> Modules linked in:
> Pid: 0, comm: swapper Not tainted 2.6.21-rc5 #8
> RIP: 0010:[<ffffffff80404587>]  [<ffffffff80404587>] nv_tx_done+0xf4/0x1cf
> RSP: 0018:ffff81003ff6be40  EFLAGS: 00010206
> RAX: 0000000000000000 RBX: ffff810002e26700 RCX: 0000000000000001
> RDX: 0000000000000042 RSI: 000000003ef00cbe RDI: ffff81003fbeb070
> RBP: ffff81003ff6be60 R08: ffff810002e26a00 R09: 0000000000000003
> R10: ffff81003ff4e100 R11: ffff810001e283f8 R12: 000000003ef00cbe
> R13: ffff810002e26000 R14: ffff810002e28fc0 R15: 0000000000000000
> FS:  00002b6cb57f1db0(0000) GS:ffff81003ff4ad40(0000) knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000088 CR3: 0000000034c87000 CR4: 00000000000006e0
> Process swapper (pid: 0, threadinfo ffff81003ff64000, task ffff81003ff4e100)
> Stack:  ffff810002e26700 0000000000000032 ffffc2000001a000 ffff810002e26000
>  ffff81003ff6bea0 ffffffff80406dae ffff810002e26700 ffff810002e26700
>  ffff810002e26000 00000000000000ff ffffc2000001a000 ffffffff80749080
> Call Trace:
>  <IRQ>  [<ffffffff80406dae>] nv_nic_irq+0x76/0x261
>  [<ffffffff8040961e>] nv_do_nic_poll+0x200/0x284
>  [<ffffffff8040941e>] nv_do_nic_poll+0x0/0x284
>  [<ffffffff80241995>] run_timer_softirq+0x167/0x1dd
>  [<ffffffff8023de45>] __do_softirq+0x5b/0xc9
>  [<ffffffff8020af0c>] call_softirq+0x1c/0x28
>  [<ffffffff8020c2b4>] do_softirq+0x31/0x84
>  [<ffffffff8023db16>] irq_exit+0x3f/0x50
>  [<ffffffff802190c2>] smp_apic_timer_interrupt+0x49/0x5b
>  [<ffffffff802087fb>] default_idle+0x0/0x44
>  [<ffffffff8020a9b6>] apic_timer_interrupt+0x66/0x70
>  <EOI>  [<ffffffff8020882a>] default_idle+0x2f/0x44
>  [<ffffffff8020804c>] enter_idle+0x22/0x24
>  [<ffffffff802088d0>] cpu_idle+0x91/0xd4
>  [<ffffffff80218572>] start_secondary+0x2e3/0x2f5
> 
> ---
>  drivers/net/forcedeth.c |   20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> Index: linux/drivers/net/forcedeth.c
> ===================================================================
> --- linux.orig/drivers/net/forcedeth.c
> +++ linux/drivers/net/forcedeth.c
> @@ -2908,6 +2908,10 @@ static irqreturn_t nv_nic_irq(int foo, v
>  			spin_unlock(&np->lock);
>  			break;
>  		}
> +		if (panic_timeout > 0) {
> +			panic_timeout--;
> +			i = max_interrupt_work+1;
> +		}
>  		if (unlikely(i > max_interrupt_work)) {
>  			spin_lock(&np->lock);
>  			/* disable interrupts on the nic */
> @@ -3026,6 +3030,10 @@ static irqreturn_t nv_nic_irq_optimized(
>  			break;
>  		}
>  
> +		if (panic_timeout > 0) {
> +			panic_timeout--;
> +			i = max_interrupt_work+1;
> +		}
>  		if (unlikely(i > max_interrupt_work)) {
>  			spin_lock(&np->lock);
>  			/* disable interrupts on the nic */
> @@ -3076,6 +3084,10 @@ static irqreturn_t nv_nic_irq_tx(int foo
>  			dprintk(KERN_DEBUG "%s: received irq with events 0x%x. Probably TX fail.\n",
>  						dev->name, events);
>  		}
> +		if (panic_timeout > 0) {
> +			panic_timeout--;
> +			i = max_interrupt_work+1;
> +		}
>  		if (unlikely(i > max_interrupt_work)) {
>  			spin_lock_irqsave(&np->lock, flags);
>  			/* disable interrupts on the nic */
> @@ -3191,6 +3203,10 @@ static irqreturn_t nv_nic_irq_rx(int foo
>  			}
>  		}
>  
> +		if (panic_timeout > 0) {
> +			panic_timeout--;
> +			i = max_interrupt_work+1;
> +		}
>  		if (unlikely(i > max_interrupt_work)) {
>  			spin_lock_irqsave(&np->lock, flags);
>  			/* disable interrupts on the nic */
> @@ -3264,6 +3280,10 @@ static irqreturn_t nv_nic_irq_other(int 
>  			printk(KERN_DEBUG "%s: received irq with unknown events 0x%x. Please report\n",
>  						dev->name, events);
>  		}
> +		if (panic_timeout > 0) {
> +			panic_timeout--;
> +			i = max_interrupt_work+1;
> +		}
>  		if (unlikely(i > max_interrupt_work)) {
>  			spin_lock_irqsave(&np->lock, flags);
>  			/* disable interrupts on the nic */