mm kernel oops with r8169 & named, PREEMPT

mm kernel oops with r8169 & named, PREEMPT

[Danny]

This is with the network driver r8169 and linux-2.6.9-rc4-mm1.  Same
thing happened with linux-2.6.9-rc3-mm3 (but also locked up). 
linux-2.6.8.1-mm4 didn't seem to have this problem.  This is very
repeatable, if this is an unknown issue let me know (CC please, not on
the list) and I will jump through the hoops to get a useful oops.

Re: mm kernel oops with r8169 & named, PREEMPT

[Jeff Garzik]

Danny wrote:
> This is with the network driver r8169 and linux-2.6.9-rc4-mm1.  Same
> thing happened with linux-2.6.9-rc3-mm3 (but also locked up). 
> linux-2.6.8.1-mm4 didn't seem to have this problem.  This is very
> repeatable, if this is an unknown issue let me know (CC please, not on
> the list) and I will jump through the hoops to get a useful oops.

What happens if you disable preempt?

lspci?  config?  Any of the other useful info mentioned in the 
REPORTING-BUGS file in the kernel tree?

	Jeff

Re: mm kernel oops with r8169 & named, PREEMPT

[Francois Romieu]

Danny <dannydaemonic@gmail.com> :
> This is with the network driver r8169 and linux-2.6.9-rc4-mm1.  Same
> thing happened with linux-2.6.9-rc3-mm3 (but also locked up). 
> linux-2.6.8.1-mm4 didn't seem to have this problem.  This is very
> repeatable, if this is an unknown issue let me know (CC please, not on
> the list) and I will jump through the hoops to get a useful oops.

Try the patch below (courtesy of Jon Mason, whitespaces may be wrong) and
see 1) if things perform better 2) if "timeout" messages appear in the
kernel log.

Oops as well as Cc: netdev@oss.sgi.com are welcome.

--- linux-2.6.9-rc4-mm1/drivers/net/r8169.c     2004-10-12 13:59:57.000000000 -0500
+++ linux-2.6.9-rc4-mm1/drivers/net/r8169.c     2004-10-12 10:51:21.000000000 -0500
@@ -1680,6 +1680,7 @@ static void rtl8169_unmap_tx_skb(struct

 	pci_unmap_single(pdev, le64_to_cpu(desc->addr), len, PCI_DMA_TODEVICE);
 	desc->opts2 = 0x00;
+	desc->opts1 = 0x00;
 	desc->addr = 0x00;
 	tx_skb->len = 0;
 }

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

On Wed, 13 Oct 2004 09:28:14 +0200, Francois Romieu wrote:
> Try the patch below (courtesy of Jon Mason, whitespaces may be wrong) and
> see 1) if things perform better 2) if "timeout" messages appear in the
> kernel log.

The patch doesn't fix or prevent the oops.  Performance might have
been better but I did no formal tests.  There were no "timeout"
messages in the kernel log, however I only ran it with this change for
35-45 minutes.

I should mention that in the kernel log, with linux-2.6.8.1-mm4, it
complains "process `named' is using obsolete setsockopt SO_BSDCOMPAT".
 However, with the most recent, 2.6.9-rc4-mm1, it doesn't get that
far.  A "Unable to handle kernel paging request at virtual address
00017f8c" happens instead.  I'm guessing the oops is just killing
named before it gets that far.

I enabled some more debug options in the kernel and I'm getting a 2nd
oops following the first.

I wasn't sure if I should paste the huge oops here, and since the raw
dmesg also shows spin lock errors, I thought I'd just post both on the
web:
http://members.cox.net/valenzdu/oops-raw
http://members.cox.net/valenzdu/oops-processed

I ran it through ksymoops but I don't have a /proc/ksyms and when I
tried using /proc/kallsyms it gave me a format error.  I hope this is
helpful, let me know if there is anything else I can do. (CC me
please.)

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

I should mention that the oops (there was actually 3 of them not just
2) was without the patch suggested by Francois Romieu.  I assumed that
would be more useful as I still get the oops with it.  Let me know if
it would help and I'll repatch it and send another oops.

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

I should mention that the oops (there was actually 3 of them not just
2) was without the patch suggested by Francois Romieu.  I assumed that
would be more useful as I still get the oops with it.  Let me know if
it would help and I'll repatch it and send another oops.

Re: mm kernel oops with r8169 & named, PREEMPT

[Jon Mason]

Can you confirm that you are running r8169 driver with NAPI and TSO turned on, 
along with Preemptable Kernel?  Also, I didn't see anything in the Oops 
specific to the r8169 driver, do you have another adapter available to run 
the same test on?  Finally, what is your setup (arch, # of cpus, etc)?

-- 
Jon Mason
jdmason@us.ibm.com

Re: mm kernel oops with r8169 & named, PREEMPT

[Francois Romieu]

Jon Mason <jdmason@us.ibm.com> :
> Can you confirm that you are running r8169 driver with NAPI and TSO turned on, 
> along with Preemptable Kernel?  Also, I didn't see anything in the Oops 
> specific to the r8169 driver, do you have another adapter available to run 
> the same test on?  Finally, what is your setup (arch, # of cpus, etc)?

Yep, it does not look closely related to the network driver (there are both
eth1 and unknown eth0 btw).

Danny, can your drop the r8169 driver from 2.6.9-rc4-mm1 into vanilla
2.6.9-rc4 and confirm that it works (preempt should not matter) ?

If it does not, disable ACPI, preempt, profiling and publish the new oops.

--
Ueimor

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

On Wed, 13 Oct 2004 20:18:40 +0200, Francois Romieu wrote:
> Danny, can your drop the r8169 driver from 2.6.9-rc4-mm1 into vanilla
> 2.6.9-rc4 and confirm that it works (preempt should not matter) ?

I see how the timing on the installation of the r8169 could have just
been bad.  Since preempt shouldn't matter, I'm going to test that
and acpi etc first (because it's easier to test heh).

See, well, I'm going to need reiser4 for the vanilla.  I found patches here:
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/

But can someone point out which patches I will need?  Do I only need
reiser4-only.patch or is it safer for my fs if I have all the other
reiser4 patches?

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

On Wed, 13 Oct 2004 22:54:33 +0200, Francois Romieu wrote:
> [snip]
> Possible :o)
> 
> OTOH, I do not want to miss an error report.

Well the options I tried disabling (ACPI, PREEMPT) didn't stop the oops.

> [snip]
> I am not a reiserfs addict but you can imho go for the complete serie
> of reiserfs patches once you have sucked the first one (3Mo, yuck).
> 
> If you have issues with the ordering of the patches, just grep the
> patches in the 'series' file.

Ok thanks, I have work to do right now so I'll have to try this out tonight.

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

I wasn't able to get the kernel to compile with just the reiser4
patches from the broken-out dir on the ftp.  The patching itself
seemed to go well though.  I tried to apply the patches in the order
they were listed in "series" but some had to go out of order anyway,
such as reiser4-only.patch had to go first.

However I was able to verify it's not a problem with the r8169 driver.
 I copied over the r8169 driver to a fresh linux-2.6.8.1-mm4 and was
able to compile and run fine with the latest r8169 driver from the
linux-2.6.9-rc4-mm1 release.  I also tested it in reverse, bringing
over r8169.c to a fresh linux-2.6.9-rc4-mm1 and it still gave me the
oops.

I'm not sure where this leaves me now.  Perhaps I should repost my
oops and my proc info in the lkml with a different subject to draw the
attention of others, since I now know this subject to be entirely
missleading.

Re: mm kernel oops with r8169 & named, PREEMPT

[Francois Romieu]

Danny <dannydaemonic@gmail.com> :
[...]
> I'm not sure where this leaves me now.  Perhaps I should repost my
> oops and my proc info in the lkml with a different subject to draw the
> attention of others, since I now know this subject to be entirely
> missleading.

May be try this one first:

http://www.uwsg.indiana.edu/hypermail/linux/kernel/0410.1/1920.html

--
Ueimor

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

On Fri, 15 Oct 2004 18:18:18 +0200, Francois Romieu wrote:
> May be try this one first:
> 
> http://www.uwsg.indiana.edu/hypermail/linux/kernel/0410.1/1920.html

Thanks, that appears to have fixed it.  I'm not getting an oops any
more.  I wish I hadn't assumed so much in my subject line, perhaps
someone would have found the solution before John Flinchbaugh even
reported his problem.

Also posting the oops directly instead of linking to it probably
allows people passing by to look at the oops, increasing the total
number of eyes.

I have a quick question though, if I'm using the kernel with all the
debug features turned on, should I still run it through ksymoops?

profile path bug again (was Re: mm kernel oops with r8169 & named, PREEMPT)

[Francois Romieu]

Danny <dannydaemonic@gmail.com> :
[profile path bug hits hard]
> Also posting the oops directly instead of linking to it probably
> allows people passing by to look at the oops, increasing the total
> number of eyes.

Yes.

Oops posting is still on-topic in l-k.

> I have a quick question though, if I'm using the kernel with all the
> debug features turned on, should I still run it through ksymoops?

Your file named "oops-raw" contained the relevant information.

--
Ueimor

Re: profile path bug again (was Re: mm kernel oops with r8169 & named, PREEMPT)

[Danny]

Thanks for your help.  Live and learn. :)

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

On Wed, 13 Oct 2004 11:29:05 -0500, Jon Mason wrote:
> Can you confirm that you are running r8169 driver with NAPI and TSO turned on,
> along with Preemptable Kernel?  Also, I didn't see anything in the Oops
> specific to the r8169 driver, do you have another adapter available to run
> the same test on?  Finally, what is your setup (arch, # of cpus, etc)?

I am using the r8169 driver with NAPI and preemptable kernel.  I don't
remember or see any option for offloading but it sounds like something
I'd turn on.  Let me know what I should look for in my config, or
check here:
http://members.cox.net/valenzdu/.config

I no longer have a 2nd nic adapter, but I did first notice the problem
when I installed this one.  However, if I don't load that module I'm
not able to duplicate the problem by running named on the other
interface.  Running named triggers it 100% of the time on the r8169
based nic.

The arch is x86 (athlon) with just 1 cpu.  Since I'm not sure what's
wrong here I'm not sure what's relevant.  Here is more info about my
setup:
http://members.cox.net/valenzdu/proc-cpuinfo
http://members.cox.net/valenzdu/proc-iomem
http://members.cox.net/valenzdu/proc-ioports
http://members.cox.net/valenzdu/proc-modules
http://members.cox.net/valenzdu/proc-version
http://members.cox.net/valenzdu/stdout-lspci
http://members.cox.net/valenzdu/stdout-ver_linux

Re: mm kernel oops with r8169 & named, PREEMPT

[Danny]

On Wed, 13 Oct 2004 17:03:21 -0500, Jon Mason wrote:
> The only thing that jumps out at me is the fact that you are running with
> Reiser4, but I don't want to point any fingers yet.  Please try recreating
> the error without NAPI and preemptable kernel, and if possible without
> Reiser4.

I can recreate it without NAPI and without the preemptable kernel, but
I have no means of recreating this with out Reiser4.  Would an oops
without the NAPI and preemptable kernel be more useful than the one I
already provided?  If so I can make another oops.

I spoke with Nikita shortly on OFTC and he said the oops "is not
related to reiser4, at least not directly. Maybe reiser4 corrupted
some internal data-structures some where in the kernel which caused
oops later."

Shrug.