Re: Oops linux 2.4.23-pre6 on amd64

Re: Oops linux 2.4.23-pre6 on amd64

[Andi Kleen]

Tony Hoyle <tmh@nodomain.org> writes:
>
> Trace; ffffffff80202fce <pci_announce_device+3e/60>

It jumped to nirvana, probably because someone passed crap 
to pci_announce_device.

My first guess would be a non matching module.  Do a make distclean
and recompile/reinstall everything.

> Trace; ffffffffa0014560 <[usbcore]hcd_data_lock+4c4c/5f5f06ec>

But the decode is useless because the module in question is not loaded.

Can you load the module whatever it is manually and then decode
the oops while it's still loaded? Or better compile in all USB
statically and see if it oopses too.

Your legacy USB problems are very likely BIOS bugs.

-Andi

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

Andi Kleen wrote:

> Can you load the module whatever it is manually and then decode
> the oops while it's still loaded? Or better compile in all USB
> statically and see if it oopses too.

Here's the decode with it loaded.  The oops happens during the 
initialisation of ehci-hcd (from an init=/bin/bash clean system, I did 
modprobe usb-uhci then modprobe ehci-hcd and it oopsed).

> Your legacy USB problems are very likely BIOS bugs.

Well this BIOS does insist I have 'sourth' bridge, so I don't have high 
hopes on that score :)

Tony


[-- Attachment #2: oops3.txt --]
[-- Type: text/plain, Size: 3559 bytes --]

ksymoops 2.4.9 on x86_64 2.4.23-pre6-amd64.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23-pre6-amd64/ (default)
     -m /boot/System.map-2.4.23-pre6-amd64 (default)
     -t elf64-x86-64 -a x86-64

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Oct  4 19:23:06 (none) kernel: Unable to handle kernel paging request<1> at 00000000a000a700 RIP: [<00000000a000a700>]PML4 3ef32067 PGD 0
Oct  4 19:23:06 (none) kernel: Oops: 0010
Oct  4 19:23:06 (none) kernel: CPU 0
Oct  4 19:23:06 (none) kernel: Pid: 104, comm: modprobe Not tainted
Oct  4 19:23:06 (none) kernel: RIP: 0010:[<00000000a000a700>]
Oct  4 19:23:06 (none) kernel: RSP: 0000:000001003efddde0  EFLAGS: 00010246
Oct  4 19:23:06 (none) kernel: RAX: ffffffffa00140e0 RBX: ffffffffa0014560 RCX: 0000000000000007
Oct  4 19:23:06 (none) kernel: RDX: ffffffffa00140e0 RSI: ffffffffa00140e0 RDI: 000001000262f800
Oct  4 19:23:06 (none) kernel: RBP: 000001000262f800 R08: 0000000000001000 R09: 00000100025a2b08
Oct  4 19:23:06 (none) kernel: R10: 000000000003ef30 R11: 0000010001000048 R12: ffffffffa00140e0
Oct  4 19:23:06 (none) kernel: R13: 0000000000000000 R14: ffffff0000a11000 R15: 000001003eed9140
Oct  4 19:23:06 (none) kernel: FS:  0000000000000000(0000) GS:ffffffff8034b540(0000) knlGS:0000000000000000
Oct  4 19:23:06 (none) kernel: CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Oct  4 19:23:06 (none) kernel: CR2: 00000000a000a700 CR3: 0000000000101000 CR4: 00000000000006e0
Oct  4 19:23:06 (none) kernel: Process modprobe (pid: 104, stackpage=1003efdd000)
Oct  4 19:23:06 (none) kernel: Stack: 000001003efddde0 0000000000000000 ffffffff80202fce 0000000000000007
Oct  4 19:23:06 (none) kernel:        000001000262f800 ffffffffa0014560 0000000000000000 00000000000000b8
Oct  4 19:23:06 (none) kernel:        ffffffff80203048 ffffffffffffffea ffffffffa0010000 0000000008086dc8
Oct  4 19:23:06 (none) kernel: Call Trace: [<ffffffff80202fce>] [<ffffffffa0014560>]
Oct  4 19:23:06 (none) kernel:        [<ffffffff80203048>] [<ffffffffa0013f4d>] [<ffffffff8012008e>]
Oct  4 19:23:06 (none) kernel:        [<ffffffff8012e21c>] [<ffffffffa00100b8>] [<ffffffff80189963>]
Oct  4 19:23:06 (none) kernel: Code:  Bad RIP value.
Oct  4 19:23:06 (none) kernel: CR2: 00000000a000a700
Warning (Oops_read): Code line not seen, dumping what data is available


>>RIP; a000a700 Before first symbol   <=====

>>RAX; ffffffffa00140e0 <[usb-uhci]uhci_pci_remove+20/e0>
>>RBX; ffffffffa0014560 <[usb-uhci]uhci_pci_probe+d0/110>
>>RDX; ffffffffa00140e0 <[usb-uhci]uhci_pci_remove+20/e0>
>>RSI; ffffffffa00140e0 <[usb-uhci]uhci_pci_remove+20/e0>
>>R12; ffffffffa00140e0 <[usb-uhci]uhci_pci_remove+20/e0>

Trace; ffffffff80202fce <pci_announce_device+3e/60>
Trace; ffffffffa0014560 <[usb-uhci]uhci_pci_probe+d0/110>
Trace; ffffffff80203048 <pci_register_driver+58/80>
Trace; ffffffffa0013f4d <[usb-uhci]uhci_interrupt+15d/180>
Trace; ffffffff8012008e <sys_init_module+62e/6f0>
Trace; ffffffff8012e21c <handle_mm_fault+bc/180>
Trace; ffffffffa00100b8 <[usbcore]hcd_data_lock+7a4/7ac>
Trace; ffffffff80189963 <ia32_syscall+67/71>


2 warnings issued.  Results may not be reliable.

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

Andi Kleen wrote:
>
> Can you load the module whatever it is manually and then decode
> the oops while it's still loaded? Or better compile in all USB
> statically and see if it oopses too.

Oops, scratch that last one... It's invalid too as I used the original 
oops rather than the new one.  This one's the right one (honest!).

Tony

[-- Attachment #2: newoops2.txt --]
[-- Type: text/plain, Size: 2945 bytes --]

ksymoops 2.4.9 on x86_64 2.4.23-pre6-amd64.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23-pre6-amd64/ (default)
     -m /boot/System.map-2.4.23-pre6-amd64 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

Unable to handle kernel paging request<1> at 00000000a000a700 RIP: [<00000000a000a700>]PML4 3f145067 PGD 0
Oops: 0010
CPU 0
Pid: 146, comm: modprobe.moduti Not tainted
RIP: 0010:[<00000000a000a700>]
Using defaults from ksymoops -t elf32-i386 -a i386
RSP: 0000:000001003f165de0  EFLAGS: 00010246
RAX: ffffffffa00220e0 RBX: ffffffffa0022560 RCX: 0000000000000007
RDX: ffffffffa00220e0 RSI: ffffffffa00220e0 RDI: 000001000262f800
RBP: 000001000262f800 R08: 0000000000000001 R09: ffffffffa00224e0
R10: ffffffffa0021fa0 R11: 0000000000000000 R12: ffffffffa00220e0
R13: 0000000000000000 R14: ffffff0000a11000 R15: 0000010002767c00
FS:  0000000000000000(0000) GS:ffffffff8034b540(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000a000a700 CR3: 0000000000101000 CR4: 00000000000006e0
Process modprobe.moduti (pid: 146, stackpage=1003f165000)
Stack: 000001003f165de0 0000000000000000 ffffffff80202fce 0000000000000007
       000001000262f800 ffffffffa0022560 0000000000000000 00000000000000b8
       ffffffff80203048 ffffffffffffffea ffffffffa001e000 0000000008088758
Call Trace: [<ffffffff80202fce>] [<ffffffffa0022560>]
       [<ffffffff80203048>] [<ffffffffa0021f4d>] [<ffffffff8012008e>]
       [<ffffffff8012e21c>] [<ffffffffa001e0b8>] [<ffffffff80189963>]
Code:  Bad RIP value.
CR2: 00000000a000a700
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; a000a700 Before first symbol   <=====

>>RAX; ffffffffa00220e0 <[ehci-hcd]pci_ids+0/40>
>>RBX; ffffffffa0022560 <[ehci-hcd]ehci_pci_driver+0/4f>
>>RDX; ffffffffa00220e0 <[ehci-hcd]pci_ids+0/40>
>>RSI; ffffffffa00220e0 <[ehci-hcd]pci_ids+0/40>
>>R09; ffffffffa00224e0 <[ehci-hcd].rodata.end+3b9/419>
>>R10; ffffffffa0021fa0 <[ehci-hcd]__module_kernel_version+0/0>
>>R12; ffffffffa00220e0 <[ehci-hcd]pci_ids+0/40>

Trace; ffffffff80202fce <pci_announce_device+3e/60>
Trace; ffffffffa0022560 <[ehci-hcd]ehci_pci_driver+0/4f>
Trace; ffffffff80203048 <pci_register_driver+58/80>
Trace; ffffffffa0021f4d <[ehci-hcd]init+d/40>
Trace; ffffffff8012008e <sys_init_module+62e/6f0>
Trace; ffffffff8012e21c <handle_mm_fault+bc/180>
Trace; ffffffffa001e0b8 <[keybdev].data.end+919/921>
Trace; ffffffff80189963 <ia32_syscall+67/71>


2 warnings issued.  Results may not be reliable.

Re: Oops linux 2.4.23-pre6 on amd64

[Andi Kleen]

> Oops, scratch that last one... It's invalid too as I used the original 
> oops rather than the new one.  This one's the right one (honest!).

I cannot see anything obviously wrong. At which kernel version did the
problem start? And is your compiler version known to be not buggy ?

-Andi

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

Andi Kleen wrote:
>>Oops, scratch that last one... It's invalid too as I used the original 
>>oops rather than the new one.  This one's the right one (honest!).
> 
> 
> I cannot see anything obviously wrong. At which kernel version did the
> problem start? And is your compiler version known to be not buggy ?

AFAIK there is only one version that supports compiling to amd64 (only 
one for debian anyway) - I'm on 3.3.2-0pre4.biarch1.  It is a bit flaky 
(in 32bit it'll happily do a make -j255 without worrying... going to 64 
bit needs about 10 attempts to do a single compile because it keeps 
falling over with internal compiler errors/segfaults/etc.).

The problem with a brand new architecture is until the toolset 
stabliizes I have to cross my fingers and hope.  Certainly if it's 
compiling dodgy code it'll explain why Alsa is about as stable as a 
drunk in a high wind...

OK I'll forget it for now & go back to 32bit for a couple of months then 
try again.

Tony

Re: Oops linux 2.4.23-pre6 on amd64

[Andi Kleen]

> AFAIK there is only one version that supports compiling to amd64 (only 
> one for debian anyway) - I'm on 3.3.2-0pre4.biarch1.  It is a bit flaky 
> (in 32bit it'll happily do a make -j255 without worrying... going to 64 
> bit needs about 10 attempts to do a single compile because it keeps 
> falling over with internal compiler errors/segfaults/etc.).

That doesn't sound good. Why did you not mention this first, it's unlikely
that such a compiler produces a working kernel.  When the segfaults
are not deterministic (go away when you try again) then you likely
have some hardware problem, like bad DIMMs (run memtest86 for 12+hours to
make sure)

To rule out the compiler you can use the compiler/binutils from

ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/

That's rpms for SuSE 8.1/i386, but I suspect you install it on Debian with
rpm2cpio or somesuch. That's an older gcc 3.2 that is known to work.

Then just put /opt/cross/bin in your $PATH and compile with
CROSS_COMPILE=x86_64-linux- ARCH=x86_64

-Andi

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

Andi Kleen wrote:

> That doesn't sound good. Why did you not mention this first, it's unlikely
> that such a compiler produces a working kernel.  When the segfaults
> are not deterministic (go away when you try again) then you likely
> have some hardware problem, like bad DIMMs (run memtest86 for 12+hours to
> make sure)

It's had 24 hours under memtest86 (I had to RMA one memory stick when I 
first got the machine) and as I mentioned handles a make -j255 in 32bit 
mode without a hitch.  The kernel does work apart from that module (and 
floppy.o which I discovered later does much the same thing as the 
ehci-hcd.o).

> To rule out the compiler you can use the compiler/binutils from
> 
> ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/
> 
> That's rpms for SuSE 8.1/i386, but I suspect you install it on Debian with
> rpm2cpio or somesuch. That's an older gcc 3.2 that is known to work.
> 
> Then just put /opt/cross/bin in your $PATH and compile with
> CROSS_COMPILE=x86_64-linux- ARCH=x86_64
> 
OK I'll try that.

Tony

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

Andi Kleen wrote:

> To rule out the compiler you can use the compiler/binutils from
> 
> ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/
> 
OK I built with that and here are the results:

1. The ehci-hcd driver fails in exactly the same place.
2. It was still v. unstable, which led me to investigate why (since I'm 
pretty sure the hardware is good & the suse compiler is supposed to be a 
good one).  I started stripping out options until eventually I found 
that it's devfs that's the culprit - with that enabled I get random 
compile errors every few seconds.  With it disabled the compile works 
perfectly, even with the debian compiler (tried -j20 and -j255 and both 
passed).

My first guess was you can't use a 32bit devfsd with a 64bit kernel, but 
stopping devfsd didn't seem to make a whole lot of difference to the 
stability... only compiling out the entire devfs system solved it.

I suppose it could be insmod breaking the ehci-hcd... I'll see if I can 
find a pure 64bit one (presumably suse have one) rather than the biarch 
one that debian uses.

Tony

Re: Oops linux 2.4.23-pre6 on amd64

[Andi Kleen]

On Sun, Oct 05, 2003 at 03:29:38PM +0100, Tony Hoyle wrote:
> Andi Kleen wrote:
> 
> >To rule out the compiler you can use the compiler/binutils from
> >
> >ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/
> >
> OK I built with that and here are the results:
> 
> 1. The ehci-hcd driver fails in exactly the same place.
> 2. It was still v. unstable, which led me to investigate why (since I'm 
> pretty sure the hardware is good & the suse compiler is supposed to be a 
> good one).  I started stripping out options until eventually I found 
> that it's devfs that's the culprit - with that enabled I get random 
> compile errors every few seconds.  With it disabled the compile works 
> perfectly, even with the debian compiler (tried -j20 and -j255 and both 
> passed).

Thanks for tracking this down. I would have never noticed
because I don't use devfs.

Marcelo, any ideas? Do you get broken devfs reports for other
64bit architectures too?

AFAIK devfs is unmaintained and I don't really plan to maintain
it myself. My proposal is to just disable it in the configuration
for x86-64 for now.

> My first guess was you can't use a 32bit devfsd with a 64bit kernel, but 
> stopping devfsd didn't seem to make a whole lot of difference to the 
> stability... only compiling out the entire devfs system solved it.

Very likely the devfs code in the kernel is buggy. It is known
to be race hell, I wouldn't be surprised if it has 64bit bugs too.

-Andi

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

Andi Kleen wrote:

> Very likely the devfs code in the kernel is buggy. It is known
> to be race hell, I wouldn't be surprised if it has 64bit bugs too.
> 
OK well I can do without devfs anyway.

The ehci-hcd problem I've eventually tracked down to a bug in the biarch 
modutils - compiling a pure 64bit version of modutils solves all the 
module loading problems, and even alsa started working :)

Tony

Re: Oops linux 2.4.23-pre6 on amd64

[Marcelo Tosatti]

On 5 Oct 2003, Andi Kleen wrote:

> On Sun, Oct 05, 2003 at 03:29:38PM +0100, Tony Hoyle wrote:
> > Andi Kleen wrote:
> > 
> > >To rule out the compiler you can use the compiler/binutils from
> > >
> > >ftp.suse.com:/pub/suse/x86-64/supplementary/CrossTools/8.1-i386/
> > >
> > OK I built with that and here are the results:
> > 
> > 1. The ehci-hcd driver fails in exactly the same place.
> > 2. It was still v. unstable, which led me to investigate why (since I'm 
> > pretty sure the hardware is good & the suse compiler is supposed to be a 
> > good one).  I started stripping out options until eventually I found 
> > that it's devfs that's the culprit - with that enabled I get random 
> > compile errors every few seconds.  With it disabled the compile works 
> > perfectly, even with the debian compiler (tried -j20 and -j255 and both 
> > passed).
> 
> Thanks for tracking this down. I would have never noticed
> because I don't use devfs.
> 
> Marcelo, any ideas? Do you get broken devfs reports for other
> 64bit architectures too?

Nope, never got such reports. 

What problem are you seeing Tony? Oopsing right? Where is the oops output?

> AFAIK devfs is unmaintained and I don't really plan to maintain
> it myself. My proposal is to just disable it in the configuration
> for x86-64 for now.

Nod

> 
> > My first guess was you can't use a 32bit devfsd with a 64bit kernel, but 
> > stopping devfsd didn't seem to make a whole lot of difference to the 
> > stability... only compiling out the entire devfs system solved it.
> 
> Very likely the devfs code in the kernel is buggy. It is known
> to be race hell, I wouldn't be surprised if it has 64bit bugs too.

Yes.

Re: Oops linux 2.4.23-pre6 on amd64

[Tony Hoyle]

Marcelo Tosatti wrote:

> 
> What problem are you seeing Tony? Oopsing right? Where is the oops output?
> 
> 
Random segfaults, internal compiler errors, general instability.  It was 
impossible to compile a kernel without something breaking...  when 
oopses happend they usually happened 3 or 4 in a row.

It's been steady as a rock since I removed devfs...

Tony

calling devinet_ioctl from a kernel module

[Vishwas Raman]

Hello all,

I'm experiencing a kernel panic in RedHat Linux 8.0 with 2.4.20 kernel 
when calling the function devinet_ioctl() with the SIOCSIFADDR command 
from within a kernel module.  The function is called within the scope of
userspace addressing in the following manner.

int res;
mm_segment_t oldfs = get_fs();
set_fs(get_ds());
res = devinet_ioctl(cmd,arg);
set_fs(oldfs);

This is how ipconfig.c illustrates how to use this function from within 
a kernel module, see ic_dev_ioctl() function in the source file.

When running in the scope of a kgdb patched kernel, I see the kernel 
panic when this function is called from within my kernel module. The 
panic happens because alloc_skb is being called non-atomically from an 
interrupt.

Is there anything grossly wrong that I am doing? Am I supposed to do 
something else before calling this function?

Thanks,
Vishwas.