Oops in 2.6.19.1

Oops in 2.6.19.1

[Alistair John Strachan]

Hi,

Any ideas?

BUG: unable to handle kernel NULL pointer dereference at virtual address 
00000009
 printing eip:
c0156f60
*pde = 00000000
Oops: 0002 [#1]
Modules linked in: ipt_recent ipt_REJECT xt_tcpudp ipt_MASQUERADE iptable_nat 
xt_state iptable_filter ip_tables x_tables prism54 yenta_socket 
rsrc_nonstatic pcmcia_core snd_via82xx snd_ac97_codec snd_ac97_bus snd_pcm 
snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd soundcore ehci_hcd 
usblp eth1394 uhci_hcd usbcore ohci1394 ieee1394 via_agp agpgart vt1211 
hwmon_vid hwmon ip_nat_ftp ip_nat ip_conntrack_ftp ip_conntrack
CPU:    0
EIP:    0060:[<c0156f60>]    Not tainted VLI
EFLAGS: 00010246   (2.6.19.1 #1)
EIP is at pipe_poll+0xa0/0xb0
eax: 00000008   ebx: 00000000   ecx: 00000008   edx: 00000000
esi: f70f3e9c   edi: f7017c00   ebp: f70f3c1c   esp: f70f3c0c
ds: 007b   es: 007b   ss: 0068
Process python (pid: 4178, ti=f70f2000 task=f70c4a90 task.ti=f70f2000)
Stack: 00000000 00000000 f70f3e9c f6e111c0 f70f3fa4 c015d7f3 f70f3c54 f70f3fac
       084c44a0 00000030 084c44d0 00000000 f70f3e94 f70f3e94 00000006 f70f3ecc
       00000000 f70f3e94 c015e580 00000000 00000000 00000006 f6e111c0 00000000
Call Trace:
 [<c015d7f3>] do_sys_poll+0x253/0x480
 [<c015da53>] sys_poll+0x33/0x50
 [<c0102c97>] syscall_call+0x7/0xb
 [<b7f6b402>] 0xb7f6b402
 =======================
Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4 89 c8 
8b 75 f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 
45 ca eb b6 8d b6 00 00 00 00 55 b8 01 00 00
EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:f70f3c0c

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Greg KH]

On Wed, Dec 20, 2006 at 02:21:03PM +0000, Alistair John Strachan wrote:
> Hi,
> 
> Any ideas?

Does the problem also happen in 2.6.19?

thanks,

greg k-h

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Wednesday 20 December 2006 16:30, Greg KH wrote:
> On Wed, Dec 20, 2006 at 02:21:03PM +0000, Alistair John Strachan wrote:
> > Hi,
> >
> > Any ideas?
>
> Does the problem also happen in 2.6.19?

No idea. I ran 2.6.19 for a couple of weeks without problems. It took 2 days 
to oops 2.6.19.1, so if it happens again within that time period I guess that 
might be indicative of a -stable patch.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Wednesday 20 December 2006 14:21, Alistair John Strachan wrote:
> Hi,
>
> Any ideas?

Pretty much like clockwork, it happened again. I think it's time to take this 
seriously as a software bug, and not some hardware problem. I've ran kernels 
since 2.6.0 on this machine without such crashes, and now two of the same in 
2.6.19.1? Pretty unlikely!

BUG: unable to handle kernel NULL pointer dereference at virtual address 
00000009
 printing eip:
c0156f60
*pde = 00000000
Oops: 0002 [#1]
Modules linked in: ipt_recent ipt_REJECT xt_tcpudp ipt_MASQUERADE iptable_nat 
xt_sta
te iptable_filter ip_tables x_tables prism54 yenta_socket rsrc_nonstatic 
pcmcia_core snd_via82xx snd_ac97_codec snd_ac97_bus
snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd soundcore 
usblp ehci_hcd eth1394 uhci_hcd usbcore ohci1394 i
eee1394 via_agp agpgart vt1211 hwmon_vid hwmon ip_nat_ftp ip_nat 
ip_conntrack_ftp ip_conntrack
CPU:    0
EIP:    0060:[<c0156f60>]    Not tainted VLI
EFLAGS: 00010246   (2.6.19.1 #1)
EIP is at pipe_poll+0xa0/0xb0
eax: 00000008   ebx: 00000000   ecx: 00000008   edx: 00000000
esi: ee1b9e9c   edi: f4d80a00   ebp: ee1b9c1c   esp: ee1b9c0c
ds: 007b   es: 007b   ss: 0068
Process java (pid: 5374, ti=ee1b8000 task=f7117560 task.ti=ee1b8000)
Stack: 00000000 00000000 ee1b9e9c f6c17160 ee1b9fa4 c015d7f3 ee1b9c54 ee1b9fac
       082dff90 00000010 082dffa0 00000000 ee1b9e94 ee1b9e94 00000002 ee1b9eac
       00000000 ee1b9e94 c015e580 00000000 00000000 00000002 f6c17160 00000000
Call Trace:
 [<c015d7f3>] do_sys_poll+0x253/0x480
 [<c015da53>] sys_poll+0x33/0x50
 [<c0102c97>] syscall_call+0x7/0xb
 [<b7f26402>] 0xb7f26402
 =======================
Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4 89 c8 
8b 75
f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 45 ca 
eb b6 8d b6 00 00 00 00 55 b8 01 00 00
EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:ee1b9c0c

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Zhang, Yanmin]

On Sat, 2006-12-23 at 15:40 +0000, Alistair John Strachan wrote:
> On Wednesday 20 December 2006 14:21, Alistair John Strachan wrote:
> > Hi,
> >
> > Any ideas?
> 
> Pretty much like clockwork, it happened again. I think it's time to take this 
> seriously as a software bug, and not some hardware problem. I've ran kernels 
> since 2.6.0 on this machine without such crashes, and now two of the same in 
> 2.6.19.1? Pretty unlikely!
> 
> BUG: unable to handle kernel NULL pointer dereference at virtual address 
> 00000009
>  printing eip:
> c0156f60
> *pde = 00000000
> Oops: 0002 [#1]
> Modules linked in: ipt_recent ipt_REJECT xt_tcpudp ipt_MASQUERADE iptable_nat 
> xt_sta
> te iptable_filter ip_tables x_tables prism54 yenta_socket rsrc_nonstatic 
> pcmcia_core snd_via82xx snd_ac97_codec snd_ac97_bus
> snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd soundcore 
> usblp ehci_hcd eth1394 uhci_hcd usbcore ohci1394 i
> eee1394 via_agp agpgart vt1211 hwmon_vid hwmon ip_nat_ftp ip_nat 
> ip_conntrack_ftp ip_conntrack
> CPU:    0
> EIP:    0060:[<c0156f60>]    Not tainted VLI
> EFLAGS: 00010246   (2.6.19.1 #1)
> EIP is at pipe_poll+0xa0/0xb0
> eax: 00000008   ebx: 00000000   ecx: 00000008   edx: 00000000
> esi: ee1b9e9c   edi: f4d80a00   ebp: ee1b9c1c   esp: ee1b9c0c
> ds: 007b   es: 007b   ss: 0068
> Process java (pid: 5374, ti=ee1b8000 task=f7117560 task.ti=ee1b8000)
> Stack: 00000000 00000000 ee1b9e9c f6c17160 ee1b9fa4 c015d7f3 ee1b9c54 ee1b9fac
>        082dff90 00000010 082dffa0 00000000 ee1b9e94 ee1b9e94 00000002 ee1b9eac
>        00000000 ee1b9e94 c015e580 00000000 00000000 00000002 f6c17160 00000000
> Call Trace:
>  [<c015d7f3>] do_sys_poll+0x253/0x480
>  [<c015da53>] sys_poll+0x33/0x50
>  [<c0102c97>] syscall_call+0x7/0xb
>  [<b7f26402>] 0xb7f26402
>  =======================
> Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4 89 c8 
> 8b 75
> f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 45 ca 
> eb b6 8d b6 00 00 00 00 55 b8 01 00 00
Above codes look weird. Could you disassemble kernel image and post
the part around address 0xc0156f60?

"87 68 01 00 00" is instruction xchg, but if I disassemble from the begining,
I couldn't see instruct xchg.


> EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:ee1b9c0c
> 

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Wednesday 27 December 2006 02:07, Zhang, Yanmin wrote:
[snip]
> > 00000000 Call Trace:
> >  [<c015d7f3>] do_sys_poll+0x253/0x480
> >  [<c015da53>] sys_poll+0x33/0x50
> >  [<c0102c97>] syscall_call+0x7/0xb
> >  [<b7f26402>] 0xb7f26402
> >  =======================
> > Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4
> > 89 c8 8b 75
> > f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 45
> > ca eb b6 8d b6 00 00 00 00 55 b8 01 00 00
>
> Above codes look weird. Could you disassemble kernel image and post
> the part around address 0xc0156f60?
>
> "87 68 01 00 00" is instruction xchg, but if I disassemble from the
> begining, I couldn't see instruct xchg.
>
> > EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:ee1b9c0c

Unfortunately, after suspecting the toolchain, I did a manual rebuild of 
binutils, gcc and glibc from the official sites, and then rebuilt 2.6.19.1. 
This might upset the decompile below, versus the original report.

Assuming it's NOT a bug in my distro's toolchain (because I am now running the 
GNU stuff), it'll crash again, so this is still useful.

Here's a current decompilation of vmlinux/pipe_poll() from the running kernel, 
the addresses have changed slightly. There's no xchg there either:

c0156ec0 <pipe_poll>:
c0156ec0:       55                      push   %ebp
c0156ec1:       89 e5                   mov    %esp,%ebp
c0156ec3:       83 ec 10                sub    $0x10,%esp
c0156ec6:       89 5d f4                mov    %ebx,0xfffffff4(%ebp)
c0156ec9:       85 d2                   test   %edx,%edx
c0156ecb:       89 d3                   mov    %edx,%ebx
c0156ecd:       89 75 f8                mov    %esi,0xfffffff8(%ebp)
c0156ed0:       89 c6                   mov    %eax,%esi
c0156ed2:       89 7d fc                mov    %edi,0xfffffffc(%ebp)
c0156ed5:       8b 40 08                mov    0x8(%eax),%eax
c0156ed8:       8b 40 08                mov    0x8(%eax),%eax
c0156edb:       8b b8 f0 00 00 00       mov    0xf0(%eax),%edi
c0156ee1:       74 0c                   je     c0156eef <pipe_poll+0x2f>
c0156ee3:       85 ff                   test   %edi,%edi
c0156ee5:       74 08                   je     c0156eef <pipe_poll+0x2f>
c0156ee7:       89 d1                   mov    %edx,%ecx
c0156ee9:       89 f0                   mov    %esi,%eax
c0156eeb:       89 fa                   mov    %edi,%edx
c0156eed:       ff 13                   call   *(%ebx)
c0156eef:       0f b7 5e 1c             movzwl 0x1c(%esi),%ebx
c0156ef3:       31 c9                   xor    %ecx,%ecx
c0156ef5:       8b 47 08                mov    0x8(%edi),%eax
c0156ef8:       f6 c3 01                test   $0x1,%bl
c0156efb:       89 45 f0                mov    %eax,0xfffffff0(%ebp)
c0156efe:       74 20                   je     c0156f20 <pipe_poll+0x60>
c0156f00:       85 c0                   test   %eax,%eax
c0156f02:       b8 41 00 00 00          mov    $0x41,%eax
c0156f07:       0f 4f c8                cmovg  %eax,%ecx
c0156f0a:       8b 87 5c 01 00 00       mov    0x15c(%edi),%eax
c0156f10:       85 c0                   test   %eax,%eax
c0156f12:       74 43                   je     c0156f57 <pipe_poll+0x97>
c0156f14:       8d b6 00 00 00 00       lea    0x0(%esi),%esi
c0156f1a:       8d bf 00 00 00 00       lea    0x0(%edi),%edi
c0156f20:       f6 c3 02                test   $0x2,%bl
c0156f23:       74 23                   je     c0156f48 <pipe_poll+0x88>
c0156f25:       83 7d f0 0f             cmpl   $0xf,0xfffffff0(%ebp)
c0156f29:       b8 04 01 00 00          mov    $0x104,%eax
c0156f2e:       ba 00 00 00 00          mov    $0x0,%edx
c0156f33:       8b 9f 58 01 00 00       mov    0x158(%edi),%ebx
c0156f39:       0f 4f c2                cmovg  %edx,%eax
c0156f3c:       09 c1                   or     %eax,%ecx
c0156f3e:       89 c8                   mov    %ecx,%eax
c0156f40:       83 c8 08                or     $0x8,%eax
c0156f43:       85 db                   test   %ebx,%ebx
c0156f45:       0f 44 c8                cmove  %eax,%ecx
c0156f48:       8b 5d f4                mov    0xfffffff4(%ebp),%ebx
c0156f4b:       89 c8                   mov    %ecx,%eax
c0156f4d:       8b 75 f8                mov    0xfffffff8(%ebp),%esi
c0156f50:       8b 7d fc                mov    0xfffffffc(%ebp),%edi
c0156f53:       89 ec                   mov    %ebp,%esp
c0156f55:       5d                      pop    %ebp
c0156f56:       c3                      ret
c0156f57:       89 ca                   mov    %ecx,%edx
c0156f59:       8b 46 6c                mov    0x6c(%esi),%eax
c0156f5c:       83 ca 10                or     $0x10,%edx
c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax
c0156f65:       0f 45 ca                cmovne %edx,%ecx
c0156f68:       eb b6                   jmp    c0156f20 <pipe_poll+0x60>
c0156f6a:       8d b6 00 00 00 00       lea    0x0(%esi),%esi

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Zhang, Yanmin]

On Wed, 2006-12-27 at 12:35 +0000, Alistair John Strachan wrote:
> On Wednesday 27 December 2006 02:07, Zhang, Yanmin wrote:
> [snip]
> > > 00000000 Call Trace:
> > >  [<c015d7f3>] do_sys_poll+0x253/0x480
> > >  [<c015da53>] sys_poll+0x33/0x50
> > >  [<c0102c97>] syscall_call+0x7/0xb
> > >  [<b7f26402>] 0xb7f26402
> > >  =======================
> > > Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4
> > > 89 c8 8b 75
> > > f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 45
> > > ca eb b6 8d b6 00 00 00 00 55 b8 01 00 00
> >
> > Above codes look weird. Could you disassemble kernel image and post
> > the part around address 0xc0156f60?
> >
> > "87 68 01 00 00" is instruction xchg, but if I disassemble from the
> > begining, I couldn't see instruct xchg.
> >
> > > EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:ee1b9c0c
> 
> Unfortunately, after suspecting the toolchain, I did a manual rebuild of 
> binutils, gcc and glibc from the official sites, and then rebuilt 2.6.19.1. 
> This might upset the decompile below, versus the original report.
> 
> Assuming it's NOT a bug in my distro's toolchain (because I am now running the 
> GNU stuff), it'll crash again, so this is still useful.
> 
> Here's a current decompilation of vmlinux/pipe_poll() from the running kernel, 
> the addresses have changed slightly. There's no xchg there either:
Could you reproduce the bug by the new kernel, so we could get the exact address
and instruction of the bug?

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Thursday 28 December 2006 02:41, Zhang, Yanmin wrote:
[snip]
> > Here's a current decompilation of vmlinux/pipe_poll() from the running
> > kernel, the addresses have changed slightly. There's no xchg there
> > either:
>
> Could you reproduce the bug by the new kernel, so we could get the exact
> address and instruction of the bug?

It crashed again, but this time with no output (machine locked solid). To be 
honest, the disassembly looks right (it's like Chuck said, it's jumping back 
half way through an instruction):

c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax

So c0156f60 is 87 68 01 00 00..

This is with the GCC recompile, so it's not a distro problem. It could still 
either be GCC 4.x, or a 2.6.19.1 specific bug, but it's serious. 2.6.19 with 
GCC 3.4.3 is 100% stable.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Thursday 28 December 2006 04:02, Alistair John Strachan wrote:
> On Thursday 28 December 2006 02:41, Zhang, Yanmin wrote:
> [snip]
>
> > > Here's a current decompilation of vmlinux/pipe_poll() from the running
> > > kernel, the addresses have changed slightly. There's no xchg there
> > > either:
> >
> > Could you reproduce the bug by the new kernel, so we could get the exact
> > address and instruction of the bug?
>
> It crashed again, but this time with no output (machine locked solid). To
> be honest, the disassembly looks right (it's like Chuck said, it's jumping
> back half way through an instruction):
>
> c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax
>
> So c0156f60 is 87 68 01 00 00..
>
> This is with the GCC recompile, so it's not a distro problem. It could
> still either be GCC 4.x, or a 2.6.19.1 specific bug, but it's serious.
> 2.6.19 with GCC 3.4.3 is 100% stable.

Looks like a similar crash here:

http://ubuntuforums.org/showthread.php?p=1803389

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Thursday 28 December 2006 04:14, Alistair John Strachan wrote:
> On Thursday 28 December 2006 04:02, Alistair John Strachan wrote:
> > On Thursday 28 December 2006 02:41, Zhang, Yanmin wrote:
> > [snip]
> >
> > > > Here's a current decompilation of vmlinux/pipe_poll() from the
> > > > running kernel, the addresses have changed slightly. There's no xchg
> > > > there either:
> > >
> > > Could you reproduce the bug by the new kernel, so we could get the
> > > exact address and instruction of the bug?
> >
> > It crashed again, but this time with no output (machine locked solid). To
> > be honest, the disassembly looks right (it's like Chuck said, it's
> > jumping back half way through an instruction):
> >
> > c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax
> >
> > So c0156f60 is 87 68 01 00 00..
> >
> > This is with the GCC recompile, so it's not a distro problem. It could
> > still either be GCC 4.x, or a 2.6.19.1 specific bug, but it's serious.
> > 2.6.19 with GCC 3.4.3 is 100% stable.
>
> Looks like a similar crash here:
>
> http://ubuntuforums.org/showthread.php?p=1803389

I've eliminated 2.6.19.1 as the culprit, and also tried toggling "optimize for 
size", various debug options. 2.6.19 compiled with GCC 4.1.1 on an Via 
Nehemiah C3-2 seems to crash in pipe_poll reliably, within approximately 12 
hours.

The machine passes 6 hours of Prime95 (a CPU stability tester), four memtest86 
passes, and there are no heat problems.

I have compiled GCC 3.4.6 and compiled 2.6.19 with an identical config using 
this compiler (but the same binutils), and will report back if it crashes. My 
bet is that it won't, however.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Saturday 30 December 2006 16:59, Alistair John Strachan wrote:
> I have compiled GCC 3.4.6 and compiled 2.6.19 with an identical config
> using this compiler (but the same binutils), and will report back if it
> crashes. My bet is that it won't, however.

Still fine after >24 hours. Linux 2.6.19, GCC 3.4.6, Binutils 2.17.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Adrian Bunk]

On Sat, Dec 30, 2006 at 04:59:35PM +0000, Alistair John Strachan wrote:
> On Thursday 28 December 2006 04:14, Alistair John Strachan wrote:
> > On Thursday 28 December 2006 04:02, Alistair John Strachan wrote:
> > > On Thursday 28 December 2006 02:41, Zhang, Yanmin wrote:
> > > [snip]
> > >
> > > > > Here's a current decompilation of vmlinux/pipe_poll() from the
> > > > > running kernel, the addresses have changed slightly. There's no xchg
> > > > > there either:
> > > >
> > > > Could you reproduce the bug by the new kernel, so we could get the
> > > > exact address and instruction of the bug?
> > >
> > > It crashed again, but this time with no output (machine locked solid). To
> > > be honest, the disassembly looks right (it's like Chuck said, it's
> > > jumping back half way through an instruction):
> > >
> > > c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax
> > >
> > > So c0156f60 is 87 68 01 00 00..
> > >
> > > This is with the GCC recompile, so it's not a distro problem. It could
> > > still either be GCC 4.x, or a 2.6.19.1 specific bug, but it's serious.
> > > 2.6.19 with GCC 3.4.3 is 100% stable.
> >
> > Looks like a similar crash here:
> >
> > http://ubuntuforums.org/showthread.php?p=1803389
> 
> I've eliminated 2.6.19.1 as the culprit, and also tried toggling "optimize for 
> size", various debug options. 2.6.19 compiled with GCC 4.1.1 on an Via 
> Nehemiah C3-2 seems to crash in pipe_poll reliably, within approximately 12 
> hours.
> 
> The machine passes 6 hours of Prime95 (a CPU stability tester), four memtest86 
> passes, and there are no heat problems.
> 
> I have compiled GCC 3.4.6 and compiled 2.6.19 with an identical config using 
> this compiler (but the same binutils), and will report back if it crashes. My 
> bet is that it won't, however.

There are occasional reports of problems with kernels compiled with 
gcc 4.1 that vanish when using older versions of gcc.

AFAIK, until now noone has ever debugged whether that's a gcc bug, 
gcc exposing a kernel bug or gcc exposing a hardware bug.

Comparing your report and [1], it seems that if these are the same 
problem, it's not a hardware bug but a gcc or kernel bug.

> Cheers,
> Alistair.

cu
Adrian

[1] http://bugzilla.kernel.org/show_bug.cgi?id=7176

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Sunday 31 December 2006 16:27, Adrian Bunk wrote:
> On Sat, Dec 30, 2006 at 04:59:35PM +0000, Alistair John Strachan wrote:
> > On Thursday 28 December 2006 04:14, Alistair John Strachan wrote:
> > > On Thursday 28 December 2006 04:02, Alistair John Strachan wrote:
> > > > On Thursday 28 December 2006 02:41, Zhang, Yanmin wrote:
> > > > [snip]
> > > >
> > > > > > Here's a current decompilation of vmlinux/pipe_poll() from the
> > > > > > running kernel, the addresses have changed slightly. There's no
> > > > > > xchg there either:
> > > > >
> > > > > Could you reproduce the bug by the new kernel, so we could get the
> > > > > exact address and instruction of the bug?
> > > >
> > > > It crashed again, but this time with no output (machine locked
> > > > solid). To be honest, the disassembly looks right (it's like Chuck
> > > > said, it's jumping back half way through an instruction):
> > > >
> > > > c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax
> > > >
> > > > So c0156f60 is 87 68 01 00 00..
> > > >
> > > > This is with the GCC recompile, so it's not a distro problem. It
> > > > could still either be GCC 4.x, or a 2.6.19.1 specific bug, but it's
> > > > serious. 2.6.19 with GCC 3.4.3 is 100% stable.
> > >
> > > Looks like a similar crash here:
> > >
> > > http://ubuntuforums.org/showthread.php?p=1803389
> >
> > I've eliminated 2.6.19.1 as the culprit, and also tried toggling
> > "optimize for size", various debug options. 2.6.19 compiled with GCC
> > 4.1.1 on an Via Nehemiah C3-2 seems to crash in pipe_poll reliably,
> > within approximately 12 hours.
> >
> > The machine passes 6 hours of Prime95 (a CPU stability tester), four
> > memtest86 passes, and there are no heat problems.
> >
> > I have compiled GCC 3.4.6 and compiled 2.6.19 with an identical config
> > using this compiler (but the same binutils), and will report back if it
> > crashes. My bet is that it won't, however.
>
> There are occasional reports of problems with kernels compiled with
> gcc 4.1 that vanish when using older versions of gcc.
>
> AFAIK, until now noone has ever debugged whether that's a gcc bug,
> gcc exposing a kernel bug or gcc exposing a hardware bug.
>
> Comparing your report and [1], it seems that if these are the same
> problem, it's not a hardware bug but a gcc or kernel bug.

This bug specifically indicates some kind of miscompilation in a driver, 
causing boot time hangs. My problem is quite different, and more subtle. The 
crash happens in the same place every time, which does suggest determinism 
(even with various options toggled on and off, and a 300K smaller kernel 
image), but it takes 8-12 hours to manifest and only happens with GCC 4.1.1.

Unless we can start narrowing this down, it would be a mammoth task to seek 
out either the kernel or GCC change that first exhibited this bug, due to the 
non-immediate reproducibility of the bug, the lack of clues, and this 
machine's role as a stable, high-availability server.

(If I had another Epia M10000 or another computer I could reproduce the bug 
on, I would be only too happy to boot as many kernels as required to fix it; 
however I cannot spare this machine).

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

kernel + gcc 4.1 = several problems

[Adrian Bunk]

On Sun, Dec 31, 2006 at 04:55:51PM +0000, Alistair John Strachan wrote:
> On Sunday 31 December 2006 16:27, Adrian Bunk wrote:
> > On Sat, Dec 30, 2006 at 04:59:35PM +0000, Alistair John Strachan wrote:
> > > On Thursday 28 December 2006 04:14, Alistair John Strachan wrote:
> > > > On Thursday 28 December 2006 04:02, Alistair John Strachan wrote:
> > > > > On Thursday 28 December 2006 02:41, Zhang, Yanmin wrote:
> > > > > [snip]
> > > > >
> > > > > > > Here's a current decompilation of vmlinux/pipe_poll() from the
> > > > > > > running kernel, the addresses have changed slightly. There's no
> > > > > > > xchg there either:
> > > > > >
> > > > > > Could you reproduce the bug by the new kernel, so we could get the
> > > > > > exact address and instruction of the bug?
> > > > >
> > > > > It crashed again, but this time with no output (machine locked
> > > > > solid). To be honest, the disassembly looks right (it's like Chuck
> > > > > said, it's jumping back half way through an instruction):
> > > > >
> > > > > c0156f5f:       3b 87 68 01 00 00       cmp    0x168(%edi),%eax
> > > > >
> > > > > So c0156f60 is 87 68 01 00 00..
> > > > >
> > > > > This is with the GCC recompile, so it's not a distro problem. It
> > > > > could still either be GCC 4.x, or a 2.6.19.1 specific bug, but it's
> > > > > serious. 2.6.19 with GCC 3.4.3 is 100% stable.
> > > >
> > > > Looks like a similar crash here:
> > > >
> > > > http://ubuntuforums.org/showthread.php?p=1803389
> > >
> > > I've eliminated 2.6.19.1 as the culprit, and also tried toggling
> > > "optimize for size", various debug options. 2.6.19 compiled with GCC
> > > 4.1.1 on an Via Nehemiah C3-2 seems to crash in pipe_poll reliably,
> > > within approximately 12 hours.
> > >
> > > The machine passes 6 hours of Prime95 (a CPU stability tester), four
> > > memtest86 passes, and there are no heat problems.
> > >
> > > I have compiled GCC 3.4.6 and compiled 2.6.19 with an identical config
> > > using this compiler (but the same binutils), and will report back if it
> > > crashes. My bet is that it won't, however.
> >
> > There are occasional reports of problems with kernels compiled with
> > gcc 4.1 that vanish when using older versions of gcc.
> >
> > AFAIK, until now noone has ever debugged whether that's a gcc bug,
> > gcc exposing a kernel bug or gcc exposing a hardware bug.
> >
> > Comparing your report and [1], it seems that if these are the same
> > problem, it's not a hardware bug but a gcc or kernel bug.
> 
> This bug specifically indicates some kind of miscompilation in a driver, 
> causing boot time hangs. My problem is quite different, and more subtle. The 
> crash happens in the same place every time, which does suggest determinism 
> (even with various options toggled on and off, and a 300K smaller kernel 
> image), but it takes 8-12 hours to manifest and only happens with GCC 4.1.1.
>...

Sorry if my point goes a bit away from your problem:

My point is that we have several reported problems only visible
with gcc 4.1.

Other bug reports are e.g. [2] and [3], but they are only present with
using gcc 4.1 _and_ using -Os.

There's simply a bunch of bugs only present with gcc 4.1, and what 
worries me most is that the estimated number of unknown cases is most 
likely very high since most people won't check different compiler 
versions when running into a problem.

> Cheers,
> Alistair.

cu
Adrian

[1] http://bugzilla.kernel.org/show_bug.cgi?id=7176
[2] http://bugzilla.kernel.org/show_bug.cgi?id=7106
[3] https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=186852

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: kernel + gcc 4.1 = several problems

[Alistair John Strachan]

On Tuesday 02 January 2007 21:10, Adrian Bunk wrote:
[snip]
> > > Comparing your report and [1], it seems that if these are the same
> > > problem, it's not a hardware bug but a gcc or kernel bug.
> >
> > This bug specifically indicates some kind of miscompilation in a driver,
> > causing boot time hangs. My problem is quite different, and more subtle.
> > The crash happens in the same place every time, which does suggest
> > determinism (even with various options toggled on and off, and a 300K
> > smaller kernel image), but it takes 8-12 hours to manifest and only
> > happens with GCC 4.1.1. ...
>
> Sorry if my point goes a bit away from your problem:
>
> My point is that we have several reported problems only visible
> with gcc 4.1.
>
> Other bug reports are e.g. [2] and [3], but they are only present with
> using gcc 4.1 _and_ using -Os.

I find [2] most compelling, and I can confirm that I do have the same problem 
with or without optimisation for size. I don't use selinux nor has it ever 
been enabled.

At any rate, I have absolute confirmation that it is GCC 4.1.1, because with 
GCC 3.4.6 the same kernel I reported booting three days ago is still 
cheerfully working. I regularly get uptimes of 60+ days on that machine, 
rebooting only for kernel upgrades. 2.6.19 seems to be no worse in this 
regard.

Perhaps fortunately, the configs I've tried have consistently failed to shake 
the crash, so I have a semi-reproducible test case here on C3-2 hardware if 
somebody wants to investigate the problem (though it still takes 6-12 hours).

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: kernel + gcc 4.1 = several problems

[D. Hazelton]

On Tuesday 02 January 2007 16:56, Alistair John Strachan wrote:
> On Tuesday 02 January 2007 21:10, Adrian Bunk wrote:
> [snip]
>
> > > > Comparing your report and [1], it seems that if these are the same
> > > > problem, it's not a hardware bug but a gcc or kernel bug.
> > >
> > > This bug specifically indicates some kind of miscompilation in a
> > > driver, causing boot time hangs. My problem is quite different, and
> > > more subtle. The crash happens in the same place every time, which does
> > > suggest determinism (even with various options toggled on and off, and
> > > a 300K smaller kernel image), but it takes 8-12 hours to manifest and
> > > only happens with GCC 4.1.1. ...
> >
> > Sorry if my point goes a bit away from your problem:
> >
> > My point is that we have several reported problems only visible
> > with gcc 4.1.
> >
> > Other bug reports are e.g. [2] and [3], but they are only present with
> > using gcc 4.1 _and_ using -Os.
>
> I find [2] most compelling, and I can confirm that I do have the same
> problem with or without optimisation for size. I don't use selinux nor has
> it ever been enabled.
>
> At any rate, I have absolute confirmation that it is GCC 4.1.1, because
> with GCC 3.4.6 the same kernel I reported booting three days ago is still
> cheerfully working. I regularly get uptimes of 60+ days on that machine,
> rebooting only for kernel upgrades. 2.6.19 seems to be no worse in this
> regard.
>
> Perhaps fortunately, the configs I've tried have consistently failed to
> shake the crash, so I have a semi-reproducible test case here on C3-2
> hardware if somebody wants to investigate the problem (though it still
> takes 6-12 hours).

The GCC code generator appears to have been rewritten between 3.4.6 and 
4.1.1....

I took a look at the dump he posted and there are some minor and some massive 
differences between the code. In one case some of the code is swapped, in 
another there is code in the 3.4.6 version that isn't in the 4.1.1... Finally 
the 4.1.1 version of the function has what appears to be function calls and 
these don't appear in the code generated by 3.4.6

In other words - the code generation for 4.1.1 appears to be broken when it 
comes to generating system code.

DRH

Re: kernel + gcc 4.1 = several problems

[Adrian Bunk]

On Tue, Jan 02, 2007 at 05:06:14PM -0500, D. Hazelton wrote:
> On Tuesday 02 January 2007 16:56, Alistair John Strachan wrote:
> > On Tuesday 02 January 2007 21:10, Adrian Bunk wrote:
> > [snip]
> >
> > > > > Comparing your report and [1], it seems that if these are the same
> > > > > problem, it's not a hardware bug but a gcc or kernel bug.
> > > >
> > > > This bug specifically indicates some kind of miscompilation in a
> > > > driver, causing boot time hangs. My problem is quite different, and
> > > > more subtle. The crash happens in the same place every time, which does
> > > > suggest determinism (even with various options toggled on and off, and
> > > > a 300K smaller kernel image), but it takes 8-12 hours to manifest and
> > > > only happens with GCC 4.1.1. ...
> > >
> > > Sorry if my point goes a bit away from your problem:
> > >
> > > My point is that we have several reported problems only visible
> > > with gcc 4.1.
> > >
> > > Other bug reports are e.g. [2] and [3], but they are only present with
> > > using gcc 4.1 _and_ using -Os.
> >
> > I find [2] most compelling, and I can confirm that I do have the same
> > problem with or without optimisation for size. I don't use selinux nor has
> > it ever been enabled.
> >
> > At any rate, I have absolute confirmation that it is GCC 4.1.1, because
> > with GCC 3.4.6 the same kernel I reported booting three days ago is still
> > cheerfully working. I regularly get uptimes of 60+ days on that machine,
> > rebooting only for kernel upgrades. 2.6.19 seems to be no worse in this
> > regard.
> >
> > Perhaps fortunately, the configs I've tried have consistently failed to
> > shake the crash, so I have a semi-reproducible test case here on C3-2
> > hardware if somebody wants to investigate the problem (though it still
> > takes 6-12 hours).
> 
> The GCC code generator appears to have been rewritten between 3.4.6 and 
> 4.1.1....
> 
> I took a look at the dump he posted and there are some minor and some massive 
> differences between the code. In one case some of the code is swapped, in 
> another there is code in the 3.4.6 version that isn't in the 4.1.1... Finally 
> the 4.1.1 version of the function has what appears to be function calls and 
> these don't appear in the code generated by 3.4.6

Differences are expected since we disable unit-at-a-time for gcc < 4 
and gcc development didn't stall between 3.4 and 4.1.

> In other words - the code generation for 4.1.1 appears to be broken when it 
> comes to generating system code.

Bug number for an either already open or created by you bug in the gcc 
Bugzilla for what you claim to be a bug in gcc?

> DRH

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: kernel + gcc 4.1 = several problems

[D. Hazelton]

On Tuesday 02 January 2007 18:24, you wrote:
> On Tue, Jan 02, 2007 at 05:06:14PM -0500, D. Hazelton wrote:
> > On Tuesday 02 January 2007 16:56, Alistair John Strachan wrote:
> > > On Tuesday 02 January 2007 21:10, Adrian Bunk wrote:
> > > [snip]
> > >
> > > > > > Comparing your report and [1], it seems that if these are the
> > > > > > same problem, it's not a hardware bug but a gcc or kernel bug.
> > > > >
> > > > > This bug specifically indicates some kind of miscompilation in a
> > > > > driver, causing boot time hangs. My problem is quite different, and
> > > > > more subtle. The crash happens in the same place every time, which
> > > > > does suggest determinism (even with various options toggled on and
> > > > > off, and a 300K smaller kernel image), but it takes 8-12 hours to
> > > > > manifest and only happens with GCC 4.1.1. ...
> > > >
> > > > Sorry if my point goes a bit away from your problem:
> > > >
> > > > My point is that we have several reported problems only visible
> > > > with gcc 4.1.
> > > >
> > > > Other bug reports are e.g. [2] and [3], but they are only present
> > > > with using gcc 4.1 _and_ using -Os.
> > >
> > > I find [2] most compelling, and I can confirm that I do have the same
> > > problem with or without optimisation for size. I don't use selinux nor
> > > has it ever been enabled.
> > >
> > > At any rate, I have absolute confirmation that it is GCC 4.1.1, because
> > > with GCC 3.4.6 the same kernel I reported booting three days ago is
> > > still cheerfully working. I regularly get uptimes of 60+ days on that
> > > machine, rebooting only for kernel upgrades. 2.6.19 seems to be no
> > > worse in this regard.
> > >
> > > Perhaps fortunately, the configs I've tried have consistently failed to
> > > shake the crash, so I have a semi-reproducible test case here on C3-2
> > > hardware if somebody wants to investigate the problem (though it still
> > > takes 6-12 hours).
> >
> > The GCC code generator appears to have been rewritten between 3.4.6 and
> > 4.1.1....
> >
> > I took a look at the dump he posted and there are some minor and some
> > massive differences between the code. In one case some of the code is
> > swapped, in another there is code in the 3.4.6 version that isn't in the
> > 4.1.1... Finally the 4.1.1 version of the function has what appears to be
> > function calls and these don't appear in the code generated by 3.4.6
>
> Differences are expected since we disable unit-at-a-time for gcc < 4
> and gcc development didn't stall between 3.4 and 4.1.

Okay. Thing is that these noted differences, aside from where 4.1.1 doesn't 
generate an opcode that 3.4.6 does aren't all that fatal, IMHO. The fact that 
there it does generate call's rather than jumps for local pointer moves 
(IIRC - been a while since I looked at the dump of pipe_poll that he 
provided) might be part of the problem

> > In other words - the code generation for 4.1.1 appears to be broken when
> > it comes to generating system code.
>
> Bug number for an either already open or created by you bug in the gcc
> Bugzilla for what you claim to be a bug in gcc?

None. I didn't file a report on this because I didn't find the big, just noted 
a problem that appears to occur. In this case the call's generated seem to 
wrap loops - something I've never heard of anyone doing. These *might* be 
causing the off-by-one that is causing the function to re-enter in the middle 
of an instruction.

Seeing this I'd guess that this follows for all system-level code generated by 
4.1.1 and this is exactly what I was reporting. If you'd like I'll go dig up 
the dumps he posted and post the two related segments side-by-side to give 
you a better example what I'm referring to.

DRH

Re: kernel + gcc 4.1 = several problems

[Horst H. von Brand]

D. Hazelton <dhazelton@enter.net> wrote:

[...]

> None. I didn't file a report on this because I didn't find the big, just
> noted a problem that appears to occur. In this case the call's generated
> seem to wrap loops - something I've never heard of anyone doing.

Example code showing this weirdness?

>                                                                  These
> *might* be causing the off-by-one that is causing the function to
> re-enter in the middle of an instruction.

If something like this happened, programs would be crashing left and right.

> Seeing this I'd guess that this follows for all system-level code
> generated by 4.1.1

Define "system-level code". What makes it different from, say,
bog-of-the-mill compiler code (yes, gcc compiles itself as part of its
sanity checking)?

>                    and this is exactly what I was reporting. If you'd
> like I'll go dig up the dumps he posted and post the two related segments
> side-by-side to give you a better example what I'm referring to.

If the related segments show code that is somehow wrong, by all means
report it /with your detailed analysis/ to the compiler people. Just a
warning, gcc is pretty smart in what it does, its code is often surprising
to the unwashed. Also, the C standard is subtle, the error might be in a
unwarranted assumption in the source code.

Re: kernel + gcc 4.1 = several problems

[Linus Torvalds]

On Tue, 2 Jan 2007, Alistair John Strachan wrote:
> 
> At any rate, I have absolute confirmation that it is GCC 4.1.1, because with 
> GCC 3.4.6 the same kernel I reported booting three days ago is still 
> cheerfully working. I regularly get uptimes of 60+ days on that machine, 
> rebooting only for kernel upgrades. 2.6.19 seems to be no worse in this 
> regard.
> 
> Perhaps fortunately, the configs I've tried have consistently failed to shake 
> the crash, so I have a semi-reproducible test case here on C3-2 hardware if 
> somebody wants to investigate the problem (though it still takes 6-12 hours).

Historically, some people have actually used horrible hacks like trying to 
figure out which particular C file gets miscompiled by basically having 
both compilers installed, and then trying out different subdirectories 
with different compilers. And once the subdirectory has been pinpointed, 
pinpointing which particular file it is.. etc.

Pretty damn horrible to do, and I'm afraid we don't have any real helpful 
scripts to do any of the work for you. So it's all effectively manual 
(basically boils down to: "compile everything with known-good compiler. 
Then replace the good compiler with the bad one, remove the object files 
from one directory, and recompile the kernel". "Rinse and repeat".

I don't think anybody has ever done that with something where triggering 
the cause then also takes that long - that just ends up making the whole 
thing even more painful. 

What are the exact crash details? That might narrow things down enough 
that maybe you could try just one or two files that are "suspect".

		Linus

Re: kernel + gcc 4.1 = several problems

[Alistair John Strachan]

Linus,

On Tuesday 02 January 2007 22:13, Linus Torvalds wrote:
[snip]
> What are the exact crash details? That might narrow things down enough
> that maybe you could try just one or two files that are "suspect".

I'll do a digest of the problem for you and anybody else that's lost track of 
the debugging story so far..

There are no hardware problems evidenced by any testing I have performed 
(memtest, prime95 CPU torture tests, temp monitors). Furthermore, kernels 
compiled with older GCCs have been running without problems for literally 
years on this machine.

Here is an example of an oops. The kernel continued to limp along after this.

BUG: unable to handle kernel NULL pointer dereference at virtual address 
00000009
 printing eip:
c0156f60
*pde = 00000000
Oops: 0002 [#1]
Modules linked in: ipt_recent ipt_REJECT xt_tcpudp ipt_MASQUERADE iptable_nat 
xt_state iptable_filter ip_tables x_tables prism54 yenta_socket 
rsrc_nonstatic pcmcia_core snd_via82xx snd_ac97_codec snd_ac97_bus snd_pcm 
snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd soundcore ehci_hcd 
usblp eth1394 uhci_hcd usbcore ohci1394 ieee1394 via_agp agpgart vt1211 
hwmon_vid hwmon ip_nat_ftp ip_nat ip_conntrack_ftp ip_conntrack
CPU:    0
EIP:    0060:[<c0156f60>]    Not tainted VLI
EFLAGS: 00010246   (2.6.19.1 #1)
EIP is at pipe_poll+0xa0/0xb0
eax: 00000008   ebx: 00000000   ecx: 00000008   edx: 00000000
esi: f70f3e9c   edi: f7017c00   ebp: f70f3c1c   esp: f70f3c0c
ds: 007b   es: 007b   ss: 0068
Process python (pid: 4178, ti=f70f2000 task=f70c4a90 task.ti=f70f2000)
Stack: 00000000 00000000 f70f3e9c f6e111c0 f70f3fa4 c015d7f3 f70f3c54 f70f3fac
       084c44a0 00000030 084c44d0 00000000 f70f3e94 f70f3e94 00000006 f70f3ecc
       00000000 f70f3e94 c015e580 00000000 00000000 00000006 f6e111c0 00000000
Call Trace:
 [<c015d7f3>] do_sys_poll+0x253/0x480
 [<c015da53>] sys_poll+0x33/0x50
 [<c0102c97>] syscall_call+0x7/0xb
 [<b7f6b402>] 0xb7f6b402
 =======================
Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4 89 c8 
8b 75 f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 
45 ca eb b6 8d b6 00 00 00 00 55 b8 01 00 00
EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:f70f3c0c

Chuck observed that the kernel tries to reenter pipe_poll half way through an 
instruction (c0156f5f->c0156f60); it's not a single-bit error but an 
off-by-one.

On Wednesday 20 December 2006 20:48, Chuck Ebbert wrote:
> In-Reply-To: <200612201421.03514.s0348365@sms.ed.ac.uk>
>
> On Wed, 20 Dec 2006 14:21:03 +0000, Alistair John Strachan wrote:
> > Any ideas?
> >
> > BUG: unable to handle kernel NULL pointer dereference at virtual address
> > 00000009
>
>     83 ca 10                  or     $0x10,%edx
>     3b                        .byte 0x3b
>     87 68 01                  xchg   %ebp,0x1(%eax)   <=====
>     00 00                     add    %al,(%eax)
>
> Somehow it is trying to execute code in the middle of an instruction.
> That almost never works, even when the resulting fragment is a legal
> opcode. :)
>
> The real instruction is:
>
>     3b 87 68 01 00 00 00        cmp    0x168(%edi),%eax

I've tried a multitude of kernel configs and compiler options, but none have 
made any difference. That first oops was pretty lucky, very often the machine 
locks up after oopsing (panic_on_oops=1 doesn't work). I've not seen oopses 
anywhere but in pipe_poll, but I've not seen many oopses.

The machine runs jabberd 2.x which uses separate python processes as 
transports to different networks. The server hosts 50-100 users. One of my 
oops reports had Java crashing in the same place, that's Azureus.

I've got binutils 2.17, gcc 4.1.1 hand bootstrapped from GNU sources (not 
distro versions). I've got another, secondary compiler (3.4.6), also compiled 
from GNU sources, installed elsewhere which I have used to build working 
kernels. So the only variable, for sure, is GCC itself.

Both compilers were built with "make bootstrap" and I built binutils with the 
resulting GCC, and GCC with the resulting binutils, just to be sure. The only 
slightly non-standard thing I do is to compile everything (GCC, binutils, the 
kernels) on a dual-opteron box, inside a 32bit chroot, which is rsync'ed over 
to the Via C3-2 box with the problem. I can't see how this would cause any 
problems (and indeed have done it successfully for years), but I thought I'd 
point it out.

The crashes take time to appear, which is why so many people suspected 
hardware initially. But the uptime of a GCC 4.1.1 kernel will always be less 
than 12 hours, where a 3.4.6 kernel will survive for months. I've had no 
other mysterious software crashes, ever.

On Sunday 31 December 2006 22:16, Alistair John Strachan wrote:
> On Sunday 31 December 2006 21:43, Chuck Ebbert wrote:
> > Those were compiled without frame pointers.  Can you post them compiled
> > with frame pointers so they match your original bug report? And confirm
> > that pipe_poll() is still at 0xc0156ec0 in vmlinux?
>
> c0156ec0 <pipe_poll>:
>
> I used the config I original sent you to rebuild it again. This time I've
> put up the whole vmlinux for both kernels, the config is replaced, the
> decompilation is re-done, I've confirmed the offset in the GCC 4.1.1 kernel
> is identical. Sorry for the confusion.
[snip]
> http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/

At the above URL can be found vmlinux images, the config used to build both, 
and decompilations of the fs/pipe.o file (with relocation information).

The suggestions I've had so far which I have not yet tried:

-	Select a different x86 CPU in the config.
		-	Unfortunately the C3-2 flags seem to simply tell GCC
			to schedule for ppro (like i686) and enabled MMX and SSE
		-	Probably useless

-	Enable as many debug options as possible ("a shot in the dark")

-	Try compiling a minimal kernel config, sans modules that are not required
	for booting. The problem with this one (whilst it might uncover some bizarre
	memory scribbling or stack corruption) is that the machine's primary role is
	that of a router, so I require most of the modules loaded for the oops to be
	reproduced (chicken, egg?).

If I can provide any more information, please do let me know.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: kernel + gcc 4.1 = several problems

[Linus Torvalds]

On Tue, 2 Jan 2007, Alistair John Strachan wrote:
>
> eax: 00000008   ebx: 00000000   ecx: 00000008   edx: 00000000
> esi: f70f3e9c   edi: f7017c00   ebp: f70f3c1c   esp: f70f3c0c
>
> Code: 58 01 00 00 0f 4f c2 09 c1 89 c8 83 c8 08 85 db 0f 44 c8 8b 5d f4 89 c8 
> 8b 75 f8 8b 7d fc 89 ec 5d c3 89 ca 8b 46 6c 83 ca 10 3b <87> 68 01 00 00 0f 
> 45 ca eb b6 8d b6 00 00 00 00 55 b8 01 00 00
> EIP: [<c0156f60>] pipe_poll+0xa0/0xb0 SS:ESP 0068:f70f3c0c
> 
> Chuck observed that the kernel tries to reenter pipe_poll half way through an 
> instruction (c0156f5f->c0156f60); it's not a single-bit error but an 
> off-by-one.

It's not an off-by-one either (eg say we're taking an exception and 
screiwing up %eip by one somehow).

The code sequence in question is

	mov    %ecx,%edx
	mov    0x6c(%esi),%eax
	or     $0x10,%edx
	cmp    0x168(%edi),%eax		<--
	cmovne %edx,%ecx
	jmp    ...

and it's in the second byte of the "cmp".

And yes, it definitely entered there, because trying other random 
entry-points will have either invalid instructions or instructions that 
would fault due to NULL pointers. HOWEVER, it's also not as simple as 
"took an interrupt, and returned with %eip incremented by one", becasue 
your %edx is zero, so it won't have done that "or $10,%edx" and then some 
interrupt happened and screwed up just %eip.

So it's literally a random %eip, but since you say it's consistently in 
that function, it's not truly "random". There's something that triggers it 
just _there_.

However, that's a damn simple function. There's _nothing_ there. The 
particular code that is involved right there is literally

	if (!pipe->writers && filp->f_version != pipe->w_counter)
		mask |= POLLHUP;

and that's it.  There's not even anything half-way interesting around it, 
except for the "poll_wait()" call, but even that is about as common as
you can humanly get..

Looking at the register set and the stack, I see:

	Stack:	00000000
		00000000  <- saved %ebx (dunno, seems dead in caller)
		f70f3e9c  <- saved %esi (== pollfd in do_pollfd)
		f6e111c0  <- saved %edi	(== filp)
		f70f3fa4  <- outer EBP (looks reasonable) 
		c015d7f3  <- return address (do_sys_poll+0x253/0x480)

and the strange thing is that when the oops happens, it really looks like 
%esi _still_ contains the value it had originally (and that is saved on 
the stack). But afaik, from your disassembly, it should have been 
overwritten by the initial %eax, which should have had the same value as 
%edi on entry...

IOW, none of it really makes any sense. The stack frames look fine, so we 
_did_ enter at the beginning of the function (and it wasn't the *poll fn 
pointer that was corrupt.

> The suggestions I've had so far which I have not yet tried:
> 
> -	Select a different x86 CPU in the config.
> 		-	Unfortunately the C3-2 flags seem to simply tell GCC
> 			to schedule for ppro (like i686) and enabled MMX and SSE
> 		-	Probably useless

Actually, try this one. Try using something that doesn't like "cmov". 
Maybe the C3-2 simply has some internal cmov bugginess. 

		Linus

Re: kernel + gcc 4.1 = several problems

[Linus Torvalds]

On Tue, 2 Jan 2007, Adrian Bunk wrote:
> 
> My point is that we have several reported problems only visible
> with gcc 4.1.
> 
> Other bug reports are e.g. [2] and [3], but they are only present with
> using gcc 4.1 _and_ using -Os.

Traditionally, afaik, -Os has tended to show compiler problems that 
_could_ happen with -O2 too, but never do in practice. It may be that 
gcc-4.1 without -Os miscompiles some very unusual code, and then with -Os 
we just hit more cases of that.

That said, I th ink gcc-4.1.1 is very common - I know it's the Fedora 
compiler. Also, CC_OPTIMIZE_FOR_SIZE defaults to 'y' if you have 
EXPERIMENTAL on, and from all the bug-reports about other features that 
are marked EXPERIMENTAL, I know that a lot of people do seem to select for 
it. So I would expect that gcc-4.1.1 and -Os is actually a fairly common 
combination. I just checked, and it's what I use personally, for example.

Of course, my main machine is an x86-64, and it has more registers. At 
least some historical -Os bug was about bad things happening under 
register pressure, iirc, and so x86-64 would show fewer problems than 
regular 32-bit x86 (which has far fewer registers for the compiler to 
use).

It is a bit worrisome. These things seem to be about 50:50 real kernel 
bugs (just hidden by some common code generation sequence) and real 
honest-to-goodness compiler bugs. But they are hard as hell to find.

		Linus

Re: kernel + gcc 4.1 = several problems

[David Rientjes]

On Tue, 2 Jan 2007, Linus Torvalds wrote:

> Traditionally, afaik, -Os has tended to show compiler problems that 
> _could_ happen with -O2 too, but never do in practice. It may be that 
> gcc-4.1 without -Os miscompiles some very unusual code, and then with -Os 
> we just hit more cases of that.
> 

gcc optimizations were almost completely rewritten between 3.4.6 and 4.1, 
and one of the subtle changes that may have been introduced is with regard 
to the heuristics used to determine whether to inline an 'inline' function 
or not when using -Os.  This problem can show up in dynamic linking and 
break on certain architectures but should be detectable by using -Winline.

		David

Re: Oops in 2.6.19.1

[Chuck Ebbert]

In-Reply-To: <200612201421.03514.s0348365@sms.ed.ac.uk>

On Wed, 20 Dec 2006 14:21:03 +0000, Alistair John Strachan wrote:

> Any ideas?
> 
> BUG: unable to handle kernel NULL pointer dereference at virtual address 
> 00000009

    83 ca 10                  or     $0x10,%edx
    3b                        .byte 0x3b
    87 68 01                  xchg   %ebp,0x1(%eax)   <=====
    00 00                     add    %al,(%eax)

Somehow it is trying to execute code in the middle of an instruction.
That almost never works, even when the resulting fragment is a legal
opcode. :)

The real instruction is:

    3b 87 68 01 00 00 00        cmp    0x168(%edi),%eax

I'd guess you have some kind of hardware problem.  It could also be
a kernel problem where the saved address was corrupted during an
interrupt, but that's not likely.
-- 
MBTI: IXTP

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Wednesday 20 December 2006 20:48, Chuck Ebbert wrote:
[snip]
> I'd guess you have some kind of hardware problem.  It could also be
> a kernel problem where the saved address was corrupted during an
> interrupt, but that's not likely.

Seems pretty unlikely on a 4 year old Via Epia. Never had any problems with it 
before now.

Maybe a cosmic ray event? ;-)

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 577 bytes --]

On Wed, 20 Dec 2006 22:15:50 GMT, Alistair John Strachan said:
> Seems pretty unlikely on a 4 year old Via Epia. Never had any problems with it
> before now.
> 
> Maybe a cosmic ray event? ;-)

More likely a stray alpha particle from a radioactive decay in the actual chip
casing - I saw some research a while back that said that the average commodity
system should *expect* to see 1 or 2 alpha-induced single-bit errors per year,
and the chance that *you* saw the event was directly related to whether the
memory had ECC, and how much of the other circuitry had ECC on it....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Oops in 2.6.19.1

[James Courtier-Dutton]

Chuck Ebbert wrote:
> In-Reply-To: <200612201421.03514.s0348365@sms.ed.ac.uk>
> 
> On Wed, 20 Dec 2006 14:21:03 +0000, Alistair John Strachan wrote:
> 
>> Any ideas?
>>
>> BUG: unable to handle kernel NULL pointer dereference at virtual address 
>> 00000009
> 
>     83 ca 10                  or     $0x10,%edx
>     3b                        .byte 0x3b
>     87 68 01                  xchg   %ebp,0x1(%eax)   <=====
>     00 00                     add    %al,(%eax)
> 
> Somehow it is trying to execute code in the middle of an instruction.
> That almost never works, even when the resulting fragment is a legal
> opcode. :)
> 
> The real instruction is:
> 
>     3b 87 68 01 00 00 00        cmp    0x168(%edi),%eax
> 
> I'd guess you have some kind of hardware problem.  It could also be
> a kernel problem where the saved address was corrupted during an
> interrupt, but that's not likely.

This looks rather strange.
The times I have seen this sort of problem is:
1) when one bit of the kernel is corrupting another part of it.
2) Kernel modules compiled with different gcc than rest of kernel.
3) kernel headers do not match the kernel being used.

One way to start tracking this down would be to run it with the fewest 
amount of kernel modules loaded as one can, but still reproduce the problem.

James

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Saturday 30 December 2006 18:06, James Courtier-Dutton wrote:
> > I'd guess you have some kind of hardware problem.  It could also be
> > a kernel problem where the saved address was corrupted during an
> > interrupt, but that's not likely.
>
> This looks rather strange.
[snip]

> 2) Kernel modules compiled with different gcc than rest of kernel.

Previously there was only one GCC version (4.1.1 totally replaced 3.4.3, and 
is the system wide GCC), now I have installed 3.4.6 into /opt/gcc-3.4.6 and 
it is only PATH'ed explicitly by me when I wish to compile a kernel using it:

export PATH=/opt/gcc-3.4.6/bin:$PATH
cp /boot/config-2.6.19-test .config
make oldconfig
make

> 3) kernel headers do not match the kernel being used.

The tree is a pristine 2.6.19.

> One way to start tracking this down would be to run it with the fewest
> amount of kernel modules loaded as one can, but still reproduce the
> problem.

Crippling the machine, though. Impractical for something that isn't 
immediately reproducible.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Chuck Ebbert]

In-Reply-To: <200612202215.50315.s0348365@sms.ed.ac.uk>

On Wed, 20 Dec 2006 22:15:50 +0000, Alistair John Strachan wrote:

> > I'd guess you have some kind of hardware problem.  It could also be
> > a kernel problem where the saved address was corrupted during an
> > interrupt, but that's not likely.
> 
> Seems pretty unlikely on a 4 year old Via Epia. Never had any problems with it 
> before now.
> 
> Maybe a cosmic ray event? ;-)

The low byte of eip should be 5f and it changed to 60, so that's
probably not it.  And the oops report is consistent with that being
the instruction that was really executed, so it's not the kernel
misreporting the address after it happened.

You weren't trying kprobes or something, were you? Have you ever
had another unexplained oops with this machine?

-- 
MBTI: IXTP

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Thursday 21 December 2006 08:05, Chuck Ebbert wrote:
> In-Reply-To: <200612202215.50315.s0348365@sms.ed.ac.uk>
>
> On Wed, 20 Dec 2006 22:15:50 +0000, Alistair John Strachan wrote:
> > > I'd guess you have some kind of hardware problem.  It could also be
> > > a kernel problem where the saved address was corrupted during an
> > > interrupt, but that's not likely.
> >
> > Seems pretty unlikely on a 4 year old Via Epia. Never had any problems
> > with it before now.
> >
> > Maybe a cosmic ray event? ;-)
>
> The low byte of eip should be 5f and it changed to 60, so that's
> probably not it.  And the oops report is consistent with that being
> the instruction that was really executed, so it's not the kernel
> misreporting the address after it happened.
>
> You weren't trying kprobes or something, were you? Have you ever
> had another unexplained oops with this machine?

Nope, it's a stock kernel and it's running on a server, kprobes isn't in use.

And no, to my knowledge there's not been another "unexplained" oops. I've had 
crashes, but they've always been known issues or BIOS trouble.

The machine was recently tampered with to install additional HDDs, but the 
memory was memtest'ed when it was installed and passed several times without 
issue. I'm rather puzzled.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Sunday 24 December 2006 04:23, Chuck Ebbert wrote:
> In-Reply-To: <200612231540.47176.s0348365@sms.ed.ac.uk>
>
> On Sat, 23 Dec 2006 15:40:46 +0000, Alistair John Strachan wrote:
> > Pretty much like clockwork, it happened again. I think it's time to take
> > this seriously as a software bug, and not some hardware problem. I've ran
> > kernels since 2.6.0 on this machine without such crashes, and now two of
> > the same in 2.6.19.1? Pretty unlikely!
>
> Stranger things have happened, e.g. your system might have started
> to overheat just recently.

True, I've considered it, I'll replace the CPU fan.

> Anyway, post your complete .config.  And exactly which one of the
> many Via cpus are you using?  Are you using the Padlock unit?

No, much older than that:

[alistair] 14:38 [~] cat /proc/cpuinfo
processor       : 0
vendor_id       : CentaurHauls
cpu family      : 6
model           : 9
model name      : VIA Nehemiah
stepping        : 1
cpu MHz         : 999.569
cache size      : 64 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu de tsc msr cx8 mtrr pge cmov mmx fxsr sse fxsr_opt
bogomips        : 2000.02

> What do those java/python programs do that are running?  What pipe
> are they polling?
>
> You could try going back to 2.6.18.x for a while in the meantime.

Well, I have had a thought. I recently upgraded the toolchain on the machine 
from binutils 2.16.x and GCC 3.4.3 (2.6.19 was built with this) to binutils 
2.17 and GCC 4.1.1. It's conceivable that this is some sort of compiler bug.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Alistair John Strachan]

[-- Attachment #1: Type: text/plain, Size: 226 bytes --]

On Sunday 24 December 2006 04:23, Chuck Ebbert wrote:
[snip]
> Anyway, post your complete .config.

Config attached.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

[-- Attachment #2: config-2.6.19.1 --]
[-- Type: text/plain, Size: 42229 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.19.1
# Sat Dec 16 19:30:00 2006
#
CONFIG_X86_32=y
CONFIG_GENERIC_TIME=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
# CONFIG_IPC_NS is not set
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_UTS_NS is not set
# CONFIG_AUDIT is not set
# CONFIG_IKCONFIG is not set
# CONFIG_RELAY is not set
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_SLAB=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
# CONFIG_SLOB is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Block layer
#
CONFIG_BLOCK=y
# CONFIG_LBD is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_LSF is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
CONFIG_DEFAULT_AS=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="anticipatory"

#
# Processor type and features
#
# CONFIG_SMP is not set
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
CONFIG_MVIAC3_2=y
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_ALIGNMENT_16=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_MCE is not set
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_REGPARM=y
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_COMPAT_VDSO is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_LEGACY is not set
# CONFIG_PM_DEBUG is not set
# CONFIG_PM_SYSFS_DEPRECATED is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
# CONFIG_ACPI_SLEEP is not set
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
# CONFIG_ACPI_BUTTON is not set
# CONFIG_ACPI_VIDEO is not set
# CONFIG_ACPI_HOTKEY is not set
# CONFIG_ACPI_FAN is not set
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
CONFIG_PCCARD=m
# CONFIG_PCMCIA_DEBUG is not set
# CONFIG_PCMCIA is not set
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PCMCIA_PROBE=y
CONFIG_PCCARD_NONSTATIC=m

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
# CONFIG_BINFMT_MISC is not set

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
# CONFIG_NETDEBUG is not set
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
# CONFIG_NETFILTER_NETLINK is not set
CONFIG_NETFILTER_XTABLES=m
# CONFIG_NETFILTER_XT_TARGET_CLASSIFY is not set
# CONFIG_NETFILTER_XT_TARGET_MARK is not set
# CONFIG_NETFILTER_XT_TARGET_NFQUEUE is not set
# CONFIG_NETFILTER_XT_MATCH_COMMENT is not set
# CONFIG_NETFILTER_XT_MATCH_CONNTRACK is not set
# CONFIG_NETFILTER_XT_MATCH_DCCP is not set
# CONFIG_NETFILTER_XT_MATCH_DSCP is not set
# CONFIG_NETFILTER_XT_MATCH_ESP is not set
# CONFIG_NETFILTER_XT_MATCH_HELPER is not set
# CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
# CONFIG_NETFILTER_XT_MATCH_LIMIT is not set
# CONFIG_NETFILTER_XT_MATCH_MAC is not set
# CONFIG_NETFILTER_XT_MATCH_MARK is not set
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
# CONFIG_NETFILTER_XT_MATCH_PHYSDEV is not set
# CONFIG_NETFILTER_XT_MATCH_PKTTYPE is not set
# CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
# CONFIG_NETFILTER_XT_MATCH_REALM is not set
# CONFIG_NETFILTER_XT_MATCH_SCTP is not set
CONFIG_NETFILTER_XT_MATCH_STATE=m
# CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
# CONFIG_NETFILTER_XT_MATCH_STRING is not set
# CONFIG_NETFILTER_XT_MATCH_TCPMSS is not set

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_CT_ACCT is not set
# CONFIG_IP_NF_CONNTRACK_MARK is not set
# CONFIG_IP_NF_CONNTRACK_EVENTS is not set
# CONFIG_IP_NF_CT_PROTO_SCTP is not set
CONFIG_IP_NF_FTP=m
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_NETBIOS_NS is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_AMANDA is not set
# CONFIG_IP_NF_PPTP is not set
# CONFIG_IP_NF_H323 is not set
# CONFIG_IP_NF_SIP is not set
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_IPRANGE=m
# CONFIG_IP_NF_MATCH_TOS is not set
CONFIG_IP_NF_MATCH_RECENT=m
# CONFIG_IP_NF_MATCH_ECN is not set
# CONFIG_IP_NF_MATCH_AH is not set
# CONFIG_IP_NF_MATCH_TTL is not set
# CONFIG_IP_NF_MATCH_OWNER is not set
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
# CONFIG_IP_NF_TARGET_LOG is not set
# CONFIG_IP_NF_TARGET_ULOG is not set
# CONFIG_IP_NF_TARGET_TCPMSS is not set
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
# CONFIG_IP_NF_TARGET_REDIRECT is not set
# CONFIG_IP_NF_TARGET_NETMAP is not set
# CONFIG_IP_NF_TARGET_SAME is not set
# CONFIG_IP_NF_NAT_SNMP_BASIC is not set
CONFIG_IP_NF_NAT_FTP=m
# CONFIG_IP_NF_MANGLE is not set
# CONFIG_IP_NF_RAW is not set
# CONFIG_IP_NF_ARPTABLES is not set

#
# Bridge: Netfilter Configuration
#
# CONFIG_BRIDGE_NF_EBTABLES is not set

#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set

#
# TIPC Configuration (EXPERIMENTAL)
#
# CONFIG_TIPC is not set
CONFIG_ATM=y
# CONFIG_ATM_CLIP is not set
# CONFIG_ATM_LANE is not set
# CONFIG_ATM_BR2684 is not set
CONFIG_BRIDGE=y
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set

#
# Queueing/Scheduling
#
# CONFIG_NET_SCH_CBQ is not set
CONFIG_NET_SCH_HTB=y
# CONFIG_NET_SCH_HFSC is not set
# CONFIG_NET_SCH_ATM is not set
CONFIG_NET_SCH_PRIO=y
# CONFIG_NET_SCH_RED is not set
CONFIG_NET_SCH_SFQ=y
# CONFIG_NET_SCH_TEQL is not set
# CONFIG_NET_SCH_TBF is not set
# CONFIG_NET_SCH_GRED is not set
# CONFIG_NET_SCH_DSMARK is not set
# CONFIG_NET_SCH_NETEM is not set
CONFIG_NET_SCH_INGRESS=y

#
# Classification
#
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
CONFIG_NET_CLS_FW=y
CONFIG_NET_CLS_U32=y
# CONFIG_CLS_U32_PERF is not set
# CONFIG_CLS_U32_MARK is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_EMATCH is not set
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_ESTIMATOR=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_IEEE80211 is not set
CONFIG_WIRELESS_EXT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_SYS_HYPERVISOR is not set

#
# Connector - unified userspace <-> kernelspace linker
#
# CONFIG_CONNECTOR is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_DEV_XD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
CONFIG_IDE_TASK_IOCTL=y

#
# IDE chipset support/bugfixes
#
# CONFIG_IDE_GENERIC is not set
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_CS5535 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
CONFIG_BLK_DEV_VIA82CXXX=y
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=m
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_PROC_FS is not set

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
CONFIG_CHR_DEV_SG=m
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set

#
# SCSI low-level drivers
#
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Serial ATA (prod) and Parallel ATA (experimental) drivers
#
# CONFIG_ATA is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
# CONFIG_MD_LINEAR is not set
# CONFIG_MD_RAID0 is not set
CONFIG_MD_RAID1=y
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID456 is not set
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
# CONFIG_BLK_DEV_DM is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_IEEE1394=m

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_IEEE1394_OUI_DB is not set
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
CONFIG_IEEE1394_CONFIG_ROM_IP1394=y
# CONFIG_IEEE1394_EXPORT_FULL_API is not set

#
# Device Drivers
#
# CONFIG_IEEE1394_PCILYNX is not set
CONFIG_IEEE1394_OHCI1394=m

#
# Protocol Drivers
#
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_SBP2=m
CONFIG_IEEE1394_SBP2_PHYS_DMA=y
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_DV1394=m
CONFIG_IEEE1394_RAWIO=m

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Network device support
#
CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# PHY device support
#
# CONFIG_PHYLIB is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_CS89x0 is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
CONFIG_VIA_RHINE=y
CONFIG_VIA_RHINE_MMIO=y
# CONFIG_VIA_RHINE_NAPI is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_R8169=y
CONFIG_R8169_NAPI=y
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set
# CONFIG_MYRI10GE is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
CONFIG_NET_RADIO=y
CONFIG_NET_WIRELESS_RTNETLINK=y

#
# Obsolete Wireless cards support (pre-802.11)
#
# CONFIG_STRIP is not set
# CONFIG_ARLAN is not set
# CONFIG_WAVELAN is not set

#
# Wireless 802.11b ISA/PCI cards support
#
# CONFIG_IPW2100 is not set
# CONFIG_IPW2200 is not set
# CONFIG_AIRO is not set
# CONFIG_HERMES is not set
# CONFIG_ATMEL is not set

#
# Prism GT/Duette 802.11(a/b/g) PCI/Cardbus support
#
CONFIG_PRISM54=m
# CONFIG_USB_ZD1201 is not set
# CONFIG_HOSTAP is not set
CONFIG_NET_WIRELESS=y

#
# Wan interfaces
#
# CONFIG_WAN is not set

#
# ATM drivers
#
# CONFIG_ATM_DUMMY is not set
# CONFIG_ATM_TCP is not set
# CONFIG_ATM_LANAI is not set
# CONFIG_ATM_ENI is not set
# CONFIG_ATM_FIRESTREAM is not set
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
# CONFIG_ATM_IDT77252 is not set
# CONFIG_ATM_AMBASSADOR is not set
# CONFIG_ATM_HORIZON is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E_MAYBE is not set
# CONFIG_ATM_HE is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=m
# CONFIG_PPP_MULTILINK is not set
# CONFIG_PPP_FILTER is not set
# CONFIG_PPP_ASYNC is not set
CONFIG_PPP_SYNC_TTY=m
# CONFIG_PPP_DEFLATE is not set
# CONFIG_PPP_BSDCOMP is not set
# CONFIG_PPP_MPPE is not set
# CONFIG_PPPOE is not set
CONFIG_PPPOATM=m
# CONFIG_SLIP is not set
CONFIG_SLHC=m
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_WISTRON_BTNS is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
# CONFIG_ROCKETPORT is not set
# CONFIG_CYCLADES is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_ESPSERIAL is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
# CONFIG_SYNCLINK is not set
# CONFIG_SYNCLINKMP is not set
# CONFIG_SYNCLINK_GT is not set
CONFIG_N_HDLC=m
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_SX is not set
# CONFIG_RIO is not set
# CONFIG_STALDRV is not set

#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=m
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
CONFIG_AGP_VIA=m
# CONFIG_AGP_EFFICEON is not set
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
# CONFIG_HANGCHECK_TIMER is not set

#
# TPM devices
#
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
# CONFIG_I2C_ALGOBIT is not set
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_ELEKTOR is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_DS1374 is not set
# CONFIG_SENSORS_EEPROM is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# SPI support
#
# CONFIG_SPI is not set
# CONFIG_SPI_MASTER is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Hardware Monitoring support
#
CONFIG_HWMON=m
CONFIG_HWMON_VID=m
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_K8TEMP is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_VIA686A is not set
CONFIG_SENSORS_VT1211=m
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set
# CONFIG_USB_DABUSB is not set

#
# Graphics support
#
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
# CONFIG_FB_MODE_HELPERS is not set
# CONFIG_FB_TILEBLITTING is not set
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
CONFIG_FB_VESA=y
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y

#
# Logo configuration
#
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
# CONFIG_SND_SEQUENCER is not set
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
# CONFIG_SND_RTCTIMER is not set
# CONFIG_SND_DYNAMIC_MINORS is not set
# CONFIG_SND_SUPPORT_OLD_API is not set
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_AC97_BUS=m
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set

#
# ISA devices
#
# CONFIG_SND_ADLIB is not set
# CONFIG_SND_AD1848 is not set
# CONFIG_SND_CMI8330 is not set
# CONFIG_SND_CS4231 is not set
# CONFIG_SND_CS4232 is not set
# CONFIG_SND_CS4236 is not set
# CONFIG_SND_ES1688 is not set
# CONFIG_SND_ES18XX is not set
# CONFIG_SND_GUSCLASSIC is not set
# CONFIG_SND_GUSEXTREME is not set
# CONFIG_SND_GUSMAX is not set
# CONFIG_SND_OPL3SA2 is not set
# CONFIG_SND_OPTI92X_AD1848 is not set
# CONFIG_SND_OPTI92X_CS4231 is not set
# CONFIG_SND_OPTI93X is not set
# CONFIG_SND_MIRO is not set
# CONFIG_SND_SB8 is not set
# CONFIG_SND_SB16 is not set
# CONFIG_SND_SBAWE is not set
# CONFIG_SND_SGALAXY is not set
# CONFIG_SND_SSCAPE is not set
# CONFIG_SND_WAVEFRONT is not set

#
# PCI devices
#
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDA_INTEL is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
CONFIG_SND_VIA82XX=m
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_AC97_POWER_SAVE is not set

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=m

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#

#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set

#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_ACECAD is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_TOUCHSCREEN is not set
# CONFIG_USB_YEALINK is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set
# CONFIG_USB_ATI_REMOTE2 is not set
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET_MII is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_MON is not set

#
# USB port drivers
#

#
# USB Serial Converter support
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_TEST is not set

#
# USB DSL modem support
#
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
# CONFIG_USB_CXACRU is not set
# CONFIG_USB_UEAGLEATM is not set
# CONFIG_USB_XUSBATM is not set

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# LED devices
#
# CONFIG_NEW_LEDS is not set

#
# LED drivers
#

#
# LED Triggers
#

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# EDAC - error detection and reporting (RAS) (EXPERIMENTAL)
#
# CONFIG_EDAC is not set

#
# Real Time Clock
#
# CONFIG_RTC_CLASS is not set

#
# DMA Engine support
#
# CONFIG_DMA_ENGINE is not set

#
# DMA Clients
#

#
# DMA Devices
#

#
# File systems
#
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4DEV_FS is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_XFS_FS=y
CONFIG_XFS_QUOTA=y
# CONFIG_XFS_SECURITY is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
CONFIG_FUSE_FS=y

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=m
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=850
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-15"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
# CONFIG_CONFIGFS_FS is not set

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=m
CONFIG_NLS_DEFAULT="iso8859-15"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=m
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m

#
# Instrumentation Support
#
# CONFIG_PROFILING is not set
# CONFIG_KPROBES is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_RWSEMS is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_LIST is not set
CONFIG_FRAME_POINTER=y
CONFIG_UNWIND_INFO=y
CONFIG_STACK_UNWIND=y
CONFIG_FORCED_INLINING=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
# CONFIG_CRYPTO is not set

#
# Library routines
#
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
CONFIG_CRC32=y
# CONFIG_LIBCRC32C is not set
CONFIG_ZLIB_INFLATE=m
CONFIG_PLIST=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_KTIME_SCALAR=y

Re: Oops in 2.6.19.1

[Chuck Ebbert]

In-Reply-To: <200612301659.35982.s0348365@sms.ed.ac.uk>

On Sat, 30 Dec 2006 16:59:35 +0000, Alistair John Strachan wrote:

> I've eliminated 2.6.19.1 as the culprit, and also tried toggling "optimize for 
> size", various debug options. 2.6.19 compiled with GCC 4.1.1 on an Via 
> Nehemiah C3-2 seems to crash in pipe_poll reliably, within approximately 12 
> hours.

Which CPU are you compiling for?  You should try different options.

Can you post disassembly of pipe_poll() for both the one that crashes
and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
relocation info and post just the one function from each for now.

-- 
MBTI: IXTP

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Saturday 30 December 2006 17:21, Chuck Ebbert wrote:
> In-Reply-To: <200612301659.35982.s0348365@sms.ed.ac.uk>
>
> On Sat, 30 Dec 2006 16:59:35 +0000, Alistair John Strachan wrote:
> > I've eliminated 2.6.19.1 as the culprit, and also tried toggling
> > "optimize for size", various debug options. 2.6.19 compiled with GCC
> > 4.1.1 on an Via Nehemiah C3-2 seems to crash in pipe_poll reliably,
> > within approximately 12 hours.
>
> Which CPU are you compiling for?  You should try different options.

I should, I haven't thought of that. Currently it's compiling for 
CONFIG_MVIAC3_2, but I could try i686 for example.

> Can you post disassembly of pipe_poll() for both the one that crashes
> and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
> relocation info and post just the one function from each for now.

Sure, no problem:

http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/

Both use identical configs, neither are optimised for size. The config is 
available from the same location.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Adrian Bunk]

On Sat, Dec 30, 2006 at 06:29:15PM +0000, Alistair John Strachan wrote:
> On Saturday 30 December 2006 17:21, Chuck Ebbert wrote:
> > In-Reply-To: <200612301659.35982.s0348365@sms.ed.ac.uk>
> >
> > On Sat, 30 Dec 2006 16:59:35 +0000, Alistair John Strachan wrote:
> > > I've eliminated 2.6.19.1 as the culprit, and also tried toggling
> > > "optimize for size", various debug options. 2.6.19 compiled with GCC
> > > 4.1.1 on an Via Nehemiah C3-2 seems to crash in pipe_poll reliably,
> > > within approximately 12 hours.
> >
> > Which CPU are you compiling for?  You should try different options.
> 
> I should, I haven't thought of that. Currently it's compiling for 
> CONFIG_MVIAC3_2, but I could try i686 for example.
> 
> > Can you post disassembly of pipe_poll() for both the one that crashes
> > and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
> > relocation info and post just the one function from each for now.
> 
> Sure, no problem:
> 
> http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/
> 
> Both use identical configs, neither are optimised for size. The config is 
> available from the same location.

Can you try enabling as many debug options as possible?

> Cheers,
> Alistair.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Sunday 31 December 2006 16:28, Adrian Bunk wrote:
> On Sat, Dec 30, 2006 at 06:29:15PM +0000, Alistair John Strachan wrote:
> > On Saturday 30 December 2006 17:21, Chuck Ebbert wrote:
> > > In-Reply-To: <200612301659.35982.s0348365@sms.ed.ac.uk>
> > >
> > > On Sat, 30 Dec 2006 16:59:35 +0000, Alistair John Strachan wrote:
> > > > I've eliminated 2.6.19.1 as the culprit, and also tried toggling
> > > > "optimize for size", various debug options. 2.6.19 compiled with GCC
> > > > 4.1.1 on an Via Nehemiah C3-2 seems to crash in pipe_poll reliably,
> > > > within approximately 12 hours.
> > >
> > > Which CPU are you compiling for?  You should try different options.
> >
> > I should, I haven't thought of that. Currently it's compiling for
> > CONFIG_MVIAC3_2, but I could try i686 for example.
> >
> > > Can you post disassembly of pipe_poll() for both the one that crashes
> > > and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
> > > relocation info and post just the one function from each for now.
> >
> > Sure, no problem:
> >
> > http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/
> >
> > Both use identical configs, neither are optimised for size. The config is
> > available from the same location.
>
> Can you try enabling as many debug options as possible?

Specifically what? I've already had:

CONFIG_DETECT_SOFTLOCKUP
CONFIG_FRAME_POINTER
CONFIG_UNWIND_INFO

Enabled. CONFIG_4KSTACKS is disabled. Are there any debugging features 
actually pertinent to this bug?

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.

Re: Oops in 2.6.19.1

[Adrian Bunk]

On Sun, Dec 31, 2006 at 04:48:43PM +0000, Alistair John Strachan wrote:
> On Sunday 31 December 2006 16:28, Adrian Bunk wrote:
> > On Sat, Dec 30, 2006 at 06:29:15PM +0000, Alistair John Strachan wrote:
> > > On Saturday 30 December 2006 17:21, Chuck Ebbert wrote:
> > > > In-Reply-To: <200612301659.35982.s0348365@sms.ed.ac.uk>
> > > >
> > > > On Sat, 30 Dec 2006 16:59:35 +0000, Alistair John Strachan wrote:
> > > > > I've eliminated 2.6.19.1 as the culprit, and also tried toggling
> > > > > "optimize for size", various debug options. 2.6.19 compiled with GCC
> > > > > 4.1.1 on an Via Nehemiah C3-2 seems to crash in pipe_poll reliably,
> > > > > within approximately 12 hours.
> > > >
> > > > Which CPU are you compiling for?  You should try different options.
> > >
> > > I should, I haven't thought of that. Currently it's compiling for
> > > CONFIG_MVIAC3_2, but I could try i686 for example.
> > >
> > > > Can you post disassembly of pipe_poll() for both the one that crashes
> > > > and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
> > > > relocation info and post just the one function from each for now.
> > >
> > > Sure, no problem:
> > >
> > > http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/
> > >
> > > Both use identical configs, neither are optimised for size. The config is
> > > available from the same location.
> >
> > Can you try enabling as many debug options as possible?
> 
> Specifically what? I've already had:
> 
> CONFIG_DETECT_SOFTLOCKUP
> CONFIG_FRAME_POINTER
> CONFIG_UNWIND_INFO
> 
> Enabled. CONFIG_4KSTACKS is disabled. Are there any debugging features 
> actually pertinent to this bug?

No, that's only an "enable as much as possible and hope one helps" shot 
in the dark.

> Cheers,
> Alistair.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Oops in 2.6.19.1

[Chuck Ebbert]

In-Reply-To: <200612301829.15980.s0348365@sms.ed.ac.uk>

On Sat, 30 Dec 2006 18:29:15 +0000, Alistair John Strachan wrote:

> > Can you post disassembly of pipe_poll() for both the one that crashes
> > and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
> > relocation info and post just the one function from each for now.
> 
> Sure, no problem:
> 
> http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/
> 
> Both use identical configs, neither are optimised for size. The config is 
> available from the same location.

Those were compiled without frame pointers.  Can you post them compiled
with frame pointers so they match your original bug report? And confirm
that pipe_poll() is still at 0xc0156ec0 in vmlinux?

-- 
MBTI: IXTP

Re: Oops in 2.6.19.1

[Alistair John Strachan]

On Sunday 31 December 2006 21:43, Chuck Ebbert wrote:
> In-Reply-To: <200612301829.15980.s0348365@sms.ed.ac.uk>
>
> On Sat, 30 Dec 2006 18:29:15 +0000, Alistair John Strachan wrote:
> > > Can you post disassembly of pipe_poll() for both the one that crashes
> > > and the one that doesn't?  Use 'objdump -D -r fs/pipe.o' so we get the
> > > relocation info and post just the one function from each for now.
> >
> > Sure, no problem:
> >
> > http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/
> >
> > Both use identical configs, neither are optimised for size. The config is
> > available from the same location.
>
> Those were compiled without frame pointers.  Can you post them compiled
> with frame pointers so they match your original bug report? And confirm
> that pipe_poll() is still at 0xc0156ec0 in vmlinux?

c0156ec0 <pipe_poll>:

I used the config I original sent you to rebuild it again. This time I've put 
up the whole vmlinux for both kernels, the config is replaced, the 
decompilation is re-done, I've confirmed the offset in the GCC 4.1.1 kernel 
is identical. Sorry for the confusion.

The reason I changed the configs was to experiment with enabling and disabling 
debugging (and other such) options that might have shaken out compiler bugs.

However none of these kernels have ever crashed gracefully again, most of them 
hang the machine (no nmi watchdog though) so I've not been able to look at 
the oops. It's the same root cause, however, as GCC 3.4.6 kernels do not 
crash.

http://devzero.co.uk/~alistair/2.6.19-via-c3-pipe_poll/

Happy new year, btw.

-- 
Cheers,
Alistair.

Final year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.