Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=10324
> 
>            Summary: kernel panic ip_route_input
>            Product: Networking
>            Version: 2.5
>      KernelVersion: 2.6.25-rc6-git6
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: IPV4
>         AssignedTo: shemminger@linux-foundation.org
>         ReportedBy: nuclearcat@nuclearcat.com
> 
> 
> Latest working kernel version: 2.6.24.x
> Earliest failing kernel version: 2.6.25-rc6

A post-2.6.24 regression.

> Distribution: Custom
> Hardware Environment: Dual CPU, e100 + e1000
> Software Environment: Custom build. FIB_TRIE enabled
> Problem Description: Kernel panic after few seconds running network on heavy
> traffic (around 350 Mbps). 
> 
> Steps to reproduce: It is complicated configuration. I can provide whole image
> of system, around 128MB (it is semi-embedded), but it will need at least few
> traffic generators similar to mine. So it is difficult to reproduce. I can
> apply any patches required, but highly prefferable if they dont make server
> unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and
> software watchdog enabled.
> I think it is critical to fix this bug before releasing 2.6.25.
> 
> Here is latest message with panic:
> CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - 
> enabled.
> 
> It reboots recently after bringing up networking. Sometimes even not reaching 
> place where i am loading netconsole.
> 
> Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
> to reproduce, but i cannot do bisect, cause this machine dont have power 
> switch and in difficult to reach area.
> 

I've snipped the log output - it was such a wordwrapped mess :(

Can you please resend it in a reply-to-all to this email, after ensuring
that it won't be wordwrapped?

Thanks.

Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

[Denys]

I upload whole netconsole file at
http://www.nuclearcat.com/files/ip_route_input_crash.txt 
There is two crashes.



On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote
> (switched to email.  Please respond via emailed reply-to-all, not 
> via the bugzilla web interface).
> 
> On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=10324
> > 
> >            Summary: kernel panic ip_route_input
> >            Product: Networking
> >            Version: 2.5
> >      KernelVersion: 2.6.25-rc6-git6
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: IPV4
> >         AssignedTo: shemminger@linux-foundation.org
> >         ReportedBy: nuclearcat@nuclearcat.com
> > 
> > 
> > Latest working kernel version: 2.6.24.x
> > Earliest failing kernel version: 2.6.25-rc6
> 
> A post-2.6.24 regression.
> 
> > Distribution: Custom
> > Hardware Environment: Dual CPU, e100 + e1000
> > Software Environment: Custom build. FIB_TRIE enabled
> > Problem Description: Kernel panic after few seconds running network on heavy
> > traffic (around 350 Mbps). 
> > 
> > Steps to reproduce: It is complicated configuration. I can provide whole image
> > of system, around 128MB (it is semi-embedded), but it will need at least few
> > traffic generators similar to mine. So it is difficult to reproduce. I can
> > apply any patches required, but highly prefferable if they dont make server
> > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and
> > software watchdog enabled.
> > I think it is critical to fix this bug before releasing 2.6.25.
> > 
> > Here is latest message with panic:
> > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - 
> > enabled.
> > 
> > It reboots recently after bringing up networking. Sometimes even not reaching 
> > place where i am loading netconsole.
> > 
> > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
> > to reproduce, but i cannot do bisect, cause this machine dont have power 
> > switch and in difficult to reach area.
> >
> 
> I've snipped the log output - it was such a wordwrapped mess :(
> 
> Can you please resend it in a reply-to-all to this email, after ensuring
> that it won't be wordwrapped?
> 
> Thanks.


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

[Denys]

Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
recent patches to FIB_TRIE broke something (i have complicated setup with ifb,
maybe it will help). If you can point me to any patch, i can reverse it and
try again with TRIE.

On Tue, 25 Mar 2008 13:03:02 -0700, Andrew Morton wrote
> (switched to email.  Please respond via emailed reply-to-all, not 
> via the bugzilla web interface).
> 
> On Tue, 25 Mar 2008 12:48:37 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=10324
> > 
> >            Summary: kernel panic ip_route_input
> >            Product: Networking
> >            Version: 2.5
> >      KernelVersion: 2.6.25-rc6-git6
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: IPV4
> >         AssignedTo: shemminger@linux-foundation.org
> >         ReportedBy: nuclearcat@nuclearcat.com
> > 
> > 
> > Latest working kernel version: 2.6.24.x
> > Earliest failing kernel version: 2.6.25-rc6
> 
> A post-2.6.24 regression.
> 
> > Distribution: Custom
> > Hardware Environment: Dual CPU, e100 + e1000
> > Software Environment: Custom build. FIB_TRIE enabled
> > Problem Description: Kernel panic after few seconds running network on heavy
> > traffic (around 350 Mbps). 
> > 
> > Steps to reproduce: It is complicated configuration. I can provide whole image
> > of system, around 128MB (it is semi-embedded), but it will need at least few
> > traffic generators similar to mine. So it is difficult to reproduce. I can
> > apply any patches required, but highly prefferable if they dont make server
> > unreachable. I have panic and oops autoreboot sysctl options, nmi-watchdog and
> > software watchdog enabled.
> > I think it is critical to fix this bug before releasing 2.6.25.
> > 
> > Here is latest message with panic:
> > CONFIG_DEBUG_RODATA, 4K stacks, Frame pointers, CONFIG_DEBUG_STACKOVERFLOW - 
> > enabled.
> > 
> > It reboots recently after bringing up networking. Sometimes even not reaching 
> > place where i am loading netconsole.
> > 
> > Always in ip_route_input. This is 2.6.25-rc6-git6, 2 panic's. They are easy 
> > to reproduce, but i cannot do bisect, cause this machine dont have power 
> > switch and in difficult to reach area.
> >
> 
> I've snipped the log output - it was such a wordwrapped mess :(
> 
> Can you please resend it in a reply-to-all to this email, after ensuring
> that it won't be wordwrapped?
> 
> Thanks.


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

[David Miller]

From: "Denys" <nuclearcat@nuclearcat.com>
Date: Wed, 26 Mar 2008 01:12:01 +0200

> Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
> recent patches to FIB_TRIE broke something (i have complicated setup with ifb,
> maybe it will help). If you can point me to any patch, i can reverse it and
> try again with TRIE.

Unfortunately there were 39 changes made to fib_trie
during this time.  So it's not like we can give you
one patch to try reverting :-/

Really, the fastest way to track this down would be
to do a bisect, focused in on net/ipv4/fib_trie.c
which git bisect does allow you to do.

Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

[Stephen Hemminger]

On Wed, 26 Mar 2008 01:12:01 +0200
"Denys" <nuclearcat@nuclearcat.com> wrote:

> Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It seems
> recent patches to FIB_TRIE broke something (i have complicated setup with ifb,
> maybe it will help). If you can point me to any patch, i can reverse it and
> try again with TRIE.

Bisect?

Re: [Bugme-new] [Bug 10324] New: kernel panic ip_route_input

[Denys]

I have to wait when power switch will be installed in that location then.


On Tue, 25 Mar 2008 16:49:09 -0700, Stephen Hemminger wrote
> On Wed, 26 Mar 2008 01:12:01 +0200
> "Denys" <nuclearcat@nuclearcat.com> wrote:
> 
> > Changing from FIB_TRIE to FIB_HASH, and it is not panicing anymore. It 
seems
> > recent patches to FIB_TRIE broke something (i have complicated setup with 
ifb,
> > maybe it will help). If you can point me to any patch, i can reverse it 
and
> > try again with TRIE.
> 
> Bisect?
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.