* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Ingo Molnar @ 2009-08-26 19:08 UTC (permalink / raw)
To: Neil Horman, David S. Miller, Steven Rostedt,
=?unknown-8bit?B?RnLDqWTDqXJpYw==?= Weisbecker
Cc: Bill Fink, Linux Network Developers, brice, gallatin
In-Reply-To: <20090826190435.GC10816@hmsreliant.think-freely.org>
* Neil Horman <nhorman@tuxdriver.com> wrote:
> On Wed, Aug 26, 2009 at 08:15:02PM +0200, Ingo Molnar wrote:
> >
> > * Neil Horman <nhorman@tuxdriver.com> wrote:
> >
> > > On Wed, Aug 26, 2009 at 07:00:13AM -0400, Neil Horman wrote:
> > > > On Wed, Aug 26, 2009 at 03:10:57AM -0400, Bill Fink wrote:
> > > > > On Fri, 21 Aug 2009, Neil Horman wrote:
> > > > >
> > > > > > On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
> > > > > > > On Thu, 20 Aug 2009, Neil Horman wrote:
> > > > > > >
> > > > > > > > On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
> > > > > > > >
> > > > > > > > > When I tried an actual nuttcp performance test, even when rate limiting
> > > > > > > > > to just 1 Mbps, I immediately got a kernel oops. I tried to get a
> > > > > > > > > crashdump via kexec/kdump, but the kexec kernel, instead of just
> > > > > > > > > generating a crashdump, fully booted the new kernel, which was
> > > > > > > > > extremely sluggish until I rebooted it through a BIOS re-init,
> > > > > > > > > and never produced a crashdump. I tried this several times and
> > > > > > > > > an immediate kernel oops was always the result (with either a TCP
> > > > > > > > > or UDP test). A ping test of 1000 9000-byte packets with an interval
> > > > > > > > > of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
> > > > > > > > > worked just fine.
> > > > > > > >
> > > > > > > > The sluggishness is expected, since the kdump kernel operates out of such
> > > > > > > > limited memory. don't know why you booted to a full system rather than did a
> > > > > > > > crash recovery. Don't suppose you got a backtrace did you?
> > > > > > >
> > > > > > > There was a backtrace on the screen but I didn't have a chance to
> > > > > > > record it. BTW did anyone ever think to print the backtrace in
> > > > > > > reverse (first to some reserved memory and then output to the display)
> > > > > > > so the more interesting parts wouldn't have scrolled off the top of
> > > > > > > the screen?
> > > > > > >
> > > > > > The real solution is to use a console to which the output doesn't scroll off the
> > > > > > screen. Normally people use a serial console they can log, or a RAC card that
> > > > > > they can record. Even on a regular vga monitor in text mode, you can set up the
> > > > > > vt iirc to allow for scrolling.
> > > > >
> > > > > None of our Asus P6T6 systems have serial consoles. I don't know of
> > > > > any RAC cards for them either, nor are there spare PCI slots available
> > > > > in many cases. I wouldn't think the Shift-PageUp trick would work
> > > > > with a crashed kernel, but I admit I didn't try it. I haven't checked
> > > > > out netconsole yet either, but I'm not sure it would help either in a
> > > > > case like this that was a network related kernel crash.
> > > > >
> > > > Any USB ports that you can attach a serial dongle to? That would work as well,
> > > > or, as previously mentioned, netconsole also does the trick.
> > > >
> > > > > In any case, a simple kernel command line that would provide a reversed
> > > > > backtrace would be a simple thing to facilitate Linux users providing
> > > > > useful info to Linux kernel developers in helping to debug kernel
> > > > > problems. The most useful info would still be on the screen, so it
> > > > > could be transcribed or a photo image of the screen could be taken.
> > > > >
> > > > I understand what your saying, I'm just saying there are currently several
> > > > options for you that have already solved this problem in differnt ways.
> > > >
> > > > > Fortunately, in this specific case, the SuperMicro X8DAH+-F system
> > > > > does have a serial console, and after a fair amount of effort I was
> > > > > able to get it to work as desired, and was able to finally capture
> > > > > a backtrace of the kernel oops. BTW I believe the reason the
> > > > > kexec/kdump didn't work was probably because it couldn't find
> > > > > a /proc/vmcore file, although I don't know why that would be,
> > > > > and the Fedora 10 /etc/init.d/kdump script will then just boot
> > > > > up normally if it fails to find the /proc/vmcore file (or it's
> > > > > zero size).
> > > > >
> > > > I take care of kdump for fedora and RHEL. If you file a bug on this, I'd be
> > > > happy to look into it further.
> > > >
> > > > > The following shows a simple ping test usage of the skb_sources
> > > > > tracing feature:
> > > > >
> > > > > [root@xeontest1 tracing]# numactl --membind=1 taskset -c 4 ping -c 5 -s 1472 192.168.1.10
> > > > > PING 192.168.1.10 (192.168.1.10) 1472(1500) bytes of data.
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.139 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.182 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=3 ttl=64 time=0.178 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=4 ttl=64 time=0.188 ms
> > > > > 1480 bytes from 192.168.1.10: icmp_seq=5 ttl=64 time=0.178 ms
> > > > >
> > > > > --- 192.168.1.10 ping statistics ---
> > > > > 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
> > > > > rtt min/avg/max/mdev = 0.139/0.173/0.188/0.017 ms
> > > > >
> > > > > [root@xeontest1 tracing]# cat trace
> > > > > # tracer: skb_sources
> > > > > #
> > > > > # PID ANID CNID IFC RXQ CCPU LEN
> > > > > # | | | | | | |
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > > 4217 1 1 eth2 0 4 1500
> > > > >
> > > > > All is as was expected.
> > > > >
> > > > > But if I try an actual nuttcp performance test (even rate limited
> > > > > to 1 Mbps), I get the following kernel oops:
> > > > >
> > > > thank you, I think I see the problem, I'll have a patch for you in just a bit
> > > >
> > > > Thanks
> > > > Neil
> > > >
> > > > > [root@xeontest1 tracing]# numactl --membind=1 nuttcp -In2 -Ri1m -xc4/0 192.168.1.10
> > > > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
> > > > > IP: [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > > PGD 337d12067 PUD 337d11067 PMD 0
> > > > > Oops: 0000 [#1] SMP
> > > > > last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:8b:00.0/0000:8c:04.0e
> > > > > CPU 4
> > > > > Modules linked in: w83627ehf hwmon_vid coretemp hwmon ipv6 dm_multipath uinput ]
> > > > > Pid: 4222, comm: nuttcp Not tainted 2.6.31-rc6-bf #3 X8DAH
> > > > > RIP: 0010:[<ffffffff810b01ab>] [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x12
> > > > > RSP: 0018:ffff8801a5811a88 EFLAGS: 00010213
> > > > > RAX: 0000000000000000 RBX: ffff88033906d154 RCX: 000000000000000d
> > > > > RDX: 000000000000f88c RSI: 000000000000000b RDI: ffff8803383d3044
> > > > > RBP: ffff8801a5811ab8 R08: 0000000000000001 R09: ffff8801ab311a00
> > > > > R10: 0000000000000005 R11: ffffc9000080e2b0 R12: ffff880337c45400
> > > > > R13: ffff88033906d150 R14: 0000000000000014 R15: ffffffff818bb890
> > > > > FS: 00007fa976d326f0(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > CR2: 0000000000000038 CR3: 000000033801e000 CR4: 00000000000006e0
> > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > Process nuttcp (pid: 4222, threadinfo ffff8801a5810000, task ffff8801ab2e5d00)
> > > > > Stack:
> > > > > ffff8801a5811ab8 ffff8801b35d4ab0 0000000000000014 0000000000000000
> > > > > <0> 0000000000000014 0000000000000014 ffff8801a5811b18 ffffffff81366ae8
> > > > > <0> ffff8801a5811ed8 0000001439084000 ffff880337c45400 00000001001416ef
> > > > > Call Trace:
> > > > > [<ffffffff81366ae8>] skb_copy_datagram_iovec+0x50/0x1f5
> > > > > [<ffffffff813ac875>] tcp_rcv_established+0x278/0x6db
> > > > > [<ffffffff813b3ef5>] tcp_v4_do_rcv+0x1b8/0x366
> > > > > [<ffffffff8135f99e>] ? release_sock+0xab/0xb4
> > > > > [<ffffffff8136004d>] ? sk_wait_data+0xc8/0xd6
> > > > > [<ffffffff813a32d6>] tcp_prequeue_process+0x79/0x8f
> > > > > [<ffffffff813a455d>] tcp_recvmsg+0x4e8/0xaa0
> > > > > [<ffffffff8135ec90>] sock_common_recvmsg+0x37/0x4c
> > > > > [<ffffffff8135cb06>] __sock_recvmsg+0x72/0x7f
> > > > > [<ffffffff8135cbdd>] sock_aio_read+0xca/0xda
> > > > > [<ffffffff810d9536>] ? vma_merge+0x2a0/0x318
> > > > > [<ffffffff810f6d4f>] do_sync_read+0xec/0x132
> > > > > [<ffffffff81067ddc>] ? autoremove_wake_function+0x0/0x3d
> > > > > [<ffffffff811b646c>] ? security_file_permission+0x16/0x18
> > > > > [<ffffffff810f785c>] vfs_read+0xc0/0x107
> > > > > [<ffffffff810f7971>] sys_read+0x4c/0x75
> > > > > [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
> > > > > Code: 44 89 73 30 89 43 14 41 0f b7 84 24 ac 00 00 00 89 43 28 65 8b 04 25 98 e
> > > > > RIP [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > > RSP <ffff8801a5811a88>
> > > > > CR2: 0000000000000038
> > > > >
> > > > > -Thanks
> > > > >
> > > > > -Bill
> > > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > >
> > >
> > >
> > > Here you go, I think this will fix your oops.
> > >
> > >
> > > Fix NULL pointer deref in skb sources ftracer
> > >
> > > Its possible that skb->sk will be null in this path, so we shouldn't just assume
> > > we can pass it to sock_net
> > >
> > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> > >
> > > trace_skb_sources.c | 6 ++++--
> > > 1 file changed, 4 insertions(+), 2 deletions(-)
> >
> > ok if this is just a temporary fix until TRACE_EVENT() is done, but
> > we'll get rid of this and do TRACE_EVENT() before net-next-2.6 it's
> > pushed to .32, right?
>
> Not sure that the two are related. I think you meant to send this
> to the other thread, didnt you?
Sigh, no. Please re-read the past discussions about this.
trace_skb_sources.c is a hack and should be converted to generic
tracepoints. Is there anything in it that cannot be expressed in
terms of TRACE_EVENT()?
Ingo
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-26 19:04 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Bill Fink, Linux Network Developers, brice, gallatin
In-Reply-To: <20090826181502.GC13632@elte.hu>
On Wed, Aug 26, 2009 at 08:15:02PM +0200, Ingo Molnar wrote:
>
> * Neil Horman <nhorman@tuxdriver.com> wrote:
>
> > On Wed, Aug 26, 2009 at 07:00:13AM -0400, Neil Horman wrote:
> > > On Wed, Aug 26, 2009 at 03:10:57AM -0400, Bill Fink wrote:
> > > > On Fri, 21 Aug 2009, Neil Horman wrote:
> > > >
> > > > > On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
> > > > > > On Thu, 20 Aug 2009, Neil Horman wrote:
> > > > > >
> > > > > > > On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
> > > > > > >
> > > > > > > > When I tried an actual nuttcp performance test, even when rate limiting
> > > > > > > > to just 1 Mbps, I immediately got a kernel oops. I tried to get a
> > > > > > > > crashdump via kexec/kdump, but the kexec kernel, instead of just
> > > > > > > > generating a crashdump, fully booted the new kernel, which was
> > > > > > > > extremely sluggish until I rebooted it through a BIOS re-init,
> > > > > > > > and never produced a crashdump. I tried this several times and
> > > > > > > > an immediate kernel oops was always the result (with either a TCP
> > > > > > > > or UDP test). A ping test of 1000 9000-byte packets with an interval
> > > > > > > > of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
> > > > > > > > worked just fine.
> > > > > > >
> > > > > > > The sluggishness is expected, since the kdump kernel operates out of such
> > > > > > > limited memory. don't know why you booted to a full system rather than did a
> > > > > > > crash recovery. Don't suppose you got a backtrace did you?
> > > > > >
> > > > > > There was a backtrace on the screen but I didn't have a chance to
> > > > > > record it. BTW did anyone ever think to print the backtrace in
> > > > > > reverse (first to some reserved memory and then output to the display)
> > > > > > so the more interesting parts wouldn't have scrolled off the top of
> > > > > > the screen?
> > > > > >
> > > > > The real solution is to use a console to which the output doesn't scroll off the
> > > > > screen. Normally people use a serial console they can log, or a RAC card that
> > > > > they can record. Even on a regular vga monitor in text mode, you can set up the
> > > > > vt iirc to allow for scrolling.
> > > >
> > > > None of our Asus P6T6 systems have serial consoles. I don't know of
> > > > any RAC cards for them either, nor are there spare PCI slots available
> > > > in many cases. I wouldn't think the Shift-PageUp trick would work
> > > > with a crashed kernel, but I admit I didn't try it. I haven't checked
> > > > out netconsole yet either, but I'm not sure it would help either in a
> > > > case like this that was a network related kernel crash.
> > > >
> > > Any USB ports that you can attach a serial dongle to? That would work as well,
> > > or, as previously mentioned, netconsole also does the trick.
> > >
> > > > In any case, a simple kernel command line that would provide a reversed
> > > > backtrace would be a simple thing to facilitate Linux users providing
> > > > useful info to Linux kernel developers in helping to debug kernel
> > > > problems. The most useful info would still be on the screen, so it
> > > > could be transcribed or a photo image of the screen could be taken.
> > > >
> > > I understand what your saying, I'm just saying there are currently several
> > > options for you that have already solved this problem in differnt ways.
> > >
> > > > Fortunately, in this specific case, the SuperMicro X8DAH+-F system
> > > > does have a serial console, and after a fair amount of effort I was
> > > > able to get it to work as desired, and was able to finally capture
> > > > a backtrace of the kernel oops. BTW I believe the reason the
> > > > kexec/kdump didn't work was probably because it couldn't find
> > > > a /proc/vmcore file, although I don't know why that would be,
> > > > and the Fedora 10 /etc/init.d/kdump script will then just boot
> > > > up normally if it fails to find the /proc/vmcore file (or it's
> > > > zero size).
> > > >
> > > I take care of kdump for fedora and RHEL. If you file a bug on this, I'd be
> > > happy to look into it further.
> > >
> > > > The following shows a simple ping test usage of the skb_sources
> > > > tracing feature:
> > > >
> > > > [root@xeontest1 tracing]# numactl --membind=1 taskset -c 4 ping -c 5 -s 1472 192.168.1.10
> > > > PING 192.168.1.10 (192.168.1.10) 1472(1500) bytes of data.
> > > > 1480 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.139 ms
> > > > 1480 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.182 ms
> > > > 1480 bytes from 192.168.1.10: icmp_seq=3 ttl=64 time=0.178 ms
> > > > 1480 bytes from 192.168.1.10: icmp_seq=4 ttl=64 time=0.188 ms
> > > > 1480 bytes from 192.168.1.10: icmp_seq=5 ttl=64 time=0.178 ms
> > > >
> > > > --- 192.168.1.10 ping statistics ---
> > > > 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
> > > > rtt min/avg/max/mdev = 0.139/0.173/0.188/0.017 ms
> > > >
> > > > [root@xeontest1 tracing]# cat trace
> > > > # tracer: skb_sources
> > > > #
> > > > # PID ANID CNID IFC RXQ CCPU LEN
> > > > # | | | | | | |
> > > > 4217 1 1 eth2 0 4 1500
> > > > 4217 1 1 eth2 0 4 1500
> > > > 4217 1 1 eth2 0 4 1500
> > > > 4217 1 1 eth2 0 4 1500
> > > > 4217 1 1 eth2 0 4 1500
> > > >
> > > > All is as was expected.
> > > >
> > > > But if I try an actual nuttcp performance test (even rate limited
> > > > to 1 Mbps), I get the following kernel oops:
> > > >
> > > thank you, I think I see the problem, I'll have a patch for you in just a bit
> > >
> > > Thanks
> > > Neil
> > >
> > > > [root@xeontest1 tracing]# numactl --membind=1 nuttcp -In2 -Ri1m -xc4/0 192.168.1.10
> > > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
> > > > IP: [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > PGD 337d12067 PUD 337d11067 PMD 0
> > > > Oops: 0000 [#1] SMP
> > > > last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:8b:00.0/0000:8c:04.0e
> > > > CPU 4
> > > > Modules linked in: w83627ehf hwmon_vid coretemp hwmon ipv6 dm_multipath uinput ]
> > > > Pid: 4222, comm: nuttcp Not tainted 2.6.31-rc6-bf #3 X8DAH
> > > > RIP: 0010:[<ffffffff810b01ab>] [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x12
> > > > RSP: 0018:ffff8801a5811a88 EFLAGS: 00010213
> > > > RAX: 0000000000000000 RBX: ffff88033906d154 RCX: 000000000000000d
> > > > RDX: 000000000000f88c RSI: 000000000000000b RDI: ffff8803383d3044
> > > > RBP: ffff8801a5811ab8 R08: 0000000000000001 R09: ffff8801ab311a00
> > > > R10: 0000000000000005 R11: ffffc9000080e2b0 R12: ffff880337c45400
> > > > R13: ffff88033906d150 R14: 0000000000000014 R15: ffffffff818bb890
> > > > FS: 00007fa976d326f0(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > CR2: 0000000000000038 CR3: 000000033801e000 CR4: 00000000000006e0
> > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > Process nuttcp (pid: 4222, threadinfo ffff8801a5810000, task ffff8801ab2e5d00)
> > > > Stack:
> > > > ffff8801a5811ab8 ffff8801b35d4ab0 0000000000000014 0000000000000000
> > > > <0> 0000000000000014 0000000000000014 ffff8801a5811b18 ffffffff81366ae8
> > > > <0> ffff8801a5811ed8 0000001439084000 ffff880337c45400 00000001001416ef
> > > > Call Trace:
> > > > [<ffffffff81366ae8>] skb_copy_datagram_iovec+0x50/0x1f5
> > > > [<ffffffff813ac875>] tcp_rcv_established+0x278/0x6db
> > > > [<ffffffff813b3ef5>] tcp_v4_do_rcv+0x1b8/0x366
> > > > [<ffffffff8135f99e>] ? release_sock+0xab/0xb4
> > > > [<ffffffff8136004d>] ? sk_wait_data+0xc8/0xd6
> > > > [<ffffffff813a32d6>] tcp_prequeue_process+0x79/0x8f
> > > > [<ffffffff813a455d>] tcp_recvmsg+0x4e8/0xaa0
> > > > [<ffffffff8135ec90>] sock_common_recvmsg+0x37/0x4c
> > > > [<ffffffff8135cb06>] __sock_recvmsg+0x72/0x7f
> > > > [<ffffffff8135cbdd>] sock_aio_read+0xca/0xda
> > > > [<ffffffff810d9536>] ? vma_merge+0x2a0/0x318
> > > > [<ffffffff810f6d4f>] do_sync_read+0xec/0x132
> > > > [<ffffffff81067ddc>] ? autoremove_wake_function+0x0/0x3d
> > > > [<ffffffff811b646c>] ? security_file_permission+0x16/0x18
> > > > [<ffffffff810f785c>] vfs_read+0xc0/0x107
> > > > [<ffffffff810f7971>] sys_read+0x4c/0x75
> > > > [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
> > > > Code: 44 89 73 30 89 43 14 41 0f b7 84 24 ac 00 00 00 89 43 28 65 8b 04 25 98 e
> > > > RIP [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > > RSP <ffff8801a5811a88>
> > > > CR2: 0000000000000038
> > > >
> > > > -Thanks
> > > >
> > > > -Bill
> > > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> >
> >
> > Here you go, I think this will fix your oops.
> >
> >
> > Fix NULL pointer deref in skb sources ftracer
> >
> > Its possible that skb->sk will be null in this path, so we shouldn't just assume
> > we can pass it to sock_net
> >
> > Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
> >
> > trace_skb_sources.c | 6 ++++--
> > 1 file changed, 4 insertions(+), 2 deletions(-)
>
> ok if this is just a temporary fix until TRACE_EVENT() is done, but
> we'll get rid of this and do TRACE_EVENT() before net-next-2.6 it's
> pushed to .32, right?
>
Not sure that the two are related. I think you meant to send this to the other
thread, didnt you?
Neil
> Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: iproute2 / tbf with large burst seems broken again
From: Jarek Poplawski @ 2009-08-26 19:03 UTC (permalink / raw)
To: Denys Fedoryschenko; +Cc: netdev
In-Reply-To: <20090825200306.GA3020@ami.dom.local>
On Tue, Aug 25, 2009 at 10:03:06PM +0200, Jarek Poplawski wrote:
> Denys Fedoryschenko wrote, On 08/25/2009 01:16 PM:
> ...
> > But this one maybe will overflow because of limitations in iproute2.
> >
> > PPoE_146 ~ # ./tc -s -d qdisc show dev ppp13
> > qdisc tbf 8004: root rate 96000bit burst 797465b/8 mpu 0b lat 275.4s
> > Sent 82867 bytes 123 pkt (dropped 0, overlimits 0 requeues 0)
> > rate 0bit 0pps backlog 0b 0p requeues 0
> > qdisc ingress ffff: parent ffff:fff1 ----------------
> > Sent 506821 bytes 1916 pkt (dropped 0, overlimits 0 requeues 0)
> > rate 0bit 0pps backlog 0b 0p requeues 0
> >
> > So maybe all of that just wrong way of using TBF.
>
> I guess so; I've just recollected you described it some time ago. If
> it were done only with TBF it would mean very large surges with line
> speed and probably a lot of drops by ISP. Since you're ISP, you
> probably drop this with HTB or something (then you should mention it
> describing the problem) or keep very long queues which means great
> latencies. Probably there is a lot of TCP resending btw. Using TBF
> with HTB etc. is considered wrong idea anyway. (But if it works for
> you shouldn't care.)
>
> > At same time this means, if HTB and policers in filters done same way, that
> > QoS in Linux cannot do similar to squid delay pools feature:
> >
> > First 10Mb give with 1Mbit/s, then slow 64Kbit/s. If user use less than 64K -
> > recharge with that unused bandwidth a "10 Mb / 1Mbit bucket".
So I thought about it a little more and I'm quite sure this idea with
large buckets is wrong/ineffective. I guess you could "describe" it
in HTB something like this:
tc class add dev ppp0 parent 1:3 classid 1:4 htb rate 64kbit\
burst 10mb cburst 10mb
tc class add dev ppp0 parent 1:4 classid 1:4 htb rate 64kbit ceil 1mbit\
cburst 10mb
(Of course, there would be this overflow problem with 2.6.31-rc and
so big buffers.)
So, the main point is: if somebody didn't send his/her 64Kbits long
time ago it usually means it's lost and can't be shared later. You
could try your luck, but e.g. if at the moment all users use their
64Kbits plus one of them 'thinks' he/she can send "saved" bits, it
means some other guy doesn't get his/her minimum (they send together
but some bytes will be dropped or queued).
It would work OK if you've reserved 1mbit per 64Kbit user but I guess
it's not what you do. So, IMHO, it should be better to use classical
methods to guarantee these 64Kbit with reasonable latency, plus
additonal borrowing with ceil and reasonable (much smaller buffers)
yet.
Jarek P.
^ permalink raw reply
* Re: [PATCH] smc91x: fix compilation on SMP
From: David Miller @ 2009-08-26 19:03 UTC (permalink / raw)
To: adobriyan; +Cc: netdev, dongdong.deng
In-Reply-To: <20090826085526.GA3209@x200.localdomain>
From: Alexey Dobriyan <adobriyan@gmail.com>
Date: Wed, 26 Aug 2009 12:55:26 +0400
> Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
Applied, thanks. :-/
^ permalink raw reply
* Re: [PATCH 7/7] MAINTAINERS: update information for sfc network driver
From: Joe Perches @ 2009-08-26 18:47 UTC (permalink / raw)
To: Ben Hutchings; +Cc: David Miller, netdev, linux-net-drivers
In-Reply-To: <1251310744.27345.21.camel@achroite>
On Wed, 2009-08-26 at 19:19 +0100, Ben Hutchings wrote:
> Update the current maintainers and add our email addresses. Robert
> Stonehouse has moved on to other projects.
>
> Move the linux-net-drivers alias to the top since it should always
> reach the current maintainers at Solarflare. Add netdev as the
> relevant mailing list.
[]
> +M: linux-net-drivers@solarflare.com (alias for current maintainers)
Hi Ben
This form: "M: bare address (extra stuff)" doesn't work
with the get_maintainers script.
Can you please use something like this instead?
M: Solarflare linux maintainers <linux-net-drivers@solarflare.com>
Perhaps this diff below.
diff --git a/MAINTAINERS b/MAINTAINERS
index 866253d..058ada4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4521,9 +4521,10 @@ S: Supported
F: drivers/net/benet/
SFC NETWORK DRIVER
-P: Steve Hodgson
-P: Ben Hutchings
-M: Robert Stonehouse <linux-net-drivers@solarflare.com>
+M: Solarflare linux maintainers <linux-net-drivers@solarflare.com>
+M: Steve Hodgson <shodgson@solarflare.com>
+M: Ben Hutchings <bhutchings@solarflare.com>
+L: netdev@vger.kernel.org
S: Supported
F: drivers/net/sfc/
^ permalink raw reply related
* Re: AlacrityVM benchmark numbers updated
From: Gregory Haskins @ 2009-08-26 18:42 UTC (permalink / raw)
To: Avi Kivity
Cc: alacrityvm-devel, linux-kernel@vger.kernel.org,
kvm@vger.kernel.org, Michael S. Tsirkin, netdev
In-Reply-To: <4A950B75.5020409@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]
Avi Kivity wrote:
> On 08/26/2009 04:01 AM, Gregory Haskins wrote:
>> We are pleased to announce the availability of the latest networking
>> benchmark numbers for AlacrityVM. We've made several tweaks to the
>> original v0.1 release to improve performance. The most notable is a
>> switch from get_user_pages to switch_mm+copy_[to/from]_user thanks to a
>> review suggestion from Michael Tsirkin (as well as his patch to
>> implement it).
>>
>> This change alone accounted for freeing up an additional 1.2Gbps, which
>> is over 25% improvement from v0.1. The previous numbers were 4560Gbps
>> before the change, and 5708Gbps after (for 1500mtu over 10GE). This
>> moves us ever closer to the goal of native performance under
>> virtualization.
>>
>
> Interesting, it's good to see that copy_*_user() works so well. Note
> that there's a possible optimization that goes in the opposite direction
> - keep using get_user_pages(), but use the dma engine API to perform the
> actual copy. I expect that it will only be a win when using tso to
> transfer full pages. Large pages may also help.
>
> Copyless tx also wants get_user_pages(). It makes sense to check if
> switch_mm() + get_user_pages_fast() gives better performance than
> get_user_pages().
Actually, I have already look at this and it does indeed seem better to
use switch_mm+gupf() over gup() by quite a large margin. You could then
couple that with your DMA-engine idea to potentially gain even more
benefits (though probably not for networking since most NICs have their
own DMA engine anyway).
Kind Regards,
-Greg
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 267 bytes --]
^ permalink raw reply
* [PATCH 7/7] MAINTAINERS: update information for sfc network driver
From: Ben Hutchings @ 2009-08-26 18:19 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
Update the current maintainers and add our email addresses. Robert
Stonehouse has moved on to other projects.
Move the linux-net-drivers alias to the top since it should always
reach the current maintainers at Solarflare. Add netdev as the
relevant mailing list.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
MAINTAINERS | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 866253d..058ada4 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4521,9 +4521,10 @@ S: Supported
F: drivers/net/benet/
SFC NETWORK DRIVER
-P: Steve Hodgson
-P: Ben Hutchings
-M: Robert Stonehouse <linux-net-drivers@solarflare.com>
+M: linux-net-drivers@solarflare.com (alias for current maintainers)
+M: Steve Hodgson <shodgson@solarflare.com>
+M: Ben Hutchings <bhutchings@solarflare.com>
+L: netdev@vger.kernel.org
S: Supported
F: drivers/net/sfc/
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* [PATCH 6/7] sfc: Improve reliability of RX queue flushing
From: Ben Hutchings @ 2009-08-26 18:18 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
Reconfiguring the port requires us to flush all DMA queues. In
repeated testing we have found that RX flushes would sometimes fail
because the RX DMA engine was not properly isolated from the MACs.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
drivers/net/sfc/efx.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/drivers/net/sfc/efx.c b/drivers/net/sfc/efx.c
index d4dc920..07a7e4b 100644
--- a/drivers/net/sfc/efx.c
+++ b/drivers/net/sfc/efx.c
@@ -1179,6 +1179,8 @@ static void efx_stop_all(struct efx_nic *efx)
/* Isolate the MAC from the TX and RX engines, so that queue
* flushes will complete in a timely fashion. */
+ falcon_deconfigure_mac_wrapper(efx);
+ msleep(10); /* Let the Rx FIFO drain */
falcon_drain_tx_fifo(efx);
/* Stop the kernel transmit interface late, so the watchdog
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* [PATCH 5/7] sfc: Work around XMAC bug causing packet loss with some peers
From: Ben Hutchings @ 2009-08-26 18:17 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
Received frames must be re-clocked by the local XGXS to the 156.25 MHz
(DDR) clock of the XGMII. If the remote clock is slightly faster this
can reduce a minimum IPG of 64 bit-times (1 cycle) to 32 bit-times
(half a cycle). If the XMAC detects that a frame has reached the
maximum RX frame length in the same cycle that it receives one of
these reduced IPGs, it may miss the IPG, causing two valid frames to
be treated as a single invalid frame (over-length with bad CRC).
We work around this by increasing the maximum RX frame length so that
peers with matched MTU will not provoke this bug.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
drivers/net/sfc/net_driver.h | 7 ++++++-
1 files changed, 6 insertions(+), 1 deletions(-)
diff --git a/drivers/net/sfc/net_driver.h b/drivers/net/sfc/net_driver.h
index 5eabede..298566d 100644
--- a/drivers/net/sfc/net_driver.h
+++ b/drivers/net/sfc/net_driver.h
@@ -984,9 +984,14 @@ static inline void clear_bit_le(unsigned nr, unsigned char *addr)
*
* The 10G MAC used in Falcon requires 8-byte alignment on the frame
* length, so we round up to the nearest 8.
+ *
+ * Re-clocking by the XGXS on RX can reduce an IPG to 32 bits (half an
+ * XGMII cycle). If the frame length reaches the maximum value in the
+ * same cycle, the XMAC can miss the IPG altogether. We work around
+ * this by adding a further 16 bytes.
*/
#define EFX_MAX_FRAME_LEN(mtu) \
- ((((mtu) + ETH_HLEN + VLAN_HLEN + 4/* FCS */) + 7) & ~7)
+ ((((mtu) + ETH_HLEN + VLAN_HLEN + 4/* FCS */ + 7) & ~7) + 16)
#endif /* EFX_NET_DRIVER_H */
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* [PATCH 4/7] sfc: QT2025C: Use hard reset only
From: Ben Hutchings @ 2009-08-26 18:17 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
At probe time, falcon_reset_hw() performs a hard reset of the PHY
along with Falcon. There is no need to perform a soft reset later,
and any access to standard MDIO registers before the PHY firmware has
booted can interrupt the boot process, making the port unusable.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
drivers/net/sfc/xfp_phy.c | 19 ++++++++++---------
1 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/drivers/net/sfc/xfp_phy.c b/drivers/net/sfc/xfp_phy.c
index bb2e6af..e6b3d5e 100644
--- a/drivers/net/sfc/xfp_phy.c
+++ b/drivers/net/sfc/xfp_phy.c
@@ -97,23 +97,24 @@ static int qt2025c_wait_reset(struct efx_nic *efx)
return 0;
}
-/* Reset the PHYXS MMD. This is documented (for the Quake PHYs) as doing
- * a complete soft reset.
- */
static int xfp_reset_phy(struct efx_nic *efx)
{
int rc;
- rc = efx_mdio_reset_mmd(efx, MDIO_MMD_PHYXS,
- XFP_MAX_RESET_TIME / XFP_RESET_WAIT,
- XFP_RESET_WAIT);
- if (rc < 0)
- goto fail;
-
if (efx->phy_type == PHY_TYPE_QT2025C) {
+ /* Wait for the reset triggered by falcon_reset_hw()
+ * to complete */
rc = qt2025c_wait_reset(efx);
if (rc < 0)
goto fail;
+ } else {
+ /* Reset the PHYXS MMD. This is documented as doing
+ * a complete soft reset. */
+ rc = efx_mdio_reset_mmd(efx, MDIO_MMD_PHYXS,
+ XFP_MAX_RESET_TIME / XFP_RESET_WAIT,
+ XFP_RESET_WAIT);
+ if (rc < 0)
+ goto fail;
}
/* Wait 250ms for the PHY to complete bootup */
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* [PATCH 3/7] mdio: Check MDIO_STAT1_FAULT in mdio45_links_ok()
From: Ben Hutchings @ 2009-08-26 18:16 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
Some PHYs will report that the link is up even though there is a fault
condition. Therefore, check the fault flag too. We must also read
STAT2 to reset this flag.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
drivers/net/mdio.c | 13 ++++++++++---
1 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/net/mdio.c b/drivers/net/mdio.c
index 6851bdb..7d2e610 100644
--- a/drivers/net/mdio.c
+++ b/drivers/net/mdio.c
@@ -109,13 +109,20 @@ int mdio45_links_ok(const struct mdio_if_info *mdio, u32 mmd_mask)
if (mmd_mask & (1 << devad)) {
mmd_mask &= ~(1 << devad);
- /* Read twice because link state is latched and a
- * read moves the current state into the register */
+ /* Reset the latched status and fault flags */
mdio->mdio_read(mdio->dev, mdio->prtad,
devad, MDIO_STAT1);
+ if (devad == MDIO_MMD_PMAPMD || devad == MDIO_MMD_PCS ||
+ devad == MDIO_MMD_PHYXS || devad == MDIO_MMD_DTEXS)
+ mdio->mdio_read(mdio->dev, mdio->prtad,
+ devad, MDIO_STAT2);
+
+ /* Check the current status and fault flags */
reg = mdio->mdio_read(mdio->dev, mdio->prtad,
devad, MDIO_STAT1);
- if (reg < 0 || !(reg & MDIO_STAT1_LSTATUS))
+ if (reg < 0 ||
+ (reg & (MDIO_STAT1_FAULT | MDIO_STAT1_LSTATUS)) !=
+ MDIO_STAT1_LSTATUS)
return false;
}
}
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* [PATCH 2/7] sfc: Do not reinitialise XAUI serdes before it has completed reset
From: Ben Hutchings @ 2009-08-26 18:16 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
falcon_reset_xaui() waits for XGXS reset to complete, but the XAUI
serdes reset may take longer. It needs to check both reset active
bits.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
drivers/net/sfc/falcon_hwdefs.h | 2 ++
drivers/net/sfc/falcon_xmac.c | 8 +++++---
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/drivers/net/sfc/falcon_hwdefs.h b/drivers/net/sfc/falcon_hwdefs.h
index 375e2a5..2d22611 100644
--- a/drivers/net/sfc/falcon_hwdefs.h
+++ b/drivers/net/sfc/falcon_hwdefs.h
@@ -700,6 +700,8 @@
/* XGXS/XAUI powerdown/reset register */
#define XX_PWR_RST_REG 0x1300
+#define XX_SD_RST_ACT_LBN 16
+#define XX_SD_RST_ACT_WIDTH 1
#define XX_PWRDND_EN_LBN 15
#define XX_PWRDND_EN_WIDTH 1
#define XX_PWRDNC_EN_LBN 14
diff --git a/drivers/net/sfc/falcon_xmac.c b/drivers/net/sfc/falcon_xmac.c
index 2b3269c..bec52ca 100644
--- a/drivers/net/sfc/falcon_xmac.c
+++ b/drivers/net/sfc/falcon_xmac.c
@@ -64,13 +64,15 @@ int falcon_reset_xaui(struct efx_nic *efx)
efx_oword_t reg;
int count;
+ /* Start reset sequence */
EFX_POPULATE_DWORD_1(reg, XX_RST_XX_EN, 1);
falcon_write(efx, ®, XX_PWR_RST_REG);
- /* Give some time for the link to establish */
- for (count = 0; count < 1000; count++) { /* wait upto 10ms */
+ /* Wait up to 10 ms for completion, then reinitialise */
+ for (count = 0; count < 1000; count++) {
falcon_read(efx, ®, XX_PWR_RST_REG);
- if (EFX_OWORD_FIELD(reg, XX_RST_XX_EN) == 0) {
+ if (EFX_OWORD_FIELD(reg, XX_RST_XX_EN) == 0 &&
+ EFX_OWORD_FIELD(reg, XX_SD_RST_ACT) == 0) {
falcon_setup_xaui(efx);
return 0;
}
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* [PATCH 1/7] sfc: Fix ordering of device registration and initial netif_carrier_off()
From: Ben Hutchings @ 2009-08-26 18:16 UTC (permalink / raw)
To: David Miller; +Cc: netdev, linux-net-drivers
We must call netif_carrier_off() after the device is registered, not
before, to set the operational state and user-space IFF_RUNNING flag
correctly.
Since we don't want observers to see an intermediate state, open-code
register_netdev() and add efx_update_name() and netif_carrier_off()
into the locked region.
Signed-off-by: Ben Hutchings <bhutchings@solarflare.com>
---
drivers/net/sfc/efx.c | 26 +++++++++++++++++---------
1 files changed, 17 insertions(+), 9 deletions(-)
diff --git a/drivers/net/sfc/efx.c b/drivers/net/sfc/efx.c
index 343e8da..d4dc920 100644
--- a/drivers/net/sfc/efx.c
+++ b/drivers/net/sfc/efx.c
@@ -1614,21 +1614,24 @@ static int efx_register_netdev(struct efx_nic *efx)
SET_NETDEV_DEV(net_dev, &efx->pci_dev->dev);
SET_ETHTOOL_OPS(net_dev, &efx_ethtool_ops);
- /* Always start with carrier off; PHY events will detect the link */
- netif_carrier_off(efx->net_dev);
-
/* Clear MAC statistics */
efx->mac_op->update_stats(efx);
memset(&efx->mac_stats, 0, sizeof(efx->mac_stats));
- rc = register_netdev(net_dev);
- if (rc) {
- EFX_ERR(efx, "could not register net dev\n");
- return rc;
- }
-
rtnl_lock();
+
+ rc = dev_alloc_name(net_dev, net_dev->name);
+ if (rc < 0)
+ goto fail_locked;
efx_update_name(efx);
+
+ rc = register_netdevice(net_dev);
+ if (rc)
+ goto fail_locked;
+
+ /* Always start with carrier off; PHY events will detect the link */
+ netif_carrier_off(efx->net_dev);
+
rtnl_unlock();
rc = device_create_file(&efx->pci_dev->dev, &dev_attr_phy_type);
@@ -1639,6 +1642,11 @@ static int efx_register_netdev(struct efx_nic *efx)
return 0;
+fail_locked:
+ rtnl_unlock();
+ EFX_ERR(efx, "could not register net dev\n");
+ return rc;
+
fail_registered:
unregister_netdev(net_dev);
return rc;
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply related
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Ingo Molnar @ 2009-08-26 18:15 UTC (permalink / raw)
To: Neil Horman; +Cc: Bill Fink, Linux Network Developers, brice, gallatin
In-Reply-To: <20090826180811.GB10816@hmsreliant.think-freely.org>
* Neil Horman <nhorman@tuxdriver.com> wrote:
> On Wed, Aug 26, 2009 at 07:00:13AM -0400, Neil Horman wrote:
> > On Wed, Aug 26, 2009 at 03:10:57AM -0400, Bill Fink wrote:
> > > On Fri, 21 Aug 2009, Neil Horman wrote:
> > >
> > > > On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
> > > > > On Thu, 20 Aug 2009, Neil Horman wrote:
> > > > >
> > > > > > On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
> > > > > >
> > > > > > > When I tried an actual nuttcp performance test, even when rate limiting
> > > > > > > to just 1 Mbps, I immediately got a kernel oops. I tried to get a
> > > > > > > crashdump via kexec/kdump, but the kexec kernel, instead of just
> > > > > > > generating a crashdump, fully booted the new kernel, which was
> > > > > > > extremely sluggish until I rebooted it through a BIOS re-init,
> > > > > > > and never produced a crashdump. I tried this several times and
> > > > > > > an immediate kernel oops was always the result (with either a TCP
> > > > > > > or UDP test). A ping test of 1000 9000-byte packets with an interval
> > > > > > > of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
> > > > > > > worked just fine.
> > > > > >
> > > > > > The sluggishness is expected, since the kdump kernel operates out of such
> > > > > > limited memory. don't know why you booted to a full system rather than did a
> > > > > > crash recovery. Don't suppose you got a backtrace did you?
> > > > >
> > > > > There was a backtrace on the screen but I didn't have a chance to
> > > > > record it. BTW did anyone ever think to print the backtrace in
> > > > > reverse (first to some reserved memory and then output to the display)
> > > > > so the more interesting parts wouldn't have scrolled off the top of
> > > > > the screen?
> > > > >
> > > > The real solution is to use a console to which the output doesn't scroll off the
> > > > screen. Normally people use a serial console they can log, or a RAC card that
> > > > they can record. Even on a regular vga monitor in text mode, you can set up the
> > > > vt iirc to allow for scrolling.
> > >
> > > None of our Asus P6T6 systems have serial consoles. I don't know of
> > > any RAC cards for them either, nor are there spare PCI slots available
> > > in many cases. I wouldn't think the Shift-PageUp trick would work
> > > with a crashed kernel, but I admit I didn't try it. I haven't checked
> > > out netconsole yet either, but I'm not sure it would help either in a
> > > case like this that was a network related kernel crash.
> > >
> > Any USB ports that you can attach a serial dongle to? That would work as well,
> > or, as previously mentioned, netconsole also does the trick.
> >
> > > In any case, a simple kernel command line that would provide a reversed
> > > backtrace would be a simple thing to facilitate Linux users providing
> > > useful info to Linux kernel developers in helping to debug kernel
> > > problems. The most useful info would still be on the screen, so it
> > > could be transcribed or a photo image of the screen could be taken.
> > >
> > I understand what your saying, I'm just saying there are currently several
> > options for you that have already solved this problem in differnt ways.
> >
> > > Fortunately, in this specific case, the SuperMicro X8DAH+-F system
> > > does have a serial console, and after a fair amount of effort I was
> > > able to get it to work as desired, and was able to finally capture
> > > a backtrace of the kernel oops. BTW I believe the reason the
> > > kexec/kdump didn't work was probably because it couldn't find
> > > a /proc/vmcore file, although I don't know why that would be,
> > > and the Fedora 10 /etc/init.d/kdump script will then just boot
> > > up normally if it fails to find the /proc/vmcore file (or it's
> > > zero size).
> > >
> > I take care of kdump for fedora and RHEL. If you file a bug on this, I'd be
> > happy to look into it further.
> >
> > > The following shows a simple ping test usage of the skb_sources
> > > tracing feature:
> > >
> > > [root@xeontest1 tracing]# numactl --membind=1 taskset -c 4 ping -c 5 -s 1472 192.168.1.10
> > > PING 192.168.1.10 (192.168.1.10) 1472(1500) bytes of data.
> > > 1480 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.139 ms
> > > 1480 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.182 ms
> > > 1480 bytes from 192.168.1.10: icmp_seq=3 ttl=64 time=0.178 ms
> > > 1480 bytes from 192.168.1.10: icmp_seq=4 ttl=64 time=0.188 ms
> > > 1480 bytes from 192.168.1.10: icmp_seq=5 ttl=64 time=0.178 ms
> > >
> > > --- 192.168.1.10 ping statistics ---
> > > 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
> > > rtt min/avg/max/mdev = 0.139/0.173/0.188/0.017 ms
> > >
> > > [root@xeontest1 tracing]# cat trace
> > > # tracer: skb_sources
> > > #
> > > # PID ANID CNID IFC RXQ CCPU LEN
> > > # | | | | | | |
> > > 4217 1 1 eth2 0 4 1500
> > > 4217 1 1 eth2 0 4 1500
> > > 4217 1 1 eth2 0 4 1500
> > > 4217 1 1 eth2 0 4 1500
> > > 4217 1 1 eth2 0 4 1500
> > >
> > > All is as was expected.
> > >
> > > But if I try an actual nuttcp performance test (even rate limited
> > > to 1 Mbps), I get the following kernel oops:
> > >
> > thank you, I think I see the problem, I'll have a patch for you in just a bit
> >
> > Thanks
> > Neil
> >
> > > [root@xeontest1 tracing]# numactl --membind=1 nuttcp -In2 -Ri1m -xc4/0 192.168.1.10
> > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
> > > IP: [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > PGD 337d12067 PUD 337d11067 PMD 0
> > > Oops: 0000 [#1] SMP
> > > last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:8b:00.0/0000:8c:04.0e
> > > CPU 4
> > > Modules linked in: w83627ehf hwmon_vid coretemp hwmon ipv6 dm_multipath uinput ]
> > > Pid: 4222, comm: nuttcp Not tainted 2.6.31-rc6-bf #3 X8DAH
> > > RIP: 0010:[<ffffffff810b01ab>] [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x12
> > > RSP: 0018:ffff8801a5811a88 EFLAGS: 00010213
> > > RAX: 0000000000000000 RBX: ffff88033906d154 RCX: 000000000000000d
> > > RDX: 000000000000f88c RSI: 000000000000000b RDI: ffff8803383d3044
> > > RBP: ffff8801a5811ab8 R08: 0000000000000001 R09: ffff8801ab311a00
> > > R10: 0000000000000005 R11: ffffc9000080e2b0 R12: ffff880337c45400
> > > R13: ffff88033906d150 R14: 0000000000000014 R15: ffffffff818bb890
> > > FS: 00007fa976d326f0(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > CR2: 0000000000000038 CR3: 000000033801e000 CR4: 00000000000006e0
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Process nuttcp (pid: 4222, threadinfo ffff8801a5810000, task ffff8801ab2e5d00)
> > > Stack:
> > > ffff8801a5811ab8 ffff8801b35d4ab0 0000000000000014 0000000000000000
> > > <0> 0000000000000014 0000000000000014 ffff8801a5811b18 ffffffff81366ae8
> > > <0> ffff8801a5811ed8 0000001439084000 ffff880337c45400 00000001001416ef
> > > Call Trace:
> > > [<ffffffff81366ae8>] skb_copy_datagram_iovec+0x50/0x1f5
> > > [<ffffffff813ac875>] tcp_rcv_established+0x278/0x6db
> > > [<ffffffff813b3ef5>] tcp_v4_do_rcv+0x1b8/0x366
> > > [<ffffffff8135f99e>] ? release_sock+0xab/0xb4
> > > [<ffffffff8136004d>] ? sk_wait_data+0xc8/0xd6
> > > [<ffffffff813a32d6>] tcp_prequeue_process+0x79/0x8f
> > > [<ffffffff813a455d>] tcp_recvmsg+0x4e8/0xaa0
> > > [<ffffffff8135ec90>] sock_common_recvmsg+0x37/0x4c
> > > [<ffffffff8135cb06>] __sock_recvmsg+0x72/0x7f
> > > [<ffffffff8135cbdd>] sock_aio_read+0xca/0xda
> > > [<ffffffff810d9536>] ? vma_merge+0x2a0/0x318
> > > [<ffffffff810f6d4f>] do_sync_read+0xec/0x132
> > > [<ffffffff81067ddc>] ? autoremove_wake_function+0x0/0x3d
> > > [<ffffffff811b646c>] ? security_file_permission+0x16/0x18
> > > [<ffffffff810f785c>] vfs_read+0xc0/0x107
> > > [<ffffffff810f7971>] sys_read+0x4c/0x75
> > > [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
> > > Code: 44 89 73 30 89 43 14 41 0f b7 84 24 ac 00 00 00 89 43 28 65 8b 04 25 98 e
> > > RIP [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > > RSP <ffff8801a5811a88>
> > > CR2: 0000000000000038
> > >
> > > -Thanks
> > >
> > > -Bill
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
> Here you go, I think this will fix your oops.
>
>
> Fix NULL pointer deref in skb sources ftracer
>
> Its possible that skb->sk will be null in this path, so we shouldn't just assume
> we can pass it to sock_net
>
> Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
>
> trace_skb_sources.c | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
ok if this is just a temporary fix until TRACE_EVENT() is done, but
we'll get rid of this and do TRACE_EVENT() before net-next-2.6 it's
pushed to .32, right?
Ingo
^ permalink raw reply
* Re: Receive side performance issue with multi-10-GigE and NUMA
From: Neil Horman @ 2009-08-26 18:08 UTC (permalink / raw)
To: Bill Fink; +Cc: Linux Network Developers, brice, gallatin
In-Reply-To: <20090826110013.GA10816@hmsreliant.think-freely.org>
On Wed, Aug 26, 2009 at 07:00:13AM -0400, Neil Horman wrote:
> On Wed, Aug 26, 2009 at 03:10:57AM -0400, Bill Fink wrote:
> > On Fri, 21 Aug 2009, Neil Horman wrote:
> >
> > > On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
> > > > On Thu, 20 Aug 2009, Neil Horman wrote:
> > > >
> > > > > On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
> > > > >
> > > > > > When I tried an actual nuttcp performance test, even when rate limiting
> > > > > > to just 1 Mbps, I immediately got a kernel oops. I tried to get a
> > > > > > crashdump via kexec/kdump, but the kexec kernel, instead of just
> > > > > > generating a crashdump, fully booted the new kernel, which was
> > > > > > extremely sluggish until I rebooted it through a BIOS re-init,
> > > > > > and never produced a crashdump. I tried this several times and
> > > > > > an immediate kernel oops was always the result (with either a TCP
> > > > > > or UDP test). A ping test of 1000 9000-byte packets with an interval
> > > > > > of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
> > > > > > worked just fine.
> > > > >
> > > > > The sluggishness is expected, since the kdump kernel operates out of such
> > > > > limited memory. don't know why you booted to a full system rather than did a
> > > > > crash recovery. Don't suppose you got a backtrace did you?
> > > >
> > > > There was a backtrace on the screen but I didn't have a chance to
> > > > record it. BTW did anyone ever think to print the backtrace in
> > > > reverse (first to some reserved memory and then output to the display)
> > > > so the more interesting parts wouldn't have scrolled off the top of
> > > > the screen?
> > > >
> > > The real solution is to use a console to which the output doesn't scroll off the
> > > screen. Normally people use a serial console they can log, or a RAC card that
> > > they can record. Even on a regular vga monitor in text mode, you can set up the
> > > vt iirc to allow for scrolling.
> >
> > None of our Asus P6T6 systems have serial consoles. I don't know of
> > any RAC cards for them either, nor are there spare PCI slots available
> > in many cases. I wouldn't think the Shift-PageUp trick would work
> > with a crashed kernel, but I admit I didn't try it. I haven't checked
> > out netconsole yet either, but I'm not sure it would help either in a
> > case like this that was a network related kernel crash.
> >
> Any USB ports that you can attach a serial dongle to? That would work as well,
> or, as previously mentioned, netconsole also does the trick.
>
> > In any case, a simple kernel command line that would provide a reversed
> > backtrace would be a simple thing to facilitate Linux users providing
> > useful info to Linux kernel developers in helping to debug kernel
> > problems. The most useful info would still be on the screen, so it
> > could be transcribed or a photo image of the screen could be taken.
> >
> I understand what your saying, I'm just saying there are currently several
> options for you that have already solved this problem in differnt ways.
>
> > Fortunately, in this specific case, the SuperMicro X8DAH+-F system
> > does have a serial console, and after a fair amount of effort I was
> > able to get it to work as desired, and was able to finally capture
> > a backtrace of the kernel oops. BTW I believe the reason the
> > kexec/kdump didn't work was probably because it couldn't find
> > a /proc/vmcore file, although I don't know why that would be,
> > and the Fedora 10 /etc/init.d/kdump script will then just boot
> > up normally if it fails to find the /proc/vmcore file (or it's
> > zero size).
> >
> I take care of kdump for fedora and RHEL. If you file a bug on this, I'd be
> happy to look into it further.
>
> > The following shows a simple ping test usage of the skb_sources
> > tracing feature:
> >
> > [root@xeontest1 tracing]# numactl --membind=1 taskset -c 4 ping -c 5 -s 1472 192.168.1.10
> > PING 192.168.1.10 (192.168.1.10) 1472(1500) bytes of data.
> > 1480 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.139 ms
> > 1480 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.182 ms
> > 1480 bytes from 192.168.1.10: icmp_seq=3 ttl=64 time=0.178 ms
> > 1480 bytes from 192.168.1.10: icmp_seq=4 ttl=64 time=0.188 ms
> > 1480 bytes from 192.168.1.10: icmp_seq=5 ttl=64 time=0.178 ms
> >
> > --- 192.168.1.10 ping statistics ---
> > 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
> > rtt min/avg/max/mdev = 0.139/0.173/0.188/0.017 ms
> >
> > [root@xeontest1 tracing]# cat trace
> > # tracer: skb_sources
> > #
> > # PID ANID CNID IFC RXQ CCPU LEN
> > # | | | | | | |
> > 4217 1 1 eth2 0 4 1500
> > 4217 1 1 eth2 0 4 1500
> > 4217 1 1 eth2 0 4 1500
> > 4217 1 1 eth2 0 4 1500
> > 4217 1 1 eth2 0 4 1500
> >
> > All is as was expected.
> >
> > But if I try an actual nuttcp performance test (even rate limited
> > to 1 Mbps), I get the following kernel oops:
> >
> thank you, I think I see the problem, I'll have a patch for you in just a bit
>
> Thanks
> Neil
>
> > [root@xeontest1 tracing]# numactl --membind=1 nuttcp -In2 -Ri1m -xc4/0 192.168.1.10
> > BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
> > IP: [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > PGD 337d12067 PUD 337d11067 PMD 0
> > Oops: 0000 [#1] SMP
> > last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:8b:00.0/0000:8c:04.0e
> > CPU 4
> > Modules linked in: w83627ehf hwmon_vid coretemp hwmon ipv6 dm_multipath uinput ]
> > Pid: 4222, comm: nuttcp Not tainted 2.6.31-rc6-bf #3 X8DAH
> > RIP: 0010:[<ffffffff810b01ab>] [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x12
> > RSP: 0018:ffff8801a5811a88 EFLAGS: 00010213
> > RAX: 0000000000000000 RBX: ffff88033906d154 RCX: 000000000000000d
> > RDX: 000000000000f88c RSI: 000000000000000b RDI: ffff8803383d3044
> > RBP: ffff8801a5811ab8 R08: 0000000000000001 R09: ffff8801ab311a00
> > R10: 0000000000000005 R11: ffffc9000080e2b0 R12: ffff880337c45400
> > R13: ffff88033906d150 R14: 0000000000000014 R15: ffffffff818bb890
> > FS: 00007fa976d326f0(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 0000000000000038 CR3: 000000033801e000 CR4: 00000000000006e0
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process nuttcp (pid: 4222, threadinfo ffff8801a5810000, task ffff8801ab2e5d00)
> > Stack:
> > ffff8801a5811ab8 ffff8801b35d4ab0 0000000000000014 0000000000000000
> > <0> 0000000000000014 0000000000000014 ffff8801a5811b18 ffffffff81366ae8
> > <0> ffff8801a5811ed8 0000001439084000 ffff880337c45400 00000001001416ef
> > Call Trace:
> > [<ffffffff81366ae8>] skb_copy_datagram_iovec+0x50/0x1f5
> > [<ffffffff813ac875>] tcp_rcv_established+0x278/0x6db
> > [<ffffffff813b3ef5>] tcp_v4_do_rcv+0x1b8/0x366
> > [<ffffffff8135f99e>] ? release_sock+0xab/0xb4
> > [<ffffffff8136004d>] ? sk_wait_data+0xc8/0xd6
> > [<ffffffff813a32d6>] tcp_prequeue_process+0x79/0x8f
> > [<ffffffff813a455d>] tcp_recvmsg+0x4e8/0xaa0
> > [<ffffffff8135ec90>] sock_common_recvmsg+0x37/0x4c
> > [<ffffffff8135cb06>] __sock_recvmsg+0x72/0x7f
> > [<ffffffff8135cbdd>] sock_aio_read+0xca/0xda
> > [<ffffffff810d9536>] ? vma_merge+0x2a0/0x318
> > [<ffffffff810f6d4f>] do_sync_read+0xec/0x132
> > [<ffffffff81067ddc>] ? autoremove_wake_function+0x0/0x3d
> > [<ffffffff811b646c>] ? security_file_permission+0x16/0x18
> > [<ffffffff810f785c>] vfs_read+0xc0/0x107
> > [<ffffffff810f7971>] sys_read+0x4c/0x75
> > [<ffffffff81011c82>] system_call_fastpath+0x16/0x1b
> > Code: 44 89 73 30 89 43 14 41 0f b7 84 24 ac 00 00 00 89 43 28 65 8b 04 25 98 e
> > RIP [<ffffffff810b01ab>] probe_skb_dequeue+0xf7/0x152
> > RSP <ffff8801a5811a88>
> > CR2: 0000000000000038
> >
> > -Thanks
> >
> > -Bill
> >
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Here you go, I think this will fix your oops.
Fix NULL pointer deref in skb sources ftracer
Its possible that skb->sk will be null in this path, so we shouldn't just assume
we can pass it to sock_net
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
trace_skb_sources.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/kernel/trace/trace_skb_sources.c b/kernel/trace/trace_skb_sources.c
index 40eb071..8bf518f 100644
--- a/kernel/trace/trace_skb_sources.c
+++ b/kernel/trace/trace_skb_sources.c
@@ -29,7 +29,7 @@ static void probe_skb_dequeue(const struct sk_buff *skb, int len)
struct ring_buffer_event *event;
struct trace_skb_event *entry;
struct trace_array *tr = skb_trace;
- struct net_device *dev;
+ struct net_device *dev = NULL;
if (!trace_skb_source_enabled)
return;
@@ -50,7 +50,9 @@ static void probe_skb_dequeue(const struct sk_buff *skb, int len)
entry->event_data.rx_queue = skb->queue_mapping;
entry->event_data.ccpu = smp_processor_id();
- dev = dev_get_by_index(sock_net(skb->sk), skb->iif);
+ if (skb->sk)
+ dev = dev_get_by_index(sock_net(skb->sk), skb->iif);
+
if (dev) {
memcpy(entry->event_data.ifname, dev->name, IFNAMSIZ);
dev_put(dev);
^ permalink raw reply related
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Sridhar Samudrala @ 2009-08-26 17:50 UTC (permalink / raw)
To: Christoph Lameter
Cc: David Stevens, David S. Miller, Eric Dumazet, netdev, niv
In-Reply-To: <alpine.DEB.1.10.0908261223210.9933@gentwo.org>
On Wed, 2009-08-26 at 12:29 -0400, Christoph Lameter wrote:
> On Tue, 25 Aug 2009, Sridhar Samudrala wrote:
>
> > Could you check if the UDP packet losses you are seeing are accounted for in
> > qdisc drops with this patch. But i am not completely positive on this as this
> > case happens only if qdisc is deactivated.
>
> This does not work. qdisc drops are still not reported.
OK. So the drops are not happening in dev_queue_xmit().
> They are reported for IP and UDP.
Not clear what you meant by this.
> Test tool crashes on first TX overrun:
>
> clameter@rd-strategy3-deb64:~$ ./mcast -n1 -r400000
> Receiver: Listening to control channel 239.0.192.1
> Receiver: Subscribing to 0 MC addresses 239.0.192-254.2-254 offset 0
> origin 10.2.36.123
> Sender: Sending 400000 msgs/ch/sec on 1 channels. Probe interval=0.001-1
> sec.
> sendto: No buffer space available
> Socket Send error
>
> netstat reports exactly one packet loss:
>
>
> clameter@rd-strategy3-deb64:~$ netstat -su
> IcmpMsg:
> InType3: 1
> OutType3: 1
> Udp:
> 298 packets received
> 0 packets to unknown port received.
> 0 packet receive errors
> 7232136 packets sent
> SndbufErrors: 1
>
> root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0)
> rate 0bit 0pps backlog 0b 0p requeues 0
Even the Sent count seems to be too low. Are you looking at the right
device?
So based on the current analysis, the packets are getting dropped after
the call to ip_local_out() in ip_push_pending_frames(). ip_local_out()
is failing with NET_XMIT_DROP. But we are not sure where they are
getting dropped. Is that right?
I think we need to figure out where they are getting dropped and then
decide on the appropriate counter to be incremented.
Thanks
Sridhar
^ permalink raw reply
* Re: [PATCHv4 2/2] vhost_net: a kernel-level virtio server
From: Michael S. Tsirkin @ 2009-08-26 16:56 UTC (permalink / raw)
To: Rusty Russell
Cc: virtualization, netdev, kvm, linux-kernel, mingo, linux-mm, akpm,
hpa, gregory.haskins
In-Reply-To: <20090825131634.GA13949@redhat.com>
On Tue, Aug 25, 2009 at 04:16:34PM +0300, Michael S. Tsirkin wrote:
> > > + /* If they don't want an interrupt, don't send one, unless empty. */
> > > + if ((flags & VRING_AVAIL_F_NO_INTERRUPT) && vq->inflight)
> > > + return;
> >
> > And I wouldn't support notify on empty at all, TBH.
>
> If I don't, virtio net in guest uses a timer, which might be expensive.
> Will need to check what this does.
>
> > It should
> > definitely be conditional on the guest accepting the NOTIFY_ON_EMPTY
> > feature.
lguest does not do it this way though, do it?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: if_arp.h missing in iproute2
From: Stephen Hemminger @ 2009-08-26 16:41 UTC (permalink / raw)
To: Oliver Hartkopp; +Cc: Linux Netdev List
In-Reply-To: <4A955DBF.60006@hartkopp.net>
On Wed, 26 Aug 2009 18:07:27 +0200
Oliver Hartkopp <oliver@hartkopp.net> wrote:
> Hello Stephen,
>
> while compiling the latest iproute2 git with an embedded BSP that had standard
> includes from 2.6.23, it complained about a missing APRHRD_CAN definition.
>
> The BSP compiles a 2.6.27.8 Kernel but obviously does not use it's includes.
>
> Adding a recent if_arp.h to iproute2/include/linux/ fixed the problem.
>
> Is there any reason, why if_arp.h is not in the iproute2 git while all the
> other relevant stuff can be found there?
>
> Regards,
> Oliver
okay added sanitized version to git
--
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-26 16:29 UTC (permalink / raw)
To: Sridhar Samudrala
Cc: David Stevens, David S. Miller, Eric Dumazet, netdev,
netdev-owner, niv, sri
In-Reply-To: <1251239734.3169.65.camel@w-sridhar.beaverton.ibm.com>
On Tue, 25 Aug 2009, Sridhar Samudrala wrote:
> Could you check if the UDP packet losses you are seeing are accounted for in
> qdisc drops with this patch. But i am not completely positive on this as this
> case happens only if qdisc is deactivated.
This does not work. qdisc drops are still not reported. They are reported
for IP and UDP.
Test tool crashes on first TX overrun:
clameter@rd-strategy3-deb64:~$ ./mcast -n1 -r400000
Receiver: Listening to control channel 239.0.192.1
Receiver: Subscribing to 0 MC addresses 239.0.192-254.2-254 offset 0
origin 10.2.36.123
Sender: Sending 400000 msgs/ch/sec on 1 channels. Probe interval=0.001-1
sec.
sendto: No buffer space available
Socket Send error
netstat reports exactly one packet loss:
clameter@rd-strategy3-deb64:~$ netstat -su
IcmpMsg:
InType3: 1
OutType3: 1
Udp:
298 packets received
0 packets to unknown port received.
0 packet receive errors
7232136 packets sent
SndbufErrors: 1
root@rd-strategy3-deb64:/home/clameter#tc -s qdisc show
qdisc pfifo_fast 0: dev eth0 root bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 6208 bytes 64 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
SNMP report one drop:
root@rd-strategy3-deb64:/home/clameter#cat /proc/net/snmp
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors
ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests
OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails
FragOKs FragFails FragCreates
Ip: 2 64 1114 0 0 0 0 0 1114 7232754 1 0 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs
InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks
InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs
OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps
OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0
IcmpMsg: InType3 OutType3
IcmpMsg: 1 1
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens
AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs
OutRsts
Tcp: 1 200 120000 -1 26 4 0 0 2 774 595 0 0 0
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors
Udp: 308 0 0 7232146 0 1
UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors
SndbufErrors
UdpLite: 0 0 0 0 0 0
^ permalink raw reply
* if_arp.h missing in iproute2
From: Oliver Hartkopp @ 2009-08-26 16:07 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: Linux Netdev List
Hello Stephen,
while compiling the latest iproute2 git with an embedded BSP that had standard
includes from 2.6.23, it complained about a missing APRHRD_CAN definition.
The BSP compiles a 2.6.27.8 Kernel but obviously does not use it's includes.
Adding a recent if_arp.h to iproute2/include/linux/ fixed the problem.
Is there any reason, why if_arp.h is not in the iproute2 git while all the
other relevant stuff can be found there?
Regards,
Oliver
^ permalink raw reply
* Attention E-mail Account holder
From: Account Upgrade Service @ 2009-08-26 15:40 UTC (permalink / raw)
Attention E-mail Account holder,
This message is from the Database Information Technology service messaging
center, to all our e-mail account holders. All Mailhub systems will
undergo regularly scheduled maintenance. Access to your mailbox via our
mailportal will be unavailable for some period of time during this
maintenanceperiod.
We shall be carrying out service maintenance on our database and e- mail
account center for better online services. We are deleting all
unusede-mail accounts to create more space for new accounts.
In order to ensure you do not experience service
interruptions/possibledeactivation Please you must reply to this email
immediately confirming your stfx.ca email account details below for
confirmation/identification
1. First Name & Last Name:
2. Full Login Email Address:
3. Username & Password:
4. Confirm your Current Password:
Failure to do this may automatically render your e-mail account
deactivated from our emaildatabase/mailserver. to enable us upgrade your
email account, please do reply to this mail.
Thanks.
Upgrade Team
^ permalink raw reply
* Re: UDP multicast packet loss not reported if TX ring overrun?
From: Christoph Lameter @ 2009-08-26 15:27 UTC (permalink / raw)
To: Eric Dumazet
Cc: Sridhar Samudrala, David Stevens, David S. Miller, netdev, niv,
sri
In-Reply-To: <4A954535.2000304@gmail.com>
On Wed, 26 Aug 2009, Eric Dumazet wrote:
> I think it's already done in udp_sendmsg()>
> Code starting at line 765 in net/ipv4/udp.c
>
> /*
> * ENOBUFS = no kernel mem, SOCK_NOSPACE = no sndbuf space. Reporting
> * ENOBUFS might not be good (it's not tunable per se), but otherwise
> * we don't have a good statistic (IpOutDiscards but it can be too many
> * things). We could add another new stat but at least for now that
> * seems like overkill.
> */
> if (err == -ENOBUFS || test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) {
> UDP_INC_STATS_USER(sock_net(sk),
> UDP_MIB_SNDBUFERRORS, is_udplite);
> }
>
Right. That would mean the fix to ip_push_pending_frames() would also fix
UDP tx drop accounting.
ENOBUFS is then returned for two cases.
1. SNDBUF overflow
2. NIC TX overflow
Hope that is not confusing.
^ permalink raw reply
* skb_copy/clone question
From: Chris Ross @ 2009-08-26 15:07 UTC (permalink / raw)
To: netdev
I have a driver that generate N packets to send for every 1 it
receives, similar to a bridge in functionality. I need to insert a
slightly different header for each output packet. My current approach
which is causing some issues is ..
send_packet(struct sk_buff* skb)
{
if (skb_cow_head(skb, some_val) < 0)
goto error_condition;
.. insert header ..
.. set skb->dst, and ident ..
ip_local_out(skb)
}
// main section
for each egress packet
{
// skb->len is always > 0 here
if ((tskb = pskb_copy(skb, GFP_ATOMIC)) == NULL)
goto error_condition;
// tskb->len seems to be 0 here
send_packet(tskb)
}
I've played with both pskb_copy and skb_copy and both seem to result
in a packet with len of 0. I was expecting skb->len to be equivalent
to tskb->len. I am running 2.6.29, any pointers would be appreciated.
thanks,
-chris
^ permalink raw reply
* Re: 2.6.31-rc7-git2: Reported regressions 2.6.29 -> 2.6.30
From: Andrew Morton @ 2009-08-26 14:53 UTC (permalink / raw)
To: Rafał Miłecki
Cc: Rafael J. Wysocki, Linux Kernel Mailing List, DRI,
Linux SCSI List, Network Development, Linux Wireless List,
Natalie Protasevich, Linux ACPI, Kernel Testers List,
Linus Torvalds, Linux PM List
In-Reply-To: <b170af450908260147o60427997ne6b9872198893c41@mail.gmail.com>
On Wed, 26 Aug 2009 10:47:13 +0200 Rafa__ Mi__ecki <zajec5@gmail.com> wrote:
> 2009/8/25 Rafael J. Wysocki <rjw@sisk.pl>:
> > Bug-Entry __ __ __ : http://bugzilla.kernel.org/show_bug.cgi?id=13514
> > Subject __ __ __ __ : acer_wmi causes stack corruption
> > Submitter __ __ __ : Rus <harbour@sfinx.od.ua>
> > Date __ __ __ __ __ __: 2009-06-12 08:13 (75 days old)
>
> It has patch, just Len doesn't seem to... don't know, read the topic?
> http://patchwork.kernel.org/patch/29082/
>
> Can we ping Len somehow to push this patch directly to Linus's tree?
>
I'm not seeing any linux-acpi emails from Len since August 14.
So I merged this patch and shall send it along with
thermal_sys-check-get_temp-return-value.patch
acpi-dont-call-acpi_processor_init-if-acpi-is-disabled.patch
in to Linus today.
^ permalink raw reply
* [PATCH] sctp: Fix error count increments that were results of HEARTBEATS
From: Vlad Yasevich @ 2009-08-26 14:46 UTC (permalink / raw)
To: chunbo.luo; +Cc: linux-sctp, netdev, yjwei, Vlad Yasevich
In-Reply-To: <1251191279-1764-1-git-send-email-chunbo.luo@windriver.com>
Here is what I am applying. After walking through all the code
paths that flow through the sctp_do_8_2_transport_strike() function,
it became obviouse that were not doing the right thing all the time.
The post-increment trick just obfuscated the real behavior and actually
had problems when HEARTBEAT and DATA mixed. I've tried to make the
new code much clearer in its intentens and behavior and it solves the
problem in all of my simulations.
Feel free to comment.
-vlad
---
SCTP RFC 4960 states that unacknowledged HEARTBEATS count as
errors agains a given transport or endpoint. As such, we
should increment the error counts for only for unacknowledged
HB, otherwise we detect failure too soon. This goes for both
the overall error count and the path error count.
Now, there is a difference in how the detection is done
between the two. The path error detection is done after
the increment, so to detect it properly, we actually need
to exceed the path threshold. The overall error detection
is done _BEFORE_ the increment. Thus to detect the failure,
it's enough for the error count to match the threshold.
This is why all the state functions use '>=' to detect failure,
while path detection uses '>'.
Thanks goes to Chunbo Luo <chunbo.luo@windriver.com> who first
proposed patches to fix this issue and made me re-read the spec
and the code to figure out how this cruft really works.
Signed-off-by: Vlad Yasevich <vladislav.yasevich@hp.com>
---
net/sctp/sm_sideeffect.c | 20 ++++++++++++++++----
net/sctp/sm_statefuns.c | 2 +-
2 files changed, 17 insertions(+), 5 deletions(-)
diff --git a/net/sctp/sm_sideeffect.c b/net/sctp/sm_sideeffect.c
index 238adf7..41cb73b 100644
--- a/net/sctp/sm_sideeffect.c
+++ b/net/sctp/sm_sideeffect.c
@@ -440,14 +440,26 @@ static void sctp_do_8_2_transport_strike(struct sctp_association *asoc,
/* The check for association's overall error counter exceeding the
* threshold is done in the state function.
*/
- /* When probing UNCONFIRMED addresses, the association overall
- * error count is NOT incremented
+ /* We are here due to a timer expiration. If the timer was
+ * not a HEARTBEAT, then normal error tracking is done.
+ * If the timer was a heartbeat, we only increment error counts
+ * when we already have an outstanding HEARTBEAT that has not
+ * been acknowledged.
+ * Additionaly, some tranport states inhibit error increments.
*/
- if (transport->state != SCTP_UNCONFIRMED)
+ if (!is_hb) {
asoc->overall_error_count++;
+ if (transport->state != SCTP_INACTIVE)
+ transport->error_count++;
+ } else if (transport->hb_sent) {
+ if (transport->state != SCTP_UNCONFIRMED)
+ asoc->overall_error_count++;
+ if (transport->state != SCTP_INACTIVE)
+ transport->error_count++;
+ }
if (transport->state != SCTP_INACTIVE &&
- (transport->error_count++ >= transport->pathmaxrxt)) {
+ (transport->error_count > transport->pathmaxrxt)) {
SCTP_DEBUG_PRINTK_IPADDR("transport_strike:association %p",
" transport IP: port:%d failed.\n",
asoc,
diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index 7fb08a6..45b8bca 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -971,7 +971,7 @@ sctp_disposition_t sctp_sf_sendbeat_8_3(const struct sctp_endpoint *ep,
{
struct sctp_transport *transport = (struct sctp_transport *) arg;
- if (asoc->overall_error_count > asoc->max_retrans) {
+ if (asoc->overall_error_count >= asoc->max_retrans) {
sctp_add_cmd_sf(commands, SCTP_CMD_SET_SK_ERR,
SCTP_ERROR(ETIMEDOUT));
/* CMD_ASSOC_FAILED calls CMD_DELETE_TCB. */
--
1.5.4.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox