* [2.6.17-git22] lock debugging output
From: Alessandro Suardi @ 2006-07-04 18:13 UTC (permalink / raw)
To: Linux Kernel, netdev
Hoping gmail doesn't mess it too badly...
eth0: tg3 (BCM5751 Gbit Ethernet)
eth1: ipw2200 (Intel PRO/Wireless 2200BG)
Sequence:
1. boot with eth0 disconnected (eth1 doesn't come up on boot)
2. ifup eth1, bring wpa-supplicant up
3. run 'dig' ---> <lock debug info gets printed on console>
Note that due to my very variable network setup, I had no /etc/resolv.conf
in place at the moment I ran 'dig'. Second execution of 'dig' did not print
any lock debug output but just (properly) stalled; then I realized I didn't
put my home resolv.conf in place, did that and 'dig' just worked.
System appears to work and I'm actually typing this report from the
same kernel that reported the following upon invoking 'dig' :
=================================
[ INFO: inconsistent lock state ]
---------------------------------
inconsistent {softirq-on-W} -> {in-softirq-R} usage.
dig/2373 [HC0[0]:SC1[2]:HE1:SE0] takes:
(&sk->sk_dst_lock){---?}, at: [<c028cf72>] sk_dst_check+0x1b/0xe6
{softirq-on-W} state was registered at:
[<c0127a6a>] lock_acquire+0x60/0x80
[<c02e151d>] _write_lock+0x19/0x28
[<c028c0af>] sock_setsockopt+0x351/0x49c
[<c0289d0d>] sys_setsockopt+0x5b/0x8d
[<c028ac22>] sys_socketcall+0x148/0x186
[<c0102699>] sysenter_past_esp+0x56/0x8d
irq event stamp: 1130
hardirqs last enabled at (1130): [<c01161ed>] local_bh_enable_ip+0xb2/0xbb
hardirqs last disabled at (1129): [<c011618e>] local_bh_enable_ip+0x53/0xbb
softirqs last enabled at (1120): [<c029423c>] dev_queue_xmit+0x205/0x211
softirqs last disabled at (1121): [<c01040e6>] do_softirq+0x4d/0xac
other info that might help us debug this:
2 locks held by dig/2373:
#0: (sk_lock-AF_INET6){--..}, at: [<f8cf1168>]
udpv6_sendmsg+0x546/0x818 [ipv6]
#1: (slock-AF_INET6){-...}, at: [<f8cf3228>] icmpv6_send+0x222/0x549 [ipv6]
stack backtrace:
[<c0102e44>] show_trace+0xd/0x10
[<c010335e>] dump_stack+0x19/0x1b
[<c01260e1>] print_usage_bug+0x1cc/0x1d9
[<c01265e2>] mark_lock+0x193/0x360
[<c01271ee>] __lock_acquire+0x3b7/0x969
[<c0127a6a>] lock_acquire+0x60/0x80
[<c02e15ff>] _read_lock+0x19/0x28
[<c028cf72>] sk_dst_check+0x1b/0xe6
[<f8ce1305>] ip6_dst_lookup+0x31/0x16d [ipv6]
[<f8cf3338>] icmpv6_send+0x332/0x549 [ipv6]
[<f8cf09a1>] udpv6_rcv+0x4ab/0x4d6 [ipv6]
[<f8ce2900>] ip6_input+0x19c/0x228 [ipv6]
[<f8ce2d61>] ipv6_rcv+0x188/0x1b7 [ipv6]
[<c02925b7>] netif_receive_skb+0x18d/0x1d8
[<c0293d6a>] process_backlog+0x80/0xf9
[<c0293f43>] net_rx_action+0x80/0x174
[<c01162fd>] __do_softirq+0x46/0x9c
[<c01040e6>] do_softirq+0x4d/0xac
=======================
[<c0116117>] local_bh_enable+0xc8/0xec
[<c029423c>] dev_queue_xmit+0x205/0x211
[<c0298a8b>] neigh_resolve_output+0x1db/0x207
[<f8ce0bee>] ip6_output2+0x1e4/0x202 [ipv6]
[<f8ce12aa>] ip6_output+0x69e/0x6c8 [ipv6]
[<f8ce1706>] ip6_push_pending_frames+0x2c5/0x377 [ipv6]
[<f8cefd8e>] udp_v6_push_pending_frames+0x154/0x176 [ipv6]
[<f8cf122a>] udpv6_sendmsg+0x608/0x818 [ipv6]
[<c02c6b1d>] inet_sendmsg+0x3b/0x48
[<c02894f9>] sock_sendmsg+0xe8/0x103
[<c0289b18>] sys_sendmsg+0x14f/0x1aa
[<c028ac45>] sys_socketcall+0x16b/0x186
[<c0102699>] sysenter_past_esp+0x56/0x8d
Hope this may be useful to lock debug devs / netdev folks...
Ciao,
--alessandro
"I can't change what makes me high and I can't change what I believe in"
(Heather Nova, "My Fidelity")
^ permalink raw reply
* Re: [2.6.17-git22] lock debugging output
From: Arjan van de Ven @ 2006-07-04 18:32 UTC (permalink / raw)
To: Alessandro Suardi; +Cc: akpm, mingo, Linux Kernel, netdev
In-Reply-To: <5a4c581d0607041113o2993cbf5m7011b2a06e96d974@mail.gmail.com>
From: Arjan van de Ven <arjan@linux.intel.com>
On Tue, 2006-07-04 at 20:13 +0200, Alessandro Suardi wrote:
> Hoping gmail doesn't mess it too badly...
>
> eth0: tg3 (BCM5751 Gbit Ethernet)
> eth1: ipw2200 (Intel PRO/Wireless 2200BG)
>
> Sequence:
> 1. boot with eth0 disconnected (eth1 doesn't come up on boot)
> 2. ifup eth1, bring wpa-supplicant up
> 3. run 'dig' ---> <lock debug info gets printed on console>
this appears to be a real deadlock:
the SO_BINDTODEVICE ioctl calls sk_dst_reset(), which looks like this:
static inline void
sk_dst_reset(struct sock *sk)
{
write_lock(&sk->sk_dst_lock);
__sk_dst_reset(sk);
write_unlock(&sk->sk_dst_lock);
}
now... ipv6 does this in softirq context:
[<c028cf72>] sk_dst_check+0x1b/0xe6
[<f8ce1305>] ip6_dst_lookup+0x31/0x16d [ipv6]
[<f8cf3338>] icmpv6_send+0x332/0x549 [ipv6]
[<f8cf09a1>] udpv6_rcv+0x4ab/0x4d6 [ipv6]
[<f8ce2900>] ip6_input+0x19c/0x228 [ipv6]
[<f8ce2d61>] ipv6_rcv+0x188/0x1b7 [ipv6]
[<c02925b7>] netif_receive_skb+0x18d/0x1d8
[<c0293d6a>] process_backlog+0x80/0xf9
[<c0293f43>] net_rx_action+0x80/0x174
[<c01162fd>] __do_softirq+0x46/0x9c
[<c01040e6>] do_softirq+0x4d/0xac
where sk_dst_check() takes the same lock for read.
that looks like a real deadlock to me...
the most obvious low impact solution is to make sk_dst_reset use an
irqsave variant; patch for that is attached below. I'll leave it to the
networking people to say if that's the real right approach
Signed-off-by: Arjan van de Ven <arjan@linux.intel.com>
---
include/net/sock.h | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
Index: linux-2.6.17-mm6/include/net/sock.h
===================================================================
--- linux-2.6.17-mm6.orig/include/net/sock.h
+++ linux-2.6.17-mm6/include/net/sock.h
@@ -1025,9 +1025,10 @@ __sk_dst_reset(struct sock *sk)
static inline void
sk_dst_reset(struct sock *sk)
{
- write_lock(&sk->sk_dst_lock);
+ unsigned long flags;
+ write_lock_irqsave(&sk->sk_dst_lock, flags);
__sk_dst_reset(sk);
- write_unlock(&sk->sk_dst_lock);
+ write_unlock_irqrestore(&sk->sk_dst_lock, flags);
}
extern struct dst_entry *__sk_dst_check(struct sock *sk, u32 cookie);
^ permalink raw reply
* Re: RDMA will be reverted
From: Andy Gay @ 2006-07-04 18:34 UTC (permalink / raw)
To: Andi Kleen; +Cc: Tom Tucker, David Miller, rdreier, netdev, akpm
In-Reply-To: <200607011626.04539.ak@suse.de>
On Sat, 2006-07-01 at 16:26 +0200, Andi Kleen wrote:
> On Saturday 01 July 2006 01:01, Tom Tucker wrote:
> > On Fri, 2006-06-30 at 14:16 -0700, David Miller wrote:
> >
> > > The TOE folks have tried to submit their hooks and drivers
> > > on several occaisions, and we've rejected it every time.
> >
> > iWARP != TOE
>
> Perhaps a good start of that discussion David asked for would
> be if you could give us an overview of the differences
> and how you avoid the TOE problems.
Interesting thread, I hope someone replies to Andi's request.
I've actually no real idea what RDMA, IWARP & TOE are, so I may be
barking up completely the wrong tree here. If so, apologies.
But it sounds like we're talking about technologies that offload some
part of the network/transport layer processing to the interface device?
And the primary objection to that is that it may bypass some of the cool
features of the Linux stack? Stuff like iptables and ... what exactly?
Presumably the reason why such offloading would be a Good Thing are to
do with very high speed network processing, 10G ethernet and the like.
Which sounds to me very like the way dedicated networking kit would do
that. So if you have a device that needs to be a very high performance
router, you dedicate it to that function and don't try to do clever
per-packet or -flow processing at the same time.
In the Cisco world, there's a network design approach where you consider
your equipment in three 'layers', I think they call them the core,
distribution and access layers, or something like that. The idea is that
the core layer handles the real high speed stuff, you don't do anything
much except routing/switching in there. The other layers aggregate flows
for the core, if you need extra processing (firewalls etc) you do it
somewhere in there. So, for example, the packet capture functions (sort
of like tcpdump) don't work if fast switching is in use, which it would
be in the core. Users accept these tradeoffs, because if you design it
right you can do the extra processing on some other device in the outer
layers.
So perhaps there's a good argument to make that a Linux system with the
right hardware could be considered a core device. Likely any place you
have such a system it would be dedicated to just moving data as well as
possible, and let other systems do the other stuff. You wouldn't want
your server farm systems to also be your firewalls.
Bottom line - these technologies seem to me to have a place in a well
designed network.
Just my 2c...
- Andy
>
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [e1000]: flow control on by default - good idea really?
From: jamal @ 2006-07-04 19:20 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, jesse.brandeburg, Robert Olsson, john.ronciak, Ben Greear,
Auke Kok, Jeff Garzik
In-Reply-To: <1152033116.5276.22.camel@jzny2>
On Tue, 2006-04-07 at 13:11 -0400, jamal wrote:
> CCing anybody who may have stakes on this. Ignore the email if this
> doesnt interest you.
> Ok, folks - i had deferred this discussion but it bit me in the ass.
> I just spend an hour debugging it (and in the process blew up a gbic i
> borrowed, so my day aint going well since i actually have to pay for
> this and cant really do the testing i was planning to;-<).
>
> I have a device connected to a e1000 that was erroneously advertising
> both tx/rx flow control but wasnt properly reacting to it.
> The default setup on the e1000 has rx flow control turned on.
> I was sending at wire rate gige from the device - which is about
> 1.48Mpps. The e1000 was in turn sending me flow control packets
> as per default/expected behavior. Unfortunately, it was sending
> a very large amount of packets. At one point i was seeing upto
> 1Mpps and on average, the flow control packets were consuming
> 60-70% of the bandwidth. Even when i fixed this behavior to act
> properly, allowing flow control on consumed up to 15% of the bandwidth.
> Clearly, this is a bad thing. Yes, the device in the first instance was
> at fault. But i have argued in the past that NAPI does just fine without
> flow control being turned on, so even chewing 5% of bandwidth on flow
> control is a bad thing..
>
> As a compromise, can we declare flow control as an advanced feature
> and turn it off by default? People who feel it is valuable and know
> what they are doing can turn it off.
I meant turn it on.
BTW, As an addendum this default behavior changed around 2.6.16 it
seems.
cheers,
jamal
^ permalink raw reply
* Re: [Patch][RFC] Disabling per-tgid stats on task exit in taskstats
From: jamal @ 2006-07-04 19:24 UTC (permalink / raw)
To: Shailabh Nagar
Cc: pj, Valdis.Kletnieks, jlan, balbir, csturtiv, linux-kernel,
netdev, Andrew Morton
In-Reply-To: <44AA9951.1060804@watson.ibm.com>
Shailabh,
On Tue, 2006-04-07 at 12:37 -0400, Shailabh Nagar wrote:
[..]
> Here's a strawman for the problem we're trying to solve: get
> notification of the close of a NETLINK_GENERIC socket that had
> been used to register interest for some cpus within taskstats.
>
> From looking at the netlink code, the way to go seems to be
>
> - it maintains a pidhash of nl_pids that are currently
> registered to listen to atleast one cpu. It also stores the
> cpumask used.
> - taskstats registers a notifier block within netlink_chain
> and receives a callback on the NETLINK_URELEASE event, similar
> to drivers/scsci/scsi_transport_iscsi.c: iscsi_rcv_nl_event()
>
> - the callback checks to see that the protocol is NETLINK_GENERIC
> and that the nl_pid for the socket is in taskstat's pidhash. If so, it
> does a cleanup using the stored cpumask and releases the nl_pid
> from the pidhash.
>
Sound quiet reasonable. I am beginning to wonder whether we should do
do the NETLINK_URELEASE in general for NETLINK_GENERIC
> We can even do away with the deregister command altogether and
> simply rely on this autocleanup.
I think if you may still need the register if you are going to allow
multiple sockets per listener process, no?
The other question is how do you correlate pid -> fd?
cheers,
jamal
^ permalink raw reply
* Re: [PATCH 0/2] NET: Accurate packet scheduling for ATM/ADSL
From: jamal @ 2006-07-04 19:29 UTC (permalink / raw)
To: Patrick McHardy
Cc: Jesper Dangaard Brouer, netdev, Stephen Hemminger, Alan Cox,
Russell Stuart, Russell Stuart
In-Reply-To: <44AA6D25.9000707@trash.net>
On Tue, 2006-04-07 at 15:29 +0200, Patrick McHardy wrote:
> Russell Stuart wrote:
[..]
> > Without seeing your actual proposal it is difficult to
> > judge whether this is a reasonable trade-off or not.
> > Hopefully we will see your code soon. Do you have any
> > idea when?
>
> Unfortunately I still didn't got to cleaning them up, so I'm sending
> them in their preliminary state. Its not much that is missing, but
> the netem usage of skb->cb needs to be integrated better, I failed
> to move it to the qdisc_skb_cb so far because of circular includes.
> But nothing unfixable. I'm mostly interested if the current size-tables
> can express what you need for ATM, I wasn't able to understand the
> big comment in tc_core.c in your patch.
>
Looks good from within the range of "change within reason" of addressed
problem. The cb on the qdisc seems only usable for netem, correct?
Also while not unreasonable, i wasnt sure how qdisc_enqueue_root()
fit in the grand scheme of things for this change (it seemed out of
place).
cheers,
jamal
^ permalink raw reply
* Re: rtl8139: NETDEV WATCHDOG: eth0: transmit timed out
From: Marcus Better @ 2006-07-04 19:38 UTC (permalink / raw)
To: netdev
In-Reply-To: <20060630211604.GA5302@electric-eye.fr.zoreil.com>
Francois Romieu wrote:
> In a better world, you would narrow the suspect with a git bissect [1]
> between v2.6.15 and v2.6.16.
Will try. It may take some time...
Marcus
^ permalink raw reply
* Re: [Patch][RFC] Disabling per-tgid stats on task exit in taskstats
From: Paul Jackson @ 2006-07-04 19:59 UTC (permalink / raw)
To: Shailabh Nagar
Cc: akpm, Valdis.Kletnieks, jlan, balbir, csturtiv, linux-kernel,
hadi, netdev
In-Reply-To: <44A9B1A9.20106@watson.ibm.com>
Shailabh wrote:
> Perhaps I should use the the other ascii format for specifying cpumasks
> since its more amenable
> to specifying an upper bound for the length of the ascii string and is
> more compact ?
Eh - basically - I don't have a strong opinion either way.
I have a slight esthetic preference toward using list of ranges format
from shell scripts and shell prompts, and using the 32-bit hex words
from C code:
17-26,44-47 # shell - list of ranges
0000f000,07fe0000 # C - 32-bit hex words
Since the primary interface you are working with is C code, that would
mean I'd slightly prefer the 32-bit hex word variant.
>From what I've seen neither of the reasons you gave for preferring
the 32-bit hex word format are persuasive (even though they both
lead to the same conclusion as I preferred ;):
Which is more compact depends on that particular bit pattern
you need to represent. See for example the examples above.
The lack of a perfect upper bound on the list of ranges format
is a theoretical problem that I have never seen in practice.
Only pathological constructs exceed six ascii characters per
set bit.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply
* Re: 2.6.17-mm6
From: Arjan van de Ven @ 2006-07-04 20:01 UTC (permalink / raw)
To: netdev, Rafael J. Wysocki; +Cc: Andrew Morton, linux-kernel
In-Reply-To: <200607042153.31848.rjw@sisk.pl>
this is one for the networking people, and thus netdev
On Tue, 2006-07-04 at 21:53 +0200, Rafael J. Wysocki wrote:
> On Monday 03 July 2006 12:03, Andrew Morton wrote:
> >
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.17/2.6.17-mm6/
> >
> >
> > - A major update to the e1000 driver.
> >
> > - 1394 updates
>
> Just found this in dmesg:
>
> =================================
> [ INFO: inconsistent lock state ]
> ---------------------------------
> inconsistent {in-hardirq-W} -> {hardirq-on-W} usage.
> nscd/4929 [HC0[0]:SC0[1]:HE1:SE0] takes:
> (&skb_queue_lock_key){++..}, at: [<ffffffff8044fe40>] udp_ioctl+0x50/0xa0
> {in-hardirq-W} state was registered at:
> [<ffffffff8024b4fa>] lock_acquire+0x8a/0xc0
> [<ffffffff80476e3f>] _spin_lock_irqsave+0x3f/0x60
> [<ffffffff80408c25>] skb_queue_tail+0x25/0x60
ok so skb_queue_lock is used in a hardirq context
> [<ffffffff881c9517>] queue_packet_complete+0x27/0x40 [ieee1394]
> [<ffffffff881c9d6b>] hpsb_packet_sent+0xab/0x100 [ieee1394]
> [<ffffffff8822a4b5>] dma_trm_reset+0x115/0x140 [ohci1394]
> [<ffffffff8822c512>] ohci_devctl+0x1c2/0x540 [ohci1394]
> [<ffffffff881c9673>] hpsb_bus_reset+0x43/0xb0 [ieee1394]
> [<ffffffff8822d7f6>] ohci_irq_handler+0x416/0x830 [ohci1394]
> [<ffffffff802631ab>] handle_IRQ_event+0x2b/0x70
> [<ffffffff80264dd4>] handle_level_irq+0xc4/0x130
> [<ffffffff8020c762>] do_IRQ+0x112/0x130
> [<ffffffff80209d90>] common_interrupt+0x64/0x65
> irq event stamp: 4280
> hardirqs last enabled at (4279): [<ffffffff8047606a>] trace_hardirqs_on_thunk+0x35/0x37
> hardirqs last disabled at (4278): [<ffffffff804760a1>] trace_hardirqs_off_thunk+0x35/0x67
> softirqs last enabled at (4258): [<ffffffff804065b5>] release_sock+0xd5/0xe0
> softirqs last disabled at (4280): [<ffffffff804764d1>] _spin_lock_bh+0x11/0x50
>
> other info that might help us debug this:
> no locks held by nscd/4929.
>
> stack backtrace:
>
> Call Trace:
> [<ffffffff8020ab9f>] show_trace+0x9f/0x240
> [<ffffffff8020af75>] dump_stack+0x15/0x20
> [<ffffffff80249e52>] print_usage_bug+0x272/0x290
> [<ffffffff8024a0d7>] mark_lock+0x267/0x5f0
> [<ffffffff8024a9a6>] __lock_acquire+0x546/0xd10
> [<ffffffff8024b4fb>] lock_acquire+0x8b/0xc0
> [<ffffffff804764f4>] _spin_lock_bh+0x34/0x50
> [<ffffffff8044fe40>] udp_ioctl+0x50/0xa0
yet udp_ioctl takes it only for _bh
> [<ffffffff80457359>] inet_ioctl+0x69/0x70
> [<ffffffff804033ac>] sock_ioctl+0x22c/0x270
> [<ffffffff802a32b1>] do_ioctl+0x31/0xa0
> [<ffffffff802a35db>] vfs_ioctl+0x2bb/0x2e0
> [<ffffffff802a366a>] sys_ioctl+0x6a/0xa0
> [<ffffffff8020985a>] system_call+0x7e/0x83
> [<00002b2d76ab98a9>]
is this a real scenario, or is this a case of "firewire is special and
needs it's own rules"?
^ permalink raw reply
* Re: [Patch][RFC] Disabling per-tgid stats on task exit in taskstats
From: Paul Jackson @ 2006-07-04 20:19 UTC (permalink / raw)
To: Andrew Morton
Cc: nagar, Valdis.Kletnieks, jlan, balbir, csturtiv, linux-kernel,
hadi, netdev
In-Reply-To: <20060703144106.cc5bd6f6.akpm@osdl.org>
Andrew wrote:
> OK, so we're passing in an ASCII string. Fair enough, I think. Paul would
> know better.
Not sure if I know better - just got stronger opinions.
I like the ASCII here - but this is one of those "he who
writes the code gets to
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply
* Re: [Patch][RFC] Disabling per-tgid stats on task exit in taskstats
From: Paul Jackson @ 2006-07-04 20:22 UTC (permalink / raw)
To: Paul Jackson
Cc: akpm, nagar, Valdis.Kletnieks, jlan, balbir, csturtiv,
linux-kernel, hadi, netdev
In-Reply-To: <20060704131913.83a604a0.pj@sgi.com>
pj wrote:
> writes the code gets to
Never mind that last incomplete post - I hit Send
when I meant to hit Cancel.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
^ permalink raw reply
* Details inside(work at home).
From: Diann @ 2006-07-04 20:26 UTC (permalink / raw)
To: netdev
Hello Everyone!
You have recieved this e-mail from us, we wanted to notify you that we
are strating our new project and we have a lot to offer. For right now
you have an amazing opportunity to earn funds by helping us. There are
a lot of people who's seeking for a good job, but still coudln't find
any. We are right here right now, you have a chance to get into our
project and help us out, thus you will earn some good money. Consider
yourself lucky, this is really a great opportunity.
Ok, closer to the deal. Our project has different kinds of opporunities
and vacancies, you can take a look at our website for more information.
You will get interested. Plus you don't need any school degree, no
education, no past experience. This is a universal opportunity. You need to
make a registration on our website and you will be included into our
project and we will contact you within 24 hours and will tell you what's
going on on your side and what you need to do. Don't think you will not
be able to handle your job, it's really very easy and interesting job,
and our agents are very friendly. You have to be 18 years old or older.
We will put you on the minimal earning plan first, which is around 2000
United States dollars per week. After two or three deals you will get
raised.
Alright. Now the question is what do you really have to do. You have to
be able to work 1-3 hours a day. The best time is in the morning. Don't
worry about your current job, feel yourself comfortable and take care
of your things first and then think about this opportunity we give you.
But enthusiasm is always appreciated. Once again, the more you work
with us - the more you earn, and this is how it goes. For more information
visit our website: http://www.wdajob.com don't forget you need to make a
registration in order to start working with us. When you will get registered
we will contact you within 24 hours, please provide your valid phone
number. You are very welcome to join our project. Thank You and have a
Great Day!
Best Regards
^ permalink raw reply
* Re: RDMA will be reverted
From: Roland Dreier @ 2006-07-04 20:34 UTC (permalink / raw)
To: Andi Kleen; +Cc: Tom Tucker, David Miller, netdev, akpm
In-Reply-To: <200607011626.04539.ak@suse.de>
Andi> Perhaps a good start of that discussion David asked for
Andi> would be if you could give us an overview of the differences
Andi> and how you avoid the TOE problems.
Well, here's a quick overview, leaving out some of the details. The
difference between TOE and iWARP/RDMA is really the interface that
they present.
A TOE ("TCP Offload Engine") is a piece of hardware that offloads TCP
processing from the main system to handle regular sockets. There is
either some way to hand off a socket from the host stack to the TOE,
or a socket is created on the TOE to start with, but in both cases,
the TOE is handling processing for normal TCP sockets. This means
that the TOE has some hardware and/or firmware to do stateful TCP
processing.
An iWARP device, or RNIC (RDMA NIC), also usually has hardware and/or
firmware TCP processing, but this isn't exposed through the BSD socket
interface. Instead, an RNIC presents an interface more like an
InfiniBand HCA: work requests (sends, receives, RDMA operations) are
passed to the RNIC via work queues, and completion notification is
returned asynchronously via completion queues. An iWARP connection
can handle both send/receive ("two-sided") and get/put (RDMA or
"one-sided") operations.
For full details of the protocol used for this, you can look at the
drafs from the IETF rddp working group, but basically an RDMA protocol
is layered on top of a connected stream protocol -- usually TCP, but
SCTP could be used as well.
A lot of the perfomance of iWARP comes from the RDMA/direct placement
capabilities -- for example an NFS/RDMA server can process requests
out of order and put data directly into the buffer that's waiting for
it, without using any CPU on the destination -- but even send/receive
operations can be useful.
As a side note, an RNIC will also typically support the same sort of
kernel bypass as an IB HCA, where work queues can be safely mapped
into a userspace process's memory so that work requests can be posted
without a system call. In fact, when people usually use RDMA as a
shorthand for the combination of the three features I described:
asynchronous work queues and completion queues, connections that
support both send/receive and RDMA, and kernel bypass.
In any case, RNIC support can be added to the existing IB stack with
fairly minor modifications -- you can search the netdev archives for
the patchsets posted by Steve Wise, but nearly all of the new code is
in the low-level hardware driver for the specific iWARP devices.
The real issues for netdev are things like Steve Wise's patch to add
route change notifiers, which could be used to tell RNICs when to
update the next hop for a connection they're handling. More
generally, it would be interesting to see if it's possible to tie an
RNIC into the kernel's packet filtering, so that disallowed
connections don't get set up. This seems very similar in spirit to
the problems around packet filtering that were raised for VJ netchannels.
- Roland
^ permalink raw reply
* Re: RDMA will be reverted
From: Roland Dreier @ 2006-07-04 20:34 UTC (permalink / raw)
To: David Miller; +Cc: tom, netdev, akpm
In-Reply-To: <20060701.144506.74722126.davem@davemloft.net>
> Roland stated that it has never been the case that we have
> rejected adding support for a certain class of devices on the
> kinds of merits being discussed in this thread. And I'm saying
> that TOE is such a case where we have emphatically done so.
Well, in the past it's seemed more like patches have been rejected not
because of a blanket refusal to consider support for certain hardware,
but rather because of issues with the patches themselves. eg last
year when Chelsio submitted some TOE hooks, you wrote the following
<http://marc.theaimsgroup.com/?l=linux-netdev&m=112382991506811&w=2>
>> There is no way you're going to be allowed to call such deep TCP
>> internals from your driver.
>> This would mean that every time we wish to change the data structures
>> and interfaces for TCP socket lookup, your drivers would need to
>> change.
which looks like a very good reason to reject the changes.
> So I am not saying iWARP or RDMA is equal to TOE, and if you had
> actually read this thread you would have understood that.
There's definitely been quite a bit of conflation between the two in
this thread, even if you're not responsible...
- R.
^ permalink raw reply
* Re: RDMA will be reverted
From: Andi Kleen @ 2006-07-04 20:47 UTC (permalink / raw)
To: Andy Gay; +Cc: Tom Tucker, David Miller, rdreier, netdev, akpm
In-Reply-To: <1152038079.3285.586.camel@tahini.andynet.net>
> So perhaps there's a good argument to make that a Linux system with the
> right hardware could be considered a core device. Likely any place you
> have such a system it would be dedicated to just moving data as well as
> possible, and let other systems do the other stuff. You wouldn't want
> your server farm systems to also be your firewalls.
Why not? While Linux firewall performance is not flawless its problems
(e.g. slow conntrack) seems to be mostly in an area where TOE cannot
do much about.
> Bottom line - these technologies seem to me to have a place in a well
> designed network.
I think there is a web page listing why it's bad, but here
a quick summary:
One worry is to debug it all together. Currently we have a single stack
to debug, although it's already difficult to control the complexity as it
grows more bells and whistles.
Just take a look at Cisco IOS release notes to see how hard
and difficult it is to get it all to work together.
Another reason is that there are general doubts that TOE can
keep up with the ever growing performance of CPUs. Even if Linux
added it today it would be likely slower again a few months later.
That is also a big difference to Cisco hardware. Linux usually
runs on fast main CPUs (or if you run it on slow CPUs you normally
don't expect the best network performance). And they get faster
and faster constantly.
Admittedly 10GB NICs are still a bit too fast for
mainstream systems, but that seems to be mostly a problem
outside the CPUs and it looks like the next generation
of systems will catch up with enough bandwidth in this area.
Also it tends to accelerate the wrong thing. On a lot of workloads
the main problem is keeping a lot of different connections under
control, and TOE tends to be slow at keeping connection
information synchronized with the host.
That is why the Linux strategy has been to ask for useful stateless offloads
instead. Examples of this are checksum offload (long time classic), TSO (TCP
segmentation offload), UFO (UDP segmentation offload), Intel iOAT (memcpy off
load), RX hashing with MSI-X (not implemented yet, but basically
it allows load balancing of incoming streams to CPU)
Note that all these are more or less stateless offloads.
iWARP is not clear yet what it is. From the meager bits of information
about it that reached netdev so far it at least sounds it does RDMA and needs
far more state than any of the other offloads we got so far and likely
got the usual TOE scaling issues. It's also likely on the wrong side
of Moore's law.
-Andi
^ permalink raw reply
* [mini-RFT] via-velocity cleanup
From: Francois Romieu @ 2006-07-04 21:20 UTC (permalink / raw)
To: netdev; +Cc: Jay Cliburn
Against 2.6.17:
http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.17/via-velocity/
The mii operations look now more familiar. There should be no functional
change. The patches do not clash with Jeff's netdev-2.6#upstream.
Please report if I have broken something.
--
Ueimor
^ permalink raw reply
* [PATCH 0/3] Action API fixes
From: Thomas Graf @ 2006-07-04 22:05 UTC (permalink / raw)
To: davem; +Cc: netdev, hadi
Dave,
Fixes for some rather serious action API bugs. Please apply.
net/sched/act_api.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
^ permalink raw reply
* [PATCH 1/3] [PKT_SCHED]: Fix illegal memory dereferences when dumping actions
From: Thomas Graf @ 2006-07-04 22:00 UTC (permalink / raw)
To: davem; +Cc: netdev, hadi
In-Reply-To: <20060704220504.787776000@postel.suug.ch>
[-- Attachment #1: act_fix_dump_null_deref --]
[-- Type: text/plain, Size: 1707 bytes --]
The TCA_ACT_KIND attribute is used without checking its
availability when dumping actions therefore leading to a
value of 0x4 being dereferenced.
The use of strcmp() in tc_lookup_action_n() isn't safe
when fed with string from an attribute without enforcing
proper NUL termination.
Both bugs can be triggered with malformed netlink message
and don't require any privileges.
Signed-off-by: Thomas Graf <tgraf@suug.ch>
Index: net-2.6.git/net/sched/act_api.c
===================================================================
--- net-2.6.git.orig/net/sched/act_api.c
+++ net-2.6.git/net/sched/act_api.c
@@ -776,7 +776,7 @@ replay:
return ret;
}
-static char *
+static struct rtattr *
find_dump_kind(struct nlmsghdr *n)
{
struct rtattr *tb1, *tb2[TCA_ACT_MAX+1];
@@ -804,7 +804,7 @@ find_dump_kind(struct nlmsghdr *n)
return NULL;
kind = tb2[TCA_ACT_KIND-1];
- return (char *) RTA_DATA(kind);
+ return kind;
}
static int
@@ -817,16 +817,15 @@ tc_dump_action(struct sk_buff *skb, stru
struct tc_action a;
int ret = 0;
struct tcamsg *t = (struct tcamsg *) NLMSG_DATA(cb->nlh);
- char *kind = find_dump_kind(cb->nlh);
+ struct rtattr *kind = find_dump_kind(cb->nlh);
if (kind == NULL) {
printk("tc_dump_action: action bad kind\n");
return 0;
}
- a_o = tc_lookup_action_n(kind);
+ a_o = tc_lookup_action(kind);
if (a_o == NULL) {
- printk("failed to find %s\n", kind);
return 0;
}
@@ -834,7 +833,7 @@ tc_dump_action(struct sk_buff *skb, stru
a.ops = a_o;
if (a_o->walk == NULL) {
- printk("tc_dump_action: %s !capable of dumping table\n", kind);
+ printk("tc_dump_action: %s !capable of dumping table\n", a_o->kind);
goto rtattr_failure;
}
--
^ permalink raw reply
* [PATCH 2/3] [PKT_SCHED]: Return ENOENT if action module is unavailable
From: Thomas Graf @ 2006-07-04 22:00 UTC (permalink / raw)
To: davem; +Cc: netdev, hadi
In-Reply-To: <20060704220504.787776000@postel.suug.ch>
[-- Attachment #1: act_fix_init_ret_val --]
[-- Type: text/plain, Size: 367 bytes --]
Signed-off-by: Thomas Graf <tgraf@suug.ch>
Index: net-2.6.git/net/sched/act_api.c
===================================================================
--- net-2.6.git.orig/net/sched/act_api.c
+++ net-2.6.git/net/sched/act_api.c
@@ -305,6 +305,7 @@ struct tc_action *tcf_action_init_1(stru
goto err_mod;
}
#endif
+ *err = -ENOENT;
goto err_out;
}
--
^ permalink raw reply
* [PATCH 3/3] [PKT_SCHED]: Fix error handling while dumping actions
From: Thomas Graf @ 2006-07-04 22:00 UTC (permalink / raw)
To: davem; +Cc: netdev, hadi
In-Reply-To: <20060704220504.787776000@postel.suug.ch>
[-- Attachment #1: act_fix_dump_err_handling --]
[-- Type: text/plain, Size: 932 bytes --]
"return -err" and blindly inheriting the error code in the netlink
failure exception handler causes errors codes to be returned as
positive value therefore making them being ignored by the caller.
May lead to sending out incomplete netlink messages.
Signed-off-by: Thomas Graf <tgraf@suug.ch>
Index: net-2.6.git/net/sched/act_api.c
===================================================================
--- net-2.6.git.orig/net/sched/act_api.c
+++ net-2.6.git/net/sched/act_api.c
@@ -250,15 +250,17 @@ tcf_action_dump(struct sk_buff *skb, str
RTA_PUT(skb, a->order, 0, NULL);
err = tcf_action_dump_1(skb, a, bind, ref);
if (err < 0)
- goto rtattr_failure;
+ goto errout;
r->rta_len = skb->tail - (u8*)r;
}
return 0;
rtattr_failure:
+ err = -EINVAL;
+errout:
skb_trim(skb, b - skb->data);
- return -err;
+ return err;
}
struct tc_action *tcf_action_init_1(struct rtattr *rta, struct rtattr *est,
--
^ permalink raw reply
* Re: RDMA will be reverted
From: Andy Gay @ 2006-07-04 22:22 UTC (permalink / raw)
To: Andi Kleen; +Cc: Tom Tucker, David Miller, rdreier, netdev, akpm
In-Reply-To: <200607042247.12296.ak@suse.de>
On Tue, 2006-07-04 at 22:47 +0200, Andi Kleen wrote:
> > So perhaps there's a good argument to make that a Linux system with the
> > right hardware could be considered a core device. Likely any place you
> > have such a system it would be dedicated to just moving data as well as
> > possible, and let other systems do the other stuff. You wouldn't want
> > your server farm systems to also be your firewalls.
>
> Why not? While Linux firewall performance is not flawless its problems
> (e.g. slow conntrack) seems to be mostly in an area where TOE cannot
> do much about.
No doubt you *can* do this, but would you want to?
My point wasn't really about performance here, more that systems needing
this level of performance (server farm is just an example) will probably
be on an 'inside' network with firewalling being done elsewhere (at the
access layer, to use the Cisco paradigm). It's just not good design to
attach such systems directly to an untrusted network, IMHO. So these
systems just don't need netfilter capabilities.
>
> > Bottom line - these technologies seem to me to have a place in a well
> > designed network.
>
> I think there is a web page listing why it's bad, but here
> a quick summary:
>
> One worry is to debug it all together. Currently we have a single stack
> to debug, although it's already difficult to control the complexity as it
> grows more bells and whistles.
>
> Just take a look at Cisco IOS release notes to see how hard
> and difficult it is to get it all to work together.
No argument there!
>
> Another reason is that there are general doubts that TOE can
> keep up with the ever growing performance of CPUs. Even if Linux
> added it today it would be likely slower again a few months later.
> That is also a big difference to Cisco hardware. Linux usually
> runs on fast main CPUs (or if you run it on slow CPUs you normally
> don't expect the best network performance). And they get faster
> and faster constantly.
>
> Admittedly 10GB NICs are still a bit too fast for
> mainstream systems, but that seems to be mostly a problem
> outside the CPUs and it looks like the next generation
> of systems will catch up with enough bandwidth in this area.
>
> Also it tends to accelerate the wrong thing. On a lot of workloads
> the main problem is keeping a lot of different connections under
> control, and TOE tends to be slow at keeping connection
> information synchronized with the host.
>
> That is why the Linux strategy has been to ask for useful stateless offloads
> instead. Examples of this are checksum offload (long time classic), TSO (TCP
> segmentation offload), UFO (UDP segmentation offload), Intel iOAT (memcpy off
> load), RX hashing with MSI-X (not implemented yet, but basically
> it allows load balancing of incoming streams to CPU)
>
> Note that all these are more or less stateless offloads.
>
> iWARP is not clear yet what it is. From the meager bits of information
> about it that reached netdev so far it at least sounds it does RDMA and needs
> far more state than any of the other offloads we got so far and likely
> got the usual TOE scaling issues. It's also likely on the wrong side
> of Moore's law.
>
> -Andi
^ permalink raw reply
* Re: [RFC] change netdevice to use struct device instead of struct class_device
From: Greg KH @ 2006-07-04 22:31 UTC (permalink / raw)
To: David Miller; +Cc: jeff, netdev, linux-kernel, akpm
In-Reply-To: <20060703.185747.74753207.davem@davemloft.net>
On Mon, Jul 03, 2006 at 06:57:47PM -0700, David Miller wrote:
> From: Greg KH <greg@kroah.com>
> Date: Mon, 3 Jul 2006 16:16:10 -0700
>
> > No, not really. According to Documentation/ABI/testing/sysfs-class all
> > code that uses /sys/class/foo/ needs to be able to handle the fact that
> > those entries might be symlinks and not just directories. Everything
> > that I know of already works properly because the input layer has had
> > symlinks in /sys/class/input for quite some time now.
> >
> > Do you know of any tools that use /sys/class/net/ that can not handle
> > symlinks there? I've been running this on my boxes for about a week now
> > with no noticeable issues. Renaming interfaces works just fine too.
>
> I do not think this change will cause any problems.
Great, thanks for looking.
Do you mind if I keep this in my tree, due to the dependancies on the
other driver core changes?
thanks,
greg k-h
^ permalink raw reply
* Fw: [Bugme-new] [Bug 6791] New: ports in SA should not be zeroed when protocol is specified
From: Andrew Morton @ 2006-07-04 22:41 UTC (permalink / raw)
To: netdev; +Cc: klimes, bugme-daemon@kernel-bugs.osdl.org
Begin forwarded message:
Date: Tue, 4 Jul 2006 06:54:01 -0700
From: bugme-daemon@bugzilla.kernel.org
To: bugme-new@lists.osdl.org
Subject: [Bugme-new] [Bug 6791] New: ports in SA should not be zeroed when protocol is specified
http://bugzilla.kernel.org/show_bug.cgi?id=6791
Summary: ports in SA should not be zeroed when protocol is
specified
Kernel Version: 2.6.17 and all previous
Status: NEW
Severity: normal
Owner: shemminger@osdl.org
Submitter: klimes@centrum.cz
Problem Description:
Security assotiations manipulation is defined by RFC2367 - PFKEY API.
However this specification has some drawbacks and unnecessary limitations.
(Maybe RFC2367 would deserve updating as it is rather old.)
Specifically there is a need for an IPsec SA which supports simultaneously TCP
and UDP. Such behaviour can be required by many applications and is e.g.
neccessary for authenticating in 3G according to 3GPP TS 33.203.
Current implementation of PFkey in linux (af_key.c) nethertheless doesn't
support usage of ports as SA selectors and moreover, if a protocol is specified,
the ports are zeroed.
I have found that a bug on this issue has been reported against solaris and has
been corrected - see bugID 6258318
(http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6258318)
So, I would appreciate to have similar behaviour in linux as well to be competitive.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
^ permalink raw reply
* (unknown),
From: Neal Sidhwaney @ 2006-07-04 22:55 UTC (permalink / raw)
To: netdev
subscribe linux-netdev
---
^ permalink raw reply
* Re: RDMA will be reverted
From: Andi Kleen @ 2006-07-04 23:01 UTC (permalink / raw)
To: Andy Gay; +Cc: Tom Tucker, David Miller, rdreier, netdev, akpm
In-Reply-To: <1152051753.3285.600.camel@tahini.andynet.net>
> My point wasn't really about performance here, more that systems needing
> this level of performance (server farm is just an example) will probably
> be on an 'inside' network with firewalling being done elsewhere (at the
> access layer, to use the Cisco paradigm). It's just not good design to
> attach such systems directly to an untrusted network, IMHO. So these
> systems just don't need netfilter capabilities.
Don't think of the highend. It is exotic and rare.
Think of the ordinary single linux box somewhere at a rackspace provider which
represents the majority of Linux boxes around.
With a not too skilled admin who mostly uses the default settings of his configuration.
For that running firewalling on the same box makes a lot of sense.
Normally it is not that loaded and it doesn't matter much how it performs,
but it might be occasionally slashdotted and then it should still hold up.
BTW basic firewalling is not really that bad as long as you don't have too many
rules. Mostly conntrack is painful right now. I'm sure at some point it will
be fixed too.
-Andi
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox