* Re: [PATCH v3 13/17] lockd: use new hashtable implementation
From: Sasha Levin @ 2012-08-22 17:32 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: snitzer-H+wXaHxf7aLQT0dZR+AlfA, neilb-l3A5Bk7waGM,
fweisbec-Re5JQEeQqe8AvxtiuMwx3w,
Trond.Myklebust-HgOvQuBEEgTQT0dZR+AlfA, J. Bruce Fields,
paul.gortmaker-CWA4WttNNZF54TAoqtyWWQ,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, agk-H+wXaHxf7aLQT0dZR+AlfA,
aarcange-H+wXaHxf7aLQT0dZR+AlfA, rds-devel-N0ozoZBvEnrZJqsBc5GL+g,
eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w,
venkat.x.venkatsubra-QHcLZuEGTsvQT0dZR+AlfA,
ccaulfie-H+wXaHxf7aLQT0dZR+AlfA, mingo-X9Un+BFzKDI,
dev-yBygre7rU0TnMu66kgdUjQ, ericvh-Re5JQEeQqe8AvxtiuMwx3w,
josh-iaAMLnmF4UmaiuxdJuQwMA, rostedt-nx8X9YLhiw1AfugRpC6u6w,
lw-BthXqXjhjHXQFUHtdCDX3A, teigland-H+wXaHxf7aLQT0dZR+AlfA,
axboe-tSWWG44O7X1aa/9Udqfwiw, linux-nfs-u79uwXL29TY76Z2rM5mHXA,
edumazet-hpIqsD4AKlfQT0dZR+AlfA, linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
netdev-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, ejt-H+wXaHxf7aLQT0dZR+AlfA,
ebiederm-aS9lmoZGLiVWk0Htik3J/w, tj-DgEjT+Ai2ygdnm+yROfE0A,
akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
torvalds-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
davem-fT/PcQaiUtIeIZ0/mPfg9Q
In-Reply-To: <20120822132243.GA2844@Krystal>
On 08/22/2012 03:22 PM, Mathieu Desnoyers wrote:
> * Sasha Levin (levinsasha928-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) wrote:
>> On 08/22/2012 01:47 PM, J. Bruce Fields wrote:
>>> On Wed, Aug 22, 2012 at 04:27:08AM +0200, Sasha Levin wrote:
>>>> +static int __init nlm_init(void)
>>>> +{
>>>> + hash_init(nlm_files);
>>>> + return 0;
>>>> +}
>>>> +
>>>> +module_init(nlm_init);
>>>
>>> That's giving me:
>>>
>>> fs/lockd/svcsubs.o: In function `nlm_init':
>>> /home/bfields/linux-2.6/fs/lockd/svcsubs.c:454: multiple definition of `init_module'
>>> fs/lockd/svc.o:/home/bfields/linux-2.6/fs/lockd/svc.c:606: first defined here
>>> make[2]: *** [fs/lockd/lockd.o] Error 1
>>> make[1]: *** [fs/lockd] Error 2
>>> make[1]: *** Waiting for unfinished jobs....
>>
>> I tested this entire patch set both with linux-next and Linus' latest master,
>> and it worked fine in both places.
>>
>> Is it possible that lockd has a -next tree which isn't pulled into linux-next?
>> (there's nothing listed in MAINTAINERS that I could see).
>
> fs/lockd/Makefile:
>
> obj-$(CONFIG_LOCKD) += lockd.o
>
> lockd-objs-y := clntlock.o clntproc.o clntxdr.o host.o svc.o svclock.o \
> svcshare.o svcproc.o svcsubs.o mon.o xdr.o grace.o
>
> your patch adds a module_init to svcsubs.c.
> However, there is already one in svc.c, pulled into the same module.
>
> in your test build, is CONFIG_LOCKD defined as "m" or "y" ? You should
> always test both.
>
> One solution here is to create a "local" init function in svcsubs.c and
> expose it to svc.c, so the latter can call it from its module init
> function.
Ah yes, it was on =y and I didn't notice :/
I'll fix that.
> Thanks,
>
> Mathieu
>
^ permalink raw reply
* Re: [PATCH 1/1] tcp: Wrong timeout for SYN segments
From: H.K. Jerry Chu @ 2012-08-22 17:29 UTC (permalink / raw)
To: Eric Dumazet
Cc: Alex Bergmann, davem, netdev, linux-kernel, Jerry Chu,
Neal Cardwell, Nandita Dukkipati
In-Reply-To: <1345629806.5158.933.camel@edumazet-glaptop>
On Wed, Aug 22, 2012 at 3:03 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Wed, 2012-08-22 at 12:00 +0200, Eric Dumazet wrote:
>> On Wed, 2012-08-22 at 11:29 +0200, Alex Bergmann wrote:
>>
>> > Actual 6 SYN frames are sent. The initial one and 5 retries.
>> >
>>
>> first one had a t0 + 0 delay. How can it count ???
>>
>> > The kernel is waiting another 32 seconds for a SYN+ACK and then gives
>> > the ETIMEDOUT back to userspace.
>> >
>> > Do you mean that we have to send another SYN packet after the 3 minutes?
>> >
>>
>> First SYN is not a retransmit
>>
>> R2 = time_of_last_SYN - time_of_initial_SYN (t0) = 31
>>
>> If you read RFC it states :
>>
>> "In particular, R2 for a SYN segment MUST
>> be set large enough to provide retransmission of the segment
>> for at least 3 minutes."
>>
>>
>> That means that the last _retransmit_ MUST happen after 180 seconds.
>>
>> And not :
>>
>> Send all the restransmits at t0 + 1, then wait 180 seconds before giving
>> connect() a timeout indication.
>>
>>
>
> Therefore, the minimal connect() timeout should be : 180 + 100 seconds
>
> (allowing 100 seconds for the SYNACKs sent in answer of the very last
> retransmit to come back)
>
> (100 seconds is the R2 for non SYN frames)
>
> RFC quote : The value of R2 SHOULD
> correspond to at least 100 seconds.
I agree if you take RFC1122 literally the last retransmission must
happen no less than 3 minutes from the 1st SYN... Oh actually it'd be
3 minutes plus initRTO because the 3 minutes applies only to
"retransmission" as in
"R2 for a SYN segment MUST be set large enough to provide retransmission
of the segment for at least 3 minutes.:
But IMHO 6 retries providing 1+2+4+8+16+32 = 63 secs retransmission plus
64 secs wait time totaling 127 secs is really plenty enough.
You have a good point on SYN-ACK.
Jerry
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Ben Hutchings @ 2012-08-22 17:27 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Linus Torvalds, David Laight, Benjamin LaHaise, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <50351304.20608@zytor.com>
On Wed, 2012-08-22 at 10:12 -0700, H. Peter Anvin wrote:
> On 08/22/2012 10:09 AM, Ben Hutchings wrote:
> >
> > Well, sure, I'm assuming that the driver is responsible for checking
> > that the device and its bus interface support an MMIO of the requested
> > width.
> >
> > But the architecture code must be responsible for reporting whether the
> > host supports it, right?
> >
>
> No, the architecture code *can't*.
So, let me check that I understand this right:
- To support 32-bit architectures, a driver should include one of two
different definitions of readq/writeq depending on which order the
device needs to receive 32-bit operations.
- On 64-bit architectures (or at least x86_64), the system might split
up readq/writeq into 32-bit operations in unspecified order, and the
driver can't control this.
If this is right, how can it be safe to use readq/writeq at all?
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Linus Torvalds @ 2012-08-22 17:26 UTC (permalink / raw)
To: Ben Hutchings
Cc: H. Peter Anvin, David Laight, Benjamin LaHaise, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <1345655343.2709.56.camel@bwh-desktop.uk.solarflarecom.com>
On Wed, Aug 22, 2012 at 10:09 AM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
>
> But the architecture code must be responsible for reporting whether the
> host supports it, right?
How? It's impossible. As far as the CPU is concerned, the writes
happen atomically. Look how you use it: you don't even query
dynamically about whether the stuff outside the CPU can handle atomic
128-bit writes. You just assume it at compile-time with an #ifdef. How
the hell do you expect that to be able to then say "oh, on this
machine the device you are doing the access to is behind an odd
PCI->PCIe bridge that will split the access"?
Not that we even tend to *know* those kinds of things. It's really
esoteric chipset knowledge. I wouldn't even expect it to be
necessarily documented in the chipset docs, it *might* be in some
NDA'd BIOS writer's guide thing.
You don't even seem to realize that things like the Intel FSB was
patented and wasn't fully documented by Intel at all? And that's for a
bus interface that was used for over a decade from the dominant CPU
manufacturer. What do you think happens with odd random chipsets? Who
do you expects to know?
I *suspect* that 128-bit writes would generally make it intact over
PCIe in real life, but I absolutely wouldn't guarantee it on all
machines. Exactly because of issues like "what happens with a nVidia
host bridge and the old FSB model on older Intel chips?" or "What does
the AMD memory pipeline do?".
Many CPU cores have 64-bit buses even *internally*, much less
externally. Yes, they have atomicity guarantees in their architecture
manual, but go look at it: those talk about memory accesses, and they
are based on cache coherency (these days - they *used* to be based on
certain bus guarantees). The MMIO side is a completely different
animal, and is still based on the bus - and nobody documents that,
afaik.
Linus
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: H. Peter Anvin @ 2012-08-22 17:12 UTC (permalink / raw)
To: Ben Hutchings
Cc: Linus Torvalds, David Laight, Benjamin LaHaise, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <1345655343.2709.56.camel@bwh-desktop.uk.solarflarecom.com>
On 08/22/2012 10:09 AM, Ben Hutchings wrote:
>
> Well, sure, I'm assuming that the driver is responsible for checking
> that the device and its bus interface support an MMIO of the requested
> width.
>
> But the architecture code must be responsible for reporting whether the
> host supports it, right?
>
No, the architecture code *can't*.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Ben Hutchings @ 2012-08-22 17:09 UTC (permalink / raw)
To: Linus Torvalds
Cc: H. Peter Anvin, David Laight, Benjamin LaHaise, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <CA+55aFxMD-Jn3fPgum7a+3EhNiWB-Mjv+1yamR7ELHwroA_+7Q@mail.gmail.com>
On Wed, 2012-08-22 at 09:55 -0700, Linus Torvalds wrote:
> On Wed, Aug 22, 2012 at 9:44 AM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
> >
> > Well, when the issue of 64-bit MMIO was discussed earlier this year, you
> > said nothing about this. I thought the conclusion was that any
> > definitions provided by <asm/io.h> *must* be atomic and drivers can use
> > <asm-generic/io-64-nonatomic-hi-lo.h> or
> > <asm-generic/io-64-nonatomic-lo-hi.h> as a fallback.
>
> Think 32-bit PCI with a 64-bit CPU.
[...]
Well, sure, I'm assuming that the driver is responsible for checking
that the device and its bus interface support an MMIO of the requested
width.
But the architecture code must be responsible for reporting whether the
host supports it, right?
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: H. Peter Anvin @ 2012-08-22 16:59 UTC (permalink / raw)
To: Linus Torvalds
Cc: Ben Hutchings, David Laight, Benjamin LaHaise, David Miller, tglx,
mingo, netdev, linux-net-drivers, x86
In-Reply-To: <CA+55aFwTyT8v9r7jW2=jp29_yMMacWjMwCka+_YktOOMzBHWBw@mail.gmail.com>
On 08/22/2012 09:51 AM, Linus Torvalds wrote:
> On Wed, Aug 22, 2012 at 8:54 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>>
>> Sorry, you fail. There are definitely systems in the field where readq()
>> and writeq() are implemented, because the CPU supports them, where the
>> fabric does not guarantee they are intact.
>
> Indeed.
>
> It's unlikely to be an issue with a PCIe driver, though. I'm pretty
> sure you can rely on 64-bit transfers there, especially with a CPU
> that is modern enough to run 64-bit mode.
>
> That said, even with PCIe, I wonder if older CPU's (think Intel with a
> front-side bus, rather than PCIe on die) necessarily always do 128-bit
> writes. The FSB is just 64 bits wide, and I could *imagine* that a
> PCIe chipset behind the FSB might end up just always generating at
> most 64-bit PCIe transactions for host accesses just because that
> would be "natural".
>
> Sounds unlikely, but hey, hardware sometimes does odd things.
>
I'm wondering how e.g. a K8 would work (CPU -> HT -> PCIe) on UC memory
there. I know for a fact that some CPU cores break up SSE transactions
into 64-bit transactions.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Linus Torvalds @ 2012-08-22 16:55 UTC (permalink / raw)
To: Ben Hutchings
Cc: H. Peter Anvin, David Laight, Benjamin LaHaise, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <1345653844.2709.51.camel@bwh-desktop.uk.solarflarecom.com>
On Wed, Aug 22, 2012 at 9:44 AM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
>
> Well, when the issue of 64-bit MMIO was discussed earlier this year, you
> said nothing about this. I thought the conclusion was that any
> definitions provided by <asm/io.h> *must* be atomic and drivers can use
> <asm-generic/io-64-nonatomic-hi-lo.h> or
> <asm-generic/io-64-nonatomic-lo-hi.h> as a fallback.
Think 32-bit PCI with a 64-bit CPU.
The CPU itself does the 64-bit access no problem. The bus? Not so
much. Even if it's a burst transaction with a single packet, the
actual device on the other side will see the 64-bit value as two
separate parts. Sometimes that matters, sometimes it doesn't (ask
yourself: "What's the atomicity guarantee at the device end? Burst
transaction or individual word of a transaction?").
Again, being limited to PCIe, you are unlikely to hit these issues,
but system bridges can do odd things sometimes, and in the *general*
case it's definitely true that "writeq()" can generate multiple
accesses at the device end even if the *CPU* only generated a single
one.
Linus
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Linus Torvalds @ 2012-08-22 16:51 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Ben Hutchings, David Laight, Benjamin LaHaise, David Miller, tglx,
mingo, netdev, linux-net-drivers, x86
In-Reply-To: <50350098.6030100@zytor.com>
On Wed, Aug 22, 2012 at 8:54 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>
> Sorry, you fail. There are definitely systems in the field where readq()
> and writeq() are implemented, because the CPU supports them, where the
> fabric does not guarantee they are intact.
Indeed.
It's unlikely to be an issue with a PCIe driver, though. I'm pretty
sure you can rely on 64-bit transfers there, especially with a CPU
that is modern enough to run 64-bit mode.
That said, even with PCIe, I wonder if older CPU's (think Intel with a
front-side bus, rather than PCIe on die) necessarily always do 128-bit
writes. The FSB is just 64 bits wide, and I could *imagine* that a
PCIe chipset behind the FSB might end up just always generating at
most 64-bit PCIe transactions for host accesses just because that
would be "natural".
Sounds unlikely, but hey, hardware sometimes does odd things.
Linus
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: H. Peter Anvin @ 2012-08-22 16:49 UTC (permalink / raw)
To: Ben Hutchings
Cc: David Laight, Benjamin LaHaise, Linus Torvalds, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <1345653844.2709.51.camel@bwh-desktop.uk.solarflarecom.com>
On 08/22/2012 09:44 AM, Ben Hutchings wrote:
>>
>> Sorry, you fail. There are definitely systems in the field where
>> readq() and writeq() are implemented, because the CPU supports them,
>> where the fabric does not guarantee they are intact.
>
> Well, when the issue of 64-bit MMIO was discussed earlier this year, you
> said nothing about this. I thought the conclusion was that any
> definitions provided by <asm/io.h> *must* be atomic and drivers can use
> <asm-generic/io-64-nonatomic-hi-lo.h> or
> <asm-generic/io-64-nonatomic-lo-hi.h> as a fallback.
>
That is true at the exit interface from the CPU core. Beyond that
drivers have to keep in mind the possible limitations of the
communications fabric between the CPU and the device.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: [PATCH 1/1] tcp: Wrong timeout for SYN segments
From: H.K. Jerry Chu @ 2012-08-22 16:44 UTC (permalink / raw)
To: Alex Bergmann; +Cc: davem, netdev, linux-kernel
In-Reply-To: <503419D3.1080700@linlab.net>
On Tue, Aug 21, 2012 at 4:29 PM, Alex Bergmann <alex@linlab.net> wrote:
> Hi David,
>
> I'm not 100% sure, but it looks like I found an RFC mismatch with the
> current default values of the TCP implementation.
>
> Alex
>
> From 8b854a525eb45f64ad29dfab16f9d9f681e84495 Mon Sep 17 00:00:00 2001
> From: Alexander Bergmann <alex@linlab.net>
> Date: Wed, 22 Aug 2012 00:29:08 +0200
> Subject: [PATCH 1/1] tcp: Wrong timeout for SYN segments
>
> Commit 9ad7c049 changed the initRTO from 3secs to 1sec in accordance to
> RFC6298 (former RFC2988bis). This introduced a gap with RFC1122 that
> defines a minimum retransmission window for SYN segments of at least
> 180secs.
>
> Prior to 9ad7c049 the timeout was defined with 189secs. Now we have only
> a timeout of 63secs.
>
> ((2 << 5) - 1) * 3 secs = 189 secs
> ((2 << 5) - 1) * 1 secs = 63 secs
>
> To fulfill the MUST constraint in RFC1122 section 4.2.3.5 about R2 for
> SYN segments, the values of TCP_SYN_RETRIES and TCP_SYNACK_RETRIES must
> be changed to 7 reties.
>
> ((2 << 7) - 1) * 1 secs = 255 secs
>
> This would result in an ETIMEDOUT of 4 minutes 15 seconds.
This issue occurred to me right after I submitted the patch for RFC6298.
I did not commit any more change because RFC compliance aside, 180secs
just seem like eternity in the Internet age.
(See my past post on this at
http://marc.info/?l=linux-netdev&m=130759078118866&w=2)
Jerry
>
> Signed-off-by: Alexander Bergmann <alex@linlab.net>
> ---
> include/net/tcp.h | 4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/include/net/tcp.h b/include/net/tcp.h
> index 1f000ff..7eaae19 100644
> --- a/include/net/tcp.h
> +++ b/include/net/tcp.h
> @@ -98,10 +98,10 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
> * 15 is ~13-30min depending on RTO.
> */
>
> -#define TCP_SYN_RETRIES 5 /* number of times to retry active opening a
> +#define TCP_SYN_RETRIES 7 /* number of times to retry active opening a
> * connection: ~180sec is RFC minimum */
>
> -#define TCP_SYNACK_RETRIES 5 /* number of times to retry passive opening a
> +#define TCP_SYNACK_RETRIES 7 /* number of times to retry passive opening a
> * connection: ~180sec is RFC minimum */
>
> #define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
> --
> 1.7.8.6
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Ben Hutchings @ 2012-08-22 16:44 UTC (permalink / raw)
To: H. Peter Anvin
Cc: David Laight, Benjamin LaHaise, Linus Torvalds, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <50350098.6030100@zytor.com>
On Wed, 2012-08-22 at 08:54 -0700, H. Peter Anvin wrote:
> On 08/22/2012 08:51 AM, Ben Hutchings wrote:
> >>
> >> FWIW can you even guarantee to do an atomic 64bit PCIe transfer
> >> on many systems (without resorting to a dma unit).
> >
> > On any architecture that implements readq and writeq these had better be
> > atomic.
> >
>
> Sorry, you fail. There are definitely systems in the field where
> readq() and writeq() are implemented, because the CPU supports them,
> where the fabric does not guarantee they are intact.
Well, when the issue of 64-bit MMIO was discussed earlier this year, you
said nothing about this. I thought the conclusion was that any
definitions provided by <asm/io.h> *must* be atomic and drivers can use
<asm-generic/io-64-nonatomic-hi-lo.h> or
<asm-generic/io-64-nonatomic-lo-hi.h> as a fallback.
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: Best way to set kernel thread affinity for handling a socket?
From: Ben Hutchings @ 2012-08-22 16:21 UTC (permalink / raw)
To: Roland Dreier; +Cc: netdev
In-Reply-To: <CAG4TOxNMRR_-bYJV1a_je2HKrpmVuGubH9ks4GdG8Shwa92K0w@mail.gmail.com>
On Wed, 2012-08-22 at 09:10 -0700, Roland Dreier wrote:
> Hi everyone,
>
> Let's say I have kernel code that's sitting in a loop doing
> kernel_accept() on a TCP socket. As each connection comes in, it
> forks off a kernel thread to deal with that socket.
>
> If I have a modern NIC with RSS and multiple queues, each TCP flow is
> going to be steered to one queue, which is probably bound to one CPU.
> So when I fork off that kernel thread, I'd like to bind it to the CPU
> where its NIC queues are going to be processed. My question is, how
> do I find out which CPU that is? Is there anything in the new socket
> structure I get back from kernel_accept() that I can look at to know
> which CPU the packets came in on?
With RFS we try to do the reverse: move the packets to match the socket
user. But it's not (yet) turned on by default. See
Documentation/networking/scaling.txt
> I'm thinking about this in the context of the kernel's iSCSI target
> code (drivers/target/iscsi), which creates threads to handle each
> iSCSI connection and sets their CPU affinity pretty much randomly
> (well, based on some "thread id", cf iscsit_thread_get_cpumask()).
Why set the affinity at all?
> And with a modern NIC, this leads to packets being received on one CPU
> but the data being consumed on another CPU, all the time, which is
> obviously far from optimal.
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Best way to set kernel thread affinity for handling a socket?
From: Roland Dreier @ 2012-08-22 16:10 UTC (permalink / raw)
To: netdev
Hi everyone,
Let's say I have kernel code that's sitting in a loop doing
kernel_accept() on a TCP socket. As each connection comes in, it
forks off a kernel thread to deal with that socket.
If I have a modern NIC with RSS and multiple queues, each TCP flow is
going to be steered to one queue, which is probably bound to one CPU.
So when I fork off that kernel thread, I'd like to bind it to the CPU
where its NIC queues are going to be processed. My question is, how
do I find out which CPU that is? Is there anything in the new socket
structure I get back from kernel_accept() that I can look at to know
which CPU the packets came in on?
I'm thinking about this in the context of the kernel's iSCSI target
code (drivers/target/iscsi), which creates threads to handle each
iSCSI connection and sets their CPU affinity pretty much randomly
(well, based on some "thread id", cf iscsit_thread_get_cpumask()).
And with a modern NIC, this leads to packets being received on one CPU
but the data being consumed on another CPU, all the time, which is
obviously far from optimal.
Thanks!
Roland
^ permalink raw reply
* Re: [PATCH 1/2] ipv6: do not hold route table lock when send ndisc probe
From: Banerjee, Debabrata @ 2012-08-22 16:04 UTC (permalink / raw)
To: Cong Wang, Debabrata Banerjee
Cc: netdev@vger.kernel.org, David S. Miller, Hideaki YOSHIFUJI,
Patrick McHardy
In-Reply-To: <1345520643.12468.6.camel@cr0>
Thanks for the patch. We're discussing how to reach to this code properly
in the lab now. Although we will probably have to modify so it's compliant
with the RFC, by checking if a ndisc_send_ns() has already been queued
within rt->rt6i_idev->cnf.rtr_probe_interval, otherwise it could flood the
network with neighbor discoveries.
-Debabrata
On 8/20/12 11:44 PM, "Cong Wang" <amwang@redhat.com> wrote:
>Hi, Debabrata,
>
>Could you help to test the attached patch below?
>
>Thanks!
>
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: H. Peter Anvin @ 2012-08-22 15:54 UTC (permalink / raw)
To: Ben Hutchings
Cc: David Laight, Benjamin LaHaise, Linus Torvalds, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <1345650689.2709.32.camel@bwh-desktop.uk.solarflarecom.com>
On 08/22/2012 08:51 AM, Ben Hutchings wrote:
>>
>> FWIW can you even guarantee to do an atomic 64bit PCIe transfer
>> on many systems (without resorting to a dma unit).
>
> On any architecture that implements readq and writeq these had better be
> atomic.
>
Sorry, you fail. There are definitely systems in the field where
readq() and writeq() are implemented, because the CPU supports them,
where the fabric does not guarantee they are intact.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: [PATCH] libceph: Fix sparse warning
From: Sage Weil @ 2012-08-22 15:53 UTC (permalink / raw)
To: Daniel Baluta; +Cc: davem, ceph-devel, netdev, Iulius Curt
In-Reply-To: <CAEnQRZBcX7vmaa2rc3-CcCSpayKZca2p11_3eSQaEsj_FAQLNg@mail.gmail.com>
On Wed, 22 Aug 2012, Daniel Baluta wrote:
> On Tue, Aug 14, 2012 at 4:27 PM, Iulius Curt <iulius.curt@gmail.com> wrote:
> > From: Iulius Curt <iulius.curt@gmail.com>
> >
> > Make ceph_monc_do_poolop() static to remove the following sparse warning:
> > * net/ceph/mon_client.c:616:5: warning: symbol 'ceph_monc_do_poolop' was not
> > declared. Should it be static?
> >
> > Signed-off-by: Iulius Curt <icurt@ixiacom.com>
> > ---
> > net/ceph/mon_client.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c
> > index 105d533..3875c60 100644
> > --- a/net/ceph/mon_client.c
> > +++ b/net/ceph/mon_client.c
> > @@ -613,7 +613,7 @@ bad:
> > /*
> > * Do a synchronous pool op.
> > */
> > -int ceph_monc_do_poolop(struct ceph_mon_client *monc, u32 op,
> > +static int ceph_monc_do_poolop(struct ceph_mon_client *monc, u32 op,
> > u32 pool, u64 snapid,
> > char *buf, int len)
> > {
> > --
> > 1.7.9.5
> >
> > --
>
> Hi Sage,
>
> Can you have a look on this? :)
Sorry, this one fell through the cracks. Yes, we can switch it to static,
but while we're doing that let's drop the ceph_monc_ prefix too (since
it's private).
Thanks!
sage
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: H. Peter Anvin @ 2012-08-22 15:51 UTC (permalink / raw)
To: David Laight
Cc: Ben Hutchings, Benjamin LaHaise, Linus Torvalds, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <AE90C24D6B3A694183C094C60CF0A2F6026B6FC1@saturn3.aculab.com>
On 08/22/2012 08:27 AM, David Laight wrote:
>> Your architecture sounds similar to one I once worked on (Orion
>> Microsystems CNIC/OPA-2). That architecture had a descriptor ring in
>> device memory, and a single trigger bit would move the head pointer.
>>
>> We used write combining to write out a set of descriptors, and then
>> used
>> a non-write-combining write to do the final write which bumps the head
>> pointer. The UC write flushes the write combiners ahead of it, so it
>> ends up with two transactions (one for the WC data and one for the UC
>> trigger) but it could frequently push quite a few descriptors in that
>> operation.
>
> The code actually looks more like a normal ethernet ring interface
> with an 'owner' bit in each entry.
> So it is important to write the owner bit last.
>
> It might be possibly to set multiple ring entries in two TLPs
> by first writing all of them (maybe with write combining)
> but without changing the ownership of the first entry.
> Then doing a second transfer to update the owner bit it
> the first entry.
> The order of the writes in the first transfer would then not
> matter.
>
The design flaw in that kind of design would be the need to set the
owner bit on every entry. Now, in the case of CNIC/OPA-2 support for
write combining was an explicit design goal, so writes are inert until
the trigger bit is written, at which point the head pointer is moved to
the entry containing the trigger bit. Very effective.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* RE: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Ben Hutchings @ 2012-08-22 15:51 UTC (permalink / raw)
To: David Laight
Cc: H. Peter Anvin, Benjamin LaHaise, Linus Torvalds, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <AE90C24D6B3A694183C094C60CF0A2F6026B6FC1@saturn3.aculab.com>
On Wed, 2012-08-22 at 16:27 +0100, David Laight wrote:
> > Your architecture sounds similar to one I once worked on (Orion
> > Microsystems CNIC/OPA-2). That architecture had a descriptor ring in
> > device memory, and a single trigger bit would move the head pointer.
> >
> > We used write combining to write out a set of descriptors, and then
> > used
> > a non-write-combining write to do the final write which bumps the head
> > pointer. The UC write flushes the write combiners ahead of it, so it
> > ends up with two transactions (one for the WC data and one for the UC
> > trigger) but it could frequently push quite a few descriptors in that
> > operation.
>
> The code actually looks more like a normal ethernet ring interface
> with an 'owner' bit in each entry.
> So it is important to write the owner bit last.
You're confused. The 'owner' field in the descriptor pointer is part of
the memory protection mechanism for user-level networking. And we don't
have up to 1024 TX descriptors in a single ring, we have up to 1024
separate rings - in host memory, of course. Which is why we have the
'TX push' feature to reduce latency for a currently empty TX queue.
> It might be possibly to set multiple ring entries in two TLPs
> by first writing all of them (maybe with write combining)
> but without changing the ownership of the first entry.
> Then doing a second transfer to update the owner bit it
> the first entry.
> The order of the writes in the first transfer would then not
> matter.
>
> FWIW can you even guarantee to do an atomic 64bit PCIe transfer
> on many systems (without resorting to a dma unit).
On any architecture that implements readq and writeq these had better be
atomic.
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: H. Peter Anvin @ 2012-08-22 15:49 UTC (permalink / raw)
To: David Laight
Cc: Ben Hutchings, Benjamin LaHaise, Linus Torvalds, David Miller,
tglx, mingo, netdev, linux-net-drivers, x86
In-Reply-To: <AE90C24D6B3A694183C094C60CF0A2F6026B6FC1@saturn3.aculab.com>
On 08/22/2012 08:27 AM, David Laight wrote:
>> Your architecture sounds similar to one I once worked on (Orion
>> Microsystems CNIC/OPA-2). That architecture had a descriptor ring in
>> device memory, and a single trigger bit would move the head pointer.
>>
>> We used write combining to write out a set of descriptors, and then
>> used
>> a non-write-combining write to do the final write which bumps the head
>> pointer. The UC write flushes the write combiners ahead of it, so it
>> ends up with two transactions (one for the WC data and one for the UC
>> trigger) but it could frequently push quite a few descriptors in that
>> operation.
>
> The code actually looks more like a normal ethernet ring interface
> with an 'owner' bit in each entry.
> So it is important to write the owner bit last.
>
> It might be possibly to set multiple ring entries in two TLPs
> by first writing all of them (maybe with write combining)
> but without changing the ownership of the first entry.
> Then doing a second transfer to update the owner bit it
> the first entry.
> The order of the writes in the first transfer would then not
> matter.
>
> FWIW can you even guarantee to do an atomic 64bit PCIe transfer
> on many systems (without resorting to a dma unit).
>
On many systems, perhaps, but I suspect that 32 bits is the maximum you
can truly guarantee.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Ben Hutchings @ 2012-08-22 15:41 UTC (permalink / raw)
To: Linus Torvalds
Cc: H. Peter Anvin, David Miller, tglx, mingo, netdev,
linux-net-drivers, x86
In-Reply-To: <CA+55aFxoWLad9vtfAzQ7qXWcspL92D5s5+MFJ-HfyH3LVqfpaA@mail.gmail.com>
On Wed, 2012-08-22 at 07:50 -0700, Linus Torvalds wrote:
> On Wed, Aug 22, 2012 at 7:24 AM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
> >
> > Sorry, I'll paste it below.
>
> The thing you pasted isn't actually the thing in the subject line.
> It's just you *using* it.
>
> I wanted to see what that "writeo()" looks like for x86-64.
>
> But I got google to find it for me by looking for "__raw_writeo", so I
> can see the patch now. It looks like it might work. Does it really
> help performance despite always doing those TS games in CR0 for each
> access?
I haven't run the experiment myself, but my colleagues observed a net
reduction of 100s of nanoseconds of latency. That may not sound like
much, but for a small packet traversing an uncongested twinax link it's
around 5-10% of the total latency from the descriptor pointer write to
DMA completion on the peer.
Later, you wrote:
> Btw, are we even certain that a 128-bit PCIe write is going to remain
> atomic across a bus (ie over various PCIe bridges etc)?
I don't think PCIe bridges are allowed to split up TLPs (this is why the
PCI core has to be so careful about programming Max Payload Size). What
happens between the processor core and the host bridge is another
matter, though.
> Do you you
> care? Is it just a "one transaction is cheaper than two", and it
> doesn't really have any ordering constraints? If the thing gets split
> into two 64-bit transactions (in whatever order) by a bridge on the
> way, would that be ok?
We care if the two transactions are not in ascending address order;
that's why we had to abandon write combining.
> We've seen buses split accesses before (ie 64-bit writes on 32-bit
> PCI).
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* Re: NULL deref in bnx2 / crashes ? ( was: netconsole leads to stalled CPU task )
From: Cong Wang @ 2012-08-22 15:40 UTC (permalink / raw)
To: netdev
In-Reply-To: <CAF6-1L7nRoG10P=+QRb=LfL4O8K877zUD6L6d5EoCq-QNt1FWA@mail.gmail.com>
On Wed, 22 Aug 2012 at 14:29 GMT, Sylvain Munaut <s.munaut@whatever-company.com> wrote:
> Hi,
>
>> my patch was incomplete, sorry :
>>
>> diff --git a/net/core/netpoll.c b/net/core/netpoll.c
>> index 346b1eb..ddc453b 100644
>> --- a/net/core/netpoll.c
>> +++ b/net/core/netpoll.c
>> @@ -335,8 +335,13 @@ void netpoll_send_skb_on_dev(struct netpoll *np, struct sk_buff *skb,
>> /* don't get messages out of order, and no recursion */
>> if (skb_queue_len(&npinfo->txq) == 0 && !netpoll_owner_active(dev)) {
>> struct netdev_queue *txq;
>> + int queue_index = skb_get_queue_mapping(skb);
>>
>> - txq = netdev_get_tx_queue(dev, skb_get_queue_mapping(skb));
>> + if (queue_index >= dev->real_num_tx_queues) {
>> + queue_index = 0;
>> + skb_set_queue_mapping(skb, 0);
>> + }
>> + txq = netdev_get_tx_queue(dev, queue_index);
>>
>> /* try until next clock tick */
>> for (tries = jiffies_to_usecs(1)/USEC_PER_POLL;
>
> Ok, I tried this.
>
> The machine with the intel card still hard freeze (no output / no nothing ...)
> The machine with the bnx2 don't crash anymore and no NULL deref, but
> the modprobe still hangs and I get this every 180 sec or so :
>
Thanks for reporting this!
I will try to see if I can reproduce it on KVM guest tomorrow.
Need to go to sleep now.
To be honest, I never try to setup netconsole on such a complex NIC,
bridge on vlan tagged bonding.
^ permalink raw reply
* TIPC handling of ethernet mac address change
From: Chris Friesen @ 2012-08-22 15:36 UTC (permalink / raw)
To: Allan Stephens, Jon Maloy, netdev
Hi,
I'm pretty new to tipc so pardon me if I use the wrong terminology. :)
I'm using TIPC 1.7.7 and bonding. I'm seeing some strange behaviour
when the bond MAC address changes--the tipc LINK_CONFIG messages still
contain the old MAC address embedded in them. This causes the other end
to send back to us on the wrong MAC and the packets get dropped.
I see recv_notification() has a case for NETDEV_CHANGEADDR so it looks
like MAC address change was intended to be handled, but I'm not entirely
clear on where the MAC address is supposed to be changed.
Any assistance on debugging/patching this would be appreciated.
Chris
--
Chris Friesen
Software Designer
3500 Carling Avenue
Ottawa, Ontario K2H 8E9
www.genband.com
^ permalink raw reply
* RE: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: David Laight @ 2012-08-22 15:27 UTC (permalink / raw)
To: H. Peter Anvin, Ben Hutchings
Cc: Benjamin LaHaise, Linus Torvalds, David Miller, tglx, mingo,
netdev, linux-net-drivers, x86
In-Reply-To: <5034F725.2090802@zytor.com>
> Your architecture sounds similar to one I once worked on (Orion
> Microsystems CNIC/OPA-2). That architecture had a descriptor ring in
> device memory, and a single trigger bit would move the head pointer.
>
> We used write combining to write out a set of descriptors, and then
> used
> a non-write-combining write to do the final write which bumps the head
> pointer. The UC write flushes the write combiners ahead of it, so it
> ends up with two transactions (one for the WC data and one for the UC
> trigger) but it could frequently push quite a few descriptors in that
> operation.
The code actually looks more like a normal ethernet ring interface
with an 'owner' bit in each entry.
So it is important to write the owner bit last.
It might be possibly to set multiple ring entries in two TLPs
by first writing all of them (maybe with write combining)
but without changing the ownership of the first entry.
Then doing a second transfer to update the owner bit it
the first entry.
The order of the writes in the first transfer would then not
matter.
FWIW can you even guarantee to do an atomic 64bit PCIe transfer
on many systems (without resorting to a dma unit).
David
^ permalink raw reply
* Re: [PATCH 2/3] x86_64: Define 128-bit memory-mapped I/O operations
From: Linus Torvalds @ 2012-08-22 15:16 UTC (permalink / raw)
To: David Laight
Cc: Ben Hutchings, H. Peter Anvin, David Miller, tglx, mingo, netdev,
linux-net-drivers, x86
In-Reply-To: <AE90C24D6B3A694183C094C60CF0A2F6026B6FC0@saturn3.aculab.com>
On Wed, Aug 22, 2012 at 8:05 AM, David Laight <David.Laight@aculab.com> wrote:
>
> PCIe transfers are basically hdlc packets containing the address,
> command and any associated data. Unless they get bridged
> though some strange PCIe<->PCI<->PCIe system they are very
> unlikely to get broken up.
It's exactly the odd kind of "mix in a non-native PCIe bridge" setups
I'd worry about. But I guess that is pretty unlikely in any modern
machine (except for thunderbolt, and I think that's going to pass any
PCIe stuff through unchanged).
Linus
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox