Best way to set kernel thread affinity for handling a socket?

Best way to set kernel thread affinity for handling a socket?

[Roland Dreier]

Hi everyone,

Let's say I have kernel code that's sitting in a loop doing
kernel_accept() on a TCP socket.  As each connection comes in, it
forks off a kernel thread to deal with that socket.

If I have a modern NIC with RSS and multiple queues, each TCP flow is
going to be steered to one queue, which is probably bound to one CPU.
So when I fork off that kernel thread, I'd like to bind it to the CPU
where its NIC queues are going to be processed.  My question is, how
do I find out which CPU that is?  Is there anything in the new socket
structure I get back from kernel_accept() that I can look at to know
which CPU the packets came in on?

I'm thinking about this in the context of the kernel's iSCSI target
code (drivers/target/iscsi), which creates threads to handle each
iSCSI connection and sets their CPU affinity pretty much randomly
(well, based on some "thread id", cf iscsit_thread_get_cpumask()).
And with a modern NIC, this leads to packets being received on one CPU
but the data being consumed on another CPU, all the time, which is
obviously far from optimal.

Thanks!
  Roland

Re: Best way to set kernel thread affinity for handling a socket?

[Ben Hutchings]

On Wed, 2012-08-22 at 09:10 -0700, Roland Dreier wrote:
> Hi everyone,
> 
> Let's say I have kernel code that's sitting in a loop doing
> kernel_accept() on a TCP socket.  As each connection comes in, it
> forks off a kernel thread to deal with that socket.
> 
> If I have a modern NIC with RSS and multiple queues, each TCP flow is
> going to be steered to one queue, which is probably bound to one CPU.
> So when I fork off that kernel thread, I'd like to bind it to the CPU
> where its NIC queues are going to be processed.  My question is, how
> do I find out which CPU that is?  Is there anything in the new socket
> structure I get back from kernel_accept() that I can look at to know
> which CPU the packets came in on?

With RFS we try to do the reverse: move the packets to match the socket
user.   But it's not (yet) turned on by default.  See
Documentation/networking/scaling.txt

> I'm thinking about this in the context of the kernel's iSCSI target
> code (drivers/target/iscsi), which creates threads to handle each
> iSCSI connection and sets their CPU affinity pretty much randomly
> (well, based on some "thread id", cf iscsit_thread_get_cpumask()).

Why set the affinity at all?

> And with a modern NIC, this leads to packets being received on one CPU
> but the data being consumed on another CPU, all the time, which is
> obviously far from optimal.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Best way to set kernel thread affinity for handling a socket?

[Roland Dreier]

On Wed, Aug 22, 2012 at 9:21 AM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
> With RFS we try to do the reverse: move the packets to match the socket
> user.   But it's not (yet) turned on by default.  See
> Documentation/networking/scaling.txt

Fair enough.  However I think at least in this case it sounds like extra
overhead: it should be easy for us to do everything on the CPU where
the packets are being received.

>> I'm thinking about this in the context of the kernel's iSCSI target
>> code (drivers/target/iscsi), which creates threads to handle each
>> iSCSI connection and sets their CPU affinity pretty much randomly
>> (well, based on some "thread id", cf iscsit_thread_get_cpumask()).
>
> Why set the affinity at all?

It's quite possible that, like a lot of other drivers/target code, this
doesn't actually make any sense and the right answer is to rip it
out completely.

Is the affinity due to waking up from a network receive event
enough to keep the threads local to the right CPU?

 - R.

Re: Best way to set kernel thread affinity for handling a socket?

[Ben Hutchings]

On Thu, 2012-08-23 at 10:04 -0700, Roland Dreier wrote:
> On Wed, Aug 22, 2012 at 9:21 AM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
> > With RFS we try to do the reverse: move the packets to match the socket
> > user.   But it's not (yet) turned on by default.  See
> > Documentation/networking/scaling.txt
> 
> Fair enough.  However I think at least in this case it sounds like extra
> overhead: it should be easy for us to do everything on the CPU where
> the packets are being received.
> 
> >> I'm thinking about this in the context of the kernel's iSCSI target
> >> code (drivers/target/iscsi), which creates threads to handle each
> >> iSCSI connection and sets their CPU affinity pretty much randomly
> >> (well, based on some "thread id", cf iscsit_thread_get_cpumask()).
> >
> > Why set the affinity at all?
> 
> It's quite possible that, like a lot of other drivers/target code, this
> doesn't actually make any sense and the right answer is to rip it
> out completely.
> 
> Is the affinity due to waking up from a network receive event
> enough to keep the threads local to the right CPU?

I expect it to have some influence on where the receiving task is
scheduled, but it doesn't necessarily outweigh other factors that the
scheduler takes into account.  So you would really have to test this
yourself.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Best way to set kernel thread affinity for handling a socket?

[Chris Friesen]

On 08/23/2012 11:04 AM, Roland Dreier wrote:
> On Wed, Aug 22, 2012 at 9:21 AM, Ben Hutchings
> <bhutchings@solarflare.com>  wrote:
>> With RFS we try to do the reverse: move the packets to match the socket
>> user.   But it's not (yet) turned on by default.  See
>> Documentation/networking/scaling.txt
>
> Fair enough.  However I think at least in this case it sounds like extra
> overhead: it should be easy for us to do everything on the CPU where
> the packets are being received.

In the general case though RFS can be useful.  We have a server with one 
application instance per core.  It's beneficial to be able to steer 
packets destined for a given instance to an eth driver queue whose 
interrupt is handled by the cpu running that instance.

Depending on the hardware being used, it may be possible to do this more 
efficiently.  The ixgbe driver (if the feature is enabled) will 
periodically look at the outgoing tcp traffic and set up hardware flow 
filtering rules to direct incoming traffic for that connection to the 
queue that sent out the messages.

Chris