[RFC PATCH] net, rps: bypass enqueue_to_backlog()

[RFC PATCH] net, rps: bypass enqueue_to_backlog()

[Zhi Yong Wu]

From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>

When local cpu is just target cpu which will handle network soft irq,
the packet should be directly injected to network stack, by bypassing
enqueue_to_backlog(), it can speed up the packet processing.

HI, guys

I checked the first several versions of RPS patch which seemed to have
this condition determination, but why was it removed later? Do i miss
anything? if yes, please correct me, thanks.

Signed-off-by: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
---
 net/core/dev.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index c482fe8..d29d61f 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3693,7 +3693,7 @@ int netif_receive_skb(struct sk_buff *skb)
 
 		cpu = get_rps_cpu(skb->dev, skb, &rflow);
 
-		if (cpu >= 0) {
+		if ((cpu >= 0) && (cpu != raw_smp_processor_id())) {
 			ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);
 			rcu_read_unlock();
 			return ret;
-- 
1.7.6.5

Re: [RFC PATCH] net, rps: bypass enqueue_to_backlog()

[Eric Dumazet]

On Wed, 2013-12-18 at 21:03 +0800, Zhi Yong Wu wrote:
> From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
> 
> When local cpu is just target cpu which will handle network soft irq,
> the packet should be directly injected to network stack, by bypassing
> enqueue_to_backlog(), it can speed up the packet processing.
> 
> HI, guys
> 
> I checked the first several versions of RPS patch which seemed to have
> this condition determination, but why was it removed later? Do i miss
> anything? if yes, please correct me, thanks.
> 
> Signed-off-by: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
> ---

Hmm... Could you elaborate ?

At which point do you think this condition was tested or removed ?

I think the idea was to drain NIC RX queues as fast as possible, then :

- Send the IPI to remote cpus
- process our queue in parallel with other cpus processing their own
queue.

If we process our packets through whole stack, packets for other cpus
will have a fair amount of extra latency.

Thats a tradeoff I suppose.

Re: [RFC PATCH] net, rps: bypass enqueue_to_backlog()

[Eric Dumazet]

On Wed, 2013-12-18 at 05:56 -0800, Eric Dumazet wrote:
> On Wed, 2013-12-18 at 21:03 +0800, Zhi Yong Wu wrote:
> > From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
> > 
> > When local cpu is just target cpu which will handle network soft irq,
> > the packet should be directly injected to network stack, by bypassing
> > enqueue_to_backlog(), it can speed up the packet processing.
> > 
> > HI, guys
> > 
> > I checked the first several versions of RPS patch which seemed to have
> > this condition determination, but why was it removed later? Do i miss
> > anything? if yes, please correct me, thanks.
> > 
> > Signed-off-by: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
> > ---
> 
> Hmm... Could you elaborate ?
> 
> At which point do you think this condition was tested or removed ?
> 
> I think the idea was to drain NIC RX queues as fast as possible, then :
> 
> - Send the IPI to remote cpus
> - process our queue in parallel with other cpus processing their own
> queue.
> 
> If we process our packets through whole stack, packets for other cpus
> will have a fair amount of extra latency.
> 
> Thats a tradeoff I suppose.
> 

Also note that going through the backlog permits thinks like

99bbc70741903c0  ("rps: selective flow shedding during softnet
overflow")

Re: [RFC PATCH] net, rps: bypass enqueue_to_backlog()

[Zhi Yong Wu]

On Wed, Dec 18, 2013 at 10:04 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Wed, 2013-12-18 at 05:56 -0800, Eric Dumazet wrote:
>> On Wed, 2013-12-18 at 21:03 +0800, Zhi Yong Wu wrote:
>> > From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
>> >
>> > When local cpu is just target cpu which will handle network soft irq,
>> > the packet should be directly injected to network stack, by bypassing
>> > enqueue_to_backlog(), it can speed up the packet processing.
>> >
>> > HI, guys
>> >
>> > I checked the first several versions of RPS patch which seemed to have
>> > this condition determination, but why was it removed later? Do i miss
>> > anything? if yes, please correct me, thanks.
>> >
>> > Signed-off-by: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
>> > ---
>>
>> Hmm... Could you elaborate ?
>>
>> At which point do you think this condition was tested or removed ?
>>
>> I think the idea was to drain NIC RX queues as fast as possible, then :
>>
>> - Send the IPI to remote cpus
>> - process our queue in parallel with other cpus processing their own
>> queue.
>>
>> If we process our packets through whole stack, packets for other cpus
>> will have a fair amount of extra latency.
>>
>> Thats a tradeoff I suppose.
>>
>
> Also note that going through the backlog permits thinks like
>
> 99bbc70741903c0  ("rps: selective flow shedding during softnet
> overflow"
it makes sense, but if cpu < 0, some packets seems to bypass the flow
shedding stuff, That is, the flow limit will be not so accurate,
right?

>
>
>



-- 
Regards,

Zhi Yong Wu

Re: [RFC PATCH] net, rps: bypass enqueue_to_backlog()

[Zhi Yong Wu]

thanks for your explanation.

by the way, pls ignore this patch, thanks.

On Wed, Dec 18, 2013 at 9:56 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Wed, 2013-12-18 at 21:03 +0800, Zhi Yong Wu wrote:
>> From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
>>
>> When local cpu is just target cpu which will handle network soft irq,
>> the packet should be directly injected to network stack, by bypassing
>> enqueue_to_backlog(), it can speed up the packet processing.
>>
>> HI, guys
>>
>> I checked the first several versions of RPS patch which seemed to have
>> this condition determination, but why was it removed later? Do i miss
>> anything? if yes, please correct me, thanks.
>>
>> Signed-off-by: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>
>> ---
>
> Hmm... Could you elaborate ?
>
> At which point do you think this condition was tested or removed ?
>
> I think the idea was to drain NIC RX queues as fast as possible, then :
>
> - Send the IPI to remote cpus
> - process our queue in parallel with other cpus processing their own
> queue.
>
> If we process our packets through whole stack, packets for other cpus
> will have a fair amount of extra latency.
>
> Thats a tradeoff I suppose.
>
>



-- 
Regards,

Zhi Yong Wu

Re: [RFC PATCH] net, rps: bypass enqueue_to_backlog()

[Eric Dumazet]

On Wed, 2013-12-18 at 22:11 +0800, Zhi Yong Wu wrote:

> it makes sense, but if cpu < 0, some packets seems to bypass the flow
> shedding stuff, That is, the flow limit will be not so accurate,
> right?

Not sure what you mean.

cpu = -1 on very rare occasions, like cpu hot unplug.

In netif_receive_skb(), we just process the packet normally.