any change in socket systemcall or packet_mmap regarding multiqueue nic?

any change in socket systemcall or packet_mmap regarding multiqueue nic?

[Jon Zhou]

hi
the multiqueue networking can utilize multi-core to process packets from multiqueue nic,
but any change in related userspace application part, such as socket system call, packet_mmap? these userspace API can also utilize multicore to process packets from kernel?
otherwise they have to read data in serialization

thanks
jon 

Re: any change in socket systemcall or packet_mmap regarding multiqueue nic?

[Eric Dumazet]

Le mardi 18 mai 2010 à 19:55 -0700, Jon Zhou a écrit :
> hi
> the multiqueue networking can utilize multi-core to process packets from multiqueue nic,
> but any change in related userspace application part, such as socket system call, packet_mmap? these userspace API can also utilize multicore to process packets from kernel?
> otherwise they have to read data in serialization
> 

Thats a bit general question. Works are in progress.

So far, you can use a new condition in filters to match a given queue
index for incoming packets. A sniffer could setup N different sockets to
receive data from N NIC queues.

For tcp flows, nothing is needed, since all packets of a given flow
should use same queue.

However the current tx queue selection is based on sk->sk_hash value, a
linux side computed value, and this differs from the rx queue selection
done by the NIC firmware. So tx packets use a different queue than rx
packets for a given tcp flow. This means this is suboptimal: tcp_ack()
can run on a different cpu than TX completion handler.

TX completion handler touches the cloned skb that TCP used to transmit
buffer. Its freeing touches the dataref atomic counter in packet.

This should be addressed somehow.

RE: any change in socket systemcall or packet_mmap regarding multiqueue nic?

[Jon Zhou]

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com] 
Sent: Wednesday, May 19, 2010 12:25 PM
To: Jon Zhou
Cc: netdev@vger.kernel.org
Subject: Re: any change in socket systemcall or packet_mmap regarding multiqueue nic?

Le mardi 18 mai 2010 à 19:55 -0700, Jon Zhou a écrit :
> hi
> the multiqueue networking can utilize multi-core to process packets from multiqueue nic,
> but any change in related userspace application part, such as socket system call, packet_mmap? these userspace API can also utilize multicore to process packets from kernel?
> otherwise they have to read data in serialization
> 

Thats a bit general question. Works are in progress.

So far, you can use a new condition in filters to match a given queue
index for incoming packets. A sniffer could setup N different sockets to
receive data from N NIC queues.

jon->is it something like "ioctl(fd,SOL_SOCKET,queue_id...),could you tell the keyword?

For tcp flows, nothing is needed, since all packets of a given flow
should use same queue.

btw,do you think RFS is helpful for this?


However the current tx queue selection is based on sk->sk_hash value, a
linux side computed value, and this differs from the rx queue selection
done by the NIC firmware. So tx packets use a different queue than rx
packets for a given tcp flow. This means this is suboptimal: tcp_ack()
can run on a different cpu than TX completion handler.

TX completion handler touches the cloned skb that TCP used to transmit
buffer. Its freeing touches the dataref atomic counter in packet.

This should be addressed somehow.

RE: any change in socket systemcall or packet_mmap regarding multiqueue nic?

[Eric Dumazet]

Le mercredi 19 mai 2010 à 03:36 -0700, Jon Zhou a écrit :
> 
> -----Original Message-----
> From: Eric Dumazet [mailto:eric.dumazet@gmail.com] 
> Sent: Wednesday, May 19, 2010 12:25 PM
> To: Jon Zhou
> Cc: netdev@vger.kernel.org
> Subject: Re: any change in socket systemcall or packet_mmap regarding multiqueue nic?
> 
> Le mardi 18 mai 2010 à 19:55 -0700, Jon Zhou a écrit :
> > hi
> > the multiqueue networking can utilize multi-core to process packets from multiqueue nic,
> > but any change in related userspace application part, such as socket system call, packet_mmap? these userspace API can also utilize multicore to process packets from kernel?
> > otherwise they have to read data in serialization
> > 
> 
> Thats a bit general question. Works are in progress.
> 
> So far, you can use a new condition in filters to match a given queue
> index for incoming packets. A sniffer could setup N different sockets to
> receive data from N NIC queues.
> 
> jon->is it something like "ioctl(fd,SOL_SOCKET,queue_id...),could you tell the keyword?

keyword is BPF (used in libpcap) and SKF_AD_QUEUE instruction

Kernel part is ready :

commit d19742fb1c68e6db83b76e06dea5a374c99e104f
Author: Eric Dumazet <eric.dumazet@gmail.com>
Date:   Tue Oct 20 01:06:22 2009 -0700

    filter: Add SKF_AD_QUEUE instruction
    
    It can help being able to filter packets on their queue_mapping.
    
    If filter performance is not good, we could add a "numqueue" field
    in struct packet_type, so that netif_nit_deliver() and other
functions
    can directly ignore packets with not expected queue number.
    
    Lets experiment this simple filter extension first.
    
    Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/include/linux/filter.h b/include/linux/filter.h
index 909193e..bb3b435 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -124,7 +124,8 @@ struct sock_fprog   /* Required for
SO_ATTACH_FILTER. */
 #define SKF_AD_NLATTR  12
 #define SKF_AD_NLATTR_NEST     16
 #define SKF_AD_MARK    20
-#define SKF_AD_MAX     24
+#define SKF_AD_QUEUE   24
+#define SKF_AD_MAX     28
 #define SKF_NET_OFF   (-0x100000)
 #define SKF_LL_OFF    (-0x200000)
 
diff --git a/net/core/filter.c b/net/core/filter.c
index e3987e1..08db7b9 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -306,6 +306,9 @@ load_b:
                case SKF_AD_MARK:
                        A = skb->mark;
                        continue;
+               case SKF_AD_QUEUE:
+                       A = skb->queue_mapping;
+                       continue;
                case SKF_AD_NLATTR: {
                        struct nlattr *nla;