* why all packets have same queue no when rps enabled?
@ 2011-02-22 4:07 Jon Zhou
2011-02-22 5:41 ` Eric Dumazet
2011-02-22 12:45 ` Ben Hutchings
0 siblings, 2 replies; 7+ messages in thread
From: Jon Zhou @ 2011-02-22 4:07 UTC (permalink / raw)
To: netdev@vger.kernel.org
Hi
I expect each incoming packet will have a different queue no. when I enabled RPS on kernel 2.6.36.4
cat /sys/class/net/eth4/queues/rx-0/rps_cpus
00000000,000000ff
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16
CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31
HI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
TIMER: 6027512 710165 2623243 542768 427807 217424 192940 217043 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
NET_TX: 1365741 59 750957 0 171 0 3 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
NET_RX: 40465750 11140803 8545859 14417762 8913471 12298691 14216845 3348431 < ---- indeed spread across cpus
I manually disable RSS on the intel X520 multiqueue supported NIC.
Cat /proc/interrupts
87: 21348294 0 0 0 0 0 0 0 PCI-MSI-edge eth4-rx-0
88: 38394 0 0 0 0 0 0 0 PCI-MSI-edge eth4-tx-0
When I tried the below program to filter packet by queue no.I got these results:
struct sock_filter BPF_code[]= {
{ 0x28,0,0,SKF_AD_OFF+SKF_AD_QUEUE},
{ 0x15, 0, 1, 0x00000001 },
{ 0x6, 0, 0, 0x0000ffff },
{ 0x6, 0, 0, 0x00000000 }
};
struct sock_fprog Filter;
Filter.len = 4;//15;
Filter.filter = BPF_code;
if ( (sock=socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP)))<0) {
perror("socket");
exit(1);
}
/* Set the network card in promiscuos mode */
strncpy(ethreq.ifr_name,"eth4",IFNAMSIZ);
if (ioctl(sock,SIOCGIFFLAGS,ðreq)==-1) {
perror("ioctl");
close(sock);
exit(1);
}
ethreq.ifr_flags|=IFF_PROMISC;
if (ioctl(sock,SIOCSIFFLAGS,ðreq)==-1) {
perror("ioctl");
close(sock);
exit(1);
}
/* Attach the filter to the socket */
if(setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER,&Filter, sizeof(Filter))<0){
perror("setsockopt");
close(sock);
exit(1);
}
static int count = 0;
while (1) {
printf("#%d----------\n",count++);
n = recvfrom(sock,buffer,2048,0,NULL,NULL);
printf("%d bytes read\n",n);
...
}
Looks almost all packets fall at same queue?
Will RPS allocate queue no for each packet? and what hash algorithm rps used? (is it Toeplitz hash algorithm?)
jon
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: why all packets have same queue no when rps enabled? 2011-02-22 4:07 why all packets have same queue no when rps enabled? Jon Zhou @ 2011-02-22 5:41 ` Eric Dumazet 2011-02-22 5:56 ` Jon Zhou 2011-02-22 12:45 ` Ben Hutchings 1 sibling, 1 reply; 7+ messages in thread From: Eric Dumazet @ 2011-02-22 5:41 UTC (permalink / raw) To: Jon Zhou; +Cc: netdev@vger.kernel.org Le lundi 21 février 2011 à 20:07 -0800, Jon Zhou a écrit : > Hi > > I expect each incoming packet will have a different queue no. when I enabled RPS on kernel 2.6.36.4 > > cat /sys/class/net/eth4/queues/rx-0/rps_cpus > 00000000,000000ff > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 > CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 > HI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > TIMER: 6027512 710165 2623243 542768 427807 217424 192940 217043 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > NET_TX: 1365741 59 750957 0 171 0 3 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > NET_RX: 40465750 11140803 8545859 14417762 8913471 12298691 14216845 3348431 < ---- indeed spread across cpus > > > I manually disable RSS on the intel X520 multiqueue supported NIC. > Cat /proc/interrupts > > 87: 21348294 0 0 0 0 0 0 0 PCI-MSI-edge eth4-rx-0 > 88: 38394 0 0 0 0 0 0 0 PCI-MSI-edge eth4-tx-0 > > > > When I tried the below program to filter packet by queue no.I got these results: > > struct sock_filter BPF_code[]= { > { 0x28,0,0,SKF_AD_OFF+SKF_AD_QUEUE}, > { 0x15, 0, 1, 0x00000001 }, > { 0x6, 0, 0, 0x0000ffff }, > { 0x6, 0, 0, 0x00000000 } > }; > > struct sock_fprog Filter; > > Filter.len = 4;//15; > Filter.filter = BPF_code; > > if ( (sock=socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP)))<0) { > perror("socket"); > exit(1); > } > > /* Set the network card in promiscuos mode */ > strncpy(ethreq.ifr_name,"eth4",IFNAMSIZ); > if (ioctl(sock,SIOCGIFFLAGS,ðreq)==-1) { > perror("ioctl"); > close(sock); > exit(1); > } > ethreq.ifr_flags|=IFF_PROMISC; > if (ioctl(sock,SIOCSIFFLAGS,ðreq)==-1) { > perror("ioctl"); > close(sock); > exit(1); > } > > /* Attach the filter to the socket */ > if(setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER,&Filter, sizeof(Filter))<0){ > perror("setsockopt"); > close(sock); > exit(1); > } > static int count = 0; > while (1) { > printf("#%d----------\n",count++); > n = recvfrom(sock,buffer,2048,0,NULL,NULL); > printf("%d bytes read\n",n); > ... > } > > > Looks almost all packets fall at same queue? > Will RPS allocate queue no for each packet? and what hash algorithm rps used? (is it Toeplitz hash algorithm?) > I believe you are mistaken. RPS is not there to spread load on _all_ cpus, but to use a hash function so that all packets of a given flow are directed to a given cpu. If you receive 1.000.000 packets of the same flow, they all are delivered to one CPU. ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: why all packets have same queue no when rps enabled? 2011-02-22 5:41 ` Eric Dumazet @ 2011-02-22 5:56 ` Jon Zhou 2011-02-22 6:13 ` Eric Dumazet 0 siblings, 1 reply; 7+ messages in thread From: Jon Zhou @ 2011-02-22 5:56 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev@vger.kernel.org > -----Original Message----- > From: Eric Dumazet [mailto:eric.dumazet@gmail.com] > Sent: Tuesday, February 22, 2011 1:41 PM > To: Jon Zhou > Cc: netdev@vger.kernel.org > Subject: Re: why all packets have same queue no when rps enabled? > > Le lundi 21 février 2011 à 20:07 -0800, Jon Zhou a écrit : > > Hi > > > > I expect each incoming packet will have a different queue no. when I > enabled RPS on kernel 2.6.36.4 > > > > cat /sys/class/net/eth4/queues/rx-0/rps_cpus > > 00000000,000000ff > > > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 > CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 > CPU13 CPU14 CPU15 CPU16 > > CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 > CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 > CPU30 CPU31 > > HI: 0 0 0 0 0 > 0 0 0 0 0 0 0 > 0 0 0 0 0 > > 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > 0 0 > > TIMER: 6027512 710165 2623243 542768 427807 > 217424 192940 217043 0 0 0 > 0 0 0 0 0 0 > > 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > 0 0 > > NET_TX: 1365741 59 750957 0 171 > 0 3 0 0 0 0 0 > 0 0 0 0 0 > > 0 0 0 0 0 0 > 0 0 0 0 0 0 0 > 0 0 > > NET_RX: 40465750 11140803 8545859 14417762 8913471 > 12298691 14216845 3348431 < ---- indeed spread across cpus > > > > > > I manually disable RSS on the intel X520 multiqueue supported NIC. > > Cat /proc/interrupts > > > > 87: 21348294 0 0 0 0 > 0 0 0 PCI-MSI-edge eth4-rx-0 > > 88: 38394 0 0 0 0 > 0 0 0 PCI-MSI-edge eth4-tx-0 > > > > > > > > When I tried the below program to filter packet by queue no.I got > these results: > > > > struct sock_filter BPF_code[]= { > > { 0x28,0,0,SKF_AD_OFF+SKF_AD_QUEUE}, > > { 0x15, 0, 1, 0x00000001 }, > > { 0x6, 0, 0, 0x0000ffff }, > > { 0x6, 0, 0, 0x00000000 } > > }; > > > > struct sock_fprog Filter; > > > > Filter.len = 4;//15; > > Filter.filter = BPF_code; > > > > if ( (sock=socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP)))<0) { > > perror("socket"); > > exit(1); > > } > > > > /* Set the network card in promiscuos mode */ > > strncpy(ethreq.ifr_name,"eth4",IFNAMSIZ); > > if (ioctl(sock,SIOCGIFFLAGS,ðreq)==-1) { > > perror("ioctl"); > > close(sock); > > exit(1); > > } > > ethreq.ifr_flags|=IFF_PROMISC; > > if (ioctl(sock,SIOCSIFFLAGS,ðreq)==-1) { > > perror("ioctl"); > > close(sock); > > exit(1); > > } > > > > /* Attach the filter to the socket */ > > if(setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER,&Filter, > sizeof(Filter))<0){ > > perror("setsockopt"); > > close(sock); > > exit(1); > > } > > static int count = 0; > > while (1) { > > printf("#%d----------\n",count++); > > n = recvfrom(sock,buffer,2048,0,NULL,NULL); > > printf("%d bytes read\n",n); > > ... > > } > > > > > > Looks almost all packets fall at same queue? > > Will RPS allocate queue no for each packet? and what hash algorithm > rps used? (is it Toeplitz hash algorithm?) > > > > I believe you are mistaken. > > RPS is not there to spread load on _all_ cpus, but to use a hash > function so that all packets of a given flow are directed to a given > cpu. > > If you receive 1.000.000 packets of the same flow, they all are > delivered to one CPU. > > With RSS supported NIC, I saw packet#0,1,2,3,~packet#7 will be delivered to different queue, but after RSS disabled and RPS turn on, all these packet will be allocated same queue no(I used the above filter program to find that) Is there any fault in the filter program? ^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: why all packets have same queue no when rps enabled? 2011-02-22 5:56 ` Jon Zhou @ 2011-02-22 6:13 ` Eric Dumazet 0 siblings, 0 replies; 7+ messages in thread From: Eric Dumazet @ 2011-02-22 6:13 UTC (permalink / raw) To: Jon Zhou; +Cc: netdev@vger.kernel.org Le lundi 21 février 2011 à 21:56 -0800, Jon Zhou a écrit : > > > -----Original Message----- > > From: Eric Dumazet [mailto:eric.dumazet@gmail.com] > > Sent: Tuesday, February 22, 2011 1:41 PM > > To: Jon Zhou > > Cc: netdev@vger.kernel.org > > Subject: Re: why all packets have same queue no when rps enabled? > > > > Le lundi 21 février 2011 à 20:07 -0800, Jon Zhou a écrit : > > > Hi > > > > > > I expect each incoming packet will have a different queue no. when I > > enabled RPS on kernel 2.6.36.4 > > > > > > cat /sys/class/net/eth4/queues/rx-0/rps_cpus > > > 00000000,000000ff > > > > > > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 > > CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 > > CPU13 CPU14 CPU15 CPU16 > > > CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 > > CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 > > CPU30 CPU31 > > > HI: 0 0 0 0 0 > > 0 0 0 0 0 0 0 > > 0 0 0 0 0 > > > 0 0 0 0 0 0 > > 0 0 0 0 0 0 0 > > 0 0 > > > TIMER: 6027512 710165 2623243 542768 427807 > > 217424 192940 217043 0 0 0 > > 0 0 0 0 0 0 > > > 0 0 0 0 0 0 > > 0 0 0 0 0 0 0 > > 0 0 > > > NET_TX: 1365741 59 750957 0 171 > > 0 3 0 0 0 0 0 > > 0 0 0 0 0 > > > 0 0 0 0 0 0 > > 0 0 0 0 0 0 0 > > 0 0 > > > NET_RX: 40465750 11140803 8545859 14417762 8913471 > > 12298691 14216845 3348431 < ---- indeed spread across cpus > > > > > > > > > I manually disable RSS on the intel X520 multiqueue supported NIC. > > > Cat /proc/interrupts > > > > > > 87: 21348294 0 0 0 0 > > 0 0 0 PCI-MSI-edge eth4-rx-0 > > > 88: 38394 0 0 0 0 > > 0 0 0 PCI-MSI-edge eth4-tx-0 > > > > > > > > > > > > When I tried the below program to filter packet by queue no.I got > > these results: > > > > > > struct sock_filter BPF_code[]= { > > > { 0x28,0,0,SKF_AD_OFF+SKF_AD_QUEUE}, > > > { 0x15, 0, 1, 0x00000001 }, > > > { 0x6, 0, 0, 0x0000ffff }, > > > { 0x6, 0, 0, 0x00000000 } > > > }; > > > > > > struct sock_fprog Filter; > > > > > > Filter.len = 4;//15; > > > Filter.filter = BPF_code; > > > > > > if ( (sock=socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP)))<0) { > > > perror("socket"); > > > exit(1); > > > } > > > > > > /* Set the network card in promiscuos mode */ > > > strncpy(ethreq.ifr_name,"eth4",IFNAMSIZ); > > > if (ioctl(sock,SIOCGIFFLAGS,ðreq)==-1) { > > > perror("ioctl"); > > > close(sock); > > > exit(1); > > > } > > > ethreq.ifr_flags|=IFF_PROMISC; > > > if (ioctl(sock,SIOCSIFFLAGS,ðreq)==-1) { > > > perror("ioctl"); > > > close(sock); > > > exit(1); > > > } > > > > > > /* Attach the filter to the socket */ > > > if(setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER,&Filter, > > sizeof(Filter))<0){ > > > perror("setsockopt"); > > > close(sock); > > > exit(1); > > > } > > > static int count = 0; > > > while (1) { > > > printf("#%d----------\n",count++); > > > n = recvfrom(sock,buffer,2048,0,NULL,NULL); > > > printf("%d bytes read\n",n); > > > ... > > > } > > > > > > > > > Looks almost all packets fall at same queue? > > > Will RPS allocate queue no for each packet? and what hash algorithm > > rps used? (is it Toeplitz hash algorithm?) > > > > > > > I believe you are mistaken. > > > > RPS is not there to spread load on _all_ cpus, but to use a hash > > function so that all packets of a given flow are directed to a given > > cpu. > > > > If you receive 1.000.000 packets of the same flow, they all are > > delivered to one CPU. > > > > > With RSS supported NIC, I saw packet#0,1,2,3,~packet#7 will be delivered to different queue, but after RSS disabled and RPS turn on, all these packet will be allocated same queue no(I used the above filter program to find that) > Is there any fault in the filter program? > I dont know... The easy way to find out is to actually do "cat /proc/net/softnet_stat" Since the last column is the number of time a cpu was given a packet by another cpu (RPS or RFS action) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: why all packets have same queue no when rps enabled? 2011-02-22 4:07 why all packets have same queue no when rps enabled? Jon Zhou 2011-02-22 5:41 ` Eric Dumazet @ 2011-02-22 12:45 ` Ben Hutchings 2011-02-22 13:01 ` Eric Dumazet 1 sibling, 1 reply; 7+ messages in thread From: Ben Hutchings @ 2011-02-22 12:45 UTC (permalink / raw) To: Jon Zhou; +Cc: netdev@vger.kernel.org On Mon, 2011-02-21 at 20:07 -0800, Jon Zhou wrote: > Hi > > I expect each incoming packet will have a different queue no. when I > enabled RPS on kernel 2.6.36.4 [...] > Looks almost all packets fall at same queue? > Will RPS allocate queue no for each packet? and what hash algorithm > rps used? (is it Toeplitz hash algorithm?) The queue number identifies a hardware queue. RPS therefore does not update this number when queueing packets for processing on other CPUs. If the hardware/driver provides a receive hash (probably Toeplitz) then this is used for RPS. Otherwise a much cheaper hash is used. Ben. -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: why all packets have same queue no when rps enabled? 2011-02-22 12:45 ` Ben Hutchings @ 2011-02-22 13:01 ` Eric Dumazet 2011-02-22 13:09 ` Ben Hutchings 0 siblings, 1 reply; 7+ messages in thread From: Eric Dumazet @ 2011-02-22 13:01 UTC (permalink / raw) To: Ben Hutchings; +Cc: Jon Zhou, netdev@vger.kernel.org Le mardi 22 février 2011 à 12:45 +0000, Ben Hutchings a écrit : > The queue number identifies a hardware queue. RPS therefore does not > update this number when queueing packets for processing on other CPUs. > > If the hardware/driver provides a receive hash (probably Toeplitz) then > this is used for RPS. Otherwise a much cheaper hash is used. The default is/should be : rxhash generated in network stack. ethtool -k eth0 | grep hash receive-hashing: off To use the device/harwdare provided rxhash, you need to ask for it. ethtool -K eth0 rxhash on BTW, I am not sure what you mean by "much cheaper hash is used"... I presume hardware provided hash is less expensive (for our cpu) than computing our rxhash... ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: why all packets have same queue no when rps enabled? 2011-02-22 13:01 ` Eric Dumazet @ 2011-02-22 13:09 ` Ben Hutchings 0 siblings, 0 replies; 7+ messages in thread From: Ben Hutchings @ 2011-02-22 13:09 UTC (permalink / raw) To: Eric Dumazet; +Cc: Jon Zhou, netdev@vger.kernel.org On Tue, 2011-02-22 at 14:01 +0100, Eric Dumazet wrote: > Le mardi 22 février 2011 à 12:45 +0000, Ben Hutchings a écrit : > > > The queue number identifies a hardware queue. RPS therefore does not > > update this number when queueing packets for processing on other CPUs. > > > > If the hardware/driver provides a receive hash (probably Toeplitz) then > > this is used for RPS. Otherwise a much cheaper hash is used. > > The default is/should be : rxhash generated in network stack. > > ethtool -k eth0 | grep hash > receive-hashing: off > > To use the device/harwdare provided rxhash, you need to ask for it. > > ethtool -K eth0 rxhash on Whether this is enabled by default depends on the driver. > BTW, I am not sure what you mean by "much cheaper hash is used"... > > I presume hardware provided hash is less expensive (for our cpu) than > computing our rxhash... I mean the hash function we use is much cheaper than Toeplitz. Ben. -- Ben Hutchings, Senior Software Engineer, Solarflare Communications Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-02-22 13:09 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-02-22 4:07 why all packets have same queue no when rps enabled? Jon Zhou 2011-02-22 5:41 ` Eric Dumazet 2011-02-22 5:56 ` Jon Zhou 2011-02-22 6:13 ` Eric Dumazet 2011-02-22 12:45 ` Ben Hutchings 2011-02-22 13:01 ` Eric Dumazet 2011-02-22 13:09 ` Ben Hutchings
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).