RE: why all packets have same queue no when rps enabled?

[Eric Dumazet <eric.dumazet@gmail.com>]

Le lundi 21 février 2011 à 21:56 -0800, Jon Zhou a écrit :
> 
> > -----Original Message-----
> > From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
> > Sent: Tuesday, February 22, 2011 1:41 PM
> > To: Jon Zhou
> > Cc: netdev@vger.kernel.org
> > Subject: Re: why all packets have same queue no when rps enabled?
> > 
> > Le lundi 21 février 2011 à 20:07 -0800, Jon Zhou a écrit :
> > > Hi
> > >
> > > I expect each incoming packet will have a different queue no. when I
> > enabled RPS on kernel 2.6.36.4
> > >
> > > cat /sys/class/net/eth4/queues/rx-0/rps_cpus
> > > 00000000,000000ff
> > >
> > > CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
> > CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12
> > CPU13      CPU14      CPU15      CPU16
> > >       CPU17      CPU18      CPU19      CPU20      CPU21      CPU22
> > CPU23      CPU24      CPU25      CPU26      CPU27      CPU28      CPU29
> > CPU30      CPU31
> > >      HI:          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0          0          0          0
> > >          0          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0
> > >    TIMER:    6027512     710165    2623243     542768     427807
> > 217424     192940     217043          0          0          0
> > 0          0          0          0          0          0
> > >          0          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0
> > >   NET_TX:    1365741         59     750957          0        171
> > 0          3          0          0          0          0          0
> > 0          0          0          0          0
> > >          0          0          0          0          0          0
> > 0          0          0          0          0          0          0
> > 0          0
> > >   NET_RX:   40465750   11140803    8545859   14417762    8913471
> > 12298691   14216845    3348431   < ---- indeed spread across cpus
> > >
> > >
> > > I manually disable RSS on the intel X520 multiqueue supported NIC.
> > > Cat /proc/interrupts
> > >
> > >   87:   21348294          0          0          0          0
> > 0          0          0   PCI-MSI-edge      eth4-rx-0
> > >   88:      38394          0          0          0          0
> > 0          0          0   PCI-MSI-edge      eth4-tx-0
> > >
> > >
> > >
> > > When I tried the below program to filter packet by queue no.I got
> > these results:
> > >
> > > struct sock_filter BPF_code[]= {
> > >     { 0x28,0,0,SKF_AD_OFF+SKF_AD_QUEUE},
> > >     { 0x15, 0, 1, 0x00000001 },
> > >     { 0x6, 0, 0, 0x0000ffff },
> > >     { 0x6, 0, 0, 0x00000000 }
> > >   };
> > >
> > >   struct sock_fprog Filter;
> > >
> > >   Filter.len = 4;//15;
> > >   Filter.filter = BPF_code;
> > >
> > >   if ( (sock=socket(PF_PACKET, SOCK_RAW, htons(ETH_P_IP)))<0) {
> > >     perror("socket");
> > >     exit(1);
> > >   }
> > >
> > >   /* Set the network card in promiscuos mode */
> > >   strncpy(ethreq.ifr_name,"eth4",IFNAMSIZ);
> > >   if (ioctl(sock,SIOCGIFFLAGS,&ethreq)==-1) {
> > >     perror("ioctl");
> > >     close(sock);
> > >     exit(1);
> > >   }
> > >   ethreq.ifr_flags|=IFF_PROMISC;
> > >   if (ioctl(sock,SIOCSIFFLAGS,&ethreq)==-1) {
> > >     perror("ioctl");
> > >     close(sock);
> > >     exit(1);
> > >   }
> > >
> > >   /* Attach the filter to the socket */
> > >   if(setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER,&Filter,
> > sizeof(Filter))<0){
> > >     perror("setsockopt");
> > >     close(sock);
> > >     exit(1);
> > >   }
> > >   static int count = 0;
> > >   while (1) {
> > >     printf("#%d----------\n",count++);
> > >     n = recvfrom(sock,buffer,2048,0,NULL,NULL);
> > >     printf("%d bytes read\n",n);
> > > ...
> > > }
> > >
> > >
> > > Looks almost all packets fall at same queue?
> > > Will RPS allocate queue no for each packet? and what hash algorithm
> > rps used? (is it Toeplitz hash algorithm?)
> > >
> > 
> > I believe you are mistaken.
> > 
> > RPS is not there to spread load on _all_ cpus, but to use a hash
> > function so that all packets of a given flow are directed to a given
> > cpu.
> > 
> > If you receive 1.000.000 packets of the same flow, they all are
> > delivered to one CPU.
> > 
> > 
> With RSS supported NIC, I saw packet#0,1,2,3,~packet#7 will be delivered to different queue, but after RSS disabled and RPS turn on, all these packet will be allocated same queue no(I used the above filter program to find that)
> Is there any fault in the filter program?  
> 

I dont know...

The easy way to find out is to actually do "cat /proc/net/softnet_stat"

Since the last column is the number of time a cpu was given a packet by
another cpu (RPS or RFS action)