how to utilize multi tx queue to sent packets

how to utilize multi tx queue to sent packets

[Jon Zhou]

hi

I am doing some test according to the
website:http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap

use packet_mmap tx_ring to send packet.

I modified the sample code "packetmmap.c" to make it send packets have different outer ip.
so that with the help of RSS, I can achieve higher throughput.

but one thing I saw at the tx side, is that all the packets are sent via the same tx_queue,
which is conflict with what I saw at the rx side.
any idea to make it sent packets via different tx_queues? (spread across the tx_queues)


thanks
jon

Re: how to utilize multi tx queue to sent packets

[Stephen Hemminger]

On Thu, 10 Mar 2011 22:52:30 -0800
Jon Zhou <Jon.Zhou@jdsu.com> wrote:

> hi
> 
> I am doing some test according to the
> website:http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap
> 
> use packet_mmap tx_ring to send packet.
> 
> I modified the sample code "packetmmap.c" to make it send packets have different outer ip.
> so that with the help of RSS, I can achieve higher throughput.
> 
> but one thing I saw at the tx side, is that all the packets are sent via the same tx_queue,
> which is conflict with what I saw at the rx side.
> any idea to make it sent packets via different tx_queues? (spread across the tx_queues)

You need to have multiple threads to get Tx scaling.
In you case that also means multiple AF_PACKET sockets and separate rings.

Or just run multiple copies of the same test each with a different IP

RE: how to utilize multi tx queue to sent packets

[Jon Zhou]

> -----Original Message-----
> From: Stephen Hemminger [mailto:shemminger@vyatta.com]
> Sent: Saturday, March 12, 2011 1:07 AM
> To: Jon Zhou
> Cc: netdev@vger.kernel.org
> Subject: Re: how to utilize multi tx queue to sent packets
> 
> On Thu, 10 Mar 2011 22:52:30 -0800
> Jon Zhou <Jon.Zhou@jdsu.com> wrote:
> 
> > hi
> >
> > I am doing some test according to the
> > website:http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap
> >
> > use packet_mmap tx_ring to send packet.
> >
> > I modified the sample code "packetmmap.c" to make it send packets
> have different outer ip.
> > so that with the help of RSS, I can achieve higher throughput.
> >
> > but one thing I saw at the tx side, is that all the packets are sent
> via the same tx_queue,
> > which is conflict with what I saw at the rx side.
> > any idea to make it sent packets via different tx_queues? (spread
> across the tx_queues)
> 
> You need to have multiple threads to get Tx scaling.
> In you case that also means multiple AF_PACKET sockets and separate
> rings.
> 
> Or just run multiple copies of the same test each with a different IP

How does kernel know which tx_queue to use?
Will dev_queue_xmit determine tx_queue(sw queue) to send packets?
Or just let NIC select a tx_queue(hardware) ?

Re: how to utilize multi tx queue to sent packets

[Stephen Hemminger]

On Sun, 13 Mar 2011 01:06:11 -0800
Jon Zhou <Jon.Zhou@jdsu.com> wrote:

> 
> 
> > -----Original Message-----
> > From: Stephen Hemminger [mailto:shemminger@vyatta.com]
> > Sent: Saturday, March 12, 2011 1:07 AM
> > To: Jon Zhou
> > Cc: netdev@vger.kernel.org
> > Subject: Re: how to utilize multi tx queue to sent packets
> > 
> > On Thu, 10 Mar 2011 22:52:30 -0800
> > Jon Zhou <Jon.Zhou@jdsu.com> wrote:
> > 
> > > hi
> > >
> > > I am doing some test according to the
> > > website:http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap
> > >
> > > use packet_mmap tx_ring to send packet.
> > >
> > > I modified the sample code "packetmmap.c" to make it send packets
> > have different outer ip.
> > > so that with the help of RSS, I can achieve higher throughput.
> > >
> > > but one thing I saw at the tx side, is that all the packets are sent
> > via the same tx_queue,
> > > which is conflict with what I saw at the rx side.
> > > any idea to make it sent packets via different tx_queues? (spread
> > across the tx_queues)
> > 
> > You need to have multiple threads to get Tx scaling.
> > In you case that also means multiple AF_PACKET sockets and separate
> > rings.
> > 
> > Or just run multiple copies of the same test each with a different IP
> 
> How does kernel know which tx_queue to use?
> Will dev_queue_xmit determine tx_queue(sw queue) to send packets?
> Or just let NIC select a tx_queue(hardware) ?
> 

On most hardware, with multi-queue there is one queue per CPU.
In that case the queue selected corresponds to the CPU.



--