Re: [PATCH RFC 0/4]: xen-net{back, front}: Multiple transmit and receive queues

[Wei Liu <wei.liu2@citrix.com>]

Cool! Finally!

On Wed, Jan 15, 2014 at 04:23:20PM +0000, Andrew J. Bennieston wrote:
> This patch series implements multiple transmit and receive queues (i.e.
> multiple shared rings) for the xen virtual network interfaces.
> 
> The series is split up as follows:
>  - Patches 1 and 3 factor out the queue-specific data for netback and
>     netfront respectively, and modify the rest of the code to use these
>     as appropriate.
>  - Patches 2 and 4 introduce new XenStore keys to negotiate and use
>    multiple shared rings and event channels, and code to connect these
>    as appropriate.
> 
> All other transmit and receive processing remains unchanged, i.e. there
> is a kthread per queue and a NAPI context per queue.
> 
> The performance of these patches has been analysed in detail, with
> results available at:
> 
> http://wiki.xenproject.org/wiki/Xen-netback_and_xen-netfront_multi-queue_performance_testing
> 
> To summarise:
>   * Using multiple queues allows a VM to transmit at line rate on a 10
>     Gbit/s NIC, compared with a maximum aggregate throughput of 6 Gbit/s
>     with a single queue.
>   * For intra-host VM--VM traffic, eight queues provide 171% of the
>     throughput of a single queue; almost 12 Gbit/s instead of 6 Gbit/s.
>   * There is a corresponding increase in total CPU usage, i.e. this is a
>     scaling out over available resources, not an efficiency improvement.
>   * Results depend on the availability of sufficient CPUs, as well as the
>     distribution of interrupts and the distribution of TCP streams across
>     the queues.
> 
> One open issue is how to deal with the tx_credit data for rate limiting.
> This used to exist on a per-VIF basis, and these patches move it to
> per-queue to avoid contention on concurrent access to the tx_credit
> data from multiple threads. This has the side effect of breaking the
> tx_credit accounting across the VIF as a whole. I cannot see a situation
> in which people would want to use both rate limiting and a
> high-performance multi-queue mode, but if this is problematic then it
> can be brought back to the VIF level, with appropriate protection.
> Obviously, it continues to work identically in the case where there is
> only one queue.
> 

I would go for per-queue limit at the stage as it simplifies things and
keep you focus on core functionality.

Wei.

> Queue selection is currently achieved via an L4 hash on the packet (i.e.
> TCP src/dst port, IP src/dst address) and is not negotiated between the
> frontend and backend, since only one option exists. Future patches to
> support other frontends (particularly Windows) will need to add some
> capability to negotiate not only the hash algorithm selection, but also
> allow the frontend to specify some parameters to this.
> 
> Queue-specific XenStore entries for ring references and event channels
> are stored hierarchically, i.e. under .../queue-N/... where N varies
> from 0 to one less than the requested number of queues (inclusive). If
> only one queue is requested, it falls back to the flat structure where
> the ring references and event channels are written at the same level as
> other vif information.
> 
> --
> Andrew J. Bennieston