[PATCH net-next] documentation: Document issues with VMs and XPS and drivers enabling it on their own

[PATCH net-next] documentation: Document issues with VMs and XPS and drivers enabling it on their own

[Rick Jones]

From: Rick Jones <rick.jones2@hpe.com>

Since XPS was first introduced two things have happened.  Some drivers
have started enabling XPS on their own initiative, and it has been
found that when a VM is sending data through a host interface with XPS
enabled, that traffic can end-up seriously out of order.

Signed-off-by: Rick Jones <rick.jones2@hpe.com>

---

diff --git a/Documentation/networking/scaling.txt b/Documentation/networking/scaling.txt
index 59f4db2..50cc888 100644
--- a/Documentation/networking/scaling.txt
+++ b/Documentation/networking/scaling.txt
@@ -400,15 +400,31 @@ transport layer is responsible for setting ooo_okay appropriately. TCP,
 for instance, sets the flag when all data for a connection has been
 acknowledged.
 
+When the traffic source is a VM running on the host, there is no
+socket structure known to the host.  In this case, unless the VM is
+itself CPU-pinned, the traffic being sent from it can end-up queued to
+multiple transmit queues and end-up being transmitted out of order.
+
+In some cases this can result in a considerable loss of performance.
+
+In such situations, XPS should not be enabled at runtime, or
+explicitly disabled if the NIC driver(s) in question enable it on
+their own.  Othersise, if possible, the VMs should be CPU pinned.
+
 ==== XPS Configuration
 
-XPS is only available if the kconfig symbol CONFIG_XPS is enabled (on by
-default for SMP). The functionality remains disabled until explicitly
-configured. To enable XPS, the bitmap of CPUs that may use a transmit
-queue is configured using the sysfs file entry:
+XPS is available only if the kconfig symbol CONFIG_XPS is enabled
+prior to building the kernel.  It is enabled by default for SMP kernel
+configurations.  In many cases the functionality remains disabled at
+runtime until explicitly configured by the system administrator. To
+enable XPS, the bitmap of CPUs that may use a transmit queue is
+configured using the sysfs file entry:
 
 /sys/class/net/<dev>/queues/tx-<n>/xps_cpus
 
+However, some NIC drivers will configure XPS at runtime for the
+interfaces they drive, via a call to netif_set_xps_queue.
+
 == Suggested Configuration
 
 For a network device with a single transmission queue, XPS configuration

Re: [PATCH net-next] documentation: Document issues with VMs and XPS and drivers enabling it on their own

[Alexander Duyck]

On Thu, Aug 25, 2016 at 2:33 PM, Rick Jones <rick.jones2@hpe.com> wrote:
> From: Rick Jones <rick.jones2@hpe.com>
>
> Since XPS was first introduced two things have happened.  Some drivers
> have started enabling XPS on their own initiative, and it has been
> found that when a VM is sending data through a host interface with XPS
> enabled, that traffic can end-up seriously out of order.
>
> Signed-off-by: Rick Jones <rick.jones2@hpe.com>
>
> ---
>
> diff --git a/Documentation/networking/scaling.txt b/Documentation/networking/scaling.txt
> index 59f4db2..50cc888 100644
> --- a/Documentation/networking/scaling.txt
> +++ b/Documentation/networking/scaling.txt
> @@ -400,15 +400,31 @@ transport layer is responsible for setting ooo_okay appropriately. TCP,
>  for instance, sets the flag when all data for a connection has been
>  acknowledged.
>
> +When the traffic source is a VM running on the host, there is no
> +socket structure known to the host.  In this case, unless the VM is
> +itself CPU-pinned, the traffic being sent from it can end-up queued to
> +multiple transmit queues and end-up being transmitted out of order.
> +
> +In some cases this can result in a considerable loss of performance.
> +
> +In such situations, XPS should not be enabled at runtime, or
> +explicitly disabled if the NIC driver(s) in question enable it on
> +their own.  Othersise, if possible, the VMs should be CPU pinned.
> +

I think you meant "Otherwise" instead of "Othersise".

>  ==== XPS Configuration
>
> -XPS is only available if the kconfig symbol CONFIG_XPS is enabled (on by
> -default for SMP). The functionality remains disabled until explicitly
> -configured. To enable XPS, the bitmap of CPUs that may use a transmit
> -queue is configured using the sysfs file entry:
> +XPS is available only if the kconfig symbol CONFIG_XPS is enabled
> +prior to building the kernel.  It is enabled by default for SMP kernel
> +configurations.  In many cases the functionality remains disabled at
> +runtime until explicitly configured by the system administrator. To
> +enable XPS, the bitmap of CPUs that may use a transmit queue is
> +configured using the sysfs file entry:
>
>  /sys/class/net/<dev>/queues/tx-<n>/xps_cpus
>
> +However, some NIC drivers will configure XPS at runtime for the
> +interfaces they drive, via a call to netif_set_xps_queue.
> +
>  == Suggested Configuration
>
>  For a network device with a single transmission queue, XPS configuration

So other than the one typo the rest of this looks good to me.

Reviewed-by: Alexander Duyck <alexander.h.duyck@intel.com>