Xen network domain performance for 10Gb NIC

Xen network domain performance for 10Gb NIC

[tosher 1]

 Hi everyone,

Lately, I have been experimenting with 10Gb NIC performance on Xen domains. I have found that network performance is very poor for PV networking when a driver domain is used as a network backend.

My experimental setup is I have two machines connected by the 10Gb network: a server running the Xen hypervisor and a desktop machine working as a client. I have Ubuntu 18.04.3 LTS running on the Dom0, Domus, Driver Domain, and client desktop, where the Xen version is 4.9. I measured the network bandwidth using iPerf3.

The network bandwidth between a DomU using Dom0 as backend and the client desktop is like 9.39Gbits/sec. However, when I use a network driver domain, which has the 10Gb NIC by PCI pass through, the bandwidth between the DomU and the client desktop is like 2.41Gbit/sec is one direction and 4.48Gbits/sec in another direction. Here, by direction, I mean the client-server direction for iPerf3.

These results indicate a huge performance degradation, which is unexpected. I am wondering if I am missing any key points here which I should have taken care of or if there is any tweak that I can apply.

Thanks,
Mehrab

Re: Xen network domain performance for 10Gb NIC

[Jürgen Groß]

On 26.04.20 21:18, tosher 1 wrote:
>   Hi everyone,
> 
> Lately, I have been experimenting with 10Gb NIC performance on Xen domains. I have found that network performance is very poor for PV networking when a driver domain is used as a network backend.
> 
> My experimental setup is I have two machines connected by the 10Gb network: a server running the Xen hypervisor and a desktop machine working as a client. I have Ubuntu 18.04.3 LTS running on the Dom0, Domus, Driver Domain, and client desktop, where the Xen version is 4.9. I measured the network bandwidth using iPerf3.
> 
> The network bandwidth between a DomU using Dom0 as backend and the client desktop is like 9.39Gbits/sec. However, when I use a network driver domain, which has the 10Gb NIC by PCI pass through, the bandwidth between the DomU and the client desktop is like 2.41Gbit/sec is one direction and 4.48Gbits/sec in another direction. Here, by direction, I mean the client-server direction for iPerf3.
> 
> These results indicate a huge performance degradation, which is unexpected. I am wondering if I am missing any key points here which I should have taken care of or if there is any tweak that I can apply.

Is the driver domain PV or HVM?

How many vcpus do dom0, the driver domain and the guest have?


Juergen

Re: Xen network domain performance for 10Gb NIC

[tosher 1]

The driver domain is HVM. Both the driver domain and 






On Monday, April 27, 2020, 1:28:13 AM EDT, Jürgen Groß <jgross@suse.com> wrote: 




> Is the driver domain PV or HVM?

The driver domain is HVM.

> How many vcpus do dom0, the driver domain and the guest have?

Dom0 has 12 vcpus, pinned. Both the driver domain and the guest have 4 vcpus, pinned as well.


Juergen

Re: Xen network domain performance for 10Gb NIC

[Roger Pau Monné]

On Sun, Apr 26, 2020 at 07:18:33PM +0000, tosher 1 wrote:
>  Hi everyone,
> 
> Lately, I have been experimenting with 10Gb NIC performance on Xen domains. I have found that network performance is very poor for PV networking when a driver domain is used as a network backend.
> 
> My experimental setup is I have two machines connected by the 10Gb network: a server running the Xen hypervisor and a desktop machine working as a client. I have Ubuntu 18.04.3 LTS running on the Dom0, Domus, Driver Domain, and client desktop, where the Xen version is 4.9. I measured the network bandwidth using iPerf3.
> 
> The network bandwidth between a DomU using Dom0 as backend and the client desktop is like 9.39Gbits/sec. However, when I use a network driver domain, which has the 10Gb NIC by PCI pass through, the bandwidth between the DomU and the client desktop is like 2.41Gbit/sec is one direction and 4.48Gbits/sec in another direction. Here, by direction, I mean the client-server direction for iPerf3.
> 
> These results indicate a huge performance degradation, which is unexpected. I am wondering if I am missing any key points here which I should have taken care of or if there is any tweak that I can apply.

Driver domains with passthrough devices need to perform IOMMU
operations in order to add/remove page table entries when doing grant
maps (ie: IOMMU TLB flushes), while dom0 doesn't need to because it
has the whole RAM identity mapped in the IOMMU tables. Depending on
how fast your IOMMU is and what capabilities it has doing such
operations can introduce a significant amount of overhead.

I would give a try to a newer version of Xen also, there have been
some changes in IOMMU management, but I would guess your bottleneck
doesn't come from the code itself, but rather from the IOMMU.

Roger.

Re: Xen network domain performance for 10Gb NIC

[tosher 1]

> Driver domains with passthrough devices need to perform IOMMU
operations in order to add/remove page table entries when doing grant
maps (ie: IOMMU TLB flushes), while dom0 doesn't need to because it
has the whole RAM identity mapped in the IOMMU tables. Depending on
how fast your IOMMU is and what capabilities it has doing such
operations can introduce a significant amount of overhead.

It makes sense to me. Do you know, in general, how to measure IOMMU performance, and what features/properties of IOMMU can contribute to getting a better network throughput? Please let me know. 

Thanks!

Re: Xen network domain performance for 10Gb NIC

[Roger Pau Monné]

On Tue, Apr 28, 2020 at 05:23:55AM +0000, tosher 1 wrote:
> > Driver domains with passthrough devices need to perform IOMMU
> operations in order to add/remove page table entries when doing grant
> maps (ie: IOMMU TLB flushes), while dom0 doesn't need to because it
> has the whole RAM identity mapped in the IOMMU tables. Depending on
> how fast your IOMMU is and what capabilities it has doing such
> operations can introduce a significant amount of overhead.
> 
> It makes sense to me. Do you know, in general, how to measure IOMMU performance, and what features/properties of IOMMU can contribute to getting a better network throughput? Please let me know. 

Hm, not really. You would have to modify Xen and/or the Linux kernel
in order to time the IOMMU or the grant map operations in order to get
an idea.

Do you get the expected performance from the driver domain when not
using it as a backend? Ie: running the iperf benchmarks directly on
the driver domain and not on the guest.

Roger.

Re: Xen network domain performance for 10Gb NIC

[tosher 1]

> Do you get the expected performance from the driver domain when not
> using it as a backend? Ie: running the iperf benchmarks directly on
> the driver domain and not on the guest.


Yes, the bandwidth between the driver domain and the client machine is close to 10Gbits/sec.

Re: Xen network domain performance for 10Gb NIC

[Roger Pau Monné]

On Tue, Apr 28, 2020 at 04:08:08PM +0000, tosher 1 wrote:
> > Do you get the expected performance from the driver domain when not
> > using it as a backend? Ie: running the iperf benchmarks directly on
> > the driver domain and not on the guest.
> 
> 
> Yes, the bandwidth between the driver domain and the client machine is close to 10Gbits/sec.

Also note that doing grant map / unmap operations from HVM domains is
much more expensive than doing them from PV domains, even without an
IOMMU. Xen needs to modify the domain p2m and issue a flush, and then
the guest (usually) needs to map the newly populated memory in it's
page tables in order to access it.

This and the addition of the IOMMU TLB flush could well explain why
you are seeing such a degraded performance. I'm afraid the only way to
figure out exactly what is causing the bottleneck will be to time the
different operations in Xen. You can compare against a PV dom0 in
order to get an idea.

Roger.