Re: [RFC/RFT PATCH v3 0/4] KVM: x86: full virtualization of guest MTRR

[Or Gerlitz <ogerlitz@mellanox.com>]

On 7/8/2015 6:18 PM, Paolo Bonzini wrote:
> This part of the MTRR patches was dropped by Xiao.  Bring SVM on feature
> parity with VMX, and then do guest MTRR virtualization for both VMX and SVM.
>
> The IPAT bit of VMX extended page tables is emulated by mangling the guest
> PAT value.
>
> I do not have any AMD machines that support an IOMMU, so I would like
> some help testing these patches.  Thanks,
>
>

Hi Paolo,

We (finally) have results showing that the patches work well and provide 
benefit.

For getting better latency with ConnectX RDMA devices, we write send 
descriptors
to a write-combining (WC) mapped buffer instead of ringing a doorbell 
and having
the HW fetch the descriptor from system memory. In the mlx4 jargon, this 
optimization
is called Blue-Flame (BF).

To test the patches, we booted two hosts with 4.2-rc, patched with
this series, over which a legacy (RHEL 6.x) guests arerunning.

Under SRIOV, the mlx4 VF driver queries the mlx4 PF driver on the
host if BF is availablefor them to use.

We did two runs:

In [1] the guest VF driver was told by the host PF driver
that BF isn't supportedand hence they didn't use WC.

In [2] the VFs were told by the host PF driver that BF is supported
and hence used WC.

The results are provided in micro-seconds and account for half RTT of
nativeRDMA latency test, the WC advantage is notable, so +1for this series!

Or.


[1] guests not-using Blue-Flame / Write-Combining

root@host-194-168-80-68 ~]# ib_send_lat -a
  #bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
  2       1000          1.13           11.53        1.16
  4       1000          1.13           6.33         1.16
  8       1000          1.13           5.17         1.17
  16      1000          1.14           4.37         1.17
  32      1000          1.15           5.01         1.18
  64      1000          1.19           7.96         1.22
  128     1000          1.28           5.44         1.31
  256     1000          1.62           6.90         1.65
  512     1000          1.78           5.65         1.82


[2] guests using Blue-Flame when the host allows Write-Combining mapping 
by VMs

root@host-194-168-80-68 ~]# ib_send_lat -a
#bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
  2       1000          0.86           16.97        0.89
  4       1000          0.87           4.81         0.90
  8       1000          0.87           4.89         0.90
  16      1000          0.87           6.46         0.90
  32      1000          0.88           4.22         0.91
  64      1000          0.94           4.03         0.97
  128     1000          1.03           6.50         1.06
  256     1000          1.36           7.71         1.39
  512     1000          1.49           5.92         1.52