Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)

[Shirley Ma <shirley.ma@oracle.com>]

On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever <chuck.lever@oracle.com>
>
>> If I understood Yan, he is trying to use NFS/RDMA in guests (kvm?).  We
>> are pretty sure that is not working at the moment,
> can you provide a short 1-2 liner why/what is broken there? the only
> thing which I can think of to be not-supported over mlx4 VFs is the
> proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> mode which uses them, right?
I've created Xen guest on DomU. Dom0 PF works which has no mtts been 
enabled, however DomU I hit this problem by just mounting the file system:
mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order 12)
mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order 12)

RDMA microbenchmark perftest works ok. I enabled mtts scripts when 
booting the Xen guest. cat /proc/mtrr:

[root@ca-nfsdev1vm1 log]# cat /proc/mtrr
reg00: base=0x0f0000000 ( 3840MB), size=  128MB, count=1: uncachable
reg01: base=0x0f8000000 ( 3968MB), size=   64MB, count=1: uncachable

lspci -v
00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2 
Virtual Function] (rev b0)
     Subsystem: Mellanox Technologies Device 61b0
     Physical Slot: 4
     Flags: bus master, fast devsel, latency 0
     Memory at f0000000 (64-bit, prefetchable) [size=128M]
     Capabilities: [60] Express Endpoint, MSI 00
     Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
     Kernel driver in use: mlx4_core
     Kernel modules: mlx4_core

I will need to find another machine to try KVM guest. Yan might hit a 
different problem.

I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he tried 
it on KVM guest.
>> but that is a priority
>> to get fixed. Shirley has a lab set up and has been looking into it.
Shirley