* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-04-30 20:49 ` Shirley Ma
0 siblings, 0 replies; 22+ messages in thread
From: Shirley Ma @ 2014-04-30 20:49 UTC (permalink / raw)
To: Or Gerlitz, Chuck Lever
Cc: Jeff Becker, <yanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
Cayton, Phil, Coulter, Susan K, Anna Schumaker,
<Devesh.Sharma-laKkSmNT4hbQT0dZR+AlfA@public.gmane.org>,
Steve Wise, Allen Andrews, Jeff Layton, linux-rdma,
Linux NFS Mailing List, Trond Myklebust,
<emossman-zcUZCMB5SQOVc3sceRu5cw@public.gmane.org>
On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
>
>> If I understood Yan, he is trying to use NFS/RDMA in guests (kvm?). We
>> are pretty sure that is not working at the moment,
> can you provide a short 1-2 liner why/what is broken there? the only
> thing which I can think of to be not-supported over mlx4 VFs is the
> proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> mode which uses them, right?
I've created Xen guest on DomU. Dom0 PF works which has no mtts been
enabled, however DomU I hit this problem by just mounting the file system:
mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order 12)
mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order 12)
RDMA microbenchmark perftest works ok. I enabled mtts scripts when
booting the Xen guest. cat /proc/mtrr:
[root@ca-nfsdev1vm1 log]# cat /proc/mtrr
reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
lspci -v
00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
Virtual Function] (rev b0)
Subsystem: Mellanox Technologies Device 61b0
Physical Slot: 4
Flags: bus master, fast devsel, latency 0
Memory at f0000000 (64-bit, prefetchable) [size=128M]
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core
I will need to find another machine to try KVM guest. Yan might hit a
different problem.
I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he tried
it on KVM guest.
>> but that is a priority
>> to get fixed. Shirley has a lab set up and has been looking into it.
Shirley
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-04-30 23:58 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-04-30 23:58 UTC (permalink / raw)
To: Shirley Ma
Cc: Jeff Becker, yanb, Phil Cayton, Susan K Coulter, Anna Schumaker,
Devesh.Sharma, Steve Wise, Allen Andrews, Jeff Layton, linux-rdma,
Linux NFS Mailing List, Trond Myklebust, emossman, Or Gerlitz,
Chuck Lever
On 04/302014 Shirley Ma wrote:
> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> > On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
> > <chuck.lever@oracle.com>
> >
> >> If I understood Yan, he is trying to use NFS/RDMA in guests
> >> (kvm?). We
> >> are pretty sure that is not working at the moment,
> > can you provide a short 1-2 liner why/what is broken there? the
> > only
> > thing which I can think of to be not-supported over mlx4 VFs is the
> > proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> > mode which uses them, right?
> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
> enabled, however DomU I hit this problem by just mounting the file
> system:
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
>
> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> booting the Xen guest. cat /proc/mtrr:
What OS/RDMA stack are you using? I'm not familiar with any mtts
scripts, however I know there is an mtrr fixup script I wrote for
the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
too, but I haven't checked). In fact, I assume that's the script
you are referring to based on the fact that your next bit of your
email cats the /proc/mtrr file. But I don't believe whether there
is an mtrr setting mixup or not that is should have any impact on
the mtts allocations in the driver. Even if your mtrr registers
were set incorrectly, the problem then becomes either A) a serious
performance bottleneck (in the case of Intel hardware that needs
write combining in order to get more than about 50MByte/s of
throughput on their cards) or B) failed operation because MMIO
writes to the card are being cached/write combined when they should
not be.
I suspect this is more likely Xen related than mtts/mtrr related.
> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>
> lspci -v
> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
> Virtual Function] (rev b0)
> Subsystem: Mellanox Technologies Device 61b0
> Physical Slot: 4
> Flags: bus master, fast devsel, latency 0
> Memory at f0000000 (64-bit, prefetchable) [size=128M]
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
> Kernel driver in use: mlx4_core
> Kernel modules: mlx4_core
>
> I will need to find another machine to try KVM guest. Yan might hit a
> different problem.
>
> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
> tried
> it on KVM guest.
> >> but that is a priority
> >> to get fixed. Shirley has a lab set up and has been looking into
> >> it.
> Shirley
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-04-30 23:58 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-04-30 23:58 UTC (permalink / raw)
To: Shirley Ma
Cc: Jeff Becker, yanb, Phil Cayton, Susan K Coulter, Anna Schumaker,
Devesh.Sharma, Steve Wise, Allen Andrews, Jeff Layton, linux-rdma,
Linux NFS Mailing List, Trond Myklebust, emossman, Or Gerlitz,
Chuck Lever
On 04/302014 Shirley Ma wrote:
> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> > On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
> > <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> >
> >> If I understood Yan, he is trying to use NFS/RDMA in guests
> >> (kvm?). We
> >> are pretty sure that is not working at the moment,
> > can you provide a short 1-2 liner why/what is broken there? the
> > only
> > thing which I can think of to be not-supported over mlx4 VFs is the
> > proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> > mode which uses them, right?
> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
> enabled, however DomU I hit this problem by just mounting the file
> system:
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
>
> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> booting the Xen guest. cat /proc/mtrr:
What OS/RDMA stack are you using? I'm not familiar with any mtts
scripts, however I know there is an mtrr fixup script I wrote for
the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
too, but I haven't checked). In fact, I assume that's the script
you are referring to based on the fact that your next bit of your
email cats the /proc/mtrr file. But I don't believe whether there
is an mtrr setting mixup or not that is should have any impact on
the mtts allocations in the driver. Even if your mtrr registers
were set incorrectly, the problem then becomes either A) a serious
performance bottleneck (in the case of Intel hardware that needs
write combining in order to get more than about 50MByte/s of
throughput on their cards) or B) failed operation because MMIO
writes to the card are being cached/write combined when they should
not be.
I suspect this is more likely Xen related than mtts/mtrr related.
> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>
> lspci -v
> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
> Virtual Function] (rev b0)
> Subsystem: Mellanox Technologies Device 61b0
> Physical Slot: 4
> Flags: bus master, fast devsel, latency 0
> Memory at f0000000 (64-bit, prefetchable) [size=128M]
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
> Kernel driver in use: mlx4_core
> Kernel modules: mlx4_core
>
> I will need to find another machine to try KVM guest. Yan might hit a
> different problem.
>
> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
> tried
> it on KVM guest.
> >> but that is a priority
> >> to get fixed. Shirley has a lab set up and has been looking into
> >> it.
> Shirley
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
2014-04-30 20:49 ` Shirley Ma
(?)
(?)
@ 2014-04-30 23:58 ` Doug Ledford
2014-05-01 5:16 ` Shirley Ma
-1 siblings, 1 reply; 22+ messages in thread
From: Doug Ledford @ 2014-04-30 23:58 UTC (permalink / raw)
To: Shirley Ma
Cc: Jeff Becker, yanb, Phil Cayton, Susan K Coulter, Anna Schumaker,
Devesh.Sharma, Steve Wise, Allen Andrews, Jeff Layton, linux-rdma,
Linux NFS Mailing List, Trond Myklebust, emossman, Or Gerlitz,
Chuck Lever
On 04/302014 Shirley Ma wrote:
> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> > On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
> > <chuck.lever@oracle.com>
> >
> >> If I understood Yan, he is trying to use NFS/RDMA in guests
> >> (kvm?). We
> >> are pretty sure that is not working at the moment,
> > can you provide a short 1-2 liner why/what is broken there? the
> > only
> > thing which I can think of to be not-supported over mlx4 VFs is the
> > proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> > mode which uses them, right?
> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
> enabled, however DomU I hit this problem by just mounting the file
> system:
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
>
> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> booting the Xen guest. cat /proc/mtrr:
What OS/RDMA stack are you using? I'm not familiar with any mtts
scripts, however I know there is an mtrr fixup script I wrote for
the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
too, but I haven't checked). In fact, I assume that's the script
you are referring to based on the fact that your next bit of your
email cats the /proc/mtrr file. But I don't believe whether there
is an mtrr setting mixup or not that is should have any impact on
the mtts allocations in the driver. Even if your mtrr registers
were set incorrectly, the problem then becomes either A) a serious
performance bottleneck (in the case of Intel hardware that needs
write combining in order to get more than about 50MByte/s of
throughput on their cards) or B) failed operation because MMIO
writes to the card are being cached/write combined when they should
not be.
I suspect this is more likely Xen related than mtts/mtrr related.
> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>
> lspci -v
> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
> Virtual Function] (rev b0)
> Subsystem: Mellanox Technologies Device 61b0
> Physical Slot: 4
> Flags: bus master, fast devsel, latency 0
> Memory at f0000000 (64-bit, prefetchable) [size=128M]
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
> Kernel driver in use: mlx4_core
> Kernel modules: mlx4_core
>
> I will need to find another machine to try KVM guest. Yan might hit a
> different problem.
>
> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
> tried
> it on KVM guest.
> >> but that is a priority
> >> to get fixed. Shirley has a lab set up and has been looking into
> >> it.
> Shirley
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
2014-04-30 23:58 ` Doug Ledford
@ 2014-05-01 5:16 ` Shirley Ma
0 siblings, 0 replies; 22+ messages in thread
From: Shirley Ma @ 2014-05-01 5:16 UTC (permalink / raw)
To: Doug Ledford
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 04/30/2014 04:58 PM, Doug Ledford wrote:
> On 04/302014 Shirley Ma wrote:
>> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
>>> On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
>>> <chuck.lever@oracle.com>
>>>
>>>> If I understood Yan, he is trying to use NFS/RDMA in guests
>>>> (kvm?). We
>>>> are pretty sure that is not working at the moment,
>>> can you provide a short 1-2 liner why/what is broken there? the
>>> only
>>> thing which I can think of to be not-supported over mlx4 VFs is the
>>> proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
>>> mode which uses them, right?
>> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
>> enabled, however DomU I hit this problem by just mounting the file
>> system:
>> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
>> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
>> 12)
>> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
>> 12)
>>
>> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
>> booting the Xen guest. cat /proc/mtrr:
> What OS/RDMA stack are you using? I'm not familiar with any mtts
> scripts, however I know there is an mtrr fixup script I wrote for
> the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> too, but I haven't checked). In fact, I assume that's the script
> you are referring to based on the fact that your next bit of your
> email cats the /proc/mtrr file. But I don't believe whether there
> is an mtrr setting mixup or not that is should have any impact on
> the mtts allocations in the driver. Even if your mtrr registers
> were set incorrectly, the problem then becomes either A) a serious
> performance bottleneck (in the case of Intel hardware that needs
> write combining in order to get more than about 50MByte/s of
> throughput on their cards) or B) failed operation because MMIO
> writes to the card are being cached/write combined when they should
> not be.
>
> I suspect this is more likely Xen related than mtts/mtrr related.
Yes. That's the script I used. I wonder whether it's possible to disable
mtrr on DomU guest to debug this. I am new to Xen.
>> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
>> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
>> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>>
>> lspci -v
>> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
>> Virtual Function] (rev b0)
>> Subsystem: Mellanox Technologies Device 61b0
>> Physical Slot: 4
>> Flags: bus master, fast devsel, latency 0
>> Memory at f0000000 (64-bit, prefetchable) [size=128M]
>> Capabilities: [60] Express Endpoint, MSI 00
>> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
>> Kernel driver in use: mlx4_core
>> Kernel modules: mlx4_core
>>
>> I will need to find another machine to try KVM guest. Yan might hit a
>> different problem.
>>
>> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
>> tried
>> it on KVM guest.
>>>> but that is a priority
>>>> to get fixed. Shirley has a lab set up and has been looking into
>>>> it.
>> Shirley
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-05-01 5:16 ` Shirley Ma
0 siblings, 0 replies; 22+ messages in thread
From: Shirley Ma @ 2014-05-01 5:16 UTC (permalink / raw)
To: Doug Ledford
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 04/30/2014 04:58 PM, Doug Ledford wrote:
> On 04/302014 Shirley Ma wrote:
>> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
>>> On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
>>> <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
>>>
>>>> If I understood Yan, he is trying to use NFS/RDMA in guests
>>>> (kvm?). We
>>>> are pretty sure that is not working at the moment,
>>> can you provide a short 1-2 liner why/what is broken there? the
>>> only
>>> thing which I can think of to be not-supported over mlx4 VFs is the
>>> proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
>>> mode which uses them, right?
>> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
>> enabled, however DomU I hit this problem by just mounting the file
>> system:
>> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
>> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
>> 12)
>> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
>> 12)
>>
>> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
>> booting the Xen guest. cat /proc/mtrr:
> What OS/RDMA stack are you using? I'm not familiar with any mtts
> scripts, however I know there is an mtrr fixup script I wrote for
> the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> too, but I haven't checked). In fact, I assume that's the script
> you are referring to based on the fact that your next bit of your
> email cats the /proc/mtrr file. But I don't believe whether there
> is an mtrr setting mixup or not that is should have any impact on
> the mtts allocations in the driver. Even if your mtrr registers
> were set incorrectly, the problem then becomes either A) a serious
> performance bottleneck (in the case of Intel hardware that needs
> write combining in order to get more than about 50MByte/s of
> throughput on their cards) or B) failed operation because MMIO
> writes to the card are being cached/write combined when they should
> not be.
>
> I suspect this is more likely Xen related than mtts/mtrr related.
Yes. That's the script I used. I wonder whether it's possible to disable
mtrr on DomU guest to debug this. I am new to Xen.
>> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
>> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
>> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>>
>> lspci -v
>> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
>> Virtual Function] (rev b0)
>> Subsystem: Mellanox Technologies Device 61b0
>> Physical Slot: 4
>> Flags: bus master, fast devsel, latency 0
>> Memory at f0000000 (64-bit, prefetchable) [size=128M]
>> Capabilities: [60] Express Endpoint, MSI 00
>> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
>> Kernel driver in use: mlx4_core
>> Kernel modules: mlx4_core
>>
>> I will need to find another machine to try KVM guest. Yan might hit a
>> different problem.
>>
>> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
>> tried
>> it on KVM guest.
>>>> but that is a priority
>>>> to get fixed. Shirley has a lab set up and has been looking into
>>>> it.
>> Shirley
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
>> in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-05-01 16:03 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-05-01 16:03 UTC (permalink / raw)
To: Shirley Ma
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 05/01/2014, Shirley Ma wrote:
> On 04/30/2014 04:58 PM, Doug Ledford wrote:
> > On 04/302014 Shirley Ma wrote:
> >> I've created Xen guest on DomU. Dom0 PF works which has no mtts
> >> been
> >> enabled, however DomU I hit this problem by just mounting the file
> >> system:
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order
> >> 7)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >>
> >> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> >> booting the Xen guest. cat /proc/mtrr:
> > What OS/RDMA stack are you using? I'm not familiar with any mtts
> > scripts, however I know there is an mtrr fixup script I wrote for
> > the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> > too, but I haven't checked). In fact, I assume that's the script
> > you are referring to based on the fact that your next bit of your
> > email cats the /proc/mtrr file. But I don't believe whether there
> > is an mtrr setting mixup or not that is should have any impact on
> > the mtts allocations in the driver. Even if your mtrr registers
> > were set incorrectly, the problem then becomes either A) a serious
> > performance bottleneck (in the case of Intel hardware that needs
> > write combining in order to get more than about 50MByte/s of
> > throughput on their cards) or B) failed operation because MMIO
> > writes to the card are being cached/write combined when they should
> > not be.
> >
> > I suspect this is more likely Xen related than mtts/mtrr related.
> Yes. That's the script I used. I wonder whether it's possible to
> disable
> mtrr on DomU guest to debug this. I am new to Xen.
No, it's not possible to disable mtrr and expect any pass through
PCI devices to work. The mtrr registers merely indicate what
portion of the memory map should be treated as normal memory (meaning
cacheable) and what should be treated as MMIO memory (meaning generally
non-cacheable). That's all they do. The mtts table allocation
failures are actually totally different. In the VF on DomU they
are passed to the PF on Dom0 and the command is done there. However,
the number of mtts available for the slave are limited (check the
code in resource_tracker.c). In addition, the number of mtts
allocated is proportionally related to memory size in the guest
and inversely related to the log2_mtts_per_seg (probably both on
Dom0 and DomU, which I suspect need to agree on the log2_mtts_per_seg
module parameter). I would try a combination of reducing the memory
in the quest, or increasing the log2_mtts_per_seg, or both, and
see if you can get it to work.
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-05-01 16:03 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-05-01 16:03 UTC (permalink / raw)
To: Shirley Ma
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 05/01/2014, Shirley Ma wrote:
> On 04/30/2014 04:58 PM, Doug Ledford wrote:
> > On 04/302014 Shirley Ma wrote:
> >> I've created Xen guest on DomU. Dom0 PF works which has no mtts
> >> been
> >> enabled, however DomU I hit this problem by just mounting the file
> >> system:
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order
> >> 7)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >>
> >> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> >> booting the Xen guest. cat /proc/mtrr:
> > What OS/RDMA stack are you using? I'm not familiar with any mtts
> > scripts, however I know there is an mtrr fixup script I wrote for
> > the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> > too, but I haven't checked). In fact, I assume that's the script
> > you are referring to based on the fact that your next bit of your
> > email cats the /proc/mtrr file. But I don't believe whether there
> > is an mtrr setting mixup or not that is should have any impact on
> > the mtts allocations in the driver. Even if your mtrr registers
> > were set incorrectly, the problem then becomes either A) a serious
> > performance bottleneck (in the case of Intel hardware that needs
> > write combining in order to get more than about 50MByte/s of
> > throughput on their cards) or B) failed operation because MMIO
> > writes to the card are being cached/write combined when they should
> > not be.
> >
> > I suspect this is more likely Xen related than mtts/mtrr related.
> Yes. That's the script I used. I wonder whether it's possible to
> disable
> mtrr on DomU guest to debug this. I am new to Xen.
No, it's not possible to disable mtrr and expect any pass through
PCI devices to work. The mtrr registers merely indicate what
portion of the memory map should be treated as normal memory (meaning
cacheable) and what should be treated as MMIO memory (meaning generally
non-cacheable). That's all they do. The mtts table allocation
failures are actually totally different. In the VF on DomU they
are passed to the PF on Dom0 and the command is done there. However,
the number of mtts available for the slave are limited (check the
code in resource_tracker.c). In addition, the number of mtts
allocated is proportionally related to memory size in the guest
and inversely related to the log2_mtts_per_seg (probably both on
Dom0 and DomU, which I suspect need to agree on the log2_mtts_per_seg
module parameter). I would try a combination of reducing the memory
in the quest, or increasing the log2_mtts_per_seg, or both, and
see if you can get it to work.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-05-01 16:03 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-05-01 16:03 UTC (permalink / raw)
To: Shirley Ma
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 05/01/2014, Shirley Ma wrote:
> On 04/30/2014 04:58 PM, Doug Ledford wrote:
> > On 04/302014 Shirley Ma wrote:
> >> I've created Xen guest on DomU. Dom0 PF works which has no mtts
> >> been
> >> enabled, however DomU I hit this problem by just mounting the file
> >> system:
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order
> >> 7)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >>
> >> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> >> booting the Xen guest. cat /proc/mtrr:
> > What OS/RDMA stack are you using? I'm not familiar with any mtts
> > scripts, however I know there is an mtrr fixup script I wrote for
> > the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> > too, but I haven't checked). In fact, I assume that's the script
> > you are referring to based on the fact that your next bit of your
> > email cats the /proc/mtrr file. But I don't believe whether there
> > is an mtrr setting mixup or not that is should have any impact on
> > the mtts allocations in the driver. Even if your mtrr registers
> > were set incorrectly, the problem then becomes either A) a serious
> > performance bottleneck (in the case of Intel hardware that needs
> > write combining in order to get more than about 50MByte/s of
> > throughput on their cards) or B) failed operation because MMIO
> > writes to the card are being cached/write combined when they should
> > not be.
> >
> > I suspect this is more likely Xen related than mtts/mtrr related.
> Yes. That's the script I used. I wonder whether it's possible to
> disable
> mtrr on DomU guest to debug this. I am new to Xen.
No, it's not possible to disable mtrr and expect any pass through
PCI devices to work. The mtrr registers merely indicate what
portion of the memory map should be treated as normal memory (meaning
cacheable) and what should be treated as MMIO memory (meaning generally
non-cacheable). That's all they do. The mtts table allocation
failures are actually totally different. In the VF on DomU they
are passed to the PF on Dom0 and the command is done there. However,
the number of mtts available for the slave are limited (check the
code in resource_tracker.c). In addition, the number of mtts
allocated is proportionally related to memory size in the guest
and inversely related to the log2_mtts_per_seg (probably both on
Dom0 and DomU, which I suspect need to agree on the log2_mtts_per_seg
module parameter). I would try a combination of reducing the memory
in the quest, or increasing the log2_mtts_per_seg, or both, and
see if you can get it to work.
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-05-01 16:03 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-05-01 16:03 UTC (permalink / raw)
To: Shirley Ma
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 05/01/2014, Shirley Ma wrote:
> On 04/30/2014 04:58 PM, Doug Ledford wrote:
> > On 04/302014 Shirley Ma wrote:
> >> I've created Xen guest on DomU. Dom0 PF works which has no mtts
> >> been
> >> enabled, however DomU I hit this problem by just mounting the file
> >> system:
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order
> >> 7)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >>
> >> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> >> booting the Xen guest. cat /proc/mtrr:
> > What OS/RDMA stack are you using? I'm not familiar with any mtts
> > scripts, however I know there is an mtrr fixup script I wrote for
> > the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> > too, but I haven't checked). In fact, I assume that's the script
> > you are referring to based on the fact that your next bit of your
> > email cats the /proc/mtrr file. But I don't believe whether there
> > is an mtrr setting mixup or not that is should have any impact on
> > the mtts allocations in the driver. Even if your mtrr registers
> > were set incorrectly, the problem then becomes either A) a serious
> > performance bottleneck (in the case of Intel hardware that needs
> > write combining in order to get more than about 50MByte/s of
> > throughput on their cards) or B) failed operation because MMIO
> > writes to the card are being cached/write combined when they should
> > not be.
> >
> > I suspect this is more likely Xen related than mtts/mtrr related.
> Yes. That's the script I used. I wonder whether it's possible to
> disable
> mtrr on DomU guest to debug this. I am new to Xen.
No, it's not possible to disable mtrr and expect any pass through
PCI devices to work. The mtrr registers merely indicate what
portion of the memory map should be treated as normal memory (meaning
cacheable) and what should be treated as MMIO memory (meaning generally
non-cacheable). That's all they do. The mtts table allocation
failures are actually totally different. In the VF on DomU they
are passed to the PF on Dom0 and the command is done there. However,
the number of mtts available for the slave are limited (check the
code in resource_tracker.c). In addition, the number of mtts
allocated is proportionally related to memory size in the guest
and inversely related to the log2_mtts_per_seg (probably both on
Dom0 and DomU, which I suspect need to agree on the log2_mtts_per_seg
module parameter). I would try a combination of reducing the memory
in the quest, or increasing the log2_mtts_per_seg, or both, and
see if you can get it to work.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
2014-05-01 5:16 ` Shirley Ma
` (2 preceding siblings ...)
(?)
@ 2014-05-01 16:03 ` Doug Ledford
-1 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-05-01 16:03 UTC (permalink / raw)
To: Shirley Ma
Cc: yanb, Phil Cayton, Susan K Coulter, Anna Schumaker, Devesh.Sharma,
Steve Wise, Allen Andrews, linux-rdma, Linux NFS Mailing List,
Or Gerlitz, Chuck Lever
On 05/01/2014, Shirley Ma wrote:
> On 04/30/2014 04:58 PM, Doug Ledford wrote:
> > On 04/302014 Shirley Ma wrote:
> >> I've created Xen guest on DomU. Dom0 PF works which has no mtts
> >> been
> >> enabled, however DomU I hit this problem by just mounting the file
> >> system:
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order
> >> 7)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096
> >> pages(order
> >> 12)
> >>
> >> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> >> booting the Xen guest. cat /proc/mtrr:
> > What OS/RDMA stack are you using? I'm not familiar with any mtts
> > scripts, however I know there is an mtrr fixup script I wrote for
> > the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
> > too, but I haven't checked). In fact, I assume that's the script
> > you are referring to based on the fact that your next bit of your
> > email cats the /proc/mtrr file. But I don't believe whether there
> > is an mtrr setting mixup or not that is should have any impact on
> > the mtts allocations in the driver. Even if your mtrr registers
> > were set incorrectly, the problem then becomes either A) a serious
> > performance bottleneck (in the case of Intel hardware that needs
> > write combining in order to get more than about 50MByte/s of
> > throughput on their cards) or B) failed operation because MMIO
> > writes to the card are being cached/write combined when they should
> > not be.
> >
> > I suspect this is more likely Xen related than mtts/mtrr related.
> Yes. That's the script I used. I wonder whether it's possible to
> disable
> mtrr on DomU guest to debug this. I am new to Xen.
No, it's not possible to disable mtrr and expect any pass through
PCI devices to work. The mtrr registers merely indicate what
portion of the memory map should be treated as normal memory (meaning
cacheable) and what should be treated as MMIO memory (meaning generally
non-cacheable). That's all they do. The mtts table allocation
failures are actually totally different. In the VF on DomU they
are passed to the PF on Dom0 and the command is done there. However,
the number of mtts available for the slave are limited (check the
code in resource_tracker.c). In addition, the number of mtts
allocated is proportionally related to memory size in the guest
and inversely related to the log2_mtts_per_seg (probably both on
Dom0 and DomU, which I suspect need to agree on the log2_mtts_per_seg
module parameter). I would try a combination of reducing the memory
in the quest, or increasing the log2_mtts_per_seg, or both, and
see if you can get it to work.
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-04-30 23:58 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-04-30 23:58 UTC (permalink / raw)
To: Shirley Ma
Cc: Jeff Becker, yanb, Phil Cayton, Susan K Coulter, Anna Schumaker,
Devesh.Sharma, Steve Wise, Allen Andrews, Jeff Layton, linux-rdma,
Linux NFS Mailing List, Trond Myklebust, emossman, Or Gerlitz,
Chuck Lever
On 04/302014 Shirley Ma wrote:
> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> > On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
> > <chuck.lever@oracle.com>
> >
> >> If I understood Yan, he is trying to use NFS/RDMA in guests
> >> (kvm?). We
> >> are pretty sure that is not working at the moment,
> > can you provide a short 1-2 liner why/what is broken there? the
> > only
> > thing which I can think of to be not-supported over mlx4 VFs is the
> > proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> > mode which uses them, right?
> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
> enabled, however DomU I hit this problem by just mounting the file
> system:
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
>
> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> booting the Xen guest. cat /proc/mtrr:
What OS/RDMA stack are you using? I'm not familiar with any mtts
scripts, however I know there is an mtrr fixup script I wrote for
the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
too, but I haven't checked). In fact, I assume that's the script
you are referring to based on the fact that your next bit of your
email cats the /proc/mtrr file. But I don't believe whether there
is an mtrr setting mixup or not that is should have any impact on
the mtts allocations in the driver. Even if your mtrr registers
were set incorrectly, the problem then becomes either A) a serious
performance bottleneck (in the case of Intel hardware that needs
write combining in order to get more than about 50MByte/s of
throughput on their cards) or B) failed operation because MMIO
writes to the card are being cached/write combined when they should
not be.
I suspect this is more likely Xen related than mtts/mtrr related.
> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>
> lspci -v
> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
> Virtual Function] (rev b0)
> Subsystem: Mellanox Technologies Device 61b0
> Physical Slot: 4
> Flags: bus master, fast devsel, latency 0
> Memory at f0000000 (64-bit, prefetchable) [size=128M]
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
> Kernel driver in use: mlx4_core
> Kernel modules: mlx4_core
>
> I will need to find another machine to try KVM guest. Yan might hit a
> different problem.
>
> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
> tried
> it on KVM guest.
> >> but that is a priority
> >> to get fixed. Shirley has a lab set up and has been looking into
> >> it.
> Shirley
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: NFSoRDMA developers bi-weekly meeting announcement (4/30)
@ 2014-04-30 23:58 ` Doug Ledford
0 siblings, 0 replies; 22+ messages in thread
From: Doug Ledford @ 2014-04-30 23:58 UTC (permalink / raw)
To: Shirley Ma
Cc: Jeff Becker, yanb, Phil Cayton, Susan K Coulter, Anna Schumaker,
Devesh.Sharma, Steve Wise, Allen Andrews, Jeff Layton, linux-rdma,
Linux NFS Mailing List, Trond Myklebust, emossman, Or Gerlitz,
Chuck Lever
On 04/302014 Shirley Ma wrote:
> On 04/30/2014 01:00 PM, Or Gerlitz wrote:
> > On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever
> > <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> >
> >> If I understood Yan, he is trying to use NFS/RDMA in guests
> >> (kvm?). We
> >> are pretty sure that is not working at the moment,
> > can you provide a short 1-2 liner why/what is broken there? the
> > only
> > thing which I can think of to be not-supported over mlx4 VFs is the
> > proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a
> > mode which uses them, right?
> I've created Xen guest on DomU. Dom0 PF works which has no mtts been
> enabled, however DomU I hit this problem by just mounting the file
> system:
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order
> 12)
>
> RDMA microbenchmark perftest works ok. I enabled mtts scripts when
> booting the Xen guest. cat /proc/mtrr:
What OS/RDMA stack are you using? I'm not familiar with any mtts
scripts, however I know there is an mtrr fixup script I wrote for
the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux
too, but I haven't checked). In fact, I assume that's the script
you are referring to based on the fact that your next bit of your
email cats the /proc/mtrr file. But I don't believe whether there
is an mtrr setting mixup or not that is should have any impact on
the mtts allocations in the driver. Even if your mtrr registers
were set incorrectly, the problem then becomes either A) a serious
performance bottleneck (in the case of Intel hardware that needs
write combining in order to get more than about 50MByte/s of
throughput on their cards) or B) failed operation because MMIO
writes to the card are being cached/write combined when they should
not be.
I suspect this is more likely Xen related than mtts/mtrr related.
> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr
> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable
> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable
>
> lspci -v
> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2
> Virtual Function] (rev b0)
> Subsystem: Mellanox Technologies Device 61b0
> Physical Slot: 4
> Flags: bus master, fast devsel, latency 0
> Memory at f0000000 (64-bit, prefetchable) [size=128M]
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked-
> Kernel driver in use: mlx4_core
> Kernel modules: mlx4_core
>
> I will need to find another machine to try KVM guest. Yan might hit a
> different problem.
>
> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he
> tried
> it on KVM guest.
> >> but that is a priority
> >> to get fixed. Shirley has a lab set up and has been looking into
> >> it.
> Shirley
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG KeyID: 0E572FDD
http://people.redhat.com/dledford
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 22+ messages in thread