Re: mellanox mlx4_core and SR-IOV

[Lukas Hejtmanek <xhejtman@ics.muni.cz>]

Hi,

On Fri, Aug 03, 2012 at 06:49:59AM -0700, Konrad Wilk wrote:
> This looks like you are using PV PCI passthrough? If so, did you
> remember to use 'iommu=soft' to enable the Xen-SWIOTLB in your guest?
> And are you booting with more than 4GB? Or is less than 3GB (so that you have
> a nice gap in E820).

good catch. I forgot to pass swiotl=force for DomU in Xen. So now, it seems
that mlx4_core works, mlx4_en (ethernet part) works as well. Unfortunately,
the IB part does not. IB layer complains that SR-IOV is currently unsupported
(kernel 3.5.0). So no luck here so far. 

There is OFED stack directly from Mellanox, that seems to support SR-IOV even
for IB layer, but they have buildable sources only for RHEL/SLES kernels
(2.6.32) and even correcting the sources to get it compile with 3.5.0 does not
make things work. The driver complains about interrupts not working in Dom0 or
even without Xen hypervisor at all.

The only good point is, that I managed to convice Supermicro (board
manufacturer), that enabling SR-IOV in BIOS leads to BIOS lockup, they
confirmed it and maybe they provide BIOS upgrade.

Thanks all.

-- 
Lukáš Hejtmánek