Re: [PATCH V4 0/6] Redesign SR-IOV on PowerNV

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On 08/19/2015 12:01 PM, Wei Yang wrote:
> In original design, it tries to group VFs to enable more number of VFs in the
> system, when VF BAR is bigger than 64MB. This design has a flaw in which one
> error on a VF will interfere other VFs in the same group.
>
> This patch series change this design by using M64 BAR in Single PE mode to
> cover only one VF BAR. By doing so, it gives absolute isolation between VFs.

With or without this patchset, this fails with a horrible loop of EEHs:
rmmod mlx4_en mlx4_ib mlx4_core
modprobe mlx4_core num_vfs=4 probe_vf=4 port_type_array=2,2 debug_level=1

No guest is needed, just boot and do these commands. The EEH error is 
pointing to a broken DMA address. iommu=nobypass fixed it for 4 VFs case 
but when I try 16 VFs, none is created.

What is the correct base tree and what hardware did you use for the testing 
_exactly_?

Mine is "Ethernet controller: Mellanox Technologies MT27520 Family 
[ConnectX-3 Pro]" with 128MB BARs and that works (just double checked - can 
create all 16 VFs) with PowerKVM 3.1 so it is not a hardware issue.



>
> v4:
>     * rebase the code on top of v4.2-rc7
>     * switch back to use the dynamic version of pe_num_map and m64_map
>     * split the memory allocation and PE assignment of pe_num_map to make it
>       more easy to read
>     * check pe_num_map value before free PE.
>     * add the rename reason for pe_num_map and m64_map in change log
> v3:
>     * return -ENOSPC when a VF has non-64bit prefetchable BAR
>     * rename offset to pe_num_map and define it staticly
>     * change commit log based on comments
>     * define m64_map staticly
> v2:
>     * clean up iov bar alignment calculation
>     * change m64s to m64_bars
>     * add a field to represent M64 Single PE mode will be used
>     * change m64_wins to m64_map
>     * calculate the gate instead of hard coded
>     * dynamically allocate m64_map
>     * dynamically allocate PE#
>     * add a case to calculate iov bar alignment when M64 Single PE is used
>     * when M64 Single PE is used, compare num_vfs with M64 BAR available number
>       in system at first
>
>
>
> Wei Yang (6):
>    powerpc/powernv: don't enable SRIOV when VF BAR has non
>      64bit-prefetchable BAR
>    powerpc/powernv: simplify the calculation of iov resource alignment
>    powerpc/powernv: use one M64 BAR in Single PE mode for one VF BAR
>    powerpc/powernv: replace the hard coded boundary with gate
>    powerpc/powernv: boundary the total VF BAR size instead of the
>      individual one
>    powerpc/powernv: allocate sparse PE# when using M64 BAR in Single PE
>      mode
>
>   arch/powerpc/include/asm/pci-bridge.h     |    7 +-
>   arch/powerpc/platforms/powernv/pci-ioda.c |  328 +++++++++++++++--------------
>   2 files changed, 175 insertions(+), 160 deletions(-)
>


-- 
Alexey