Re: [RFC PATCH kernel] Revert "net/mlx4_core: Set UAR page size to 4KB regardless of system page size"

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On 03/16/2016 05:09 PM, Eli Cohen wrote:
> On Wed, Mar 16, 2016 at 04:49:00PM +1100, Alexey Kardashevskiy wrote:
>> On 03/16/2016 04:10 PM, Eli Cohen wrote:
>>> On Wed, Mar 16, 2016 at 01:07:58PM +1100, Alexey Kardashevskiy wrote:
>>>>
>>>> So with v4.5 as a host, there is no actual distro available today to
>>>> use as a guest in the next 6 months (or whatever it takes to
>>>> backport this partucular patch back there).
>>>>
>>>> You could have added a module parameter to enforce the old behavoir,
>>>> at least...
>>>>
>>>> And sorry but from the original commit log I could not understand
>>>> why exactly all existing guests need to be broken. Could you please
>>>> point me to a piece of documentation describing all this UAR
>>>> bisuness (what is UAR, why 128 UARs are required and for what, etc).
>>>> Thanks.
>>>>
>>>
>>> We are going to send a patch that fixes this using a module parameter.
>>> The patch will be on top of Huy's patch.
>>>
>>> Some background to the problem: mlx4 supported devices require 128 UAR
>>
>> What does UAR stand for?
> User Access Region. It's the way you interface with the hardware.
>>
>>> pages from PCI memory space defined by BAR2-3. Each UAR page can be
>>> any power of 2 value from 4K up to 64K. Before Huy's patch the driver
>>> chose UAR page size to be equal to system page size. Since PowerPC's
>>> page size is 64K this means minimum requirement of UAR pages is not
>>> met (default UAR BAR is 8MB and only half of it is really reserved for
>>> UARs).
>>
>> And what was the downside? afaict the performance was good...
>>
>
> It's not a performance issue. Defining 64KB for a UAR is not required
> and wastes pci memory mapped i/o space.
>
>>
>>> More details can be found in the programmer's manual.
>>
>> Can you please point me to this manual on the website? I tried,
>> honestly, could not find it. Thanks.
>>
> It's not publically available. If you have an FAE that work with your
> company you can ask him how to get the doc.


Oh. ok. It also looks like even with the reverted patch, mlx4 VF does not 
work in a guest:

root@le-dbg:~# dhclient eth0
mlx4_en: eth0:   frag:0 - size:1518 prefix:0 stride:1536

mlx4_core 0000:00:00.0: Internal error detected on the communication channel
mlx4_core 0000:00:00.0: device is going to be reset
mlx4_core 0000:00:00.0: VF reset is not needed
mlx4_core 0000:00:00.0: device was reset successfully
mlx4_en 0000:00:00.0: Internal error detected, restarting device
mlx4_core 0000:00:00.0: command 0x5 failed: fw status = 0x1
mlx4_core 0000:00:00.0: Failed to close slave function
mlx4_core 0000:00:00.0: Detected virtual function - running in slave mode
mlx4_core 0000:00:00.0: Sending reset


mlx4_core 0000:00:00.0: slave is currently in the middle of FLR - Deferring 
probe
mlx4_core 0000:00:00.0: mlx4_restart_one: ERROR: mlx4_load_one failed, 
pci_name=0000:00:00.0, err=-517
mlx4_core 0000:00:00.0: mlx4_restart_one was ended, ret=-517

root@le-dbg:~# ifconfig -a
lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@le-dbg:~# lspci -v
00:00.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family 
[ConnectX-3/ConnectX-3 Pro Virtual Function]
         Subsystem: IBM Device 61b0
         Physical Slot: C16
         Flags: bus master, fast devsel, latency 0
         Memory at 10120000000 (64-bit, prefetchable) [size=64M]
         Capabilities: [60] Express Endpoint, MSI 00
         Capabilities: [9c] MSI-X: Enable- Count=52 Masked-
         Capabilities: [40] Power Management version 0
         Kernel driver in use: mlx4_core



-- 
Alexey