Manual loading of mlx4_en

Manual loading of mlx4_en

[Jean Delvare]

Hi all,

I have a usability concern with the mlx4_en ethernet network driver. The
module aliases for all the supported devices are on the mlx4_core
module, so mlx4_core gets loaded automatically, but mlx4_en does not. I
must load mlx4_en manually (kernel 3.12.)

May I ask why the mlx4 driver was split into mlx4_core and mlx4_en in
the first place?

I have worked around the problem with the following modprobe directive:

install mlx4_core /sbin/modprobe --ignore-install mlx4_core && /sbin/modprobe mlx4_en

But this is kind of a hack and I would expect the right driver to load
automatically without magic.

Shouldn't the module aliases be attached to mlx4_en instead of
mlx4_core?

Thanks,
-- 
Jean Delvare
SUSE L3 Support

Re: Manual loading of mlx4_en

[Amir Vadai]

Hi,

On 7/29/2014 12:53 PM, Jean Delvare wrote:
> Hi all,
> 
> I have a usability concern with the mlx4_en ethernet network driver. The
> module aliases for all the supported devices are on the mlx4_core
> module, so mlx4_core gets loaded automatically, but mlx4_en does not. I
> must load mlx4_en manually (kernel 3.12.)
> 
> May I ask why the mlx4 driver was split into mlx4_core and mlx4_en in
> the first place?
Connect-X card can support both Ethernet and Infiniband link layer - you
can switch between link layers without replacing the hardware.

To prevent code duplication, there is a single lower layer driver -
mlx4_core and two separate upper layer drivers - mlx4_en and mlx4_ib.

Since kernel 3.13 mlx4_core is loading automatically the upper layer
driver using request_module():
b046ffe net/mlx4_core: Load higher level modules according to ports type

> 
> I have worked around the problem with the following modprobe directive:
> 
> install mlx4_core /sbin/modprobe --ignore-install mlx4_core && /sbin/modprobe mlx4_en
> 
> But this is kind of a hack and I would expect the right driver to load
> automatically without magic.
> 
> Shouldn't the module aliases be attached to mlx4_en instead of
> mlx4_core?
> 
> Thanks,
> 

Amir

Re: Manual loading of mlx4_en

[Jean Delvare]

Hi Amir,

On Tue, 29 Jul 2014 13:16:42 +0300, Amir Vadai wrote:
> On 7/29/2014 12:53 PM, Jean Delvare wrote:
> > I have a usability concern with the mlx4_en ethernet network driver. The
> > module aliases for all the supported devices are on the mlx4_core
> > module, so mlx4_core gets loaded automatically, but mlx4_en does not. I
> > must load mlx4_en manually (kernel 3.12.)
> > 
> > May I ask why the mlx4 driver was split into mlx4_core and mlx4_en in
> > the first place?
> Connect-X card can support both Ethernet and Infiniband link layer - you
> can switch between link layers without replacing the hardware.
> 
> To prevent code duplication, there is a single lower layer driver -
> mlx4_core and two separate upper layer drivers - mlx4_en and mlx4_ib.
> 
> Since kernel 3.13 mlx4_core is loading automatically the upper layer
> driver using request_module():
> b046ffe net/mlx4_core: Load higher level modules according to ports type

Many thanks for the fast and detailed answer! I'll go backport this
commit to kernel 3.12 (and 3.0, hopefully it doesn't have too many
dependencies?)

-- 
Jean Delvare
SUSE L3 Support

Re: Manual loading of mlx4_en

[Amir Vadai]

On 7/29/2014 1:34 PM, Jean Delvare wrote:
> Hi Amir,
> 
> On Tue, 29 Jul 2014 13:16:42 +0300, Amir Vadai wrote:
>> On 7/29/2014 12:53 PM, Jean Delvare wrote:
>>> I have a usability concern with the mlx4_en ethernet network driver. The
>>> module aliases for all the supported devices are on the mlx4_core
>>> module, so mlx4_core gets loaded automatically, but mlx4_en does not. I
>>> must load mlx4_en manually (kernel 3.12.)
>>>
>>> May I ask why the mlx4 driver was split into mlx4_core and mlx4_en in
>>> the first place?
>> Connect-X card can support both Ethernet and Infiniband link layer - you
>> can switch between link layers without replacing the hardware.
>>
>> To prevent code duplication, there is a single lower layer driver -
>> mlx4_core and two separate upper layer drivers - mlx4_en and mlx4_ib.
>>
>> Since kernel 3.13 mlx4_core is loading automatically the upper layer
>> driver using request_module():
>> b046ffe net/mlx4_core: Load higher level modules according to ports type
> 
> Many thanks for the fast and detailed answer! I'll go backport this
> commit to kernel 3.12 (and 3.0, hopefully it doesn't have too many
> dependencies?)
> 

I don't think there is any dependency for that commit. If you're
backporting it, also take a small fix to it:
f24f790 net/mlx4_core: Load the Eth driver first

Amir

Re: Manual loading of mlx4_en

[Jean Delvare]

Le Tuesday 29 July 2014 à 13:44 +0300, Amir Vadai a écrit :
> On 7/29/2014 1:34 PM, Jean Delvare wrote:
> > Many thanks for the fast and detailed answer! I'll go backport this
> > commit to kernel 3.12 (and 3.0, hopefully it doesn't have too many
> > dependencies?)
>
> I don't think there is any dependency for that commit. If you're
> backporting it, also take a small fix to it:
> f24f790 net/mlx4_core: Load the Eth driver first

Thanks for the hint. I suppose I should pick 7855bff4 "net/mlx4_core:
Load the IB driver when the device supports IBoE" as well?

-- 
Jean Delvare
SUSE L3 Support

Re: Manual loading of mlx4_en

[Amir Vadai]

On 7/29/2014 3:06 PM, Jean Delvare wrote:
> Le Tuesday 29 July 2014 à 13:44 +0300, Amir Vadai a écrit :
>> On 7/29/2014 1:34 PM, Jean Delvare wrote:
>>> Many thanks for the fast and detailed answer! I'll go backport this
>>> commit to kernel 3.12 (and 3.0, hopefully it doesn't have too many
>>> dependencies?)
>>
>> I don't think there is any dependency for that commit. If you're
>> backporting it, also take a small fix to it:
>> f24f790 net/mlx4_core: Load the Eth driver first
> 
> Thanks for the hint. I suppose I should pick 7855bff4 "net/mlx4_core:
> Load the IB driver when the device supports IBoE" as well?
> 
Right - missed it.

Amir

Re: Manual loading of mlx4_en

[Benjamin Poirier]

On 2014/07/29 16:42, Amir Vadai wrote:
> On 7/29/2014 3:06 PM, Jean Delvare wrote:
> > Le Tuesday 29 July 2014 à 13:44 +0300, Amir Vadai a écrit :
> >> On 7/29/2014 1:34 PM, Jean Delvare wrote:
> >>> Many thanks for the fast and detailed answer! I'll go backport this
> >>> commit to kernel 3.12 (and 3.0, hopefully it doesn't have too many
> >>> dependencies?)
> >>
> >> I don't think there is any dependency for that commit. If you're
> >> backporting it, also take a small fix to it:
> >> f24f790 net/mlx4_core: Load the Eth driver first
> > 
> > Thanks for the hint. I suppose I should pick 7855bff4 "net/mlx4_core:
> > Load the IB driver when the device supports IBoE" as well?
> > 
> Right - missed it.

Thanks for the pointers.
Looking at f24f790f, I'm wondering if request_module(EN_DRV_NAME) ought
to be used instead of _nowait to guarantee the desired effect. Suppose
for example that the process is migrated to a different cpu between the
two request_module calls, the second one could complete before the first
one no?

Re: Manual loading of mlx4_en

[Amir Vadai]

On 7/30/2014 5:14 AM, Benjamin Poirier wrote:
> On 2014/07/29 16:42, Amir Vadai wrote:
>> On 7/29/2014 3:06 PM, Jean Delvare wrote:
>>> Le Tuesday 29 July 2014 à 13:44 +0300, Amir Vadai a écrit :
>>>> On 7/29/2014 1:34 PM, Jean Delvare wrote:
>>>>> Many thanks for the fast and detailed answer! I'll go backport this
>>>>> commit to kernel 3.12 (and 3.0, hopefully it doesn't have too many
>>>>> dependencies?)
>>>>
>>>> I don't think there is any dependency for that commit. If you're
>>>> backporting it, also take a small fix to it:
>>>> f24f790 net/mlx4_core: Load the Eth driver first
>>>
>>> Thanks for the hint. I suppose I should pick 7855bff4 "net/mlx4_core:
>>> Load the IB driver when the device supports IBoE" as well?
>>>
>> Right - missed it.
> 
> Thanks for the pointers.
> Looking at f24f790f, I'm wondering if request_module(EN_DRV_NAME) ought
> to be used instead of _nowait to guarantee the desired effect. Suppose
> for example that the process is migrated to a different cpu between the
> two request_module calls, the second one could complete before the first
> one no?
> 

The issue was addressed after the commit above, through these two
commits in a proper and robust way:
9433c18 IB/mlx4: Invoke UPDATE_QP for proxy QP1 on MAC changes
ce8d9e0 net/mlx4_core: Add UPDATE_QP SRIOV wrapper support

Thanks,
Amir