Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Alexei Starovoitov]

On Thu, Aug 30, 2018 at 1:35 AM, Tariq Toukan <tariqt@mellanox.com> wrote:
>
>
> On 29/08/2018 6:05 PM, Jesper Dangaard Brouer wrote:
>>
>> Hi Saeed,
>>
>> I'm having issues loading mlx5 driver on v4.19 kernels (tested both
>> net-next and bpf-next), while kernel v4.18 seems to work.  It happens
>> with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
>> from the system now).
>>
>
> Hi Jesper,
>
> Thanks for your report!
>
> We are working to analyze and debug the issue.

looks like serious issue to me... while no news in 2 weeks.
any update?

Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Jesper Dangaard Brouer]

On Thu, 13 Sep 2018 15:55:29 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:

> On Thu, Aug 30, 2018 at 1:35 AM, Tariq Toukan <tariqt@mellanox.com> wrote:
> >
> >
> > On 29/08/2018 6:05 PM, Jesper Dangaard Brouer wrote:  
> >>
> >> Hi Saeed,
> >>
> >> I'm having issues loading mlx5 driver on v4.19 kernels (tested both
> >> net-next and bpf-next), while kernel v4.18 seems to work.  It happens
> >> with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
> >> from the system now).
> >>  
> >
> > Hi Jesper,
> >
> > Thanks for your report!
> >
> > We are working to analyze and debug the issue.  
> 
> looks like serious issue to me... while no news in 2 weeks.
> any update?

Mellanox took it offlist, and Sep 6th found that this is a regression
introduced by commit 269d26f47f6f ("net/mlx5: Reduce command polling
interval"), but only if CONFIG_PREEMPT is on. 

I can confirm that reverting this commit fixed the issue (and not the
firmware upgrade I also did).

I think Moshe (Cc) is responsible for this case, and I expect to soon
see a revert or alternative solution to this!?

Thanks for the kick Alexei :-)
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Saeed Mahameed]

On Thu, Sep 13, 2018 at 11:36 PM, Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
> On Thu, 13 Sep 2018 15:55:29 -0700
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
>> On Thu, Aug 30, 2018 at 1:35 AM, Tariq Toukan <tariqt@mellanox.com> wrote:
>> >
>> >
>> > On 29/08/2018 6:05 PM, Jesper Dangaard Brouer wrote:
>> >>
>> >> Hi Saeed,
>> >>
>> >> I'm having issues loading mlx5 driver on v4.19 kernels (tested both
>> >> net-next and bpf-next), while kernel v4.18 seems to work.  It happens
>> >> with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
>> >> from the system now).
>> >>
>> >
>> > Hi Jesper,
>> >
>> > Thanks for your report!
>> >
>> > We are working to analyze and debug the issue.
>>
>> looks like serious issue to me... while no news in 2 weeks.
>> any update?
>
> Mellanox took it offlist, and Sep 6th found that this is a regression
> introduced by commit 269d26f47f6f ("net/mlx5: Reduce command polling
> interval"), but only if CONFIG_PREEMPT is on.
>
> I can confirm that reverting this commit fixed the issue (and not the
> firmware upgrade I also did).
>
> I think Moshe (Cc) is responsible for this case, and I expect to soon
> see a revert or alternative solution to this!?
>
> Thanks for the kick Alexei :-)

Thanks you Alexei and Jesper for following up,
the fix is already being tested [1] and will be submitted tomorrow,
as Jesper pointed out the issue happens only with 269d26f47f6f
("net/mlx5: Reduce command polling
interval"), and only if CONFIG_PREEMPT is on.
the only affected kernel is 4.19 which is not GA yet.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=net-mlx5

> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Jesper Dangaard Brouer]

On Fri, 14 Sep 2018 01:22:15 -0700
Saeed Mahameed <saeedm@dev.mellanox.co.il> wrote:

> On Thu, Sep 13, 2018 at 11:36 PM, Jesper Dangaard Brouer
> <brouer@redhat.com> wrote:
> > On Thu, 13 Sep 2018 15:55:29 -0700
> > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> >  
> >> On Thu, Aug 30, 2018 at 1:35 AM, Tariq Toukan <tariqt@mellanox.com> wrote:  
> >> >
> >> >
> >> > On 29/08/2018 6:05 PM, Jesper Dangaard Brouer wrote:  
> >> >>
> >> >> Hi Saeed,
> >> >>
> >> >> I'm having issues loading mlx5 driver on v4.19 kernels (tested both
> >> >> net-next and bpf-next), while kernel v4.18 seems to work.  It happens
> >> >> with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
> >> >> from the system now).
> >> >>  
> >> >
> >> > Hi Jesper,
> >> >
> >> > Thanks for your report!
> >> >
> >> > We are working to analyze and debug the issue.  
> >>
> >> looks like serious issue to me... while no news in 2 weeks.
> >> any update?  
> >
> > Mellanox took it offlist, and Sep 6th found that this is a regression
> > introduced by commit 269d26f47f6f ("net/mlx5: Reduce command polling
> > interval"), but only if CONFIG_PREEMPT is on.
> >
> > I can confirm that reverting this commit fixed the issue (and not the
> > firmware upgrade I also did).
> >
> > I think Moshe (Cc) is responsible for this case, and I expect to soon
> > see a revert or alternative solution to this!?
> >
> > Thanks for the kick Alexei :-)  
> 
> Thanks you Alexei and Jesper for following up,
> the fix is already being tested [1] and will be submitted tomorrow,
> as Jesper pointed out the issue happens only with 269d26f47f6f
> ("net/mlx5: Reduce command polling
> interval"), and only if CONFIG_PREEMPT is on.
> the only affected kernel is 4.19 which is not GA yet.
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=net-mlx5

Sound good.

I will appreciate if you add a:

Reported-by: Jesper Dangaard Brouer <brouer@redhat.com>

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Saeed Mahameed]

On Fri, Sep 14, 2018 at 1:52 AM, Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
> On Fri, 14 Sep 2018 01:22:15 -0700
> Saeed Mahameed <saeedm@dev.mellanox.co.il> wrote:
>
>> On Thu, Sep 13, 2018 at 11:36 PM, Jesper Dangaard Brouer
>> <brouer@redhat.com> wrote:
>> > On Thu, 13 Sep 2018 15:55:29 -0700
>> > Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>> >
>> >> On Thu, Aug 30, 2018 at 1:35 AM, Tariq Toukan <tariqt@mellanox.com> wrote:
>> >> >
>> >> >
>> >> > On 29/08/2018 6:05 PM, Jesper Dangaard Brouer wrote:
>> >> >>
>> >> >> Hi Saeed,
>> >> >>
>> >> >> I'm having issues loading mlx5 driver on v4.19 kernels (tested both
>> >> >> net-next and bpf-next), while kernel v4.18 seems to work.  It happens
>> >> >> with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
>> >> >> from the system now).
>> >> >>
>> >> >
>> >> > Hi Jesper,
>> >> >
>> >> > Thanks for your report!
>> >> >
>> >> > We are working to analyze and debug the issue.
>> >>
>> >> looks like serious issue to me... while no news in 2 weeks.
>> >> any update?
>> >
>> > Mellanox took it offlist, and Sep 6th found that this is a regression
>> > introduced by commit 269d26f47f6f ("net/mlx5: Reduce command polling
>> > interval"), but only if CONFIG_PREEMPT is on.
>> >
>> > I can confirm that reverting this commit fixed the issue (and not the
>> > firmware upgrade I also did).
>> >
>> > I think Moshe (Cc) is responsible for this case, and I expect to soon
>> > see a revert or alternative solution to this!?
>> >
>> > Thanks for the kick Alexei :-)
>>
>> Thanks you Alexei and Jesper for following up,
>> the fix is already being tested [1] and will be submitted tomorrow,
>> as Jesper pointed out the issue happens only with 269d26f47f6f
>> ("net/mlx5: Reduce command polling
>> interval"), and only if CONFIG_PREEMPT is on.
>> the only affected kernel is 4.19 which is not GA yet.
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=net-mlx5
>
> Sound good.
>
> I will appreciate if you add a:
>
> Reported-by: Jesper Dangaard Brouer <brouer@redhat.com>
>

Of course i will add it, simply the patch was in my review queue
before your report :).

> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Jesper Dangaard Brouer]

Hi Saeed,

I'm having issues loading mlx5 driver on v4.19 kernels (tested both
net-next and bpf-next), while kernel v4.18 seems to work.  It happens
with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
from the system now).

One pain point is very long boot-time, caused by some timeout code in
the driver. The kernel console log (dmesg) says:

[    5.763330] mlx5_core 0000:03:00.0: firmware version: 16.22.1002
[    5.769367] mlx5_core 0000:03:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:02.0 (capable of 252.048 Gb/s with 16 GT/s x16 link)

(...) other drivers loading

[   66.816635] mlx5_core 0000:03:00.0: wait_func:964:(pid 112): ENABLE_HCA(0x104) timeout. Will cause a leak of a command resource
[   66.828123] mlx5_core 0000:03:00.0: enable hca failed
[   66.845516] mlx5_core 0000:03:00.0: mlx5_load_one failed with error code -110
[   66.852802] mlx5_core: probe of 0000:03:00.0 failed with error -110

[   66.859347] mlx5_core 0000:03:00.1: firmware version: 16.22.1002
[   66.865388] mlx5_core 0000:03:00.1: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:02.0 (capable of 252.048 Gb/s with 16 GT/s x16 link)

[  125.787395] XFS (sda3): Mounting V5 Filesystem
[  125.848509] XFS (sda3): Ending clean mount
[  127.984784] mlx5_core 0000:03:00.1: wait_func:964:(pid 5): ENABLE_HCA(0x104) timeout. Will cause a leak of a command resource
[  127.996090] mlx5_core 0000:03:00.1: enable hca failed
[  128.013819] mlx5_core 0000:03:00.1: mlx5_load_one failed with error code -110
[  128.021076] mlx5_core: probe of 0000:03:00.1 failed with error -110


Do you have any idea what could be causing this?

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: mlx5 driver loading failing on v4.19 / net-next / bpf-next

[Tariq Toukan]

On 29/08/2018 6:05 PM, Jesper Dangaard Brouer wrote:
> Hi Saeed,
> 
> I'm having issues loading mlx5 driver on v4.19 kernels (tested both
> net-next and bpf-next), while kernel v4.18 seems to work.  It happens
> with a Mellanox ConnectX-5 NIC (and also a CX4-Lx but I removed that
> from the system now).
> 

Hi Jesper,

Thanks for your report!

We are working to analyze and debug the issue.

> One pain point is very long boot-time, caused by some timeout code in
> the driver. The kernel console log (dmesg) says:
> 
> [    5.763330] mlx5_core 0000:03:00.0: firmware version: 16.22.1002
> [    5.769367] mlx5_core 0000:03:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:02.0 (capable of 252.048 Gb/s with 16 GT/s x16 link)
> 
> (...) other drivers loading
> 
> [   66.816635] mlx5_core 0000:03:00.0: wait_func:964:(pid 112): ENABLE_HCA(0x104) timeout. Will cause a leak of a command resource
> [   66.828123] mlx5_core 0000:03:00.0: enable hca failed
> [   66.845516] mlx5_core 0000:03:00.0: mlx5_load_one failed with error code -110
> [   66.852802] mlx5_core: probe of 0000:03:00.0 failed with error -110
> 
> [   66.859347] mlx5_core 0000:03:00.1: firmware version: 16.22.1002
> [   66.865388] mlx5_core 0000:03:00.1: 126.016 Gb/s available PCIe bandwidth, limited by 8 GT/s x16 link at 0000:00:02.0 (capable of 252.048 Gb/s with 16 GT/s x16 link)
> 
> [  125.787395] XFS (sda3): Mounting V5 Filesystem
> [  125.848509] XFS (sda3): Ending clean mount
> [  127.984784] mlx5_core 0000:03:00.1: wait_func:964:(pid 5): ENABLE_HCA(0x104) timeout. Will cause a leak of a command resource
> [  127.996090] mlx5_core 0000:03:00.1: enable hca failed
> [  128.013819] mlx5_core 0000:03:00.1: mlx5_load_one failed with error code -110
> [  128.021076] mlx5_core: probe of 0000:03:00.1 failed with error -110
> 
> 
> Do you have any idea what could be causing this?
> 

We'll update regarding any progress.

Thanks!