Re: [BUG] mlx5_core general protection fault in mlx5_cmd_comp_handler

[Moshe Shemesh <moshe@nvidia.com>]

On 11/15/2022 5:08 PM, Jinpu Wang wrote:
> On Tue, Nov 15, 2022 at 6:46 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>> On Tue, Nov 15, 2022 at 6:15 AM Moshe Shemesh <moshe@nvidia.com> wrote:
>>>
>>> On 11/9/2022 11:51 AM, Jinpu Wang wrote:
>>>> On Mon, Oct 17, 2022 at 7:54 AM Jinpu Wang <jinpu.wang@ionos.com> wrote:
>>>>> On Thu, Oct 13, 2022 at 12:27 PM Leon Romanovsky <leon@kernel.org> wrote:
>>>>>> On Thu, Oct 13, 2022 at 10:32:55AM +0200, Jinpu Wang wrote:
>>>>>>> On Thu, Oct 13, 2022 at 10:18 AM Leon Romanovsky <leon@kernel.org> wrote:
>>>>>>>> On Wed, Oct 12, 2022 at 01:55:55PM +0200, Jinpu Wang wrote:
>>>>>>>>> Hi Leon, hi Saeed,
>>>>>>>>>
>>>>>>>>> We have seen crashes during server shutdown on both kernel 5.10 and
>>>>>>>>> kernel 5.15 with GPF in mlx5 mlx5_cmd_comp_handler function.
>>>>>>>>>
>>>>>>>>> All of the crashes point to
>>>>>>>>>
>>>>>>>>> 1606                         memcpy(ent->out->first.data,
>>>>>>>>> ent->lay->out, sizeof(ent->lay->out));
>>>>>>>>>
>>>>>>>>> I guess, it's kind of use after free for ent buffer. I tried to reprod
>>>>>>>>> by repeatedly reboot the testing servers, but no success  so far.
>>>>>>>> My guess is that command interface is not flushed, but Moshe and me
>>>>>>>> didn't see how it can happen.
>>>>>>>>
>>>>>>>>     1206         INIT_DELAYED_WORK(&ent->cb_timeout_work, cb_timeout_handler);
>>>>>>>>     1207         INIT_WORK(&ent->work, cmd_work_handler);
>>>>>>>>     1208         if (page_queue) {
>>>>>>>>     1209                 cmd_work_handler(&ent->work);
>>>>>>>>     1210         } else if (!queue_work(cmd->wq, &ent->work)) {
>>>>>>>>                             ^^^^^^^ this is what is causing to the splat
>>>>>>>>     1211                 mlx5_core_warn(dev, "failed to queue work\n");
>>>>>>>>     1212                 err = -EALREADY;
>>>>>>>>     1213                 goto out_free;
>>>>>>>>     1214         }
>>>>>>>>
>>>>>>>> <...>
>>>>>>>>> Is this problem known, maybe already fixed?
>>>>>>>> I don't see any missing Fixes that exist in 6.0 and don't exist in 5.5.32.
>>>>>> Sorry it is 5.15.32
>>>>>>
>>>>>>>> Is it possible to reproduce this on latest upstream code?
>>>>>>> I haven't been able to reproduce it, as mentioned above, I tried to
>>>>>>> reproduce by simply reboot in loop, no luck yet.
>>>>>>> do you have suggestions to speedup the reproduction?
>>>>>> Maybe try to shutdown during filling command interface.
>>>>>> I think that any query command will do the trick.
>>>>> Just an update.
>>>>> I tried to run "saquery" in a loop in one session and do "modproble -r
>>>>> mlx5_ib && modprobe mlx5_ib" in loop in another session during last
>>>>> days , but still no luck. --c
>>>>>>> Once I can reproduce, I can also try with kernel 6.0.
>>>>>> It will be great.
>>>>>>
>>>>>> Thanks
>>>>> Thanks!
>>>> Just want to mention, we see more crash during reboot, all the crash
>>>> we saw are all
>>>> Intel  Intel(R) Xeon(R) Gold 6338 CPU. We use the same HCA on
>>>> different servers. So I suspect the bug is related to Ice Lake server.
>>>>
>>>> In case it matters, here is lspci attached.
>>>
>>> Please try the following change on 5.15.32, let me know if it solves the
>>> failure :
>> Thank you Moshe, I will test it on affected servers and report back the result.
> Hi Moshe,
>
> I've been running the reboot tests on 4 affected machines in parallel
> for more than 6 hours,  in total did 300+ reboot, I can no longer
> reproduce the crash. without the fix, I was able to reproduce 2 times
> in 20 reboots.
> So I think the bug is fixed.


Great !

> I also did some basic functional test via RNBD/IPOIB, all look good.
> Tested-by: Jack Wang <jinpu.wang@ionos.com>
> Please provide a formal fix.


Will do.

Thanks!

>
> Thx!
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>>> b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>>> index e06a6104e91f..d45ca9c52a21 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
>>> @@ -971,6 +971,7 @@ static void cmd_work_handler(struct work_struct *work)
>>>                   cmd_ent_get(ent);
>>>           set_bit(MLX5_CMD_ENT_STATE_PENDING_COMP, &ent->state);
>>>
>>> +       cmd_ent_get(ent); /* for the _real_ FW event on completion */
>>>           /* Skip sending command to fw if internal error */
>>>           if (mlx5_cmd_is_down(dev) || !opcode_allowed(&dev->cmd, ent->op)) {
>>>                   u8 status = 0;
>>> @@ -984,7 +985,6 @@ static void cmd_work_handler(struct work_struct *work)
>>>                   return;
>>>           }
>>>
>>> -       cmd_ent_get(ent); /* for the _real_ FW event on completion */
>>>           /* ring doorbell after the descriptor is valid */
>>>           mlx5_core_dbg(dev, "writing 0x%x to command doorbell\n", 1 <<
>>> ent->idx);
>>>           wmb();
>>> @@ -1598,8 +1598,8 @@ static void mlx5_cmd_comp_handler(struct
>>> mlx5_core_dev *dev, u64 vec, bool force
>>>                                   cmd_ent_put(ent); /* timeout work was
>>> canceled */
>>>
>>>                           if (!forced || /* Real FW completion */
>>> -                           pci_channel_offline(dev->pdev) || /* FW is
>>> inaccessible */
>>> -                           dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR)
>>> +                            mlx5_cmd_is_down(dev) || /* No real FW
>>> completion is expected */
>>> +                            !opcode_allowed(cmd, ent->op))
>>>                                   cmd_ent_put(ent);
>>>
>>>                           ent->ts2 = ktime_get_ns();
>>>
>>>> Thx!