From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96B1AC433E0 for ; Wed, 12 Aug 2020 09:34:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6589120885 for ; Wed, 12 Aug 2020 09:34:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="hBYZrFO4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727884AbgHLJep (ORCPT ); Wed, 12 Aug 2020 05:34:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726629AbgHLJeo (ORCPT ); Wed, 12 Aug 2020 05:34:44 -0400 Received: from mail-lj1-x242.google.com (mail-lj1-x242.google.com [IPv6:2a00:1450:4864:20::242]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44B46C06174A for ; Wed, 12 Aug 2020 02:34:44 -0700 (PDT) Received: by mail-lj1-x242.google.com with SMTP id w14so1480293ljj.4 for ; Wed, 12 Aug 2020 02:34:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=kn22iNgLDoT2FylS2rp2aS0ljPEs4sl8pUcenyynzLc=; b=hBYZrFO4My7bEJ67J/RTZHMceJ4pDA1vsETQ4pVBLoGZ02rc792kDvN16F2inJzPSE aCn0piArp/T4z/EfRwNx9cb2SnUMuabP8T2wFEnZURgo0/sBc6SaDG3vx1TZ7yth/C2r cPHe1g6y98imhFhfoSPA8dT5U+yQHARjwm0FQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kn22iNgLDoT2FylS2rp2aS0ljPEs4sl8pUcenyynzLc=; b=G+f+tTD60Va0VkYAl78LGLcDQDqQIwJxoB89q41kj/sgM4Kf8lNhlZNVZtjfnmNc8v QJ5KyJXcKZYUohGC7Kpm74PbdMwcswBexEUpc0jCsdoJoaWT0Va1KfacSd71BdfsaIw3 bxn5XSba3K8+0WFriSuLG8EArWZfUDNStDG6FvlbTI+uyhwmv0Qv9qhlJVq4jeq1vVPi +s1x9qTBANZTy7SdVw9HUxEpw6Gwin1JCxqv2fvDgrO0Jqbl1sFSbudyWpWFEdATiVcI mqya1GWOK3HBu/WlRIcaUvPq8+Q+HY3QEMroC0xkV4MzW400546gvrWl73oEaRe5JTKS nYOA== X-Gm-Message-State: AOAM532MUwe69Fkn6c8QF8TtPsoa+wRcjSmTxsyZofPQKUsapkswTsLS WYF5p2Tyo4Q8+97WDi7mk2G66W7QPj4US8WSNJaAzQ== X-Google-Smtp-Source: ABdhPJwU0yxulG1HBQCvdYfMzzVBm9t1Urj5zRA+7b8hNJ/06CUxaad9o0JJ2SxSER75I3iHVdcoq7vIXTieMeZTjXI= X-Received: by 2002:a2e:9010:: with SMTP id h16mr5056506ljg.316.1597224882556; Wed, 12 Aug 2020 02:34:42 -0700 (PDT) MIME-Version: 1.0 References: <1595847753-2234-1-git-send-email-moshe@mellanox.com> <7a9c315f-fa29-7bd5-31be-3748b8841b29@mellanox.com> <7fd63d16-f9fa-9d55-0b30-fe190d0fb1cb@mellanox.com> In-Reply-To: From: Vasundhara Volam Date: Wed, 12 Aug 2020 15:04:31 +0530 Message-ID: Subject: Re: [PATCH net-next RFC 00/13] Add devlink reload level option To: Moshe Shemesh Cc: Jacob Keller , "David S. Miller" , Jiri Pirko , Netdev , open list Content-Type: text/plain; charset="UTF-8" Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Wed, Aug 5, 2020 at 1:51 PM Moshe Shemesh wrote: > > > On 8/5/2020 9:55 AM, Vasundhara Volam wrote: > > On Wed, Aug 5, 2020 at 12:02 PM Moshe Shemesh wrote: > >> > >> On 8/4/2020 1:13 PM, Vasundhara Volam wrote: > >>> On Mon, Aug 3, 2020 at 7:23 PM Moshe Shemesh wrote: > >>>> On 8/3/2020 3:47 PM, Vasundhara Volam wrote: > >>>>> On Mon, Aug 3, 2020 at 5:47 PM Moshe Shemesh wrote: > >>>>>> On 8/3/2020 1:24 PM, Vasundhara Volam wrote: > >>>>>>> On Tue, Jul 28, 2020 at 10:13 PM Jacob Keller wrote: > >>>>>>>> On 7/27/2020 10:25 PM, Vasundhara Volam wrote: > >>>>>>>>> On Mon, Jul 27, 2020 at 4:36 PM Moshe Shemesh wrote: > >>>>>>>>>> Introduce new option on devlink reload API to enable the user to select the > >>>>>>>>>> reload level required. Complete support for all levels in mlx5. > >>>>>>>>>> The following reload levels are supported: > >>>>>>>>>> driver: Driver entities re-instantiation only. > >>>>>>>>>> fw_reset: Firmware reset and driver entities re-instantiation. > >>>>>>>>> The Name is a little confusing. I think it should be renamed to > >>>>>>>>> fw_live_reset (in which both firmware and driver entities are > >>>>>>>>> re-instantiated). For only fw_reset, the driver should not undergo > >>>>>>>>> reset (it requires a driver reload for firmware to undergo reset). > >>>>>>>>> > >>>>>>>> So, I think the differentiation here is that "live_patch" doesn't reset > >>>>>>>> anything. > >>>>>>> This seems similar to flashing the firmware and does not reset anything. > >>>>>> The live patch is activating fw change without reset. > >>>>>> > >>>>>> It is not suitable for any fw change but fw gaps which don't require reset. > >>>>>> > >>>>>> I can query the fw to check if the pending image change is suitable or > >>>>>> require fw reset. > >>>>> Okay. > >>>>>>>>>> fw_live_patch: Firmware live patching only. > >>>>>>>>> This level is not clear. Is this similar to flashing?? > >>>>>>>>> > >>>>>>>>> Also I have a basic query. The reload command is split into > >>>>>>>>> reload_up/reload_down handlers (Please correct me if this behaviour is > >>>>>>>>> changed with this patchset). What if the vendor specific driver does > >>>>>>>>> not support up/down and needs only a single handler to fire a firmware > >>>>>>>>> reset or firmware live reset command? > >>>>>>>> In the "reload_down" handler, they would trigger the appropriate reset, > >>>>>>>> and quiesce anything that needs to be done. Then on reload up, it would > >>>>>>>> restore and bring up anything quiesced in the first stage. > >>>>>>> Yes, I got the "reload_down" and "reload_up". Similar to the device > >>>>>>> "remove" and "re-probe" respectively. > >>>>>>> > >>>>>>> But our requirement is a similar "ethtool reset" command, where > >>>>>>> ethtool calls a single callback in driver and driver just sends a > >>>>>>> firmware command for doing the reset. Once firmware receives the > >>>>>>> command, it will initiate the reset of driver and firmware entities > >>>>>>> asynchronously. > >>>>>> It is similar to mlx5 case here for fw_reset. The driver triggers the fw > >>>>>> command to reset and all PFs drivers gets events to handle and do > >>>>>> re-initialization. To fit it to the devlink reload_down and reload_up, > >>>>>> I wait for the event handler to complete and it stops at driver unload > >>>>>> to have the driver up by devlink reload_up. See patch 8 in this patchset. > >>>>>> > >>>>> Yes, I see reload_down is triggering the reset. In our driver, after > >>>>> triggering the reset through a firmware command, reset is done in > >>>>> another context as the driver initiates the reset only after receiving > >>>>> an ASYNC event from the firmware. > >>>> Same here. > >>>> > >>>>> Probably, we have to use reload_down() to send firmware command to > >>>>> trigger reset and do nothing in reload_up. > >>>> I had that in previous version, but its wrong to use devlink reload this > >>>> way, so I added wait with timeout for the event handling to complete > >>>> before unload_down function ends. See mlx5_fw_wait_fw_reset_done(). Also > >>>> the event handler stops before load back to have that done by devlink > >>>> reload_up. > >>> But "devlink dev reload" will be invoked by the user only on a single > >>> dev handler and all function drivers will be re-instantiated upon the > >>> ASYNC event. reload_down and reload_up are invoked only the function > >>> which the user invoked. > >>> > >>> Take an example of a 2-port (PF0 and PF1) adapter on a single host and > >>> with some VFs loaded on the device. User invokes "devlink dev reload" > >>> on PF0, ASYNC event is received on 2 PFs and VFs for reset. All the > >>> function drivers will be re-instantiated including PF0. > >>> > >>> If we wait for some time in reload_down() of PF0 and then call load in > >>> reload_up(), this code will be different from other function drivers. > >> > >> I see your point here, but the user run devlink reload command on one > >> PF, in this case of fw-reset it will influence other PFs, but that's a > >> result of the fw-reset, the user if asked for params change or namespace > >> change that was for this PF. > > Right, if any driver is implementing only fw-reset have to leave > > reload_up as an empty function. > > > No, its not only up the driver. The netns option is implemented by > devlink and its running between reload_down and reload_up. What I mean is, driver will provide a reload_up handler but it will not do anything and simply return 0. > > >>>>> And returning from reload > >>>>> does not mean that reset is complete as it is done in another context > >>>>> and the driver notifies the health reporter once the reset is > >>>>> complete. devlink framework may have to allow drivers to implement > >>>>> reload_down only to look more clean or call reload_up only if the > >>>>> driver notifies the devlink once reset is completed from another > >>>>> context. Please suggest.