Re: [PATCH v9 00/13] Make PCI's devres API more consistent

Philipp Stanner
2024-06-14T11:38:49Z

On Thu, 2024-06-13 at 16:57 -0500, Bjorn Helgaas wrote:
> On Thu, Jun 13, 2024 at 01:50:13PM +0200, Philipp Stanner wrote:
> > Changes in v9:
> >   - Remove forgotten dead code ('enabled' bit in struct pci_dev) in
> >     patch No.8 ("Move pinned status bit...")
> >   - Rework patch No.3:
> >       - Change title from "Reimplement plural devres functions"
> >         to "Add partial-BAR devres support".
> >       - Drop excessive details about the general cleanup from the
> > commit
> >         message. Only motivate why this patch's new infrastructure
> > is
> >         necessary.
> >   - Fix some minor spelling issues (s/pci/PCI ...)
> >
> > Changes in v8:
> >   - Rebase the series on the already merged patches which were
> > slightly
> >     modified by Bjorn Helgaas.
> >   - Reword the pci_intx() commit message so it clearly states it's
> > about
> >     reworking pci_intx().
> >   - Move the removal of find_pci_dr() from patch "Remove legacy
> >     pcim_release()" to patch "Give pci_intx() its own devres
> > callback"
> >     since this later patch already removed all calls to that
> > function.
> >   - In patch "Give pci_intx() its own devres callback": use
> >     pci_is_enabled() (and, thus, the enabled_cnt in struct pci_dev)
> >     instead of a separate enabled field. (Bjorn)
> >
> > Changes in v7:
> >   - Split the entire series in smaller, more atomic chunks /
> > patches
> >     (Bjorn)
> >   - Remove functions (such as pcim_iomap_region_range()) that do
> > not yet
> >     have a user (Bjorn)
> >   - Don't export interfaces publicly anymore, except for
> >     pcim_iomap_range(), needed by vboxvideo (Bjorn)
> >   - Mention the actual (vboxvideo) bug in "PCI: Warn users..."
> > commit
> >     (Bjorn)
> >   - Drop docstring warnings on PCI-internal functions (Bjorn)
> >   - Rework docstring warnings
> >   - Fix spelling in a few places. Rewrapp paragraphs (Bjorn)
> >
> > Changes in v6:
> >   - Restructure the cleanup in pcim_iomap_regions_request_all() so
> > that
> >     it doesn't trigger a (false positive) test robot warning. No
> >     behavior change intended. (Dan Carpenter)
> >
> > Changes in v5:
> >   - Add Hans's Reviewed-by to vboxvideo patch (Hans de Goede)
> >   - Remove stable-kernel from CC in vboxvideo patch (Hans de Goede)
> >
> > Changes in v4:
> >   - Rebase against linux-next
> >
> > Changes in v3:
> >   - Use the term "PCI devres API" at some forgotten places.
> >   - Fix more grammar errors in patch #3.
> >   - Remove the comment advising to call (the outdated) pcim_intx()
> > in pci.c
> >   - Rename __pcim_request_region_range() flags-field "exclusive" to
> >     "req_flags", since this is what the int actually represents.
> >   - Remove the call to pcim_region_request() from patch #10. (Hans)
> >
> > Changes in v2:
> >   - Make commit head lines congruent with PCI's style (Bjorn)
> >   - Add missing error checks for devm_add_action(). (Andy)
> >   - Repair the "Returns: " marks for docu generation (Andy)
> >   - Initialize the addr_devres struct with memset(). (Andy)
> >   - Make pcim_intx() a PCI-internal function so that new drivers
> > won't
> >     be encouraged to use the outdated pci_intx() mechanism.
> >     (Andy / Philipp)
> >   - Fix grammar and spelling (Bjorn)
> >   - Be more precise on why pcim_iomap_table() is problematic
> > (Bjorn)
> >   - Provide the actual structs' and functions' names in the commit
> >     messages (Bjorn)
> >   - Remove redundant variable initializers (Andy)
> >   - Regroup PM bitfield members in struct pci_dev (Andy)
> >   - Make pcim_intx() visible only for the PCI subsystem so that
> > new
> >     drivers won't use this outdated API (Andy, Myself)
> >   - Add a NOTE to pcim_iomap() to warn about this function being
> > the one
> >     exception that does just return NULL.
> >   - Consistently use the term "PCI devres API"; also in Patch #10
> > (Bjorn)
> >
> >
> > ¡Hola!
> >
> > PCI's devres API suffers several weaknesses:
> >
> > 1. There are functions prefixed with pcim_. Those are always
> > managed
> >    counterparts to never-managed functions prefixed with pci_ – or
> > so one
> >    would like to think. There are some apparently unmanaged
> > functions
> >    (all region-request / release functions, and pci_intx()) which
> >    suddenly become managed once the user has initialized the device
> > with
> >    pcim_enable_device() instead of pci_enable_device(). This
> > "sometimes
> >    yes, sometimes no" nature of those functions is confusing and
> >    therefore bug-provoking. In fact, it has already caused a bug in
> > DRM.
> >    The last patch in this series fixes that bug.
> > 2. iomappings: Instead of giving each mapping its own callback, the
> >    existing API uses a statically allocated struct tracking one
> > mapping
> >    per bar. This is not extensible. Especially, you can't create
> >    _ranged_ managed mappings that way, which many drivers want.
> > 3. Managed request functions only exist as "plural versions" with a
> >    bit-mask as a parameter. That's quite over-engineered
> > considering
> >    that each user only ever mapps one, maybe two bars.
> >
> > This series:
> > - add a set of new "singular" devres functions that use devres the
> > way
> >   its intended, with one callback per resource.
> > - deprecates the existing iomap-table mechanism.
> > - deprecates the hybrid nature of pci_ functions.
> > - preserves backwards compatibility so that drivers using the
> > existing
> >   API won't notice any changes.
> > - adds documentation, especially some warning users about the
> >   complicated nature of PCI's devres.
> >
> >
> > Note that this series is based on my "unify pci_iounmap"-series
> > from a
> > few weeks ago. [1]
> >
> > I tested this on a x86 VM with a simple pci test-device with two
> > regions. Operates and reserves resources as intended on my system.
> > Kasan and kmemleak didn't find any problems.
> >
> > I believe this series cleans the API up as much as possible without
> > having to port all existing drivers to the new API. Especially, I
> > think
> > that this implementation is easy to extend if the need for new
> > managed
> > functions arises :)
> >
> > Greetings,
> > P.
> >
> > Philipp Stanner (13):
> >   PCI: Add and use devres helper for bit masks
> >   PCI: Add devres helpers for iomap table
> >   PCI: Add partial-BAR devres support
> >   PCI: Deprecate two surplus devres functions
> >   PCI: Make devres region requests consistent
> >   PCI: Warn users about complicated devres nature
> >   PCI: Remove enabled status bit from pci_devres
> >   PCI: Move pinned status bit to struct pci_dev
> >   PCI: Give pcim_set_mwi() its own devres callback
> >   PCI: Give pci_intx() its own devres callback
> >   PCI: Remove legacy pcim_release()
> >   PCI: Add pcim_iomap_range()
> >   drm/vboxvideo: fix mapping leaks
> >
> >  drivers/gpu/drm/vboxvideo/vbox_main.c |  20 +-
> >  drivers/pci/devres.c                  | 903 +++++++++++++++++++++-
> > ----
> >  drivers/pci/iomap.c                   |  16 +
> >  drivers/pci/pci.c                     |  94 ++-
> >  drivers/pci/pci.h                     |  23 +-
> >  include/linux/pci.h                   |   5 +-
> >  6 files changed, 858 insertions(+), 203 deletions(-)
>
> This is on pci/devres with some commit log rework and the following
> diffs.  I think the bar short/int thing is the only actual code
> change.  Happy to squash in any other updates or things I botched.

I looked through your tree and only found the following nit:

In commit "PCI: Remove struct pci_devres.enabled status bit" you
changed the line

"The PCI devres implementation has a separate boolean to track whether
a"

to:

"The pci_devres struct has a separate boolean to track whether a device
is"

In past reviews that has been criticized and I was told to always call
it "struct pci_devres", not the other way around. That's also how it's
put in the following paragraph.

>
> Planned for v6.11.

\o/

P.

>
> diff --git a/drivers/pci/devres.c b/drivers/pci/devres.c
> index 2f0379a4e58f..d9b78a0d903a 100644
> --- a/drivers/pci/devres.c
> +++ b/drivers/pci/devres.c
> @@ -11,7 +11,7 @@
>   * 1. It is very strongly tied to the statically allocated mapping
> table in
>   *    struct pcim_iomap_devres below. This is mostly solved in the
> sense of the
>   *    pcim_ functions in this file providing things like ranged
> mapping by
> - *    bypassing this table, wheras the functions that were present
> in the old
> + *    bypassing this table, whereas the functions that were present
> in the old
>   *    API still enter the mapping addresses into the table for users
> of the old
>   *    API.
>   *
> @@ -25,10 +25,11 @@
>   *    Consequently, in the new API, region requests performed by the
> pcim_
>   *    functions are automatically cleaned up through the devres
> callback
>   *    pcim_addr_resource_release().
> - *    Users utilizing pcim_enable_device() + pci_*region*() are
> redirected in
> + *
> + *    Users of pcim_enable_device() + pci_*region*() are redirected
> in
>   *    pci.c to the managed functions here in this file. This isn't
> exactly
> - *    perfect, but the only alternative way would be to port ALL
> drivers using
> - *    said combination to pcim_ functions.
> + *    perfect, but the only alternative way would be to port ALL
> drivers
> + *    using said combination to pcim_ functions.
>   *
>   * TODO:
>   * Remove the legacy table entirely once all calls to
> pcim_iomap_table() in
> @@ -42,7 +43,7 @@ struct pcim_iomap_devres {
>         void __iomem *table[PCI_STD_NUM_BARS];
>  };
>
> -/* Used to restore the old intx state on driver detach. */
> +/* Used to restore the old INTx state on driver detach. */
>  struct pcim_intx_devres {
>         int orig_intx;
>  };
> @@ -77,7 +78,7 @@ struct pcim_addr_devres {
>         void __iomem *baseaddr;
>         unsigned long offset;
>         unsigned long len;
> -       short bar;
> +       int bar;
>  };
>
>  static inline void pcim_addr_devres_clear(struct pcim_addr_devres
> *res)
> @@ -108,8 +109,9 @@ static inline void pcim_addr_devres_clear(struct
> pcim_addr_devres *res)
>   * Request a range within a device's PCI BAR.  Sanity check the
> input.
>   */
>  static int __pcim_request_region_range(struct pci_dev *pdev, int
> bar,
> -               unsigned long offset, unsigned long maxlen,
> -               const char *name, int req_flags)
> +                                      unsigned long offset,
> +                                      unsigned long maxlen,
> +                                      const char *name, int
> req_flags)
>  {
>         resource_size_t start = pci_resource_start(pdev, bar);
>         resource_size_t len = pci_resource_len(pdev, bar);
> @@ -118,7 +120,7 @@ static int __pcim_request_region_range(struct
> pci_dev *pdev, int bar,
>         if (start == 0 || len == 0) /* Unused BAR. */
>                 return 0;
>         if (len <= offset)
> -               return  -EINVAL;
> +               return -EINVAL;
>
>         start += offset;
>         len -= offset;
> @@ -141,7 +143,8 @@ static int __pcim_request_region_range(struct
> pci_dev *pdev, int bar,
>  }
>
>  static void __pcim_release_region_range(struct pci_dev *pdev, int
> bar,
> -               unsigned long offset, unsigned long maxlen)
> +                                       unsigned long offset,
> +                                       unsigned long maxlen)
>  {
>         resource_size_t start = pci_resource_start(pdev, bar);
>         resource_size_t len = pci_resource_len(pdev, bar);
> @@ -166,7 +169,7 @@ static void __pcim_release_region_range(struct
> pci_dev *pdev, int bar,
>  }
>
>  static int __pcim_request_region(struct pci_dev *pdev, int bar,
> -               const char *name, int flags)
> +                                const char *name, int flags)
>  {
>         unsigned long offset = 0;
>         unsigned long len = pci_resource_len(pdev, bar);
> @@ -208,7 +211,7 @@ static struct pcim_addr_devres
> *pcim_addr_devres_alloc(struct pci_dev *pdev)
>         struct pcim_addr_devres *res;
>
>         res = devres_alloc_node(pcim_addr_resource_release,
> sizeof(*res),
> -                       GFP_KERNEL, dev_to_node(&pdev->dev));
> +                               GFP_KERNEL, dev_to_node(&pdev->dev));
>         if (res)
>                 pcim_addr_devres_clear(res);
>         return res;
> @@ -223,7 +226,8 @@ static inline void pcim_addr_devres_free(struct
> pcim_addr_devres *res)
>  /*
>   * Used by devres to identify a pcim_addr_devres.
>   */
> -static int pcim_addr_resources_match(struct device *dev, void
> *a_raw, void *b_raw)
> +static int pcim_addr_resources_match(struct device *dev,
> +                                    void *a_raw, void *b_raw)
>  {
>         struct pcim_addr_devres *a, *b;
>
> @@ -402,7 +406,6 @@ int pcim_set_mwi(struct pci_dev *pdev)
>  }
>  EXPORT_SYMBOL(pcim_set_mwi);
>
> -
>  static inline bool mask_contains_bar(int mask, int bar)
>  {
>         return mask & BIT(bar);
> @@ -438,8 +441,8 @@ static struct pcim_intx_devres
> *get_or_create_intx_devres(struct device *dev)
>   *
>   * Returns: 0 on success, -ENOMEM on error.
>   *
> - * Enables/disables PCI INTx for device @pdev.
> - * Restores the original state on driver detach.
> + * Enable/disable PCI INTx for device @pdev.
> + * Restore the original state on driver detach.
>   */
>  int pcim_intx(struct pci_dev *pdev, int enable)
>  {
> @@ -492,7 +495,7 @@ int pcim_enable_device(struct pci_dev *pdev)
>
>         /*
>          * We prefer removing the action in case of an error over
> -        * devm_add_action_or_reset() because the later could
> theoretically be
> +        * devm_add_action_or_reset() because the latter could
> theoretically be
>          * disturbed by users having pinned the device too soon.
>          */
>         ret = pci_enable_device(pdev);
> @@ -618,7 +621,7 @@ static void
> pcim_remove_mapping_from_legacy_table(struct pci_dev *pdev,
>   * The same as pcim_remove_mapping_from_legacy_table(), but
> identifies the
>   * mapping by its BAR index.
>   */
> -static void pcim_remove_bar_from_legacy_table(struct pci_dev *pdev,
> short bar)
> +static void pcim_remove_bar_from_legacy_table(struct pci_dev *pdev,
> int bar)
>  {
>         void __iomem **legacy_iomap_table;
>
> @@ -783,7 +786,7 @@ static void pcim_iounmap_region(struct pci_dev
> *pdev, int bar)
>  int pcim_iomap_regions(struct pci_dev *pdev, int mask, const char
> *name)
>  {
>         int ret;
> -       short bar;
> +       int bar;
>         void __iomem *mapping;
>
>         for (bar = 0; bar < DEVICE_COUNT_RESOURCE; bar++) {
> @@ -813,7 +816,7 @@ int pcim_iomap_regions(struct pci_dev *pdev, int
> mask, const char *name)
>  EXPORT_SYMBOL(pcim_iomap_regions);
>
>  static int _pcim_request_region(struct pci_dev *pdev, int bar, const
> char *name,
> -               int request_flags)
> +                               int request_flags)
>  {
>         int ret;
>         struct pcim_addr_devres *res;
> @@ -903,7 +906,7 @@ void pcim_release_region(struct pci_dev *pdev,
> int bar)
>   */
>  static void pcim_release_all_regions(struct pci_dev *pdev)
>  {
> -       short bar;
> +       int bar;
>
>         for (bar = 0; bar < PCI_STD_NUM_BARS; bar++)
>                 pcim_release_region(pdev, bar);
> @@ -923,7 +926,7 @@ static void pcim_release_all_regions(struct
> pci_dev *pdev)
>  static int pcim_request_all_regions(struct pci_dev *pdev, const char
> *name)
>  {
>         int ret;
> -       short bar;
> +       int bar;
>
>         for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
>                 ret = pcim_request_region(pdev, bar, name);
> @@ -960,7 +963,7 @@ static int pcim_request_all_regions(struct
> pci_dev *pdev, const char *name)
>  int pcim_iomap_regions_request_all(struct pci_dev *pdev, int mask,
>                                    const char *name)
>  {
> -       short bar;
> +       int bar;
>         int ret;
>         void __iomem **legacy_iomap_table;
>
> @@ -1004,14 +1007,14 @@
> EXPORT_SYMBOL(pcim_iomap_regions_request_all);
>   */
>  void pcim_iounmap_regions(struct pci_dev *pdev, int mask)
>  {
> -       short bar;
> +       int i;
>
> -       for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
> -               if (!mask_contains_bar(mask, bar))
> +       for (i = 0; i < PCI_STD_NUM_BARS; i++) {
> +               if (!mask_contains_bar(mask, i))
>                         continue;
>
> -               pcim_iounmap_region(pdev, bar);
> -               pcim_remove_bar_from_legacy_table(pdev, bar);
> +               pcim_iounmap_region(pdev, i);
> +               pcim_remove_bar_from_legacy_table(pdev, i);
>         }
>  }
>  EXPORT_SYMBOL(pcim_iounmap_regions);
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 1b4832a60047..807f8be043cd 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -4073,6 +4073,11 @@ EXPORT_SYMBOL(pci_release_regions);
>   *
>   * Returns 0 on success, or %EBUSY on error.  A warning
>   * message is also printed on failure.
> + *
> + * NOTE:
> + * This is a "hybrid" function: It's normally unmanaged, but becomes
> managed
> + * when pcim_enable_device() has been called in advance. This hybrid
> feature is
> + * DEPRECATED! If you want managed cleanup, use the pcim_* functions
> instead.
>   */
>  int pci_request_regions(struct pci_dev *pdev, const char *res_name)
>  {
> @@ -4437,17 +4442,13 @@ void pci_disable_parity(struct pci_dev *dev)
>   * NOTE:
>   * This is a "hybrid" function: It's normally unmanaged, but becomes
> managed
>   * when pcim_enable_device() has been called in advance. This hybrid
> feature is
> - * DEPRECATED!
> + * DEPRECATED! If you want managed cleanup, use pcim_intx() instead.
>   */
>  void pci_intx(struct pci_dev *pdev, int enable)
>  {
>         u16 pci_command, new;
>
> -       /*
> -        * This is done for backwards compatibility, because the old
> PCI devres
> -        * API had a mode in which this function became managed if
> the dev had
> -        * been enabled with pcim_enable_device() instead of
> pci_enable_device().
> -        */
> +       /* Preserve the "hybrid" behavior for backwards compatibility
> */
>         if (pci_is_managed(pdev)) {
>                 WARN_ON_ONCE(pcim_intx(pdev, enable) != 0);
>                 return;
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index e51e6fa79fcc..e6d299b93c21 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -813,7 +813,8 @@ static inline pci_power_t
> mid_pci_get_power_state(struct pci_dev *pdev)
>  int pcim_intx(struct pci_dev *dev, int enable);
>
>  int pcim_request_region(struct pci_dev *pdev, int bar, const char
> *name);
> -int pcim_request_region_exclusive(struct pci_dev *pdev, int bar,
> const char *name);
> +int pcim_request_region_exclusive(struct pci_dev *pdev, int bar,
> +                                 const char *name);
>  void pcim_release_region(struct pci_dev *pdev, int bar);
>
>  /*
>