* User SIW fails matching device
@ 2019-07-12 14:27 Potnuri Bharat Teja
2019-07-12 14:35 ` Jason Gunthorpe
0 siblings, 1 reply; 5+ messages in thread
From: Potnuri Bharat Teja @ 2019-07-12 14:27 UTC (permalink / raw)
To: jgg, linux-rdma; +Cc: BMT, monis, nirranjan, bharat
Hi all,
I observe the following behavior on one of my machines configured for siw.
Issue:
SIW device gets wrong device ops (HW/real rdma driver device ops) instead of
siw device ops due to improper device matching.
Root-cause:
In libibverbs, during user cma initialisation, for each entry from the driver
list, sysfs device is checked for matching name or device.
If the siw/rxe driver is at the head of the list, then sysfs device matches
properly with the corresponding siw driver and gets the corresponding siw/rxe
device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5
respectively in the driver list, then siw sysfs device matches pci device and
wrongly gets the device ops of HW driver (cxgb4/mlx5).
Below debug prints from verbs_register_driver() and driver_list entries, where
siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context
instead of siw_alloc_context, thus breaking user siw.
<debug> verbs_register_driver_22: 184: driver 0x176e370
<debug> verbs_register_driver_22: 185: name ipathverbs
<debug> verbs_register_driver_22: 184: driver 0x176f6a0
<debug> verbs_register_driver_22: 185: name cxgb4
<debug> verbs_register_driver_22: 184: driver 0x176fd50
<debug> verbs_register_driver_22: 185: name cxgb3
<debug> verbs_register_driver_22: 184: driver 0x1777020
<debug> verbs_register_driver_22: 185: name rxe
<debug> verbs_register_driver_22: 184: driver 0x1770a30
<debug> verbs_register_driver_22: 185: name siw
<debug> verbs_register_driver_22: 184: driver 0x1771120
<debug> verbs_register_driver_22: 185: name mlx4
<debug> verbs_register_driver_22: 184: driver 0x1771990
<debug> verbs_register_driver_22: 185: name mlx5
<debug> verbs_register_driver_22: 184: driver 0x1771ff0
<debug> verbs_register_driver_22: 185: name efa
<debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs
<debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4
<debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3
<debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe
<debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw
<debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4
<debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5
<debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
Proposed fix:
I have the below fix that works. It adds siw/rxe driver to the HEAD of the
driver list and the rest to the tail. I am not sure if this fix is the ideal
one, so I am attaching it to this mail.
diff --git a/libibverbs/init.c b/libibverbs/init.c
index 930d91811ca9..e44f0d743063 100644
--- a/libibverbs/init.c
+++ b/libibverbs/init.c
@@ -182,7 +182,10 @@ void verbs_register_driver(const struct verbs_device_ops *ops)
driver->ops = ops;
- list_add_tail(&driver_list, &driver->entry);
+ if (!strcmp(ops->name, "siw") || !strcmp(ops->name, "rxe"))
+ list_add(&driver_list, &driver->entry);
+ else
+ list_add_tail(&driver_list, &driver->entry);
}
---
Thanks,
Bharat
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: User SIW fails matching device
2019-07-12 14:27 User SIW fails matching device Potnuri Bharat Teja
@ 2019-07-12 14:35 ` Jason Gunthorpe
2019-07-12 15:24 ` Potnuri Bharat Teja
0 siblings, 1 reply; 5+ messages in thread
From: Jason Gunthorpe @ 2019-07-12 14:35 UTC (permalink / raw)
To: Potnuri Bharat Teja; +Cc: linux-rdma, BMT, monis, nirranjan
On Fri, Jul 12, 2019 at 07:57:19PM +0530, Potnuri Bharat Teja wrote:
> Hi all,
> I observe the following behavior on one of my machines configured for siw.
>
> Issue:
> SIW device gets wrong device ops (HW/real rdma driver device ops) instead of
> siw device ops due to improper device matching.
>
> Root-cause:
> In libibverbs, during user cma initialisation, for each entry from the driver
> list, sysfs device is checked for matching name or device.
> If the siw/rxe driver is at the head of the list, then sysfs device matches
> properly with the corresponding siw driver and gets the corresponding siw/rxe
> device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5
> respectively in the driver list, then siw sysfs device matches pci device and
> wrongly gets the device ops of HW driver (cxgb4/mlx5).
>
> Below debug prints from verbs_register_driver() and driver_list entries, where
> siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context
> instead of siw_alloc_context, thus breaking user siw.
>
> <debug> verbs_register_driver_22: 184: driver 0x176e370
> <debug> verbs_register_driver_22: 185: name ipathverbs
> <debug> verbs_register_driver_22: 184: driver 0x176f6a0
> <debug> verbs_register_driver_22: 185: name cxgb4
> <debug> verbs_register_driver_22: 184: driver 0x176fd50
> <debug> verbs_register_driver_22: 185: name cxgb3
> <debug> verbs_register_driver_22: 184: driver 0x1777020
> <debug> verbs_register_driver_22: 185: name rxe
> <debug> verbs_register_driver_22: 184: driver 0x1770a30
> <debug> verbs_register_driver_22: 185: name siw
> <debug> verbs_register_driver_22: 184: driver 0x1771120
> <debug> verbs_register_driver_22: 185: name mlx4
> <debug> verbs_register_driver_22: 184: driver 0x1771990
> <debug> verbs_register_driver_22: 185: name mlx5
> <debug> verbs_register_driver_22: 184: driver 0x1771ff0
> <debug> verbs_register_driver_22: 185: name efa
>
> <debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs
> <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4
> <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3
> <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe
> <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw
> <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4
> <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5
> <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
>
> Proposed fix:
> I have the below fix that works. It adds siw/rxe driver to the HEAD of the
> driver list and the rest to the tail. I am not sure if this fix is the ideal
> one, so I am attaching it to this mail.
Update your rdma-core to latest and this will be fixed fully by using
netlink to match the siw device..
Jason
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: User SIW fails matching device
2019-07-12 14:35 ` Jason Gunthorpe
@ 2019-07-12 15:24 ` Potnuri Bharat Teja
2019-07-12 15:30 ` Jason Gunthorpe
0 siblings, 1 reply; 5+ messages in thread
From: Potnuri Bharat Teja @ 2019-07-12 15:24 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma@vger.kernel.org, BMT@zurich.ibm.com,
monis@mellanox.com, Nirranjan Kirubaharan
On Friday, July 07/12/19, 2019 at 20:05:46 +0530, Jason Gunthorpe wrote:
> On Fri, Jul 12, 2019 at 07:57:19PM +0530, Potnuri Bharat Teja wrote:
> > Hi all,
> > I observe the following behavior on one of my machines configured for siw.
> >
> > Issue:
> > SIW device gets wrong device ops (HW/real rdma driver device ops) instead of
> > siw device ops due to improper device matching.
> >
> > Root-cause:
> > In libibverbs, during user cma initialisation, for each entry from the driver
> > list, sysfs device is checked for matching name or device.
> > If the siw/rxe driver is at the head of the list, then sysfs device matches
> > properly with the corresponding siw driver and gets the corresponding siw/rxe
> > device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5
> > respectively in the driver list, then siw sysfs device matches pci device and
> > wrongly gets the device ops of HW driver (cxgb4/mlx5).
> >
> > Below debug prints from verbs_register_driver() and driver_list entries, where
> > siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context
> > instead of siw_alloc_context, thus breaking user siw.
> >
> > <debug> verbs_register_driver_22: 184: driver 0x176e370
> > <debug> verbs_register_driver_22: 185: name ipathverbs
> > <debug> verbs_register_driver_22: 184: driver 0x176f6a0
> > <debug> verbs_register_driver_22: 185: name cxgb4
> > <debug> verbs_register_driver_22: 184: driver 0x176fd50
> > <debug> verbs_register_driver_22: 185: name cxgb3
> > <debug> verbs_register_driver_22: 184: driver 0x1777020
> > <debug> verbs_register_driver_22: 185: name rxe
> > <debug> verbs_register_driver_22: 184: driver 0x1770a30
> > <debug> verbs_register_driver_22: 185: name siw
> > <debug> verbs_register_driver_22: 184: driver 0x1771120
> > <debug> verbs_register_driver_22: 185: name mlx4
> > <debug> verbs_register_driver_22: 184: driver 0x1771990
> > <debug> verbs_register_driver_22: 185: name mlx5
> > <debug> verbs_register_driver_22: 184: driver 0x1771ff0
> > <debug> verbs_register_driver_22: 185: name efa
> >
> > <debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs
> > <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4
> > <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3
> > <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe
> > <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw
> > <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4
> > <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5
> > <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
> >
> > Proposed fix:
> > I have the below fix that works. It adds siw/rxe driver to the HEAD of the
> > driver list and the rest to the tail. I am not sure if this fix is the ideal
> > one, so I am attaching it to this mail.
>
> Update your rdma-core to latest and this will be fixed fully by using
> netlink to match the siw device..
>
I pulled the latest rdma-core, still see the issue.
commit 7ef6077ec3201f661458297fea776746ba752843 (HEAD, upstream/master)
Merge: 837954ff677c 95934b61a74e
Author: Jason Gunthorpe <jgg@mellanox.com>
Date: Thu Jul 11 16:18:06 2019 -0300
Merge pull request #539 from jgunthorpe/netlink
Use netlink to learn about ibdevs and their related chardevs
-----------
Is there any corresponding kernel change or package dependency? I am currently
on Doug's wip/dl-for-next branch.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: User SIW fails matching device
2019-07-12 15:24 ` Potnuri Bharat Teja
@ 2019-07-12 15:30 ` Jason Gunthorpe
2019-07-12 15:52 ` Potnuri Bharat Teja
0 siblings, 1 reply; 5+ messages in thread
From: Jason Gunthorpe @ 2019-07-12 15:30 UTC (permalink / raw)
To: Potnuri Bharat Teja
Cc: linux-rdma@vger.kernel.org, BMT@zurich.ibm.com,
monis@mellanox.com, Nirranjan Kirubaharan
On Fri, Jul 12, 2019 at 08:54:20PM +0530, Potnuri Bharat Teja wrote:
> On Friday, July 07/12/19, 2019 at 20:05:46 +0530, Jason Gunthorpe wrote:
> > On Fri, Jul 12, 2019 at 07:57:19PM +0530, Potnuri Bharat Teja wrote:
> > > Hi all,
> > > I observe the following behavior on one of my machines configured for siw.
> > >
> > > Issue:
> > > SIW device gets wrong device ops (HW/real rdma driver device ops) instead of
> > > siw device ops due to improper device matching.
> > >
> > > Root-cause:
> > > In libibverbs, during user cma initialisation, for each entry from the driver
> > > list, sysfs device is checked for matching name or device.
> > > If the siw/rxe driver is at the head of the list, then sysfs device matches
> > > properly with the corresponding siw driver and gets the corresponding siw/rxe
> > > device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5
> > > respectively in the driver list, then siw sysfs device matches pci device and
> > > wrongly gets the device ops of HW driver (cxgb4/mlx5).
> > >
> > > Below debug prints from verbs_register_driver() and driver_list entries, where
> > > siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context
> > > instead of siw_alloc_context, thus breaking user siw.
> > >
> > > <debug> verbs_register_driver_22: 184: driver 0x176e370
> > > <debug> verbs_register_driver_22: 185: name ipathverbs
> > > <debug> verbs_register_driver_22: 184: driver 0x176f6a0
> > > <debug> verbs_register_driver_22: 185: name cxgb4
> > > <debug> verbs_register_driver_22: 184: driver 0x176fd50
> > > <debug> verbs_register_driver_22: 185: name cxgb3
> > > <debug> verbs_register_driver_22: 184: driver 0x1777020
> > > <debug> verbs_register_driver_22: 185: name rxe
> > > <debug> verbs_register_driver_22: 184: driver 0x1770a30
> > > <debug> verbs_register_driver_22: 185: name siw
> > > <debug> verbs_register_driver_22: 184: driver 0x1771120
> > > <debug> verbs_register_driver_22: 185: name mlx4
> > > <debug> verbs_register_driver_22: 184: driver 0x1771990
> > > <debug> verbs_register_driver_22: 185: name mlx5
> > > <debug> verbs_register_driver_22: 184: driver 0x1771ff0
> > > <debug> verbs_register_driver_22: 185: name efa
> > >
> > > <debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs
> > > <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4
> > > <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3
> > > <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe
> > > <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw
> > > <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4
> > > <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5
> > > <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
> > >
> > > Proposed fix:
> > > I have the below fix that works. It adds siw/rxe driver to the HEAD of the
> > > driver list and the rest to the tail. I am not sure if this fix is the ideal
> > > one, so I am attaching it to this mail.
> >
> > Update your rdma-core to latest and this will be fixed fully by using
> > netlink to match the siw device..
> >
> I pulled the latest rdma-core, still see the issue.
>
> commit 7ef6077ec3201f661458297fea776746ba752843 (HEAD, upstream/master)
> Merge: 837954ff677c 95934b61a74e
> Author: Jason Gunthorpe <jgg@mellanox.com>
> Date: Thu Jul 11 16:18:06 2019 -0300
>
> Merge pull request #539 from jgunthorpe/netlink
>
> Use netlink to learn about ibdevs and their related chardevs
>
>
> Is there any corresponding kernel change or package dependency? I am currently
> on Doug's wip/dl-for-next branch.
That should be good enough for the kernel.. Hmm.. The siw stuff didn't
get updated, you need this rdma-core patch too. Please confirm
diff --git a/providers/siw/siw.c b/providers/siw/siw.c
index 23e4dd976caf84..41f33fa16123e9 100644
--- a/providers/siw/siw.c
+++ b/providers/siw/siw.c
@@ -907,7 +907,7 @@ static void siw_device_free(struct verbs_device *vdev)
}
static const struct verbs_match_ent rnic_table[] = {
- VERBS_NAME_MATCH("siw", NULL),
+ VERBS_DRIVER_ID(RDMA_DRIVER_SIW),
{},
};
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: User SIW fails matching device
2019-07-12 15:30 ` Jason Gunthorpe
@ 2019-07-12 15:52 ` Potnuri Bharat Teja
0 siblings, 0 replies; 5+ messages in thread
From: Potnuri Bharat Teja @ 2019-07-12 15:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma@vger.kernel.org, BMT@zurich.ibm.com,
monis@mellanox.com, Nirranjan Kirubaharan
On Friday, July 07/12/19, 2019 at 21:00:32 +0530, Jason Gunthorpe wrote:
> On Fri, Jul 12, 2019 at 08:54:20PM +0530, Potnuri Bharat Teja wrote:
> > On Friday, July 07/12/19, 2019 at 20:05:46 +0530, Jason Gunthorpe wrote:
> > > On Fri, Jul 12, 2019 at 07:57:19PM +0530, Potnuri Bharat Teja wrote:
> > > > Hi all,
> > > > I observe the following behavior on one of my machines configured for siw.
> > > >
> > > > Issue:
> > > > SIW device gets wrong device ops (HW/real rdma driver device ops) instead of
> > > > siw device ops due to improper device matching.
> > > >
> > > > Root-cause:
> > > > In libibverbs, during user cma initialisation, for each entry from the driver
> > > > list, sysfs device is checked for matching name or device.
> > > > If the siw/rxe driver is at the head of the list, then sysfs device matches
> > > > properly with the corresponding siw driver and gets the corresponding siw/rxe
> > > > device ops. Now, If the siw/rxe driver is after the real HW driver cxgb4/mlx5
> > > > respectively in the driver list, then siw sysfs device matches pci device and
> > > > wrongly gets the device ops of HW driver (cxgb4/mlx5).
> > > >
> > > > Below debug prints from verbs_register_driver() and driver_list entries, where
> > > > siw is after cxgb4. I see verbs alloc context landing in cxgb4_alloc_context
> > > > instead of siw_alloc_context, thus breaking user siw.
> > > >
> > > > <debug> verbs_register_driver_22: 184: driver 0x176e370
> > > > <debug> verbs_register_driver_22: 185: name ipathverbs
> > > > <debug> verbs_register_driver_22: 184: driver 0x176f6a0
> > > > <debug> verbs_register_driver_22: 185: name cxgb4
> > > > <debug> verbs_register_driver_22: 184: driver 0x176fd50
> > > > <debug> verbs_register_driver_22: 185: name cxgb3
> > > > <debug> verbs_register_driver_22: 184: driver 0x1777020
> > > > <debug> verbs_register_driver_22: 185: name rxe
> > > > <debug> verbs_register_driver_22: 184: driver 0x1770a30
> > > > <debug> verbs_register_driver_22: 185: name siw
> > > > <debug> verbs_register_driver_22: 184: driver 0x1771120
> > > > <debug> verbs_register_driver_22: 185: name mlx4
> > > > <debug> verbs_register_driver_22: 184: driver 0x1771990
> > > > <debug> verbs_register_driver_22: 185: name mlx5
> > > > <debug> verbs_register_driver_22: 184: driver 0x1771ff0
> > > > <debug> verbs_register_driver_22: 185: name efa
> > > >
> > > > <debug> try_drivers: 372: driver 0x176e370, sysfs_dev 0x1776b20, name: ipathverbs
> > > > <debug> try_drivers: 372: driver 0x176f6a0, sysfs_dev 0x1776b20, name: cxgb4
> > > > <debug> try_drivers: 372: driver 0x176fd50, sysfs_dev 0x1776b20, name: cxgb3
> > > > <debug> try_drivers: 372: driver 0x1777020, sysfs_dev 0x1776b20, name: rxe
> > > > <debug> try_drivers: 372: driver 0x1770a30, sysfs_dev 0x1776b20, name: siw
> > > > <debug> try_drivers: 372: driver 0x1771120, sysfs_dev 0x1776b20, name: mlx4
> > > > <debug> try_drivers: 372: driver 0x1771990, sysfs_dev 0x1776b20, name: mlx5
> > > > <debug> try_drivers: 372: driver 0x1771ff0, sysfs_dev 0x1776b20, name: efa
> > > >
> > > > Proposed fix:
> > > > I have the below fix that works. It adds siw/rxe driver to the HEAD of the
> > > > driver list and the rest to the tail. I am not sure if this fix is the ideal
> > > > one, so I am attaching it to this mail.
> > >
> > > Update your rdma-core to latest and this will be fixed fully by using
> > > netlink to match the siw device..
> > >
> > I pulled the latest rdma-core, still see the issue.
> >
> > commit 7ef6077ec3201f661458297fea776746ba752843 (HEAD, upstream/master)
> > Merge: 837954ff677c 95934b61a74e
> > Author: Jason Gunthorpe <jgg@mellanox.com>
> > Date: Thu Jul 11 16:18:06 2019 -0300
> >
> > Merge pull request #539 from jgunthorpe/netlink
> >
> > Use netlink to learn about ibdevs and their related chardevs
> >
> >
> > Is there any corresponding kernel change or package dependency? I am currently
> > on Doug's wip/dl-for-next branch.
>
> That should be good enough for the kernel.. Hmm.. The siw stuff didn't
> get updated, you need this rdma-core patch too. Please confirm
>
> diff --git a/providers/siw/siw.c b/providers/siw/siw.c
> index 23e4dd976caf84..41f33fa16123e9 100644
> --- a/providers/siw/siw.c
> +++ b/providers/siw/siw.c
> @@ -907,7 +907,7 @@ static void siw_device_free(struct verbs_device *vdev)
> }
>
> static const struct verbs_match_ent rnic_table[] = {
> - VERBS_NAME_MATCH("siw", NULL),
> + VERBS_DRIVER_ID(RDMA_DRIVER_SIW),
> {},
> };
>
It works with above change. Thanks!
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-07-12 15:53 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-07-12 14:27 User SIW fails matching device Potnuri Bharat Teja
2019-07-12 14:35 ` Jason Gunthorpe
2019-07-12 15:24 ` Potnuri Bharat Teja
2019-07-12 15:30 ` Jason Gunthorpe
2019-07-12 15:52 ` Potnuri Bharat Teja
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).