Re: [PATCH for-next 3/3] IB/hfi1: Use the ibdev in hfi1_devdata as the parent of cdev

[Jason Gunthorpe <jgg@ziepe.ca>]

On Wed, Mar 18, 2020 at 04:02:42PM +0000, Wan, Kaike wrote:
> 
> 
> > From: Jason Gunthorpe <jgg@ziepe.ca>
> > Sent: Wednesday, March 18, 2020 9:32 AM
> > To: Dalessandro, Dennis <dennis.dalessandro@intel.com>
> > Cc: dledford@redhat.com; linux-rdma@vger.kernel.org; Marciniszyn, Mike
> > <mike.marciniszyn@intel.com>; Wan, Kaike <kaike.wan@intel.com>
> > Subject: Re: [PATCH for-next 3/3] IB/hfi1: Use the ibdev in hfi1_devdata as
> > the parent of cdev
> > 
> > On Mon, Mar 16, 2020 at 05:05:07PM -0400, Dennis Dalessandro wrote:
> > > From: Kaike Wan <kaike.wan@intel.com>
> > >
> > > This patch is implemented to address the concerns raised in:
> > >   https://marc.info/?l=linux-rdma&m=158101337614772&w=2
> > >
> > > The hfi1 driver dynammically allocates a struct device to represent
> > > the cdev in sysfs and devtmpfs (/dev/hfi1_x). On the other hand, the
> > > hfi1_devdata already contains a struct device in its ibdev field
> > > (hfi1_devdata.verbs_dev.rdi.ibdev.dev), and it is therefore possible
> > > to eliminate the dynamical allocation when creating the cdev. Since
> > > each device could be added to the sysfs only once and the function
> > > device_add() is already called for the ibdev in ib_register_device(),
> > > the function cdev_device_add() could not be used to create the cdev,
> > > even though the hfi1_devdata contains both cdev and ibdev in the same
> > > structure.
> > >
> > > This patch eliminates the dynamic allocation by creating the cdev
> > > first, setting up the ibdev, and then calling the ib_register_device()
> > > to add the device to sysfs and devtmpfs.
> > 
> > What do the sysfs paths for the cdev look like now?
> 
> ls -l /sys/dev/char/243:0
> lrwxrwxrwx 1 root root 0 Mar 15 14:30 /sys/dev/char/243:0 -> ../../devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/hfi1_0
> 
> It points back to the IB device (hfi1_0 ).
> 
> Before this change, it pointed back to a virtual device:
> 
> ls /sys/dev/char/243:0 -l
> lrwxrwxrwx 1 root root 0 Mar 18 11:52 /sys/dev/char/243:0 -> ../../devices/virtual/hfi1_user/hfi1_0

Great, yes this looks right to me

So this came up due to PSM having problems.. The right way for PSM
to work is now to find the hfi1_0 devices under /sys/class/hfi1_user/*
and then map then back to RDMA devices and the physical card by doing
realpath and learning the
'/sys/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/XXX/' path

Yes?

Jason