How to make a bus with heterogeneous devices?

How to make a bus with heterogeneous devices?

[Alessandro Rubini]

Hello Greg. 

Federico Vaga and me are working on the ZIO framework
(ohwr.org/projects/zio) and he's currently adding the bus abstraction,
so we can support several cards of the same type.  Under ZIO,
different devices may have different attributes: an SPI ADC chip may
allow to choose from two voltage references while a PCI digital-output
device may have a device-wide clocking rate.

But the list of attributes associated to a device is only known when
the driver is matched. In other workds, the "device" part is only
claiming "here I am and I'm called <sth>", then the probe method of the
<sth> driver will instantiate the device with its attributes.

However, the device appears in sysfs whether or not there is a driver
for it (which we agree is correct), but the list of attributes is only
known when the driver is matched -- because the list is known to the
driver, not to the device.

Here are some paths we evaluated:

1- /sys/bus/zio/devices may include place-holders, while a successful
match and probe will create real stuff is in /sys/zio/devices/<sth>-0/
But this would mean kobjects and you mandated us to only use devices
instead.

2- Like above, but falling in /sys/devices/virtual/zio/ ?

3- Attributes may be added after match succeeds, before calling probe.
This cam be done with a "raw" call to sysfs_create_group(); then
groups cna be added to dev->groups so they will be removed on
device_unregister().

4- Attributes may be added to the device before drv->probe is called
(or after it succeeds), using FIXMEFEDE. But .... FIXMEFEDE

5- The real attributes may live in another device, which is a child of
the one used within the bus abstraction.  Here the drawback is using
an additional device and directory level for no reason.

6- We may use a fake (almost empty) device for the match function, and
then register the real one, with full attributes, when the match
succeeds (so match is called again and must be worked around). This
would work but is clearly a horrible hack.

We looked at other bus abstractions, and all of them host homogeneous
devices (they may have sub-devices, but none of them has different
attributes).  Even the platform bus has no per-device attribute list
instantiated by the driver.  In the cases where the attribute list is
different for each device, the list is known to the device, before the
match function is called.

Any suggestion is welcome. Meanwhile, Federico started implementing 3
above, but it still looks a dirty approach, and possibly not future-proof.

thanks
/alessandro (and federico)

Re: How to make a bus with heterogeneous devices?

[Greg Kroah-Hartman]

On Fri, Feb 24, 2012 at 04:20:51PM +0100, Alessandro Rubini wrote:
> Hello Greg. 
> 
> Federico Vaga and me are working on the ZIO framework
> (ohwr.org/projects/zio) and he's currently adding the bus abstraction,
> so we can support several cards of the same type.  Under ZIO,
> different devices may have different attributes: an SPI ADC chip may
> allow to choose from two voltage references while a PCI digital-output
> device may have a device-wide clocking rate.
> 
> But the list of attributes associated to a device is only known when
> the driver is matched. In other workds, the "device" part is only
> claiming "here I am and I'm called <sth>", then the probe method of the
> <sth> driver will instantiate the device with its attributes.

That's fine, it is what happens with lots of drivers, you can do this
with the default attribute group assigned to that driver, right?

> However, the device appears in sysfs whether or not there is a driver
> for it (which we agree is correct), but the list of attributes is only
> known when the driver is matched -- because the list is known to the
> driver, not to the device.
> 
> Here are some paths we evaluated:
> 
> 1- /sys/bus/zio/devices may include place-holders, while a successful
> match and probe will create real stuff is in /sys/zio/devices/<sth>-0/
> But this would mean kobjects and you mandated us to only use devices
> instead.
> 
> 2- Like above, but falling in /sys/devices/virtual/zio/ ?
> 
> 3- Attributes may be added after match succeeds, before calling probe.
> This cam be done with a "raw" call to sysfs_create_group(); then
> groups cna be added to dev->groups so they will be removed on
> device_unregister().
> 
> 4- Attributes may be added to the device before drv->probe is called
> (or after it succeeds), using FIXMEFEDE. But .... FIXMEFEDE
> 
> 5- The real attributes may live in another device, which is a child of
> the one used within the bus abstraction.  Here the drawback is using
> an additional device and directory level for no reason.
> 
> 6- We may use a fake (almost empty) device for the match function, and
> then register the real one, with full attributes, when the match
> succeeds (so match is called again and must be worked around). This
> would work but is clearly a horrible hack.

Ick, don't do that...

> We looked at other bus abstractions, and all of them host homogeneous
> devices (they may have sub-devices, but none of them has different
> attributes).  Even the platform bus has no per-device attribute list
> instantiated by the driver.  In the cases where the attribute list is
> different for each device, the list is known to the device, before the
> match function is called.

I think you need to look at how the system bus is handled.  It just went
into 3.3-rc and allows for "different" types of devices to all be on the
same bus, with different drivers and attributes.  Let me know if how
that works does not work out for you.

thanks,

greg k-h

Re: How to make a bus with heterogeneous devices?

[Federico Vaga]

> I think you need to look at how the system bus is handled.  It just went
> into 3.3-rc and allows for "different" types of devices to all be on the
> same bus, with different drivers and attributes.  Let me know if how
> that works does not work out for you.

Can I have a reference to the patch(s) which introduces this? 
I taked a look (3.3-rc5) but I didn't find it

Thank you

-- 
Federico Vaga

Re: How to make a bus with heterogeneous devices?

[Greg Kroah-Hartman]

On Wed, Feb 29, 2012 at 02:57:01PM +0100, Federico Vaga wrote:
> > I think you need to look at how the system bus is handled.  It just went
> > into 3.3-rc and allows for "different" types of devices to all be on the
> > same bus, with different drivers and attributes.  Let me know if how
> > that works does not work out for you.
> 
> Can I have a reference to the patch(s) which introduces this? 
> I taked a look (3.3-rc5) but I didn't find it

Look at ca22e56debc57b47c422b749c93217ba62644be2 and the patches after
it.

Hope this helps,

greg k-h

Re: How to make a bus with heterogeneous devices?

[Federico Vaga]

> That's fine, it is what happens with lots of drivers, you can do this
> with the default attribute group assigned to that driver, right?

We can't because a driver can handle devices with different set of 
attributes. We  can't use device->groups because only if match succeed 
we know which attributes add to the device.
That's because the same driver handles several similar devices, to avoid 
proliferation of almost-identical drivers. So the drivers attaches 
attributes to the device after match succeeds.

The solution that I temporarily  implemented is the following:

> > [...]
> > 3- Attributes may be added after match succeeds, before calling
> > probe.
> > This cam be done with a "raw" call to sysfs_create_group(); then
> > groups cna be added to dev->groups so they will be removed on
> > device_unregister().
> > [...]

(with calling probe I mean calling "my" driver probe)

I know it's not a nice solution and I know it's not nice add groups to 
device->groups after registration. I can use sysfs_create_group() and 
sysfs_remove_group() without touching the device.
 
> I think you need to look at how the system bus is handled.  It just 
> went into 3.3-rc and allows for "different" types of devices to all be 
> on the same bus, with different drivers and attributes.  Let me know if 
> how that works does not work out for you.

(thank you for the reference to the patches)

I think it doesn't work in our case for the same reason we can't use 
device_driver->groups: we know too late in the register process which 
attributes add to a device, practically at the end of the registration.
Maybe I'm missing something.

-- 
Federico Vaga

Re: How to make a bus with heterogeneous devices?

[Greg Kroah-Hartman]

On Fri, Mar 02, 2012 at 11:18:59AM +0100, Federico Vaga wrote:
> > That's fine, it is what happens with lots of drivers, you can do this
> > with the default attribute group assigned to that driver, right?
> 
> We can't because a driver can handle devices with different set of 
> attributes. We  can't use device->groups because only if match succeed 
> we know which attributes add to the device.
> That's because the same driver handles several similar devices, to avoid 
> proliferation of almost-identical drivers. So the drivers attaches 
> attributes to the device after match succeeds.

Unless you manage the uevents by "hand", you can run into big problems
with userspace tools if you add the attributes after the device is
matched to a driver.  Be very careful here.

> The solution that I temporarily  implemented is the following:
> 
> > > [...]
> > > 3- Attributes may be added after match succeeds, before calling
> > > probe.
> > > This cam be done with a "raw" call to sysfs_create_group(); then
> > > groups cna be added to dev->groups so they will be removed on
> > > device_unregister().
> > > [...]
> 
> (with calling probe I mean calling "my" driver probe)
> 
> I know it's not a nice solution and I know it's not nice add groups to 
> device->groups after registration. I can use sysfs_create_group() and 
> sysfs_remove_group() without touching the device.

It's not an issue of "touching" the device, it's an issue of "userspace
just got missed all of those attributes", so your tools will be all
confused.  The kernel has no problem with it :)

> > I think you need to look at how the system bus is handled.  It just 
> > went into 3.3-rc and allows for "different" types of devices to all be 
> > on the same bus, with different drivers and attributes.  Let me know if 
> > how that works does not work out for you.
> 
> (thank you for the reference to the patches)
> 
> I think it doesn't work in our case for the same reason we can't use 
> device_driver->groups: we know too late in the register process which 
> attributes add to a device, practically at the end of the registration.
> Maybe I'm missing something.

I think you are.

Kay, you know this area of the driver core better than I do at the
moment, given your most recent changes here.  Any reason why Alessandro
and Federico can't do the same thing that the sysdev rework did for
their devices here?

thanks,

greg k-h

Re: How to make a bus with heterogeneous devices?

[Kay Sievers]

On Fri, Mar 2, 2012 at 23:38, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Fri, Mar 02, 2012 at 11:18:59AM +0100, Federico Vaga wrote:

>> Maybe I'm missing something.
>
> I think you are.
>
> Kay, you know this area of the driver core better than I do at the
> moment, given your most recent changes here.  Any reason why Alessandro
> and Federico can't do the same thing that the sysdev rework did for
> their devices here?

Maybe I miss something but it sounds not that different from USB.

You have a bus that enumerates devices and a "bus driver" creates a
generic device on that bus for everything it finds. These devices stay
generic, do not have type-specific attributes, but export matches
again to bind other device-specific drivers, which create _new_ child
devices below the generic ones with the proper attributes matching
their individual type of devices. So every "logical" device is always
represented by one generic device and one type-specific child device.

All these devices, the generic ones and the type-specific ones can be
on the same bus (struct bus_type) when they get a "struct device_type"
assigned which carries the device-type specific set of attributes.

It's like USB, where we get a plain "usb_device" for every device
connected, then we look at each device to find out how many
"usb_interface" childs we need to create, and every interface device
offers again matches for more drivers again, like tty, storage, video,
... to bind to as child devices.

Wouldn't that work?

Kay

Re: How to make a bus with heterogeneous devices?

[Federico Vaga]

> Maybe I miss something but it sounds not that different from USB.
> 
> You have a bus that enumerates devices and a "bus driver" creates a
> generic device on that bus for everything it finds. These devices stay
> generic, do not have type-specific attributes, but export matches
> again to bind other device-specific drivers, which create _new_ child
> devices below the generic ones with the proper attributes matching
> their individual type of devices. So every "logical" device is always
> represented by one generic device and one type-specific child device.

It sounds like our option 5:

> 5- The real attributes may live in another device, which is a child of
> the one used within the bus abstraction.  Here the drawback is using
> an additional device and directory level for no reason.

- I register the generic (empty) device on the bus
- when the device match with a driver 
  - I register a new device child of the matched one but without connect     
    it directly on the bus.

The result will be something like (zio is our bus):

/sys/bus/zio/devices/<generic-dev1>/<real-device1>/<attributes>
/sys/bus/zio/devices/<generic-dev2>/<real-device2>/<attributes>
/sys/bus/zio/devices/<generic-devN>/<real-deviceN>/<attributes>

When there is not a driver:

/sys/bus/zio/devices/<generic-dev1>/<device-stuff>
/sys/bus/zio/devices/<generic-devN>/<device-stuff>

Where the generic-dev is an empty device used to make the bus work. Is 
it correct?

> All these devices, the generic ones and the type-specific ones can be
> on the same bus (struct bus_type) when they get a "struct device_type"
> assigned which carries the device-type specific set of attributes.

But if all devices can live on the bus (set device->bus for all 
devices), when I register the "real-device" (type-specific)  the bus try 
to match it with a driver (again); but we already matched the device 
with its driver. it is better if the type-specific devices are not 
directly on the bus but only a child of a device on the bus.

-- 
Federico Vaga

Re: How to make a bus with heterogeneous devices?

[Kay Sievers]

On Sat, Mar 3, 2012 at 16:44, Federico Vaga <federico.vaga@gmail.com> wrote:
>> Maybe I miss something but it sounds not that different from USB.
>>
>> You have a bus that enumerates devices and a "bus driver" creates a
>> generic device on that bus for everything it finds. These devices stay
>> generic, do not have type-specific attributes, but export matches
>> again to bind other device-specific drivers, which create _new_ child
>> devices below the generic ones with the proper attributes matching
>> their individual type of devices. So every "logical" device is always
>> represented by one generic device and one type-specific child device.

>> 5- The real attributes may live in another device, which is a child of
>> the one used within the bus abstraction.  Here the drawback is using
>> an additional device and directory level for no reason.
>
> - I register the generic (empty) device on the bus

You can always add some generic, no-type specific attributes there, if
that fits in your model.

> - when the device match with a driver
>  - I register a new device child of the matched one but without connect
>    it directly on the bus.

You can assign all these devices to the same bus (see details below).

> The result will be something like (zio is our bus):
>
> /sys/bus/zio/devices/<generic-dev1>/<real-device1>/<attributes>
> /sys/bus/zio/devices/<generic-dev2>/<real-device2>/<attributes>
> /sys/bus/zio/devices/<generic-devN>/<real-deviceN>/<attributes>
>
> When there is not a driver:
>
> /sys/bus/zio/devices/<generic-dev1>/<device-stuff>
> /sys/bus/zio/devices/<generic-devN>/<device-stuff>
>
> Where the generic-dev is an empty device used to make the bus work. Is
> it correct?

All buses have have a 'subsystem directory' at:
  /sys/bus/
This location exposes the 'device tree classification', like the
'collection of devices by subsystem'

The bus "zio" has its directory at:
  /sys/bus/zio/

All devices known to the bus "zio" are referenced at:
  /sys/bus/zio/devices/*

Which is a bunch of symlinks pointing into the:
  /sys/devices/
directory, which is the tree of all devices from all subsystems. This
tree exposes the 'device tree hierarchy'.

So your registered devices would all show up in:
  /sys/bus/zio/devices/generic0 --> ../../../devices/.../generic0
  /sys/bus/zio/devices/generic1 --> ../../../devices/.../generic1
 /sys/bus/zio/devices/generic2 --> ../../../devices/.../generic2
  /sys/bus/zio/devices/type-a0 --> ../../../devices/.../generic0/type-a0
  /sys/bus/zio/devices/type-a1 --> ../../../devices/.../generic1/type-a1
  /sys/bus/zio/devices/type-b0 --> ../../../devices/.../generic2/type-b0

They all show up on the same bus, but the type-specific devices are
child devices of the generic ones.

>> All these devices, the generic ones and the type-specific ones can be
>> on the same bus (struct bus_type) when they get a "struct device_type"
>> assigned which carries the device-type specific set of attributes.
>
> But if all devices can live on the bus (set device->bus for all
> devices), when I register the "real-device" (type-specific)  the bus try
> to match it with a driver (again); but we already matched the device
> with its driver. it is better if the type-specific devices are not
> directly on the bus but only a child of a device on the bus.

You can sort them out by 'struct device_type' in your probe functions,
that should  not be a problem.

We generally do not like devices which do not belong to a bus or
class, they are kind of 'invisible' to userspace, and udev can not use
them directly, it sees them only as a parent device of a device with a
bus or class.

Kay

Re: How to make a bus with heterogeneous devices?

[Federico Vaga]

We moved to this solution. Thank you

-- 
Federico Vaga