Hardware Abstraction Layer

Hardware Abstraction Layer

[David Zeuthen]

Hi,

I'm currently working on a hardware abstraction layer, see

  http://pdx.freedesktop.org/~hal/

for use in desktop environments such as GNOME or KDE.  The HAL project
is in it's early stages - there is a draft spec planned and I got an
implementation running as well.

I want to hear how you think we can cooperate, by e.g. having
/sbin/hotplug send events to hald. Note that the HAL aims at a range of
UNIX-like systems, but I think we can handle that requirement... I use
GNU/Linux myself.

There is also a discussion going over at the xdg-list at
freedesktop.org, you might want to out check the archives.

Let me know what you think. Comments are appreciated. Note that I'm out
traveling until late Tuesday with no access to e-mail so I won't be able
to reply for while.

Thank you for your time!

Regards,
David

-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-    David Zeuthen             pgp key:     http://fubar.dk/pgpkey.asc
-  david(at)fubar.dk           pgp key id:                    b89bab82




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Joerg Sommer]

begin  David Zeuthen <david@fubar.dk> wrote:
> I'm currently working on a hardware abstraction layer, see
> 
>   http://pdx.freedesktop.org/~hal/

Nice.

> I want to hear how you think we can cooperate, by e.g. having
> /sbin/hotplug send events to hald. Note that the HAL aims at a range of
> UNIX-like systems, but I think we can handle that requirement... I use
> GNU/Linux myself.

if you take a look at /sbin/hotplug, you see it calls all files in
/etc/hotplug.d/default/*.hotplug. So if you place their a script you
receive all hotplug events and it then up to you to react on them.

Jörg.



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Sat, Sep 13, 2003 at 03:30:38PM +0200, David Zeuthen wrote:
> Hi,
> 
> I'm currently working on a hardware abstraction layer, see
> 
>   http://pdx.freedesktop.org/~hal/
> 
> for use in desktop environments such as GNOME or KDE.  The HAL project
> is in it's early stages - there is a draft spec planned and I got an
> implementation running as well.

How does this differ from D-BUS?  Hm, in looking at your page, it says
you are using D-BUS messages.  Oh, have you looked at udev too?  It will
handle naming the devices for you in /dev.

> I want to hear how you think we can cooperate, by e.g. having
> /sbin/hotplug send events to hald. Note that the HAL aims at a range of
> UNIX-like systems, but I think we can handle that requirement... I use
> GNU/Linux myself.

As was already posted, anyone can get /sbin/hotplug events by just
dropping a link into the proper directory.  Also, D-BUS messages will be
created for all /sbin/hotplug events too.  You might want to work with
that if it's easier for you.

Hope this helps,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[David Zeuthen]

On Tue, 2003-09-16 at 18:39, Greg KH wrote:
> On Sat, Sep 13, 2003 at 03:30:38PM +0200, David Zeuthen wrote:
> > Hi,
> > 
> > I'm currently working on a hardware abstraction layer, see
> > 
> >   http://pdx.freedesktop.org/~hal/
> > 
> > for use in desktop environments such as GNOME or KDE.  The HAL project
> > is in it's early stages - there is a draft spec planned and I got an
> > implementation running as well.
> 
> How does this differ from D-BUS?  Hm, in looking at your page, it says
> you are using D-BUS messages.  

I use D-BUS as the IPC mechanism between hotplug agents and applications
wanting to query a device database. In the middle there is a daemon. 

In a sense I want the entire HAL not to care about specific hardware
issues at all, but build on existing software like linux-hotplug ;-) 
(see my recent post to the xdg-list for detail)

> Oh, have you looked at udev too?  It will
> handle naming the devices for you in /dev.
> 

Excellent - will look into this later. Will this help me, e.g., getting
the information that my CF card reader is at /dev/sda1 when using the
kernel module usb-storage? Or have I misunderstood?

> > I want to hear how you think we can cooperate, by e.g. having
> > /sbin/hotplug send events to hald. Note that the HAL aims at a range of
> > UNIX-like systems, but I think we can handle that requirement... I use
> > GNU/Linux myself.
> 
> As was already posted, anyone can get /sbin/hotplug events by just
> dropping a link into the proper directory.  

Cool.

> Also, D-BUS messages will be
> created for all /sbin/hotplug events too.  You might want to work with
> that if it's easier for you.
> 

I'll be interested in this, definately. Are you using the service
org.freedesktop.DBus.Broadcast for this? 

I'll also need a unique ID of each device being hotplugged, and I have
some problems seeing this is even possible on some busses like USB. The
unique ID must be the same over plugs/unplugs.

Say, that I plug two identical cameras into the system with no
device-instance specific information. Corner case, but nasty one... 

In the event this is not possible the hotplug agent must somehow convey
this information so I can append a counter to make the ID unique in that
way.. 

Any suggestions / comments ?

> Hope this helps,
> 

Very much, Thanks,
David




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Wed, Sep 17, 2003 at 02:40:12AM +0200, David Zeuthen wrote:
> On Tue, 2003-09-16 at 18:39, Greg KH wrote:
> > On Sat, Sep 13, 2003 at 03:30:38PM +0200, David Zeuthen wrote:
> > > Hi,
> > > 
> > > I'm currently working on a hardware abstraction layer, see
> > > 
> > >   http://pdx.freedesktop.org/~hal/
> > > 
> > > for use in desktop environments such as GNOME or KDE.  The HAL project
> > > is in it's early stages - there is a draft spec planned and I got an
> > > implementation running as well.
> > 
> > How does this differ from D-BUS?  Hm, in looking at your page, it says
> > you are using D-BUS messages.  
> 
> I use D-BUS as the IPC mechanism between hotplug agents and applications
> wanting to query a device database. In the middle there is a daemon. 
> 
> In a sense I want the entire HAL not to care about specific hardware
> issues at all, but build on existing software like linux-hotplug ;-) 
> (see my recent post to the xdg-list for detail)

Hm, in looking over your post I think you are going to be duplicating a
lot of existing information that is dynamically available to you from
the kernel.

For example, the kernel already knows what USB devices are supported by
what drivers.  It exports that info to userspace in the modules.*map
files in /lib/modules/<KERNEL_VERSION>/

The existing hotplug scripts already handle the mapping from device to
driver, so you don't need to duplicate that functionaity.

> > Oh, have you looked at udev too?  It will
> > handle naming the devices for you in /dev.
> > 
> 
> Excellent - will look into this later. Will this help me, e.g., getting
> the information that my CF card reader is at /dev/sda1 when using the
> kernel module usb-storage? Or have I misunderstood?

Yes it will.  See my OLS paper from this year for lots more information
about it.

> > Also, D-BUS messages will be
> > created for all /sbin/hotplug events too.  You might want to work with
> > that if it's easier for you.
> > 
> 
> I'll be interested in this, definately. Are you using the service
> org.freedesktop.DBus.Broadcast for this? 

I'm not using anything :)

There was a simple command line program that generated dbus messages.
Someone needs to take that and have it convert hotplug messages into
dbus messages.  I think it's going to use the org.kernel namespace.

> I'll also need a unique ID of each device being hotplugged, and I have
> some problems seeing this is even possible on some busses like USB. The
> unique ID must be the same over plugs/unplugs.

What "unique ID"?  In short, you are never guaranteed such a thing, but
you can get close.  See my udev paper for more info on this.

> Say, that I plug two identical cameras into the system with no
> device-instance specific information. Corner case, but nasty one... 

Use the bus topology to distinguish between those two devices.

> In the event this is not possible the hotplug agent must somehow convey
> this information so I can append a counter to make the ID unique in that
> way.. 

The kernel will tell you that this is a new device.  It's up to
userspace to do something with that info.  udev will determine the /dev
name for the device.  It's up to other programs to do other things with
this info (which is what I'm thinking your proposal does, correct?)

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[David Zeuthen]

On Thu, 2003-09-18 at 02:29, Greg KH wrote:
> On Wed, Sep 17, 2003 at 02:40:12AM +0200, David Zeuthen wrote:
> > On Tue, 2003-09-16 at 18:39, Greg KH wrote:
> > > On Sat, Sep 13, 2003 at 03:30:38PM +0200, David Zeuthen wrote:
> > > > Hi,
> > > > 
> > > > I'm currently working on a hardware abstraction layer, see
> > > > 
> > > >   http://pdx.freedesktop.org/~hal/
> > > > 
> > > > for use in desktop environments such as GNOME or KDE.  The HAL project
> > > > is in it's early stages - there is a draft spec planned and I got an
> > > > implementation running as well.
> > > 
> > > How does this differ from D-BUS?  Hm, in looking at your page, it says
> > > you are using D-BUS messages.  
> > 
> > I use D-BUS as the IPC mechanism between hotplug agents and applications
> > wanting to query a device database. In the middle there is a daemon. 
> > 
> > In a sense I want the entire HAL not to care about specific hardware
> > issues at all, but build on existing software like linux-hotplug ;-) 
> > (see my recent post to the xdg-list for detail)
> 
> Hm, in looking over your post I think you are going to be duplicating a
> lot of existing information that is dynamically available to you from
> the kernel.
> 

I see your point. You specifically mention the word ''post'', so I'll
assume that you are talking about my initial message to the xdg-list.
This post mentions both ''drivers'' and ''kernel modules'', which I
admit is confusing.

In the latest version, available at the link above, I specifically avoid
using these words and only talk about ''device info files'' and ''boot
programs''. 

The contention is that properties for a device should contain a)
bus-device-specific properties (such as usb.idVendor, usb.bcdDevice,
usb.linux_devfs etc.) and category-specific properties (such as
Storage.Device=/dev/sda1 and Storage.MountPoint=/mnt/flash). 

Applications of the HAL (e.g. desktop environments) will then use
standard kernel and OS interfaces to e.g. display the amount of free
space on a Storage device. 

I hope this clear matters up?

> For example, the kernel already knows what USB devices are supported by
> what drivers.  It exports that info to userspace in the modules.*map
> files in /lib/modules/<KERNEL_VERSION>/
> 
> The existing hotplug scripts already handle the mapping from device to
> driver, so you don't need to duplicate that functionaity.
> 

You are right. Note that the HAL needs to be kernel / OS agnostic but
the idea is to handle this at the device-info level. So, for Linux the
BootProgram for a storage-device simply mounts the device somewhere
chosen by the user or the desktop environment. On other OS'es the
bootprogram may need to insert modules or whatever into a running
kernel.

This implies that, for instance, /etc/hotplug/usb.hotplug will need to
send a DBUS message *after* the kernel modules have been successfully
loaded - see proposal below.

Side question: If a device vendor ships a (kernel) driver for his USB
device will modules.*map get updated?

> > > Oh, have you looked at udev too?  It will
> > > handle naming the devices for you in /dev.
> > > 
> > 
> > Excellent - will look into this later. Will this help me, e.g., getting
> > the information that my CF card reader is at /dev/sda1 when using the
> > kernel module usb-storage? Or have I misunderstood?
> 
> Yes it will.  See my OLS paper from this year for lots more information
> about it.
> 

Great! I'll look into this soon.

> > > Also, D-BUS messages will be
> > > created for all /sbin/hotplug events too.  You might want to work with
> > > that if it's easier for you.
> > > 
> > 
> > I'll be interested in this, definately. Are you using the service
> > org.freedesktop.DBus.Broadcast for this? 
> 
> I'm not using anything :)
> 
> There was a simple command line program that generated dbus messages.
> Someone needs to take that and have it convert hotplug messages into
> dbus messages.  I think it's going to use the org.kernel namespace.
> 

My initial experiments was actually to patch /etc/hotplug/usb.hotplug
and used the same command-line tool. The tool is a bit weak still (but
DBUS is not at 1.0 yet) in so far that obvious data-types like
dictionaries are not available yet. Now I just have a hackish C program
using libdbus to parse the environment given from the kernel and send it
off to the HAL daemon.

IMHO, the best possible scenario for HAL (wrt. hotplugging) would be to
have a well-defined DBUS message scheme for each bus type and
linux-hotplug to implement this broadcasting it through the service
org.freedesktop.DBus.Broadcast. 

Data in these messages would contain the usb.* fields and information
about what (linux) kernel modules was loaded if any (cannot find device
driver etc.).

If there is not a well-defined scheme yet, I'll volunteer to write a
spec and implement it as part of linux-hotplug. We can put this spec at
either the HAL site or the linux-hotplug site. The former would IMHO be
better since we want the scheme to be OS agnostic, but I'm flexible. I'm
quite certain the scheme, to an extent, can be OS agnostic.

What is your take on this?

> > I'll also need a unique ID of each device being hotplugged, and I have
> > some problems seeing this is even possible on some busses like USB. The
> > unique ID must be the same over plugs/unplugs.
> 
> What "unique ID"?  In short, you are never guaranteed such a thing, but
> you can get close.  See my udev paper for more info on this.
> 
> > Say, that I plug two identical cameras into the system with no
> > device-instance specific information. Corner case, but nasty one... 
> 
> Use the bus topology to distinguish between those two devices.
> 
> > In the event this is not possible the hotplug agent must somehow convey
> > this information so I can append a counter to make the ID unique in that
> > way.. 
> 
> The kernel will tell you that this is a new device.  It's up to
> userspace to do something with that info.  udev will determine the /dev
> name for the device.  It's up to other programs to do other things with
> this info (which is what I'm thinking your proposal does, correct?)
> 

Ok, the HAL daemon can take care of this as long as we agree on the DBUS
message format. HAL is strictly user-space.

Thanks,
David




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Thu, Sep 18, 2003 at 01:24:39PM +0200, David Zeuthen wrote:
> On Thu, 2003-09-18 at 02:29, Greg KH wrote:
> > On Wed, Sep 17, 2003 at 02:40:12AM +0200, David Zeuthen wrote:
> > > On Tue, 2003-09-16 at 18:39, Greg KH wrote:
> > > > On Sat, Sep 13, 2003 at 03:30:38PM +0200, David Zeuthen wrote:
> > > > > Hi,
> > > > > 
> > > > > I'm currently working on a hardware abstraction layer, see
> > > > > 
> > > > >   http://pdx.freedesktop.org/~hal/
> > > > > 
> > > > > for use in desktop environments such as GNOME or KDE.  The HAL project
> > > > > is in it's early stages - there is a draft spec planned and I got an
> > > > > implementation running as well.
> > > > 
> > > > How does this differ from D-BUS?  Hm, in looking at your page, it says
> > > > you are using D-BUS messages.  
> > > 
> > > I use D-BUS as the IPC mechanism between hotplug agents and applications
> > > wanting to query a device database. In the middle there is a daemon. 
> > > 
> > > In a sense I want the entire HAL not to care about specific hardware
> > > issues at all, but build on existing software like linux-hotplug ;-) 
> > > (see my recent post to the xdg-list for detail)
> > 
> > Hm, in looking over your post I think you are going to be duplicating a
> > lot of existing information that is dynamically available to you from
> > the kernel.
> > 
> 
> I see your point. You specifically mention the word ''post'', so I'll
> assume that you are talking about my initial message to the xdg-list.
> This post mentions both ''drivers'' and ''kernel modules'', which I
> admit is confusing.

Yes, that is what I was referring to.

> In the latest version, available at the link above, I specifically avoid
> using these words and only talk about ''device info files'' and ''boot
> programs''. 
> 
> The contention is that properties for a device should contain a)
> bus-device-specific properties (such as usb.idVendor, usb.bcdDevice,
> usb.linux_devfs etc.) and category-specific properties (such as
> Storage.Device=/dev/sda1 and Storage.MountPoint=/mnt/flash). 

But my main point is, why store this info anywhere, when it is already
stored on your machine today?  And it is automatically already handled
properly today, without any need for "yet another abstraction layer" to
handle it?

Yes, I know this only works on Linux, and you want to be
"cross-platform".  Might I suggest you get the other OSs that you want
to support to also support what Linux already does, instead of trying to
duplicate work?

Oh, what oses are you wanting to support?  I think FreeBSD already
handles much of what Linux does, with it's devfs implementation, but I
haven't looked at that in a long time to verify this.

> Applications of the HAL (e.g. desktop environments) will then use
> standard kernel and OS interfaces to e.g. display the amount of free
> space on a Storage device. 
> 
> I hope this clear matters up?

Not really.  I'm getting the impression that you don't realize the
current state of Linux and how hotplugging and driver loading and device
naming currently works.  A lot of what you are saying you want to do,
already works today.

The only "major" piece that is missing, is what udev is working on
solving (the persistant device naming issue.)  Now I agree that udev and
the hotplug scripts, and the raw hotplug events should get sent using
dbus events, but I was thinking that userspace programs (like GNOME and
KDE) that cared about those events would be listening for them.  I
didn't think there needed to be a whole intermediate layer for these
messages to have to go through.  But hey, I'm probably missing something
big here :)

> > For example, the kernel already knows what USB devices are supported by
> > what drivers.  It exports that info to userspace in the modules.*map
> > files in /lib/modules/<KERNEL_VERSION>/
> > 
> > The existing hotplug scripts already handle the mapping from device to
> > driver, so you don't need to duplicate that functionaity.
> 
> You are right. Note that the HAL needs to be kernel / OS agnostic but
> the idea is to handle this at the device-info level. So, for Linux the
> BootProgram for a storage-device simply mounts the device somewhere
> chosen by the user or the desktop environment. On other OS'es the
> bootprogram may need to insert modules or whatever into a running
> kernel.
> 
> This implies that, for instance, /etc/hotplug/usb.hotplug will need to
> send a DBUS message *after* the kernel modules have been successfully
> loaded - see proposal below.

That's fine.

> Side question: If a device vendor ships a (kernel) driver for his USB
> device will modules.*map get updated?

Side question to your side question: What device vendor ships kernel
drivers for USB devices?  :)

Anyway, yes, it will get updated, next boot time when depmod is run.
Actually in 2.6, those files are not needed at all, and people activly
want to get rid of them.  And I agree with them, it's just that the
current hotplug scripts rely too much on them at this time...  They
should be rewritten as to not need them at all.

So be aware that they might go away in the future.

> > > > Also, D-BUS messages will be
> > > > created for all /sbin/hotplug events too.  You might want to work with
> > > > that if it's easier for you.
> > > > 
> > > 
> > > I'll be interested in this, definately. Are you using the service
> > > org.freedesktop.DBus.Broadcast for this? 
> > 
> > I'm not using anything :)
> > 
> > There was a simple command line program that generated dbus messages.
> > Someone needs to take that and have it convert hotplug messages into
> > dbus messages.  I think it's going to use the org.kernel namespace.
> > 
> 
> My initial experiments was actually to patch /etc/hotplug/usb.hotplug
> and used the same command-line tool. The tool is a bit weak still (but
> DBUS is not at 1.0 yet) in so far that obvious data-types like
> dictionaries are not available yet. Now I just have a hackish C program
> using libdbus to parse the environment given from the kernel and send it
> off to the HAL daemon.

Nice.

> IMHO, the best possible scenario for HAL (wrt. hotplugging) would be to
> have a well-defined DBUS message scheme for each bus type and
> linux-hotplug to implement this broadcasting it through the service
> org.freedesktop.DBus.Broadcast. 

Hm, why that address?  I thought the org.kernel namespace was going to
be reserved for these types of messages.  Ah, but the org.kernel name
should be used for "raw" hotplug events (which some people might want to
listen for).  Ok, I understand now.

> Data in these messages would contain the usb.* fields and information
> about what (linux) kernel modules was loaded if any (cannot find device
> driver etc.).

Why would loading a new driver really be of any interest to anyone?
Userspace can't do anything with that.  Userspace can only interact with
/dev nodes.

But I can understand that if no driver is found for a device, that would
be useful.  Also remember that not all USB devices use kernel drivers
(like all of the cameras that use libusb and gphoto2 to talk to devices,
there is no kernel driver in use there.)

> If there is not a well-defined scheme yet, I'll volunteer to write a
> spec and implement it as part of linux-hotplug. We can put this spec at
> either the HAL site or the linux-hotplug site. The former would IMHO be
> better since we want the scheme to be OS agnostic, but I'm flexible. I'm
> quite certain the scheme, to an extent, can be OS agnostic.
> 
> What is your take on this?

I think I need to see your spec first :)

> > > In the event this is not possible the hotplug agent must somehow convey
> > > this information so I can append a counter to make the ID unique in that
> > > way.. 
> > 
> > The kernel will tell you that this is a new device.  It's up to
> > userspace to do something with that info.  udev will determine the /dev
> > name for the device.  It's up to other programs to do other things with
> > this info (which is what I'm thinking your proposal does, correct?)
> > 
> 
> Ok, the HAL daemon can take care of this as long as we agree on the DBUS
> message format. HAL is strictly user-space.

So is udev.  And yes, I have no problem agreeing on the dbus message
format.

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Havoc Pennington]

On Thu, 2003-09-18 at 13:36, Greg KH wrote:
> But my main point is, why store this info anywhere, when it is already
> stored on your machine today? 

Keep in mind the original proposal at http://ometer.com/hardware.html
which is that the HAL _merges_ information from various sources into a
single list/tree of devices, and property set for each device.

So we may read some stuff from the USB hardware itself, then add some
info provided by the kernel, then add some metadata from some systemwide
files, then add some data that has been obtained by the desktop and
stored per-user, then look at some blacklist, and finally we have a
complete picture of everything known about that particular device.

But the exact set of things merged varies by device. The suggestion
someone else posted about having "interfaces" is interesting in this
respect.

One point of the HAL is to abstract where this information is coming
from. Each info source is part of the "backend" not the
application-visible API.

>  And it is automatically already handled
> properly today, without any need for "yet another abstraction layer" to
> handle it?

Things aren't handled properly today in the sense that there's no UI on
it, there's no integration with applications. To add that on the
desktop/app level, we first have to have a single API that will work
with different kinds of device and on different operating systems.

One principle to keep in mind is: application code should not know the
details of specific devices. If I add a new device model, apps should
automatically use it. Ideally, I can even add a new kind of bus and apps
can use devices on that bus; e.g. an app doesn't care if a scanner is
SCSI or USB.

From an OEM standpoint, if I submit a patch to one or a small
well-defined set of projects, then my device should get used by all the
interesting apps.

You may be lumping all userspace code together. Remember that the main
desktop apps are built on dozens of libraries, all with opaque
long-term-frozen ABIs that create additional abstraction layers.

> Yes, I know this only works on Linux, and you want to be
> "cross-platform".  Might I suggest you get the other OSs that you want
> to support to also support what Linux already does, instead of trying to
> duplicate work?

That doesn't help, unless the other OSs have _exactly_ the same
filenames, interfaces, executables, and all other details. If there's
_any_ difference between any interesting versions of Linux and any
interesting versions of any other kernel, then we have to have the
abstraction barrier. Apps can't contain #ifdef hell, it's not practical.
App authors know _nothing_ about any specific hardware models or buses,
and understand a single OS if that. They are on the level of "list the
cameras, get the pictures from each"

Interesting kernels at the moment seem to be Solaris, Linux, and *BSD
including Darwin primarily. I don't see coordinating exact interfaces
between those as practical.

If it is practical, it definitely involves writing a spec for what the
interfaces are, and a conformance suite, and committing to maintaining
spec conformance over time. I haven't seen anything like this for the
Linux stuff yet. ;-)

> Oh, what oses are you wanting to support?  I think FreeBSD already
> handles much of what Linux does, with it's devfs implementation, but I
> haven't looked at that in a long time to verify this.

Isn't devfs a device naming solution? HAL really has little or nothing
to do with device naming; from an app standpoint I don't think we care
at all what's in /dev other than being able to find out what to use in
there, see the "implementation goals" point #2 in
http://ometer.com/hardware.html

Think of it this way: it's a bug if we have a field in the UI like:
 Device Node:   [   /dev/foo  ] 
that the user has to fill in. Sometimes we might have to do it, but it's
a UI bug for the OS/desktop combo as a whole system. Instead the UI
should be:
  Cameras:  [x] Minolta XYZ [ ] Canon
where those may be on different buses/kernel subsystems.

>  Now I agree that udev and
> the hotplug scripts, and the raw hotplug events should get sent using
> dbus events, but I was thinking that userspace programs (like GNOME and
> KDE) that cared about those events would be listening for them.  I
> didn't think there needed to be a whole intermediate layer for these
> messages to have to go through.  But hey, I'm probably missing something
> big here :)

Possibly you're expecting apps to understand the specifics of each
kernel subsystem (USB, etc.) and have significant Linux-specific code.
That's not feasible from the GNOME/KDE/OpenOffice.org/Mozilla/etc.
standpoint.
To the user we display "here are your cameras"; and at some level there
has to be an abstraction layer that puts things on the level of
"cameras" not "USB devices" - that layer needs to be below the
application, so that all apps have the same list of cameras.

(Cameras are just an example of course.)

The motivation is partly portability, and partly simply shielding app
developers from having to have a deep understanding of hardware in order
to write the UI for it. App developers _won't_ have such an
understanding, so if they have to, not much UI will happen.

Another angle on the motivation is the OEM one, allowing hardware to
just drop in and work after I bought it at the store, at least when
possible (e.g. the kernel driver is a generic one).

Maybe the most important motivation is getting the UI right, which
requires us to know as much as we can about each device.

> Why would loading a new driver really be of any interest to anyone?
> Userspace can't do anything with that.  Userspace can only interact with
> /dev nodes.

If you have a device and you don't know which driver to use, the desktop
could ask the user which driver to use. Also, if I'm an OEM and I know
my device works with a standard Linux driver, but that some deployed
kernels won't know that, I could ideally ship the information in a
little file of some kind for the user to install.

> But I can understand that if no driver is found for a device, that would
> be useful.

Right.

>   Also remember that not all USB devices use kernel drivers
> (like all of the cameras that use libusb and gphoto2 to talk to devices,
> there is no kernel driver in use there.)

Right, this is another reason to have the HAL. Apps do not want to sort
out all this which-device-uses-what stuff. They want to get a list of
cameras, then pass a camera in to a camera API (e.g. gphoto) that
magically gets the photos. And nobody wants to know what kind of plug is
on the camera or who implemented the driver where. ;-)


Hopefully this mail helps clarify some of the goals. To use hardware
effectively, apps need to have a hard dependency on an ABI for doing so.
Hard deps must be portable and well-encapsulated/well-defined in terms
of the semantics/ABI exported. Below this abstraction barrier, the
various OS-specific stuff would be hooked in. So "HAL" essentially just
means "the project that contains the abstraction barrier, and chains out
to the per-OS per-device-model implementations" - my writeup has one
speculation on what this project should look like, but I won't pretend
to know for sure.

Havoc




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[David Zeuthen]

Greg,

Havoc really touched all your points much better than me, so I'm only
replying to a subset of them.

On Thu, 2003-09-18 at 19:36, Greg KH wrote:
> But my main point is, why store this info anywhere, when it is already
> stored on your machine today?  And it is automatically already handled
> properly today, without any need for "yet another abstraction layer" to
> handle it?
> 

The point of HAL is really to *complement* the information that is
available in, say, Linux kernel 2.6 with sysfs.

Applications care about devices with specific capabilities, not on which
bus they are on, and certainly not how to retrieve it from this or that
bus. They use device-category specific support libraries to do that.
 
The contention is to have properties for each device that links to where
bus-information information is. For example, in Linux Kernel 2.6 you
would have a OS.Linux.devpath property that points somewhere into /sys
for *every* device. 

The advantage of this: When you ask an established device-access library
to do something, you can give it a device-id and it does the job. It
simply looks at the OS.Linux.devpath property from the given device and
does it's magic. Nothing less, Nothing more and quite future proof.

> Yes, I know this only works on Linux, and you want to be
> "cross-platform".  Might I suggest you get the other OSs that you want
> to support to also support what Linux already does, instead of trying to
> duplicate work?
> 

This is not trying to duplicate work at all. It's simply shielding
applications from caring about bus information etc. by moving such
information into libraries. No application should know what USB,
IEEE1394 and so on is...

Also getting everyone to do and emulate what Linux does is not friendly
and it might slow down new and better interfaces both in future versions
of Linux and other places.

> Oh, what oses are you wanting to support?  I think FreeBSD already
> handles much of what Linux does, with it's devfs implementation, but I
> haven't looked at that in a long time to verify this.
> 
> > Applications of the HAL (e.g. desktop environments) will then use
> > standard kernel and OS interfaces to e.g. display the amount of free
> > space on a Storage device. 
> > 
> > I hope this clear matters up?
> 
> Not really.  I'm getting the impression that you don't realize the
> current state of Linux and how hotplugging and driver loading and device
> naming currently works.  A lot of what you are saying you want to do,
> already works today.
> 

Well, you are right that udev + sysfs is news to me... 

All, I have tried to do is to formulate requirements regarding hardware
from the point of view of desktop environments. I was inspired by
Havoc's paper.

I have tried to be careful to specify everything and initiating
discussion before just coding away. Which is why we are having this
conversation, so I got at least one of my goals right ;-) 

Oh, and I'm also new to open source development!

> The only "major" piece that is missing, is what udev is working on
> solving (the persistant device naming issue.)  Now I agree that udev and
> the hotplug scripts, and the raw hotplug events should get sent using
> dbus events, but I was thinking that userspace programs (like GNOME and
> KDE) that cared about those events would be listening for them.  I
> didn't think there needed to be a whole intermediate layer for these
> messages to have to go through.  But hey, I'm probably missing something
> big here :)
> 

I'm not sure, but these are the key points that HAL tries to solve.

 1. Make access to devices as simple as possible by shielding
    applications from bus-specific things by moving such specifics
    into libraries.

 2. Having a single device info file concept that OEM's ship with
    their devices. This device info file should cover most popular 
    distros and OS'es and versions thereof.
    It should specify what the device is and what the user needs
    to do in order to interact with it. In order to match a device
    do this it need specifics of the hardware like usb.idVendor etc.

 3. Provide some place where user-made settings are stored and persisted
    for a device. For instance UI-things like device name, where to
    mount a storage device and with what options (storage only readable
    by the user or not etc.). 

<rambling>
In fact, things that pop on in the device-list managed by the HAL could
be devices managed by other hosts (USB ports on your remote display,
your fridge, a remote webserver etc.)..
</rambling>

> So is udev.  And yes, I have no problem agreeing on the dbus message
> format.
> 

That sounds good. 

Regards,
David




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Joerg Sommer]

begin  Greg KH <greg@kroah.com> wrote:
> Yes it will.  See my OLS paper from this year for lots more information
> about it.

Nice to meet you here. :) I read this paper some time ago and one
question stayed open: How does udev handle device requests? devfs has the
nice feature to tell modprobe that a device is requested. But I don't
found any similar in your paper. How gets udev informed that a program
wants to open /dev/foobar?

Bye, Jörg.

end.
-- 
NetBSD ist für Frauen: es läuft auf Waschmaschinen


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Fri, Sep 19, 2003 at 08:11:14PM +0000, Joerg Sommer wrote:
> begin  Greg KH <greg@kroah.com> wrote:
> > Yes it will.  See my OLS paper from this year for lots more information
> > about it.
> 
> Nice to meet you here. :) I read this paper some time ago and one
> question stayed open: How does udev handle device requests? devfs has the
> nice feature to tell modprobe that a device is requested. But I don't
> found any similar in your paper. How gets udev informed that a program
> wants to open /dev/foobar?

It doesn't :)

If you _really_ need that kind of functionality, then you will have to
use devfs.

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Thu, Sep 18, 2003 at 02:30:40PM -0400, Havoc Pennington wrote:
> On Thu, 2003-09-18 at 13:36, Greg KH wrote:
> > But my main point is, why store this info anywhere, when it is already
> > stored on your machine today? 
> 
> Keep in mind the original proposal at http://ometer.com/hardware.html
> which is that the HAL _merges_ information from various sources into a
> single list/tree of devices, and property set for each device.
> 
> So we may read some stuff from the USB hardware itself, then add some
> info provided by the kernel, then add some metadata from some systemwide
> files, then add some data that has been obtained by the desktop and
> stored per-user, then look at some blacklist, and finally we have a
> complete picture of everything known about that particular device.
> 
> But the exact set of things merged varies by device. The suggestion
> someone else posted about having "interfaces" is interesting in this
> respect.

Heh, watch out, that word is just too overloaded these days :)

For USB devices, it makes sense to do something like an "interface" as
USB devices can be compound devices with different drivers binding to
different "usb interfaces".  A USB speaker is the easiest example, the
usb audio driver binds to the audio portion of the device, and the hid
driver binds to the buttons part of the device.

Anyway, yes, you are right, if you want to merge from lots of different
sources, that is a good thing.

> One point of the HAL is to abstract where this information is coming
> from. Each info source is part of the "backend" not the
> application-visible API.

Ok, makes sense.

> >  And it is automatically already handled
> > properly today, without any need for "yet another abstraction layer" to
> > handle it?
> 
> Things aren't handled properly today in the sense that there's no UI on
> it, there's no integration with applications. To add that on the
> desktop/app level, we first have to have a single API that will work
> with different kinds of device and on different operating systems.

Hm, for a lot of this a UI isn't needed, right?  The user doesn't need
to know that a driver was just automatically loaded for their device,
correct?

Now I agree, that if something goes wrong (no driver found), a UI is a
very good thing to have.  I just don't want to see you all duplicate
work that has already been done :)

> One principle to keep in mind is: application code should not know the
> details of specific devices. If I add a new device model, apps should
> automatically use it. Ideally, I can even add a new kind of bus and apps
> can use devices on that bus; e.g. an app doesn't care if a scanner is
> SCSI or USB.

I completely agree.  That's one big reason why most USB devices use
existing kernel interfaces (SCSI, V4L, serial ports, etc.)

> From an OEM standpoint, if I submit a patch to one or a small
> well-defined set of projects, then my device should get used by all the
> interesting apps.
> 
> You may be lumping all userspace code together. Remember that the main
> desktop apps are built on dozens of libraries, all with opaque
> long-term-frozen ABIs that create additional abstraction layers.

I probably am, sorry.  Partly because I know what's going on down low in
the kernel, and don't pay attention to userspace much at all, "it's just
a test load for the kernel" is how most kernel developers feel :)

But remember, most OEMs do not write kernel drivers.  There are a few
exceptions, thankfully.  And that is what is required for almost all
devices to get working with linux today.

Again, there are very notable exceptions, the digital cameras supported
by gphoto2 and the USB scanners supported by xsane, both using userspace
only code.  I'm just pretty skeptical that something can be created that
would allow OEMs to plop into a userspace location that would enable
device support across multiple operating systems.  But please, prove me
wrong, I will be very happy.


> > Yes, I know this only works on Linux, and you want to be
> > "cross-platform".  Might I suggest you get the other OSs that you want
> > to support to also support what Linux already does, instead of trying to
> > duplicate work?
> 
> That doesn't help, unless the other OSs have _exactly_ the same
> filenames, interfaces, executables, and all other details. If there's
> _any_ difference between any interesting versions of Linux and any
> interesting versions of any other kernel, then we have to have the
> abstraction barrier. Apps can't contain #ifdef hell, it's not practical.
> App authors know _nothing_ about any specific hardware models or buses,
> and understand a single OS if that. They are on the level of "list the
> cameras, get the pictures from each"

I understand this.  The kernel doesn't contain #ifdef hell for that very
reason.  But this is my main point as to how this might get very tough
for having cross platform support.  The BSDs support lots of USB devices
today in a very different manner than Linux does.

> Interesting kernels at the moment seem to be Solaris, Linux, and *BSD
> including Darwin primarily. I don't see coordinating exact interfaces
> between those as practical.
> 
> If it is practical, it definitely involves writing a spec for what the
> interfaces are, and a conformance suite, and committing to maintaining
> spec conformance over time. I haven't seen anything like this for the
> Linux stuff yet. ;-)

I haven't either, but I would be interested in seeing such a thing.

> > Oh, what oses are you wanting to support?  I think FreeBSD already
> > handles much of what Linux does, with it's devfs implementation, but I
> > haven't looked at that in a long time to verify this.
> 
> Isn't devfs a device naming solution?

Yes it is.

> HAL really has little or nothing to do with device naming; from an app
> standpoint I don't think we care at all what's in /dev other than
> being able to find out what to use in there, see the "implementation
> goals" point #2 in http://ometer.com/hardware.html

Ok, David mentioned device naming, and I pointed out udev.  udev's goal
is to solve your point #2 for Linux.  Yes, it would be great to move
udev configuration into your HAL so that it can be configured in a
common manner.  I'll help out with this in any way possible.

> Think of it this way: it's a bug if we have a field in the UI like:
>  Device Node:   [   /dev/foo  ] 
> that the user has to fill in. Sometimes we might have to do it, but it's
> a UI bug for the OS/desktop combo as a whole system. Instead the UI
> should be:
>   Cameras:  [x] Minolta XYZ [ ] Canon
> where those may be on different buses/kernel subsystems.

That would be very nice.  Hopefully udev can help provide some of the
underlying glue that you need to determine this kind of information for
Linux.  It will contain a device database that you will be able to query
to get this kind of information.

> >  Now I agree that udev and
> > the hotplug scripts, and the raw hotplug events should get sent using
> > dbus events, but I was thinking that userspace programs (like GNOME and
> > KDE) that cared about those events would be listening for them.  I
> > didn't think there needed to be a whole intermediate layer for these
> > messages to have to go through.  But hey, I'm probably missing something
> > big here :)
> 
> Possibly you're expecting apps to understand the specifics of each
> kernel subsystem (USB, etc.) and have significant Linux-specific code.
> That's not feasible from the GNOME/KDE/OpenOffice.org/Mozilla/etc.
> standpoint.

Hm, yeah, sorry.  Again, I live too low in the OS stack :)

> To the user we display "here are your cameras"; and at some level there
> has to be an abstraction layer that puts things on the level of
> "cameras" not "USB devices" - that layer needs to be below the
> application, so that all apps have the same list of cameras.
> 
> (Cameras are just an example of course.)

And one of the most difficult examples, as previously mentioned (some
use gphoto2, some show up as scsi devices and must be mounted, some
others are not even supported at all...)

> > Why would loading a new driver really be of any interest to anyone?
> > Userspace can't do anything with that.  Userspace can only interact with
> > /dev nodes.
> 
> If you have a device and you don't know which driver to use, the desktop
> could ask the user which driver to use. Also, if I'm an OEM and I know
> my device works with a standard Linux driver, but that some deployed
> kernels won't know that, I could ideally ship the information in a
> little file of some kind for the user to install.

But if the driver was successfully loaded and is working, the user should
not care at all, right?

> Hopefully this mail helps clarify some of the goals. To use hardware
> effectively, apps need to have a hard dependency on an ABI for doing so.
> Hard deps must be portable and well-encapsulated/well-defined in terms
> of the semantics/ABI exported. Below this abstraction barrier, the
> various OS-specific stuff would be hooked in. So "HAL" essentially just
> means "the project that contains the abstraction barrier, and chains out
> to the per-OS per-device-model implementations" - my writeup has one
> speculation on what this project should look like, but I won't pretend
> to know for sure.

Yes, it does help a lot.

I'm happy to see this work happening, and if there's anything that you
need from the kernel, hotplug scripts, or udev that I can help out with,
please let me know.

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Thu, Sep 18, 2003 at 10:35:45PM +0200, David Zeuthen wrote:
> 
> Greg,
> 
> Havoc really touched all your points much better than me, so I'm only
> replying to a subset of them.

Ok, I just responded back to his message.

> On Thu, 2003-09-18 at 19:36, Greg KH wrote:
> > But my main point is, why store this info anywhere, when it is already
> > stored on your machine today?  And it is automatically already handled
> > properly today, without any need for "yet another abstraction layer" to
> > handle it?
> > 
> 
> The point of HAL is really to *complement* the information that is
> available in, say, Linux kernel 2.6 with sysfs.
> 
> Applications care about devices with specific capabilities, not on which
> bus they are on, and certainly not how to retrieve it from this or that
> bus. They use device-category specific support libraries to do that.

Agreed.

> The contention is to have properties for each device that links to where
> bus-information information is. For example, in Linux Kernel 2.6 you
> would have a OS.Linux.devpath property that points somewhere into /sys
> for *every* device. 

Makes sense.  udev will be able to provide this to you if you so desire.

> > Not really.  I'm getting the impression that you don't realize the
> > current state of Linux and how hotplugging and driver loading and device
> > naming currently works.  A lot of what you are saying you want to do,
> > already works today.
> > 
> 
> Well, you are right that udev + sysfs is news to me... 

Glad you found it.  Hopefully it will make your life easier for your
Linux implementation.

> All, I have tried to do is to formulate requirements regarding hardware
> from the point of view of desktop environments. I was inspired by
> Havoc's paper.
> 
> I have tried to be careful to specify everything and initiating
> discussion before just coding away. Which is why we are having this
> conversation, so I got at least one of my goals right ;-) 
> 
> Oh, and I'm also new to open source development!

Well, welcome.  Looking forward to a working implementation :)

> > The only "major" piece that is missing, is what udev is working on
> > solving (the persistant device naming issue.)  Now I agree that udev and
> > the hotplug scripts, and the raw hotplug events should get sent using
> > dbus events, but I was thinking that userspace programs (like GNOME and
> > KDE) that cared about those events would be listening for them.  I
> > didn't think there needed to be a whole intermediate layer for these
> > messages to have to go through.  But hey, I'm probably missing something
> > big here :)
> > 
> 
> I'm not sure, but these are the key points that HAL tries to solve.
> 
>  1. Make access to devices as simple as possible by shielding
>     applications from bus-specific things by moving such specifics
>     into libraries.

This seems doable.

>  2. Having a single device info file concept that OEM's ship with
>     their devices. This device info file should cover most popular 
>     distros and OS'es and versions thereof.
>     It should specify what the device is and what the user needs
>     to do in order to interact with it. In order to match a device
>     do this it need specifics of the hardware like usb.idVendor etc.

This might be much more difficult due to the ways different OSs
implement different device support.  But hey, it's good to have dreams.

>  3. Provide some place where user-made settings are stored and persisted
>     for a device. For instance UI-things like device name, where to
>     mount a storage device and with what options (storage only readable
>     by the user or not etc.). 

Some of that will be taken care of in udev for Linux.  But others (like
where to mount the device) would be up to you.

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Joerg Sommer]

begin  Greg KH <greg@kroah.com> wrote:
> On Fri, Sep 19, 2003 at 08:11:14PM +0000, Joerg Sommer wrote:
>> begin  Greg KH <greg@kroah.com> wrote:
>> > Yes it will.  See my OLS paper from this year for lots more information
>> > about it.
>> 
>> Nice to meet you here. :) I read this paper some time ago and one
>> question stayed open: How does udev handle device requests? devfs has the
>> nice feature to tell modprobe that a device is requested. But I don't
>> found any similar in your paper. How gets udev informed that a program
>> wants to open /dev/foobar?
>
> It doesn't :)
>
> If you _really_ need that kind of functionality, then you will have to
> use devfs.

Why you removed this? IMO it is a very useful feature to only load a
module if it is requested. Do you really think this is not needed?

Jörg.
end.
-- 
"Computer games don't affect kids. If Pacman would have affected us as
children, we would now run around in darkened rooms, munching pills and
listening to repetetive music."


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Sat, Sep 20, 2003 at 07:31:26PM +0000, Joerg Sommer wrote:
> begin  Greg KH <greg@kroah.com> wrote:
> > On Fri, Sep 19, 2003 at 08:11:14PM +0000, Joerg Sommer wrote:
> >> begin  Greg KH <greg@kroah.com> wrote:
> >> > Yes it will.  See my OLS paper from this year for lots more information
> >> > about it.
> >> 
> >> Nice to meet you here. :) I read this paper some time ago and one
> >> question stayed open: How does udev handle device requests? devfs has the
> >> nice feature to tell modprobe that a device is requested. But I don't
> >> found any similar in your paper. How gets udev informed that a program
> >> wants to open /dev/foobar?
> >
> > It doesn't :)
> >
> > If you _really_ need that kind of functionality, then you will have to
> > use devfs.
> 
> Why you removed this? IMO it is a very useful feature to only load a
> module if it is requested. Do you really think this is not needed?

I have not removed anything, this is just something that can not be done
from userspace alone (which is what udev is.)

And yes, I really do not think this is needed, as things have moved from
being a "load when we open the dev node" type system, to a "load when we
have found the hardware" type system.

But hey, if you really need this functionality, then use devfs.  I'm not
saying that udev can duplicate all of devfs's functionality, just the
majority of it (this is really the only exception that I can think of
right now.)

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[David Zeuthen]

On Sat, 2003-09-20 at 02:17, Greg KH wrote:
> Well, welcome.  Looking forward to a working implementation :)
> 

Thanks! I hope not to be too far away from some proof of concept
hot-pluggable usb storage. Only hotplug and device info stuff is
missing.. All the other stuff seem stable enough now.. (stable, but not
exactly production quality yet)

> >  2. Having a single device info file concept that OEM's ship with
> >     their devices. This device info file should cover most popular 
> >     distros and OS'es and versions thereof.
> >     It should specify what the device is and what the user needs
> >     to do in order to interact with it. In order to match a device
> >     do this it need specifics of the hardware like usb.idVendor etc.
> 
> This might be much more difficult due to the ways different OSs
> implement different device support.  But hey, it's good to have dreams.
> 

It sure is :-)

In fact, I think it need not be too difficult - remember the HAL I'm
working on is essentially just a database of information (and links to
information) plus some well-defined properties that makes sense for many
OS'es. Plus some rules on what happens when properties change due to
external events such as hot-plugging or desktop application interaction.

For, e.g., a storage device to be usable, some existing software below
my HAL would have to mount into the file system.. We even have to assume
a file system that you can mount stuff into. I just want a device info
file format that allow people to do this in different ways for different
OS'es by specifying how and where to mount it in an OS-neutral way.

(assuming we also do device interfaces, similar nice stories for other
categories of devices can be made)

Plus, having device info files, the OEM can tweak, he can put in
properties about addresses for tech support, logo to display etc. This
will make us appear a little bit more OEM friendly. (I know that Windows
support this)

The big question that remains, is really who's life we are going to make
easier or harder? the app developers, the users, the kernel people, the
OEM's or the device library people? I don't think necessarily everyone
can win.. And we need adoption from most, if not all, of these people..

So, yeah, it's going to be very difficult. (I'm done rambling now :-))

> >  3. Provide some place where user-made settings are stored and persisted
> >     for a device. For instance UI-things like device name, where to
> >     mount a storage device and with what options (storage only readable
> >     by the user or not etc.). 
> 
> Some of that will be taken care of in udev for Linux.  But others (like
> where to mount the device) would be up to you.
> 

Yes. I did install 2.6.0-test5 over the weekend and played around with
udev 0.2, /sys and libsysfs + tools. It seems udev can't handle USB
storage devices yet? It didn't worked for me at least.

Side question: Is it really true that the only way to obtain where the
SCSI subsystem mounts the usb-storage is by inspecting /var/log/messages
on 2.4? I couldn't find anything on 2.5/2.6 through googling either...
It's very frustrating

Further, and this is because I known little or nothing of Linux kernel
programming, /sys seems a little intimidating (but very cool - I looked
at it in awe for over an hour).

So, what is your opinion on this: I plan, for some devices, to have
well-defined properties (that makes sense on other OS'es also) that is
dynamically retrieved from well-known sources such as /sys or /proc. 

An obvious example is the CPU-frequency for a CPU device - ya know,
stuff that desktop application programmers like to use but a) is quite
difficult and a pain to obtain (parsing /proc/cpuinfo); and b) varies
over a lot of OS'es. So in this example, when the laptop user
disconnects power and his CPU slows down, the property
CPU.ClockFrequency is automatically updated and the desktop application
will be notified. Another example might be free space for storage
devices.
(Just to stress it, the keyword here is linking rather than duplication,
which I why I'd like your opinion.)

More on udev: how will it fit with the linux-hotplug project?

Maybe it's already there, but it could be nice to have a set of
prioritized hotplug scripts that is invoked a'la /etc/init.d.. So one
for loading a kernel module, udev for naming the device and one for
sending a D-BUS message, notifying, among others, the HAL. In the chain
of invoking these, the environment would grow. 

While we're on the topic, I'll look into the D-BUS messages I'd like
from hot-plugging. Will send a proposal one of these days. Since D-BUS
is a moving target, we'll probably have to change it at some point
though.

Thanks,
David




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Greg KH]

On Sun, Sep 21, 2003 at 10:56:26PM +0200, David Zeuthen wrote:
> 
> The big question that remains, is really who's life we are going to make
> easier or harder? the app developers, the users, the kernel people, the
> OEM's or the device library people? I don't think necessarily everyone
> can win.. And we need adoption from most, if not all, of these people..

The users are the most important ones here.

> > >  3. Provide some place where user-made settings are stored and persisted
> > >     for a device. For instance UI-things like device name, where to
> > >     mount a storage device and with what options (storage only readable
> > >     by the user or not etc.). 
> > 
> > Some of that will be taken care of in udev for Linux.  But others (like
> > where to mount the device) would be up to you.
> > 
> 
> Yes. I did install 2.6.0-test5 over the weekend and played around with
> udev 0.2, /sys and libsysfs + tools. It seems udev can't handle USB
> storage devices yet? It didn't worked for me at least.

Should work just fine.  They show up as block devices, just like any
other disk (look in /sys/block).  My demo at OLS 2003 used my usb camera
which is a usb storage device.

> Side question: Is it really true that the only way to obtain where the
> SCSI subsystem mounts the usb-storage is by inspecting /var/log/messages
> on 2.4? I couldn't find anything on 2.5/2.6 through googling either...
> It's very frustrating

Don't remember, sorry.  There are a bunch of different scsi programs for
doing this.  Search the linux-scsi archives for links to them.

> Further, and this is because I known little or nothing of Linux kernel
> programming, /sys seems a little intimidating (but very cool - I looked
> at it in awe for over an hour).

Heh, thanks.  It's pretty impressive to finally see all of those links
that were always internal to the kernel.

> So, what is your opinion on this: I plan, for some devices, to have
> well-defined properties (that makes sense on other OS'es also) that is
> dynamically retrieved from well-known sources such as /sys or /proc. 
> 
> An obvious example is the CPU-frequency for a CPU device - ya know,
> stuff that desktop application programmers like to use but a) is quite
> difficult and a pain to obtain (parsing /proc/cpuinfo); and b) varies
> over a lot of OS'es. So in this example, when the laptop user
> disconnects power and his CPU slows down, the property
> CPU.ClockFrequency is automatically updated and the desktop application
> will be notified. Another example might be free space for storage
> devices.
> (Just to stress it, the keyword here is linking rather than duplication,
> which I why I'd like your opinion.)

I think there's already a user library to get the cpu frequency stuff
out of /proc or /sys to userspace.  You might want to look into that.

But yes, unifying this for user apps would be a good thing.

> More on udev: how will it fit with the linux-hotplug project?

It uses the hotplug-base package, that's all.

> Maybe it's already there, but it could be nice to have a set of
> prioritized hotplug scripts that is invoked a'la /etc/init.d.. So one
> for loading a kernel module, udev for naming the device and one for
> sending a D-BUS message, notifying, among others, the HAL. In the chain
> of invoking these, the environment would grow. 

Heh, we already have that.  See the latest release of the linux-hotplug
package :)

thanks,

greg k-h


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[David Zeuthen]

On Wed, 2003-09-24 at 23:08, Greg KH wrote:
> > > >  3. Provide some place where user-made settings are stored and persisted
> > > >     for a device. For instance UI-things like device name, where to
> > > >     mount a storage device and with what options (storage only readable
> > > >     by the user or not etc.). 
> > > 
> > > Some of that will be taken care of in udev for Linux.  But others (like
> > > where to mount the device) would be up to you.
> > > 
> > 
> > Yes. I did install 2.6.0-test5 over the weekend and played around with
> > udev 0.2, /sys and libsysfs + tools. It seems udev can't handle USB
> > storage devices yet? It didn't worked for me at least.
> 
> Should work just fine.  They show up as block devices, just like any
> other disk (look in /sys/block).  My demo at OLS 2003 used my usb camera
> which is a usb storage device.
> 

My bad. I hadn't loaded the kernel module sd_mod. Works like a charm
with this module..

> > Side question: Is it really true that the only way to obtain where the
> > SCSI subsystem mounts the usb-storage is by inspecting /var/log/messages
> > on 2.4? I couldn't find anything on 2.5/2.6 through googling either...
> > It's very frustrating
> 
> Don't remember, sorry.  There are a bunch of different scsi programs for
> doing this.  Search the linux-scsi archives for links to them.
> 

Hmm yes.. I searched for a long time, but found this message

 http://www.dragoninc.on.ca/mail-archives/linux-usb/2002-06/0122.html

where you say this is not possible in 2.4.

> > More on udev: how will it fit with the linux-hotplug project?
> 
> It uses the hotplug-base package, that's all.
> 

Ah, I see.. You symlink udev to /etc/hotplug.d/default/udev.hotplug and
on hotplug ACTION­d, $1=block invocations query the sys/$DEVPATH/dev
file and do a mknod. 

I can see that it's a clever thing to have a (pluggable) naming database
and strategy, especially for those sites you mention with thousands of
disks.

Do you intend udev to send a hotplug message to notify that a new device
file have been created? Or is it only for kernel messages? The thinking
is that one could make a generic script that broadcast each and every
hotplug event. It could be as simple as

  dbus-send --dest=org.freedesktop.DBus.Broadcast org.kernel.Hotplug
  string:$1 string:"`env`"

to broadcast the hotplug event on the dbus system bus. Of course, in a
proper script one would test for /usr/bin/dbus-send and
/var/run/messagebus.pid.

This way, I wouldn't need my hal_hotplug program, but just listen for
org.kernel.Hotplug messages on the dbus system bus and still enjoy the
functionality of udev.

> > Maybe it's already there, but it could be nice to have a set of
> > prioritized hotplug scripts that is invoked a'la /etc/init.d.. So one
> > for loading a kernel module, udev for naming the device and one for
> > sending a D-BUS message, notifying, among others, the HAL. In the chain
> > of invoking these, the environment would grow. 
> 
> Heh, we already have that.  See the latest release of the linux-hotplug
> package :)
> 

OK. One thing though.. When I plug in my USB storage device I get a
number hotplug invocations on 2.6 (instead of only one on 2.4):

 o  usb, DEVPATH=/devices/pci0000:00/0000:00:10.0/0000:02:07.0/usb2/
                 2-1/2-1.3

 o  usb, DEVPATH=/devices/pci0000:00/0000:00:10.0/0000:02:07.0/
                 usb2/2-1/2-1.3/2-1.3:0

 o  scsi_host, DEVPATH=/class/scsi_host/host2

 o  scsi, DEVPATH=/devices/pci0000:00/0000:00:10.0/0000:02:07.0/
                  usb2/2-1/2-1.3/2-1.3:0/host2/2:0:0:0

 o  block, DEVPATH=/block/sda

 o  block, DEVPATH=/block/sda/sda1

 o  scsi_device, DEVPATH=/class/scsi_device/2:0:0:0


These are all handled by stuff loading modules and creating device files
(udev) which is all good. But when do I know when all messages for a
device is sent?

My understanding is that is doesn't make send to talk about hotplug
start and hotplug end messages (though it would be nice), so I guess
when receiving the second usb hotplug message I wait for a sec or two to
received the first block message (since $DEVPATH/device points to the
USB device). Or even better, a message from udev! (otherwise I would
have to create my own device file with mknod?)

Thanks,
David





-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Hardware Abstraction Layer

[Havoc Pennington]

Hi,

I just learned from a comment on gnomedesktop.org that I missed most of
this thread, it landed in the wrong mail folder or something. ;-)

On Fri, 2003-09-19 at 20:12, Greg KH wrote:
> > Things aren't handled properly today in the sense that there's no UI on
> > it, there's no integration with applications. To add that on the
> > desktop/app level, we first have to have a single API that will work
> > with different kinds of device and on different operating systems.
> 
> Hm, for a lot of this a UI isn't needed, right?  The user doesn't need
> to know that a driver was just automatically loaded for their device,
> correct?

That's right. The bulk of the UI should be about using devices rather
than getting them connected and working.

> Again, there are very notable exceptions, the digital cameras supported
> by gphoto2 and the USB scanners supported by xsane, both using userspace
> only code.  I'm just pretty skeptical that something can be created that
> would allow OEMs to plop into a userspace location that would enable
> device support across multiple operating systems.  But please, prove me
> wrong, I will be very happy.

Ah, I don't really mean that the OEM interface would be OS-independent;
either the file plopped would have to contain OS-specific portions, or
there would be a per-OS file. That's OK since we can expect an OEM to
deal with each OS independently, as they only have 1 device * N OSes.
It's the app developers that have to be insulated from M devices * N
OSes.

The OS independent portion is the part the app sees. Some properties
exported by the HAL are probably even OS-specific, but the point is
there's an OS-independent core/subset. (i.e. the idea is to let you
write portable code, not to keep you from punching down to OS details if
you want)

> I understand this.  The kernel doesn't contain #ifdef hell for that very
> reason.  But this is my main point as to how this might get very tough
> for having cross platform support.  The BSDs support lots of USB devices
> today in a very different manner than Linux does.

I think of it as "portability by subdirectory" ;-)

You have an API that apps talk to, and then some generic code, and then
a per-OS backend with a subdirectory for each backend:

 hal/
   include/
     hal/hal.h
   generic/
     hal.c
   linux/
     usb.c
   freebsd/
     usb.c

It isn't so much that the freebsd/ subdir is populated immediately, but
rather that when someone appears with the skills and motivation they can
add that subdir without having to rearchitect everything.

The desktop userspace doesn't have a list of required platforms so much
as the requirement that "if someone is actively maintaining a platform
they can keep that platform working"

> I'm happy to see this work happening, and if there's anything that you
> need from the kernel, hotplug scripts, or udev that I can help out with,
> please let me know.

I really appreciate your interest/help, btw.

I'm curious how many iterations of all this we'll go through before
we're happy with it. Wait, I've worked on software long enough to know
the answer is "infinite"... ;-)

Havoc




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel