Re: ANNOUNCE: User-space System Device Enumation (uSDE)

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[David Dodge]

Greg KH writes:
> On Wed, Oct 29, 2003 at 01:12:26PM +0800, Guo, Min wrote:
> > 2.For non-hotplug device
[...]
> >  uDEV:
> >          not deal with it
> 
> See Robert Love's very simple script to populate stuff from sysfs.  It
> can run from initscript just like SDE.  But in the end, udev will end up
> in initramfs and we will not need to do this.

So the intent is to have compiled-in drivers for already-attached
devices (framebuffer, system disks, loop, whatever) generate calls to
/sbin/hotplug within initramfs?

Mainly I'm asking because I did try putting a hotplug script into an
initramfs a few weeks ago (using -test7), and it didn't appear to be
invoked for e.g. the VESA framebuffer. So I want to make sure this is a
"future" capability and not something that should have worked :-)

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Greg KH]

On Thu, Oct 30, 2003 at 07:45:08PM -0500, David Dodge wrote:
> Greg KH writes:
> > On Wed, Oct 29, 2003 at 01:12:26PM +0800, Guo, Min wrote:
> > > 2.For non-hotplug device
> [...]
> > >  uDEV:
> > >          not deal with it
> > 
> > See Robert Love's very simple script to populate stuff from sysfs.  It
> > can run from initscript just like SDE.  But in the end, udev will end up
> > in initramfs and we will not need to do this.
> 
> So the intent is to have compiled-in drivers for already-attached
> devices (framebuffer, system disks, loop, whatever) generate calls to
> /sbin/hotplug within initramfs?
> 
> Mainly I'm asking because I did try putting a hotplug script into an
> initramfs a few weeks ago (using -test7), and it didn't appear to be
> invoked for e.g. the VESA framebuffer. So I want to make sure this is a
> "future" capability and not something that should have worked :-)

This is something that should have worked for you today, /sbin/hotplug
does get called during early boot, before init is started up.

thanks,

greg k-h

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Dave Dodge]

Greg KH writes: 
> On Thu, Oct 30, 2003 at 07:45:08PM -0500, David Dodge wrote:
[...]
> > Mainly I'm asking because I did try putting a hotplug script into an
> > initramfs a few weeks ago (using -test7), and it didn't appear to be
> > invoked for e.g. the VESA framebuffer. So I want to make sure this is a
> > "future" capability and not something that should have worked :-)
> 
> This is something that should have worked for you today, /sbin/hotplug
> does get called during early boot, before init is started up.

Okay, I'll keep working at it.  Unfortunately my main test system has
since had a major hardware failure and I haven't gotten a replacement
set up yet.

In the meantime, from a quick look at the kernel code I see this:

  - invocations of /sbin/hotplug normally go through call_usermodehelper.

  - kernel/kmod.c: call_usermodehelper does this prior to scheduling
    any work:

        if (!system_running)
                return -EBUSY;

  - init/main.c: system_running is not set non-zero until just
    prior to starting init. 

So it looks like calls to hotplug are dropped while compiled-in
drivers are initializing.  Or am I missing something obvious?  I
needed a one-line patch to be able to boot into initramfs; do I need
another to enable call_usermodehelper earlier?

RE: ANNOUNCE: User-space System Device Enumation (uSDE)

[Guo, Min]

Here I try to summary out some difference for uSDE and uDEV,any comments are welcome!
In my opinion, competition is good and user can choose one they like because both of them
are user-space applications with a little minor kernel changes,is that right?

1.For the IDE/SCSI/PCI hotplug devices. 
 uDEV:
	edit namedev.config manually to specify certain map,
	if slot change or device change, user can re-edit namedev.config
 uSDE
	record the id or slot at the first time, when move device to new slot or change to a new same type device,
	automatically persist the name.
	user can also specify map manually.

2.For non-hotplug device
 uDEV:
	   not deal with it
 uSDE
	   scan at boot time and also perform policy method. so when moved or replaced devcie when the machine is
down, the name can also be persisted.


3. for multipath device.
uDEV
	    not support it.
uSDE
	    automatically detect mulitpath device and create md device. support hot add a new path and remove a path.

4. ethernet
 uDEV
	    not deal with it 
 nameif
	   name interface based on MAC, 
 uSDE
	   can set map based on MAC, SLOT.
	   support both setting manually map and automatic processing
	   support hotplug, eg. when exchange two device, the name can also be exchanged automatically.
  . 
5.devfs simulation
  uDEV
	   No such function
  uSDE
	    provide devfs simulation method.

>From the above comparsion, we can see uSDE really have some advantages.As far maintaince,
I think that more codes don't mean lacking maintainability.

Thanks
Guo Min

> -----Original Message-----
> From: linux-hotplug-devel-admin@lists.sourceforge.net 
> [mailto:linux-hotplug-devel-admin@lists.sourceforge.net]On 
> Behalf Of Greg KH
> Sent: Wednesday, October 29, 2003 6:44 AM
> To: Steven Dake
> Cc: Lars Marowsky-Bree; Mark Bellon; 
> linux-raid@vger.kernel.org; linux-kernel@vger.kernel.org; 
> linux-hotplug-devel@lists.sourceforge.net
> Subject: Re: ANNOUNCE: User-space System Device Enumation (uSDE)
> 
> 
> On Tue, Oct 28, 2003 at 11:12:07AM -0700, Steven Dake wrote:
> > On Tue, 2003-10-28 at 04:00, Lars Marowsky-Bree wrote:
> > > On 2003-10-27T14:14:18,
> > >    Mark Bellon <mbellon@mvista.com> said:
> > > 
> > > > The uSDE and udev are simlar in some respects. The uSDE 
> allows for 
> > > > complete control of the policy handling a device - not 
> just its naming. 
> > > 
> > > Well, so could udev in theory, and I had this plan to 
> enhance it to do
> > > so for the specific case of multipathing one day in the 
> not too distant
> > > future (ie, before q1/04).
> > > 
> > > In as far as I can see, udev and uSDE really do not have 
> too different
> > > goals. Competition is good, but only if they explore 
> distinct approaches
> > > ;-)
> > > 
> > There are several distinct approaches which have been enumerated in
> > other mails.
> > 
> > Since this point has not been addressed, I'd like to focus 
> on the major
> > difference in philosophy.
> > 
> > SDE places all policy in the hands of the policy developer 
> in a seperate
> > policy program.  udev places the policies in the main 
> processing loop of
> > the system, effectively implementing whatever policy is 
> desired by the
> > udev maintainers.
> 
> What do you mean by "policy"?
> 
> If you are saying that SDE allows programmers the ability to write new
> programs to plug into your framework to create new types of policies,
> then I understand that.  Your loadable plugins look like they support
> that.  But they require a dynamic loader :)
> 
> What udev is doing is trying to provide a flexible policy 
> that will work
> for everyone, with a heirchy of rules that can be easily 
> controlled and
> changed by anyone who can operate a text editor (and soon to 
> be changed
> by GUI applications, like the HAL project).
> 
> I have not heard of any situation in which the current udev 
> set of rules
> do not work out for them.  And if you can think of one, can't it be
> covered by the CALLOUT rule?  For example, someone has sent me a small
> userspace program that works with the CALLOUT rule that handles
> multipath devices by talking to the dm code.  Now that's pretty
> flexible.
> 
> If you (or anyone else) thinks of something that the existing 
> udev rules
> do not handle, please let me know.  If it's too complex, then yes, the
> user should use write their own SDE plugin.  But remember, 
> 99.9% of the
> people out there just want the LSB device names, with possibly a
> persistent entry for their digital camera and USB joystick, which udev
> handles just fine today.
> 
> For people with 4000 real scsi disks, udev also works well.  
> That seems
> to cover the wide range of users.
> 
> > Without seperating policies from the core executive of device naming
> > system, the core of udev suffers from the same issues as 
> placing policy
> > in the kernel suffers.   Lack of maintainability, lack of 
> user-defined
> > functionality, bloat, etc.
> 
> Easy Separation?  Hm, in looking at udev, if you just replace 
> 2 .c files
> with your new naming scheme, everything works just fine.
> 
> And if you want to go down the path of accusations about lack of
> maintainability, bloat, etc. I will be glad to point people 
> at your tree
> and then they can see these kinds of numbers:
> 
> For SDE:
> $ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
>   57328 total
> 
> Wow, you build 18 shared libraries:
> $ find . -type f | grep '\.so' | wc
>      18      18     625
> 
> For udev:
> $ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
>   17632 total
> 
> And that includes all of klibc, which really is not fair for udev to
> calculate.  So let's just look at the udev code size:
> $ ls | egrep '[.c|.h]$' | xargs wc -l | tail -1
>    2613 total
> 
> And to be complete, let's add the totals of libsysfs and tdb, 
> but to be
> fair any udev developer never has to look into those files:
> 
> libsysfs is this big:
>    3798 total
> 
> And tdb is this big:
>    2679 total
> 
> So adding those numbers up we get these kinds of numbers for 
> size of the
> .c and .h files in the different projects:
> 	SDE:	57328 lines
> 	udev:	 9090 lines
> 
> That makes SDE over 6 times bigger in source code alone than all of
> udev (including tdb and libsysfs).
> 
> I can compare executable size too, if you really want to still claim
> that udev is suffering from "lack of maintainability and bloat" if you
> really want :)
> 
> Oh, any reason you all haven't shown a working uSDE system in public
> anywhere?
> 
> thanks,
> 
> greg k-h
> 
> p.s. yes, I know lines of code is a horrible metric, and 
> doesn't really
> mean squat.  I just want to point out the huge size difference between
> the current state of udev and SDE, with pretty much identical
> functionality from what I can tell.
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> Does SourceForge.net help you be more productive?  Does it
> help you create better code?   SHARE THE LOVE, and help us help
> YOU!  Click Here: http://sourceforge.net/donate/
> _______________________________________________
> Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
> Linux-hotplug-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
> 

RE: ANNOUNCE: User-space System Device Enumation (uSDE)

[Steven Dake]

On Tue, 2003-10-28 at 22:12, Guo, Min wrote:
> Here I try to summary out some difference for uSDE and uDEV,any comments are welcome!
> In my opinion, competition is good and user can choose one they like because both of them
> are user-space applications with a little minor kernel changes,is that right?
> 

No kernel changes necessary for either usde or udev; I believe any bug
fixes have been merged into 2.6.0-test6 or later.

> 1.For the IDE/SCSI/PCI hotplug devices. 
>  uDEV:
> 	edit namedev.config manually to specify certain map,
> 	if slot change or device change, user can re-edit namedev.config
>  uSDE
> 	record the id or slot at the first time, when move device to new slot or change to a new same type device,
> 	automatically persist the name.
> 	user can also specify map manually.
> 
> 2.For non-hotplug device
>  uDEV:
> 	   not deal with it
>  uSDE
> 	   scan at boot time and also perform policy method. so when moved or replaced devcie when the machine is
> down, the name can also be persisted.
> 
> 
> 3. for multipath device.
> uDEV
> 	    not support it.
> uSDE
> 	    automatically detect mulitpath device and create md device. support hot add a new path and remove a path.
> 
> 4. ethernet
>  uDEV
> 	    not deal with it 
>  nameif
> 	   name interface based on MAC, 
>  uSDE
> 	   can set map based on MAC, SLOT.
> 	   support both setting manually map and automatic processing
> 	   support hotplug, eg. when exchange two device, the name can also be exchanged automatically.
>   . 
> 5.devfs simulation
>   uDEV
> 	   No such function
>   uSDE
> 	    provide devfs simulation method.
> 
> >From the above comparsion, we can see uSDE really have some advantages.As far maintaince,
> I think that more codes don't mean lacking maintainability.
> 
> Thanks
> Guo Min
> 
> > -----Original Message-----
> > From: linux-hotplug-devel-admin@lists.sourceforge.net 
> > [mailto:linux-hotplug-devel-admin@lists.sourceforge.net]On 
> > Behalf Of Greg KH
> > Sent: Wednesday, October 29, 2003 6:44 AM
> > To: Steven Dake
> > Cc: Lars Marowsky-Bree; Mark Bellon; 
> > linux-raid@vger.kernel.org; linux-kernel@vger.kernel.org; 
> > linux-hotplug-devel@lists.sourceforge.net
> > Subject: Re: ANNOUNCE: User-space System Device Enumation (uSDE)
> > 
> > 
> > On Tue, Oct 28, 2003 at 11:12:07AM -0700, Steven Dake wrote:
> > > On Tue, 2003-10-28 at 04:00, Lars Marowsky-Bree wrote:
> > > > On 2003-10-27T14:14:18,
> > > >    Mark Bellon <mbellon@mvista.com> said:
> > > > 
> > > > > The uSDE and udev are simlar in some respects. The uSDE 
> > allows for 
> > > > > complete control of the policy handling a device - not 
> > just its naming. 
> > > > 
> > > > Well, so could udev in theory, and I had this plan to 
> > enhance it to do
> > > > so for the specific case of multipathing one day in the 
> > not too distant
> > > > future (ie, before q1/04).
> > > > 
> > > > In as far as I can see, udev and uSDE really do not have 
> > too different
> > > > goals. Competition is good, but only if they explore 
> > distinct approaches
> > > > ;-)
> > > > 
> > > There are several distinct approaches which have been enumerated in
> > > other mails.
> > > 
> > > Since this point has not been addressed, I'd like to focus 
> > on the major
> > > difference in philosophy.
> > > 
> > > SDE places all policy in the hands of the policy developer 
> > in a seperate
> > > policy program.  udev places the policies in the main 
> > processing loop of
> > > the system, effectively implementing whatever policy is 
> > desired by the
> > > udev maintainers.
> > 
> > What do you mean by "policy"?
> > 
> > If you are saying that SDE allows programmers the ability to write new
> > programs to plug into your framework to create new types of policies,
> > then I understand that.  Your loadable plugins look like they support
> > that.  But they require a dynamic loader :)
> > 
> > What udev is doing is trying to provide a flexible policy 
> > that will work
> > for everyone, with a heirchy of rules that can be easily 
> > controlled and
> > changed by anyone who can operate a text editor (and soon to 
> > be changed
> > by GUI applications, like the HAL project).
> > 
> > I have not heard of any situation in which the current udev 
> > set of rules
> > do not work out for them.  And if you can think of one, can't it be
> > covered by the CALLOUT rule?  For example, someone has sent me a small
> > userspace program that works with the CALLOUT rule that handles
> > multipath devices by talking to the dm code.  Now that's pretty
> > flexible.
> > 
> > If you (or anyone else) thinks of something that the existing 
> > udev rules
> > do not handle, please let me know.  If it's too complex, then yes, the
> > user should use write their own SDE plugin.  But remember, 
> > 99.9% of the
> > people out there just want the LSB device names, with possibly a
> > persistent entry for their digital camera and USB joystick, which udev
> > handles just fine today.
> > 
> > For people with 4000 real scsi disks, udev also works well.  
> > That seems
> > to cover the wide range of users.
> > 
> > > Without seperating policies from the core executive of device naming
> > > system, the core of udev suffers from the same issues as 
> > placing policy
> > > in the kernel suffers.   Lack of maintainability, lack of 
> > user-defined
> > > functionality, bloat, etc.
> > 
> > Easy Separation?  Hm, in looking at udev, if you just replace 
> > 2 .c files
> > with your new naming scheme, everything works just fine.
> > 
> > And if you want to go down the path of accusations about lack of
> > maintainability, bloat, etc. I will be glad to point people 
> > at your tree
> > and then they can see these kinds of numbers:
> > 
> > For SDE:
> > $ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
> >   57328 total
> > 
> > Wow, you build 18 shared libraries:
> > $ find . -type f | grep '\.so' | wc
> >      18      18     625
> > 
> > For udev:
> > $ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
> >   17632 total
> > 
> > And that includes all of klibc, which really is not fair for udev to
> > calculate.  So let's just look at the udev code size:
> > $ ls | egrep '[.c|.h]$' | xargs wc -l | tail -1
> >    2613 total
> > 
> > And to be complete, let's add the totals of libsysfs and tdb, 
> > but to be
> > fair any udev developer never has to look into those files:
> > 
> > libsysfs is this big:
> >    3798 total
> > 
> > And tdb is this big:
> >    2679 total
> > 
> > So adding those numbers up we get these kinds of numbers for 
> > size of the
> > .c and .h files in the different projects:
> > 	SDE:	57328 lines
> > 	udev:	 9090 lines
> > 
> > That makes SDE over 6 times bigger in source code alone than all of
> > udev (including tdb and libsysfs).
> > 
> > I can compare executable size too, if you really want to still claim
> > that udev is suffering from "lack of maintainability and bloat" if you
> > really want :)
> > 
> > Oh, any reason you all haven't shown a working uSDE system in public
> > anywhere?
> > 
> > thanks,
> > 
> > greg k-h
> > 
> > p.s. yes, I know lines of code is a horrible metric, and 
> > doesn't really
> > mean squat.  I just want to point out the huge size difference between
> > the current state of udev and SDE, with pretty much identical
> > functionality from what I can tell.
> > 
> > 
> > -------------------------------------------------------
> > This SF.net email is sponsored by: SF.net Giveback Program.
> > Does SourceForge.net help you be more productive?  Does it
> > help you create better code?   SHARE THE LOVE, and help us help
> > YOU!  Click Here: http://sourceforge.net/donate/
> > _______________________________________________
> > Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
> > Linux-hotplug-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
> > 
> 
> 

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Greg KH]

On Wed, Oct 29, 2003 at 01:12:26PM +0800, Guo, Min wrote:
> Here I try to summary out some difference for uSDE and uDEV,any comments are welcome!
> In my opinion, competition is good and user can choose one they like because both of them
> are user-space applications with a little minor kernel changes,is that right?
> 
> 1.For the IDE/SCSI/PCI hotplug devices. 
>  uDEV:

What's with the wierd intercaps of the udev name?

> 	edit namedev.config manually to specify certain map,
> 	if slot change or device change, user can re-edit namedev.config

Then use something that will not change if you move the device.  Like a
serial number, or other unique identifier.  It's not udev's fault that
you used a non-flexible rule :)

Oh, and the file is called udev.config.

>  uSDE
> 	record the id or slot at the first time, when move device to new slot or change to a new same type device,
> 	automatically persist the name.
> 	user can also specify map manually.
> 
> 2.For non-hotplug device

What do you mean by this?  Memory devices?

>  uDEV:
> 	   not deal with it

See Robert Love's very simple script to populate stuff from sysfs.  It
can run from initscript just like SDE.  But in the end, udev will end up
in initramfs and we will not need to do this.

>  uSDE
> 	   scan at boot time and also perform policy method. so when moved or replaced devcie when the machine is
> down, the name can also be persisted.
> 
> 
> 3. for multipath device.
> uDEV
> 	    not support it.

Not true.  The CALLOUT rule handles this just fine.  I have a small
userspace program here from someone else that handles multipath devices
through the CALLOUT rule.

In fact I think this shows the flexibility of udev.  If you come up with
some new kind of device, or subsystem, or way of determining that you
want to name a device, udev can run _any_ program to do this.  No
rebuilding the code, or creating a shared library.  Small simple
programs all talking together in a universal manner.  Hm, where have I
heard that design decision before....

> uSDE
> 	    automatically detect mulitpath device and create md device. support hot add a new path and remove a path.
> 
> 4. ethernet
>  uDEV
> 	    not deal with it 

And this is on purpose.  Why would we, when there are so many other
programs out there that do deal with network devices.   Remember, not
all network connections are ethernet, which shows the limitation of SDE
in not handling all of them (ppp, ipsec, usb network devices, isdn, atm,
wireless, etc.)

>  nameif
> 	   name interface based on MAC, 
>  uSDE
> 	   can set map based on MAC, SLOT.
> 	   support both setting manually map and automatic processing
> 	   support hotplug, eg. when exchange two device, the name can also be exchanged automatically.
>   . 
> 5.devfs simulation
>   uDEV
> 	   No such function

Huh?  All it needs is a single config file to be created.  As the
current installed base of devfs users can probably all fit into my
basement with room to spare, and no one is coming up with such a file
for udev yet, I don't think this is a real need yet :)

But again, no rebuilding needed, if such a config file shows up, udev
will do this just fine.

>   uSDE
> 	    provide devfs simulation method.
> 
> >From the above comparsion, we can see uSDE really have some advantages.As far maintaince,
> I think that more codes don't mean lacking maintainability.

But it is at least one indicator, correct?

> Thanks
> Guo Min

In the interest of full disclosure, Min is one of the SDE authors, and I
am one of the udev authors.

Min, maybe you can answer why Intel has spent effort on this project
instead of offering to help udev, which has been public for a long time
now?

thanks,

greg k-h

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Lars Marowsky-Bree]

On 2003-10-27T14:14:18,
   Mark Bellon <mbellon@mvista.com> said:

> The uSDE and udev are simlar in some respects. The uSDE allows for 
> complete control of the policy handling a device - not just its naming. 

Well, so could udev in theory, and I had this plan to enhance it to do
so for the specific case of multipathing one day in the not too distant
future (ie, before q1/04).

In as far as I can see, udev and uSDE really do not have too different
goals. Competition is good, but only if they explore distinct approaches
;-)

> >How does this integrate with DM, md, EVMS, LVM...?
> As devices appear in sysfs the uSDE reacts to them via their hotplug 
> events. The policy for each device handles any device issues including 
> dealing with any device nodes.  It is possible to track and maintain 
> multiported devices and automatically provide multipath devices nodes 
> for instance.

Yes, I know that, I was asking whether you had done any discussion with
the EVMS2 folks for example to have a policy plugin to interact with
EVMS2 accordingly and do the magic.



Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering	      \ ever tried. ever failed. no matter.
SUSE Labs			      | try again. fail again. fail better.
Research & Development, SUSE LINUX AG \ 	-- Samuel Beckett

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Steven Dake]

On Tue, 2003-10-28 at 04:00, Lars Marowsky-Bree wrote:
> On 2003-10-27T14:14:18,
>    Mark Bellon <mbellon@mvista.com> said:
> 
> > The uSDE and udev are simlar in some respects. The uSDE allows for 
> > complete control of the policy handling a device - not just its naming. 
> 
> Well, so could udev in theory, and I had this plan to enhance it to do
> so for the specific case of multipathing one day in the not too distant
> future (ie, before q1/04).
> 
> In as far as I can see, udev and uSDE really do not have too different
> goals. Competition is good, but only if they explore distinct approaches
> ;-)
> 
There are several distinct approaches which have been enumerated in
other mails.

Since this point has not been addressed, I'd like to focus on the major
difference in philosophy.

SDE places all policy in the hands of the policy developer in a seperate
policy program.  udev places the policies in the main processing loop of
the system, effectively implementing whatever policy is desired by the
udev maintainers.

Without seperating policies from the core executive of device naming
system, the core of udev suffers from the same issues as placing policy
in the kernel suffers.   Lack of maintainability, lack of user-defined
functionality, bloat, etc.

> > >How does this integrate with DM, md, EVMS, LVM...?
> > As devices appear in sysfs the uSDE reacts to them via their hotplug 
> > events. The policy for each device handles any device issues including 
> > dealing with any device nodes.  It is possible to track and maintain 
> > multiported devices and automatically provide multipath devices nodes 
> > for instance.
> 
> Yes, I know that, I was asking whether you had done any discussion with
> the EVMS2 folks for example to have a policy plugin to interact with
> EVMS2 accordingly and do the magic.
> 
> 
No but this is definately a good idea.
Thanks!
-steve
> 
> Sincerely,
>     Lars Marowsky-Brée <lmb@suse.de>

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Greg KH]

On Tue, Oct 28, 2003 at 11:12:07AM -0700, Steven Dake wrote:
> On Tue, 2003-10-28 at 04:00, Lars Marowsky-Bree wrote:
> > On 2003-10-27T14:14:18,
> >    Mark Bellon <mbellon@mvista.com> said:
> > 
> > > The uSDE and udev are simlar in some respects. The uSDE allows for 
> > > complete control of the policy handling a device - not just its naming. 
> > 
> > Well, so could udev in theory, and I had this plan to enhance it to do
> > so for the specific case of multipathing one day in the not too distant
> > future (ie, before q1/04).
> > 
> > In as far as I can see, udev and uSDE really do not have too different
> > goals. Competition is good, but only if they explore distinct approaches
> > ;-)
> > 
> There are several distinct approaches which have been enumerated in
> other mails.
> 
> Since this point has not been addressed, I'd like to focus on the major
> difference in philosophy.
> 
> SDE places all policy in the hands of the policy developer in a seperate
> policy program.  udev places the policies in the main processing loop of
> the system, effectively implementing whatever policy is desired by the
> udev maintainers.

What do you mean by "policy"?

If you are saying that SDE allows programmers the ability to write new
programs to plug into your framework to create new types of policies,
then I understand that.  Your loadable plugins look like they support
that.  But they require a dynamic loader :)

What udev is doing is trying to provide a flexible policy that will work
for everyone, with a heirchy of rules that can be easily controlled and
changed by anyone who can operate a text editor (and soon to be changed
by GUI applications, like the HAL project).

I have not heard of any situation in which the current udev set of rules
do not work out for them.  And if you can think of one, can't it be
covered by the CALLOUT rule?  For example, someone has sent me a small
userspace program that works with the CALLOUT rule that handles
multipath devices by talking to the dm code.  Now that's pretty
flexible.

If you (or anyone else) thinks of something that the existing udev rules
do not handle, please let me know.  If it's too complex, then yes, the
user should use write their own SDE plugin.  But remember, 99.9% of the
people out there just want the LSB device names, with possibly a
persistent entry for their digital camera and USB joystick, which udev
handles just fine today.

For people with 4000 real scsi disks, udev also works well.  That seems
to cover the wide range of users.

> Without seperating policies from the core executive of device naming
> system, the core of udev suffers from the same issues as placing policy
> in the kernel suffers.   Lack of maintainability, lack of user-defined
> functionality, bloat, etc.

Easy Separation?  Hm, in looking at udev, if you just replace 2 .c files
with your new naming scheme, everything works just fine.

And if you want to go down the path of accusations about lack of
maintainability, bloat, etc. I will be glad to point people at your tree
and then they can see these kinds of numbers:

For SDE:
$ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
  57328 total

Wow, you build 18 shared libraries:
$ find . -type f | grep '\.so' | wc
     18      18     625

For udev:
$ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
  17632 total

And that includes all of klibc, which really is not fair for udev to
calculate.  So let's just look at the udev code size:
$ ls | egrep '[.c|.h]$' | xargs wc -l | tail -1
   2613 total

And to be complete, let's add the totals of libsysfs and tdb, but to be
fair any udev developer never has to look into those files:

libsysfs is this big:
   3798 total

And tdb is this big:
   2679 total

So adding those numbers up we get these kinds of numbers for size of the
.c and .h files in the different projects:
	SDE:	57328 lines
	udev:	 9090 lines

That makes SDE over 6 times bigger in source code alone than all of
udev (including tdb and libsysfs).

I can compare executable size too, if you really want to still claim
that udev is suffering from "lack of maintainability and bloat" if you
really want :)

Oh, any reason you all haven't shown a working uSDE system in public
anywhere?

thanks,

greg k-h

p.s. yes, I know lines of code is a horrible metric, and doesn't really
mean squat.  I just want to point out the huge size difference between
the current state of udev and SDE, with pretty much identical
functionality from what I can tell.

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Andreas Jellinghaus]

On Tue, 28 Oct 2003 22:52:33 +0000, Greg KH wrote:
> For SDE:
> $ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
>   57328 total
...

> For udev:
> $ find . -type f | egrep '[.c|.h]$' | xargs wc -l | tail -1
>   17632 total

Here is a config file and a 41 lines shell script, that
will populate /udev with all devices found via /sys.
A version to add or remove only one file should be about
the same size.

---cat /etc/makedev.conf---
ttyS0 root dialout 0660
zero root root 0666
null root root 0666

---cat makedev---
#!/bin/sh

set -e

DEV=/udev
CONFIG=/etc/makedev.conf

cd $DEV

find /sys/class -name dev |while read A;
do
	B=`dirname $A`
	B=`basename $B`
	MM=`cat $A|tr ":" " "`
	mknod --mode=0600 $B c $MM
	if grep -q "$B " $CONFIG
	then
		USER=`grep "$B " $CONFIG|cut -d" " -f2`
		GROUP=`grep "$B " $CONFIG|cut -d" " -f3`
		MODE=`grep "$B " $CONFIG|cut -d" " -f4`
		chown $USER.$GROUP $B
		chmod $MODE $B
	fi
done

find /sys/block -name dev |while read A;
do
	B=`dirname $A`
	B=`basename $B`
	MM=`cat $A|tr ":" " "`
	mknod --mode=0600 $B b $MM
	if grep -q "$B " $CONFIG
	then
		USER=`grep "$B " $CONFIG|cut -d" " -f2`
		GROUP=`grep "$B " $CONFIG|cut -d" " -f3`
		MODE=`grep "$B " $CONFIG|cut -d" " -f4`
		chown $USER.$GROUP $B
		chmod $MODE $B
	fi
done
---cut---

So udev is 99% overhead?

sure, it's fast and small, and has lots of features that I don't need.

> 	SDE:	57328 lines
> 	udev:	 9090 lines
 shell script:     41 lines

> that udev is suffering from "lack of maintainability and bloat" if you
> really want :)

bloat. lots of bloat. what is that tdb database for?
filesystems are persistent. if you want to save space,
create a tar file :-) 

> p.s. yes, I know lines of code is a horrible metric, and doesn't really
> mean squat.  I just want to point out the huge size difference between
> the current state of udev and SDE, with pretty much identical
> functionality from what I can tell.

I agree. lines of codes is a horrible metric, and comparing a shell
script that uses many external commands to a c application with
everything build is makes absolutely no sense. but I wonder why
the off the shelf machine needs a c applications, if all those
external commands are installed anyway.

Regards, Andreas

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Daniel Stekloff]

On Wednesday 29 October 2003 06:30 am, Andreas Jellinghaus wrote:
> On Tue, 28 Oct 2003 22:52:33 +0000, Greg KH wrote:
[snip]
> > that udev is suffering from "lack of maintainability and bloat" if you
> > really want :)
>
> bloat. lots of bloat. what is that tdb database for?
> filesystems are persistent. if you want to save space,
> create a tar file :-)

[snip]


The tdb database is for storing current device information, udev needs to 
reference names to devices. The database also enables an api for applications 
to query what devices are on the system, their names, and their nodes. 

Using tdb has its advantages too; it's small, it's flexible, it's fast, it can 
be in memory or on disk, and it has locking for multiple accesses.

IMVHO - tdb isn't bloat.

Thanks,

Dan

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Andreas Jellinghaus]

On Wed, 2003-10-29 at 18:20, Daniel Stekloff wrote:
> The tdb database is for storing current device information, udev needs to 
> reference names to devices. The database also enables an api for applications 
> to query what devices are on the system, their names, and their nodes. 
> 
> Using tdb has its advantages too; it's small, it's flexible, it's fast, it can 
> be in memory or on disk, and it has locking for multiple accesses.
> 
> IMVHO - tdb isn't bloat.

Hi Dan,

thanks for your email.
I took a look at tdb. Upon adding devices, the DEVPATH is resolved via
config files etc. to a final /dev filename. That combination is stored
in tdb, and when the device is remove, the same resolution process is
not done, but the tdb is looked up to find the filename, and remove
the device. Is that right?

So the advantage would be resistance against config file changes - if
the nameing scheme is changed while a devices is added, the remove would
get the new name, and that way try to remove the wrong device.

Also this mechanism could be used to implement counting device names
like "disc/0", where the final name depends on the devices currently
available, so there is no static translation from devpath to the
filename.

I'd prefer the kernel giving up the old device names, and migrating
to counter names i.e. disc/0, cdrom/0, printer/0, etc. Those who
still want the old names could use /sys/ to determine the details
on the device, and that way create devices per the old naming schema.
That way tdb wouldn't be necessary for counting device names, at least
if sysfs still has the full information on the device while the hotplug
event runs. I guess that is not the case or not guaranteed?

Also I have to admit, if symlinks like "hpdeskjet" to some usb device
are configured in the config, the device is attached, and the config
is changed, then a remove event will not find the old symlink and
cannot remove it, without tdb.

But maybe like a coldplug / fulling an empty /dev, there should be
rerun command? I.e. like coldplug determine what device and symlinks
shold be in /dev, and the remove unnecessary, add missing, and modify
outdated entries (devices,files)? If that existed, configuration changes
wouldn't be a reason for udev to use tdb?

So why is tdb currently required? I only see the possibility to use
naming schemes like disc/0 as a reason, but that isn't implemented
in udev so far...

other than a theological discussion about needed or not, I guess nobody
will complain about it - even people with /dev on tmpfs and a readonly
/ will have a writable or tmpfs /var so they can live with it anyway.
but I'm still not sure, if it isn't unnecessary.

Regards, Andreas

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Daniel Stekloff]

On Sunday 02 November 2003 09:28 am, Andreas Jellinghaus wrote:
> On Wed, 2003-10-29 at 18:20, Daniel Stekloff wrote:
> > The tdb database is for storing current device information, udev needs to
> > reference names to devices. The database also enables an api for
> > applications to query what devices are on the system, their names, and
> > their nodes.
> >
> > Using tdb has its advantages too; it's small, it's flexible, it's fast,
> > it can be in memory or on disk, and it has locking for multiple accesses.
> >
> > IMVHO - tdb isn't bloat.
>
> Hi Dan,


Hi Andreas, 

Sorry for my late reply. 


> thanks for your email.
> I took a look at tdb. Upon adding devices, the DEVPATH is resolved via
> config files etc. to a final /dev filename. That combination is stored
> in tdb, and when the device is remove, the same resolution process is
> not done, but the tdb is looked up to find the filename, and remove
> the device. Is that right?


The device path and device name are stored in the tdb database. The path is 
used as the primary key, the unique identifier to locate the specific device. 
A device is added by path and then removed using the path. 


> So the advantage would be resistance against config file changes - if
> the nameing scheme is changed while a devices is added, the remove would
> get the new name, and that way try to remove the wrong device.


Adding and removing device from the tdb database is done based on the device's 
path and not the generated name. Changing the config file shouldn't impact 
currently loaded and configured devices. 



> Also this mechanism could be used to implement counting device names
> like "disc/0", where the final name depends on the devices currently
> available, so there is no static translation from devpath to the
> filename.
>
> I'd prefer the kernel giving up the old device names, and migrating
> to counter names i.e. disc/0, cdrom/0, printer/0, etc. Those who
> still want the old names could use /sys/ to determine the details
> on the device, and that way create devices per the old naming schema.
> That way tdb wouldn't be necessary for counting device names, at least
> if sysfs still has the full information on the device while the hotplug
> event runs. I guess that is not the case or not guaranteed?
>
> Also I have to admit, if symlinks like "hpdeskjet" to some usb device
> are configured in the config, the device is attached, and the config
> is changed, then a remove event will not find the old symlink and
> cannot remove it, without tdb.
>
> But maybe like a coldplug / fulling an empty /dev, there should be
> rerun command? I.e. like coldplug determine what device and symlinks
> shold be in /dev, and the remove unnecessary, add missing, and modify
> outdated entries (devices,files)? If that existed, configuration changes
> wouldn't be a reason for udev to use tdb?
>
> So why is tdb currently required? I only see the possibility to use
> naming schemes like disc/0 as a reason, but that isn't implemented
> in udev so far...
>
> other than a theological discussion about needed or not, I guess nobody
> will complain about it - even people with /dev on tmpfs and a readonly
> / will have a writable or tmpfs /var so they can live with it anyway.
> but I'm still not sure, if it isn't unnecessary.


Honestly, I'm not exactly sure what you're questioning - is tdb specifically 
unnecessary or is having a backing store unnecessary. 

It's true, you could do without a backing store. You could build the reference 
between device and name every time you wish to work with the device. In some 
cases that would be easy, such as simply using the kernel naming system. In 
other cases it might not be so easy, especially if you use a specific and 
complicated naming scheme. You'd have to pay the processing hit it would take 
to calculate the names every time you access a device. You'd also have to 
deal with issues like current devices and adjusting the config files, which 
you've already brought up. 

I believe having a backing store - whether tdb, gdbm, a table in memory, or a 
flat file - is useful for storing the configured devices and their names. 
It's cheap to submit a small query to find the necessary device, rather than 
having to go through the naming process every time a device is added, 
removed, or queried. Having the backing store removes the problem you 
mentioned about changing config files on the fly, there's no loose ends or 
missing or changed device names - they are stored in the database. I believe 
the trade off between backing store complexity and storage versus on the fly 
calculation is worth it. I can understand if you feel differently.

As for tdb specifically being unnecessary, you could most certainly use 
something else to store device information. You could use a flat file. You 
could use gdbm. You could use a simple table in memory. I chose tdb for the 
following reasons: 

- it is a proven mechanism. 
- it could handle thousands of devices.
- it is fast to query.
- it has a small footprint.

It seemed to me like a good solution for udev's backing store need. 

If you believe udev doesn't need the backing store and/or tdb, let's see your 
solution and we can decide on its technical merit by looking it over. I'm 
certainly open to new ideas and solutions. I'm sorry if you felt this was a 
"theological" discussion, I certainly don't feel the same way.

Cheers,

Dan

 


> Regards, Andreas

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Greg KH]

On Wed, Oct 29, 2003 at 03:30:30PM +0100, Andreas Jellinghaus wrote:
> ---cut---

Sweet shell script, nice job.

> So udev is 99% overhead?

To you, sure, it might be.  Don't use it then, I'm not forcing anyone.

> > 	SDE:	57328 lines
> > 	udev:	 9090 lines
>  shell script:     41 lines

Hm, how about the size of bash?

> > that udev is suffering from "lack of maintainability and bloat" if you
> > really want :)
> 
> bloat. lots of bloat. what is that tdb database for?
> filesystems are persistent. if you want to save space,
> create a tar file :-) 

Sweet, and then run everything on a in-ram compress filesystem just to
save that precious disk space.

> > p.s. yes, I know lines of code is a horrible metric, and doesn't really
> > mean squat.  I just want to point out the huge size difference between
> > the current state of udev and SDE, with pretty much identical
> > functionality from what I can tell.
> 
> I agree. lines of codes is a horrible metric, and comparing a shell
> script that uses many external commands to a c application with
> everything build is makes absolutely no sense. but I wonder why
> the off the shelf machine needs a c applications, if all those
> external commands are installed anyway.

Remember, userspace is just a load test for the kernel, who really needs
applications anyway.

</sarcasm>

greg k-h

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Andreas Jellinghaus]

Hi Greg!

On Wed, 2003-10-29 at 20:18, Greg KH wrote:
> Sweet shell script, nice job.
> 
> > So udev is 99% overhead?
> To you, sure, it might be.  Don't use it then, I'm not forcing anyone.

Thats right! So I don't understand all those discussions why some people
created an alternative to udev, as everyone is free to. All
implementations seem to be quit young, and I enjoy many different
approaches to solve these related problems. a bit of diversity or
evolution and natural selection doesn't hurt.

But, all these different user space tools share one thing: the same
kernel. the linux kernel has to work for all of us, and I even think
it is very important to have many different user space tools at the
current time, so everyone can voice their opinions and suggest how the
kernel should be changed, so it will work for him, and preferable for
all of us.

Now, I'd love if the kernel would keep the permissions that devfs
has. you don't. nobody else wrote something about it? but that issue
isn't that important.

I'd like to see a different namespace. I need some info on
block devices. currently there is no way to see if hdc is
a cdrom or a disc in sysfs, so that should be added.

how do you call that distinction cdrom/disc? type? class? I don't now.
I was under the impression, that this should be part of the path to
the dev file, i.e. cdroms should be in a cdrom/ directory and 
discs in a disc/ directory. would that be a major change?

the alternative would be an additional attribute to the kobject
with that information. 

I'm no kernel hacker, I can bareley understand some parts of the code
and do trivial changes. But I will try to learn more and maybe be
of some help.

Andreas

Re: ANNOUNCE: User-space System Device Enumation (uSDE)

[Greg KH]

On Wed, Oct 29, 2003 at 11:12:53PM +0100, Andreas Jellinghaus wrote:
> Hi Greg!
> 
> On Wed, 2003-10-29 at 20:18, Greg KH wrote:
> > Sweet shell script, nice job.
> > 
> > > So udev is 99% overhead?
> > To you, sure, it might be.  Don't use it then, I'm not forcing anyone.
> 
> Thats right! So I don't understand all those discussions why some people
> created an alternative to udev, as everyone is free to. All
> implementations seem to be quit young, and I enjoy many different
> approaches to solve these related problems. a bit of diversity or
> evolution and natural selection doesn't hurt.

I agree, evolution is good in this case.  I'm just really curious as to
why Monta Vista and Intel decided that udev wasn't an approiate project
to work with, and decided to spend time and resources to develop an
alternative one.  Just a bit frustrated, as udev really could have used
a few people working on it full time as SDE had.

But hey, that's all past, if those companies like to live with the NIH
symptom, I can't change that.  And udev development will go on
regardless.

> But, all these different user space tools share one thing: the same
> kernel. the linux kernel has to work for all of us, and I even think
> it is very important to have many different user space tools at the
> current time, so everyone can voice their opinions and suggest how the
> kernel should be changed, so it will work for him, and preferable for
> all of us.

Agreed.

> I'd like to see a different namespace.

Different namespace where?  In /sys?  Or somewhere else?

> I need some info on block devices. currently there is no way to see if
> hdc is a cdrom or a disc in sysfs, so that should be added.

Not true at all.  Just look at the device symlink, and the type file in
that directory.  Shows it quite well for all of my scsi devices.  Now it
looks like IDE needs to add that same type of file, but that's an IDE
issue.  Actually IDE's sysfs code probably could use some good cleanups,
now that I look at it some more.  Want to work on that?

thanks,

greg k-h