Non-hardware devices like loop: another idea

Non-hardware devices like loop: another idea

[Alexander E. Patrakov]

Recently I stated that I am working on solution to the "chicken-and-egg" 
problem related to devices like /dev/loop*, /dev/nbd*, /dev/ppp and so on, 
which require no hardware and therefore are not detectable by the hotplug 
scripts.

My (bad) idea was to include the sysfs entries that would appear after loading 
the module in the module itself by means of the MODULE_INFO macro, and then 
modify depmod to populate /lib/modules/<version>/sys and use it to 
populate /dev with the corresponding entries (the UDEV_TEST environment 
variable already allows us to override the location of the sysfs directory). 
Then the char-major magic will load these modules on demand.

I have a proof-of-concept implementation (will mail it here if there is any 
demand). The list of files altered:

module-init-tools-3.0-pre5/depmod.c
linux-2.6.4/include/linux/module.h
linux-2.6.4/drivers/block/loop.c
linux-2.6.4/drivers/char/rtc.c
linux-2.6.4/drivers/net/ppp_generic.c
...and my udev initscript

This is not a real, but a proof-of-concept implementation because there are 
much more such non-hardware modules. This works,

--- BUT! ---

I found my implementation rather clumsy and while implementing it I decided 
that it would be better to just mark such modules and modprobe tham all, and 
make the hotplug script modprobe other stuff. I decided to start implementing 
such mark... and found that it has already been done, so actually no patches 
are needed except for the udev initscript!

I am talking about /lib/modules/<version>/modules.alias.

The simple idea is to grep this file (and probably /etc/modprobe.conf also) 
for lines containing char-major and block-major. The list pretty much 
coincides with the list of non-hardware-related modules that nevertheless 
provide devices.

Does anyone care to include this into the future release of udev?

-- 
Alexander E. Patrakov


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Mike Snitzer]

What if you configure this options into the kernel, e.g.:

CONFIG_BLK_DEV_LOOP=y
CONFIG_TUN=y
... etc ...

rather than building all non-hardware backed devices as modules?  Relying
on them being modules seems quite restrictive.

Mike


On Fri, Mar 12 2004 at 08:03,
Alexander E. Patrakov <patrakov@ums.usu.ru> wrote:

> Recently I stated that I am working on solution to the "chicken-and-egg" 
> problem related to devices like /dev/loop*, /dev/nbd*, /dev/ppp and so on, 
> which require no hardware and therefore are not detectable by the hotplug 
> scripts.
> 
> My (bad) idea was to include the sysfs entries that would appear after loading 
> the module in the module itself by means of the MODULE_INFO macro, and then 
> modify depmod to populate /lib/modules/<version>/sys and use it to 
> populate /dev with the corresponding entries (the UDEV_TEST environment 
> variable already allows us to override the location of the sysfs directory). 
> Then the char-major magic will load these modules on demand.
> 
> I have a proof-of-concept implementation (will mail it here if there is any 
> demand). The list of files altered:
> 
> module-init-tools-3.0-pre5/depmod.c
> linux-2.6.4/include/linux/module.h
> linux-2.6.4/drivers/block/loop.c
> linux-2.6.4/drivers/char/rtc.c
> linux-2.6.4/drivers/net/ppp_generic.c
> ...and my udev initscript
> 
> This is not a real, but a proof-of-concept implementation because there are 
> much more such non-hardware modules. This works,
> 
> --- BUT! ---
> 
> I found my implementation rather clumsy and while implementing it I decided 
> that it would be better to just mark such modules and modprobe tham all, and 
> make the hotplug script modprobe other stuff. I decided to start implementing 
> such mark... and found that it has already been done, so actually no patches 
> are needed except for the udev initscript!
> 
> I am talking about /lib/modules/<version>/modules.alias.
> 
> The simple idea is to grep this file (and probably /etc/modprobe.conf also) 
> for lines containing char-major and block-major. The list pretty much 
> coincides with the list of non-hardware-related modules that nevertheless 
> provide devices.
> 
> Does anyone care to include this into the future release of udev?
> 
> -- 
> Alexander E. Patrakov
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
> _______________________________________________
> Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
> Linux-hotplug-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
> 




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Alexander E. Patrakov]

On Wednesday 17 March 2004 06:47, Mike Snitzer wrote:
> What if you configure this options into the kernel, e.g.:
>
> CONFIG_BLK_DEV_LOOP=y
> CONFIG_TUN=y
> ... etc ...
>
> rather than building all non-hardware backed devices as modules?  Relying
> on them being modules seems quite restrictive.
>
> Mike

This will also work. If you do so, then the corresponding entries will not 
appear in the "modules.alias" file.

Also, I don't completely understand you when you say that something relies 
upon these options being modules. Can you explain it more clear?

-- 
Alexander E. Patrakov


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Mike Snitzer]

On Tue, Mar 16 2004 at 21:13,
Alexander E. Patrakov <patrakov@ums.usu.ru> wrote:

> On Wednesday 17 March 2004 06:47, Mike Snitzer wrote:
> > What if you configure this options into the kernel, e.g.:
> >
> > CONFIG_BLK_DEV_LOOP=y
> > CONFIG_TUN=y
> > ... etc ...
> >
> > rather than building all non-hardware backed devices as modules?  Relying
> > on them being modules seems quite restrictive.
> >
> > Mike
> 
> This will also work. If you do so, then the corresponding entries will not 
> appear in the "modules.alias" file.

Right; they won't appear in modules.alias and therefore your solution
won't work.

> Also, I don't completely understand you when you say that something relies 
> upon these options being modules. Can you explain it more clear?

You were suggesting that modules.alias and modules.conf be parsed to
determine which non-hardware backward devices should get created by udev.
By the nature of these module related files; it implies those options be
configured as modules, no?

So I suppose what you're suggesting _could_ work (if modules) but its not
a complete solution.

Mike 




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Alexander E. Patrakov]

Mike Snitzer wrote:

>On Tue, Mar 16 2004 at 21:13,
>Alexander E. Patrakov <patrakov@ums.usu.ru> wrote:
>
>  
>
>>On Wednesday 17 March 2004 06:47, Mike Snitzer wrote:
>>    
>>
>>>What if you configure this options into the kernel, e.g.:
>>>
>>>CONFIG_BLK_DEV_LOOP=y
>>>CONFIG_TUN=y
>>>... etc ...
>>>
>>>rather than building all non-hardware backed devices as modules?  Relying
>>>on them being modules seems quite restrictive.
>>>
>>>Mike
>>>      
>>>
>>This will also work. If you do so, then the corresponding entries will not 
>>appear in the "modules.alias" file.
>>    
>>
>
>Right; they won't appear in modules.alias and therefore your solution
>won't work.
>
>  
>
>>Also, I don't completely understand you when you say that something relies 
>>upon these options being modules. Can you explain it more clear?
>>    
>>
>
>You were suggesting that modules.alias and modules.conf be parsed to
>determine which non-hardware backward devices should get created by udev.
>By the nature of these module related files; it implies those options be
>configured as modules, no?
>
>So I suppose what you're suggesting _could_ work (if modules) but its not
>a complete solution.
>  
>
Now I see where the confusion comes from. You misunderstood my proposal. 
Sorry for not making it clear enough.

I don't propose to parse these files at all. I want to probe the modules 
corresponding to the *-major-* aliases unconditionally. Let's review the 
scenarios for the loop module.

1) It is in the kernel. Then the /sys/block/loop0/dev file will be 
present upon boot, the udev initscript as supplied with the current 
version of udev will notice it while walking the /sys/block hierarchy, 
and /dev/loop0 will be created from the initscript.

2) It is a module. Then the modified initscript will notice that there 
is a block-major line in /lib/modules/<version>/modules.alias with the 
"loop" as its third field. You suggest to create /dev/loop0 based on 
this information. I suggest to just call "modprobe loop" instead. The 
kernel will send the hotplug event and the udev program as supplied in 
the current udev package will react upon that event by creating /dev/loop0.

As you can see, looking at modules.alias and modprobe.conf is just a 
good (?) heuristic that helps to decide which modules don't depend upon 
some specific hardware and therefore are not detectable by the current 
hotplug package.

The question mark above is because of the ide-cd module: it is not 
detectable by hotplug and doesn't have a standard alias in modules.alias.

-- 
Alexander E. Patrakov



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Mike Snitzer]

On Tue, Mar 16 2004 at 23:39,
Alexander E. Patrakov <patrakov@ums.usu.ru> wrote:

> Now I see where the confusion comes from. You misunderstood my proposal. 
> Sorry for not making it clear enough.
> 
> I don't propose to parse these files at all. I want to probe the modules 
> corresponding to the *-major-* aliases unconditionally. Let's review the 
> scenarios for the loop module.
> 
> 1) It is in the kernel. Then the /sys/block/loop0/dev file will be 
> present upon boot, the udev initscript as supplied with the current 
> version of udev will notice it while walking the /sys/block hierarchy, 
> and /dev/loop0 will be created from the initscript.

Cool, I didn't realize this.

> 2) It is a module. Then the modified initscript will notice that there 
> is a block-major line in /lib/modules/<version>/modules.alias with the 
> "loop" as its third field. You suggest to create /dev/loop0 based on 
> this information. I suggest to just call "modprobe loop" instead. The 
> kernel will send the hotplug event and the udev program as supplied in 
> the current udev package will react upon that event by creating /dev/loop0.

I wasn't suggesting anything; merely pointed out that your solution was
dependent on modules.  Ultimately udev will create /dev entries as it
walks sysfs (so it takes care of /dev/loopX, etc); but there are aspects
of the kernel that aren't sysfs-ified (e.g. tun).  Not to mention if the
devices don't support hotplug udev will never get triggered.

So wouldn't a better option be to make the various devices that don't
support hotplug and/or sysfs support them? Then udev just needs some new
rules.  Infering information from the module load (via modules.alias) is
fragile.  

Are there devices that can't and/or shouldn't be hotplug/sysfs-ified?

Mike



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Alexander E. Patrakov]

On Wednesday 17 March 2004 22:01, Mike Snitzer wrote:
> Alexander E. Patrakov wrote:
> > I don't propose to parse these files at all. I want to probe the modules
> > corresponding to the *-major-* aliases unconditionally.

> I wasn't suggesting anything; merely pointed out that your solution was 
> dependent on modules.  Ultimately udev will create /dev entries as it
> walks sysfs (so it takes care of /dev/loopX, etc); but there are aspects
> of the kernel that aren't sysfs-ified (e.g. tun).  Not to mention if the
> devices don't support hotplug udev will never get triggered.

We are talking about different problems here. I am talking about the fact that 
the FAQ says:

Q: But wait, I really want udev to automatically load drivers when they
   are not present but the device node is opened.  It's the only reason I
   like using devfs.  Please make udev do this.
A: No.  udev is for managing /dev, not loading kernel drivers.

Q: Oh come on, pretty please.  It can't be that hard to do.
A: Such a functionality isn't needed on a properly configured system. All
   devices present on the system should generate hotplug events, loading
   the appropriate driver, and udev will notice and create the
   appropriate device node.

I just propose a way of achieving the presence of some devices in /dev when 
the corresponding modules can't support PCI/USB/whatever hardware hotplug 
events at all (is loop a PCI device?). I _assume_ that sysfs is supported 
properly.

> So wouldn't a better option be to make the various devices that don't
> support hotplug and/or sysfs support them? Then udev just needs some new
> rules.  Infering information from the module load (via modules.alias) is
> fragile.

The loop module supports sysfs, and generated hotplug events when inserted or 
removed. The problem is that you cannot make the current hotplug scripts 
insert this module at hardware detection phase. If you know something 
different, please tell us how (based on what information) would you make 
devices such as loop, lp, ppdev (note that they properly support sysfs, but 
can't be detected as hardware) present in /dev? The only other solution 
mentioned here was to keep a big tarball of such undetectable devices and 
untar it every boot.

My approach is better because it also works without modification if I patch my 
kernel and add the pktcdvd module. I will not have to add the pktcdvd devices 
to that tarball.

-- 
Alexander E. Patrakov


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Greg KH]

On Fri, Mar 12, 2004 at 08:03:00PM +0500, Alexander E. Patrakov wrote:
> 
> I found my implementation rather clumsy and while implementing it I decided 
> that it would be better to just mark such modules and modprobe tham all, and 
> make the hotplug script modprobe other stuff. I decided to start implementing 
> such mark... and found that it has already been done, so actually no patches 
> are needed except for the udev initscript!
> 
> I am talking about /lib/modules/<version>/modules.alias.

Take a look at a distro's modules.alias file.  With your scheme you
would have them load every possible module.  Not a good idea :)

Sorry,

greg k-h


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Mike Snitzer]

On Wed, Mar 17 2004 at 20:56,
Alexander E. Patrakov <patrakov@ums.usu.ru> wrote:

> > So wouldn't a better option be to make the various devices that don't
> > support hotplug and/or sysfs support them? Then udev just needs some new
> > rules.  Infering information from the module load (via modules.alias) is
> > fragile.
> 
> The loop module supports sysfs, and generated hotplug events when inserted or 
> removed. The problem is that you cannot make the current hotplug scripts 
> insert this module at hardware detection phase. If you know something 
> different, please tell us how (based on what information) would you make 
> devices such as loop, lp, ppdev (note that they properly support sysfs, but 
> can't be detected as hardware) present in /dev? The only other solution 
> mentioned here was to keep a big tarball of such undetectable devices and 
> untar it every boot.

Ok, so at system boot umlstart walks sysfs and populates /dev accordingly;
loop devices work great.  GregKH says tun/tap devices now work in 2.6.4;
so ultimately its just a matter of adding sysfs support to the various
device drivers.

But your desire for loading modules when a device node is accessed isn't
doable.

Mike


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Alexander E. Patrakov]

Greg KH wrote:

>On Fri, Mar 12, 2004 at 08:03:00PM +0500, Alexander E. Patrakov wrote:
>  
>
>>I found my implementation rather clumsy and while implementing it I decided 
>>that it would be better to just mark such modules and modprobe tham all, and 
>>make the hotplug script modprobe other stuff. I decided to start implementing 
>>such mark... and found that it has already been done, so actually no patches 
>>are needed except for the udev initscript!
>>
>>I am talking about /lib/modules/<version>/modules.alias.
>>    
>>
>
>Take a look at a distro's modules.alias file.  With your scheme you
>would have them load every possible module.  Not a good idea :)
>  
>
Did that (both for Debian distribution and for the kernel source - 
grepped for MODULE_*ALIAS*'es that result in char-major or block-major 
entry in the modules.alias file). The junk is:
1) watchdog timers - they share the same alias which is not a good thing 
anyway
2) old cdroms like sbpcd - they won't load anyway on my computer
3) dv1394 and friends - we do need to blacklist them

Missing things are:
1) psmouse
2) ide-cd
3) maybe ACPI, but it is not accessible via /dev and therefore it is of 
no interest to udev

So maybe we do need to mark such software-only modules via special 
entries in .modinfo sections and not rely upon existing *major aliases. 
The idea remains the same: probe all modules with such mark set. I 
proposed to (mis-)use the presence of *major alias as such a mark.

Yes, I agree that not everybody needs the nbd module and with my 
approach, it will get loaded. But we already disagree with loading the 
module upon demand from an application accessing the non-existing device 
node, and therefore we have either to load the module at boot time or 
not to load it at all.

Oh, there is also a possibility of utilizing these char-major aliases 
for autoloading modules on demand with device nodes being already 
created, and this is what my earlier clumsy approach is about. Do you 
want to see my module-init-tools and kernel patches for that approach?

-- 
Alexander E. Patrakov



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Alexander E. Patrakov]

Mike Snitzer wrote:

>Ok, so at system boot udevstart walks sysfs and populates /dev 
>accordingly;
>loop devices work great.  GregKH says tun/tap devices now work in 2.6.4;
>so ultimately its just a matter of adding sysfs support to the various
>device drivers.
>
>But your desire for loading modules when a device node is accessed isn't
>doable.

This is not my desire. My wish is that all devices supported by kernel
(even by means of modules) should be present in /dev, before (not when!)
a program tries to access them, as it is the case now e.g. for PCI
soundcards where the hotplug initscript probes the needed modules when
the system boots.

-- 
Alexander E. Patrakov












-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Non-hardware devices like loop: another idea

[Marco d'Itri]

On Mar 18, "Alexander E. Patrakov" <patrakov@ums.usu.ru> wrote:

> >Take a look at a distro's modules.alias file.  With your scheme you
> >would have them load every possible module.  Not a good idea :)
I'm not sure about what looking at this would buy you. Can you be more
explicit?

> Missing things are:
> 1) psmouse
Just yesterday I asked Vojtech Pavlik about this and he explained that
the kernel still lacks hotplug support for the serio layer. There are
patches but they are not finished and help would be appreciated.

> 2) ide-cd
Last month Greg said that this should be handled by the distribution
init scripts, but did not explain how.
I could not find anything in the hotplug packages of the major RPM-based
distributions, so I'm looking for clues about this.

-- 
ciao, |
Marco | [5177 l'P4w3ETnUl/E]


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id\x1470&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel