how to have the kernel do udev's job and autoload the right modules ?

how to have the kernel do udev's job and autoload the right modules ?

[linux cbon]

Hi,

I am experimenting a rc.sysinit without udev.
Only creating /dev with mount -t devtmpfs dev /dev
It also mounts /proc and /sys and /tmp and /var .
So the kernel boots up loading a lot of hardware, but some important
modules are not loaded , like sound, network and video.
I am not sure how to have them auto loaded by the kernel without udev ?
I though kernel would have some kind of auto-loading of the right
modules, without needing any help like udev.
I don't know the simplest and easiest way to achieve this ?
Thanks for your help.

Best regard.

Re: how to have the kernel do udev's job and autoload the right modules ?

[Richard Weinberger]

On Mon, May 4, 2015 at 7:31 AM, linux cbon <linuxcbon@gmail.com> wrote:
> Hi,
>
> I am experimenting a rc.sysinit without udev.
> Only creating /dev with mount -t devtmpfs dev /dev
> It also mounts /proc and /sys and /tmp and /var .
> So the kernel boots up loading a lot of hardware, but some important
> modules are not loaded , like sound, network and video.
> I am not sure how to have them auto loaded by the kernel without udev ?
> I though kernel would have some kind of auto-loading of the right
> modules, without needing any help like udev.

We have request_module() but it is not a magic bullet.
You can re-invent a mini udev or just use udev.

> I don't know the simplest and easiest way to achieve this ?
> Thanks for your help.

We have udev for a reason. :-)

-- 
Thanks,
//richard

Re: how to have the kernel do udev's job and autoload the right modules ?

[linux cbon]

On Mon, May 4, 2015 at 8:34 AM, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
> On Mon, May 4, 2015 at 7:31 AM, linux cbon <linuxcbon@gmail.com> wrote:
>> Hi,
>>
>> I am experimenting a rc.sysinit without udev.
>> Only creating /dev with mount -t devtmpfs dev /dev
>> It also mounts /proc and /sys and /tmp and /var .
>> So the kernel boots up loading a lot of hardware, but some important
>> modules are not loaded , like sound, network and video.
>> I am not sure how to have them auto loaded by the kernel without udev ?
>> I though kernel would have some kind of auto-loading of the right
>> modules, without needing any help like udev.
>
> We have request_module() but it is not a magic bullet.
> You can re-invent a mini udev or just use udev.
>
>> I don't know the simplest and easiest way to achieve this ?
>> Thanks for your help.
>
> We have udev for a reason. :-)
>
> --
> Thanks,
> //richard



Hi Richard,

thanks for your answer.

use request_module() and reinvent a userspace mini udev ? Well thats
above my skills,
and I though the kernel was more magical. Do you know why not give a
"minimal" network,
 sound, and amd video support in the kernel, so that loading those
modules would be optional ?

I read there is mdev from busybox, but not sure if it's better or easier.
One last question : compiling the kernel with modules included would help ?

Best regards
Linuxcbon

Re: how to have the kernel do udev's job and autoload the right modules ?

[Richard Weinberger]

Am 04.05.2015 um 19:27 schrieb linux cbon:
> use request_module() and reinvent a userspace mini udev ? Well thats
> above my skills,
> and I though the kernel was more magical. Do you know why not give a
> "minimal" network,
>  sound, and amd video support in the kernel, so that loading those
> modules would be optional ?

You can build in the needed modules or just use udev...

> I read there is mdev from busybox, but not sure if it's better or easier.
> One last question : compiling the kernel with modules included would help ?

Yes.

Thanks,
//richard

Re: how to have the kernel do udev's job and autoload the right modules ?

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

On 2015-05-04 14:21, Richard Weinberger wrote:
> Am 04.05.2015 um 19:27 schrieb linux cbon:
>> use request_module() and reinvent a userspace mini udev ? Well thats
>> above my skills,
>> and I though the kernel was more magical. Do you know why not give a
>> "minimal" network,
>>   sound, and amd video support in the kernel, so that loading those
>> modules would be optional ?
>
> You can build in the needed modules or just use udev...
>
You might consider looking into eudev, it's a Gentoo hosted fork of udev 
targeted at embedded systems, and therefore very lightweight (and it 
doesn't depend on systemd).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: how to have the kernel do udev's job and autoload the right modules ?

[linux cbon]

On Mon, May 4, 2015 at 7:21 PM, Richard Weinberger <richard@nod.at> wrote:
> You can build in the needed modules or just use udev...

for example, I need these modules to be loaded automatically by the kernel,
but they are not : r8169 , radeon , snd_hda_codec etc.
Why doesn't the kernel do it, instead of using a userspace program like udev ?
Sorry if my question is silly.

Re: how to have the kernel do udev's job and autoload the right modules ?

[Richard Weinberger]

Am 05.05.2015 um 01:14 schrieb linux cbon:
> On Mon, May 4, 2015 at 7:21 PM, Richard Weinberger <richard@nod.at> wrote:
>> You can build in the needed modules or just use udev...
> 
> for example, I need these modules to be loaded automatically by the kernel,
> but they are not : r8169 , radeon , snd_hda_codec etc.
> Why doesn't the kernel do it, instead of using a userspace program like udev ?
> Sorry if my question is silly.

Because in some cases you want to blacklist modules, have special parameters for
them etc...
It is a typical "kernel offers mechanism and userspace policy" thing.

If you want them blindly loaded add them to your local.rc or build them in.

Thanks,
//richard

Re: how to have the kernel do udev's job and autoload the right modules ?

[linux cbon]

On Mon, May 4, 2015 at 11:24 PM, Richard Weinberger <richard@nod.at> wrote:
> You can build in the needed modules or just use udev...

Sorry , but I don't want a monolithic or a huge kernel with many modules inside.
I want a minimal and modular kernel which only loads the needed modules.
If I understand, there are 2 choices left :
1/ the kernel without modules has a minimal builtin support for my
network (RTL8111/8168B)
and my sound (RS780 and SBx00) ... but it doesn't seem the case.
2/ I can use udev in userspace to detect and load needed modules.

For 2/, for my rc.sysinit , can you please give me 2 or 3 lines I need to add
, the minimal possible, to have udev working and my modules loaded ?
I would like to have the smallest and minimal sysinit as possible. Maybe you can
help me for udev minimal setup ?
I have read that busybox mdev creates /dev only but doesn't load modules,
so  that's not what I want.
I already have /dev with devtmpfs , so udev or mdev shouldn't be doing
this anymore.

> Because in some cases you want to blacklist modules, have special parameters for
> them etc...
> It is a typical "kernel offers mechanism and userspace policy" thing.
> If you want them blindly loaded add them to your local.rc or build them in.
> Thanks,
> //richard

I don't add modprobes in my sysinit, because I find it's a dirty
workaround, it's manual,
it works only for one kind of hardware and not for another etc.
I also don't need to blacklist any modules, I don't see why I should
ever do this.
(to be honest , I also think the kernel should take care of the HW and
not the user...)
Once again I am not an expert in kernel internals and all the coding insides,
 so please correct me if I am wrong.

Thanks
linuxcbon

Re: how to have the kernel do udev's job and autoload the right modules ?

[Ken Moffat]

On Tue, May 05, 2015 at 06:08:01PM +0000, linux cbon wrote:
> On Mon, May 4, 2015 at 11:24 PM, Richard Weinberger <richard@nod.at> wrote:
> > You can build in the needed modules or just use udev...
> 
> Sorry , but I don't want a monolithic or a huge kernel with many modules inside.
> I want a minimal and modular kernel which only loads the needed modules.
> If I understand, there are 2 choices left :
> 1/ the kernel without modules has a minimal builtin support for my
> network (RTL8111/8168B)
> and my sound (RS780 and SBx00) ... but it doesn't seem the case.

I don't think you have ever given any context about what you are
trying to do.  It looks to me as if there are two alternatives:

1. You are building for a particular machine.  Either build
everything into the kernel, or use an initscript to load the modules.
For an x86 machine, the only reason to use modules in this situation
(apart from giving them a compile-test in the kernel) would be to
save memory, e.g. by not loading sound in runlevel 1.  Using modules
seems to offer minimal benefits in that situation.

2. You are building a distro.  For most distros, the need to support
as many users as possible is important.  Limiting the network to
juut 2 modules, and the same with audio, does not seem to make any
sense.  So, if this is a distro, the target is a tiny number of
machines, and you got lucky in only having two different network
drivers and two sound drivers.

If this is a "one build will run on all my own machines" system, you
could identify the particular machine in an initscript (check `uname
-n`) and modprobe the required modules.  Or, you could perhaps build
both these options into a monolithic kernel.  You said you don't
want a huge kernel, but you appear to be saying that only two things
differ.  Personally, I think that using a *specific* .config for each
of my machines is the way to go.  But then, I happily use eudev
(with some of my own initscripts, e.g. to control cpufreq).

>From your comment, I assume you have already removed all the config
options you do not use - if so, you are doing better than I am.  My
point here is that, at least on a machine from the last few years,
you probably have enough RAM that wasting a little of it by building
too much into your kernel is not a significant problem.

...
> I don't add modprobes in my sysinit, because I find it's a dirty
> workaround, it's manual,
> it works only for one kind of hardware and not for another etc.

If you think modprobe is a dirty workaround (really?) but you want
to use a semi-generic kernel without a udev variant, you don't seem
to leave yourself a lot of space to do things on a desktop machine.

What is so dirty about modprobe ?  Make a list of your hardware, per
machine (by this stage I'm fairly sure you want to use one kernel for
both, or all, your machines) and write a simple shell script to work
out which machine it is running on, and load the correct modules.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: how to have the kernel do udev's job and autoload the right modules ?

[linuxcbon linuxcbon]

On Tue, May 5, 2015 at 11:26 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> I don't think you have ever given any context about what you are
> trying to do.  It looks to me as if there are two alternatives:
(...)
I am trying to write a minimal rc.sysinit
Only the udev part is missing , because the kernel doesn't
know how to modprobe many modules by itself.
(Without udev, I can boot up to desktop, but sound , network and
 full screen video are missing.)

> 1. You are building for a particular machine.
> 2. You are building a distro.
> ĸen
(...)
I need it to work for any hardware, (well normal pc), so answer 2.
I need udev to detect and modprobe which modules are needed.

linuxcbon

Re: how to have the kernel do udev's job and autoload the right modules ?

[Ken Moffat]

On Wed, May 06, 2015 at 01:54:27AM +0100, linuxcbon linuxcbon wrote:
> On Tue, May 5, 2015 at 11:26 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> > I don't think you have ever given any context about what you are
> > trying to do.  It looks to me as if there are two alternatives:
> (...)
> I am trying to write a minimal rc.sysinit
> Only the udev part is missing , because the kernel doesn't
> know how to modprobe many modules by itself.
> (Without udev, I can boot up to desktop, but sound , network and
>  full screen video are missing.)
> 
> > 1. You are building for a particular machine.
> > 2. You are building a distro.
> > ĸen
> (...)
> I need it to work for any hardware, (well normal pc), so answer 2.
> I need udev to detect and modprobe which modules are needed.
> 
> linuxcbon

So, your earlier comment that only two different network drivers and
two sound drivers are needed will turn out to be incorrect when you
have access to more hardware.

And when that happens, you will discover that there are all sorts of
things which differ - since you asked on the kernel list, I will
mention KMS for the video, of which we have at least three variants
on x86 hardware.

I suggest that you take the time to look at eudev and mdev, and
think about how you can use the facilities they offer.

Alternatively, get over your dislike of modprobe and write some
scripts - perhaps using lspci as well as reading /sys - to work out
what hardware is present.

If you have not compiled all of the necesary disk drivers into the
kernel, you are likely to find that some (older) machines will not
be able to access their disk(s).  But we all learn best when we try
things out, so I wish you an enjoyable learning experience!

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: how to have the kernel do udev's job and autoload the right modules ?

[linuxcbon linuxcbon]

On Wed, May 6, 2015 at 5:55 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> I suggest that you take the time to look at eudev and mdev, and
> think about how you can use the facilities they offer.
I was wishing the kernel would offer some minimal support for
network, sound and full screen video for my hw :(.
But it seems I need to load modules to achieve this. And to load modules,
 it needs some kind of "hotplug" called udev or mdev.

> Alternatively, get over your dislike of modprobe and write some
> scripts - perhaps using lspci as well as reading /sys - to work out
> what hardware is present.
No manual modprobe, I need hotplug. Why have udev then,
 if we have to manually modprobe ?
Yes I think reading /sys will be the job, but why the user has to do
such things and not the kernel ? (get crazy and put udev
and all the hotplug stuff inside the kernel.)

> If you have not compiled all of the necesary disk drivers into the
> kernel, you are likely to find that some (older) machines will not
> be able to access their disk(s).  But we all learn best when we try
> things out, so I wish you an enjoyable learning experience!
> ĸen
yes still learning, testing and will never stop.
linuxcbon

Re: how to have the kernel do udev's job and autoload the right modules ?

[Ken Moffat]

On Wed, May 06, 2015 at 06:09:39PM +0100, linuxcbon linuxcbon wrote:
> On Wed, May 6, 2015 at 5:55 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> > I suggest that you take the time to look at eudev and mdev, and
> > think about how you can use the facilities they offer.
> I was wishing the kernel would offer some minimal support for
> network, sound and full screen video for my hw :(.
> But it seems I need to load modules to achieve this. And to load modules,
>  it needs some kind of "hotplug" called udev or mdev.
> 
Because it is something that _can_ be done in userspace, so it is
done there.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

Re: how to have the kernel do udev's job and autoload the right modules ?

[David Lang]

On Wed, 6 May 2015, linuxcbon linuxcbon wrote:

> On Wed, May 6, 2015 at 5:55 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
>> I suggest that you take the time to look at eudev and mdev, and
>> think about how you can use the facilities they offer.

> I was wishing the kernel would offer some minimal support for
> network, sound and full screen video for my hw :(.
> But it seems I need to load modules to achieve this. And to load modules,
> it needs some kind of "hotplug" called udev or mdev.

I've been building my own kernels for production systems for a long time. It is 
absolutly possible to have a kernel provide support for your hardware without 
modules.

The problem is the question of how much hardware you want to support. Modules 
were created because compiling everything into the kernel at once has multiple 
problems

1. sometimes different drivers can handle the same hardware, and you can only 
use one driver for the hardware

2. sometimes different hardware conflicts in that drivers for one piece of 
hardware will think that they've found their hardware, and prevent the proper 
drivers from working (sometimes doing 'strange' things to the hardware in the 
process)

3. the resulting kernel is VERY large. Back in the day, the problem was that the 
kernel would no longer fit on a floppy. We don't have that limit, but we still 
don't want to waste time reading a huge amount of data into RAM (at which point 
it prevents the RAM from being used for other things)

4. boot time would be horrible as all the drivers try to detect their hardware 
and time out.


so if you want to cover your hardware, you have two choices.

1. If you have a relatively small variation of hardware, just compile in all the 
drivers you need. This even works for most hotplugged items.

2. use modules

If you use modules, then you need to have some way of loading them. It's a very 
bad idea to have this happen by magic, without any control over the policies 
(sometimes you don't want drivers to load just because hardware exists). So you 
need to have a place to set the policy. Since the kernel provides mechanisms, 
not policy, the result is that the kernel tells userspace what it thinks it's 
found and it's up to userspace to then 'do the right thing'

So if you don't want to use udev, then you need to have something that replaces 
it to load the right module with the right options.

It's perfectly legitimate to not want to use udev, but that doesn't mean that 
the kernel will (or should) do it for you.

David Lang

Re: how to have the kernel do udev's job and autoload the right modules ?

[linuxcbon linuxcbon]

On Wed, May 6, 2015 at 7:53 PM, David Lang <david@lang.hm> wrote:
> It's perfectly legitimate to not want to use udev, but that doesn't mean
> that the kernel will (or should) do it for you.
> David Lang

When I boot the kernel without modules, I don't have anything working
except "minimal video".
I think the kernel should give a minimal support for network, sound and
video, even if 0 modules are loaded. I am just dreaming,
 so I will move on to using udev.
cheers
linuxcbon

Re: how to have the kernel do udev's job and autoload the right modules ?

[Richard Weinberger]

Am 06.05.2015 um 22:34 schrieb linuxcbon linuxcbon:
> On Wed, May 6, 2015 at 7:53 PM, David Lang <david@lang.hm> wrote:
>> It's perfectly legitimate to not want to use udev, but that doesn't mean
>> that the kernel will (or should) do it for you.
>> David Lang
> 
> When I boot the kernel without modules, I don't have anything working
> except "minimal video".
> I think the kernel should give a minimal support for network, sound and
> video, even if 0 modules are loaded. I am just dreaming,
>  so I will move on to using udev.

Sounds much you want a allyesconfig kernel...

Thanks,
//richard

Re: how to have the kernel do udev's job and autoload the right modules ?

[David Lang]

On Wed, 6 May 2015, linuxcbon linuxcbon wrote:

> On Wed, May 6, 2015 at 7:53 PM, David Lang <david@lang.hm> wrote:
>> It's perfectly legitimate to not want to use udev, but that doesn't mean
>> that the kernel will (or should) do it for you.
>> David Lang
>
> When I boot the kernel without modules, I don't have anything working
> except "minimal video".
> I think the kernel should give a minimal support for network, sound and
> video, even if 0 modules are loaded. I am just dreaming,

You can do that, you just need to build in all the network and sound drivers 
(and pick which driver in the case of conflicts)

There isn't such a thing as a 'generic' network or sound card. For video there 
is 'VGA video' which is used by default on x86 systems, but even that's a driver 
that could be disabled.

David Lang

Re: how to have the kernel do udev's job and autoload the right modules ?

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

On 2015-05-06 16:49, David Lang wrote:
> On Wed, 6 May 2015, linuxcbon linuxcbon wrote:
>
>> On Wed, May 6, 2015 at 7:53 PM, David Lang <david@lang.hm> wrote:
>>> It's perfectly legitimate to not want to use udev, but that doesn't mean
>>> that the kernel will (or should) do it for you.
>>> David Lang
>>
>> When I boot the kernel without modules, I don't have anything working
>> except "minimal video".
>> I think the kernel should give a minimal support for network, sound and
>> video, even if 0 modules are loaded. I am just dreaming,
>
> You can do that, you just need to build in all the network and sound
> drivers (and pick which driver in the case of conflicts)
>
> There isn't such a thing as a 'generic' network or sound card. For video
> there is 'VGA video' which is used by default on x86 systems, but even
> that's a driver that could be disabled.
>
To explain further, video has a standardized hardware level API (VGA and 
VBE) because it is considered critical system functionality (which is BS 
in my opinion, you can get by just fine with a serial console, but 
that's irrelevant to this discussion).  Sound is traditionally not 
considered critical, and therefore doesn't have a standardized hardware 
API.  Networking is (traditionally) only considered critical if the 
system is booting off the network, and therefore only has a standardized 
API (part of the PXE spec, known as UNDI) on some systems, and even then 
only when they are configured to netboot (and IIRC, also only when the 
processor is in real mode, just like for all other BIOS calls).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: how to have the kernel do udev's job and autoload the right modules ?

[David Lang]

On Thu, 7 May 2015, Austin S Hemmelgarn wrote:

> On 2015-05-06 16:49, David Lang wrote:
>> On Wed, 6 May 2015, linuxcbon linuxcbon wrote:
>> 
>>> On Wed, May 6, 2015 at 7:53 PM, David Lang <david@lang.hm> wrote:
>>>> It's perfectly legitimate to not want to use udev, but that doesn't mean
>>>> that the kernel will (or should) do it for you.
>>>> David Lang
>>> 
>>> When I boot the kernel without modules, I don't have anything working
>>> except "minimal video".
>>> I think the kernel should give a minimal support for network, sound and
>>> video, even if 0 modules are loaded. I am just dreaming,
>> 
>> You can do that, you just need to build in all the network and sound
>> drivers (and pick which driver in the case of conflicts)
>> 
>> There isn't such a thing as a 'generic' network or sound card. For video
>> there is 'VGA video' which is used by default on x86 systems, but even
>> that's a driver that could be disabled.
>> 
> To explain further, video has a standardized hardware level API (VGA and VBE) 
> because it is considered critical system functionality (which is BS in my 
> opinion, you can get by just fine with a serial console, but that's 
> irrelevant to this discussion).  Sound is traditionally not considered 
> critical, and therefore doesn't have a standardized hardware API.  Networking 
> is (traditionally) only considered critical if the system is booting off the 
> network, and therefore only has a standardized API (part of the PXE spec, 
> known as UNDI) on some systems, and even then only when they are configured 
> to netboot (and IIRC, also only when the processor is in real mode, just like 
> for all other BIOS calls).

I don't think that it has anything to do with critical system functionality, but 
rather just the legacy history of the PC clones. At one point VGA was the 
standard, and at that point the different video card manufacturers got into the 
game, but since they all had to boot the system, and the BIOS only knew how to 
talk to a VGA card, all the enhanced cards had to implement VGA so that DOS and 
the BIOS could function. That legacy has continued on the PC clone systems to 
today. Non PC clones didn't have such a standard, and they don't implement VGA 
on their video cards (unless it's a card ported from a PC)

Network cards were never standardized, and were optional add-ons. They also 
weren't needed for the system to boot, so there was never any standard for 
newcomers to implement.

David Lang