add devname module aliases to allow module on-demand auto-loading

add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

From: Kay Sievers <kay.sievers@vrfy.org>
Subject: add devname module aliases to allow module on-demand auto-loading

This adds:
  alias: devname:<name>
to some common kernel modules, which will allow the on-demand loading
of the kernel module when the device node is accessed.

Ideally all these modules would be compiled-in, but distros seems too
much in love with their modularization that we need to cover the common
cases with this new facility. It will allow us to remove a bunch of pretty
useless init scripts and modprobes from init scripts.

The static device node aliases will be carried in the module itself. The
program depmod will extract this information to a file in the module directory:
  $ cat /lib/modules/2.6.34-00650-g537b60d-dirty/modules.devname 
  # Device nodes to trigger on-demand module loading.
  microcode cpu/microcode c10:184
  fuse fuse c10:229
  ppp_generic ppp c108:0
  tun net/tun c10:200
  dm_mod mapper/control c10:235

Udev will pick up the depmod created file on startup and create all the
static device nodes which the kernel modules specify, so that these modules
get automatically loaded when the device node is accessed:
  $ /sbin/udevd --debug
  ...
  static_dev_create_from_modules: mknod '/dev/cpu/microcode' c10:184
  static_dev_create_from_modules: mknod '/dev/fuse' c10:229
  static_dev_create_from_modules: mknod '/dev/ppp' c108:0
  static_dev_create_from_modules: mknod '/dev/net/tun' c10:200
  static_dev_create_from_modules: mknod '/dev/mapper/control' c10:235
  udev_rules_apply_static_dev_perms: chmod '/dev/net/tun' 0666
  udev_rules_apply_static_dev_perms: chmod '/dev/fuse' 0666

A few device nodes are switched to statically allocated numbers, to allow
the static nodes to work. This might also useful for systems which still run
a plain static /dev, which is completely unsafe to use with any dynamic minor
numbers.

Note:
The devname aliases must be limited to the *common* and *single*instance*
device nodes, like the misc devices, and never be used for conceptually limited
systems like the loop devices, which should rather get fixed properly and get a
control node for losetup to talk to, instead of creating a random number of
device nodes in advance, regardless if they are ever used.

This facility is to hide the mess distros are creating with too modualized
kernels, and just to hide that these modules are not compiled-in, and not to
paper-over broken concepts. Thanks! :)

Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Alasdair G Kergon <agk@redhat.com>
Cc: Tigran Aivazian <tigran@aivazian.fsnet.co.uk>
Cc: Ian Kent <raven@themaw.net>
Signed-Off-By: Kay Sievers <kay.sievers@vrfy.org>
---

 arch/x86/kernel/microcode_core.c |    1 +
 drivers/md/dm-ioctl.c            |    5 ++++-
 drivers/net/ppp_generic.c        |    4 ++--
 drivers/net/tun.c                |    1 +
 fs/autofs4/dev-ioctl.c           |    5 ++++-
 fs/btrfs/super.c                 |    5 ++++-
 fs/fuse/dev.c                    |    1 +
 include/linux/miscdevice.h       |    3 +++
 8 files changed, 20 insertions(+), 5 deletions(-)

--- a/arch/x86/kernel/microcode_core.c
+++ b/arch/x86/kernel/microcode_core.c
@@ -260,6 +260,7 @@ static void microcode_dev_exit(void)
 }
 
 MODULE_ALIAS_MISCDEV(MICROCODE_MINOR);
+MODULE_ALIAS("devname:cpu/microcode");
 #else
 #define microcode_dev_init()	0
 #define microcode_dev_exit()	do { } while (0)
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1599,12 +1599,15 @@ static const struct file_operations _ctl
 };
 
 static struct miscdevice _dm_misc = {
-	.minor 		= MISC_DYNAMIC_MINOR,
+	.minor 		= MAPPER_CTRL_MINOR,
 	.name  		= DM_NAME,
 	.nodename	= "mapper/control",
 	.fops  		= &_ctl_fops
 };
 
+MODULE_ALIAS_MISCDEV(MAPPER_CTRL_MINOR);
+MODULE_ALIAS("devname:mapper/control");
+
 /*
  * Create misc character device and link to DM_DIR/control.
  */
--- a/drivers/net/ppp_generic.c
+++ b/drivers/net/ppp_generic.c
@@ -2907,5 +2907,5 @@ EXPORT_SYMBOL(ppp_output_wakeup);
 EXPORT_SYMBOL(ppp_register_compressor);
 EXPORT_SYMBOL(ppp_unregister_compressor);
 MODULE_LICENSE("GPL");
-MODULE_ALIAS_CHARDEV_MAJOR(PPP_MAJOR);
-MODULE_ALIAS("/dev/ppp");
+MODULE_ALIAS_CHARDEV(PPP_MAJOR, 0);
+MODULE_ALIAS("devname:ppp");
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1624,3 +1624,4 @@ MODULE_DESCRIPTION(DRV_DESCRIPTION);
 MODULE_AUTHOR(DRV_COPYRIGHT);
 MODULE_LICENSE("GPL");
 MODULE_ALIAS_MISCDEV(TUN_MINOR);
+MODULE_ALIAS("devname:net/tun");
--- a/fs/autofs4/dev-ioctl.c
+++ b/fs/autofs4/dev-ioctl.c
@@ -736,11 +736,14 @@ static const struct file_operations _dev
 };
 
 static struct miscdevice _autofs_dev_ioctl_misc = {
-	.minor 		= MISC_DYNAMIC_MINOR,
+	.minor 		= AUTOFS_MINOR,
 	.name  		= AUTOFS_DEVICE_NAME,
 	.fops  		= &_dev_ioctl_fops
 };
 
+MODULE_ALIAS_MISCDEV(BTRFS_MINOR);
+MODULE_ALIAS("devname:autofs");
+
 /* Register/deregister misc character device */
 int autofs_dev_ioctl_init(void)
 {
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -832,11 +832,14 @@ static const struct file_operations btrf
 };
 
 static struct miscdevice btrfs_misc = {
-	.minor		= MISC_DYNAMIC_MINOR,
+	.minor		= BTRFS_MINOR,
 	.name		= "btrfs-control",
 	.fops		= &btrfs_ctl_fops
 };
 
+MODULE_ALIAS_MISCDEV(BTRFS_MINOR);
+MODULE_ALIAS("devname:btrfs-control");
+
 static int btrfs_interface_init(void)
 {
 	return misc_register(&btrfs_misc);
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -18,6 +18,7 @@
 #include <linux/slab.h>
 
 MODULE_ALIAS_MISCDEV(FUSE_MINOR);
+MODULE_ALIAS("devname:fuse");
 
 static struct kmem_cache *fuse_req_cachep;
 
--- a/include/linux/miscdevice.h
+++ b/include/linux/miscdevice.h
@@ -31,6 +31,9 @@
 #define FUSE_MINOR		229
 #define KVM_MINOR		232
 #define VHOST_NET_MINOR		233
+#define BTRFS_MINOR		234
+#define MAPPER_CTRL_MINOR	235
+#define AUTOFS_MINOR		236
 #define MISC_DYNAMIC_MINOR	255
 
 struct device;

Re: add devname module aliases to allow module on-demand auto-loading

[Ian Kent]

On Thu, 2010-05-20 at 18:07 +0200, Kay Sievers wrote:
> From: Kay Sievers <kay.sievers@vrfy.org>
> Subject: add devname module aliases to allow module on-demand auto-loading
> 
> This adds:
>   alias: devname:<name>
> to some common kernel modules, which will allow the on-demand loading
> of the kernel module when the device node is accessed.
> 

snip ...

> --- a/fs/autofs4/dev-ioctl.c
> +++ b/fs/autofs4/dev-ioctl.c
> @@ -736,11 +736,14 @@ static const struct file_operations _dev
>  };
>  
>  static struct miscdevice _autofs_dev_ioctl_misc = {
> -	.minor 		= MISC_DYNAMIC_MINOR,
> +	.minor 		= AUTOFS_MINOR,
>  	.name  		= AUTOFS_DEVICE_NAME,
>  	.fops  		= &_dev_ioctl_fops
>  };
>  
> +MODULE_ALIAS_MISCDEV(BTRFS_MINOR);

Is this a typo or am I missing something?

Shouldn't it be:
+MODULE_ALIAS_MISCDEV(AUTOFS_MINOR);

> +MODULE_ALIAS("devname:autofs");
> +
>  /* Register/deregister misc character device */
>  int autofs_dev_ioctl_init(void)
>  {
> --- a/fs/btrfs/super.c
> +++ b/fs/btrfs/super.c
> @@ -832,11 +832,14 @@ static const struct file_operations btrf
>  };
>  
>  static struct miscdevice btrfs_misc = {
> -	.minor		= MISC_DYNAMIC_MINOR,
> +	.minor		= BTRFS_MINOR,
>  	.name		= "btrfs-control",
>  	.fops		= &btrfs_ctl_fops
>  };
>  
> +MODULE_ALIAS_MISCDEV(BTRFS_MINOR);
> +MODULE_ALIAS("devname:btrfs-control");
> +
>  static int btrfs_interface_init(void)
>  {
>  	return misc_register(&btrfs_misc);
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -18,6 +18,7 @@
>  #include <linux/slab.h>
>  
>  MODULE_ALIAS_MISCDEV(FUSE_MINOR);
> +MODULE_ALIAS("devname:fuse");
>  
>  static struct kmem_cache *fuse_req_cachep;
>  
> --- a/include/linux/miscdevice.h
> +++ b/include/linux/miscdevice.h
> @@ -31,6 +31,9 @@
>  #define FUSE_MINOR		229
>  #define KVM_MINOR		232
>  #define VHOST_NET_MINOR		233
> +#define BTRFS_MINOR		234
> +#define MAPPER_CTRL_MINOR	235
> +#define AUTOFS_MINOR		236
>  #define MISC_DYNAMIC_MINOR	255
>  
>  struct device;
> 
> 

Re: add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Fri, 2010-05-21 at 19:11 +0800, Ian Kent wrote:
> On Thu, 2010-05-20 at 18:07 +0200, Kay Sievers wrote:

> > --- a/fs/autofs4/dev-ioctl.c
> > +MODULE_ALIAS_MISCDEV(BTRFS_MINOR);
> 
> Is this a typo or am I missing something?

Yikes, sorry, that's just broken. New patch. Thanks a lot for checking.

Kay


From: Kay Sievers <kay.sievers@vrfy.org>
Subject: add devname module aliases to allow module on-demand auto-loading

This adds:
  alias: devname:<name>
to some common kernel modules, which will allow the on-demand loading
of the kernel module when the device node is accessed.

Ideally all these modules would be compiled-in, but distros seems too
much in love with their modularization that we need to cover the common
cases with this new facility. It will allow us to remove a bunch of pretty
useless init scripts and modprobes from init scripts.

The static device node aliases will be carried in the module itself. The
program depmod will extract this information to a file in the module directory:
  $ cat /lib/modules/2.6.34-00650-g537b60d-dirty/modules.devname
  # Device nodes to trigger on-demand module loading.
  microcode cpu/microcode c10:184
  fuse fuse c10:229
  ppp_generic ppp c108:0
  tun net/tun c10:200
  dm_mod mapper/control c10:235

Udev will pick up the depmod created file on startup and create all the
static device nodes which the kernel modules specify, so that these modules
get automatically loaded when the device node is accessed:
  $ /sbin/udevd --debug
  ...
  static_dev_create_from_modules: mknod '/dev/cpu/microcode' c10:184
  static_dev_create_from_modules: mknod '/dev/fuse' c10:229
  static_dev_create_from_modules: mknod '/dev/ppp' c108:0
  static_dev_create_from_modules: mknod '/dev/net/tun' c10:200
  static_dev_create_from_modules: mknod '/dev/mapper/control' c10:235
  udev_rules_apply_static_dev_perms: chmod '/dev/net/tun' 0666
  udev_rules_apply_static_dev_perms: chmod '/dev/fuse' 0666

A few device nodes are switched to statically allocated numbers, to allow
the static nodes to work. This might also useful for systems which still run
a plain static /dev, which is completely unsafe to use with any dynamic minor
numbers.

Note:
The devname aliases must be limited to the *common* and *single*instance*
device nodes, like the misc devices, and never be used for conceptually limited
systems like the loop devices, which should rather get fixed properly and get a
control node for losetup to talk to, instead of creating a random number of
device nodes in advance, regardless if they are ever used.

This facility is to hide the mess distros are creating with too modualized
kernels, and just to hide that these modules are not compiled-in, and not to
paper-over broken concepts. Thanks! :)

Cc: Greg Kroah-Hartman <gregkh@suse.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Alasdair G Kergon <agk@redhat.com>
Cc: Tigran Aivazian <tigran@aivazian.fsnet.co.uk>
Cc: Ian Kent <raven@themaw.net>
Signed-Off-By: Kay Sievers <kay.sievers@vrfy.org>
---

 Documentation/devices.txt        |    3 +++
 arch/x86/kernel/microcode_core.c |    1 +
 drivers/md/dm-ioctl.c            |    5 ++++-
 drivers/net/ppp_generic.c        |    4 ++--
 drivers/net/tun.c                |    1 +
 fs/autofs4/dev-ioctl.c           |    5 ++++-
 fs/btrfs/super.c                 |    5 ++++-
 fs/fuse/dev.c                    |    1 +
 include/linux/miscdevice.h       |    3 +++
 9 files changed, 23 insertions(+), 5 deletions(-)

--- a/Documentation/devices.txt
+++ b/Documentation/devices.txt
@@ -443,6 +443,9 @@ Your cooperation is appreciated.
 		231 = /dev/snapshot	System memory snapshot device
 		232 = /dev/kvm		Kernel-based virtual machine (hardware virtualization extensions)
 		233 = /dev/kmview	View-OS A process with a view
+		234 = /dev/btrfs-control	Btrfs control device
+		235 = /dev/mapper/control	Device-Mapper control device
+		236 = /dev/autofs	Autofs control device
 		240-254			Reserved for local use
 		255			Reserved for MISC_DYNAMIC_MINOR
 
--- a/arch/x86/kernel/microcode_core.c
+++ b/arch/x86/kernel/microcode_core.c
@@ -260,6 +260,7 @@ static void microcode_dev_exit(void)
 }
 
 MODULE_ALIAS_MISCDEV(MICROCODE_MINOR);
+MODULE_ALIAS("devname:cpu/microcode");
 #else
 #define microcode_dev_init()	0
 #define microcode_dev_exit()	do { } while (0)
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1599,12 +1599,15 @@ static const struct file_operations _ctl
 };
 
 static struct miscdevice _dm_misc = {
-	.minor 		= MISC_DYNAMIC_MINOR,
+	.minor 		= MAPPER_CTRL_MINOR,
 	.name  		= DM_NAME,
 	.nodename	= "mapper/control",
 	.fops  		= &_ctl_fops
 };
 
+MODULE_ALIAS_MISCDEV(MAPPER_CTRL_MINOR);
+MODULE_ALIAS("devname:mapper/control");
+
 /*
  * Create misc character device and link to DM_DIR/control.
  */
--- a/drivers/net/ppp_generic.c
+++ b/drivers/net/ppp_generic.c
@@ -2907,5 +2907,5 @@ EXPORT_SYMBOL(ppp_output_wakeup);
 EXPORT_SYMBOL(ppp_register_compressor);
 EXPORT_SYMBOL(ppp_unregister_compressor);
 MODULE_LICENSE("GPL");
-MODULE_ALIAS_CHARDEV_MAJOR(PPP_MAJOR);
-MODULE_ALIAS("/dev/ppp");
+MODULE_ALIAS_CHARDEV(PPP_MAJOR, 0);
+MODULE_ALIAS("devname:ppp");
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1624,3 +1624,4 @@ MODULE_DESCRIPTION(DRV_DESCRIPTION);
 MODULE_AUTHOR(DRV_COPYRIGHT);
 MODULE_LICENSE("GPL");
 MODULE_ALIAS_MISCDEV(TUN_MINOR);
+MODULE_ALIAS("devname:net/tun");
--- a/fs/autofs4/dev-ioctl.c
+++ b/fs/autofs4/dev-ioctl.c
@@ -736,11 +736,14 @@ static const struct file_operations _dev
 };
 
 static struct miscdevice _autofs_dev_ioctl_misc = {
-	.minor 		= MISC_DYNAMIC_MINOR,
+	.minor 		= AUTOFS_MINOR,
 	.name  		= AUTOFS_DEVICE_NAME,
 	.fops  		= &_dev_ioctl_fops
 };
 
+MODULE_ALIAS_MISCDEV(AUTOFS_MINOR);
+MODULE_ALIAS("devname:autofs");
+
 /* Register/deregister misc character device */
 int autofs_dev_ioctl_init(void)
 {
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -832,11 +832,14 @@ static const struct file_operations btrf
 };
 
 static struct miscdevice btrfs_misc = {
-	.minor		= MISC_DYNAMIC_MINOR,
+	.minor		= BTRFS_MINOR,
 	.name		= "btrfs-control",
 	.fops		= &btrfs_ctl_fops
 };
 
+MODULE_ALIAS_MISCDEV(BTRFS_MINOR);
+MODULE_ALIAS("devname:btrfs-control");
+
 static int btrfs_interface_init(void)
 {
 	return misc_register(&btrfs_misc);
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -18,6 +18,7 @@
 #include <linux/slab.h>
 
 MODULE_ALIAS_MISCDEV(FUSE_MINOR);
+MODULE_ALIAS("devname:fuse");
 
 static struct kmem_cache *fuse_req_cachep;
 
--- a/include/linux/miscdevice.h
+++ b/include/linux/miscdevice.h
@@ -31,6 +31,9 @@
 #define FUSE_MINOR		229
 #define KVM_MINOR		232
 #define VHOST_NET_MINOR		233
+#define BTRFS_MINOR		234
+#define MAPPER_CTRL_MINOR	235
+#define AUTOFS_MINOR		236
 #define MISC_DYNAMIC_MINOR	255
 
 struct device;

Re: add devname module aliases to allow module on-demand auto-loading

[Alasdair G Kergon]

On Thu, May 20, 2010 at 06:07:20PM +0200, Kay Sievers wrote:
> This adds:
>   alias: devname:<name>
> to some common kernel modules, which will allow the on-demand loading
> of the kernel module when the device node is accessed.

I don't see any need for this for device-mapper: please leave dm out of this.
 
> Ideally all these modules would be compiled-in, 

Why do you think that?  Currently that's a matter for the user/distro to
decide!  IMHO It's really not for the kernel to force a policy like this on its
users.  If that's what you think, why does your patch instead not go the whole
way and refuse to allow these items even to be compiled as modules?

> but distros seems too
> much in love with their modularization that we need to cover the common
> cases with this new facility. It will allow us to remove a bunch of pretty
> useless init scripts and modprobes from init scripts.

Again, I don't see why this needs any kernel changes.  If this was 
important, any distro could deal with it itself trivially without needing
a kernel patch.
 
Nack for the dm part of this from my point of view as it removes flexibility
with a 'one size fits all' approach that introduces a fixed minor number
into a dynamic world.

Alasdair

Re: add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Fri, May 21, 2010 at 13:34, Alasdair G Kergon <agk@redhat.com> wrote:
> On Thu, May 20, 2010 at 06:07:20PM +0200, Kay Sievers wrote:
>> This adds:
>>   alias: devname:<name>
>> to some common kernel modules, which will allow the on-demand loading
>> of the kernel module when the device node is accessed.
>
> I don't see any need for this for device-mapper: please leave dm out of this.
>
>> Ideally all these modules would be compiled-in,
>
> Why do you think that?  Currently that's a matter for the user/distro to
> decide!  IMHO It's really not for the kernel to force a policy like this on its
> users.  If that's what you think, why does your patch instead not go the whole
> way and refuse to allow these items even to be compiled as modules?

Well, they will work fine as modules, and we need them rto work as
such. It just does not make much sense if you are not a developer, and
distros should not do what they do, but that's a different story, I
don't want to get into. As a developer modules are more than useful,
they make kernel development possible without constantly rebooting.

This patch just brings the both needlessly different cases closer to
each other, and does not require special init scripts anymore, to
activate a specific sybsystem prior to using it.

>> but distros seems too
>> much in love with their modularization that we need to cover the common
>> cases with this new facility. It will allow us to remove a bunch of pretty
>> useless init scripts and modprobes from init scripts.
>
> Again, I don't see why this needs any kernel changes.  If this was
> important, any distro could deal with it itself trivially without needing
> a kernel patch.

That is actually to make systemd work on Fedora. And that's driven by
the company you work for. You might just not work in the area where
people fight against such problems, and don't know about them now.

> Nack for the dm part of this from my point of view as it removes flexibility
> with a 'one size fits all' approach that introduces a fixed minor number
> into a dynamic world.

There is no harm to make a well-know device node static, it just
solves a lot of problems, and also makes it possible to work off of a
static /dev. It's nothing different from the statically allocated
numbers for /dev/dm-*

Thanks for considering,
Kay

Re: add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Fri, May 21, 2010 at 13:51, Kay Sievers <kay.sievers@vrfy.org> wrote:
> On Fri, May 21, 2010 at 13:34, Alasdair G Kergon <agk@redhat.com> wrote:

> There is no harm to make a well-know device node static, it just
> solves a lot of problems, and also makes it possible to work off of a
> static /dev.

To illustrate:

On my box without this patch:
  dmsetup version
  Library version:   1.02.42 (2010-01-14)
  /proc/misc: No entry for device-mapper found
  Is device-mapper driver missing from kernel?
  Failure to communicate with kernel device-mapper driver.

And the same box just with this patch, nothing else changed:
  dmsetup version
  Library version:   1.02.42 (2010-01-14)
  Driver version:    4.17.0

But its up to you to care if device-mapper just works, or if there is
stuff like an init script with modprobe needed to load stuff that
might never be needed. :)

This is surely not about dynamic vs. static, it is about race-free
on-demand activation of services and subsystems.

Thanks,
Kay

Re: add devname module aliases to allow module on-demand auto-loading

[Nikanth Karthikesan]

On Friday 21 May 2010 18:41:38 Kay Sievers wrote:
> On Fri, May 21, 2010 at 13:51, Kay Sievers <kay.sievers@vrfy.org> wrote:
> > On Fri, May 21, 2010 at 13:34, Alasdair G Kergon <agk@redhat.com> wrote:
> >
> > There is no harm to make a well-know device node static, it just
> > solves a lot of problems, and also makes it possible to work off of a
> > static /dev.
> 
> To illustrate:
> 
> On my box without this patch:
>   dmsetup version
>   Library version:   1.02.42 (2010-01-14)
>   /proc/misc: No entry for device-mapper found
>   Is device-mapper driver missing from kernel?
>   Failure to communicate with kernel device-mapper driver.
> 
> And the same box just with this patch, nothing else changed:
>   dmsetup version
>   Library version:   1.02.42 (2010-01-14)
>   Driver version:    4.17.0
> 
> But its up to you to care if device-mapper just works, or if there is
> stuff like an init script with modprobe needed to load stuff that
> might never be needed. :)
>

If this is needed, the dmsetup itself can do a `modprobe dm` instead of 
printing the message, "Is device-mapper driver missing from kernel?"?

> This is surely not about dynamic vs. static, it is about race-free
> on-demand activation of services and subsystems.
> 

Loading dm-mod alone is enough for `dmsetup version`. But for other operations 
dm-mod may not be enough, as various other modules like dm-crypt, dm-
mirror,... would also be required, depending on the dm table, which may or may 
not be installed.

Thanks
Nikanth

> Thanks,
> Kay
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Fri, May 21, 2010 at 15:39, Nikanth Karthikesan <knikanth@suse.de> wrote:
> On Friday 21 May 2010 18:41:38 Kay Sievers wrote:
>> On Fri, May 21, 2010 at 13:51, Kay Sievers <kay.sievers@vrfy.org> wrote:
>> > On Fri, May 21, 2010 at 13:34, Alasdair G Kergon <agk@redhat.com> wrote:
>> >
>> > There is no harm to make a well-know device node static, it just
>> > solves a lot of problems, and also makes it possible to work off of a
>> > static /dev.
>>
>> To illustrate:
>>
>> On my box without this patch:
>>   dmsetup version
>>   Library version:   1.02.42 (2010-01-14)
>>   /proc/misc: No entry for device-mapper found
>>   Is device-mapper driver missing from kernel?
>>   Failure to communicate with kernel device-mapper driver.
>>
>> And the same box just with this patch, nothing else changed:
>>   dmsetup version
>>   Library version:   1.02.42 (2010-01-14)
>>   Driver version:    4.17.0
>>
>> But its up to you to care if device-mapper just works, or if there is
>> stuff like an init script with modprobe needed to load stuff that
>> might never be needed. :)
>>
>
> If this is needed, the dmsetup itself can do a `modprobe dm` instead of
> printing the message, "Is device-mapper driver missing from kernel?"?

No, it can't. There is a delay until the node appears, unless you use
devtmpfs. This can not be handled reliably by usual non-hotplug-aware
tools.

Also it could be any user of libdevmapper doing this. And running
processes from libraries without the user of the lib knowing all the
problems sub-processes create is not recommended at all.

There is a good reason to let the kernel load the module on-demand,
and not to code all sorts of racy stuff into tools. It's just open(),
nothing else, and it is guaranteed by the kenrel to work, without any
further synchronization.

>> This is surely not about dynamic vs. static, it is about race-free
>> on-demand activation of services and subsystems.
>>
> Loading dm-mod alone is enough for `dmsetup version`. But for other operations
> dm-mod may not be enough, as various other modules like dm-crypt, dm-
> mirror,... would also be required, depending on the dm table, which may or may
> not be installed.

Sure, but that is a detail, that can be solved by kernel-driven
on-demand loading as well.

Kay

Re: add devname module aliases to allow module on-demand auto-loading

[Greg KH]

On Fri, May 21, 2010 at 01:24:13PM +0200, Kay Sievers wrote:
> On Fri, 2010-05-21 at 19:11 +0800, Ian Kent wrote:
> > On Thu, 2010-05-20 at 18:07 +0200, Kay Sievers wrote:
> 
> > > --- a/fs/autofs4/dev-ioctl.c
> > > +MODULE_ALIAS_MISCDEV(BTRFS_MINOR);
> > 
> > Is this a typo or am I missing something?
> 
> Yikes, sorry, that's just broken. New patch. Thanks a lot for checking.

Thanks, I've updated the patch in my tree to this version.

greg k-h

Re: add devname module aliases to allow module on-demand auto-loading

[Alasdair G Kergon]

On Fri, May 21, 2010 at 03:55:21PM +0200, Kay Sievers wrote:
> On Fri, May 21, 2010 at 15:39, Nikanth Karthikesan <knikanth@suse.de> wrote:
> > On Friday 21 May 2010 18:41:38 Kay Sievers wrote:
> >> On Fri, May 21, 2010 at 13:51, Kay Sievers <kay.sievers@vrfy.org> wrote:
> >> > On Fri, May 21, 2010 at 13:34, Alasdair G Kergon <agk@redhat.com> wrote:
> >> >
> >> > There is no harm to make a well-know device node static, it just
> >> > solves a lot of problems, and also makes it possible to work off of a
> >> > static /dev.

Well until now we've tried to make everything possible treat device numbers as
dynamic, so I'd like to see some compelling logic behind the change.

> >> To illustrate:
> >>
> >> On my box without this patch:
> >>   dmsetup version
> >>   Library version:   1.02.42 (2010-01-14)
> >>   /proc/misc: No entry for device-mapper found
> >>   Is device-mapper driver missing from kernel?
> >>   Failure to communicate with kernel device-mapper driver.

(Curiously this is not an issue I remember anyone raising with me as a problem
before.)

So what concepts are at stake here.
The module is available to the kernel but has not been loaded.
Some users do not need the module, so it should not be loaded by default.
Users who *do* need it would like it to get loaded automatically.

The matter under discussion is what mechanism to use to load it automatically.

The unique information is the module name so the mechanism should
ideally be tied directly to that.  Anything wanting to use dm already
knows that name.  The character device only becomes available later,
after the module is loaded, and userspace obtains it from /proc/misc.

The modprobe mechanism is tied to the name, so we should really look for
a solution based on that in the first instance. 

A related matter - not yet mentioned or discussed between us - is how
the /dev/mapper/control node gets created and what it should be called,
as that name obviously goes against the udev standard of placing
everything in a flat /dev namespace with kernel-based names (so
something like '/dev/miscNNN' in this case) and creating symlinks from
the traditional filesystem locations like /dev/mapper/control.

Currently libdevmapper creates /dev/mapper/control as required based on
/proc/misc.  Presumably we should now hand that over to udev as a
special case of the way we transferred the entries for the actual
devices with similar waiting & notification.

Currently tools (not libdevmapper) take responsibility for checking and
autoloading.  E.g. lvm issues modprobes to autoload dm target modules.
(The kernel does also handle this but lvm doesn't use it because
it wants to know earlier that the operation would fail to avoid more
complex error-handling code.)

So I think we should first try to make an extension of the existing
mechanisms work.  Everything is keyed off a single piece of information,
the module name, shared between userspace and kernel.

Alasdair

Re: add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Fri, May 21, 2010 at 20:23, Alasdair G Kergon <agk@redhat.com> wrote:
> On Fri, May 21, 2010 at 03:55:21PM +0200, Kay Sievers wrote:
>> On Fri, May 21, 2010 at 15:39, Nikanth Karthikesan <knikanth@suse.de> wrote:
>> > On Friday 21 May 2010 18:41:38 Kay Sievers wrote:
>> >> On Fri, May 21, 2010 at 13:51, Kay Sievers <kay.sievers@vrfy.org> wrote:
>> >> > On Fri, May 21, 2010 at 13:34, Alasdair G Kergon <agk@redhat.com> wrote:
>> >> >
>> >> > There is no harm to make a well-know device node static, it just
>> >> > solves a lot of problems, and also makes it possible to work off of a
>> >> > static /dev.
>
> Well until now we've tried to make everything possible treat device numbers as
> dynamic, so I'd like to see some compelling logic behind the change.

It's a well known single-instance device, and the static assignment
solves a few problems with on-demand subsystem activation (which is
the new fashion :)) and makes it possible to hide the
compiled-in/modular issues we see, and have no proper solution for it.

>> >> To illustrate:
>> >>
>> >> On my box without this patch:
>> >>   dmsetup version
>> >>   Library version:   1.02.42 (2010-01-14)
>> >>   /proc/misc: No entry for device-mapper found
>> >>   Is device-mapper driver missing from kernel?
>> >>   Failure to communicate with kernel device-mapper driver.
>
> (Curiously this is not an issue I remember anyone raising with me as a problem
> before.)

Because distros go the easy and stupid way and just load everything
they configured as a module. :) That is something to solve, or at
least avoid and hide a bit.

> So what concepts are at stake here.
> The module is available to the kernel but has not been loaded.
> Some users do not need the module, so it should not be loaded by default.
> Users who *do* need it would like it to get loaded automatically.

Yes. Like many other things too. Ideally we just hide as much as we
can that stuff is a module and not compiled-in.

> The matter under discussion is what mechanism to use to load it automatically.
>
> The unique information is the module name so the mechanism should
> ideally be tied directly to that.  Anything wanting to use dm already
> knows that name.  The character device only becomes available later,
> after the module is loaded, and userspace obtains it from /proc/misc.
>
> The modprobe mechanism is tied to the name, so we should really look for
> a solution based on that in the first instance.
>
> A related matter - not yet mentioned or discussed between us - is how
> the /dev/mapper/control node gets created and what it should be called,
> as that name obviously goes against the udev standard of placing
> everything in a flat /dev namespace with kernel-based names (so
> something like '/dev/miscNNN' in this case) and creating symlinks from
> the traditional filesystem locations like /dev/mapper/control.

The kernel itself creates /dev/mapper/control today. And if devtmpfs
is not used (all recent released or upcoming distro releases depend on
it) the kernel will supply this name to udev, and udev will create it.

> Currently libdevmapper creates /dev/mapper/control as required based on
> /proc/misc.  Presumably we should now hand that over to udev as a
> special case of the way we transferred the entries for the actual
> devices with similar waiting & notification.

It like this already for a while. But it's all in the kernel, and is
race free regarding the module loading, At the moment modprobe comes
back, we are sure the node exists.

> Currently tools (not libdevmapper) take responsibility for checking and
> autoloading.  E.g. lvm issues modprobes to autoload dm target modules.
> (The kernel does also handle this but lvm doesn't use it because
> it wants to know earlier that the operation would fail to avoid more
> complex error-handling code.)

That's what this is about. The kernel will block the process that
accesses /dev/mapper/control until it is loaded. No need to sync
anything or then, or handle complex error cases.

> So I think we should first try to make an extension of the existing
> mechanisms work.  Everything is keyed off a single piece of information,
> the module name, shared between userspace and kernel.

I think its all covered with this patch. :) But if yo have any better
idea, let us know.

Thanks,
Kay

Re: [dm-devel] add devname module aliases to allow module on-demand auto-loading

[Alasdair G Kergon]

On Fri, May 21, 2010 at 08:51:49PM +0200, Kay Sievers wrote:
> The kernel itself creates /dev/mapper/control today. 

Must have missed that patch:)
Userspace lvm and dm code is certainly not tied to that today.

The device node that userspace lvm and dm tools use is under the
complete control of this userspace code: it chooses the name, and
creates or fixes it if it doesn't already exist with the required
properties: anything devtmpfs is relevant only if it guessed the
right name that this *completely independent* userspace code
chooses to use and already created it exactly as required.

If we were to move to a fixed name for the control device and hand
control of that name to the kernel, then obviously this userspace
code would need adjusting to respect that.

Alasdair

Re: [dm-devel] add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Fri, May 21, 2010 at 21:32, Alasdair G Kergon <agk@redhat.com> wrote:
> On Fri, May 21, 2010 at 08:51:49PM +0200, Kay Sievers wrote:
>> The kernel itself creates /dev/mapper/control today.
>
> Must have missed that patch:)
> Userspace lvm and dm code is certainly not tied to that today.
>
> The device node that userspace lvm and dm tools use is under the
> complete control of this userspace code: it chooses the name, and
> creates or fixes it if it doesn't already exist with the required
> properties: anything devtmpfs is relevant only if it guessed the
> right name that this *completely independent* userspace code
> chooses to use and already created it exactly as required.
>
> If we were to move to a fixed name for the control device and hand
> control of that name to the kernel, then obviously this userspace
> code would need adjusting to respect that.

Upstream udev has no primary device node naming anymore, it's all
controlled by the kernel. In fact, recent udev versions log errors if
userspace defines names which disagree with the kernel-provided names,
or the kernel-created nodes.

There is no "abstraction" anymore, the kernel defines the API today,
and device naming is 100% inside the kernel.

Udev's job is manage permissions, and to create meaningful symlinks
which may change during lifetime of a kernel device.

The primary kernel device name is always in sync with "dmesg",
"proc/sys" and "/dev" these days. Only the symlinks may change at any
time, reflecting the current state of the device.

Kay

Re: [dm-devel] add devname module aliases to allow module on-demand auto-loading

[Luca Berra]

On Fri, May 21, 2010 at 08:51:49PM +0200, Kay Sievers wrote:
>On Fri, May 21, 2010 at 20:23, Alasdair G Kergon <agk@redhat.com> wrote:
>> On Fri, May 21, 2010 at 03:55:21PM +0200, Kay Sievers wrote:
>>> >> On my box without this patch:
>>> >>   dmsetup version
>>> >>   Library version:   1.02.42 (2010-01-14)
>>> >>   /proc/misc: No entry for device-mapper found
>>> >>   Is device-mapper driver missing from kernel?
>>> >>   Failure to communicate with kernel device-mapper driver.
>>
>> (Curiously this is not an issue I remember anyone raising with me as a problem
>> before.)
I am fairly sure i did report that, i even tried moving the module
loading framework from lvm2 to device mapper, but the resulting code
sucked too much to submit, the idea was to load device-mapper module,
and all target modules on first use.

>Because distros go the easy and stupid way and just load everything
>they configured as a module. :) That is something to solve, or at
>least avoid and hide a bit.
In this case we were forced to preload dm-mod to work around the issue
;)

>
>I think its all covered with this patch. :) But if yo have any better
>idea, let us know.

what patch?

L.
-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: [dm-devel] add devname module aliases to allow module on-demand auto-loading

[Alasdair G Kergon]

On Tue, May 25, 2010 at 07:33:41AM +0200, Luca Berra wrote:
> what patch?

Peter Rajnoha is working on a final version of this, which includes
tweaking the userspace logic to give supremacy to devtmpfs.

Alasdair

Re: [dm-devel] add devname module aliases to allow module on-demand auto-loading

[Kay Sievers]

On Tue, May 25, 2010 at 11:48, Alasdair G Kergon <agk@redhat.com> wrote:
> On Tue, May 25, 2010 at 07:33:41AM +0200, Luca Berra wrote:
>> what patch?
>
> Peter Rajnoha is working on a final version of this, which includes
> tweaking the userspace logic to give supremacy to devtmpfs.

Note, that this will also work without devtmpfs, it will be just udev
pre-creating the nodes on startup, even when the modules aren't
loaded. Any access to the "dead" device nodes will cause the kernel to
load the corresponding module.

We need this facility mainly for other subsystems, which depend on
this behavior for some time already.

Thanks,
Kay