[PATCH 0/6] Clean up virtio device object handling [was Re: [PATCH] virtio: make PCI devices take a virtio_pci module ref]

[PATCH 0/6] Clean up virtio device object handling [was Re: [PATCH] virtio: make PCI devices take a virtio_pci module ref]

[Mark McLoughlin]

(Moved from kvm@vger to virtualization@linux-foundation, changed
subject, cleaned up cc list)

On Wed, 2008-12-10 at 13:02 +0100, Kay Sievers wrote:
> On Wed, Dec 10, 2008 at 10:49, Mark McLoughlin <markmc@redhat.com> wrote:
> > On Tue, 2008-12-09 at 19:16 +0100, Kay Sievers wrote:
> >> On Tue, Dec 9, 2008 at 17:41, Mark McLoughlin <markmc@redhat.com> wrote:
> >> > On Mon, 2008-12-08 at 08:46 -0600, Anthony Liguori wrote:
> >> >> Mark McLoughlin wrote:
> >> >> > On Sun, 2008-12-07 at 18:52 +1030, Rusty Russell wrote:
> >> >> >> On Saturday 06 December 2008 01:37:06 Mark McLoughlin wrote:
> >> >> >>
> >> >> >>> Another example of a lack of an explicit dependency causing problems is
> >> >> >>> Fedora's mkinitrd having this hack:
> >> >> >>>
> >> >> >>>     if echo $PWD | grep -q /virtio-pci/ ; then
> >> >> >>>         findmodule virtio_pci
> >> >> >>>     fi
> >> >> >>>
> >> >> >>> which basically says "if this is a virtio device, don't forget to
> >> >> >>> include virtio_pci in the initrd too!". Now, mkinitrd is full of hacks,
> >> >> >>> but this is a particularly unusual one.
> >> >> >>>
> >> >> >> Um, I don't know what this does, sorry.
> >> >> >>
> >> >> >> I have no idea how Fedora chooses what to put in an initrd; I can't think
> >> >> >> of a sensible way of deciding what goes in and what doesn't other than
> >> >> >> lists and heuristics.
> >> >> >>
> >> >> >
> >> >> > Fedora's mkinitrd creates an initrd suitable to boot the machine you run
> >> >> > mkinitrd on, rather than creating an initrd suitable to boot any
> >> >> > machine.
> >> >> >
> >> >> > So, it goes "ah, / is mounted from /dev/vda, we need to include
> >> >> > virtio_blk and it's dependencies". It does that in a generic way that
> >> >> > works well for most setups:
> >> >> >
> >> >> >   1) Find the device name (e.g. vda) below /sys/block
> >> >> >
> >> >> >   2) Follow the 'device' link to e.g. /sys/devices/virtio-pci/virtio1
> >> >> >
> >> >> >   3) Find the module need for this through either 'modalias' or the
> >> >> >      'driver/module' symlink
> >> >> >
> >> >> >   4) Use modprobe to list any dependencies of that module
> >> >> >
> >> >> > Clearly, virtio-pci won't be pulled in by any of this so we've added a
> >> >> > hack to say "oh, it's a virtio device, let's include virtio_pci just in
> >> >> > case".
> >> >> >
> >> >> > It's not even the case that mkinitrd needs to know how to include the
> >> >> > the module for the bus, because in our case that's virtio.ko ... we've
> >> >> > pretty effectively hidden the the bus *implementation* from userspace.
> >> >> >
> >> >> > I don't think this is worth wasting too much time fixing, that's why I'm
> >> >> > thinking we should just make virtio_pci built-in by default with
> >> >> > CONFIG_KVM_GUEST.
> >> >> >
> >> >>
> >> >> What if we have multiple virtio transports?
> >> >
> >> > I don't think that's so much an an issue (just build in any transport
> >> > supported by KVM), but rather that you might build a non-pv_ops kernel
> >> > to run on QEMU which would benefit from using virtio drivers ...
> >> >
> >> >>   Is there a way that we can
> >> >> expose the relationship with virtio-blk and virtio-pci in sysfs?  We
> >> >> have a struct device for the PCI device, it's just a matter of making
> >> >> the link visible.
> >> >
> >> > It feels a bit like busy work to generalise this since only virtio_pci
> >> > can be built as a module, but here's a patch.
> >> >
> >> > The mkinitrd hack turns into:
> >> >
> >> >        # Handle finding virtio bus implementations
> >> >        if [ -L ./virtio_module ] ; then
> >> >            findmodule $(basename $(readlink ./virtio_module))
> >> >        else if echo $PWD | grep -q /virtio-pci/ ; then
> >> >            findmodule virtio_pci
> >> >        fi; fi
> >> >
> >> > [PATCH] virtio: add a 'virtio_module' sysfs symlink
> >>
> >> Doesn't the device have a "driver" link already? If yes, the driver it
> >> points to should have a "module" link.
> >
> > The virtio bus is an abstraction that has several different backend
> > implementations - currently virtio-pci, lguest and kvm-s390.
> >
> > So yes, the driver/module link gives us the device driver, but the
> > virtio_module link is to the virtio bus driver (aka implementation,
> > transport, backend, ...):
> >
> >  $> basename $(readlink virtio_module)
> >  virtio_pci
> >  $> basename $(readlink driver/module)
> >  virtio_net
> 
> I see. But why not just call it "module", like we do in all other
> places, when it points to /sys/module/.
> 
> To find dependent modules, you would walk up the chain of parents, and
> include everything that is found by looking for "driver/module" and
> "module" links?
> 
> Wouldn't that make it completely generic, without any virtio specific hacks?

Yeah, that sounds much better - a minor detail is that it'd be better to
hang the symlink off each virtio implementation's root object rather
than off each device.

To that end, I've hacked up register_virtio_root_device() which fixes
the fact that we statically allocate root objects and gives us a sane
place to add this generic symlink.

It might make sense to add this to the core, though - e.g.
device_register_root() - and that would also allow us use the same
approach as module_add_driver() to add the module symlink for built-in
modules.

Cheers,
Mark.

Re: [PATCH 0/6] Clean up virtio device object handling [was Re: [PATCH] virtio: make PCI devices take a virtio_pci module ref]

[Kay Sievers]

On Wed, Dec 10, 2008 at 18:44, Mark McLoughlin <markmc@redhat.com> wrote:
> (Moved from kvm@vger to virtualization@linux-foundation, changed
> subject, cleaned up cc list)
> On Wed, 2008-12-10 at 13:02 +0100, Kay Sievers wrote:
>> On Wed, Dec 10, 2008 at 10:49, Mark McLoughlin <markmc@redhat.com> wrote:
>> > On Tue, 2008-12-09 at 19:16 +0100, Kay Sievers wrote:
>> >> On Tue, Dec 9, 2008 at 17:41, Mark McLoughlin <markmc@redhat.com> wrote:
>> >> > On Mon, 2008-12-08 at 08:46 -0600, Anthony Liguori wrote:
>> >> >> Mark McLoughlin wrote:
>> >> >> > On Sun, 2008-12-07 at 18:52 +1030, Rusty Russell wrote:
>> >> >> >> On Saturday 06 December 2008 01:37:06 Mark McLoughlin wrote:
>> >> >> >>
>> >> >> >>> Another example of a lack of an explicit dependency causing problems is
>> >> >> >>> Fedora's mkinitrd having this hack:
>> >> >> >>>
>> >> >> >>>     if echo $PWD | grep -q /virtio-pci/ ; then
>> >> >> >>>         findmodule virtio_pci
>> >> >> >>>     fi
>> >> >> >>>
>> >> >> >>> which basically says "if this is a virtio device, don't forget to
>> >> >> >>> include virtio_pci in the initrd too!". Now, mkinitrd is full of hacks,
>> >> >> >>> but this is a particularly unusual one.
>> >> >> >>>
>> >> >> >> Um, I don't know what this does, sorry.
>> >> >> >>
>> >> >> >> I have no idea how Fedora chooses what to put in an initrd; I can't think
>> >> >> >> of a sensible way of deciding what goes in and what doesn't other than
>> >> >> >> lists and heuristics.
>> >> >> >>
>> >> >> >
>> >> >> > Fedora's mkinitrd creates an initrd suitable to boot the machine you run
>> >> >> > mkinitrd on, rather than creating an initrd suitable to boot any
>> >> >> > machine.
>> >> >> >
>> >> >> > So, it goes "ah, / is mounted from /dev/vda, we need to include
>> >> >> > virtio_blk and it's dependencies". It does that in a generic way that
>> >> >> > works well for most setups:
>> >> >> >
>> >> >> >   1) Find the device name (e.g. vda) below /sys/block
>> >> >> >
>> >> >> >   2) Follow the 'device' link to e.g. /sys/devices/virtio-pci/virtio1
>> >> >> >
>> >> >> >   3) Find the module need for this through either 'modalias' or the
>> >> >> >      'driver/module' symlink
>> >> >> >
>> >> >> >   4) Use modprobe to list any dependencies of that module
>> >> >> >
>> >> >> > Clearly, virtio-pci won't be pulled in by any of this so we've added a
>> >> >> > hack to say "oh, it's a virtio device, let's include virtio_pci just in
>> >> >> > case".
>> >> >> >
>> >> >> > It's not even the case that mkinitrd needs to know how to include the
>> >> >> > the module for the bus, because in our case that's virtio.ko ... we've
>> >> >> > pretty effectively hidden the the bus *implementation* from userspace.
>> >> >> >
>> >> >> > I don't think this is worth wasting too much time fixing, that's why I'm
>> >> >> > thinking we should just make virtio_pci built-in by default with
>> >> >> > CONFIG_KVM_GUEST.
>> >> >> >
>> >> >>
>> >> >> What if we have multiple virtio transports?
>> >> >
>> >> > I don't think that's so much an an issue (just build in any transport
>> >> > supported by KVM), but rather that you might build a non-pv_ops kernel
>> >> > to run on QEMU which would benefit from using virtio drivers ...
>> >> >
>> >> >>   Is there a way that we can
>> >> >> expose the relationship with virtio-blk and virtio-pci in sysfs?  We
>> >> >> have a struct device for the PCI device, it's just a matter of making
>> >> >> the link visible.
>> >> >
>> >> > It feels a bit like busy work to generalise this since only virtio_pci
>> >> > can be built as a module, but here's a patch.
>> >> >
>> >> > The mkinitrd hack turns into:
>> >> >
>> >> >        # Handle finding virtio bus implementations
>> >> >        if [ -L ./virtio_module ] ; then
>> >> >            findmodule $(basename $(readlink ./virtio_module))
>> >> >        else if echo $PWD | grep -q /virtio-pci/ ; then
>> >> >            findmodule virtio_pci
>> >> >        fi; fi
>> >> >
>> >> > [PATCH] virtio: add a 'virtio_module' sysfs symlink
>> >>
>> >> Doesn't the device have a "driver" link already? If yes, the driver it
>> >> points to should have a "module" link.
>> >
>> > The virtio bus is an abstraction that has several different backend
>> > implementations - currently virtio-pci, lguest and kvm-s390.
>> >
>> > So yes, the driver/module link gives us the device driver, but the
>> > virtio_module link is to the virtio bus driver (aka implementation,
>> > transport, backend, ...):
>> >
>> >  $> basename $(readlink virtio_module)
>> >  virtio_pci
>> >  $> basename $(readlink driver/module)
>> >  virtio_net
>>
>> I see. But why not just call it "module", like we do in all other
>> places, when it points to /sys/module/.
>>
>> To find dependent modules, you would walk up the chain of parents, and
>> include everything that is found by looking for "driver/module" and
>> "module" links?
>>
>> Wouldn't that make it completely generic, without any virtio specific hacks?
>
> Yeah, that sounds much better - a minor detail is that it'd be better to
> hang the symlink off each virtio implementation's root object rather
> than off each device.

Sounds good, if the devices below can never be from different modules
at the same time.

> To that end, I've hacked up register_virtio_root_device() which fixes
> the fact that we statically allocate root objects and gives us a sane
> place to add this generic symlink.

Fine.

> It might make sense to add this to the core, though - e.g.
> device_register_root() - and that would also allow us use the same
> approach as module_add_driver() to add the module symlink for built-in
> modules.

Sure, I see no problem moving that over when things work out as expected.

Looks all fine from a short look over it. Also thanks for converting
to allocated struct device, and the bus_id conversion.

Kay

Re: [PATCH 5/6] kvm-s390: use register_virtio_root_device()

[Christian Borntraeger]

Am Mittwoch, 10. Dezember 2008 schrieb Mark McLoughlin:
> This is basically a no-op change, since it does exactly the
> same thing as s390_root_dev_register() when the caller isn't
> a module.

Ok, I gave it a short test and it seems to work.
Some comments: I agree with your comment in patch0, that a generic 
device_register_root() function might be useful. 

> --- a/drivers/s390/kvm/kvm_virtio.c
> +++ b/drivers/s390/kvm/kvm_virtio.c
[...]
> -	kvm_root = s390_root_dev_register("kvm_s390");
> +	kvm_root = register_virtio_root_device("kvm_s390");
[...]
> -		s390_root_dev_unregister(kvm_root);
> +		unregister_virtio_root_device(kvm_root);

You can now remove the include <asm/s390_rdev.h>

Re: [PATCH 5/6] kvm-s390: use register_virtio_root_device()

[Cornelia Huck]

On Thu, 11 Dec 2008 10:05:49 +0100,
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> Am Mittwoch, 10. Dezember 2008 schrieb Mark McLoughlin:
> > This is basically a no-op change, since it does exactly the
> > same thing as s390_root_dev_register() when the caller isn't
> > a module.
> 
> Ok, I gave it a short test and it seems to work.
> Some comments: I agree with your comment in patch0, that a generic 
> device_register_root() function might be useful. 

Indeed, if this is a simple replacement, we want a generic function.
I'll take a look at the patches.

> 
> > --- a/drivers/s390/kvm/kvm_virtio.c
> > +++ b/drivers/s390/kvm/kvm_virtio.c
> [...]
> > -	kvm_root = s390_root_dev_register("kvm_s390");
> > +	kvm_root = register_virtio_root_device("kvm_s390");
> [...]
> > -		s390_root_dev_unregister(kvm_root);
> > +		unregister_virtio_root_device(kvm_root);
> 
> You can now remove the include <asm/s390_rdev.h>

Re: [PATCH 2/6] virtio: add register_virtio_root_device()

[Cornelia Huck]

On Wed, 10 Dec 2008 17:45:35 +0000,
Mark McLoughlin <markmc@redhat.com> wrote:

> Add a function to allocate a root device object to group the
> devices from a given virtio implementation.
> 
> Also add a 'module' sysfs symlink to allow so that userspace
> can generically determine which virtio implementation a
> device is associated with. This will be used by Fedora
> mkinitrd to generically determine e.g. that virtio_pci is
> needed to mount a given root filesystem.

Nothing about this is really virtio-specific (just as
s390_root_dev_register() is not really s390-specific), and a 'module'
symlink doesn't really hurt in a generic implementation, even if it is
unneeded. I'm voting to put this in some generic, always built-in code
(or have the users select it) so we could also use it from s390.

> 
> Signed-off-by: Mark McLoughlin <markmc@redhat.com>
> ---
>  drivers/virtio/virtio.c |   71 +++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/virtio.h  |   10 ++++++
>  2 files changed, 81 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
> index 018c070..61e6597 100644
> --- a/drivers/virtio/virtio.c
> +++ b/drivers/virtio/virtio.c
> @@ -1,6 +1,7 @@
>  #include <linux/virtio.h>
>  #include <linux/spinlock.h>
>  #include <linux/virtio_config.h>
> +#include <linux/err.h>
> 
>  /* Unique numbering for virtio devices. */
>  static unsigned int dev_index;
> @@ -200,6 +201,76 @@ void unregister_virtio_device(struct virtio_device *dev)
>  }
>  EXPORT_SYMBOL_GPL(unregister_virtio_device);
> 
> +/* A root device for virtio devices from a given backend.  This makes them
> + * appear as /sys/devices/{name}/0,1,2 not /sys/devices/0,1,2. It also allows
> + * us to have a /sys/devices/{name}/module symlink to the backend module. */
> +struct virtio_root_device
> +{
> +	struct device dev;
> +	struct module *owner;
> +};
> +
> +static struct virtio_root_device *to_virtio_root(struct device *dev)
> +{
> +        return container_of(dev, struct virtio_root_device, dev);
> +}
> +
> +static void release_virtio_root_device(struct device *dev)
> +{
> +	struct virtio_root_device *root = to_virtio_root(dev);
> +	if (root->owner)
> +		sysfs_remove_link(&root->dev.kobj, "module");
> +	kfree(root);
> +}

Can this code be a module? If yes, move the release callback to a
build-in as there are races with release-functions in modules.

> +
> +struct device *__register_virtio_root_device(const char *name,
> +					     struct module *owner)
> +{
> +	struct virtio_root_device *root;
> +	int err = -ENOMEM;
> +
> +	root = kzalloc(sizeof(struct virtio_root_device), GFP_KERNEL);
> +	if (!root)
> +		goto out;
> +
> +	err = dev_set_name(&root->dev, name);
> +	if (err)
> +		goto free_root;
> +
> +	err = device_register(&root->dev);
> +	if (err)
> +		goto free_root;
> +
> +	root->dev.parent  = NULL;
> +	root->dev.release = release_virtio_root_device;

You must set ->release before calling device_register(), and setting
the parent is unneeded.

> +
> +	if (owner) {
> +		struct module_kobject *mk = &owner->mkobj;
> +
> +		err = sysfs_create_link(&root->dev.kobj, &mk->kobj, "module");
> +                if (err) {
> +			device_unregister(&root->dev);
> +			return ERR_PTR(err);
> +		}
> +
> +		root->owner = owner;
> +	}
> +
> +	return &root->dev;
> +
> +free_root:
> +	kfree(root);

You need to call device_put() if you called device_register().

> +out:
> +	return ERR_PTR(err);
> +}
> +EXPORT_SYMBOL_GPL(__register_virtio_root_device);
> +
> +void unregister_virtio_root_device(struct device *root)
> +{
> +	device_unregister(root);
> +}
> +EXPORT_SYMBOL_GPL(unregister_virtio_root_device);
> +
>  static int virtio_init(void)
>  {
>  	if (bus_register(&virtio_bus) != 0)
> diff --git a/include/linux/virtio.h b/include/linux/virtio.h
> index 06005fa..66e6c67 100644
> --- a/include/linux/virtio.h
> +++ b/include/linux/virtio.h
> @@ -93,6 +93,16 @@ struct virtio_device
>  int register_virtio_device(struct virtio_device *dev);
>  void unregister_virtio_device(struct virtio_device *dev);
> 
> +/* A root device is a dummy device used to group virtio devices from each
> + * implementation. */
> +struct device *__register_virtio_root_device(const char *name,
> +					     struct module *owner);
> +static inline struct device *register_virtio_root_device(const char *name)
> +{
> +	return __register_virtio_root_device(name, THIS_MODULE);
> +}
> +void unregister_virtio_root_device(struct device *root);
> +
>  /**
>   * virtio_driver - operations for a virtio I/O driver
>   * @driver: underlying device driver (populate name and owner).
> -- 
> 1.5.4.3

Re: [PATCH 2/6] virtio: add register_virtio_root_device()

[Mark McLoughlin]

On Thu, 2008-12-11 at 13:59 +0100, Cornelia Huck wrote:
> On Wed, 10 Dec 2008 17:45:35 +0000,
> Mark McLoughlin <markmc@redhat.com> wrote:
> 
> > Add a function to allocate a root device object to group the
> > devices from a given virtio implementation.
> > 
> > Also add a 'module' sysfs symlink to allow so that userspace
> > can generically determine which virtio implementation a
> > device is associated with. This will be used by Fedora
> > mkinitrd to generically determine e.g. that virtio_pci is
> > needed to mount a given root filesystem.
> 
> Nothing about this is really virtio-specific (just as
> s390_root_dev_register() is not really s390-specific), and a 'module'
> symlink doesn't really hurt in a generic implementation, even if it is
> unneeded. I'm voting to put this in some generic, always built-in code
> (or have the users select it) so we could also use it from s390.

Okay, coming up ...

> > Signed-off-by: Mark McLoughlin <markmc@redhat.com>
> > ---
> >  drivers/virtio/virtio.c |   71 +++++++++++++++++++++++++++++++++++++++++++++++
> >  include/linux/virtio.h  |   10 ++++++
> >  2 files changed, 81 insertions(+), 0 deletions(-)
> > 
> > diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
> > index 018c070..61e6597 100644
> > --- a/drivers/virtio/virtio.c
> > +++ b/drivers/virtio/virtio.c
> > @@ -1,6 +1,7 @@
> >  #include <linux/virtio.h>
> >  #include <linux/spinlock.h>
> >  #include <linux/virtio_config.h>
> > +#include <linux/err.h>
> > 
> >  /* Unique numbering for virtio devices. */
> >  static unsigned int dev_index;
> > @@ -200,6 +201,76 @@ void unregister_virtio_device(struct virtio_device *dev)
> >  }
> >  EXPORT_SYMBOL_GPL(unregister_virtio_device);
> > 
> > +/* A root device for virtio devices from a given backend.  This makes them
> > + * appear as /sys/devices/{name}/0,1,2 not /sys/devices/0,1,2. It also allows
> > + * us to have a /sys/devices/{name}/module symlink to the backend module. */
> > +struct virtio_root_device
> > +{
> > +	struct device dev;
> > +	struct module *owner;
> > +};
> > +
> > +static struct virtio_root_device *to_virtio_root(struct device *dev)
> > +{
> > +        return container_of(dev, struct virtio_root_device, dev);
> > +}
> > +
> > +static void release_virtio_root_device(struct device *dev)
> > +{
> > +	struct virtio_root_device *root = to_virtio_root(dev);
> > +	if (root->owner)
> > +		sysfs_remove_link(&root->dev.kobj, "module");
> > +	kfree(root);
> > +}
> 
> Can this code be a module? If yes, move the release callback to a
> build-in as there are races with release-functions in modules.

Not sure I fully understand the issue here, but it won't be an problem
with it if we move to driver core.

> > +struct device *__register_virtio_root_device(const char *name,
> > +					     struct module *owner)
> > +{
> > +	struct virtio_root_device *root;
> > +	int err = -ENOMEM;
> > +
> > +	root = kzalloc(sizeof(struct virtio_root_device), GFP_KERNEL);
> > +	if (!root)
> > +		goto out;
> > +
> > +	err = dev_set_name(&root->dev, name);
> > +	if (err)
> > +		goto free_root;
> > +
> > +	err = device_register(&root->dev);
> > +	if (err)
> > +		goto free_root;
> > +
> > +	root->dev.parent  = NULL;
> > +	root->dev.release = release_virtio_root_device;
> 
> You must set ->release before calling device_register(), and setting
> the parent is unneeded.

Okay.

> > +	if (owner) {
> > +		struct module_kobject *mk = &owner->mkobj;
> > +
> > +		err = sysfs_create_link(&root->dev.kobj, &mk->kobj, "module");
> > +                if (err) {
> > +			device_unregister(&root->dev);
> > +			return ERR_PTR(err);
> > +		}
> > +
> > +		root->owner = owner;
> > +	}
> > +
> > +	return &root->dev;
> > +
> > +free_root:
> > +	kfree(root);
> 
> You need to call device_put() if you called device_register().

Oh, I missed that subtlety. So the rules are:

  1) To release before calling device_register(), use kfree()

  2) To release if device_register() failed, put_device()

  3) To release once device_register() succeeds, device_unregister()

Cheers,
Mark.