Automatic loading of xen-evtchn module in Xen 4.0.0?

Automatic loading of xen-evtchn module in Xen 4.0.0?

[Pasi Kärkkäinen]

Hello,

xend automatically mounts /proc/xen, but it doesn't try loading evtchn or xen-evtchn modules.
Should we add automatic loading of these modules? 

xend doesn't start if xen event channel functionality is missing.

Fedora Xen rpms do load modules automatically, like this:

# cat /etc/sysconfig/modules/xen.modules
#!/bin/sh

test -d /proc/xen                               || exit 0
test -f /proc/xen/capabilities                  || mount -t xenfs xen /proc/xen
test -f /proc/xen/capabilities                  || exit 0
grep -q "control_d" /proc/xen/capabilities      || exit 0

for module in xen-evtchn evtchn blkbk netbk; do
        modprobe ${module} >/dev/null 2>&1
done

-- Pasi

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Jeremy Fitzhardinge]

On 03/27/2010 10:32 AM, Pasi Kärkkäinen wrote:
> Hello,
>
> xend automatically mounts /proc/xen, but it doesn't try loading evtchn or xen-evtchn modules.
> Should we add automatic loading of these modules?
>
> xend doesn't start if xen event channel functionality is missing.
>
> Fedora Xen rpms do load modules automatically, like this:
>
> # cat /etc/sysconfig/modules/xen.modules
> #!/bin/sh
>
> test -d /proc/xen                               || exit 0
> test -f /proc/xen/capabilities                  || mount -t xenfs xen /proc/xen
> test -f /proc/xen/capabilities                  || exit 0
> grep -q "control_d" /proc/xen/capabilities      || exit 0
>
> for module in xen-evtchn evtchn blkbk netbk; do
>          modprobe ${module}>/dev/null 2>&1
> done
>    

We should be able to make these mount automatically with udev rules, I 
think, but I'm not sure whether you can do udev rules to mount misc 
devices (which is what /dev/xen/evtchn is).

I think the front/back xenbus and device drivers are still a bit of a 
mess with regard to modules; I have to admit I tend to just compile them 
all in.  It would be nice to get all that properly sorted out so they 
can be sensibly modularized and auto-loaded as required.  (A good first 
step would be to systematically work out what compile combinations 
actually work or fail, hint, hint ;)

     J

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Ian Campbell]

On Sat, 2010-03-27 at 22:18 +0000, Jeremy Fitzhardinge wrote:
> On 03/27/2010 10:32 AM, Pasi Kärkkäinen wrote:
> > Hello,
> >
> > xend automatically mounts /proc/xen, but it doesn't try loading evtchn or xen-evtchn modules.
> > Should we add automatic loading of these modules?
> >
> > xend doesn't start if xen event channel functionality is missing.
> >
> > Fedora Xen rpms do load modules automatically, like this:
> >
> > # cat /etc/sysconfig/modules/xen.modules
> > #!/bin/sh
> >
> > test -d /proc/xen                               || exit 0
> > test -f /proc/xen/capabilities                  || mount -t xenfs xen /proc/xen
> > test -f /proc/xen/capabilities                  || exit 0
> > grep -q "control_d" /proc/xen/capabilities      || exit 0
> >
> > for module in xen-evtchn evtchn blkbk netbk; do
> >          modprobe ${module}>/dev/null 2>&1
> > done
> >    
> 
> We should be able to make these mount automatically with udev rules, I 
> think, but I'm not sure whether you can do udev rules to mount misc 
> devices (which is what /dev/xen/evtchn is).

I think you mean "modprobe misc devices" not mount.

I see various uses of the MODULE_ALIAS_MISCDEV macro in the kernel --
but I'm not sure what actually causes a matching hotplug event to be
generated, other than trying to open a matching device node which with
udev won't exist until the module is loaded.

Perhaps something in the xen core code (or xenbus?) could generate the
necessary hotplug events for the Xen "platform devices" (like the evtchn
one)?

> I think the front/back xenbus and device drivers are still a bit of a 
> mess with regard to modules; I have to admit I tend to just compile them 
> all in.  It would be nice to get all that properly sorted out so they 
> can be sensibly modularized and auto-loaded as required.  (A good first 
> step would be to systematically work out what compile combinations 
> actually work or fail, hint, hint ;)

AFAIK the frontend driver modules work correctly as modules and distros
have been compiling them modular for a while (Debian certainly has).

The situation is certainly not as rosy for *back. It'd certainly be nice
if some (or all) of the xenbus stuff could become modular too although I
would guess there will be a core which must be compiled in.

Ian.

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Konrad Rzeszutek Wilk]

On Tue, Mar 30, 2010 at 10:44:57AM +0100, Ian Campbell wrote:
> On Sat, 2010-03-27 at 22:18 +0000, Jeremy Fitzhardinge wrote:
> > On 03/27/2010 10:32 AM, Pasi Kärkkäinen wrote:
> > > Hello,
> > >
> > > xend automatically mounts /proc/xen, but it doesn't try loading evtchn or xen-evtchn modules.
> > > Should we add automatic loading of these modules?
> > >
> > > xend doesn't start if xen event channel functionality is missing.
> > >
> > > Fedora Xen rpms do load modules automatically, like this:
> > >
> > > # cat /etc/sysconfig/modules/xen.modules
> > > #!/bin/sh
> > >
> > > test -d /proc/xen                               || exit 0
> > > test -f /proc/xen/capabilities                  || mount -t xenfs xen /proc/xen
> > > test -f /proc/xen/capabilities                  || exit 0
> > > grep -q "control_d" /proc/xen/capabilities      || exit 0
> > >
> > > for module in xen-evtchn evtchn blkbk netbk; do
> > >          modprobe ${module}>/dev/null 2>&1
> > > done
> > >    
> > 
> > We should be able to make these mount automatically with udev rules, I 
> > think, but I'm not sure whether you can do udev rules to mount misc 
> > devices (which is what /dev/xen/evtchn is).
> 
> I think you mean "modprobe misc devices" not mount.
> 
> I see various uses of the MODULE_ALIAS_MISCDEV macro in the kernel --
> but I'm not sure what actually causes a matching hotplug event to be
> generated, other than trying to open a matching device node which with
> udev won't exist until the module is loaded.

One idea that I submitted to dracut mailing list (the initramfs
creation/running tool under FC12 and above) was to leverage 'xen-detect'
and load the appropiate modules. This way during bootup, bamm, you get
all your modules in.

But they righly suggested that there should be an uevent to kick udev to
start loading them and use the MODULE_ALIAS. But to use MODULE_ALIAS,
you have to get an uevent, and to get an uevent, you have to load the
module, so it looks to be a chicken-and-egg problem.

> 
> Perhaps something in the xen core code (or xenbus?) could generate the
> necessary hotplug events for the Xen "platform devices" (like the evtchn
> one)?

But I think that comes back to the chicken-and-egg. If you have all of
the front-end drivers compiled as modules, how would they (front-end
drivers) generate the hotplug events when they are not loaded?

Perhaps have the xenbus try to load the modules if it detects that is
running under Xen? Something similar to how the kernel tries to mount
various filesystem modules when mounting a disk?

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Ian Campbell]

On Tue, 2010-03-30 at 15:22 +0100, Konrad Rzeszutek Wilk wrote:
> On Tue, Mar 30, 2010 at 10:44:57AM +0100, Ian Campbell wrote:
> > On Sat, 2010-03-27 at 22:18 +0000, Jeremy Fitzhardinge wrote:
> > > On 03/27/2010 10:32 AM, Pasi Kärkkäinen wrote:
> > > > Hello,
> > > >
> > > > xend automatically mounts /proc/xen, but it doesn't try loading evtchn or xen-evtchn modules.
> > > > Should we add automatic loading of these modules?
> > > >
> > > > xend doesn't start if xen event channel functionality is missing.
> > > >
> > > > Fedora Xen rpms do load modules automatically, like this:
> > > >
> > > > # cat /etc/sysconfig/modules/xen.modules
> > > > #!/bin/sh
> > > >
> > > > test -d /proc/xen                               || exit 0
> > > > test -f /proc/xen/capabilities                  || mount -t xenfs xen /proc/xen
> > > > test -f /proc/xen/capabilities                  || exit 0
> > > > grep -q "control_d" /proc/xen/capabilities      || exit 0
> > > >
> > > > for module in xen-evtchn evtchn blkbk netbk; do
> > > >          modprobe ${module}>/dev/null 2>&1
> > > > done
> > > >    
> > > 
> > > We should be able to make these mount automatically with udev rules, I 
> > > think, but I'm not sure whether you can do udev rules to mount misc 
> > > devices (which is what /dev/xen/evtchn is).
> > 
> > I think you mean "modprobe misc devices" not mount.
> > 
> > I see various uses of the MODULE_ALIAS_MISCDEV macro in the kernel --
> > but I'm not sure what actually causes a matching hotplug event to be
> > generated, other than trying to open a matching device node which with
> > udev won't exist until the module is loaded.
> 
> One idea that I submitted to dracut mailing list (the initramfs
> creation/running tool under FC12 and above) was to leverage 'xen-detect'
> and load the appropiate modules. This way during bootup, bamm, you get
> all your modules in.
> 
> But they righly suggested that there should be an uevent to kick udev to
> start loading them and use the MODULE_ALIAS. But to use MODULE_ALIAS,
> you have to get an uevent, and to get an uevent, you have to load the
> module, so it looks to be a chicken-and-egg problem.
> 
> > 
> > Perhaps something in the xen core code (or xenbus?) could generate the
> > necessary hotplug events for the Xen "platform devices" (like the evtchn
> > one)?
> 
> But I think that comes back to the chicken-and-egg. If you have all of
> the front-end drivers compiled as modules, how would they (front-end
> drivers) generate the hotplug events when they are not loaded?

The xenbus infrastructure sees e.g. the /vif/ node appear in xenstore
and generates an event for "xen:vif" which xen-netfront declares as an
alias so the module will be loaded.

In other words it is up to the _bus_ implementation not the drivers to
generate the events to cause the drivers to be loaded. 

For frontend drivers this works today, AFAIK.

The issue with the evtchn driver is that there is no bus as such. We
could potentially make use of the existing platform device
infrastructure which is kind of like a bus for system devices which
aren't really on a bus.

I guess the xenbus driver itself falls into the same sort of category as
the evtchn driver.

Ian.

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Jeremy Fitzhardinge]

On 03/30/2010 02:44 AM, Ian Campbell wrote:
> On Sat, 2010-03-27 at 22:18 +0000, Jeremy Fitzhardinge wrote:
>    
>> On 03/27/2010 10:32 AM, Pasi Kärkkäinen wrote:
>>      
>>> Hello,
>>>
>>> xend automatically mounts /proc/xen, but it doesn't try loading evtchn or xen-evtchn modules.
>>> Should we add automatic loading of these modules?
>>>
>>> xend doesn't start if xen event channel functionality is missing.
>>>
>>> Fedora Xen rpms do load modules automatically, like this:
>>>
>>> # cat /etc/sysconfig/modules/xen.modules
>>> #!/bin/sh
>>>
>>> test -d /proc/xen                               || exit 0
>>> test -f /proc/xen/capabilities                  || mount -t xenfs xen /proc/xen
>>> test -f /proc/xen/capabilities                  || exit 0
>>> grep -q "control_d" /proc/xen/capabilities      || exit 0
>>>
>>> for module in xen-evtchn evtchn blkbk netbk; do
>>>           modprobe ${module}>/dev/null 2>&1
>>> done
>>>
>>>        
>> We should be able to make these mount automatically with udev rules, I
>> think, but I'm not sure whether you can do udev rules to mount misc
>> devices (which is what /dev/xen/evtchn is).
>>      
> I think you mean "modprobe misc devices" not mount.
>    

Yep.

> I see various uses of the MODULE_ALIAS_MISCDEV macro in the kernel --
> but I'm not sure what actually causes a matching hotplug event to be
> generated, other than trying to open a matching device node which with
> udev won't exist until the module is loaded.
>
> Perhaps something in the xen core code (or xenbus?) could generate the
> necessary hotplug events for the Xen "platform devices" (like the evtchn
> one)?
>
>    
>> I think the front/back xenbus and device drivers are still a bit of a
>> mess with regard to modules; I have to admit I tend to just compile them
>> all in.  It would be nice to get all that properly sorted out so they
>> can be sensibly modularized and auto-loaded as required.  (A good first
>> step would be to systematically work out what compile combinations
>> actually work or fail, hint, hint ;)
>>      
> AFAIK the frontend driver modules work correctly as modules and distros
> have been compiling them modular for a while (Debian certainly has).
>    

Yes, I think frontend drivers are OK.  But the various xenbus bits seem 
a bit fragile.

> The situation is certainly not as rosy for *back. It'd certainly be nice
> if some (or all) of the xenbus stuff could become modular too although I
> would guess there will be a core which must be compiled in.
>    

In practice there's no need to get very modular.  There's some value in 
not loading all that backend support code unless we're actually dom0, 
but I don't think there's much value in getting too fine-grained beyond 
that.  The actual drivers themselves are fairly ordinary, so they will 
probably just work if we can make xenbus generate the right request 
events.  Hm, I guess that would be the tool stack setting things up for 
a new domain, and requesting backend devices in the process, no?

But it doesn't help for things like evtchn and gntdev.  But there's no 
huge burden in just making the toolstack startup scripts do modprobes 
for them, just as it should be mounting /proc/xen.

     J

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Ian Campbell]

On Tue, 2010-03-30 at 23:47 +0100, Jeremy Fitzhardinge wrote:
> 
> In practice there's no need to get very modular.  There's some value
> in 
> not loading all that backend support code unless we're actually dom0, 
> but I don't think there's much value in getting too fine-grained
> beyond that.  The actual drivers themselves are fairly ordinary, so
> they will probably just work if we can make xenbus generate the right
> request events.  Hm, I guess that would be the tool stack setting
> things up for a new domain, and requesting backend devices in the
> process, no? 

What happens with frontend hotplug, and should happen with backends too,
is that the xenbus core watches the "device" (or "backend") node in
xenstore (relative to the domain's root path "/local/domain/<X>/"). The
tools then create the necessary nodes to create the device which
triggers the xenstore watch, e.g. on a path starting with "vif", which
causes xenbus to generate a uevent with the necessary MODALIAS, see
drivers/xen/xenbus/xenbus_probe.c:xenbus_uevent().

The only problem with this is getting the initial xenbus-backend.ko
loaded so that it can do the watch -- perhaps that falls in the same
boat as evtchn and gntdev etc.

Ian.

Re: Automatic loading of xen-evtchn module in Xen 4.0.0?

[Konrad Rzeszutek Wilk]

On Wed, Mar 31, 2010 at 09:40:40AM +0100, Ian Campbell wrote:
> On Tue, 2010-03-30 at 23:47 +0100, Jeremy Fitzhardinge wrote:
> > 
> > In practice there's no need to get very modular.  There's some value
> > in 
> > not loading all that backend support code unless we're actually dom0, 
> > but I don't think there's much value in getting too fine-grained
> > beyond that.  The actual drivers themselves are fairly ordinary, so
> > they will probably just work if we can make xenbus generate the right
> > request events.  Hm, I guess that would be the tool stack setting
> > things up for a new domain, and requesting backend devices in the
> > process, no? 
> 
> What happens with frontend hotplug, and should happen with backends too,
> is that the xenbus core watches the "device" (or "backend") node in
> xenstore (relative to the domain's root path "/local/domain/<X>/"). The
> tools then create the necessary nodes to create the device which
> triggers the xenstore watch, e.g. on a path starting with "vif", which
> causes xenbus to generate a uevent with the necessary MODALIAS, see
> drivers/xen/xenbus/xenbus_probe.c:xenbus_uevent().
> 
> The only problem with this is getting the initial xenbus-backend.ko
> loaded so that it can do the watch -- perhaps that falls in the same
> boat as evtchn and gntdev etc.

.. which goes back to what I suggested some time ago, that might do the
trick. That is to make the owner of /sys/hypervisor do a kobject_uevent
and have an udev rule that would listen to this and load the
xenbus-backend.ko. That should start the avalanche of Xen backend module
loading, I think?