2.6.37 PV on HVM and emul_unplug=unnecessary

2.6.37 PV on HVM and emul_unplug=unnecessary

[Alex Bligh]

On modern mainline, when a PV on HVM kernel boots, it will detect the
ability to PCI unplug the PCI SD devices, so to avoid producing
/dev/xvdX and /dev/sdX referring to the same device. On Xen Dom0 prior
to 3.4.something, this fails because PCI unplug is not supported.
The check can be worked around by using emulunplug=unnecessary, but
this is not optimal as
 a) you still end up with 2 devices; and
 b) it requires modifying the kernel boot line, which is not always
    easy (e.g. if your bootloader is supplied by a third party IAAS
    provider and you wish to use a mainline kernel).

Another approach would be to stop sd_mod loading another way
(e.g. blacklist sd_mod). This does not work well as many kernels have
sd_mod built in.

I am trying to think of a non-intrusive way to prevent sd_mod initialising
if we are doing PV on HVM and PCI unplug is not supported (other than
upgrade Xen). One possibility would be to manually register dummy
block major device with the same major number as used by sd. This would
cause register_blkdev to return EBUSY and the module not to init. However,
it would leave the dummy block device around. Another more intrusive
method would simply be to put a check into sd_mod. Yet more intrusive
would be to modify the entire do_initcalls/module_init subsystem
to provide a generalised method for one bit of code to disable another.

Has someone already looked into this and concluded it's impossible? Or
is this worth a little time?


-- 
Alex Bligh

Re: 2.6.37 PV on HVM and emul_unplug=unnecessary

[Paolo Bonzini]

On 01/12/2011 12:45 PM, Alex Bligh wrote:
> On modern mainline, when a PV on HVM kernel boots, it will detect the
> ability to PCI unplug the PCI SD devices, so to avoid producing
> /dev/xvdX and /dev/sdX referring to the same device. On Xen Dom0 prior
> to 3.4.something, this fails because PCI unplug is not supported.
> The check can be worked around by using emulunplug=unnecessary, but
> this is not optimal as
> a) you still end up with 2 devices; and
> b) it requires modifying the kernel boot line, which is not always
> easy (e.g. if your bootloader is supplied by a third party IAAS
> provider and you wish to use a mainline kernel).
>
> Another approach would be to stop sd_mod loading another way
> (e.g. blacklist sd_mod). This does not work well as many kernels have
> sd_mod built in.
>
> I am trying to think of a non-intrusive way to prevent sd_mod initialising
> if we are doing PV on HVM and PCI unplug is not supported (other than
> upgrade Xen). One possibility would be to manually register dummy
> block major device with the same major number as used by sd. This would
> cause register_blkdev to return EBUSY and the module not to init. However,
> it would leave the dummy block device around.

On Windows it is possible to add a "filter driver" above the PCI and IDE 
buses and hide the emulated devices this way.  It doesn't leave a dummy 
disk, but it does leave a dummy device in the PCI tree.  It's a kind of 
kernel-level blacklist.  Also, it couldn't be an external module if 
sd_mod is built in, which makes it much less interesting.

> Has someone already looked into this and concluded it's impossible? Or
> is this worth a little time?

Something similar was done in the 2.6.18 "XenoLinux" kernels, but it's 
ugly and it touches into areas of the kernell that Xen support shouldn't 
affect.

If your "Xen Dom0 prior to 3.4.something" is RHEL/CentOS, the 5.6 
version will include a backport of PCI unplug support.  Then you only 
need to bug your cloud provider. :)

Paolo

Re: 2.6.37 PV on HVM and emul_unplug=unnecessary

[Alex Bligh]

--On 12 January 2011 14:04:57 +0100 Paolo Bonzini <pbonzini@redhat.com> 
wrote:

> Something similar was done in the 2.6.18 "XenoLinux" kernels, but it's
> ugly and it touches into areas of the kernell that Xen support shouldn't
> affect.
>
> If your "Xen Dom0 prior to 3.4.something" is RHEL/CentOS, the 5.6 version
> will include a backport of PCI unplug support.  Then you only need to bug
> your cloud provider. :)

I am the cloud provider. The real fix is to move to Xen 4.0.1. Part of
the issue, however, is moving to 4.0.1 will make a subset of guests
behave differently from others, as suddenly their /dev/sda will disappear.
We know there are people who don't use standard installs so possibly
unknowingly are not using pv drivers (e.g. they have /dev/sdaX in their
fstab). It would be nice if I could separate this change from the rest
of the 4.0.1 upgrade, as this would make a 4.0.1 upgrade less scary.
It would be oh-so-much-easier-for-me if /dev/sdaX disappeared when
they upgraded their kernel, rather than when I make a later change.

However, if could point me at a working changeset which contains the
backport, that would be really useful. Stefano posted one which I
think he marked as "untested".

-- 
Alex Bligh