xen_emul_unplug on xen 4.1, HVM guest 2.6.38

xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Alex Bligh]

I am running Xen 4 and want users to be able to install arbitrary HVM
images. We want both PV and emulated drivers to appear. The reason
for this is for compatibility with images that were prepared for
Xen 3.3, which assume emulated devices exist at boot time (particularly
for running the boot loader).

If I do not specify on the boot line xen_emul_unplug=never, then
under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
unplugs the non-PV devices, which is undesirable in my application.

I cannot arbitrarily change the kernel command line because it is
in effect set by the user's image.

Therefore what I need to do is prevent Xen4 offering the facility
to unplug the devices in the first place. Setting "(unpluggable 0)"
in the config for the emulated NICs and disks does not appear
to prevent them getting unplugged; what may be happening is that
domU tries to unplug them (but fails) but still doesn't use them.
Is there a way to signal from dom0 that the kernel really shouldn't
unplug this stuff.

-- 
Alex Bligh

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Konrad Rzeszutek Wilk]

On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
> I am running Xen 4 and want users to be able to install arbitrary HVM
> images. We want both PV and emulated drivers to appear. The reason
> for this is for compatibility with images that were prepared for
> Xen 3.3, which assume emulated devices exist at boot time (particularly
> for running the boot loader).
> 
> If I do not specify on the boot line xen_emul_unplug=never, then
> under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
> unplugs the non-PV devices, which is undesirable in my application.
> 
> I cannot arbitrarily change the kernel command line because it is
> in effect set by the user's image.
> 
> Therefore what I need to do is prevent Xen4 offering the facility
> to unplug the devices in the first place. Setting "(unpluggable 0)"
> in the config for the emulated NICs and disks does not appear
> to prevent them getting unplugged; what may be happening is that
> domU tries to unplug them (but fails) but still doesn't use them.
> Is there a way to signal from dom0 that the kernel really shouldn't
> unplug this stuff.

xen_pci_platform=0 in your guest config should do it.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Pasi Kärkkäinen]

On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
> > I am running Xen 4 and want users to be able to install arbitrary HVM
> > images. We want both PV and emulated drivers to appear. The reason
> > for this is for compatibility with images that were prepared for
> > Xen 3.3, which assume emulated devices exist at boot time (particularly
> > for running the boot loader).
> > 
> > If I do not specify on the boot line xen_emul_unplug=never, then
> > under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
> > unplugs the non-PV devices, which is undesirable in my application.
> > 
> > I cannot arbitrarily change the kernel command line because it is
> > in effect set by the user's image.
> > 
> > Therefore what I need to do is prevent Xen4 offering the facility
> > to unplug the devices in the first place. Setting "(unpluggable 0)"
> > in the config for the emulated NICs and disks does not appear
> > to prevent them getting unplugged; what may be happening is that
> > domU tries to unplug them (but fails) but still doesn't use them.
> > Is there a way to signal from dom0 that the kernel really shouldn't
> > unplug this stuff.
> 
> xen_pci_platform=0 in your guest config should do it.
> 

It's actually called:
xen_platform_pci=0

:)

-- Pasi

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Alex Bligh]

Pasi, Conrad,

--On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:
>> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
>> > I am running Xen 4 and want users to be able to install arbitrary HVM
>> > images. We want both PV and emulated drivers to appear. The reason
>> > for this is for compatibility with images that were prepared for
>> > Xen 3.3, which assume emulated devices exist at boot time (particularly
>> > for running the boot loader).
>> >
>> > If I do not specify on the boot line xen_emul_unplug=never, then
>> > under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
>> > unplugs the non-PV devices, which is undesirable in my application.
>> >
>> > I cannot arbitrarily change the kernel command line because it is
>> > in effect set by the user's image.
>> >
>> > Therefore what I need to do is prevent Xen4 offering the facility
>> > to unplug the devices in the first place. Setting "(unpluggable 0)"
>> > in the config for the emulated NICs and disks does not appear
>> > to prevent them getting unplugged; what may be happening is that
>> > domU tries to unplug them (but fails) but still doesn't use them.
>> > Is there a way to signal from dom0 that the kernel really shouldn't
>> > unplug this stuff.
>>
>> xen_pci_platform=0 in your guest config should do it.
>>
>
> It's actually called:
> xen_platform_pci=0

Thanks. Just to confirm, that will still allow HVM PV drivers to
operate (as well)?

-- 
Alex Bligh

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Ian Campbell]

On Wed, 2011-10-26 at 15:47 +0100, Alex Bligh wrote:
> Pasi, Conrad,
> 
> --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote:
> 
> > On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:
> >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
> >> > I am running Xen 4 and want users to be able to install arbitrary HVM
> >> > images. We want both PV and emulated drivers to appear. The reason
> >> > for this is for compatibility with images that were prepared for
> >> > Xen 3.3, which assume emulated devices exist at boot time (particularly
> >> > for running the boot loader).
> >> >
> >> > If I do not specify on the boot line xen_emul_unplug=never, then
> >> > under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
> >> > unplugs the non-PV devices, which is undesirable in my application.
> >> >
> >> > I cannot arbitrarily change the kernel command line because it is
> >> > in effect set by the user's image.
> >> >
> >> > Therefore what I need to do is prevent Xen4 offering the facility
> >> > to unplug the devices in the first place. Setting "(unpluggable 0)"
> >> > in the config for the emulated NICs and disks does not appear
> >> > to prevent them getting unplugged; what may be happening is that
> >> > domU tries to unplug them (but fails) but still doesn't use them.
> >> > Is there a way to signal from dom0 that the kernel really shouldn't
> >> > unplug this stuff.
> >>
> >> xen_pci_platform=0 in your guest config should do it.
> >>
> >
> > It's actually called:
> > xen_platform_pci=0
> 
> Thanks. Just to confirm, that will still allow HVM PV drivers to
> operate (as well)?

No, this will disable PV drivers.

The decision to unplug is a kernel side decision and in PVHVM Linux
kernels is not currently possible to have both types of devices by
default due to the risk of dataloss if the guest is not correctly
configured (i.e. the kernel can't tell if it is mounting the same
filesystem via two paths). The xen_emul_unplug option is the current way
you can override this once you have confirmed that your guest
configuration is not dangerous. I'm afraid this necessarily involves
guest config and guest admin interaction.

In principal we might be able to extend the unplug protocol (which would
involve patches to qemu, the kernel(s) and the toolstack) to allow
devices to be marked as being not necessary to unplug. Someone would
have to send patches though and it would be opening up a way for people
to lose data so we'd need to be careful.

I'm sure that the unplug protocol is documented somewhere in the source
tree but I can't for the life of me find it :-(

Ian.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Pasi Kärkkäinen]

On Wed, Oct 26, 2011 at 03:56:44PM +0100, Ian Campbell wrote:
> On Wed, 2011-10-26 at 15:47 +0100, Alex Bligh wrote:
> > Pasi, Conrad,
> > 
> > --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > 
> > > On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:
> > >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
> > >> > I am running Xen 4 and want users to be able to install arbitrary HVM
> > >> > images. We want both PV and emulated drivers to appear. The reason
> > >> > for this is for compatibility with images that were prepared for
> > >> > Xen 3.3, which assume emulated devices exist at boot time (particularly
> > >> > for running the boot loader).
> > >> >
> > >> > If I do not specify on the boot line xen_emul_unplug=never, then
> > >> > under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
> > >> > unplugs the non-PV devices, which is undesirable in my application.
> > >> >
> > >> > I cannot arbitrarily change the kernel command line because it is
> > >> > in effect set by the user's image.
> > >> >
> > >> > Therefore what I need to do is prevent Xen4 offering the facility
> > >> > to unplug the devices in the first place. Setting "(unpluggable 0)"
> > >> > in the config for the emulated NICs and disks does not appear
> > >> > to prevent them getting unplugged; what may be happening is that
> > >> > domU tries to unplug them (but fails) but still doesn't use them.
> > >> > Is there a way to signal from dom0 that the kernel really shouldn't
> > >> > unplug this stuff.
> > >>
> > >> xen_pci_platform=0 in your guest config should do it.
> > >>
> > >
> > > It's actually called:
> > > xen_platform_pci=0
> > 
> > Thanks. Just to confirm, that will still allow HVM PV drivers to
> > operate (as well)?
> 
> No, this will disable PV drivers.
> 
> The decision to unplug is a kernel side decision and in PVHVM Linux
> kernels is not currently possible to have both types of devices by
> default due to the risk of dataloss if the guest is not correctly
> configured (i.e. the kernel can't tell if it is mounting the same
> filesystem via two paths). The xen_emul_unplug option is the current way
> you can override this once you have confirmed that your guest
> configuration is not dangerous. I'm afraid this necessarily involves
> guest config and guest admin interaction.
> 
> In principal we might be able to extend the unplug protocol (which would
> involve patches to qemu, the kernel(s) and the toolstack) to allow
> devices to be marked as being not necessary to unplug. Someone would
> have to send patches though and it would be opening up a way for people
> to lose data so we'd need to be careful.
> 
> I'm sure that the unplug protocol is documented somewhere in the source
> tree but I can't for the life of me find it :-(
> 

"disable qemu PCI devices in HVM domains":
http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=e7911109f4321e9ba0cc56a253b653600aa46bea

"disable qemu PCI devices in HVM domains - protocol document":
http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=bc5c93ede039525765a7aa98e1da3386efa1465f


-- Pasi

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Ian Campbell]

On Wed, 2011-10-26 at 16:59 +0100, Pasi Kärkkäinen wrote:
> On Wed, Oct 26, 2011 at 03:56:44PM +0100, Ian Campbell wrote:
> > On Wed, 2011-10-26 at 15:47 +0100, Alex Bligh wrote:
> > > Pasi, Conrad,
> > > 
> > > --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > 
> > > > On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:
> > > >> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
> > > >> > I am running Xen 4 and want users to be able to install arbitrary HVM
> > > >> > images. We want both PV and emulated drivers to appear. The reason
> > > >> > for this is for compatibility with images that were prepared for
> > > >> > Xen 3.3, which assume emulated devices exist at boot time (particularly
> > > >> > for running the boot loader).
> > > >> >
> > > >> > If I do not specify on the boot line xen_emul_unplug=never, then
> > > >> > under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
> > > >> > unplugs the non-PV devices, which is undesirable in my application.
> > > >> >
> > > >> > I cannot arbitrarily change the kernel command line because it is
> > > >> > in effect set by the user's image.
> > > >> >
> > > >> > Therefore what I need to do is prevent Xen4 offering the facility
> > > >> > to unplug the devices in the first place. Setting "(unpluggable 0)"
> > > >> > in the config for the emulated NICs and disks does not appear
> > > >> > to prevent them getting unplugged; what may be happening is that
> > > >> > domU tries to unplug them (but fails) but still doesn't use them.
> > > >> > Is there a way to signal from dom0 that the kernel really shouldn't
> > > >> > unplug this stuff.
> > > >>
> > > >> xen_pci_platform=0 in your guest config should do it.
> > > >>
> > > >
> > > > It's actually called:
> > > > xen_platform_pci=0
> > > 
> > > Thanks. Just to confirm, that will still allow HVM PV drivers to
> > > operate (as well)?
> > 
> > No, this will disable PV drivers.
> > 
> > The decision to unplug is a kernel side decision and in PVHVM Linux
> > kernels is not currently possible to have both types of devices by
> > default due to the risk of dataloss if the guest is not correctly
> > configured (i.e. the kernel can't tell if it is mounting the same
> > filesystem via two paths). The xen_emul_unplug option is the current way
> > you can override this once you have confirmed that your guest
> > configuration is not dangerous. I'm afraid this necessarily involves
> > guest config and guest admin interaction.
> > 
> > In principal we might be able to extend the unplug protocol (which would
> > involve patches to qemu, the kernel(s) and the toolstack) to allow
> > devices to be marked as being not necessary to unplug. Someone would
> > have to send patches though and it would be opening up a way for people
> > to lose data so we'd need to be careful.
> > 
> > I'm sure that the unplug protocol is documented somewhere in the source
> > tree but I can't for the life of me find it :-(
> > 
> 
> "disable qemu PCI devices in HVM domains":
> http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=e7911109f4321e9ba0cc56a253b653600aa46bea
> 
> "disable qemu PCI devices in HVM domains - protocol document":
> http://xenbits.xen.org/gitweb/?p=qemu-xen-unstable.git;a=commit;h=bc5c93ede039525765a7aa98e1da3386efa1465f

Thanks, I found it and am just about to send a patch to add it to
xen-unstable.hg...

Ian.

> 
> 
> -- Pasi
> 
> 

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Alex Bligh]

Ian,

> No, this will disable PV drivers.

I can confirm that our testing illustrates this :-(

> The decision to unplug is a kernel side decision and in PVHVM Linux
> kernels is not currently possible to have both types of devices by
> default due to the risk of dataloss if the guest is not correctly
> configured (i.e. the kernel can't tell if it is mounting the same
> filesystem via two paths). The xen_emul_unplug option is the current way
> you can override this once you have confirmed that your guest
> configuration is not dangerous. I'm afraid this necessarily involves
> guest config and guest admin interaction.
>
> In principal we might be able to extend the unplug protocol (which would
> involve patches to qemu, the kernel(s) and the toolstack) to allow
> devices to be marked as being not necessary to unplug. Someone would
> have to send patches though and it would be opening up a way for people
> to lose data so we'd need to be careful.
>
> I'm sure that the unplug protocol is documented somewhere in the source
> tree but I can't for the life of me find it :-(

So, the issue is this. We have thousands (literally) of disks in use
by third parties on xen 3.3. Some are Windows, some are ancient linux,
some are modern linux, etc. The hypervisor has no way of whether the
images are going to use /dev/sda or /dev/xvda (i.e. PV or emulated)
drivers. Indeed the most common linux case is that grub uses the
emulated devices to load the kernel, then uses /dev/xvda as a root
device, i.e. both are used (but not simultaneously).

We need to have the xen pci stuff on, so PV drivers operate (in both
new and old kernels). But as modern linux kernels detect the unplug 
functionality, they will unplug the emulated devices and then fail to
boot because (for instance) under Xen3.3 using /dev/sdaX to access
(say) your /boot partition worked perfectly well. What we need is a
switch to revert to the old Xen3.3 (pre-unplug) behaviour, so any
Linux kernel will see the same set of devices. I cannot believe this
is a unique requirement for people attempting to do a Xen3.3 to
Xen4.1 migration.

I think this is in xen_unplug_emulated_devices() in
arch/x86/xen/platform-pci-unplug.c

This uses check_platform_magic(), which I have appended. In order
to avoid unplugging (without relying on the boot line), I need
this to return a non-zero value (XEN_PLATFORM_ERR_MAGIC is
irrelevant as xen_emul_unplug is 0 by assumption).

I can achieve that by either (a) returning a bad magic number,
(b) making the host 'blacklist' the product (how does that work?)
or (c) using a protocol value of (say) 0. I take it Xen 3.3 simply
returns a bad magic number as I don't think XEN_IOPORT_MAGIC existed
in 3.3. As far as I can tell, XEN_IOPORT_MAGIC is only use for
PCI unplug.

So, is the correct approach to disable XEN_IOPORT_MAGIC (or rather
make it return a different value) depending on a configuration option?
If so, I am happy to submit a patch to do that. Or can I do this
without a patch by "blacklisting" everything? (not sure how that is done).

Out of interest, with a default guest Ubuntu Natty install CD, using the
default Xen 4.1 settings, we are seeing the guest (a) unplugging the
emulated devices (fine), then (b) failing to find the emulated devices,
and (c) the install failing. Is that to be expected?

-- 
Alex Bligh


static int __init check_platform_magic(void)
{
        short magic;
        char protocol;

        magic = inw(XEN_IOPORT_MAGIC);
        if (magic != XEN_IOPORT_MAGIC_VAL) {
                printk(KERN_ERR "Xen Platform PCI: unrecognised magic 
value\n");
                return XEN_PLATFORM_ERR_MAGIC;
        }

        protocol = inb(XEN_IOPORT_PROTOVER);

        printk(KERN_DEBUG "Xen Platform PCI: I/O protocol version %d\n",
                        protocol);

        switch (protocol) {
        case 1:
                outw(XEN_IOPORT_LINUX_PRODNUM, XEN_IOPORT_PRODNUM);
                outl(XEN_IOPORT_LINUX_DRVVER, XEN_IOPORT_DRVVER);
                if (inw(XEN_IOPORT_MAGIC) != XEN_IOPORT_MAGIC_VAL) {
                        printk(KERN_ERR "Xen Platform: blacklisted by 
host\n");
                        return XEN_PLATFORM_ERR_BLACKLIST;
                }
                break;
        default:
                printk(KERN_WARNING "Xen Platform PCI: unknown I/O protocol 
version");
                return XEN_PLATFORM_ERR_PROTOCOL;
        }

        return 0;
}

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Ian Campbell]

On Wed, 2011-10-26 at 17:25 +0100, Alex Bligh wrote:
> Ian,
> 
> > No, this will disable PV drivers.
> 
> I can confirm that our testing illustrates this :-(
> 
> > The decision to unplug is a kernel side decision and in PVHVM Linux
> > kernels is not currently possible to have both types of devices by
> > default due to the risk of dataloss if the guest is not correctly
> > configured (i.e. the kernel can't tell if it is mounting the same
> > filesystem via two paths). The xen_emul_unplug option is the current way
> > you can override this once you have confirmed that your guest
> > configuration is not dangerous. I'm afraid this necessarily involves
> > guest config and guest admin interaction.
> >
> > In principal we might be able to extend the unplug protocol (which would
> > involve patches to qemu, the kernel(s) and the toolstack) to allow
> > devices to be marked as being not necessary to unplug. Someone would
> > have to send patches though and it would be opening up a way for people
> > to lose data so we'd need to be careful.
> >
> > I'm sure that the unplug protocol is documented somewhere in the source
> > tree but I can't for the life of me find it :-(
> 
> So, the issue is this. We have thousands (literally) of disks in use
> by third parties on xen 3.3. Some are Windows, some are ancient linux,
> some are modern linux, etc. The hypervisor has no way of whether the
> images are going to use /dev/sda or /dev/xvda (i.e. PV or emulated)
> drivers. Indeed the most common linux case is that grub uses the
> emulated devices to load the kernel, then uses /dev/xvda as a root
> device, i.e. both are used (but not simultaneously).
> 
> We need to have the xen pci stuff on, so PV drivers operate (in both
> new and old kernels). But as modern linux kernels detect the unplug 
> functionality, they will unplug the emulated devices and then fail to
> boot because (for instance) under Xen3.3 using /dev/sdaX to access
> (say) your /boot partition worked perfectly well. What we need is a
> switch to revert to the old Xen3.3 (pre-unplug) behaviour, so any
> Linux kernel will see the same set of devices. I cannot believe this
> is a unique requirement for people attempting to do a Xen3.3 to
> Xen4.1 migration.

I'm a bit fuzzy on the details but I'm not sure what this has to do with
the host, the device naming and behaviour on unplug are kernel side
things, I'd expect that if /dev/sdaX as /boot worked on 3.3 it'll work
on 4.1 too. (I believe you that it doesn't work, I'm just wondering
aloud what I'm missing).

Can you give us the specifics of a setup which fails, e.g. a complete
guest cfg file, the kernel version, command line options, /etc/fstab,
dmesg etc.

> I think this is in xen_unplug_emulated_devices() in
> arch/x86/xen/platform-pci-unplug.c
> 
> This uses check_platform_magic(), which I have appended. In order
> to avoid unplugging (without relying on the boot line), I need
> this to return a non-zero value (XEN_PLATFORM_ERR_MAGIC is
> irrelevant as xen_emul_unplug is 0 by assumption).
> 
> I can achieve that by either (a) returning a bad magic number,
> (b) making the host 'blacklist' the product (how does that work?)
> or (c) using a protocol value of (say) 0. I take it Xen 3.3 simply
> returns a bad magic number as I don't think XEN_IOPORT_MAGIC existed
> in 3.3. As far as I can tell, XEN_IOPORT_MAGIC is only use for
> PCI unplug.
>
> So, is the correct approach to disable XEN_IOPORT_MAGIC (or rather
> make it return a different value) depending on a configuration option?
> If so, I am happy to submit a patch to do that. Or can I do this
> without a patch by "blacklisting" everything? (not sure how that is done).

Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on
the kernel side is what I was missing.

It might make sense to have a guest level config option which disables
these magic ports, i.e. makes them return 0xffff like they would have
done in 3.3 (I think 0xffff is what you'll get from an invalid port in
general).

> Out of interest, with a default guest Ubuntu Natty install CD, using the
> default Xen 4.1 settings, we are seeing the guest (a) unplugging the
> emulated devices (fine), then (b) failing to find the emulated devices,
> and (c) the install failing. Is that to be expected?

Sounds like an Ubuntu bug to me, but I don't follow Ubuntu closely
enough to know if it is known or not.

Ian.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefan Bader]

On 26.10.2011 18:43, Ian Campbell wrote:
> On Wed, 2011-10-26 at 17:25 +0100, Alex Bligh wrote:
>> Ian,
>>
>>> No, this will disable PV drivers.
>>
>> I can confirm that our testing illustrates this :-(
>>
>>> The decision to unplug is a kernel side decision and in PVHVM Linux
>>> kernels is not currently possible to have both types of devices by
>>> default due to the risk of dataloss if the guest is not correctly
>>> configured (i.e. the kernel can't tell if it is mounting the same
>>> filesystem via two paths). The xen_emul_unplug option is the current way
>>> you can override this once you have confirmed that your guest
>>> configuration is not dangerous. I'm afraid this necessarily involves
>>> guest config and guest admin interaction.
>>>
>>> In principal we might be able to extend the unplug protocol (which would
>>> involve patches to qemu, the kernel(s) and the toolstack) to allow
>>> devices to be marked as being not necessary to unplug. Someone would
>>> have to send patches though and it would be opening up a way for people
>>> to lose data so we'd need to be careful.
>>>
>>> I'm sure that the unplug protocol is documented somewhere in the source
>>> tree but I can't for the life of me find it :-(
>>
>> So, the issue is this. We have thousands (literally) of disks in use
>> by third parties on xen 3.3. Some are Windows, some are ancient linux,
>> some are modern linux, etc. The hypervisor has no way of whether the
>> images are going to use /dev/sda or /dev/xvda (i.e. PV or emulated)
>> drivers. Indeed the most common linux case is that grub uses the
>> emulated devices to load the kernel, then uses /dev/xvda as a root
>> device, i.e. both are used (but not simultaneously).
>>
>> We need to have the xen pci stuff on, so PV drivers operate (in both
>> new and old kernels). But as modern linux kernels detect the unplug 
>> functionality, they will unplug the emulated devices and then fail to
>> boot because (for instance) under Xen3.3 using /dev/sdaX to access
>> (say) your /boot partition worked perfectly well. What we need is a
>> switch to revert to the old Xen3.3 (pre-unplug) behaviour, so any
>> Linux kernel will see the same set of devices. I cannot believe this
>> is a unique requirement for people attempting to do a Xen3.3 to
>> Xen4.1 migration.
> 
> I'm a bit fuzzy on the details but I'm not sure what this has to do with
> the host, the device naming and behaviour on unplug are kernel side
> things, I'd expect that if /dev/sdaX as /boot worked on 3.3 it'll work
> on 4.1 too. (I believe you that it doesn't work, I'm just wondering
> aloud what I'm missing).
> 
> Can you give us the specifics of a setup which fails, e.g. a complete
> guest cfg file, the kernel version, command line options, /etc/fstab,
> dmesg etc.
> 
>> I think this is in xen_unplug_emulated_devices() in
>> arch/x86/xen/platform-pci-unplug.c
>>
>> This uses check_platform_magic(), which I have appended. In order
>> to avoid unplugging (without relying on the boot line), I need
>> this to return a non-zero value (XEN_PLATFORM_ERR_MAGIC is
>> irrelevant as xen_emul_unplug is 0 by assumption).
>>
>> I can achieve that by either (a) returning a bad magic number,
>> (b) making the host 'blacklist' the product (how does that work?)
>> or (c) using a protocol value of (say) 0. I take it Xen 3.3 simply
>> returns a bad magic number as I don't think XEN_IOPORT_MAGIC existed
>> in 3.3. As far as I can tell, XEN_IOPORT_MAGIC is only use for
>> PCI unplug.
>>
>> So, is the correct approach to disable XEN_IOPORT_MAGIC (or rather
>> make it return a different value) depending on a configuration option?
>> If so, I am happy to submit a patch to do that. Or can I do this
>> without a patch by "blacklisting" everything? (not sure how that is done).
> 
> Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on
> the kernel side is what I was missing.
> 
> It might make sense to have a guest level config option which disables
> these magic ports, i.e. makes them return 0xffff like they would have
> done in 3.3 (I think 0xffff is what you'll get from an invalid port in
> general).
> 
>> Out of interest, with a default guest Ubuntu Natty install CD, using the
>> default Xen 4.1 settings, we are seeing the guest (a) unplugging the
>> emulated devices (fine), then (b) failing to find the emulated devices,
>> and (c) the install failing. Is that to be expected?
> 
> Sounds like an Ubuntu bug to me, but I don't follow Ubuntu closely
> enough to know if it is known or not.
> 
At least one part is not Ubuntu specific. And that is that the unplug logic
decides to unplug emulated devices based on having the pci and the blkfront
driver *available* (built-in or module). But later on the blkfront driver
ignores all devices that are not *named* in a way to map to the xvd major.
Which leaves you without any usable devices when you named your disk hda in
the config file and you do not prevent unplugging.

The other part of the problem is that even when you name the disk xvda in the
config file, the installer does not know about blkfront. This is Ubuntu specific
and we either need to have blkfront built-in or put them into a special udeb
which is a special handling just for the installer.

Still I would love to see this unplug handling become a bit more obvious. If
unplug was successful, then blkfront should not ignore the devices. Or maybe
just make the config more what-you-write-is-what-you-get and having hd or sd
there only gives you emulated devices and xvd gives you pv devices.

-Stefan

> Ian.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefano Stabellini]

I take we are still talking about PV on HVM guests here.

On Thu, 27 Oct 2011, Stefan Bader wrote:
> At least one part is not Ubuntu specific. And that is that the unplug logic
> decides to unplug emulated devices based on having the pci and the blkfront
> driver *available* (built-in or module). But later on the blkfront driver
> ignores all devices that are not *named* in a way to map to the xvd major.
> Which leaves you without any usable devices when you named your disk hda in
> the config file and you do not prevent unplugging.

If you name your disk hda (as you should), blkfront is going to create
/dev/xvda in your guest.
It is not ignoring your disk, it just using "xvd" to name the device in
the guest.


> Still I would love to see this unplug handling become a bit more obvious. If
> unplug was successful, then blkfront should not ignore the devices. Or maybe
> just make the config more what-you-write-is-what-you-get and having hd or sd
> there only gives you emulated devices and xvd gives you pv devices.

Yes, if the unplug is unsuccessful blkfront should not ignore the
device: it is going to create a /dev/xvd* for you.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefan Bader]

On 27.10.2011 15:42, Stefano Stabellini wrote:
> I take we are still talking about PV on HVM guests here.
> 
> On Thu, 27 Oct 2011, Stefan Bader wrote:
>> At least one part is not Ubuntu specific. And that is that the unplug logic
>> decides to unplug emulated devices based on having the pci and the blkfront
>> driver *available* (built-in or module). But later on the blkfront driver
>> ignores all devices that are not *named* in a way to map to the xvd major.
>> Which leaves you without any usable devices when you named your disk hda in
>> the config file and you do not prevent unplugging.
> 
> If you name your disk hda (as you should), blkfront is going to create
> /dev/xvda in your guest.
> It is not ignoring your disk, it just using "xvd" to name the device in
> the guest.
> 
> 
>> Still I would love to see this unplug handling become a bit more obvious. If
>> unplug was successful, then blkfront should not ignore the devices. Or maybe
>> just make the config more what-you-write-is-what-you-get and having hd or sd
>> there only gives you emulated devices and xvd gives you pv devices.
> 
> Yes, if the unplug is unsuccessful blkfront should not ignore the
> device: it is going to create a /dev/xvd* for you.

The problem I saw in my test was that in blkfront_probe the following case was
hit when the device name was hda in the cfg:

                if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) {
                        int major;

                        if (!VDEV_IS_EXTENDED(vdevice))
                                major = BLKIF_MAJOR(vdevice);
                        else
                                major = XENVBD_MAJOR;

                        if (major != XENVBD_MAJOR) {
                                printk(KERN_INFO
                                                "%s: HVM does not support vbd %d
as xen block device\n",
                                                __FUNCTION__, vdevice);
                                return -ENODEV;
                        }
                }

So major is not XENVBD_MAJOR and the device is ignored.

-Stefan

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefano Stabellini]

On Thu, 27 Oct 2011, Stefan Bader wrote:
> On 27.10.2011 15:42, Stefano Stabellini wrote:
> > I take we are still talking about PV on HVM guests here.
> > 
> > On Thu, 27 Oct 2011, Stefan Bader wrote:
> >> At least one part is not Ubuntu specific. And that is that the unplug logic
> >> decides to unplug emulated devices based on having the pci and the blkfront
> >> driver *available* (built-in or module). But later on the blkfront driver
> >> ignores all devices that are not *named* in a way to map to the xvd major.
> >> Which leaves you without any usable devices when you named your disk hda in
> >> the config file and you do not prevent unplugging.
> > 
> > If you name your disk hda (as you should), blkfront is going to create
> > /dev/xvda in your guest.
> > It is not ignoring your disk, it just using "xvd" to name the device in
> > the guest.
> > 
> > 
> >> Still I would love to see this unplug handling become a bit more obvious. If
> >> unplug was successful, then blkfront should not ignore the devices. Or maybe
> >> just make the config more what-you-write-is-what-you-get and having hd or sd
> >> there only gives you emulated devices and xvd gives you pv devices.
> > 
> > Yes, if the unplug is unsuccessful blkfront should not ignore the
> > device: it is going to create a /dev/xvd* for you.
> 
> The problem I saw in my test was that in blkfront_probe the following case was
> hit when the device name was hda in the cfg:
> 
>                 if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) {
>                         int major;
> 
>                         if (!VDEV_IS_EXTENDED(vdevice))
>                                 major = BLKIF_MAJOR(vdevice);
>                         else
>                                 major = XENVBD_MAJOR;
> 
>                         if (major != XENVBD_MAJOR) {
>                                 printk(KERN_INFO
>                                                 "%s: HVM does not support vbd %d
> as xen block device\n",
>                                                 __FUNCTION__, vdevice);
>                                 return -ENODEV;
>                         }
>                 }
> 
> So major is not XENVBD_MAJOR and the device is ignored.
> 

Why are you passing xen_emul_unplug=unnecessary?
The idea is that if you pass that option you have to use the emulated
path to access your disk unless you esplicitely add an xvd disk to your
config.

In the normal case, if you don't specify xen_emul_unplug, you are going
to get xvda for your hda disk.
Then if you use LABEL or UUID in your root= argument, you don't need to
change any configuration to make it work.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefan Bader]

On 27.10.2011 16:08, Stefano Stabellini wrote:
> On Thu, 27 Oct 2011, Stefan Bader wrote:
>> On 27.10.2011 15:42, Stefano Stabellini wrote:
>>> I take we are still talking about PV on HVM guests here.
>>>
>>> On Thu, 27 Oct 2011, Stefan Bader wrote:
>>>> At least one part is not Ubuntu specific. And that is that the unplug logic
>>>> decides to unplug emulated devices based on having the pci and the blkfront
>>>> driver *available* (built-in or module). But later on the blkfront driver
>>>> ignores all devices that are not *named* in a way to map to the xvd major.
>>>> Which leaves you without any usable devices when you named your disk hda in
>>>> the config file and you do not prevent unplugging.
>>>
>>> If you name your disk hda (as you should), blkfront is going to create
>>> /dev/xvda in your guest.
>>> It is not ignoring your disk, it just using "xvd" to name the device in
>>> the guest.
>>>
>>>
>>>> Still I would love to see this unplug handling become a bit more obvious. If
>>>> unplug was successful, then blkfront should not ignore the devices. Or maybe
>>>> just make the config more what-you-write-is-what-you-get and having hd or sd
>>>> there only gives you emulated devices and xvd gives you pv devices.
>>>
>>> Yes, if the unplug is unsuccessful blkfront should not ignore the
>>> device: it is going to create a /dev/xvd* for you.
>>
>> The problem I saw in my test was that in blkfront_probe the following case was
>> hit when the device name was hda in the cfg:
>>
>>                 if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) {
>>                         int major;
>>
>>                         if (!VDEV_IS_EXTENDED(vdevice))
>>                                 major = BLKIF_MAJOR(vdevice);
>>                         else
>>                                 major = XENVBD_MAJOR;
>>
>>                         if (major != XENVBD_MAJOR) {
>>                                 printk(KERN_INFO
>>                                                 "%s: HVM does not support vbd %d
>> as xen block device\n",
>>                                                 __FUNCTION__, vdevice);
>>                                 return -ENODEV;
>>                         }
>>                 }
>>
>> So major is not XENVBD_MAJOR and the device is ignored.
>>
> 
> Why are you passing xen_emul_unplug=unnecessary?
> The idea is that if you pass that option you have to use the emulated
> path to access your disk unless you esplicitely add an xvd disk to your
> config.
> 
> In the normal case, if you don't specify xen_emul_unplug, you are going
> to get xvda for your hda disk.
> Then if you use LABEL or UUID in your root= argument, you don't need to
> change any configuration to make it work.

Because of the above I have to specify the xen_emul_unplug to get *any* device.
If I do not specify anything, then (because pci and pv driver are available) the
emulated disk gets unplugged. Ok. But then the probe does not give me any xvd
because the major does not match.

The only form I could make this work was: use xvda in the config file.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefano Stabellini]

On Thu, 27 Oct 2011, Stefan Bader wrote:
> On 27.10.2011 16:08, Stefano Stabellini wrote:
> > On Thu, 27 Oct 2011, Stefan Bader wrote:
> >> On 27.10.2011 15:42, Stefano Stabellini wrote:
> >>> I take we are still talking about PV on HVM guests here.
> >>>
> >>> On Thu, 27 Oct 2011, Stefan Bader wrote:
> >>>> At least one part is not Ubuntu specific. And that is that the unplug logic
> >>>> decides to unplug emulated devices based on having the pci and the blkfront
> >>>> driver *available* (built-in or module). But later on the blkfront driver
> >>>> ignores all devices that are not *named* in a way to map to the xvd major.
> >>>> Which leaves you without any usable devices when you named your disk hda in
> >>>> the config file and you do not prevent unplugging.
> >>>
> >>> If you name your disk hda (as you should), blkfront is going to create
> >>> /dev/xvda in your guest.
> >>> It is not ignoring your disk, it just using "xvd" to name the device in
> >>> the guest.
> >>>
> >>>
> >>>> Still I would love to see this unplug handling become a bit more obvious. If
> >>>> unplug was successful, then blkfront should not ignore the devices. Or maybe
> >>>> just make the config more what-you-write-is-what-you-get and having hd or sd
> >>>> there only gives you emulated devices and xvd gives you pv devices.
> >>>
> >>> Yes, if the unplug is unsuccessful blkfront should not ignore the
> >>> device: it is going to create a /dev/xvd* for you.
> >>
> >> The problem I saw in my test was that in blkfront_probe the following case was
> >> hit when the device name was hda in the cfg:
> >>
> >>                 if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) {
> >>                         int major;
> >>
> >>                         if (!VDEV_IS_EXTENDED(vdevice))
> >>                                 major = BLKIF_MAJOR(vdevice);
> >>                         else
> >>                                 major = XENVBD_MAJOR;
> >>
> >>                         if (major != XENVBD_MAJOR) {
> >>                                 printk(KERN_INFO
> >>                                                 "%s: HVM does not support vbd %d
> >> as xen block device\n",
> >>                                                 __FUNCTION__, vdevice);
> >>                                 return -ENODEV;
> >>                         }
> >>                 }
> >>
> >> So major is not XENVBD_MAJOR and the device is ignored.
> >>
> > 
> > Why are you passing xen_emul_unplug=unnecessary?
> > The idea is that if you pass that option you have to use the emulated
> > path to access your disk unless you esplicitely add an xvd disk to your
> > config.
> > 
> > In the normal case, if you don't specify xen_emul_unplug, you are going
> > to get xvda for your hda disk.
> > Then if you use LABEL or UUID in your root= argument, you don't need to
> > change any configuration to make it work.
> 
> Because of the above I have to specify the xen_emul_unplug to get *any* device.

Why?
If you do *not* specify xen_emul_unplug, you should get /dev/xvda
(provided you have blkfront available and hda in your disk config line).


> If I do not specify anything, then (because pci and pv driver are available) the
> emulated disk gets unplugged. Ok. But then the probe does not give me any xvd
> because the major does not match.

that is only because you passed xen_emul_unplug=unnecessary; you
shouldn't do that

> The only form I could make this work was: use xvda in the config file.

you should use hda in the disk config line, and then configure the guest
kernel root= argument with /dev/xvda1 directly or LABEL or UUID
(better).

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefan Bader]

On 27.10.2011 16:25, Stefano Stabellini wrote:
> On Thu, 27 Oct 2011, Stefan Bader wrote:
>> On 27.10.2011 16:08, Stefano Stabellini wrote:
>>> On Thu, 27 Oct 2011, Stefan Bader wrote:
>>>> On 27.10.2011 15:42, Stefano Stabellini wrote:
>>>>> I take we are still talking about PV on HVM guests here.
>>>>>
>>>>> On Thu, 27 Oct 2011, Stefan Bader wrote:
>>>>>> At least one part is not Ubuntu specific. And that is that the unplug logic
>>>>>> decides to unplug emulated devices based on having the pci and the blkfront
>>>>>> driver *available* (built-in or module). But later on the blkfront driver
>>>>>> ignores all devices that are not *named* in a way to map to the xvd major.
>>>>>> Which leaves you without any usable devices when you named your disk hda in
>>>>>> the config file and you do not prevent unplugging.
>>>>>
>>>>> If you name your disk hda (as you should), blkfront is going to create
>>>>> /dev/xvda in your guest.
>>>>> It is not ignoring your disk, it just using "xvd" to name the device in
>>>>> the guest.
>>>>>
>>>>>
>>>>>> Still I would love to see this unplug handling become a bit more obvious. If
>>>>>> unplug was successful, then blkfront should not ignore the devices. Or maybe
>>>>>> just make the config more what-you-write-is-what-you-get and having hd or sd
>>>>>> there only gives you emulated devices and xvd gives you pv devices.
>>>>>
>>>>> Yes, if the unplug is unsuccessful blkfront should not ignore the
>>>>> device: it is going to create a /dev/xvd* for you.
>>>>
>>>> The problem I saw in my test was that in blkfront_probe the following case was
>>>> hit when the device name was hda in the cfg:
>>>>
>>>>                 if (xen_platform_pci_unplug & XEN_UNPLUG_UNNECESSARY) {
>>>>                         int major;
>>>>
>>>>                         if (!VDEV_IS_EXTENDED(vdevice))
>>>>                                 major = BLKIF_MAJOR(vdevice);
>>>>                         else
>>>>                                 major = XENVBD_MAJOR;
>>>>
>>>>                         if (major != XENVBD_MAJOR) {
>>>>                                 printk(KERN_INFO
>>>>                                                 "%s: HVM does not support vbd %d
>>>> as xen block device\n",
>>>>                                                 __FUNCTION__, vdevice);
>>>>                                 return -ENODEV;
>>>>                         }
>>>>                 }
>>>>
>>>> So major is not XENVBD_MAJOR and the device is ignored.
>>>>
>>>
>>> Why are you passing xen_emul_unplug=unnecessary?
>>> The idea is that if you pass that option you have to use the emulated
>>> path to access your disk unless you esplicitely add an xvd disk to your
>>> config.
>>> (and force the blkfront to get loaded but that is some other patches missing
>>> In the normal case, if you don't specify xen_emul_unplug, you are going
>>> to get xvda for your hda disk.
>>> Then if you use LABEL or UUID in your root= argument, you don't need to
>>> change any configuration to make it work.
>>
>> Because of the above I have to specify the xen_emul_unplug to get *any* device.
> 
> Why?
> If you do *not* specify xen_emul_unplug, you should get /dev/xvda
> (provided you have blkfront available and hda in your disk config line).
> 
Ah! Sorry, yes. I think I remember now how my confusion started. While trying to
find the lost interrupt problem for the network devices I had two of them
configured (8139cp and e1000) and during those tests there was no way I was able
to get any xvd device. But that was because for some reason with both NICs the
pci device would fail to get an IRQ, so xenbus was never started.

Removing the second NIC made the pci device work, but then with allowing unplug,
there was still some problem with the pv nic not working. So I probably ended up
with the unnecessary, and when the disk finally worked when defined as xvda, and
I was already too confused by other things to realize that that segment only
applies for the unecessary case. :/

> 
>> If I do not specify anything, then (because pci and pv driver are available) the
>> emulated disk gets unplugged. Ok. But then the probe does not give me any xvd
>> because the major does not match.
> 
> that is only because you passed xen_emul_unplug=unnecessary; you
> shouldn't do that
> 
>> The only form I could make this work was: use xvda in the config file.
> 
> you should use hda in the disk config line, and then configure the guest
> kernel root= argument with /dev/xvda1 directly or LABEL or UUID
> (better).

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefan Bader]

On 27.10.2011 15:42, Stefano Stabellini wrote:
> I take we are still talking about PV on HVM guests here.
> 
> On Thu, 27 Oct 2011, Stefan Bader wrote:
>> At least one part is not Ubuntu specific. And that is that the unplug logic
>> decides to unplug emulated devices based on having the pci and the blkfront
>> driver *available* (built-in or module). But later on the blkfront driver
>> ignores all devices that are not *named* in a way to map to the xvd major.
>> Which leaves you without any usable devices when you named your disk hda in
>> the config file and you do not prevent unplugging.
> 
> If you name your disk hda (as you should), blkfront is going to create
> /dev/xvda in your guest.
> It is not ignoring your disk, it just using "xvd" to name the device in
> the guest.
> 
> 
>> Still I would love to see this unplug handling become a bit more obvious. If
>> unplug was successful, then blkfront should not ignore the devices. Or maybe
>> just make the config more what-you-write-is-what-you-get and having hd or sd
>> there only gives you emulated devices and xvd gives you pv devices.
> 
> Yes, if the unplug is unsuccessful blkfront should not ignore the
> device: it is going to create a /dev/xvd* for you.

Hm, did you mean unsuccessful? Because then I think it *should* ignore the pv
disk. If it was successful then it *should not*. Or am I understanding this wrong?

> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefano Stabellini]

On Thu, 27 Oct 2011, Stefan Bader wrote:
> On 27.10.2011 15:42, Stefano Stabellini wrote:
> > I take we are still talking about PV on HVM guests here.
> > 
> > On Thu, 27 Oct 2011, Stefan Bader wrote:
> >> At least one part is not Ubuntu specific. And that is that the unplug logic
> >> decides to unplug emulated devices based on having the pci and the blkfront
> >> driver *available* (built-in or module). But later on the blkfront driver
> >> ignores all devices that are not *named* in a way to map to the xvd major.
> >> Which leaves you without any usable devices when you named your disk hda in
> >> the config file and you do not prevent unplugging.
> > 
> > If you name your disk hda (as you should), blkfront is going to create
> > /dev/xvda in your guest.
> > It is not ignoring your disk, it just using "xvd" to name the device in
> > the guest.
> > 
> > 
> >> Still I would love to see this unplug handling become a bit more obvious. If
> >> unplug was successful, then blkfront should not ignore the devices. Or maybe
> >> just make the config more what-you-write-is-what-you-get and having hd or sd
> >> there only gives you emulated devices and xvd gives you pv devices.
> > 
> > Yes, if the unplug is unsuccessful blkfront should not ignore the
> > device: it is going to create a /dev/xvd* for you.
> 
> Hm, did you mean unsuccessful? Because then I think it *should* ignore the pv
> disk. If it was successful then it *should not*. Or am I understanding this wrong?

Yes, sorry for the misspell.
If the unplug is successful then it should not ignore the disk.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Pasi Kärkkäinen]

On Wed, Oct 26, 2011 at 03:47:13PM +0100, Alex Bligh wrote:
> Pasi, Conrad,
>
> --On 26 October 2011 17:12:33 +0300 Pasi Kärkkäinen <pasik@iki.fi> wrote:
>
>> On Wed, Oct 26, 2011 at 09:49:42AM -0400, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Oct 26, 2011 at 02:17:25PM +0100, Alex Bligh wrote:
>>> > I am running Xen 4 and want users to be able to install arbitrary HVM
>>> > images. We want both PV and emulated drivers to appear. The reason
>>> > for this is for compatibility with images that were prepared for
>>> > Xen 3.3, which assume emulated devices exist at boot time (particularly
>>> > for running the boot loader).
>>> >
>>> > If I do not specify on the boot line xen_emul_unplug=never, then
>>> > under some guest OS's (e.g. Ubuntu Natty (2.6.38)), the domU kernel
>>> > unplugs the non-PV devices, which is undesirable in my application.
>>> >
>>> > I cannot arbitrarily change the kernel command line because it is
>>> > in effect set by the user's image.
>>> >
>>> > Therefore what I need to do is prevent Xen4 offering the facility
>>> > to unplug the devices in the first place. Setting "(unpluggable 0)"
>>> > in the config for the emulated NICs and disks does not appear
>>> > to prevent them getting unplugged; what may be happening is that
>>> > domU tries to unplug them (but fails) but still doesn't use them.
>>> > Is there a way to signal from dom0 that the kernel really shouldn't
>>> > unplug this stuff.
>>>
>>> xen_pci_platform=0 in your guest config should do it.
>>>
>>
>> It's actually called:
>> xen_platform_pci=0
>
> Thanks. Just to confirm, that will still allow HVM PV drivers to
> operate (as well)?
>

Nope. PVHVM drivers will require the xen pci platform device to operate.

-- Pasi

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Alex Bligh]

Ian,

> I'm a bit fuzzy on the details but I'm not sure what this has to do with
> the host, the device naming and behaviour on unplug are kernel side
> things, I'd expect that if /dev/sdaX as /boot worked on 3.3 it'll work
> on 4.1 too. (I believe you that it doesn't work, I'm just wondering
> aloud what I'm missing).
>
> Can you give us the specifics of a setup which fails, e.g. a complete
> guest cfg file, the kernel version, command line options, /etc/fstab,
> dmesg etc.

I am not avoiding answering your question (I will get you this) but
what is /meant/ to happen in the following scenario:

* Install on recent kernel (e.g. 2.6.37) running on Xen 3.3
* No fancy boot options, xen_emul_unplug not set
* No XEN_IOPORT_MAGIC implemented, so check_platform_magic()
  returns an error
* Therefore xen_platform_pci_unplug=0
* Therefore blkfront etc. don't init (probe returns
  -ENODEV)
* Therefore OS boots with root=/dev/sda

Now Xen 3.3 is upgraded to Xen 4
* Kernel boots, and XEN_IOPORT_MAGIC now exists
* Therefore unplug occurs, and xen_platform_pci_unplug is non zero
* Therefore blkfront etc. inits, and PV drivers start
* OS still boots with root=/dev/sda

Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?
If so, how in the code does this happen? If not, won't the boot fail?

> Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on
> the kernel side is what I was missing.
>
> It might make sense to have a guest level config option which disables
> these magic ports, i.e. makes them return 0xffff like they would have
> done in 3.3 (I think 0xffff is what you'll get from an invalid port in
> general).

Actually I don't think this will work. If we do this,
xen_plaftofm_pci_unplug will still be zero (as it's only set on exit
of the function after a successful unplug), and that's enough to
prevent blkfront and xenbus_probe_frontend from doing anything useful,
so will effectively disable PV drivers even where they should be enabled.

>> Out of interest, with a default guest Ubuntu Natty install CD, using the
>> default Xen 4.1 settings, we are seeing the guest (a) unplugging the
>> emulated devices (fine), then (b) failing to find the emulated devices,
>> and (c) the install failing. Is that to be expected?
>
> Sounds like an Ubuntu bug to me, but I don't follow Ubuntu closely
> enough to know if it is known or not.

We will investigate further. Currently we can't seem to get /any/
distro using PV drivers, and only old ones using emulated drivers.

-- 
Alex Bligh

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Ian Campbell]

On Wed, 2011-10-26 at 18:43 +0100, Alex Bligh wrote:
> Ian,
> 
> > I'm a bit fuzzy on the details but I'm not sure what this has to do with
> > the host, the device naming and behaviour on unplug are kernel side
> > things, I'd expect that if /dev/sdaX as /boot worked on 3.3 it'll work
> > on 4.1 too. (I believe you that it doesn't work, I'm just wondering
> > aloud what I'm missing).
> >
> > Can you give us the specifics of a setup which fails, e.g. a complete
> > guest cfg file, the kernel version, command line options, /etc/fstab,
> > dmesg etc.
> 
> I am not avoiding answering your question (I will get you this) but
> what is /meant/ to happen in the following scenario:

Lets call this scenario A:

> * Install on recent kernel (e.g. 2.6.37) running on Xen 3.3
> * No fancy boot options, xen_emul_unplug not set
> * No XEN_IOPORT_MAGIC implemented, so check_platform_magic()
>   returns an error

Correct, specifically it returns XEN_PLATFORM_ERR_MAGIC.

> * Therefore xen_platform_pci_unplug=0

Correct. There is some special casing around XEN_PLATFORM_ERR_MAGIC but
lets assume xen_emul_unplug=unnecessary has not been passed to the
kernel so it does not take effect.

> * Therefore blkfront etc. don't init (probe returns
>   -ENODEV)
> * Therefore OS boots with root=/dev/sda

sda must be an emulated IDE device since the 2.6.37 kernel does not
support PV devices with names other than xvd*.

I think this is as expected.

> Now Xen 3.3 is upgraded to Xen 4

Lets call this scenario B:

> * Kernel boots, and XEN_IOPORT_MAGIC now exists
> * Therefore unplug occurs, and xen_platform_pci_unplug is non zero
> * Therefore blkfront etc. inits, and PV drivers start
> * OS still boots with root=/dev/sda
> 
> Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?

No, there is no support for this in upstream kernels (in general all the
old behaviours of Xen kernels where they would hijack other drivers
device names are not upstreamble)

> If so, how in the code does this happen? If not, won't the boot fail?

You need to be using root=/dev/xvda here for it to work. Or even better
use root=UUID=thing or root=LABEL=thong (many distros do one of these by
default these days).

(I'd forgotten about the UUID=/LABEL= option til just now -- that might
be the bit of magic which was missing to make this work)

> > Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on
> > the kernel side is what I was missing.
> >
> > It might make sense to have a guest level config option which disables
> > these magic ports, i.e. makes them return 0xffff like they would have
> > done in 3.3 (I think 0xffff is what you'll get from an invalid port in
> > general).
> 
> Actually I don't think this will work. If we do this,
> xen_plaftofm_pci_unplug will still be zero (as it's only set on exit
> of the function after a successful unplug), and that's enough to
> prevent blkfront and xenbus_probe_frontend from doing anything useful,
> so will effectively disable PV drivers even where they should be enabled.

Correct, this will take you back to scenario A, however if that is how
the guest is configured (to use emulated devices) then this is what you
wanted (or at least it is what the guest configuration is expecting).

If the guest were configured to use PV devices then it would be using
root=/dev/xvda. Such a configuration would have needed
xen_emul_unplug=unnecessary in order to have worked on 3.3 before the
upgrade and this option would be harmless but unnecessary on 4.1.

If you were using UUID=/LABEL= then I think things would have worked in
both cases (emulated on 3.3 and pv on 4.1) without additional kernel
parameters.

One detail worth mentioning is that if the guest is using PV drivers and
expecting xvd* named devices then prior to the unplug the devices
xvd[a-d] are also exposed as the IDE devices hd[a-d]. This is how the
bootloader is still able to access the emulated device.

Ian.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Konrad Rzeszutek Wilk]

> > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?
> 
> No, there is no support for this in upstream kernels (in general all the
> old behaviours of Xen kernels where they would hijack other drivers
> device names are not upstreamble)

Wait a minute. I think these two:

konrad@phenom:~/work/linux$ git shortlog v3.0..v3.1 drivers/block/xen-blkfront.c
Stefan Bader (2):
      xen-blkfront: Drop name and minor adjustments for emulated scsi devices
      xen-blkfront: Fix one off warning about name clash


Made it happen.. as without them the Amazon EC2 would not boot?

Details: "    xen-blkfront: Drop name and minor adjustments for emulated scsi devices
    
    These were intended to avoid the namespace clash when representing
    emulated IDE and SCSI devices. However that seems to confuse users
    more than expected (a disk defined as sda becomes xvde).
    So for now go back to the scheme which does no adjustments. This
    will break when mixing IDE and SCSI names in the configuration of
    guests but should be by now expected"

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefano Stabellini]

On Thu, 27 Oct 2011, Konrad Rzeszutek Wilk wrote:
> > > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?
> > 
> > No, there is no support for this in upstream kernels (in general all the
> > old behaviours of Xen kernels where they would hijack other drivers
> > device names are not upstreamble)
> 
> Wait a minute. I think these two:
> 
> konrad@phenom:~/work/linux$ git shortlog v3.0..v3.1 drivers/block/xen-blkfront.c
> Stefan Bader (2):
>       xen-blkfront: Drop name and minor adjustments for emulated scsi devices
>       xen-blkfront: Fix one off warning about name clash
> 
> 
> Made it happen.. as without them the Amazon EC2 would not boot?
> 
> Details: "    xen-blkfront: Drop name and minor adjustments for emulated scsi devices
>     
>     These were intended to avoid the namespace clash when representing
>     emulated IDE and SCSI devices. However that seems to confuse users
>     more than expected (a disk defined as sda becomes xvde).
>     So for now go back to the scheme which does no adjustments. This
>     will break when mixing IDE and SCSI names in the configuration of
>     guests but should be by now expected"

This commit should only cause sda to become xvda rather than xvde.

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Ian Campbell]

On Thu, 2011-10-27 at 14:13 +0100, Konrad Rzeszutek Wilk wrote:
> > > Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?
> > 
> > No, there is no support for this in upstream kernels (in general all the
> > old behaviours of Xen kernels where they would hijack other drivers
> > device names are not upstreamble)
> 
> Wait a minute. I think these two:
> 
> konrad@phenom:~/work/linux$ git shortlog v3.0..v3.1 drivers/block/xen-blkfront.c
> Stefan Bader (2):
>       xen-blkfront: Drop name and minor adjustments for emulated scsi devices
>       xen-blkfront: Fix one off warning about name clash
> 
> 
> Made it happen.. as without them the Amazon EC2 would not boot?

No, that was to do with not renaming hda devices into xvdHDa (or
something like that) and instead calling them hda.

I was talking about blkfront literally taking over the IDE major which
is not something that's upstream.

> 
> Details: "    xen-blkfront: Drop name and minor adjustments for emulated scsi devices
>     
>     These were intended to avoid the namespace clash when representing
>     emulated IDE and SCSI devices. However that seems to confuse users
>     more than expected (a disk defined as sda becomes xvde).
>     So for now go back to the scheme which does no adjustments. This
>     will break when mixing IDE and SCSI names in the configuration of
>     guests but should be by now expected"

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Alex Bligh]

Ian,

--On 27 October 2011 10:27:08 +0100 Ian Campbell <Ian.Campbell@citrix.com> 
wrote:


>> I am not avoiding answering your question (I will get you this) but
>> what is /meant/ to happen in the following scenario:

(now answered in another email) :-)

> Lets call this scenario A:
>
>> * Install on recent kernel (e.g. 2.6.37) running on Xen 3.3
>> * No fancy boot options, xen_emul_unplug not set
>> * No XEN_IOPORT_MAGIC implemented, so check_platform_magic()
>>   returns an error
>
> Correct, specifically it returns XEN_PLATFORM_ERR_MAGIC.
>
>> * Therefore xen_platform_pci_unplug=0
>
> Correct. There is some special casing around XEN_PLATFORM_ERR_MAGIC but
> lets assume xen_emul_unplug=unnecessary has not been passed to the
> kernel so it does not take effect.
>
>> * Therefore blkfront etc. don't init (probe returns
>>   -ENODEV)
>> * Therefore OS boots with root=/dev/sda
>
> sda must be an emulated IDE device since the 2.6.37 kernel does not
> support PV devices with names other than xvd*.
>
> I think this is as expected.

Yes, correct. So an in image created for xen 3.3 will have root=/dev/sda

>> Now Xen 3.3 is upgraded to Xen 4
>
> Lets call this scenario B:
>
>> * Kernel boots, and XEN_IOPORT_MAGIC now exists
>> * Therefore unplug occurs, and xen_platform_pci_unplug is non zero
>> * Therefore blkfront etc. inits, and PV drivers start
>> * OS still boots with root=/dev/sda
>>
>> Are the PV devices meant to appear as /dev/sdX rather than /dev/xvdX?
>
> No, there is no support for this in upstream kernels (in general all the
> old behaviours of Xen kernels where they would hijack other drivers
> device names are not upstreamble)

I thought so (from reading the code)

>> If so, how in the code does this happen? If not, won't the boot fail?
>
> You need to be using root=/dev/xvda here for it to work. Or even better
> use root=UUID=thing or root=LABEL=thong (many distros do one of these by
> default these days).
>
> (I'd forgotten about the UUID=/LABEL= option til just now -- that might
> be the bit of magic which was missing to make this work)

Right, so the same image that worked fine on Xen 3 will not work if
Xen 3 is upgraded to Xen 4 - it will require changes to the image.
It's not only root= on the command line, but also /etc/fstab (we have
partitioned disks in our images).

Sure, using UUIDs works, but unfortunately we have thousands of VMs run by 
users who, not possessing crystal balls, did not know the names of their 
block devices were to change :-) For completeness, there is a also a 
problem with certain versions of grub (pre grub2) where UUID config doesn't 
work;
whilst grub of course loads before the kernel and before the unplug, the
grub config file still references a device which has been unplugged, so
the next time grub is reconfigured, it can't find the disk to write to.

>> > Hmm, yes I think the special treatment of XEN_IOPORT_MAGIC mismatch on
>> > the kernel side is what I was missing.
>> >
>> > It might make sense to have a guest level config option which disables
>> > these magic ports, i.e. makes them return 0xffff like they would have
>> > done in 3.3 (I think 0xffff is what you'll get from an invalid port in
>> > general).
>>
>> Actually I don't think this will work. If we do this,
>> xen_plaftofm_pci_unplug will still be zero (as it's only set on exit
>> of the function after a successful unplug), and that's enough to
>> prevent blkfront and xenbus_probe_frontend from doing anything useful,
>> so will effectively disable PV drivers even where they should be enabled.
>
> Correct, this will take you back to scenario A, however if that is how
> the guest is configured (to use emulated devices) then this is what you
> wanted (or at least it is what the guest configuration is expecting).
>
> If the guest were configured to use PV devices then it would be using
> root=/dev/xvda. Such a configuration would have needed
> xen_emul_unplug=unnecessary in order to have worked on 3.3 before the
> upgrade and this option would be harmless but unnecessary on 4.1.

Yes.

The trouble is some guests are running Windows, some old xenlinux
kernels, and some new kernels that talk to XEN_IOPORT_MAGIC. I need
one config that works for all of them, as we have no way of telling
them apart.

If I disable XEN_IOPORT_MAGIC then new installs of newer operating
systems will not run with PV drivers. I agree that these won't
run (without tweaked kernel boot lines) on Xen 3.3 with PV drivers
either, but I don't want to be stuck this way for ever.

I /think/ what you are saying is that the on new kernels that talk
to XEN_IOPORT_MAGIC, they will use /dev/sdX (emulated disks). If
we upgrade to Xen4, these will all break, because we can't distinguish
them from the behaviour of a new kernel doing a new install, where
we want the unplug to occur. The fix is either to fix the root
device, or to pass xen_emul_unplug=unnecessary in which case it should
continue to use to the emulated devices.

It would have been really nice to have been able to pass something
through XEN_IOPORT_MAGIC to effectively cause the same effect as
xen_emul_unplug=unnecessary on the command line, but I don't see how
that can be done with code already in the wild - unless I'm wrong?

> If you were using UUID=/LABEL= then I think things would have worked in
> both cases (emulated on 3.3 and pv on 4.1) without additional kernel
> parameters.
>
> One detail worth mentioning is that if the guest is using PV drivers and
> expecting xvd* named devices then prior to the unplug the devices
> xvd[a-d] are also exposed as the IDE devices hd[a-d]. This is how the
> bootloader is still able to access the emulated device.

Sure

-- 
Alex Bligh

Re: xen_emul_unplug on xen 4.1, HVM guest 2.6.38

[Stefano Stabellini]

On Thu, 27 Oct 2011, Alex Bligh wrote:
> It would have been really nice to have been able to pass something
> through XEN_IOPORT_MAGIC to effectively cause the same effect as
> xen_emul_unplug=unnecessary on the command line, but I don't see how
> that can be done with code already in the wild - unless I'm wrong?

You are correct: with the current linux code you need to change the
linux command line options in order to be able to accomplish what you
want.