Upgrade to xen unstable, what changes are needed?

Upgrade to xen unstable, what changes are needed?

[Bruce Edge]

It's obvious that some config changes are needed to move from testing
to unstable. Is this documented anywhere?
I diff'd the docs dir in the xen-testing and xen-unstable branches and
the only change of note is Stefano's
xen-unstable.hg/docs/misc/console.txt doc.
Is there a wiki/README/man page on what a user needs to fix after
upgrading to unstable?

Aside from obvious things like xencommons and xend needing to be
manually started, what else is there?

I do have a bridge still: (that was one of the complaints I saw from some users)
#> brctl show
bridge name     bridge id               STP enabled     interfaces
eth0            8000.002481e8a51c       no              peth0

Here's the error I'm getting trying to start an Ubuntu pv-ops VM that
was working fine with testing:

xenconsole: Could not read tty from store: No such file or directory
#> Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

The README isn't exaclty helpful :-)
 %> diff xen-4.0-testing.hg/README xen-unstable.hg/README
2,7c2,7
<  __  __            _  _    ___
<  \ \/ /___ _ __   | || |  / _ \
<   \  // _ \ '_ \  | || |_| | | |
<   /  \  __/ | | | |__   _| |_| |
<  /_/\_\___|_| |_|    |_|(_)___/
<
---
>  __  __            _  _    _
>  \ \/ /___ _ __   | || |  / |
>   \  // _ \ '_ \  | || |_ | |
>   /  \  __/ | | | |__   _|| |
>  /_/\_\___|_| |_|    |_|(_)_|
>
22c22
< The 4.0 release offers excellent performance, hardware support and
---
> The 4.1 release offers excellent performance, hardware support and

Thanks

-Bruce

Re: Upgrade to xen unstable, what changes are needed?

[Pasi Kärkkäinen]

On Fri, Sep 10, 2010 at 04:26:12PM -0700, Bruce Edge wrote:
> It's obvious that some config changes are needed to move from testing
> to unstable. Is this documented anywhere?
> I diff'd the docs dir in the xen-testing and xen-unstable branches and
> the only change of note is Stefano's
> xen-unstable.hg/docs/misc/console.txt doc.
> Is there a wiki/README/man page on what a user needs to fix after
> upgrading to unstable?
> 
> Aside from obvious things like xencommons and xend needing to be
> manually started, what else is there?
> 
> I do have a bridge still: (that was one of the complaints I saw from some users)
> #> brctl show
> bridge name     bridge id               STP enabled     interfaces
> eth0            8000.002481e8a51c       no              peth0
> 
> Here's the error I'm getting trying to start an Ubuntu pv-ops VM that
> was working fine with testing:
> 
> xenconsole: Could not read tty from store: No such file or directory
> #> Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
> 

try reading "xm log" to figure out why vif hotplug doesn't work.

-- Pasi

Re: Upgrade to xen unstable, what changes are needed?

[Bruce Edge]

On Sat, Sep 11, 2010 at 3:15 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Fri, Sep 10, 2010 at 04:26:12PM -0700, Bruce Edge wrote:
>> It's obvious that some config changes are needed to move from testing
>> to unstable. Is this documented anywhere?
>> I diff'd the docs dir in the xen-testing and xen-unstable branches and
>> the only change of note is Stefano's
>> xen-unstable.hg/docs/misc/console.txt doc.
>> Is there a wiki/README/man page on what a user needs to fix after
>> upgrading to unstable?
>>
>> Aside from obvious things like xencommons and xend needing to be
>> manually started, what else is there?
>>
>> I do have a bridge still: (that was one of the complaints I saw from some users)
>> #> brctl show
>> bridge name     bridge id               STP enabled     interfaces
>> eth0            8000.002481e8a51c       no              peth0
>>
>> Here's the error I'm getting trying to start an Ubuntu pv-ops VM that
>> was working fine with testing:
>>
>> xenconsole: Could not read tty from store: No such file or directory
>> #> Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
>>
>
> try reading "xm log" to figure out why vif hotplug doesn't work.
>
> -- Pasi
>
>

I diff'd the xend.log files from testing vs unstable and normalized
out all of the runtime differences and ended up with this:

xen-testing (works):

DEBUG (DevController:642) hotplugStatusCallback 1.
DEBUG (DevController:139) Waiting for devices vkbd.
DEBUG (DevController:139) Waiting for devices ioports.
DEBUG (DevController:139) Waiting for devices tap.
DEBUG (DevController:139) Waiting for devices vif2.
DEBUG (DevController:139) Waiting for devices console.
DEBUG (DevController:144) Waiting for 0.
DEBUG (DevController:139) Waiting for devices vscsi.
DEBUG (DevController:139) Waiting for devices vbd.
DEBUG (DevController:144) Waiting for 51712.
DEBUG (DevController:628) hotplugStatusCallback
/local/domain/0/backend/vbd/1/51712/hotplug-status.
DEBUG (DevController:642) hotplugStatusCallback 1.
DEBUG (DevController:139) Waiting for devices irq.
DEBUG (DevController:139) Waiting for devices vfb.
DEBUG (DevController:139) Waiting for devices pci.
DEBUG (DevController:144) Waiting for 0.
DEBUG (DevController:139) Waiting for devices vusb.
DEBUG (DevController:139) Waiting for devices vtpm.

xen-unstable (not working):

DEBUG (XendDomainInfo:3071) XendDomainInfo.destroy: domid=1
DEBUG (XendDomainInfo:2401) Destroying device model
DEBUG (XendDomainInfo:2408) Releasing devices
DEBUG (XendDomainInfo:2414) Removing vif/0
DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass
= vif, device = vif/0
DEBUG (XendDomainInfo:2414) Removing console/0
DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass
= console, device = console/0
DEBUG (XendDomainInfo:2414) Removing vbd/51712
DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass
= vbd, device = vbd/51712
DEBUG (XendDomainInfo:2414) Removing pci/0
DEBUG (XendDomainInfo:1276) XendDomainInfo.destroyDevice: deviceClass
= pci, device = pci/0
DEBUG (pciif:460) XendDomainInfo.handleAerStateWatch
DEBUG (pciif:578) pci: unregister aer watch
DEBUG (DevController:628) hotplugStatusCallback
/local/domain/0/backend/vif/1/0/hotplug-status.
DEBUG (DevController:628) hotplugStatusCallback
/local/domain/0/backend/vif/1/0/hotplug-status.
DEBUG (DevController:628) hotplugStatusCallback
/local/domain/0/backend/vif/1/0/hotplug-status.
DEBUG (DevController:628) hotplugStatusCallback
/local/domain/0/backend/vif/1/0/hotplug-status.

Up to that point they were identical except for differences in PID's,
UUID's etc.

The only message on the domU console after pygrub is :

Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

The dom0 serial console prints this when the I start the VM:

(XEN) tmem: all pools thawed for all domains
(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains
(XEN) tmem: all pools frozen for all domains


One thing that keeps nagging at me is that I did not create my own
bridge. I'm still relying on manually starting xendomains and xend to
do this for me for the simple reason that it works.
Is there any down side to letting these scripts setup the bridge?

The xm list -l outputs are quire different.

Here is xen-testing:

(domain
    (domid 0)
    (cpu_weight 256)
    (cpu_cap 0)
    (bootloader )
    (on_crash restart)
    (uuid 00000000-0000-0000-0000-000000000000)
    (bootloader_args )
    (vcpus 1)
    (name Domain-0)
    (cpus ((0)))
    (on_reboot restart)
    (on_poweroff destroy)
    (maxmem 16777215)
    (memory 508)
    (shadow_memory 0)
    (features )
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (cpu_time 153.45433386)
    (online_vcpus 1)
    (image (linux (kernel ) (superpages 0) (nomigrate 0) (tsc_mode 0)))
    (status 2)
    (state r-----)
)
(domain
    (domid 2)
    (cpu_weight 256)
    (cpu_cap 0)
    (bootloader /usr/bin/pygrub)
    (on_crash destroy)
    (uuid 68462626-c380-f4a6-56cd-f7b8f255ff14)
    (bootloader_args )
    (vcpus 14)
    (description )
    (name dpm)
    (cpus ((2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15)))
    (on_reboot restart)
    (on_poweroff destroy)
    (maxmem 2048)
    (memory 2048)
    (shadow_memory 0)
    (features )
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (start_time 1284140737.75)
    (cpu_time 203.775702892)
    (online_vcpus 14)
    (image
        (linux
            (kernel )
            (args 'ip=:127.0.255.255::::eth0:dhcp ')
            (superpages 0)
            (videoram 4)
            (pci ())
            (serial pty)
            (nomigrate 0)
            (tsc_mode 0)
            (notes
                (HV_START_LOW 18446603336221196288)
                (FEATURES '!writable_page_tables|pae_pgdir_above_4gb')
                (VIRT_BASE 18446744071562067968)
                (GUEST_VERSION 2.6)
                (PADDR_OFFSET 0)
                (GUEST_OS linux)
                (HYPERCALL_PAGE 18446744071578882048)
                (LOADER generic)
                (SUSPEND_CANCEL 1)
                (PAE_MODE yes)
                (ENTRY 18446744071590900224)
                (XEN_VERSION xen-3.0)
            )
        )
    )
    (status 2)
    (state -b----)
    (store_mfn 1652929)
    (console_mfn 1652928)
    (device
        (vif
            (bridge eth0)
            (uuid 817ae372-10c2-24ff-d59f-a0a4853f8d25)
            (script /etc/xen/scripts/vif-bridge)
            (mac 00:16:3e:00:18:01)
            (vifname vif-dpm)
            (model e1000)
            (backend 0)
        )
    )
    (device
        (console
            (protocol vt100)
            (location 2)
            (uuid 6a1c6e22-9a20-a956-a2b3-89e93a38448a)
        )
    )
    (device
        (vbd
            (protocol x86_64-abi)
            (uuid 2683c56a-34f6-52df-edda-6d6d0710d001)
            (bootable 1)
            (dev xvda:disk)
            (uname phy:system/dpm_0)
            (mode w)
            (backend 0)
            (VDI )
        )
    )
)

Here is xen-unstable:


(domain
    (domid 0)
    (cpu_weight 256)
    (cpu_cap 0)
    (pool_name Pool-0)
    (bootloader '')
    (vcpus 1)
    (cpus ((0)))
    (on_poweroff destroy)
    (on_crash restart)
    (uuid 00000000-0000-0000-0000-000000000000)
    (bootloader_args '')
    (name Domain-0)
    (on_reboot restart)
    (maxmem 16777215)
    (memory 508)
    (shadow_memory 0)
    (features '')
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (cpu_time 37.522339757)
    (online_vcpus 1)
    (image (linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0)))
    (status 2)
    (state r-----)
)
(domain
    (domid 1)
    (cpu_weight 256)
    (cpu_cap 0)
    (pool_name Pool-0)
    (bootloader /usr/bin/pygrub)
    (vcpus 14)
    (cpus ((2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15)))
    (on_poweroff destroy)
    (description '')
    (on_crash destroy)
    (uuid ee0c00b4-13e1-6b0e-28ff-3cb961931ce4)
    (bootloader_args '')
    (name dpm)
    (on_reboot restart)
    (maxmem 2048)
    (memory 2048)
    (shadow_memory 0)
    (features '')
    (on_xend_start ignore)
    (on_xend_stop ignore)
    (cpu_time 0.0)
    (online_vcpus 0)
    (image
        (linux
            (kernel '')
            (args 'ip=:127.0.255.255::::eth0:dhcp ')
            (superpages 0)
            (videoram 4)
            (pci ())
            (serial pty)
            (nomigrate 0)
            (tsc_mode 0)
        )
    )
    (status 1)
    (state --p---)
    (device
        (vif
            (uuid 2f3d40b7-e2be-ace3-6bdb-f0c19f5a2705)
            (bridge eth0)
            (model e1000)
            (mac 00:16:3e:00:18:01)
            (vifname vif-dpm)
        )
    )
    (device
        (vbd
            (uuid d1f9e0a5-384e-15f7-959b-8b75ccd925ed)
            (bootable 1)
            (driver paravirtualised)
            (dev xvda)
            (uname phy:system/dpm_0)
            (mode w)
        )
    )
)

I am building my own xen with a patch to fix the etc/sysconfig ->
etc/default issue, but that doesn't seem to be the issue here as there
are no errors relating to these files or the runtime config that I can
see in the logs.

Here's my VM's .cfg file:

#!/usr/bin/env python
import os
from platform_defs import *
from get_mac import *
from socket import gethostname
from osaMountUtils import *

# Non-xen vars
nopci = os.getenv('NOPCI')
name  = 'dpm'

# Xen bits
pci, memory, memory_dynamic_max, vcpus, cpus =
get_platform_settings('pciback', nopci)  <== Script that sets platform
dependent data

mac         = get_mac(name)
vif         = [ 'model = e1000, bridge = eth0, mac = ' + mac +
',vifname = vif-dpm' ]
dhcp            = "dhcp"
sdl         = 0
vnc         = 0
serial      = 'pty' # ...or else "xenconsole: Could not read tty from
store: No such file or directory"
usbdevice   = 'tablet'

on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'destroy'

boot            = 'c'
disk            = [ 'phy:system/dpm_0,xvda,w' ]

#disk           = [ 'phy:system/dpm_1,xvda1,w', 'phy:system/dom0_1,xvda2,w' ]

memory          = 2048
builder         = 'linux'
bootloader = '/usr/bin/pygrub'

I tried setting cpus/vcpus to 1 with no change.

-Bruce

Re: Upgrade to xen unstable, what changes are needed?

[Konrad Rzeszutek Wilk]

On Mon, Sep 13, 2010 at 12:12:55PM -0700, Bruce Edge wrote:
> On Sat, Sep 11, 2010 at 3:15 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > On Fri, Sep 10, 2010 at 04:26:12PM -0700, Bruce Edge wrote:
> >> It's obvious that some config changes are needed to move from testing
> >> to unstable. Is this documented anywhere?
> >> I diff'd the docs dir in the xen-testing and xen-unstable branches and
> >> the only change of note is Stefano's
> >> xen-unstable.hg/docs/misc/console.txt doc.
> >> Is there a wiki/README/man page on what a user needs to fix after
> >> upgrading to unstable?
> >>
> >> Aside from obvious things like xencommons and xend needing to be
> >> manually started, what else is there?
> >>
> >> I do have a bridge still: (that was one of the complaints I saw from some users)
> >> #> brctl show
> >> bridge name     bridge id               STP enabled     interfaces
> >> eth0            8000.002481e8a51c       no              peth0
> >>
> >> Here's the error I'm getting trying to start an Ubuntu pv-ops VM that
> >> was working fine with testing:
> >>
> >> xenconsole: Could not read tty from store: No such file or directory
> >> #> Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

Is the xen backend device loaded or built in? Is the tap/tun device loaded?

Re: Upgrade to xen unstable, what changes are needed?

[Bruce Edge]

On Mon, Sep 20, 2010 at 8:38 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Mon, Sep 13, 2010 at 12:12:55PM -0700, Bruce Edge wrote:
>> On Sat, Sep 11, 2010 at 3:15 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>> > On Fri, Sep 10, 2010 at 04:26:12PM -0700, Bruce Edge wrote:
>> >> It's obvious that some config changes are needed to move from testing
>> >> to unstable. Is this documented anywhere?
>> >> I diff'd the docs dir in the xen-testing and xen-unstable branches and
>> >> the only change of note is Stefano's
>> >> xen-unstable.hg/docs/misc/console.txt doc.
>> >> Is there a wiki/README/man page on what a user needs to fix after
>> >> upgrading to unstable?
>> >>
>> >> Aside from obvious things like xencommons and xend needing to be
>> >> manually started, what else is there?
>> >>
>> >> I do have a bridge still: (that was one of the complaints I saw from some users)
>> >> #> brctl show
>> >> bridge name     bridge id               STP enabled     interfaces
>> >> eth0            8000.002481e8a51c       no              peth0
>> >>
>> >> Here's the error I'm getting trying to start an Ubuntu pv-ops VM that
>> >> was working fine with testing:
>> >>
>> >> xenconsole: Could not read tty from store: No such file or directory
>> >> #> Error: Device 0 (vif) could not be connected. Hotplug scripts not working.
>
> Is the xen backend device loaded or built in? Is the tap/tun device loaded?
>

Turned out to be some missing modules. The xen udev rules were not
loading the modules.
Once I fixed that (packaging problems), most other problems went away.

The final problems were related to the VM's VIF line. I had to change it from:
    vif         = [ 'mac = ' + mac + ',vifname = vif-dpm' ]
to
    vif         = [ 'mac = ' + mac  ]


For whatever reason I can't specify the VIF name with unstable.

-Bruce