Inplace upgrading 4.4.x -> 4.5.0

Inplace upgrading 4.4.x -> 4.5.0

[Steven Haigh]

[-- Attachment #1.1: Type: text/plain, Size: 599 bytes --]

Hi all,

I was under the impression that you should be able to do in-place
upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
manage DomUs...

This would support upgrades from running systems from Xen 4.4.x to 4.5.0
- only requiring a reboot to boot into the 4.5.0 hypervisor.

When I try this in practice, I get a whole heap of permission denied
errors and lose control of any running DomUs.

Is there some secret sauce that will allow this to work?

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Inplace upgrading 4.4.x -> 4.5.0

[Stefano Stabellini]

Hello Steven,
upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.

Please post more details and we'll try to help you figure out what's
wrong.

Cheers,

Stefano

On Sun, 8 Feb 2015, Steven Haigh wrote:
> Hi all,
> 
> I was under the impression that you should be able to do in-place
> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
> manage DomUs...
> 
> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
> - only requiring a reboot to boot into the 4.5.0 hypervisor.
> 
> When I try this in practice, I get a whole heap of permission denied
> errors and lose control of any running DomUs.
> 
> Is there some secret sauce that will allow this to work?
> 
> -- 
> Steven Haigh
> 
> Email: netwiz@crc.id.au
> Web: http://www.crc.id.au
> Phone: (03) 9001 6090 - 0412 935 897
> 
> 

Re: Inplace upgrading 4.4.x -> 4.5.0

[Steven Haigh]

[-- Attachment #1.1: Type: text/plain, Size: 1264 bytes --]

Thanks Stefano,

I wanted to make sure I was correct with this belief before I posted all
the info.

I'll gather as much info as I can in the next day or so and see what we
can do.

On 9/02/2015 7:35 PM, Stefano Stabellini wrote:
> Hello Steven,
> upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.
> 
> Please post more details and we'll try to help you figure out what's
> wrong.
> 
> Cheers,
> 
> Stefano
> 
> On Sun, 8 Feb 2015, Steven Haigh wrote:
>> Hi all,
>>
>> I was under the impression that you should be able to do in-place
>> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
>> manage DomUs...
>>
>> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
>> - only requiring a reboot to boot into the 4.5.0 hypervisor.
>>
>> When I try this in practice, I get a whole heap of permission denied
>> errors and lose control of any running DomUs.
>>
>> Is there some secret sauce that will allow this to work?
>>
>> -- 
>> Steven Haigh
>>
>> Email: netwiz@crc.id.au
>> Web: http://www.crc.id.au
>> Phone: (03) 9001 6090 - 0412 935 897
>>
>>


-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Inplace upgrading 4.4.x -> 4.5.0

[Sander Eikelenboom]

Monday, February 9, 2015, 9:35:33 AM, you wrote:

> Hello Steven,
> upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.

> Please post more details and we'll try to help you figure out what's
> wrong.

> Cheers,

> Stefano

> On Sun, 8 Feb 2015, Steven Haigh wrote:
>> Hi all,
>> 
>> I was under the impression that you should be able to do in-place
>> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
>> manage DomUs...
>> 
>> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
>> - only requiring a reboot to boot into the 4.5.0 hypervisor.
>> 
>> When I try this in practice, I get a whole heap of permission denied
>> errors and lose control of any running DomUs.
>> 
>> Is there some secret sauce that will allow this to work?
>> 
>> -- 
>> Steven Haigh
>> 
>> Email: netwiz@crc.id.au
>> Web: http://www.crc.id.au
>> Phone: (03) 9001 6090 - 0412 935 897
>> 
>> 

You are probably running into a mismatch between the running hypervisor (4.4) and 
the now installed toolstack (4.5) .. for instance when trying to shutdown the VM's
to do the reboot. 
(Since the newly installed hypervisor parts are only loaded and run on the next boot).

--
Sander

Re: Inplace upgrading 4.4.x -> 4.5.0

[Steven Haigh]

[-- Attachment #1.1: Type: text/plain, Size: 1895 bytes --]

On 9/02/2015 7:59 PM, Sander Eikelenboom wrote:
> Monday, February 9, 2015, 9:35:33 AM, you wrote:
> 
>> Hello Steven,
>> upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.
> 
>> Please post more details and we'll try to help you figure out what's
>> wrong.
> 
>> Cheers,
> 
>> Stefano
> 
>> On Sun, 8 Feb 2015, Steven Haigh wrote:
>>> Hi all,
>>>
>>> I was under the impression that you should be able to do in-place
>>> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
>>> manage DomUs...
>>>
>>> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
>>> - only requiring a reboot to boot into the 4.5.0 hypervisor.
>>>
>>> When I try this in practice, I get a whole heap of permission denied
>>> errors and lose control of any running DomUs.
>>>
>>> Is there some secret sauce that will allow this to work?
> 
> You are probably running into a mismatch between the running hypervisor (4.4) and 
> the now installed toolstack (4.5) .. for instance when trying to shutdown the VM's
> to do the reboot. 
> (Since the newly installed hypervisor parts are only loaded and run on the next boot).

Correct - It is the 4.4 Hypervisor with 4.5 toolstack. After a reboot,
all is good. However this causes the problem - once you update the
packages from 4.4 -> 4.5, you lose the ability to manage any running DomUs.

This is problematic - if only for the fact that you can't shut down
running DomUs for the Dom0 reboot.

I understand that large jumps in versions isn't supported - but I
believe that point versions should be supported using the same toolset.
ie 4.2 -> 4.3, 4.4 -> 4.5 etc.

I'm just about to gather some data for it - and I'll make a new thread
with what I can gather.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Inplace upgrading 4.4.x -> 4.5.0

[Andrew Cooper]

On 09/02/2015 09:09, Steven Haigh wrote:
> On 9/02/2015 7:59 PM, Sander Eikelenboom wrote:
>> Monday, February 9, 2015, 9:35:33 AM, you wrote:
>>
>>> Hello Steven,
>>> upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.
>>> Please post more details and we'll try to help you figure out what's
>>> wrong.
>>> Cheers,
>>> Stefano
>>> On Sun, 8 Feb 2015, Steven Haigh wrote:
>>>> Hi all,
>>>>
>>>> I was under the impression that you should be able to do in-place
>>>> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
>>>> manage DomUs...
>>>>
>>>> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
>>>> - only requiring a reboot to boot into the 4.5.0 hypervisor.
>>>>
>>>> When I try this in practice, I get a whole heap of permission denied
>>>> errors and lose control of any running DomUs.
>>>>
>>>> Is there some secret sauce that will allow this to work?
>> You are probably running into a mismatch between the running hypervisor (4.4) and 
>> the now installed toolstack (4.5) .. for instance when trying to shutdown the VM's
>> to do the reboot. 
>> (Since the newly installed hypervisor parts are only loaded and run on the next boot).
> Correct - It is the 4.4 Hypervisor with 4.5 toolstack. After a reboot,
> all is good. However this causes the problem - once you update the
> packages from 4.4 -> 4.5, you lose the ability to manage any running DomUs.
>
> This is problematic - if only for the fact that you can't shut down
> running DomUs for the Dom0 reboot.
>
> I understand that large jumps in versions isn't supported - but I
> believe that point versions should be supported using the same toolset.
> ie 4.2 -> 4.3, 4.4 -> 4.5 etc.
>
> I'm just about to gather some data for it - and I'll make a new thread
> with what I can gather.

This is because of the removal of Xend.  In the past, Xend was a daemon
process and would have continued to use the old libraries it loaded when
it started.

With xl and libxl, the new process will fall over an EACCES from Xen
(mismatched tools and hypervisor) before it can identify the correct
daemonised libxl process to talk to to shut the VMs down.

~Andrew

Re: Inplace upgrading 4.4.x -> 4.5.0

[Ian Campbell]

On Mon, 2015-02-09 at 20:09 +1100, Steven Haigh wrote:
> On 9/02/2015 7:59 PM, Sander Eikelenboom wrote:
> > Monday, February 9, 2015, 9:35:33 AM, you wrote:
> > 
> >> Hello Steven,
> >> upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.
> > 
> >> Please post more details and we'll try to help you figure out what's
> >> wrong.
> > 
> >> Cheers,
> > 
> >> Stefano
> > 
> >> On Sun, 8 Feb 2015, Steven Haigh wrote:
> >>> Hi all,
> >>>
> >>> I was under the impression that you should be able to do in-place
> >>> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
> >>> manage DomUs...
> >>>
> >>> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
> >>> - only requiring a reboot to boot into the 4.5.0 hypervisor.
> >>>
> >>> When I try this in practice, I get a whole heap of permission denied
> >>> errors and lose control of any running DomUs.
> >>>
> >>> Is there some secret sauce that will allow this to work?
> > 
> > You are probably running into a mismatch between the running hypervisor (4.4) and 
> > the now installed toolstack (4.5) .. for instance when trying to shutdown the VM's
> > to do the reboot. 
> > (Since the newly installed hypervisor parts are only loaded and run on the next boot).
> 
> Correct - It is the 4.4 Hypervisor with 4.5 toolstack. After a reboot,
> all is good. However this causes the problem - once you update the
> packages from 4.4 -> 4.5, you lose the ability to manage any running DomUs.

This sounds like a packaging issue -- Debian's packages for example jump
through some hoops to make sure multiple tools packages can be installed
in parallel and the correct ones selected for the currently running
hypervisor.

Otherwise I think the upgrade path is:
      * shutdown all VMs (or migrate them away)
      * install new Xen + tools
      * reboot
      * restart domains with new tools.

I'm afraid that using old tools on a new Xen is not something which is
supported, even in the midst of an upgrade and AFAIK never has been. The
N->N+1->N+2 statement is normally with reference to live migration (i.e.
you can live migrate from a 4.4 system to a 4.5 one).

Ian.

Re: Inplace upgrading 4.4.x -> 4.5.0

[Steven Haigh]

[-- Attachment #1.1: Type: text/plain, Size: 4268 bytes --]

On 9/02/2015 8:16 PM, Ian Campbell wrote:
> On Mon, 2015-02-09 at 20:09 +1100, Steven Haigh wrote:
>> On 9/02/2015 7:59 PM, Sander Eikelenboom wrote:
>>> Monday, February 9, 2015, 9:35:33 AM, you wrote:
>>>
>>>> Hello Steven,
>>>> upgrades from Xen 4.4 to 4.5 are supposed to work out of the box.
>>>
>>>> Please post more details and we'll try to help you figure out what's
>>>> wrong.
>>>
>>>> Cheers,
>>>
>>>> Stefano
>>>
>>>> On Sun, 8 Feb 2015, Steven Haigh wrote:
>>>>> Hi all,
>>>>>
>>>>> I was under the impression that you should be able to do in-place
>>>>> upgrades from Xen 4.4 to 4.5 on a system without losing the ability to
>>>>> manage DomUs...
>>>>>
>>>>> This would support upgrades from running systems from Xen 4.4.x to 4.5.0
>>>>> - only requiring a reboot to boot into the 4.5.0 hypervisor.
>>>>>
>>>>> When I try this in practice, I get a whole heap of permission denied
>>>>> errors and lose control of any running DomUs.
>>>>>
>>>>> Is there some secret sauce that will allow this to work?
>>>
>>> You are probably running into a mismatch between the running hypervisor (4.4) and 
>>> the now installed toolstack (4.5) .. for instance when trying to shutdown the VM's
>>> to do the reboot. 
>>> (Since the newly installed hypervisor parts are only loaded and run on the next boot).
>>
>> Correct - It is the 4.4 Hypervisor with 4.5 toolstack. After a reboot,
>> all is good. However this causes the problem - once you update the
>> packages from 4.4 -> 4.5, you lose the ability to manage any running DomUs.
> 
> This sounds like a packaging issue -- Debian's packages for example jump
> through some hoops to make sure multiple tools packages can be installed
> in parallel and the correct ones selected for the currently running
> hypervisor.

Hmmm - that sounds very hacky :\

> Otherwise I think the upgrade path is:
>       * shutdown all VMs (or migrate them away)
>       * install new Xen + tools
>       * reboot
>       * restart domains with new tools.
> 
> I'm afraid that using old tools on a new Xen is not something which is
> supported, even in the midst of an upgrade and AFAIK never has been. The
> N->N+1->N+2 statement is normally with reference to live migration (i.e.
> you can live migrate from a 4.4 system to a 4.5 one).

Hmmm Andrew is correct, the errors are all:

============= xl info ==============
libxl: error: libxl.c:5044:libxl_get_physinfo: getting physinfo:
Permission denied
libxl_physinfo failed.
libxl: error: libxl.c:5534:libxl_get_scheduler: getting domain info
list: Permission denied
host                   : xenhost
release                : 3.14.32-1.el6xen.x86_64
version                : #1 SMP Sun Feb 8 15:41:07 AEDT 2015
machine                : x86_64
xen_major              : 4
xen_minor              : 4
xen_extra              : .1
xen_version            : 4.4.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : (null)
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : dom0_mem=1024M cpufreq=xen dom0_max_vcpus=1
dom0_vcpus_pin console=tty0 console=com1 com1=115200,8n1
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11)
cc_compile_by          : mockbuild
cc_compile_domain      : crc.id.au
cc_compile_date        : Thu Jan  1 18:19:30 AEDT 2015
xend_config_format     : 4

============= xl list ==============
libxl: error: libxl.c:669:libxl_list_domain: getting domain info list:
Permission denied
libxl_list_domain failed.

============= xl dmesg ==============
libxl: error: libxl.c:6061:libxl_xen_console_read_line: reading console
ring buffer: Permission denied

So, this leads me to wonder - as I'm sure MANY people get bitten by this
- how to control (at least to shutdown) DomUs after an in-place upgrade?

Even if no other functions are implemented other than shutdown, I would
call that an acceptable functionality. At least this way, you're not
hard killing running VMs on reboot.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Inplace upgrading 4.4.x -> 4.5.0

[Ian Campbell]

On Mon, 2015-02-09 at 20:36 +1100, Steven Haigh wrote:
> > This sounds like a packaging issue -- Debian's packages for example jump
> > through some hoops to make sure multiple tools packages can be installed
> > in parallel and the correct ones selected for the currently running
> > hypervisor.
> 
> Hmmm - that sounds very hacky :\

I've been slowly unpicking the Debian patches and upstreaming bits of
them, I'm not sure if I'll manage to get this stuff upstream though
since it is a bit more invasive than the other stuff.
> Hmmm Andrew is correct, the errors are all:
> 
> ============= xl info ==============
> libxl: error: libxl.c:5044:libxl_get_physinfo: getting physinfo:
> Permission denied

EPERM is essential "tools/hypervisor version mismatch" in most contexts.

[...]
> So, this leads me to wonder - as I'm sure MANY people get bitten by this
> - how to control (at least to shutdown) DomUs after an in-place upgrade?

You should evacuate the host before upgrading it, which is what I
suppose most people do as the first step in their maintenance window.
Evacuation might involve migrating VMs to another host (perhaps as part
of a pool rolling upgrade type manoeuvre) or just shutting them down.

> Even if no other functions are implemented other than shutdown, I would
> call that an acceptable functionality. At least this way, you're not
> hard killing running VMs on reboot.

I'd expect that it might be possible to arrange to connect to the VM
console and shut it down from within, or possibly to use the xenstore
CLI tools to initiate the shutdown externally.

After that then you would still end up with some zombie domains since
after they have shutdown actually reaping them would require toolstack
actions to talk to the hypervisor and you'd hit the version mismatch.

Ian.

Re: Inplace upgrading 4.4.x -> 4.5.0

[Steven Haigh]

[-- Attachment #1.1: Type: text/plain, Size: 2640 bytes --]

On 18/02/2015 11:38 PM, Ian Campbell wrote:
> On Mon, 2015-02-09 at 20:36 +1100, Steven Haigh wrote:
>>> This sounds like a packaging issue -- Debian's packages for example jump
>>> through some hoops to make sure multiple tools packages can be installed
>>> in parallel and the correct ones selected for the currently running
>>> hypervisor.
>>
>> Hmmm - that sounds very hacky :\
> 
> I've been slowly unpicking the Debian patches and upstreaming bits of
> them, I'm not sure if I'll manage to get this stuff upstream though
> since it is a bit more invasive than the other stuff.

Anything that helps out here is a good thing.

>> Hmmm Andrew is correct, the errors are all:
>>
>> ============= xl info ==============
>> libxl: error: libxl.c:5044:libxl_get_physinfo: getting physinfo:
>> Permission denied
> 
> EPERM is essential "tools/hypervisor version mismatch" in most contexts.
> 
> [...]
>> So, this leads me to wonder - as I'm sure MANY people get bitten by this
>> - how to control (at least to shutdown) DomUs after an in-place upgrade?
> 
> You should evacuate the host before upgrading it, which is what I
> suppose most people do as the first step in their maintenance window.
> Evacuation might involve migrating VMs to another host (perhaps as part
> of a pool rolling upgrade type manoeuvre) or just shutting them down.
> 
>> Even if no other functions are implemented other than shutdown, I would
>> call that an acceptable functionality. At least this way, you're not
>> hard killing running VMs on reboot.
> 
> I'd expect that it might be possible to arrange to connect to the VM
> console and shut it down from within, or possibly to use the xenstore
> CLI tools to initiate the shutdown externally.
> 
> After that then you would still end up with some zombie domains since
> after they have shutdown actually reaping them would require toolstack
> actions to talk to the hypervisor and you'd hit the version mismatch.

In large scale organisations (10+ systems), then yes, I'd say you're
probably right. This problem hits those who are smaller than that
however. I could list countless people who have been bitten by this in
the past - the majority of which are small businesses / hobbyists that
don't have the kind of equipment to do any of the above.

The expected upgrade path for those types is "yum -y update && reboot"

As we know, this falls over in a heap - however I don't think it is
beyond the realms of expectation for this to work.

-- 
Steven Haigh

Email: netwiz@crc.id.au
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel