Distro upgrade strategies

Distro upgrade strategies

[hboddington]

Hello all,

I'm struggling conceptually with a recommended upgrade strategy for our distribution (i.e. the resultant distribution on an ARM-based embedded device, not the development SDK itself).  I had initially assumed that a package manager (we're using DNF) could be used to upgrade a distribution from one version to the next, but every time I've used it to try to deploy a large upgrade (e.g.,  Zeus->Dunfell) I've run into problems.  Currently, if I try to use DNF to upgrade from a Zeus-based distribution my DNF installation breaks ("ModuleNotFoundError: No module named 'dnf'") and I can't even shut the device down ("/run/initctl: No such file or directory").  However, if I install the Dunfell-based distribution from a raw image (without upgrading through Zeus) then everything works fine.  I'm not getting an errors in my build or indication that something went wrong, but obviously something is amiss.

Questions:
  1) Am I missing something?  Is it reasonable to expect that I can upgrade the whole distro using DNF?  I've read the wiki page about System Updates (https://wiki.yoctoproject.org/wiki/System_Update) and DNF (or any other package manager) is not listed - does that imply it's not up to the job?
  2) If I do continue using DNF, how do I troubleshoot the above issues?  I don't mean specifically those issues, but I want to learn to fish.  Some packages got broken.  Was it caused by the distro I'm upgrading from?  To?  What's the best way of tracing it back?  I tried looking through the various commits but obviously there were a lot of changes made from one distro to another.  The above scenario seems also to be a perfect storm because now that DNF is broken I don't have a mechanism to downgrade anything.

I can't help but think that I'm swimming against the current here, or that I've done something fundamentally wrong - I'd appreciate any insight or advice.  Thank you!

Hugh

Re: [yocto] Distro upgrade strategies

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]

Yocto does not support or test upgrading from one yocto release to the next
through dnf. Nothing conceptually prevents it, but you need to set up and
test such upgrades yourself, and send any needed fixes upstream.

Alex

On Fri 19. Jun 2020 at 14.16, Hugh Boddington <
hboddington@orangephotonics.com> wrote:

> Hello all,
>
> I'm struggling conceptually with a recommended upgrade strategy for our
> distribution (i.e. the resultant distribution on an ARM-based embedded
> device, not the development SDK itself).  I had initially assumed that a
> package manager (we're using DNF) could be used to upgrade a distribution
> from one version to the next, but every time I've used it to try to deploy
> a large upgrade (e.g.,  Zeus->Dunfell) I've run into problems.  Currently,
> if I try to use DNF to upgrade from a Zeus-based distribution my DNF
> installation breaks ("ModuleNotFoundError: No module named 'dnf'") and I
> can't even shut the device down ("/run/initctl: No such file or
> directory").  However, if I install the Dunfell-based distribution from a
> raw image (without upgrading through Zeus) then everything works fine.  I'm
> not getting an errors in my build or indication that something went wrong,
> but obviously something is amiss.
>
> Questions:
>   1) Am I missing something?  Is it reasonable to expect that I can
> upgrade the whole distro using DNF?  I've read the wiki page about System
> Updates (https://wiki.yoctoproject.org/wiki/System_Update) and DNF (or
> any other package manager) is not listed - does that imply it's not up to
> the job?
>   2) If I do continue using DNF, how do I troubleshoot the above issues?
> I don't mean specifically those issues, but I want to learn to fish.  Some
> packages got broken.  Was it caused by the distro I'm upgrading from?  To?
> What's the best way of tracing it back?  I tried looking through the
> various commits but obviously there were a lot of changes made from one
> distro to another.  The above scenario seems also to be a perfect storm
> because now that DNF is broken I don't have a mechanism to downgrade
> anything.
>
> I can't help but think that I'm swimming against the current here, or that
> I've done something fundamentally wrong - I'd appreciate any insight or
> advice.  Thank you!
>
> Hugh
> 
>

[-- Attachment #2: Type: text/html, Size: 2897 bytes --]

Re: [yocto] Distro upgrade strategies

[csimmonds]

I would like to add that this is a design decision. Package updates, be
they based on rpm, deb, or ipk, are not atomic. Which means that if you
power off during an update the system will usually not boot afterwards.
This is not a problem with servers in data centers with redundant power
supplies and whatever, but it is a problem for an embedded device on the
top of a mountain or at the bottom of the sea. OpenEmbedded/Yocto
Project expect the update will be done by programming a new root
filesystem image. There are several ways to do this, including OTA
solutions like swupdate and mender.io.

HTH,
Chris

On 19/06/2020 15:05, Alexander Kanavin wrote:
> Yocto does not support or test upgrading from one yocto release to the
> next through dnf. Nothing conceptually prevents it, but you need to set
> up and test such upgrades yourself, and send any needed fixes upstream.
> 
> Alex
> 
> On Fri 19. Jun 2020 at 14.16, Hugh Boddington
> <hboddington@orangephotonics.com
> <mailto:hboddington@orangephotonics.com>> wrote:
> 
>     Hello all,
> 
>     I'm struggling conceptually with a recommended upgrade strategy for
>     our distribution (i.e. the resultant distribution on an ARM-based
>     embedded device, not the development SDK itself).  I had initially
>     assumed that a package manager (we're using DNF) could be used to
>     upgrade a distribution from one version to the next, but every time
>     I've used it to try to deploy a large upgrade (e.g.,  Zeus->Dunfell)
>     I've run into problems.  Currently, if I try to use DNF to upgrade
>     from a Zeus-based distribution my DNF installation breaks
>     ("ModuleNotFoundError: No module named 'dnf'") and I can't even shut
>     the device down ("/run/initctl: No such file or directory"). 
>     However, if I install the Dunfell-based distribution from a raw
>     image (without upgrading through Zeus) then everything works fine. 
>     I'm not getting an errors in my build or indication that something
>     went wrong, but obviously something is amiss.
> 
>     Questions:
>       1) Am I missing something?  Is it reasonable to expect that I can
>     upgrade the whole distro using DNF?  I've read the wiki page about
>     System Updates (https://wiki.yoctoproject.org/wiki/System_Update)
>     and DNF (or any other package manager) is not listed - does that
>     imply it's not up to the job?
>       2) If I do continue using DNF, how do I troubleshoot the above
>     issues?  I don't mean specifically those issues, but I want to learn
>     to fish.  Some packages got broken.  Was it caused by the distro I'm
>     upgrading from?  To?  What's the best way of tracing it back?  I
>     tried looking through the various commits but obviously there were a
>     lot of changes made from one distro to another.  The above scenario
>     seems also to be a perfect storm because now that DNF is broken I
>     don't have a mechanism to downgrade anything.
> 
>     I can't help but think that I'm swimming against the current here,
>     or that I've done something fundamentally wrong - I'd appreciate any
>     insight or advice.  Thank you!
> 
>     Hugh
> 
> 
> 
> 


-- 
Chris Simmonds, trainer and consultant at 2net
http://www.2net.co.uk
Author of "Mastering Embedded Linux Programming"

Re: [yocto] Distro upgrade strategies

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3782 bytes --]

Right. Where I work, the updates would be performed through an A/B
partition scheme: if the update is botched, it is always possible to roll
back. This doubles the disk space requirement, but is far more robust.

Alex

On Fri, 19 Jun 2020 at 16:35, csimmonds <chris@2net.co.uk> wrote:

> I would like to add that this is a design decision. Package updates, be
> they based on rpm, deb, or ipk, are not atomic. Which means that if you
> power off during an update the system will usually not boot afterwards.
> This is not a problem with servers in data centers with redundant power
> supplies and whatever, but it is a problem for an embedded device on the
> top of a mountain or at the bottom of the sea. OpenEmbedded/Yocto
> Project expect the update will be done by programming a new root
> filesystem image. There are several ways to do this, including OTA
> solutions like swupdate and mender.io.
>
> HTH,
> Chris
>
> On 19/06/2020 15:05, Alexander Kanavin wrote:
> > Yocto does not support or test upgrading from one yocto release to the
> > next through dnf. Nothing conceptually prevents it, but you need to set
> > up and test such upgrades yourself, and send any needed fixes upstream.
> >
> > Alex
> >
> > On Fri 19. Jun 2020 at 14.16, Hugh Boddington
> > <hboddington@orangephotonics.com
> > <mailto:hboddington@orangephotonics.com>> wrote:
> >
> >     Hello all,
> >
> >     I'm struggling conceptually with a recommended upgrade strategy for
> >     our distribution (i.e. the resultant distribution on an ARM-based
> >     embedded device, not the development SDK itself).  I had initially
> >     assumed that a package manager (we're using DNF) could be used to
> >     upgrade a distribution from one version to the next, but every time
> >     I've used it to try to deploy a large upgrade (e.g.,  Zeus->Dunfell)
> >     I've run into problems.  Currently, if I try to use DNF to upgrade
> >     from a Zeus-based distribution my DNF installation breaks
> >     ("ModuleNotFoundError: No module named 'dnf'") and I can't even shut
> >     the device down ("/run/initctl: No such file or directory").
> >     However, if I install the Dunfell-based distribution from a raw
> >     image (without upgrading through Zeus) then everything works fine.
> >     I'm not getting an errors in my build or indication that something
> >     went wrong, but obviously something is amiss.
> >
> >     Questions:
> >       1) Am I missing something?  Is it reasonable to expect that I can
> >     upgrade the whole distro using DNF?  I've read the wiki page about
> >     System Updates (https://wiki.yoctoproject.org/wiki/System_Update)
> >     and DNF (or any other package manager) is not listed - does that
> >     imply it's not up to the job?
> >       2) If I do continue using DNF, how do I troubleshoot the above
> >     issues?  I don't mean specifically those issues, but I want to learn
> >     to fish.  Some packages got broken.  Was it caused by the distro I'm
> >     upgrading from?  To?  What's the best way of tracing it back?  I
> >     tried looking through the various commits but obviously there were a
> >     lot of changes made from one distro to another.  The above scenario
> >     seems also to be a perfect storm because now that DNF is broken I
> >     don't have a mechanism to downgrade anything.
> >
> >     I can't help but think that I'm swimming against the current here,
> >     or that I've done something fundamentally wrong - I'd appreciate any
> >     insight or advice.  Thank you!
> >
> >     Hugh
> >
> >
> >
> >
>
>
> --
> Chris Simmonds, trainer and consultant at 2net
> http://www.2net.co.uk
> Author of "Mastering Embedded Linux Programming"
> 
>

[-- Attachment #2: Type: text/html, Size: 5001 bytes --]

Re: [yocto] Distro upgrade strategies

[Adrian Bunk]

On Fri, Jun 19, 2020 at 03:35:33PM +0100, csimmonds wrote:
> I would like to add that this is a design decision. Package updates, be
> they based on rpm, deb, or ipk, are not atomic. Which means that if you
> power off during an update the system will usually not boot afterwards.

This is not true, package manager like dpkg are trying very very hard to 
make things atomic where they have to be and allow recovery from any 
kind of interruption.

I do not recall anyone ever describing such a problem with Debian
(or Ubuntu) in over 20 years as Debian user/administrator/developer.

>...
> OpenEmbedded/Yocto
> Project expect the update will be done by programming a new root
> filesystem image.
>...

Note that what you write would imply that package management must not
ever be used on OpenEmbedded/Yocto devices.

If there is any potential problem, it would equally apply to the 
situation where you are upgrading just one package for a security
update.

> HTH,
> Chris

cu
Adrian

Re: [yocto] Distro upgrade strategies

[Stefano Babic]

Hi Adrian,

On 21.06.20 09:11, Adrian Bunk wrote:
> On Fri, Jun 19, 2020 at 03:35:33PM +0100, csimmonds wrote:
>> I would like to add that this is a design decision. Package updates, be
>> they based on rpm, deb, or ipk, are not atomic. Which means that if you
>> power off during an update the system will usually not boot afterwards.
> 
> This is not true, package manager like dpkg are trying very very hard to
> make things atomic where they have to be and allow recovery from any
> kind of interruption.

I agree with Chris. Yes, dpkg had a lot of improvements, and it is more 
difficult to break a Debian / Ubuntu during an "apt-get update". But let 
me say it is not enough for embedded systems.

I think you mean that dpkg has improved to make the update of a single 
package more atomic, but what about if you upgrade a lot of packages at 
once and you have a power-cut in between ?

If it can work in many case, it won't in many other cases. Hash issues 
with packages ? System does not boot and a grub-rescue shell is started 
? Not a very big problem (in many cases) on server / pc. We check what 
happens, we rework maybe the grub menu, or if nothing works we start a 
live distro. But if this is ok on server / pc for a linux developer, how 
can it work on unattended (embedded) systems, without shell access (due 
to security reasons), and with end users ? It is not enough if it works 
in most cases, it must always work.

> 
> I do not recall anyone ever describing such a problem with Debian
> (or Ubuntu) in over 20 years as Debian user/administrator/developer.
> 

Yes, but if you say that we can update safe between two Debian / Ubuntu 
versions, I have just a couple of cases on my desk that it could not 
work without human intervention...

>> ...
>> OpenEmbedded/Yocto
>> Project expect the update will be done by programming a new root
>> filesystem image.
>> ...
> 
> Note that what you write would imply that package management must not
> ever be used on OpenEmbedded/Yocto devices.

I do not think Chris thought this - you can use package management, it 
speeds up the development process. How this is possible to be used in 
field, it depends on specific situation (maybe just the application is 
udated via package ?), but it is generally not thought to be atomic from 
one distro version to another one (including downgrading).

> 
> If there is any potential problem, it would equally apply to the
> situation where you are upgrading just one package for a security
> update.

In fact, it could apply...

Best regards,
Stefano

> 
>> HTH,
>> Chris
> 
> cu
> Adrian
> 
> 
> 
> 

-- 
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
=====================================================================

Re: [yocto] Distro upgrade strategies

[Adrian Bunk]

On Sun, Jun 21, 2020 at 02:09:44PM +0200, Stefano Babic wrote:
> Hi Adrian,

Hi Stefano,

>...
> If it can work in many case, it won't in many other cases. Hash issues with
> packages ? System does not boot and a grub-rescue shell is started ? Not a
> very big problem (in many cases) on server / pc. We check what happens, we
> rework maybe the grub menu, or if nothing works we start a live distro. But
> if this is ok on server / pc for a linux developer, how can it work on
> unattended (embedded) systems, without shell access (due to security
> reasons), and with end users ? It is not enough if it works in most cases,
> it must always work.
>...

one major difference between Yocto and most Linux distributions is that
most Linux distributions enable automatic security updates by default.

A normal Ubuntu system gets security updates installed several times
every week.

I know many nontechnical people working on an Ubuntu laptop or desktop,
and I have seen many embedded systems running Ubuntu.

>...
> I think you mean that dpkg has improved to make the update of a single
> package more atomic, but what about if you upgrade a lot of packages at once
> and you have a power-cut in between ?
>...

What I mean is that what Steve claimed to be "usually" is something 
I do not remember seeing in over 20 years with Debian and Ubuntu.

Not just "improved", this is something dpkg gets right since the 1990s.

As long as any highlevel tool calls "dpkg --configure -a" on unclean
dpkg state, chances are the upgrade can just continue after the power-cut.

With "atomic where they have to be" I meant the normal fsync+rename
that ensures that you can never end up with a half-written file
after a power-cut.

> Best regards,
> Stefano

cu
Adrian

Re: [yocto] Distro upgrade strategies

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 547 bytes --]

On Sun, 21 Jun 2020 at 15:46, Adrian Bunk <bunk@stusta.de> wrote:

>
> With "atomic where they have to be" I meant the normal fsync+rename
> that ensures that you can never end up with a half-written file
> after a power-cut.
>

What about half-installed packages? Does dpkg guarantee atomicity on that
level, and if so, how? The situation where the system boots with a core
package being a mix of old and upgraded version isn't acceptable. And same
for the overall package set, where some packages are old and others are new.

Alex

[-- Attachment #2: Type: text/html, Size: 865 bytes --]