[PATCH 0/1] V2 opkg 0.1.8: respect to the arch when choose the alternatives

[PATCH 0/1] V2 opkg 0.1.8: respect to the arch when choose the alternatives

[Robert Yang]

* Changes of V2:
  - Add the test plan and more explanations in the cover letter as Saul suggested.

* Test info (PACKAGE_CLASSES = "package_ipk")
  - Test to make sure the bug has been fixed
    1) bitbake core-image-sato-sdk with MACHINE=qemux86
    2) bitbake core-image-sato with MACHINE=crownbay

    === Test result:
    Without this patch, the xserver-xorg-*_1.11.2-r3_i586.ipk would be
    install to the crownbay's image, this is incorrect since what
    crownbay needs are xserver-xorg*_1.9.3-r1.core2.ipk, the core2 pkgs
    would be installed correctly on crownbay's image. And check
    tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/temp/log.do_rootfs,
    no i586 pkgs should be installed.

  - Test the upgrade (with nothing changed in the feeds)
    # opkg update
    1) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs update

    # opkg upgrade
    2) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs upgrade

    === Test result:
    Nothing to do though there are xserver-xorg-*_1.11.2-r3_i586.ipk
    which have a higher version than the installed
    xserver-xorg*_1.9.3-r1.core2.ipk, becuase the priority of
    i586 is lower than core2 in core2's image.

  - Test the installation
    # opkg remove
    1) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs \
       remove --force-depends xserver-xorg

    # opkg install
    2) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs \
       install xserver-xorg

    === Test result:
    The xserver-xorg_1.9.3-r1_core2.ipk will be installed, the
    reason is the same as the previous one

* More explanations

  This patch only affects it's good_pkg_by_name behaviour, Here are its
  selecting priorities: (From high to low)
  
  1, The pkg set manually
  2, The good_pkg_by_name
  3, The held pkg
  4, Choose the higher arch one
  5, Use the latest one
  
  If of them is matched, then the left ones would be ignored, for
  example, if the first one matches, others would be ignored.
      
  This patch affects the second one(good_by_pkg_name), what it did in
  the past: If there are both core2/pkg-1.0.apk and i586/pkg-1.1.apk
  match, it would use i586/pkg-1.1.apk since its version is higher and
  ignore the arch priorities, but we have set the arch priorities in
  opkg.conf clearly:
  
  arch i586 31
  arch core2 41
  
  What we need is core2/pkg-1.0.apk since we may set the specific pkg
  version for the board (for example, the xorg-server for crownbay), let
  it respect to the arch priorities in such a case.

* Both rpm(fixed) and ipk have the similar issue, dpkg is OK.

* TODO
  - In the following upgrade scene (with pkg changed in the feeds)
    # opkg remove
    1) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs \
       remove --force-depends xserver-xorg
    # Remove xserver-xorg_1.9.3-r1_core2.ipk to make
    # xserver-xorg_1.11.2-r3_i586.ipk install to the image.
    2) mv tmp/deploy/ipk/core2/xserver-xorg_1.9.3-r1_core2.ipk tmp/bak/
    # Run opkg-make-index to re-create the feeds
    3) ./tmp/sysroots/i686-linux/usr/bin/opkg-make-index \
       -r tmp/deploy/ipk/core2/Packages \
       -p tmp/deploy/ipk/core2/Packages \
       -m tmp/deploy/ipk/core2
    # Install xserver-xorg_1.11.2-r3_i586.ipk
    4) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs \
       install xserver-xorg
    # Add xserver-xorg_1.9.3-r1_core2.ipk back to the feeds
    5) mv tmp/bak/xserver-xorg_1.9.3-r1_core2.ipk tmp/deploy/ipk/core2/
    # Run opkg-make-index to re-create the feeds
    6) ./tmp/sysroots/i686-linux/usr/bin/opkg-make-index \
       -r tmp/deploy/ipk/core2/Packages \
       -p tmp/deploy/ipk/core2/Packages \
       -m tmp/deploy/ipk/core2
    # opkg update
    7) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs update
    # opkg upgrade
    8) ./tmp/sysroots/i686-linux/usr/bin/opkg-cl -f \
       /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/opkg.conf \
       -o /path/to/tmp/work/crownbay-poky-linux/core-image-sato-1.0-r0/rootfs upgrade

    Nothing will be done currently, since the opkg_upgrade.c will check
    the version again before really download the pkg and upgrade it,
    I'm not sure whether this is what we need, if we think that it is
    better, I'd like to work on it in another patch. There is another
    way to achieve:
    opkg install --force-downgrade xserver-xorg

    Then the xserver-xorg_1.9.3-r1_core2.ipk will be installed.

// Robert

The following changes since commit 363424cdc78cafa278008f973c2b4288297ebf8c:

  image.bbclass: Ensure ${S} is cleaned at the start of rootfs generation (2012-05-25 11:28:00 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib robert/ipk_arch
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/ipk_arch

Robert Yang (1):
  opkg 0.1.8: respect to the arch when choose the alternatives

 .../opkg/opkg/respect-to-arch.patch                |   47 ++++++++++++++++++++
 meta/recipes-devtools/opkg/opkg_0.1.8.bb           |    3 +-
 meta/recipes-devtools/opkg/opkg_svn.bb             |    3 +-
 3 files changed, 51 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/opkg/opkg/respect-to-arch.patch

[PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Robert Yang]

There is a bug if we:
1) bitbake core-image-sato-sdk MACHINE=qemux86
2) bitbake core-image-sato with MACHINE=crownbay

Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
image even if there is one in deploy/ipk/core2 and we have set the
core2's priority higher than i586, when the version in deploy/ipk/i586 is
higher. This doesn't work for us, for example, what the crownbay need is
xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.

This is caused by opkg's selecting mechanism, if there are more than one
candidates which have the same pkg name in the candidate list, for
example, the same pkg with different versions, then it will use the last
one which is the highest version in the list, this doesn't work for us,
it should respect to the arch priorities in such a case.

This is for both denzil and master branch.

[YOCTO #2360]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../opkg/opkg/respect-to-arch.patch                |   47 ++++++++++++++++++++
 meta/recipes-devtools/opkg/opkg_0.1.8.bb           |    3 +-
 meta/recipes-devtools/opkg/opkg_svn.bb             |    3 +-
 3 files changed, 51 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-devtools/opkg/opkg/respect-to-arch.patch

diff --git a/meta/recipes-devtools/opkg/opkg/respect-to-arch.patch b/meta/recipes-devtools/opkg/opkg/respect-to-arch.patch
new file mode 100644
index 0000000..6f4537f
--- /dev/null
+++ b/meta/recipes-devtools/opkg/opkg/respect-to-arch.patch
@@ -0,0 +1,47 @@
+pkg_hash.c: respect to the arch priorities when good_pkg_by_name
+
+If there are more than one candidates which have the same pkg name in
+the candidate list, for example, the same pkg with different versions,
+then it will use the last one which is the highest version in the list,
+it should respect to the arch priorities in such a case.
+
+Upstream Status: Pending
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ libopkg/pkg_hash.c |   18 +++++++++++++++---
+ 1 files changed, 15 insertions(+), 3 deletions(-)
+
+diff --git a/libopkg/pkg_hash.c b/libopkg/pkg_hash.c
+index a99cf6b..cc048c8 100644
+--- a/libopkg/pkg_hash.c
++++ b/libopkg/pkg_hash.c
+@@ -376,10 +376,22 @@ pkg_hash_fetch_best_installation_candidate(abstract_pkg_t *apkg,
+           if (constraint_fcn(matching, cdata)) {
+              opkg_msg(DEBUG, "Candidate: %s %s.\n",
+ 			     matching->name, matching->version) ;
+-             good_pkg_by_name = matching;
++             /* Respect to the arch priorities when given alternatives */
++             if (good_pkg_by_name) {
++                 if (matching->arch_priority >= good_pkg_by_name->arch_priority) {
++                     good_pkg_by_name = matching;
++                     opkg_msg(DEBUG, "%s %s wins by priority.\n",
++                         matching->name, matching->version) ;
++                 } else
++                     opkg_msg(DEBUG, "%s %s wins by priority.\n",
++                         good_pkg_by_name->name, good_pkg_by_name->version) ;
++             } else
++                 good_pkg_by_name = matching;
+ 	     /* It has been provided by hand, so it is what user want */
+-             if (matching->provided_by_hand == 1)
+-                break;
++             if (matching->provided_by_hand == 1) {
++                 good_pkg_by_name = matching;
++                 break;
++             }
+           }
+      }
+ 
+-- 
+1.7.1
+
diff --git a/meta/recipes-devtools/opkg/opkg_0.1.8.bb b/meta/recipes-devtools/opkg/opkg_0.1.8.bb
index c206b37..09ff3b5 100644
--- a/meta/recipes-devtools/opkg/opkg_0.1.8.bb
+++ b/meta/recipes-devtools/opkg/opkg_0.1.8.bb
@@ -3,6 +3,7 @@ require opkg.inc
 SRC_URI = "http://opkg.googlecode.com/files/opkg-${PV}.tar.gz \
            file://add_vercmp.patch \
            file://headerfix.patch \
+           file://respect-to-arch.patch \
           "
 
-PR = "${INC_PR}.0"
+PR = "${INC_PR}.1"
diff --git a/meta/recipes-devtools/opkg/opkg_svn.bb b/meta/recipes-devtools/opkg/opkg_svn.bb
index c07d393..890ed52 100644
--- a/meta/recipes-devtools/opkg/opkg_svn.bb
+++ b/meta/recipes-devtools/opkg/opkg_svn.bb
@@ -6,6 +6,7 @@ SRC_URI = "svn://opkg.googlecode.com/svn;module=trunk;proto=http \
            file://fix_installorder.patch \
            file://offline_postinstall.patch\
            file://track_parents.patch \
+           file://respect-to-arch.patch \
 "
 
 S = "${WORKDIR}/trunk"
@@ -13,4 +14,4 @@ S = "${WORKDIR}/trunk"
 SRCREV = "633"
 PV = "0.1.8+svnr${SRCPV}"
 
-PR = "${INC_PR}.0"
+PR = "${INC_PR}.1"
-- 
1.7.1

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Koen Kooi]

Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:

> There is a bug if we:
> 1) bitbake core-image-sato-sdk MACHINE=qemux86
> 2) bitbake core-image-sato with MACHINE=crownbay
> 
> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
> image even if there is one in deploy/ipk/core2 and we have set the
> core2's priority higher than i586, when the version in deploy/ipk/i586 is
> higher. This doesn't work for us, for example, what the crownbay need is
> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.

And this is working exactly as intended. Don't break opkg because your hardware driver situation sucks.

So: NAK on this patch.

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Robert Yang]

On 05/31/2012 11:01 PM, Koen Kooi wrote:
>
> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
>
>> There is a bug if we:
>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
>> 2) bitbake core-image-sato with MACHINE=crownbay
>>
>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
>> image even if there is one in deploy/ipk/core2 and we have set the
>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
>> higher. This doesn't work for us, for example, what the crownbay need is
>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
>
> And this is working exactly as intended. Don't break opkg because your hardware driver situation sucks.
>

Hi Koen,

Thanks for your great patient on this patch:-), I'm afraid that this is
not working exactly as intended, since the rpm or dpkg doesn't have this
issue. (I have fixed other issue for rpm, but not the same as this one,
the xserver-xorg-1.9.3.rpm had installed correctly before the fix for rpm).
and dpkg works well without any fix.

I'm also glad that if we won't fix it.

// Robert


> So: NAK on this patch.
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Richard Purdie]

On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
> 
> > There is a bug if we:
> > 1) bitbake core-image-sato-sdk MACHINE=qemux86
> > 2) bitbake core-image-sato with MACHINE=crownbay
> > 
> > Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
> > image even if there is one in deploy/ipk/core2 and we have set the
> > core2's priority higher than i586, when the version in deploy/ipk/i586 is
> > higher. This doesn't work for us, for example, what the crownbay need is
> > xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
> 
> And this is working exactly as intended. Don't break opkg because your
> hardware driver situation sucks.
> 
> So: NAK on this patch.

I think we do have a problem here. For example, the system is ignoring a
PREFERRED_VERSION directive here by building one thing and then
installing another. We're also inconsistent between the dpkg/rpm and
opkg backends. There is therefore definitely some kind of user
experience issue at stake here since this behaviour is not obvious,
expected or particularly correct.

The fact the example is hardware related is not particularly relevant,
its the bigger picture I worry about. I know that hardware issue sucks
and many people on this list have experienced pain due to it, we'd all
like it to go away. Using this as a reason not to examine and
potentially fix some problematic package manager behaviour is not right
IMO though. The world isn't perfect, sucky hardware/software exists, we
need to work with it.

So to be honest I'm leaning towards taking the patch. I would be
interested in other people's opinions though...

Cheers,

Richard

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Koen Kooi]

Op 1 jun. 2012, om 10:17 heeft Richard Purdie het volgende geschreven:

> On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
>> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
>> 
>>> There is a bug if we:
>>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
>>> 2) bitbake core-image-sato with MACHINE=crownbay
>>> 
>>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
>>> image even if there is one in deploy/ipk/core2 and we have set the
>>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
>>> higher. This doesn't work for us, for example, what the crownbay need is
>>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
>> 
>> And this is working exactly as intended. Don't break opkg because your
>> hardware driver situation sucks.
>> 
>> So: NAK on this patch.
> 
> I think we do have a problem here. For example, the system is ignoring a
> PREFERRED_VERSION directive here

A PREFERRED_VERSION set in a machine.conf, which is the wrong place. Let's change the above build sequence to this:

1) MACHINE=qemux86 bitbake xserver-xorg
2) MACHINE=othercore2machine bitbake xserver-xorg
3) MACHINE=crownbay bitbake xserver-xorg

Oh look, I get 1.11.2 on crownbay regardless of this patch.

> by building one thing and then
> installing another. We're also inconsistent between the dpkg/rpm and
> opkg backends. There is therefore definitely some kind of user
> experience issue at stake here since this behaviour is not obvious,
> expected or particularly correct.
> 
> The fact the example is hardware related is not particularly relevant,
> its the bigger picture I worry about

I also worry about the bigger picture, especially about what Martin said about this breaking feeds.

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Richard Purdie]

On Fri, 2012-06-01 at 11:04 +0200, Koen Kooi wrote:
> Op 1 jun. 2012, om 10:17 heeft Richard Purdie het volgende geschreven:
> 
> > On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
> >> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
> >> 
> >>> There is a bug if we:
> >>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
> >>> 2) bitbake core-image-sato with MACHINE=crownbay
> >>> 
> >>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
> >>> image even if there is one in deploy/ipk/core2 and we have set the
> >>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
> >>> higher. This doesn't work for us, for example, what the crownbay need is
> >>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
> >> 
> >> And this is working exactly as intended. Don't break opkg because your
> >> hardware driver situation sucks.
> >> 
> >> So: NAK on this patch.
> > 
> > I think we do have a problem here. For example, the system is ignoring a
> > PREFERRED_VERSION directive here
> 
> A PREFERRED_VERSION set in a machine.conf, which is the wrong place.

Its strongly not recommended. You can do it and if things are setup
correctly, we do support machine specific alterations however...

>  Let's change the above build sequence to this:
> 
> 1) MACHINE=qemux86 bitbake xserver-xorg
> 2) MACHINE=othercore2machine bitbake xserver-xorg
> 3) MACHINE=crownbay bitbake xserver-xorg
> 
> Oh look, I get 1.11.2 on crownbay regardless of this patch.

I've been assuming this xserver is marked as machine specific. If its
not, that is a bug and we can fix that.

> > by building one thing and then
> > installing another. We're also inconsistent between the dpkg/rpm and
> > opkg backends. There is therefore definitely some kind of user
> > experience issue at stake here since this behaviour is not obvious,
> > expected or particularly correct.
> > 
> > The fact the example is hardware related is not particularly relevant,
> > its the bigger picture I worry about
> 
> I also worry about the bigger picture, especially about what Martin
> said about this breaking feeds.

As far as I understood Martin's comments, this actually helps avoid some
of the issues he's been experiencing with feeds?

Martin has a problem where machines are ending up with unoptimised
versions of code on them and it would be good to fix opkg behaviour to
do what people expect...

Cheers,

Richard

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Koen Kooi]

Op 1 jun. 2012, om 12:02 heeft Richard Purdie het volgende geschreven:

> On Fri, 2012-06-01 at 11:04 +0200, Koen Kooi wrote:
>> Op 1 jun. 2012, om 10:17 heeft Richard Purdie het volgende geschreven:
>> 
>>> On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
>>>> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
>>>> 
>>>>> There is a bug if we:
>>>>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
>>>>> 2) bitbake core-image-sato with MACHINE=crownbay
>>>>> 
>>>>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
>>>>> image even if there is one in deploy/ipk/core2 and we have set the
>>>>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
>>>>> higher. This doesn't work for us, for example, what the crownbay need is
>>>>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
>>>> 
>>>> And this is working exactly as intended. Don't break opkg because your
>>>> hardware driver situation sucks.
>>>> 
>>>> So: NAK on this patch.
>>> 
>>> I think we do have a problem here. For example, the system is ignoring a
>>> PREFERRED_VERSION directive here
>> 
>> A PREFERRED_VERSION set in a machine.conf, which is the wrong place.
> 
> Its strongly not recommended. You can do it and if things are setup
> correctly, we do support machine specific alterations however...
> 
>> Let's change the above build sequence to this:
>> 
>> 1) MACHINE=qemux86 bitbake xserver-xorg
>> 2) MACHINE=othercore2machine bitbake xserver-xorg
>> 3) MACHINE=crownbay bitbake xserver-xorg
>> 
>> Oh look, I get 1.11.2 on crownbay regardless of this patch.
> 
> I've been assuming this xserver is marked as machine specific. If its
> not, that is a bug and we can fix that.

the X server not machine specific and that is NOT a bug. The recipe for the DDX only works with a specific X version. The real problem here is that xf86-video-something (again, not machine specific, think pci cards) has a strict requirement on the x server version. This is now purely a DISTRO problem. A few possible solutions are:

1) Lock X to 1.9.3 for all compatible x86 archs in your DISTRO
2) Lock X to 1.11.2 for all compatible x86 archs in your DISTRO, live with the crownbay breakage
3) Only use a package manager with no notion of compatible archs (dpkg) or with strong version pinning (rpm)
4) Be really careful with the build sequence and structure your feed configs to not have compatible archs in their feed URIs
5) remove 'x11' from DISTRO_FEATURES

This is a point where you cannot make everybody happy. But as you say below, let's decouple the above problem from the arch vs version discussion.

>>> by building one thing and then
>>> installing another. We're also inconsistent between the dpkg/rpm and
>>> opkg backends. There is therefore definitely some kind of user
>>> experience issue at stake here since this behaviour is not obvious,
>>> expected or particularly correct.
>>> 
>>> The fact the example is hardware related is not particularly relevant,
>>> its the bigger picture I worry about
>> 
>> I also worry about the bigger picture, especially about what Martin
>> said about this breaking feeds.
> 
> As far as I understood Martin's comments, this actually helps avoid some
> of the issues he's been experiencing with feeds?

No, it allows you to add a package-of-death. Example:

broken-app_1.0_machine.ipk is in the feeds, being machine specific
broken-apps author fixes the machine specific bits properly
broken-app_2.0_i586.ipk hits the feeds and is not picked up.

> Martin has a problem where machines are ending up with unoptimised
> versions of code on them and it would be good to fix opkg behaviour to
> do what people expect...

Yes, feed updates are not atomic, but this patch breaks moving packages between archs like the broken-app example above. In angstrom a machine doesn't see feeds with compatible archs at runtime, e.g. beagleboard sees beagleboard, armv7a and noarch URIs. This reduces these kind of problems to images built with OE. I must note that this setup for feed URIs was done out of bandwidth, storage and ram concerns (good old ipkg), not to avoid Martins problems :)

regards,

Koen

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Robert Yang]

On 06/01/2012 06:35 PM, Koen Kooi wrote:
>
> Op 1 jun. 2012, om 12:02 heeft Richard Purdie het volgende geschreven:
>
>> On Fri, 2012-06-01 at 11:04 +0200, Koen Kooi wrote:
>>> Op 1 jun. 2012, om 10:17 heeft Richard Purdie het volgende geschreven:
>>>
>>>> On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
>>>>> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
>>>>>
>>>>>> There is a bug if we:
>>>>>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
>>>>>> 2) bitbake core-image-sato with MACHINE=crownbay
>>>>>>
>>>>>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
>>>>>> image even if there is one in deploy/ipk/core2 and we have set the
>>>>>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
>>>>>> higher. This doesn't work for us, for example, what the crownbay need is
>>>>>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
>>>>>
>>>>> And this is working exactly as intended. Don't break opkg because your
>>>>> hardware driver situation sucks.
>>>>>
>>>>> So: NAK on this patch.
>>>>
>>>> I think we do have a problem here. For example, the system is ignoring a
>>>> PREFERRED_VERSION directive here
>>>
>>> A PREFERRED_VERSION set in a machine.conf, which is the wrong place.
>>
>> Its strongly not recommended. You can do it and if things are setup
>> correctly, we do support machine specific alterations however...
>>
>>> Let's change the above build sequence to this:
>>>
>>> 1) MACHINE=qemux86 bitbake xserver-xorg
>>> 2) MACHINE=othercore2machine bitbake xserver-xorg
>>> 3) MACHINE=crownbay bitbake xserver-xorg
>>>
>>> Oh look, I get 1.11.2 on crownbay regardless of this patch.
>>
>> I've been assuming this xserver is marked as machine specific. If its
>> not, that is a bug and we can fix that.
>
> the X server not machine specific and that is NOT a bug. The recipe for the DDX only works with a specific X version. The real problem here is that xf86-video-something (again, not machine specific, think pci cards) has a strict requirement on the x server version. This is now purely a DISTRO problem. A few possible solutions are:
>
> 1) Lock X to 1.9.3 for all compatible x86 archs in your DISTRO
> 2) Lock X to 1.11.2 for all compatible x86 archs in your DISTRO, live with the crownbay breakage
> 3) Only use a package manager with no notion of compatible archs (dpkg) or with strong version pinning (rpm)
> 4) Be really careful with the build sequence and structure your feed configs to not have compatible archs in their feed URIs
> 5) remove 'x11' from DISTRO_FEATURES
>
> This is a point where you cannot make everybody happy. But as you say below, let's decouple the above problem from the arch vs version discussion.
>
>>>> by building one thing and then
>>>> installing another. We're also inconsistent between the dpkg/rpm and
>>>> opkg backends. There is therefore definitely some kind of user
>>>> experience issue at stake here since this behaviour is not obvious,
>>>> expected or particularly correct.
>>>>
>>>> The fact the example is hardware related is not particularly relevant,
>>>> its the bigger picture I worry about
>>>
>>> I also worry about the bigger picture, especially about what Martin
>>> said about this breaking feeds.
>>
>> As far as I understood Martin's comments, this actually helps avoid some
>> of the issues he's been experiencing with feeds?
>
> No, it allows you to add a package-of-death. Example:
>
> broken-app_1.0_machine.ipk is in the feeds, being machine specific
> broken-apps author fixes the machine specific bits properly
> broken-app_2.0_i586.ipk hits the feeds and is not picked up.

I think that the broken-app_2.0_i586.ipk came unexpectedly,
there was a broken-app_1.0_machine.ipk (maybe also broken-app_1.0_i586.ipk,
but it didn't matter), if it has been fixed to version 2.0, then there should
be also a broken-app_2.0_machine.ipk (and it would be picked up).

// Robert

>
>> Martin has a problem where machines are ending up with unoptimised
>> versions of code on them and it would be good to fix opkg behaviour to
>> do what people expect...
>
> Yes, feed updates are not atomic, but this patch breaks moving packages between archs like the broken-app example above. In angstrom a machine doesn't see feeds with compatible archs at runtime, e.g. beagleboard sees beagleboard, armv7a and noarch URIs. This reduces these kind of problems to images built with OE. I must note that this setup for feed URIs was done out of bandwidth, storage and ram concerns (good old ipkg), not to avoid Martins problems :)
>
> regards,
>
> Koen
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5874 bytes --]

On Mon, Jun 04, 2012 at 05:31:26PM +0800, Robert Yang wrote:
> 
> 
> On 06/01/2012 06:35 PM, Koen Kooi wrote:
> >
> > Op 1 jun. 2012, om 12:02 heeft Richard Purdie het volgende geschreven:
> >
> >> On Fri, 2012-06-01 at 11:04 +0200, Koen Kooi wrote:
> >>> Op 1 jun. 2012, om 10:17 heeft Richard Purdie het volgende geschreven:
> >>>
> >>>> On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
> >>>>> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
> >>>>>
> >>>>>> There is a bug if we:
> >>>>>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
> >>>>>> 2) bitbake core-image-sato with MACHINE=crownbay
> >>>>>>
> >>>>>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
> >>>>>> image even if there is one in deploy/ipk/core2 and we have set the
> >>>>>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
> >>>>>> higher. This doesn't work for us, for example, what the crownbay need is
> >>>>>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
> >>>>>
> >>>>> And this is working exactly as intended. Don't break opkg because your
> >>>>> hardware driver situation sucks.
> >>>>>
> >>>>> So: NAK on this patch.
> >>>>
> >>>> I think we do have a problem here. For example, the system is ignoring a
> >>>> PREFERRED_VERSION directive here
> >>>
> >>> A PREFERRED_VERSION set in a machine.conf, which is the wrong place.
> >>
> >> Its strongly not recommended. You can do it and if things are setup
> >> correctly, we do support machine specific alterations however...
> >>
> >>> Let's change the above build sequence to this:
> >>>
> >>> 1) MACHINE=qemux86 bitbake xserver-xorg
> >>> 2) MACHINE=othercore2machine bitbake xserver-xorg
> >>> 3) MACHINE=crownbay bitbake xserver-xorg
> >>>
> >>> Oh look, I get 1.11.2 on crownbay regardless of this patch.
> >>
> >> I've been assuming this xserver is marked as machine specific. If its
> >> not, that is a bug and we can fix that.
> >
> > the X server not machine specific and that is NOT a bug. The recipe for the DDX only works with a specific X version. The real problem here is that xf86-video-something (again, not machine specific, think pci cards) has a strict requirement on the x server version. This is now purely a DISTRO problem. A few possible solutions are:
> >
> > 1) Lock X to 1.9.3 for all compatible x86 archs in your DISTRO
> > 2) Lock X to 1.11.2 for all compatible x86 archs in your DISTRO, live with the crownbay breakage
> > 3) Only use a package manager with no notion of compatible archs (dpkg) or with strong version pinning (rpm)
> > 4) Be really careful with the build sequence and structure your feed configs to not have compatible archs in their feed URIs
> > 5) remove 'x11' from DISTRO_FEATURES
> >
> > This is a point where you cannot make everybody happy. But as you say below, let's decouple the above problem from the arch vs version discussion.
> >
> >>>> by building one thing and then
> >>>> installing another. We're also inconsistent between the dpkg/rpm and
> >>>> opkg backends. There is therefore definitely some kind of user
> >>>> experience issue at stake here since this behaviour is not obvious,
> >>>> expected or particularly correct.
> >>>>
> >>>> The fact the example is hardware related is not particularly relevant,
> >>>> its the bigger picture I worry about
> >>>
> >>> I also worry about the bigger picture, especially about what Martin
> >>> said about this breaking feeds.
> >>
> >> As far as I understood Martin's comments, this actually helps avoid some
> >> of the issues he's been experiencing with feeds?
> >
> > No, it allows you to add a package-of-death. Example:
> >
> > broken-app_1.0_machine.ipk is in the feeds, being machine specific
> > broken-apps author fixes the machine specific bits properly
> > broken-app_2.0_i586.ipk hits the feeds and is not picked up.
> 
> I think that the broken-app_2.0_i586.ipk came unexpectedly,
> there was a broken-app_1.0_machine.ipk (maybe also broken-app_1.0_i586.ipk,
> but it didn't matter), if it has been fixed to version 2.0, then there should
> be also a broken-app_2.0_machine.ipk (and it would be picked up).

If broken-app developers are now detecting machine capabilities (or
whatever) since 2.0 in runtime, then we don't need to build broken-app
for every single machine we support, so changing PACKAGE_ARCH from
MACHINE_ARCH to TUNE_PKGARCH is right thing to do to save compile time
and feed disk space.

Same use-case broken by this patch reported here:
http://lists.linuxtogo.org/pipermail/openembedded-core/2012-May/023154.html

Cheers,

> 
> // Robert
> 
> >
> >> Martin has a problem where machines are ending up with unoptimised
> >> versions of code on them and it would be good to fix opkg behaviour to
> >> do what people expect...
> >
> > Yes, feed updates are not atomic, but this patch breaks moving packages between archs like the broken-app example above. In angstrom a machine doesn't see feeds with compatible archs at runtime, e.g. beagleboard sees beagleboard, armv7a and noarch URIs. This reduces these kind of problems to images built with OE. I must note that this setup for feed URIs was done out of bandwidth, storage and ram concerns (good old ipkg), not to avoid Martins problems :)
> >
> > regards,
> >
> > Koen
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
> >
> >
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 1/1] opkg 0.1.8: respect to the arch when choose the alternatives

[Koen Kooi]

Op 4 jun. 2012, om 12:39 heeft Martin Jansa het volgende geschreven:

> On Mon, Jun 04, 2012 at 05:31:26PM +0800, Robert Yang wrote:
>> 
>> 
>> On 06/01/2012 06:35 PM, Koen Kooi wrote:
>>> 
>>> Op 1 jun. 2012, om 12:02 heeft Richard Purdie het volgende geschreven:
>>> 
>>>> On Fri, 2012-06-01 at 11:04 +0200, Koen Kooi wrote:
>>>>> Op 1 jun. 2012, om 10:17 heeft Richard Purdie het volgende geschreven:
>>>>> 
>>>>>> On Thu, 2012-05-31 at 17:01 +0200, Koen Kooi wrote:
>>>>>>> Op 31 mei 2012, om 16:13 heeft Robert Yang het volgende geschreven:
>>>>>>> 
>>>>>>>> There is a bug if we:
>>>>>>>> 1) bitbake core-image-sato-sdk MACHINE=qemux86
>>>>>>>> 2) bitbake core-image-sato with MACHINE=crownbay
>>>>>>>> 
>>>>>>>> Then several pkgs in deploy/ipk/i586 would be installed to crownbay's
>>>>>>>> image even if there is one in deploy/ipk/core2 and we have set the
>>>>>>>> core2's priority higher than i586, when the version in deploy/ipk/i586 is
>>>>>>>> higher. This doesn't work for us, for example, what the crownbay need is
>>>>>>>> xserver-xorg-1.9.3, but it installs xserver-xorg-1.11.2.
>>>>>>> 
>>>>>>> And this is working exactly as intended. Don't break opkg because your
>>>>>>> hardware driver situation sucks.
>>>>>>> 
>>>>>>> So: NAK on this patch.
>>>>>> 
>>>>>> I think we do have a problem here. For example, the system is ignoring a
>>>>>> PREFERRED_VERSION directive here
>>>>> 
>>>>> A PREFERRED_VERSION set in a machine.conf, which is the wrong place.
>>>> 
>>>> Its strongly not recommended. You can do it and if things are setup
>>>> correctly, we do support machine specific alterations however...
>>>> 
>>>>> Let's change the above build sequence to this:
>>>>> 
>>>>> 1) MACHINE=qemux86 bitbake xserver-xorg
>>>>> 2) MACHINE=othercore2machine bitbake xserver-xorg
>>>>> 3) MACHINE=crownbay bitbake xserver-xorg
>>>>> 
>>>>> Oh look, I get 1.11.2 on crownbay regardless of this patch.
>>>> 
>>>> I've been assuming this xserver is marked as machine specific. If its
>>>> not, that is a bug and we can fix that.
>>> 
>>> the X server not machine specific and that is NOT a bug. The recipe for the DDX only works with a specific X version. The real problem here is that xf86-video-something (again, not machine specific, think pci cards) has a strict requirement on the x server version. This is now purely a DISTRO problem. A few possible solutions are:
>>> 
>>> 1) Lock X to 1.9.3 for all compatible x86 archs in your DISTRO
>>> 2) Lock X to 1.11.2 for all compatible x86 archs in your DISTRO, live with the crownbay breakage
>>> 3) Only use a package manager with no notion of compatible archs (dpkg) or with strong version pinning (rpm)
>>> 4) Be really careful with the build sequence and structure your feed configs to not have compatible archs in their feed URIs
>>> 5) remove 'x11' from DISTRO_FEATURES
>>> 
>>> This is a point where you cannot make everybody happy. But as you say below, let's decouple the above problem from the arch vs version discussion.
>>> 
>>>>>> by building one thing and then
>>>>>> installing another. We're also inconsistent between the dpkg/rpm and
>>>>>> opkg backends. There is therefore definitely some kind of user
>>>>>> experience issue at stake here since this behaviour is not obvious,
>>>>>> expected or particularly correct.
>>>>>> 
>>>>>> The fact the example is hardware related is not particularly relevant,
>>>>>> its the bigger picture I worry about
>>>>> 
>>>>> I also worry about the bigger picture, especially about what Martin
>>>>> said about this breaking feeds.
>>>> 
>>>> As far as I understood Martin's comments, this actually helps avoid some
>>>> of the issues he's been experiencing with feeds?
>>> 
>>> No, it allows you to add a package-of-death. Example:
>>> 
>>> broken-app_1.0_machine.ipk is in the feeds, being machine specific
>>> broken-apps author fixes the machine specific bits properly
>>> broken-app_2.0_i586.ipk hits the feeds and is not picked up.
>> 
>> I think that the broken-app_2.0_i586.ipk came unexpectedly,
>> there was a broken-app_1.0_machine.ipk (maybe also broken-app_1.0_i586.ipk,
>> but it didn't matter), if it has been fixed to version 2.0, then there should
>> be also a broken-app_2.0_machine.ipk (and it would be picked up).
> 
> If broken-app developers are now detecting machine capabilities (or
> whatever) since 2.0 in runtime, then we don't need to build broken-app
> for every single machine we support, so changing PACKAGE_ARCH from
> MACHINE_ARCH to TUNE_PKGARCH is right thing to do to save compile time
> and feed disk space.

Exactly

> Same use-case broken by this patch reported here:
> http://lists.linuxtogo.org/pipermail/openembedded-core/2012-May/023154.html

I borrowed heavily from your post since both Richard and Robert don't seem to see the damage this will do to feeds when a package/recipe changes archs.

regards,

Koen