How do I change the "Architecture" of a package?

How do I change the "Architecture" of a package?

[Mike Looijmans]

What if the architecture of a package was accidentally left at its 
default, but it should have been "all" for example?

Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the 
recipe is not enough. You get stuck with a "more specific" older 
version, so that no device wants to upgrade to the newer version that's 
"all" architecture compatible.

Only way to fix this was to manually remove the old version from the 
feeds and also manually remove it from the device, and then reinstall 
it. This works for me personally, but what if I want to apply this to 
about a million devices that are already in the field?

What can I do to get them to upgrade from the (in this case) "mipsel32" 
to "all" package?

(and I have a few dozen of these mis-architectured packages...)

-- 
Mike Looijmans

Re: How do I change the "Architecture" of a package?

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
> What if the architecture of a package was accidentally left at its
> default, but it should have been "all" for example?
> 
> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
> recipe is not enough. You get stuck with a "more specific" older
> version, so that no device wants to upgrade to the newer version
> that's "all" architecture compatible.
>

What package manager are you using on the device? If you're using opkg it should
prioritise by version not arch unless the command line option
'--prefer-arch-to-version' is passed. If you're using opkg and it's not doing
that, let me know and I'll look into it when I get chance to.

> Only way to fix this was to manually remove the old version from the
> feeds and also manually remove it from the device, and then
> reinstall it. This works for me personally, but what if I want to
> apply this to about a million devices that are already in the field?
> 
> What can I do to get them to upgrade from the (in this case)
> "mipsel32" to "all" package?

As an absolute hack, you could rename the package and set it to
replace/conflict/provide the old package name. I think there should be better
solutions though.

> 
> (and I have a few dozen of these mis-architectured packages...)
> 
> -- 
> Mike Looijmans
> -- 

Thanks,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: How do I change the "Architecture" of a package?

[Mike Looijmans]

On 30-12-2014 18:59, Paul Barker wrote:
> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>> What if the architecture of a package was accidentally left at its
>> default, but it should have been "all" for example?
>>
>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>> recipe is not enough. You get stuck with a "more specific" older
>> version, so that no device wants to upgrade to the newer version
>> that's "all" architecture compatible.
>>
>
> What package manager are you using on the device? If you're using opkg it should
> prioritise by version not arch unless the command line option
> '--prefer-arch-to-version' is passed. If you're using opkg and it's not doing
> that, let me know and I'll look into it when I get chance to.

It's opkg.

But on closer inspection I noticed that the "git" version is also 
mysteriously reset to 0, so that the package also gets a lower version 
number instead of a higher one. Seems to be the PR server borking things 
again or so, I'll have to investigate that next year...

>> Only way to fix this was to manually remove the old version from the
>> feeds and also manually remove it from the device, and then
>> reinstall it. This works for me personally, but what if I want to
>> apply this to about a million devices that are already in the field?
>>
>> What can I do to get them to upgrade from the (in this case)
>> "mipsel32" to "all" package?
>
> As an absolute hack, you could rename the package and set it to
> replace/conflict/provide the old package name. I think there should be better
> solutions though.

I don't think so, I've never seen one in the wild. I'll send a free 
Linux TV satellite settopbox to whomever provides a solution ;)

Re: How do I change the "Architecture" of a package?

[Mike Looijmans]

On 12/31/2014 08:13 PM, Mike Looijmans wrote:
> On 30-12-2014 18:59, Paul Barker wrote:
>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>>> What if the architecture of a package was accidentally left at its
>>> default, but it should have been "all" for example?
>>>
>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>>> recipe is not enough. You get stuck with a "more specific" older
>>> version, so that no device wants to upgrade to the newer version
>>> that's "all" architecture compatible.
>>>
>>
>> What package manager are you using on the device? If you're using opkg
>> it should
>> prioritise by version not arch unless the command line option
>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
>> not doing
>> that, let me know and I'll look into it when I get chance to.
>
> It's opkg.
>
> But on closer inspection I noticed that the "git" version is also
> mysteriously reset to 0, so that the package also gets a lower version
> number instead of a higher one. Seems to be the PR server borking things
> again or so, I'll have to investigate that next year...

Weird, something in OE killed "gitpkgv".

in the recipe, I have this:

inherit gitpkgv
PV = "2.0+git${SRCPV}"
PKGV = "2.0+git${GITPKGV}"


$ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV

delivers correct information:

PKGV="2.0+git68+2e7a1db"
GITPKGVTAG="0.0-68-g2e7a1db"
GITPKGV="68+2e7a1db"


But after building and deploying the package, the version number will 
eventually end up being this one:

2.0+git5+2e7a1db509-r0.2


What in OE is replacing a perfectly good PKGV tag with something 
completely different bearing no relation whatsoever? Even the number of 
digits in the git tag differs from the one I put in the recipe!

Even if I put some random text into PKGV, it gets replaced.

-- 
Mike Looijmans

Re: How do I change the "Architecture" of a package?

[Richard Purdie]

On Fri, 2015-01-02 at 09:48 +0100, Mike Looijmans wrote:
> On 12/31/2014 08:13 PM, Mike Looijmans wrote:
> > On 30-12-2014 18:59, Paul Barker wrote:
> >> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
> >>> What if the architecture of a package was accidentally left at its
> >>> default, but it should have been "all" for example?
> >>>
> >>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
> >>> recipe is not enough. You get stuck with a "more specific" older
> >>> version, so that no device wants to upgrade to the newer version
> >>> that's "all" architecture compatible.
> >>>
> >>
> >> What package manager are you using on the device? If you're using opkg
> >> it should
> >> prioritise by version not arch unless the command line option
> >> '--prefer-arch-to-version' is passed. If you're using opkg and it's
> >> not doing
> >> that, let me know and I'll look into it when I get chance to.
> >
> > It's opkg.
> >
> > But on closer inspection I noticed that the "git" version is also
> > mysteriously reset to 0, so that the package also gets a lower version
> > number instead of a higher one. Seems to be the PR server borking things
> > again or so, I'll have to investigate that next year...
> 
> Weird, something in OE killed "gitpkgv".
> 
> in the recipe, I have this:
> 
> inherit gitpkgv
> PV = "2.0+git${SRCPV}"
> PKGV = "2.0+git${GITPKGV}"
> 
> 
> $ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV
> 
> delivers correct information:
> 
> PKGV="2.0+git68+2e7a1db"
> GITPKGVTAG="0.0-68-g2e7a1db"
> GITPKGV="68+2e7a1db"
> 
> 
> But after building and deploying the package, the version number will 
> eventually end up being this one:
> 
> 2.0+git5+2e7a1db509-r0.2
> 
> 
> What in OE is replacing a perfectly good PKGV tag with something 
> completely different bearing no relation whatsoever? Even the number of 
> digits in the git tag differs from the one I put in the recipe!
> 
> Even if I put some random text into PKGV, it gets replaced.

Did something come from sstate?

Cheers,

Richard

Re: How do I change the "Architecture" of a package?

[Mike Looijmans]

On 01/02/2015 10:16 AM, Richard Purdie wrote:
> On Fri, 2015-01-02 at 09:48 +0100, Mike Looijmans wrote:
>> On 12/31/2014 08:13 PM, Mike Looijmans wrote:
>>> On 30-12-2014 18:59, Paul Barker wrote:
>>>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>>>>> What if the architecture of a package was accidentally left at its
>>>>> default, but it should have been "all" for example?
>>>>>
>>>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>>>>> recipe is not enough. You get stuck with a "more specific" older
>>>>> version, so that no device wants to upgrade to the newer version
>>>>> that's "all" architecture compatible.
>>>>>
>>>>
>>>> What package manager are you using on the device? If you're using opkg
>>>> it should
>>>> prioritise by version not arch unless the command line option
>>>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
>>>> not doing
>>>> that, let me know and I'll look into it when I get chance to.
>>>
>>> It's opkg.
>>>
>>> But on closer inspection I noticed that the "git" version is also
>>> mysteriously reset to 0, so that the package also gets a lower version
>>> number instead of a higher one. Seems to be the PR server borking things
>>> again or so, I'll have to investigate that next year...
>>
>> Weird, something in OE killed "gitpkgv".
>>
>> in the recipe, I have this:
>>
>> inherit gitpkgv
>> PV = "2.0+git${SRCPV}"
>> PKGV = "2.0+git${GITPKGV}"
>>
>>
>> $ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV
>>
>> delivers correct information:
>>
>> PKGV="2.0+git68+2e7a1db"
>> GITPKGVTAG="0.0-68-g2e7a1db"
>> GITPKGV="68+2e7a1db"
>>
>>
>> But after building and deploying the package, the version number will
>> eventually end up being this one:
>>
>> 2.0+git5+2e7a1db509-r0.2
>>
>>
>> What in OE is replacing a perfectly good PKGV tag with something
>> completely different bearing no relation whatsoever? Even the number of
>> digits in the git tag differs from the one I put in the recipe!
>>
>> Even if I put some random text into PKGV, it gets replaced.
>
> Did something come from sstate?

Nope, I do a "bitbake -c cleansstate [package]" first.

i've been experimenting for a while now. It appears that whatever I put 
into PKGV gets ignored and replaced with what was in PV.

Here's a recipe where I stubbed the actual building and install phases 
to demonstrate the issue:

(cut here)

DESCRIPTION = "Version numbers are borked"

inherit gitpkgv
PV = "z-pv+${SRCPV}"
PKGV = "z-pkgv+${GITPKGV}"

# Completely random public GIT repo
SRCREV = "${AUTOREV}"
MODULE = "AutoBackup"
SRC_URI = "git://github.com/E2OpenPlugins/e2openplugin-${MODULE}.git"

LICENSE = "GPLv2"
LIC_FILES_CHKSUM = 
"file://../LICENSE.GPLv2;md5=eb723b61539feef013de476e68b5c50a"
SRC_URI_append = " file://LICENSE.GPLv2"

S="${WORKDIR}/git"

do_compile() {
	touch ${B}/helloworld
}

do_install() {
	install -d ${D}${sysconfdir}
	install -m 644 ${B}/helloworld ${D}${sysconfdir}/
}

(EOF)

If I build this package, it ends up getting version 
"z-pv+0+2e7a1db509-r0.1" instead of "z-pkgv+68+2e7a1db509-r0.1" which 
I'd expect.


-- 
Mike Looijmans

Re: How do I change the "Architecture" of a package?

[Mike Looijmans]

On 01/02/2015 10:24 AM, Mike Looijmans wrote:
> On 01/02/2015 10:16 AM, Richard Purdie wrote:
>> On Fri, 2015-01-02 at 09:48 +0100, Mike Looijmans wrote:
>>> On 12/31/2014 08:13 PM, Mike Looijmans wrote:
>>>> On 30-12-2014 18:59, Paul Barker wrote:
>>>>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>>>>>> What if the architecture of a package was accidentally left at its
>>>>>> default, but it should have been "all" for example?
>>>>>>
>>>>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>>>>>> recipe is not enough. You get stuck with a "more specific" older
>>>>>> version, so that no device wants to upgrade to the newer version
>>>>>> that's "all" architecture compatible.
>>>>>>
>>>>>
>>>>> What package manager are you using on the device? If you're using opkg
>>>>> it should
>>>>> prioritise by version not arch unless the command line option
>>>>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
>>>>> not doing
>>>>> that, let me know and I'll look into it when I get chance to.
>>>>
>>>> It's opkg.
>>>>
>>>> But on closer inspection I noticed that the "git" version is also
>>>> mysteriously reset to 0, so that the package also gets a lower version
>>>> number instead of a higher one. Seems to be the PR server borking
>>>> things
>>>> again or so, I'll have to investigate that next year...
>>>
>>> Weird, something in OE killed "gitpkgv".
>>>
>>> in the recipe, I have this:
>>>
>>> inherit gitpkgv
>>> PV = "2.0+git${SRCPV}"
>>> PKGV = "2.0+git${GITPKGV}"
>>>
>>>
>>> $ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV
>>>
>>> delivers correct information:
>>>
>>> PKGV="2.0+git68+2e7a1db"
>>> GITPKGVTAG="0.0-68-g2e7a1db"
>>> GITPKGV="68+2e7a1db"
>>>
>>>
>>> But after building and deploying the package, the version number will
>>> eventually end up being this one:
>>>
>>> 2.0+git5+2e7a1db509-r0.2
>>>
>>>
>>> What in OE is replacing a perfectly good PKGV tag with something
>>> completely different bearing no relation whatsoever? Even the number of
>>> digits in the git tag differs from the one I put in the recipe!
>>>
>>> Even if I put some random text into PKGV, it gets replaced.
>>
>> Did something come from sstate?
>
> Nope, I do a "bitbake -c cleansstate [package]" first.
>
> i've been experimenting for a while now. It appears that whatever I put
> into PKGV gets ignored and replaced with what was in PV.
>
> Here's a recipe where I stubbed the actual building and install phases
> to demonstrate the issue:
>
> (cut here)
>
> DESCRIPTION = "Version numbers are borked"
>
> inherit gitpkgv
> PV = "z-pv+${SRCPV}"
> PKGV = "z-pkgv+${GITPKGV}"
>
> # Completely random public GIT repo
> SRCREV = "${AUTOREV}"
> MODULE = "AutoBackup"
> SRC_URI = "git://github.com/E2OpenPlugins/e2openplugin-${MODULE}.git"
>
> LICENSE = "GPLv2"
> LIC_FILES_CHKSUM =
> "file://../LICENSE.GPLv2;md5=eb723b61539feef013de476e68b5c50a"
> SRC_URI_append = " file://LICENSE.GPLv2"
>
> S="${WORKDIR}/git"
>
> do_compile() {
>      touch ${B}/helloworld
> }
>
> do_install() {
>      install -d ${D}${sysconfdir}
>      install -m 644 ${B}/helloworld ${D}${sysconfdir}/
> }
>
> (EOF)
>
> If I build this package, it ends up getting version
> "z-pv+0+2e7a1db509-r0.1" instead of "z-pkgv+68+2e7a1db509-r0.1" which
> I'd expect.
>
>

Just to be complete, I'm a bit behind on OE, using tag 
"2cc1bd9dd060f5002c2fde7aacba86fe230c12af" currently. Don't think it'd 
make a difference though.


-- 
Mike Looijmans

Re: How do I change the "Architecture" of a package?

[Mike Looijmans]

On 12/31/2014 08:13 PM, Mike Looijmans wrote:
> On 30-12-2014 18:59, Paul Barker wrote:
>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>>> What if the architecture of a package was accidentally left at its
>>> default, but it should have been "all" for example?
>>>
>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>>> recipe is not enough. You get stuck with a "more specific" older
>>> version, so that no device wants to upgrade to the newer version
>>> that's "all" architecture compatible.
>>>
>>
>> What package manager are you using on the device? If you're using opkg
>> it should
>> prioritise by version not arch unless the command line option
>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
>> not doing
>> that, let me know and I'll look into it when I get chance to.
>
> It's opkg.
>
> But on closer inspection I noticed that the "git" version is also
> mysteriously reset to 0, so that the package also gets a lower version
> number instead of a higher one. Seems to be the PR server borking things
> again or so, I'll have to investigate that next year...

Well, changing the architecture will always fail because of a bug in the 
PR server. It resets the number to "0" when PACKAGE_ARCH changes, and 
thus breaks the upgrade path.

-- 
Mike Looijmans

Bug: PR server changes the PKGV variable too

[Mike Looijmans]

On 01/02/2015 10:16 AM, Richard Purdie wrote:
> On Fri, 2015-01-02 at 09:48 +0100, Mike Looijmans wrote:
>> On 12/31/2014 08:13 PM, Mike Looijmans wrote:
>>> On 30-12-2014 18:59, Paul Barker wrote:
>>>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>>>>> What if the architecture of a package was accidentally left at its
>>>>> default, but it should have been "all" for example?
>>>>>
>>>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>>>>> recipe is not enough. You get stuck with a "more specific" older
>>>>> version, so that no device wants to upgrade to the newer version
>>>>> that's "all" architecture compatible.
>>>>>
>>>>
>>>> What package manager are you using on the device? If you're using opkg
>>>> it should
>>>> prioritise by version not arch unless the command line option
>>>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
>>>> not doing
>>>> that, let me know and I'll look into it when I get chance to.
>>>
>>> It's opkg.
>>>
>>> But on closer inspection I noticed that the "git" version is also
>>> mysteriously reset to 0, so that the package also gets a lower version
>>> number instead of a higher one. Seems to be the PR server borking things
>>> again or so, I'll have to investigate that next year...
>>
>> Weird, something in OE killed "gitpkgv".
>>
>> in the recipe, I have this:
>>
>> inherit gitpkgv
>> PV = "2.0+git${SRCPV}"
>> PKGV = "2.0+git${GITPKGV}"
>>
>>
>> $ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV
>>
>> delivers correct information:
>>
>> PKGV="2.0+git68+2e7a1db"
>> GITPKGVTAG="0.0-68-g2e7a1db"
>> GITPKGV="68+2e7a1db"
>>
>>
>> But after building and deploying the package, the version number will
>> eventually end up being this one:
>>
>> 2.0+git5+2e7a1db509-r0.2
>>
>>
>> What in OE is replacing a perfectly good PKGV tag with something
>> completely different bearing no relation whatsoever? Even the number of
>> digits in the git tag differs from the one I put in the recipe!
>>
>> Even if I put some random text into PKGV, it gets replaced.
>
> Did something come from sstate?

I got todays master from openembedded-core and meta-openembedded, and 
the recipe above (with a fix for the license). I leave everything as the 
script "oe-init-build-env" does, I only add meta-openembedded/meta-oe to 
the bblayers.conf list to get access to the gitpkgv class.

I named the recipe above "weirdversion.bb", and when I build it "as is", 
I get the correct version number "z-pkgv+68+2e7a1db-r0" (with "pkgv" and 
"68" in it).

However, when I add the following line to local.conf to activate the 
PR-SERVER, things go wrong:

PRSERV_HOST = "localhost:0"

After this change, the package suddenly gets "z-pv+0+2e7a1db509-r0.0" as 
version string, so apparently the PR server kills the PKGV variable and 
replaces it with something derived from PV alone.

Since this is a completely clean situation with nothing but core code, 
it must be a bug in the PR server (it seems to act as PV server instead...).


-- 
Mike Looijmans

Re: Bug: PR server changes the PKGV variable too

[Richard Purdie]

On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
> On 01/02/2015 10:16 AM, Richard Purdie wrote:
> > On Fri, 2015-01-02 at 09:48 +0100, Mike Looijmans wrote:
> >> On 12/31/2014 08:13 PM, Mike Looijmans wrote:
> >>> On 30-12-2014 18:59, Paul Barker wrote:
> >>>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
> >>>>> What if the architecture of a package was accidentally left at its
> >>>>> default, but it should have been "all" for example?
> >>>>>
> >>>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
> >>>>> recipe is not enough. You get stuck with a "more specific" older
> >>>>> version, so that no device wants to upgrade to the newer version
> >>>>> that's "all" architecture compatible.
> >>>>>
> >>>>
> >>>> What package manager are you using on the device? If you're using opkg
> >>>> it should
> >>>> prioritise by version not arch unless the command line option
> >>>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
> >>>> not doing
> >>>> that, let me know and I'll look into it when I get chance to.
> >>>
> >>> It's opkg.
> >>>
> >>> But on closer inspection I noticed that the "git" version is also
> >>> mysteriously reset to 0, so that the package also gets a lower version
> >>> number instead of a higher one. Seems to be the PR server borking things
> >>> again or so, I'll have to investigate that next year...
> >>
> >> Weird, something in OE killed "gitpkgv".
> >>
> >> in the recipe, I have this:
> >>
> >> inherit gitpkgv
> >> PV = "2.0+git${SRCPV}"
> >> PKGV = "2.0+git${GITPKGV}"
> >>
> >>
> >> $ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV
> >>
> >> delivers correct information:
> >>
> >> PKGV="2.0+git68+2e7a1db"
> >> GITPKGVTAG="0.0-68-g2e7a1db"
> >> GITPKGV="68+2e7a1db"
> >>
> >>
> >> But after building and deploying the package, the version number will
> >> eventually end up being this one:
> >>
> >> 2.0+git5+2e7a1db509-r0.2
> >>
> >>
> >> What in OE is replacing a perfectly good PKGV tag with something
> >> completely different bearing no relation whatsoever? Even the number of
> >> digits in the git tag differs from the one I put in the recipe!
> >>
> >> Even if I put some random text into PKGV, it gets replaced.
> >
> > Did something come from sstate?
> 
> I got todays master from openembedded-core and meta-openembedded, and 
> the recipe above (with a fix for the license). I leave everything as the 
> script "oe-init-build-env" does, I only add meta-openembedded/meta-oe to 
> the bblayers.conf list to get access to the gitpkgv class.
> 
> I named the recipe above "weirdversion.bb", and when I build it "as is", 
> I get the correct version number "z-pkgv+68+2e7a1db-r0" (with "pkgv" and 
> "68" in it).
> 
> However, when I add the following line to local.conf to activate the 
> PR-SERVER, things go wrong:
> 
> PRSERV_HOST = "localhost:0"
> 
> After this change, the package suddenly gets "z-pv+0+2e7a1db509-r0.0" as 
> version string, so apparently the PR server kills the PKGV variable and 
> replaces it with something derived from PV alone.
> 
> Since this is a completely clean situation with nothing but core code, 
> it must be a bug in the PR server (it seems to act as PV server instead...).

Imagine you're not using gitpkgv. You set:

PV = "x.y+${SRCPV}"

Since SRCPV contains a revision hash, you can end up in a situation
where the version changes and you cannot upgrade the package since the
hash didn't 'increase'.

The PR server therefore combines with the git fetcher to add an
incremental number at the start of the SRCPV string and yes, in that
scenario, it acts as a PV server. This is actually working as designed.

When you add in gitpkgv, something is obviously going wrong. gitpkgv is
in meta-oe since I've refused to add it to core on the several occasions
its been requested. I've said this before but I will say is again, it
*needs* become part of the standard fetcher API rather than a hacked in
afterthought which doesn't integrate well.

Cheers,

Richard

Re: Bug: PR server changes the PKGV variable too

[Mike Looijmans]

On 01/05/2015 10:27 AM, Richard Purdie wrote:
> On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
>> On 01/02/2015 10:16 AM, Richard Purdie wrote:
>>> On Fri, 2015-01-02 at 09:48 +0100, Mike Looijmans wrote:
>>>> On 12/31/2014 08:13 PM, Mike Looijmans wrote:
>>>>> On 30-12-2014 18:59, Paul Barker wrote:
>>>>>> On Tue, Dec 30, 2014 at 04:24:34PM +0100, Mike Looijmans wrote:
>>>>>>> What if the architecture of a package was accidentally left at its
>>>>>>> default, but it should have been "all" for example?
>>>>>>>
>>>>>>> Just putting "inherit allarch" or simply PACKAGE_ARCH="all" into the
>>>>>>> recipe is not enough. You get stuck with a "more specific" older
>>>>>>> version, so that no device wants to upgrade to the newer version
>>>>>>> that's "all" architecture compatible.
>>>>>>>
>>>>>>
>>>>>> What package manager are you using on the device? If you're using opkg
>>>>>> it should
>>>>>> prioritise by version not arch unless the command line option
>>>>>> '--prefer-arch-to-version' is passed. If you're using opkg and it's
>>>>>> not doing
>>>>>> that, let me know and I'll look into it when I get chance to.
>>>>>
>>>>> It's opkg.
>>>>>
>>>>> But on closer inspection I noticed that the "git" version is also
>>>>> mysteriously reset to 0, so that the package also gets a lower version
>>>>> number instead of a higher one. Seems to be the PR server borking things
>>>>> again or so, I'll have to investigate that next year...
>>>>
>>>> Weird, something in OE killed "gitpkgv".
>>>>
>>>> in the recipe, I have this:
>>>>
>>>> inherit gitpkgv
>>>> PV = "2.0+git${SRCPV}"
>>>> PKGV = "2.0+git${GITPKGV}"
>>>>
>>>>
>>>> $ bitbake enigma2-plugin-extensions-autobackup -e | grep PKGV
>>>>
>>>> delivers correct information:
>>>>
>>>> PKGV="2.0+git68+2e7a1db"
>>>> GITPKGVTAG="0.0-68-g2e7a1db"
>>>> GITPKGV="68+2e7a1db"
>>>>
>>>>
>>>> But after building and deploying the package, the version number will
>>>> eventually end up being this one:
>>>>
>>>> 2.0+git5+2e7a1db509-r0.2
>>>>
>>>>
>>>> What in OE is replacing a perfectly good PKGV tag with something
>>>> completely different bearing no relation whatsoever? Even the number of
>>>> digits in the git tag differs from the one I put in the recipe!
>>>>
>>>> Even if I put some random text into PKGV, it gets replaced.
>>>
>>> Did something come from sstate?
>>
>> I got todays master from openembedded-core and meta-openembedded, and
>> the recipe above (with a fix for the license). I leave everything as the
>> script "oe-init-build-env" does, I only add meta-openembedded/meta-oe to
>> the bblayers.conf list to get access to the gitpkgv class.
>>
>> I named the recipe above "weirdversion.bb", and when I build it "as is",
>> I get the correct version number "z-pkgv+68+2e7a1db-r0" (with "pkgv" and
>> "68" in it).
>>
>> However, when I add the following line to local.conf to activate the
>> PR-SERVER, things go wrong:
>>
>> PRSERV_HOST = "localhost:0"
>>
>> After this change, the package suddenly gets "z-pv+0+2e7a1db509-r0.0" as
>> version string, so apparently the PR server kills the PKGV variable and
>> replaces it with something derived from PV alone.
>>
>> Since this is a completely clean situation with nothing but core code,
>> it must be a bug in the PR server (it seems to act as PV server instead...).
>
> Imagine you're not using gitpkgv. You set:
>
> PV = "x.y+${SRCPV}"
>
> Since SRCPV contains a revision hash, you can end up in a situation
> where the version changes and you cannot upgrade the package since the
> hash didn't 'increase'.
>
> The PR server therefore combines with the git fetcher to add an
> incremental number at the start of the SRCPV string and yes, in that
> scenario, it acts as a PV server. This is actually working as designed.

Then the design is wrong. If a package chose to override PKGV manually, then 
the rest of the system should leave that value as is, and not touch it. 
Apparently the recipe author knows better, so please let him use that wisdom.

Also, if you change the architecture of the package, the PR server will reset 
the version counter to 0 and break the upgrade path too. That was the problem 
that caused me to discover this problem, the PR server is making it hard to 
fix arch errors in recipes.

> When you add in gitpkgv, something is obviously going wrong. gitpkgv is
> in meta-oe since I've refused to add it to core on the several occasions
> its been requested. I've said this before but I will say is again, it
> *needs* become part of the standard fetcher API rather than a hacked in
> afterthought which doesn't integrate well.

I already volunteered and tried to do that, but got stuck in lack of 
understanding how the fetcher works, and did not get any help so abandoned it 
in favor of keep using gitpkgv "as is". I'm still volunteering, but without 
help I can't do it.

M.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Bug: PR server changes the PKGV variable too

[Richard Purdie]

On Mon, 2015-01-05 at 10:41 +0100, Mike Looijmans wrote:
> On 01/05/2015 10:27 AM, Richard Purdie wrote:
> > On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
> > Imagine you're not using gitpkgv. You set:
> >
> > PV = "x.y+${SRCPV}"
> >
> > Since SRCPV contains a revision hash, you can end up in a situation
> > where the version changes and you cannot upgrade the package since the
> > hash didn't 'increase'.
> >
> > The PR server therefore combines with the git fetcher to add an
> > incremental number at the start of the SRCPV string and yes, in that
> > scenario, it acts as a PV server. This is actually working as designed.
> 
> Then the design is wrong. If a package chose to override PKGV manually, then 
> the rest of the system should leave that value as is, and not touch it. 
> Apparently the recipe author knows better, so please let him use that wisdom.
> 
> Also, if you change the architecture of the package, the PR server will reset 
> the version counter to 0 and break the upgrade path too. That was the problem 
> that caused me to discover this problem, the PR server is making it hard to 
> fix arch errors in recipes.

You realise why it does that though? Imagine multiple MACHINE values
being built against a PR server. Each MACHINE will have different
package architecture values and different hashes. The 'PR' values should
therefore be seen as different. If the system didn't do this, it would
increment PR for every MACHINE change.

Ok, so you say, lets make it work against MACHINE. That fails in the
case where multiple MACHINES share a package architecture :/. Its not a
simple problem to try and deal with :(

I make no claim the system is perfect or free from bugs but these
behaviours you're referring to are like this for various reasons. I'm
open to changing them but we need to address the underlying problems
like changing MACHINE above.

There are several other issues with changing package architecture such
as the sstate files in the sysroot. There is an open bug for this and
right now, I simply don't know how to solve it properly :(.

> > When you add in gitpkgv, something is obviously going wrong. gitpkgv is
> > in meta-oe since I've refused to add it to core on the several occasions
> > its been requested. I've said this before but I will say is again, it
> > *needs* become part of the standard fetcher API rather than a hacked in
> > afterthought which doesn't integrate well.
> 
> I already volunteered and tried to do that, but got stuck in lack of 
> understanding how the fetcher works, and did not get any help so abandoned it 
> in favor of keep using gitpkgv "as is". I'm still volunteering, but without 
> help I can't do it.

I'm struggling since to provide the help you need, I'd nearly have to
solve the problem myself. I've helped several different people
understand the fetcher in the past, only to have most of them move onto
other things :(. Add in the 101 other distractions I have and its
frustrating for everyone including me. Can you remind me where you were
at (links to the right emails would help)?

Cheers,

Richard

Re: Bug: PR server changes the PKGV variable too

[Mike Looijmans]

On 01/05/2015 11:07 AM, Richard Purdie wrote:
> On Mon, 2015-01-05 at 10:41 +0100, Mike Looijmans wrote:
>> On 01/05/2015 10:27 AM, Richard Purdie wrote:
>>> On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
>>> Imagine you're not using gitpkgv. You set:
>>>
>>> PV = "x.y+${SRCPV}"
>>>
>>> Since SRCPV contains a revision hash, you can end up in a situation
>>> where the version changes and you cannot upgrade the package since the
>>> hash didn't 'increase'.
>>>
>>> The PR server therefore combines with the git fetcher to add an
>>> incremental number at the start of the SRCPV string and yes, in that
>>> scenario, it acts as a PV server. This is actually working as designed.
>>
>> Then the design is wrong. If a package chose to override PKGV manually, then
>> the rest of the system should leave that value as is, and not touch it.
>> Apparently the recipe author knows better, so please let him use that wisdom.
>>
>> Also, if you change the architecture of the package, the PR server will reset
>> the version counter to 0 and break the upgrade path too. That was the problem
>> that caused me to discover this problem, the PR server is making it hard to
>> fix arch errors in recipes.
>
> You realise why it does that though? Imagine multiple MACHINE values
> being built against a PR server. Each MACHINE will have different
> package architecture values and different hashes. The 'PR' values should
> therefore be seen as different. If the system didn't do this, it would
> increment PR for every MACHINE change.
>
> Ok, so you say, lets make it work against MACHINE. That fails in the
> case where multiple MACHINES share a package architecture :/. Its not a
> simple problem to try and deal with :(
>
> I make no claim the system is perfect or free from bugs but these
> behaviours you're referring to are like this for various reasons. I'm
> open to changing them but we need to address the underlying problems
> like changing MACHINE above.
>
> There are several other issues with changing package architecture such
> as the sstate files in the sysroot. There is an open bug for this and
> right now, I simply don't know how to solve it properly :(.

Yeah, I stand corrected, this is not something easily fixed.

I'd still like to have a way to override the PR server. Is there a way to make 
the PR server ignore a package, or at least, have it NOT modify the PKGV 
variable for a package?
And if not, would you accept a patch to add that functionality to the prserver 
system?

>>> When you add in gitpkgv, something is obviously going wrong. gitpkgv is
>>> in meta-oe since I've refused to add it to core on the several occasions
>>> its been requested. I've said this before but I will say is again, it
>>> *needs* become part of the standard fetcher API rather than a hacked in
>>> afterthought which doesn't integrate well.
>>
>> I already volunteered and tried to do that, but got stuck in lack of
>> understanding how the fetcher works, and did not get any help so abandoned it
>> in favor of keep using gitpkgv "as is". I'm still volunteering, but without
>> help I can't do it.
>
> I'm struggling since to provide the help you need, I'd nearly have to
> solve the problem myself. I've helped several different people
> understand the fetcher in the past, only to have most of them move onto
> other things :(. Add in the 101 other distractions I have and its
> frustrating for everyone including me. Can you remind me where you were
> at (links to the right emails would help)?

This is basically as far as I got:

http://lists.openembedded.org/pipermail/openembedded-core/2014-October/098110.html





Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Bug: PR server changes the PKGV variable too

[Richard Purdie]

On Mon, 2015-01-05 at 11:36 +0100, Mike Looijmans wrote:
> On 01/05/2015 11:07 AM, Richard Purdie wrote:
> > On Mon, 2015-01-05 at 10:41 +0100, Mike Looijmans wrote:
> >> On 01/05/2015 10:27 AM, Richard Purdie wrote:
> >>> On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
> >>> Imagine you're not using gitpkgv. You set:
> >>>
> >>> PV = "x.y+${SRCPV}"
> >>>
> >>> Since SRCPV contains a revision hash, you can end up in a situation
> >>> where the version changes and you cannot upgrade the package since the
> >>> hash didn't 'increase'.
> >>>
> >>> The PR server therefore combines with the git fetcher to add an
> >>> incremental number at the start of the SRCPV string and yes, in that
> >>> scenario, it acts as a PV server. This is actually working as designed.
> >>
> >> Then the design is wrong. If a package chose to override PKGV manually, then
> >> the rest of the system should leave that value as is, and not touch it.
> >> Apparently the recipe author knows better, so please let him use that wisdom.
> >>
> >> Also, if you change the architecture of the package, the PR server will reset
> >> the version counter to 0 and break the upgrade path too. That was the problem
> >> that caused me to discover this problem, the PR server is making it hard to
> >> fix arch errors in recipes.
> >
> > You realise why it does that though? Imagine multiple MACHINE values
> > being built against a PR server. Each MACHINE will have different
> > package architecture values and different hashes. The 'PR' values should
> > therefore be seen as different. If the system didn't do this, it would
> > increment PR for every MACHINE change.
> >
> > Ok, so you say, lets make it work against MACHINE. That fails in the
> > case where multiple MACHINES share a package architecture :/. Its not a
> > simple problem to try and deal with :(
> >
> > I make no claim the system is perfect or free from bugs but these
> > behaviours you're referring to are like this for various reasons. I'm
> > open to changing them but we need to address the underlying problems
> > like changing MACHINE above.
> >
> > There are several other issues with changing package architecture such
> > as the sstate files in the sysroot. There is an open bug for this and
> > right now, I simply don't know how to solve it properly :(.
> 
> Yeah, I stand corrected, this is not something easily fixed.
> 
> I'd still like to have a way to override the PR server. Is there a way to make 
> the PR server ignore a package, or at least, have it NOT modify the PKGV 
> variable for a package?
> And if not, would you accept a patch to add that functionality to the prserver 
> system?

I've not tested it but would something like:

PRSERV_HOST_pn-XXX = ""

work?

There looks to be code to allow:

PRSERV_HOST_XXX = ""

to work too, looking at package.bbclass. Why, I have no idea :/. We
should probably remove that in favour of the standard override.

Since I've just been looking at this, I'll write down an overview of
what goes on:

PKGR contains ${PRAUTO}. The PKG{E,V,R} variables are only meant to be
accessed at packaging time. The code in package.bbclass,
package_get_auto_pr() runs at packaging time and sets PRAUTO with a
value from the PR server.

The only exception to this is if the magic string "AUTOINC" is in PV. If
it is, AUTOINC is replaced with the value from the PR server. You could
ask why it sets PKGV with the value of PV and I honestly don't remember,
that could be an oversight.

What this did mean is that we could jettison the autoincrement code from
the fetcher and just have one thing doing it.

The PKGV being overwritten by PV does look like a bug and is worth
looking into. Its not the PR server at fault though, its what
package.bbclass is doing with the data. I would like to look at why it
was added that way as I worry there was some underlying reason :/.

> >>> When you add in gitpkgv, something is obviously going wrong. gitpkgv is
> >>> in meta-oe since I've refused to add it to core on the several occasions
> >>> its been requested. I've said this before but I will say is again, it
> >>> *needs* become part of the standard fetcher API rather than a hacked in
> >>> afterthought which doesn't integrate well.
> >>
> >> I already volunteered and tried to do that, but got stuck in lack of
> >> understanding how the fetcher works, and did not get any help so abandoned it
> >> in favor of keep using gitpkgv "as is". I'm still volunteering, but without
> >> help I can't do it.
> >
> > I'm struggling since to provide the help you need, I'd nearly have to
> > solve the problem myself. I've helped several different people
> > understand the fetcher in the past, only to have most of them move onto
> > other things :(. Add in the 101 other distractions I have and its
> > frustrating for everyone including me. Can you remind me where you were
> > at (links to the right emails would help)?
> 
> This is basically as far as I got:
> 
> http://lists.openembedded.org/pipermail/openembedded-core/2014-October/098110.html

Thanks, I do remember looking at this. I also remember getting horribly
confused so I did start doing some things to try and help:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=593f14b2e3d1474d0c21d8d872dc7685163ffad2
(which was merged into master, trying to unravel some horrible function
chains)

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=89a20582e4fb3b222ccfeaf068fc5cfc7ae2dbf9
(this one should get merged, its just a comment tweak in bitbake).

I also noticed this in my WIP branch:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=00deb71e6e9a6b7f8aa5509288b59c6e4526c300
but that is perhaps best ignored, I think I had other ideas here.

So to explain what happens, you write:

PV = "x.y+{SRCPV}"

and

SRCPV = "${@bb.fetch2.get_srcrev(d)}"

and this triggers the code in fetch2/__init__.py. That function returns
a string which starts with AUTOINC+ and then a set of revisions.

The problem you've run into is that PV is expanded during parsing and is
used all over the place (for good reason) (e.g. in the value of WORKDIR)
and fetching may not have occurred yet. That means as you observe you
can't run git commands on a repo that may not yet exist (this function
only uses remote commands like ls-remote if needed).

So how to solve this? You need to add a function in parallel to
get_srcrev(), lets call it get_pretty_srcrev() which basically does the
same thing, but calls pretty_revision() instead of sortable_revision().
You can probably parameterise get_srcrev() internally. You'd then add a
pretty_revision to the git fetcher. At the metadata level you'd have
something like:

PRETTYSRCPV = "${@bb.fetch2.get_pretty_srcrev(d)}"

The recipe would then set PKGV = "x.y.+${PRETTYSRCPV}" to get the
gitpkgv behaviour.

All this is very approximate and I'm sure it won't be that simple but
hopefully it may let you move forward.

Cheers,

Richard

Re: Bug: PR server changes the PKGV variable too

[Mike Looijmans]

On 01/05/2015 12:37 PM, Richard Purdie wrote:
> On Mon, 2015-01-05 at 11:36 +0100, Mike Looijmans wrote:
>> On 01/05/2015 11:07 AM, Richard Purdie wrote:
>>> On Mon, 2015-01-05 at 10:41 +0100, Mike Looijmans wrote:
>>>> On 01/05/2015 10:27 AM, Richard Purdie wrote:
>>>>> On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
>>>>> Imagine you're not using gitpkgv. You set:
>>>>>
>>>>> PV = "x.y+${SRCPV}"
>>>>>
>>>>> Since SRCPV contains a revision hash, you can end up in a situation
>>>>> where the version changes and you cannot upgrade the package since the
>>>>> hash didn't 'increase'.
>>>>>
>>>>> The PR server therefore combines with the git fetcher to add an
>>>>> incremental number at the start of the SRCPV string and yes, in that
>>>>> scenario, it acts as a PV server. This is actually working as designed.
>>>>
>>>> Then the design is wrong. If a package chose to override PKGV manually, then
>>>> the rest of the system should leave that value as is, and not touch it.
>>>> Apparently the recipe author knows better, so please let him use that wisdom.
>>>>
>>>> Also, if you change the architecture of the package, the PR server will reset
>>>> the version counter to 0 and break the upgrade path too. That was the problem
>>>> that caused me to discover this problem, the PR server is making it hard to
>>>> fix arch errors in recipes.
>>>
>>> You realise why it does that though? Imagine multiple MACHINE values
>>> being built against a PR server. Each MACHINE will have different
>>> package architecture values and different hashes. The 'PR' values should
>>> therefore be seen as different. If the system didn't do this, it would
>>> increment PR for every MACHINE change.
>>>
>>> Ok, so you say, lets make it work against MACHINE. That fails in the
>>> case where multiple MACHINES share a package architecture :/. Its not a
>>> simple problem to try and deal with :(
>>>
>>> I make no claim the system is perfect or free from bugs but these
>>> behaviours you're referring to are like this for various reasons. I'm
>>> open to changing them but we need to address the underlying problems
>>> like changing MACHINE above.
>>>
>>> There are several other issues with changing package architecture such
>>> as the sstate files in the sysroot. There is an open bug for this and
>>> right now, I simply don't know how to solve it properly :(.
>>
>> Yeah, I stand corrected, this is not something easily fixed.
>>
>> I'd still like to have a way to override the PR server. Is there a way to make
>> the PR server ignore a package, or at least, have it NOT modify the PKGV
>> variable for a package?
>> And if not, would you accept a patch to add that functionality to the prserver
>> system?
>
> I've not tested it but would something like:
>
> PRSERV_HOST_pn-XXX = ""
>
> work?
>
> There looks to be code to allow:
>
> PRSERV_HOST_XXX = ""
>
> to work too, looking at package.bbclass. Why, I have no idea :/. We
> should probably remove that in favour of the standard override.

That means I could also set PRSERV_HOST inside the recipe, I guess. That might 
be easier to manage. Anyway, good suggestions, I'll try them. That might be 
useful in reducing the strain on network and flash now that 300+ packages get 
upgraded if someone happens to patch libc.

> Since I've just been looking at this, I'll write down an overview of
> what goes on:
>
> PKGR contains ${PRAUTO}. The PKG{E,V,R} variables are only meant to be
> accessed at packaging time. The code in package.bbclass,
> package_get_auto_pr() runs at packaging time and sets PRAUTO with a
> value from the PR server.
>
> The only exception to this is if the magic string "AUTOINC" is in PV. If
> it is, AUTOINC is replaced with the value from the PR server. You could
> ask why it sets PKGV with the value of PV and I honestly don't remember,
> that could be an oversight.
>
> What this did mean is that we could jettison the autoincrement code from
> the fetcher and just have one thing doing it.
>
> The PKGV being overwritten by PV does look like a bug and is worth
> looking into. Its not the PR server at fault though, its what
> package.bbclass is doing with the data. I would like to look at why it
> was added that way as I worry there was some underlying reason :/.

I see what you mean, in package.bbclass in  the "package_get_auto_pr" function:

             if "AUTOINC" in pv:
                 srcpv = bb.fetch2.get_srcrev(d)
                 base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
                 value = conn.getPR(base_ver, pkgarch, srcpv)
                 d.setVar("PKGV", pv.replace("AUTOINC", str(value)))

I think this is the line that does it. It's surprising that it looks at PV in 
this part of the method, while it looks at PKGV for the "PR Server not active" 
case.

And from looking at this code, the simple way to prevent this from happening 
is making sure that PV does NOT contain the AUTOINC string, which SRCPV will 
have by default when using git, and PV usually includes SRCPV.

That would mean that if I set SRCPV to something that doesn't contain AUTOINC, 
this part of the packager would not fire and leave the PKGV as is. Now that 
would fix the issue for my particular case.

It still makes one wonder whether that code snippet above was accidentally 
done this way, or that it was done for a particular reason... I worry right 
along with you there.


>>>>> When you add in gitpkgv, something is obviously going wrong. gitpkgv is
>>>>> in meta-oe since I've refused to add it to core on the several occasions
>>>>> its been requested. I've said this before but I will say is again, it
>>>>> *needs* become part of the standard fetcher API rather than a hacked in
>>>>> afterthought which doesn't integrate well.
>>>>
>>>> I already volunteered and tried to do that, but got stuck in lack of
>>>> understanding how the fetcher works, and did not get any help so abandoned it
>>>> in favor of keep using gitpkgv "as is". I'm still volunteering, but without
>>>> help I can't do it.
>>>
>>> I'm struggling since to provide the help you need, I'd nearly have to
>>> solve the problem myself. I've helped several different people
>>> understand the fetcher in the past, only to have most of them move onto
>>> other things :(. Add in the 101 other distractions I have and its
>>> frustrating for everyone including me. Can you remind me where you were
>>> at (links to the right emails would help)?
>>
>> This is basically as far as I got:
>>
>> http://lists.openembedded.org/pipermail/openembedded-core/2014-October/098110.html
>
> Thanks, I do remember looking at this. I also remember getting horribly
> confused so I did start doing some things to try and help:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=593f14b2e3d1474d0c21d8d872dc7685163ffad2
> (which was merged into master, trying to unravel some horrible function
> chains)
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=89a20582e4fb3b222ccfeaf068fc5cfc7ae2dbf9
> (this one should get merged, its just a comment tweak in bitbake).
>
> I also noticed this in my WIP branch:
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=00deb71e6e9a6b7f8aa5509288b59c6e4526c300
> but that is perhaps best ignored, I think I had other ideas here.
>
> So to explain what happens, you write:
>
> PV = "x.y+{SRCPV}"
>
> and
>
> SRCPV = "${@bb.fetch2.get_srcrev(d)}"
>
> and this triggers the code in fetch2/__init__.py. That function returns
> a string which starts with AUTOINC+ and then a set of revisions.
>
> The problem you've run into is that PV is expanded during parsing and is
> used all over the place (for good reason) (e.g. in the value of WORKDIR)
> and fetching may not have occurred yet. That means as you observe you
> can't run git commands on a repo that may not yet exist (this function
> only uses remote commands like ls-remote if needed).
>
> So how to solve this? You need to add a function in parallel to
> get_srcrev(), lets call it get_pretty_srcrev() which basically does the
> same thing, but calls pretty_revision() instead of sortable_revision().
> You can probably parameterise get_srcrev() internally. You'd then add a
> pretty_revision to the git fetcher. At the metadata level you'd have
> something like:
>
> PRETTYSRCPV = "${@bb.fetch2.get_pretty_srcrev(d)}"
>
> The recipe would then set PKGV = "x.y.+${PRETTYSRCPV}" to get the
> gitpkgv behaviour.
>
> All this is very approximate and I'm sure it won't be that simple but
> hopefully it may let you move forward.

I'll give it another try one of these days with the information you added, and 
return to it.





Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Bug: PR server changes the PKGV variable too

[Mike Looijmans]

On 01/05/2015 01:09 PM, Mike Looijmans wrote:
> On 01/05/2015 12:37 PM, Richard Purdie wrote:
>> On Mon, 2015-01-05 at 11:36 +0100, Mike Looijmans wrote:
>>> On 01/05/2015 11:07 AM, Richard Purdie wrote:
>>>> On Mon, 2015-01-05 at 10:41 +0100, Mike Looijmans wrote:
>>>>> On 01/05/2015 10:27 AM, Richard Purdie wrote:
>>>>>> On Sun, 2015-01-04 at 16:20 +0100, Mike Looijmans wrote:
>>>>>> Imagine you're not using gitpkgv. You set:
>>>>>>
>>>>>> PV = "x.y+${SRCPV}"
>>>>>>
>>>>>> Since SRCPV contains a revision hash, you can end up in a situation
>>>>>> where the version changes and you cannot upgrade the package since
>>>>>> the
>>>>>> hash didn't 'increase'.
>>>>>>
>>>>>> The PR server therefore combines with the git fetcher to add an
>>>>>> incremental number at the start of the SRCPV string and yes, in that
>>>>>> scenario, it acts as a PV server. This is actually working as
>>>>>> designed.
>>>>>
>>>>> Then the design is wrong. If a package chose to override PKGV
>>>>> manually, then
>>>>> the rest of the system should leave that value as is, and not touch
>>>>> it.
>>>>> Apparently the recipe author knows better, so please let him use
>>>>> that wisdom.
>>>>>
>>>>> Also, if you change the architecture of the package, the PR server
>>>>> will reset
>>>>> the version counter to 0 and break the upgrade path too. That was
>>>>> the problem
>>>>> that caused me to discover this problem, the PR server is making it
>>>>> hard to
>>>>> fix arch errors in recipes.
>>>>
>>>> You realise why it does that though? Imagine multiple MACHINE values
>>>> being built against a PR server. Each MACHINE will have different
>>>> package architecture values and different hashes. The 'PR' values
>>>> should
>>>> therefore be seen as different. If the system didn't do this, it would
>>>> increment PR for every MACHINE change.
>>>>
>>>> Ok, so you say, lets make it work against MACHINE. That fails in the
>>>> case where multiple MACHINES share a package architecture :/. Its not a
>>>> simple problem to try and deal with :(
>>>>
>>>> I make no claim the system is perfect or free from bugs but these
>>>> behaviours you're referring to are like this for various reasons. I'm
>>>> open to changing them but we need to address the underlying problems
>>>> like changing MACHINE above.
>>>>
>>>> There are several other issues with changing package architecture such
>>>> as the sstate files in the sysroot. There is an open bug for this and
>>>> right now, I simply don't know how to solve it properly :(.
>>>
>>> Yeah, I stand corrected, this is not something easily fixed.
>>>
>>> I'd still like to have a way to override the PR server. Is there a
>>> way to make
>>> the PR server ignore a package, or at least, have it NOT modify the PKGV
>>> variable for a package?
>>> And if not, would you accept a patch to add that functionality to the
>>> prserver
>>> system?
>>
>> I've not tested it but would something like:
>>
>> PRSERV_HOST_pn-XXX = ""
>>
>> work?
>>
>> There looks to be code to allow:
>>
>> PRSERV_HOST_XXX = ""
>>
>> to work too, looking at package.bbclass. Why, I have no idea :/. We
>> should probably remove that in favour of the standard override.
>
> That means I could also set PRSERV_HOST inside the recipe, I guess. That
> might be easier to manage. Anyway, good suggestions, I'll try them. That
> might be useful in reducing the strain on network and flash now that
> 300+ packages get upgraded if someone happens to patch libc.
>
>> Since I've just been looking at this, I'll write down an overview of
>> what goes on:
>>
>> PKGR contains ${PRAUTO}. The PKG{E,V,R} variables are only meant to be
>> accessed at packaging time. The code in package.bbclass,
>> package_get_auto_pr() runs at packaging time and sets PRAUTO with a
>> value from the PR server.
>>
>> The only exception to this is if the magic string "AUTOINC" is in PV. If
>> it is, AUTOINC is replaced with the value from the PR server. You could
>> ask why it sets PKGV with the value of PV and I honestly don't remember,
>> that could be an oversight.
>>
>> What this did mean is that we could jettison the autoincrement code from
>> the fetcher and just have one thing doing it.
>>
>> The PKGV being overwritten by PV does look like a bug and is worth
>> looking into. Its not the PR server at fault though, its what
>> package.bbclass is doing with the data. I would like to look at why it
>> was added that way as I worry there was some underlying reason :/.
>
> I see what you mean, in package.bbclass in  the "package_get_auto_pr"
> function:
>
>              if "AUTOINC" in pv:
>                  srcpv = bb.fetch2.get_srcrev(d)
>                  base_ver = "AUTOINC-%s" % version[:version.find(srcpv)]
>                  value = conn.getPR(base_ver, pkgarch, srcpv)
>                  d.setVar("PKGV", pv.replace("AUTOINC", str(value)))
>
> I think this is the line that does it. It's surprising that it looks at
> PV in this part of the method, while it looks at PKGV for the "PR Server
> not active" case.
>
> And from looking at this code, the simple way to prevent this from
> happening is making sure that PV does NOT contain the AUTOINC string,
> which SRCPV will have by default when using git, and PV usually includes
> SRCPV.
>
> That would mean that if I set SRCPV to something that doesn't contain
> AUTOINC, this part of the packager would not fire and leave the PKGV as
> is. Now that would fix the issue for my particular case.

Indeed, if I set:
   SRCPV = "${@bb.fetch2.get_srcrev(d).replace('AUTOINC','X')}"
then the resulting PKGV is as i'd have expected because PV no longer 
triggers the substitution.

-- 
Mike Looijmans