PRServer's problem

PRServer's problem

[Robert Yang]

The PRServer bumps PR according to do_package's task hash, that
causes it bumps *all* packages' PR when recipes like pseudo-native
and rpm-native is changed. It is a very bad user experience when we
run "smart/opkg upgrade" on running target, for example, when we apply
a CVE patch to pseudo-native or rpm-native, or do some slight changes
in their do_compile, "smart/opkg upgrade" will download/install *all*
the packages since all of the packages' PR are bumped.

Here are some rough suggestions to fix this problem, and please feel
free to give your suggestions.
1) Do not use do_package's task for bumping PR, the easiest way
    is simulate manually bump PR -- only bump PR when the recipe
    itself's checksum is changed.

2) Add a new task for PRServer, redefine its task hash for bumping
    PR, for example, this task hash only considers RDEPENDS (no
    DEPENDS), and drop any native dependencies.

I prefer the first way, and an alternative way maybe add a var so that
the user can configure it:
PR_CHECKSUM = "${BB_TASKHASH}" (current way)
Or
PR_CHECKSUM = "<recipe checksum>"

-- 
Thanks

Robert

Re: PRServer's problem

[Joshua G Lock]

Hi Robert,

On Wed, 2016-05-18 at 14:09 +0800, Robert Yang wrote:
> The PRServer bumps PR according to do_package's task hash, that
> causes it bumps *all* packages' PR when recipes like pseudo-native
> and rpm-native is changed. It is a very bad user experience when we
> run "smart/opkg upgrade" on running target, for example, when we
> apply
> a CVE patch to pseudo-native or rpm-native, or do some slight changes
> in their do_compile, "smart/opkg upgrade" will download/install *all*
> the packages since all of the packages' PR are bumped.

Paul began working to address the package feed churn issue some time
back using the tool build-compare from the openSUSE project.

You can find links to various trees and a summary of the progress so
far in this bugzilla entry:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8318

It still needs some more work; most notably I'd like to see some tests
that validates the functionality.

Regards,

Joshua

Re: PRServer's problem

[Robert Yang]

Hi Joshua,

Thanks, I read packagefeed-stability.bbclass just now. If
we can change PRserver's checksum to consider RDEPENDS only
(or make it optional) I will run a testing on that script.

// Robert

On 05/18/2016 03:34 PM, Joshua G Lock wrote:
> Hi Robert,
>
> On Wed, 2016-05-18 at 14:09 +0800, Robert Yang wrote:
>> The PRServer bumps PR according to do_package's task hash, that
>> causes it bumps *all* packages' PR when recipes like pseudo-native
>> and rpm-native is changed. It is a very bad user experience when we
>> run "smart/opkg upgrade" on running target, for example, when we
>> apply
>> a CVE patch to pseudo-native or rpm-native, or do some slight changes
>> in their do_compile, "smart/opkg upgrade" will download/install *all*
>> the packages since all of the packages' PR are bumped.
>
> Paul began working to address the package feed churn issue some time
> back using the tool build-compare from the openSUSE project.
>
> You can find links to various trees and a summary of the progress so
> far in this bugzilla entry:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8318
>
> It still needs some more work; most notably I'd like to see some tests
> that validates the functionality.
>
> Regards,
>
> Joshua
>

Re: PRServer's problem

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1798 bytes --]

See:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970

Just using recipe checksum wont work, because the main reason for PR bumps
is to automatically upgrade the packages when one of the dependencies
changes .so version, which you won't detect from recipe checksum of the app
which is just using the library.

On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com>
wrote:

> The PRServer bumps PR according to do_package's task hash, that
> causes it bumps *all* packages' PR when recipes like pseudo-native
> and rpm-native is changed. It is a very bad user experience when we
> run "smart/opkg upgrade" on running target, for example, when we apply
> a CVE patch to pseudo-native or rpm-native, or do some slight changes
> in their do_compile, "smart/opkg upgrade" will download/install *all*
> the packages since all of the packages' PR are bumped.
>
> Here are some rough suggestions to fix this problem, and please feel
> free to give your suggestions.
> 1) Do not use do_package's task for bumping PR, the easiest way
>    is simulate manually bump PR -- only bump PR when the recipe
>    itself's checksum is changed.
>
> 2) Add a new task for PRServer, redefine its task hash for bumping
>    PR, for example, this task hash only considers RDEPENDS (no
>    DEPENDS), and drop any native dependencies.
>
> I prefer the first way, and an alternative way maybe add a var so that
> the user can configure it:
> PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> Or
> PR_CHECKSUM = "<recipe checksum>"
>
> --
> Thanks
>
> Robert
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2589 bytes --]

Re: PRServer's problem

[Robert Yang]

Hi Martin,

On 05/18/2016 03:39 PM, Martin Jansa wrote:
> See:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>
> Just using recipe checksum wont work, because the main reason for PR bumps is to
> automatically upgrade the packages when one of the dependencies changes .so
> version, which you won't detect from recipe checksum of the app which is just
> using the library.

For the development branch like master, yes, that would happen. But for
the stable release like jethro and krogoth, it is unlikely that would
happen, and if if does, the user can manually bump the impacted recipe's
PR to fix the problem. The current problem is that when *all* recipes'
PR are bumped, there is no way to fix the problem.

// Robert

>
> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
>     The PRServer bumps PR according to do_package's task hash, that
>     causes it bumps *all* packages' PR when recipes like pseudo-native
>     and rpm-native is changed. It is a very bad user experience when we
>     run "smart/opkg upgrade" on running target, for example, when we apply
>     a CVE patch to pseudo-native or rpm-native, or do some slight changes
>     in their do_compile, "smart/opkg upgrade" will download/install *all*
>     the packages since all of the packages' PR are bumped.
>
>     Here are some rough suggestions to fix this problem, and please feel
>     free to give your suggestions.
>     1) Do not use do_package's task for bumping PR, the easiest way
>         is simulate manually bump PR -- only bump PR when the recipe
>         itself's checksum is changed.
>
>     2) Add a new task for PRServer, redefine its task hash for bumping
>         PR, for example, this task hash only considers RDEPENDS (no
>         DEPENDS), and drop any native dependencies.
>
>     I prefer the first way, and an alternative way maybe add a var so that
>     the user can configure it:
>     PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>     Or
>     PR_CHECKSUM = "<recipe checksum>"
>
>     --
>     Thanks
>
>     Robert
>     --
>     _______________________________________________
>     Openembedded-core mailing list
>     Openembedded-core@lists.openembedded.org
>     <mailto:Openembedded-core@lists.openembedded.org>
>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

Re: PRServer's problem

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2822 bytes --]

On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
> Hi Martin,
> 
> On 05/18/2016 03:39 PM, Martin Jansa wrote:
> > See:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
> >
> > Just using recipe checksum wont work, because the main reason for PR bumps is to
> > automatically upgrade the packages when one of the dependencies changes .so
> > version, which you won't detect from recipe checksum of the app which is just
> > using the library.
> 
> For the development branch like master, yes, that would happen. But for
> the stable release like jethro and krogoth, it is unlikely that would
> happen, and if if does, the user can manually bump the impacted recipe's
> PR to fix the problem. The current problem is that when *all* recipes'
> PR are bumped, there is no way to fix the problem.

You can still stop using PR service and start doing manual PR bumps, but
it's quite annoying if you need to bump a lot of recipes you don't
control :).

> > On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
> > <mailto:liezhi.yang@windriver.com>> wrote:
> >
> >     The PRServer bumps PR according to do_package's task hash, that
> >     causes it bumps *all* packages' PR when recipes like pseudo-native
> >     and rpm-native is changed. It is a very bad user experience when we
> >     run "smart/opkg upgrade" on running target, for example, when we apply
> >     a CVE patch to pseudo-native or rpm-native, or do some slight changes
> >     in their do_compile, "smart/opkg upgrade" will download/install *all*
> >     the packages since all of the packages' PR are bumped.
> >
> >     Here are some rough suggestions to fix this problem, and please feel
> >     free to give your suggestions.
> >     1) Do not use do_package's task for bumping PR, the easiest way
> >         is simulate manually bump PR -- only bump PR when the recipe
> >         itself's checksum is changed.
> >
> >     2) Add a new task for PRServer, redefine its task hash for bumping
> >         PR, for example, this task hash only considers RDEPENDS (no
> >         DEPENDS), and drop any native dependencies.
> >
> >     I prefer the first way, and an alternative way maybe add a var so that
> >     the user can configure it:
> >     PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> >     Or
> >     PR_CHECKSUM = "<recipe checksum>"
> >
> >     --
> >     Thanks
> >
> >     Robert
> >     --
> >     _______________________________________________
> >     Openembedded-core mailing list
> >     Openembedded-core@lists.openembedded.org
> >     <mailto:Openembedded-core@lists.openembedded.org>
> >     http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >
> >

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: PRServer's problem

[Robert Yang]

On 05/18/2016 05:20 PM, Martin Jansa wrote:
> On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>> Hi Martin,
>>
>> On 05/18/2016 03:39 PM, Martin Jansa wrote:
>>> See:
>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>>>
>>> Just using recipe checksum wont work, because the main reason for PR bumps is to
>>> automatically upgrade the packages when one of the dependencies changes .so
>>> version, which you won't detect from recipe checksum of the app which is just
>>> using the library.
>>
>> For the development branch like master, yes, that would happen. But for
>> the stable release like jethro and krogoth, it is unlikely that would
>> happen, and if if does, the user can manually bump the impacted recipe's
>> PR to fix the problem. The current problem is that when *all* recipes'
>> PR are bumped, there is no way to fix the problem.
>
> You can still stop using PR service and start doing manual PR bumps, but

We can't stop PR service and start doing manual PR bumps since we need keep
update to date with upstream, the changes from upstream don't do the manual
PR, and I don't think that we have to if they can be done automatically.

> it's quite annoying if you need to bump a lot of recipes you don't
> control :).

What's your opinion about only consider RDEPENDS for PR service's checksum,
please ?

// Robert

>
>>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>
>>>      The PRServer bumps PR according to do_package's task hash, that
>>>      causes it bumps *all* packages' PR when recipes like pseudo-native
>>>      and rpm-native is changed. It is a very bad user experience when we
>>>      run "smart/opkg upgrade" on running target, for example, when we apply
>>>      a CVE patch to pseudo-native or rpm-native, or do some slight changes
>>>      in their do_compile, "smart/opkg upgrade" will download/install *all*
>>>      the packages since all of the packages' PR are bumped.
>>>
>>>      Here are some rough suggestions to fix this problem, and please feel
>>>      free to give your suggestions.
>>>      1) Do not use do_package's task for bumping PR, the easiest way
>>>          is simulate manually bump PR -- only bump PR when the recipe
>>>          itself's checksum is changed.
>>>
>>>      2) Add a new task for PRServer, redefine its task hash for bumping
>>>          PR, for example, this task hash only considers RDEPENDS (no
>>>          DEPENDS), and drop any native dependencies.
>>>
>>>      I prefer the first way, and an alternative way maybe add a var so that
>>>      the user can configure it:
>>>      PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>>>      Or
>>>      PR_CHECKSUM = "<recipe checksum>"
>>>
>>>      --
>>>      Thanks
>>>
>>>      Robert
>>>      --
>>>      _______________________________________________
>>>      Openembedded-core mailing list
>>>      Openembedded-core@lists.openembedded.org
>>>      <mailto:Openembedded-core@lists.openembedded.org>
>>>      http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>>
>

Re: PRServer's problem

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3611 bytes --]

On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
> 
> 
> On 05/18/2016 05:20 PM, Martin Jansa wrote:
> > On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
> >> Hi Martin,
> >>
> >> On 05/18/2016 03:39 PM, Martin Jansa wrote:
> >>> See:
> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
> >>>
> >>> Just using recipe checksum wont work, because the main reason for PR bumps is to
> >>> automatically upgrade the packages when one of the dependencies changes .so
> >>> version, which you won't detect from recipe checksum of the app which is just
> >>> using the library.
> >>
> >> For the development branch like master, yes, that would happen. But for
> >> the stable release like jethro and krogoth, it is unlikely that would
> >> happen, and if if does, the user can manually bump the impacted recipe's
> >> PR to fix the problem. The current problem is that when *all* recipes'
> >> PR are bumped, there is no way to fix the problem.
> >
> > You can still stop using PR service and start doing manual PR bumps, but
> 
> We can't stop PR service and start doing manual PR bumps since we need keep
> update to date with upstream, the changes from upstream don't do the manual
> PR, and I don't think that we have to if they can be done automatically.
> 
> > it's quite annoying if you need to bump a lot of recipes you don't
> > control :).
> 
> What's your opinion about only consider RDEPENDS for PR service's checksum,
> please ?

I would like to have separate handler as described in that bug, option
c).

Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.

> >>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
> >>> <mailto:liezhi.yang@windriver.com>> wrote:
> >>>
> >>>      The PRServer bumps PR according to do_package's task hash, that
> >>>      causes it bumps *all* packages' PR when recipes like pseudo-native
> >>>      and rpm-native is changed. It is a very bad user experience when we
> >>>      run "smart/opkg upgrade" on running target, for example, when we apply
> >>>      a CVE patch to pseudo-native or rpm-native, or do some slight changes
> >>>      in their do_compile, "smart/opkg upgrade" will download/install *all*
> >>>      the packages since all of the packages' PR are bumped.
> >>>
> >>>      Here are some rough suggestions to fix this problem, and please feel
> >>>      free to give your suggestions.
> >>>      1) Do not use do_package's task for bumping PR, the easiest way
> >>>          is simulate manually bump PR -- only bump PR when the recipe
> >>>          itself's checksum is changed.
> >>>
> >>>      2) Add a new task for PRServer, redefine its task hash for bumping
> >>>          PR, for example, this task hash only considers RDEPENDS (no
> >>>          DEPENDS), and drop any native dependencies.
> >>>
> >>>      I prefer the first way, and an alternative way maybe add a var so that
> >>>      the user can configure it:
> >>>      PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> >>>      Or
> >>>      PR_CHECKSUM = "<recipe checksum>"
> >>>
> >>>      --
> >>>      Thanks
> >>>
> >>>      Robert
> >>>      --
> >>>      _______________________________________________
> >>>      Openembedded-core mailing list
> >>>      Openembedded-core@lists.openembedded.org
> >>>      <mailto:Openembedded-core@lists.openembedded.org>
> >>>      http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >>>
> >>>
> >

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: PRServer's problem

[Robert Yang]

Hi Martin,

I found this patch in the bug:

http://git.openembedded.org/openembedded-core/commit/meta/lib/oe/sstatesig.py?id=336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6

Too many PR bumps and rebuildings are caused by this patch. I'm
not very sure about what this patch tries to fix, it seems that
it is trying to fix problems when 32bit and 64bit uses the same
sstate cache? Would you please provide a simple reproducer, please?

Things will become much better after revert this patch. Be less
stricter or more is a hard problem, if we still need the patch,
can we leave such a choice to the end user? We can add some vars
like DROP_NATIVE_SIG ? = "0/1". This would be good to the end user
who uses stable release like jethro or krogoth to make their
distributions, and PR Service really matters here. Even if they
switch the build between 32 and 64 builds (This is unlikely to
happen for production environment) and got problems, they still
can fix the problem by rebuilding, this is still much better than
current status: run a "smart/opkg/apt-get upgrade" on the target,
*all* of the packages are downloaded and installed again after a
CVE patch applies on native recipes like pseudo-native or rpm-native,
but in fact, nothing is changed on the target this is really a bad
experience.

// Robert

On 05/18/2016 06:15 PM, Martin Jansa wrote:
> On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
>>
>>
>> On 05/18/2016 05:20 PM, Martin Jansa wrote:
>>> On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>>>> Hi Martin,
>>>>
>>>> On 05/18/2016 03:39 PM, Martin Jansa wrote:
>>>>> See:
>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>>>>>
>>>>> Just using recipe checksum wont work, because the main reason for PR bumps is to
>>>>> automatically upgrade the packages when one of the dependencies changes .so
>>>>> version, which you won't detect from recipe checksum of the app which is just
>>>>> using the library.
>>>>
>>>> For the development branch like master, yes, that would happen. But for
>>>> the stable release like jethro and krogoth, it is unlikely that would
>>>> happen, and if if does, the user can manually bump the impacted recipe's
>>>> PR to fix the problem. The current problem is that when *all* recipes'
>>>> PR are bumped, there is no way to fix the problem.
>>>
>>> You can still stop using PR service and start doing manual PR bumps, but
>>
>> We can't stop PR service and start doing manual PR bumps since we need keep
>> update to date with upstream, the changes from upstream don't do the manual
>> PR, and I don't think that we have to if they can be done automatically.
>>
>>> it's quite annoying if you need to bump a lot of recipes you don't
>>> control :).
>>
>> What's your opinion about only consider RDEPENDS for PR service's checksum,
>> please ?
>
> I would like to have separate handler as described in that bug, option
> c).
>
> Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary rebuilds.
>
>>>>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
>>>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>>>
>>>>>       The PRServer bumps PR according to do_package's task hash, that
>>>>>       causes it bumps *all* packages' PR when recipes like pseudo-native
>>>>>       and rpm-native is changed. It is a very bad user experience when we
>>>>>       run "smart/opkg upgrade" on running target, for example, when we apply
>>>>>       a CVE patch to pseudo-native or rpm-native, or do some slight changes
>>>>>       in their do_compile, "smart/opkg upgrade" will download/install *all*
>>>>>       the packages since all of the packages' PR are bumped.
>>>>>
>>>>>       Here are some rough suggestions to fix this problem, and please feel
>>>>>       free to give your suggestions.
>>>>>       1) Do not use do_package's task for bumping PR, the easiest way
>>>>>           is simulate manually bump PR -- only bump PR when the recipe
>>>>>           itself's checksum is changed.
>>>>>
>>>>>       2) Add a new task for PRServer, redefine its task hash for bumping
>>>>>           PR, for example, this task hash only considers RDEPENDS (no
>>>>>           DEPENDS), and drop any native dependencies.
>>>>>
>>>>>       I prefer the first way, and an alternative way maybe add a var so that
>>>>>       the user can configure it:
>>>>>       PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>>>>>       Or
>>>>>       PR_CHECKSUM = "<recipe checksum>"
>>>>>
>>>>>       --
>>>>>       Thanks
>>>>>
>>>>>       Robert
>>>>>       --
>>>>>       _______________________________________________
>>>>>       Openembedded-core mailing list
>>>>>       Openembedded-core@lists.openembedded.org
>>>>>       <mailto:Openembedded-core@lists.openembedded.org>
>>>>>       http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>
>>>>>
>>>
>

Re: PRServer's problem

[Robert Yang]

On 05/19/2016 10:33 AM, Robert Yang wrote:
>
> Hi Martin,
>
> I found this patch in the bug:
>
> http://git.openembedded.org/openembedded-core/commit/meta/lib/oe/sstatesig.py?id=336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6

And this patch causes another inconsistent:

PACKAGE_CLASSES = "package_ipk"

1) After make some changes in opkg-utils-native, foo.do_package_write_ipk
will re-run, but PR doesn't bump, this is because PR is bumped in
do_package, and do_package doesn't depend on opkg-utils-native, but
do_package_write_ipk does, so do_package_write_ipk will rerun but no
PR is bumped.

2) Another more funny problem is, after make some changes in rpm-native
(PACKAGE_CLASSES = "package_ipk"), both foo.do_package and
foo.do_package_write_ipk will run and PR is bumped. This is because
do_package depends on rpm-native (for FILERDEPENDS), so do_package
re-runs and PR bumps, thus caused do_package_write_ipk re-runs.

Revert the patch will fix both 1) and 2).

For 2), the only one uses FILERDEPENDS is do_package_qa, maybe we need
move the generation of FILERDEPENDS there. This is another bug.

// Robert

>
>
> Too many PR bumps and rebuildings are caused by this patch. I'm
> not very sure about what this patch tries to fix, it seems that
> it is trying to fix problems when 32bit and 64bit uses the same
> sstate cache? Would you please provide a simple reproducer, please?
>
> Things will become much better after revert this patch. Be less
> stricter or more is a hard problem, if we still need the patch,
> can we leave such a choice to the end user? We can add some vars
> like DROP_NATIVE_SIG ? = "0/1". This would be good to the end user
> who uses stable release like jethro or krogoth to make their
> distributions, and PR Service really matters here. Even if they
> switch the build between 32 and 64 builds (This is unlikely to
> happen for production environment) and got problems, they still
> can fix the problem by rebuilding, this is still much better than
> current status: run a "smart/opkg/apt-get upgrade" on the target,
> *all* of the packages are downloaded and installed again after a
> CVE patch applies on native recipes like pseudo-native or rpm-native,
> but in fact, nothing is changed on the target this is really a bad
> experience.
>
> // Robert
>
> On 05/18/2016 06:15 PM, Martin Jansa wrote:
>> On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
>>>
>>>
>>> On 05/18/2016 05:20 PM, Martin Jansa wrote:
>>>> On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>>>>> Hi Martin,
>>>>>
>>>>> On 05/18/2016 03:39 PM, Martin Jansa wrote:
>>>>>> See:
>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>>>>>>
>>>>>> Just using recipe checksum wont work, because the main reason for PR bumps
>>>>>> is to
>>>>>> automatically upgrade the packages when one of the dependencies changes .so
>>>>>> version, which you won't detect from recipe checksum of the app which is just
>>>>>> using the library.
>>>>>
>>>>> For the development branch like master, yes, that would happen. But for
>>>>> the stable release like jethro and krogoth, it is unlikely that would
>>>>> happen, and if if does, the user can manually bump the impacted recipe's
>>>>> PR to fix the problem. The current problem is that when *all* recipes'
>>>>> PR are bumped, there is no way to fix the problem.
>>>>
>>>> You can still stop using PR service and start doing manual PR bumps, but
>>>
>>> We can't stop PR service and start doing manual PR bumps since we need keep
>>> update to date with upstream, the changes from upstream don't do the manual
>>> PR, and I don't think that we have to if they can be done automatically.
>>>
>>>> it's quite annoying if you need to bump a lot of recipes you don't
>>>> control :).
>>>
>>> What's your opinion about only consider RDEPENDS for PR service's checksum,
>>> please ?
>>
>> I would like to have separate handler as described in that bug, option
>> c).
>>
>> Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary
>> rebuilds.
>>
>>>>>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
>>>>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>>>>
>>>>>>       The PRServer bumps PR according to do_package's task hash, that
>>>>>>       causes it bumps *all* packages' PR when recipes like pseudo-native
>>>>>>       and rpm-native is changed. It is a very bad user experience when we
>>>>>>       run "smart/opkg upgrade" on running target, for example, when we apply
>>>>>>       a CVE patch to pseudo-native or rpm-native, or do some slight changes
>>>>>>       in their do_compile, "smart/opkg upgrade" will download/install *all*
>>>>>>       the packages since all of the packages' PR are bumped.
>>>>>>
>>>>>>       Here are some rough suggestions to fix this problem, and please feel
>>>>>>       free to give your suggestions.
>>>>>>       1) Do not use do_package's task for bumping PR, the easiest way
>>>>>>           is simulate manually bump PR -- only bump PR when the recipe
>>>>>>           itself's checksum is changed.
>>>>>>
>>>>>>       2) Add a new task for PRServer, redefine its task hash for bumping
>>>>>>           PR, for example, this task hash only considers RDEPENDS (no
>>>>>>           DEPENDS), and drop any native dependencies.
>>>>>>
>>>>>>       I prefer the first way, and an alternative way maybe add a var so that
>>>>>>       the user can configure it:
>>>>>>       PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>>>>>>       Or
>>>>>>       PR_CHECKSUM = "<recipe checksum>"
>>>>>>
>>>>>>       --
>>>>>>       Thanks
>>>>>>
>>>>>>       Robert
>>>>>>       --
>>>>>>       _______________________________________________
>>>>>>       Openembedded-core mailing list
>>>>>>       Openembedded-core@lists.openembedded.org
>>>>>>       <mailto:Openembedded-core@lists.openembedded.org>
>>>>>>       http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>>
>>>>>>
>>>>
>>

Re: PRServer's problem

[Richard Purdie]

The bottom line is that the system is setup to be sensitive to changes.
Where we've had cases where we haven't reacted to changes, people have
complained and we've ended up making sure we do react to them. The
patch you reference was one such case where users complained we didn't
react enough.

You can't have things both ways, where it reacts to all changes but
also never reacts to changes which don't have any visible effect on the
end result (how do you know?).

The binary diff tool is likely going to be the only way to ultimately
control a binary package feed and is still the only way I can see of
solving this problem. I'd love some help on making that work. Making
deterministic binaries is a large part in making that tool effectively
ultimate and we had some great work on that in the last release.

To be really clear, OE-Core will not have a different signature policy
on release branches since that differing policy would break user
expectations and also wouldn't get tested apart from on the branch so
we'd have less confidence it was working.

Users are free to set their own policies, the system was designed to do
that. If WindRiver wants to have a much more permissive policy, I'm
more than happy for them to do so.

Cheers,

Richard

Re: PRServer's problem

[Robert Yang]

On 05/19/2016 05:45 PM, Richard Purdie wrote:
>
> The bottom line is that the system is setup to be sensitive to changes.
> Where we've had cases where we haven't reacted to changes, people have
> complained and we've ended up making sure we do react to them. The
> patch you reference was one such case where users complained we didn't
> react enough.
>
> You can't have things both ways, where it reacts to all changes but
> also never reacts to changes which don't have any visible effect on the
> end result (how do you know?).
>
> The binary diff tool is likely going to be the only way to ultimately
> control a binary package feed and is still the only way I can see of
> solving this problem. I'd love some help on making that work. Making
> deterministic binaries is a large part in making that tool effectively
> ultimate and we had some great work on that in the last release.
>
> To be really clear, OE-Core will not have a different signature policy
> on release branches since that differing policy would break user
> expectations and also wouldn't get tested apart from on the branch so
> we'd have less confidence it was working.

Yes, I agree with this, I just used stable release as an example (big
changes won't happen on a stable release).

>
> Users are free to set their own policies, the system was designed to do
> that. If WindRiver wants to have a much more permissive policy, I'm
> more than happy for them to do so.

Thanks, frankly speaking, not only WindRiver wants this. After cloud
computing and virtualization gets hot, more and more users want to
customize their own images (for saving disk space, memory and security
reason), oe/yocto is very good at customizing images, so more and
more people try to use it to build their own distros, where live
upgrades becomes very important.

// Robert

>
> Cheers,
>
> Richard
>

Re: PRServer's problem

[Richard Purdie]

On Thu, 2016-05-19 at 18:12 +0800, Robert Yang wrote:
> On 05/19/2016 05:45 PM, Richard Purdie wrote:
> > To be really clear, OE-Core will not have a different signature
> > policy
> > on release branches since that differing policy would break user
> > expectations and also wouldn't get tested apart from on the branch
> > so
> > we'd have less confidence it was working.
> 
> Yes, I agree with this, I just used stable release as an example (big
> changes won't happen on a stable release).
> 
> > 
> > Users are free to set their own policies, the system was designed
> > to do
> > that. If WindRiver wants to have a much more permissive policy, I'm
> > more than happy for them to do so.
> 
> Thanks, frankly speaking, not only WindRiver wants this. After cloud
> computing and virtualization gets hot, more and more users want to
> customize their own images (for saving disk space, memory and
> security
> reason), oe/yocto is very good at customizing images, so more and
> more people try to use it to build their own distros, where live
> upgrades becomes very important.

I understand that. So are we going to get binary diff working? Its the
only viable solution we have to the problem at the moment that I know
of. Randomly hacking bits of the signature generation isn't going to
help this unfortunately.

Cheers,

Richard

Re: PRServer's problem

[Robert Yang]

On 05/19/2016 06:17 PM, Richard Purdie wrote:
> On Thu, 2016-05-19 at 18:12 +0800, Robert Yang wrote:
>> On 05/19/2016 05:45 PM, Richard Purdie wrote:
>>> To be really clear, OE-Core will not have a different signature
>>> policy
>>> on release branches since that differing policy would break user
>>> expectations and also wouldn't get tested apart from on the branch
>>> so
>>> we'd have less confidence it was working.
>>
>> Yes, I agree with this, I just used stable release as an example (big
>> changes won't happen on a stable release).
>>
>>>
>>> Users are free to set their own policies, the system was designed
>>> to do
>>> that. If WindRiver wants to have a much more permissive policy, I'm
>>> more than happy for them to do so.
>>
>> Thanks, frankly speaking, not only WindRiver wants this. After cloud
>> computing and virtualization gets hot, more and more users want to
>> customize their own images (for saving disk space, memory and
>> security
>> reason), oe/yocto is very good at customizing images, so more and
>> more people try to use it to build their own distros, where live
>> upgrades becomes very important.
>
> I understand that. So are we going to get binary diff working? Its the

Thanks everyone, it seems that this is the best way we have, I will
try packagefeed-stability class.

// Robert

> only viable solution we have to the problem at the moment that I know
> of. Randomly hacking bits of the signature generation isn't going to
> help this unfortunately.
>
> Cheers,
>
> Richard
>
>

Re: PRServer's problem

[Joshua G Lock]

On Thu, 2016-05-19 at 18:12 +0800, Robert Yang wrote:
> 
> On 05/19/2016 05:45 PM, Richard Purdie wrote:
> > Users are free to set their own policies, the system was designed
> > to do
> > that. If WindRiver wants to have a much more permissive policy, I'm
> > more than happy for them to do so.
> Thanks, frankly speaking, not only WindRiver wants this. After cloud
> computing and virtualization gets hot, more and more users want to
> customize their own images (for saving disk space, memory and
> security
> reason), oe/yocto is very good at customizing images, so more and
> more people try to use it to build their own distros, where live
> upgrades becomes very important.
> 

The desire for this is well understood, which is why work began on the
packagefeed-stability class. 

Paul's initial effort is solid and works for simple cases, I had a
brief look at it and tried to identify some of the issues and wrote
some minor patches (for things like versioned depends and copying all
packages for a recipe as soon as a difference is found) but
unfortunately I haven't had time to finish the work.

The main piece we're missing is to define tests (in the test suite) to
validate the class works as expected with each of the package backends
and through those tests identify areas where it doesn't work (and of
course fix them).

Regards,

Joshua

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8318

Re: PRServer's problem

[Paul Eggleton]

On Thu, 19 May 2016 11:37:03 Joshua G Lock wrote:
> On Thu, 2016-05-19 at 18:12 +0800, Robert Yang wrote:
> > On 05/19/2016 05:45 PM, Richard Purdie wrote:
> > > Users are free to set their own policies, the system was designed
> > > to do
> > > that. If WindRiver wants to have a much more permissive policy, I'm
> > > more than happy for them to do so.
> > 
> > Thanks, frankly speaking, not only WindRiver wants this. After cloud
> > computing and virtualization gets hot, more and more users want to
> > customize their own images (for saving disk space, memory and
> > security
> > reason), oe/yocto is very good at customizing images, so more and
> > more people try to use it to build their own distros, where live
> > upgrades becomes very important.
> 
> The desire for this is well understood, which is why work began on the
> packagefeed-stability class. 
> 
> Paul's initial effort is solid and works for simple cases, I had a
> brief look at it and tried to identify some of the issues and wrote
> some minor patches (for things like versioned depends and copying all
> packages for a recipe as soon as a difference is found) but
> unfortunately I haven't had time to finish the work.
> 
> The main piece we're missing is to define tests (in the test suite) to
> validate the class works as expected with each of the package backends
> and through those tests identify areas where it doesn't work (and of
> course fix them).
> 
> Regards,
> 
> Joshua
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8318

Also as I believe I mentioned in my RFC (IIRC) whilst it does basically work I 
was not exactly impressed by the build-compare tool's error handling, there 
were errors produced during some of my runs and yet the script itself 
succeeded, which gives me less confidence about the results. I haven't looked 
to see if there's a new version upstream that is improved in this regard 
though, and even if there isn't we do have the option of patching it ourselves 
(and sending the fix upstream, naturally).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: PRServer's problem

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 6434 bytes --]

As the commit says, small change in package.bbclass also causes all
packages to be recreated with PR bump even when the content is most likely
the same.

Fixing bug in gcc may at least provide different binaries so it might be
useful to upgrade them on target (or at least distinguish if they were
already rebuilt with fixed gcc-cross or not).

Reverting the commit doesn't fix the issue that sstate handler cannot
compare if the change in signature produced "significantly different"
binaries. If it's reverted in oe-core I'll just revert the revert in our
builds, because we care about reproducible builds and we already gave up on
working upgrade paths, which are broken too often anyway.

Reflash of "system" partitions and storing user data/configuration
somewhere else is faster and safer since the sstate was invented (since we
stopped using OE classic). You can find some rant e-mails from me about
this e.g.
http://lists.openembedded.org/pipermail/openembedded-core/2011-November/051354.html
and the Yocto bug I gave you.

Regards,

On Thu, May 19, 2016 at 4:33 AM, Robert Yang <liezhi.yang@windriver.com>
wrote:

>
> Hi Martin,
>
> I found this patch in the bug:
>
>
> http://git.openembedded.org/openembedded-core/commit/meta/lib/oe/sstatesig.py?id=336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6
>
> Too many PR bumps and rebuildings are caused by this patch. I'm
> not very sure about what this patch tries to fix, it seems that
> it is trying to fix problems when 32bit and 64bit uses the same
> sstate cache? Would you please provide a simple reproducer, please?
>
> Things will become much better after revert this patch. Be less
> stricter or more is a hard problem, if we still need the patch,
> can we leave such a choice to the end user? We can add some vars
> like DROP_NATIVE_SIG ? = "0/1". This would be good to the end user
> who uses stable release like jethro or krogoth to make their
> distributions, and PR Service really matters here. Even if they
> switch the build between 32 and 64 builds (This is unlikely to
> happen for production environment) and got problems, they still
> can fix the problem by rebuilding, this is still much better than
> current status: run a "smart/opkg/apt-get upgrade" on the target,
> *all* of the packages are downloaded and installed again after a
> CVE patch applies on native recipes like pseudo-native or rpm-native,
> but in fact, nothing is changed on the target this is really a bad
> experience.
>
> // Robert
>
>
> On 05/18/2016 06:15 PM, Martin Jansa wrote:
>
>> On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
>>
>>>
>>>
>>> On 05/18/2016 05:20 PM, Martin Jansa wrote:
>>>
>>>> On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>>>>
>>>>> Hi Martin,
>>>>>
>>>>> On 05/18/2016 03:39 PM, Martin Jansa wrote:
>>>>>
>>>>>> See:
>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>>>>>>
>>>>>> Just using recipe checksum wont work, because the main reason for PR
>>>>>> bumps is to
>>>>>> automatically upgrade the packages when one of the dependencies
>>>>>> changes .so
>>>>>> version, which you won't detect from recipe checksum of the app which
>>>>>> is just
>>>>>> using the library.
>>>>>>
>>>>>
>>>>> For the development branch like master, yes, that would happen. But for
>>>>> the stable release like jethro and krogoth, it is unlikely that would
>>>>> happen, and if if does, the user can manually bump the impacted
>>>>> recipe's
>>>>> PR to fix the problem. The current problem is that when *all* recipes'
>>>>> PR are bumped, there is no way to fix the problem.
>>>>>
>>>>
>>>> You can still stop using PR service and start doing manual PR bumps, but
>>>>
>>>
>>> We can't stop PR service and start doing manual PR bumps since we need
>>> keep
>>> update to date with upstream, the changes from upstream don't do the
>>> manual
>>> PR, and I don't think that we have to if they can be done automatically.
>>>
>>> it's quite annoying if you need to bump a lot of recipes you don't
>>>> control :).
>>>>
>>>
>>> What's your opinion about only consider RDEPENDS for PR service's
>>> checksum,
>>> please ?
>>>
>>
>> I would like to have separate handler as described in that bug, option
>> c).
>>
>> Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary
>> rebuilds.
>>
>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
>>>>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>>>>
>>>>>>       The PRServer bumps PR according to do_package's task hash, that
>>>>>>       causes it bumps *all* packages' PR when recipes like
>>>>>> pseudo-native
>>>>>>       and rpm-native is changed. It is a very bad user experience
>>>>>> when we
>>>>>>       run "smart/opkg upgrade" on running target, for example, when
>>>>>> we apply
>>>>>>       a CVE patch to pseudo-native or rpm-native, or do some slight
>>>>>> changes
>>>>>>       in their do_compile, "smart/opkg upgrade" will download/install
>>>>>> *all*
>>>>>>       the packages since all of the packages' PR are bumped.
>>>>>>
>>>>>>       Here are some rough suggestions to fix this problem, and please
>>>>>> feel
>>>>>>       free to give your suggestions.
>>>>>>       1) Do not use do_package's task for bumping PR, the easiest way
>>>>>>           is simulate manually bump PR -- only bump PR when the recipe
>>>>>>           itself's checksum is changed.
>>>>>>
>>>>>>       2) Add a new task for PRServer, redefine its task hash for
>>>>>> bumping
>>>>>>           PR, for example, this task hash only considers RDEPENDS (no
>>>>>>           DEPENDS), and drop any native dependencies.
>>>>>>
>>>>>>       I prefer the first way, and an alternative way maybe add a var
>>>>>> so that
>>>>>>       the user can configure it:
>>>>>>       PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>>>>>>       Or
>>>>>>       PR_CHECKSUM = "<recipe checksum>"
>>>>>>
>>>>>>       --
>>>>>>       Thanks
>>>>>>
>>>>>>       Robert
>>>>>>       --
>>>>>>       _______________________________________________
>>>>>>       Openembedded-core mailing list
>>>>>>       Openembedded-core@lists.openembedded.org
>>>>>>       <mailto:Openembedded-core@lists.openembedded.org>
>>>>>>
>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>>
>>>>>>
>>>>>>
>>>>
>>

[-- Attachment #2: Type: text/html, Size: 8904 bytes --]

Re: PRServer's problem

[Richard Purdie]

On Thu, 2016-05-19 at 10:47 +0200, Martin Jansa wrote:
> As the commit says, small change in package.bbclass also causes all
> packages to be recreated with PR bump even when the content is most
> likely the same.
> 
> Fixing bug in gcc may at least provide different binaries so it might
> be useful to upgrade them on target (or at least distinguish if they
> were already rebuilt with fixed gcc-cross or not).
>
> Reverting the commit doesn't fix the issue that sstate handler cannot
> compare if the change in signature produced "significantly different"
> binaries. If it's reverted in oe-core I'll just revert the revert in
> our builds, because we care about reproducible builds and we already
> gave up on working upgrade paths, which are broken too often anyway.
> 
> Reflash of "system" partitions and storing user data/configuration
> somewhere else is faster and safer since the sstate was invented
> (since we stopped using OE classic). You can find some rant e-mails
> from me about this e.g.
> http://lists.openembedded.org/pipermail/openembedded-core/2011-Novemb
> er/051354.html
> and the Yocto bug I gave you.

FWIW I've not given up on upgrade paths and I do try and ensure basic
things do get spotted and addressed during review.

IMO, the only way we'll improve the upgrade situation is if someone
writes some automated tests for it, such that when we test changes, we
test the upgrade paths work. We have a ton of test automation now so we
have the basic framework to build something like this on top of. I do
appreciate there are some challenges since it does mean maintaining a
baseline state which the upgrade is tested from (previous state and
previous release being the two obvious ones).

Cheers,

Richard

Re: PRServer's problem

[Robert Yang]

Hi Martin,

On 05/19/2016 04:47 PM, Martin Jansa wrote:
> As the commit says, small change in package.bbclass also causes all packages to
> be recreated with PR bump even when the content is most likely the same.

That is another case we need work on.

>
> Fixing bug in gcc may at least provide different binaries so it might be useful
> to upgrade them on target (or at least distinguish if they were already rebuilt
> with fixed gcc-cross or not).

The binary might be different when the compiler fixed a bug, but that may don't
make any sense to the end user. AFAIK, ccache doesn't check /path/to/gcc's hash
value.

>
> Reverting the commit doesn't fix the issue that sstate handler cannot compare if
> the change in signature produced "significantly different" binaries. If it's
> reverted in oe-core I'll just revert the revert in our builds, because we care
> about reproducible builds and we already gave up on working upgrade paths, which
> are broken too often anyway.

Be less or more stricter is a hard problem, so we'd better let's leave this
choice to the end user, something like DROP_NATIVE_SIG ? = "0/1" would help.

I'm afraid that there is nearly nothing we can do to enhance PR server's
user experience based on the current stricter mode. Think about the
distros like Debian, CentOS, Opensuse and so on, their
"apt-get/yum/yast upgrade" never re-install all the pkgs, but the distros
built out by oe-core usually does (especially, when a patch is applied to
a native recipe, then all pkgs are downloaded and re-installed, it's very hard
to explain such a problem to the distro's user).

// Robert

>
> Reflash of "system" partitions and storing user data/configuration somewhere
> else is faster and safer since the sstate was invented (since we stopped using
> OE classic). You can find some rant e-mails from me about this e.g.
> http://lists.openembedded.org/pipermail/openembedded-core/2011-November/051354.html
> and the Yocto bug I gave you.
>
> Regards,
>
> On Thu, May 19, 2016 at 4:33 AM, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
>
>     Hi Martin,
>
>     I found this patch in the bug:
>
>     http://git.openembedded.org/openembedded-core/commit/meta/lib/oe/sstatesig.py?id=336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6
>
>     Too many PR bumps and rebuildings are caused by this patch. I'm
>     not very sure about what this patch tries to fix, it seems that
>     it is trying to fix problems when 32bit and 64bit uses the same
>     sstate cache? Would you please provide a simple reproducer, please?
>
>     Things will become much better after revert this patch. Be less
>     stricter or more is a hard problem, if we still need the patch,
>     can we leave such a choice to the end user? We can add some vars
>     like DROP_NATIVE_SIG ? = "0/1". This would be good to the end user
>     who uses stable release like jethro or krogoth to make their
>     distributions, and PR Service really matters here. Even if they
>     switch the build between 32 and 64 builds (This is unlikely to
>     happen for production environment) and got problems, they still
>     can fix the problem by rebuilding, this is still much better than
>     current status: run a "smart/opkg/apt-get upgrade" on the target,
>     *all* of the packages are downloaded and installed again after a
>     CVE patch applies on native recipes like pseudo-native or rpm-native,
>     but in fact, nothing is changed on the target this is really a bad
>     experience.
>
>     // Robert
>
>
>     On 05/18/2016 06:15 PM, Martin Jansa wrote:
>
>         On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
>
>
>
>             On 05/18/2016 05:20 PM, Martin Jansa wrote:
>
>                 On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>
>                     Hi Martin,
>
>                     On 05/18/2016 03:39 PM, Martin Jansa wrote:
>
>                         See:
>                         https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>
>                         Just using recipe checksum wont work, because the main
>                         reason for PR bumps is to
>                         automatically upgrade the packages when one of the
>                         dependencies changes .so
>                         version, which you won't detect from recipe checksum of
>                         the app which is just
>                         using the library.
>
>
>                     For the development branch like master, yes, that would
>                     happen. But for
>                     the stable release like jethro and krogoth, it is unlikely
>                     that would
>                     happen, and if if does, the user can manually bump the
>                     impacted recipe's
>                     PR to fix the problem. The current problem is that when
>                     *all* recipes'
>                     PR are bumped, there is no way to fix the problem.
>
>
>                 You can still stop using PR service and start doing manual PR
>                 bumps, but
>
>
>             We can't stop PR service and start doing manual PR bumps since we
>             need keep
>             update to date with upstream, the changes from upstream don't do the
>             manual
>             PR, and I don't think that we have to if they can be done automatically.
>
>                 it's quite annoying if you need to bump a lot of recipes you don't
>                 control :).
>
>
>             What's your opinion about only consider RDEPENDS for PR service's
>             checksum,
>             please ?
>
>
>         I would like to have separate handler as described in that bug, option
>         c).
>
>         Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary
>         rebuilds.
>
>                         On Wed, May 18, 2016 at 8:09 AM, Robert Yang
>                         <liezhi.yang@windriver.com
>                         <mailto:liezhi.yang@windriver.com>
>                         <mailto:liezhi.yang@windriver.com
>                         <mailto:liezhi.yang@windriver.com>>> wrote:
>
>                                The PRServer bumps PR according to do_package's
>                         task hash, that
>                                causes it bumps *all* packages' PR when recipes
>                         like pseudo-native
>                                and rpm-native is changed. It is a very bad user
>                         experience when we
>                                run "smart/opkg upgrade" on running target, for
>                         example, when we apply
>                                a CVE patch to pseudo-native or rpm-native, or do
>                         some slight changes
>                                in their do_compile, "smart/opkg upgrade" will
>                         download/install *all*
>                                the packages since all of the packages' PR are
>                         bumped.
>
>                                Here are some rough suggestions to fix this
>                         problem, and please feel
>                                free to give your suggestions.
>                                1) Do not use do_package's task for bumping PR,
>                         the easiest way
>                                    is simulate manually bump PR -- only bump PR
>                         when the recipe
>                                    itself's checksum is changed.
>
>                                2) Add a new task for PRServer, redefine its task
>                         hash for bumping
>                                    PR, for example, this task hash only
>                         considers RDEPENDS (no
>                                    DEPENDS), and drop any native dependencies.
>
>                                I prefer the first way, and an alternative way
>                         maybe add a var so that
>                                the user can configure it:
>                                PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>                                Or
>                                PR_CHECKSUM = "<recipe checksum>"
>
>                                --
>                                Thanks
>
>                                Robert
>                                --
>                                _______________________________________________
>                                Openembedded-core mailing list
>                         Openembedded-core@lists.openembedded.org
>                         <mailto:Openembedded-core@lists.openembedded.org>
>                                <mailto:Openembedded-core@lists.openembedded.org
>                         <mailto:Openembedded-core@lists.openembedded.org>>
>                         http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
>
>
>

Re: PRServer's problem

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 10095 bytes --]

On Thu, May 19, 2016 at 05:45:11PM +0800, Robert Yang wrote:
> Hi Martin,
> 
> On 05/19/2016 04:47 PM, Martin Jansa wrote:
> > As the commit says, small change in package.bbclass also causes all packages to
> > be recreated with PR bump even when the content is most likely the same.
> 
> That is another case we need work on.

It's not another case, because the root cause is the same and you're
asking to revert one symptom of it, instead of trying to resolve the
root cause.

> > Fixing bug in gcc may at least provide different binaries so it might be useful
> > to upgrade them on target (or at least distinguish if they were already rebuilt
> > with fixed gcc-cross or not).
> 
> The binary might be different when the compiler fixed a bug, but that may don't
> make any sense to the end user. AFAIK, ccache doesn't check /path/to/gcc's hash
> value.

That's exactly the same case as small change in package.bbclass which often doesn't
change the binary output at all, which is even worse for end user to
re-download and re-install exactly the same bits just because of EXTENDPRAUTO bump.

> > Reverting the commit doesn't fix the issue that sstate handler cannot compare if
> > the change in signature produced "significantly different" binaries. If it's
> > reverted in oe-core I'll just revert the revert in our builds, because we care
> > about reproducible builds and we already gave up on working upgrade paths, which
> > are broken too often anyway.
> 
> Be less or more stricter is a hard problem, so we'd better let's leave this
> choice to the end user, something like DROP_NATIVE_SIG ? = "0/1" would help.
> 
> I'm afraid that there is nearly nothing we can do to enhance PR server's
> user experience based on the current stricter mode. Think about the
> distros like Debian, CentOS, Opensuse and so on, their
> "apt-get/yum/yast upgrade" never re-install all the pkgs, but the distros
> built out by oe-core usually does (especially, when a patch is applied to
> a native recipe, then all pkgs are downloaded and re-installed, it's very hard
> to explain such a problem to the distro's user).
> 
> // Robert
> 
> >
> > Reflash of "system" partitions and storing user data/configuration somewhere
> > else is faster and safer since the sstate was invented (since we stopped using
> > OE classic). You can find some rant e-mails from me about this e.g.
> > http://lists.openembedded.org/pipermail/openembedded-core/2011-November/051354.html
> > and the Yocto bug I gave you.
> >
> > Regards,
> >
> > On Thu, May 19, 2016 at 4:33 AM, Robert Yang <liezhi.yang@windriver.com
> > <mailto:liezhi.yang@windriver.com>> wrote:
> >
> >
> >     Hi Martin,
> >
> >     I found this patch in the bug:
> >
> >     http://git.openembedded.org/openembedded-core/commit/meta/lib/oe/sstatesig.py?id=336a7897e39b9e42dcfcba9e2520ea96b0c6a8d6
> >
> >     Too many PR bumps and rebuildings are caused by this patch. I'm
> >     not very sure about what this patch tries to fix, it seems that
> >     it is trying to fix problems when 32bit and 64bit uses the same
> >     sstate cache? Would you please provide a simple reproducer, please?
> >
> >     Things will become much better after revert this patch. Be less
> >     stricter or more is a hard problem, if we still need the patch,
> >     can we leave such a choice to the end user? We can add some vars
> >     like DROP_NATIVE_SIG ? = "0/1". This would be good to the end user
> >     who uses stable release like jethro or krogoth to make their
> >     distributions, and PR Service really matters here. Even if they
> >     switch the build between 32 and 64 builds (This is unlikely to
> >     happen for production environment) and got problems, they still
> >     can fix the problem by rebuilding, this is still much better than
> >     current status: run a "smart/opkg/apt-get upgrade" on the target,
> >     *all* of the packages are downloaded and installed again after a
> >     CVE patch applies on native recipes like pseudo-native or rpm-native,
> >     but in fact, nothing is changed on the target this is really a bad
> >     experience.
> >
> >     // Robert
> >
> >
> >     On 05/18/2016 06:15 PM, Martin Jansa wrote:
> >
> >         On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
> >
> >
> >
> >             On 05/18/2016 05:20 PM, Martin Jansa wrote:
> >
> >                 On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
> >
> >                     Hi Martin,
> >
> >                     On 05/18/2016 03:39 PM, Martin Jansa wrote:
> >
> >                         See:
> >                         https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
> >
> >                         Just using recipe checksum wont work, because the main
> >                         reason for PR bumps is to
> >                         automatically upgrade the packages when one of the
> >                         dependencies changes .so
> >                         version, which you won't detect from recipe checksum of
> >                         the app which is just
> >                         using the library.
> >
> >
> >                     For the development branch like master, yes, that would
> >                     happen. But for
> >                     the stable release like jethro and krogoth, it is unlikely
> >                     that would
> >                     happen, and if if does, the user can manually bump the
> >                     impacted recipe's
> >                     PR to fix the problem. The current problem is that when
> >                     *all* recipes'
> >                     PR are bumped, there is no way to fix the problem.
> >
> >
> >                 You can still stop using PR service and start doing manual PR
> >                 bumps, but
> >
> >
> >             We can't stop PR service and start doing manual PR bumps since we
> >             need keep
> >             update to date with upstream, the changes from upstream don't do the
> >             manual
> >             PR, and I don't think that we have to if they can be done automatically.
> >
> >                 it's quite annoying if you need to bump a lot of recipes you don't
> >                 control :).
> >
> >
> >             What's your opinion about only consider RDEPENDS for PR service's
> >             checksum,
> >             please ?
> >
> >
> >         I would like to have separate handler as described in that bug, option
> >         c).
> >
> >         Not only because of unnecessary EXTENDPRAUTO bumps, but also unnecessary
> >         rebuilds.
> >
> >                         On Wed, May 18, 2016 at 8:09 AM, Robert Yang
> >                         <liezhi.yang@windriver.com
> >                         <mailto:liezhi.yang@windriver.com>
> >                         <mailto:liezhi.yang@windriver.com
> >                         <mailto:liezhi.yang@windriver.com>>> wrote:
> >
> >                                The PRServer bumps PR according to do_package's
> >                         task hash, that
> >                                causes it bumps *all* packages' PR when recipes
> >                         like pseudo-native
> >                                and rpm-native is changed. It is a very bad user
> >                         experience when we
> >                                run "smart/opkg upgrade" on running target, for
> >                         example, when we apply
> >                                a CVE patch to pseudo-native or rpm-native, or do
> >                         some slight changes
> >                                in their do_compile, "smart/opkg upgrade" will
> >                         download/install *all*
> >                                the packages since all of the packages' PR are
> >                         bumped.
> >
> >                                Here are some rough suggestions to fix this
> >                         problem, and please feel
> >                                free to give your suggestions.
> >                                1) Do not use do_package's task for bumping PR,
> >                         the easiest way
> >                                    is simulate manually bump PR -- only bump PR
> >                         when the recipe
> >                                    itself's checksum is changed.
> >
> >                                2) Add a new task for PRServer, redefine its task
> >                         hash for bumping
> >                                    PR, for example, this task hash only
> >                         considers RDEPENDS (no
> >                                    DEPENDS), and drop any native dependencies.
> >
> >                                I prefer the first way, and an alternative way
> >                         maybe add a var so that
> >                                the user can configure it:
> >                                PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> >                                Or
> >                                PR_CHECKSUM = "<recipe checksum>"
> >
> >                                --
> >                                Thanks
> >
> >                                Robert
> >                                --
> >                                _______________________________________________
> >                                Openembedded-core mailing list
> >                         Openembedded-core@lists.openembedded.org
> >                         <mailto:Openembedded-core@lists.openembedded.org>
> >                                <mailto:Openembedded-core@lists.openembedded.org
> >                         <mailto:Openembedded-core@lists.openembedded.org>>
> >                         http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >
> >
> >
> >
> >

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: PRServer's problem

[Sergey 'Jin' Bostandzhyan]

Hi,

just wanted to say that we are facing the same issue with PRServer and
live updates, looking forward to a possible solution.

Kind regards,
Sergey

On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
> 
> 
> On 05/18/2016 05:20 PM, Martin Jansa wrote:
> >On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
> >>Hi Martin,
> >>
> >>On 05/18/2016 03:39 PM, Martin Jansa wrote:
> >>>See:
> >>>https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
> >>>
> >>>Just using recipe checksum wont work, because the main reason for PR bumps is to
> >>>automatically upgrade the packages when one of the dependencies changes .so
> >>>version, which you won't detect from recipe checksum of the app which is just
> >>>using the library.
> >>
> >>For the development branch like master, yes, that would happen. But for
> >>the stable release like jethro and krogoth, it is unlikely that would
> >>happen, and if if does, the user can manually bump the impacted recipe's
> >>PR to fix the problem. The current problem is that when *all* recipes'
> >>PR are bumped, there is no way to fix the problem.
> >
> >You can still stop using PR service and start doing manual PR bumps, but
> 
> We can't stop PR service and start doing manual PR bumps since we need keep
> update to date with upstream, the changes from upstream don't do the manual
> PR, and I don't think that we have to if they can be done automatically.
> 
> >it's quite annoying if you need to bump a lot of recipes you don't
> >control :).
> 
> What's your opinion about only consider RDEPENDS for PR service's checksum,
> please ?
> 
> // Robert
> 
> >
> >>>On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
> >>><mailto:liezhi.yang@windriver.com>> wrote:
> >>>
> >>>     The PRServer bumps PR according to do_package's task hash, that
> >>>     causes it bumps *all* packages' PR when recipes like pseudo-native
> >>>     and rpm-native is changed. It is a very bad user experience when we
> >>>     run "smart/opkg upgrade" on running target, for example, when we apply
> >>>     a CVE patch to pseudo-native or rpm-native, or do some slight changes
> >>>     in their do_compile, "smart/opkg upgrade" will download/install *all*
> >>>     the packages since all of the packages' PR are bumped.
> >>>
> >>>     Here are some rough suggestions to fix this problem, and please feel
> >>>     free to give your suggestions.
> >>>     1) Do not use do_package's task for bumping PR, the easiest way
> >>>         is simulate manually bump PR -- only bump PR when the recipe
> >>>         itself's checksum is changed.
> >>>
> >>>     2) Add a new task for PRServer, redefine its task hash for bumping
> >>>         PR, for example, this task hash only considers RDEPENDS (no
> >>>         DEPENDS), and drop any native dependencies.
> >>>
> >>>     I prefer the first way, and an alternative way maybe add a var so that
> >>>     the user can configure it:
> >>>     PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> >>>     Or
> >>>     PR_CHECKSUM = "<recipe checksum>"
> >>>
> >>>     --
> >>>     Thanks
> >>>
> >>>     Robert
> >>>     --
> >>>     _______________________________________________
> >>>     Openembedded-core mailing list
> >>>     Openembedded-core@lists.openembedded.org
> >>>     <mailto:Openembedded-core@lists.openembedded.org>
> >>>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >>>
> >>>
> >
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: PRServer's problem

[Robert Yang]

Hi Sergey,

On 05/18/2016 08:27 PM, Sergey 'Jin' Bostandzhyan wrote:
> Hi,
>
> just wanted to say that we are facing the same issue with PRServer and
> live updates, looking forward to a possible solution.

You can try packagefeed-stability.bbclass on master branch now, and please
feel free to let me know if you find any bugs.

Usage:

INHERIT += "packagefeed-stability"

// Robert

>
> Kind regards,
> Sergey
>
> On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
>>
>>
>> On 05/18/2016 05:20 PM, Martin Jansa wrote:
>>> On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>>>> Hi Martin,
>>>>
>>>> On 05/18/2016 03:39 PM, Martin Jansa wrote:
>>>>> See:
>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>>>>>
>>>>> Just using recipe checksum wont work, because the main reason for PR bumps is to
>>>>> automatically upgrade the packages when one of the dependencies changes .so
>>>>> version, which you won't detect from recipe checksum of the app which is just
>>>>> using the library.
>>>>
>>>> For the development branch like master, yes, that would happen. But for
>>>> the stable release like jethro and krogoth, it is unlikely that would
>>>> happen, and if if does, the user can manually bump the impacted recipe's
>>>> PR to fix the problem. The current problem is that when *all* recipes'
>>>> PR are bumped, there is no way to fix the problem.
>>>
>>> You can still stop using PR service and start doing manual PR bumps, but
>>
>> We can't stop PR service and start doing manual PR bumps since we need keep
>> update to date with upstream, the changes from upstream don't do the manual
>> PR, and I don't think that we have to if they can be done automatically.
>>
>>> it's quite annoying if you need to bump a lot of recipes you don't
>>> control :).
>>
>> What's your opinion about only consider RDEPENDS for PR service's checksum,
>> please ?
>>
>> // Robert
>>
>>>
>>>>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
>>>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>>>
>>>>>      The PRServer bumps PR according to do_package's task hash, that
>>>>>      causes it bumps *all* packages' PR when recipes like pseudo-native
>>>>>      and rpm-native is changed. It is a very bad user experience when we
>>>>>      run "smart/opkg upgrade" on running target, for example, when we apply
>>>>>      a CVE patch to pseudo-native or rpm-native, or do some slight changes
>>>>>      in their do_compile, "smart/opkg upgrade" will download/install *all*
>>>>>      the packages since all of the packages' PR are bumped.
>>>>>
>>>>>      Here are some rough suggestions to fix this problem, and please feel
>>>>>      free to give your suggestions.
>>>>>      1) Do not use do_package's task for bumping PR, the easiest way
>>>>>          is simulate manually bump PR -- only bump PR when the recipe
>>>>>          itself's checksum is changed.
>>>>>
>>>>>      2) Add a new task for PRServer, redefine its task hash for bumping
>>>>>          PR, for example, this task hash only considers RDEPENDS (no
>>>>>          DEPENDS), and drop any native dependencies.
>>>>>
>>>>>      I prefer the first way, and an alternative way maybe add a var so that
>>>>>      the user can configure it:
>>>>>      PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>>>>>      Or
>>>>>      PR_CHECKSUM = "<recipe checksum>"
>>>>>
>>>>>      --
>>>>>      Thanks
>>>>>
>>>>>      Robert
>>>>>      --
>>>>>      _______________________________________________
>>>>>      Openembedded-core mailing list
>>>>>      Openembedded-core@lists.openembedded.org
>>>>>      <mailto:Openembedded-core@lists.openembedded.org>
>>>>>      http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>
>>>>>
>>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: PRServer's problem

[Sergey 'Jin' Bostandzhyan]

Hi Robert,

On Wed, Jul 27, 2016 at 04:06:19PM +0800, Robert Yang wrote:
> >just wanted to say that we are facing the same issue with PRServer and
> >live updates, looking forward to a possible solution.
> 
> You can try packagefeed-stability.bbclass on master branch now, and please
> feel free to let me know if you find any bugs.
> 
> Usage:
> 
> INHERIT += "packagefeed-stability"

thank you, this sounds great! Is it safe to pull in this class to 2.0.2?
This is what we are currently using, so I could only try it by patching my
2.0.2 based tree.

Kind regards,
Sergey


> >On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
> >>
> >>
> >>On 05/18/2016 05:20 PM, Martin Jansa wrote:
> >>>On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
> >>>>Hi Martin,
> >>>>
> >>>>On 05/18/2016 03:39 PM, Martin Jansa wrote:
> >>>>>See:
> >>>>>https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
> >>>>>
> >>>>>Just using recipe checksum wont work, because the main reason for PR bumps is to
> >>>>>automatically upgrade the packages when one of the dependencies changes .so
> >>>>>version, which you won't detect from recipe checksum of the app which is just
> >>>>>using the library.
> >>>>
> >>>>For the development branch like master, yes, that would happen. But for
> >>>>the stable release like jethro and krogoth, it is unlikely that would
> >>>>happen, and if if does, the user can manually bump the impacted recipe's
> >>>>PR to fix the problem. The current problem is that when *all* recipes'
> >>>>PR are bumped, there is no way to fix the problem.
> >>>
> >>>You can still stop using PR service and start doing manual PR bumps, but
> >>
> >>We can't stop PR service and start doing manual PR bumps since we need keep
> >>update to date with upstream, the changes from upstream don't do the manual
> >>PR, and I don't think that we have to if they can be done automatically.
> >>
> >>>it's quite annoying if you need to bump a lot of recipes you don't
> >>>control :).
> >>
> >>What's your opinion about only consider RDEPENDS for PR service's checksum,
> >>please ?
> >>
> >>// Robert
> >>
> >>>
> >>>>>On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
> >>>>><mailto:liezhi.yang@windriver.com>> wrote:
> >>>>>
> >>>>>     The PRServer bumps PR according to do_package's task hash, that
> >>>>>     causes it bumps *all* packages' PR when recipes like pseudo-native
> >>>>>     and rpm-native is changed. It is a very bad user experience when we
> >>>>>     run "smart/opkg upgrade" on running target, for example, when we apply
> >>>>>     a CVE patch to pseudo-native or rpm-native, or do some slight changes
> >>>>>     in their do_compile, "smart/opkg upgrade" will download/install *all*
> >>>>>     the packages since all of the packages' PR are bumped.
> >>>>>
> >>>>>     Here are some rough suggestions to fix this problem, and please feel
> >>>>>     free to give your suggestions.
> >>>>>     1) Do not use do_package's task for bumping PR, the easiest way
> >>>>>         is simulate manually bump PR -- only bump PR when the recipe
> >>>>>         itself's checksum is changed.
> >>>>>
> >>>>>     2) Add a new task for PRServer, redefine its task hash for bumping
> >>>>>         PR, for example, this task hash only considers RDEPENDS (no
> >>>>>         DEPENDS), and drop any native dependencies.
> >>>>>
> >>>>>     I prefer the first way, and an alternative way maybe add a var so that
> >>>>>     the user can configure it:
> >>>>>     PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> >>>>>     Or
> >>>>>     PR_CHECKSUM = "<recipe checksum>"
> >>>>>
> >>>>>     --
> >>>>>     Thanks
> >>>>>
> >>>>>     Robert
> >>>>>     --
> >>>>>     _______________________________________________
> >>>>>     Openembedded-core mailing list
> >>>>>     Openembedded-core@lists.openembedded.org
> >>>>>     <mailto:Openembedded-core@lists.openembedded.org>
> >>>>>     http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >>>>>
> >>>>>
> >>>
> >>--
> >>_______________________________________________
> >>Openembedded-core mailing list
> >>Openembedded-core@lists.openembedded.org
> >>http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >

Re: PRServer's problem

[Robert Yang]

On 07/27/2016 04:30 PM, Sergey 'Jin' Bostandzhyan wrote:
> Hi Robert,
>
> On Wed, Jul 27, 2016 at 04:06:19PM +0800, Robert Yang wrote:
>>> just wanted to say that we are facing the same issue with PRServer and
>>> live updates, looking forward to a possible solution.
>>
>> You can try packagefeed-stability.bbclass on master branch now, and please
>> feel free to let me know if you find any bugs.
>>
>> Usage:
>>
>> INHERIT += "packagefeed-stability"
>
> thank you, this sounds great! Is it safe to pull in this class to 2.0.2?
> This is what we are currently using, so I could only try it by patching my
> 2.0.2 based tree.

Maybe it's not very easy to backport to 2.0.2, these are patches
on master branch:

packagefeed-stability.bbclass: copy all packages of a recipe
packagefeed-stability.bbclass: minor fixes
package_deb.bbclass/package_ipk.bbclass: sort RPROVIDES
build-compare: improve deb and ipk checking
build-compare: fix checking for named pipe and others
build-compare: make pkg-diff.sh 75% faster
build-compare: remove space at head
rpm: make --nosignature work

And there is another one which I sent today for fixing rpm + multilib:
[OE-core] [PATCH 0/1] packagefeed-stability.bbclass: fix multilib + rpm

Also another patch in bitbake:
bitbake: lib/bb/utils.py: return sorted OrderedDict in explode_dep_versions2

// Robert

>
> Kind regards,
> Sergey
>
>
>>> On Wed, May 18, 2016 at 05:31:09PM +0800, Robert Yang wrote:
>>>>
>>>>
>>>> On 05/18/2016 05:20 PM, Martin Jansa wrote:
>>>>> On Wed, May 18, 2016 at 04:03:58PM +0800, Robert Yang wrote:
>>>>>> Hi Martin,
>>>>>>
>>>>>> On 05/18/2016 03:39 PM, Martin Jansa wrote:
>>>>>>> See:
>>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5970
>>>>>>>
>>>>>>> Just using recipe checksum wont work, because the main reason for PR bumps is to
>>>>>>> automatically upgrade the packages when one of the dependencies changes .so
>>>>>>> version, which you won't detect from recipe checksum of the app which is just
>>>>>>> using the library.
>>>>>>
>>>>>> For the development branch like master, yes, that would happen. But for
>>>>>> the stable release like jethro and krogoth, it is unlikely that would
>>>>>> happen, and if if does, the user can manually bump the impacted recipe's
>>>>>> PR to fix the problem. The current problem is that when *all* recipes'
>>>>>> PR are bumped, there is no way to fix the problem.
>>>>>
>>>>> You can still stop using PR service and start doing manual PR bumps, but
>>>>
>>>> We can't stop PR service and start doing manual PR bumps since we need keep
>>>> update to date with upstream, the changes from upstream don't do the manual
>>>> PR, and I don't think that we have to if they can be done automatically.
>>>>
>>>>> it's quite annoying if you need to bump a lot of recipes you don't
>>>>> control :).
>>>>
>>>> What's your opinion about only consider RDEPENDS for PR service's checksum,
>>>> please ?
>>>>
>>>> // Robert
>>>>
>>>>>
>>>>>>> On Wed, May 18, 2016 at 8:09 AM, Robert Yang <liezhi.yang@windriver.com
>>>>>>> <mailto:liezhi.yang@windriver.com>> wrote:
>>>>>>>
>>>>>>>      The PRServer bumps PR according to do_package's task hash, that
>>>>>>>      causes it bumps *all* packages' PR when recipes like pseudo-native
>>>>>>>      and rpm-native is changed. It is a very bad user experience when we
>>>>>>>      run "smart/opkg upgrade" on running target, for example, when we apply
>>>>>>>      a CVE patch to pseudo-native or rpm-native, or do some slight changes
>>>>>>>      in their do_compile, "smart/opkg upgrade" will download/install *all*
>>>>>>>      the packages since all of the packages' PR are bumped.
>>>>>>>
>>>>>>>      Here are some rough suggestions to fix this problem, and please feel
>>>>>>>      free to give your suggestions.
>>>>>>>      1) Do not use do_package's task for bumping PR, the easiest way
>>>>>>>          is simulate manually bump PR -- only bump PR when the recipe
>>>>>>>          itself's checksum is changed.
>>>>>>>
>>>>>>>      2) Add a new task for PRServer, redefine its task hash for bumping
>>>>>>>          PR, for example, this task hash only considers RDEPENDS (no
>>>>>>>          DEPENDS), and drop any native dependencies.
>>>>>>>
>>>>>>>      I prefer the first way, and an alternative way maybe add a var so that
>>>>>>>      the user can configure it:
>>>>>>>      PR_CHECKSUM = "${BB_TASKHASH}" (current way)
>>>>>>>      Or
>>>>>>>      PR_CHECKSUM = "<recipe checksum>"
>>>>>>>
>>>>>>>      --
>>>>>>>      Thanks
>>>>>>>
>>>>>>>      Robert
>>>>>>>      --
>>>>>>>      _______________________________________________
>>>>>>>      Openembedded-core mailing list
>>>>>>>      Openembedded-core@lists.openembedded.org
>>>>>>>      <mailto:Openembedded-core@lists.openembedded.org>
>>>>>>>      http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>>>>
>>>>>>>
>>>>>
>>>> --
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>

Re: PRServer's problem

[Mike Looijmans]

In addition, the introduction of PR server also increased the load on the 
OpenPLi "feed" servers with a few terabytes per month. Luckily, bandwidth is 
cheap nowadays :)

On 18-05-16 08:09, Robert Yang wrote:
> The PRServer bumps PR according to do_package's task hash, that
> causes it bumps *all* packages' PR when recipes like pseudo-native
> and rpm-native is changed. It is a very bad user experience when we
> run "smart/opkg upgrade" on running target, for example, when we apply
> a CVE patch to pseudo-native or rpm-native, or do some slight changes
> in their do_compile, "smart/opkg upgrade" will download/install *all*
> the packages since all of the packages' PR are bumped.
>
> Here are some rough suggestions to fix this problem, and please feel
> free to give your suggestions.
> 1) Do not use do_package's task for bumping PR, the easiest way
>     is simulate manually bump PR -- only bump PR when the recipe
>     itself's checksum is changed.
>
> 2) Add a new task for PRServer, redefine its task hash for bumping
>     PR, for example, this task hash only considers RDEPENDS (no
>     DEPENDS), and drop any native dependencies.
>
> I prefer the first way, and an alternative way maybe add a var so that
> the user can configure it:
> PR_CHECKSUM = "${BB_TASKHASH}" (current way)
> Or
> PR_CHECKSUM = "<recipe checksum>"
>



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail