All of lore.kernel.org
 help / color / mirror / Atom feed
* RFC: design of network based PR service
@ 2011-04-28  7:08 Lu, Lianhao
  2011-04-28  7:15   ` Koen Kooi
                   ` (2 more replies)
  0 siblings, 3 replies; 18+ messages in thread
From: Lu, Lianhao @ 2011-04-28  7:08 UTC (permalink / raw)
  To: poky@yoctoproject.org

Hi Guys,

Here is the design of network based PR service, please help to review and 
give you comment. Thanks a lot!

By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the 
task checksum will be in the stamp file, so this implies the end of PR bumps 
since whenever a recipe change happens, the checksum of the appropriate 
tasks will change so everything depends on that tasks will automatically got 
rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).

But there is one piece of building block missing there. Because the package 
feeds needs the PR bumps because some relevant fields in the package file(i.e. 
Version field in .ipk and .deb file, Release field in .rpm file) need it so the 
generated package feeds can be used to upgrade/install new packages in the 
user's linux distribution. So RP proposed this network based PR service at 
https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html. 

Each embedded linux distribution entity using Yocto to build their distributions 
should have a single network based PR service. Multiple build sources could send 
the tuple (PN, task checksum) to this network based PR service. If the PR service 
finds the corresponding (PN, task checksum) in its database, it will return the PR 
values to the client. If not, it will bump the PR for that PN and return the new value 
to the client. The returned value will only goes into the generated package files.

The PR network service will be a separate process other than bitbake, using the 
XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the 
backend database. The table definition would be something like
(
  PN TEXT,
  CheckSum TEXT PRIMARY KEY,
  PR INT
)

Because different type of package files are generated in different tasks, (e.g.
do_package_write_ipk, do_package_write_rpm), we will use the task checksum 
of do_package to ask for PR value, so all the package files will have the same PR value.

To be backward compatible, if the PR value is already set in the recipe, the final PR value 
that goes into the package files will be "PR in recipe.PR returned by the service".

========================
Lianhao Lu
OTC SSG, Intel APAC R&D Ltd.
86-21-61167139







^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [poky] RFC: design of network based PR service
  2011-04-28  7:08 RFC: design of network based PR service Lu, Lianhao
@ 2011-04-28  7:15   ` Koen Kooi
  2011-04-28  9:22   ` Martin Jansa
  2011-04-29  3:10 ` Zhang, Jessica
  2 siblings, 0 replies; 18+ messages in thread
From: Koen Kooi @ 2011-04-28  7:15 UTC (permalink / raw)
  To: Lu, Lianhao; +Cc: poky, Patches and discussions about the oe-core layer

Adding oe-core list to CC:

Op 28 apr 2011, om 09:08 heeft Lu, Lianhao het volgende geschreven:

> Hi Guys,
> 
> Here is the design of network based PR service, please help to review and 
> give you comment. Thanks a lot!
> 
> By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the 
> task checksum will be in the stamp file, so this implies the end of PR bumps 
> since whenever a recipe change happens, the checksum of the appropriate 
> tasks will change so everything depends on that tasks will automatically got 
> rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).
> 
> But there is one piece of building block missing there. Because the package 
> feeds needs the PR bumps because some relevant fields in the package file(i.e. 
> Version field in .ipk and .deb file, Release field in .rpm file) need it so the 
> generated package feeds can be used to upgrade/install new packages in the 
> user's linux distribution. So RP proposed this network based PR service at 
> https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html. 
> 
> Each embedded linux distribution entity using Yocto to build their distributions 
> should have a single network based PR service. Multiple build sources could send 
> the tuple (PN, task checksum) to this network based PR service. If the PR service 
> finds the corresponding (PN, task checksum) in its database, it will return the PR 
> values to the client. If not, it will bump the PR for that PN and return the new value 
> to the client. The returned value will only goes into the generated package files.
> 
> The PR network service will be a separate process other than bitbake, using the 
> XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the 
> backend database. The table definition would be something like
> (
>  PN TEXT,
>  CheckSum TEXT PRIMARY KEY,
>  PR INT
> )
> 
> Because different type of package files are generated in different tasks, (e.g.
> do_package_write_ipk, do_package_write_rpm), we will use the task checksum 
> of do_package to ask for PR value, so all the package files will have the same PR value.
> 
> To be backward compatible, if the PR value is already set in the recipe, the final PR value 
> that goes into the package files will be "PR in recipe.PR returned by the service".
> 
> ========================
> Lianhao Lu
> OTC SSG, Intel APAC R&D Ltd.
> 86-21-61167139
> 
> 
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky




^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: RFC: design of network based PR service
@ 2011-04-28  7:15   ` Koen Kooi
  0 siblings, 0 replies; 18+ messages in thread
From: Koen Kooi @ 2011-04-28  7:15 UTC (permalink / raw)
  To: Lu, Lianhao; +Cc: poky, Patches and discussions about the oe-core layer

Adding oe-core list to CC:

Op 28 apr 2011, om 09:08 heeft Lu, Lianhao het volgende geschreven:

> Hi Guys,
> 
> Here is the design of network based PR service, please help to review and 
> give you comment. Thanks a lot!
> 
> By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the 
> task checksum will be in the stamp file, so this implies the end of PR bumps 
> since whenever a recipe change happens, the checksum of the appropriate 
> tasks will change so everything depends on that tasks will automatically got 
> rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).
> 
> But there is one piece of building block missing there. Because the package 
> feeds needs the PR bumps because some relevant fields in the package file(i.e. 
> Version field in .ipk and .deb file, Release field in .rpm file) need it so the 
> generated package feeds can be used to upgrade/install new packages in the 
> user's linux distribution. So RP proposed this network based PR service at 
> https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html. 
> 
> Each embedded linux distribution entity using Yocto to build their distributions 
> should have a single network based PR service. Multiple build sources could send 
> the tuple (PN, task checksum) to this network based PR service. If the PR service 
> finds the corresponding (PN, task checksum) in its database, it will return the PR 
> values to the client. If not, it will bump the PR for that PN and return the new value 
> to the client. The returned value will only goes into the generated package files.
> 
> The PR network service will be a separate process other than bitbake, using the 
> XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the 
> backend database. The table definition would be something like
> (
>  PN TEXT,
>  CheckSum TEXT PRIMARY KEY,
>  PR INT
> )
> 
> Because different type of package files are generated in different tasks, (e.g.
> do_package_write_ipk, do_package_write_rpm), we will use the task checksum 
> of do_package to ask for PR value, so all the package files will have the same PR value.
> 
> To be backward compatible, if the PR value is already set in the recipe, the final PR value 
> that goes into the package files will be "PR in recipe.PR returned by the service".
> 
> ========================
> Lianhao Lu
> OTC SSG, Intel APAC R&D Ltd.
> 86-21-61167139
> 
> 
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [poky] RFC: design of network based PR service
  2011-04-28  7:08 RFC: design of network based PR service Lu, Lianhao
@ 2011-04-28  9:22   ` Martin Jansa
  2011-04-28  9:22   ` Martin Jansa
  2011-04-29  3:10 ` Zhang, Jessica
  2 siblings, 0 replies; 18+ messages in thread
From: Martin Jansa @ 2011-04-28  9:22 UTC (permalink / raw)
  To: Lu, Lianhao; +Cc: poky@yoctoproject.org, openembedded-core

[-- Attachment #1: Type: text/plain, Size: 3440 bytes --]

On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
> Hi Guys,
> 
> Here is the design of network based PR service, please help to review and 
> give you comment. Thanks a lot!

Hi,

looks good, just wondering if we can use the same mechanism for
LOCALCOUNT numbers in SRCPV, we can even use same table if we prefix
first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column would be
actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd table with
similar structure.

But it wont work very well if one builder is using AUTOREV and another
one isn't, but that can be resolved by allowing only trusted builders
with same configuration (wrt AUTOREV) to send such tuples.

Do we have at least 2 groups of "users".
1) trusted which increments PR when hash is not found
2) public which can query PR for given hash (not sure what they should
   do if hash is not found)

Regards,

> By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the 
> task checksum will be in the stamp file, so this implies the end of PR bumps 
> since whenever a recipe change happens, the checksum of the appropriate 
> tasks will change so everything depends on that tasks will automatically got 
> rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).
> 
> But there is one piece of building block missing there. Because the package 
> feeds needs the PR bumps because some relevant fields in the package file(i.e. 
> Version field in .ipk and .deb file, Release field in .rpm file) need it so the 
> generated package feeds can be used to upgrade/install new packages in the 
> user's linux distribution. So RP proposed this network based PR service at 
> https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html. 
> 
> Each embedded linux distribution entity using Yocto to build their distributions 
> should have a single network based PR service. Multiple build sources could send 
> the tuple (PN, task checksum) to this network based PR service. If the PR service 
> finds the corresponding (PN, task checksum) in its database, it will return the PR 
> values to the client. If not, it will bump the PR for that PN and return the new value 
> to the client. The returned value will only goes into the generated package files.
> 
> The PR network service will be a separate process other than bitbake, using the 
> XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the 
> backend database. The table definition would be something like
> (
>   PN TEXT,
>   CheckSum TEXT PRIMARY KEY,
>   PR INT
> )
> 
> Because different type of package files are generated in different tasks, (e.g.
> do_package_write_ipk, do_package_write_rpm), we will use the task checksum 
> of do_package to ask for PR value, so all the package files will have the same PR value.
> 
> To be backward compatible, if the PR value is already set in the recipe, the final PR value 
> that goes into the package files will be "PR in recipe.PR returned by the service".
> 
> ========================
> Lianhao Lu
> OTC SSG, Intel APAC R&D Ltd.
> 86-21-61167139
> 
> 
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: RFC: design of network based PR service
@ 2011-04-28  9:22   ` Martin Jansa
  0 siblings, 0 replies; 18+ messages in thread
From: Martin Jansa @ 2011-04-28  9:22 UTC (permalink / raw)
  To: Lu, Lianhao; +Cc: poky@yoctoproject.org, openembedded-core

[-- Attachment #1: Type: text/plain, Size: 3440 bytes --]

On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
> Hi Guys,
> 
> Here is the design of network based PR service, please help to review and 
> give you comment. Thanks a lot!

Hi,

looks good, just wondering if we can use the same mechanism for
LOCALCOUNT numbers in SRCPV, we can even use same table if we prefix
first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column would be
actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd table with
similar structure.

But it wont work very well if one builder is using AUTOREV and another
one isn't, but that can be resolved by allowing only trusted builders
with same configuration (wrt AUTOREV) to send such tuples.

Do we have at least 2 groups of "users".
1) trusted which increments PR when hash is not found
2) public which can query PR for given hash (not sure what they should
   do if hash is not found)

Regards,

> By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the 
> task checksum will be in the stamp file, so this implies the end of PR bumps 
> since whenever a recipe change happens, the checksum of the appropriate 
> tasks will change so everything depends on that tasks will automatically got 
> rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).
> 
> But there is one piece of building block missing there. Because the package 
> feeds needs the PR bumps because some relevant fields in the package file(i.e. 
> Version field in .ipk and .deb file, Release field in .rpm file) need it so the 
> generated package feeds can be used to upgrade/install new packages in the 
> user's linux distribution. So RP proposed this network based PR service at 
> https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html. 
> 
> Each embedded linux distribution entity using Yocto to build their distributions 
> should have a single network based PR service. Multiple build sources could send 
> the tuple (PN, task checksum) to this network based PR service. If the PR service 
> finds the corresponding (PN, task checksum) in its database, it will return the PR 
> values to the client. If not, it will bump the PR for that PN and return the new value 
> to the client. The returned value will only goes into the generated package files.
> 
> The PR network service will be a separate process other than bitbake, using the 
> XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the 
> backend database. The table definition would be something like
> (
>   PN TEXT,
>   CheckSum TEXT PRIMARY KEY,
>   PR INT
> )
> 
> Because different type of package files are generated in different tasks, (e.g.
> do_package_write_ipk, do_package_write_rpm), we will use the task checksum 
> of do_package to ask for PR value, so all the package files will have the same PR value.
> 
> To be backward compatible, if the PR value is already set in the recipe, the final PR value 
> that goes into the package files will be "PR in recipe.PR returned by the service".
> 
> ========================
> Lianhao Lu
> OTC SSG, Intel APAC R&D Ltd.
> 86-21-61167139
> 
> 
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: RFC: design of network based PR service
  2011-04-28  9:22   ` Martin Jansa
  (?)
@ 2011-04-28 15:19   ` Mark Hatle
  2011-04-28 21:04     ` Joshua Lock
  -1 siblings, 1 reply; 18+ messages in thread
From: Mark Hatle @ 2011-04-28 15:19 UTC (permalink / raw)
  To: poky

On 4/28/11 4:22 AM, Martin Jansa wrote:
> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>> Hi Guys,
>>
>> Here is the design of network based PR service, please help to review and 
>> give you comment. Thanks a lot!
> 
> Hi,
> 
> looks good, just wondering if we can use the same mechanism for
> LOCALCOUNT numbers in SRCPV, we can even use same table if we prefix
> first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column would be
> actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd table with
> similar structure.
> 
> But it wont work very well if one builder is using AUTOREV and another
> one isn't, but that can be resolved by allowing only trusted builders
> with same configuration (wrt AUTOREV) to send such tuples.
> 
> Do we have at least 2 groups of "users".
> 1) trusted which increments PR when hash is not found
> 2) public which can query PR for given hash (not sure what they should
>    do if hash is not found)

I think this points our something necessary.  There are manual indications of a
change.  I.e. the current "PR" usage.  If I change the recipe, patches, etc I
should expect to update the 'PR' to the next value.  However, if something ELSE
changes (affecting the checksum), or an end user forgets to change the PR, the
auto increment should be used.

The issue is how do you avoid manual conflicts with the automatic increments.
One way this is being done for other systems is using the "." syntax.  User
recipe revisions are captured as such:

PR="r1"

While automatic revisions using a ".N", so the first would be ".0", ".1", ".2", etc.

This way the Version, and PR are still being used.  The checksum is also being
used by the increment server, and the returned value is a global counter based
on the checksum simply changing with the assumption that "newer" is "better".
(A reasonable policy decision!)

I would suggest that the value of the automatically generated PR be constructed
in some way that can be overridden by project (if necessary).  So if someone
wants user the revision format of "PR.CUSTOM.AUTOMATIC" it would be easy to
retrofit into the environment.  (I've seen this format before in distros of
different versions that want to tag the package/recipe with a revision, add in
[custom] distribution version, and then have the build incrementing...)

So the suggestion above of using a new variable to handle the incrementing, a
separate "format" value that the packaging backends would use instead of simply
PR... and I think we have a good extensible solution to the problem.

--Mark

> Regards,
> 
>> By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the 
>> task checksum will be in the stamp file, so this implies the end of PR bumps 
>> since whenever a recipe change happens, the checksum of the appropriate 
>> tasks will change so everything depends on that tasks will automatically got 
>> rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).
>>
>> But there is one piece of building block missing there. Because the package 
>> feeds needs the PR bumps because some relevant fields in the package file(i.e. 
>> Version field in .ipk and .deb file, Release field in .rpm file) need it so the 
>> generated package feeds can be used to upgrade/install new packages in the 
>> user's linux distribution. So RP proposed this network based PR service at 
>> https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html. 
>>
>> Each embedded linux distribution entity using Yocto to build their distributions 
>> should have a single network based PR service. Multiple build sources could send 
>> the tuple (PN, task checksum) to this network based PR service. If the PR service 
>> finds the corresponding (PN, task checksum) in its database, it will return the PR 
>> values to the client. If not, it will bump the PR for that PN and return the new value 
>> to the client. The returned value will only goes into the generated package files.
>>
>> The PR network service will be a separate process other than bitbake, using the 
>> XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the 
>> backend database. The table definition would be something like
>> (
>>   PN TEXT,
>>   CheckSum TEXT PRIMARY KEY,
>>   PR INT
>> )
>>
>> Because different type of package files are generated in different tasks, (e.g.
>> do_package_write_ipk, do_package_write_rpm), we will use the task checksum 
>> of do_package to ask for PR value, so all the package files will have the same PR value.
>>
>> To be backward compatible, if the PR value is already set in the recipe, the final PR value 
>> that goes into the package files will be "PR in recipe.PR returned by the service".
>>
>> ========================
>> Lianhao Lu
>> OTC SSG, Intel APAC R&D Ltd.
>> 86-21-61167139
>>
>>
>>
>>
>>
>> _______________________________________________
>> poky mailing list
>> poky@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/poky
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: RFC: design of network based PR service
  2011-04-28 15:19   ` Mark Hatle
@ 2011-04-28 21:04     ` Joshua Lock
  2011-04-28 21:23         ` Mark Hatle
  0 siblings, 1 reply; 18+ messages in thread
From: Joshua Lock @ 2011-04-28 21:04 UTC (permalink / raw)
  To: Mark Hatle; +Cc: poky

On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
> On 4/28/11 4:22 AM, Martin Jansa wrote:
> > On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
> >> Hi Guys,
> >>
> >> Here is the design of network based PR service, please help to review and 
> >> give you comment. Thanks a lot!
> > 
> > Hi,
> > 
> > looks good, just wondering if we can use the same mechanism for
> > LOCALCOUNT numbers in SRCPV, we can even use same table if we prefix
> > first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column would be
> > actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd table with
> > similar structure.
> > 
> > But it wont work very well if one builder is using AUTOREV and another
> > one isn't, but that can be resolved by allowing only trusted builders
> > with same configuration (wrt AUTOREV) to send such tuples.
> > 
> > Do we have at least 2 groups of "users".
> > 1) trusted which increments PR when hash is not found
> > 2) public which can query PR for given hash (not sure what they should
> >    do if hash is not found)
> 
> I think this points our something necessary.  There are manual indications of a
> change.  I.e. the current "PR" usage.  If I change the recipe, patches, etc I
> should expect to update the 'PR' to the next value.  However, if something ELSE
> changes (affecting the checksum), or an end user forgets to change the PR, the
> auto increment should be used.

Why would you need to update the PR manually? Detecting changes in the
recipe/patches/etc should all be handled by the checksumming.

Cheers,
Joshua
-- 
Joshua Lock
        Yocto Build System Monkey
        Intel Open Source Technology Center



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [poky] RFC: design of network based PR service
  2011-04-28 21:04     ` Joshua Lock
@ 2011-04-28 21:23         ` Mark Hatle
  0 siblings, 0 replies; 18+ messages in thread
From: Mark Hatle @ 2011-04-28 21:23 UTC (permalink / raw)
  To: Joshua Lock; +Cc: poky, Patches and discussions about the oe-core layer

(adding oe-core back to the list as it got dropped off my previous reply)

On 4/28/11 4:04 PM, Joshua Lock wrote:
> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>> Hi Guys,
>>>>
>>>> Here is the design of network based PR service, please help to review and 
>>>> give you comment. Thanks a lot!
>>>
>>> Hi,
>>>
>>> looks good, just wondering if we can use the same mechanism for
>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we prefix
>>> first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column would be
>>> actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd table with
>>> similar structure.
>>>
>>> But it wont work very well if one builder is using AUTOREV and another
>>> one isn't, but that can be resolved by allowing only trusted builders
>>> with same configuration (wrt AUTOREV) to send such tuples.
>>>
>>> Do we have at least 2 groups of "users".
>>> 1) trusted which increments PR when hash is not found
>>> 2) public which can query PR for given hash (not sure what they should
>>>    do if hash is not found)
>>
>> I think this points our something necessary.  There are manual indications of a
>> change.  I.e. the current "PR" usage.  If I change the recipe, patches, etc I
>> should expect to update the 'PR' to the next value.  However, if something ELSE
>> changes (affecting the checksum), or an end user forgets to change the PR, the
>> auto increment should be used.
> 
> Why would you need to update the PR manually? Detecting changes in the
> recipe/patches/etc should all be handled by the checksumming.

Checksums and timestamps are not enough to determine a logical progression of
the components.

We need something that informs the user where these changes fit in the grand
scheme of things.  Are they newer or older then a recipe of the same name (and
package version)?

So this really allows us to visualize multiple levels of upstream work:

Level 1 - Upstream "version"  (PV)

Level 2 - Upstream (oe-core) recipe versioning (PR)

...

Level N - Local build and configuration versioning (checksum based & auto
incrementing)


In a lot of projects 1, 2 and N are enough.. but it's understandable to add in
other intermediate levels to indicate different upstreams along the way.  (Such
as if a company has their own internal distribution..)

I've seen cases where a level 3 exists to indicate a "distribution version"..

Level N (when done manually) has usually captured major system configuration
changes and rebuilding in order to enable solver tools to upgrade properly.


While the checksum will be able to point a unique instance of the recipe to a
given PR... it doesn't guaranty any type of logical numbering.. other then a new
checksum is a new (presumably larger) PR number.  By adding in the various
levels of version information the checksum becomes "more" unique as it only
refers to specific 'upstream' revisions, and make it more logical that a level
1, 2, ... N change means it's a "newer" version of the package.

--Mark

> Cheers,
> Joshua




^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: RFC: design of network based PR service
@ 2011-04-28 21:23         ` Mark Hatle
  0 siblings, 0 replies; 18+ messages in thread
From: Mark Hatle @ 2011-04-28 21:23 UTC (permalink / raw)
  To: Joshua Lock; +Cc: poky, Patches and discussions about the oe-core layer

(adding oe-core back to the list as it got dropped off my previous reply)

On 4/28/11 4:04 PM, Joshua Lock wrote:
> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>> Hi Guys,
>>>>
>>>> Here is the design of network based PR service, please help to review and 
>>>> give you comment. Thanks a lot!
>>>
>>> Hi,
>>>
>>> looks good, just wondering if we can use the same mechanism for
>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we prefix
>>> first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column would be
>>> actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd table with
>>> similar structure.
>>>
>>> But it wont work very well if one builder is using AUTOREV and another
>>> one isn't, but that can be resolved by allowing only trusted builders
>>> with same configuration (wrt AUTOREV) to send such tuples.
>>>
>>> Do we have at least 2 groups of "users".
>>> 1) trusted which increments PR when hash is not found
>>> 2) public which can query PR for given hash (not sure what they should
>>>    do if hash is not found)
>>
>> I think this points our something necessary.  There are manual indications of a
>> change.  I.e. the current "PR" usage.  If I change the recipe, patches, etc I
>> should expect to update the 'PR' to the next value.  However, if something ELSE
>> changes (affecting the checksum), or an end user forgets to change the PR, the
>> auto increment should be used.
> 
> Why would you need to update the PR manually? Detecting changes in the
> recipe/patches/etc should all be handled by the checksumming.

Checksums and timestamps are not enough to determine a logical progression of
the components.

We need something that informs the user where these changes fit in the grand
scheme of things.  Are they newer or older then a recipe of the same name (and
package version)?

So this really allows us to visualize multiple levels of upstream work:

Level 1 - Upstream "version"  (PV)

Level 2 - Upstream (oe-core) recipe versioning (PR)

...

Level N - Local build and configuration versioning (checksum based & auto
incrementing)


In a lot of projects 1, 2 and N are enough.. but it's understandable to add in
other intermediate levels to indicate different upstreams along the way.  (Such
as if a company has their own internal distribution..)

I've seen cases where a level 3 exists to indicate a "distribution version"..

Level N (when done manually) has usually captured major system configuration
changes and rebuilding in order to enable solver tools to upgrade properly.


While the checksum will be able to point a unique instance of the recipe to a
given PR... it doesn't guaranty any type of logical numbering.. other then a new
checksum is a new (presumably larger) PR number.  By adding in the various
levels of version information the checksum becomes "more" unique as it only
refers to specific 'upstream' revisions, and make it more logical that a level
1, 2, ... N change means it's a "newer" version of the package.

--Mark

> Cheers,
> Joshua



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [poky] RFC: design of network based PR service
  2011-04-28 21:23         ` Mark Hatle
@ 2011-04-28 22:02           ` Chris Larson
  -1 siblings, 0 replies; 18+ messages in thread
From: Chris Larson @ 2011-04-28 22:02 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer; +Cc: poky

On Thu, Apr 28, 2011 at 2:23 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> Checksums and timestamps are not enough to determine a logical progression of
> the components.
>
> We need something that informs the user where these changes fit in the grand
> scheme of things.  Are they newer or older then a recipe of the same name (and
> package version)?

I think we might want to stop using the term PR to describe what
you're talking about here.  PR has historically had a quite specific
meaning to us, given how bitbake has operated, and how stamps worked.
It sounds like you want to formalize what we've likely all been doing
manually -- PR .= ".1" or whatever in the .bbappend of a given layer.
Do you think we really need a format string for this, or would
introducing a new variable that's simply a list of extra version
components, and which is used by the packaging classes, likely not by
bitbake itself, get the job done? Am I grasping your need correctly?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [OE-core] RFC: design of network based PR service
@ 2011-04-28 22:02           ` Chris Larson
  0 siblings, 0 replies; 18+ messages in thread
From: Chris Larson @ 2011-04-28 22:02 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer; +Cc: poky

On Thu, Apr 28, 2011 at 2:23 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> Checksums and timestamps are not enough to determine a logical progression of
> the components.
>
> We need something that informs the user where these changes fit in the grand
> scheme of things.  Are they newer or older then a recipe of the same name (and
> package version)?

I think we might want to stop using the term PR to describe what
you're talking about here.  PR has historically had a quite specific
meaning to us, given how bitbake has operated, and how stamps worked.
It sounds like you want to formalize what we've likely all been doing
manually -- PR .= ".1" or whatever in the .bbappend of a given layer.
Do you think we really need a format string for this, or would
introducing a new variable that's simply a list of extra version
components, and which is used by the packaging classes, likely not by
bitbake itself, get the job done? Am I grasping your need correctly?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [OE-core] RFC: design of network based PR service
  2011-04-28 22:02           ` [OE-core] " Chris Larson
  (?)
@ 2011-04-28 22:08           ` Mark Hatle
  -1 siblings, 0 replies; 18+ messages in thread
From: Mark Hatle @ 2011-04-28 22:08 UTC (permalink / raw)
  To: poky

On 4/28/11 5:02 PM, Chris Larson wrote:
> On Thu, Apr 28, 2011 at 2:23 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
>> Checksums and timestamps are not enough to determine a logical progression of
>> the components.
>>
>> We need something that informs the user where these changes fit in the grand
>> scheme of things.  Are they newer or older then a recipe of the same name (and
>> package version)?
> 
> I think we might want to stop using the term PR to describe what
> you're talking about here.  PR has historically had a quite specific
> meaning to us, given how bitbake has operated, and how stamps worked.
> It sounds like you want to formalize what we've likely all been doing
> manually -- PR .= ".1" or whatever in the .bbappend of a given layer.
> Do you think we really need a format string for this, or would
> introducing a new variable that's simply a list of extra version
> components, and which is used by the packaging classes, likely not by
> bitbake itself, get the job done? Am I grasping your need correctly?

I would like a format string so that for my products I can choose to do
arbitrary revision's in packages based on the data present in the recipes (and
elsewhere)..

But the way I see this, you are correct.  This is an effort to help automate the
PR .= ".1" and similar, but based on checksums and system configurations changes....

To me, "PR" stays as it is used (mostly) today.  As a way to indicate revisions
of a particular recipe...  The other changes can optionally site on top of that
based on the format string that the packaging back ends will adopt.  (We can
even make the default that the packaging revision = PR as the default string.)

--Mark


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: RFC: design of network based PR service
  2011-04-28  7:08 RFC: design of network based PR service Lu, Lianhao
  2011-04-28  7:15   ` Koen Kooi
  2011-04-28  9:22   ` Martin Jansa
@ 2011-04-29  3:10 ` Zhang, Jessica
  2 siblings, 0 replies; 18+ messages in thread
From: Zhang, Jessica @ 2011-04-29  3:10 UTC (permalink / raw)
  To: Lu, Lianhao, poky@yoctoproject.org

Lianhao,

I thought RP mentioned that checksum already contains PN so in your db table, do we still need the PN?

- Jessica

-----Original Message-----
From: poky-bounces@yoctoproject.org [mailto:poky-bounces@yoctoproject.org] On Behalf Of Lu, Lianhao
Sent: Thursday, April 28, 2011 12:08 AM
To: poky@yoctoproject.org
Subject: [poky] RFC: design of network based PR service

Hi Guys,

Here is the design of network based PR service, please help to review and
give you comment. Thanks a lot!

By changing the BB_SIGNATURE_HANDLER from "basic" to "basichash", the
task checksum will be in the stamp file, so this implies the end of PR bumps
since whenever a recipe change happens, the checksum of the appropriate
tasks will change so everything depends on that tasks will automatically got
rebuilt. (The checksum mechanism was introduced here at https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.html).

But there is one piece of building block missing there. Because the package
feeds needs the PR bumps because some relevant fields in the package file(i.e.
Version field in .ipk and .deb file, Release field in .rpm file) need it so the
generated package feeds can be used to upgrade/install new packages in the
user's linux distribution. So RP proposed this network based PR service at
https://lists.yoctoproject.org/pipermail/poky/2011-January/002123.html.

Each embedded linux distribution entity using Yocto to build their distributions
should have a single network based PR service. Multiple build sources could send
the tuple (PN, task checksum) to this network based PR service. If the PR service
finds the corresponding (PN, task checksum) in its database, it will return the PR
values to the client. If not, it will bump the PR for that PN and return the new value
to the client. The returned value will only goes into the generated package files.

The PR network service will be a separate process other than bitbake, using the
XMLPRC to communicate with the client(i.e. bitbake process) and sqlite3 as the
backend database. The table definition would be something like
(
  PN TEXT,
  CheckSum TEXT PRIMARY KEY,
  PR INT
)

Because different type of package files are generated in different tasks, (e.g.
do_package_write_ipk, do_package_write_rpm), we will use the task checksum
of do_package to ask for PR value, so all the package files will have the same PR value.

To be backward compatible, if the PR value is already set in the recipe, the final PR value
that goes into the package files will be "PR in recipe.PR returned by the service".

========================
Lianhao Lu
OTC SSG, Intel APAC R&D Ltd.
86-21-61167139





_______________________________________________
poky mailing list
poky@yoctoproject.org
https://lists.yoctoproject.org/listinfo/poky


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [poky] RFC: design of network based PR service
  2011-04-28 21:23         ` Mark Hatle
@ 2011-04-29  5:28           ` Lu, Lianhao
  -1 siblings, 0 replies; 18+ messages in thread
From: Lu, Lianhao @ 2011-04-29  5:28 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer; +Cc: poky@yoctoproject.org

Mark Hatle wrote on 2011-04-29:
> (adding oe-core back to the list as it got dropped off my previous reply)
> 
> On 4/28/11 4:04 PM, Joshua Lock wrote:
>> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>>> Hi Guys,
>>>>> 
>>>>> Here is the design of network based PR service, please help to
>>>>> review and give you comment. Thanks a lot!
>>>> 
>>>> Hi,
>>>> 
>>>> looks good, just wondering if we can use the same mechanism for
>>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we
>>>> prefix first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column
>>>> would be actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd
>>>> table with similar structure.
>>>> 
>>>> But it wont work very well if one builder is using AUTOREV and
>>>> another one isn't, but that can be resolved by allowing only
>>>> trusted builders with same configuration (wrt AUTOREV) to send such tuples.
>>>> 
>>>> Do we have at least 2 groups of "users".
>>>> 1) trusted which increments PR when hash is not found
>>>> 2) public which can query PR for given hash (not sure what they should
>>>>    do if hash is not found)
>>> 
>>> I think this points our something necessary.  There are manual
>>> indications of a change.  I.e. the current "PR" usage.  If I change
>>> the recipe, patches, etc I should expect to update the 'PR' to the
>>> next value.  However, if something ELSE changes (affecting the
>>> checksum), or an end user forgets to change the PR, the auto
>>> increment
> should be used.
>> 
>> Why would you need to update the PR manually? Detecting changes in
>> the recipe/patches/etc should all be handled by the checksumming.
> 
> Checksums and timestamps are not enough to determine a logical
> progression of the components.
> 
> We need something that informs the user where these changes fit in the
> grand scheme of things.  Are they newer or older then a recipe of the
> same name (and package version)?
> 
> So this really allows us to visualize multiple levels of upstream work:
> 
> Level 1 - Upstream "version"  (PV)
> 
> Level 2 - Upstream (oe-core) recipe versioning (PR)
> 
> ...
> 
> Level N - Local build and configuration versioning (checksum based &
> auto incrementing)
> 
> 
> In a lot of projects 1, 2 and N are enough.. but it's understandable
> to add in other intermediate levels to indicate different upstreams along the way.
> (Such as if a company has their own internal distribution..)
> 
> I've seen cases where a level 3 exists to indicate a "distribution version"..
> 
> Level N (when done manually) has usually captured major system
> configuration changes and rebuilding in order to enable solver tools to upgrade properly.
> 
> 
> While the checksum will be able to point a unique instance of the
> recipe to a given PR... it doesn't guaranty any type of logical
> numbering.. other then a new checksum is a new (presumably larger) PR
> number.  By adding in the various levels of version information the
> checksum becomes "more" unique as it only refers to specific
> 'upstream' revisions, and make it more logical that a level 1, 2, ... N change means it's a "newer" version of the package.
> 

If a new PV(and/or PR) value comes, does the level N value need to be reset to 0? Or should it continue to increase?

Best Regards,
Lianhao





^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [OE-core] RFC: design of network based PR service
@ 2011-04-29  5:28           ` Lu, Lianhao
  0 siblings, 0 replies; 18+ messages in thread
From: Lu, Lianhao @ 2011-04-29  5:28 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer; +Cc: poky@yoctoproject.org

Mark Hatle wrote on 2011-04-29:
> (adding oe-core back to the list as it got dropped off my previous reply)
> 
> On 4/28/11 4:04 PM, Joshua Lock wrote:
>> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>>> Hi Guys,
>>>>> 
>>>>> Here is the design of network based PR service, please help to
>>>>> review and give you comment. Thanks a lot!
>>>> 
>>>> Hi,
>>>> 
>>>> looks good, just wondering if we can use the same mechanism for
>>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we
>>>> prefix first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column
>>>> would be actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd
>>>> table with similar structure.
>>>> 
>>>> But it wont work very well if one builder is using AUTOREV and
>>>> another one isn't, but that can be resolved by allowing only
>>>> trusted builders with same configuration (wrt AUTOREV) to send such tuples.
>>>> 
>>>> Do we have at least 2 groups of "users".
>>>> 1) trusted which increments PR when hash is not found
>>>> 2) public which can query PR for given hash (not sure what they should
>>>>    do if hash is not found)
>>> 
>>> I think this points our something necessary.  There are manual
>>> indications of a change.  I.e. the current "PR" usage.  If I change
>>> the recipe, patches, etc I should expect to update the 'PR' to the
>>> next value.  However, if something ELSE changes (affecting the
>>> checksum), or an end user forgets to change the PR, the auto
>>> increment
> should be used.
>> 
>> Why would you need to update the PR manually? Detecting changes in
>> the recipe/patches/etc should all be handled by the checksumming.
> 
> Checksums and timestamps are not enough to determine a logical
> progression of the components.
> 
> We need something that informs the user where these changes fit in the
> grand scheme of things.  Are they newer or older then a recipe of the
> same name (and package version)?
> 
> So this really allows us to visualize multiple levels of upstream work:
> 
> Level 1 - Upstream "version"  (PV)
> 
> Level 2 - Upstream (oe-core) recipe versioning (PR)
> 
> ...
> 
> Level N - Local build and configuration versioning (checksum based &
> auto incrementing)
> 
> 
> In a lot of projects 1, 2 and N are enough.. but it's understandable
> to add in other intermediate levels to indicate different upstreams along the way.
> (Such as if a company has their own internal distribution..)
> 
> I've seen cases where a level 3 exists to indicate a "distribution version"..
> 
> Level N (when done manually) has usually captured major system
> configuration changes and rebuilding in order to enable solver tools to upgrade properly.
> 
> 
> While the checksum will be able to point a unique instance of the
> recipe to a given PR... it doesn't guaranty any type of logical
> numbering.. other then a new checksum is a new (presumably larger) PR
> number.  By adding in the various levels of version information the
> checksum becomes "more" unique as it only refers to specific
> 'upstream' revisions, and make it more logical that a level 1, 2, ... N change means it's a "newer" version of the package.
> 

If a new PV(and/or PR) value comes, does the level N value need to be reset to 0? Or should it continue to increase?

Best Regards,
Lianhao




^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [OE-core] RFC: design of network based PR service
  2011-04-29  5:28           ` [OE-core] " Lu, Lianhao
  (?)
@ 2011-04-29 12:44           ` Mark Hatle
  -1 siblings, 0 replies; 18+ messages in thread
From: Mark Hatle @ 2011-04-29 12:44 UTC (permalink / raw)
  To: poky

On 4/29/11 12:28 AM, Lu, Lianhao wrote:
> Mark Hatle wrote on 2011-04-29:
>> (adding oe-core back to the list as it got dropped off my previous reply)
>>
>> On 4/28/11 4:04 PM, Joshua Lock wrote:
>>> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>>>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> Here is the design of network based PR service, please help to
>>>>>> review and give you comment. Thanks a lot!
>>>>>
>>>>> Hi,
>>>>>
>>>>> looks good, just wondering if we can use the same mechanism for
>>>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we
>>>>> prefix first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column
>>>>> would be actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd
>>>>> table with similar structure.
>>>>>
>>>>> But it wont work very well if one builder is using AUTOREV and
>>>>> another one isn't, but that can be resolved by allowing only
>>>>> trusted builders with same configuration (wrt AUTOREV) to send such tuples.
>>>>>
>>>>> Do we have at least 2 groups of "users".
>>>>> 1) trusted which increments PR when hash is not found
>>>>> 2) public which can query PR for given hash (not sure what they should
>>>>>    do if hash is not found)
>>>>
>>>> I think this points our something necessary.  There are manual
>>>> indications of a change.  I.e. the current "PR" usage.  If I change
>>>> the recipe, patches, etc I should expect to update the 'PR' to the
>>>> next value.  However, if something ELSE changes (affecting the
>>>> checksum), or an end user forgets to change the PR, the auto
>>>> increment
>> should be used.
>>>
>>> Why would you need to update the PR manually? Detecting changes in
>>> the recipe/patches/etc should all be handled by the checksumming.
>>
>> Checksums and timestamps are not enough to determine a logical
>> progression of the components.
>>
>> We need something that informs the user where these changes fit in the
>> grand scheme of things.  Are they newer or older then a recipe of the
>> same name (and package version)?
>>
>> So this really allows us to visualize multiple levels of upstream work:
>>
>> Level 1 - Upstream "version"  (PV)
>>
>> Level 2 - Upstream (oe-core) recipe versioning (PR)
>>
>> ...
>>
>> Level N - Local build and configuration versioning (checksum based &
>> auto incrementing)
>>
>>
>> In a lot of projects 1, 2 and N are enough.. but it's understandable
>> to add in other intermediate levels to indicate different upstreams along the way.
>> (Such as if a company has their own internal distribution..)
>>
>> I've seen cases where a level 3 exists to indicate a "distribution version"..
>>
>> Level N (when done manually) has usually captured major system
>> configuration changes and rebuilding in order to enable solver tools to upgrade properly.
>>
>>
>> While the checksum will be able to point a unique instance of the
>> recipe to a given PR... it doesn't guaranty any type of logical
>> numbering.. other then a new checksum is a new (presumably larger) PR
>> number.  By adding in the various levels of version information the
>> checksum becomes "more" unique as it only refers to specific
>> 'upstream' revisions, and make it more logical that a level 1, 2, ... N change means it's a "newer" version of the package.
>>
> 
> If a new PV(and/or PR) value comes, does the level N value need to be reset to 0? Or should it continue to increase?

Either behavior will work.  However, I'm used to seeing it reset on a new PV-PR
combination.

This likely means that beyond the checksum, the server will need the PN, PV and
existing PR (the overall 'levels' listed above is really whats needed.. so it
might be arbitrary based on a specific formatting string.)  Then this
combination can be used to generate an existing, the first or the "next" N level
number.

--Mark

> Best Regards,
> Lianhao
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [poky] RFC: design of network based PR service
  2011-04-29  5:28           ` [OE-core] " Lu, Lianhao
@ 2011-05-02 22:40             ` Khem Raj
  -1 siblings, 0 replies; 18+ messages in thread
From: Khem Raj @ 2011-05-02 22:40 UTC (permalink / raw)
  To: Lu, Lianhao
  Cc: poky@yoctoproject.org,
	Patches and discussions about the oe-core layer

On Thu, Apr 28, 2011 at 10:28 PM, Lu, Lianhao <lianhao.lu@intel.com> wrote:
> Mark Hatle wrote on 2011-04-29:
>> (adding oe-core back to the list as it got dropped off my previous reply)
>>
>> On 4/28/11 4:04 PM, Joshua Lock wrote:
>>> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>>>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> Here is the design of network based PR service, please help to
>>>>>> review and give you comment. Thanks a lot!
>>>>>
>>>>> Hi,
>>>>>
>>>>> looks good, just wondering if we can use the same mechanism for
>>>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we
>>>>> prefix first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column
>>>>> would be actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd
>>>>> table with similar structure.
>>>>>
>>>>> But it wont work very well if one builder is using AUTOREV and
>>>>> another one isn't, but that can be resolved by allowing only
>>>>> trusted builders with same configuration (wrt AUTOREV) to send such tuples.
>>>>>
>>>>> Do we have at least 2 groups of "users".
>>>>> 1) trusted which increments PR when hash is not found
>>>>> 2) public which can query PR for given hash (not sure what they should
>>>>>    do if hash is not found)
>>>>
>>>> I think this points our something necessary.  There are manual
>>>> indications of a change.  I.e. the current "PR" usage.  If I change
>>>> the recipe, patches, etc I should expect to update the 'PR' to the
>>>> next value.  However, if something ELSE changes (affecting the
>>>> checksum), or an end user forgets to change the PR, the auto
>>>> increment
>> should be used.
>>>
>>> Why would you need to update the PR manually? Detecting changes in
>>> the recipe/patches/etc should all be handled by the checksumming.
>>
>> Checksums and timestamps are not enough to determine a logical
>> progression of the components.
>>
>> We need something that informs the user where these changes fit in the
>> grand scheme of things.  Are they newer or older then a recipe of the
>> same name (and package version)?
>>
>> So this really allows us to visualize multiple levels of upstream work:
>>
>> Level 1 - Upstream "version"  (PV)
>>
>> Level 2 - Upstream (oe-core) recipe versioning (PR)
>>
>> ...
>>
>> Level N - Local build and configuration versioning (checksum based &
>> auto incrementing)
>>
>>
>> In a lot of projects 1, 2 and N are enough.. but it's understandable
>> to add in other intermediate levels to indicate different upstreams along the way.
>> (Such as if a company has their own internal distribution..)
>>
>> I've seen cases where a level 3 exists to indicate a "distribution version"..
>>
>> Level N (when done manually) has usually captured major system
>> configuration changes and rebuilding in order to enable solver tools to upgrade properly.
>>
>>
>> While the checksum will be able to point a unique instance of the
>> recipe to a given PR... it doesn't guaranty any type of logical
>> numbering.. other then a new checksum is a new (presumably larger) PR
>> number.  By adding in the various levels of version information the
>> checksum becomes "more" unique as it only refers to specific
>> 'upstream' revisions, and make it more logical that a level 1, 2, ... N change means it's a "newer" version of the package.
>>
>
> If a new PV(and/or PR) value comes, does the level N value need to be reset to 0? Or should it continue to increase?
>

I am in favor of reseting it to 0

> Best Regards,
> Lianhao
>
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [OE-core] RFC: design of network based PR service
@ 2011-05-02 22:40             ` Khem Raj
  0 siblings, 0 replies; 18+ messages in thread
From: Khem Raj @ 2011-05-02 22:40 UTC (permalink / raw)
  To: Lu, Lianhao
  Cc: poky@yoctoproject.org,
	Patches and discussions about the oe-core layer

On Thu, Apr 28, 2011 at 10:28 PM, Lu, Lianhao <lianhao.lu@intel.com> wrote:
> Mark Hatle wrote on 2011-04-29:
>> (adding oe-core back to the list as it got dropped off my previous reply)
>>
>> On 4/28/11 4:04 PM, Joshua Lock wrote:
>>> On Thu, 2011-04-28 at 10:19 -0500, Mark Hatle wrote:
>>>> On 4/28/11 4:22 AM, Martin Jansa wrote:
>>>>> On Thu, Apr 28, 2011 at 03:08:03PM +0800, Lu, Lianhao wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> Here is the design of network based PR service, please help to
>>>>>> review and give you comment. Thanks a lot!
>>>>>
>>>>> Hi,
>>>>>
>>>>> looks good, just wondering if we can use the same mechanism for
>>>>> LOCALCOUNT numbers in SRCPV, we can even use same table if we
>>>>> prefix first column ie LOCALCOUNT_${PN} and CheckSum in 2nd column
>>>>> would be actuall git hash (from SRCREV or HEAD for AUTOREV) or 2nd
>>>>> table with similar structure.
>>>>>
>>>>> But it wont work very well if one builder is using AUTOREV and
>>>>> another one isn't, but that can be resolved by allowing only
>>>>> trusted builders with same configuration (wrt AUTOREV) to send such tuples.
>>>>>
>>>>> Do we have at least 2 groups of "users".
>>>>> 1) trusted which increments PR when hash is not found
>>>>> 2) public which can query PR for given hash (not sure what they should
>>>>>    do if hash is not found)
>>>>
>>>> I think this points our something necessary.  There are manual
>>>> indications of a change.  I.e. the current "PR" usage.  If I change
>>>> the recipe, patches, etc I should expect to update the 'PR' to the
>>>> next value.  However, if something ELSE changes (affecting the
>>>> checksum), or an end user forgets to change the PR, the auto
>>>> increment
>> should be used.
>>>
>>> Why would you need to update the PR manually? Detecting changes in
>>> the recipe/patches/etc should all be handled by the checksumming.
>>
>> Checksums and timestamps are not enough to determine a logical
>> progression of the components.
>>
>> We need something that informs the user where these changes fit in the
>> grand scheme of things.  Are they newer or older then a recipe of the
>> same name (and package version)?
>>
>> So this really allows us to visualize multiple levels of upstream work:
>>
>> Level 1 - Upstream "version"  (PV)
>>
>> Level 2 - Upstream (oe-core) recipe versioning (PR)
>>
>> ...
>>
>> Level N - Local build and configuration versioning (checksum based &
>> auto incrementing)
>>
>>
>> In a lot of projects 1, 2 and N are enough.. but it's understandable
>> to add in other intermediate levels to indicate different upstreams along the way.
>> (Such as if a company has their own internal distribution..)
>>
>> I've seen cases where a level 3 exists to indicate a "distribution version"..
>>
>> Level N (when done manually) has usually captured major system
>> configuration changes and rebuilding in order to enable solver tools to upgrade properly.
>>
>>
>> While the checksum will be able to point a unique instance of the
>> recipe to a given PR... it doesn't guaranty any type of logical
>> numbering.. other then a new checksum is a new (presumably larger) PR
>> number.  By adding in the various levels of version information the
>> checksum becomes "more" unique as it only refers to specific
>> 'upstream' revisions, and make it more logical that a level 1, 2, ... N change means it's a "newer" version of the package.
>>
>
> If a new PV(and/or PR) value comes, does the level N value need to be reset to 0? Or should it continue to increase?
>

I am in favor of reseting it to 0

> Best Regards,
> Lianhao
>
>
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>


^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2011-05-02 22:43 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-04-28  7:08 RFC: design of network based PR service Lu, Lianhao
2011-04-28  7:15 ` [poky] " Koen Kooi
2011-04-28  7:15   ` Koen Kooi
2011-04-28  9:22 ` [poky] " Martin Jansa
2011-04-28  9:22   ` Martin Jansa
2011-04-28 15:19   ` Mark Hatle
2011-04-28 21:04     ` Joshua Lock
2011-04-28 21:23       ` [poky] " Mark Hatle
2011-04-28 21:23         ` Mark Hatle
2011-04-28 22:02         ` [poky] " Chris Larson
2011-04-28 22:02           ` [OE-core] " Chris Larson
2011-04-28 22:08           ` Mark Hatle
2011-04-29  5:28         ` [poky] " Lu, Lianhao
2011-04-29  5:28           ` [OE-core] " Lu, Lianhao
2011-04-29 12:44           ` Mark Hatle
2011-05-02 22:40           ` [poky] " Khem Raj
2011-05-02 22:40             ` [OE-core] " Khem Raj
2011-04-29  3:10 ` Zhang, Jessica

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.