PRINC migration questions

PRINC migration questions

[Bryan Evenson]

All,

I am on poky/dylan and have not yet started using the PR server.  I really do want to start using the PR server and stop using PRINC.  However, even more so I really don't want to break my package feeds.  I want to make sure I do my migration correctly and I don't do something that looks like it works okay only to find out it causes problems later.  With that in mind, I am looking for advice on the proper way to migrate away from using PRINC.

Let's say I have a .bbappend in my private layer with the line:

PRINC := "${@int(PRINC) + 4}"

The mainline layer's .bb has the line:

PR = "${INC_PR}.0"

And the mainline layer's .inc has the line:

INC_PR = "r8"

In this scenario, the resulting PR will be "r12.0".  Now let's say I want to start using the PR server and get rid of PRINC in this recipe.  The resulting package contents are to be the same, so I want the resulting PR to be "r12.0".  From some limited testing, if I start the PR server and remove the PRINC line from my .bbappend the resulting PR is "r8.0".  So in my .bbappend should I change the PRINC line to:

INC_PR = "r12"

Or should I change it to something else?  If I change it to a hardcoded value, then I'll have to be careful about what to do if the mainline recipe changes INC_PR.  How have other people handled this situation?

Thanks,
Bryan

Re: PRINC migration questions

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

On Fri, Nov 07, 2014 at 01:31:02PM +0000, Bryan Evenson wrote:
> All,
> 
> I am on poky/dylan and have not yet started using the PR server.  I really do want to start using the PR server and stop using PRINC.  However, even more so I really don't want to break my package feeds.  I want to make sure I do my migration correctly and I don't do something that looks like it works okay only to find out it causes problems later.  With that in mind, I am looking for advice on the proper way to migrate away from using PRINC.
> 
> Let's say I have a .bbappend in my private layer with the line:
> 
> PRINC := "${@int(PRINC) + 4}"
> 
> The mainline layer's .bb has the line:
> 
> PR = "${INC_PR}.0"
> 
> And the mainline layer's .inc has the line:
> 
> INC_PR = "r8"
> 
> In this scenario, the resulting PR will be "r12.0".  Now let's say I want to start using the PR server and get rid of PRINC in this recipe.  The resulting package contents are to be the same, so I want the resulting PR to be "r12.0".  From some limited testing, if I start the PR server and remove the PRINC line from my .bbappend the resulting PR is "r8.0".  So in my .bbappend should I change the PRINC line to:
> 
> INC_PR = "r12"
> 
> Or should I change it to something else?  If I change it to a hardcoded value, then I'll have to be careful about what to do if the mainline recipe changes INC_PR.  How have other people handled this situation?

PR service won't you help at all with this (it adds another .N to PR
value, so it cannot fix when the level of .N goes backwards).

The only really working solution is to increment INC_PR/PR values in upstream
recipes atomically with PRINC drop in your layers.

For PR service migration the more important part is to correctly migrate
LOCALCOUNT numbers (used in SRCPVs), especially if you're also changing
package architecture (e.g. t2 suffix added in daisy or selecting
different DEFAULTTUNE and all your LOCALCOUNTs in PR server DB will
reset to 0, because of different db key to find them. Luckily you can
pre-populate them with single SQL command if you realize this situation
before first build.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: PRINC migration questions

[Bryan Evenson]

Martin,

> -----Original Message-----
> From: openembedded-devel-bounces@lists.openembedded.org
> [mailto:openembedded-devel-bounces@lists.openembedded.org] On
> Behalf Of Martin Jansa
> Sent: Friday, November 07, 2014 9:36 AM
> To: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] PRINC migration questions
> 
> On Fri, Nov 07, 2014 at 01:31:02PM +0000, Bryan Evenson wrote:
> > All,
> >
> > I am on poky/dylan and have not yet started using the PR server.  I really do
> want to start using the PR server and stop using PRINC.  However, even more
> so I really don't want to break my package feeds.  I want to make sure I do
> my migration correctly and I don't do something that looks like it works okay
> only to find out it causes problems later.  With that in mind, I am looking for
> advice on the proper way to migrate away from using PRINC.
> >
> > Let's say I have a .bbappend in my private layer with the line:
> >
> > PRINC := "${@int(PRINC) + 4}"
> >
> > The mainline layer's .bb has the line:
> >
> > PR = "${INC_PR}.0"
> >
> > And the mainline layer's .inc has the line:
> >
> > INC_PR = "r8"
> >
> > In this scenario, the resulting PR will be "r12.0".  Now let's say I want to start
> using the PR server and get rid of PRINC in this recipe.  The resulting package
> contents are to be the same, so I want the resulting PR to be "r12.0".  From
> some limited testing, if I start the PR server and remove the PRINC line from
> my .bbappend the resulting PR is "r8.0".  So in my .bbappend should I change
> the PRINC line to:
> >
> > INC_PR = "r12"
> >
> > Or should I change it to something else?  If I change it to a hardcoded value,
> then I'll have to be careful about what to do if the mainline recipe changes
> INC_PR.  How have other people handled this situation?
> 
> PR service won't you help at all with this (it adds another .N to PR value, so it
> cannot fix when the level of .N goes backwards).
> 
> The only really working solution is to increment INC_PR/PR values in
> upstream recipes atomically with PRINC drop in your layers.
> 

Thanks for the information.  Just a few more questions to make sure I get this right.  We have a local Git server which has clones of all the Yocto/oe layers that we are using.  So your suggestion for the example scenario would be to:
1. Remove PRINC reference from our .bbappend
2. Change .inc in mainline recipe to INC_PR = "r12"
3. Push changes to our local Git repositories.

Then if later I were to merge in updates from the official layers, there would be a merge conflict and I would know at merge time that I need to inspect INC_PR for being the correct value.  Does that sound right?

> For PR service migration the more important part is to correctly migrate
> LOCALCOUNT numbers (used in SRCPVs), especially if you're also changing
> package architecture (e.g. t2 suffix added in daisy or selecting different
> DEFAULTTUNE and all your LOCALCOUNTs in PR server DB will reset to 0,
> because of different db key to find them. Luckily you can pre-populate them
> with single SQL command if you realize this situation before first build.
> 

OK, now I'm confused.  I see no reference to LOCALCOUNT anywhere in the Yocto Project Mega Manual.  The only reference I see to LOCALCOUNT is on the PR Service wiki which states that you can call "bitbake-prserv-tool migrate_localcount" to migrate LOCALCOUNTs to AUTOINCs (and AUTOINC is another variable I cannot find mentioned anywhere else).  So I don't fully understand how to recognize if I need to migrate LOCALCOUNT numbers or what the SQL command would be to perform the migration.  But it sounds like this is a pretty important piece to get right during migration.  Could you explain this piece for PR service migration in more detail, or point me to the proper place in the documentation that describes this part of migration?

Thanks,
Bryan 

Re: PRINC migration questions

[Michael_E_Brown]

I have an additional question on the same subject.

We are migrating from poky 1.3, and I don't see any way I can start using the PR server (multi-site config, firewalled off, IT restrictions on running random network daemons, etc).

What is the solution for this case?
--
Michael
________________________________________
From: openembedded-devel-bounces@lists.openembedded.org [openembedded-devel-bounces@lists.openembedded.org] On Behalf Of Bryan Evenson [bevenson@melinkcorp.com]
Sent: Friday, November 07, 2014 7:31 AM
To: openembedded-devel@lists.openembedded.org
Subject: [oe] PRINC migration questions

All,

I am on poky/dylan and have not yet started using the PR server.  I really do want to start using the PR server and stop using PRINC.  However, even more so I really don't want to break my package feeds.  I want to make sure I do my migration correctly and I don't do something that looks like it works okay only to find out it causes problems later.  With that in mind, I am looking for advice on the proper way to migrate away from using PRINC.

Let's say I have a .bbappend in my private layer with the line:

PRINC := "${@int(PRINC) + 4}"

The mainline layer's .bb has the line:

PR = "${INC_PR}.0"

And the mainline layer's .inc has the line:

INC_PR = "r8"

In this scenario, the resulting PR will be "r12.0".  Now let's say I want to start using the PR server and get rid of PRINC in this recipe.  The resulting package contents are to be the same, so I want the resulting PR to be "r12.0".  From some limited testing, if I start the PR server and remove the PRINC line from my .bbappend the resulting PR is "r8.0".  So in my .bbappend should I change the PRINC line to:

INC_PR = "r12"

Or should I change it to something else?  If I change it to a hardcoded value, then I'll have to be careful about what to do if the mainline recipe changes INC_PR.  How have other people handled this situation?

Thanks,
Bryan
--
_______________________________________________
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: PRINC migration questions

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 6036 bytes --]

On Fri, Nov 07, 2014 at 03:36:14PM +0000, Bryan Evenson wrote:
> Martin,
> 
> > -----Original Message-----
> > From: openembedded-devel-bounces@lists.openembedded.org
> > [mailto:openembedded-devel-bounces@lists.openembedded.org] On
> > Behalf Of Martin Jansa
> > Sent: Friday, November 07, 2014 9:36 AM
> > To: openembedded-devel@lists.openembedded.org
> > Subject: Re: [oe] PRINC migration questions
> > 
> > On Fri, Nov 07, 2014 at 01:31:02PM +0000, Bryan Evenson wrote:
> > > All,
> > >
> > > I am on poky/dylan and have not yet started using the PR server.  I really do
> > want to start using the PR server and stop using PRINC.  However, even more
> > so I really don't want to break my package feeds.  I want to make sure I do
> > my migration correctly and I don't do something that looks like it works okay
> > only to find out it causes problems later.  With that in mind, I am looking for
> > advice on the proper way to migrate away from using PRINC.
> > >
> > > Let's say I have a .bbappend in my private layer with the line:
> > >
> > > PRINC := "${@int(PRINC) + 4}"
> > >
> > > The mainline layer's .bb has the line:
> > >
> > > PR = "${INC_PR}.0"
> > >
> > > And the mainline layer's .inc has the line:
> > >
> > > INC_PR = "r8"
> > >
> > > In this scenario, the resulting PR will be "r12.0".  Now let's say I want to start
> > using the PR server and get rid of PRINC in this recipe.  The resulting package
> > contents are to be the same, so I want the resulting PR to be "r12.0".  From
> > some limited testing, if I start the PR server and remove the PRINC line from
> > my .bbappend the resulting PR is "r8.0".  So in my .bbappend should I change
> > the PRINC line to:
> > >
> > > INC_PR = "r12"
> > >
> > > Or should I change it to something else?  If I change it to a hardcoded value,
> > then I'll have to be careful about what to do if the mainline recipe changes
> > INC_PR.  How have other people handled this situation?
> > 
> > PR service won't you help at all with this (it adds another .N to PR value, so it
> > cannot fix when the level of .N goes backwards).
> > 
> > The only really working solution is to increment INC_PR/PR values in
> > upstream recipes atomically with PRINC drop in your layers.
> > 
> 
> Thanks for the information.  Just a few more questions to make sure I get this right.  We have a local Git server which has clones of all the Yocto/oe layers that we are using.  So your suggestion for the example scenario would be to:
> 1. Remove PRINC reference from our .bbappend
> 2. Change .inc in mainline recipe to INC_PR = "r12"
> 3. Push changes to our local Git repositories.
> 
> Then if later I were to merge in updates from the official layers, there would be a merge conflict and I would know at merge time that I need to inspect INC_PR for being the correct value.  Does that sound right?

Yes that's possible, a bit of pain to maintain it, but as you say, at
least you get the warning in form of merge conflict. Just keep in mind
that you can drop these PR, INC_PR bumps when the upstream recipe
increments PE or PV so you can at least drop some of them in time.

> > For PR service migration the more important part is to correctly migrate
> > LOCALCOUNT numbers (used in SRCPVs), especially if you're also changing
> > package architecture (e.g. t2 suffix added in daisy or selecting different
> > DEFAULTTUNE and all your LOCALCOUNTs in PR server DB will reset to 0,
> > because of different db key to find them. Luckily you can pre-populate them
> > with single SQL command if you realize this situation before first build.
> > 
> 
> OK, now I'm confused.  I see no reference to LOCALCOUNT anywhere in the Yocto Project Mega Manual.  The only reference I see to LOCALCOUNT is on the PR Service wiki which states that you can call "bitbake-prserv-tool migrate_localcount" to migrate LOCALCOUNTs to AUTOINCs (and AUTOINC is another variable I cannot find mentioned anywhere else).  So I don't fully understand how to recognize if I need to migrate LOCALCOUNT numbers or what the SQL command would be to perform the migration.  But it sounds like this is a pretty important piece to get right during migration.  Could you explain this piece for PR service migration in more detail, or point me to the proper place in the documentation that describes this part of migration?

The important question is if you're using SRCPV in some recipes you care
about upgrade-path (e.g. linux-yocto is using it).

LOCALCOUNT was name of variable which had the N number in SRCPV
(e.g. "gitrN+<hash>"), originally it was stored in separate sqlite
database cache/bb_persist_data.sqlite3 and maintenance of this number
was moved also to PR service, that's why it's not referenced in current
documentation anymore.

AUTOINC has the same meaning, basically the fetcher will ask PR service:
"do you have AUTOINC number for revision "deadcafe1234", recipe "abc",
architecture "cortexa9t2-neon" and PR service will reply either with
already stored number or max+1 for given recipe+architecture tuple and
store the returned number.

It's important for the migration, because if you depend on N numbers in
gitrN to be from always increasing sequence you'll get zero for all
recipes if you don't migrate your old cache/bb_persist_data.sqlite3 to
PR service's sqlite db.

Honestly I did that all and it was sort of working after few manual
fixes, but I've already gave up on trusting and testing upgrade paths :/.

It's possible to use public PR service with multiple builders, but there
are some cases where the current design wont work and other cases where
it just isn't reliable enough to trust it - if you want reliable images
on target you have to reflash them from time to time (or really know
what you're doing and what you're expecting to get every time you type
"opkg update && opkg upgrade" and most end-users don't know).

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: PRINC migration questions

[Bryan Evenson]

Martin,

> -----Original Message-----
> From: Martin Jansa [mailto:martin.jansa@gmail.com]
> Sent: Friday, November 07, 2014 6:21 PM
> To: Bryan Evenson
> Cc: openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] PRINC migration questions
> 
> On Fri, Nov 07, 2014 at 03:36:14PM +0000, Bryan Evenson wrote:
> > Martin,
> >
> > > -----Original Message-----
> > > From: openembedded-devel-bounces@lists.openembedded.org
> > > [mailto:openembedded-devel-bounces@lists.openembedded.org] On
> Behalf
> > > Of Martin Jansa
> > > Sent: Friday, November 07, 2014 9:36 AM
> > > To: openembedded-devel@lists.openembedded.org
> > > Subject: Re: [oe] PRINC migration questions
> > >
> > > On Fri, Nov 07, 2014 at 01:31:02PM +0000, Bryan Evenson wrote:
> > > > All,
> > > >
> > > > I am on poky/dylan and have not yet started using the PR server.
> > > > I really do
> > > want to start using the PR server and stop using PRINC.  However,
> > > even more so I really don't want to break my package feeds.  I want
> > > to make sure I do my migration correctly and I don't do something
> > > that looks like it works okay only to find out it causes problems
> > > later.  With that in mind, I am looking for advice on the proper way to
> migrate away from using PRINC.
> > > >
> > > > Let's say I have a .bbappend in my private layer with the line:
> > > >
> > > > PRINC := "${@int(PRINC) + 4}"
> > > >
> > > > The mainline layer's .bb has the line:
> > > >
> > > > PR = "${INC_PR}.0"
> > > >
> > > > And the mainline layer's .inc has the line:
> > > >
> > > > INC_PR = "r8"
> > > >
> > > > In this scenario, the resulting PR will be "r12.0".  Now let's say
> > > > I want to start
> > > using the PR server and get rid of PRINC in this recipe.  The
> > > resulting package contents are to be the same, so I want the
> > > resulting PR to be "r12.0".  From some limited testing, if I start
> > > the PR server and remove the PRINC line from my .bbappend the
> > > resulting PR is "r8.0".  So in my .bbappend should I change the PRINC line
> to:
> > > >
> > > > INC_PR = "r12"
> > > >
> > > > Or should I change it to something else?  If I change it to a
> > > > hardcoded value,
> > > then I'll have to be careful about what to do if the mainline recipe
> > > changes INC_PR.  How have other people handled this situation?
> > >
> > > PR service won't you help at all with this (it adds another .N to PR
> > > value, so it cannot fix when the level of .N goes backwards).
> > >
> > > The only really working solution is to increment INC_PR/PR values in
> > > upstream recipes atomically with PRINC drop in your layers.
> > >
> >
> > Thanks for the information.  Just a few more questions to make sure I get
> this right.  We have a local Git server which has clones of all the Yocto/oe
> layers that we are using.  So your suggestion for the example scenario would
> be to:
> > 1. Remove PRINC reference from our .bbappend 2. Change .inc in
> > mainline recipe to INC_PR = "r12"
> > 3. Push changes to our local Git repositories.
> >
> > Then if later I were to merge in updates from the official layers, there
> would be a merge conflict and I would know at merge time that I need to
> inspect INC_PR for being the correct value.  Does that sound right?
> 
> Yes that's possible, a bit of pain to maintain it, but as you say, at least you get
> the warning in form of merge conflict. Just keep in mind that you can drop
> these PR, INC_PR bumps when the upstream recipe increments PE or PV so
> you can at least drop some of them in time.
> 
> > > For PR service migration the more important part is to correctly
> > > migrate LOCALCOUNT numbers (used in SRCPVs), especially if you're
> > > also changing package architecture (e.g. t2 suffix added in daisy or
> > > selecting different DEFAULTTUNE and all your LOCALCOUNTs in PR
> > > server DB will reset to 0, because of different db key to find them.
> > > Luckily you can pre-populate them with single SQL command if you realize
> this situation before first build.
> > >
> >
> > OK, now I'm confused.  I see no reference to LOCALCOUNT anywhere in
> the Yocto Project Mega Manual.  The only reference I see to LOCALCOUNT is
> on the PR Service wiki which states that you can call "bitbake-prserv-tool
> migrate_localcount" to migrate LOCALCOUNTs to AUTOINCs (and AUTOINC is
> another variable I cannot find mentioned anywhere else).  So I don't fully
> understand how to recognize if I need to migrate LOCALCOUNT numbers or
> what the SQL command would be to perform the migration.  But it sounds
> like this is a pretty important piece to get right during migration.  Could you
> explain this piece for PR service migration in more detail, or point me to the
> proper place in the documentation that describes this part of migration?
> 
> The important question is if you're using SRCPV in some recipes you care
> about upgrade-path (e.g. linux-yocto is using it).
> 
> LOCALCOUNT was name of variable which had the N number in SRCPV (e.g.
> "gitrN+<hash>"), originally it was stored in separate sqlite database
> cache/bb_persist_data.sqlite3 and maintenance of this number was moved
> also to PR service, that's why it's not referenced in current documentation
> anymore.
> 
> AUTOINC has the same meaning, basically the fetcher will ask PR service:
> "do you have AUTOINC number for revision "deadcafe1234", recipe "abc",
> architecture "cortexa9t2-neon" and PR service will reply either with already
> stored number or max+1 for given recipe+architecture tuple and store the
> returned number.
> 
> It's important for the migration, because if you depend on N numbers in
> gitrN to be from always increasing sequence you'll get zero for all recipes if
> you don't migrate your old cache/bb_persist_data.sqlite3 to PR service's
> sqlite db.
> 
> Honestly I did that all and it was sort of working after few manual fixes, but
> I've already gave up on trusting and testing upgrade paths :/.
> 
> It's possible to use public PR service with multiple builders, but there are
> some cases where the current design wont work and other cases where it
> just isn't reliable enough to trust it - if you want reliable images on target you
> have to reflash them from time to time (or really know what you're doing
> and what you're expecting to get every time you type "opkg update && opkg
> upgrade" and most end-users don't know).
> 

Thanks for the detailed explanation.  In other words, before release take a close look at what recipes changed and what PRs updated in the resulting packages.  I think all this information would be helpful in a "PRINC Migration" section in the manual.  Given personal experience, I think a lot of people have not migrated away from using PRINC because they may not know how to without breaking package feeds.

Regards,
Bryan