[PATCH] scripts/oe-setup-layers: add option to skip revs in json

[PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Andrey Zhizhikin]

Current script implementation uses revisions recorded in JSON file to
replicate previously recorded setup. While this is useful for complete
layer setup reproducibility, this does not allow the setup "upgrade" where
this script can be used to use remote branch information and latest HEAD,
have the setup restored, and then saved using updated revisions.

Add new '--skip-revs' parameter which can be used to obtain the updated
layer setup.

Following set of operations could be executed to achieve this:
1. Restore initial layer setup using remote branch HEAD instead of recorded
   revisions by specifying --skip-revs.
2. After layer setup is created, JSON file containing updated revisions is
   retrieved via `bitbake-layers create-layers-setup`
3. Updated JSON file can be checked in to bootstrap repository and used
   later without '--skip-revs' to replicated updated setup.

Signed-off-by: Andrey Zhizhikin <andrey.z@gmail.com>
---
 scripts/oe-setup-layers | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
index 6d49688a32..a4941450d3 100755
--- a/scripts/oe-setup-layers
+++ b/scripts/oe-setup-layers
@@ -69,6 +69,13 @@ def _do_checkout(args, json):
         remotes = r_remote['remotes']
 
         print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch))
+        if args['skip_revs']:
+            if branch and (branch != 'HEAD'):
+                print('\nSkip revision option used, remote {} branch {} HEAD will be used instead of {}'.format(r_name, branch, rev))
+                rev = branch
+            else:
+                print('\nBranch info is missing in json, --skip-revs option is not used')
+
         if not _is_repo_git_repo(repodir):
             cmd = 'git init -q {}'.format(repodir)
             print("Running '{}'".format(cmd))
@@ -108,6 +115,7 @@ except subprocess.CalledProcessError as e:
 
 parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest))
 parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json"))
+parser.add_argument('--skip-revs', action='store_true', help='Skip revisions recorded in json manifest, and use latest revision on branch to restore setup')
 
 args = parser.parse_args()
 
-- 
2.34.1

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexander Kanavin]

Unfortunately I have to say no to this. The problem is that this will
update to latest revisions on all the layers, which I believe is not
what people would want, as everything including poky etc. will be
shifting uncontrollably.

Conceptually, I would not want oe-setup-layers to suffer from feature
creep and become bloated and difficult to maintain. Every little
enhancement is pushing in that direction. It has a json, it does what
the json prescribes, end of story.

We've previously discussed with Jermain how this can be solved on the
bitbake-layers side, e.g. that tool would write out a json where some
layers don't have a hardcoded revision. I think work to implement that
has stalled, not sure. But it's not difficult to add.

Alex



On Wed, 6 Dec 2023 at 10:25, Andrey Zhizhikin <andrey.z@gmail.com> wrote:
>
> Current script implementation uses revisions recorded in JSON file to
> replicate previously recorded setup. While this is useful for complete
> layer setup reproducibility, this does not allow the setup "upgrade" where
> this script can be used to use remote branch information and latest HEAD,
> have the setup restored, and then saved using updated revisions.
>
> Add new '--skip-revs' parameter which can be used to obtain the updated
> layer setup.
>
> Following set of operations could be executed to achieve this:
> 1. Restore initial layer setup using remote branch HEAD instead of recorded
>    revisions by specifying --skip-revs.
> 2. After layer setup is created, JSON file containing updated revisions is
>    retrieved via `bitbake-layers create-layers-setup`
> 3. Updated JSON file can be checked in to bootstrap repository and used
>    later without '--skip-revs' to replicated updated setup.
>
> Signed-off-by: Andrey Zhizhikin <andrey.z@gmail.com>
> ---
>  scripts/oe-setup-layers | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
> index 6d49688a32..a4941450d3 100755
> --- a/scripts/oe-setup-layers
> +++ b/scripts/oe-setup-layers
> @@ -69,6 +69,13 @@ def _do_checkout(args, json):
>          remotes = r_remote['remotes']
>
>          print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch))
> +        if args['skip_revs']:
> +            if branch and (branch != 'HEAD'):
> +                print('\nSkip revision option used, remote {} branch {} HEAD will be used instead of {}'.format(r_name, branch, rev))
> +                rev = branch
> +            else:
> +                print('\nBranch info is missing in json, --skip-revs option is not used')
> +
>          if not _is_repo_git_repo(repodir):
>              cmd = 'git init -q {}'.format(repodir)
>              print("Running '{}'".format(cmd))
> @@ -108,6 +115,7 @@ except subprocess.CalledProcessError as e:
>
>  parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest))
>  parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json"))
> +parser.add_argument('--skip-revs', action='store_true', help='Skip revisions recorded in json manifest, and use latest revision on branch to restore setup')
>
>  args = parser.parse_args()
>
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191879): https://lists.openembedded.org/g/openembedded-core/message/191879
> Mute This Topic: https://lists.openembedded.org/mt/103009912/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Andrey Zhizhikin]

Hello Alex,

On Wed, Dec 6, 2023 at 11:08 AM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> Unfortunately I have to say no to this. The problem is that this will
> update to latest revisions on all the layers, which I believe is not
> what people would want, as everything including poky etc. will be
> shifting uncontrollably.

Correct, and that is exactly the reason why this mechanism is only
offered via additional switch. If it is not supplied - then behavior is
exactly the same, and this is intended. Only those users who would
really like to exercise the shift would use it.

I've also prepared a patch for documenting this switch, but held it back
until the functionality is in. There I do explain the case for CI systems
which would want to capture updated setup of all layers in collection.

>
> Conceptually, I would not want oe-setup-layers to suffer from feature
> creep and become bloated and difficult to maintain. Every little
> enhancement is pushing in that direction. It has a json, it does what
> the json prescribes, end of story.

In this regard I think that json prescription is too strict.

While this totally suites the needs and purpose of passing over the layer
setup to any other client, it really limits the usability of setup-layers only
to that specific case. If this was originally an intended design, then
perhaps a note in documentation would be needed.

>
> We've previously discussed with Jermain how this can be solved on the
> bitbake-layers side, e.g. that tool would write out a json where some
> layers don't have a hardcoded revision. I think work to implement that
> has stalled, not sure. But it's not difficult to add.

Is there any mail thread or patch stack where this has been discussed?

I would like to take a look at what was proposed and perhaps re-use and
re-establish that part instead.

Thanks a lot!

>
> Alex
>
>
>
> On Wed, 6 Dec 2023 at 10:25, Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> >
> > Current script implementation uses revisions recorded in JSON file to
> > replicate previously recorded setup. While this is useful for complete
> > layer setup reproducibility, this does not allow the setup "upgrade" where
> > this script can be used to use remote branch information and latest HEAD,
> > have the setup restored, and then saved using updated revisions.
> >
> > Add new '--skip-revs' parameter which can be used to obtain the updated
> > layer setup.
> >
> > Following set of operations could be executed to achieve this:
> > 1. Restore initial layer setup using remote branch HEAD instead of recorded
> >    revisions by specifying --skip-revs.
> > 2. After layer setup is created, JSON file containing updated revisions is
> >    retrieved via `bitbake-layers create-layers-setup`
> > 3. Updated JSON file can be checked in to bootstrap repository and used
> >    later without '--skip-revs' to replicated updated setup.
> >
> > Signed-off-by: Andrey Zhizhikin <andrey.z@gmail.com>
> > ---
> >  scripts/oe-setup-layers | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >
> > diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
> > index 6d49688a32..a4941450d3 100755
> > --- a/scripts/oe-setup-layers
> > +++ b/scripts/oe-setup-layers
> > @@ -69,6 +69,13 @@ def _do_checkout(args, json):
> >          remotes = r_remote['remotes']
> >
> >          print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch))
> > +        if args['skip_revs']:
> > +            if branch and (branch != 'HEAD'):
> > +                print('\nSkip revision option used, remote {} branch {} HEAD will be used instead of {}'.format(r_name, branch, rev))
> > +                rev = branch
> > +            else:
> > +                print('\nBranch info is missing in json, --skip-revs option is not used')
> > +
> >          if not _is_repo_git_repo(repodir):
> >              cmd = 'git init -q {}'.format(repodir)
> >              print("Running '{}'".format(cmd))
> > @@ -108,6 +115,7 @@ except subprocess.CalledProcessError as e:
> >
> >  parser.add_argument('--destdir', default=defaultdest, help='Where to check out the layers (default is {defaultdest}).'.format(defaultdest=defaultdest))
> >  parser.add_argument('--jsondata', default=__file__+".json", help='File containing the layer data in json format (default is {defaultjson}).'.format(defaultjson=__file__+".json"))
> > +parser.add_argument('--skip-revs', action='store_true', help='Skip revisions recorded in json manifest, and use latest revision on branch to restore setup')
> >
> >  args = parser.parse_args()
> >
> > --
> > 2.34.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#191879): https://lists.openembedded.org/g/openembedded-core/message/191879
> > Mute This Topic: https://lists.openembedded.org/mt/103009912/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >



-- 
Regards,
Andrey.

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexander Kanavin]

On Wed, 6 Dec 2023 at 12:51, Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> > Unfortunately I have to say no to this. The problem is that this will
> > update to latest revisions on all the layers, which I believe is not
> > what people would want, as everything including poky etc. will be
> > shifting uncontrollably.
>
> Correct, and that is exactly the reason why this mechanism is only
> offered via additional switch. If it is not supplied - then behavior is
> exactly the same, and this is intended. Only those users who would
> really like to exercise the shift would use it.

If you want to add something like this, it can't be an all or nothing
switch. There has to be a way to update only some of the layers, and
have others stay where they are. Also, you might want to update to
latest tag on a branch, rather than latest revision. There are many
scenarios, and there's no way oe-setup-layers can cover them, and it
shouldn't try.

> I've also prepared a patch for documenting this switch, but held it back
> until the functionality is in. There I do explain the case for CI systems
> which would want to capture updated setup of all layers in collection.

I understand and agree with the use case, I just don't think updating
all layers to their latest revisions is what many users want, or that
oe-setup-layers is the right place for this functionality.

> In this regard I think that json prescription is too strict.
>
> While this totally suites the needs and purpose of passing over the layer
> setup to any other client, it really limits the usability of setup-layers only
> to that specific case. If this was originally an intended design, then
> perhaps a note in documentation would be needed.

json schema does not require having precise revisions of the layers.
You can specify tags, branches or anything else that 'git checkout'
understands, and oe-setup-layers would follow that. It's just that we
currently don't have tools to write out anything except precise
revisions. But you can edit the json after the fact to point to
something else, then give it to oe-setup-layers, and it will work.

> Is there any mail thread or patch stack where this has been discussed?
>
> I would like to take a look at what was proposed and perhaps re-use and
> re-establish that part instead.

Look for the patchset '[RFC 0/7] bitbake-layers: Add
update-layers-setup' posted on 7 November, particularly the comments
to the individual patches, where we get to the design that should work
and cover the use cases.

Alex

RE: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Jermain Horsman]

> Look for the patchset '[RFC 0/7] bitbake-layers: Add
> update-layers-setup' posted on 7 November, particularly the comments
> to the individual patches, where we get to the design that should work
> and cover the use cases.

I had to work on some other issues that needed my attention,
so not too much progress since, however, I have reserved some
time this month and was planning to work on this.

Sincerely,

Jermain Horsman

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexander Kanavin]

On Wed, 6 Dec 2023 at 13:38, Jermain Horsman <jermain.horsman@nedap.com> wrote:
>
> > Look for the patchset '[RFC 0/7] bitbake-layers: Add
> > update-layers-setup' posted on 7 November, particularly the comments
> > to the individual patches, where we get to the design that should work
> > and cover the use cases.
>
> I had to work on some other issues that needed my attention,
> so not too much progress since, however, I have reserved some
> time this month and was planning to work on this.

The patches are now in testing so Andrey can take a look and check if
they handle he use case he wants to solve:

https://git.yoctoproject.org/poky-contrib/log/?h=abelloni%2Fmaster-next&qt=author&q=horsman

Alex

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexander Kanavin]

On Wed, 31 Jan 2024 at 15:30, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> > > Look for the patchset '[RFC 0/7] bitbake-layers: Add
> > > update-layers-setup' posted on 7 November, particularly the comments
> > > to the individual patches, where we get to the design that should work
> > > and cover the use cases.
> >
> > I had to work on some other issues that needed my attention,
> > so not too much progress since, however, I have reserved some
> > time this month and was planning to work on this.
>
> The patches are now in testing so Andrey can take a look and check if
> they handle he use case he wants to solve:
>
> https://git.yoctoproject.org/poky-contrib/log/?h=abelloni%2Fmaster-next&qt=author&q=horsman

And now looks like they're not. Alexandre, was there some reason to drop them?

[PATCH 1/2] bitbake-layers: Add ability to update the reference of repositories
[PATCH 2/2] bitbake-layers: Add test case layers setup for custom references


Alex

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexandre Belloni]

On 06/02/2024 15:54:27+0100, Alexander Kanavin wrote:
> On Wed, 31 Jan 2024 at 15:30, Alexander Kanavin via
> lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
> wrote:
> > > > Look for the patchset '[RFC 0/7] bitbake-layers: Add
> > > > update-layers-setup' posted on 7 November, particularly the comments
> > > > to the individual patches, where we get to the design that should work
> > > > and cover the use cases.
> > >
> > > I had to work on some other issues that needed my attention,
> > > so not too much progress since, however, I have reserved some
> > > time this month and was planning to work on this.
> >
> > The patches are now in testing so Andrey can take a look and check if
> > they handle he use case he wants to solve:
> >
> > https://git.yoctoproject.org/poky-contrib/log/?h=abelloni%2Fmaster-next&qt=author&q=horsman
> 
> And now looks like they're not. Alexandre, was there some reason to drop them?
> 
> [PATCH 1/2] bitbake-layers: Add ability to update the reference of repositories
> [PATCH 2/2] bitbake-layers: Add test case layers setup for custom references
> 

RP didn't take them for a while and they conflict with your other series
that I wanted to test. I moved those to my staging branch so I remember
to come back to them.



-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexander Kanavin]

On Tue, 6 Feb 2024 at 20:02, Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
> RP didn't take them for a while and they conflict with your other series
> that I wanted to test. I moved those to my staging branch so I remember
> to come back to them.

Thanks, that's fine.

Alex

RE: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Jermain Horsman]

> RP didn't take them for a while and they conflict with your other series
> that I wanted to test. I moved those to my staging branch so I remember
> to come back to them.

At a quick glance it's probably because we both updated the end of the selftest file.
Anyway let me know if/when I need to update anything.

Sincerely,

Jermain Horsman

Re: [OE-core] [PATCH] scripts/oe-setup-layers: add option to skip revs in json

[Alexandre Belloni]

On 06/02/2024 19:23:58+0000, Jermain Horsman wrote:
> > RP didn't take them for a while and they conflict with your other series
> > that I wanted to test. I moved those to my staging branch so I remember
> > to come back to them.
> 
> At a quick glance it's probably because we both updated the end of the selftest file.
> Anyway let me know if/when I need to update anything.
> 

Yes it would be great if you could rebase on top of my master-next.

> Sincerely,
> 
> Jermain Horsman

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com