[yocto-autobuilder-helper][PATCH] prepare-shared-repos: Allow to tag poky git

[yocto-autobuilder-helper][PATCH] prepare-shared-repos: Allow to tag poky git

[mathieu.dubois-briand]

From: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>

Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
 scripts/prepare-shared-repos |  6 ++++++
 scripts/utils.py             | 10 ++++++++++
 2 files changed, 16 insertions(+)

diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
index 1b756520b701..3d10af4e6d63 100755
--- a/scripts/prepare-shared-repos
+++ b/scripts/prepare-shared-repos
@@ -16,6 +16,7 @@ import tempfile
 
 import utils
 
+poky_tag_remote = "git@push.yoctoproject.org:poky-ci-archive"
 
 parser = utils.ArgParser(description='Iterates over a set of repositories in a json file and sets up a shared directory containing them.')
 
@@ -26,6 +27,9 @@ parser.add_argument('sharedsrcdir',
 parser.add_argument('-p', '--publish-dir',
                     action='store',
                     help="Where to publish artefacts to (optional)")
+parser.add_argument('-t', '--tag',
+                    action='store',
+                    help="Git tag to create (optional)")
 
 args = parser.parse_args()
 
@@ -49,6 +53,8 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil
             utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir, depth=1)
         if args.publish_dir:
             utils.publishrepo(tempdir, repo, args.publish_dir)
+        if repo == "poky" and args.tag:
+            utils.taggitrepo(tempdir, repo, repos[repo], poky_tag_remote, args.tag)
 
     utils.printheader("Creating shared src tarball")
     subprocess.check_call("tar -I zstd -cf " + args.sharedsrcdir.rstrip("/") + ".tar.zst ./*", shell=True, cwd=tempdir)
diff --git a/scripts/utils.py b/scripts/utils.py
index 8e6463d8c62c..122a56d3c3e4 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -280,6 +280,16 @@ def fetchgitrepo(clonedir, repo, params, stashdir, depth=None):
     subprocess.check_call(["git", "reset", "origin/" + branch, "--hard"], cwd=sharedrepo)
     subprocess.check_call(["git", "reset", revision, "--hard"], cwd=sharedrepo)
 
+def taggitrepo(clonedir, repo, params, tagremote, tagname):
+    sharedrepo = "%s/%s" % (clonedir, repo)
+    revision = params["revision"]
+    print("Creating tag...")
+    subprocess.check_call(["git", "tag", tagname, revision], cwd=sharedrepo)
+    print("Pushing tag...")
+    subprocess.check_call(["git", "remote", "add", "tag", tagremote], cwd=sharedrepo)
+    subprocess.check_call(["git", "push", "tag", tagname], cwd=sharedrepo)
+    subprocess.check_call(["git", "remote", "rm", "tag"], cwd=sharedrepo)
+
 def publishrepo(clonedir, repo, publishdir):
     sharedrepo = "%s/%s" % (clonedir, repo)
     revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=sharedrepo).decode('utf-8').strip()
-- 
2.39.5

Re: [yocto-patches] [yocto-autobuilder-helper][PATCH] prepare-shared-repos: Allow to tag poky git

[Quentin Schulz]

Hi Mathieu,

On 11/15/24 3:02 PM, Mathieu Dubois-Briand via lists.yoctoproject.org wrote:
> From: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
> 
> Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
> ---
>   scripts/prepare-shared-repos |  6 ++++++
>   scripts/utils.py             | 10 ++++++++++
>   2 files changed, 16 insertions(+)
> 
> diff --git a/scripts/prepare-shared-repos b/scripts/prepare-shared-repos
> index 1b756520b701..3d10af4e6d63 100755
> --- a/scripts/prepare-shared-repos
> +++ b/scripts/prepare-shared-repos
> @@ -16,6 +16,7 @@ import tempfile
>   
>   import utils
>   
> +poky_tag_remote = "git@push.yoctoproject.org:poky-ci-archive"
>   
>   parser = utils.ArgParser(description='Iterates over a set of repositories in a json file and sets up a shared directory containing them.')
>   
> @@ -26,6 +27,9 @@ parser.add_argument('sharedsrcdir',
>   parser.add_argument('-p', '--publish-dir',
>                       action='store',
>                       help="Where to publish artefacts to (optional)")
> +parser.add_argument('-t', '--tag',
> +                    action='store',
> +                    help="Git tag to create (optional)")
>   
>   args = parser.parse_args()
>   
> @@ -49,6 +53,8 @@ with tempfile.TemporaryDirectory(prefix="shared-repo-temp-", dir="/home/pokybuil
>               utils.fetchgitrepo(tempdir, repo, repos[repo], stashdir, depth=1)
>           if args.publish_dir:
>               utils.publishrepo(tempdir, repo, args.publish_dir)
> +        if repo == "poky" and args.tag:
> +            utils.taggitrepo(tempdir, repo, repos[repo], poky_tag_remote, args.tag)
>   
>       utils.printheader("Creating shared src tarball")
>       subprocess.check_call("tar -I zstd -cf " + args.sharedsrcdir.rstrip("/") + ".tar.zst ./*", shell=True, cwd=tempdir)
> diff --git a/scripts/utils.py b/scripts/utils.py
> index 8e6463d8c62c..122a56d3c3e4 100644
> --- a/scripts/utils.py
> +++ b/scripts/utils.py
> @@ -280,6 +280,16 @@ def fetchgitrepo(clonedir, repo, params, stashdir, depth=None):
>       subprocess.check_call(["git", "reset", "origin/" + branch, "--hard"], cwd=sharedrepo)
>       subprocess.check_call(["git", "reset", revision, "--hard"], cwd=sharedrepo)
>   
> +def taggitrepo(clonedir, repo, params, tagremote, tagname):
> +    sharedrepo = "%s/%s" % (clonedir, repo)
> +    revision = params["revision"]
> +    print("Creating tag...")
> +    subprocess.check_call(["git", "tag", tagname, revision], cwd=sharedrepo)
> +    print("Pushing tag...")
> +    subprocess.check_call(["git", "remote", "add", "tag", tagremote], cwd=sharedrepo)
> +    subprocess.check_call(["git", "push", "tag", tagname], cwd=sharedrepo)
> +    subprocess.check_call(["git", "remote", "rm", "tag"], cwd=sharedrepo)
> +

Apparently git-push supports pushing without a remote, c.f. the manpage:

        git push [--all | --branches | --mirror | --tags] 
[--follow-tags] [--atomic] [-n | --dry-run] 
[--receive-pack=<git-receive-pack>]
                   [--repo=<repository>] [-f | --force] [-d | --delete] 
[--prune] [-q | --quiet] [-v | --verbose]
                   [-u | --set-upstream] [-o <string> | 
--push-option=<string>]
                   [--[no-]signed|--signed=(true|false|if-asked)]
                   [--force-with-lease[=<refname>[:<expect>]] 
[--force-if-includes]]
                   [--no-verify] [<repository> [<refspec>...]]

[...]

        <repository>
            The "remote" repository that is the destination of a push 
operation. This parameter can be either a URL (see the section GIT URLS
below) or the name of a remote (see the section REMOTES below).

This would avoid leftovers from stopped/interrupted/broken runs to mess 
with the git remote addition which could fail if the remote already exists.

Either don't use an explicit local remote, or remove it beforehand. 
Except that removing a non-existing remote will fail check_call, so you 
probably want to simply run the command and ignore the return code.

Cheers,
Quentin

Re: [yocto-patches] [yocto-autobuilder-helper][PATCH] prepare-shared-repos: Allow to tag poky git

[Mathieu Dubois-Briand]

On Fri, Nov 15, 2024 at 03:21:15PM +0100, Quentin Schulz wrote:
> Hi Mathieu,
> 
> Apparently git-push supports pushing without a remote, c.f. the manpage:
> 
>        git push [--all | --branches | --mirror | --tags] [--follow-tags]
> [--atomic] [-n | --dry-run] [--receive-pack=<git-receive-pack>]
>                   [--repo=<repository>] [-f | --force] [-d | --delete]
> [--prune] [-q | --quiet] [-v | --verbose]
>                   [-u | --set-upstream] [-o <string> |
> --push-option=<string>]
>                   [--[no-]signed|--signed=(true|false|if-asked)]
>                   [--force-with-lease[=<refname>[:<expect>]]
> [--force-if-includes]]
>                   [--no-verify] [<repository> [<refspec>...]]
> 
> [...]
> 
>        <repository>
>            The "remote" repository that is the destination of a push
> operation. This parameter can be either a URL (see the section GIT URLS
> below) or the name of a remote (see the section REMOTES below).

Oh, that's nice! I was definitely not aware of this possibility. I will
send a new version.

> 
> This would avoid leftovers from stopped/interrupted/broken runs to mess with
> the git remote addition which could fail if the remote already exists.
> 

Um right, I did not think about builds potentially interrupted in the
mean time, thanks.

> Either don't use an explicit local remote, or remove it beforehand. Except
> that removing a non-existing remote will fail check_call, so you probably
> want to simply run the command and ignore the return code.
> 
> Cheers,
> Quentin

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com