[PATCH 0/3] scripts: Add a root dir in archve-source.sh

[PATCH 0/3] scripts: Add a root dir in archve-source.sh

[Joel Granados]

When creating an archive source to use in a nix overlay, it is necessary
that all files are appended to the tar.gz file under a root dir. This
series adds a argument (--dir) that will create a root directory and
append all files under it.

The series does not change the default behavior where all files are
added to the tar.gz root.

Comments are greatly appreciated.

PS: Had initialy missed adding qemu-devel to CC. Appologies if you
    received this twice.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
--

---
Joel Granados (3):
      scripts: nixify archive-source.sh
      scripts: Add root dir to arguments
      scripts: Clarify doc messages in archive-source.sh

 scripts/archive-source.sh | 92 +++++++++++++++++++++++++++++++++++------------
 1 file changed, 69 insertions(+), 23 deletions(-)
---
base-commit: ea35a5082a5fe81ce8fd184b0e163cd7b08b7ff7
change-id: 20250408-jag-sysctl-36c35b960c28

Best regards,
-- 
Joel Granados <joel.granados@kernel.org>

[PATCH 1/3] scripts: nixify archive-source.sh

[Joel Granados]

Use "#!/usr/bin/env bash" instead of "#!/bin/bash". This is necessary
for nix environments as they only provide /usr/bin/env at the standard
location.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 scripts/archive-source.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 30677c3ec9032ea01090f74602d839d1c571d012..a469a5e2dec4b05e51474f0a1af190c1ccf23c7e 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 #
 # Author: Fam Zheng <famz@redhat.com>
 #

-- 
2.44.1

[PATCH 2/3] scripts: Add root dir to arguments

[Joel Granados]

The archive-source script appended everything directly on the tar root
making it unusable for creating nix overlays; nix expects all files to
be under a directory in the tarbal.

Add a "-d|--dir" argument that places all files under a custom root.
Behaviour is unchanged when the argument is not given.

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 scripts/archive-source.sh | 78 ++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 63 insertions(+), 15 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index a469a5e2dec4b05e51474f0a1af190c1ccf23c7e..7143aaefc1133fe991b329df3c78e6ed7726322f 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -9,19 +9,6 @@
 # This code is licensed under the GPL version 2 or later.  See
 # the COPYING file in the top-level directory.
 
-error() {
-    printf %s\\n "$*" >&2
-    exit 1
-}
-
-if test $# -lt 1; then
-    error "Usage: $0 <output tarball>"
-fi
-
-tar_file=$(realpath "$1")
-sub_tdir=$(mktemp -d "${tar_file%.tar}.sub.XXXXXXXX")
-sub_file="${sub_tdir}/submodule.tar"
-
 # We want a predictable list of submodules for builds, that is
 # independent of what the developer currently has initialized
 # in their checkout, because the build environment is completely
@@ -32,6 +19,61 @@ subprojects="keycodemapdb libvfio-user berkeley-softfloat-3
   proc-macro-error-1-rs proc-macro-error-attr-1-rs quote-1-rs
   syn-2-rs unicode-ident-1-rs"
 sub_deinit=""
+git_archive_prefix=""
+tar_transform=""
+
+BASENAME="$(basename "${BASH_SOURCE[0]}")"
+USAGE="Usage: ${BASENAME} [OPTIONS] <tarball>
+  <tarball>             Resulting tarball name
+
+  OPTIONS:
+    -d, --dir <DIR>     Files will be placed into DIR inside the tarball
+"
+
+error() {
+    printf %s\\n "$*" >&2
+    exit 1
+}
+
+get_opts() {
+    local short="d:"
+    local long="dir:"
+
+    if ! tmp=$(getopt -o "$short" --long "$long" -n "$BASENAME" -- "$@"); then
+        exit 1
+    fi
+    eval set -- "$tmp"
+    unset tmp
+
+    while true; do
+      case "$1" in
+        '-d' | '--dir' )
+            local tar_dir_name=$"$2/"; shift 2
+            git_archive_prefix="--prefix ${tar_dir_name}"
+            tar_transform="--transform=s|^|${tar_dir_name}|"
+            ;;
+
+        '--' )
+            shift 1
+            ;;
+
+        * )
+            tar_file_name="$1"; shift
+            break
+            ;;
+
+      esac
+    done
+
+    if [ $# -gt 0 ]; then
+        echo "Superfluous args: $@"
+        error "${USAGE}"
+    fi
+
+    tar_file=$(realpath "$tar_file_name")
+    sub_tdir=$(mktemp -d "${tar_file%.tar}.sub.XXXXXXXX")
+    sub_file="${sub_tdir}/submodule.tar"
+}
 
 function cleanup() {
     local status=$?
@@ -73,13 +115,19 @@ function subproject_dir() {
     echo "${dir:-$1}"
 }
 
-git archive --format tar "$(tree_ish)" > "$tar_file"
+get_opts "$@"
+
+git archive --format tar ${git_archive_prefix} "$(tree_ish)" > "$tar_file"
 test $? -ne 0 && error "failed to archive qemu"
 
 for sp in $subprojects; do
     meson subprojects download $sp
     test $? -ne 0 && error "failed to download subproject $sp"
-    tar --append --file "$tar_file" --exclude=.git subprojects/"$(subproject_dir $sp)"
+    tar --append \
+        --file "$tar_file" \
+        --exclude=.git \
+        ${tar_transform} \
+        subprojects/"$(subproject_dir $sp)"
     test $? -ne 0 && error "failed to append subproject $sp to $tar_file"
 done
 exit 0

-- 
2.44.1

[PATCH 3/3] scripts: Clarify doc messages in archive-source.sh

[Joel Granados]

Signed-off-by: Joel Granados <joel.granados@kernel.org>
---
 scripts/archive-source.sh | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 7143aaefc1133fe991b329df3c78e6ed7726322f..eb6ee715a8a3a33d8abe333fd896ad1048821b0f 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -2,17 +2,15 @@
 #
 # Author: Fam Zheng <famz@redhat.com>
 #
-# Archive source tree, including submodules. This is created for test code to
-# export the source files, in order to be built in a different environment,
-# such as in a docker instance or VM.
+# This script exports source files and submodules used by test code to build in
+# "different" environments such as docker or VM.
 #
 # This code is licensed under the GPL version 2 or later.  See
 # the COPYING file in the top-level directory.
 
-# We want a predictable list of submodules for builds, that is
-# independent of what the developer currently has initialized
-# in their checkout, because the build environment is completely
-# different to the host OS.
+# subprojects is a predictable list of submodules that is independent of what
+# the developer has in their checkout. This is needed because the build
+# environment is completely different to the host OS.
 subprojects="keycodemapdb libvfio-user berkeley-softfloat-3
   berkeley-testfloat-3 arbitrary-int-1-rs bilge-0.2-rs
   bilge-impl-0.2-rs either-1-rs itertools-0.11-rs proc-macro2-1-rs

-- 
2.44.1

Re: [PATCH 1/3] scripts: nixify archive-source.sh

[Thomas Huth]

On 08/04/2025 22.14, Joel Granados wrote:
> Use "#!/usr/bin/env bash" instead of "#!/bin/bash". This is necessary
> for nix environments as they only provide /usr/bin/env at the standard
> location.
> 
> Signed-off-by: Joel Granados <joel.granados@kernel.org>
> ---
>   scripts/archive-source.sh | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index 30677c3ec9032ea01090f74602d839d1c571d012..a469a5e2dec4b05e51474f0a1af190c1ccf23c7e 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -1,4 +1,4 @@
> -#!/bin/bash
> +#!/usr/bin/env bash
>   #
>   # Author: Fam Zheng <famz@redhat.com>
>   #
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 1/3] scripts: nixify archive-source.sh

[Paolo Bonzini]

On 4/8/25 22:14, Joel Granados wrote:
> Use "#!/usr/bin/env bash" instead of "#!/bin/bash". This is necessary
> for nix environments as they only provide /usr/bin/env at the standard
> location.

I am confused, how does this not break everything else?  All the test 
scripts in tests/docker/test-* have "#!/bin/bash", and configure has 
"/bin/sh".  How is the environment that runs scripts/archive-source.sh 
different, and why should it be fixed in scripts/archive-source.sh?

These are genuine questions - it would help if the commit message 
explained those... In fact, what is a nix overlay and why would you use 
scripts/archive-source.sh to prepare one? :)

> 
> Signed-off-by: Joel Granados <joel.granados@kernel.org>
> ---
>   scripts/archive-source.sh | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index 30677c3ec9032ea01090f74602d839d1c571d012..a469a5e2dec4b05e51474f0a1af190c1ccf23c7e 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -1,4 +1,4 @@
> -#!/bin/bash
> +#!/usr/bin/env bash
>   #
>   # Author: Fam Zheng <famz@redhat.com>
>   #
> 

Re: [PATCH 1/3] scripts: nixify archive-source.sh

[Joel Granados]

[-- Attachment #1: Type: text/plain, Size: 2375 bytes --]

On Thu, Apr 10, 2025 at 02:18:01PM +0200, Paolo Bonzini wrote:
> On 4/8/25 22:14, Joel Granados wrote:
> > Use "#!/usr/bin/env bash" instead of "#!/bin/bash". This is necessary
> > for nix environments as they only provide /usr/bin/env at the standard
> > location.
> 
> I am confused, how does this not break everything else?  All the test
> scripts in tests/docker/test-* have "#!/bin/bash", and configure has
> "/bin/sh".
Not sure what "break" means here, but the "env" shebang should just work
in distros that follow (or try to) POSIX. AFAIK, to be POSIX compliant,
you need to provide an "env" for command invocation [1]. The two
previous changes that I know of [2], [3] have not had any adverse
effects that I'm aware of.

> How is the environment that runs scripts/archive-source.sh
> different, and why should it be fixed in scripts/archive-source.sh?
It is different from the other files in that I hit the issue only on
that file. I'm happy to provide a treewide change if it would make more
sense.

> 
> These are genuine questions - it would help if the commit message explained
> those...
I can weave the explanation into the commit for V2.

> In fact, what is a nix overlay and why would you use
> scripts/archive-source.sh to prepare one? :)
I now realize that mentioning a nix overlay in the commit message might
be to far from a general context. How about if I change the commit
message of the second patch to:

```
  The archive-source script appends everything directly on the tar root
  making it unsuitable for cases where a different directory structure
  is expected, like during nix packages customization.

  Add a "-d|--dir" argument that places all files under a custom root.
  Behaviour is unchanged when the argument is not given.
```

To answer the question of what an nix overlay is: It is a way of
customizing nix packages. I use it to override specific elements from a
nix package and replace them with my stuff [4]. I am working with the qemu
package and am overriding the qemu tarbal.

Best

[1] https://pubs.opengroup.org/onlinepubs/9799919799/utilities/env.html
[2] https://lore.kernel.org/all/20250122-jag-nix-ify-v1-1-addb3170f93c@kernel.org/
[3] https://lore.kernel.org/all/20240817215025.161628-1-kent.overstreet@linux.dev/
[4] https://nixos.wiki/wiki/Overlays
-- 

Joel Granados

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]