[RFC PATCH v1] tools: update-subtree.sh improve user experience

[RFC PATCH v1] tools: update-subtree.sh improve user experience

[E Shattow]

Makeover of the script:
- Usage now explains at each step what is expected from the user.
- Default branch name is no longer hard-coded and is queried when omitted.
- Additional operations 'update' and 'log' close the logic gap between
  expecting the user to provide a commit id and where from this commit id
  should be obtained.
- Use 'set -x' context to show git invocation where appropriate
- Sort options for usage output
- 'pick' operation detects merge commit id and displays suggested commits

Additionally the LwIP remote is now using the Savannah mirror system:

https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html

Signed-off-by: E Shattow <e@freeshell.de>
---

TODO v1:

- 'update dts <tag>' should attempt to substitute tag with the branch containing the suspected tag.  How to do this?

- 'log dts <tag>' should attempt to substitute tag with the branch containing the suspected tag.  How to do this?

- list branch names where branch name given is invalid or possibly as an info operation i.e. "see info operation for list of tags and/or/branches".  How to do this?

- 'pull dts invalid' takes awhile to fail, maybe pre-check if the tag exists before doing the pull action but only if that would be faster to fail. How to do this?

- behavior of operations when disconnected from network is not well-characterized, yet.

I think this script can add value by making it give helpful output and guide the user to a successful outcome. In most situations it should not be necessary to read documentation to use the script properly.

-E

---
 tools/update-subtree.sh | 187 +++++++++++++++++++++++++++++-----------
 1 file changed, 138 insertions(+), 49 deletions(-)

diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
index 536b3318573..e0c59330f16 100755
--- a/tools/update-subtree.sh
+++ b/tools/update-subtree.sh
@@ -1,42 +1,39 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0+
 #
-# Copyright (c) 2024 Linaro Limited
+# update-subtree.sh: Pull changes from subtree repo into U-Boot
 #
-# Usage: from the top level U-Boot source tree, run:
-# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
-# Or:
-# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
+# Copyright (c) 2024 Linaro Limited
 #
-# The script will pull changes from subtree repo into U-Boot.
-# It will automatically create a squash/merge commit listing the commits
-# imported.
-
-set -e
-
-print_usage() {
-    echo "usage: $0 <op> <subtree-name> <ref>"
-    echo "  <op>           pull or pick"
-    echo "  <subtree-name> mbedtls or dts or lwip"
-    echo "  <ref>          release tag [pull] or commit id [pick]"
-}
-
-if [ $# -ne 3 ]; then
-    print_usage
-    exit 1
-fi
 
 op=$1
 subtree_name=$2
 ref=$3
 
+print_usage() {
+    echo "usage: $0 <op> <subtree> [ref]"
+    echo "    op        update    fetch <subtree> for [ref] branch"
+    echo "                        add remote <subtree> as needed"
+    echo "                        query default branch from remote if omitted"
+    echo "              log       view log for <subtree>/[ref] branch"
+    echo "                        add remote <subtree> as needed"
+    echo "                        query default branch from remote if omitted"
+    echo "              pick      create cherry-pick commit into U-Boot with"
+    echo "                        <subtree> for <ref> commit-id"
+    echo "              pull      create squash/merge commit into U-Boot with"
+    echo "                        <subtree> for <ref> release-tag"
+    echo "                        add remote <subtree> as needed"
+    echo ""
+    echo "    subtree   dts       device tree definitions extracted from the"
+    echo "                        Linux kernel source tree"
+    echo "              lwip      A Lightweight TCPIP stack"
+    echo "              mbedtls   C library that implements cryptographic"
+    echo "                        primatives, X.509 certificate manipulation"
+    echo "                        and the SSL/TLS and DTLS protocols"
+}
+
 set_params() {
     case "$subtree_name" in
-        mbedtls)
-            path=lib/mbedtls/external/mbedtls
-            repo_url=https://github.com/Mbed-TLS/mbedtls.git
-            remote_name="mbedtls_upstream"
-            ;;
         dts)
             path=dts/upstream
             repo_url=https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
@@ -44,9 +41,14 @@ set_params() {
             ;;
         lwip)
             path=lib/lwip/lwip
-            repo_url=https://git.savannah.gnu.org/git/lwip.git
+            repo_url=https://https.git.savannah.gnu.org/git/lwip.git
             remote_name="lwip_upstream"
             ;;
+        mbedtls)
+            path=lib/mbedtls/external/mbedtls
+            repo_url=https://github.com/Mbed-TLS/mbedtls.git
+            remote_name="mbedtls_upstream"
+            ;;
         *)
             echo "Invalid subtree name: $subtree_name"
             print_usage
@@ -54,32 +56,119 @@ set_params() {
     esac
 }
 
-set_params
-
-merge_commit_msg=$(cat << EOF
-Subtree merge tag '$ref' of $subtree_name repo [1] into $path
+set_remote() {
+    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
+        (set -x
+            git remote add $remote_name $repo_url
+        )
+    fi
+}
 
-[1] $repo_url
-EOF
-)
+set_branch() {
+    if [ -z "$ref" ]; then
+        echo -n "Branch name omitted, query remote for default branch name... "
+        ref=$(git remote show $remote_name | grep -Po '(?<=HEAD branch:\s).*')
+        if [ $? -eq 0 ]; then
+            echo "$ref"
+        else
+            echo "Failed to query."
+            exit 1
+        fi
+    fi
+}
 
-remote_add_and_fetch() {
-    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
-        echo "Warning: Script automatically adds new git remote via:"
-        echo "    git remote add $remote_name \\"
-        echo "        $repo_url"
-        git remote add $remote_name $repo_url
+check_branch() {
+    (set -x
+        git show-ref --quiet "$remote_name"/"$ref"
+    )
+    if [ $? -ne 0 ] ; then
+        echo "TODO: lookup remote to determine if branch exists there"
+        echo "No such branch $remote_name/$ref"
+        echo "TODO: resolve if $ref is a tag and what branch it is from"
+        echo "Did you run \"$0 update $subtree_name $ref\" ?"
+        exit 1
     fi
-    git fetch $remote_name master
 }
 
-if [ "$op" = "pull" ]; then
-    remote_add_and_fetch
-    git subtree pull --prefix $path $remote_name "$ref" --squash -m "$merge_commit_msg"
-elif [ "$op" = "pick" ]; then
-    remote_add_and_fetch
-    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
-else
+if [ $# -lt 1 ]; then
     print_usage
     exit 1
 fi
+
+case "$op" in
+    update)
+        if [ $# -lt 2 ]; then
+            print_usage
+            exit 1
+        fi
+        set_params
+        set_remote
+        set_branch
+        (set -x
+            git fetch $remote_name $ref
+        )
+        ;;
+    log)
+        if [ $# -lt 2 ]; then
+            print_usage
+            exit 1
+        fi
+        set_params
+        set_remote
+        set_branch
+        check_branch
+        (set -x
+            git log $remote_name/$ref
+        )
+        ;;
+    pick)
+        if [ $# -lt 3 ]; then
+            print_usage
+            exit 1
+        fi
+        set_params
+        is_merge=$(set -x
+            git rev-list --no-walk --count --merges "$ref" 2>/dev/null
+        )
+        if [ $? -ne 0 ]; then
+            echo "Merge commit detection in pick failed for $ref."
+            echo "TODO: resolve if $ref exists in remote to guide suggestion."
+            echo "Did you run \"$0 update $subtree_name\" ?"
+            exit 1
+        fi
+        if [ $(($is_merge)) -gt 0 ]; then
+            echo "Merge commits are not allowed for pick operation."
+            echo "Did you want to pick one of the following commit-id instead?"
+            (set -x
+                git log --oneline "$ref"^1.."$ref"^2
+            )
+            exit 1
+        fi
+        (set -x
+            git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
+        )
+        ;;
+    pull)
+        if [ $# -lt 3 ]; then
+            print_usage
+            exit 1
+        fi
+        set_params
+        set_remote
+        # HEREDOC with tabs indent and syntax to ignore tabs:
+        merge_msg=$(cat <<-EOF
+		Subtree merge tag '$ref' of $subtree_name repo [1] into $path
+		
+		[1] $repo_url
+		EOF
+        )
+        (set -x
+            git subtree pull --prefix $path $remote_name "$ref" --squash -m "$merge_msg"
+        )
+        ;;
+    *)
+        echo "Invalid op: $op"
+        print_usage
+        exit 1
+        ;;
+esac

base-commit: b1cc9a53d7ba8b85e77980417242420889273e86
-- 
2.50.0

Re: [RFC PATCH v1] tools: update-subtree.sh improve user experience

[Quentin Schulz]

Hi E,

On 10/15/25 10:38 AM, E Shattow wrote:
> Makeover of the script:
> - Usage now explains at each step what is expected from the user.
> - Default branch name is no longer hard-coded and is queried when omitted.
> - Additional operations 'update' and 'log' close the logic gap between
>    expecting the user to provide a commit id and where from this commit id
>    should be obtained.
> - Use 'set -x' context to show git invocation where appropriate
> - Sort options for usage output
> - 'pick' operation detects merge commit id and displays suggested commits
> 
> Additionally the LwIP remote is now using the Savannah mirror system:
> 
> https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html
> 

Please try to split changes into multiple commits. There's a lot of 
unnecessary noise and it makes difficult to review your patch otherwise.

Here I can identify:

- Cleaning up the comments at the top
- Removing set -e (why?)
- Moving print_usage somewhere else
- Changing the output of print_usage
- Reordering cases in the switch..case for the subtree-name
- Changing URL of GNU mirror (which very much looks like a typo as it 
doesn't seem to be what the link pasted above says the new URLs should 
look like, albeit working somehow)
- Adding new features (each its own commit please)

You also do not explain what your pain points were when using this 
script. This would be useful information now for reviewing and for later 
if/when this gets merged so we know what kind of confusion happened in 
the past so that potential future reworks don't reintroduce the same 
problems.

If I remember correctly, the biggest issue was that a pull dts was 
attempted instead of multiple cherry-picks? I believe nobody except Tom 
(for DTS) and whoever are the maintainers of lwip/mbedtls for the 
respective projects should be running the pull option. The diff is too 
big for DTS to be meaningful so for now Tom is essentially pushing to 
the branch and then notifying people on the ML that this was done and 
that the new base is available in master, c.f. 
https://lore.kernel.org/u-boot/20251008220903.GM298503@bill-the-cat/. 
Maybe we should also be conveying this better (including 
https://docs.u-boot.org/en/latest/develop/devicetree/control.html#resyncing-with-devicetree-rebasing).

Cheers,
Quentin

Re: [RFC PATCH v1] tools: update-subtree.sh improve user experience

[E Shattow]

Hi Quentin, thanks for the review.

On 10/20/25 09:01, Quentin Schulz wrote:
> Hi E,
> 
> On 10/15/25 10:38 AM, E Shattow wrote:
>> Makeover of the script:
>> - Usage now explains at each step what is expected from the user.
>> - Default branch name is no longer hard-coded and is queried when
>> omitted.
>> - Additional operations 'update' and 'log' close the logic gap between
>>    expecting the user to provide a commit id and where from this
>> commit id
>>    should be obtained.
>> - Use 'set -x' context to show git invocation where appropriate
>> - Sort options for usage output
>> - 'pick' operation detects merge commit id and displays suggested commits
>>
>> Additionally the LwIP remote is now using the Savannah mirror system:
>>
>> https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html
>>
> 
> Please try to split changes into multiple commits. There's a lot of
> unnecessary noise and it makes difficult to review your patch otherwise.

Yes this is incomprehensible as a diff. It should be reviewed as a
replacement for the existing script (I do not know if that is possible
with git format-patch, ?) I'll try to spoon feed it in a series of more
easily reviewable patches to get where I want to.

> 
> Here I can identify:
> 
> - Cleaning up the comments at the top
> - Removing set -e (why?)

set -e fails the script immediately which prevents giving the user some
facts about an error

> - Moving print_usage somewhere else
> - Changing the output of print_usage
> - Reordering cases in the switch..case for the subtree-name
> - Changing URL of GNU mirror (which very much looks like a typo as it
> doesn't seem to be what the link pasted above says the new URLs should
> look like, albeit working somehow)

I got this information from the person working on it for FSF. I will
review to see if it is better documented somewhere, but this is their
suggestion both the load balance URL to use and the public announcement
reference.

> - Adding new features (each its own commit please)
> 
> You also do not explain what your pain points were when using this
> script. This would be useful information now for reviewing and for later
> if/when this gets merged so we know what kind of confusion happened in
> the past so that potential future reworks don't reintroduce the same
> problems.

I was never able to get this to work by the description in build
documentation. I have a choice to do a lengthy explanation in
documentation to improve that or just make the command more
user-friendly - I will try making it better before giving up and
increasing the word count of the documentation.

> 
> If I remember correctly, the biggest issue was that a pull dts was
> attempted instead of multiple cherry-picks? I believe nobody except Tom
> (for DTS) and whoever are the maintainers of lwip/mbedtls for the
> respective projects should be running the pull option. The diff is too

Yes, this is not clear from documentation presently. The tool should not
punish the user for misuse or ignorance about commit ids before getting
to the moment where they can begin to learn about such things.

> big for DTS to be meaningful so for now Tom is essentially pushing to
> the branch and then notifying people on the ML that this was done and
> that the new base is available in master, c.f. https://lore.kernel.org/
> u-boot/20251008220903.GM298503@bill-the-cat/. Maybe we should also be
> conveying this better (including https://docs.u-boot.org/en/latest/
> develop/devicetree/control.html#resyncing-with-devicetree-rebasing).
> 
> Cheers,
> Quentin

Yes I also plan to reduce the documentation to what is helpful only to
start using the tool and with changes to the tool it will be easier for
project maintainer and anyone else to use. More information does not
make the tool any better or easier to use and for what it does now could
be better to just explain the git commands. I will first attempt to
improve the tool to be more helpful when the user does not already know
what git commands or commit ids are involved.

-E

Re: [RFC PATCH v1] tools: update-subtree.sh improve user experience

[Quentin Schulz]

Hi E,

On 10/27/25 7:56 AM, E Shattow wrote:
> Hi Quentin, thanks for the review.
> 
> On 10/20/25 09:01, Quentin Schulz wrote:
>> Hi E,
>>
>> On 10/15/25 10:38 AM, E Shattow wrote:
>>> Makeover of the script:
>>> - Usage now explains at each step what is expected from the user.
>>> - Default branch name is no longer hard-coded and is queried when
>>> omitted.
>>> - Additional operations 'update' and 'log' close the logic gap between
>>>     expecting the user to provide a commit id and where from this
>>> commit id
>>>     should be obtained.
>>> - Use 'set -x' context to show git invocation where appropriate
>>> - Sort options for usage output
>>> - 'pick' operation detects merge commit id and displays suggested commits
>>>
>>> Additionally the LwIP remote is now using the Savannah mirror system:
>>>
>>> https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html
>>>
>>
>> Please try to split changes into multiple commits. There's a lot of
>> unnecessary noise and it makes difficult to review your patch otherwise.
> 
> Yes this is incomprehensible as a diff. It should be reviewed as a
> replacement for the existing script (I do not know if that is possible
> with git format-patch, ?) I'll try to spoon feed it in a series of more
> easily reviewable patches to get where I want to.
> 

Create two patches, one which deletes the current file and the second 
which adds your new changes. Then in the cover letter say that they 
should be squashed before being applied.

Note that this is very unusual, I believe projects typically would 
rather see small incremental changes than starting from scratch.

>>
>> Here I can identify:
>>
>> - Cleaning up the comments at the top
>> - Removing set -e (why?)
> 
> set -e fails the script immediately which prevents giving the user some
> facts about an error
> 

It fails the script only if you don't do proper error handling. Which we 
should be doing.

If error handling is missing and set -e is set, the script will stop 
where it fails.
If error handling is missing and set -e is not set, the script will 
happily continue and fail in some random places later on.

I prefer the first one. Personal preference though (and likely of the 
author(s) of the original script).

>> - Moving print_usage somewhere else
>> - Changing the output of print_usage
>> - Reordering cases in the switch..case for the subtree-name
>> - Changing URL of GNU mirror (which very much looks like a typo as it
>> doesn't seem to be what the link pasted above says the new URLs should
>> look like, albeit working somehow)
> 
> I got this information from the person working on it for FSF. I will
> review to see if it is better documented somewhere, but this is their
> suggestion both the load balance URL to use and the public announcement
> reference.
> 

https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html 
says

     https://gitweb.git.savannah.gnu.org/gitweb/
     https://cgit.git.savannah.gnu.org/cgit/

not

https://https.git.savannah.gnu.org/git/lwip.git

I needed to go to the cgit page of the project to find the clone 
instructions where
https://https.git.savannah.gnu.org/git/lwip.git is mentioned, c.f. 
https://cgit.git.savannah.gnu.org/cgit/lwip.git/

or also on gitweb:
https://gitweb.git.savannah.gnu.org/gitweb/?p=lwip.git&view=view+git+repository

Please mention this instead of simply linking to the mailing list 
archive which doesn't mention that at all.

>> - Adding new features (each its own commit please)
>>
>> You also do not explain what your pain points were when using this
>> script. This would be useful information now for reviewing and for later
>> if/when this gets merged so we know what kind of confusion happened in
>> the past so that potential future reworks don't reintroduce the same
>> problems.
> 
> I was never able to get this to work by the description in build
> documentation. I have a choice to do a lengthy explanation in
> documentation to improve that or just make the command more
> user-friendly - I will try making it better before giving up and
> increasing the word count of the documentation.
> 

You should justify your patch. Why is it better than the current 
implementation? How is that "improving the user experience"? It improves 
yours, but maybe it's worse for everybody else now? What was confusing?

I'm not asking to write documentation, I'm asking for the reasons you 
wrote this patch so we know better than reintroducing these issues later 
on or for other scripts. With this patch here, the project cannot learn 
what it did wrong and cannot improve.

>>
>> If I remember correctly, the biggest issue was that a pull dts was
>> attempted instead of multiple cherry-picks? I believe nobody except Tom
>> (for DTS) and whoever are the maintainers of lwip/mbedtls for the
>> respective projects should be running the pull option. The diff is too
> 
> Yes, this is not clear from documentation presently. The tool should not

Agreed.

> punish the user for misuse or ignorance about commit ids before getting
> to the moment where they can begin to learn about such things.
> 

 From that, I assume you mean that you would like the tool to catch 
whether the attempted cherry-pick is a merge commit and error in that case?

Cheers,
Quentin