[PATCH v2] tools: Add script to update git subtree projects

[PATCH v2] tools: Add script to update git subtree projects

[Raymond Mao]

Recently we are introducing multiple git subtree projects and
it is the right time to have a universal script to update
various subtrees and replace the dts/update-dts-subtree.sh.

update-subtree.sh is a wrapper of git subtree commands.

Usage: From U-Boot top directory,
run
$ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
for pulling a tag from the upstream.
Or run
$ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
for cherry-pick a commit from the upstream.

Currently <subtree-name> supports dts, mbedtls and lwip.

Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
---
Changes in v2
- Refactored the script.
- Update the control doc.

 doc/develop/devicetree/control.rst |  8 +--
 dts/update-dts-subtree.sh          | 48 -----------------
 tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
 3 files changed, 90 insertions(+), 52 deletions(-)
 delete mode 100755 dts/update-dts-subtree.sh
 create mode 100755 tools/update-subtree.sh

diff --git a/doc/develop/devicetree/control.rst b/doc/develop/devicetree/control.rst
index ca4fb0b5b10..211f7e4909c 100644
--- a/doc/develop/devicetree/control.rst
+++ b/doc/develop/devicetree/control.rst
@@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the devicetree-rebasing repo whenever
 the next branch opens (refer: :doc:`../release_cycle`) with the latest mainline
 Linux kernel release. To sync the `dts/upstream/` subtree, run::
 
-    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
+    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
 
 If required it is also possible to cherry-pick fixes from the
 devicetree-rebasing repository prior to next sync, usage::
 
-    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
+    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
 
 
 Configuration
@@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received devicetree source file for your
 newly added board support then one option is that you can add the corresponding
 devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that add `#
 CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
-prompted by Kconfig. Another option is that you can use use the "pick" option of
-`dts/update-dts-subtree.sh` mentioned above to bring in the commits that you
+prompted by Kconfig. Another option is that you can use the "pick" option of
+`tools/update-subtree.sh` mentioned above to bring in the commits that you
 need.
 
 This should include your CPU or SoC's devicetree file. On top of that any U-Boot
diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
deleted file mode 100755
index a57b78a41d3..00000000000
--- a/dts/update-dts-subtree.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0+
-#
-# Copyright 2024 Linaro Ltd.
-#
-# Usage: from the top level U-Boot source tree, run:
-# $ ./dts/update-dts-subtree.sh pull <release-tag>
-# $ ./dts/update-dts-subtree.sh pick <commit-id>
-#
-# The script will pull changes from devicetree-rebasing repo into U-Boot
-# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
-# automatically create a squash/merge commit listing the commits imported.
-
-set -e
-
-merge_commit_msg=$(cat << EOF
-Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
-
-[1] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
-EOF
-)
-
-remote_add_and_fetch() {
-    if ! git remote get-url devicetree-rebasing 2>/dev/null
-    then
-        echo "Warning: Script automatically adds new git remote via:"
-        echo "    git remote add devicetree-rebasing \\"
-        echo "        https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git"
-        git remote add devicetree-rebasing \
-            https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
-    fi
-    git fetch devicetree-rebasing master
-}
-
-if [ "$1" = "pull" ]
-then
-    remote_add_and_fetch
-    git subtree pull --prefix dts/upstream devicetree-rebasing \
-        "$2" --squash -m "${merge_commit_msg}"
-elif [ "$1" = "pick" ]
-then
-    remote_add_and_fetch
-    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
-else
-    echo "usage: $0 <op> <ref>"
-    echo "  <op>     pull or pick"
-    echo "  <ref>    release tag [pull] or commit id [pick]"
-fi
diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
new file mode 100755
index 00000000000..5946aa1a9e2
--- /dev/null
+++ b/tools/update-subtree.sh
@@ -0,0 +1,86 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0+
+#
+# Copyright (c) 2024 Linaro Limited
+# Author: Raymond Mao <raymond.mao@linaro.org>
+#
+# Usage: from the top level U-Boot source tree, run:
+# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
+# Or:
+# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
+#
+# The script will pull changes from subtree repo into U-Boot.
+# It will automatically create a squash/merge commit listing the commits
+# imported.
+
+set -e
+
+print_usage() {
+    echo "usage: $0 <op> <subtree-name> <ref>"
+    echo "  <op>           pull or pick"
+    echo "  <subtree-name> mbedtls or dts or lwip"
+    echo "  <ref>          release tag [pull] or commit id [pick]"
+}
+
+if [ $# -ne 3 ]; then
+    print_usage
+    exit 1
+fi
+
+op=$1
+subtree_name=$2
+ref=$3
+
+set_params() {
+    case "$subtree_name" in
+        mbedtls)
+            path=lib/mbedtls/external/mbedtls
+            repo_url=https://github.com/Mbed-TLS/mbedtls.git
+            remote_name="mbedtls_upstream"
+            ;;
+        dts)
+            path=dts/upstream
+            repo_url=https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
+            remote_name="devicetree-rebasing"
+            ;;
+        lwip)
+            path=lib/lwip/lwip
+            repo_url=https://git.savannah.gnu.org/git/lwip.git
+            remote_name="lwip_upstream"
+            ;;
+        *)
+            echo "Invalid subtree name: $subtree_name"
+            print_usage
+            exit 1
+    esac
+}
+
+set_params
+
+merge_commit_msg=$(cat << EOF
+Subtree merge tag '$ref' of $subtree_name repo [1] into $path
+
+[1] $repo_url
+EOF
+)
+
+remote_add_and_fetch() {
+    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
+        echo "Warning: Script automatically adds new git remote via:"
+        echo "    git remote add $remote_name \\"
+        echo "        $repo_url"
+        git remote add $remote_name $repo_url
+    fi
+    git fetch $remote_name master
+}
+
+if [ "$op" = "pull" ]; then
+    remote_add_and_fetch
+    git subtree pull --prefix $path $remote_name "$ref" --squash -m "$merge_commit_msg"
+elif [ "$op" = "pick" ]; then
+    remote_add_and_fetch
+    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
+else
+    print_usage
+    exit 1
+fi
-- 
2.25.1

Re: [PATCH v2] tools: Add script to update git subtree projects

[Jerome Forissier]

On 7/22/24 21:30, Raymond Mao wrote:
> Recently we are introducing multiple git subtree projects and
> it is the right time to have a universal script to update
> various subtrees and replace the dts/update-dts-subtree.sh.
> 
> update-subtree.sh is a wrapper of git subtree commands.
> 
> Usage: From U-Boot top directory,
> run
> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> for pulling a tag from the upstream.
> Or run
> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> for cherry-pick a commit from the upstream.
> 
> Currently <subtree-name> supports dts, mbedtls and lwip.
> 
> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> ---
> Changes in v2
> - Refactored the script.
> - Update the control doc.
> 
>  doc/develop/devicetree/control.rst |  8 +--
>  dts/update-dts-subtree.sh          | 48 -----------------
>  tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>  3 files changed, 90 insertions(+), 52 deletions(-)
>  delete mode 100755 dts/update-dts-subtree.sh
>  create mode 100755 tools/update-subtree.sh

Acked-by: Jerome Forissier <jerome.forissier@linaro.org>

-- 
Jerome

Re: [PATCH v2] tools: Add script to update git subtree projects

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Mon, Jul 22, 2024 at 12:30:50PM -0700, Raymond Mao wrote:

> Recently we are introducing multiple git subtree projects and
> it is the right time to have a universal script to update
> various subtrees and replace the dts/update-dts-subtree.sh.
> 
> update-subtree.sh is a wrapper of git subtree commands.
> 
> Usage: From U-Boot top directory,
> run
> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> for pulling a tag from the upstream.
> Or run
> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> for cherry-pick a commit from the upstream.
> 
> Currently <subtree-name> supports dts, mbedtls and lwip.
> 
> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>

Reviewed-by: Tom Rini <trini@konsulko.com>

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH v2] tools: Add script to update git subtree projects

[Ilias Apalodimas]

On Mon, 22 Jul 2024 at 22:31, Raymond Mao <raymond.mao@linaro.org> wrote:
>
> Recently we are introducing multiple git subtree projects and
> it is the right time to have a universal script to update
> various subtrees and replace the dts/update-dts-subtree.sh.
>
> update-subtree.sh is a wrapper of git subtree commands.
>
> Usage: From U-Boot top directory,
> run
> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> for pulling a tag from the upstream.
> Or run
> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> for cherry-pick a commit from the upstream.
>
> Currently <subtree-name> supports dts, mbedtls and lwip.
>
> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> ---
> Changes in v2
> - Refactored the script.
> - Update the control doc.
>
>  doc/develop/devicetree/control.rst |  8 +--
>  dts/update-dts-subtree.sh          | 48 -----------------
>  tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>  3 files changed, 90 insertions(+), 52 deletions(-)
>  delete mode 100755 dts/update-dts-subtree.sh
>  create mode 100755 tools/update-subtree.sh
>
> diff --git a/doc/develop/devicetree/control.rst b/doc/develop/devicetree/control.rst
> index ca4fb0b5b10..211f7e4909c 100644
> --- a/doc/develop/devicetree/control.rst
> +++ b/doc/develop/devicetree/control.rst
> @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the devicetree-rebasing repo whenever
>  the next branch opens (refer: :doc:`../release_cycle`) with the latest mainline
>  Linux kernel release. To sync the `dts/upstream/` subtree, run::
>
> -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
> +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
>
>  If required it is also possible to cherry-pick fixes from the
>  devicetree-rebasing repository prior to next sync, usage::
>
> -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
> +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
>
>
>  Configuration
> @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received devicetree source file for your
>  newly added board support then one option is that you can add the corresponding
>  devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that add `#
>  CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
> -prompted by Kconfig. Another option is that you can use use the "pick" option of
> -`dts/update-dts-subtree.sh` mentioned above to bring in the commits that you
> +prompted by Kconfig. Another option is that you can use the "pick" option of
> +`tools/update-subtree.sh` mentioned above to bring in the commits that you
>  need.
>
>  This should include your CPU or SoC's devicetree file. On top of that any U-Boot
> diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
> deleted file mode 100755
> index a57b78a41d3..00000000000
> --- a/dts/update-dts-subtree.sh
> +++ /dev/null
> @@ -1,48 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0+
> -#
> -# Copyright 2024 Linaro Ltd.
> -#
> -# Usage: from the top level U-Boot source tree, run:
> -# $ ./dts/update-dts-subtree.sh pull <release-tag>
> -# $ ./dts/update-dts-subtree.sh pick <commit-id>
> -#
> -# The script will pull changes from devicetree-rebasing repo into U-Boot
> -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
> -# automatically create a squash/merge commit listing the commits imported.
> -
> -set -e
> -
> -merge_commit_msg=$(cat << EOF
> -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
> -
> -[1] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
> -EOF
> -)
> -
> -remote_add_and_fetch() {
> -    if ! git remote get-url devicetree-rebasing 2>/dev/null
> -    then
> -        echo "Warning: Script automatically adds new git remote via:"
> -        echo "    git remote add devicetree-rebasing \\"
> -        echo "        https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git"
> -        git remote add devicetree-rebasing \
> -            https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> -    fi
> -    git fetch devicetree-rebasing master
> -}
> -
> -if [ "$1" = "pull" ]
> -then
> -    remote_add_and_fetch
> -    git subtree pull --prefix dts/upstream devicetree-rebasing \
> -        "$2" --squash -m "${merge_commit_msg}"
> -elif [ "$1" = "pick" ]
> -then
> -    remote_add_and_fetch
> -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
> -else
> -    echo "usage: $0 <op> <ref>"
> -    echo "  <op>     pull or pick"
> -    echo "  <ref>    release tag [pull] or commit id [pick]"
> -fi
> diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
> new file mode 100755
> index 00000000000..5946aa1a9e2
> --- /dev/null
> +++ b/tools/update-subtree.sh
> @@ -0,0 +1,86 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Copyright (c) 2024 Linaro Limited
> +# Author: Raymond Mao <raymond.mao@linaro.org>
> +#
> +# Usage: from the top level U-Boot source tree, run:
> +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> +# Or:
> +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> +#
> +# The script will pull changes from subtree repo into U-Boot.
> +# It will automatically create a squash/merge commit listing the commits
> +# imported.
> +
> +set -e
> +
> +print_usage() {
> +    echo "usage: $0 <op> <subtree-name> <ref>"
> +    echo "  <op>           pull or pick"
> +    echo "  <subtree-name> mbedtls or dts or lwip"
> +    echo "  <ref>          release tag [pull] or commit id [pick]"
> +}
> +
> +if [ $# -ne 3 ]; then
> +    print_usage
> +    exit 1
> +fi
> +
> +op=$1
> +subtree_name=$2
> +ref=$3
> +
> +set_params() {
> +    case "$subtree_name" in
> +        mbedtls)
> +            path=lib/mbedtls/external/mbedtls
> +            repo_url=https://github.com/Mbed-TLS/mbedtls.git
> +            remote_name="mbedtls_upstream"
> +            ;;
> +        dts)
> +            path=dts/upstream
> +            repo_url=https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> +            remote_name="devicetree-rebasing"
> +            ;;
> +        lwip)
> +            path=lib/lwip/lwip
> +            repo_url=https://git.savannah.gnu.org/git/lwip.git
> +            remote_name="lwip_upstream"
> +            ;;
> +        *)
> +            echo "Invalid subtree name: $subtree_name"
> +            print_usage
> +            exit 1
> +    esac
> +}
> +
> +set_params
> +
> +merge_commit_msg=$(cat << EOF
> +Subtree merge tag '$ref' of $subtree_name repo [1] into $path
> +
> +[1] $repo_url
> +EOF
> +)
> +
> +remote_add_and_fetch() {
> +    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
> +        echo "Warning: Script automatically adds new git remote via:"
> +        echo "    git remote add $remote_name \\"
> +        echo "        $repo_url"
> +        git remote add $remote_name $repo_url
> +    fi
> +    git fetch $remote_name master
> +}
> +
> +if [ "$op" = "pull" ]; then
> +    remote_add_and_fetch
> +    git subtree pull --prefix $path $remote_name "$ref" --squash -m "$merge_commit_msg"
> +elif [ "$op" = "pick" ]; then
> +    remote_add_and_fetch
> +    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
> +else
> +    print_usage
> +    exit 1
> +fi
> --
> 2.25.1
>


Acked-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>

Re: [PATCH v2] tools: Add script to update git subtree projects

[Quentin Schulz]

Hi Raymond,

On 7/22/24 9:30 PM, Raymond Mao wrote:
> Recently we are introducing multiple git subtree projects and
> it is the right time to have a universal script to update
> various subtrees and replace the dts/update-dts-subtree.sh.
> 
> update-subtree.sh is a wrapper of git subtree commands.
> 
> Usage: From U-Boot top directory,
> run
> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> for pulling a tag from the upstream.
> Or run
> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> for cherry-pick a commit from the upstream.
> 
> Currently <subtree-name> supports dts, mbedtls and lwip.
> 
> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> ---
> Changes in v2
> - Refactored the script.
> - Update the control doc.
> 
>   doc/develop/devicetree/control.rst |  8 +--
>   dts/update-dts-subtree.sh          | 48 -----------------
>   tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>   3 files changed, 90 insertions(+), 52 deletions(-)
>   delete mode 100755 dts/update-dts-subtree.sh
>   create mode 100755 tools/update-subtree.sh
> 
> diff --git a/doc/develop/devicetree/control.rst b/doc/develop/devicetree/control.rst
> index ca4fb0b5b10..211f7e4909c 100644
> --- a/doc/develop/devicetree/control.rst
> +++ b/doc/develop/devicetree/control.rst
> @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the devicetree-rebasing repo whenever
>   the next branch opens (refer: :doc:`../release_cycle`) with the latest mainline
>   Linux kernel release. To sync the `dts/upstream/` subtree, run::
>   
> -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
> +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
>   
>   If required it is also possible to cherry-pick fixes from the
>   devicetree-rebasing repository prior to next sync, usage::
>   
> -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
> +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
>   
>   
>   Configuration
> @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received devicetree source file for your
>   newly added board support then one option is that you can add the corresponding
>   devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that add `#
>   CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
> -prompted by Kconfig. Another option is that you can use use the "pick" option of
> -`dts/update-dts-subtree.sh` mentioned above to bring in the commits that you
> +prompted by Kconfig. Another option is that you can use the "pick" option of
> +`tools/update-subtree.sh` mentioned above to bring in the commits that you
>   need.
>   
>   This should include your CPU or SoC's devicetree file. On top of that any U-Boot
> diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
> deleted file mode 100755
> index a57b78a41d3..00000000000
> --- a/dts/update-dts-subtree.sh
> +++ /dev/null
> @@ -1,48 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0+
> -#
> -# Copyright 2024 Linaro Ltd.
> -#
> -# Usage: from the top level U-Boot source tree, run:
> -# $ ./dts/update-dts-subtree.sh pull <release-tag>
> -# $ ./dts/update-dts-subtree.sh pick <commit-id>
> -#
> -# The script will pull changes from devicetree-rebasing repo into U-Boot
> -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
> -# automatically create a squash/merge commit listing the commits imported.
> -
> -set -e
> -
> -merge_commit_msg=$(cat << EOF
> -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
> -
> -[1] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
> -EOF
> -)
> -
> -remote_add_and_fetch() {
> -    if ! git remote get-url devicetree-rebasing 2>/dev/null
> -    then
> -        echo "Warning: Script automatically adds new git remote via:"
> -        echo "    git remote add devicetree-rebasing \\"
> -        echo "        https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git"
> -        git remote add devicetree-rebasing \
> -            https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> -    fi
> -    git fetch devicetree-rebasing master
> -}
> -
> -if [ "$1" = "pull" ]
> -then
> -    remote_add_and_fetch
> -    git subtree pull --prefix dts/upstream devicetree-rebasing \
> -        "$2" --squash -m "${merge_commit_msg}"
> -elif [ "$1" = "pick" ]
> -then
> -    remote_add_and_fetch
> -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
> -else
> -    echo "usage: $0 <op> <ref>"
> -    echo "  <op>     pull or pick"
> -    echo "  <ref>    release tag [pull] or commit id [pick]"
> -fi
> diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
> new file mode 100755
> index 00000000000..5946aa1a9e2
> --- /dev/null
> +++ b/tools/update-subtree.sh

We have scripts/ and tools/ directories, this being a shell script I 
would have intuitively added it to scripts/. So my question is, what's 
the difference between those two directories :) ?

> @@ -0,0 +1,86 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Copyright (c) 2024 Linaro Limited
> +# Author: Raymond Mao <raymond.mao@linaro.org>
> +#
> +# Usage: from the top level U-Boot source tree, run:
> +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> +# Or:
> +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> +#
> +# The script will pull changes from subtree repo into U-Boot.
> +# It will automatically create a squash/merge commit listing the commits
> +# imported.
> +
> +set -e
> +
> +print_usage() {
> +    echo "usage: $0 <op> <subtree-name> <ref>"
> +    echo "  <op>           pull or pick"

I think pulls are supposed to be done and sent to the ML by maintainers 
only? At least that's what we're supposed to do for the dts/upstream/ 
directory AFAIR. Is it the same for mbedtls and lwip? If so, maybe 
explicit this in the usage that only maintainers are supposed to run 
this script with that option? And if you need to do it, ask the 
maintainers to do it for you? Does this make sense?

> +    echo "  <subtree-name> mbedtls or dts or lwip"
> +    echo "  <ref>          release tag [pull] or commit id [pick]"
> +}
> +
> +if [ $# -ne 3 ]; then
> +    print_usage
> +    exit 1
> +fi
> +
> +op=$1
> +subtree_name=$2
> +ref=$3
> +
> +set_params() {
> +    case "$subtree_name" in
> +        mbedtls)
> +            path=lib/mbedtls/external/mbedtls
> +            repo_url=https://github.com/Mbed-TLS/mbedtls.git
> +            remote_name="mbedtls_upstream"
> +            ;;
> +        dts)
> +            path=dts/upstream
> +            repo_url=https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> +            remote_name="devicetree-rebasing"
> +            ;;
> +        lwip)
> +            path=lib/lwip/lwip
> +            repo_url=https://git.savannah.gnu.org/git/lwip.git
> +            remote_name="lwip_upstream"
> +            ;;
> +        *)
> +            echo "Invalid subtree name: $subtree_name"
> +            print_usage
> +            exit 1
> +    esac
> +}
> +
> +set_params
> +
> +merge_commit_msg=$(cat << EOF
> +Subtree merge tag '$ref' of $subtree_name repo [1] into $path
> +

I would recommend adding at the very least the path of the tools used to 
generate this merge commit (so ./tools/update-subtree.sh or maybe just 
$0?) or maybe even the full command line that was used to do this merge?

> +[1] $repo_url
> +EOF
> +)
> +
> +remote_add_and_fetch() {
> +    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
> +        echo "Warning: Script automatically adds new git remote via:"
> +        echo "    git remote add $remote_name \\"
> +        echo "        $repo_url"
> +        git remote add $remote_name $repo_url
> +    fi
> +    git fetch $remote_name master
> +}
> +
> +if [ "$op" = "pull" ]; then
> +    remote_add_and_fetch
> +    git subtree pull --prefix $path $remote_name "$ref" --squash -m "$merge_commit_msg"
> +elif [ "$op" = "pick" ]; then
> +    remote_add_and_fetch
> +    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"

Should we add -s there too? (or --signoff)

Those are just questions, no blocker for me.

Cheers,
Quentin

Re: [PATCH v2] tools: Add script to update git subtree projects

[Raymond Mao]

Hi Quentin,

On Tue, 23 Jul 2024 at 03:16, Quentin Schulz <quentin.schulz@cherry.de>
wrote:

> Hi Raymond,
>
> On 7/22/24 9:30 PM, Raymond Mao wrote:
> > Recently we are introducing multiple git subtree projects and
> > it is the right time to have a universal script to update
> > various subtrees and replace the dts/update-dts-subtree.sh.
> >
> > update-subtree.sh is a wrapper of git subtree commands.
> >
> > Usage: From U-Boot top directory,
> > run
> > $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> > for pulling a tag from the upstream.
> > Or run
> > $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> > for cherry-pick a commit from the upstream.
> >
> > Currently <subtree-name> supports dts, mbedtls and lwip.
> >
> > Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> > ---
> > Changes in v2
> > - Refactored the script.
> > - Update the control doc.
> >
> >   doc/develop/devicetree/control.rst |  8 +--
> >   dts/update-dts-subtree.sh          | 48 -----------------
> >   tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
> >   3 files changed, 90 insertions(+), 52 deletions(-)
> >   delete mode 100755 dts/update-dts-subtree.sh
> >   create mode 100755 tools/update-subtree.sh
> >
> > diff --git a/doc/develop/devicetree/control.rst
> b/doc/develop/devicetree/control.rst
> > index ca4fb0b5b10..211f7e4909c 100644
> > --- a/doc/develop/devicetree/control.rst
> > +++ b/doc/develop/devicetree/control.rst
> > @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the
> devicetree-rebasing repo whenever
> >   the next branch opens (refer: :doc:`../release_cycle`) with the latest
> mainline
> >   Linux kernel release. To sync the `dts/upstream/` subtree, run::
> >
> > -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
> > +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
> >
> >   If required it is also possible to cherry-pick fixes from the
> >   devicetree-rebasing repository prior to next sync, usage::
> >
> > -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
> > +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
> >
> >
> >   Configuration
> > @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received
> devicetree source file for your
> >   newly added board support then one option is that you can add the
> corresponding
> >   devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that
> add `#
> >   CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>`
> when
> > -prompted by Kconfig. Another option is that you can use use the "pick"
> option of
> > -`dts/update-dts-subtree.sh` mentioned above to bring in the commits
> that you
> > +prompted by Kconfig. Another option is that you can use the "pick"
> option of
> > +`tools/update-subtree.sh` mentioned above to bring in the commits that
> you
> >   need.
> >
> >   This should include your CPU or SoC's devicetree file. On top of that
> any U-Boot
> > diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
> > deleted file mode 100755
> > index a57b78a41d3..00000000000
> > --- a/dts/update-dts-subtree.sh
> > +++ /dev/null
> > @@ -1,48 +0,0 @@
> > -#!/bin/sh
> > -# SPDX-License-Identifier: GPL-2.0+
> > -#
> > -# Copyright 2024 Linaro Ltd.
> > -#
> > -# Usage: from the top level U-Boot source tree, run:
> > -# $ ./dts/update-dts-subtree.sh pull <release-tag>
> > -# $ ./dts/update-dts-subtree.sh pick <commit-id>
> > -#
> > -# The script will pull changes from devicetree-rebasing repo into U-Boot
> > -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
> > -# automatically create a squash/merge commit listing the commits
> imported.
> > -
> > -set -e
> > -
> > -merge_commit_msg=$(cat << EOF
> > -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
> > -
> > -[1]
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
> > -EOF
> > -)
> > -
> > -remote_add_and_fetch() {
> > -    if ! git remote get-url devicetree-rebasing 2>/dev/null
> > -    then
> > -        echo "Warning: Script automatically adds new git remote via:"
> > -        echo "    git remote add devicetree-rebasing \\"
> > -        echo "
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> "
> > -        git remote add devicetree-rebasing \
> > -
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> > -    fi
> > -    git fetch devicetree-rebasing master
> > -}
> > -
> > -if [ "$1" = "pull" ]
> > -then
> > -    remote_add_and_fetch
> > -    git subtree pull --prefix dts/upstream devicetree-rebasing \
> > -        "$2" --squash -m "${merge_commit_msg}"
> > -elif [ "$1" = "pick" ]
> > -then
> > -    remote_add_and_fetch
> > -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
> > -else
> > -    echo "usage: $0 <op> <ref>"
> > -    echo "  <op>     pull or pick"
> > -    echo "  <ref>    release tag [pull] or commit id [pick]"
> > -fi
> > diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
> > new file mode 100755
> > index 00000000000..5946aa1a9e2
> > --- /dev/null
> > +++ b/tools/update-subtree.sh
>
> We have scripts/ and tools/ directories, this being a shell script I
> would have intuitively added it to scripts/. So my question is, what's
> the difference between those two directories :) ?
>
> Good question. Personally I think 'scripts' are for those scripts needed
by other components and 'tools' are for the general purpose utilities
(not limited to scripts). That is the reason I prefer 'tools'.

> @@ -0,0 +1,86 @@
> > +#!/bin/sh
> > +# SPDX-License-Identifier: GPL-2.0+
> > +#
> > +# Copyright (c) 2024 Linaro Limited
> > +# Author: Raymond Mao <raymond.mao@linaro.org>
> > +#
> > +# Usage: from the top level U-Boot source tree, run:
> > +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> > +# Or:
> > +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> > +#
> > +# The script will pull changes from subtree repo into U-Boot.
> > +# It will automatically create a squash/merge commit listing the commits
> > +# imported.
> > +
> > +set -e
> > +
> > +print_usage() {
> > +    echo "usage: $0 <op> <subtree-name> <ref>"
> > +    echo "  <op>           pull or pick"
>
> I think pulls are supposed to be done and sent to the ML by maintainers
> only? At least that's what we're supposed to do for the dts/upstream/
> directory AFAIR. Is it the same for mbedtls and lwip? If so, maybe
> explicit this in the usage that only maintainers are supposed to run
> this script with that option? And if you need to do it, ask the
> maintainers to do it for you? Does this make sense?
>
> The rules of running this script are the same as the original one
'dts/update-dts-subtree.sh'. We have the documentation for dts now,
and will have similar ones for MbedTLS and LWIP soon.
I didn't add this part yet since both MbedTLS and LWIP patch series
are still under review.


> > +    echo "  <subtree-name> mbedtls or dts or lwip"
> > +    echo "  <ref>          release tag [pull] or commit id [pick]"
> > +}
> > +
> > +if [ $# -ne 3 ]; then
> > +    print_usage
> > +    exit 1
> > +fi
> > +
> > +op=$1
> > +subtree_name=$2
> > +ref=$3
> > +
> > +set_params() {
> > +    case "$subtree_name" in
> > +        mbedtls)
> > +            path=lib/mbedtls/external/mbedtls
> > +            repo_url=https://github.com/Mbed-TLS/mbedtls.git
> > +            remote_name="mbedtls_upstream"
> > +            ;;
> > +        dts)
> > +            path=dts/upstream
> > +            repo_url=
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> > +            remote_name="devicetree-rebasing"
> > +            ;;
> > +        lwip)
> > +            path=lib/lwip/lwip
> > +            repo_url=https://git.savannah.gnu.org/git/lwip.git
> > +            remote_name="lwip_upstream"
> > +            ;;
> > +        *)
> > +            echo "Invalid subtree name: $subtree_name"
> > +            print_usage
> > +            exit 1
> > +    esac
> > +}
> > +
> > +set_params
> > +
> > +merge_commit_msg=$(cat << EOF
> > +Subtree merge tag '$ref' of $subtree_name repo [1] into $path
> > +
>
> I would recommend adding at the very least the path of the tools used to
> generate this merge commit (so ./tools/update-subtree.sh or maybe just
> $0?) or maybe even the full command line that was used to do this merge?
>
> Sorry but in any case would we need this commit message only without
knowing
the variables?


> > +[1] $repo_url
> > +EOF
> > +)
> > +
> > +remote_add_and_fetch() {
> > +    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
> > +        echo "Warning: Script automatically adds new git remote via:"
> > +        echo "    git remote add $remote_name \\"
> > +        echo "        $repo_url"
> > +        git remote add $remote_name $repo_url
> > +    fi
> > +    git fetch $remote_name master
> > +}
> > +
> > +if [ "$op" = "pull" ]; then
> > +    remote_add_and_fetch
> > +    git subtree pull --prefix $path $remote_name "$ref" --squash -m
> "$merge_commit_msg"
> > +elif [ "$op" = "pick" ]; then
> > +    remote_add_and_fetch
> > +    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
>
> Should we add -s there too? (or --signoff)
>
> As I search for all 'merge' commits in history, they have not
signed-off tags.
This patch is to extend the original script to support more git upstream
repos.
So I intended to keep the commit message format as the original one unless
we have reasons to improve it.

Regards,
Raymond

Re: [PATCH v2] tools: Add script to update git subtree projects

[Quentin Schulz]

Hi Raymond,

On 7/23/24 4:05 PM, Raymond Mao wrote:
> Hi Quentin,
> 
> On Tue, 23 Jul 2024 at 03:16, Quentin Schulz <quentin.schulz@cherry.de>
> wrote:
> 
>> Hi Raymond,
>>
>> On 7/22/24 9:30 PM, Raymond Mao wrote:
>>> Recently we are introducing multiple git subtree projects and
>>> it is the right time to have a universal script to update
>>> various subtrees and replace the dts/update-dts-subtree.sh.
>>>
>>> update-subtree.sh is a wrapper of git subtree commands.
>>>
>>> Usage: From U-Boot top directory,
>>> run
>>> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
>>> for pulling a tag from the upstream.
>>> Or run
>>> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
>>> for cherry-pick a commit from the upstream.
>>>
>>> Currently <subtree-name> supports dts, mbedtls and lwip.
>>>
>>> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
>>> ---
>>> Changes in v2
>>> - Refactored the script.
>>> - Update the control doc.
>>>
>>>    doc/develop/devicetree/control.rst |  8 +--
>>>    dts/update-dts-subtree.sh          | 48 -----------------
>>>    tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>>>    3 files changed, 90 insertions(+), 52 deletions(-)
>>>    delete mode 100755 dts/update-dts-subtree.sh
>>>    create mode 100755 tools/update-subtree.sh
>>>
>>> diff --git a/doc/develop/devicetree/control.rst
>> b/doc/develop/devicetree/control.rst
>>> index ca4fb0b5b10..211f7e4909c 100644
>>> --- a/doc/develop/devicetree/control.rst
>>> +++ b/doc/develop/devicetree/control.rst
>>> @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the
>> devicetree-rebasing repo whenever
>>>    the next branch opens (refer: :doc:`../release_cycle`) with the latest
>> mainline
>>>    Linux kernel release. To sync the `dts/upstream/` subtree, run::
>>>
>>> -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
>>> +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
>>>
>>>    If required it is also possible to cherry-pick fixes from the
>>>    devicetree-rebasing repository prior to next sync, usage::
>>>
>>> -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
>>> +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
>>>
>>>
>>>    Configuration
>>> @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received
>> devicetree source file for your
>>>    newly added board support then one option is that you can add the
>> corresponding
>>>    devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that
>> add `#
>>>    CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>`
>> when
>>> -prompted by Kconfig. Another option is that you can use use the "pick"
>> option of
>>> -`dts/update-dts-subtree.sh` mentioned above to bring in the commits
>> that you
>>> +prompted by Kconfig. Another option is that you can use the "pick"
>> option of
>>> +`tools/update-subtree.sh` mentioned above to bring in the commits that
>> you
>>>    need.
>>>
>>>    This should include your CPU or SoC's devicetree file. On top of that
>> any U-Boot
>>> diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
>>> deleted file mode 100755
>>> index a57b78a41d3..00000000000
>>> --- a/dts/update-dts-subtree.sh
>>> +++ /dev/null
>>> @@ -1,48 +0,0 @@
>>> -#!/bin/sh
>>> -# SPDX-License-Identifier: GPL-2.0+
>>> -#
>>> -# Copyright 2024 Linaro Ltd.
>>> -#
>>> -# Usage: from the top level U-Boot source tree, run:
>>> -# $ ./dts/update-dts-subtree.sh pull <release-tag>
>>> -# $ ./dts/update-dts-subtree.sh pick <commit-id>
>>> -#
>>> -# The script will pull changes from devicetree-rebasing repo into U-Boot
>>> -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
>>> -# automatically create a squash/merge commit listing the commits
>> imported.
>>> -
>>> -set -e
>>> -
>>> -merge_commit_msg=$(cat << EOF
>>> -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
>>> -
>>> -[1]
>> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
>>> -EOF
>>> -)
>>> -
>>> -remote_add_and_fetch() {
>>> -    if ! git remote get-url devicetree-rebasing 2>/dev/null
>>> -    then
>>> -        echo "Warning: Script automatically adds new git remote via:"
>>> -        echo "    git remote add devicetree-rebasing \\"
>>> -        echo "
>> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
>> "
>>> -        git remote add devicetree-rebasing \
>>> -
>> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
>>> -    fi
>>> -    git fetch devicetree-rebasing master
>>> -}
>>> -
>>> -if [ "$1" = "pull" ]
>>> -then
>>> -    remote_add_and_fetch
>>> -    git subtree pull --prefix dts/upstream devicetree-rebasing \
>>> -        "$2" --squash -m "${merge_commit_msg}"
>>> -elif [ "$1" = "pick" ]
>>> -then
>>> -    remote_add_and_fetch
>>> -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
>>> -else
>>> -    echo "usage: $0 <op> <ref>"
>>> -    echo "  <op>     pull or pick"
>>> -    echo "  <ref>    release tag [pull] or commit id [pick]"
>>> -fi
>>> diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
>>> new file mode 100755
>>> index 00000000000..5946aa1a9e2
>>> --- /dev/null
>>> +++ b/tools/update-subtree.sh
>>
>> We have scripts/ and tools/ directories, this being a shell script I
>> would have intuitively added it to scripts/. So my question is, what's
>> the difference between those two directories :) ?
>>
>> Good question. Personally I think 'scripts' are for those scripts needed
> by other components and 'tools' are for the general purpose utilities
> (not limited to scripts). That is the reason I prefer 'tools'.
> 
>> @@ -0,0 +1,86 @@
>>> +#!/bin/sh
>>> +# SPDX-License-Identifier: GPL-2.0+
>>> +#
>>> +# Copyright (c) 2024 Linaro Limited
>>> +# Author: Raymond Mao <raymond.mao@linaro.org>
>>> +#
>>> +# Usage: from the top level U-Boot source tree, run:
>>> +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
>>> +# Or:
>>> +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
>>> +#
>>> +# The script will pull changes from subtree repo into U-Boot.
>>> +# It will automatically create a squash/merge commit listing the commits
>>> +# imported.
>>> +
>>> +set -e
>>> +
>>> +print_usage() {
>>> +    echo "usage: $0 <op> <subtree-name> <ref>"
>>> +    echo "  <op>           pull or pick"
>>
>> I think pulls are supposed to be done and sent to the ML by maintainers
>> only? At least that's what we're supposed to do for the dts/upstream/
>> directory AFAIR. Is it the same for mbedtls and lwip? If so, maybe
>> explicit this in the usage that only maintainers are supposed to run
>> this script with that option? And if you need to do it, ask the
>> maintainers to do it for you? Does this make sense?
>>
>> The rules of running this script are the same as the original one
> 'dts/update-dts-subtree.sh'. We have the documentation for dts now,
> and will have similar ones for MbedTLS and LWIP soon.
> I didn't add this part yet since both MbedTLS and LWIP patch series
> are still under review.
> 

I'm suggesting we could "improve" this, not that it needs to be done as 
part of this patch :)

> 
>>> +    echo "  <subtree-name> mbedtls or dts or lwip"
>>> +    echo "  <ref>          release tag [pull] or commit id [pick]"
>>> +}
>>> +
>>> +if [ $# -ne 3 ]; then
>>> +    print_usage
>>> +    exit 1
>>> +fi
>>> +
>>> +op=$1
>>> +subtree_name=$2
>>> +ref=$3
>>> +
>>> +set_params() {
>>> +    case "$subtree_name" in
>>> +        mbedtls)
>>> +            path=lib/mbedtls/external/mbedtls
>>> +            repo_url=https://github.com/Mbed-TLS/mbedtls.git
>>> +            remote_name="mbedtls_upstream"
>>> +            ;;
>>> +        dts)
>>> +            path=dts/upstream
>>> +            repo_url=
>> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
>>> +            remote_name="devicetree-rebasing"
>>> +            ;;
>>> +        lwip)
>>> +            path=lib/lwip/lwip
>>> +            repo_url=https://git.savannah.gnu.org/git/lwip.git
>>> +            remote_name="lwip_upstream"
>>> +            ;;
>>> +        *)
>>> +            echo "Invalid subtree name: $subtree_name"
>>> +            print_usage
>>> +            exit 1
>>> +    esac
>>> +}
>>> +
>>> +set_params
>>> +
>>> +merge_commit_msg=$(cat << EOF
>>> +Subtree merge tag '$ref' of $subtree_name repo [1] into $path
>>> +
>>
>> I would recommend adding at the very least the path of the tools used to
>> generate this merge commit (so ./tools/update-subtree.sh or maybe just
>> $0?) or maybe even the full command line that was used to do this merge?
>>
>> Sorry but in any case would we need this commit message only without
> knowing
> the variables?
> 

Less magic for people not aware of those pulls and how they are made.

> 
>>> +[1] $repo_url
>>> +EOF
>>> +)
>>> +
>>> +remote_add_and_fetch() {
>>> +    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
>>> +        echo "Warning: Script automatically adds new git remote via:"
>>> +        echo "    git remote add $remote_name \\"
>>> +        echo "        $repo_url"
>>> +        git remote add $remote_name $repo_url
>>> +    fi
>>> +    git fetch $remote_name master
>>> +}
>>> +
>>> +if [ "$op" = "pull" ]; then
>>> +    remote_add_and_fetch
>>> +    git subtree pull --prefix $path $remote_name "$ref" --squash -m
>> "$merge_commit_msg"
>>> +elif [ "$op" = "pick" ]; then
>>> +    remote_add_and_fetch
>>> +    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
>>
>> Should we add -s there too? (or --signoff)
>>
>> As I search for all 'merge' commits in history, they have not
> signed-off tags.

Yes, and I think that's a mistake. Though.... I'm not entirely sure if 
we can add this Signed-off-by tag to the merge commit as well? (with 
what I'm suggesting, we would only fix one of the ways to bring patches 
into dts/upstream or the two other subtrees).

> This patch is to extend the original script to support more git upstream
> repos.

Yup, was just suggesting "improving" this, not necessarily as part of 
this one patch though.

Forgot to say in my earlier mail:
Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>

Thanks!
Quentin

Re: [PATCH v2] tools: Add script to update git subtree projects

[Sumit Garg]

Hi Raymond,

Thanks for doing this.

On Tue, 23 Jul 2024 at 01:01, Raymond Mao <raymond.mao@linaro.org> wrote:
>
> Recently we are introducing multiple git subtree projects and
> it is the right time to have a universal script to update
> various subtrees and replace the dts/update-dts-subtree.sh.
>
> update-subtree.sh is a wrapper of git subtree commands.
>
> Usage: From U-Boot top directory,
> run
> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> for pulling a tag from the upstream.
> Or run
> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> for cherry-pick a commit from the upstream.
>
> Currently <subtree-name> supports dts, mbedtls and lwip.
>
> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> ---
> Changes in v2
> - Refactored the script.
> - Update the control doc.
>
>  doc/develop/devicetree/control.rst |  8 +--
>  dts/update-dts-subtree.sh          | 48 -----------------
>  tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>  3 files changed, 90 insertions(+), 52 deletions(-)
>  delete mode 100755 dts/update-dts-subtree.sh
>  create mode 100755 tools/update-subtree.sh
>
> diff --git a/doc/develop/devicetree/control.rst b/doc/develop/devicetree/control.rst
> index ca4fb0b5b10..211f7e4909c 100644
> --- a/doc/develop/devicetree/control.rst
> +++ b/doc/develop/devicetree/control.rst
> @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the devicetree-rebasing repo whenever
>  the next branch opens (refer: :doc:`../release_cycle`) with the latest mainline
>  Linux kernel release. To sync the `dts/upstream/` subtree, run::
>
> -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
> +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
>
>  If required it is also possible to cherry-pick fixes from the
>  devicetree-rebasing repository prior to next sync, usage::
>
> -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
> +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
>
>
>  Configuration
> @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received devicetree source file for your
>  newly added board support then one option is that you can add the corresponding
>  devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that add `#
>  CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
> -prompted by Kconfig. Another option is that you can use use the "pick" option of
> -`dts/update-dts-subtree.sh` mentioned above to bring in the commits that you
> +prompted by Kconfig. Another option is that you can use the "pick" option of
> +`tools/update-subtree.sh` mentioned above to bring in the commits that you
>  need.
>
>  This should include your CPU or SoC's devicetree file. On top of that any U-Boot
> diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
> deleted file mode 100755
> index a57b78a41d3..00000000000
> --- a/dts/update-dts-subtree.sh
> +++ /dev/null
> @@ -1,48 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0+
> -#
> -# Copyright 2024 Linaro Ltd.
> -#
> -# Usage: from the top level U-Boot source tree, run:
> -# $ ./dts/update-dts-subtree.sh pull <release-tag>
> -# $ ./dts/update-dts-subtree.sh pick <commit-id>
> -#
> -# The script will pull changes from devicetree-rebasing repo into U-Boot
> -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
> -# automatically create a squash/merge commit listing the commits imported.
> -
> -set -e
> -
> -merge_commit_msg=$(cat << EOF
> -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
> -
> -[1] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
> -EOF
> -)
> -
> -remote_add_and_fetch() {
> -    if ! git remote get-url devicetree-rebasing 2>/dev/null
> -    then
> -        echo "Warning: Script automatically adds new git remote via:"
> -        echo "    git remote add devicetree-rebasing \\"
> -        echo "        https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git"
> -        git remote add devicetree-rebasing \
> -            https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> -    fi
> -    git fetch devicetree-rebasing master
> -}
> -
> -if [ "$1" = "pull" ]
> -then
> -    remote_add_and_fetch
> -    git subtree pull --prefix dts/upstream devicetree-rebasing \
> -        "$2" --squash -m "${merge_commit_msg}"
> -elif [ "$1" = "pick" ]
> -then
> -    remote_add_and_fetch
> -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
> -else
> -    echo "usage: $0 <op> <ref>"
> -    echo "  <op>     pull or pick"
> -    echo "  <ref>    release tag [pull] or commit id [pick]"
> -fi
> diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
> new file mode 100755
> index 00000000000..5946aa1a9e2
> --- /dev/null
> +++ b/tools/update-subtree.sh
> @@ -0,0 +1,86 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# Copyright (c) 2024 Linaro Limited
> +# Author: Raymond Mao <raymond.mao@linaro.org>
> +#
> +# Usage: from the top level U-Boot source tree, run:
> +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> +# Or:
> +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> +#
> +# The script will pull changes from subtree repo into U-Boot.
> +# It will automatically create a squash/merge commit listing the commits
> +# imported.
> +
> +set -e
> +
> +print_usage() {
> +    echo "usage: $0 <op> <subtree-name> <ref>"
> +    echo "  <op>           pull or pick"
> +    echo "  <subtree-name> mbedtls or dts or lwip"
> +    echo "  <ref>          release tag [pull] or commit id [pick]"
> +}
> +
> +if [ $# -ne 3 ]; then
> +    print_usage
> +    exit 1
> +fi
> +
> +op=$1
> +subtree_name=$2
> +ref=$3
> +
> +set_params() {
> +    case "$subtree_name" in
> +        mbedtls)
> +            path=lib/mbedtls/external/mbedtls

Although I am in favour of making this script generic, this subdir
path doesn't exist yet. Wouldn't it make more sense to add mbedtls
here alongside the corresponding patch-set? Otherwise the script will
just fail.

> +            repo_url=https://github.com/Mbed-TLS/mbedtls.git
> +            remote_name="mbedtls_upstream"
> +            ;;
> +        dts)
> +            path=dts/upstream
> +            repo_url=https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> +            remote_name="devicetree-rebasing"
> +            ;;
> +        lwip)
> +            path=lib/lwip/lwip

Ditto here.

-Sumit

> +            repo_url=https://git.savannah.gnu.org/git/lwip.git
> +            remote_name="lwip_upstream"
> +            ;;
> +        *)
> +            echo "Invalid subtree name: $subtree_name"
> +            print_usage
> +            exit 1
> +    esac
> +}
> +
> +set_params
> +
> +merge_commit_msg=$(cat << EOF
> +Subtree merge tag '$ref' of $subtree_name repo [1] into $path
> +
> +[1] $repo_url
> +EOF
> +)
> +
> +remote_add_and_fetch() {
> +    if [ -z "$(git remote get-url $remote_name 2>/dev/null)" ]; then
> +        echo "Warning: Script automatically adds new git remote via:"
> +        echo "    git remote add $remote_name \\"
> +        echo "        $repo_url"
> +        git remote add $remote_name $repo_url
> +    fi
> +    git fetch $remote_name master
> +}
> +
> +if [ "$op" = "pull" ]; then
> +    remote_add_and_fetch
> +    git subtree pull --prefix $path $remote_name "$ref" --squash -m "$merge_commit_msg"
> +elif [ "$op" = "pick" ]; then
> +    remote_add_and_fetch
> +    git cherry-pick -x --strategy=subtree -Xsubtree=$path/ "$ref"
> +else
> +    print_usage
> +    exit 1
> +fi
> --
> 2.25.1
>

Re: [PATCH v2] tools: Add script to update git subtree projects

[Raymond Mao]

Hi Sumit,

On Wed, 24 Jul 2024 at 03:08, Sumit Garg <sumit.garg@linaro.org> wrote:

> Hi Raymond,
>
> Thanks for doing this.
>
> On Tue, 23 Jul 2024 at 01:01, Raymond Mao <raymond.mao@linaro.org> wrote:
> >
> > Recently we are introducing multiple git subtree projects and
> > it is the right time to have a universal script to update
> > various subtrees and replace the dts/update-dts-subtree.sh.
> >
> > update-subtree.sh is a wrapper of git subtree commands.
> >
> > Usage: From U-Boot top directory,
> > run
> > $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> > for pulling a tag from the upstream.
> > Or run
> > $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> > for cherry-pick a commit from the upstream.
> >
> > Currently <subtree-name> supports dts, mbedtls and lwip.
> >
> > Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> > ---
> > Changes in v2
> > - Refactored the script.
> > - Update the control doc.
> >
> >  doc/develop/devicetree/control.rst |  8 +--
> >  dts/update-dts-subtree.sh          | 48 -----------------
> >  tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
> >  3 files changed, 90 insertions(+), 52 deletions(-)
> >  delete mode 100755 dts/update-dts-subtree.sh
> >  create mode 100755 tools/update-subtree.sh
> >
> > diff --git a/doc/develop/devicetree/control.rst
> b/doc/develop/devicetree/control.rst
> > index ca4fb0b5b10..211f7e4909c 100644
> > --- a/doc/develop/devicetree/control.rst
> > +++ b/doc/develop/devicetree/control.rst
> > @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the
> devicetree-rebasing repo whenever
> >  the next branch opens (refer: :doc:`../release_cycle`) with the latest
> mainline
> >  Linux kernel release. To sync the `dts/upstream/` subtree, run::
> >
> > -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
> > +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
> >
> >  If required it is also possible to cherry-pick fixes from the
> >  devicetree-rebasing repository prior to next sync, usage::
> >
> > -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
> > +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
> >
> >
> >  Configuration
> > @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received
> devicetree source file for your
> >  newly added board support then one option is that you can add the
> corresponding
> >  devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that
> add `#
> >  CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
> > -prompted by Kconfig. Another option is that you can use use the "pick"
> option of
> > -`dts/update-dts-subtree.sh` mentioned above to bring in the commits
> that you
> > +prompted by Kconfig. Another option is that you can use the "pick"
> option of
> > +`tools/update-subtree.sh` mentioned above to bring in the commits that
> you
> >  need.
> >
> >  This should include your CPU or SoC's devicetree file. On top of that
> any U-Boot
> > diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
> > deleted file mode 100755
> > index a57b78a41d3..00000000000
> > --- a/dts/update-dts-subtree.sh
> > +++ /dev/null
> > @@ -1,48 +0,0 @@
> > -#!/bin/sh
> > -# SPDX-License-Identifier: GPL-2.0+
> > -#
> > -# Copyright 2024 Linaro Ltd.
> > -#
> > -# Usage: from the top level U-Boot source tree, run:
> > -# $ ./dts/update-dts-subtree.sh pull <release-tag>
> > -# $ ./dts/update-dts-subtree.sh pick <commit-id>
> > -#
> > -# The script will pull changes from devicetree-rebasing repo into U-Boot
> > -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
> > -# automatically create a squash/merge commit listing the commits
> imported.
> > -
> > -set -e
> > -
> > -merge_commit_msg=$(cat << EOF
> > -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
> > -
> > -[1]
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
> > -EOF
> > -)
> > -
> > -remote_add_and_fetch() {
> > -    if ! git remote get-url devicetree-rebasing 2>/dev/null
> > -    then
> > -        echo "Warning: Script automatically adds new git remote via:"
> > -        echo "    git remote add devicetree-rebasing \\"
> > -        echo "
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> "
> > -        git remote add devicetree-rebasing \
> > -
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> > -    fi
> > -    git fetch devicetree-rebasing master
> > -}
> > -
> > -if [ "$1" = "pull" ]
> > -then
> > -    remote_add_and_fetch
> > -    git subtree pull --prefix dts/upstream devicetree-rebasing \
> > -        "$2" --squash -m "${merge_commit_msg}"
> > -elif [ "$1" = "pick" ]
> > -then
> > -    remote_add_and_fetch
> > -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
> > -else
> > -    echo "usage: $0 <op> <ref>"
> > -    echo "  <op>     pull or pick"
> > -    echo "  <ref>    release tag [pull] or commit id [pick]"
> > -fi
> > diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
> > new file mode 100755
> > index 00000000000..5946aa1a9e2
> > --- /dev/null
> > +++ b/tools/update-subtree.sh
> > @@ -0,0 +1,86 @@
> > +#!/bin/sh
> > +# SPDX-License-Identifier: GPL-2.0+
> > +#
> > +# Copyright (c) 2024 Linaro Limited
> > +# Author: Raymond Mao <raymond.mao@linaro.org>
> > +#
> > +# Usage: from the top level U-Boot source tree, run:
> > +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> > +# Or:
> > +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> > +#
> > +# The script will pull changes from subtree repo into U-Boot.
> > +# It will automatically create a squash/merge commit listing the commits
> > +# imported.
> > +
> > +set -e
> > +
> > +print_usage() {
> > +    echo "usage: $0 <op> <subtree-name> <ref>"
> > +    echo "  <op>           pull or pick"
> > +    echo "  <subtree-name> mbedtls or dts or lwip"
> > +    echo "  <ref>          release tag [pull] or commit id [pick]"
> > +}
> > +
> > +if [ $# -ne 3 ]; then
> > +    print_usage
> > +    exit 1
> > +fi
> > +
> > +op=$1
> > +subtree_name=$2
> > +ref=$3
> > +
> > +set_params() {
> > +    case "$subtree_name" in
> > +        mbedtls)
> > +            path=lib/mbedtls/external/mbedtls
>
> Although I am in favour of making this script generic, this subdir
> path doesn't exist yet. Wouldn't it make more sense to add mbedtls
> here alongside the corresponding patch-set? Otherwise the script will
> just fail.
>
> Actually this patch is split from the MbedTLS series which is under review.
I think this should be fine before merging the MbedTLS/LWIP series, since
it is supposed to be run by maintainers only. And the failing message
explains
that the sub-project dir does not exist which should be sufficient at the
moment.

Regards,
Raymond

Re: [PATCH v2] tools: Add script to update git subtree projects

[Sumit Garg]

On Wed, 24 Jul 2024 at 19:40, Raymond Mao <raymond.mao@linaro.org> wrote:
>
> Hi Sumit,
>
> On Wed, 24 Jul 2024 at 03:08, Sumit Garg <sumit.garg@linaro.org> wrote:
>>
>> Hi Raymond,
>>
>> Thanks for doing this.
>>
>> On Tue, 23 Jul 2024 at 01:01, Raymond Mao <raymond.mao@linaro.org> wrote:
>> >
>> > Recently we are introducing multiple git subtree projects and
>> > it is the right time to have a universal script to update
>> > various subtrees and replace the dts/update-dts-subtree.sh.
>> >
>> > update-subtree.sh is a wrapper of git subtree commands.
>> >
>> > Usage: From U-Boot top directory,
>> > run
>> > $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
>> > for pulling a tag from the upstream.
>> > Or run
>> > $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
>> > for cherry-pick a commit from the upstream.
>> >
>> > Currently <subtree-name> supports dts, mbedtls and lwip.
>> >
>> > Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
>> > ---
>> > Changes in v2
>> > - Refactored the script.
>> > - Update the control doc.
>> >
>> >  doc/develop/devicetree/control.rst |  8 +--
>> >  dts/update-dts-subtree.sh          | 48 -----------------
>> >  tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>> >  3 files changed, 90 insertions(+), 52 deletions(-)
>> >  delete mode 100755 dts/update-dts-subtree.sh
>> >  create mode 100755 tools/update-subtree.sh
>> >
>> > diff --git a/doc/develop/devicetree/control.rst b/doc/develop/devicetree/control.rst
>> > index ca4fb0b5b10..211f7e4909c 100644
>> > --- a/doc/develop/devicetree/control.rst
>> > +++ b/doc/develop/devicetree/control.rst
>> > @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the devicetree-rebasing repo whenever
>> >  the next branch opens (refer: :doc:`../release_cycle`) with the latest mainline
>> >  Linux kernel release. To sync the `dts/upstream/` subtree, run::
>> >
>> > -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
>> > +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
>> >
>> >  If required it is also possible to cherry-pick fixes from the
>> >  devicetree-rebasing repository prior to next sync, usage::
>> >
>> > -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
>> > +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
>> >
>> >
>> >  Configuration
>> > @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received devicetree source file for your
>> >  newly added board support then one option is that you can add the corresponding
>> >  devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that add `#
>> >  CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
>> > -prompted by Kconfig. Another option is that you can use use the "pick" option of
>> > -`dts/update-dts-subtree.sh` mentioned above to bring in the commits that you
>> > +prompted by Kconfig. Another option is that you can use the "pick" option of
>> > +`tools/update-subtree.sh` mentioned above to bring in the commits that you
>> >  need.
>> >
>> >  This should include your CPU or SoC's devicetree file. On top of that any U-Boot
>> > diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
>> > deleted file mode 100755
>> > index a57b78a41d3..00000000000
>> > --- a/dts/update-dts-subtree.sh
>> > +++ /dev/null
>> > @@ -1,48 +0,0 @@
>> > -#!/bin/sh
>> > -# SPDX-License-Identifier: GPL-2.0+
>> > -#
>> > -# Copyright 2024 Linaro Ltd.
>> > -#
>> > -# Usage: from the top level U-Boot source tree, run:
>> > -# $ ./dts/update-dts-subtree.sh pull <release-tag>
>> > -# $ ./dts/update-dts-subtree.sh pick <commit-id>
>> > -#
>> > -# The script will pull changes from devicetree-rebasing repo into U-Boot
>> > -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
>> > -# automatically create a squash/merge commit listing the commits imported.
>> > -
>> > -set -e
>> > -
>> > -merge_commit_msg=$(cat << EOF
>> > -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
>> > -
>> > -[1] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
>> > -EOF
>> > -)
>> > -
>> > -remote_add_and_fetch() {
>> > -    if ! git remote get-url devicetree-rebasing 2>/dev/null
>> > -    then
>> > -        echo "Warning: Script automatically adds new git remote via:"
>> > -        echo "    git remote add devicetree-rebasing \\"
>> > -        echo "        https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git"
>> > -        git remote add devicetree-rebasing \
>> > -            https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
>> > -    fi
>> > -    git fetch devicetree-rebasing master
>> > -}
>> > -
>> > -if [ "$1" = "pull" ]
>> > -then
>> > -    remote_add_and_fetch
>> > -    git subtree pull --prefix dts/upstream devicetree-rebasing \
>> > -        "$2" --squash -m "${merge_commit_msg}"
>> > -elif [ "$1" = "pick" ]
>> > -then
>> > -    remote_add_and_fetch
>> > -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
>> > -else
>> > -    echo "usage: $0 <op> <ref>"
>> > -    echo "  <op>     pull or pick"
>> > -    echo "  <ref>    release tag [pull] or commit id [pick]"
>> > -fi
>> > diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
>> > new file mode 100755
>> > index 00000000000..5946aa1a9e2
>> > --- /dev/null
>> > +++ b/tools/update-subtree.sh
>> > @@ -0,0 +1,86 @@
>> > +#!/bin/sh
>> > +# SPDX-License-Identifier: GPL-2.0+
>> > +#
>> > +# Copyright (c) 2024 Linaro Limited
>> > +# Author: Raymond Mao <raymond.mao@linaro.org>

Something I forgot earlier, if you are going to add explicit
authorship then you should retain the prior author too from where this
script is derived.

>> > +#
>> > +# Usage: from the top level U-Boot source tree, run:
>> > +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
>> > +# Or:
>> > +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
>> > +#
>> > +# The script will pull changes from subtree repo into U-Boot.
>> > +# It will automatically create a squash/merge commit listing the commits
>> > +# imported.
>> > +
>> > +set -e
>> > +
>> > +print_usage() {
>> > +    echo "usage: $0 <op> <subtree-name> <ref>"
>> > +    echo "  <op>           pull or pick"
>> > +    echo "  <subtree-name> mbedtls or dts or lwip"
>> > +    echo "  <ref>          release tag [pull] or commit id [pick]"
>> > +}
>> > +
>> > +if [ $# -ne 3 ]; then
>> > +    print_usage
>> > +    exit 1
>> > +fi
>> > +
>> > +op=$1
>> > +subtree_name=$2
>> > +ref=$3
>> > +
>> > +set_params() {
>> > +    case "$subtree_name" in
>> > +        mbedtls)
>> > +            path=lib/mbedtls/external/mbedtls
>>
>> Although I am in favour of making this script generic, this subdir
>> path doesn't exist yet. Wouldn't it make more sense to add mbedtls
>> here alongside the corresponding patch-set? Otherwise the script will
>> just fail.
>>
> Actually this patch is split from the MbedTLS series which is under review.
> I think this should be fine before merging the MbedTLS/LWIP series, since
> it is supposed to be run by maintainers only.

Developers also use this script to cherry-pick DTS patches from
upstream to enable certain features in U-Boot.

> And the failing message explains
> that the sub-project dir does not exist which should be sufficient at the moment.

Sure, I am only afraid that if this patch lands in a prior U-Boot
release than the actual MbedTLS/LWIP series then we are just giving
users a somewhat broken tool. If your worry is about merge conflicts
then I suppose that should be taken care off while merging
corresponding patch series.

-Sumit

>
> Regards,
> Raymond

Re: [PATCH v2] tools: Add script to update git subtree projects

[Jerome Forissier]

On 7/25/24 07:45, Sumit Garg wrote:
> On Wed, 24 Jul 2024 at 19:40, Raymond Mao <raymond.mao@linaro.org> wrote:
>>
>> Hi Sumit,
>>
>> On Wed, 24 Jul 2024 at 03:08, Sumit Garg <sumit.garg@linaro.org> wrote:
>>>
>>> Hi Raymond,
>>>
>>> Thanks for doing this.
>>>
>>> On Tue, 23 Jul 2024 at 01:01, Raymond Mao <raymond.mao@linaro.org> wrote:
>>>>
>>>> Recently we are introducing multiple git subtree projects and
>>>> it is the right time to have a universal script to update
>>>> various subtrees and replace the dts/update-dts-subtree.sh.
>>>>
>>>> update-subtree.sh is a wrapper of git subtree commands.
>>>>
>>>> Usage: From U-Boot top directory,
>>>> run
>>>> $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
>>>> for pulling a tag from the upstream.
>>>> Or run
>>>> $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
>>>> for cherry-pick a commit from the upstream.
>>>>
>>>> Currently <subtree-name> supports dts, mbedtls and lwip.
>>>>
>>>> Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
>>>> ---
>>>> Changes in v2
>>>> - Refactored the script.
>>>> - Update the control doc.
>>>>
>>>>  doc/develop/devicetree/control.rst |  8 +--
>>>>  dts/update-dts-subtree.sh          | 48 -----------------
>>>>  tools/update-subtree.sh            | 86 ++++++++++++++++++++++++++++++
>>>>  3 files changed, 90 insertions(+), 52 deletions(-)
>>>>  delete mode 100755 dts/update-dts-subtree.sh
>>>>  create mode 100755 tools/update-subtree.sh
>>>>
>>>> diff --git a/doc/develop/devicetree/control.rst b/doc/develop/devicetree/control.rst
>>>> index ca4fb0b5b10..211f7e4909c 100644
>>>> --- a/doc/develop/devicetree/control.rst
>>>> +++ b/doc/develop/devicetree/control.rst
>>>> @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the devicetree-rebasing repo whenever
>>>>  the next branch opens (refer: :doc:`../release_cycle`) with the latest mainline
>>>>  Linux kernel release. To sync the `dts/upstream/` subtree, run::
>>>>
>>>> -    ./dts/update-dts-subtree.sh pull <devicetree-rebasing-release-tag>
>>>> +    ./tools/update-subtree.sh pull dts <devicetree-rebasing-release-tag>
>>>>
>>>>  If required it is also possible to cherry-pick fixes from the
>>>>  devicetree-rebasing repository prior to next sync, usage::
>>>>
>>>> -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
>>>> +    ./tools/update-subtree.sh pick dts <devicetree-rebasing-commit-id>
>>>>
>>>>
>>>>  Configuration
>>>> @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received devicetree source file for your
>>>>  newly added board support then one option is that you can add the corresponding
>>>>  devicetree source file as `arch/<arch>/dts/<name>.dts`. To select that add `#
>>>>  CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>` when
>>>> -prompted by Kconfig. Another option is that you can use use the "pick" option of
>>>> -`dts/update-dts-subtree.sh` mentioned above to bring in the commits that you
>>>> +prompted by Kconfig. Another option is that you can use the "pick" option of
>>>> +`tools/update-subtree.sh` mentioned above to bring in the commits that you
>>>>  need.
>>>>
>>>>  This should include your CPU or SoC's devicetree file. On top of that any U-Boot
>>>> diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
>>>> deleted file mode 100755
>>>> index a57b78a41d3..00000000000
>>>> --- a/dts/update-dts-subtree.sh
>>>> +++ /dev/null
>>>> @@ -1,48 +0,0 @@
>>>> -#!/bin/sh
>>>> -# SPDX-License-Identifier: GPL-2.0+
>>>> -#
>>>> -# Copyright 2024 Linaro Ltd.
>>>> -#
>>>> -# Usage: from the top level U-Boot source tree, run:
>>>> -# $ ./dts/update-dts-subtree.sh pull <release-tag>
>>>> -# $ ./dts/update-dts-subtree.sh pick <commit-id>
>>>> -#
>>>> -# The script will pull changes from devicetree-rebasing repo into U-Boot
>>>> -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It will
>>>> -# automatically create a squash/merge commit listing the commits imported.
>>>> -
>>>> -set -e
>>>> -
>>>> -merge_commit_msg=$(cat << EOF
>>>> -Subtree merge tag '$2' of devicetree-rebasing repo [1] into dts/upstream
>>>> -
>>>> -[1] https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
>>>> -EOF
>>>> -)
>>>> -
>>>> -remote_add_and_fetch() {
>>>> -    if ! git remote get-url devicetree-rebasing 2>/dev/null
>>>> -    then
>>>> -        echo "Warning: Script automatically adds new git remote via:"
>>>> -        echo "    git remote add devicetree-rebasing \\"
>>>> -        echo "        https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git"
>>>> -        git remote add devicetree-rebasing \
>>>> -            https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
>>>> -    fi
>>>> -    git fetch devicetree-rebasing master
>>>> -}
>>>> -
>>>> -if [ "$1" = "pull" ]
>>>> -then
>>>> -    remote_add_and_fetch
>>>> -    git subtree pull --prefix dts/upstream devicetree-rebasing \
>>>> -        "$2" --squash -m "${merge_commit_msg}"
>>>> -elif [ "$1" = "pick" ]
>>>> -then
>>>> -    remote_add_and_fetch
>>>> -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/ "$2"
>>>> -else
>>>> -    echo "usage: $0 <op> <ref>"
>>>> -    echo "  <op>     pull or pick"
>>>> -    echo "  <ref>    release tag [pull] or commit id [pick]"
>>>> -fi
>>>> diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
>>>> new file mode 100755
>>>> index 00000000000..5946aa1a9e2
>>>> --- /dev/null
>>>> +++ b/tools/update-subtree.sh
>>>> @@ -0,0 +1,86 @@
>>>> +#!/bin/sh
>>>> +# SPDX-License-Identifier: GPL-2.0+
>>>> +#
>>>> +# Copyright (c) 2024 Linaro Limited
>>>> +# Author: Raymond Mao <raymond.mao@linaro.org>
> 
> Something I forgot earlier, if you are going to add explicit
> authorship then you should retain the prior author too from where this
> script is derived.
> 
>>>> +#
>>>> +# Usage: from the top level U-Boot source tree, run:
>>>> +# $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
>>>> +# Or:
>>>> +# $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
>>>> +#
>>>> +# The script will pull changes from subtree repo into U-Boot.
>>>> +# It will automatically create a squash/merge commit listing the commits
>>>> +# imported.
>>>> +
>>>> +set -e
>>>> +
>>>> +print_usage() {
>>>> +    echo "usage: $0 <op> <subtree-name> <ref>"
>>>> +    echo "  <op>           pull or pick"
>>>> +    echo "  <subtree-name> mbedtls or dts or lwip"
>>>> +    echo "  <ref>          release tag [pull] or commit id [pick]"
>>>> +}
>>>> +
>>>> +if [ $# -ne 3 ]; then
>>>> +    print_usage
>>>> +    exit 1
>>>> +fi
>>>> +
>>>> +op=$1
>>>> +subtree_name=$2
>>>> +ref=$3
>>>> +
>>>> +set_params() {
>>>> +    case "$subtree_name" in
>>>> +        mbedtls)
>>>> +            path=lib/mbedtls/external/mbedtls
>>>
>>> Although I am in favour of making this script generic, this subdir
>>> path doesn't exist yet. Wouldn't it make more sense to add mbedtls
>>> here alongside the corresponding patch-set? Otherwise the script will
>>> just fail.
>>>
>> Actually this patch is split from the MbedTLS series which is under review.
>> I think this should be fine before merging the MbedTLS/LWIP series, since
>> it is supposed to be run by maintainers only.
> 
> Developers also use this script to cherry-pick DTS patches from
> upstream to enable certain features in U-Boot.
> 
>> And the failing message explains
>> that the sub-project dir does not exist which should be sufficient at the moment.
> 
> Sure, I am only afraid that if this patch lands in a prior U-Boot
> release than the actual MbedTLS/LWIP series then we are just giving
> users a somewhat broken tool. If your worry is about merge conflicts
> then I suppose that should be taken care off while merging
> corresponding patch series.

I can certainly take care of adding the lwIP chunk in the lwIP series
if that helps.

Thanks,
-- 
Jerome

> 
> -Sumit
> 
>>
>> Regards,
>> Raymond

Re: [PATCH v2] tools: Add script to update git subtree projects

[Raymond Mao]

Hi Sumit,

On Thu, 25 Jul 2024 at 01:46, Sumit Garg <sumit.garg@linaro.org> wrote:

> On Wed, 24 Jul 2024 at 19:40, Raymond Mao <raymond.mao@linaro.org> wrote:
> >
> > Hi Sumit,
> >
> > On Wed, 24 Jul 2024 at 03:08, Sumit Garg <sumit.garg@linaro.org> wrote:
> >>
> >> Hi Raymond,
> >>
> >> Thanks for doing this.
> >>
> >> On Tue, 23 Jul 2024 at 01:01, Raymond Mao <raymond.mao@linaro.org>
> wrote:
> >> >
> >> > Recently we are introducing multiple git subtree projects and
> >> > it is the right time to have a universal script to update
> >> > various subtrees and replace the dts/update-dts-subtree.sh.
> >> >
> >> > update-subtree.sh is a wrapper of git subtree commands.
> >> >
> >> > Usage: From U-Boot top directory,
> >> > run
> >> > $ ./tools/update-subtree.sh pull <subtree-name> <release-tag>
> >> > for pulling a tag from the upstream.
> >> > Or run
> >> > $ ./tools/update-subtree.sh pick <subtree-name> <commit-id>
> >> > for cherry-pick a commit from the upstream.
> >> >
> >> > Currently <subtree-name> supports dts, mbedtls and lwip.
> >> >
> >> > Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
> >> > ---
> >> > Changes in v2
> >> > - Refactored the script.
> >> > - Update the control doc.
> >> >
> >> >  doc/develop/devicetree/control.rst |  8 +--
> >> >  dts/update-dts-subtree.sh          | 48 -----------------
> >> >  tools/update-subtree.sh            | 86
> ++++++++++++++++++++++++++++++
> >> >  3 files changed, 90 insertions(+), 52 deletions(-)
> >> >  delete mode 100755 dts/update-dts-subtree.sh
> >> >  create mode 100755 tools/update-subtree.sh
> >> >
> >> > diff --git a/doc/develop/devicetree/control.rst
> b/doc/develop/devicetree/control.rst
> >> > index ca4fb0b5b10..211f7e4909c 100644
> >> > --- a/doc/develop/devicetree/control.rst
> >> > +++ b/doc/develop/devicetree/control.rst
> >> > @@ -96,12 +96,12 @@ sync the `dts/upstream/` subtree from the
> devicetree-rebasing repo whenever
> >> >  the next branch opens (refer: :doc:`../release_cycle`) with the
> latest mainline
> >> >  Linux kernel release. To sync the `dts/upstream/` subtree, run::
> >> >
> >> > -    ./dts/update-dts-subtree.sh pull
> <devicetree-rebasing-release-tag>
> >> > +    ./tools/update-subtree.sh pull dts
> <devicetree-rebasing-release-tag>
> >> >
> >> >  If required it is also possible to cherry-pick fixes from the
> >> >  devicetree-rebasing repository prior to next sync, usage::
> >> >
> >> > -    ./dts/update-dts-subtree.sh pick <devicetree-rebasing-commit-id>
> >> > +    ./tools/update-subtree.sh pick dts
> <devicetree-rebasing-commit-id>
> >> >
> >> >
> >> >  Configuration
> >> > @@ -116,8 +116,8 @@ However, if `dts/upstream/` hasn't yet received
> devicetree source file for your
> >> >  newly added board support then one option is that you can add the
> corresponding
> >> >  devicetree source file as `arch/<arch>/dts/<name>.dts`. To select
> that add `#
> >> >  CONFIG_OF_UPSTREAM is not set` and set `DEFAULT_DEVICE_TREE=<name>`
> when
> >> > -prompted by Kconfig. Another option is that you can use use the
> "pick" option of
> >> > -`dts/update-dts-subtree.sh` mentioned above to bring in the commits
> that you
> >> > +prompted by Kconfig. Another option is that you can use the "pick"
> option of
> >> > +`tools/update-subtree.sh` mentioned above to bring in the commits
> that you
> >> >  need.
> >> >
> >> >  This should include your CPU or SoC's devicetree file. On top of
> that any U-Boot
> >> > diff --git a/dts/update-dts-subtree.sh b/dts/update-dts-subtree.sh
> >> > deleted file mode 100755
> >> > index a57b78a41d3..00000000000
> >> > --- a/dts/update-dts-subtree.sh
> >> > +++ /dev/null
> >> > @@ -1,48 +0,0 @@
> >> > -#!/bin/sh
> >> > -# SPDX-License-Identifier: GPL-2.0+
> >> > -#
> >> > -# Copyright 2024 Linaro Ltd.
> >> > -#
> >> > -# Usage: from the top level U-Boot source tree, run:
> >> > -# $ ./dts/update-dts-subtree.sh pull <release-tag>
> >> > -# $ ./dts/update-dts-subtree.sh pick <commit-id>
> >> > -#
> >> > -# The script will pull changes from devicetree-rebasing repo into
> U-Boot
> >> > -# as a subtree located as <U-Boot>/dts/upstream sub-directory. It
> will
> >> > -# automatically create a squash/merge commit listing the commits
> imported.
> >> > -
> >> > -set -e
> >> > -
> >> > -merge_commit_msg=$(cat << EOF
> >> > -Subtree merge tag '$2' of devicetree-rebasing repo [1] into
> dts/upstream
> >> > -
> >> > -[1]
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
> >> > -EOF
> >> > -)
> >> > -
> >> > -remote_add_and_fetch() {
> >> > -    if ! git remote get-url devicetree-rebasing 2>/dev/null
> >> > -    then
> >> > -        echo "Warning: Script automatically adds new git remote via:"
> >> > -        echo "    git remote add devicetree-rebasing \\"
> >> > -        echo "
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> "
> >> > -        git remote add devicetree-rebasing \
> >> > -
> https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git
> >> > -    fi
> >> > -    git fetch devicetree-rebasing master
> >> > -}
> >> > -
> >> > -if [ "$1" = "pull" ]
> >> > -then
> >> > -    remote_add_and_fetch
> >> > -    git subtree pull --prefix dts/upstream devicetree-rebasing \
> >> > -        "$2" --squash -m "${merge_commit_msg}"
> >> > -elif [ "$1" = "pick" ]
> >> > -then
> >> > -    remote_add_and_fetch
> >> > -    git cherry-pick -x --strategy=subtree -Xsubtree=dts/upstream/
> "$2"
> >> > -else
> >> > -    echo "usage: $0 <op> <ref>"
> >> > -    echo "  <op>     pull or pick"
> >> > -    echo "  <ref>    release tag [pull] or commit id [pick]"
> >> > -fi
> >> > diff --git a/tools/update-subtree.sh b/tools/update-subtree.sh
> >> > new file mode 100755
> >> > index 00000000000..5946aa1a9e2
> >> > --- /dev/null
> >> > +++ b/tools/update-subtree.sh
> >> > @@ -0,0 +1,86 @@
> >> > +#!/bin/sh
> >> > +# SPDX-License-Identifier: GPL-2.0+
> >> > +#
> >> > +# Copyright (c) 2024 Linaro Limited
> >> > +# Author: Raymond Mao <raymond.mao@linaro.org>
>
> Something I forgot earlier, if you are going to add explicit
> authorship then you should retain the prior author too from where this
> script is derived.
>
> Sorry about it, I will update the file header.

Raymond