[yocto-autobuilder-helper][PATCH] oe-mirror: Add job to mirror openembedded sources

[yocto-autobuilder-helper][PATCH] oe-mirror: Add job to mirror openembedded sources

[David Pierret]

- Add a job that call run-fetchuniverse
- Add script run-fetchuniverse
   - loop over each LTS branches
   - Download all the meta-openembedded layers sources.
   - Call to oe-selftest test_yocto_source_mirror after fetching.
We are not using a second step to keep checkout consistency with potential merge.

Signed-off-by: David Pierret <david.pierret@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
---
 config.json               | 13 +++++++++++
 scripts/run-fetchuniverse | 49 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
 create mode 100755 scripts/run-fetchuniverse

diff --git a/config.json b/config.json
index 3acb710..8d59be8 100644
--- a/config.json
+++ b/config.json
@@ -1420,6 +1420,19 @@
                 "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
             ]
         },
+        "oe-mirror" : {
+            "SDKMACHINE" : "x86_64",
+            "MACHINE" : "qemux86-64",
+            "NEEDREPOS" : ["poky", "meta-openembedded"],
+
+            "step1" : {
+                "shortname" : "Sources pre-fetching",
+                "EXTRAPLAINCMDS" : ["${SCRIPTSDIR}/run-fetchuniverse ${BUILDDIR} ${SCRIPTSDIR}"],
+                "extravars" : [
+                    "SOURCE_MIRROR_FETCH = '1'"
+                ]
+            }
+        },
         "a-quick" : {
             "TEMPLATE" : "trigger-build"
         },
diff --git a/scripts/run-fetchuniverse b/scripts/run-fetchuniverse
new file mode 100755
index 0000000..9ef82de
--- /dev/null
+++ b/scripts/run-fetchuniverse
@@ -0,0 +1,49 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+BUILDDIR=`realpath $1`
+SCRIPTSDIR=`realpath $2`
+ROOTDIR=$BUILDDIR/..
+
+#
+# Fetch mirror for each LTS branch and test integrity
+#
+for branch in master mickledore langdale kirkstone dunfell; do
+  # For each layer
+  for repo in bitbake poky meta-openembedded ; do
+    pushd $ROOTDIR/$repo || exit 1
+      # reset branch
+      git reset origin/$branch --hard
+    popd || exit 1
+  done
+  # remove config to be re-generated by oe-init-build-env
+  rm conf/local.conf
+  rm conf/bblayers.conf
+  rm -f conf/templateconf.cfg
+  rm tmp/ -rf
+
+  # got back to ROOTDIR
+  pushd $ROOTDIR || exit 1
+  # generate config for current branch
+  . ./poky/oe-init-build-env build
+
+  # We need to add the meta-openembedded layers before fetch
+  bitbake-layers add-layer ${BUILDDIR}/../meta-openembedded/meta-filesystems ${BUILDDIR}/../meta-openembedded/meta-gnome \
+                           ${BUILDDIR}/../meta-openembedded/meta-initramfs ${BUILDDIR}/../meta-openembedded/meta-multimedia \
+                           ${BUILDDIR}/../meta-openembedded/meta-networking ${BUILDDIR}/../meta-openembedded/meta-oe \
+                           ${BUILDDIR}/../meta-openembedded/meta-perl ${BUILDDIR}/../meta-openembedded/meta-python \
+                           ${BUILDDIR}/../meta-openembedded/meta-webserver ${BUILDDIR}/../meta-openembedded/meta-xfce
+  # call bitbake to fetch universe
+  bitbake universe -c fetch -k
+
+  # call oe-selftest to execute test_yocto_source_mirror
+  # We need the selftest layer
+  bitbake-layers add-layer $BUILDDIR/../meta-selftest
+  # ensure no previous build-st directory exist
+  rm -rf ${BUILDDIR}/../build-st
+  ${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror || exit 1
+  popd || exit 1
+done
+
+
-- 
2.39.2

Re: [yocto] [yocto-autobuilder-helper][PATCH] oe-mirror: Add job to mirror openembedded sources

[Richard Purdie]

On Tue, 2023-10-10 at 16:49 +0200, David Pierret wrote:
> - Add a job that call run-fetchuniverse
> - Add script run-fetchuniverse
>    - loop over each LTS branches
>    - Download all the meta-openembedded layers sources.
>    - Call to oe-selftest test_yocto_source_mirror after fetching.
> We are not using a second step to keep checkout consistency with potential merge.
> 
> Signed-off-by: David Pierret <david.pierret@smile.fr>
> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
> ---
>  config.json               | 13 +++++++++++
>  scripts/run-fetchuniverse | 49 +++++++++++++++++++++++++++++++++++++++
>  2 files changed, 62 insertions(+)
>  create mode 100755 scripts/run-fetchuniverse
> 
> diff --git a/config.json b/config.json
> index 3acb710..8d59be8 100644
> --- a/config.json
> +++ b/config.json
> @@ -1420,6 +1420,19 @@
>                  "${SCRIPTSDIR}/setup-auh ${HELPERBUILDDIR}; ${SCRIPTSDIR}/run-auh ${HELPERBUILDDIR} ${WEBPUBLISH_DIR}/pub/auh/"
>              ]
>          },
> +        "oe-mirror" : {

This needs to be called something like meta-oe-mirror to differentiate
it from other "OE" things.


> +            "SDKMACHINE" : "x86_64",
> +            "MACHINE" : "qemux86-64",
> +            "NEEDREPOS" : ["poky", "meta-openembedded"],
> +
> +            "step1" : {
> +                "shortname" : "Sources pre-fetching",
> +                "EXTRAPLAINCMDS" : ["${SCRIPTSDIR}/run-fetchuniverse ${BUILDDIR} ${SCRIPTSDIR}"],
> +                "extravars" : [
> +                    "SOURCE_MIRROR_FETCH = '1'"
> +                ]
> +            }
> +        },
>          "a-quick" : {
>              "TEMPLATE" : "trigger-build"
>          },
> diff --git a/scripts/run-fetchuniverse b/scripts/run-fetchuniverse
> new file mode 100755
> index 0000000..9ef82de
> --- /dev/null
> +++ b/scripts/run-fetchuniverse
> @@ -0,0 +1,49 @@
> +#!/bin/bash
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +BUILDDIR=`realpath $1`
> +SCRIPTSDIR=`realpath $2`
> +ROOTDIR=$BUILDDIR/..
> +
> +#
> +# Fetch mirror for each LTS branch and test integrity
> +#
> +for branch in master mickledore langdale kirkstone dunfell; do

Rather than iterate here, we should add a configuration to each branch
of -helper where we want to enable this.

> +  # For each layer
> +  for repo in bitbake poky meta-openembedded ; do
> +    pushd $ROOTDIR/$repo || exit 1
> +      # reset branch
> +      git reset origin/$branch --hard
> +    popd || exit 1
> +  done
> +  # remove config to be re-generated by oe-init-build-env
> +  rm conf/local.conf
> +  rm conf/bblayers.conf
> +  rm -f conf/templateconf.cfg
> +  rm tmp/ -rf
> +
> +  # got back to ROOTDIR
> +  pushd $ROOTDIR || exit 1
> +  # generate config for current branch
> +  . ./poky/oe-init-build-env build
> +
> +  # We need to add the meta-openembedded layers before fetch
> +  bitbake-layers add-layer ${BUILDDIR}/../meta-openembedded/meta-filesystems ${BUILDDIR}/../meta-openembedded/meta-gnome \
> +                           ${BUILDDIR}/../meta-openembedded/meta-initramfs ${BUILDDIR}/../meta-openembedded/meta-multimedia \
> +                           ${BUILDDIR}/../meta-openembedded/meta-networking ${BUILDDIR}/../meta-openembedded/meta-oe \
> +                           ${BUILDDIR}/../meta-openembedded/meta-perl ${BUILDDIR}/../meta-openembedded/meta-python \
> +                           ${BUILDDIR}/../meta-openembedded/meta-webserver ${BUILDDIR}/../meta-openembedded/meta-xfce

Then hopefully we can massively simplify the script!

Cheers,

Richard

> +  # call bitbake to fetch universe
> +  bitbake universe -c fetch -k
> +
> +  # call oe-selftest to execute test_yocto_source_mirror
> +  # We need the selftest layer
> +  bitbake-layers add-layer $BUILDDIR/../meta-selftest
> +  # ensure no previous build-st directory exist
> +  rm -rf ${BUILDDIR}/../build-st
> +  ${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror || exit 1
> +  popd || exit 1
> +done
> +
> +
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#61306): https://lists.yoctoproject.org/g/yocto/message/61306
> Mute This Topic: https://lists.yoctoproject.org/mt/101876565/1686473
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [richard.purdie@linuxfoundation.org]
> -=-=-=-=-=-=-=-=-=-=-=-
>