[PATCH 1/4] ci: azure: Update to use stages

[PATCH 1/4] ci: azure: Update to use stages

[Tom Rini]

Follow what we do in GitLab CI where we break the jobs up in to stages
such that if earlier and often quicker sanity tests fail we don't run
everything else.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .azure-pipelines.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index 670bbc0e1636..d97115668167 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -9,7 +9,9 @@ variables:
   container_option: -u 0
   work_dir: /u
 
-jobs:
+stages:
+- stage: testsuites
+  jobs:
   - job: tools_only_windows
     displayName: 'Ensure host tools build for Windows'
     pool:
@@ -199,6 +201,8 @@ jobs:
           export PATH=/opt/gcc-11.1.0-nolibc/arm-linux-gnueabi/bin:$PATH
           test/nokia_rx51_test.sh
 
+- stage: test_py
+  jobs:
   - job: test_py
     displayName: 'test.py'
     pool:
@@ -381,6 +385,8 @@ jobs:
           # Some tests using libguestfs-tools need the fuse device to run
           docker run "$@" --device /dev/fuse:/dev/fuse -v $PWD:$(work_dir) $(ci_runner_image) /bin/bash $(work_dir)/test.sh
 
+- stage: world_build
+  jobs:
   - job: build_the_world
     displayName: 'Build the World'
     pool:
-- 
2.25.1

[PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

[Tom Rini]

Bring us to the focial-20220105 tag and rebuild our images on top of
this.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .azure-pipelines.yml    | 2 +-
 .gitlab-ci.yml          | 2 +-
 tools/docker/Dockerfile | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.azure-pipelines.yml b/.azure-pipelines.yml
index d97115668167..c0f72a811363 100644
--- a/.azure-pipelines.yml
+++ b/.azure-pipelines.yml
@@ -2,7 +2,7 @@ variables:
   windows_vm: windows-2019
   ubuntu_vm: ubuntu-18.04
   macos_vm: macOS-10.15
-  ci_runner_image: trini/u-boot-gitlab-ci-runner:focal-20211006-14Nov2021
+  ci_runner_image: trini/u-boot-gitlab-ci-runner:focal-20220105-10Jan2022
   # Add '-u 0' options for Azure pipelines, otherwise we get "permission
   # denied" error when it tries to "useradd -m -u 1001 vsts_azpcontainer",
   # since our $(ci_runner_image) user is not root.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d06cca45fd04..4c44c01e7bf5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,7 +2,7 @@
 
 # Grab our configured image.  The source for this is found at:
 # https://source.denx.de/u-boot/gitlab-ci-runner
-image: trini/u-boot-gitlab-ci-runner:focal-20211006-14Nov2021
+image: trini/u-boot-gitlab-ci-runner:focal-20220105-10Jan2022
 
 # We run some tests in different order, to catch some failures quicker.
 stages:
diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
index fb422e758813..f19e618ffba3 100644
--- a/tools/docker/Dockerfile
+++ b/tools/docker/Dockerfile
@@ -2,7 +2,7 @@
 # This Dockerfile is used to build an image containing basic stuff to be used
 # to build U-Boot and run our test suites.
 
-FROM ubuntu:focal-20211006
+FROM ubuntu:focal-20220105
 MAINTAINER Tom Rini <trini@konsulko.com>
 LABEL Description=" This image is for building U-Boot inside a container"
 
-- 
2.25.1

Re: [PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

[Fabio Estevam]

On Tue, Jan 11, 2022 at 9:14 PM Tom Rini <trini@konsulko.com> wrote:
>
> Bring us to the focial-20220105 tag and rebuild our images on top of

s/focial/focal

Re: [PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

On Tue, Jan 11, 2022 at 10:55:08PM -0300, Fabio Estevam wrote:

> On Tue, Jan 11, 2022 at 9:14 PM Tom Rini <trini@konsulko.com> wrote:
> >
> > Bring us to the focial-20220105 tag and rebuild our images on top of
> 
> s/focial/focal

Whoops, thanks.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 2/4] CI, Dockerfile: Update to latest "focal" tag

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

On Tue, Jan 11, 2022 at 07:14:29PM -0500, Tom Rini wrote:

> Bring us to the focal-20220105 tag and rebuild our images on top of
> this.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

[PATCH 3/4] binman: Have faked binaries live in the output directory.

[Tom Rini]

In general, and for Azure specifically, we need to have files created in
the output directory and cannot assume a writable source directory.
Rework the faked blob support to put the faked binary in to the output
directory and then stop the test from deleting the now non-existent
file.

Cc: Heiko Thiery <heiko.thiery@gmail.com>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 tools/binman/etype/blob.py | 4 +++-
 tools/binman/ftest.py      | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/tools/binman/etype/blob.py b/tools/binman/etype/blob.py
index 65ebb2ecf4d8..ca8023b1a0b2 100644
--- a/tools/binman/etype/blob.py
+++ b/tools/binman/etype/blob.py
@@ -39,11 +39,13 @@ class Entry_blob(Entry):
 
     def ObtainContents(self):
         if self.allow_fake and not pathlib.Path(self._filename).is_file():
+            self._filename = tools.GetOutputFilename(self._filename)
             with open(self._filename, "wb") as out:
                 out.truncate(1024)
             self.faked = True
+        else:
+            self._filename = self.GetDefaultFilename()
 
-        self._filename = self.GetDefaultFilename()
         self._pathname = tools.GetInputFilename(self._filename,
             self.external and self.section.GetAllowMissing())
         # Allow the file to be missing
diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py
index a9b7880f362f..d03ce6f05f8d 100644
--- a/tools/binman/ftest.py
+++ b/tools/binman/ftest.py
@@ -4675,7 +4675,6 @@ class TestFunctional(unittest.TestCase):
         err = stderr.getvalue()
         self.assertRegex(err,
                          "Image '.*' has faked external blobs and is non-functional: .*")
-        os.remove('binman_faking_test_blob')
 
     def testVersion(self):
         """Test we can get the binman version"""
-- 
2.25.1

Re: [PATCH 3/4] binman: Have faked binaries live in the output directory.

[Simon Glass]

Hi Tom,

On Tue, 11 Jan 2022 at 17:14, Tom Rini <trini@konsulko.com> wrote:
>
> In general, and for Azure specifically, we need to have files created in
> the output directory and cannot assume a writable source directory.
> Rework the faked blob support to put the faked binary in to the output
> directory and then stop the test from deleting the now non-existent
> file.
>
> Cc: Heiko Thiery <heiko.thiery@gmail.com>
> Cc: Simon Glass <sjg@chromium.org>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  tools/binman/etype/blob.py | 4 +++-
>  tools/binman/ftest.py      | 1 -
>  2 files changed, 3 insertions(+), 2 deletions(-)

Please see this:

https://patchwork.ozlabs.org/project/uboot/patch/20220110031413.1970836-10-sjg@chromium.org/

Regards,
Simon

Re: [PATCH 3/4] binman: Have faked binaries live in the output directory.

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

On Wed, Jan 12, 2022 at 01:01:17PM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Tue, 11 Jan 2022 at 17:14, Tom Rini <trini@konsulko.com> wrote:
> >
> > In general, and for Azure specifically, we need to have files created in
> > the output directory and cannot assume a writable source directory.
> > Rework the faked blob support to put the faked binary in to the output
> > directory and then stop the test from deleting the now non-existent
> > file.
> >
> > Cc: Heiko Thiery <heiko.thiery@gmail.com>
> > Cc: Simon Glass <sjg@chromium.org>
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> > ---
> >  tools/binman/etype/blob.py | 4 +++-
> >  tools/binman/ftest.py      | 1 -
> >  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> Please see this:
> 
> https://patchwork.ozlabs.org/project/uboot/patch/20220110031413.1970836-10-sjg@chromium.org/

Ah, I thought you might have had it fixed but I didn't read well enough.
My only concern is I would like to get Azure passing again ASAP.  Should
I just grab that patch by itself then?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 3/4] binman: Have faked binaries live in the output directory.

[Simon Glass]

Hi Tom,

On Wed, 12 Jan 2022 at 13:03, Tom Rini <trini@konsulko.com> wrote:
>
> On Wed, Jan 12, 2022 at 01:01:17PM -0700, Simon Glass wrote:
> > Hi Tom,
> >
> > On Tue, 11 Jan 2022 at 17:14, Tom Rini <trini@konsulko.com> wrote:
> > >
> > > In general, and for Azure specifically, we need to have files created in
> > > the output directory and cannot assume a writable source directory.
> > > Rework the faked blob support to put the faked binary in to the output
> > > directory and then stop the test from deleting the now non-existent
> > > file.
> > >
> > > Cc: Heiko Thiery <heiko.thiery@gmail.com>
> > > Cc: Simon Glass <sjg@chromium.org>
> > > Signed-off-by: Tom Rini <trini@konsulko.com>
> > > ---
> > >  tools/binman/etype/blob.py | 4 +++-
> > >  tools/binman/ftest.py      | 1 -
> > >  2 files changed, 3 insertions(+), 2 deletions(-)
> >
> > Please see this:
> >
> > https://patchwork.ozlabs.org/project/uboot/patch/20220110031413.1970836-10-sjg@chromium.org/
>
> Ah, I thought you might have had it fixed but I didn't read well enough.
> My only concern is I would like to get Azure passing again ASAP.  Should
> I just grab that patch by itself then?

Yes, but I just sent v2 rebased to master (two patches).

Regards,
Simon

[PATCH 4/4] .readthedocs.yml: Set our requirements file

[Tom Rini]

We provide a requirements.txt file for doc building, but had not been
configuring readthedocs to know where it is.

Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .readthedocs.yml | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/.readthedocs.yml b/.readthedocs.yml
index 44949ea239d8..c6f47b3e07e9 100644
--- a/.readthedocs.yml
+++ b/.readthedocs.yml
@@ -12,8 +12,7 @@ sphinx:
 # Optionally build your docs in additional formats such as PDF and ePub
 formats: []
 
-# Optionally set the version of Python and requirements required to build your docs
-# python:
-#   version: 3.7
-#   install:
-#     - requirements: docs/requirements.txt
+# Explicitly set the requirements file
+python:
+  install:
+    - requirements: doc/sphinx/requirements.txt
-- 
2.25.1

Re: [PATCH 4/4] .readthedocs.yml: Set our requirements file

[Heinrich Schuchardt]

On 1/12/22 01:14, Tom Rini wrote:
> We provide a requirements.txt file for doc building, but had not been
> configuring readthedocs to know where it is.
>
> Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>   .readthedocs.yml | 9 ++++----- >   1 file changed, 4 insertions(+), 5 deletions(-)
>
> diff --git a/.readthedocs.yml b/.readthedocs.yml
> index 44949ea239d8..c6f47b3e07e9 100644
> --- a/.readthedocs.yml
> +++ b/.readthedocs.yml
> @@ -12,8 +12,7 @@ sphinx:
>   # Optionally build your docs in additional formats such as PDF and ePub
>   formats: []
>
> -# Optionally set the version of Python and requirements required to build your docs
> -# python:
> -#   version: 3.7
> -#   install:
> -#     - requirements: docs/requirements.txt
> +# Explicitly set the requirements file
> +python:
> +  install:
> +    - requirements: doc/sphinx/requirements.txt

I could not get it running without install python3-six:

# .readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

build:
   os: "ubuntu-20.04"
   apt_packages:
     - python3-six
   tools:
     python: "3.9"

# Build documentation in the docs/ directory with Sphinx
sphinx:
   configuration: doc/conf.py

# Optionally build your docs in additional formats such as PDF and ePub
formats: []

python:
   install:
     - requirements: doc/sphinx/requirements.txt

Best regards

Heinrich

Re: [PATCH 1/4] ci: azure: Update to use stages

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

On Tue, Jan 11, 2022 at 07:14:28PM -0500, Tom Rini wrote:

> Follow what we do in GitLab CI where we break the jobs up in to stages
> such that if earlier and often quicker sanity tests fail we don't run
> everything else.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  .azure-pipelines.yml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)

I want to note that in my testing this didn't change the overall
run-time of the CI job.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [PATCH 1/4] ci: azure: Update to use stages

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

On Tue, Jan 11, 2022 at 07:14:28PM -0500, Tom Rini wrote:

> Follow what we do in GitLab CI where we break the jobs up in to stages
> such that if earlier and often quicker sanity tests fail we don't run
> everything else.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]