Re: [PATCH v2 07/12] Support building arbitrary Linux branch/tag/commit

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 4820 bytes --]

On Tue, Dec 09, 2025 at 04:29:02PM -0800, Stefano Stabellini wrote:
> On Thu, 4 Dec 2025, Marek Marczykowski-Górecki wrote:
> > If LINUX_URL is set, fetch LINUX_VERSION from there. Go with "git
> > init" + "git fetch" instead of "git clone" to support any of
> > branch/tag/commit.
> > 
> > This also defines optional linux-git-* jobs which will build the thing
> > if LINUX_GIT_VERSION and LINUX_GIT_URL variables are provided for the
> > pipeline.
> > 
> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> > ---
> > The script variable and job variable need to have different names, so a
> > pipeline variable won't override it for all jobs. While LINUX_VERSION /
> > LINUX_GIT_VERSION is IMO okay, I'm not very happy about LINUX_URL /
> > LINUX_GIT_URL. Any better ideas?
> 
> The problem is not LINUX_GIT_URL and LINUX_GIT_VERSION, those are good
> names. The issue is ...
> 
> > ---
> >  .gitlab-ci.yml         | 22 ++++++++++++++++++++++
> >  scripts/build-linux.sh | 18 +++++++++++++-----
> >  2 files changed, 35 insertions(+), 5 deletions(-)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 184d0b3..8d1deee 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -1,5 +1,9 @@
> >  variables:
> >    REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> > +  LINUX_GIT_VERSION:
> > +    description: "branch/tag/commit for the linux-git jobs"
> > +  LINUX_GIT_URL:
> > +    description: "git url for the linux-git jobs"
> >  
> >  stages:
> >    - build
> > @@ -53,6 +57,15 @@ linux-6.6.86-arm64:
> >    variables:
> >      LINUX_VERSION: 6.6.86
> >  
> > +linux-git-arm64:
> > +  extends: .arm64-artifacts
> > +  script: ./scripts/build-linux.sh
> > +  variables:
> > +    LINUX_VERSION: $LINUX_GIT_VERSION
> > +    LINUX_URL: $LINUX_GIT_URL
> > +  rules:
> > +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> > +
> >  #
> >  # x86_64 artifacts
> >  #
> > @@ -91,6 +104,15 @@ linux-6.12.60-x86_64:
> >    variables:
> >      LINUX_VERSION: 6.12.60
> >  
> > +linux-git-x86_64:
> > +  extends: .x86_64-artifacts
> > +  script: ./scripts/build-linux.sh
> > +  variables:
> > +    LINUX_VERSION: $LINUX_GIT_VERSION
> > +    LINUX_URL: $LINUX_GIT_URL
> > +  rules:
> > +  - if: $LINUX_GIT_VERSION && $LINUX_GIT_URL
> > +
> >  microcode-x86:
> >    extends: .x86_64-artifacts
> >    script: ./scripts/x86-microcode.sh
> > diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
> > index cf0e744..1fc96d1 100755
> > --- a/scripts/build-linux.sh
> > +++ b/scripts/build-linux.sh
> > @@ -12,11 +12,19 @@ COPYDIR="${WORKDIR}/binaries"
> >  UNAME=$(uname -m)
> >  
> >  # Build Linux
> > -MAJOR=${LINUX_VERSION%%.*}
> > -curl -fsSLO \
> > -    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > -tar xf linux-"${LINUX_VERSION}".tar.xz
> > -cd linux-"${LINUX_VERSION}"
> > +if [[ -n "${LINUX_URL}" ]]; then
> > +    mkdir linux
> > +    cd linux
> > +    git init
> > +    git fetch --depth=1 "${LINUX_URL}" "${LINUX_VERSION}"
> > +    git checkout FETCH_HEAD
> > +else
> > +    MAJOR=${LINUX_VERSION%%.*}
> > +    curl -fsSLO \
> > +        https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> > +    tar xf linux-"${LINUX_VERSION}".tar.xz
> > +    cd linux-"${LINUX_VERSION}"
> > +fi
> 
> ... the issue is detecting to fetch via git or via curl based on the
> presence of a variable called "LINUX_URL". Technically 
> 
> https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> 
> is a a valid URL as well.
> 
> So I think you should keep LINUX_GIT_URL and LINUX_GIT_VERSION named as
> they are, expose them to scripts/build-linux.sh, and detect the fetch
> program based on the presence of LINUX_GIT_URL.
> 
> Ideally, we should not have LINUX_GIT_VERSION. Instead we should have a
> a common LINUX_VERSION used in both git and curl cases.

The problem here is conflicting variables for different jobs. If you
specify a variable when starting a pipeline (either manually, or via
schedule, or via settings), the variable will be set for all the jobs.
So, to be able to schedule a pipeline with both linux-6.12.60-x86_64 and
linux-git-x86_64 (for example based on linux-next, or maybe some rc),
the pipeline variable needs to be named differently than the one used by
this script. And IMO it's more important to have clear naming
(LINUX_GIT_VERSION+LINUX_GIT_URL) at the pipeline level.

I can change this script to use arguments instead of variables to avoid
this issue, but it will result in slightly more duplication (in
.gitlab-ci.yml file).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]