[ANN] Introducing build scripts to test

[ANN] Introducing build scripts to test

[Hans Verkuil]

Hi all,

We have been working on simplifying the media maintenance, and one part of that is
standardizing on build tools, in particular to make it easier for patch submitters
to run their patches through the same set of tests that the daily build does.

This helps detect issues before you submit your patches.

I have been working since July on transforming my hackish scripts to something
that is easier to use and of better quality. While there are still a few rough
edges, I consider it good enough to have others start to use it.

To get the build scripts run:

git clone git://linuxtv.org/hverkuil/build-scripts.git

All the test builds will happen within this directory. It is completely separate
from where you do you normal development, instead you point it to where your
git repository is.

See the README contained in the build-scripts git repo for all the details on
how to set it up.

Currently the scripts expect a debian 12-based distro (likely debian 11 will work
as well). I have no idea if it works well on Red Hat or Suse. If you use one of
those distros, and you get it to work, then a patch updating the README file with
the correct list of packages to install would be welcome.

Please note that running the regression tests using virtme-ng is currently only
supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
hopefully we can support that in the future. Running regressions tests are
primarily useful when making changes to core frameworks and public APIs, and
it is possible to run them manually (see the README).

Since running this locally can take a fair amount of time, we hope to have
build servers available in the future so this can be offloaded.

To give an idea of the expected build times:

On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
(build.sh -test all) takes 39 minutes.

On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.

Regards,

	Hans

Re: [ANN] Introducing build scripts to test

[Jacopo Mondi]

Hi Hans

On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> Hi all,
>
> We have been working on simplifying the media maintenance, and one part of that is
> standardizing on build tools, in particular to make it easier for patch submitters
> to run their patches through the same set of tests that the daily build does.
>
> This helps detect issues before you submit your patches.
>
> I have been working since July on transforming my hackish scripts to something
> that is easier to use and of better quality. While there are still a few rough
> edges, I consider it good enough to have others start to use it.
>
> To get the build scripts run:
>
> git clone git://linuxtv.org/hverkuil/build-scripts.git
>
> All the test builds will happen within this directory. It is completely separate
> from where you do you normal development, instead you point it to where your
> git repository is.
>
> See the README contained in the build-scripts git repo for all the details on
> how to set it up.
>

I've been using your scripts since after ELC-E and I can tell they're
useful!

> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> those distros, and you get it to work, then a patch updating the README file with
> the correct list of packages to install would be welcome.
>

Speaking about distros, I was wondering if you still consider a
requirement to build all compiler or we should instead try to use the
distro provided ones when possible to test the distro-shipped version
?

> Please note that running the regression tests using virtme-ng is currently only
> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> hopefully we can support that in the future. Running regressions tests are
> primarily useful when making changes to core frameworks and public APIs, and
> it is possible to run them manually (see the README).
>
> Since running this locally can take a fair amount of time, we hope to have
> build servers available in the future so this can be offloaded.
>
> To give an idea of the expected build times:
>
> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> (build.sh -test all) takes 39 minutes.
>
> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
>
> Regards,
>
> 	Hans
>
> _______________________________________________
> linuxtv-ci mailing list
> linuxtv-ci@linuxtv.org
> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci

Re: [ANN] Introducing build scripts to test

[Hans Verkuil]

Hi Jacopo,

On 28/08/2023 16:05, Jacopo Mondi wrote:
> Hi Hans
> 
> On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
>> Hi all,
>>
>> We have been working on simplifying the media maintenance, and one part of that is
>> standardizing on build tools, in particular to make it easier for patch submitters
>> to run their patches through the same set of tests that the daily build does.
>>
>> This helps detect issues before you submit your patches.
>>
>> I have been working since July on transforming my hackish scripts to something
>> that is easier to use and of better quality. While there are still a few rough
>> edges, I consider it good enough to have others start to use it.
>>
>> To get the build scripts run:
>>
>> git clone git://linuxtv.org/hverkuil/build-scripts.git
>>
>> All the test builds will happen within this directory. It is completely separate
>> from where you do you normal development, instead you point it to where your
>> git repository is.
>>
>> See the README contained in the build-scripts git repo for all the details on
>> how to set it up.
>>
> 
> I've been using your scripts since after ELC-E and I can tell they're
> useful!
> 
>> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
>> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
>> those distros, and you get it to work, then a patch updating the README file with
>> the correct list of packages to install would be welcome.
>>
> 
> Speaking about distros, I was wondering if you still consider a
> requirement to build all compiler or we should instead try to use the
> distro provided ones when possible to test the distro-shipped version
> ?

I strongly believe we should build the cross compilers. The reason is that
otherwise you get a wide variety of compiler versions, each with typically
different compiler warnings. It's a pain for a developer to see different
warnings than the person that merges those patches.

It's a a regular problem that the daily build sees different warnings than
you do locally with the distro's compiler.

Doing it this way also makes it easier to upgrade to the latest compiler
version, certainly quicker than a distro would do.

It's about reproducibility, really.

Regards,

	Hans

> 
>> Please note that running the regression tests using virtme-ng is currently only
>> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
>> hopefully we can support that in the future. Running regressions tests are
>> primarily useful when making changes to core frameworks and public APIs, and
>> it is possible to run them manually (see the README).
>>
>> Since running this locally can take a fair amount of time, we hope to have
>> build servers available in the future so this can be offloaded.
>>
>> To give an idea of the expected build times:
>>
>> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
>> (build.sh -test all) takes 39 minutes.
>>
>> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
>>
>> Regards,
>>
>> 	Hans
>>
>> _______________________________________________
>> linuxtv-ci mailing list
>> linuxtv-ci@linuxtv.org
>> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci

Re: [ANN] Introducing build scripts to test

[Laurent Pinchart]

On Mon, Aug 28, 2023 at 04:05:53PM +0200, Jacopo Mondi wrote:
> On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> > Hi all,
> >
> > We have been working on simplifying the media maintenance, and one part of that is
> > standardizing on build tools, in particular to make it easier for patch submitters
> > to run their patches through the same set of tests that the daily build does.
> >
> > This helps detect issues before you submit your patches.
> >
> > I have been working since July on transforming my hackish scripts to something
> > that is easier to use and of better quality. While there are still a few rough
> > edges, I consider it good enough to have others start to use it.
> >
> > To get the build scripts run:
> >
> > git clone git://linuxtv.org/hverkuil/build-scripts.git
> >
> > All the test builds will happen within this directory. It is completely separate
> > from where you do you normal development, instead you point it to where your
> > git repository is.
> >
> > See the README contained in the build-scripts git repo for all the details on
> > how to set it up.
> 
> I've been using your scripts since after ELC-E and I can tell they're
> useful!
> 
> > Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> > as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> > those distros, and you get it to work, then a patch updating the README file with
> > the correct list of packages to install would be welcome.
> 
> Speaking about distros, I was wondering if you still consider a
> requirement to build all compiler or we should instead try to use the
> distro provided ones when possible to test the distro-shipped version
> ?

I have a large set of cross-compilers (built by buildroot), and I would
like to use those instead of having to build separate ones.

> > Please note that running the regression tests using virtme-ng is currently only
> > supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> > hopefully we can support that in the future. Running regressions tests are
> > primarily useful when making changes to core frameworks and public APIs, and
> > it is possible to run them manually (see the README).
> >
> > Since running this locally can take a fair amount of time, we hope to have
> > build servers available in the future so this can be offloaded.
> >
> > To give an idea of the expected build times:
> >
> > On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> > (build.sh -test all) takes 39 minutes.
> >
> > On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.

I'll try on my main development machine, which is a quad core i7-8550U.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Introducing build scripts to test

[Laurent Pinchart]

On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> On 28/08/2023 16:05, Jacopo Mondi wrote:
> > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> >> Hi all,
> >>
> >> We have been working on simplifying the media maintenance, and one part of that is
> >> standardizing on build tools, in particular to make it easier for patch submitters
> >> to run their patches through the same set of tests that the daily build does.
> >>
> >> This helps detect issues before you submit your patches.
> >>
> >> I have been working since July on transforming my hackish scripts to something
> >> that is easier to use and of better quality. While there are still a few rough
> >> edges, I consider it good enough to have others start to use it.
> >>
> >> To get the build scripts run:
> >>
> >> git clone git://linuxtv.org/hverkuil/build-scripts.git
> >>
> >> All the test builds will happen within this directory. It is completely separate
> >> from where you do you normal development, instead you point it to where your
> >> git repository is.
> >>
> >> See the README contained in the build-scripts git repo for all the details on
> >> how to set it up.
> >>
> > 
> > I've been using your scripts since after ELC-E and I can tell they're
> > useful!
> > 
> >> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> >> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> >> those distros, and you get it to work, then a patch updating the README file with
> >> the correct list of packages to install would be welcome.
> > 
> > Speaking about distros, I was wondering if you still consider a
> > requirement to build all compiler or we should instead try to use the
> > distro provided ones when possible to test the distro-shipped version
> > ?
> 
> I strongly believe we should build the cross compilers. The reason is that
> otherwise you get a wide variety of compiler versions, each with typically
> different compiler warnings. It's a pain for a developer to see different
> warnings than the person that merges those patches.
> 
> It's a a regular problem that the daily build sees different warnings than
> you do locally with the distro's compiler.
> 
> Doing it this way also makes it easier to upgrade to the latest compiler
> version, certainly quicker than a distro would do.
> 
> It's about reproducibility, really.

There's value in testing with different compiler versions though. The
kernel's documented minimum gcc version is v5.1 at the moment. I
certainly don't want to build myself with all versions between v5.1 and
v13.2, but collectively we could cover more ground.

Regardless of this, I already have recent cross compilers (built with
buildroot) for ARM and ARM64, and I'd rather use those than duplicating
compilers. Anything that consumes extra disk space is a serious
hinderance.

> >> Please note that running the regression tests using virtme-ng is currently only
> >> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> >> hopefully we can support that in the future. Running regressions tests are
> >> primarily useful when making changes to core frameworks and public APIs, and
> >> it is possible to run them manually (see the README).
> >>
> >> Since running this locally can take a fair amount of time, we hope to have
> >> build servers available in the future so this can be offloaded.
> >>
> >> To give an idea of the expected build times:
> >>
> >> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> >> (build.sh -test all) takes 39 minutes.
> >>
> >> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Introducing build scripts to test

[Chen-Yu Tsai]

On Mon, Aug 28, 2023 at 10:16 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
>
> Hi Jacopo,
>
> On 28/08/2023 16:05, Jacopo Mondi wrote:
> > Hi Hans
> >
> > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> >> Hi all,
> >>
> >> We have been working on simplifying the media maintenance, and one part of that is
> >> standardizing on build tools, in particular to make it easier for patch submitters
> >> to run their patches through the same set of tests that the daily build does.
> >>
> >> This helps detect issues before you submit your patches.
> >>
> >> I have been working since July on transforming my hackish scripts to something
> >> that is easier to use and of better quality. While there are still a few rough
> >> edges, I consider it good enough to have others start to use it.
> >>
> >> To get the build scripts run:
> >>
> >> git clone git://linuxtv.org/hverkuil/build-scripts.git
> >>
> >> All the test builds will happen within this directory. It is completely separate
> >> from where you do you normal development, instead you point it to where your
> >> git repository is.
> >>
> >> See the README contained in the build-scripts git repo for all the details on
> >> how to set it up.
> >>
> >
> > I've been using your scripts since after ELC-E and I can tell they're
> > useful!
> >
> >> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> >> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> >> those distros, and you get it to work, then a patch updating the README file with
> >> the correct list of packages to install would be welcome.
> >>
> >
> > Speaking about distros, I was wondering if you still consider a
> > requirement to build all compiler or we should instead try to use the
> > distro provided ones when possible to test the distro-shipped version
> > ?
>
> I strongly believe we should build the cross compilers. The reason is that
> otherwise you get a wide variety of compiler versions, each with typically
> different compiler warnings. It's a pain for a developer to see different
> warnings than the person that merges those patches.
>
> It's a a regular problem that the daily build sees different warnings than
> you do locally with the distro's compiler.
>
> Doing it this way also makes it easier to upgrade to the latest compiler
> version, certainly quicker than a distro would do.
>
> It's about reproducibility, really.

If it's stability and/or reproducibility we want, maybe we could utilize the
prebuilt toolchains from Linaro or Bootlin? That should cover a good array
of versions, while saving time by not having to build them.

ChenYu


> >> Please note that running the regression tests using virtme-ng is currently only
> >> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> >> hopefully we can support that in the future. Running regressions tests are
> >> primarily useful when making changes to core frameworks and public APIs, and
> >> it is possible to run them manually (see the README).
> >>
> >> Since running this locally can take a fair amount of time, we hope to have
> >> build servers available in the future so this can be offloaded.
> >>
> >> To give an idea of the expected build times:
> >>
> >> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> >> (build.sh -test all) takes 39 minutes.
> >>
> >> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> >>
> >> Regards,
> >>
> >>      Hans
> >>
> >> _______________________________________________
> >> linuxtv-ci mailing list
> >> linuxtv-ci@linuxtv.org
> >> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci
>

Re: [ANN] Introducing build scripts to test

[Hans Verkuil]

On 28/08/2023 16:26, Laurent Pinchart wrote:
> On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
>> On 28/08/2023 16:05, Jacopo Mondi wrote:
>>> On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
>>>> Hi all,
>>>>
>>>> We have been working on simplifying the media maintenance, and one part of that is
>>>> standardizing on build tools, in particular to make it easier for patch submitters
>>>> to run their patches through the same set of tests that the daily build does.
>>>>
>>>> This helps detect issues before you submit your patches.
>>>>
>>>> I have been working since July on transforming my hackish scripts to something
>>>> that is easier to use and of better quality. While there are still a few rough
>>>> edges, I consider it good enough to have others start to use it.
>>>>
>>>> To get the build scripts run:
>>>>
>>>> git clone git://linuxtv.org/hverkuil/build-scripts.git
>>>>
>>>> All the test builds will happen within this directory. It is completely separate
>>>> from where you do you normal development, instead you point it to where your
>>>> git repository is.
>>>>
>>>> See the README contained in the build-scripts git repo for all the details on
>>>> how to set it up.
>>>>
>>>
>>> I've been using your scripts since after ELC-E and I can tell they're
>>> useful!
>>>
>>>> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
>>>> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
>>>> those distros, and you get it to work, then a patch updating the README file with
>>>> the correct list of packages to install would be welcome.
>>>
>>> Speaking about distros, I was wondering if you still consider a
>>> requirement to build all compiler or we should instead try to use the
>>> distro provided ones when possible to test the distro-shipped version
>>> ?
>>
>> I strongly believe we should build the cross compilers. The reason is that
>> otherwise you get a wide variety of compiler versions, each with typically
>> different compiler warnings. It's a pain for a developer to see different
>> warnings than the person that merges those patches.
>>
>> It's a a regular problem that the daily build sees different warnings than
>> you do locally with the distro's compiler.
>>
>> Doing it this way also makes it easier to upgrade to the latest compiler
>> version, certainly quicker than a distro would do.
>>
>> It's about reproducibility, really.
> 
> There's value in testing with different compiler versions though. The
> kernel's documented minimum gcc version is v5.1 at the moment. I
> certainly don't want to build myself with all versions between v5.1 and
> v13.2, but collectively we could cover more ground.
> 
> Regardless of this, I already have recent cross compilers (built with
> buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> compilers. Anything that consumes extra disk space is a serious
> hinderance.

Feel free, but you run the risk that your PR is rejected because when I
run with these compiler versions I see new warnings. The whole point is
to be able to do the same tests before you make the PR to reduce the risk
of having to make a v2.

FYI: the cross directory takes about 10 GB on my system. That can be
streamlined a bit by deleting some temporary directories needed while
building, probably to something closer to 5-6 GB.

Regards,

	Hans

> 
>>>> Please note that running the regression tests using virtme-ng is currently only
>>>> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
>>>> hopefully we can support that in the future. Running regressions tests are
>>>> primarily useful when making changes to core frameworks and public APIs, and
>>>> it is possible to run them manually (see the README).
>>>>
>>>> Since running this locally can take a fair amount of time, we hope to have
>>>> build servers available in the future so this can be offloaded.
>>>>
>>>> To give an idea of the expected build times:
>>>>
>>>> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
>>>> (build.sh -test all) takes 39 minutes.
>>>>
>>>> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> 

Re: [ANN] Introducing build scripts to test

[Laurent Pinchart]

On Mon, Aug 28, 2023 at 04:38:32PM +0200, Hans Verkuil wrote:
> On 28/08/2023 16:26, Laurent Pinchart wrote:
> > On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> >> On 28/08/2023 16:05, Jacopo Mondi wrote:
> >>> On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> >>>> Hi all,
> >>>>
> >>>> We have been working on simplifying the media maintenance, and one part of that is
> >>>> standardizing on build tools, in particular to make it easier for patch submitters
> >>>> to run their patches through the same set of tests that the daily build does.
> >>>>
> >>>> This helps detect issues before you submit your patches.
> >>>>
> >>>> I have been working since July on transforming my hackish scripts to something
> >>>> that is easier to use and of better quality. While there are still a few rough
> >>>> edges, I consider it good enough to have others start to use it.
> >>>>
> >>>> To get the build scripts run:
> >>>>
> >>>> git clone git://linuxtv.org/hverkuil/build-scripts.git
> >>>>
> >>>> All the test builds will happen within this directory. It is completely separate
> >>>> from where you do you normal development, instead you point it to where your
> >>>> git repository is.
> >>>>
> >>>> See the README contained in the build-scripts git repo for all the details on
> >>>> how to set it up.
> >>>>
> >>>
> >>> I've been using your scripts since after ELC-E and I can tell they're
> >>> useful!
> >>>
> >>>> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> >>>> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> >>>> those distros, and you get it to work, then a patch updating the README file with
> >>>> the correct list of packages to install would be welcome.
> >>>
> >>> Speaking about distros, I was wondering if you still consider a
> >>> requirement to build all compiler or we should instead try to use the
> >>> distro provided ones when possible to test the distro-shipped version
> >>> ?
> >>
> >> I strongly believe we should build the cross compilers. The reason is that
> >> otherwise you get a wide variety of compiler versions, each with typically
> >> different compiler warnings. It's a pain for a developer to see different
> >> warnings than the person that merges those patches.
> >>
> >> It's a a regular problem that the daily build sees different warnings than
> >> you do locally with the distro's compiler.
> >>
> >> Doing it this way also makes it easier to upgrade to the latest compiler
> >> version, certainly quicker than a distro would do.
> >>
> >> It's about reproducibility, really.
> > 
> > There's value in testing with different compiler versions though. The
> > kernel's documented minimum gcc version is v5.1 at the moment. I
> > certainly don't want to build myself with all versions between v5.1 and
> > v13.2, but collectively we could cover more ground.
> > 
> > Regardless of this, I already have recent cross compilers (built with
> > buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> > compilers. Anything that consumes extra disk space is a serious
> > hinderance.
> 
> Feel free, but you run the risk that your PR is rejected because when I
> run with these compiler versions I see new warnings. The whole point is
> to be able to do the same tests before you make the PR to reduce the risk
> of having to make a v2.
> 
> FYI: the cross directory takes about 10 GB on my system. That can be
> streamlined a bit by deleting some temporary directories needed while
> building, probably to something closer to 5-6 GB.

It may not be huge by itself, but it quickly adds up when you need to
maintain multiple userspace cross-built enviroments, including Chrome
OS, Android, Yocto, ... :-( I have half a TB of disk on my main
development machine, and I would need at least 4 times that to cover my
current needs comfortably.

> >>>> Please note that running the regression tests using virtme-ng is currently only
> >>>> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> >>>> hopefully we can support that in the future. Running regressions tests are
> >>>> primarily useful when making changes to core frameworks and public APIs, and
> >>>> it is possible to run them manually (see the README).
> >>>>
> >>>> Since running this locally can take a fair amount of time, we hope to have
> >>>> build servers available in the future so this can be offloaded.
> >>>>
> >>>> To give an idea of the expected build times:
> >>>>
> >>>> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> >>>> (build.sh -test all) takes 39 minutes.
> >>>>
> >>>> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.

-- 
Regards,

Laurent Pinchart

Re: [ANN] Introducing build scripts to test

[Mauro Carvalho Chehab]

Em Mon, 28 Aug 2023 16:14:56 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> Hi Jacopo,
> 
> On 28/08/2023 16:05, Jacopo Mondi wrote:
> > Hi Hans
> > 
> > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:  
> >> Hi all,
> >>
> >> We have been working on simplifying the media maintenance, and one part of that is
> >> standardizing on build tools, in particular to make it easier for patch submitters
> >> to run their patches through the same set of tests that the daily build does.
> >>
> >> This helps detect issues before you submit your patches.
> >>
> >> I have been working since July on transforming my hackish scripts to something
> >> that is easier to use and of better quality. While there are still a few rough
> >> edges, I consider it good enough to have others start to use it.
> >>
> >> To get the build scripts run:
> >>
> >> git clone git://linuxtv.org/hverkuil/build-scripts.git
> >>
> >> All the test builds will happen within this directory. It is completely separate
> >> from where you do you normal development, instead you point it to where your
> >> git repository is.
> >>
> >> See the README contained in the build-scripts git repo for all the details on
> >> how to set it up.
> >>  
> > 
> > I've been using your scripts since after ELC-E and I can tell they're
> > useful!
> >   
> >> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> >> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> >> those distros, and you get it to work, then a patch updating the README file with
> >> the correct list of packages to install would be welcome.
> >>  
> > 
> > Speaking about distros, I was wondering if you still consider a
> > requirement to build all compiler or we should instead try to use the
> > distro provided ones when possible to test the distro-shipped version
> > ?  
> 
> I strongly believe we should build the cross compilers. The reason is that
> otherwise you get a wide variety of compiler versions, each with typically
> different compiler warnings. It's a pain for a developer to see different
> warnings than the person that merges those patches.

I disagree. We don't want to break build for any supported compiler version. 
With Werror enabled, this can be tough. So, having people using different
versions is actually a good thing.

> It's a a regular problem that the daily build sees different warnings than
> you do locally with the distro's compiler.

Sparse (or smatch) will produce different warnings anyway, even with the
same version, depending on how many times you run it, and if you create
a database or not.

> 
> Doing it this way also makes it easier to upgrade to the latest compiler
> version, certainly quicker than a distro would do.

Usually, Fedora is very quick upgrading to the latest compiler. Yet,
I don't think this should be the main goal. I bet people will be a lot
pickier if we break compilation against a gcc version on Ubuntu,
Suse, Fedora and variants than if you break compilation about an upcoming
gcc version that are used by just a bunch of early testers.

Btw, there are early testers for both gcc and clang that periodically
test them during compilers -rc stages. They'll likely identify issues
before us (as it already happened in the past).

> 
> It's about reproducibility, really.
> 
> Regards,
> 
> 	Hans
> 
> >   
> >> Please note that running the regression tests using virtme-ng is currently only
> >> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> >> hopefully we can support that in the future. Running regressions tests are
> >> primarily useful when making changes to core frameworks and public APIs, and
> >> it is possible to run them manually (see the README).
> >>
> >> Since running this locally can take a fair amount of time, we hope to have
> >> build servers available in the future so this can be offloaded.
> >>
> >> To give an idea of the expected build times:
> >>
> >> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> >> (build.sh -test all) takes 39 minutes.
> >>
> >> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> >>
> >> Regards,
> >>
> >> 	Hans
> >>
> >> _______________________________________________
> >> linuxtv-ci mailing list
> >> linuxtv-ci@linuxtv.org
> >> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci  
> 
> 
> _______________________________________________
> linuxtv-ci mailing list
> linuxtv-ci@linuxtv.org
> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci

Re: [ANN] Introducing build scripts to test

[Hans Verkuil]

Hi Mauro,

On 28/08/2023 19:46, Mauro Carvalho Chehab wrote:
> Em Mon, 28 Aug 2023 16:14:56 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> Hi Jacopo,
>>
>> On 28/08/2023 16:05, Jacopo Mondi wrote:
>>> Hi Hans
>>>
>>> On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:  
>>>> Hi all,
>>>>
>>>> We have been working on simplifying the media maintenance, and one part of that is
>>>> standardizing on build tools, in particular to make it easier for patch submitters
>>>> to run their patches through the same set of tests that the daily build does.
>>>>
>>>> This helps detect issues before you submit your patches.
>>>>
>>>> I have been working since July on transforming my hackish scripts to something
>>>> that is easier to use and of better quality. While there are still a few rough
>>>> edges, I consider it good enough to have others start to use it.
>>>>
>>>> To get the build scripts run:
>>>>
>>>> git clone git://linuxtv.org/hverkuil/build-scripts.git
>>>>
>>>> All the test builds will happen within this directory. It is completely separate
>>>> from where you do you normal development, instead you point it to where your
>>>> git repository is.
>>>>
>>>> See the README contained in the build-scripts git repo for all the details on
>>>> how to set it up.
>>>>  
>>>
>>> I've been using your scripts since after ELC-E and I can tell they're
>>> useful!
>>>   
>>>> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
>>>> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
>>>> those distros, and you get it to work, then a patch updating the README file with
>>>> the correct list of packages to install would be welcome.
>>>>  
>>>
>>> Speaking about distros, I was wondering if you still consider a
>>> requirement to build all compiler or we should instead try to use the
>>> distro provided ones when possible to test the distro-shipped version
>>> ?  
>>
>> I strongly believe we should build the cross compilers. The reason is that
>> otherwise you get a wide variety of compiler versions, each with typically
>> different compiler warnings. It's a pain for a developer to see different
>> warnings than the person that merges those patches.
> 
> I disagree. We don't want to break build for any supported compiler version. 
> With Werror enabled, this can be tough. So, having people using different
> versions is actually a good thing.

Of course, but if you want to get your code into the kernel it is good if you
can reproduce whatever system the subsystem maintainers use to avoid wasting
time. That's the reason for using the same toolchain. The other reason is that
it allows us to switch to a new compiler quickly. Especially a new gcc major
release will often introduce a bunch of new warnings, and it is nice not having
to wait for distros to pick up a new version.

For day to day development you use whatever compiler version you want, it
is not intended that you use these cross compilers for your normal development
process. It is purely to ensure you have the same environment as the CI
environment.

> 
>> It's a a regular problem that the daily build sees different warnings than
>> you do locally with the distro's compiler.
> 
> Sparse (or smatch) will produce different warnings anyway, even with the
> same version, depending on how many times you run it, and if you create
> a database or not.
> 
>>
>> Doing it this way also makes it easier to upgrade to the latest compiler
>> version, certainly quicker than a distro would do.
> 
> Usually, Fedora is very quick upgrading to the latest compiler. Yet,
> I don't think this should be the main goal. I bet people will be a lot
> pickier if we break compilation against a gcc version on Ubuntu,
> Suse, Fedora and variants than if you break compilation about an upcoming
> gcc version that are used by just a bunch of early testers.

I only use released gcc versions, I never use pre-release versions. I leave
that to those early testers.

And this is not about breaking compilation: it is very, very rare that
a source compiles for a newer version but not for an older version.

It is primarily about compiler warnings as there is much more variation
there between compiler versions. We want to keep the number of warnings
at 0, both for -Werrors and for making it easy to detect new warnings.

By and large, newer compilers will have more warnings than older compilers,
so using the latest released compiler has been my preference over the years
that I ran the daily build.

Regards,

	Hans

> 
> Btw, there are early testers for both gcc and clang that periodically
> test them during compilers -rc stages. They'll likely identify issues
> before us (as it already happened in the past).
> 
>>
>> It's about reproducibility, really.
>>
>> Regards,
>>
>> 	Hans
>>
>>>   
>>>> Please note that running the regression tests using virtme-ng is currently only
>>>> supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
>>>> hopefully we can support that in the future. Running regressions tests are
>>>> primarily useful when making changes to core frameworks and public APIs, and
>>>> it is possible to run them manually (see the README).
>>>>
>>>> Since running this locally can take a fair amount of time, we hope to have
>>>> build servers available in the future so this can be offloaded.
>>>>
>>>> To give an idea of the expected build times:
>>>>
>>>> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
>>>> (build.sh -test all) takes 39 minutes.
>>>>
>>>> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
>>>>
>>>> Regards,
>>>>
>>>> 	Hans
>>>>
>>>> _______________________________________________
>>>> linuxtv-ci mailing list
>>>> linuxtv-ci@linuxtv.org
>>>> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci  
>>
>>
>> _______________________________________________
>> linuxtv-ci mailing list
>> linuxtv-ci@linuxtv.org
>> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci

Re: [ANN] Introducing build scripts to test

[Sakari Ailus]

Hi Laurent, Hans,

On Mon, Aug 28, 2023 at 05:26:22PM +0300, Laurent Pinchart wrote:
> On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> > On 28/08/2023 16:05, Jacopo Mondi wrote:
> > > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> > >> Hi all,
> > >>
> > >> We have been working on simplifying the media maintenance, and one part of that is
> > >> standardizing on build tools, in particular to make it easier for patch submitters
> > >> to run their patches through the same set of tests that the daily build does.
> > >>
> > >> This helps detect issues before you submit your patches.
> > >>
> > >> I have been working since July on transforming my hackish scripts to something
> > >> that is easier to use and of better quality. While there are still a few rough
> > >> edges, I consider it good enough to have others start to use it.
> > >>
> > >> To get the build scripts run:
> > >>
> > >> git clone git://linuxtv.org/hverkuil/build-scripts.git
> > >>
> > >> All the test builds will happen within this directory. It is completely separate
> > >> from where you do you normal development, instead you point it to where your
> > >> git repository is.
> > >>
> > >> See the README contained in the build-scripts git repo for all the details on
> > >> how to set it up.
> > >>
> > > 
> > > I've been using your scripts since after ELC-E and I can tell they're
> > > useful!
> > > 
> > >> Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> > >> as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> > >> those distros, and you get it to work, then a patch updating the README file with
> > >> the correct list of packages to install would be welcome.
> > > 
> > > Speaking about distros, I was wondering if you still consider a
> > > requirement to build all compiler or we should instead try to use the
> > > distro provided ones when possible to test the distro-shipped version
> > > ?
> > 
> > I strongly believe we should build the cross compilers. The reason is that
> > otherwise you get a wide variety of compiler versions, each with typically
> > different compiler warnings. It's a pain for a developer to see different
> > warnings than the person that merges those patches.
> > 
> > It's a a regular problem that the daily build sees different warnings than
> > you do locally with the distro's compiler.
> > 
> > Doing it this way also makes it easier to upgrade to the latest compiler
> > version, certainly quicker than a distro would do.
> > 
> > It's about reproducibility, really.
> 
> There's value in testing with different compiler versions though. The
> kernel's documented minimum gcc version is v5.1 at the moment. I
> certainly don't want to build myself with all versions between v5.1 and
> v13.2, but collectively we could cover more ground.
> 
> Regardless of this, I already have recent cross compilers (built with
> buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> compilers. Anything that consumes extra disk space is a serious
> hinderance.

I have to say nearly all issues I've come across I have not seen myself in
my own (trivial) tests are related to either different architecture or
different kernel configuration.

There have been a couple of issues that were related to compiling the
kernel with LLVM, too, but they are much more rare. I suppose there may
have been some related to different GCC versions but I don't remember any
over the last ~ 5 years. :-)

So IMO varying configuration and building for more architectures is
beneficial, possibly compilers, too, but for different compiler versions
not so much.

-- 
Kind regards,

Sakari Ailus

Re: [ANN] Introducing build scripts to test

[Nicolas Dufresne]

Le lundi 28 août 2023 à 17:45 +0300, Laurent Pinchart a écrit :
> On Mon, Aug 28, 2023 at 04:38:32PM +0200, Hans Verkuil wrote:
> > On 28/08/2023 16:26, Laurent Pinchart wrote:
> > > On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> > > > On 28/08/2023 16:05, Jacopo Mondi wrote:
> > > > > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> > > > > > Hi all,
> > > > > > 
> > > > > > We have been working on simplifying the media maintenance, and one part of that is
> > > > > > standardizing on build tools, in particular to make it easier for patch submitters
> > > > > > to run their patches through the same set of tests that the daily build does.
> > > > > > 
> > > > > > This helps detect issues before you submit your patches.
> > > > > > 
> > > > > > I have been working since July on transforming my hackish scripts to something
> > > > > > that is easier to use and of better quality. While there are still a few rough
> > > > > > edges, I consider it good enough to have others start to use it.
> > > > > > 
> > > > > > To get the build scripts run:
> > > > > > 
> > > > > > git clone git://linuxtv.org/hverkuil/build-scripts.git
> > > > > > 
> > > > > > All the test builds will happen within this directory. It is completely separate
> > > > > > from where you do you normal development, instead you point it to where your
> > > > > > git repository is.
> > > > > > 
> > > > > > See the README contained in the build-scripts git repo for all the details on
> > > > > > how to set it up.
> > > > > > 
> > > > > 
> > > > > I've been using your scripts since after ELC-E and I can tell they're
> > > > > useful!
> > > > > 
> > > > > > Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> > > > > > as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> > > > > > those distros, and you get it to work, then a patch updating the README file with
> > > > > > the correct list of packages to install would be welcome.
> > > > > 
> > > > > Speaking about distros, I was wondering if you still consider a
> > > > > requirement to build all compiler or we should instead try to use the
> > > > > distro provided ones when possible to test the distro-shipped version
> > > > > ?
> > > > 
> > > > I strongly believe we should build the cross compilers. The reason is that
> > > > otherwise you get a wide variety of compiler versions, each with typically
> > > > different compiler warnings. It's a pain for a developer to see different
> > > > warnings than the person that merges those patches.
> > > > 
> > > > It's a a regular problem that the daily build sees different warnings than
> > > > you do locally with the distro's compiler.
> > > > 
> > > > Doing it this way also makes it easier to upgrade to the latest compiler
> > > > version, certainly quicker than a distro would do.
> > > > 
> > > > It's about reproducibility, really.
> > > 
> > > There's value in testing with different compiler versions though. The
> > > kernel's documented minimum gcc version is v5.1 at the moment. I
> > > certainly don't want to build myself with all versions between v5.1 and
> > > v13.2, but collectively we could cover more ground.
> > > 
> > > Regardless of this, I already have recent cross compilers (built with
> > > buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> > > compilers. Anything that consumes extra disk space is a serious
> > > hinderance.
> > 
> > Feel free, but you run the risk that your PR is rejected because when I
> > run with these compiler versions I see new warnings. The whole point is
> > to be able to do the same tests before you make the PR to reduce the risk
> > of having to make a v2.
> > 
> > FYI: the cross directory takes about 10 GB on my system. That can be
> > streamlined a bit by deleting some temporary directories needed while
> > building, probably to something closer to 5-6 GB.
> 
> It may not be huge by itself, but it quickly adds up when you need to
> maintain multiple userspace cross-built enviroments, including Chrome
> OS, Android, Yocto, ... :-( I have half a TB of disk on my main
> development machine, and I would need at least 4 times that to cover my
> current needs comfortably.

I suppose this is irrelevant if you have a mean to send you PR to a machine that
will validate it for you. This is something I'd like to see happen in the
future. Considering the very tiny number of devs doing PR, a first step could be
to have a shared server in the cloud with the appropriate distro, compilers, and
just one more script to run test against a PR URI. This is quite minimal infra
and maintenance, since this is identical to what everyone may have locally, plus
having to install an SSH server and manages keys. Of course, scripts remains,
and can be used locally with of course the possible oops of running something
slightly different, but with the benefit of not having to "push somewhere" to
validate.

We could also have an FDO project and use their infra, which would be a lot
nicer imho, but we can't enter FDO without bringing matching sponsorship for the
resources we'd be using. At least we should ask first, not serve ourself there.

Nicolas
> 
> > > > > > Please note that running the regression tests using virtme-ng is currently only
> > > > > > supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> > > > > > hopefully we can support that in the future. Running regressions tests are
> > > > > > primarily useful when making changes to core frameworks and public APIs, and
> > > > > > it is possible to run them manually (see the README).
> > > > > > 
> > > > > > Since running this locally can take a fair amount of time, we hope to have
> > > > > > build servers available in the future so this can be offloaded.
> > > > > > 
> > > > > > To give an idea of the expected build times:
> > > > > > 
> > > > > > On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> > > > > > (build.sh -test all) takes 39 minutes.
> > > > > > 
> > > > > > On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> 

Re: [ANN] Introducing build scripts to test

[Ricardo Ribalda]

Hi Nicolas

On Fri, 8 Sept 2023 at 17:44, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le lundi 28 août 2023 à 17:45 +0300, Laurent Pinchart a écrit :
> > On Mon, Aug 28, 2023 at 04:38:32PM +0200, Hans Verkuil wrote:
> > > On 28/08/2023 16:26, Laurent Pinchart wrote:
> > > > On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> > > > > On 28/08/2023 16:05, Jacopo Mondi wrote:
> > > > > > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have been working on simplifying the media maintenance, and one part of that is
> > > > > > > standardizing on build tools, in particular to make it easier for patch submitters
> > > > > > > to run their patches through the same set of tests that the daily build does.
> > > > > > >
> > > > > > > This helps detect issues before you submit your patches.
> > > > > > >
> > > > > > > I have been working since July on transforming my hackish scripts to something
> > > > > > > that is easier to use and of better quality. While there are still a few rough
> > > > > > > edges, I consider it good enough to have others start to use it.
> > > > > > >
> > > > > > > To get the build scripts run:
> > > > > > >
> > > > > > > git clone git://linuxtv.org/hverkuil/build-scripts.git
> > > > > > >
> > > > > > > All the test builds will happen within this directory. It is completely separate
> > > > > > > from where you do you normal development, instead you point it to where your
> > > > > > > git repository is.
> > > > > > >
> > > > > > > See the README contained in the build-scripts git repo for all the details on
> > > > > > > how to set it up.
> > > > > > >
> > > > > >
> > > > > > I've been using your scripts since after ELC-E and I can tell they're
> > > > > > useful!
> > > > > >
> > > > > > > Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> > > > > > > as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> > > > > > > those distros, and you get it to work, then a patch updating the README file with
> > > > > > > the correct list of packages to install would be welcome.
> > > > > >
> > > > > > Speaking about distros, I was wondering if you still consider a
> > > > > > requirement to build all compiler or we should instead try to use the
> > > > > > distro provided ones when possible to test the distro-shipped version
> > > > > > ?
> > > > >
> > > > > I strongly believe we should build the cross compilers. The reason is that
> > > > > otherwise you get a wide variety of compiler versions, each with typically
> > > > > different compiler warnings. It's a pain for a developer to see different
> > > > > warnings than the person that merges those patches.
> > > > >
> > > > > It's a a regular problem that the daily build sees different warnings than
> > > > > you do locally with the distro's compiler.
> > > > >
> > > > > Doing it this way also makes it easier to upgrade to the latest compiler
> > > > > version, certainly quicker than a distro would do.
> > > > >
> > > > > It's about reproducibility, really.
> > > >
> > > > There's value in testing with different compiler versions though. The
> > > > kernel's documented minimum gcc version is v5.1 at the moment. I
> > > > certainly don't want to build myself with all versions between v5.1 and
> > > > v13.2, but collectively we could cover more ground.
> > > >
> > > > Regardless of this, I already have recent cross compilers (built with
> > > > buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> > > > compilers. Anything that consumes extra disk space is a serious
> > > > hinderance.
> > >
> > > Feel free, but you run the risk that your PR is rejected because when I
> > > run with these compiler versions I see new warnings. The whole point is
> > > to be able to do the same tests before you make the PR to reduce the risk
> > > of having to make a v2.
> > >
> > > FYI: the cross directory takes about 10 GB on my system. That can be
> > > streamlined a bit by deleting some temporary directories needed while
> > > building, probably to something closer to 5-6 GB.
> >
> > It may not be huge by itself, but it quickly adds up when you need to
> > maintain multiple userspace cross-built enviroments, including Chrome
> > OS, Android, Yocto, ... :-( I have half a TB of disk on my main
> > development machine, and I would need at least 4 times that to cover my
> > current needs comfortably.
>
> I suppose this is irrelevant if you have a mean to send you PR to a machine that
> will validate it for you. This is something I'd like to see happen in the
> future. Considering the very tiny number of devs doing PR, a first step could be
> to have a shared server in the cloud with the appropriate distro, compilers, and
> just one more script to run test against a PR URI. This is quite minimal infra
> and maintenance, since this is identical to what everyone may have locally, plus
> having to install an SSH server and manages keys. Of course, scripts remains,
> and can be used locally with of course the possible oops of running something
> slightly different, but with the benefit of not having to "push somewhere" to
> validate.

This is something we have just started to work on:

https://docs.google.com/document/d/1HTpk73qqfZLjrrvUwbd4g11wd8e9TkXTXvz5FZBd52g/edit#heading=h.4v9g2243whew

The plan is to be able to test locally and in gitlab.

>
> We could also have an FDO project and use their infra, which would be a lot
> nicer imho, but we can't enter FDO without bringing matching sponsorship for the
> resources we'd be using. At least we should ask first, not serve ourself there.

I aleady got some some google cloud sponsorship for the project if we
can land it ;)


>
> Nicolas
> >
> > > > > > > Please note that running the regression tests using virtme-ng is currently only
> > > > > > > supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> > > > > > > hopefully we can support that in the future. Running regressions tests are
> > > > > > > primarily useful when making changes to core frameworks and public APIs, and
> > > > > > > it is possible to run them manually (see the README).
> > > > > > >
> > > > > > > Since running this locally can take a fair amount of time, we hope to have
> > > > > > > build servers available in the future so this can be offloaded.
> > > > > > >
> > > > > > > To give an idea of the expected build times:
> > > > > > >
> > > > > > > On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> > > > > > > (build.sh -test all) takes 39 minutes.
> > > > > > >
> > > > > > > On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> >
>
>
> _______________________________________________
> linuxtv-ci mailing list
> linuxtv-ci@linuxtv.org
> https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci



-- 
Ricardo Ribalda

Re: [ANN] Introducing build scripts to test

[Nicolas Dufresne]

Le vendredi 08 septembre 2023 à 21:44 +0200, Ricardo Ribalda a écrit :
> Hi Nicolas
> 
> On Fri, 8 Sept 2023 at 17:44, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > 
> > Le lundi 28 août 2023 à 17:45 +0300, Laurent Pinchart a écrit :
> > > On Mon, Aug 28, 2023 at 04:38:32PM +0200, Hans Verkuil wrote:
> > > > On 28/08/2023 16:26, Laurent Pinchart wrote:
> > > > > On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> > > > > > On 28/08/2023 16:05, Jacopo Mondi wrote:
> > > > > > > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> > > > > > > > Hi all,
> > > > > > > > 
> > > > > > > > We have been working on simplifying the media maintenance, and one part of that is
> > > > > > > > standardizing on build tools, in particular to make it easier for patch submitters
> > > > > > > > to run their patches through the same set of tests that the daily build does.
> > > > > > > > 
> > > > > > > > This helps detect issues before you submit your patches.
> > > > > > > > 
> > > > > > > > I have been working since July on transforming my hackish scripts to something
> > > > > > > > that is easier to use and of better quality. While there are still a few rough
> > > > > > > > edges, I consider it good enough to have others start to use it.
> > > > > > > > 
> > > > > > > > To get the build scripts run:
> > > > > > > > 
> > > > > > > > git clone git://linuxtv.org/hverkuil/build-scripts.git
> > > > > > > > 
> > > > > > > > All the test builds will happen within this directory. It is completely separate
> > > > > > > > from where you do you normal development, instead you point it to where your
> > > > > > > > git repository is.
> > > > > > > > 
> > > > > > > > See the README contained in the build-scripts git repo for all the details on
> > > > > > > > how to set it up.
> > > > > > > > 
> > > > > > > 
> > > > > > > I've been using your scripts since after ELC-E and I can tell they're
> > > > > > > useful!
> > > > > > > 
> > > > > > > > Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> > > > > > > > as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> > > > > > > > those distros, and you get it to work, then a patch updating the README file with
> > > > > > > > the correct list of packages to install would be welcome.
> > > > > > > 
> > > > > > > Speaking about distros, I was wondering if you still consider a
> > > > > > > requirement to build all compiler or we should instead try to use the
> > > > > > > distro provided ones when possible to test the distro-shipped version
> > > > > > > ?
> > > > > > 
> > > > > > I strongly believe we should build the cross compilers. The reason is that
> > > > > > otherwise you get a wide variety of compiler versions, each with typically
> > > > > > different compiler warnings. It's a pain for a developer to see different
> > > > > > warnings than the person that merges those patches.
> > > > > > 
> > > > > > It's a a regular problem that the daily build sees different warnings than
> > > > > > you do locally with the distro's compiler.
> > > > > > 
> > > > > > Doing it this way also makes it easier to upgrade to the latest compiler
> > > > > > version, certainly quicker than a distro would do.
> > > > > > 
> > > > > > It's about reproducibility, really.
> > > > > 
> > > > > There's value in testing with different compiler versions though. The
> > > > > kernel's documented minimum gcc version is v5.1 at the moment. I
> > > > > certainly don't want to build myself with all versions between v5.1 and
> > > > > v13.2, but collectively we could cover more ground.
> > > > > 
> > > > > Regardless of this, I already have recent cross compilers (built with
> > > > > buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> > > > > compilers. Anything that consumes extra disk space is a serious
> > > > > hinderance.
> > > > 
> > > > Feel free, but you run the risk that your PR is rejected because when I
> > > > run with these compiler versions I see new warnings. The whole point is
> > > > to be able to do the same tests before you make the PR to reduce the risk
> > > > of having to make a v2.
> > > > 
> > > > FYI: the cross directory takes about 10 GB on my system. That can be
> > > > streamlined a bit by deleting some temporary directories needed while
> > > > building, probably to something closer to 5-6 GB.
> > > 
> > > It may not be huge by itself, but it quickly adds up when you need to
> > > maintain multiple userspace cross-built enviroments, including Chrome
> > > OS, Android, Yocto, ... :-( I have half a TB of disk on my main
> > > development machine, and I would need at least 4 times that to cover my
> > > current needs comfortably.
> > 
> > I suppose this is irrelevant if you have a mean to send you PR to a machine that
> > will validate it for you. This is something I'd like to see happen in the
> > future. Considering the very tiny number of devs doing PR, a first step could be
> > to have a shared server in the cloud with the appropriate distro, compilers, and
> > just one more script to run test against a PR URI. This is quite minimal infra
> > and maintenance, since this is identical to what everyone may have locally, plus
> > having to install an SSH server and manages keys. Of course, scripts remains,
> > and can be used locally with of course the possible oops of running something
> > slightly different, but with the benefit of not having to "push somewhere" to
> > validate.
> 
> This is something we have just started to work on:
> 
> https://docs.google.com/document/d/1HTpk73qqfZLjrrvUwbd4g11wd8e9TkXTXvz5FZBd52g/edit#heading=h.4v9g2243whew
> 
> The plan is to be able to test locally and in gitlab.

Ok,let comment in there, though, I hope something will be sent to the ML form
time to time, since that document will turn down many. I saw the quickly reading
"life of a patch", and wasn't very impressed. I'd like to see something a bit
more forward looking, get out of the bubble of "maintainer" testing.

Currently, reading this document, all the benefit of gitlab endup being "for
maintainer only". I'd like to see a better vision for future of patch submission
that helps the submitter directly. It is the latency between reviewers to
submitters that kills the flow, the more the submitters can fix by themself, the
better.

> 
> > 
> > We could also have an FDO project and use their infra, which would be a lot
> > nicer imho, but we can't enter FDO without bringing matching sponsorship for the
> > resources we'd be using. At least we should ask first, not serve ourself there.
> 
> I aleady got some some google cloud sponsorship for the project if we
> can land it ;)

That is great news. Make sure to contact FDO admins. What about having a
namespace there ?

Nicolas

> 
> 
> > 
> > Nicolas
> > > 
> > > > > > > > Please note that running the regression tests using virtme-ng is currently only
> > > > > > > > supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> > > > > > > > hopefully we can support that in the future. Running regressions tests are
> > > > > > > > primarily useful when making changes to core frameworks and public APIs, and
> > > > > > > > it is possible to run them manually (see the README).
> > > > > > > > 
> > > > > > > > Since running this locally can take a fair amount of time, we hope to have
> > > > > > > > build servers available in the future so this can be offloaded.
> > > > > > > > 
> > > > > > > > To give an idea of the expected build times:
> > > > > > > > 
> > > > > > > > On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> > > > > > > > (build.sh -test all) takes 39 minutes.
> > > > > > > > 
> > > > > > > > On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> > > 
> > 
> > 
> > _______________________________________________
> > linuxtv-ci mailing list
> > linuxtv-ci@linuxtv.org
> > https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci
> 
> 
> 

Re: [ANN] Introducing build scripts to test

[Ricardo Ribalda]

Hi Nicolas

On Fri, 8 Sept 2023 at 22:36, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le vendredi 08 septembre 2023 à 21:44 +0200, Ricardo Ribalda a écrit :
> > Hi Nicolas
> >
> > On Fri, 8 Sept 2023 at 17:44, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> > >
> > > Le lundi 28 août 2023 à 17:45 +0300, Laurent Pinchart a écrit :
> > > > On Mon, Aug 28, 2023 at 04:38:32PM +0200, Hans Verkuil wrote:
> > > > > On 28/08/2023 16:26, Laurent Pinchart wrote:
> > > > > > On Mon, Aug 28, 2023 at 04:14:56PM +0200, Hans Verkuil wrote:
> > > > > > > On 28/08/2023 16:05, Jacopo Mondi wrote:
> > > > > > > > On Mon, Aug 28, 2023 at 03:29:41PM +0200, Hans Verkuil wrote:
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > We have been working on simplifying the media maintenance, and one part of that is
> > > > > > > > > standardizing on build tools, in particular to make it easier for patch submitters
> > > > > > > > > to run their patches through the same set of tests that the daily build does.
> > > > > > > > >
> > > > > > > > > This helps detect issues before you submit your patches.
> > > > > > > > >
> > > > > > > > > I have been working since July on transforming my hackish scripts to something
> > > > > > > > > that is easier to use and of better quality. While there are still a few rough
> > > > > > > > > edges, I consider it good enough to have others start to use it.
> > > > > > > > >
> > > > > > > > > To get the build scripts run:
> > > > > > > > >
> > > > > > > > > git clone git://linuxtv.org/hverkuil/build-scripts.git
> > > > > > > > >
> > > > > > > > > All the test builds will happen within this directory. It is completely separate
> > > > > > > > > from where you do you normal development, instead you point it to where your
> > > > > > > > > git repository is.
> > > > > > > > >
> > > > > > > > > See the README contained in the build-scripts git repo for all the details on
> > > > > > > > > how to set it up.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I've been using your scripts since after ELC-E and I can tell they're
> > > > > > > > useful!
> > > > > > > >
> > > > > > > > > Currently the scripts expect a debian 12-based distro (likely debian 11 will work
> > > > > > > > > as well). I have no idea if it works well on Red Hat or Suse. If you use one of
> > > > > > > > > those distros, and you get it to work, then a patch updating the README file with
> > > > > > > > > the correct list of packages to install would be welcome.
> > > > > > > >
> > > > > > > > Speaking about distros, I was wondering if you still consider a
> > > > > > > > requirement to build all compiler or we should instead try to use the
> > > > > > > > distro provided ones when possible to test the distro-shipped version
> > > > > > > > ?
> > > > > > >
> > > > > > > I strongly believe we should build the cross compilers. The reason is that
> > > > > > > otherwise you get a wide variety of compiler versions, each with typically
> > > > > > > different compiler warnings. It's a pain for a developer to see different
> > > > > > > warnings than the person that merges those patches.
> > > > > > >
> > > > > > > It's a a regular problem that the daily build sees different warnings than
> > > > > > > you do locally with the distro's compiler.
> > > > > > >
> > > > > > > Doing it this way also makes it easier to upgrade to the latest compiler
> > > > > > > version, certainly quicker than a distro would do.
> > > > > > >
> > > > > > > It's about reproducibility, really.
> > > > > >
> > > > > > There's value in testing with different compiler versions though. The
> > > > > > kernel's documented minimum gcc version is v5.1 at the moment. I
> > > > > > certainly don't want to build myself with all versions between v5.1 and
> > > > > > v13.2, but collectively we could cover more ground.
> > > > > >
> > > > > > Regardless of this, I already have recent cross compilers (built with
> > > > > > buildroot) for ARM and ARM64, and I'd rather use those than duplicating
> > > > > > compilers. Anything that consumes extra disk space is a serious
> > > > > > hinderance.
> > > > >
> > > > > Feel free, but you run the risk that your PR is rejected because when I
> > > > > run with these compiler versions I see new warnings. The whole point is
> > > > > to be able to do the same tests before you make the PR to reduce the risk
> > > > > of having to make a v2.
> > > > >
> > > > > FYI: the cross directory takes about 10 GB on my system. That can be
> > > > > streamlined a bit by deleting some temporary directories needed while
> > > > > building, probably to something closer to 5-6 GB.
> > > >
> > > > It may not be huge by itself, but it quickly adds up when you need to
> > > > maintain multiple userspace cross-built enviroments, including Chrome
> > > > OS, Android, Yocto, ... :-( I have half a TB of disk on my main
> > > > development machine, and I would need at least 4 times that to cover my
> > > > current needs comfortably.
> > >
> > > I suppose this is irrelevant if you have a mean to send you PR to a machine that
> > > will validate it for you. This is something I'd like to see happen in the
> > > future. Considering the very tiny number of devs doing PR, a first step could be
> > > to have a shared server in the cloud with the appropriate distro, compilers, and
> > > just one more script to run test against a PR URI. This is quite minimal infra
> > > and maintenance, since this is identical to what everyone may have locally, plus
> > > having to install an SSH server and manages keys. Of course, scripts remains,
> > > and can be used locally with of course the possible oops of running something
> > > slightly different, but with the benefit of not having to "push somewhere" to
> > > validate.
> >
> > This is something we have just started to work on:
> >
> > https://docs.google.com/document/d/1HTpk73qqfZLjrrvUwbd4g11wd8e9TkXTXvz5FZBd52g/edit#heading=h.4v9g2243whew
> >
> > The plan is to be able to test locally and in gitlab.
>
> Ok,let comment in there, though, I hope something will be sent to the ML form
> time to time, since that document will turn down many. I saw the quickly reading

It will become part of our documentation once everything is setup. I
really like google docs for the early stages of design docs. There is
no need to have a gmail account to use it.

> "life of a patch", and wasn't very impressed. I'd like to see something a bit
> more forward looking, get out of the bubble of "maintainer" testing.

Not sure what you mean. Any developer can push their code to gitlab and test it.

The only "super-power" of the uploader is being able to press on
merge, after the PR has been validated by the CI/CD.

Coordinating when all the different PR land should be the job of
someone used to our community. Please note that the bar to become a
uploader will be much lower than becoming a core maintainer.

>
> Currently, reading this document, all the benefit of gitlab endup being "for
> maintainer only". I'd like to see a better vision for future of patch submission

Anyone can use gitlab if they want, but we are not forcing any
developer to use it. Anyone can still send patches to the ML and they
will eventually find their way on the media tree.

> that helps the submitter directly. It is the latency between reviewers to
> submitters that kills the flow, the more the submitters can fix by themself, the
> better.

We also have a long latency from until code is reviewed.

Looking forward to your comments on the doc :)




>
> >
> > >
> > > We could also have an FDO project and use their infra, which would be a lot
> > > nicer imho, but we can't enter FDO without bringing matching sponsorship for the
> > > resources we'd be using. At least we should ask first, not serve ourself there.
> >
> > I aleady got some some google cloud sponsorship for the project if we
> > can land it ;)
>
> That is great news. Make sure to contact FDO admins. What about having a
> namespace there ?
>
> Nicolas
>
> >
> >
> > >
> > > Nicolas
> > > >
> > > > > > > > > Please note that running the regression tests using virtme-ng is currently only
> > > > > > > > > supported on Debian 12, not on e.g. Ubuntu. Someone is looking into that, and
> > > > > > > > > hopefully we can support that in the future. Running regressions tests are
> > > > > > > > > primarily useful when making changes to core frameworks and public APIs, and
> > > > > > > > > it is possible to run them manually (see the README).
> > > > > > > > >
> > > > > > > > > Since running this locally can take a fair amount of time, we hope to have
> > > > > > > > > build servers available in the future so this can be offloaded.
> > > > > > > > >
> > > > > > > > > To give an idea of the expected build times:
> > > > > > > > >
> > > > > > > > > On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> > > > > > > > > (build.sh -test all) takes 39 minutes.
> > > > > > > > >
> > > > > > > > > On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.
> > > >
> > >
> > >
> > > _______________________________________________
> > > linuxtv-ci mailing list
> > > linuxtv-ci@linuxtv.org
> > > https://www.linuxtv.org/cgi-bin/mailman/listinfo/linuxtv-ci
> >
> >
> >
>


-- 
Ricardo Ribalda

Re: [ANN] Introducing build scripts to test

[Benjamin Mugnier]

Hi Hans,

A bit late to the party, I tried it this morning and everything went
flawlessly. Instructions are clear and command line tools are well designed.
This will be useful for us. Thank you for releasing it.

On 8/28/23 15:29, Hans Verkuil wrote
> To give an idea of the expected build times:
> 
> On an AMD Ryzen 9 6900HX (8 cores) a standard build of the staging tree
> (build.sh -test all) takes 39 minutes.
> 
> On an AMD Ryzen Threadripper 3970X (32 cores) it takes a bit over 13 minutes.

For the record on my Intel i7-11850H :

$ time ./build.sh -test all
real	36m21.633s
user	452m2.855s
sys	39m30.233s

-- 
Regards,

Benjamin