[QUERY] How many CI mails is too many?

[QUERY] How many CI mails is too many?

[Arkadiusz Hiler]

Hey all,

For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
results and "full IGT" results (if BAT has not failed).

Recently we have added 32bit build check, and if that fails it sends out
additional mail In-Reply-To the series.

I am working on adding some static checks to the CI (spare and checkpatch at the
moment, more may come in the future), which may generate even more commotion on
the mailing list.

How much of CI noise is too much and how you would like to have the results
grouped?

Couple of options to start the discussion:

 1. Group all static checks (and the 32bit build?) into one mail:
    - just one additional mail,
    - may be hard to read in case of catastrophic failure,
    - we can send it only when something actually fails.

 2. Send out the results as a part of BAT results:
    - even less noise than (1),
    - BAT results already feel cluttered, this may decrease readability.

 3. Have each check as a separate mail, but send it only if the check fails:
    - noisy: may result in many mails, depending how many checks fail,
    - easier to read and easier to follow on patchwork.

Any opinions? Any other ideas?

-- 
Cheers,
Arek
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Sagar Arun Kamble]

I feel we generally tend to ignore the results mails for series that we are not actively involved on (although we might be
interested in series itself). Also if number of revisions some series can undergo is high, this tendency can grow.

How about the option of sending the results mails to only author and all committers.
Ideally, author should include at lease one committer and in that case we can possibly avoid mail to all committers.

Another option would be no results mails at all and enforce authors to ensure all green at https://patchwork.freedesktop.org/project/intel-gfx/series/?ordering=-last_updated

Thanks
Sagar

On 11/27/2017 8:24 PM, Arkadiusz Hiler wrote:
> Hey all,
>
> For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> results and "full IGT" results (if BAT has not failed).
>
> Recently we have added 32bit build check, and if that fails it sends out
> additional mail In-Reply-To the series.
>
> I am working on adding some static checks to the CI (spare and checkpatch at the
> moment, more may come in the future), which may generate even more commotion on
> the mailing list.
>
> How much of CI noise is too much and how you would like to have the results
> grouped?
>
> Couple of options to start the discussion:
>
>   1. Group all static checks (and the 32bit build?) into one mail:
>      - just one additional mail,
>      - may be hard to read in case of catastrophic failure,
>      - we can send it only when something actually fails.
>
>   2. Send out the results as a part of BAT results:
>      - even less noise than (1),
>      - BAT results already feel cluttered, this may decrease readability.
>
>   3. Have each check as a separate mail, but send it only if the check fails:
>      - noisy: may result in many mails, depending how many checks fail,
>      - easier to read and easier to follow on patchwork.
>
> Any opinions? Any other ideas?
>

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Rodrigo Vivi]

On Mon, Nov 27, 2017 at 02:54:02PM +0000, Arkadiusz Hiler wrote:
> Hey all,
> 
> For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> results and "full IGT" results (if BAT has not failed).
> 
> Recently we have added 32bit build check, and if that fails it sends out
> additional mail In-Reply-To the series.
> 
> I am working on adding some static checks to the CI (spare and checkpatch at the
> moment, more may come in the future), which may generate even more commotion on
> the mailing list.
> 
> How much of CI noise is too much and how you would like to have the results
> grouped?
> 
> Couple of options to start the discussion:
> 
>  1. Group all static checks (and the 32bit build?) into one mail:
>     - just one additional mail,
>     - may be hard to read in case of catastrophic failure,
>     - we can send it only when something actually fails.
> 
>  2. Send out the results as a part of BAT results:
>     - even less noise than (1),
>     - BAT results already feel cluttered, this may decrease readability.
> 
>  3. Have each check as a separate mail, but send it only if the check fails:
>     - noisy: may result in many mails, depending how many checks fail,
>     - easier to read and easier to follow on patchwork.

I'd vote for number 3. And let's see how noisy that gets and maybe
that end up in a good thing because we will start seeing what causes noise
and possibly work to avoid those.

> 
> Any opinions? Any other ideas?
> 
> -- 
> Cheers,
> Arek
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Joonas Lahtinen]

On Mon, 2017-11-27 at 16:54 +0200, Arkadiusz Hiler wrote:
> Hey all,
> 
> For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> results and "full IGT" results (if BAT has not failed).
> 
> Recently we have added 32bit build check, and if that fails it sends out
> additional mail In-Reply-To the series.
> 
> I am working on adding some static checks to the CI (spare and checkpatch at the
> moment, more may come in the future), which may generate even more commotion on
> the mailing list.
> 
> How much of CI noise is too much and how you would like to have the results
> grouped?
> 
> Couple of options to start the discussion:
> 
>  1. Group all static checks (and the 32bit build?) into one mail:
>     - just one additional mail,
>     - may be hard to read in case of catastrophic failure,
>     - we can send it only when something actually fails.
> 
>  2. Send out the results as a part of BAT results:
>     - even less noise than (1),
>     - BAT results already feel cluttered, this may decrease readability.
> 
>  3. Have each check as a separate mail, but send it only if the check fails:
>     - noisy: may result in many mails, depending how many checks fail,
>     - easier to read and easier to follow on patchwork.

The best user experience I could think of;

1. If all CI checks succeed, delay and only send one mail with all the
results. This would indicate it's good to merge, go do it.
2. When a CI checks fail, immediately send that out so the developer
gets to work on the fix.

Above requires that all the checks complete rather quickly and a trust
is gained to the system so that the absence of e-mail always means the
series is doing good, not that the system is clogged in some way :)

Regards, Joonas

> 
> Any opinions? Any other ideas?
> 
-- 
Joonas Lahtinen
Open Source Technology Center
Intel Corporation
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Tvrtko Ursulin]

On 27/11/2017 14:54, Arkadiusz Hiler wrote:
> Hey all,
> 
> For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> results and "full IGT" results (if BAT has not failed).
> 
> Recently we have added 32bit build check, and if that fails it sends out
> additional mail In-Reply-To the series.
> 
> I am working on adding some static checks to the CI (spare and checkpatch at the
> moment, more may come in the future), which may generate even more commotion on
> the mailing list.
> 
> How much of CI noise is too much and how you would like to have the results
> grouped?
> 
> Couple of options to start the discussion:
> 
>   1. Group all static checks (and the 32bit build?) into one mail:
>      - just one additional mail,
>      - may be hard to read in case of catastrophic failure,
>      - we can send it only when something actually fails.
> 
>   2. Send out the results as a part of BAT results:
>      - even less noise than (1),
>      - BAT results already feel cluttered, this may decrease readability.
> 
>   3. Have each check as a separate mail, but send it only if the check fails:
>      - noisy: may result in many mails, depending how many checks fail,
>      - easier to read and easier to follow on patchwork.

Option 3 sounds good to me.

I don't see it as noisy - if something fails a dedicated notification is 
in order, and since it gets assigned into a respective thread by the 
MUA, the noise is minimal. And you only see self-inflicted noise, unless 
you don't use threading, or are keeping all the threads expanded, which 
is already non-workable to start with.

Regards,

Tvrtko


_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Daniel Vetter]

On Mon, Nov 27, 2017 at 3:54 PM, Arkadiusz Hiler
<arkadiusz.hiler@intel.com> wrote:
> Hey all,
>
> For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> results and "full IGT" results (if BAT has not failed).
>
> Recently we have added 32bit build check, and if that fails it sends out
> additional mail In-Reply-To the series.
>
> I am working on adding some static checks to the CI (spare and checkpatch at the
> moment, more may come in the future), which may generate even more commotion on
> the mailing list.
>
> How much of CI noise is too much and how you would like to have the results
> grouped?
>
> Couple of options to start the discussion:
>
>  1. Group all static checks (and the 32bit build?) into one mail:
>     - just one additional mail,
>     - may be hard to read in case of catastrophic failure,
>     - we can send it only when something actually fails.
>
>  2. Send out the results as a part of BAT results:
>     - even less noise than (1),
>     - BAT results already feel cluttered, this may decrease readability.
>
>  3. Have each check as a separate mail, but send it only if the check fails:
>     - noisy: may result in many mails, depending how many checks fail,
>     - easier to read and easier to follow on patchwork.
>
> Any opinions? Any other ideas?

All static checks grouped, and only sent out when there's something
iffy. I think for BAT&functional tests positive confirmation from CI
is good, but for static checkers it's a bit more meh imo. So 1&3 for
me.

And yes I'd group all the static checks and build checks into 1. E.g.
on dri-devel we also want to build-test arm and arm64 using the
drm-misc configs.
-Daniel

>
> --
> Cheers,
> Arek
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Chris Wilson]

Quoting Joonas Lahtinen (2017-11-28 08:15:13)
> On Mon, 2017-11-27 at 16:54 +0200, Arkadiusz Hiler wrote:
> > Hey all,
> > 
> > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> > results and "full IGT" results (if BAT has not failed).
> > 
> > Recently we have added 32bit build check, and if that fails it sends out
> > additional mail In-Reply-To the series.
> > 
> > I am working on adding some static checks to the CI (spare and checkpatch at the
> > moment, more may come in the future), which may generate even more commotion on
> > the mailing list.
> > 
> > How much of CI noise is too much and how you would like to have the results
> > grouped?
> > 
> > Couple of options to start the discussion:
> > 
> >  1. Group all static checks (and the 32bit build?) into one mail:
> >     - just one additional mail,
> >     - may be hard to read in case of catastrophic failure,
> >     - we can send it only when something actually fails.
> > 
> >  2. Send out the results as a part of BAT results:
> >     - even less noise than (1),
> >     - BAT results already feel cluttered, this may decrease readability.
> > 
> >  3. Have each check as a separate mail, but send it only if the check fails:
> >     - noisy: may result in many mails, depending how many checks fail,
> >     - easier to read and easier to follow on patchwork.
> 
> The best user experience I could think of;
> 
> 1. If all CI checks succeed, delay and only send one mail with all the
> results. This would indicate it's good to merge, go do it.
> 2. When a CI checks fail, immediately send that out so the developer
> gets to work on the fix.
> 
> Above requires that all the checks complete rather quickly and a trust
> is gained to the system so that the absence of e-mail always means the
> series is doing good, not that the system is clogged in some way :)

Or just 2. The first being the compilation report; saying we
have received your patch and it compiles fine, it will be queued to the
farm currently in slot N (or it doesn't even compile!). The second being
the success or failure of the CI run.

From the user pov, we can't do anything until the CI report so
intermediate emails saying congrats are just fluff. Useful simply to
know the patch hasn't fall out of the system, but not supplying any
actionable information.
-Chris
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Daniel Vetter]

On Tue, Nov 28, 2017 at 11:06 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> Quoting Joonas Lahtinen (2017-11-28 08:15:13)
>> On Mon, 2017-11-27 at 16:54 +0200, Arkadiusz Hiler wrote:
>> > Hey all,
>> >
>> > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
>> > results and "full IGT" results (if BAT has not failed).
>> >
>> > Recently we have added 32bit build check, and if that fails it sends out
>> > additional mail In-Reply-To the series.
>> >
>> > I am working on adding some static checks to the CI (spare and checkpatch at the
>> > moment, more may come in the future), which may generate even more commotion on
>> > the mailing list.
>> >
>> > How much of CI noise is too much and how you would like to have the results
>> > grouped?
>> >
>> > Couple of options to start the discussion:
>> >
>> >  1. Group all static checks (and the 32bit build?) into one mail:
>> >     - just one additional mail,
>> >     - may be hard to read in case of catastrophic failure,
>> >     - we can send it only when something actually fails.
>> >
>> >  2. Send out the results as a part of BAT results:
>> >     - even less noise than (1),
>> >     - BAT results already feel cluttered, this may decrease readability.
>> >
>> >  3. Have each check as a separate mail, but send it only if the check fails:
>> >     - noisy: may result in many mails, depending how many checks fail,
>> >     - easier to read and easier to follow on patchwork.
>>
>> The best user experience I could think of;
>>
>> 1. If all CI checks succeed, delay and only send one mail with all the
>> results. This would indicate it's good to merge, go do it.
>> 2. When a CI checks fail, immediately send that out so the developer
>> gets to work on the fix.
>>
>> Above requires that all the checks complete rather quickly and a trust
>> is gained to the system so that the absence of e-mail always means the
>> series is doing good, not that the system is clogged in some way :)
>
> Or just 2. The first being the compilation report; saying we
> have received your patch and it compiles fine, it will be queued to the
> farm currently in slot N (or it doesn't even compile!). The second being
> the success or failure of the CI run.
>
> From the user pov, we can't do anything until the CI report so
> intermediate emails saying congrats are just fluff. Useful simply to
> know the patch hasn't fall out of the system, but not supplying any
> actionable information.

BAT was meant to be that mail, with the added benefit that if a series
fails the basic sanity check you can ignore it for review and
everything. Still not quite there yet (and the recently undone change
of ratelimiting didn't help).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Chris Wilson]

Quoting Daniel Vetter (2017-11-28 10:08:56)
> On Tue, Nov 28, 2017 at 11:06 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> > Quoting Joonas Lahtinen (2017-11-28 08:15:13)
> >> On Mon, 2017-11-27 at 16:54 +0200, Arkadiusz Hiler wrote:
> >> > Hey all,
> >> >
> >> > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> >> > results and "full IGT" results (if BAT has not failed).
> >> >
> >> > Recently we have added 32bit build check, and if that fails it sends out
> >> > additional mail In-Reply-To the series.
> >> >
> >> > I am working on adding some static checks to the CI (spare and checkpatch at the
> >> > moment, more may come in the future), which may generate even more commotion on
> >> > the mailing list.
> >> >
> >> > How much of CI noise is too much and how you would like to have the results
> >> > grouped?
> >> >
> >> > Couple of options to start the discussion:
> >> >
> >> >  1. Group all static checks (and the 32bit build?) into one mail:
> >> >     - just one additional mail,
> >> >     - may be hard to read in case of catastrophic failure,
> >> >     - we can send it only when something actually fails.
> >> >
> >> >  2. Send out the results as a part of BAT results:
> >> >     - even less noise than (1),
> >> >     - BAT results already feel cluttered, this may decrease readability.
> >> >
> >> >  3. Have each check as a separate mail, but send it only if the check fails:
> >> >     - noisy: may result in many mails, depending how many checks fail,
> >> >     - easier to read and easier to follow on patchwork.
> >>
> >> The best user experience I could think of;
> >>
> >> 1. If all CI checks succeed, delay and only send one mail with all the
> >> results. This would indicate it's good to merge, go do it.
> >> 2. When a CI checks fail, immediately send that out so the developer
> >> gets to work on the fix.
> >>
> >> Above requires that all the checks complete rather quickly and a trust
> >> is gained to the system so that the absence of e-mail always means the
> >> series is doing good, not that the system is clogged in some way :)
> >
> > Or just 2. The first being the compilation report; saying we
> > have received your patch and it compiles fine, it will be queued to the
> > farm currently in slot N (or it doesn't even compile!). The second being
> > the success or failure of the CI run.
> >
> > From the user pov, we can't do anything until the CI report so
> > intermediate emails saying congrats are just fluff. Useful simply to
> > know the patch hasn't fall out of the system, but not supplying any
> > actionable information.
> 
> BAT was meant to be that mail, with the added benefit that if a series
> fails the basic sanity check you can ignore it for review and
> everything. Still not quite there yet (and the recently undone change
> of ratelimiting didn't help).

The compile check should take 30s(?) on the build host with all the
distcc/ccache. It's going to be rare to develop a long queue and
significant latency; whereas one developer can flood the system with 5
different series they happen to have queued, repeat for everyone getting
to work in the morning.
-Chris
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Arkadiusz Hiler]

On Mon, Nov 27, 2017 at 09:10:37PM +0530, Sagar Arun Kamble wrote:
> I feel we generally tend to ignore the results mails for series that
> we are not actively involved on (although we might be interested in
> series itself). Also if number of revisions some series can undergo is
> high, this tendency can grow.

It is true that I don't care that much about results tied to series I am
not interested in, but I don't find this too distracting. They are
nested nicely in the thread and are also very easy to distinguish
visually.

> How about the option of sending the results mails to only author and
> all committers. Ideally, author should include at lease one committer
> and in that case we can possibly avoid mail to all committers.
> 
> Another option would be no results mails at all and enforce authors to
> ensure all green at
> https://patchwork.freedesktop.org/project/intel-gfx/series/?ordering=-last_updated
> 
> Thanks
> Sagar

In my mind the CI system should complement mailing list, not change it
or require unnecessary, external interactions. By sending the result to
intel-gfx we get the gist of the results here, with the direct links to
the patchwork and intel-gfx-ci grids provided (so no need to hunt for
those).

The results also stores nicely in the online mailing list archives if we
send it to the intel-gfx@fdo.

Searchability and easy categorization is an added bonus if you subscribe
to dozen or so mailing lists.

Cheers,
Arek

> On 11/27/2017 8:24 PM, Arkadiusz Hiler wrote:
> > Hey all,
> > 
> > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> > results and "full IGT" results (if BAT has not failed).
> > 
> > Recently we have added 32bit build check, and if that fails it sends out
> > additional mail In-Reply-To the series.
> > 
> > I am working on adding some static checks to the CI (spare and checkpatch at the
> > moment, more may come in the future), which may generate even more commotion on
> > the mailing list.
> > 
> > How much of CI noise is too much and how you would like to have the results
> > grouped?
> > 
> > Couple of options to start the discussion:
> > 
> >   1. Group all static checks (and the 32bit build?) into one mail:
> >      - just one additional mail,
> >      - may be hard to read in case of catastrophic failure,
> >      - we can send it only when something actually fails.
> > 
> >   2. Send out the results as a part of BAT results:
> >      - even less noise than (1),
> >      - BAT results already feel cluttered, this may decrease readability.
> > 
> >   3. Have each check as a separate mail, but send it only if the check fails:
> >      - noisy: may result in many mails, depending how many checks fail,
> >      - easier to read and easier to follow on patchwork.
> > 
> > Any opinions? Any other ideas?
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Ewelina Musial]

On Tue, Nov 28, 2017 at 01:16:50PM +0200, Arkadiusz Hiler wrote:
> On Mon, Nov 27, 2017 at 09:10:37PM +0530, Sagar Arun Kamble wrote:
> > I feel we generally tend to ignore the results mails for series that
> > we are not actively involved on (although we might be interested in
> > series itself). Also if number of revisions some series can undergo is
> > high, this tendency can grow.
> 
> It is true that I don't care that much about results tied to series I am
> not interested in, but I don't find this too distracting. They are
> nested nicely in the thread and are also very easy to distinguish
> visually.
> 
> > How about the option of sending the results mails to only author and
> > all committers. Ideally, author should include at lease one committer
> > and in that case we can possibly avoid mail to all committers.

I had the same idea but then I thought that sometimes if I want to apply some series of patches
I would like to know what is a status of that patch (fail or not) so maybe sending that to
author only can be not really good idea.

-Ewelina
> > 
> > Another option would be no results mails at all and enforce authors to
> > ensure all green at
> > https://patchwork.freedesktop.org/project/intel-gfx/series/?ordering=-last_updated
> > 
> > Thanks
> > Sagar
> 
> In my mind the CI system should complement mailing list, not change it
> or require unnecessary, external interactions. By sending the result to
> intel-gfx we get the gist of the results here, with the direct links to
> the patchwork and intel-gfx-ci grids provided (so no need to hunt for
> those).
> 
> The results also stores nicely in the online mailing list archives if we
> send it to the intel-gfx@fdo.
> 
> Searchability and easy categorization is an added bonus if you subscribe
> to dozen or so mailing lists.
> 
> Cheers,
> Arek
> 
> > On 11/27/2017 8:24 PM, Arkadiusz Hiler wrote:
> > > Hey all,
> > > 
> > > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> > > results and "full IGT" results (if BAT has not failed).
> > > 
> > > Recently we have added 32bit build check, and if that fails it sends out
> > > additional mail In-Reply-To the series.
> > > 
> > > I am working on adding some static checks to the CI (spare and checkpatch at the
> > > moment, more may come in the future), which may generate even more commotion on
> > > the mailing list.
> > > 
> > > How much of CI noise is too much and how you would like to have the results
> > > grouped?
> > > 
> > > Couple of options to start the discussion:
> > > 
> > >   1. Group all static checks (and the 32bit build?) into one mail:
> > >      - just one additional mail,
> > >      - may be hard to read in case of catastrophic failure,
> > >      - we can send it only when something actually fails.
> > > 
> > >   2. Send out the results as a part of BAT results:
> > >      - even less noise than (1),
> > >      - BAT results already feel cluttered, this may decrease readability.
> > > 
> > >   3. Have each check as a separate mail, but send it only if the check fails:
> > >      - noisy: may result in many mails, depending how many checks fail,
> > >      - easier to read and easier to follow on patchwork.
> > > 
> > > Any opinions? Any other ideas?
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [QUERY] How many CI mails is too many?

[Chris Wilson]

Quoting Chris Wilson (2017-11-28 10:16:17)
> Quoting Daniel Vetter (2017-11-28 10:08:56)
> > On Tue, Nov 28, 2017 at 11:06 AM, Chris Wilson <chris@chris-wilson.co.uk> wrote:
> > > Quoting Joonas Lahtinen (2017-11-28 08:15:13)
> > >> On Mon, 2017-11-27 at 16:54 +0200, Arkadiusz Hiler wrote:
> > >> > Hey all,
> > >> >
> > >> > For some time already CI sends out 1-2 mails per series per (re)run, i.e. BAT
> > >> > results and "full IGT" results (if BAT has not failed).
> > >> >
> > >> > Recently we have added 32bit build check, and if that fails it sends out
> > >> > additional mail In-Reply-To the series.
> > >> >
> > >> > I am working on adding some static checks to the CI (spare and checkpatch at the
> > >> > moment, more may come in the future), which may generate even more commotion on
> > >> > the mailing list.
> > >> >
> > >> > How much of CI noise is too much and how you would like to have the results
> > >> > grouped?
> > >> >
> > >> > Couple of options to start the discussion:
> > >> >
> > >> >  1. Group all static checks (and the 32bit build?) into one mail:
> > >> >     - just one additional mail,
> > >> >     - may be hard to read in case of catastrophic failure,
> > >> >     - we can send it only when something actually fails.
> > >> >
> > >> >  2. Send out the results as a part of BAT results:
> > >> >     - even less noise than (1),
> > >> >     - BAT results already feel cluttered, this may decrease readability.
> > >> >
> > >> >  3. Have each check as a separate mail, but send it only if the check fails:
> > >> >     - noisy: may result in many mails, depending how many checks fail,
> > >> >     - easier to read and easier to follow on patchwork.
> > >>
> > >> The best user experience I could think of;
> > >>
> > >> 1. If all CI checks succeed, delay and only send one mail with all the
> > >> results. This would indicate it's good to merge, go do it.
> > >> 2. When a CI checks fail, immediately send that out so the developer
> > >> gets to work on the fix.
> > >>
> > >> Above requires that all the checks complete rather quickly and a trust
> > >> is gained to the system so that the absence of e-mail always means the
> > >> series is doing good, not that the system is clogged in some way :)
> > >
> > > Or just 2. The first being the compilation report; saying we
> > > have received your patch and it compiles fine, it will be queued to the
> > > farm currently in slot N (or it doesn't even compile!). The second being
> > > the success or failure of the CI run.
> > >
> > > From the user pov, we can't do anything until the CI report so
> > > intermediate emails saying congrats are just fluff. Useful simply to
> > > know the patch hasn't fall out of the system, but not supplying any
> > > actionable information.
> > 
> > BAT was meant to be that mail, with the added benefit that if a series
> > fails the basic sanity check you can ignore it for review and
> > everything. Still not quite there yet (and the recently undone change
> > of ratelimiting didn't help).
> 
> The compile check should take 30s(?) on the build host with all the
> distcc/ccache. It's going to be rare to develop a long queue and
> significant latency; whereas one developer can flood the system with 5
> different series they happen to have queued, repeat for everyone getting
> to work in the morning.

One thing I forgot to mention, is that trybot has a large latency for
that single BAT success email. Having a quick "patch received; compiles"
response from the system would give a nice bit of reassurance.
-Chris
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx