Yocto Technical Team Minutes, Engineering Sync, for June 23 2020

Yocto Technical Team Minutes, Engineering Sync, for June 23 2020

[Trevor Woerner]

Yocto Technical Team Minutes, Engineering Sync, for June 23 2020
archive: https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit

== disclaimer ==
Best efforts are made to ensure the below is accurate and valid. However,
errors sometimes happen. If any errors or omissions are found, please feel
free to reply to this email with any corrections.

== attendees ==
Trevor Woerner, Stephen Jolly, Jan-Simon Möller, Armin Kuster, Jeremy
Puhlman, Trevor Gamblin, Scott Murray, Richard Purdie, Steve Sakoman, Bruce
Ashfield, Paul Baker, Randy MacLeod, Peter Kjellerstedt, David Reyna, Mark
Morton, Tim Orling, Joshua Watt, Michael Halstead, Vineela (?), Alejandro H

== notes ==
- 3.1.1 was released last week
- 3.2.1 out of QA, looks clean, for TSC review, release probably in next few
    days
- still issues with AB
- still looking for more maintainers
- lots of unassigned M2 bugs
- new testing manual
- ELC + devday next week

== general ==
TW: meeting next week?
Stephen: will look if there’s a timezone/agenda conflict…
(various input)
Stephen: okay, next week’s meeting cancelled

David: ELC, virtual booth and slack session, looking for people to hang out
TW: do we have to be registered for ELC?
David: no, certainly not for slack, Tracey and I will work it out. email to
    come later today with details, contact me if interested

RP: links in status report to devday and to new test manual. please have
    a look through the test manual. kudos to Mark for pulling it together
    (always getting questions about testing). this was started a while back
    with ScottR

RP: re AB. still seeing intermittent issues, causes lots of builds to fail.
    impacts me and SteveS. couple issues with repositories and/or branches
    (upstream) disappearing.

RP: issues around the “stdin” bug (wrt testing)

RP: Ross still working on RCU bug

RP: Alex working on a “warnings” issue

Scott: can we still add 3.2 features?
RP: nobody’s mentioning anything. we’ve never been so bleeding edge (AUH
    kudos: AlexK). apt upgrade. Armin working on dhcp and bind updates. But no
    “3.2 sound-bite”
JPEW: multiconfig fixes
RP: true. and also some RDEPENDS support (should be straight-forward except
    when recursing)
JPEW: not sold on the feature
RP: simple use-case is to put 2 toolchains into one sdk. my answer is to build
    the two then combine later (with other code). i agree that we might need
    to look for a different approach
Timo: the code to bring it all together could do other stuff too, but i agree
    with Joshua that “there be dragons”

Timo: perl has a new release 5.32, we might want to get that in
Scott: but it has some new language features, might not just be a point
    release
Timo: haven’t dug in, we’ve had issues with perl so hopefully things are
    better

RP: 3.2. held up due to valgrind regressions, new distros (gcc 10 fixes),
    reproducible builds, etc… so there’s lots of good stuff going in, just
    nothing “new” (i.e. a feature), but that’s okay

Scott: new autoconf (Ross)?
RP: don’t think it’s released yet, we’re testing the pre-release. if
    upstream releases then we’ll upgrade, but don’t know their timescale

RP: trying to reduce patch overhead (AlexK has been helping). it has been
    slowing moving down, but recently up-ticked a tiny bit. we’ll never get
    to 0, but every reduction helps

Timo: was looking at Sphinx update yesterday. looks good, some glitches.
    noticed the doc revision history was missing. how important is that?
RP: i’ll ask MarkM to take a look. if it’s lost it won’t break the deal,
    but ideally we’d keep it
MarkM: yes, that and some other things (e.g. html version doesn’t have
    section links)
Timo: was looking at Toaster manual, noticed some images were missing too.
    also found a broken URL. overall very good, nice to see a modern format
    (i.e. readthedocs)
MarkM: we’re also looking at a colour scheme, Nico simply picked the
    default, but something else will probably look better
RP: Nico, MarkM, and I met (it was an open meeting… *wink wink*) and decided
    to move forward and are hoping to have it done for Dunfell (LTS) and
    master
Timo: agreed. XML is a dying art, everything looks good, kudos! (will try to
    get some help from some Intel folks)

JPEW: sstate found/miss cache?
JPEW: a patch to not go out and look for every sstate object, keep a cache if
    one had already been found
RP: the found cache isn’t the issue, the not-found could be an issue
    especially where multiple builds are using the same sstate. we don’t
    have a mechanism to generate an sstate object in isolation, but it might
    be a step away
JPEW: but it’s the not-found that is the slow path. if the sstate object
    isn’t there, the build goes out and searches the filesystem for the
    missing object
RP: maybe we could make the cache time-limited (e.g. 30 seconds?)
JPEW: yes, that might help. it’s like there’s 2 scenarios: local builds
    with one builder, group builds
PaulB: maybe we could simply watch the sstate directory for changes?
PaulB: there’s also an issue of a remote sstate cache over https where
    packets are dropped so it fails, the machine builds it locally. could be
    a “time of check” vs “time of use” issue: the check says it’s
    there, but at use time the network blips and then it’s not retrieved,
    and then there’s no retry
JPEW: we saw same issue here: solution for us was to run bitbake twice: once
    with the setscene only, then once without it
PaulB: we’re trying to offer our customers a common sstate but we can’t
    necessarily ask users to do the “2 bitbake” build
RP: maybe we need a config setting
PaulB: i think the issue is we’re checking a URL twice, it’s the time
    difference between checking and using that is the issue. perhaps we could
    just do the poll once?
RP: absolutely not, that’s effectively not possible. decisions are made
    based on those checks so the two fetches have to be separate. if we
    combine them, then there’s no savings, each build would be very long
PaulB: perhaps better reporting
RP: people are just going to set the config to ignore the warning then builds
    will fail and it’ll look like the build is flaky. i think the answer
    is we’re going to have to accept a config option, but we’ll have to
    document it and be aware
Alejandro: could we switch run-queue classes (by a flag for e.g.)
RP: the patch isn’t the issue, it’s a philosophical question. i’m
    worried about the implications from a usability standpoint (e.g. “why is
    my build rebuilding all this stuff?” “because your network is bad”)
RP: over the years we’ve reduced the number of useless warnings to the point
    where warnings are actual warnings now. maybe it wouldn’t be bad to have
    a warning for this situation
PaulB: some layers cause lots of warnings to be emitted
RP: heads up for 3.2, lots of warnings are errors now in poky
Timo: very similar to gcc’s situation, code warnings are ignored, code
    doesn’t work, so now more warnings are fatal for the build
RP: 5 years ago i wouldn’t have taken that patch, now i think it’s
    reasonable
PaulB: okay, I’ll put a patch together

RP: getting back to JPEW’s issue, i think we’ll simply need a config
    option. for the future I see bitbake needing some sort of dynamic
    directory monitoring code, perhaps notifications from the hash equivalence
    server (if there was a mechanism for sending messages)
JPEW: true, but if event-driven (not polling)
TW: but if the network is flaky…
RP: depends on scenario, AB should work reliably, over the Internet?
    anyone’s guess
RP: maybe a username/password, or web token, or some other auth method

PaulB: what would it take to replace pseudo with some sort of container
    (flakiness with LD_PRELOAD issues and others). 2 issues: 1: build needs
    the sub-uids and sub-gids needed before the build 2: even with root i
    don’t think you can run mknod
Timo: could be bind-mount in the devices and then add groups, we’d like to
    keep it rootless if possible
PaulB: but even with root we can’t create device nodes (device name-space
    inside and outside the contain is the same (shared?))
Randy: is the same namespace used on purpose?
JPEW: not sure how to name-space physical devices?
PaulB: if mknod is the only thing that could be an issue, then it’s the only
    thing we’ll need to get working with LD_PRELOAD
RP: not just that, but anything mknod would create would need to be wrapped
PaulB: oh, true
Scott: do we need the full system image in the container?
Timo: no
RP: all things considered, pseudo is still holding up pretty well, is it worth
    wholesale replacement?
PaulB: still seeing seccomp issue
RP: there is a work-around for the seccomp issue
Scott: i like the idea of using the container since it will definitely avoid
    host contamination
Randy: a container would be faster too?
JPEW: we still have to record everything
RP: won’t completely solve the host contamination thing because the
    cross-compile is still happening on the host, depends on which problem
    we’re solving (LD_PRELOAD? host contamination? performance? …)
PaulB: my issue was mostly LD_PRELOAD, and wic. bitbake runs inside fakeroot,
    but wic doesn’t
RP: true. i think the tools need to be more granular

Re: [yocto] Yocto Technical Team Minutes, Engineering Sync, for June 23 2020

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 10442 bytes --]

Le mar. 23 juin 2020 à 21:05, Trevor Woerner <twoerner@gmail.com> a écrit :

> Yocto Technical Team Minutes, Engineering Sync, for June 23 2020
> archive:
> https://docs.google.com/document/d/1ly8nyhO14kDNnFcW2QskANXW3ZT7QwKC5wWVDg9dDH4/edit
>
> == disclaimer ==
> Best efforts are made to ensure the below is accurate and valid. However,
> errors sometimes happen. If any errors or omissions are found, please feel
> free to reply to this email with any corrections.
>
> == attendees ==
> Trevor Woerner, Stephen Jolly, Jan-Simon Möller, Armin Kuster, Jeremy
> Puhlman, Trevor Gamblin, Scott Murray, Richard Purdie, Steve Sakoman, Bruce
> Ashfield, Paul Baker, Randy MacLeod, Peter Kjellerstedt, David Reyna, Mark
> Morton, Tim Orling, Joshua Watt, Michael Halstead, Vineela (?), Alejandro H
>
> == notes ==
> - 3.1.1 was released last week
> - 3.2.1 out of QA, looks clean, for TSC review, release probably in next
> few
>     days
> - still issues with AB
> - still looking for more maintainers
> - lots of unassigned M2 bugs
> - new testing manual
> - ELC + devday next week
>
> == general ==
> TW: meeting next week?
> Stephen: will look if there’s a timezone/agenda conflict…
> (various input)
> Stephen: okay, next week’s meeting cancelled
>
> David: ELC, virtual booth and slack session, looking for people to hang out
> TW: do we have to be registered for ELC?
> David: no, certainly not for slack, Tracey and I will work it out. email to
>     come later today with details, contact me if interested
>
> RP: links in status report to devday and to new test manual. please have
>     a look through the test manual. kudos to Mark for pulling it together
>     (always getting questions about testing). this was started a while back
>     with ScottR
>
> RP: re AB. still seeing intermittent issues, causes lots of builds to fail.
>     impacts me and SteveS. couple issues with repositories and/or branches
>     (upstream) disappearing.
>
> RP: issues around the “stdin” bug (wrt testing)
>
> RP: Ross still working on RCU bug
>
> RP: Alex working on a “warnings” issue
>
> Scott: can we still add 3.2 features?
> RP: nobody’s mentioning anything. we’ve never been so bleeding edge (AUH
>     kudos: AlexK). apt upgrade. Armin working on dhcp and bind updates.
> But no
>     “3.2 sound-bite”
> JPEW: multiconfig fixes
> RP: true. and also some RDEPENDS support (should be straight-forward except
>     when recursing)
> JPEW: not sold on the feature
> RP: simple use-case is to put 2 toolchains into one sdk. my answer is to
> build
>     the two then combine later (with other code). i agree that we might
> need
>     to look for a different approach
> Timo: the code to bring it all together could do other stuff too, but i
> agree
>     with Joshua that “there be dragons”
>
> Timo: perl has a new release 5.32, we might want to get that in
> Scott: but it has some new language features, might not just be a point
>     release
> Timo: haven’t dug in, we’ve had issues with perl so hopefully things are
>     better
>
> RP: 3.2. held up due to valgrind regressions, new distros (gcc 10 fixes),
>     reproducible builds, etc… so there’s lots of good stuff going in, just
>     nothing “new” (i.e. a feature), but that’s okay
>
> Scott: new autoconf (Ross)?
> RP: don’t think it’s released yet, we’re testing the pre-release. if
>     upstream releases then we’ll upgrade, but don’t know their timescale
>
> RP: trying to reduce patch overhead (AlexK has been helping). it has been
>     slowing moving down, but recently up-ticked a tiny bit. we’ll never get
>     to 0, but every reduction helps
>
> Timo: was looking at Sphinx update yesterday. looks good, some glitches.
>     noticed the doc revision history was missing. how important is that?
> RP: i’ll ask MarkM to take a look. if it’s lost it won’t break the deal,
>     but ideally we’d keep it


It is not lost. It was not migrated yet. It is on the doc “front page “ and
i have one item in the todo list to deal with docs boilerplate, the history
section is part of that.


> MarkM: yes, that and some other things (e.g. html version doesn’t have
>     section links)
> Timo: was looking at Toaster manual, noticed some images were missing too.
>     also found a broken URL. overall very good, nice to see a modern format
>     (i.e. readthedocs)


Many thanks for looking at it Timo!


> MarkM: we’re also looking at a colour scheme, Nico simply picked the
>     default, but something else will probably look better
> RP: Nico, MarkM, and I met (it was an open meeting… *wink wink*) and
> decided
>     to move forward and are hoping to have it done for Dunfell (LTS) and
>     master
> Timo: agreed. XML is a dying art, everything looks good, kudos! (will try
> to
>     get some help from some Intel folks)
>
> JPEW: sstate found/miss cache?
> JPEW: a patch to not go out and look for every sstate object, keep a cache
> if
>     one had already been found
> RP: the found cache isn’t the issue, the not-found could be an issue
>     especially where multiple builds are using the same sstate. we don’t
>     have a mechanism to generate an sstate object in isolation, but it
> might
>     be a step away
> JPEW: but it’s the not-found that is the slow path. if the sstate object
>     isn’t there, the build goes out and searches the filesystem for the
>     missing object
> RP: maybe we could make the cache time-limited (e.g. 30 seconds?)
> JPEW: yes, that might help. it’s like there’s 2 scenarios: local builds
>     with one builder, group builds
> PaulB: maybe we could simply watch the sstate directory for changes?
> PaulB: there’s also an issue of a remote sstate cache over https where
>     packets are dropped so it fails, the machine builds it locally. could
> be
>     a “time of check” vs “time of use” issue: the check says it’s
>     there, but at use time the network blips and then it’s not retrieved,
>     and then there’s no retry
> JPEW: we saw same issue here: solution for us was to run bitbake twice:
> once
>     with the setscene only, then once without it
> PaulB: we’re trying to offer our customers a common sstate but we can’t
>     necessarily ask users to do the “2 bitbake” build
> RP: maybe we need a config setting
> PaulB: i think the issue is we’re checking a URL twice, it’s the time
>     difference between checking and using that is the issue. perhaps we
> could
>     just do the poll once?
> RP: absolutely not, that’s effectively not possible. decisions are made
>     based on those checks so the two fetches have to be separate. if we
>     combine them, then there’s no savings, each build would be very long
> PaulB: perhaps better reporting
> RP: people are just going to set the config to ignore the warning then
> builds
>     will fail and it’ll look like the build is flaky. i think the answer
>     is we’re going to have to accept a config option, but we’ll have to
>     document it and be aware
> Alejandro: could we switch run-queue classes (by a flag for e.g.)
> RP: the patch isn’t the issue, it’s a philosophical question. i’m
>     worried about the implications from a usability standpoint (e.g. “why
> is
>     my build rebuilding all this stuff?” “because your network is bad”)
> RP: over the years we’ve reduced the number of useless warnings to the
> point
>     where warnings are actual warnings now. maybe it wouldn’t be bad to
> have
>     a warning for this situation
> PaulB: some layers cause lots of warnings to be emitted
> RP: heads up for 3.2, lots of warnings are errors now in poky
> Timo: very similar to gcc’s situation, code warnings are ignored, code
>     doesn’t work, so now more warnings are fatal for the build
> RP: 5 years ago i wouldn’t have taken that patch, now i think it’s
>     reasonable
> PaulB: okay, I’ll put a patch together
>
> RP: getting back to JPEW’s issue, i think we’ll simply need a config
>     option. for the future I see bitbake needing some sort of dynamic
>     directory monitoring code, perhaps notifications from the hash
> equivalence
>     server (if there was a mechanism for sending messages)
> JPEW: true, but if event-driven (not polling)
> TW: but if the network is flaky…
> RP: depends on scenario, AB should work reliably, over the Internet?
>     anyone’s guess
> RP: maybe a username/password, or web token, or some other auth method
>
> PaulB: what would it take to replace pseudo with some sort of container
>     (flakiness with LD_PRELOAD issues and others). 2 issues: 1: build needs
>     the sub-uids and sub-gids needed before the build 2: even with root i
>     don’t think you can run mknod
> Timo: could be bind-mount in the devices and then add groups, we’d like to
>     keep it rootless if possible
> PaulB: but even with root we can’t create device nodes (device name-space
>     inside and outside the contain is the same (shared?))
> Randy: is the same namespace used on purpose?
> JPEW: not sure how to name-space physical devices?
> PaulB: if mknod is the only thing that could be an issue, then it’s the
> only
>     thing we’ll need to get working with LD_PRELOAD
> RP: not just that, but anything mknod would create would need to be wrapped
> PaulB: oh, true
> Scott: do we need the full system image in the container?
> Timo: no
> RP: all things considered, pseudo is still holding up pretty well, is it
> worth
>     wholesale replacement?
> PaulB: still seeing seccomp issue
> RP: there is a work-around for the seccomp issue
> Scott: i like the idea of using the container since it will definitely
> avoid
>     host contamination
> Randy: a container would be faster too?
> JPEW: we still have to record everything
> RP: won’t completely solve the host contamination thing because the
>     cross-compile is still happening on the host, depends on which problem
>     we’re solving (LD_PRELOAD? host contamination? performance? …)
> PaulB: my issue was mostly LD_PRELOAD, and wic. bitbake runs inside
> fakeroot,
>     but wic doesn’t
> RP: true. i think the tools need to be more granular
> 
>

[-- Attachment #2: Type: text/html, Size: 11853 bytes --]