Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Richard Purdie]

On Thu, 2013-09-05 at 06:15 +0000, nitin.a.kamble@intel.com wrote:
> From: Nitin A Kamble <nitin.a.kamble@intel.com>
> 
> When this recipe is rebuilt for a different machine with same arch,
> then the previous change to EXTRA_OECONF was causing rerun of
> all the tasks starting from do_configure.
> 
> This commit avoids this behavior, needing only rerun of the
> do_populate_sysroot task. This is achieved by letting the mesa
> recipe build the GL components provided by the emgd-driver-bin
> recipe, but avoid these conflicting files getting populated in
> the machine specific sysroot.
> 
> Fixes bug:
> [YOCTO #5120]
> 
> Signed-off-by: Nitin A Kamble <nitin.a.kamble@intel.com>
> ---
>  common/recipes-graphics/mesa/mesa_9.1.6.bbappend | 69 +++++++++++++++++-------
>  1 file changed, 49 insertions(+), 20 deletions(-)
> 
> diff --git a/common/recipes-graphics/mesa/mesa_9.1.6.bbappend b/common/recipes-graphics/mesa/mesa_9.1.6.bbappend
> index b92831d..8024f0f 100644
> --- a/common/recipes-graphics/mesa/mesa_9.1.6.bbappend
> +++ b/common/recipes-graphics/mesa/mesa_9.1.6.bbappend
> @@ -1,24 +1,53 @@
> -
>  # The emgd binary driver also provides egl, gles1, gles2 library & headers.
> -# To avoid conflict disable egl, gles1, gles2 from meta-dri if the BSP image
> -# is bundling the emgd driver.
> +# To avoid conflict do not populate sysroot with  egl, gles1, gles2 files
> +# from mesa recipe if the BSP image is bundling the emgd driver.
> +
> +EXCLUDE_FROM_STAGING="${libdir}/libGLESv2.so.2 \
> +                             ${libdir}/libEGL.so.1 \
> +                             ${libdir}/libGLESv2.so \
> +                             ${libdir}/libEGL.so \
> +                             ${libdir}/pkgconfig/egl.pc \
> +                             ${libdir}/pkgconfig/glesv2.pc \
> +                             ${includedir}/KHR/khrplatform.h \
> +                             ${includedir}/GLES/glext.h \
> +                             ${includedir}/GLES/glplatform.h \
> +                             ${includedir}/GLES/gl.h \
> +                             ${includedir}/EGL/eglext.h \
> +                             ${includedir}/EGL/eglplatform.h \
> +                             ${includedir}/EGL/egl.h \
> +                             ${includedir}/GLES2/gl2.h \
> +                             ${includedir}/GLES2/gl2platform.h \
> +                             ${includedir}/GLES2/gl2ext.h \
> +                             "
> +
> +# override the function from the staging.bbclass
> +# An exclude list is conveyed to the tar
> +sysroot_stage_dir () {
> +        src="$1"
> +        dest="$2"
> +        # if the src doesn't exist don't do anything
> +        if [ ! -d "$src" ]; then
> +                 return
> +        fi
>  
> -python __anonymous () {
> -    import re
> -    xserver = d.getVar('XSERVER', True)
> -    if xserver and 'emgd-driver-bin' in xserver.split(' '):
> -        extra_oeconf = d.getVar('EXTRA_OECONF', True).split()
> -        take_out = ["--enable-egl", "--enable-gles1", "--enable-gles2"]
> -        put_in = ["--disable-egl", "--disable-gles1", "--disable-gles2"]
> -        pattern = re.compile("--with-egl-platforms")
> -        new_extra_oeconf = [ ]
> -        for i in extra_oeconf:
> -            if ( i not in take_out ) and ( not pattern.match(i)):
> -                new_extra_oeconf.append(i)
> -        for i in put_in:
> -            new_extra_oeconf.append(i)
> +        # We only want to stage the contents of $src if it's non-empty so first rmdir $src
> +        # then if it still exists (rmdir on non-empty dir fails) we can copy its contents
> +        rmdir "$src" 2> /dev/null || true
> +        # However we always want to stage a $src itself, even if it's empty
> +        mkdir -p "$dest"
> +
> +        if [ -d "$src" ]; then
> +                tar -X ${T}/exclude_from_staging -cf - -C "$src" -ps . | tar -xf - -C "$dest"
> +        fi
> +}
>  
> -        d.setVar('EXTRA_OECONF', ' '.join(new_extra_oeconf))
> -        depends = d.getVar('DEPENDS', True)
> -        d.setVar('DEPENDS', depends + " emgd-driver-bin")
> +# here the exclude list for tar is prepared
> +sysroot_stage_all_prepend() {
> +        truncate --size 0 ${T}/exclude_from_staging
> +        if [[ "${XSERVER}" =~ "emgd-driver-bin" ]]; then
> +                for i in ${EXCLUDE_FROM_STAGING}
> +                do
> +                        echo ${D}${i} >> ${T}/exclude_from_staging
> +                done
> +        fi
>  }

This is certainly better and is heading in the right direction. The two
issues I can see are:

a) do_populate_sysroot still needs to run again
b) do_packagedata installs pkgdata (shlibs) which says mesa provides 
   some of these libs yet is no shouldn't for the given MACHINE.

I did think this shouldn't be too difficult to fix however life is never
quite straightforward.

For a), rather than change things before we create the sstate package, I
was thinking we could change than at installation of the sstate package.
We have SSTATEPOSTINSTFUNCS however these run after we have placed the
files in the sysroot so its too late. SSTATEPREINSTFUNCS sounds better
but this actually runs before unpacking the sstate files which is too
early for us.

There is only one user of SSTATEPREINSTFUNCS (useradd.bbclass) so I'm
going to propose we rename that one to a more logical
SSTATEPREUNPACKFUNCS and add PREINSTFUNCS at a more sensible place. We
can then add a SSTATEPREINSTFUNCS to the mesa recipe which for
do_populate_sysroot and do_populate_sysroot_setscene deletes the files
in question. That should resolve a).

b) is more tricky since pkgdata is arch specific, not machine specific.
I had thought it was machine specific but checking now, its not.

To be quite honest, I don't like what pkgdata does at the moment. We
have cleaned things up a lot compared to where they used to be but the
search process the system uses to find information is truly horrid.

It would be rather tempting to decide to install pkgdata in a machine
specific directory much like we do with populate_sysroot already. There
would be minimal performance overhead since the files are small and we
could then bin all the "search all, then arch, then machine" type lookup
code we have right now. If we did make it machine specific, we could
then use the same trick we use in a) but for do_packagedata and instead
of removing files, it would run some carefully crafted sed operations.

Right now, we might not need to fix this 100% properly. If emgd installs
itself as machine specific, it should be first in the providers search
paths so the second lot of pkgdata should get ignored. emgd is of course
not machine specific at the moment, nor should it be apart from this
issue :/.

This also raises some other questions. The existing code in meta-intel
adds EMGD into the DEPENDS. It does this so the normal PROVIDES for
virtual/libgl and friends don't have to change. It does mean the sstate
checksums for anything depending on mesa change however. If we don't
have the dependency, things may try and build before emgd is built,
libGL would be missing and things would break.

So somehow we need to add in the dependency let leave the sstate
checksums unchanged. This is possible in the sstate checksum validation
code but its probably easiest to change that in the core, not
meta-intel :/.

I'd also appreciate someone (Ross maybe?) confirming that if we build
application A against mesa, then change over to a machine that uses emgd
and swizzle the libs around in the sysroot, do we need to recompile the
app? This approach is assuming we do not need to do that. If we do,
there is a different approach we need to take.

So in summary, this problem is far from simple, we do have most of the
pieces we need to fix it with some tweaking but some of the changes are
invasive and might need to be deferred to 1.6. As it stands, the change
will cause issues due to the changed DEPENDS. We do need to think
carefully about this and solve it once and for all in a way which works
optimally but the existing workaround might be best until we can pull
all the better pieces together.

I hope this was a relatively clear explanation of the problem, let me
know if I need to clarify anything. I've cc'd OE-Core since I think more
people probably need to see/think about this.

Cheers,

Richard

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Richard Purdie]

On Thu, 2013-09-05 at 09:33 +0100, Richard Purdie wrote:
> This is certainly better and is heading in the right direction. The two
> issues I can see are:
> 
> a) do_populate_sysroot still needs to run again
> b) do_packagedata installs pkgdata (shlibs) which says mesa provides 
>    some of these libs yet is no shouldn't for the given MACHINE.
> 
> I did think this shouldn't be too difficult to fix however life is never
> quite straightforward.
> 
> For a), rather than change things before we create the sstate package, I
> was thinking we could change than at installation of the sstate package.
> We have SSTATEPOSTINSTFUNCS however these run after we have placed the
> files in the sysroot so its too late. SSTATEPREINSTFUNCS sounds better
> but this actually runs before unpacking the sstate files which is too
> early for us.
> 
> There is only one user of SSTATEPREINSTFUNCS (useradd.bbclass) so I'm
> going to propose we rename that one to a more logical
> SSTATEPREUNPACKFUNCS and add PREINSTFUNCS at a more sensible place. We
> can then add a SSTATEPREINSTFUNCS to the mesa recipe which for
> do_populate_sysroot and do_populate_sysroot_setscene deletes the files
> in question. That should resolve a).
> 
> b) is more tricky since pkgdata is arch specific, not machine specific.
> I had thought it was machine specific but checking now, its not.
> 
> To be quite honest, I don't like what pkgdata does at the moment. We
> have cleaned things up a lot compared to where they used to be but the
> search process the system uses to find information is truly horrid.
> 
> It would be rather tempting to decide to install pkgdata in a machine
> specific directory much like we do with populate_sysroot already. There
> would be minimal performance overhead since the files are small and we
> could then bin all the "search all, then arch, then machine" type lookup
> code we have right now. If we did make it machine specific, we could
> then use the same trick we use in a) but for do_packagedata and instead
> of removing files, it would run some carefully crafted sed operations.
> 
> Right now, we might not need to fix this 100% properly. If emgd installs
> itself as machine specific, it should be first in the providers search
> paths so the second lot of pkgdata should get ignored. emgd is of course
> not machine specific at the moment, nor should it be apart from this
> issue :/.
> 
> This also raises some other questions. The existing code in meta-intel
> adds EMGD into the DEPENDS. It does this so the normal PROVIDES for
> virtual/libgl and friends don't have to change. It does mean the sstate
> checksums for anything depending on mesa change however. If we don't
> have the dependency, things may try and build before emgd is built,
> libGL would be missing and things would break.
> 
> So somehow we need to add in the dependency let leave the sstate
> checksums unchanged. This is possible in the sstate checksum validation
> code but its probably easiest to change that in the core, not
> meta-intel :/.
> 
> I'd also appreciate someone (Ross maybe?) confirming that if we build
> application A against mesa, then change over to a machine that uses emgd
> and swizzle the libs around in the sysroot, do we need to recompile the
> app? This approach is assuming we do not need to do that. If we do,
> there is a different approach we need to take.
> 
> So in summary, this problem is far from simple, we do have most of the
> pieces we need to fix it with some tweaking but some of the changes are
> invasive and might need to be deferred to 1.6. As it stands, the change
> will cause issues due to the changed DEPENDS. We do need to think
> carefully about this and solve it once and for all in a way which works
> optimally but the existing workaround might be best until we can pull
> all the better pieces together.
> 
> I hope this was a relatively clear explanation of the problem, let me
> know if I need to clarify anything. I've cc'd OE-Core since I think more
> people probably need to see/think about this.

It actually get worse (or maybe slightly better) when you think about
the shlibs code.

Imagine application A links against libGL. It therefore gets an RDEPENDS
on the package that provides libGL. With emgd in the mix, the package
providing libGL changes and as things stand today, I suspect it gets
rebuild.

So lets assume that we add leave the mesa pkgdata/shlibs code in place
and actively stop emgd putting conflicting shlibs provides in place
(easy to do).

We could then make the emgd-binary packages RPROVIDE/RCONFLICT the right
mesa packages. We'd then need to make sure the package manager behaviour
is correct in that:

a) If an image is mesa based, installing the app pulls in libGL as
normal, doesn't touch emgd.
b) In an EMGD image, installing the app pulls in libgl, however XSERVER
also pulls in emgd-binary and this then replaces the libGL package.

In theory this should all be possible and would resolve the problem in
b) in my original email without the need to make pkgdata machine
specific (we should do that for other reasons but we're getting off
track).

So perhaps there is a way out of this but it isn't simple...

Cheers,

Richard

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Otavio Salvador]

On Thu, Sep 5, 2013 at 5:33 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
....
> This also raises some other questions. The existing code in meta-intel
> adds EMGD into the DEPENDS. It does this so the normal PROVIDES for
> virtual/libgl and friends don't have to change. It does mean the sstate
> checksums for anything depending on mesa change however. If we don't
> have the dependency, things may try and build before emgd is built,
> libGL would be missing and things would break.
>
> So somehow we need to add in the dependency let leave the sstate
> checksums unchanged. This is possible in the sstate checksum validation
> code but its probably easiest to change that in the core, not
> meta-intel :/.
>
> I'd also appreciate someone (Ross maybe?) confirming that if we build
> application A against mesa, then change over to a machine that uses emgd
> and swizzle the libs around in the sysroot, do we need to recompile the
> app? This approach is assuming we do not need to do that. If we do,
> there is a different approach we need to take.

In theory both cases are possible; the API must be the same but we
cannot guarantee ABI will match.

> So in summary, this problem is far from simple, we do have most of the
> pieces we need to fix it with some tweaking but some of the changes are
> invasive and might need to be deferred to 1.6. As it stands, the change
> will cause issues due to the changed DEPENDS. We do need to think
> carefully about this and solve it once and for all in a way which works
> optimally but the existing workaround might be best until we can pull
> all the better pieces together.
>
> I hope this was a relatively clear explanation of the problem, let me
> know if I need to clarify anything. I've cc'd OE-Core since I think more
> people probably need to see/think about this.

We have same issues in meta-fsl-arm; there we made mesa and gpu
packages machine specific as we /mangle/ them.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Burton, Ross]

On 5 September 2013 09:33, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I'd also appreciate someone (Ross maybe?) confirming that if we build
> application A against mesa, then change over to a machine that uses emgd
> and swizzle the libs around in the sysroot, do we need to recompile the
> app? This approach is assuming we do not need to do that. If we do,
> there is a different approach we need to take.

So what's happening is that we're revising the original OpenGL policy
that I posted in October last year:

http://lists.openembedded.org/pipermail/openembedded-core/2012-October/070122.html

tl;dr: I proposed that 1) we rename all GL packages so they identify
their provider (ie libgl-mesa, we do that), 2) no dependencies on
specific GL implementations, 3) only Mesa stages.  The theory was that
Mesa can be considered a canonical implementation of the various GL
platforms and that we can *build* against Mesa but *install* the right
hardware-specific GL implementation into the images.

Nice in theory and it should work, but sadly that isn't the case - the
thread has more details.  Short version is that some applications *do*
rely on specific GL headers, so you can't just swap them around in
reality.

Maybe it is time to have a mesa-gl recipe alongside mesa that *just*
builds the GL libraries.  EMGD can depend on it for the driver modules
it installs, and presumably other vendors with binary drivers can
install it for the software rendering/GLX support (Otavio etc, please
step in here!).

Ross

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Otavio Salvador]

On Fri, Sep 6, 2013 at 3:36 PM, Burton, Ross <ross.burton@intel.com> wrote:
> On 5 September 2013 09:33, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> I'd also appreciate someone (Ross maybe?) confirming that if we build
>> application A against mesa, then change over to a machine that uses emgd
>> and swizzle the libs around in the sysroot, do we need to recompile the
>> app? This approach is assuming we do not need to do that. If we do,
>> there is a different approach we need to take.
>
> So what's happening is that we're revising the original OpenGL policy
> that I posted in October last year:
>
> http://lists.openembedded.org/pipermail/openembedded-core/2012-October/070122.html
>
> tl;dr: I proposed that 1) we rename all GL packages so they identify
> their provider (ie libgl-mesa, we do that), 2) no dependencies on
> specific GL implementations, 3) only Mesa stages.  The theory was that
> Mesa can be considered a canonical implementation of the various GL
> platforms and that we can *build* against Mesa but *install* the right
> hardware-specific GL implementation into the images.
>
> Nice in theory and it should work, but sadly that isn't the case - the
> thread has more details.  Short version is that some applications *do*
> rely on specific GL headers, so you can't just swap them around in
> reality.
>
> Maybe it is time to have a mesa-gl recipe alongside mesa that *just*
> builds the GL libraries.  EMGD can depend on it for the driver modules
> it installs, and presumably other vendors with binary drivers can
> install it for the software rendering/GLX support (Otavio etc, please
> step in here!).

Yes; we have both situations at Freescale ARM BSP.

For the AMD GPU binaries, which are used in MX5 SoCs we *just* use
mesa GL and do software rendering. For Vivante GPU binaries, used for
MX6, we use mesa GL *headers* but Vivante provides a libGL binary.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Burton, Ross]

On 7 September 2013 00:56, Otavio Salvador <otavio@ossystems.com.br> wrote:
>> Maybe it is time to have a mesa-gl recipe alongside mesa that *just*
>> builds the GL libraries.  EMGD can depend on it for the driver modules
>> it installs, and presumably other vendors with binary drivers can
>> install it for the software rendering/GLX support (Otavio etc, please
>> step in here!).
>
> Yes; we have both situations at Freescale ARM BSP.
>
> For the AMD GPU binaries, which are used in MX5 SoCs we *just* use
> mesa GL and do software rendering.

So mesa-gl will work fine there.

> For Vivante GPU binaries, used for
> MX6, we use mesa GL *headers* but Vivante provides a libGL binary.

My, that's horrific.  I'd forgotten about this, for good reason clearly.

Ross

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Richard Purdie]

On Mon, 2013-09-09 at 12:36 +0100, Burton, Ross wrote:
> On 7 September 2013 00:56, Otavio Salvador <otavio@ossystems.com.br> wrote:
> >> Maybe it is time to have a mesa-gl recipe alongside mesa that *just*
> >> builds the GL libraries.  EMGD can depend on it for the driver modules
> >> it installs, and presumably other vendors with binary drivers can
> >> install it for the software rendering/GLX support (Otavio etc, please
> >> step in here!).
> >
> > Yes; we have both situations at Freescale ARM BSP.
> >
> > For the AMD GPU binaries, which are used in MX5 SoCs we *just* use
> > mesa GL and do software rendering.
> 
> So mesa-gl will work fine there.
> 
> > For Vivante GPU binaries, used for
> > MX6, we use mesa GL *headers* but Vivante provides a libGL binary.
> 
> My, that's horrific.  I'd forgotten about this, for good reason clearly.

So I now have actual code which is better than any thought experiment :)

First, I have this applied:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t1&id=3e2c058ffab5b4e92523dac3078c1bc02d0eb8b8

This creates a machine specific pkgdata/shlibs directory instead of the
current multiple directories and iterations we use. Its not complete yet
(buildhistory is broken for example), its another proof of concept. What
this does is ensure that the main TUNE_PKGARCH directory doesn't get
corrupted with emgd shlibs below, accidentally or otherwise.

We then have two PoC patches one for OE-Core, one for meta-intel. These
put the gl bits into "i586-emgd" instead of "i586" so they're isolated.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t1&id=258e068bbb309877b6e36eeec1107806fc710cde
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/metaintel&id=917286eebc5e4eeb13fcc1b0db6adb50ccd847a0

I hacked emenlow so it shared qemux86's TUNE_PKGARCH for testing
purposes. You can then build both qemux86 and emenlow in the same build
directory without them trampling clutter.

cogl would also need to inherit the gl class so its rough around the
edges but basically works.

Thoughts?

Cheers,

Richard

Re: [Patch-v2 1/1] mesa: avoid unnecessary rerunning of tasks

[Otavio Salvador]

On Wed, Sep 11, 2013 at 1:46 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Mon, 2013-09-09 at 12:36 +0100, Burton, Ross wrote:
>> On 7 September 2013 00:56, Otavio Salvador <otavio@ossystems.com.br> wrote:
>> >> Maybe it is time to have a mesa-gl recipe alongside mesa that *just*
>> >> builds the GL libraries.  EMGD can depend on it for the driver modules
>> >> it installs, and presumably other vendors with binary drivers can
>> >> install it for the software rendering/GLX support (Otavio etc, please
>> >> step in here!).
>> >
>> > Yes; we have both situations at Freescale ARM BSP.
>> >
>> > For the AMD GPU binaries, which are used in MX5 SoCs we *just* use
>> > mesa GL and do software rendering.
>>
>> So mesa-gl will work fine there.
>>
>> > For Vivante GPU binaries, used for
>> > MX6, we use mesa GL *headers* but Vivante provides a libGL binary.
>>
>> My, that's horrific.  I'd forgotten about this, for good reason clearly.
>
> So I now have actual code which is better than any thought experiment :)
>
> First, I have this applied:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t1&id=3e2c058ffab5b4e92523dac3078c1bc02d0eb8b8
>
> This creates a machine specific pkgdata/shlibs directory instead of the
> current multiple directories and iterations we use. Its not complete yet
> (buildhistory is broken for example), its another proof of concept. What
> this does is ensure that the main TUNE_PKGARCH directory doesn't get
> corrupted with emgd shlibs below, accidentally or otherwise.
>
> We then have two PoC patches one for OE-Core, one for meta-intel. These
> put the gl bits into "i586-emgd" instead of "i586" so they're isolated.
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t1&id=258e068bbb309877b6e36eeec1107806fc710cde
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/metaintel&id=917286eebc5e4eeb13fcc1b0db6adb50ccd847a0
>
> I hacked emenlow so it shared qemux86's TUNE_PKGARCH for testing
> purposes. You can then build both qemux86 and emenlow in the same build
> directory without them trampling clutter.
>
> cogl would also need to inherit the gl class so its rough around the
> edges but basically works.
>
> Thoughts?

Sorry by delayed reply.

I did look at the changes and I did enjoy it a lot. I had something
like this in O.S. Systems fork of OpenEmbedded-Classic to share
multiple builds against compatible geode machines... so let's stop
with nostalgy and focus in 2013 ;-)

It does looks great. I like the idea however I'd prefer if you could
name it like "subarch" or something like that. For example we have
some uses for this in FSL layer:

 * xserver-xorg
 * gpu-viv
 * amd-gpu
 * gst-fsl-plugin

All these are subarch compatible but currently are marked as machine
specific (which slow down builds).

Regards,

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750