Recipe specific sysroot: handling recipes creating same files

Recipe specific sysroot: handling recipes creating same files

[Andreas Müller]

Hi,

I have the case that two recipes are trying to install same files.

In my case:

samba and libldb install

/usr/lib/python2.7/site-packages/_ldb_text.py

I bet samba doe that due to missing dependency on libldb but I don't
want to discuss that here - I'll find a proper solution.

What I would like to discuss is that both recipes build fine without
an error. The error comes up on gvfs depending on both. gvfs fails
with cryptic python backtrace:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/home/superandy/data/oe-core/sources/openembedded-core/meta/classes/staging.bbclass',
lineno: 623, function: extend_recipe_sysroot
     0619:                    dest = newmanifest[l]
     0620:                    if l.endswith("/"):
     0621:                        staging_copydir(l, targetdir, dest, seendirs)
     0622:                        continue
 *** 0623:                    staging_copyfile(l, targetdir, dest,
postinsts, seendirs)
     0624:
     0625:    for f in fixme:
     0626:        if f == '':
     0627:            staging_processfixme(fixme[f], recipesysroot,
recipesysroot, recipesysrootnative, d)
File: '/home/superandy/data/oe-core/sources/openembedded-core/meta/classes/staging.bbclass',
lineno: 269, function: staging_copyfile
     0265:        os.symlink(linkto, dest)
     0266:        #bb.warn(c)
     0267:    else:
     0268:        try:
 *** 0269:            os.link(c, dest)
     0270:        except OSError as err:
     0271:            if err.errno == errno.EXDEV:
     0272:                bb.utils.copyfile(c, dest)
     0273:            else:
Exception: FileExistsError: [Errno 17] File exists:
'/home/superandy/tmp/oe-core-glibc/sysroots-components/cortexa7t2hf-neon-vfpv4/libldb/usr/lib/python2.7/site-packages/_ldb_text.py'
-> '/home/superandy/tmp/oe-core-glibc/work/cortexa7t2hf-neon-vfpv4-angstrom-linux-gnueabi/gvfs/1.28.3-r0/recipe-sysroot/usr/lib/python2.7/site-packages/_ldb_text.py'

This needs love: One can guess that libldb is trying to install stuff
already there - nothing mentions samba and the error pops up for gvfs
which does nothing really wrong. I consider this as bug introduced by
RSS.

Andreas

Re: Recipe specific sysroot: handling recipes creating same files

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On 19 February 2017 at 22:04, Andreas Müller <schnitzeltony@googlemail.com>
wrote:

> This needs love: One can guess that libldb is trying to install stuff
> already there - nothing mentions samba and the error pops up for gvfs
> which does nothing really wrong. I consider this as bug introduced by
> RSS.
>

Yes: without RSS this would result in a fatal error when the second recipe
wrote to the sysroot.  Can you file a bug?

Ross

[-- Attachment #2: Type: text/html, Size: 864 bytes --]

Re: Recipe specific sysroot: handling recipes creating same files

[Patrick Ohly]

On Mon, 2017-02-20 at 17:08 +0000, Burton, Ross wrote:
> 
> On 19 February 2017 at 22:04, Andreas Müller
> <schnitzeltony@googlemail.com> wrote:
>         This needs love: One can guess that libldb is trying to
>         install stuff
>         already there - nothing mentions samba and the error pops up
>         for gvfs
>         which does nothing really wrong. I consider this as bug
>         introduced by
>         RSS.
> 
> Yes: without RSS this would result in a fatal error when the second
> recipe wrote to the sysroot.  Can you file a bug?

So it is still considered an error when two recipes produce the same
file?

One (IMHO valid) use-case for allowing this are configuration packages.
You could have a /etc/motd packaged in foo-motd and another in bar-motd
with different content, and then build different images where a suitable
motd config package is added.

With a single sysroot, one had to introduce alternatives, which is more
complicated and introduces unnecessary symlinks in read-only images.
With RSS, it is possible more easily.

Having said that, the error report generated when files overlap in the
same real sysroot definitely needs to be improved.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Recipe specific sysroot: handling recipes creating same files

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

On 20 February 2017 at 21:46, Patrick Ohly <patrick.ohly@intel.com> wrote:

> So it is still considered an error when two recipes produce the same
> file?
>

Only if they both installed to the sysroot.  As you say, RSS makes the
build and use of these deterministic, so some of the checks need to be
reviewed - we're obviously missing some and possibly some can go away.

Ross

[-- Attachment #2: Type: text/html, Size: 802 bytes --]