Re: [PATCH 2/3] package.bbclass: use oe.path.realpath()

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2013-03-12 at 11:25 +0100, Enrico Scholz wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
> >> Old implementation suffered from lot of problems; e.g.
> >> 
> >> * redundant code
> >> 
> >> * calls 'os.stat()' which references files on host; this can give wrong
> >>   results about existing/non-existing and can cause EPERM (instead of
> >>   the catched ENONENT) exceptions
> >> 
> >> * does not deal with special cases like '..' leaving the sysroot.
> >
> > Whilst these changes are good, they do come at a cost:
> >
> > post symlink package changes
> > ./perfscript -c 8c22531e491e6b0cfffaaa80d6bc75db757fc1d1
> > 49:38.46,17:12.15
> >
> > pre symlink package changes
> > ./perfscript -c 1a80329b3fcf23ecc23e409a260b9b2182652f65
> > 48:16.33,13:39.97
> >
> > So it added 1m20 to the overall build time, but more worryingly, added
> > added nearly 3m30 to the time for:
> >
> > bitbake virtual/kernel -c cleansstate; bitbake virtual/kernel
> >
> > These tests are based on the script linked from
> > https://wiki.yoctoproject.org/wiki/Performance_Test where the kernel
> > test is this is the second number above list, overall build time is the
> > first.
> >
> > Have you any time to look into this performance regression?
> 
> Well, patch replaced a call to os.path.realpath() with a more accurate
> variant honoring a sysroot. There must be more work be done to replace
> things previously solved by the operating system with custom (python)
> code.
> 
> I will try to write a C implementation of oe.path.realpath() but this
> can take some time and I do not have an idea how to integrate it into
> oe.

The issue is not an interpreted language verses C, the problem is the
shear number of syscalls. Each isdir() and islink() call means a stat
call into the kernel. For "bitbake virtual/kernel -c package", I'm
seeing 400,000 stat calls for 23,000 files. We know the filesystem
doesn't change so we could cache the stat calls and hopefully hence the
speed. I actually have some code I tried this with but its not working
as well as I'd like so I need to do some further investigation about
what is happening.

Cheers,

Richard