Re: package: Add cachedpath optimisation

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2013-03-14 at 20:02 -0700, Chris Larson wrote:
> 
> On Thu, Mar 14, 2013 at 6:57 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>         Currently, various standard library operations like os.walk(),
>         os.path.isdir() and os.path.islink() each call stat or lstat
>         which
>         involves a syscall into the kernel. There is no caching since
>         they could
>         conceivably have changed on disk. The result is that for
>         something like
>         the do_package task of the kernel we're spending over two
>         minutes making
>         868,000 individual stat calls for 23,000 files. This is
>         suboptimal.
>         
>         This patch adds lib/oe/cachedpath.py which are a set of
>         replacement
>         functions for these operations which use cached stat data
>         rather than
>         hitting the kernel each time. It gives a nice performance
>         improvement
>         halving the build time of the kernel do_package.
>         
>         Signed-off-by: Richard Purdie
>         <richard.purdie@linuxfoundation.org>
> 
> Have you considered having the initcache() return a cache object,
> either with the rest as methods on it, or pass that in? I think it'd
> be substantially cleaner than messing with globals, particularly given
> with globals we tend to have issues with cache lifetime and
> invalidation — if the cache was returned, it would go away when the
> object gets collected.

Agreed. Due to the way package.bbclass is structured, we currently need
a global there but that is no reason we shouldn't have a sensible
class/object structure for the new code. I've sent out a v2 which
hopefully does better in that regard.

Cheers,

Richard
>