Processing extra field requested by bitbake -u in Cache

Processing extra field requested by bitbake -u in Cache

[Ke, Liping]

Hi, all

I am looking @ bug 770 Add mechanism to enable UI's to request extra data be stored in the cache. We are now thinking about the possible solutions.

There are two choices:
1. Store extra cache fields in a separated small cache file, so non-ui mode will only fetch the base cache file, if in ui-mode, it will load the extra small cache (with the extra 3 fields, such as summary)
  Advantage: 1) Will 'not' change the cache invalidation mechanism. For non-ui mode, it will go the old path. If extra fields are required (ui-mode), we need to load the extra field into the memory.
            2) disk file size is optimized
  Disadvantage: cache code change will be big, since all data load/sync/ methods need to be modified.

2. Have two big cache files, one is for ui mode, one is for non-ui mode
  Advantage: Code change will be a little smaller. When loading, if it's non-ui mode, load smaller cache file. Otherwise, load bigger cache file.  
  Disadvantage: 1) Cache invalidation mechanism (timestamp) is broken. When mode switching, we need to invalidate all caches, reparsing, reloading time 30s.
              s) disk file size is bigger. 2*original_size (2*6M)

Any suggestions?

Thanks a lot for your help!

criping

Re: Processing extra field requested by bitbake -u in Cache

[Joshua Lock]

On Wed, 2011-04-27 at 17:34 +0800, Ke, Liping wrote:
> Hi, all
> 
> I am looking @ bug 770 Add mechanism to enable UI's to request extra
> data be stored in the cache. We are now thinking about the possible
> solutions.

For people wishing to follow:
http://bugzilla.pokylinux.org/show_bug.cgi?id=770

> 
> There are two choices:
> 1. Store extra cache fields in a separated small cache file, so non-ui mode will only fetch the base cache file, if in ui-mode, it will load the extra small cache (with the extra 3 fields, such as summary)
>   Advantage: 1) Will 'not' change the cache invalidation mechanism. For non-ui mode, it will go the old path. If extra fields are required (ui-mode), we need to load the extra field into the memory.
>             2) disk file size is optimized
>   Disadvantage: cache code change will be big, since all data load/sync/ methods need to be modified.

Why must we change the invalidation mechanisms when having a separate
cache file per UI.

My thought is that we'll have a cache file name which encodes the UI and
then run the usual invalidation tests against that cache.

Say that the UI name is known by the cache and stored in variable
currentui we can achieve this by:

-        self.cachefile = os.path.join(self.cachedir, "bb_cache.dat")
+        self.cachefile = os.path.join(self.cachedir, "bb_cache_%s.dat"
% currentui)

Also, surely we *do* need alter the cache invalidation mechanisms for
multiple files, at least insofar as we must test more than one cache
file some times.

> 
> 2. Have two big cache files, one is for ui mode, one is for non-ui mode
>   Advantage: Code change will be a little smaller. When loading, if it's non-ui mode, load smaller cache file. Otherwise, load bigger cache file.  
>   Disadvantage: 1) Cache invalidation mechanism (timestamp) is broken. When mode switching, we need to invalidate all caches, reparsing, reloading time 30s.
>               s) disk file size is bigger. 2*original_size (2*6M)

I'm not sure that 6MB disk overhead is worth the code complexity
increase by having multiple cache files stitched together.

I am definitely keen to hear others opinions here.

Cheers,
Joshua
-- 
Joshua Lock
        Yocto Build System Monkey
        Intel Open Source Technology Centre

Re: Processing extra field requested by bitbake -u in Cache

[Ke, Liping]

> Why must we change the invalidation mechanisms when having a separate
> cache file per UI.
Hi, Josh

Considering the following usage sequence, ui 1 -> non-ui (made changes) 2 -> ui (made changes)  3 -> non-ui (made changes) 4

If we use option 1, no cache invalidation needed.

But if we use option 2, we need invalidate cache each time. I collect some local feedback here, they said it's unbearable. 

if we want to reduce the unnecessary invalidation, we need very careful tricky mechanism,
such as, could it be possible that when ui->no-ui, we don't' invalidation cache when the mode switching? It's not a two step state machine,
but more steps?

And also, yes extra 6 M files are the disk is not a big problem. But actually the extra three fields are not a big burden in memory, 
it's only about size(fields of the 3) * file_no (<1000)? So the extra 6M vs extra three fields in memory is a dilemma from my me plus the extra invalidation.
That's why I feel RP's suggestion is reasonable at that time?

I am still in a dilemma...

Thanks indeed for your idea!
criping



> 
> My thought is that we'll have a cache file name which encodes the UI and
> then run the usual invalidation tests against that cache.
> 
> Say that the UI name is known by the cache and stored in variable
> currentui we can achieve this by:
> 
> -        self.cachefile = os.path.join(self.cachedir, "bb_cache.dat")
> +        self.cachefile = os.path.join(self.cachedir, "bb_cache_%s.dat"
> % currentui)
> 
> Also, surely we *do* need alter the cache invalidation mechanisms for
> multiple files, at least insofar as we must test more than one cache
> file some times.
> 
> >
> > 2. Have two big cache files, one is for ui mode, one is for non-ui mode
> >   Advantage: Code change will be a little smaller. When loading, if it's non-ui
> mode, load smaller cache file. Otherwise, load bigger cache file.
> >   Disadvantage: 1) Cache invalidation mechanism (timestamp) is broken.
> When mode switching, we need to invalidate all caches, reparsing, reloading
> time 30s.
> >               s) disk file size is bigger. 2*original_size (2*6M)
> 
> I'm not sure that 6MB disk overhead is worth the code complexity
> increase by having multiple cache files stitched together.
> 
> I am definitely keen to hear others opinions here.
> 
> Cheers,
> Joshua
> --
> Joshua Lock
>         Yocto Build System Monkey
>         Intel Open Source Technology Centre
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: Processing extra field requested by bitbake -u in Cache

[Joshua Lock]

On Thu, 2011-04-28 at 09:22 +0800, Ke, Liping wrote:
> > Why must we change the invalidation mechanisms when having a separate
> > cache file per UI.
> Hi, Josh
> 
> Considering the following usage sequence, ui 1 -> non-ui (made changes) 2 -> ui (made changes)  3 -> non-ui (made changes) 4
> 
> If we use option 1, no cache invalidation needed.

If there are changes you'll still need to invalidate the cache.
I guess you're trying  to show something like the following?

== Option 1 (addition cache file for extra fields)
launch gui
<make changes>
launch gui (cache rebuild)
use cli
<make changes>
use cli (cache rebuild)
launch gui
use cli
launch gui
<make changes>
launch gui (cache rebuild)
use cli

=> 3 cache rebuilds

== Option 2 (per ui cache files)
launch gui
<make changes>
launch gui (cache rebuild)
use cli
<make changes>
use cli (cache rebuild)
launch gui (cache rebuild)
use cli
launch gui
<make changes>
launch gui (cache rebuild)
use cli (cache rebuild)

=> 5 cache rebuilds

However I can't imagine people switching UI's this frequently.

> 
> But if we use option 2, we need invalidate cache each time. I collect some local feedback here, they said it's unbearable. 

What's unbearable? The cache rebuilding time?

I'm still not clear why we need to invalidate the cache each time we
switch UI's. The cache invalidation test is the same: if any of the
dependencies have been modified or the file doesn't exist build the
cache, else load the available cache. If we are using a per UI cache
file we just run the invalidation tests against a different file each
time the UI is changed. Or am I missing something here?

Regards,
Joshua
-- 
Joshua Lock
        Yocto Build System Monkey
        Intel Open Source Technology Centre

Re: Processing extra field requested by bitbake -u in Cache

[Ke, Liping]

> >
> > Considering the following usage sequence, ui 1 -> non-ui (made changes) 2 ->
> ui (made changes)  3 -> non-ui (made changes) 4
> >
> > If we use option 1, no cache invalidation needed.
> 
> If there are changes you'll still need to invalidate the cache.
> I guess you're trying  to show something like the following?
> 
> == Option 1 (addition cache file for extra fields)
> launch gui
> <make changes>
> launch gui (cache rebuild)
> use cli
> <make changes>
> use cli (cache rebuild)
> launch gui
> use cli
> launch gui
> <make changes>
> launch gui (cache rebuild)
> use cli
> 
> => 3 cache rebuilds
> 
> == Option 2 (per ui cache files)
> launch gui
> <make changes>
> launch gui (cache rebuild)
> use cli
> <make changes>
> use cli (cache rebuild)
> launch gui (cache rebuild)
> use cli
> launch gui
> <make changes>
> launch gui (cache rebuild)
> use cli (cache rebuild)
> 
> => 5 cache rebuilds
> 
> However I can't imagine people switching UI's this frequently.
> 
Hi, Josh
Yes, I asked whether we would like to use the image creator, some of them said yes, they will.
We're all waiting on functionality: select a image type, and add or move a recipe and then make rootfs...
If this functionality is ok, we guess image creator would be very useful.
I guess it's really very hard to judge users' usage scenario?

Yes. Reparsing will take extra 30s. I found it's a hard decision since we're always trying to improve the
Performance and reduce the disk size. 

criping
> >
> > But if we use option 2, we need invalidate cache each time. I collect some
> local feedback here, they said it's unbearable.
> 
> What's unbearable? The cache rebuilding time?