bitbake eating more than 6GB ram while parsing recipes

bitbake eating more than 6GB ram while parsing recipes

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

Hi,

I know this was reported a while ago and I had this issue for few days
with 4 GB RAM on my workstation it was hard to do anything when bitbake
was parsing and today I've "resolved" it by removing cache.

first I've tried to remove
#PARALLEL_MAKE="-j3"
#BB_NUMBER_THREADS = "2"
from my local.conf and it didn't help I still had 6 bitbake processes
4 eating ~ 1GB ad 2 with ~ 512MB. So I've removed tmp/cache except
bb_persist_data.sqlite3:

OE om-gta02@shr ~/shr-core $ du -hs tmp/cache/*
35M     tmp/cache/bb_codeparser.dat
0       tmp/cache/bb_codeparser.dat.lock.13293
0       tmp/cache/bb_codeparser.dat.lock.13294
0       tmp/cache/bb_codeparser.dat.lock.13295
0       tmp/cache/bb_codeparser.dat.lock.15708
0       tmp/cache/bb_codeparser.dat.lock.16878
0       tmp/cache/bb_codeparser.dat.lock.16879
0       tmp/cache/bb_codeparser.dat.lock.16880
0       tmp/cache/bb_codeparser.dat.lock.25605
68K     tmp/cache/bb_persist_data.sqlite3
87M     tmp/cache/default-eglibc
OE om-gta02@shr ~/shr-core $ rm -rf tmp/cache/bb_codeparser.dat*

OE om-gta02@shr ~/shr-core $ du -hs tmp/cache/default-eglibc/*
13M     tmp/cache/default-eglibc/nokia900
13M     tmp/cache/default-eglibc/om-gta02
13M     tmp/cache/default-eglibc/palmpre
13M     tmp/cache/default-eglibc/qemuarm
13M     tmp/cache/default-eglibc/qemux86
13M     tmp/cache/default-eglibc/qemux86-64
13M     tmp/cache/default-eglibc/spitz
OE om-gta02@shr ~/shr-core $ rm -rf tmp/cache/default-eglibc/

and now it's sane again and memory used is < 2GB.

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: bitbake eating more than 6GB ram while parsing recipes

[Khem Raj]

On Sun, Oct 23, 2011 at 12:57 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
> Hi,
>
> I know this was reported a while ago and I had this issue for few days
> with 4 GB RAM on my workstation it was hard to do anything when bitbake
> was parsing and today I've "resolved" it by removing cache.

yes I have exact same problem. As you add more machines to same tmpdir
build. It keeps becoming s bigger pig. And so far I have been removing
cache dir too since s parse from scratch is way faster in this case.

I think we have to profile bitbake and find the memory wastage even after
parse right now it keeps like 600M through out the build process which
could be reduced too.

>
> first I've tried to remove
> #PARALLEL_MAKE="-j3"
> #BB_NUMBER_THREADS = "2"
> from my local.conf and it didn't help I still had 6 bitbake processes
> 4 eating ~ 1GB ad 2 with ~ 512MB. So I've removed tmp/cache except
> bb_persist_data.sqlite3:
>
> OE om-gta02@shr ~/shr-core $ du -hs tmp/cache/*
> 35M     tmp/cache/bb_codeparser.dat
> 0       tmp/cache/bb_codeparser.dat.lock.13293
> 0       tmp/cache/bb_codeparser.dat.lock.13294
> 0       tmp/cache/bb_codeparser.dat.lock.13295
> 0       tmp/cache/bb_codeparser.dat.lock.15708
> 0       tmp/cache/bb_codeparser.dat.lock.16878
> 0       tmp/cache/bb_codeparser.dat.lock.16879
> 0       tmp/cache/bb_codeparser.dat.lock.16880
> 0       tmp/cache/bb_codeparser.dat.lock.25605
> 68K     tmp/cache/bb_persist_data.sqlite3
> 87M     tmp/cache/default-eglibc
> OE om-gta02@shr ~/shr-core $ rm -rf tmp/cache/bb_codeparser.dat*
>
> OE om-gta02@shr ~/shr-core $ du -hs tmp/cache/default-eglibc/*
> 13M     tmp/cache/default-eglibc/nokia900
> 13M     tmp/cache/default-eglibc/om-gta02
> 13M     tmp/cache/default-eglibc/palmpre
> 13M     tmp/cache/default-eglibc/qemuarm
> 13M     tmp/cache/default-eglibc/qemux86
> 13M     tmp/cache/default-eglibc/qemux86-64
> 13M     tmp/cache/default-eglibc/spitz
> OE om-gta02@shr ~/shr-core $ rm -rf tmp/cache/default-eglibc/
>
> and now it's sane again and memory used is < 2GB.
>
> Regards,
> --
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>