I hava an idea to Make OpenEmbedded Run Faster

I hava an idea to Make OpenEmbedded Run Faster

[杨同峰]

EveryOne: 
      By two week  of using OE , I hava notice that task was topology
sort to a queue .and  Run Task One By One. 
draw on the experience of Design of CPU . we can make a multiply thread
run of it ! 

      I notice that the biggest bottleneck of OE is do_fetch . then we
can use a mutiply thread downloader instead of wget 
      And about task schedule ,use method like this:

     bb_No1 do_fetch   do_patch do_configure do_build do_install
     bb_No2                  do_fetch  do_patch       do_configure
do_build do_install
     bb_No3                                 do_fetch      do_patch
do_configure do_build do_install
     bb_No4
do_fetch  .......

     Idea 1 :    focus on download one file one time ,and make it more
fast use mutiply thread downloader 
     Idea 2 :    make 0ther task of bb files  to run  asynchronously 
     Idea 3 :   deal with task  topology more carefully to make sure
that  the bb file that was dependend was finish before
                    this one start to do_patch (Notice : not before
do_fetch ) 
     Idea 4 :   If topology can't make sure Idea 3 ,we must tell OE how
and when  to wait 

     If the Idea can come true ,The speed can go 3 times faster then
now 

Tongfeng Yang
2007-8-27

Re: I hava an idea to Make OpenEmbedded Run Faster

[Justin Patrin]

On 8/26/07, 杨同峰 <yangtf@envision.com.cn> wrote:
> EveryOne:
>       By two week  of using OE , I hava notice that task was topology
> sort to a queue .and  Run Task One By One.
> draw on the experience of Design of CPU . we can make a multiply thread
> run of it !
>
>       I notice that the biggest bottleneck of OE is do_fetch . then we
> can use a mutiply thread downloader instead of wget
>       And about task schedule ,use method like this:
>
>      bb_No1 do_fetch   do_patch do_configure do_build do_install
>      bb_No2                  do_fetch  do_patch       do_configure
> do_build do_install
>      bb_No3                                 do_fetch      do_patch
> do_configure do_build do_install
>      bb_No4
> do_fetch  .......
>
>      Idea 1 :    focus on download one file one time ,and make it more
> fast use mutiply thread downloader
>      Idea 2 :    make 0ther task of bb files  to run  asynchronously
>      Idea 3 :   deal with task  topology more carefully to make sure
> that  the bb file that was dependend was finish before
>                     this one start to do_patch (Notice : not before
> do_fetch )
>      Idea 4 :   If topology can't make sure Idea 3 ,we must tell OE how
> and when  to wait
>
>      If the Idea can come true ,The speed can go 3 times faster then
> now
>

bitbake already does this.

-- 
Justin Patrin

Re: I hava an idea to Make OpenEmbedded Run Faster

[杨同峰]

really?? Bitbake same to be single thread . My Main Idear is like this  
when some task is compiling ,Some task is fetching ,and other maybe
do_installing .
but  bitbake don't look like this .

Tongfeng.Yang

在 2007-08-26日的 21:04 -0700，Justin Patrin写道：

> On 8/26/07, 杨同峰 <yangtf@envision.com.cn> wrote:
> > EveryOne:
> >       By two week  of using OE , I hava notice that task was topology
> > sort to a queue .and  Run Task One By One.
> > draw on the experience of Design of CPU . we can make a multiply thread
> > run of it !
> >
> >       I notice that the biggest bottleneck of OE is do_fetch . then we
> > can use a mutiply thread downloader instead of wget
> >       And about task schedule ,use method like this:
> >
> >      bb_No1 do_fetch   do_patch do_configure do_build do_install
> >      bb_No2                  do_fetch  do_patch       do_configure
> > do_build do_install
> >      bb_No3                                 do_fetch      do_patch
> > do_configure do_build do_install
> >      bb_No4
> > do_fetch  .......
> >
> >      Idea 1 :    focus on download one file one time ,and make it more
> > fast use mutiply thread downloader
> >      Idea 2 :    make 0ther task of bb files  to run  asynchronously
> >      Idea 3 :   deal with task  topology more carefully to make sure
> > that  the bb file that was dependend was finish before
> >                     this one start to do_patch (Notice : not before
> > do_fetch )
> >      Idea 4 :   If topology can't make sure Idea 3 ,we must tell OE how
> > and when  to wait
> >
> >      If the Idea can come true ,The speed can go 3 times faster then
> > now
> >
> 
> bitbake already does this.
> 

Re: I hava an idea to Make OpenEmbedded Run Faster

[杨同峰]

KEEP DOWNLOADING UNTIL ALL WAS DOWNLOAD,DON'T LET NET REST !
-------------------------------

We See that , all the  times when  in compiling . No Task is
downloading .
This is a big waste of time . 

My Idear is KEEP DOWNLOADING UNTIL ALL WAS DOWNLOAD,DON'T LET NET REST !

If the whole compile is not finish ,Keep Downling !!!!!

The one who has a big width of net ,can't understand my feeling who 
has only 5k/s width of net .

R

Re: I hava an idea to Make OpenEmbedded Run Faster

[Justin Patrin]

On 8/26/07, 杨同峰 <yangtf@envision.com.cn> wrote:
> really?? Bitbake same to be single thread . My Main Idear is like this
> when some task is compiling ,Some task is fetching ,and other maybe
> do_installing .
> but  bitbake don't look like this .
>

What version of bitbake are you using? I know that 1.8 supports multithreading.

>
> 在 2007-08-26日的 21:04 -0700，Justin Patrin写道：
>
> > On 8/26/07, 杨同峰 <yangtf@envision.com.cn> wrote:
> > > EveryOne:
> > >       By two week  of using OE , I hava notice that task was topology
> > > sort to a queue .and  Run Task One By One.
> > > draw on the experience of Design of CPU . we can make a multiply thread
> > > run of it !
> > >
> > >       I notice that the biggest bottleneck of OE is do_fetch . then we
> > > can use a mutiply thread downloader instead of wget
> > >       And about task schedule ,use method like this:
> > >
> > >      bb_No1 do_fetch   do_patch do_configure do_build do_install
> > >      bb_No2                  do_fetch  do_patch       do_configure
> > > do_build do_install
> > >      bb_No3                                 do_fetch      do_patch
> > > do_configure do_build do_install
> > >      bb_No4
> > > do_fetch  .......
> > >
> > >      Idea 1 :    focus on download one file one time ,and make it more
> > > fast use mutiply thread downloader
> > >      Idea 2 :    make 0ther task of bb files  to run  asynchronously
> > >      Idea 3 :   deal with task  topology more carefully to make sure
> > > that  the bb file that was dependend was finish before
> > >                     this one start to do_patch (Notice : not before
> > > do_fetch )
> > >      Idea 4 :   If topology can't make sure Idea 3 ,we must tell OE how
> > > and when  to wait
> > >
> > >      If the Idea can come true ,The speed can go 3 times faster then
> > > now
> > >
> >
> > bitbake already does this.
> >
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>


-- 
Justin Patrin

Re: I hava an idea to Make OpenEmbedded Run Faster

[杨同峰]

Oh , My bitbake is 1.6.6 
I'll try  1.8  thank you ! 

Re: I hava an idea to Make OpenEmbedded Run Faster

[杨同峰]

Thanks to Justin Patrin !  
I update the bitbake to 1.8.6 and  export BB_NUMBER_THREADS=10,
Now it's Run fast !!