Re: Bitbake and task offloading onto multiple cloud-based servers

[Alex J Lennon <ajlennon@dynamicdevices.co.uk>]

On 04/01/2013 21:08, Chris Larson wrote:
> 
> 
> On Fri, Jan 4, 2013 at 1:56 PM, Alex J Lennon
> <ajlennon@dynamicdevices.co.uk <mailto:ajlennon@dynamicdevices.co.uk>>
> wrote:
> 
>     Can anybody advise on whether bitbake currently supports offloading of
>     build tasks onto multiple systems? Perhaps cloud based?
> 
>     I'm thinking that it would be more efficient for me if I could bring up
>     a number of Amazon EC2 servers (or similar) then have bitbake
>     parallelise the build onto those servers to significantly reduce my
>     build times?
> 
>     I see bitbake supports a level of task parallelisation on a single box.
> 
>     Can parallelisation of build onto multiple systems be achieved?
> 
>     Is it something that should even be a goal?
> 
> 
> It's not supported today. It could be implemented, but nobody has made
> it a priority and done so.

Do you have any feeling for the level of difficulty of such an
implementation / what would have to change / how invasive it would be to
the codebase ?

I'm wondering if it could be along the lines of creating a "remote task"
class and then, say, having that class ssh into one of a pool of servers
(running a standard image with all tools preinstalled maybe) then
bitbaking the recipes for the particular and waiting on completion
before pulling back the output rpm/deb/ipkg ?

Things are usually more complex than expected when you get into the
nitty gritty though. What would the challenges be do you think?

Where would one start to look in the bitbake code to add this kind of
support in?

Thanks,

Alex


> -- 
> Christopher Larson