[RFC] bbclass for automatic generation of binary recipes

[RFC] bbclass for automatic generation of binary recipes

[Erik Botö]

Hi,

Background:
If you use OE to build software which includes e.g. a closed source
library, but still want to others to be able to build software based
on that library in a OE environment you would have to create recipes
that contains a binary version of your library + headers manually.
This would have to be done manually for each package, and as the
development of the library continues the binary recipes will have to
be manually updated to keep in sync.

Idea:
Create a new bbclass that handles this automatically as much as
possible. The basic idea is to:
- Create a tarball of ${WORKDIR}/image/ alternatively
  ${WORKDIR}/packages-split/
- By parsing the "source recipe", create a "binary recipe" that uses
that tarball

The binary recipe should work as a drop in replacement of the source
recipe. This means that the other recipes depending on the closed
source software shouldn't need to be adapted to work with one or the
other.

I'd like feedback on whether you think the idea is feasible, and if
you think it could be useful for others. And anything else you think
of.

Cheers,
Erik Botö

Re: [RFC] bbclass for automatic generation of binary recipes

[Robert Yang]

Is it similar to meta/classes/package.bbclass ?

// Robert

On 05/08/2013 01:53 PM, Erik Botö wrote:
> Hi,
>
> Background:
> If you use OE to build software which includes e.g. a closed source
> library, but still want to others to be able to build software based
> on that library in a OE environment you would have to create recipes
> that contains a binary version of your library + headers manually.
> This would have to be done manually for each package, and as the
> development of the library continues the binary recipes will have to
> be manually updated to keep in sync.
>
> Idea:
> Create a new bbclass that handles this automatically as much as
> possible. The basic idea is to:
> - Create a tarball of ${WORKDIR}/image/ alternatively
>    ${WORKDIR}/packages-split/
> - By parsing the "source recipe", create a "binary recipe" that uses
> that tarball
>
> The binary recipe should work as a drop in replacement of the source
> recipe. This means that the other recipes depending on the closed
> source software shouldn't need to be adapted to work with one or the
> other.
>
> I'd like feedback on whether you think the idea is feasible, and if
> you think it could be useful for others. And anything else you think
> of.
>
> Cheers,
> Erik Botö
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: [RFC] bbclass for automatic generation of binary recipes

[Erik Botö]

Hi,

On Wed, May 8, 2013 at 8:11 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>
> Is it similar to meta/classes/package.bbclass ?
>
> // Robert

I would say it's in addition to package.bbclass.

The idea is that the developers of the closed source software uses OE
to build their software from source, and in this build process the
files needed for a binary distribution in OE is automatically
generated. These generated files can be distributed to others that
wish to use that closed source software in OE. So it would use some of
the output from package.bbclass, like the package split, but then
generate some extra files.

Cheers,
Erik


>
>
> On 05/08/2013 01:53 PM, Erik Botö wrote:
>>
>> Hi,
>>
>> Background:
>> If you use OE to build software which includes e.g. a closed source
>> library, but still want to others to be able to build software based
>> on that library in a OE environment you would have to create recipes
>> that contains a binary version of your library + headers manually.
>> This would have to be done manually for each package, and as the
>> development of the library continues the binary recipes will have to
>> be manually updated to keep in sync.
>>
>> Idea:
>> Create a new bbclass that handles this automatically as much as
>> possible. The basic idea is to:
>> - Create a tarball of ${WORKDIR}/image/ alternatively
>>    ${WORKDIR}/packages-split/
>> - By parsing the "source recipe", create a "binary recipe" that uses
>> that tarball
>>
>> The binary recipe should work as a drop in replacement of the source
>> recipe. This means that the other recipes depending on the closed
>> source software shouldn't need to be adapted to work with one or the
>> other.
>>
>> I'd like feedback on whether you think the idea is feasible, and if
>> you think it could be useful for others. And anything else you think
>> of.
>>
>> Cheers,
>> Erik Botö
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>>
>

Re: [RFC] bbclass for automatic generation of binary recipes

[Robert Yang]

On 05/08/2013 02:36 PM, Erik Botö wrote:
> Hi,
>
> On Wed, May 8, 2013 at 8:11 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>>
>> Is it similar to meta/classes/package.bbclass ?
>>
>> // Robert
>
> I would say it's in addition to package.bbclass.
>

Hi Erik,

Sorry, I meant this one meta/classes/bin_package.bbclass :-)

// Robert

> The idea is that the developers of the closed source software uses OE
> to build their software from source, and in this build process the
> files needed for a binary distribution in OE is automatically
> generated. These generated files can be distributed to others that
> wish to use that closed source software in OE. So it would use some of
> the output from package.bbclass, like the package split, but then
> generate some extra files.
>
> Cheers,
> Erik
>
>
>>
>>
>> On 05/08/2013 01:53 PM, Erik Botö wrote:
>>>
>>> Hi,
>>>
>>> Background:
>>> If you use OE to build software which includes e.g. a closed source
>>> library, but still want to others to be able to build software based
>>> on that library in a OE environment you would have to create recipes
>>> that contains a binary version of your library + headers manually.
>>> This would have to be done manually for each package, and as the
>>> development of the library continues the binary recipes will have to
>>> be manually updated to keep in sync.
>>>
>>> Idea:
>>> Create a new bbclass that handles this automatically as much as
>>> possible. The basic idea is to:
>>> - Create a tarball of ${WORKDIR}/image/ alternatively
>>>     ${WORKDIR}/packages-split/
>>> - By parsing the "source recipe", create a "binary recipe" that uses
>>> that tarball
>>>
>>> The binary recipe should work as a drop in replacement of the source
>>> recipe. This means that the other recipes depending on the closed
>>> source software shouldn't need to be adapted to work with one or the
>>> other.
>>>
>>> I'd like feedback on whether you think the idea is feasible, and if
>>> you think it could be useful for others. And anything else you think
>>> of.
>>>
>>> Cheers,
>>> Erik Botö
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>
>>>
>>
>
>

Re: [RFC] bbclass for automatic generation of binary recipes

[Erik Botö]

On Wed, May 8, 2013 at 9:12 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>
>
> On 05/08/2013 02:36 PM, Erik Botö wrote:
>>
>> Hi,
>>
>> On Wed, May 8, 2013 at 8:11 AM, Robert Yang <liezhi.yang@windriver.com>
>> wrote:
>>>
>>>
>>> Is it similar to meta/classes/package.bbclass ?
>>>
>>> // Robert
>>
>>
>> I would say it's in addition to package.bbclass.
>>
>
> Hi Erik,
>
> Sorry, I meant this one meta/classes/bin_package.bbclass :-)
>
> // Robert

Oh, hadn't seen that one. That could definitely be useful, the only
thing left to do is to try to automate the generation of the recipe
that uses the binary package.

Thanks,
Erik

>
>
>> The idea is that the developers of the closed source software uses OE
>> to build their software from source, and in this build process the
>> files needed for a binary distribution in OE is automatically
>> generated. These generated files can be distributed to others that
>> wish to use that closed source software in OE. So it would use some of
>> the output from package.bbclass, like the package split, but then
>> generate some extra files.
>>
>> Cheers,
>> Erik
>>
>>
>>>
>>>
>>> On 05/08/2013 01:53 PM, Erik Botö wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> Background:
>>>> If you use OE to build software which includes e.g. a closed source
>>>> library, but still want to others to be able to build software based
>>>> on that library in a OE environment you would have to create recipes
>>>> that contains a binary version of your library + headers manually.
>>>> This would have to be done manually for each package, and as the
>>>> development of the library continues the binary recipes will have to
>>>> be manually updated to keep in sync.
>>>>
>>>> Idea:
>>>> Create a new bbclass that handles this automatically as much as
>>>> possible. The basic idea is to:
>>>> - Create a tarball of ${WORKDIR}/image/ alternatively
>>>>     ${WORKDIR}/packages-split/
>>>> - By parsing the "source recipe", create a "binary recipe" that uses
>>>> that tarball
>>>>
>>>> The binary recipe should work as a drop in replacement of the source
>>>> recipe. This means that the other recipes depending on the closed
>>>> source software shouldn't need to be adapted to work with one or the
>>>> other.
>>>>
>>>> I'd like feedback on whether you think the idea is feasible, and if
>>>> you think it could be useful for others. And anything else you think
>>>> of.
>>>>
>>>> Cheers,
>>>> Erik Botö
>>>>
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org
>>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>>
>>>>
>>>
>>
>>
>

Re: [RFC] bbclass for automatic generation of binary recipes

[Robert Yang]

On 05/08/2013 03:35 PM, Erik Botö wrote:
> On Wed, May 8, 2013 at 9:12 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
>>
>>
>> On 05/08/2013 02:36 PM, Erik Botö wrote:
>>>
>>> Hi,
>>>
>>> On Wed, May 8, 2013 at 8:11 AM, Robert Yang <liezhi.yang@windriver.com>
>>> wrote:
>>>>
>>>>
>>>> Is it similar to meta/classes/package.bbclass ?
>>>>
>>>> // Robert
>>>
>>>
>>> I would say it's in addition to package.bbclass.
>>>
>>
>> Hi Erik,
>>
>> Sorry, I meant this one meta/classes/bin_package.bbclass :-)
>>
>> // Robert
>
> Oh, hadn't seen that one. That could definitely be useful, the only
> thing left to do is to try to automate the generation of the recipe
> that uses the binary package.
>

How about the scripts/create-recipe ? It works fine with the tarball, but
not very well with .rpm/.deb/ipk.

// Robert

> Thanks,
> Erik
>
>>
>>
>>> The idea is that the developers of the closed source software uses OE
>>> to build their software from source, and in this build process the
>>> files needed for a binary distribution in OE is automatically
>>> generated. These generated files can be distributed to others that
>>> wish to use that closed source software in OE. So it would use some of
>>> the output from package.bbclass, like the package split, but then
>>> generate some extra files.
>>>
>>> Cheers,
>>> Erik
>>>
>>>
>>>>
>>>>
>>>> On 05/08/2013 01:53 PM, Erik Botö wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> Background:
>>>>> If you use OE to build software which includes e.g. a closed source
>>>>> library, but still want to others to be able to build software based
>>>>> on that library in a OE environment you would have to create recipes
>>>>> that contains a binary version of your library + headers manually.
>>>>> This would have to be done manually for each package, and as the
>>>>> development of the library continues the binary recipes will have to
>>>>> be manually updated to keep in sync.
>>>>>
>>>>> Idea:
>>>>> Create a new bbclass that handles this automatically as much as
>>>>> possible. The basic idea is to:
>>>>> - Create a tarball of ${WORKDIR}/image/ alternatively
>>>>>      ${WORKDIR}/packages-split/
>>>>> - By parsing the "source recipe", create a "binary recipe" that uses
>>>>> that tarball
>>>>>
>>>>> The binary recipe should work as a drop in replacement of the source
>>>>> recipe. This means that the other recipes depending on the closed
>>>>> source software shouldn't need to be adapted to work with one or the
>>>>> other.
>>>>>
>>>>> I'd like feedback on whether you think the idea is feasible, and if
>>>>> you think it could be useful for others. And anything else you think
>>>>> of.
>>>>>
>>>>> Cheers,
>>>>> Erik Botö
>>>>>
>>>>> _______________________________________________
>>>>> Openembedded-core mailing list
>>>>> Openembedded-core@lists.openembedded.org
>>>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

Re: [RFC] bbclass for automatic generation of binary recipes

[Erik Botö]

On Wed, May 8, 2013 at 10:01 AM, Robert Yang <liezhi.yang@windriver.com> wrote:
> On 05/08/2013 03:35 PM, Erik Botö wrote:
>>
>> On Wed, May 8, 2013 at 9:12 AM, Robert Yang <liezhi.yang@windriver.com>
>> wrote:
>>>
>>> On 05/08/2013 02:36 PM, Erik Botö wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> On Wed, May 8, 2013 at 8:11 AM, Robert Yang <liezhi.yang@windriver.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Is it similar to meta/classes/package.bbclass ?
>>>>>
>>>>> // Robert
>>>>
>>>>
>>>>
>>>> I would say it's in addition to package.bbclass.
>>>>
>>>
>>> Hi Erik,
>>>
>>> Sorry, I meant this one meta/classes/bin_package.bbclass :-)
>>>
>>> // Robert
>>
>>
>> Oh, hadn't seen that one. That could definitely be useful, the only
>> thing left to do is to try to automate the generation of the recipe
>> that uses the binary package.
>>
>
> How about the scripts/create-recipe ? It works fine with the tarball, but
> not very well with .rpm/.deb/ipk.
>
> // Robert

What I have in mind is that the recipes will be created automatically
when building a source recipe that inherits a certain class. The
recipe and tarball would then be placed in a layer structure together
with other recipes that includes that class. Fields like RDEPENDS,
LICENSE; PR etc should be carried over from the source recipe.

Maybe it should be done in a way similar to populate_sdk, so if I
bitbake an image with e.g. -c populate_binary_layer it would create a
meta-layer with binary recipes for all packages inheriting that class.

-- Erik

>
>
>> Thanks,
>> Erik
>>
>>>
>>>
>>>> The idea is that the developers of the closed source software uses OE
>>>> to build their software from source, and in this build process the
>>>> files needed for a binary distribution in OE is automatically
>>>> generated. These generated files can be distributed to others that
>>>> wish to use that closed source software in OE. So it would use some of
>>>> the output from package.bbclass, like the package split, but then
>>>> generate some extra files.
>>>>
>>>> Cheers,
>>>> Erik
>>>>
>>>>
>>>>>
>>>>>
>>>>> On 05/08/2013 01:53 PM, Erik Botö wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Background:
>>>>>> If you use OE to build software which includes e.g. a closed source
>>>>>> library, but still want to others to be able to build software based
>>>>>> on that library in a OE environment you would have to create recipes
>>>>>> that contains a binary version of your library + headers manually.
>>>>>> This would have to be done manually for each package, and as the
>>>>>> development of the library continues the binary recipes will have to
>>>>>> be manually updated to keep in sync.
>>>>>>
>>>>>> Idea:
>>>>>> Create a new bbclass that handles this automatically as much as
>>>>>> possible. The basic idea is to:
>>>>>> - Create a tarball of ${WORKDIR}/image/ alternatively
>>>>>>      ${WORKDIR}/packages-split/
>>>>>> - By parsing the "source recipe", create a "binary recipe" that uses
>>>>>> that tarball
>>>>>>
>>>>>> The binary recipe should work as a drop in replacement of the source
>>>>>> recipe. This means that the other recipes depending on the closed
>>>>>> source software shouldn't need to be adapted to work with one or the
>>>>>> other.
>>>>>>
>>>>>> I'd like feedback on whether you think the idea is feasible, and if
>>>>>> you think it could be useful for others. And anything else you think
>>>>>> of.
>>>>>>
>>>>>> Cheers,
>>>>>> Erik Botö
>>>>>>
>>>>>> _______________________________________________
>>>>>> Openembedded-core mailing list
>>>>>> Openembedded-core@lists.openembedded.org
>>>>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>



--
=============================================
Erik Botö
Senior Software Engineer
Pelagicore AB
Ekelundsgatan 4, 6tr, SE-411 18 Gothenburg, Sweden
Mobile: +46 (0)76 881 72 03
E-Mail: erik.boto@pelagicore.com
=============================================