Automatically creating tar files for bin_package.bbclass

Automatically creating tar files for bin_package.bbclass

[Andre McCurdy]

Say I have a proprietary library which only I can build from source
and I have a conventional recipe which builds it within OE. I would
now like to create a tar file of the library and headers etc, plus a
companion recipe based on bin_package.bbclass to allow others to make
use of the library in their OE builds. ie the situation described
here:

  http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.html#packaging-externally-produced-binaries

Is there a recommended way to create the tar file to support that?
e.g. a class which automatically creates a deployable tar file of a
recipe's ${D} directory?

Re: Automatically creating tar files for bin_package.bbclass

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com> wrote:

> Say I have a proprietary library which only I can build from source
> and I have a conventional recipe which builds it within OE. I would
> now like to create a tar file of the library and headers etc, plus a
> companion recipe based on bin_package.bbclass to allow others to make
> use of the library in their OE builds. ie the situation described
> here:
>
>
> http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.html#packaging-externally-produced-binaries
>
> Is there a recommended way to create the tar file to support that?
> e.g. a class which automatically creates a deployable tar file of a
> recipe's ${D} directory?
>

You *may* be able to use package_tar in the recipe in some way to achieve
this.

Ross

[-- Attachment #2: Type: text/html, Size: 1397 bytes --]

Re: Automatically creating tar files for bin_package.bbclass

[Andre McCurdy]

On Mon, Jul 4, 2016 at 8:58 AM, Burton, Ross <ross.burton@intel.com> wrote:
>
> On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com> wrote:
>>
>> Say I have a proprietary library which only I can build from source
>> and I have a conventional recipe which builds it within OE. I would
>> now like to create a tar file of the library and headers etc, plus a
>> companion recipe based on bin_package.bbclass to allow others to make
>> use of the library in their OE builds. ie the situation described
>> here:
>>
>> http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.html#packaging-externally-produced-binaries
>>
>> Is there a recommended way to create the tar file to support that?
>> e.g. a class which automatically creates a deployable tar file of a
>> recipe's ${D} directory?
>
> You *may* be able to use package_tar in the recipe in some way to achieve
> this.

It only seems to support creating tar files after packages have been
split ( ie individual tar files for each subdirectory under
${WORKDIR}/packages-split ). I don't think it's going to be directly
useful for creating a tar file of ${D}.

> Ross

Re: Automatically creating tar files for bin_package.bbclass

[Khem Raj]

On Tue, Jul 5, 2016 at 1:21 AM, Andre McCurdy <armccurdy@gmail.com> wrote:
> On Mon, Jul 4, 2016 at 8:58 AM, Burton, Ross <ross.burton@intel.com> wrote:
>>
>> On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com> wrote:
>>>
>>> Say I have a proprietary library which only I can build from source
>>> and I have a conventional recipe which builds it within OE. I would
>>> now like to create a tar file of the library and headers etc, plus a
>>> companion recipe based on bin_package.bbclass to allow others to make
>>> use of the library in their OE builds. ie the situation described
>>> here:
>>>
>>> http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.html#packaging-externally-produced-binaries
>>>
>>> Is there a recommended way to create the tar file to support that?
>>> e.g. a class which automatically creates a deployable tar file of a
>>> recipe's ${D} directory?
>>
>> You *may* be able to use package_tar in the recipe in some way to achieve
>> this.
>
> It only seems to support creating tar files after packages have been
> split ( ie individual tar files for each subdirectory under
> ${WORKDIR}/packages-split ). I don't think it's going to be directly
> useful for creating a tar file of ${D}.

package_tar is just another packaging format here so you will get
individual package separately
you might want to add another class to package everything into 1 thats
a separate issue, you have
to either add this via a bbclass or additional function
>
>> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Automatically creating tar files for bin_package.bbclass

[Paul Eggleton]

On Tue, 05 Jul 2016 08:12:14 Khem Raj wrote:
> On Tue, Jul 5, 2016 at 1:21 AM, Andre McCurdy <armccurdy@gmail.com> wrote:
> > On Mon, Jul 4, 2016 at 8:58 AM, Burton, Ross <ross.burton@intel.com> 
wrote:
> >> On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com> wrote:
> >>> Say I have a proprietary library which only I can build from source
> >>> and I have a conventional recipe which builds it within OE. I would
> >>> now like to create a tar file of the library and headers etc, plus a
> >>> companion recipe based on bin_package.bbclass to allow others to make
> >>> use of the library in their OE builds. ie the situation described
> >>> here:
> >>> 
> >>> http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.html#packag
> >>> ing-externally-produced-binaries
> >>> 
> >>> Is there a recommended way to create the tar file to support that?
> >>> e.g. a class which automatically creates a deployable tar file of a
> >>> recipe's ${D} directory?
> >> 
> >> You *may* be able to use package_tar in the recipe in some way to achieve
> >> this.
> > 
> > It only seems to support creating tar files after packages have been
> > split ( ie individual tar files for each subdirectory under
> > ${WORKDIR}/packages-split ). I don't think it's going to be directly
> > useful for creating a tar file of ${D}.
> 
> package_tar is just another packaging format here so you will get
> individual package separately
> you might want to add another class to package everything into 1 thats
> a separate issue, you have
> to either add this via a bbclass or additional function

Right. Another thing - you probably want to tar up ${PKGD} rather than ${D}; 
then you get the result of all of the processing that happens during packaging  
(other than the splitting to individual packages) such as debug symbol 
stripping.

I think we'd take a patch to have a variable that told package_tar to do this, 
it does seem useful in this context.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Automatically creating tar files for bin_package.bbclass

[Richard Purdie]

On Tue, 2016-07-05 at 08:12 -0700, Khem Raj wrote:
> On Tue, Jul 5, 2016 at 1:21 AM, Andre McCurdy <armccurdy@gmail.com>
> wrote:
> > On Mon, Jul 4, 2016 at 8:58 AM, Burton, Ross <ross.burton@intel.com
> > > wrote:
> > > 
> > > On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com>
> > > wrote:
> > > > 
> > > > Say I have a proprietary library which only I can build from
> > > > source
> > > > and I have a conventional recipe which builds it within OE. I
> > > > would
> > > > now like to create a tar file of the library and headers etc,
> > > > plus a
> > > > companion recipe based on bin_package.bbclass to allow others
> > > > to make
> > > > use of the library in their OE builds. ie the situation
> > > > described
> > > > here:
> > > > 
> > > > http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.ht
> > > > ml#packaging-externally-produced-binaries
> > > > 
> > > > Is there a recommended way to create the tar file to support
> > > > that?
> > > > e.g. a class which automatically creates a deployable tar file
> > > > of a
> > > > recipe's ${D} directory?
> > > 
> > > You *may* be able to use package_tar in the recipe in some way to
> > > achieve
> > > this.
> > 
> > It only seems to support creating tar files after packages have
> > been
> > split ( ie individual tar files for each subdirectory under
> > ${WORKDIR}/packages-split ). I don't think it's going to be
> > directly
> > useful for creating a tar file of ${D}.
> 
> package_tar is just another packaging format here so you will get
> individual package separately
> you might want to add another class to package everything into 1
> thats
> a separate issue, you have
> to either add this via a bbclass or additional function

Personally, I think that involving package_tar is likely going to
overcomplicate this, as is packaging up the already separated out
packages.

Better to provide the output as do_install would leave it in ${D}, and
then let packaging happen as the user has configured. Only tricky part
would be the lack of source for the debugging symbols but if you can't
release the source you'd want to "fix" that anyway.

This way you don't need to worry if your tarball matches the users
package configuration options (which backend, which packaging options
etc.).

Cheers,

Richard

Re: Automatically creating tar files for bin_package.bbclass

[Andre McCurdy]

On Tue, Jul 5, 2016 at 3:27 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-07-05 at 08:12 -0700, Khem Raj wrote:
>> On Tue, Jul 5, 2016 at 1:21 AM, Andre McCurdy <armccurdy@gmail.com>
>> wrote:
>> > On Mon, Jul 4, 2016 at 8:58 AM, Burton, Ross <ross.burton@intel.com
>> > > wrote:
>> > >
>> > > On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com>
>> > > wrote:
>> > > >
>> > > > Say I have a proprietary library which only I can build from
>> > > > source
>> > > > and I have a conventional recipe which builds it within OE. I
>> > > > would
>> > > > now like to create a tar file of the library and headers etc,
>> > > > plus a
>> > > > companion recipe based on bin_package.bbclass to allow others
>> > > > to make
>> > > > use of the library in their OE builds. ie the situation
>> > > > described
>> > > > here:
>> > > >
>> > > > http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.ht
>> > > > ml#packaging-externally-produced-binaries
>> > > >
>> > > > Is there a recommended way to create the tar file to support
>> > > > that?
>> > > > e.g. a class which automatically creates a deployable tar file
>> > > > of a
>> > > > recipe's ${D} directory?
>> > >
>> > > You *may* be able to use package_tar in the recipe in some way to
>> > > achieve
>> > > this.
>> >
>> > It only seems to support creating tar files after packages have
>> > been
>> > split ( ie individual tar files for each subdirectory under
>> > ${WORKDIR}/packages-split ). I don't think it's going to be
>> > directly
>> > useful for creating a tar file of ${D}.
>>
>> package_tar is just another packaging format here so you will get
>> individual package separately
>> you might want to add another class to package everything into 1
>> thats
>> a separate issue, you have
>> to either add this via a bbclass or additional function
>
> Personally, I think that involving package_tar is likely going to
> overcomplicate this, as is packaging up the already separated out
> packages.
>
> Better to provide the output as do_install would leave it in ${D}, and
> then let packaging happen as the user has configured. Only tricky part
> would be the lack of source for the debugging symbols but if you can't
> release the source you'd want to "fix" that anyway.

I think the common case is that you wouldn't want to include source or
debug symbols, so using ${D} directly is still not perfect.

Here's the generate-prebuilt-tarfile.bbclass I'm currently using. It's
certainly not suitable for merging yet.

----

DEPLOY_DIR_PREBUILTS ?= "${DEPLOY_DIR}/prebuilts"

do_genprebuilt[depends] += "pbzip2-native:do_populate_sysroot"
do_genprebuilt[dirs] = "${DEPLOY_DIR_PREBUILTS}/${PACKAGE_ARCH} ${WORKDIR}"

do_genprebuilt() {

    if [ "${CLASSOVERRIDE}" != "class-target" ]; then
        exit 0
    fi

    TARFILE_BASENAME="$(echo ${PN}-prebuilt-${PV} | sed 's/AUTOINC+//')"

    tar --transform "s|^image|$TARFILE_BASENAME|" --owner 0 --group 0
-cvf - $(find image ! -type d | LC_ALL=C sort) | pbzip2 >
${DEPLOY_DIR_PREBUILTS}/${PACKAGE_ARCH}/$TARFILE_BASENAME.tar.bz2
}

addtask do_genprebuilt after do_install before do_populate_sysroot

----

> This way you don't need to worry if your tarball matches the users
> package configuration options (which backend, which packaging options
> etc.).
>
> Cheers,
>
> Richard
>
>

Re: Automatically creating tar files for bin_package.bbclass

[Nicolas Dechesne]

hi Andre!

On Wed, Jul 6, 2016 at 10:21 AM Andre McCurdy <armccurdy@gmail.com> wrote:
>
> On Tue, Jul 5, 2016 at 3:27 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Tue, 2016-07-05 at 08:12 -0700, Khem Raj wrote:
> >> On Tue, Jul 5, 2016 at 1:21 AM, Andre McCurdy <armccurdy@gmail.com>
> >> wrote:
> >> > On Mon, Jul 4, 2016 at 8:58 AM, Burton, Ross <ross.burton@intel.com
> >> > > wrote:
> >> > >
> >> > > On 4 July 2016 at 16:00, Andre McCurdy <armccurdy@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > Say I have a proprietary library which only I can build from
> >> > > > source
> >> > > > and I have a conventional recipe which builds it within OE. I
> >> > > > would
> >> > > > now like to create a tar file of the library and headers etc,
> >> > > > plus a
> >> > > > companion recipe based on bin_package.bbclass to allow others
> >> > > > to make
> >> > > > use of the library in their OE builds. ie the situation
> >> > > > described
> >> > > > here:
> >> > > >
> >> > > > http://www.yoctoproject.org/docs/2.1/mega-manual/mega-manual.ht
> >> > > > ml#packaging-externally-produced-binaries
> >> > > >
> >> > > > Is there a recommended way to create the tar file to support
> >> > > > that?
> >> > > > e.g. a class which automatically creates a deployable tar file
> >> > > > of a
> >> > > > recipe's ${D} directory?
> >> > >
> >> > > You *may* be able to use package_tar in the recipe in some way to
> >> > > achieve
> >> > > this.
> >> >
> >> > It only seems to support creating tar files after packages have
> >> > been
> >> > split ( ie individual tar files for each subdirectory under
> >> > ${WORKDIR}/packages-split ). I don't think it's going to be
> >> > directly
> >> > useful for creating a tar file of ${D}.
> >>
> >> package_tar is just another packaging format here so you will get
> >> individual package separately
> >> you might want to add another class to package everything into 1
> >> thats
> >> a separate issue, you have
> >> to either add this via a bbclass or additional function
> >
> > Personally, I think that involving package_tar is likely going to
> > overcomplicate this, as is packaging up the already separated out
> > packages.
> >
> > Better to provide the output as do_install would leave it in ${D}, and
> > then let packaging happen as the user has configured. Only tricky part
> > would be the lack of source for the debugging symbols but if you can't
> > release the source you'd want to "fix" that anyway.
>
> I think the common case is that you wouldn't want to include source or
> debug symbols, so using ${D} directly is still not perfect.
>
> Here's the generate-prebuilt-tarfile.bbclass I'm currently using. It's
> certainly not suitable for merging yet.
>
> ----
>
> DEPLOY_DIR_PREBUILTS ?= "${DEPLOY_DIR}/prebuilts"
>
> do_genprebuilt[depends] += "pbzip2-native:do_populate_sysroot"
> do_genprebuilt[dirs] = "${DEPLOY_DIR_PREBUILTS}/${PACKAGE_ARCH} ${WORKDIR}"
>
> do_genprebuilt() {
>
>     if [ "${CLASSOVERRIDE}" != "class-target" ]; then
>         exit 0
>     fi
>
>     TARFILE_BASENAME="$(echo ${PN}-prebuilt-${PV} | sed 's/AUTOINC+//')"
>
>     tar --transform "s|^image|$TARFILE_BASENAME|" --owner 0 --group 0
> -cvf - $(find image ! -type d | LC_ALL=C sort) | pbzip2 >
> ${DEPLOY_DIR_PREBUILTS}/${PACKAGE_ARCH}/$TARFILE_BASENAME.tar.bz2
> }
>
> addtask do_genprebuilt after do_install before do_populate_sysroot
>

has anything more been done on that topic? I happen to now need
something similar. E.g. packaging for proprietary content, and google
led me to this discussion. What's the recommended approach nowadays to
deal with recipes for proprietary content? The use case is similar,
when building internally, we want to build all sources, but when we
distribute, we want to distribute all open source content +
proprietary as blob, but with minimal (if not none) changes in the
meta data.


> ----
>
> > This way you don't need to worry if your tarball matches the users
> > package configuration options (which backend, which packaging options
> > etc.).
> >
> > Cheers,
> >
> > Richard
> >
> >
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Automatically creating tar files for bin_package.bbclass

[Andre McCurdy]

 On Fri, Jul 19, 2019 at 1:09 AM Nicolas Dechesne
<nicolas.dechesne@linaro.org> wrote:
>
> hi Andre!

Hi Nico! :-)

> has anything more been done on that topic? I happen to now need
> something similar. E.g. packaging for proprietary content, and google
> led me to this discussion. What's the recommended approach nowadays to
> deal with recipes for proprietary content? The use case is similar,
> when building internally, we want to build all sources, but when we
> distribute, we want to distribute all open source content +
> proprietary as blob, but with minimal (if not none) changes in the
> meta data.

I don't think there was any further public work on this topic. My own
private solution evolved somewhat to the class copied and pasted
below.

In the end it became a self contained solution which doesn't rely on
bin_package.bbclass. Instead it directly creates a tar file of the
proprietary binaries (based on the contents of ${PKGD}) and a
dedicated "mini recipe" to extract it. The party with access to the
proprietary source includes the class in the main recipe which builds
from source and then arranges for the resulting tar file and mini
recipe to be distributed to the parties who don't have source access.
In my case, hooks run on a CI build server pushed the mini recipe to a
meta layer git repo and the tar file to a server accessible via scp.
The consumers of the prebuilts therefore only needed to repo sync the
meta layer to get the latest updates.

----

DEPLOY_DIR_PREBUILT_RECIPES ?= "${DEPLOY_DIR}/prebuilt-recipes"
DEPLOY_DIR_PREBUILT_TARFILES ?= "${DEPLOY_DIR}/prebuilt-tarfiles"

BASEPV = "${@ d.getVar('PV', True).replace('AUTOINC+', '')}"

PREBUILT_PV ?= "prebuilt-${MACHINE}-${BASEPV}"
PREBUILT_TARFILE_BASENAME ?= "${BPN}-${PREBUILT_PV}"

PREBUILT_RECIPE ?=
"${DEPLOY_DIR_PREBUILT_RECIPES}/${BPN}_prebuilt-${MACHINE}.bb"
PREBUILT_TARFILE ?=
"${DEPLOY_DIR_PREBUILT_TARFILES}/${PREBUILT_TARFILE_BASENAME}.tar.bz2"

PREBUILT_SRC_URI_PREFIX ?= "http://localhost:8000/"
PREBUILT_SRC_URI_PATH ?= ""

PREBUILT_CREATE_TARFILE_DOTDONE ?= "true"

do_genprebuilt[depends] += "pbzip2-native:do_populate_sysroot"
do_genprebuilt[dirs] = "${DEPLOY_DIR_PREBUILT_RECIPES}
${DEPLOY_DIR_PREBUILT_TARFILES} ${WORKDIR}"

do_genprebuilt() {

    [ "${CLASSOVERRIDE}" != "class-target" ] && exit 0

    find package -type d -empty -o ! -type d | grep -v
'^package/usr/src' | grep -v '\.debug' | LC_ALL=C sort >
${PREBUILT_TARFILE}.files
    tar --transform "s|^package|${PREBUILT_TARFILE_BASENAME}|" --owner
0 --group 0 -T ${PREBUILT_TARFILE}.files -cvf - | pbzip2 >
${PREBUILT_TARFILE}
    rm ${PREBUILT_TARFILE}.files

    [ "${PREBUILT_CREATE_TARFILE_DOTDONE}" = "true" ] && touch
${PREBUILT_TARFILE}.done

    # ------------------------------------------------------------------------

    s='$''{S}'
    d='$''{D}'
    bp='$''{BP}'
    ma='$''{MACHINE_ARCH}'
    pn='$''{PN}'
    wd='$''{WORKDIR}'
    openbracket='{'
    closebracket='}'

    md5=`md5sum < ${PREBUILT_TARFILE} | cut -c-32`
    sha256=`sha256sum < ${PREBUILT_TARFILE} | cut -c-64`

    # ------------------------------------------------------------------------

    cat << EOF > ${PREBUILT_RECIPE}

COMPATIBLE_MACHINE = "${MACHINE}"

PV = "${PREBUILT_PV}"

SRC_URI = "${PREBUILT_SRC_URI_PREFIX}${PREBUILT_SRC_URI_PATH}$bp.tar.bz2"

SRC_URI[md5sum] = "$md5"
SRC_URI[sha256sum] = "$sha256"

do_configure[noexec] = "1"
do_compile[noexec] = "1"

do_install () $openbracket
    tar -C $s --exclude='./patches' --exclude='./.pc' -cpf - . | tar
-C $d --no-same-owner -xpvf -
$closebracket

PACKAGE_ARCH = "$ma"

INSANE_SKIP_$pn += "already-stripped"
EOF

    # ------------------------------------------------------------------------
}

addtask do_genprebuilt after do_package before do_package_qa

----