Re: [RFC] Source packages

[Haris Okanovic <haris.okanovic@ni.com>]

Hi Andre,

Thanks for the quick response!

On 12/09/2015 11:56 AM, Andre McCurdy wrote:
> Hi Haris,
>
> On Tue, Dec 8, 2015 at 12:53 PM, Haris Okanovic <haris.okanovic@ni.com> wrote:
>> This change enables creation of ${PN}-src packages which provide recipe
>> source code on target machines. A distribution might use this facility
>> to provide a feed of source packages for user reference and
>> debugging/development activities.
>>
>> The packaged source code is copy of SRC_URI files + a manifest declaring
>> the contents in order of appearance in SRC_URI. For example, it might
>> include a tarball + patch files, which could be used to reconstruct the
>> state of a recipe's working dir after do_patch() by unpacking the
>> archive and applying patches in order.
>>
>> At the moment, unpacking need to be done by hand. However, RPM and Dpkg
>> can both do this automatically upon install when the source is packaged
>> in a src.rpm or dsc file, respectively, as opposed to a binary archive
>> (rpm or deb). A good future improvement might be to provide proper
>> src.rpm's or .dsc's for distributions using those package manager.
>>
>> I don't believe opkg can distinguish between binary and source archives
>> at the moment, so this may be the best we can do at the moment for IPK
>> distributions.
>>
>>
>> Classic OE provided a similar facility for IPKs only with
>> sourceipk.bbclass [1]. It was dropped at some point in modern OE and I'm
>> not familiar enough with OE's history pre Yocto to track down exactly
>> why. I'm curious if this was this a deliberate change or just an
>> accidental regression, and the reasons it was removed in the former
>> case.
>>
>> One significant difference between the classic sourceipk.bbclass and my
>> proposal is what gets packaged: sourceipk simply archived the entire
>> ${S} dir after do_patch() but before do_configure() pollutes it. I opted
>> not to go this route because it's seems at odds with RPM and Dpkg, both
>> of which use the aforementioned tarball+patch file scheme. This may
>> allow us to re-use the staging logic should we implement proper
>> src.rpm's and dsc's in the future.
>>
>>
>> Not all recipes have useful source to package: For example,
>> linux-libc-headers build from the kernel source, which is is already
>> provided by the kernel recipe. base-files just provides initial system
>> configuration, so it's source package provides no additional info than
>> it's binary.
>>
>> I couldn't find a clever way to filter out these edge cases, other than
>> disabling source packages inside those recipes which would require a
>> lengthy review of OE meta layers.
>>
>> I imagine many OE distro also target small, resource constrained,
>> systems, that couldn't benefit from source packaging.
>>
>> Therefore I've made this an opt-in feature for individual distro
>> maintainers to enable as needed. It can be enabled per recipe via the
>> ENABLE_SRC_INSTALL_${PN} var or globally via ENABLE_SRC_INSTALL from
>> distribution confs.
>>
>>
>> === Implementation details ===
>>
>> [PATCH 1/3] package.bbclass/package.py: Add do_install_source() task
>>
>> Add do_install_source() task to stage a recipe's SRC_URI files under
>> SRC_D.
>>
>> Dependencies:
>>   After do_fetch() to ensure SRC_URI files are downloaded
>>   After do_install() because it resets ${D} that's also used by this task
>>   Before do_package() to stage files before writing installers
>>
>> No-ops unless ENABLE_SRC_INSTALL_${PN} = 1, which needs to be set in
>> distro config or recipes wanting to use this facility.
>>
>> [PATCH 2/3] documentation.conf: Document do_install_source()
>>
>> [PATCH 3/3] bitbake.conf: Define source package, disabled by default
>>
>> Add ${PN}-src to PACKAGES with a default set of FILES_{PV}-src
>> per FHS [2].
>>
>> Define a staging directory SRC_D under ${D}.
>>
>> Define ENABLE_SRC_INSTALL to enable source staging via package.bbclass's
>> do_install_source() task. It's disabled by default, but may be
>> overridden by distro configs or individual recipes as needed.
>>
>>
>> === References ===
>>
>> [1] https://github.com/openembedded/openembedded/blob/master/classes/sourceipk.bbclass
>> [2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#usrsrcSourceCode
>>
>>
>> Comments, opinions, and other meandering thoughts welcome....
>>
>> Thanks,
>> Haris
>> ---
>>   meta/classes/package.bbclass |  5 +++
>>   meta/conf/bitbake.conf       |  9 ++++-
>>   meta/conf/documentation.conf |  1 +
>>   meta/lib/oe/package.py       | 96 ++++++++++++++++++++++++++++++++++++++++++++
>>   4 files changed, 110 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
>> index d731757..98f01e5 100644
>> --- a/meta/classes/package.bbclass
>> +++ b/meta/classes/package.bbclass
>> @@ -2079,3 +2079,8 @@ def mapping_rename_hook(d):
>>       runtime_mapping_rename("RRECOMMENDS", pkg, d)
>>       runtime_mapping_rename("RSUGGESTS", pkg, d)
>>
>> +addtask do_install_source after do_fetch after do_install before do_package
>> +
>> +python do_install_source () {
>> +    oe.package.do_install_source(d)
>> +}
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 06971da..431865d 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -36,6 +36,7 @@ export systemd_unitdir = "${nonarch_base_libdir}/systemd"
>>   export systemd_system_unitdir = "${nonarch_base_libdir}/systemd/system"
>>   export nonarch_libdir = "${exec_prefix}/lib"
>>   export systemd_user_unitdir = "${nonarch_libdir}/systemd/user"
>> +export srcdir = "${prefix}/src"
>
> I wonder if globally exporting a "srcdir" environment variable is the
> correct thing to do. If it's not expected to be recognised or used
> directly by any package's "make install", etc, then perhaps it should
> just be a bitbake internal variable to avoid any concerns about
> namespace pollution.
>
> (AFAIK, variables like "prefix" are only exported to support legacy
> packages which rely on environment variables + "make -e" to ensure
> that "make install", etc, do the right thing. For autotools packages,
> for example, prefix etc are all passed directly on the configure
> command line so exporting them too is mostly redundant. If we succeed
> in getting rid of "make -e" then I guess eventually none of these
> directory variables will need to be exported any more?).
>

srcdir is only used by bitbake.conf atm, so I agree there's really no 
practical benefit to exporting it. Removed.

--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -36,7 +36,7 @@ export systemd_unitdir = "${nonarch_base_libdir}/systemd"
  export systemd_system_unitdir = "${nonarch_base_libdir}/systemd/system"
  export nonarch_libdir = "${exec_prefix}/lib"
  export systemd_user_unitdir = "${nonarch_libdir}/systemd/user"
-export srcdir = "${prefix}/src"
+srcdir = "${prefix}/src"

  # Architecture dependent paths
  export bindir = "${exec_prefix}/bin"
---

>>   # Architecture dependent paths
>>   export bindir = "${exec_prefix}/bin"
>> @@ -267,7 +268,7 @@ SOLIBSDEV = ".so"
>>   SOLIBSDEV_darwin = ".dylibbroken"
>>
>>   PACKAGE_BEFORE_PN ?= ""
>> -PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
>> +PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
>>   PACKAGES_DYNAMIC = "^${PN}-locale-.*"
>>   FILES = ""
>>
>> @@ -313,6 +314,12 @@ ALLOW_EMPTY_${PN}-dbg = "1"
>>
>>   FILES_${PN}-locale = "${datadir}/locale"
>>
>> +FILES_${PN}-src = "${srcdir}/${PN}"
>> +SRC_D = "${D}${srcdir}/${PN}"
>> +
>> +ENABLE_SRC_INSTALL = "0"
>> +ENABLE_SRC_INSTALL_${PN} = "${ENABLE_SRC_INSTALL}"
>> +
>>   # File manifest
>>
>>   FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
>> diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
>> index 845559a..0df8a2f 100644
>> --- a/meta/conf/documentation.conf
>> +++ b/meta/conf/documentation.conf
>> @@ -26,6 +26,7 @@ do_fetchall[doc] = "Fetches all remote sources required to build a target"
>>   do_generate_qt_config_file[doc] = "Writes a qt.conf file for building a Qt-based application"
>>   do_install[doc] = "Copies files from the compilation directory to a holding area"
>>   do_install_ptest_base[doc] = "Copies the runtime test suite files from the compilation directory to a holding area"
>> +do_install_source[doc] = "Stages source code for packaging"
>>   do_kernel_checkout[doc] = "Checks out source/meta branches for a linux-yocto style kernel"
>>   do_kernel_configcheck[doc] = "Validates the kernel configuration for a linux-yocto style kernel"
>>   do_kernel_configme[doc] = "Assembles the kernel configuration for a linux-yocto style kernel"
>> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
>> index ea6feaa..f980f25 100644
>> --- a/meta/lib/oe/package.py
>> +++ b/meta/lib/oe/package.py
>> @@ -123,3 +123,99 @@ def read_shlib_providers(d):
>>                           shlib_provider[s[0]] = {}
>>                       shlib_provider[s[0]][s[1]] = (dep_pkg, s[2])
>>       return shlib_provider
>> +
>> +def archive_dir(dirPath, archivePath):
>> +    ''' Create tar.bz2 archive at archivePath from dirPath '''
>> +    import os, oe, bb
>> +
>> +    arDir = os.path.dirname(dirPath)
>> +    arName = os.path.basename(dirPath)
>> +
>> +    cmd = 'tar -c -j -f \"%s\" -C \"%s\" -p \"%s\"' % (archivePath, arDir, arName)
>
> Perhaps try to make use of pbzip2 instead of "tar -j" to create
> tar.bz2 files. See:
>
>    http://git.openembedded.org/openembedded-core/commit/?id=670f5cda06070ae888d17cca7a07aa74d751c2f7
>

I think that's a good idea, it might be a little faster that way. 
Setting `-I pbzip2` instead of `-j` in cmd.

Built xinetd and verified it's Git source got tarred up as before.

--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -131,7 +131,7 @@ def archive_dir(dirPath, archivePath):
      arDir = os.path.dirname(dirPath)
      arName = os.path.basename(dirPath)

-    cmd = 'tar -c -j -f \"%s\" -C \"%s\" -p \"%s\"' % (archivePath, 
arDir, arName)
+    cmd = 'tar -c -I pbzip2 -f \"%s\" -C \"%s\" -p \"%s\"' % 
(archivePath, arDir, arName)
      (retval, output) = oe.utils.getstatusoutput(cmd)
      if retval:
          bb.fatal('Failed to archive %s --> %s: %s %s' % (dirPath, 
archivePath, cmd, output))
---

>> +    (retval, output) = oe.utils.getstatusoutput(cmd)
>> +    if retval:
>> +        bb.fatal('Failed to archive %s --> %s: %s %s' % (dirPath, archivePath, cmd, output))
>> +
>> +def do_install_source(d):
>> +    ''' Stage recipe's source for packaging '''
>> +    import os, oe, bb
>> +
>> +    pn = d.getVar("PN", True)
>> +
>> +    if d.getVar("ENABLE_SRC_INSTALL_%s" % pn, True) != "1":
>> +        return
>> +
>> +    packages = (d.getVar("PACKAGES") or "").split()
>> +    if ("%s-src" % pn) not in packages:
>> +        # Some recipes redefine PACKAGES without ${PN}-src. Don't stage
>> +        # anything in this case to avoid installed-vs-shipped warning.
>> +        return
>> +
>> +    urls = (d.getVar('SRC_URI', True) or "").split()
>> +    if len(urls) == 0:
>> +        return
>> +
>> +    workdir = d.getVar('WORKDIR', True)
>> +
>> +    # TODO rm_work() should clean this up
>> +    unpackTempDir = os.path.join(workdir, 'install-source-unpack-temp')
>> +    if os.path.exists(unpackTempDir):
>> +        bb.utils.remove(unpackTempDir, recurse=True)
>> +    os.makedirs(unpackTempDir, 0755)
>> +
>> +    src_d = d.getVar("SRC_D", True)
>> +    if os.path.exists(src_d):
>> +        bb.warn("SRC_D already exist. Removing.")
>> +        bb.utils.remove(src_d, recurse=True)
>> +    os.makedirs(src_d, 0755)
>> +
>> +    fetcher = bb.fetch2.Fetch(urls, d)
>> +
>> +    fileManif = []
>> +    for url in urls:
>> +        urlScheme = bb.fetch2.decodeurl(url)[0]
>> +        srcPath = fetcher.localpath(url)
>> +        srcName = os.path.basename(srcPath)
>> +
>> +        dstName = srcName
>> +        if os.path.isdir(srcPath):
>> +            dstName += '.tar.bz2'
>> +
>> +        dstPath = os.path.join(src_d, dstName)
>> +
>> +        # fetch() doesn't retrieve any actual files from git:// URLs,
>> +        # so we do an additional unpack() step to get something useful
>> +        # for these.
>> +        # TODO: May need to pre-process other revision control schemes
>> +        if urlScheme == 'git':
>> +            unpackPath = os.path.join(unpackTempDir, srcName)
>> +            if os.path.exists(unpackPath):
>> +                bb.utils.remove(unpackPath, recurse=True)
>> +            os.makedirs(unpackPath, 0755)
>> +
>> +            fetcher.unpack(unpackPath, [url])
>> +
>> +            # unpack() puts actual source in a 'git' subdir
>> +            srcPath = os.path.join(unpackPath, 'git')
>> +
>> +        if os.path.exists(dstPath):
>> +            bb.warn('Duplicate file %s in SRC_URI. Overwriting.' % dstName)
>> +            bb.utils.remove(dstPath, recurse=True)
>> +
>> +        if not dstName in fileManif:
>> +            fileManif.append(dstName)
>> +
>> +        if os.path.isdir(srcPath):
>> +            archive_dir(srcPath, dstPath)
>> +        else:
>> +            bb.utils.copyfile(srcPath, dstPath)
>> +
>> +    manifFilePath = os.path.join(src_d, 'manifest')
>> +    if os.path.exists(manifFilePath):
>> +        bb.warn('manifest file found in SRC_URI. Overwriting.')
>> +        bb.utils.remove(manifFilePath, recurse=True)
>> +
>> +    with open(manifFilePath, 'wb') as manif:
>> +        for fname in fileManif:
>> +            manif.write(fname)
>> +            manif.write('\n')
>> --
>> 2.6.2
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- Haris