[PATCH 0/2] IPK_HIERARCHICAL_FEED support

[PATCH 0/2] IPK_HIERARCHICAL_FEED support

[Paul Barker]

This is a follow up to the RFC patches I sent out a few weeks ago. I've finally
had time to finish up the work and give it another round of testing. Everything
looks good both with and without IPK_HIERARCHICAL_FEED set. I've tested that it
is possible to install from a hierarchical feed using opkg on qemuarm.

Paul Barker (2):
  opkg-utils: Update SRCREV
  package_ipk.bbclass: Support hierarchical feed

 meta/classes/package_ipk.bbclass                   | 27 +++++++++++++++++++++-
 meta/conf/local.conf.sample.extended               | 23 ++++++++++++++++++
 meta/recipes-devtools/opkg-utils/opkg-utils_git.bb |  2 +-
 3 files changed, 50 insertions(+), 2 deletions(-)

-- 
1.9.3

[PATCH 1/2] opkg-utils: Update SRCREV

[Paul Barker]

Recent changes in opkg-utils allow package files to be stored in a different
directory to the package index if desired.

Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
---
 meta/recipes-devtools/opkg-utils/opkg-utils_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
index 54cfe54..2d16660 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
@@ -7,7 +7,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f \
                     file://opkg.py;beginline=1;endline=18;md5=15917491ad6bf7acc666ca5f7cc1e083"
 PROVIDES += "virtual/update-alternatives"
 
-SRCREV = "c33b217016ee911718b10c9d57f9912935baf5a9"
+SRCREV = "d14458499af7dd41d8e424946c164961421b0ddb"
 PV = "0.1.8+git${SRCPV}"
 
 SRC_URI = "git://git.yoctoproject.org/opkg-utils"
-- 
1.9.3

[PATCH 2/2] package_ipk.bbclass: Support hierarchical feed

[Paul Barker]

This patch allows for an optional new layout for ipk feed directories which I've
called a 'hierarchical feed' and is based on how Debian pools package files. It
is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".

In the traditional feed layout, package files are placed in <outdir>/<arch>/.
This can lead to several thousand files existing in a single directory which is
often a problem if developers want to upload a package feed to a shared web
hosting provider. For example, in my case, listing files via FTP only shows the
first 2000 files, breaking my scripts which attempt to upload only new and
changed files via FTP.

In the hierarchical feed, package files are written to
<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>, where pkg_prefix is the first letter
of the package file name for non-lib packages or "lib" plus the 4th letter of
the package file name for lib packages (eg, 'l' for less, 'libc' for libc6).
pkg_subdir is the root of the package file name, discarding the version and
architecture parts and the common suffixes '-dbg', '-dev', '-doc', '-staticdev',
'-locale' and '-locale-*' which are listed in meta/conf/bitbake.conf.

This change relies on recent patches to opkg-utils which support hierarchical
package feeds.

Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
---
 meta/classes/package_ipk.bbclass     | 27 ++++++++++++++++++++++++++-
 meta/conf/local.conf.sample.extended | 23 +++++++++++++++++++++++
 2 files changed, 49 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 2949d1d..9586e06 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -63,7 +63,32 @@ python do_package_ipk () {
         bb.data.update_data(localdata)
         basedir = os.path.join(os.path.dirname(root))
         arch = localdata.getVar('PACKAGE_ARCH', True)
-        pkgoutdir = "%s/%s" % (outdir, arch)
+
+        if localdata.getVar('IPK_HIERARCHICAL_FEED') == "1":
+            # Spread packages across subdirectories so each isn't too crowded
+            if pkgname.startswith('lib'):
+                pkg_prefix = 'lib' + pkgname[3]
+            else:
+                pkg_prefix = pkgname[0]
+
+            # Keep -dbg, -dev, -doc, -staticdev, -locale and -locale-* packages
+            # together. These package suffixes are taken from the definitions of
+            # PACKAGES and PACKAGES_DYNAMIC in meta/conf/bitbake.conf
+            if pkgname[-4:] in ('-dbg', '-dev', '-doc'):
+                pkg_subdir = pkgname[:-4]
+            elif pkgname.endswith('-staticdev'):
+                pkg_subdir = pkgname[:-10]
+            elif pkgname.endswith('-locale'):
+                pkg_subdir = pkgname[:-7]
+            elif '-locale-' in pkgname:
+                pkg_subdir = pkgname[:pkgname.find('-locale-')]
+            else:
+                pkg_subdir = pkgname
+
+            pkgoutdir = "%s/%s/%s/%s" % (outdir, arch, pkg_prefix, pkg_subdir)
+        else:
+            pkgoutdir = "%s/%s" % (outdir, arch)
+
         bb.utils.mkdirhier(pkgoutdir)
         os.chdir(root)
         cleanupcontrol(root)
diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index a1f2464..e4f596a 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -310,3 +310,26 @@
 #INITRAMFS_IMAGE = "core-image-minimal-initramfs"
 #INITRAMFS_IMAGE_BUNDLE = "1"
 
+#
+# IPK Hierarchical feed
+#
+# In some cases it may be desirable not to have all package files in the same
+# directory. An example would be when package feeds are to be uploaded to a
+# shared webhosting service or transferred to a Windows machine which may have
+# problems with directories containing multiple thousands of files.
+#
+# If the IPK_HIERARCHICAL_FEED variable is set to "1", packages will be split
+# between subdirectories in a similar way to how Debian package feeds are
+# organised. In the hierarchical feed, package files are written to
+# <outdir>/<arch>/<pkg_prefix>/<pkg_subdir>, where pkg_prefix is the first
+# letter of the package file name for non-lib packages or "lib" plus the 4th
+# letter of the package file name for lib packages (eg, 'l' for less, 'libc' for
+# libc6).  pkg_subdir is the root of the package file name, discarding the
+# version and architecture parts and the common suffixes '-dbg', '-dev', '-doc',
+# '-staticdev', '-locale' and '-locale-*' which are listed in
+# meta/conf/bitbake.conf.
+#
+# If IPK_HIERARCHICAL_FEED is unset or set to any other value, the traditional
+# feed layout is used where package files are placed in <outdir>/<arch>/.
+#
+#IPK_HIERARCHICAL_FEED = "1"
-- 
1.9.3

Re: [PATCH 2/2] package_ipk.bbclass: Support hierarchical feed

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Wed, May 28, 2014 at 03:19:50PM +0000, Paul Barker wrote:
> This patch allows for an optional new layout for ipk feed directories which I've
> called a 'hierarchical feed' and is based on how Debian pools package files. It
> is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".
> 

Has anyone had chance to look at this and see if they think it's a good idea?
The other patch in the series has been merged but I haven't had any feedback on
this one to say why it hasn't been merged.

Thanks,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

[-- Attachment #2: Type: application/pgp-signature, Size: 484 bytes --]