[PATCH RFC] Support hierarchical ipk feeds

[PATCH RFC] Support hierarchical ipk feeds

[Paul Barker]

This is an RFC only and is not suitable for merging yet. The necessary patches
for opkg-utils have also been posted for review but have not been merged yet. I
want to know if there is general approval for adding this feature to oe-core. If
there is, I'll go ahead and merge the opkg-utils patches and post a new patch
series which includes the update to the SRCREV used for opkg-utils in oe-core.

This patch has been tested on qemuarm:

  A feed was created with IPK_HIERARCHICAL_FEED unset and opkg was pointed at
  that feed. 'opkg update && opkg install ncurses-terminfo-base' worked
  correctly, downloading a package file from the same directory as the package
  feed.
  
  IPK_HIERARCHICAL_FEED was then set to "1" in local.conf and the feed was
  rebuilt by running 'bitbake core-image-minimal && bitbake package-index'. On
  qemu 'opkg update && opkg install ncurses-terminfo' worked correctly,
  downloading the package from the subdirectory 'n/ncurses-terminfo'.

  Finally, IPK_HIERARCHICAL_FEED was unset again and the feed was rebuilt. On
  qemu 'opkg update && opkg install ncurses' worked correctly, downloading
  package files from the same directory as the package feed again.

Those tests prove that changes to IPK_HIERARCHICAL_FEED do not break the upgrade
path of an existing system provided that 'bitbake pacakge-index' is re-ran after
the change.

The proposed commit message follows...



package_ipk.bbclass: Support hierarchical feed

This patch allows for an optional new layout for ipk feed directories which I've
called a 'hierarchical feed' and is based on how Debian pools package files. It
is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".

In the traditional feed layout, package files are placed in <outdir>/<arch>/.
This can lead to several thousand files existing in a single directory which is
often a problem if developers want to upload a package feed to a shared web
hosting provider. For example, in my case, listing files via FTP only shows the
first 2000 files, breaking my scripts which attempt to upload only new and
changed files via FTP.

In the hierarchical feed, package files are written to
'<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>', where pkg_prefix is the first
letter of the package file name for non-lib packages or "lib" plus the 4th
letter of the pacakge file name for lib pacakges (eg, 'l' for less, 'libc' for
libc6).  pkg_subdir is the root of the pacakge file name, discarding the version
and architecture parts and the common suffixes '-dbg', '-dev', '-doc',
'-staticdev' and '-locale-*'.

This change relies on recent patches to opkg-utils which support hierarchical
package feeds.

Signed-off-by: Paul Barker <paul@paulbarker.me.uk>
---
 meta/classes/package_ipk.bbclass | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 2949d1d..680b9bf 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -63,7 +63,26 @@ python do_package_ipk () {
         bb.data.update_data(localdata)
         basedir = os.path.join(os.path.dirname(root))
         arch = localdata.getVar('PACKAGE_ARCH', True)
-        pkgoutdir = "%s/%s" % (outdir, arch)
+        if localdata.getVar('IPK_HIERARCHICAL_FEED') == "1":
+            # Spread packages across subdirectories so each isn't too crowded
+            if pkgname.startswith('lib'):
+                pkg_prefix = 'lib' + pkgname[3]
+            else:
+                pkg_prefix = pkgname[0]
+
+            # Keep -dbg, -dev, -doc, -staticdev and -locale-* packages together
+            if pkgname[-4:] in ('-dbg', '-dev', '-doc'):
+                pkg_subdir = pkgname[:-4]
+            elif pkgname.endswith('-staticdev'):
+                pkg_subdir = pkgname[:-10]
+            elif '-locale-' in pkgname:
+                pkg_subdir = pkgname[:pkgname.find('-locale-')]
+            else:
+                pkg_subdir = pkgname
+
+            pkgoutdir = "%s/%s/%s/%s" % (outdir, arch, pkg_prefix, pkg_subdir)
+        else:
+            pkgoutdir = "%s/%s" % (outdir, arch)
         bb.utils.mkdirhier(pkgoutdir)
         os.chdir(root)
         cleanupcontrol(root)
-- 
1.9.2

Re: [PATCH RFC] Support hierarchical ipk feeds

[Richard Purdie]

On Thu, 2014-05-08 at 16:55 +0000, Paul Barker wrote:
> This is an RFC only and is not suitable for merging yet. The necessary patches
> for opkg-utils have also been posted for review but have not been merged yet. I
> want to know if there is general approval for adding this feature to oe-core. If
> there is, I'll go ahead and merge the opkg-utils patches and post a new patch
> series which includes the update to the SRCREV used for opkg-utils in oe-core.
> 
> This patch has been tested on qemuarm:
> 
>   A feed was created with IPK_HIERARCHICAL_FEED unset and opkg was pointed at
>   that feed. 'opkg update && opkg install ncurses-terminfo-base' worked
>   correctly, downloading a package file from the same directory as the package
>   feed.
>   
>   IPK_HIERARCHICAL_FEED was then set to "1" in local.conf and the feed was
>   rebuilt by running 'bitbake core-image-minimal && bitbake package-index'. On
>   qemu 'opkg update && opkg install ncurses-terminfo' worked correctly,
>   downloading the package from the subdirectory 'n/ncurses-terminfo'.
> 
>   Finally, IPK_HIERARCHICAL_FEED was unset again and the feed was rebuilt. On
>   qemu 'opkg update && opkg install ncurses' worked correctly, downloading
>   package files from the same directory as the package feed again.
> 
> Those tests prove that changes to IPK_HIERARCHICAL_FEED do not break the upgrade
> path of an existing system provided that 'bitbake pacakge-index' is re-ran after
> the change.
> 
> The proposed commit message follows...
> 
> 
> 
> package_ipk.bbclass: Support hierarchical feed
> 
> This patch allows for an optional new layout for ipk feed directories which I've
> called a 'hierarchical feed' and is based on how Debian pools package files. It
> is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".
> 
> In the traditional feed layout, package files are placed in <outdir>/<arch>/.
> This can lead to several thousand files existing in a single directory which is
> often a problem if developers want to upload a package feed to a shared web
> hosting provider. For example, in my case, listing files via FTP only shows the
> first 2000 files, breaking my scripts which attempt to upload only new and
> changed files via FTP.
> 
> In the hierarchical feed, package files are written to
> '<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>', where pkg_prefix is the first
> letter of the package file name for non-lib packages or "lib" plus the 4th
> letter of the pacakge file name for lib pacakges (eg, 'l' for less, 'libc' for
> libc6).  pkg_subdir is the root of the pacakge file name, discarding the version
> and architecture parts and the common suffixes '-dbg', '-dev', '-doc',
> '-staticdev' and '-locale-*'.
> 
> This change relies on recent patches to opkg-utils which support hierarchical
> package feeds.

FWIW I'm ok with this and its something we probably want to enable by
default however:

a) I'm not sure <pkg_subdir> is that useful, we tend to only have one
version of things around in most cases and I can't imagine people having
a lot of these.

b) How are the "common suffixes" configured? I guess its just a loose
thing but I worry about the "why did my package get put into directory X
when its really part of Y?" type questions. Perhaps splitting on the
first dash in the name might be good enough? I'm thinking about
kernel-modules-xxx for example and some of our other dynamically
generated packages.

Cheers,

Richard

Re: [PATCH RFC] Support hierarchical ipk feeds

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 5610 bytes --]

On Thu, May 08, 2014 at 11:03:05PM +0100, Richard Purdie wrote:
> On Thu, 2014-05-08 at 16:55 +0000, Paul Barker wrote:
> > This is an RFC only and is not suitable for merging yet. The necessary patches
> > for opkg-utils have also been posted for review but have not been merged yet. I
> > want to know if there is general approval for adding this feature to oe-core. If
> > there is, I'll go ahead and merge the opkg-utils patches and post a new patch
> > series which includes the update to the SRCREV used for opkg-utils in oe-core.
> > 
> > This patch has been tested on qemuarm:
> > 
> >   A feed was created with IPK_HIERARCHICAL_FEED unset and opkg was pointed at
> >   that feed. 'opkg update && opkg install ncurses-terminfo-base' worked
> >   correctly, downloading a package file from the same directory as the package
> >   feed.
> >   
> >   IPK_HIERARCHICAL_FEED was then set to "1" in local.conf and the feed was
> >   rebuilt by running 'bitbake core-image-minimal && bitbake package-index'. On
> >   qemu 'opkg update && opkg install ncurses-terminfo' worked correctly,
> >   downloading the package from the subdirectory 'n/ncurses-terminfo'.
> > 
> >   Finally, IPK_HIERARCHICAL_FEED was unset again and the feed was rebuilt. On
> >   qemu 'opkg update && opkg install ncurses' worked correctly, downloading
> >   package files from the same directory as the package feed again.
> > 
> > Those tests prove that changes to IPK_HIERARCHICAL_FEED do not break the upgrade
> > path of an existing system provided that 'bitbake pacakge-index' is re-ran after
> > the change.
> > 
> > The proposed commit message follows...
> > 
> > 
> > 
> > package_ipk.bbclass: Support hierarchical feed
> > 
> > This patch allows for an optional new layout for ipk feed directories which I've
> > called a 'hierarchical feed' and is based on how Debian pools package files. It
> > is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".
> > 
> > In the traditional feed layout, package files are placed in <outdir>/<arch>/.
> > This can lead to several thousand files existing in a single directory which is
> > often a problem if developers want to upload a package feed to a shared web
> > hosting provider. For example, in my case, listing files via FTP only shows the
> > first 2000 files, breaking my scripts which attempt to upload only new and
> > changed files via FTP.
> > 
> > In the hierarchical feed, package files are written to
> > '<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>', where pkg_prefix is the first
> > letter of the package file name for non-lib packages or "lib" plus the 4th
> > letter of the pacakge file name for lib pacakges (eg, 'l' for less, 'libc' for
> > libc6).  pkg_subdir is the root of the pacakge file name, discarding the version
> > and architecture parts and the common suffixes '-dbg', '-dev', '-doc',
> > '-staticdev' and '-locale-*'.
> > 
> > This change relies on recent patches to opkg-utils which support hierarchical
> > package feeds.
> 
> FWIW I'm ok with this and its something we probably want to enable by
> default however:
> 
> a) I'm not sure <pkg_subdir> is that useful, we tend to only have one
> version of things around in most cases and I can't imagine people having
> a lot of these.

pkg_subdir does reduce the number of files in a single directory quite a bit,
especially when several locale-specific packages are created.

Also, one of the features I'm currently working on in opkg is per-package
signatures. These would be separate '.sig' or '.asc' files next to each package
file so that will immediately double the number of files per directory if it is
used.

I do keep an archive of old packages around so that if something breaks I can
just tell people to revert back to the old version.

There is one thing slightly specific to my use case and that is that the number
of directories in each directory doesn't matter but the number of files in each
directory does matter. This is because the mirror operation will do a separate
'ls' for each subdirectory (cheap) whether or not it saw it in the 'ls' of the
parent directory, however it will do a full upload (expensive) for each file it
didn't see in the 'ls'. Thus every file past number 2000 is uploaded in full
even if it was already on the server. Fixing this would require patching lftp
which I haven't had time to look at yet.

> 
> b) How are the "common suffixes" configured? I guess its just a loose
> thing but I worry about the "why did my package get put into directory X
> when its really part of Y?" type questions. Perhaps splitting on the
> first dash in the name might be good enough? I'm thinking about
> kernel-modules-xxx for example and some of our other dynamically
> generated packages.
> 

The common suffixes are just my best guess at the minute. They certainly don't
group all related packages together. They just handle the suffixes that are
automatically packaged ('-dev', '-dbg', etc), plus locales. I've not even looked
through the packaging code to see if I've got the full list of automatic
suffixes - I'll do that before this is merged.

Splitting on the first dash could be confusing for packages which have a dash in
their 'well known' name (eg. util-linux, update-rc.d, etc).

'kernel-modules-xxx' looks good at first glance but it would immediately end up
with many many packages in one directory.

> Cheers,
> 
> Richard
> 
> 

Thanks,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

[-- Attachment #2: Type: application/pgp-signature, Size: 501 bytes --]