[PATCH 0/3] man/info: fix not support xz/bz2 compression

[PATCH 0/3] man/info: fix not support xz/bz2 compression

[Hongxu Jia]

Test Case:

vim local.conf
..
IMAGE_INSTALL_append = " man man-doc texinfo texinfo-doc info info-doc"
INHERIT += "compress_doc"
DOC_COMPRESS = "xz" or DOC_COMPRESS = "bz2"
..

bitbake core-image-minimal

runqemu qemux86

On target,
...
root@qemux86:~# man man

root@qemux86:~# info info
...

//Hongxu

The following changes since commit afdbe3112b940987f3f3cddcb32c91a5e17a297d:

  bitbake: bitbake-worker: Fix bitbake -n (2014-09-22 15:54:40 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/fix-doc
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-doc

Hongxu Jia (3):
  busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default
  man: fix not support xz/bz2 compression
  texinfo: fix info not work

 meta/recipes-core/busybox/busybox/defconfig  | 6 +++---
 meta/recipes-extended/man/man/man.conf       | 3 ++-
 meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
 3 files changed, 12 insertions(+), 5 deletions(-)

-- 
1.9.1

[PATCH 1/3] busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default

[Hongxu Jia]

It was required by info/man command to compress/decompress
doc pages with xz/bz2 compression.

[YOCTO #6750]
[YOCTO #6751]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-core/busybox/busybox/defconfig | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/busybox/busybox/defconfig b/meta/recipes-core/busybox/busybox/defconfig
index 8394067..676f770 100644
--- a/meta/recipes-core/busybox/busybox/defconfig
+++ b/meta/recipes-core/busybox/busybox/defconfig
@@ -134,7 +134,7 @@ CONFIG_AR=y
 # CONFIG_FEATURE_AR_LONG_FILENAMES is not set
 # CONFIG_FEATURE_AR_CREATE is not set
 CONFIG_BUNZIP2=y
-# CONFIG_BZIP2 is not set
+CONFIG_BZIP2=y
 CONFIG_CPIO=y
 # CONFIG_FEATURE_CPIO_O is not set
 # CONFIG_FEATURE_CPIO_P is not set
@@ -165,8 +165,8 @@ CONFIG_FEATURE_TAR_NOPRESERVE_TIME=y
 # CONFIG_UNLZMA is not set
 # CONFIG_FEATURE_LZMA_FAST is not set
 # CONFIG_LZMA is not set
-# CONFIG_UNXZ is not set
-# CONFIG_XZ is not set
+CONFIG_UNXZ=y
+CONFIG_XZ=y
 CONFIG_UNZIP=y
 
 #
-- 
1.9.1

Re: [PATCH 1/3] busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default

[Burton, Ross]

On 23 September 2014 15:19, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> It was required by info/man command to compress/decompress
> doc pages with xz/bz2 compression.

When changing busybox configuration, it's helpful to list the size change.

So for anyone else curious, buildhistory reports the package size (not
the binary size) as:

-PKGSIZE = 573632
+PKGSIZE = 585144

Ross

Re: [PATCH 1/3] busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default

[Hongxu Jia]

On 09/23/2014 10:47 PM, Burton, Ross wrote:
> On 23 September 2014 15:19, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>> It was required by info/man command to compress/decompress
>> doc pages with xz/bz2 compression.
> When changing busybox configuration, it's helpful to list the size change.
>
> So for anyone else curious, buildhistory reports the package size (not
> the binary size) as:
>
> -PKGSIZE = 573632
> +PKGSIZE = 585144

Since we care about the size, and affect of the fix is global.

How about conditionally add 'xz/bz2/gzip' to info/man's
RDEPENDS according DOC_COMPRESS value.

V2 incoming

//Hongxu


> Ross

[PATCH 2/3] man: fix not support xz/bz2 compression

[Hongxu Jia]

In oe-core, bunzip and unzx located in /usr/bin/
rather than /usr, so tweak man's config.

[YOCTO #6750]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-extended/man/man/man.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-extended/man/man/man.conf b/meta/recipes-extended/man/man/man.conf
index bb3c697..fc10be2 100644
--- a/meta/recipes-extended/man/man/man.conf
+++ b/meta/recipes-extended/man/man/man.conf
@@ -133,7 +133,8 @@ MANSECT		1:1p:8:2:3:3p:4:5:6:7:9:0p:tcl:n:l:p:o
 # The command given must act as a filter.
 #
 .gz		/bin/gunzip -c
-.bz2		/bin/bzip2 -c -d
+.bz2		/usr/bin/bunzip2 -c
+.xz		/usr/bin/unxz -c
 .z		
 .Z		/bin/zcat
 .F		
-- 
1.9.1

[PATCH 3/3] texinfo: fix info not work

[Hongxu Jia]

In texinfo, we didn't have info/dir entries, which caused
info not work.
...
root@qemux86:~# info info
info: dir: No such file or directory
...
Invoke install-info to create info/dir entries in pkg_postinst_info.

Also fix a QA Issue
...
ERROR: QA Issue: texinfo: Files/directories were installed but not shipped
  /usr/share/info/info.info.bz2
  /usr/share/info/info-stnd.info.bz2 [installed-vs-shipped]
...

[YOCTO #6751]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-extended/texinfo/texinfo_5.2.bb b/meta/recipes-extended/texinfo/texinfo_5.2.bb
index be90aba..91552a9 100644
--- a/meta/recipes-extended/texinfo/texinfo_5.2.bb
+++ b/meta/recipes-extended/texinfo/texinfo_5.2.bb
@@ -51,7 +51,7 @@ do_install_append_class-native() {
 PACKAGES += "info info-doc"
 
 FILES_info = "${bindir}/info ${bindir}/infokey ${bindir}/install-info"
-FILES_info-doc = "${infodir}/info.info ${infodir}/dir ${infodir}/info-*.info \
+FILES_info-doc = "${infodir}/info.info* ${infodir}/dir ${infodir}/info-*.info* \
                   ${mandir}/man1/info.1* ${mandir}/man5/info.5* \
                   ${mandir}/man1/infokey.1* ${mandir}/man1/install-info.1*"
 
@@ -61,4 +61,10 @@ FILES_${PN}-doc = "${infodir}/texinfo* \
                    ${datadir}/${tex_texinfo} \
                    ${mandir}/man1 ${mandir}/man5"
 
+pkg_postinst_info () {
+    for f in `find $D/${infodir} -name "*.info*"`; do
+        install-info $f $D/${infodir}/dir
+    done
+}
+
 BBCLASSEXTEND = "native"
-- 
1.9.1

Re: [PATCH 3/3] texinfo: fix info not work

[Burton, Ross]

On 23 September 2014 15:19, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> +pkg_postinst_info () {
> +    for f in `find $D/${infodir} -name "*.info*"`; do
> +        install-info $f $D/${infodir}/dir
> +    done
> +}

For future, this seems like the sort of thing we need a class for.
There is a texinfo.bbclass now...

Ross

Re: [PATCH 3/3] texinfo: fix info not work

[Richard Purdie]

On Tue, 2014-09-23 at 22:19 +0800, Hongxu Jia wrote:
> In texinfo, we didn't have info/dir entries, which caused
> info not work.
> ...
> root@qemux86:~# info info
> info: dir: No such file or directory
> ...
> Invoke install-info to create info/dir entries in pkg_postinst_info.
> 
> Also fix a QA Issue
> ...
> ERROR: QA Issue: texinfo: Files/directories were installed but not shipped
>   /usr/share/info/info.info.bz2
>   /usr/share/info/info-stnd.info.bz2 [installed-vs-shipped]
> ...
> 
> [YOCTO #6751]
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-extended/texinfo/texinfo_5.2.bb b/meta/recipes-extended/texinfo/texinfo_5.2.bb
> index be90aba..91552a9 100644
> --- a/meta/recipes-extended/texinfo/texinfo_5.2.bb
> +++ b/meta/recipes-extended/texinfo/texinfo_5.2.bb
> @@ -51,7 +51,7 @@ do_install_append_class-native() {
>  PACKAGES += "info info-doc"
>  
>  FILES_info = "${bindir}/info ${bindir}/infokey ${bindir}/install-info"
> -FILES_info-doc = "${infodir}/info.info ${infodir}/dir ${infodir}/info-*.info \
> +FILES_info-doc = "${infodir}/info.info* ${infodir}/dir ${infodir}/info-*.info* \
>                    ${mandir}/man1/info.1* ${mandir}/man5/info.5* \
>                    ${mandir}/man1/infokey.1* ${mandir}/man1/install-info.1*"
>  
> @@ -61,4 +61,10 @@ FILES_${PN}-doc = "${infodir}/texinfo* \
>                     ${datadir}/${tex_texinfo} \
>                     ${mandir}/man1 ${mandir}/man5"
>  
> +pkg_postinst_info () {
> +    for f in `find $D/${infodir} -name "*.info*"`; do
> +        install-info $f $D/${infodir}/dir
> +    done
> +}
> +
>  BBCLASSEXTEND = "native"


I think there are a few issues here. Firstly, this postinst can run at
build time which means there is a dependency on texinfo-native. I guess
if this fails, it will just run on target so its probably "harmless" but
not deterministic. Secondly, anything installing an info directory
really needs to update the dir files with its own postinst. For optimal
handling at rootfs time, that will need trigger/intercept support.

So this probably can merge as it improves things however we need some
enhacement bugs opening related to this, we may want to force the
postinst to run on target for now so things are deterministic too.

Cheers,

Richard

Re: [PATCH 3/3] texinfo: fix info not work

[Burton, Ross]

On 23 September 2014 21:37, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I think there are a few issues here. Firstly, this postinst can run at
> build time which means there is a dependency on texinfo-native. I guess
> if this fails, it will just run on target so its probably "harmless" but
> not deterministic. Secondly, anything installing an info directory
> really needs to update the dir files with its own postinst. For optimal
> handling at rootfs time, that will need trigger/intercept support.
>
> So this probably can merge as it improves things however we need some
> enhacement bugs opening related to this, we may want to force the
> postinst to run on target for now so things are deterministic too.

There's a texinfo.bbclass which all recipes installing info files are
probably inheriting anyway, so moving the postinst there should solve
the race on the directory.  That class says that texinfo-native is
assume-provided but you'll want to check that usual configurations
ship install-info alongside makeinfo on the distributions we support.

Ross

Re: [PATCH 3/3] texinfo: fix info not work

[Hongxu Jia]

On 09/24/2014 04:40 AM, Burton, Ross wrote:
> On 23 September 2014 21:37, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> I think there are a few issues here. Firstly, this postinst can run at
>> build time which means there is a dependency on texinfo-native. I guess
>> if this fails, it will just run on target so its probably "harmless" but
>> not deterministic. Secondly, anything installing an info directory
>> really needs to update the dir files with its own postinst. For optimal
>> handling at rootfs time, that will need trigger/intercept support.
>>
>> So this probably can merge as it improves things however we need some
>> enhacement bugs opening related to this, we may want to force the
>> postinst to run on target for now so things are deterministic too.
> There's a texinfo.bbclass which all recipes installing info files are
> probably inheriting anyway, so moving the postinst there should solve
> the race on the directory.  That class says that texinfo-native is
> assume-provided but you'll want to check that usual configurations
> ship install-info alongside makeinfo on the distributions we support.

Hi Ross,

I am afraid moving the postinst to texinfo.bbclass will cause some
potential issues:

- It will be invoked many times if multiple packages installed.

- We could not use a generic way to name pkg_postinst.
   Such as texinfo recipe, we need pkg_postinst_info.

I will refer what Ubuntu/Debian did, add postinst/postrm to info
packages.

//Hongxu

> Ross

Re: [PATCH 3/3] texinfo: fix info not work

[Burton, Ross]

On 24 September 2014 10:16, Hongxu Jia <hongxu.jia@windriver.com> wrote:
> I am afraid moving the postinst to texinfo.bbclass will cause some
> potential issues:
>
> - It will be invoked many times if multiple packages installed.

Sure, we could make it a trigger in the future.

> - We could not use a generic way to name pkg_postinst.
>   Such as texinfo recipe, we need pkg_postinst_info.

Many classes extend the postinst scripts, so this isn't a problem.

Ross

Re: [PATCH 3/3] texinfo: fix info not work

[Hongxu Jia]

On 09/24/2014 04:37 AM, Richard Purdie wrote:
> I think there are a few issues here. Firstly, this postinst can run at
> build time which means there is a dependency on texinfo-native. I guess
> if this fails, it will just run on target so its probably "harmless" but
> not deterministic. Secondly, anything installing an info directory
> really needs to update the dir files with its own postinst. For optimal
> handling at rootfs time, that will need trigger/intercept support.

Hi Richard,

Yes, you are right, I will refer Ubuntu to add a script to 
postinst_intercept
to do that, run it after all packages have been installed at build time.

And about trigger, but I have no idea about how to implement it. It seems
we support pkg_postinst/pkg_preinst but don't support pkg_triggers,
Refer triggers in install-info_5.2.0.dfsg.1-4_amd64.deb, we got:
...
interest-noawait /usr/share/info
...

V2 incoming

//Hongxu

> So this probably can merge as it improves things however we need some
> enhacement bugs opening related to this, we may want to force the
> postinst to run on target for now so things are deterministic too.
>
> Cheers,
>
> Richard
>
>

Re: [PATCH 0/3] man/info: fix not support xz/bz2 compression

[Mark Hatle]

On 9/23/14, 9:19 AM, Hongxu Jia wrote:
> Test Case:
>
> vim local.conf
> ..
> IMAGE_INSTALL_append = " man man-doc texinfo texinfo-doc info info-doc"
> INHERIT += "compress_doc"
> DOC_COMPRESS = "xz" or DOC_COMPRESS = "bz2"
> ..

Patch 1/3 the busybox change.  I'm not sure we really want to make that change 
as it will increase the size of busybox.

(following is me thinking aloud and hopefully someone will have a suggest on how 
best to unify this..)

I'm wondering if instead of forcing the change for everyone via the busybox 
change, if it makes sense to add a image validation routine or something that 
verifies if the DOC_COMPRESS is set, that the corresponding tool was installed 
into the target environment (along w/ man, info or other appropriate items.. and 
the pkgs-docs was set as well.. if any of those are missing then a warning 
should probably be presented.)

I'm sure there is probably a better way to set this via a distribution 
configuration switch or something as well.. maybe something like?

DISTRO += "man info"

That could then force the man and info systems to add a dependency on bz2 (or xz 
or whatever else was set in DOC_COMPRESS)..  and also add man, info and the 
associated man and info pages to the image itself.

(As I see it, DOC_COMPRESS is really a distribution wide setting, so we should 
treat it as such...)


--Mark

> bitbake core-image-minimal
>
> runqemu qemux86
>
> On target,
> ...
> root@qemux86:~# man man
>
> root@qemux86:~# info info
> ...
>
> //Hongxu
>
> The following changes since commit afdbe3112b940987f3f3cddcb32c91a5e17a297d:
>
>    bitbake: bitbake-worker: Fix bitbake -n (2014-09-22 15:54:40 +0100)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/fix-doc
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-doc
>
> Hongxu Jia (3):
>    busybox: enable CONFIG_BZIP2/CONFIG_XZ/CONFIG_UNXZ by default
>    man: fix not support xz/bz2 compression
>    texinfo: fix info not work
>
>   meta/recipes-core/busybox/busybox/defconfig  | 6 +++---
>   meta/recipes-extended/man/man/man.conf       | 3 ++-
>   meta/recipes-extended/texinfo/texinfo_5.2.bb | 8 +++++++-
>   3 files changed, 12 insertions(+), 5 deletions(-)
>