[PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[Alejandro Hernandez]

If a the number of hard links decreases or increases while creating
the tar files used for an ipk package, tar fails with error code 1,
if this is the case we ignore the error and continue to create the ipk file

[YOCTO #7933]

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
---
 .../opkg-utils/opkg-utils/tar_ignore_error.patch   | 47 ++++++++++++++++++++++
 meta/recipes-devtools/opkg-utils/opkg-utils_git.bb |  4 +-
 2 files changed, 50 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch

diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
new file mode 100644
index 0000000..4bcae4954
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
@@ -0,0 +1,47 @@
+If a the number of hard links decreases or increases while creating
+the tar files used for an ipk package, tar fails with error code 1:
+
+| DEBUG: Executing python function do_package_ipk
+| tar: ./usr/src/debug/gperf/3.0.4-r0/gperf-3.0.4/src/main.cc: file changed as we read it
+NOTE: recipe gperf-3.0.4-r0: task do_package_write_ipk: Failed
+ERROR: Task 6539 (recipes-extended/gperf/gperf_3.0.4.bb, do_package_write_ipk) failed with exit code '1'
+
+
+We detect if the error code produced by tar is 1 and in this case ignore it.
+
+This a similar behavior to the one on dpkg:
+http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
+
+Upsteam-Status: Inappropriate
+
+Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
+
+
+Index: git/opkg-build
+===================================================================
+--- git.orig/opkg-build
++++ git/opkg-build
+@@ -250,8 +250,21 @@ tmp_dir=$dest_dir/IPKG_BUILD.$$
+ mkdir $tmp_dir
+ 
+ echo $CONTROL > $tmp_dir/tarX
+-( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . )
+-( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . )
++
++
++# Ignore error code 1, caused by modifying the number of hard links while creating the tar file
++rc=0
++( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . ) || rc=$?
++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then
++        exit $rc 
++fi
++
++rc=0
++( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . ) || rc=$?
++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then
++        exit $rc
++fi
++
+ rm $tmp_dir/tarX
+ 
+ echo "2.0" > $tmp_dir/debian-binary
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
index 2800a5d..74b974d 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
@@ -10,7 +10,9 @@ PROVIDES += "virtual/update-alternatives"
 SRCREV = "53274f087565fd45d8452c5367997ba6a682a37a"
 PV = "0.1.8+git${SRCPV}"
 
-SRC_URI = "git://git.yoctoproject.org/opkg-utils"
+SRC_URI = "git://git.yoctoproject.org/opkg-utils " 
+
+SRC_URI_append_class-native = "file://tar_ignore_error.patch"
 
 S = "${WORKDIR}/git"
 
-- 
1.8.4.5

Re: [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4060 bytes --]

On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
> If a the number of hard links decreases or increases while creating
> the tar files used for an ipk package, tar fails with error code 1,
> if this is the case we ignore the error and continue to create the ipk file
> 
> [YOCTO #7933]
> 
> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
> ---
>  .../opkg-utils/opkg-utils/tar_ignore_error.patch   | 47 ++++++++++++++++++++++
>  meta/recipes-devtools/opkg-utils/opkg-utils_git.bb |  4 +-
>  2 files changed, 50 insertions(+), 1 deletion(-)
>  create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
> 
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
> new file mode 100644
> index 0000000..4bcae4954
> --- /dev/null
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
> @@ -0,0 +1,47 @@
> +If a the number of hard links decreases or increases while creating
> +the tar files used for an ipk package, tar fails with error code 1:
> +
> +| DEBUG: Executing python function do_package_ipk
> +| tar: ./usr/src/debug/gperf/3.0.4-r0/gperf-3.0.4/src/main.cc: file changed as we read it
> +NOTE: recipe gperf-3.0.4-r0: task do_package_write_ipk: Failed
> +ERROR: Task 6539 (recipes-extended/gperf/gperf_3.0.4.bb, do_package_write_ipk) failed with exit code '1'
> +
> +
> +We detect if the error code produced by tar is 1 and in this case ignore it.
> +
> +This a similar behavior to the one on dpkg:
> +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
> +
> +Upsteam-Status: Inappropriate

Typo and why is it inappropriate? Looks like bugfix to me, even for
issue which usually happens in OE way of using opkg-build.

> +
> +Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
> +
> +
> +Index: git/opkg-build
> +===================================================================
> +--- git.orig/opkg-build
> ++++ git/opkg-build
> +@@ -250,8 +250,21 @@ tmp_dir=$dest_dir/IPKG_BUILD.$$
> + mkdir $tmp_dir
> + 
> + echo $CONTROL > $tmp_dir/tarX
> +-( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . )
> +-( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . )
> ++
> ++
> ++# Ignore error code 1, caused by modifying the number of hard links while creating the tar file
> ++rc=0
> ++( cd $pkg_dir && tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f $tmp_dir/data.tar.gz . ) || rc=$?
> ++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then
> ++        exit $rc 
> ++fi
> ++
> ++rc=0
> ++( cd $pkg_dir/$CONTROL && tar $ogargs -cz $tarformat -f $tmp_dir/control.tar.gz . ) || rc=$?
> ++if [[ $rc -ne 1 ]] && [[ $rc -ne 0 ]] ;then
> ++        exit $rc
> ++fi
> ++
> + rm $tmp_dir/tarX
> + 
> + echo "2.0" > $tmp_dir/debian-binary
> diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
> index 2800a5d..74b974d 100644
> --- a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
> +++ b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
> @@ -10,7 +10,9 @@ PROVIDES += "virtual/update-alternatives"
>  SRCREV = "53274f087565fd45d8452c5367997ba6a682a37a"
>  PV = "0.1.8+git${SRCPV}"
>  
> -SRC_URI = "git://git.yoctoproject.org/opkg-utils"
> +SRC_URI = "git://git.yoctoproject.org/opkg-utils " 
> +
> +SRC_URI_append_class-native = "file://tar_ignore_error.patch"

Add leading space in append not in original SRC_URI variable.

Why is it applied only for native anyway?

>  
>  S = "${WORKDIR}/git"
>  
> -- 
> 1.8.4.5
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[Alejandro Hernandez]

On 06/07/15 15:10, Martin Jansa wrote:
> On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
>
> +
> +This a similar behavior to the one on dpkg:
> +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
> +
> +Upsteam-Status: Inappropriate
> Typo and why is it inappropriate? Looks like bugfix to me, even for
> issue which usually happens in OE way of using opkg-build.

It seemed inappropriate to me, since this behavior is not expected 
(number of hardlinks changing), we can't say for sure that this error 
will be caused by this reason on when using opkg on another environment
>   
> -SRC_URI = "git://git.yoctoproject.org/opkg-utils"
> +SRC_URI = "git://git.yoctoproject.org/opkg-utils "
> +
> +SRC_URI_append_class-native = "file://tar_ignore_error.patch"
> Add leading space in append not in original SRC_URI variable.
>
> Why is it applied only for native anyway?

Same argument as before, modifying the number of hardlinks while the tar 
is created happens when building using bitbake, which should not happen 
on target, if files changed when creating an ipk for any other reason it 
shouldn't be ignored; space was a typo

Re: [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[Gary Thomas]

On 2015-07-06 15:10, Alejandro Hernandez wrote:
> On 06/07/15 15:10, Martin Jansa wrote:
>> On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
>>
>> +
>> +This a similar behavior to the one on dpkg:
>> +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
>> +
>> +Upsteam-Status: Inappropriate
>> Typo and why is it inappropriate? Looks like bugfix to me, even for
>> issue which usually happens in OE way of using opkg-build.
>
> It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using
> opkg on another environment
>> -SRC_URI = "git://git.yoctoproject.org/opkg-utils"
>> +SRC_URI = "git://git.yoctoproject.org/opkg-utils "
>> +
>> +SRC_URI_append_class-native = "file://tar_ignore_error.patch"
>> Add leading space in append not in original SRC_URI variable.
>>
>> Why is it applied only for native anyway?
>
> Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when
> creating an ipk for any other reason it shouldn't be ignored; space was a typo

Why is it happening when using bitbake?  This seems like an
error to me - one should not be generating the tarball (i.e.
packing up the final results) if the set of files being packaged
is changing in any way.

Note: I'm pretty sure that error can also happen if the inode
changes in many different ways, not just the number of links
changing.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[Otavio Salvador]

Hello,

On Mon, Jul 6, 2015 at 6:16 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> On 2015-07-06 15:10, Alejandro Hernandez wrote:
>>
>> On 06/07/15 15:10, Martin Jansa wrote:
>>>
>>> On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
>>>
>>> +
>>> +This a similar behavior to the one on dpkg:
>>>
>>> +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
>>> +
>>> +Upsteam-Status: Inappropriate
>>> Typo and why is it inappropriate? Looks like bugfix to me, even for
>>> issue which usually happens in OE way of using opkg-build.
>>
>>
>> It seemed inappropriate to me, since this behavior is not expected (number
>> of hardlinks changing), we can't say for sure that this error will be caused
>> by this reason on when using
>> opkg on another environment
>>>
>>> -SRC_URI = "git://git.yoctoproject.org/opkg-utils"
>>> +SRC_URI = "git://git.yoctoproject.org/opkg-utils "
>>> +
>>> +SRC_URI_append_class-native = "file://tar_ignore_error.patch"
>>> Add leading space in append not in original SRC_URI variable.
>>>
>>> Why is it applied only for native anyway?
>>
>>
>> Same argument as before, modifying the number of hardlinks while the tar
>> is created happens when building using bitbake, which should not happen on
>> target, if files changed when
>> creating an ipk for any other reason it shouldn't be ignored; space was a
>> typo
>
>
> Why is it happening when using bitbake?  This seems like an
> error to me - one should not be generating the tarball (i.e.
> packing up the final results) if the set of files being packaged
> is changing in any way.
>
> Note: I'm pretty sure that error can also happen if the inode
> changes in many different ways, not just the number of links
> changing.

I fully agree with Gary; this also makes unreproducible the number of
shared hard links in a final rootfs which is also important.

This (and dpkg related patch) seems to be covering the real bug.

(Adding Richard so he can also comment on this and to bring his
attention to the concerns exposed here)

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files

[Richard Purdie]

On Mon, 2015-07-06 at 18:20 -0300, Otavio Salvador wrote:
> Hello,
> 
> On Mon, Jul 6, 2015 at 6:16 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> > On 2015-07-06 15:10, Alejandro Hernandez wrote:
> >>
> >> On 06/07/15 15:10, Martin Jansa wrote:
> >>>
> >>> On Mon, Jul 06, 2015 at 10:17:19AM +0000, Alejandro Hernandez wrote:
> >>>
> >>> +
> >>> +This a similar behavior to the one on dpkg:
> >>>
> >>> +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
> >>> +
> >>> +Upsteam-Status: Inappropriate
> >>> Typo and why is it inappropriate? Looks like bugfix to me, even for
> >>> issue which usually happens in OE way of using opkg-build.
> >>
> >>
> >> It seemed inappropriate to me, since this behavior is not expected (number
> >> of hardlinks changing), we can't say for sure that this error will be caused
> >> by this reason on when using
> >> opkg on another environment
> >>>
> >>> -SRC_URI = "git://git.yoctoproject.org/opkg-utils"
> >>> +SRC_URI = "git://git.yoctoproject.org/opkg-utils "
> >>> +
> >>> +SRC_URI_append_class-native = "file://tar_ignore_error.patch"
> >>> Add leading space in append not in original SRC_URI variable.
> >>>
> >>> Why is it applied only for native anyway?
> >>
> >>
> >> Same argument as before, modifying the number of hardlinks while the tar
> >> is created happens when building using bitbake, which should not happen on
> >> target, if files changed when
> >> creating an ipk for any other reason it shouldn't be ignored; space was a
> >> typo
> >
> >
> > Why is it happening when using bitbake?  This seems like an
> > error to me - one should not be generating the tarball (i.e.
> > packing up the final results) if the set of files being packaged
> > is changing in any way.
> >
> > Note: I'm pretty sure that error can also happen if the inode
> > changes in many different ways, not just the number of links
> > changing.
> 
> I fully agree with Gary; this also makes unreproducible the number of
> shared hard links in a final rootfs which is also important.
> 
> This (and dpkg related patch) seems to be covering the real bug.
> 
> (Adding Richard so he can also comment on this and to bring his
> attention to the concerns exposed here)

The issue here is some of the attempts to decrease disk usage and
increase the build speed. To do that we hardlink files in the package
directories rather than create individual copies in packages/ and
package-split/. There are cases where other parts of the build (e.g.
populate-sysroot) can create or remote copies of the hardlinked file
which changes the link count.

The files themselves do not change, only the link does and then only
rarely. We fixed dpkg a while ago for this issue, this is the same fix
being applied to opkg-build. The resulting packages will be the same
since the hardlink count is internal to the tarball and not related to
the number of original file copies. Since we control the build env, it
is safe to do what we're doing here, however I would agree with
Alejandro that it isn't something upstream would want to do.

Cheers,

Richard