opkg 0.3.0 or rootfs task

opkg 0.3.0 or rootfs task

[Ahsan, Noor]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

Hello,

I noticed that at the time of rootfs creation symbolic links of the ipk files present in deploy/ipk. The problem what have it while creation of symbolic link it take the full path till that ipk and remove slashes and convert them into underscores. Use that as the name of the symbolic link. This make a very long file names. If we have very long path then name of the symlink exceed from max filename limits. And we get following error

Collected errors:
* file_link: unable to stat `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk': File name too long.

Can anyone tell me why the addition of full path was added to the symlink name and can we remove it cause it is cause issues?

Noor

[-- Attachment #2: Type: text/html, Size: 3151 bytes --]

Re: opkg 0.3.0 or rootfs task

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 1515 bytes --]


> On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com> wrote:
> 
> Hello,
> 
> I noticed that at the time of rootfs creation symbolic links of the ipk files present in deploy/ipk. The problem what have it while creation of symbolic link it take the full path till that ipk and remove slashes and convert them into underscores. Use that as the name of the symbolic link. This make a very long file names. If we have very long path then name of the symlink exceed from max filename limits. And we get following error
> 
> Collected errors:
> * file_link: unable to stat `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk': File name too long.
> 
> Can anyone tell me why the addition of full path was added to the symlink name and can we remove it cause it is cause issues?

what does

getconf PATH_MAX /

show ?

> 
> Noor
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
> https://lists.yoctoproject.org/listinfo/yocto <https://lists.yoctoproject.org/listinfo/yocto>

[-- Attachment #1.2: Type: text/html, Size: 7270 bytes --]

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: opkg 0.3.0 or rootfs task

[Ahsan, Noor]

[-- Attachment #1: Type: text/plain, Size: 1786 bytes --]



From: Khem Raj [mailto:raj.khem@gmail.com]
Sent: Friday, October 16, 2015 9:54 PM
To: Ahsan, Noor
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] opkg 0.3.0 or rootfs task


On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com<mailto:Noor_Ahsan@mentor.com>> wrote:

Hello,

I noticed that at the time of rootfs creation symbolic links of the ipk files present in deploy/ipk. The problem what have it while creation of symbolic link it take the full path till that ipk and remove slashes and convert them into underscores. Use that as the name of the symbolic link. This make a very long file names. If we have very long path then name of the symlink exceed from max filename limits. And we get following error

Collected errors:
* file_link: unable to stat `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk': File name too long.

Can anyone tell me why the addition of full path was added to the symlink name and can we remove it cause it is cause issues?

what does

getconf PATH_MAX /

show ?


jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
PATH_MAX                           4096
_POSIX_PATH_MAX                    4096

Noor
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto


[-- Attachment #2: Type: text/html, Size: 7749 bytes --]

Re: opkg 0.3.0 or rootfs task

[Ahsan, Noor]

[-- Attachment #1: Type: text/plain, Size: 2245 bytes --]



From: yocto-bounces@yoctoproject.org [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Ahsan, Noor
Sent: Friday, October 16, 2015 11:55 PM
To: Khem Raj
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] opkg 0.3.0 or rootfs task



From: Khem Raj [mailto:raj.khem@gmail.com]
Sent: Friday, October 16, 2015 9:54 PM
To: Ahsan, Noor
Cc: yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
Subject: Re: [yocto] opkg 0.3.0 or rootfs task


On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com<mailto:Noor_Ahsan@mentor.com>> wrote:

Hello,

I noticed that at the time of rootfs creation symbolic links of the ipk files present in deploy/ipk. The problem what have it while creation of symbolic link it take the full path till that ipk and remove slashes and convert them into underscores. Use that as the name of the symbolic link. This make a very long file names. If we have very long path then name of the symlink exceed from max filename limits. And we get following error

Collected errors:
* file_link: unable to stat `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk': File name too long.

Can anyone tell me why the addition of full path was added to the symlink name and can we remove it cause it is cause issues?

what does

getconf PATH_MAX /

show ?

jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
PATH_MAX                           4096
_POSIX_PATH_MAX                    4096


I think the issue is with file name not the path.

Secondly the googling which I did reveals that symlink creation can't be stopped. I just wanna confirm that is my findings correct?

Noor
--
_______________________________________________
yocto mailing list
yocto@yoctoproject.org<mailto:yocto@yoctoproject.org>
https://lists.yoctoproject.org/listinfo/yocto


[-- Attachment #2: Type: text/html, Size: 9860 bytes --]

Re: opkg 0.3.0 or rootfs task

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 2531 bytes --]

On Fri, Oct 16, 2015 at 07:38:19PM +0000, Ahsan, Noor wrote:
> On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com<mailto:Noor_Ahsan@mentor.com>> wrote:
> 
> Hello,
> 
> I noticed that at the time of rootfs creation symbolic links of the ipk files present in deploy/ipk. The problem what have it while creation of symbolic link it take the full path till that ipk and remove slashes and convert them into underscores. Use that as the name of the symbolic link. This make a very long file names. If we have very long path then name of the symlink exceed from max filename limits. And we get following error
> 
> Collected errors:
> * file_link: unable to stat `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk': File name too long.
> 
> Can anyone tell me why the addition of full path was added to the symlink name and can we remove it cause it is cause issues?
> 
> what does
> 
> getconf PATH_MAX /
> 
> show ?
> 
> jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
> PATH_MAX                           4096
> _POSIX_PATH_MAX                    4096
> 
> 
> I think the issue is with file name not the path.
> 
> Secondly the googling which I did reveals that symlink creation can't be stopped. I just wanna confirm that is my findings correct?
> 

This looks like something we overlooked in opkg. When we added the caching code
we didn't think about how long the paths and filenames might get during the
rootfs step. It's not currently possible to reduce the length of the symlink
filenames, but it is possible to change the directory in which the symlinks are
created.

Currently the opkg cache dir can only be set in the opkg.conf file. I think we
should add a '--cache-dir' argument to opkg. If this is added you'll be able to
set the following in your local.conf file to change the cache location. Eg. to
use '/tmp/opkg' on the host during rootfs creation.

    OPKG_ARGS = "--cache-dir=/tmp/opkg"

I'll submit a patch to opkg to add this option.

Thanks,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: opkg 0.3.0 or rootfs task

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3087 bytes --]

On Sun, Oct 18, 2015 at 2:57 AM, Paul Barker <paul@paulbarker.me.uk> wrote:

> On Fri, Oct 16, 2015 at 07:38:19PM +0000, Ahsan, Noor wrote:
> > On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com<mailto:
> Noor_Ahsan@mentor.com>> wrote:
> >
> > Hello,
> >
> > I noticed that at the time of rootfs creation symbolic links of the ipk
> files present in deploy/ipk. The problem what have it while creation of
> symbolic link it take the full path till that ipk and remove slashes and
> convert them into underscores. Use that as the name of the symbolic link.
> This make a very long file names. If we have very long path then name of
> the symlink exceed from max filename limits. And we get following error
> >
> > Collected errors:
> > * file_link: unable to stat
> `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/
> opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk':
> File name too long.
> >
> > Can anyone tell me why the addition of full path was added to the
> symlink name and can we remove it cause it is cause issues?
> >
> > what does
> >
> > getconf PATH_MAX /
> >
> > show ?
> >
> > jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
> > PATH_MAX                           4096
> > _POSIX_PATH_MAX                    4096
> >
> >
> > I think the issue is with file name not the path.
> >
> > Secondly the googling which I did reveals that symlink creation can't be
> stopped. I just wanna confirm that is my findings correct?
> >
>
> This looks like something we overlooked in opkg. When we added the
> caching code
> we didn't think about how long the paths and filenames might get during the
> rootfs step. It's not currently possible to reduce the length of the
> symlink
> filenames, but it is possible to change the directory in which the
> symlinks are
> created.
>
> Currently the opkg cache dir can only be set in the opkg.conf file. I
> think we
> should add a '--cache-dir' argument to opkg. If this is added you'll be
> able to
> set the following in your local.conf file to change the cache location.
> Eg. to
> use '/tmp/opkg' on the host during rootfs creation.
>
>     OPKG_ARGS = "--cache-dir=/tmp/opkg"
>
> I'll submit a patch to opkg to add this option.
>

This will only shorten the full path, not the filename length, so I doubt
this'll solve it. That said, I can't actually successfully test this today,
because cache_dir is made relative to offline_root, so setting such a path
as you suggest doesn't shorten the full path either.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4029 bytes --]

Re: opkg 0.3.0 or rootfs task

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 3523 bytes --]

On Tue, Oct 20, 2015 at 10:14 AM, Christopher Larson <clarson@kergoth.com>
wrote:

>
> On Sun, Oct 18, 2015 at 2:57 AM, Paul Barker <paul@paulbarker.me.uk>
> wrote:
>
>> On Fri, Oct 16, 2015 at 07:38:19PM +0000, Ahsan, Noor wrote:
>> > On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com<mailto:
>> Noor_Ahsan@mentor.com>> wrote:
>> >
>> > Hello,
>> >
>> > I noticed that at the time of rootfs creation symbolic links of the ipk
>> files present in deploy/ipk. The problem what have it while creation of
>> symbolic link it take the full path till that ipk and remove slashes and
>> convert them into underscores. Use that as the name of the symbolic link.
>> This make a very long file names. If we have very long path then name of
>> the symlink exceed from max filename limits. And we get following error
>> >
>> > Collected errors:
>> > * file_link: unable to stat
>> `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/
>> opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk':
>> File name too long.
>> >
>> > Can anyone tell me why the addition of full path was added to the
>> symlink name and can we remove it cause it is cause issues?
>> >
>> > what does
>> >
>> > getconf PATH_MAX /
>> >
>> > show ?
>> >
>> > jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
>> > PATH_MAX                           4096
>> > _POSIX_PATH_MAX                    4096
>> >
>> >
>> > I think the issue is with file name not the path.
>> >
>> > Secondly the googling which I did reveals that symlink creation can't
>> be stopped. I just wanna confirm that is my findings correct?
>> >
>>
>> This looks like something we overlooked in opkg. When we added the
>> caching code
>> we didn't think about how long the paths and filenames might get during
>> the
>> rootfs step. It's not currently possible to reduce the length of the
>> symlink
>> filenames, but it is possible to change the directory in which the
>> symlinks are
>> created.
>>
>> Currently the opkg cache dir can only be set in the opkg.conf file. I
>> think we
>> should add a '--cache-dir' argument to opkg. If this is added you'll be
>> able to
>> set the following in your local.conf file to change the cache location.
>> Eg. to
>> use '/tmp/opkg' on the host during rootfs creation.
>>
>>     OPKG_ARGS = "--cache-dir=/tmp/opkg"
>>
>> I'll submit a patch to opkg to add this option.
>>
>
> This will only shorten the full path, not the filename length, so I doubt
> this'll solve it. That said, I can't actually successfully test this today,
> because cache_dir is made relative to offline_root, so setting such a path
> as you suggest doesn't shorten the full path either.


Also, did a touch of just the cache filename and it gives the same filename
length error, so where the cache dir is really isn't going to matter, it's
the filename including the full path to a deep BUILDDIR, and therefore
DEPLOY_DIR_IPK, which is the problem.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4832 bytes --]

Re: opkg 0.3.0 or rootfs task

[Ahsan, Noor]

[-- Attachment #1: Type: text/plain, Size: 3993 bytes --]

We are hitting this issue on another BSP

file:_var_jenkins_workspace_ce-main_buildhost_SB64_buildtype_industrial_machine_zedboard-zynq7-mel_build_zedboard-zynq7-mel_tmp_deploy_ipk_zedboard_zynq7_mel_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_zedboard_zynq7_mel.ipk

need its solution quickly

Noor

From: kergoth@gmail.com [mailto:kergoth@gmail.com] On Behalf Of Christopher Larson
Sent: Tuesday, October 20, 2015 10:20 PM
To: Paul Barker
Cc: Ahsan, Noor; yocto@yoctoproject.org
Subject: Re: [yocto] opkg 0.3.0 or rootfs task


On Tue, Oct 20, 2015 at 10:14 AM, Christopher Larson <clarson@kergoth.com<mailto:clarson@kergoth.com>> wrote:

On Sun, Oct 18, 2015 at 2:57 AM, Paul Barker <paul@paulbarker.me.uk<mailto:paul@paulbarker.me.uk>> wrote:
On Fri, Oct 16, 2015 at 07:38:19PM +0000, Ahsan, Noor wrote:
> On Oct 16, 2015, at 5:47 AM, Ahsan, Noor <Noor_Ahsan@mentor.com<mailto:Noor_Ahsan@mentor.com><mailto:Noor_Ahsan@mentor.com<mailto:Noor_Ahsan@mentor.com>>> wrote:
>
> Hello,
>
> I noticed that at the time of rootfs creation symbolic links of the ipk files present in deploy/ipk. The problem what have it while creation of symbolic link it take the full path till that ipk and remove slashes and convert them into underscores. Use that as the name of the symbolic link. This make a very long file names. If we have very long path then name of the symlink exceed from max filename limits. And we get following error
>
> Collected errors:
> * file_link: unable to stat `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk': File name too long.
>
> Can anyone tell me why the addition of full path was added to the symlink name and can we remove it cause it is cause issues?
>
> what does
>
> getconf PATH_MAX /
>
> show ?
>
> jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
> PATH_MAX                           4096
> _POSIX_PATH_MAX                    4096
>
>
> I think the issue is with file name not the path.
>
> Secondly the googling which I did reveals that symlink creation can't be stopped. I just wanna confirm that is my findings correct?
>

This looks like something we overlooked in opkg. When we added the caching code
we didn't think about how long the paths and filenames might get during the
rootfs step. It's not currently possible to reduce the length of the symlink
filenames, but it is possible to change the directory in which the symlinks are
created.

Currently the opkg cache dir can only be set in the opkg.conf file. I think we
should add a '--cache-dir' argument to opkg. If this is added you'll be able to
set the following in your local.conf file to change the cache location. Eg. to
use '/tmp/opkg' on the host during rootfs creation.

    OPKG_ARGS = "--cache-dir=/tmp/opkg"

I'll submit a patch to opkg to add this option.

This will only shorten the full path, not the filename length, so I doubt this'll solve it. That said, I can't actually successfully test this today, because cache_dir is made relative to offline_root, so setting such a path as you suggest doesn't shorten the full path either.

Also, did a touch of just the cache filename and it gives the same filename length error, so where the cache dir is really isn't going to matter, it's the filename including the full path to a deep BUILDDIR, and therefore DEPLOY_DIR_IPK, which is the problem.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 8642 bytes --]

Re: opkg 0.3.0 or rootfs task

[Alejandro del Castillo]

On 10/21/2015 02:49 AM, Ahsan, Noor wrote:
> We are hitting this issue on another BSP
>
> file:_var_jenkins_workspace_ce-main_buildhost_SB64_buildtype_industrial_machine_zedboard-zynq7-mel_build_zedboard-zynq7-mel_tmp_deploy_ipk_zedboard_zynq7_mel_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_zedboard_zynq7_mel.ipk
>
> need its solution quickly

If you are in need of a quick workaround, try setting:

option cache_local_files 0

on opkg.conf. This will make a copy of your ipk's instead of creating 
symlinks (haven't try it myself, but that's my understanding).

Also, please open a bug report for opkg on 
https://bugzilla.yoctoproject.org to track the issue.

> *From:*kergoth@gmail.com [mailto:kergoth@gmail.com] *On Behalf Of
> *Christopher Larson
> *Sent:* Tuesday, October 20, 2015 10:20 PM
> *To:* Paul Barker
> *Cc:* Ahsan, Noor; yocto@yoctoproject.org
> *Subject:* Re: [yocto] opkg 0.3.0 or rootfs task
>
> On Tue, Oct 20, 2015 at 10:14 AM, Christopher Larson
> <clarson@kergoth.com <mailto:clarson@kergoth.com>> wrote:
>
>     On Sun, Oct 18, 2015 at 2:57 AM, Paul Barker <paul@paulbarker.me.uk
>     <mailto:paul@paulbarker.me.uk>> wrote:
>
>         On Fri, Oct 16, 2015 at 07:38:19PM +0000, Ahsan, Noor wrote:
>          > On Oct 16, 2015, at 5:47 AM, Ahsan, Noor
>         <Noor_Ahsan@mentor.com
>         <mailto:Noor_Ahsan@mentor.com><mailto:Noor_Ahsan@mentor.com
>         <mailto:Noor_Ahsan@mentor.com>>> wrote:
>          >
>          > Hello,
>          >
>          > I noticed that at the time of rootfs creation symbolic links
>         of the ipk files present in deploy/ipk. The problem what have it
>         while creation of symbolic link it take the full path till that
>         ipk and remove slashes and convert them into underscores. Use
>         that as the name of the symbolic link. This make a very long
>         file names. If we have very long path then name of the symlink
>         exceed from max filename limits. And we get following error
>          >
>          > Collected errors:
>          > * file_link: unable to stat
>         `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk':
>         File name too long.
>          >
>          > Can anyone tell me why the addition of full path was added to
>         the symlink name and can we remove it cause it is cause issues?
>          >
>          > what does
>          >
>          > getconf PATH_MAX /
>          >
>          > show ?
>          >
>          > jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
>          > PATH_MAX                           4096
>          > _POSIX_PATH_MAX                    4096
>          >
>          >
>          > I think the issue is with file name not the path.
>          >
>          > Secondly the googling which I did reveals that symlink
>         creation can't be stopped. I just wanna confirm that is my
>         findings correct?
>          >
>
>         This looks like something we overlooked in opkg. When we added
>         the caching code
>         we didn't think about how long the paths and filenames might get
>         during the
>         rootfs step. It's not currently possible to reduce the length of
>         the symlink
>         filenames, but it is possible to change the directory in which
>         the symlinks are
>         created.
>
>         Currently the opkg cache dir can only be set in the opkg.conf
>         file. I think we
>         should add a '--cache-dir' argument to opkg. If this is added
>         you'll be able to
>         set the following in your local.conf file to change the cache
>         location. Eg. to
>         use '/tmp/opkg' on the host during rootfs creation.
>
>              OPKG_ARGS = "--cache-dir=/tmp/opkg"
>
>         I'll submit a patch to opkg to add this option.
>
>     This will only shorten the full path, not the filename length, so I
>     doubt this'll solve it. That said, I can't actually successfully
>     test this today, because cache_dir is made relative to offline_root,
>     so setting such a path as you suggest doesn't shorten the full path
>     either.
>
>
> Also, did a touch of just the cache filename and it gives the same
> filename length error, so where the cache dir is really isn't going to
> matter, it's the filename including the full path to a deep BUILDDIR,
> and therefore DEPLOY_DIR_IPK, which is the problem.
> --
>
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
>
>
>

Re: opkg 0.3.0 or rootfs task

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 5575 bytes --]

Not likely to help. The problem is the filename length, which will be the
same whether it's a file or a symlink.
On Sat, Oct 24, 2015 at 10:53 AM Alejandro del Castillo <
alejandro.delcastillo@ni.com> wrote:

>
>
> On 10/21/2015 02:49 AM, Ahsan, Noor wrote:
> > We are hitting this issue on another BSP
> >
> >
> file:_var_jenkins_workspace_ce-main_buildhost_SB64_buildtype_industrial_machine_zedboard-zynq7-mel_build_zedboard-zynq7-mel_tmp_deploy_ipk_zedboard_zynq7_mel_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_zedboard_zynq7_mel.ipk
> >
> > need its solution quickly
>
> If you are in need of a quick workaround, try setting:
>
> option cache_local_files 0
>
> on opkg.conf. This will make a copy of your ipk's instead of creating
> symlinks (haven't try it myself, but that's my understanding).
>
> Also, please open a bug report for opkg on
> https://bugzilla.yoctoproject.org to track the issue.
>
> > *From:*kergoth@gmail.com [mailto:kergoth@gmail.com] *On Behalf Of
> > *Christopher Larson
> > *Sent:* Tuesday, October 20, 2015 10:20 PM
> > *To:* Paul Barker
> > *Cc:* Ahsan, Noor; yocto@yoctoproject.org
> > *Subject:* Re: [yocto] opkg 0.3.0 or rootfs task
> >
> > On Tue, Oct 20, 2015 at 10:14 AM, Christopher Larson
> > <clarson@kergoth.com <mailto:clarson@kergoth.com>> wrote:
> >
> >     On Sun, Oct 18, 2015 at 2:57 AM, Paul Barker <paul@paulbarker.me.uk
> >     <mailto:paul@paulbarker.me.uk>> wrote:
> >
> >         On Fri, Oct 16, 2015 at 07:38:19PM +0000, Ahsan, Noor wrote:
> >          > On Oct 16, 2015, at 5:47 AM, Ahsan, Noor
> >         <Noor_Ahsan@mentor.com
> >         <mailto:Noor_Ahsan@mentor.com><mailto:Noor_Ahsan@mentor.com
> >         <mailto:Noor_Ahsan@mentor.com>>> wrote:
> >          >
> >          > Hello,
> >          >
> >          > I noticed that at the time of rootfs creation symbolic links
> >         of the ipk files present in deploy/ipk. The problem what have it
> >         while creation of symbolic link it take the full path till that
> >         ipk and remove slashes and convert them into underscores. Use
> >         that as the name of the symbolic link. This make a very long
> >         file names. If we have very long path then name of the symlink
> >         exceed from max filename limits. And we get following error
> >          >
> >          > Collected errors:
> >          > * file_link: unable to stat
> >
>  `/var/jenkins/workspace/mel_cedar-main/buildhost/amd-ubuntu14-32b/buildtype/iot/machine/mel-dra7xx-evm/build_mel-dra7xx-evm/tmp/work/mel_dra7xx_evm-mel-linux-gnueabi/console-image/1.0-r0/rootfs//var/cache/opkg/volatile/file:_var_jenkins_workspace_mel_cedar-main_buildhost_amd-ubuntu14-32b_buildtype_iot_machine_mel-dra7xx-evm_build_mel-dra7xx-evm_tmp_deploy_ipk_mel_dra7xx_evm_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_mel_dra7xx_evm.ipk':
> >         File name too long.
> >          >
> >          > Can anyone tell me why the addition of full path was added to
> >         the symlink name and can we remove it cause it is cause issues?
> >          >
> >          > what does
> >          >
> >          > getconf PATH_MAX /
> >          >
> >          > show ?
> >          >
> >          > jenkins@amd-ubu14-32-3:~$ getconf -a | grep  PATH_MAX
> >          > PATH_MAX                           4096
> >          > _POSIX_PATH_MAX                    4096
> >          >
> >          >
> >          > I think the issue is with file name not the path.
> >          >
> >          > Secondly the googling which I did reveals that symlink
> >         creation can't be stopped. I just wanna confirm that is my
> >         findings correct?
> >          >
> >
> >         This looks like something we overlooked in opkg. When we added
> >         the caching code
> >         we didn't think about how long the paths and filenames might get
> >         during the
> >         rootfs step. It's not currently possible to reduce the length of
> >         the symlink
> >         filenames, but it is possible to change the directory in which
> >         the symlinks are
> >         created.
> >
> >         Currently the opkg cache dir can only be set in the opkg.conf
> >         file. I think we
> >         should add a '--cache-dir' argument to opkg. If this is added
> >         you'll be able to
> >         set the following in your local.conf file to change the cache
> >         location. Eg. to
> >         use '/tmp/opkg' on the host during rootfs creation.
> >
> >              OPKG_ARGS = "--cache-dir=/tmp/opkg"
> >
> >         I'll submit a patch to opkg to add this option.
> >
> >     This will only shorten the full path, not the filename length, so I
> >     doubt this'll solve it. That said, I can't actually successfully
> >     test this today, because cache_dir is made relative to offline_root,
> >     so setting such a path as you suggest doesn't shorten the full path
> >     either.
> >
> >
> > Also, did a touch of just the cache filename and it gives the same
> > filename length error, so where the cache dir is really isn't going to
> > matter, it's the filename including the full path to a deep BUILDDIR,
> > and therefore DEPLOY_DIR_IPK, which is the problem.
> > --
> >
> > Christopher Larson
> > clarson at kergoth dot com
> > Founder - BitBake, OpenEmbedded, OpenZaurus
> > Maintainer - Tslib
> > Senior Software Engineer, Mentor Graphics
> >
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 7621 bytes --]

Re: opkg 0.3.0 or rootfs task

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 1348 bytes --]

On Sat, Oct 24, 2015 at 06:54:16PM +0000, Christopher Larson wrote:
> Not likely to help. The problem is the filename length, which will be the
> same whether it's a file or a symlink.
> On Sat, Oct 24, 2015 at 10:53 AM Alejandro del Castillo <
> alejandro.delcastillo@ni.com> wrote:
> 
> >
> >
> > On 10/21/2015 02:49 AM, Ahsan, Noor wrote:
> > > We are hitting this issue on another BSP
> > >
> > >
> > file:_var_jenkins_workspace_ce-main_buildhost_SB64_buildtype_industrial_machine_zedboard-zynq7-mel_build_zedboard-zynq7-mel_tmp_deploy_ipk_zedboard_zynq7_mel_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_zedboard_zynq7_mel.ipk
> > >
> > > need its solution quickly
> >
> > If you are in need of a quick workaround, try setting:
> >
> > option cache_local_files 0
> >
> > on opkg.conf. This will make a copy of your ipk's instead of creating
> > symlinks (haven't try it myself, but that's my understanding).
> >
> > Also, please open a bug report for opkg on
> > https://bugzilla.yoctoproject.org to track the issue.
> >

I've just posted patches to the opkg-devel list which should fix this. I had
intended to give them a bit more testing before I sent them but a quick fix is
needed.

Thanks,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: opkg 0.3.0 or rootfs task

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]

On Sat, Oct 24, 2015 at 12:19 PM, Paul Barker <paul@paulbarker.me.uk> wrote:

> On Sat, Oct 24, 2015 at 06:54:16PM +0000, Christopher Larson wrote:
> > Not likely to help. The problem is the filename length, which will be the
> > same whether it's a file or a symlink.
> > On Sat, Oct 24, 2015 at 10:53 AM Alejandro del Castillo <
> > alejandro.delcastillo@ni.com> wrote:
> >
> > >
> > >
> > > On 10/21/2015 02:49 AM, Ahsan, Noor wrote:
> > > > We are hitting this issue on another BSP
> > > >
> > > >
> > >
> file:_var_jenkins_workspace_ce-main_buildhost_SB64_buildtype_industrial_machine_zedboard-zynq7-mel_build_zedboard-zynq7-mel_tmp_deploy_ipk_zedboard_zynq7_mel_kernel-module-lttng-ring-buffer-client-mmap-overwrite_2.6.2+git0+7a88f8b506-r0.0_zedboard_zynq7_mel.ipk
> > > >
> > > > need its solution quickly
> > >
> > > If you are in need of a quick workaround, try setting:
> > >
> > > option cache_local_files 0
> > >
> > > on opkg.conf. This will make a copy of your ipk's instead of creating
> > > symlinks (haven't try it myself, but that's my understanding).
> > >
> > > Also, please open a bug report for opkg on
> > > https://bugzilla.yoctoproject.org to track the issue.
> > >
>
> I've just posted patches to the opkg-devel list which should fix this. I
> had
> intended to give them a bit more testing before I sent them but a quick
> fix is
> needed.


Thanks for your quick work on this, Paul, it's much appreciated. From some
initial testing, builds are completing now. We'll keep testing it out.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2437 bytes --]

Re: opkg 0.3.0 or rootfs task

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On Mon, Oct 26, 2015 at 12:46 AM, Christopher Larson <clarson@kergoth.com>
wrote:

>
>> I've just posted patches to the opkg-devel list which should fix this. I
>> had
>> intended to give them a bit more testing before I sent them but a quick
>> fix is
>> needed.
>
>
> Thanks for your quick work on this, Paul, it's much appreciated. From some
> initial testing, builds are completing now. We'll keep testing it out.



I just hit the same issue. I was pointed out to this thread on IRC today. I
found the patches on the 0.3.x branch in the opkg git tree. Apart from
applying these patches, do i need to do anything in any OE variables to use
the newly added options? I am not entirely sure after reading this thread
and looking at the patches..

thanks

[-- Attachment #2: Type: text/html, Size: 1335 bytes --]

Re: opkg 0.3.0 or rootfs task

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On Mon, Nov 2, 2015 at 8:25 AM, Nicolas Dechesne <
nicolas.dechesne@linaro.org> wrote:

> On Mon, Oct 26, 2015 at 12:46 AM, Christopher Larson <clarson@kergoth.com>
> wrote:
>
>>
>>> I've just posted patches to the opkg-devel list which should fix this. I
>>> had
>>> intended to give them a bit more testing before I sent them but a quick
>>> fix is
>>> needed.
>>
>>
>> Thanks for your quick work on this, Paul, it's much appreciated. From
>> some initial testing, builds are completing now. We'll keep testing it out.
>
>
>
> I just hit the same issue. I was pointed out to this thread on IRC today.
> I found the patches on the 0.3.x branch in the opkg git tree. Apart from
> applying these patches, do i need to do anything in any OE variables to use
> the newly added options? I am not entirely sure after reading this thread
> and looking at the patches..
>

No, there's nothing else you need to do. The patches fromPaul change the
opkg download cache to use checksums rather than the entire url as the
filename, which avoids the file name length problem.

I would not apply the first patch in the series, however, since that's not
related to this. The first patch (memory leak fix) caused double-free
failures in glibc in our testing.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2308 bytes --]

Re: opkg 0.3.0 or rootfs task

[Alejandro del Castillo]

On 11/02/2015 09:31 AM, Christopher Larson wrote:
> 
> On Mon, Nov 2, 2015 at 8:25 AM, Nicolas Dechesne <nicolas.dechesne@linaro.org <mailto:nicolas.dechesne@linaro.org>> wrote:
> 
>     On Mon, Oct 26, 2015 at 12:46 AM, Christopher Larson <clarson@kergoth.com <mailto:clarson@kergoth.com>> wrote:
> 
> 
>             I've just posted patches to the opkg-devel list which should fix this. I had
>             intended to give them a bit more testing before I sent them but a quick fix is
>             needed.
> 
> 
>         Thanks for your quick work on this, Paul, it's much appreciated. From some initial testing, builds are completing now. We'll keep testing it out.
> 
> 
> 
>     I just hit the same issue. I was pointed out to this thread on IRC today. I found the patches on the 0.3.x branch in the opkg git tree. Apart from applying these patches, do i need to do anything in any OE variables to use the newly added options? I am not entirely sure after reading this thread and looking at the patches..
> 
> 
> No, there's nothing else you need to do. The patches fromPaul change the opkg download cache to use checksums rather than the entire url as the filename, which avoids the file name length problem.

The patches are under review on the opkg mailing list: https://groups.google.com/forum/#!topic/opkg-devel/UzDigiuKBcs. I asked Paul for a small modification on one of the patches, once that's done, I will pull them into the opkg repo (and we'll need to update the opkg recipe).

> I would not apply the first patch in the series, however, since that's not related to this. The first patch (memory leak fix) caused double-free failures in glibc in our testing.

yes, the first patch needs to be dropped

-- 
Cheers,

Alejandro

Re: opkg 0.3.0 or rootfs task

[Nicolas Dechesne]

On Mon, Nov 2, 2015 at 4:53 PM, Alejandro del Castillo
<alejandro.delcastillo@ni.com> wrote:
>
> > No, there's nothing else you need to do. The patches fromPaul change the opkg download cache to use checksums rather than the entire url as the filename, which avoids the file name length problem.
>
> The patches are under review on the opkg mailing list: https://groups.google.com/forum/#!topic/opkg-devel/UzDigiuKBcs. I asked Paul for a small modification on one of the patches, once that's done, I will pull them into the opkg repo (and we'll need to update the opkg recipe).
>

thanks. It works, after I pulled these patches:

       file://0001-md5-Add-md5_to_string-function.patch \
       file://0002-sha256-Add-sha256_to_string-function.patch \
       file://0003-opkg_download-Use-md5sum-of-src-URI-as-cache-file-na.patch \

So, basically I took patches 2,3 and 4 from the series.

Any chance to get these patches in jethro? Or more generally what's
the plan to get them merged?

thanks a lot!

nico