can not build yocto on NFS mounted NAS

can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

I tried to build to build bitbake core-image-minimal-dev on some nfs
mounted NAS.

Pseudo is not present but is required, building this first before the
main build
Parsing recipes: 100%
|###################################################################################################################################|
Time: 00:00:20
Parsing of 819 .bb files complete (0 cached, 819 parsed). 1126 targets,
47 skipped, 0 masked, 0 errors.

Build Configuration:
BB_VERSION        = "1.18.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-12.04"
TARGET_SYS        = "arm-poky-linux-gnueabi"
MACHINE           = "beagle-xm-ml"
DISTRO            = "poky"
DISTRO_VERSION    = "1.4.1"
TUNE_FEATURES     = "armv7a vfp neon"
TARGET_FPU        = "vfp-neon"
meta
meta-yocto
meta-yocto-bsp    =
"beagleboard_dylan-9.0.1_LOCAL:73f103bf9b2cdf985464dc53bf4f1cfd71d4531f"

It fails pretty soon with:

ERROR: Error executing a python function in
/yoctotmp/manual-build/poky/meta/recipes-devtools/quilt/quilt-native_0.60.bb:
CalledProcessError: Command 'cp -al
/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beagle-xm/tmp/sysroots/x86_64-linux/*
/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-sysroot/x86_64-linux'
returned non-zero
 exit status 1 with output cp: cannot create hard link
`/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sstate-build-populate-sysroot/x86
_64-linux/usr/share/quilt/compat/awk' to
`/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beag
le-xm/tmp/sysroots/x86_64-linux/usr/share/quilt/compat/awk': Unknown
error 524

A bit of further digging reveals that underneath
/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beagle-xm/tmp/sysroots/x86_64-linux/*
we have a symbolic link to
/yoctotmp/manual-build/poky/beagle-xm/tmp/work/x86_64-linux/quilt-native/0.60-r0/sysroot-destdir/yoctotmp/manual-build/poky/beagle-xm/tmp/sysroots/x86_64-linux/usr/share/quilt/compat/awk
-> /usr/bin/gawk

Since hard links can not be made between different file systems (local
ext4 and nfs) the cp -al fails.

Is it like this by design, or should I file a bug in bugzilla?

Previous experiments (only SSTATE and DL from NFS/NAS) seem to work and
I guess that's also your autobuilder cluster setup.

Regards,

Robert
...Under a government which imprisons any unjustly, the true place for a
just man is also a prison. -- Henry David Thoreau

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On Thu, Aug 8, 2013 at 6:57 AM, Robert Berger <
gmane@reliableembeddedsystems.com> wrote:

> Is it like this by design, or should I file a bug in bugzilla?
>
> Previous experiments (only SSTATE and DL from NFS/NAS) seem to work and
> I guess that's also your autobuilder cluster setup.
>

TMPDIR on NFS has never been supported with oe/yocto, and I doubt it will
be at any time in the near future, but if that's changed, I'm sure someone
will jump into the thread and mention it.

As you say, SSTATE_DIR and DL_DIR work fine there.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1147 bytes --]

Re: can not build yocto on NFS mounted NAS

[Gary Thomas]

On 2013-08-08 08:03, Chris Larson wrote:
> On Thu, Aug 8, 2013 at 6:57 AM, Robert Berger <gmane@reliableembeddedsystems.com <mailto:gmane@reliableembeddedsystems.com>> wrote:
>
>     Is it like this by design, or should I file a bug in bugzilla?
>
>     Previous experiments (only SSTATE and DL from NFS/NAS) seem to work and
>     I guess that's also your autobuilder cluster setup.
>
>
> TMPDIR on NFS has never been supported with oe/yocto, and I doubt it will be at any time in the near future, but if that's changed, I'm sure someone will jump into the thread and
> mention it.
>
> As you say, SSTATE_DIR and DL_DIR work fine there.

Interesting.  I'm building now - OEROOT on a local ext3 file system,
build/tmp on NFS - and it's working just fine.  My host is Fedora (15).

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

On 08/08/2013 05:53 PM, Gary Thomas wrote:
> 
> Interesting.  I'm building now - OEROOT on a local ext3 file system,
> build/tmp on NFS - and it's working just fine.  My host is Fedora (15).
> 

/poky
  /beagle-xm
    /cache
    /conf
    /tmp    <--- this /tmp?

How you do this?

Do you use some magic build setting for this or a symlink to the nfs?

Regards,

Robert
..."It is practically impossible to teach good programming style to
students that have had prior exposure to BASIC; as potential programmers
they are mentally mutilated beyond hope of regeneration." - Edsger Dijkstra

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[Gary Thomas]

On 2013-08-08 09:09, Robert Berger wrote:
> Hi,
>
> On 08/08/2013 05:53 PM, Gary Thomas wrote:
>>
>> Interesting.  I'm building now - OEROOT on a local ext3 file system,
>> build/tmp on NFS - and it's working just fine.  My host is Fedora (15).
>>
>
> /poky
>    /beagle-xm
>      /cache
>      /conf
>      /tmp    <--- this /tmp?
>
> How you do this?
>
> Do you use some magic build setting for this or a symlink to the nfs?


I used the standard setup script, like this:
   % . /local/poky/oe-init-build-env /MY_NFS_VOLUME/build_dir

This will create a build tree on MY_NFS_VOLUME named 'build_dir'.  Then
I adjusted /MY_NFS_VOLUME/build_dir/conf/local.conf as needed and ran
bitbake...

Note: I NEVER build inside of my Poky/Yocto tree - always in a separate
build directory and quite often on a separate file system.

(sorry for any duplicate messages, I forgot to follow my own advice
and CC the list!)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

On 08/08/2013 06:23 PM, Gary Thomas wrote:
>>
>> How you do this?
>>
>> Do you use some magic build setting for this or a symlink to the nfs?
> 
> 
> I used the standard setup script, like this:
>   % . /local/poky/oe-init-build-env /MY_NFS_VOLUME/build_dir
> 
> This will create a build tree on MY_NFS_VOLUME named 'build_dir'.  Then
> I adjusted /MY_NFS_VOLUME/build_dir/conf/local.conf as needed and ran
> bitbake...
> 
> Note: I NEVER build inside of my Poky/Yocto tree - always in a separate
> build directory and quite often on a separate file system.
> 

I just tried this and with Ubuntu 12.04.2 LTS it does __NOT__ work (as
expected). I'm really surprised that you can have
/MY_NFS_VOLUME/build_dir due to the fact that there are hard links
required between the /MY_NFS_VOLUME/build_dir and where your host Linux
rootfs resides.

So either we have a different NFS setup or cp -al behaves differently.

Do you happen to have an Ubuntu build machine to test? I'm currently on
the road so it's a bit tricky for me to try with Fedora.

Regards,

Robert



..."As far as I know we never had an undetected error." - Anonymous

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[Gary Thomas]

On 2013-08-08 11:29, Robert Berger wrote:
> Hi,
>
> On 08/08/2013 06:23 PM, Gary Thomas wrote:
>>>
>>> How you do this?
>>>
>>> Do you use some magic build setting for this or a symlink to the nfs?
>>
>>
>> I used the standard setup script, like this:
>>    % . /local/poky/oe-init-build-env /MY_NFS_VOLUME/build_dir
>>
>> This will create a build tree on MY_NFS_VOLUME named 'build_dir'.  Then
>> I adjusted /MY_NFS_VOLUME/build_dir/conf/local.conf as needed and ran
>> bitbake...
>>
>> Note: I NEVER build inside of my Poky/Yocto tree - always in a separate
>> build directory and quite often on a separate file system.
>>
>
> I just tried this and with Ubuntu 12.04.2 LTS it does __NOT__ work (as
> expected). I'm really surprised that you can have
> /MY_NFS_VOLUME/build_dir due to the fact that there are hard links
> required between the /MY_NFS_VOLUME/build_dir and where your host Linux
> rootfs resides.
>
> So either we have a different NFS setup or cp -al behaves differently.
>
> Do you happen to have an Ubuntu build machine to test? I'm currently on
> the road so it's a bit tricky for me to try with Fedora.

I'm trying this now with my Ubuntu 12.04 laptop and the same NFS server.
Looks like it's working just fine, it's already made it through the first
phase of a fresh build (building all the necessary 'native' tools), so I
think it will work as expected (it's way past where you had problems).
I'll let it complete and report back if there were any problems.

n.b. building this way is glacially slow :-(

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

On 08/08/2013 08:52 PM, Gary Thomas wrote:
> 
> I'm trying this now with my Ubuntu 12.04 laptop and the same NFS server.
> Looks like it's working just fine, it's already made it through the first
> phase of a fresh build (building all the necessary 'native' tools), so I
> think it will work as expected (it's way past where you had problems).
> I'll let it complete and report back if there were any problems.
> 
> n.b. building this way is glacially slow :-(
> 

So let's assume it's the nfs server. Can you please tell me what's the
file system on your nfs server and how you export it?

A possible problem is, that with nfs symbolic links are not followed by
default.

My problem might be that I use a FreeBSD based nfs server.

Maybe with something like ext3, ext4 it might work.

Also do you use dylan-9.0.1 or something more recent?

Regards,

Robert
..."Premature optimisation is the root of all evil" -- Knuth

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[Gary Thomas]

On 2013-08-08 14:56, Robert Berger wrote:
> Hi,
>
> On 08/08/2013 08:52 PM, Gary Thomas wrote:
>>
>> I'm trying this now with my Ubuntu 12.04 laptop and the same NFS server.
>> Looks like it's working just fine, it's already made it through the first
>> phase of a fresh build (building all the necessary 'native' tools), so I
>> think it will work as expected (it's way past where you had problems).
>> I'll let it complete and report back if there were any problems.
>>
>> n.b. building this way is glacially slow :-(
>>
>
> So let's assume it's the nfs server. Can you please tell me what's the
> file system on your nfs server and how you export it?
>
> A possible problem is, that with nfs symbolic links are not followed by
> default.
>
> My problem might be that I use a FreeBSD based nfs server.
>
> Maybe with something like ext3, ext4 it might work.

My NFS server is on a Fedora 16 system.  The exported disk is EXT3 file
system, using these exports:
   /shared *(sync,insecure,rw,no_root_squash)

>
> Also do you use dylan-9.0.1 or something more recent?

I always work from master, but I've been doing this (and running similar
scenarios) since at least "pinky"

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

Just for the record I was able to remove the hard link with a small
patch and now things look good (so far) - although it will consume more
disk space.

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index faa0f61..3225993 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -90,7 +90,7 @@ def copyhardlinktree(src, dst):
         if not len(os.listdir(src)):
             return
         src = src + "/*"
-    cmd = 'cp -al %s %s' % (src, dst)
+    cmd = 'cp -a %s %s' % (src, dst)
     check_output(cmd, shell=True, stderr=subprocess.STDOUT)

 def remove(path, recurse=True):

Regards,

Robert
..."But I have a slowly coagulating theory that the size of a project is
directly proportional to the possibility that significant bugs will crop
up. Exponentiate for each additional programmer involved." - Steven K.
Halliburton

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[lothar]

Am 2013-08-18 22:44, schrieb Robert Berger:
> Hi,
> 
> Just for the record I was able to remove the hard link with a small
> patch and now things look good (so far) - although it will consume more
> disk space.
> 
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index faa0f61..3225993 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -90,7 +90,7 @@ def copyhardlinktree(src, dst):
>          if not len(os.listdir(src)):
>              return
>          src = src + "/*"
> -    cmd = 'cp -al %s %s' % (src, dst)
> +    cmd = 'cp -a %s %s' % (src, dst)
>      check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> 
>  def remove(path, recurse=True):
> 
> Regards,
> 
> Robert
> ..."But I have a slowly coagulating theory that the size of a project 
> is
> directly proportional to the possibility that significant bugs will 
> crop
> up. Exponentiate for each additional programmer involved." - Steven K.
> Halliburton
> 
> My public pgp key is available,at:
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
> 
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Hi,
  I have just two points here to ask:

1) Wouldn't simply using SSTATE_MIRRORS be a better solution here, 
instead?

2) Patch: wouldn't it be nicer to try the cp -al, and catch the 
CalledProcessError Exception, if it is thrown, and then run the brute 
force cp -a? Perhaps even as a general approach?

[I didn't test it either practically, just, sometimes one may get the 
impression in yocto / bitbake, that a bit more exception handling and 
directly related error messages already are one of the worlds most 
expensive things to do.. pls, improve this situation!! ;) ]

BR,
Lothar

Re: can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

On 08/20/2013 07:45 PM, lothar@denx.de wrote:
> 
> Hi,
>  I have just two points here to ask:
> 
> 1) Wouldn't simply using SSTATE_MIRRORS be a better solution here, instead?

It's not the same.

SSTATE and DL_DIR are on an nfs export anyhow and this works.

The problem here is with TMP dir (and an nfs server with a funny
underlying filesystem like UFS) and hard links.

> 
> 2) Patch: wouldn't it be nicer to try the cp -al, and catch the
> CalledProcessError Exception, if it is thrown, and then run the brute
> force cp -a? Perhaps even as a general approach?

In theory yes, in practice you will find out that if you build something
like core-image-sato-sdk "cp -al" is not the only place which creates
hard links.

Regards,

Robert
..."There's an infinite number of ways to make a simple problem seem
difficult; only a handful to make a difficult problem seem simple" -
Jack Crenshaw's Law #1

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[Robert Berger]

Hi,

Gary is, as usual, right ;)

Now I have the following test setup:

nfs server:

Ubuntu 12.04.2 LTS server

<export_dir> *(sync,insecure,rw,no_root_squash)

client:

<nfs_server>:<export_dir> <mount_point>  nfs
auto,noatime,nolock,intr,tcp,actimeo=1800 0 0

mkdir -p <mount_point>/${HOSTNAME}/manual-build/

source oe-init-build-env <mount_point>/${HOSTNAME}/manual-build/<board>

I build there happily (no patch to remove hard links):

bitbake core-image-minimal-dev
bitbake core-image-sato-sdk

Regards,

Robert
..."It is practically impossible to teach good programming style to
students that have had prior exposure to BASIC; as potential programmers
they are mentally mutilated beyond hope of regeneration." - Edsger Dijkstra

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: can not build yocto on NFS mounted NAS

[lothar]

Am 2013-08-21 07:58, schrieb Robert Berger:
> Hi,
> 
> On 08/20/2013 07:45 PM, lothar@denx.de wrote:
>> 
>> Hi,
>>  I have just two points here to ask:
>> 
>> 1) Wouldn't simply using SSTATE_MIRRORS be a better solution here, 
>> instead?
> 
> It's not the same.
> 
> SSTATE and DL_DIR are on an nfs export anyhow and this works.
> 
> The problem here is with TMP dir (and an nfs server with a funny
> underlying filesystem like UFS) and hard links.
> 
>> 
>> 2) Patch: wouldn't it be nicer to try the cp -al, and catch the
>> CalledProcessError Exception, if it is thrown, and then run the brute
>> force cp -a? Perhaps even as a general approach?
> 
> In theory yes, in practice you will find out that if you build 
> something
> like core-image-sato-sdk "cp -al" is not the only place which creates
> hard links.
> 
> Regards,
> 
> Robert
> ..."There's an infinite number of ways to make a simple problem seem
> difficult; only a handful to make a difficult problem seem simple" -
> Jack Crenshaw's Law #1
> 
> My public pgp key is available,at:
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1
> 
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Hi Robert,

Thank you for the answer. About my second point, I think you 
missunderstood, or I do not get your answer correctly, respectively.

In brief, I mean:
TRY.... CATCH... EXCEPTION

In detail:
The issue you discovered here, IMHO is definitely a real problem, when 
it comes to do hardlinks across filesystems.

Grepping for 'cp -al', though, to me only reviels this single place 
where hardlinks are done (do you know more?), which I showed you in the 
chat we had apart and for which you posted the fix here in the ML. To my 
understanding, it is only used by the sstate .bbclass, to refer to the 
build products after recipe execution. Means, say, using for instance a 
central SSTATE_DIR folder, you may remove the 'tmp' folder in a 
particular build folder (referring to the SSTATE_DIR), and still be able 
to build in your other build folders (having set SSTATE_DIR) relatively 
fast, since the build products were referenced in SSTATE_DIR by 
hardlinks. This is done by this 'cp -al', which is applied to every 
single recipe's build, means it is pretty central!

Thus, simply exchanging 'cp -al' by 'cp -a' would affect all recipes' 
execution. But running 'cp -al' and extending it by a fallback to 'cp 
-a' using the already thrown EXCEPTION, does not change the original 
behavior, but extends it. Perhaps this remark seems a bit hairsplitting. 
Ok, but, not handling any exception here, IMHO seems to be wrong, too, 
since it will already be thrown. And if so, it should be fixed, or not? 
This was my point about. Keeping in mind that if both approaches fail, 
there could be at least e.g. printstacktrace() like output..
AFAIR, the git fetcher uses a similar fallback approach for handling 
unaccessable repositories.

Just some remarks, maybe I'm wrong.. but, also currently messing a bit 
with sstate.
BR,
Lothar Rubusch