Shared DL_DIR with other users

Shared DL_DIR with other users

[Vanveerdeghem, Simon]

[-- Attachment #1: Type: text/plain, Size: 2110 bytes --]

Hi,

 

We are using Yocto with a couple of users, to prevent that our build
server has duplicates of downloads, we share the DL_DIR.

But we have problems with permissions. When a user run yocto, it will
checkout sources, or download tar files with permissions 644 with the
username of the user who is running yocto. 

When another user will do a cleanall of a bitbake recipe, yocto will
remove the tarball and the svn workingcopy, but because the permissions
aren't 777 It cannot remove all the files (yocto doesn't throw an error
here).

When that other user will do a fetch of thqt recipe, yocto will checkout
the svn directory, it will leave as good as clean because it says
internally "Skipped ." (yocto doesn't throw an error here) then it wil
take a tarball of it, again the tarball has only a directory named of
the checked out module, but have no sources. (yocto doesn't throw an
error here)

When yocto will building the sources it will fail because the tar was
clean, and there is nothing to build.

 

The rootcause of this problem is that the shared download directory only
use 644. (We can workaround this with the .profile (we can set in the
.profile a standard permission mask, but then you always use that mask,
something we don't want)

 

 

Someone already problems with this?

 

 



DISCLAIMER:
Unless indicated otherwise, the information contained in this message is privileged and confidential, and is intended only for the use of the addressee(s) named above and others who have been specifically authorized to receive it. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message and/or attachments is strictly prohibited. The company accepts no liability for any damage caused by any virus transmitted by this email. Furthermore, the company does not warrant a proper and complete transmission of this information, nor does it accept liability for any delays. If you have received this message in error, please contact the sender and delete the message. Thank you.

[-- Attachment #2: Type: text/html, Size: 4110 bytes --]

Re: Shared DL_DIR with other users

[Paul Eggleton]

Hi Simon,

On Tuesday 26 March 2013 14:37:36 Vanveerdeghem, Simon wrote:
> We are using Yocto with a couple of users, to prevent that our build
> server has duplicates of downloads, we share the DL_DIR.
> 
> But we have problems with permissions. When a user run yocto, it will
> checkout sources, or download tar files with permissions 644 with the
> username of the user who is running yocto.
> 
> When another user will do a cleanall of a bitbake recipe, yocto will
> remove the tarball and the svn workingcopy, but because the permissions
> aren't 777 It cannot remove all the files (yocto doesn't throw an error
> here).
> 
> When that other user will do a fetch of thqt recipe, yocto will checkout
> the svn directory, it will leave as good as clean because it says
> internally "Skipped ." (yocto doesn't throw an error here) then it wil
> take a tarball of it, again the tarball has only a directory named of
> the checked out module, but have no sources. (yocto doesn't throw an
> error here)
> 
> When yocto will building the sources it will fail because the tar was
> clean, and there is nothing to build.
> 
> The rootcause of this problem is that the shared download directory only
> use 644. (We can workaround this with the .profile (we can set in the
> .profile a standard permission mask, but then you always use that mask,
> something we don't want)

It's likely that sharing DL_DIR directly between multiple users is not a
heavily tested scenario. One way around it which is fairly widely used 
though is to set up a local mirror shared by all of the build machines
and keep DL_DIR separate on each machine. There is some information on
how to do that here:

https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Shared DL_DIR with other users

[Mark Hatle]

On 3/26/13 9:25 AM, Paul Eggleton wrote:
> Hi Simon,
>
> On Tuesday 26 March 2013 14:37:36 Vanveerdeghem, Simon wrote:
>> We are using Yocto with a couple of users, to prevent that our build
>> server has duplicates of downloads, we share the DL_DIR.
>>
>> But we have problems with permissions. When a user run yocto, it will
>> checkout sources, or download tar files with permissions 644 with the
>> username of the user who is running yocto.
>>
>> When another user will do a cleanall of a bitbake recipe, yocto will
>> remove the tarball and the svn workingcopy, but because the permissions
>> aren't 777 It cannot remove all the files (yocto doesn't throw an error
>> here).
>>
>> When that other user will do a fetch of thqt recipe, yocto will checkout
>> the svn directory, it will leave as good as clean because it says
>> internally "Skipped ." (yocto doesn't throw an error here) then it wil
>> take a tarball of it, again the tarball has only a directory named of
>> the checked out module, but have no sources. (yocto doesn't throw an
>> error here)
>>
>> When yocto will building the sources it will fail because the tar was
>> clean, and there is nothing to build.
>>
>> The rootcause of this problem is that the shared download directory only
>> use 644. (We can workaround this with the .profile (we can set in the
>> .profile a standard permission mask, but then you always use that mask,
>> something we don't want)
>
> It's likely that sharing DL_DIR directly between multiple users is not a
> heavily tested scenario. One way around it which is fairly widely used
> though is to set up a local mirror shared by all of the build machines
> and keep DL_DIR separate on each machine. There is some information on
> how to do that here:
>
> https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F

This is how we do it.  We have one user who routinely does a fetchall against 
the trees and populates the mirror.  Then the users have a layer which adds:

PREMIRRORS_append = "\
      git://.*/.* file://path/ \n \
      svn://.*/.* file://path/ \n \
      ftp://.*/.* file://path/ \n \
      http://.*/.* file://path/ \n \
      https://.*/.* file://path/ \n \
"

--Mark

> Cheers,
> Paul
>