Write-protect $DL_DIR

Write-protect $DL_DIR

[Gary Thomas]

In my quest for a well-packaged Poky that I can distribute
to my customers, I'd like to give them a pre-populated $DL_DIR
tree (also in response to those customers that have little or
no net access).  This would likely be in /opt/XYZ and practice
would dictate that it be read-only.

Sadly, this doesn't work.  When I try to build Poky with a
read-only $DL_DIR, it simply hangs on every 'fetch' step.

Note: I could avoid this problem (although I think it's a
bug even if it should never be experienced) by being able
to easily provide source mirrors for the packages.  So far,
I've not figured out how to do this; I'd like to add my
own mirror lists, similar to those in meta/classes/poky.bbclass
Would this be a way forward?  If so, how?

Thanks for any ideas

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Write-protect $DL_DIR

[Eric Bénard]

Hi Gary,

On 21/12/2010 16:45, Gary Thomas wrote:
> In my quest for a well-packaged Poky that I can distribute
> to my customers, I'd like to give them a pre-populated $DL_DIR
> tree (also in response to those customers that have little or
> no net access). This would likely be in /opt/XYZ and practice
> would dictate that it be read-only.
>
> Sadly, this doesn't work. When I try to build Poky with a
> read-only $DL_DIR, it simply hangs on every 'fetch' step.
>
> Note: I could avoid this problem (although I think it's a
> bug even if it should never be experienced) by being able
> to easily provide source mirrors for the packages. So far,
> I've not figured out how to do this; I'd like to add my
> own mirror lists, similar to those in meta/classes/poky.bbclass
> Would this be a way forward? If so, how?
>
maybe something like own-mirrors in OE could be interesting :
http://cgit.openembedded.net/cgit.cgi/openembedded/tree/classes/own-mirrors.bbclass

Eric

Re: Write-protect $DL_DIR

[Scott Garman]

On 12/21/2010 07:45 AM, Gary Thomas wrote:
> In my quest for a well-packaged Poky that I can distribute
> to my customers, I'd like to give them a pre-populated $DL_DIR
> tree (also in response to those customers that have little or
> no net access).  This would likely be in /opt/XYZ and practice
> would dictate that it be read-only.
>
> Sadly, this doesn't work.  When I try to build Poky with a
> read-only $DL_DIR, it simply hangs on every 'fetch' step.
>
> Note: I could avoid this problem (although I think it's a
> bug even if it should never be experienced) by being able
> to easily provide source mirrors for the packages.  So far,
> I've not figured out how to do this; I'd like to add my
> own mirror lists, similar to those in meta/classes/poky.bbclass
> Would this be a way forward?  If so, how?
>
> Thanks for any ideas

Hi Gary,

Being able to do this is an important priority. Please file a bug to 
track it if there isn't a current workaround.

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: Write-protect $DL_DIR

[Richard Purdie]

On Tue, 2010-12-21 at 08:45 -0700, Gary Thomas wrote:
> In my quest for a well-packaged Poky that I can distribute
> to my customers, I'd like to give them a pre-populated $DL_DIR
> tree (also in response to those customers that have little or
> no net access).  This would likely be in /opt/XYZ and practice
> would dictate that it be read-only.
> 
> Sadly, this doesn't work.  When I try to build Poky with a
> read-only $DL_DIR, it simply hangs on every 'fetch' step.
> 
> Note: I could avoid this problem (although I think it's a
> bug even if it should never be experienced) by being able
> to easily provide source mirrors for the packages.  So far,
> I've not figured out how to do this; I'd like to add my
> own mirror lists, similar to those in meta/classes/poky.bbclass
> Would this be a way forward?  If so, how?
> 
> Thanks for any ideas

Make the readonly directory a file:// source in PREMIRRORS. You can use
the SSTATE_MIRRORS syntax in local.conf to do this without a class file.

It won't actually copy them to DL_DIR if they're already local.

Cheers,

Richard

Re: Write-protect $DL_DIR

[Gary Thomas]

On 12/21/2010 09:44 AM, Richard Purdie wrote:
> On Tue, 2010-12-21 at 08:45 -0700, Gary Thomas wrote:
>> In my quest for a well-packaged Poky that I can distribute
>> to my customers, I'd like to give them a pre-populated $DL_DIR
>> tree (also in response to those customers that have little or
>> no net access).  This would likely be in /opt/XYZ and practice
>> would dictate that it be read-only.
>>
>> Sadly, this doesn't work.  When I try to build Poky with a
>> read-only $DL_DIR, it simply hangs on every 'fetch' step.
>>
>> Note: I could avoid this problem (although I think it's a
>> bug even if it should never be experienced) by being able
>> to easily provide source mirrors for the packages.  So far,
>> I've not figured out how to do this; I'd like to add my
>> own mirror lists, similar to those in meta/classes/poky.bbclass
>> Would this be a way forward?  If so, how?
>>
>> Thanks for any ideas
>
> Make the readonly directory a file:// source in PREMIRRORS. You can use
> the SSTATE_MIRRORS syntax in local.conf to do this without a class file.
>
> It won't actually copy them to DL_DIR if they're already local.

This sounds good.  Do you have an example of what to put into
local.conf?  I'm not sure what you mean by using the SSTATE_MIRRORS
syntax above.

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Write-protect $DL_DIR

[Richard Purdie]

On Tue, 2010-12-21 at 09:52 -0700, Gary Thomas wrote:
> On 12/21/2010 09:44 AM, Richard Purdie wrote:
> > Make the readonly directory a file:// source in PREMIRRORS. You can use
> > the SSTATE_MIRRORS syntax in local.conf to do this without a class file.
> >
> > It won't actually copy them to DL_DIR if they're already local.
> 
> This sounds good.  Do you have an example of what to put into
> local.conf?  I'm not sure what you mean by using the SSTATE_MIRRORS
> syntax above.

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=642dd32920f62e991e4ca629ed304b95b37ce1ef

Cheers,

Richard

Re: Write-protect $DL_DIR

[Gary Thomas]

On 12/21/2010 09:39 AM, Scott Garman wrote:
> On 12/21/2010 07:45 AM, Gary Thomas wrote:
>> In my quest for a well-packaged Poky that I can distribute
>> to my customers, I'd like to give them a pre-populated $DL_DIR
>> tree (also in response to those customers that have little or
>> no net access). This would likely be in /opt/XYZ and practice
>> would dictate that it be read-only.
>>
>> Sadly, this doesn't work. When I try to build Poky with a
>> read-only $DL_DIR, it simply hangs on every 'fetch' step.
>>
>> Note: I could avoid this problem (although I think it's a
>> bug even if it should never be experienced) by being able
>> to easily provide source mirrors for the packages. So far,
>> I've not figured out how to do this; I'd like to add my
>> own mirror lists, similar to those in meta/classes/poky.bbclass
>> Would this be a way forward? If so, how?
>>
>> Thanks for any ideas
>
> Hi Gary,
>
> Being able to do this is an important priority. Please file a bug to track it if there isn't a current workaround.

Done, bug #606.

I'll also look at Richard's new support for MIRRORS/PREMIRRORS
to see how to make it work for my environment.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Write-protect $DL_DIR

[Richard Purdie]

On Tue, 2010-12-21 at 11:56 -0700, Gary Thomas wrote:
> On 12/21/2010 09:39 AM, Scott Garman wrote:
> > On 12/21/2010 07:45 AM, Gary Thomas wrote:
> >> In my quest for a well-packaged Poky that I can distribute
> >> to my customers, I'd like to give them a pre-populated $DL_DIR
> >> tree (also in response to those customers that have little or
> >> no net access). This would likely be in /opt/XYZ and practice
> >> would dictate that it be read-only.
> >>
> >> Sadly, this doesn't work. When I try to build Poky with a
> >> read-only $DL_DIR, it simply hangs on every 'fetch' step.
> >>
> >> Note: I could avoid this problem (although I think it's a
> >> bug even if it should never be experienced) by being able
> >> to easily provide source mirrors for the packages. So far,
> >> I've not figured out how to do this; I'd like to add my
> >> own mirror lists, similar to those in meta/classes/poky.bbclass
> >> Would this be a way forward? If so, how?
> >>
> >> Thanks for any ideas
> >
> > Hi Gary,
> >
> > Being able to do this is an important priority. Please file a bug to
> track it if there isn't a current workaround.
> 
> Done, bug #606.

Sorry but this isn't what is being asked and I don't think that bug is
reasonable. DL_DIR needs to be writeable. You can implement a readonly
mirror just fine without requiring that so we have what I think is a
very reasonable way to solve the problem.

> I'll also look at Richard's new support for MIRRORS/PREMIRRORS
> to see how to make it work for my environment.

Please do this and report back on how you get one. I'm fine with
documenting this and making things clearer. Turning DL_DIR into some new
mirror and having a new writable area for state information isn't the
right way to solve this problem though when we have mirror handling for
exactly this kind of use case already.

Cheers,

Richard

Re: Write-protect $DL_DIR

[Gary Thomas]

On 12/21/2010 02:27 PM, Richard Purdie wrote:
> On Tue, 2010-12-21 at 11:56 -0700, Gary Thomas wrote:
>> On 12/21/2010 09:39 AM, Scott Garman wrote:
>>> On 12/21/2010 07:45 AM, Gary Thomas wrote:
>>>> In my quest for a well-packaged Poky that I can distribute
>>>> to my customers, I'd like to give them a pre-populated $DL_DIR
>>>> tree (also in response to those customers that have little or
>>>> no net access). This would likely be in /opt/XYZ and practice
>>>> would dictate that it be read-only.
>>>>
>>>> Sadly, this doesn't work. When I try to build Poky with a
>>>> read-only $DL_DIR, it simply hangs on every 'fetch' step.
>>>>
>>>> Note: I could avoid this problem (although I think it's a
>>>> bug even if it should never be experienced) by being able
>>>> to easily provide source mirrors for the packages. So far,
>>>> I've not figured out how to do this; I'd like to add my
>>>> own mirror lists, similar to those in meta/classes/poky.bbclass
>>>> Would this be a way forward? If so, how?
>>>>
>>>> Thanks for any ideas
>>>
>>> Hi Gary,
>>>
>>> Being able to do this is an important priority. Please file a bug to
>> track it if there isn't a current workaround.
>>
>> Done, bug #606.
>
> Sorry but this isn't what is being asked and I don't think that bug is
> reasonable. DL_DIR needs to be writeable. You can implement a readonly
> mirror just fine without requiring that so we have what I think is a
> very reasonable way to solve the problem.

>> I'll also look at Richard's new support for MIRRORS/PREMIRRORS
>> to see how to make it work for my environment.
>
> Please do this and report back on how you get one. I'm fine with
> documenting this and making things clearer. Turning DL_DIR into some new
> mirror and having a new writable area for state information isn't the
> right way to solve this problem though when we have mirror handling for
> exactly this kind of use case already.

I understand that these are really [now] two separate things.
I had been trying to use $DL_DIR as a pure mirror, but your change
of today will make that no longer necessary.

I went ahead and filed the bug about a read-only $DL_DIR because
it causes the build to hang/build silently which could cause confusion.
The build should either fail at fetch time with a message or be
sanity checked.

I've tried your changes and they solve my problem well, thanks.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Write-protect $DL_DIR

[Gary Thomas]

On 12/21/2010 03:34 PM, Gary Thomas wrote:
  ... snip
>
> I've tried your changes and they solve my problem well, thanks.
>

I have had a problem with this (after more testing).  Originally,
I had DL_DIR="/work/misc/Poky/sources/", using it somewhat like
a mirror.  With your changes yesterday, I added these lines to
my distro.conf:

   # Provide pre-staged sources
   PREMIRRORS_append = "\
   bzr://.*/.*   file:///work/misc/Poky/sources/ \n \
   cvs://.*/.*   file:///work/misc/Poky/sources/ \n \
   git://.*/.*   file:///work/misc/Poky/sources/ \n \
   hg://.*/.*    file:///work/misc/Poky/sources/ \n \
   osc://.*/.*   file:///work/misc/Poky/sources/ \n \
   p4://.*/.*    file:///work/misc/Poky/sources/ \n \
   svk://.*/.*   file:///work/misc/Poky/sources/ \n \
   svn://.*/.*   file:///work/misc/Poky/sources/ \n"

   MIRRORS_append = "\
   ftp://.*/.*      file:///work/misc/Poky/sources/ \n \
   http://.*/.*     file:///work/misc/Poky/sources/ \n \
   https://.*/.*    file:///work/misc/Poky/sources/ \n"

   require conf/distro/include/my-fixed-revisions.inc
   require conf/distro/poky.conf

I also disabled DL_DIR in local.conf, falling back to using
"downloads" in my build tree (/work/local/new_p60).

As you can see, my distro uses Poky as its base, just overriding
and/or adding some features for our environment.

With this setup, the build for gconf-dbus_svn fails in the
unpack stage, with this error:
   NOTE: Unpacking /work/local/new_p60/downloads/trunk_developer.imendio.com_.svn.gconf-dbus_705_.tar.gz to 
/work/local/new_p60/tmp/work/armv7a-poky-linux-gnueabi/gconf-dbus-2.16.0+svnr705-r0/
   tar: /work/local/new_p60/downloads/trunk_developer.imendio.com_.svn.gconf-dbus_705_.tar.gz: Cannot open: No such file or directory

Did I set this up incorrectly?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Write-protect $DL_DIR

[Gary Thomas]

On 12/22/2010 07:51 AM, Gary Thomas wrote:
> On 12/21/2010 03:34 PM, Gary Thomas wrote:
> ... snip
>>
>> I've tried your changes and they solve my problem well, thanks.
>>
>
> I have had a problem with this (after more testing). Originally,
> I had DL_DIR="/work/misc/Poky/sources/", using it somewhat like
> a mirror. With your changes yesterday, I added these lines to
> my distro.conf:
>
> # Provide pre-staged sources
> PREMIRRORS_append = "\
> bzr://.*/.* file:///work/misc/Poky/sources/ \n \
> cvs://.*/.* file:///work/misc/Poky/sources/ \n \
> git://.*/.* file:///work/misc/Poky/sources/ \n \
> hg://.*/.* file:///work/misc/Poky/sources/ \n \
> osc://.*/.* file:///work/misc/Poky/sources/ \n \
> p4://.*/.* file:///work/misc/Poky/sources/ \n \
> svk://.*/.* file:///work/misc/Poky/sources/ \n \
> svn://.*/.* file:///work/misc/Poky/sources/ \n"
>
> MIRRORS_append = "\
> ftp://.*/.* file:///work/misc/Poky/sources/ \n \
> http://.*/.* file:///work/misc/Poky/sources/ \n \
> https://.*/.* file:///work/misc/Poky/sources/ \n"
>
> require conf/distro/include/my-fixed-revisions.inc
> require conf/distro/poky.conf
>
> I also disabled DL_DIR in local.conf, falling back to using
> "downloads" in my build tree (/work/local/new_p60).
>
> As you can see, my distro uses Poky as its base, just overriding
> and/or adding some features for our environment.
>
> With this setup, the build for gconf-dbus_svn fails in the
> unpack stage, with this error:
> NOTE: Unpacking /work/local/new_p60/downloads/trunk_developer.imendio.com_.svn.gconf-dbus_705_.tar.gz to
> /work/local/new_p60/tmp/work/armv7a-poky-linux-gnueabi/gconf-dbus-2.16.0+svnr705-r0/
> tar: /work/local/new_p60/downloads/trunk_developer.imendio.com_.svn.gconf-dbus_705_.tar.gz: Cannot open: No such file or directory
>
> Did I set this up incorrectly?
>

Note: it also fails if I only use the pure poky.conf
(not touching PREMIRRORS or MIRRORS in my distro.conf)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Write-protect $DL_DIR

[Gary Thomas]

On 12/22/2010 07:51 AM, Gary Thomas wrote:
> On 12/21/2010 03:34 PM, Gary Thomas wrote:
> ... snip
>>
>> I've tried your changes and they solve my problem well, thanks.
>>
>
> I have had a problem with this (after more testing). Originally,
> I had DL_DIR="/work/misc/Poky/sources/", using it somewhat like
> a mirror. With your changes yesterday, I added these lines to
> my distro.conf:
>
> # Provide pre-staged sources
> PREMIRRORS_append = "\
> bzr://.*/.* file:///work/misc/Poky/sources/ \n \
> cvs://.*/.* file:///work/misc/Poky/sources/ \n \
> git://.*/.* file:///work/misc/Poky/sources/ \n \
> hg://.*/.* file:///work/misc/Poky/sources/ \n \
> osc://.*/.* file:///work/misc/Poky/sources/ \n \
> p4://.*/.* file:///work/misc/Poky/sources/ \n \
> svk://.*/.* file:///work/misc/Poky/sources/ \n \
> svn://.*/.* file:///work/misc/Poky/sources/ \n"
>
> MIRRORS_append = "\
> ftp://.*/.* file:///work/misc/Poky/sources/ \n \
> http://.*/.* file:///work/misc/Poky/sources/ \n \
> https://.*/.* file:///work/misc/Poky/sources/ \n"
>
> require conf/distro/include/my-fixed-revisions.inc
> require conf/distro/poky.conf
>
> I also disabled DL_DIR in local.conf, falling back to using
> "downloads" in my build tree (/work/local/new_p60).
>
> As you can see, my distro uses Poky as its base, just overriding
> and/or adding some features for our environment.
>
> With this setup, the build for gconf-dbus_svn fails in the
> unpack stage, with this error:
> NOTE: Unpacking /work/local/new_p60/downloads/trunk_developer.imendio.com_.svn.gconf-dbus_705_.tar.gz to
> /work/local/new_p60/tmp/work/armv7a-poky-linux-gnueabi/gconf-dbus-2.16.0+svnr705-r0/
> tar: /work/local/new_p60/downloads/trunk_developer.imendio.com_.svn.gconf-dbus_705_.tar.gz: Cannot open: No such file or directory
>
> Did I set this up incorrectly?

I've tracked down why this happens, but I've no idea how
to best fix it.

It seems that in bitbake/lib/bb/fetch/__init__.py, the PREMIRRORS
are used by the function go() to figure out where to find a file.
In the case of the SVN: URL used by the gconf-dbus_svn recipe,
this could be a staged tar file representing the SVN tree, which
does exist in my local mirror.  During the 'fetch' phase, all
works properly and fetch decides that the file is available
from the mirror as above.

The problem comes during unpack which is run as a separate step.
All the decisions about where to find the file that were computed
in fetch have been lost (new step, new context?) and unpack goes
looking for the file in downloads where it assumed fetch left it.
The confusion seems to only happen with URLs that need PREMIRROR
support.

Does this make any sense?


-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------