[PATCH] rootfs: always update the opkg index

[PATCH] rootfs: always update the opkg index

[Ioan-Adrian Ratiu]

The previous logic assumed that if $BUILD_IMAGES_FROM_FEEDS=1 then a
complete set of ipk feeds from which to build the image is already
present under $IPK_FEED_URIS at do_rootfs runtime.

$IPK_FEED_URIS usually contains "file://${DEPLOY_DIR_IPK}" which
renders the above assumption bad because some recipes in the current
build can contain code like do_install[nostamp] = "1" which will cause
rebuilds bumping $PR and invalidating the index.

Even when the index is manually re-created before an image build
("bitbake package-index"), the nostamp will cause failures because the
dependency gets rebuilt before do_rootfs in the "bitbake <image>" call.

So make the opkg rootfs index logic the same as for rpm/deb, to always
update the index in $DEPLOY_DIR_IPK to fix the above nostamp failure.

Feeds outside $DEPLOY_DIR_IPK continue to work as usual.

Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com>
---
 meta/lib/oe/rootfs.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index cdb86f7715..67ae281e47 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -853,9 +853,8 @@ class OpkgRootfs(DpkgOpkgRootfs):
         opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS')
         opkg_post_process_cmds = self.d.getVar('OPKG_POSTPROCESS_COMMANDS')
 
-        # update PM index files, unless users provide their own feeds
-        if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS') or "") != "1":
-            self.pm.write_index()
+        # update PM index files
+        self.pm.write_index()
 
         execute_pre_post_process(self.d, opkg_pre_process_cmds)
 
-- 
2.19.0

Re: [PATCH] rootfs: always update the opkg index

[Richard Purdie]

On Mon, 2018-09-24 at 16:25 +0300, Ioan-Adrian Ratiu wrote:
> The previous logic assumed that if $BUILD_IMAGES_FROM_FEEDS=1 then a
> complete set of ipk feeds from which to build the image is already
> present under $IPK_FEED_URIS at do_rootfs runtime.
> 
> $IPK_FEED_URIS usually contains "file://${DEPLOY_DIR_IPK}" which
> renders the above assumption bad because some recipes in the current
> build can contain code like do_install[nostamp] = "1" which will
> cause
> rebuilds bumping $PR and invalidating the index.
> 
> Even when the index is manually re-created before an image build
> ("bitbake package-index"), the nostamp will cause failures because
> the
> dependency gets rebuilt before do_rootfs in the "bitbake <image>"
> call.
> 
> So make the opkg rootfs index logic the same as for rpm/deb, to
> always
> update the index in $DEPLOY_DIR_IPK to fix the above nostamp failure.
> 
> Feeds outside $DEPLOY_DIR_IPK continue to work as usual.

Is this with master or an older branch?

With master, a copy of the feed is constructed under WORKDIR which is
then indexed since other image generation or package writes could occur
and the index isn't static. I'm guessing it doesn't do this for
external feeds and assumes those feeds are static. 

I am wondering what happens if you put a remote (http://) url into this
though? Does that still work or does it break?

I'm also worried about what happens if the file:// path you point at is
owned by another user and isn't writable.

Cheers,

Richard

Re: [PATCH] rootfs: always update the opkg index

[Ioan-Adrian Ratiu]

On Mon, 24 Sep 2018, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> On Mon, 2018-09-24 at 16:25 +0300, Ioan-Adrian Ratiu wrote:
>> The previous logic assumed that if $BUILD_IMAGES_FROM_FEEDS=1 then a
>> complete set of ipk feeds from which to build the image is already
>> present under $IPK_FEED_URIS at do_rootfs runtime.
>> 
>> $IPK_FEED_URIS usually contains "file://${DEPLOY_DIR_IPK}" which
>> renders the above assumption bad because some recipes in the current
>> build can contain code like do_install[nostamp] = "1" which will
>> cause
>> rebuilds bumping $PR and invalidating the index.
>> 
>> Even when the index is manually re-created before an image build
>> ("bitbake package-index"), the nostamp will cause failures because
>> the
>> dependency gets rebuilt before do_rootfs in the "bitbake <image>"
>> call.
>> 
>> So make the opkg rootfs index logic the same as for rpm/deb, to
>> always
>> update the index in $DEPLOY_DIR_IPK to fix the above nostamp failure.
>> 
>> Feeds outside $DEPLOY_DIR_IPK continue to work as usual.
>
> Is this with master or an older branch?

I've extensively tested this with Sumo, on master I could only verify
the build passes and a minimal image boots.

>
> With master, a copy of the feed is constructed under WORKDIR which is
> then indexed since other image generation or package writes could occur
> and the index isn't static. I'm guessing it doesn't do this for
> external feeds and assumes those feeds are static.

My description of the problem mirrors my understanding of the feed
construction process which might be limited especially if it changed on
master post-Sumo, but the answer is yes, external feeds are assumed
static.

>
> I am wondering what happens if you put a remote (http://) url into this
> though? Does that still work or does it break?

It still works, we're using http paths all the time in our distro
(public feed server is at http://download.ni.com/ni-linux-rt/)

>
> I'm also worried about what happens if the file:// path you point at is
> owned by another user and isn't writable.

Thanks for pointing this out. Looks like it enters an infinite loop: the
do_rootfs task is stuck at 3% and a worker process pegs a core to 100%.

Without this change, do_rootfs gets stuck at 12%, worker pegging a core
if the feed is owned by another user with no write permissions.

This was unexpected :) I'll roll up my sleeves and start digging.

Do you have any pointers about what might be happening?

>
> Cheers,
>
> Richard

Re: [PATCH] rootfs: always update the opkg index

[richard.purdie]

On Mon, 2018-09-24 at 18:00 +0300, Ioan-Adrian Ratiu wrote:
> > With master, a copy of the feed is constructed under WORKDIR which
> > is
> > then indexed since other image generation or package writes could
> > occur
> > and the index isn't static. I'm guessing it doesn't do this for
> > external feeds and assumes those feeds are static.
> 
> My description of the problem mirrors my understanding of the feed
> construction process which might be limited especially if it changed
> on master post-Sumo, but the answer is yes, external feeds are
> assumed static.

Ok.

> > I am wondering what happens if you put a remote (http://) url into
> > this
> > though? Does that still work or does it break?
> 
> It still works, we're using http paths all the time in our distro
> (public feed server is at http://download.ni.com/ni-linux-rt/)

That is good to know and worth mentioning in the commit message as that
isn't clear.

> > 
> > I'm also worried about what happens if the file:// path you point
> > at is
> > owned by another user and isn't writable.
> 
> Thanks for pointing this out. Looks like it enters an infinite loop:
> the
> do_rootfs task is stuck at 3% and a worker process pegs a core to
> 100%.
> 
> Without this change, do_rootfs gets stuck at 12%, worker pegging a
> core
> if the feed is owned by another user with no write permissions.
> 
> This was unexpected :) I'll roll up my sleeves and start digging.
> 
> Do you have any pointers about what might be happening?

I was expecting it to error, not hang. If I had to totally guess, I'd
guess at the bb.utils.lockfile() hanging.

Cheers,

Richard

Re: [PATCH] rootfs: always update the opkg index

[Ioan-Adrian Ratiu]

On Mon, 24 Sep 2018, richard.purdie@linuxfoundation.org wrote:
> On Mon, 2018-09-24 at 18:00 +0300, Ioan-Adrian Ratiu wrote:
>> > With master, a copy of the feed is constructed under WORKDIR which
>> > is
>> > then indexed since other image generation or package writes could
>> > occur
>> > and the index isn't static. I'm guessing it doesn't do this for
>> > external feeds and assumes those feeds are static.
>> 
>> My description of the problem mirrors my understanding of the feed
>> construction process which might be limited especially if it changed
>> on master post-Sumo, but the answer is yes, external feeds are
>> assumed static.
>
> Ok.
>
>> > I am wondering what happens if you put a remote (http://) url into
>> > this
>> > though? Does that still work or does it break?
>> 
>> It still works, we're using http paths all the time in our distro
>> (public feed server is at http://download.ni.com/ni-linux-rt/)
>
> That is good to know and worth mentioning in the commit message as that
> isn't clear.
>
>> > 
>> > I'm also worried about what happens if the file:// path you point
>> > at is
>> > owned by another user and isn't writable.
>> 
>> Thanks for pointing this out. Looks like it enters an infinite loop:
>> the
>> do_rootfs task is stuck at 3% and a worker process pegs a core to
>> 100%.
>> 
>> Without this change, do_rootfs gets stuck at 12%, worker pegging a
>> core
>> if the feed is owned by another user with no write permissions.
>> 
>> This was unexpected :) I'll roll up my sleeves and start digging.
>> 
>> Do you have any pointers about what might be happening?
>
> I was expecting it to error, not hang. If I had to totally guess, I'd
> guess at the bb.utils.lockfile() hanging.

I can't reproduce the hang anymore (with or without my change)... I
tried re-using the sstate, clearing it, all combinations of calling
bitbake packagegroups, package-index, image, etc and still no hang.

I'm thinking the hang was caused by a build race I can't easily
reproduce.

Now I get "operation not permitted" errors every time like this one:

Exception: subprocess.CalledProcessError: Command 'cd
(...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks; 
find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks
-p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xhf - -C (...)/tmp-glibc/deploy/ipk' returned non-zero exit status 2.

Subprocess output:
tar: ./core2-64: Cannot utime: Operation not permitted
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors

ERROR: safemode-image-1.0-r0 do_package_write_ipk: Function failed:
sstate_task_postfunc
ERROR: Logfile of failure stored in:
(...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/temp/log.do_package_write_ipk.21668

Seeing that the hang is not related to this change, are you ok with
integrating v2 in which I'll update the commit message to mention
http:// URI's still work as expected?

>
> Cheers,
>
> Richard

Re: [PATCH] rootfs: always update the opkg index

[Ioan-Adrian Ratiu]

On Tue, 25 Sep 2018, Ioan-Adrian Ratiu <adrian.ratiu@ni.com> wrote:
> On Mon, 24 Sep 2018, richard.purdie@linuxfoundation.org wrote:
>> On Mon, 2018-09-24 at 18:00 +0300, Ioan-Adrian Ratiu wrote:
>>> > With master, a copy of the feed is constructed under WORKDIR which
>>> > is
>>> > then indexed since other image generation or package writes could
>>> > occur
>>> > and the index isn't static. I'm guessing it doesn't do this for
>>> > external feeds and assumes those feeds are static.
>>> 
>>> My description of the problem mirrors my understanding of the feed
>>> construction process which might be limited especially if it changed
>>> on master post-Sumo, but the answer is yes, external feeds are
>>> assumed static.
>>
>> Ok.
>>
>>> > I am wondering what happens if you put a remote (http://) url into
>>> > this
>>> > though? Does that still work or does it break?
>>> 
>>> It still works, we're using http paths all the time in our distro
>>> (public feed server is at http://download.ni.com/ni-linux-rt/)
>>
>> That is good to know and worth mentioning in the commit message as that
>> isn't clear.
>>
>>> > 
>>> > I'm also worried about what happens if the file:// path you point
>>> > at is
>>> > owned by another user and isn't writable.
>>> 
>>> Thanks for pointing this out. Looks like it enters an infinite loop:
>>> the
>>> do_rootfs task is stuck at 3% and a worker process pegs a core to
>>> 100%.
>>> 
>>> Without this change, do_rootfs gets stuck at 12%, worker pegging a
>>> core
>>> if the feed is owned by another user with no write permissions.
>>> 
>>> This was unexpected :) I'll roll up my sleeves and start digging.
>>> 
>>> Do you have any pointers about what might be happening?
>>
>> I was expecting it to error, not hang. If I had to totally guess, I'd
>> guess at the bb.utils.lockfile() hanging.
>
> I can't reproduce the hang anymore (with or without my change)... I
> tried re-using the sstate, clearing it, all combinations of calling
> bitbake packagegroups, package-index, image, etc and still no hang.
>
> I'm thinking the hang was caused by a build race I can't easily
> reproduce.
>
> Now I get "operation not permitted" errors every time like this one:
>
> Exception: subprocess.CalledProcessError: Command 'cd
> (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks; 
> find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks
> -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xhf - -C (...)/tmp-glibc/deploy/ipk' returned non-zero exit status 2.
>
> Subprocess output:
> tar: ./core2-64: Cannot utime: Operation not permitted
> tar: .: Cannot utime: Operation not permitted
> tar: Exiting with failure status due to previous errors
>
> ERROR: safemode-image-1.0-r0 do_package_write_ipk: Function failed:
> sstate_task_postfunc
> ERROR: Logfile of failure stored in:
> (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/temp/log.do_package_write_ipk.21668
>
> Seeing that the hang is not related to this change, are you ok with
> integrating v2 in which I'll update the commit message to mention
> http:// URI's still work as expected?

Please ignore this message, I was misled by the bad naming (the
safemode-image is actually a package), the hang reproduces, I'm on it.

>
>>
>> Cheers,
>>
>> Richard