Re: [PATCH] rootfs: always update the opkg index

[Ioan-Adrian Ratiu <adrian.ratiu@ni.com>]

On Mon, 24 Sep 2018, richard.purdie@linuxfoundation.org wrote:
> On Mon, 2018-09-24 at 18:00 +0300, Ioan-Adrian Ratiu wrote:
>> > With master, a copy of the feed is constructed under WORKDIR which
>> > is
>> > then indexed since other image generation or package writes could
>> > occur
>> > and the index isn't static. I'm guessing it doesn't do this for
>> > external feeds and assumes those feeds are static.
>> 
>> My description of the problem mirrors my understanding of the feed
>> construction process which might be limited especially if it changed
>> on master post-Sumo, but the answer is yes, external feeds are
>> assumed static.
>
> Ok.
>
>> > I am wondering what happens if you put a remote (http://) url into
>> > this
>> > though? Does that still work or does it break?
>> 
>> It still works, we're using http paths all the time in our distro
>> (public feed server is at http://download.ni.com/ni-linux-rt/)
>
> That is good to know and worth mentioning in the commit message as that
> isn't clear.
>
>> > 
>> > I'm also worried about what happens if the file:// path you point
>> > at is
>> > owned by another user and isn't writable.
>> 
>> Thanks for pointing this out. Looks like it enters an infinite loop:
>> the
>> do_rootfs task is stuck at 3% and a worker process pegs a core to
>> 100%.
>> 
>> Without this change, do_rootfs gets stuck at 12%, worker pegging a
>> core
>> if the feed is owned by another user with no write permissions.
>> 
>> This was unexpected :) I'll roll up my sleeves and start digging.
>> 
>> Do you have any pointers about what might be happening?
>
> I was expecting it to error, not hang. If I had to totally guess, I'd
> guess at the bb.utils.lockfile() hanging.

I can't reproduce the hang anymore (with or without my change)... I
tried re-using the sstate, clearing it, all combinations of calling
bitbake packagegroups, package-index, image, etc and still no hang.

I'm thinking the hang was caused by a build race I can't easily
reproduce.

Now I get "operation not permitted" errors every time like this one:

Exception: subprocess.CalledProcessError: Command 'cd
(...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks; 
find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks
-p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xhf - -C (...)/tmp-glibc/deploy/ipk' returned non-zero exit status 2.

Subprocess output:
tar: ./core2-64: Cannot utime: Operation not permitted
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors

ERROR: safemode-image-1.0-r0 do_package_write_ipk: Function failed:
sstate_task_postfunc
ERROR: Logfile of failure stored in:
(...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/temp/log.do_package_write_ipk.21668

Seeing that the hang is not related to this change, are you ok with
integrating v2 in which I'll update the commit message to mention
http:// URI's still work as expected?

>
> Cheers,
>
> Richard