"qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

"qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Steffen Sledz]

After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(

The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call

qemu-arm -r 2.6.32 -L /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs -E LD_LIBRARY_PATH=/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/usr/lib:/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/lib /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/bin/udevadm hwdb --update --root /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs

which eats swap space without end till the system crashes (more than 100GB swap space ist available).

I've no idea how to inspect this problem further. Any ideas?

Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Gaurang Shastri]

Dear Steffen,

I have already raised some similar issue where because of un-appropriate use of "QEMU_OPTIONS" led the post process of udev-hwdb failed.

Anyhow, Richard has created a patch and you can go through the same here at:
http://lists.openembedded.org/pipermail/openembedded-core/2014-December/099544.html

Let's see if that solves your issue.

Regards,
Gaurang Shastri

-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Steffen Sledz
Sent: Monday, December 15, 2014 4:24 PM
To: openembedded-core
Subject: [OE-core] "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(

The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call

qemu-arm -r 2.6.32 -L /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs -E LD_LIBRARY_PATH=/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/usr/lib:/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/lib /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/bin/udevadm hwdb --update --root /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs

which eats swap space without end till the system crashes (more than 100GB swap space ist available).

I've no idea how to inspect this problem further. Any ideas?

Steffen

--
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle; Amtsgericht Berlin Charlottenburg; HRB 130120 B; Ust.-IDNr. DE273952058
--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Steffen Sledz]

On 15.12.2014 12:00, Gaurang Shastri wrote:
> From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Steffen Sledz
>> After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(
>>
>> The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call
>>
>> qemu-arm -r 2.6.32 -L /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs -E LD_LIBRARY_PATH=/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/usr/lib:/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/lib /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/bin/udevadm hwdb --update --root /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs
>>
>> which eats swap space without end till the system crashes (more than 100GB swap space ist available).
>>
>> I've no idea how to inspect this problem further. Any ideas?
>
> I have already raised some similar issue where because of un-appropriate use of "QEMU_OPTIONS" led the post process of udev-hwdb failed.
> 
> Anyhow, Richard has created a patch and you can go through the same here at:
> http://lists.openembedded.org/pipermail/openembedded-core/2014-December/099544.html
> 
> Let's see if that solves your issue.

Unfortunately not.

I've used the latest dizzy-Branch and applied this patch, but the behaviour did not change. :(

commit 219e793907406eabf632e784e3a11ab9acb77cfb
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Thu Dec 4 19:49:19 2014 +0000

    qemu/libc-package: Fix qemu option handling

    The 'overrides' here are PACKAGE_ARCH based and hence not overrides
    as such and the _append wasn't working in many cases. This adjusts the
    code to use PACKAGE_ARCH as the accessor and ensures the variables
    work as expected. This fixes various segfaults and ensures postinsts
    run at build time rather than on the target system.

    The bug was introduced in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7efad8a1b56df6ee07c12ad360c0493d7b1d6d23.

    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
    Signed-off-by: Ross Burton <ross.burton@intel.com>

Regards,
Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Enrico Scholz]

Steffen Sledz <sledz-mQBY8sqGERgRVMum7q6B+Q@public.gmane.org> writes:

> The first thing we observed was that the do_rootfs stage made a very
> big load (more than 500) on the machine but the cpu's aren't working
> for 100%. I could track down the problem to the call
> ...
> which eats swap space without end till the system crashes (more than 100GB swap space ist available).
>
> I've no idea how to inspect this problem further. Any ideas?

I had a similar issue when /var/tmp in the rootfs was an absolute symlink
pointing to toplevel /var/tmp which contained some millions of files.

Can you look (lsof) which files are open by qemu?


Enrico

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Steffen Sledz]

On 15.12.2014 12:00, Gaurang Shastri wrote:
> From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Steffen Sledz
>> After updating our build host from openSUSE 13.1 to 13.2 we're no longer able to build our images with OE (with 13.1 on the same host everything worked well). :(
>>
>> The first thing we observed was that the do_rootfs stage made a very big load (more than 500) on the machine but the cpu's aren't working for 100%. I could track down the problem to the call
>>
>> qemu-arm -r 2.6.32 -L /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs -E LD_LIBRARY_PATH=/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/usr/lib:/pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/lib /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs/bin/udevadm hwdb --update --root /pm/sledz/hipos-dfe.dizzy/build-closed/tmp-glibc/work/hikirk-angstrom-linux-gnueabi/hydraip-prodimage/1.0-r0+r16+r17.5/rootfs
>>
>> which eats swap space without end till the system crashes (more than 100GB swap space ist available).
>>
>> I've no idea how to inspect this problem further. Any ideas?
>
> I have already raised some similar issue where because of un-appropriate use of "QEMU_OPTIONS" led the post process of udev-hwdb failed.
> 
> Anyhow, Richard has created a patch and you can go through the same here at:
> http://lists.openembedded.org/pipermail/openembedded-core/2014-December/099544.html
> 
> Let's see if that solves your issue.

Unfortunately not.

I've used the latest dizzy-Branch and applied this patch, but the behaviour did not change.

commit 219e793907406eabf632e784e3a11ab9acb77cfb
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Thu Dec 4 19:49:19 2014 +0000

    qemu/libc-package: Fix qemu option handling

    The 'overrides' here are PACKAGE_ARCH based and hence not overrides
    as such and the _append wasn't working in many cases. This adjusts the
    code to use PACKAGE_ARCH as the accessor and ensures the variables
    work as expected. This fixes various segfaults and ensures postinsts
    run at build time rather than on the target system.

    The bug was introduced in http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7efad8a1b56df6ee07c12ad360c0493d7b1d6d23.

    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
    Signed-off-by: Ross Burton <ross.burton@intel.com>

Regards,
Steffen

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Steffen Sledz]

On 15.12.2014 17:19, Enrico Scholz wrote:
> Steffen Sledz <sledz-mQBY8sqGERgRVMum7q6B+Q@public.gmane.org> writes:
> 
>> The first thing we observed was that the do_rootfs stage made a very
>> big load (more than 500) on the machine but the cpu's aren't working
>> for 100%. I could track down the problem to the call
>> ...
>> which eats swap space without end till the system crashes (more than 100GB swap space ist available).
>>
>> I've no idea how to inspect this problem further. Any ideas?
> 
> I had a similar issue when /var/tmp in the rootfs was an absolute symlink
> pointing to toplevel /var/tmp which contained some millions of files.
> 
> Can you look (lsof) which files are open by qemu?

Crazy! This really seems to be the problem.

I had a big subtree from earlier local openSUSE Build Service runs below toplevel /var/run. After deleting this the build succeeds.

But manually deleting such files before running OE builds cannot really be a reasonable solution. :(

-- 
DResearch Fahrzeugelektronik GmbH
Otto-Schmirgal-Str. 3, 10319 Berlin, Germany
Tel: +49 30 515932-237 mailto:sledz@dresearch-fe.de
Fax: +49 30 515932-299
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 130120 B;
Ust.-IDNr. DE273952058

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Richard Purdie]

On Tue, 2014-12-16 at 09:36 +0100, Steffen Sledz wrote:
> On 15.12.2014 17:19, Enrico Scholz wrote:
> > Steffen Sledz <sledz-mQBY8sqGERgRVMum7q6B+Q@public.gmane.org> writes:
> > 
> >> The first thing we observed was that the do_rootfs stage made a very
> >> big load (more than 500) on the machine but the cpu's aren't working
> >> for 100%. I could track down the problem to the call
> >> ...
> >> which eats swap space without end till the system crashes (more than 100GB swap space ist available).
> >>
> >> I've no idea how to inspect this problem further. Any ideas?
> > 
> > I had a similar issue when /var/tmp in the rootfs was an absolute symlink
> > pointing to toplevel /var/tmp which contained some millions of files.
> > 
> > Can you look (lsof) which files are open by qemu?
> 
> Crazy! This really seems to be the problem.
> 
> I had a big subtree from earlier local openSUSE Build Service runs below toplevel /var/run. After deleting this the build succeeds.
> 
> But manually deleting such files before running OE builds cannot really be a reasonable solution. :(

Agreed, we need to figure out which tool is suffering issues with
circular symlinks, then fix that tool. Its not so much a problem with
the core build system as the tools we're using though :(

Cheers,

Richard

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>> I had a big subtree from earlier local openSUSE Build Service runs
>> below toplevel /var/run. After deleting this the build succeeds.
>> 
>> But manually deleting such files before running OE builds cannot
>> really be a reasonable solution. :(
>
> Agreed, we need to figure out which tool is suffering issues with
> circular symlinks,

Circular symlinks are not the issue but absolute symlinks (like /tmp ->
/var/tmp due to local changes).  I can not find absolute symlinks to
directories in my images anymore, so Steffen's problem might be caused
by local changes too.

Chroot unaware tools (qemu) are the real problem... 


Enrico

Re: "qemu-arm udevadm hwdb --update" eats swap space after build host update to openSUSE 13.2

[Khem Raj]

> On Dec 18, 2014, at 3:19 AM, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> wrote:
> 
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
>>> I had a big subtree from earlier local openSUSE Build Service runs
>>> below toplevel /var/run. After deleting this the build succeeds.
>>> 
>>> But manually deleting such files before running OE builds cannot
>>> really be a reasonable solution. :(
>> 
>> Agreed, we need to figure out which tool is suffering issues with
>> circular symlinks,
> 
> Circular symlinks are not the issue but absolute symlinks (like /tmp ->
> /var/tmp due to local changes).  I can not find absolute symlinks to
> directories in my images anymore, so Steffen's problem might be caused
> by local changes too.
> 
> Chroot unaware tools (qemu) are the real problem… 

I think in this case one could use qemu’s -E option ( -E TMPDIR=/x/y/z) to pass target specific TMPDIR
to udevadm