Machine stuck when userspace 100% busy

Machine stuck when userspace 100% busy

[Rolf Eike Beer]

My C3600 runs the CMake nightly builds. Basically this is a master 
process (ctest) that forks other binaries that do the actual tests. 
Afterwards it collects the output. If the child does not respond for 
some time (usually set to 30 minutes) it will get killed by ctest.

Yesterday someone accidentially introduced an endless loop into CMake, 
so some of the called tests will run at 100% CPU load forever. The 
master process was not affected by this, so these childs should have 
eventually got killed. But this did not happen. It did happen on all 
other machines building those tests 
(http://open.cdash.org/index.php?project=CMake&date=2012-10-21, e.g. 
http://open.cdash.org/viewTest.php?onlyfailed&buildid=2621607), but not 
on my machine. And from all what I can tell it does not look as if it is 
a ctest bug, but something in the scheduler or something like that not 
working properly.

$ ping voyager
PING voyager (192.168.2.119) 56(84) bytes of data.
64 bytes from voyager (192.168.2.119): icmp_seq=1 ttl=64 time=0.504 ms
64 bytes from voyager (192.168.2.119): icmp_seq=2 ttl=64 time=0.268 ms
64 bytes from voyager (192.168.2.119): icmp_seq=3 ttl=64 time=0.274 ms

So, the machine is alive and the ping time is ok. Doing ssh to it will 
get stuck for hours (literally). So, sadly, I have currently no way to 
get into userland of the machine. What I know is:

-ssh doesn't work
-kernel is alive
-the machine is very likely running at 100% CPU load from a normal user 
account (with not too excessive RAM usage AFAIK)

So for me it looks like this "userspace is at 100%" does something 
utterly bad to the scheduling, as it seems that no other processes will 
get their chance of running. If ctest would get it's chance it should 
have killed the slave after ~30 minutes, and ssh should definitely work. 
 From what I see on other machines the worst case scenario would be 18 of 
these amok processes, so after ~9 hours the dust should start to clear. 
That would have been nearly 20 hours ago, so something is not working 
there.

Any ideas?

Eike

Re: Machine stuck when userspace 100% busy

[Carlos O'Donell]

On Mon, Oct 22, 2012 at 3:41 AM, Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
> My C3600 runs the CMake nightly builds. Basically this is a master process
> (ctest) that forks other binaries that do the actual tests. Afterwards it
> collects the output. If the child does not respond for some time (usually
> set to 30 minutes) it will get killed by ctest.
>
> Yesterday someone accidentially introduced an endless loop into CMake, so
> some of the called tests will run at 100% CPU load forever. The master
> process was not affected by this, so these childs should have eventually got
> killed. But this did not happen. It did happen on all other machines
> building those tests
> (http://open.cdash.org/index.php?project=CMake&date=2012-10-21, e.g.
> http://open.cdash.org/viewTest.php?onlyfailed&buildid=2621607), but not on
> my machine. And from all what I can tell it does not look as if it is a
> ctest bug, but something in the scheduler or something like that not working
> properly.
>
> $ ping voyager
> PING voyager (192.168.2.119) 56(84) bytes of data.
> 64 bytes from voyager (192.168.2.119): icmp_seq=1 ttl=64 time=0.504 ms
> 64 bytes from voyager (192.168.2.119): icmp_seq=2 ttl=64 time=0.268 ms
> 64 bytes from voyager (192.168.2.119): icmp_seq=3 ttl=64 time=0.274 ms
>
> So, the machine is alive and the ping time is ok. Doing ssh to it will get
> stuck for hours (literally). So, sadly, I have currently no way to get into
> userland of the machine. What I know is:
>
> -ssh doesn't work
> -kernel is alive
> -the machine is very likely running at 100% CPU load from a normal user
> account (with not too excessive RAM usage AFAIK)
>
> So for me it looks like this "userspace is at 100%" does something utterly
> bad to the scheduling, as it seems that no other processes will get their
> chance of running. If ctest would get it's chance it should have killed the
> slave after ~30 minutes, and ssh should definitely work. From what I see on
> other machines the worst case scenario would be 18 of these amok processes,
> so after ~9 hours the dust should start to clear. That would have been
> nearly 20 hours ago, so something is not working there.
>
> Any ideas?

Get on the serial console and see if you can login?

Then issue a magic-sysrq+t? What's userspace doing?

Cheers,
Carlos.

Re: Machine stuck when userspace 100% busy

[Rolf Eike Beer]

Am , schrieb Carlos O'Donell:
> On Mon, Oct 22, 2012 at 3:41 AM, Rolf Eike Beer 
> <eike-kernel@sf-tec.de> wrote:
>> My C3600 runs the CMake nightly builds. Basically this is a master 
>> process
>> (ctest) that forks other binaries that do the actual tests. 
>> Afterwards it
>> collects the output. If the child does not respond for some time 
>> (usually
>> set to 30 minutes) it will get killed by ctest.
>>
>> Yesterday someone accidentially introduced an endless loop into 
>> CMake, so
>> some of the called tests will run at 100% CPU load forever. The 
>> master
>> process was not affected by this, so these childs should have 
>> eventually got
>> killed. But this did not happen. It did happen on all other machines
>> building those tests
>> (http://open.cdash.org/index.php?project=CMake&date=2012-10-21, e.g.
>> http://open.cdash.org/viewTest.php?onlyfailed&buildid=2621607), but 
>> not on
>> my machine. And from all what I can tell it does not look as if it 
>> is a
>> ctest bug, but something in the scheduler or something like that not 
>> working
>> properly.
>>
>> $ ping voyager
>> PING voyager (192.168.2.119) 56(84) bytes of data.
>> 64 bytes from voyager (192.168.2.119): icmp_seq=1 ttl=64 time=0.504 
>> ms
>> 64 bytes from voyager (192.168.2.119): icmp_seq=2 ttl=64 time=0.268 
>> ms
>> 64 bytes from voyager (192.168.2.119): icmp_seq=3 ttl=64 time=0.274 
>> ms
>>
>> So, the machine is alive and the ping time is ok. Doing ssh to it 
>> will get
>> stuck for hours (literally). So, sadly, I have currently no way to 
>> get into
>> userland of the machine. What I know is:
>>
>> -ssh doesn't work
>> -kernel is alive
>> -the machine is very likely running at 100% CPU load from a normal 
>> user
>> account (with not too excessive RAM usage AFAIK)
>>
>> So for me it looks like this "userspace is at 100%" does something 
>> utterly
>> bad to the scheduling, as it seems that no other processes will get 
>> their
>> chance of running. If ctest would get it's chance it should have 
>> killed the
>> slave after ~30 minutes, and ssh should definitely work. From what I 
>> see on
>> other machines the worst case scenario would be 18 of these amok 
>> processes,
>> so after ~9 hours the dust should start to clear. That would have 
>> been
>> nearly 20 hours ago, so something is not working there.
>>
>> Any ideas?
>
> Get on the serial console and see if you can login?

Was not possible, but because of something I broke on the other side of 
the serial line (i.e. no fault of the C3600).

> Then issue a magic-sysrq+t? What's userspace doing?

I went down to the machine and it was happily heartbeating as expected 
when the kernel still is able to send out ping replies. I pushed the 
power button and it shut down cleanly in something like half a minute. 
Poweron did work fine.

I see in the logs one OOM event during the CMake tests. Afterwards I 
see for another 10 minutes some nagios events, so it took a while after 
the OOM for the machine to freak out, and things like creating new 
processes was no problem until then.

I have no deeper insight in what happened later beyond what I already 
have written.

Eike

Re: Machine stuck when userspace 100% busy

[Carlos O'Donell]

On Mon, Oct 22, 2012 at 10:07 AM, Rolf Eike Beer <eike-kernel@sf-tec.de> wrote:
>> Get on the serial console and see if you can login?
>
>
> Was not possible, but because of something I broke on the other side of the
> serial line (i.e. no fault of the C3600).
>
>
>> Then issue a magic-sysrq+t? What's userspace doing?
>
>
> I went down to the machine and it was happily heartbeating as expected when
> the kernel still is able to send out ping replies. I pushed the power button
> and it shut down cleanly in something like half a minute. Poweron did work
> fine.
>
> I see in the logs one OOM event during the CMake tests. Afterwards I see for
> another 10 minutes some nagios events, so it took a while after the OOM for
> the machine to freak out, and things like creating new processes was no
> problem until then.
>
> I have no deeper insight in what happened later beyond what I already have
> written.

Without more information it's hard to determine what went wrong.

Cheers,
Carlos.

Re: Machine stuck when userspace 100% busy

[Rolf Eike Beer]

Am , schrieb Carlos O'Donell:
> On Mon, Oct 22, 2012 at 10:07 AM, Rolf Eike Beer 
> <eike-kernel@sf-tec.de> wrote:
>>> Get on the serial console and see if you can login?
>>
>>
>> Was not possible, but because of something I broke on the other side 
>> of the
>> serial line (i.e. no fault of the C3600).
>>
>>
>>> Then issue a magic-sysrq+t? What's userspace doing?
>>
>>
>> I went down to the machine and it was happily heartbeating as 
>> expected when
>> the kernel still is able to send out ping replies. I pushed the 
>> power button
>> and it shut down cleanly in something like half a minute. Poweron 
>> did work
>> fine.
>>
>> I see in the logs one OOM event during the CMake tests. Afterwards I 
>> see for
>> another 10 minutes some nagios events, so it took a while after the 
>> OOM for
>> the machine to freak out, and things like creating new processes was 
>> no
>> problem until then.
>>
>> I have no deeper insight in what happened later beyond what I 
>> already have
>> written.
>
> Without more information it's hard to determine what went wrong.

Ok, this is the way I can reproduce it:

git clone git://cmake.org/cmake.git
cd cmake
git checkout cb4bff9882ecb7dbd674cc9d97ac5cd42248f0dc
mkdir build
cd build
cmake ..
make
ctest

The test RunCMake.ObjectLibrary (208 of 253 for me) will then hang the 
machine. It will not hang the machine when run on it's own.

Kernel: 3.6.2
glibc: GNU C Library stable release version 2.14.1, by Roland McGrath 
et al.
RAM: 512 MiB
Swap: 2GiB

Eike

Re: Machine stuck when userspace 100% busy

[John David Anglin]

On 23-Oct-12, at 11:23 AM, Rolf Eike Beer wrote:

> Ok, this is the way I can reproduce it:
>
> git clone git://cmake.org/cmake.git
> cd cmake
> git checkout cb4bff9882ecb7dbd674cc9d97ac5cd42248f0dc
> mkdir build
> cd build
> cmake ..
> make
> ctest


I ran this test on my system with the following results:

94% tests passed, 15 tests failed out of 261

Label Time Summary:
Label1    =   0.26 sec
Label2    =   0.26 sec

Total Test time (real) = 11715.14 sec

The following tests FAILED:
          57 - ExportImport (Failed)
          65 - Qt4Deploy (OTHER_FAULT)
          69 - Module.GenerateExportHeader (OTHER_FAULT)
          91 - SimpleInstall-Stage2 (OTHER_FAULT)
         111 - complex (OTHER_FAULT)
         112 - complexOneConfig (OTHER_FAULT)
         115 - QtAutomoc (OTHER_FAULT)
         133 - Plugin (OTHER_FAULT)
         191 - BootstrapTest (Failed)
         196 - IncludeDirectories (OTHER_FAULT)
         202 - CMakeOnly.LinkInterfaceLoop (Timeout)
         210 - CMakeOnly.AllFindModules (Failed)
         211 - CMakeOnly.TargetScope (Failed)
         217 - RunCMake.ObjectLibrary (Failed)
         261 - CMake.CheckSourceTree (Failed)
Errors while running CTest

System didn't hang.

Machine: rp3440
Kernel: 3.6.2+ (JDA changes)
libc: debian 2.13-35

If you have fixes for any of the above fails let me know.

Dave
--
John David Anglin	dave.anglin@bell.net

Re: Machine stuck when userspace 100% busy

[Rolf Eike Beer]

Am , schrieb John David Anglin:
> On 23-Oct-12, at 11:23 AM, Rolf Eike Beer wrote:
>
>> Ok, this is the way I can reproduce it:
>>
>> git clone git://cmake.org/cmake.git
>> cd cmake
>> git checkout cb4bff9882ecb7dbd674cc9d97ac5cd42248f0dc
>> mkdir build
>> cd build
>> cmake ..
>> make
>> ctest
>
>
> I ran this test on my system with the following results:
>
> 94% tests passed, 15 tests failed out of 261
>
> Label Time Summary:
> Label1    =   0.26 sec
> Label2    =   0.26 sec
>
> Total Test time (real) = 11715.14 sec
>
> The following tests FAILED:
>          57 - ExportImport (Failed)
>          65 - Qt4Deploy (OTHER_FAULT)
>          69 - Module.GenerateExportHeader (OTHER_FAULT)
>          91 - SimpleInstall-Stage2 (OTHER_FAULT)
>         111 - complex (OTHER_FAULT)
>         112 - complexOneConfig (OTHER_FAULT)
>         115 - QtAutomoc (OTHER_FAULT)
>         133 - Plugin (OTHER_FAULT)
>         191 - BootstrapTest (Failed)
>         196 - IncludeDirectories (OTHER_FAULT)
>         202 - CMakeOnly.LinkInterfaceLoop (Timeout)
>         210 - CMakeOnly.AllFindModules (Failed)
>         211 - CMakeOnly.TargetScope (Failed)
>         217 - RunCMake.ObjectLibrary (Failed)
>         261 - CMake.CheckSourceTree (Failed)
> Errors while running CTest

This is more or less expected as the version under test contains a bug, 
so of course some of the tests will fail.

BootstrapTest, the Qt*, and CMakeOnly.AllFindModules are not in the 
list of things that I expected to fail. Please send me your 
Testing/Temporary/LastTest.log so I can look what the problem was with 
them. CMake.CheckSourceTree will fail if you used my commands, as you 
then have the untracked "build" directory in your source tree. So if you 
kill the build dir and put it somewhere outside the checkout and use 
"cmake /path/to/source" instead of "cmake .." this will succeed. The 
CMakeOnly.AllFindModules _could_ be a problem related to multiarch 
include directories, but that one is usually harmless.

> System didn't hang.
>
> Machine: rp3440
> Kernel: 3.6.2+ (JDA changes)
> libc: debian 2.13-35
>
> If you have fixes for any of the above fails let me know.

This is an SMP system, I bet that makes a difference for this sort of 
problem. Can you try booting with nosmp?

Btw: boot with nr_cpu=1 will make the kernel go mad, although I would 
have expected it to behave exactly like nosmp. At least when I last 
tested it.

Eike