Re: kernel freeze when running docker

[Vignesh Raman <vignesh.raman.in@gmail.com>]

Hi Julia,

I enabled Linux Kernel Dump Test Tool (CONFIG_LKDTM=y) and triggered a
hard and soft lockup using,
echo SOFTLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT
echo HARDLOCKUP > /sys/kernel/debug/provoke-crash/DIRECT

With this the kernel panic/oops message is seen in serial console 2 out
of 5 times (for both soft and hard lockup case). In /sys/fs/pstore the
kernel panic messages are not captured at all.

If I try with,
echo PANIC > /sys/kernel/debug/provoke-crash/DIRECT

the kernel panic/oops message is always dumped in serial console and in
/sys/fs/pstore the crash logs are captured always.

It looks like with rt kernel during soft/hard lockup, the dump messages
are not captured sometimes. Not sure if this is related to the issue I'm
seeing (kernel panic logs not seen in serial with docker test case).

I have not performed these tests on non-rt kernel.

Thanks.

Regards,
Vignesh

On Friday 24 August 2018 12:11 PM, Vignesh Raman wrote:
> Hi Julia,
> 
> On Wednesday 22 August 2018 10:12 PM, Julia Cartwright wrote:
>> I never explicitly asked...have you tried a similar test on a non-RT
>> kernel?  Does it exhibit the same behavior?  The linked moby github
>> issue would seem to indicate this isn't RT specific...
> 
> Yes I tried the tests on non-RT kernel and the issue was not reproduced
> with 14 hours of testing with docker running kubernetes. I also tested
> with the test case mentioned in moby github issue and the kernel freeze
> was not seen. Config file is https://paste.debian.net/1039104/
> 
> root@debian:~# uname -a
> Linux debian 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02)
> x86_64 GNU/Linux
> 
>> Can you send your full config?  If I get some time to try it, I'd like
>> to see if I can reproduce your issue.
> 
> Thanks. Config file is https://paste.debian.net/1039103/ (output of
> 'grep "=[y|m]" config-4.14.59-rt37')
> This issue was reproduced with 4.9.0-6-rt-amd64 kernel also.
> 
> root@debian:~# uname -a
> Linux debian 4.14.59-rt37 #1 SMP PREEMPT RT Fri Aug 24 00:31:29 UTC 2018
> x86_64 GNU/Linux
> 
> Below are the steps to reproduce the issue,
> Install docker using,
> curl
> https://raw.githubusercontent.com/rancher/install-docker/master/17.03.2.sh
> | /bin/bash
> 
> Run
> "for f in $(seq 1 10000);do docker run -it --rm ubuntu echo $f; done"
> consecutively in 3 terminals.
> 
> The freeze/lockup occurs sporadically.
> 
> The other test case is to run docker with kubernetes.
> 
> Regards,
> Vignesh
>