Re: [lxc-devel] segfault on shutdown if containers running

Re: [lxc-devel] segfault on shutdown if containers running

[Daniel Lezcano]

Adam Majer wrote:
> Daniel Lezcano wrote:
>>> Is this a known problem? If not, I'll provide the backtrace.
>> Yes, please, maybe I am missing something but I was not able to
>> reproduce it.
> 
> I see to be failing at being able to safe this segfault. The computer
> has no serial port so I've been trying to get netconsole to work, and it
> does, but the entire OOPS is only visible on local console, not
> transmitted. Only part of the oops is transmitted :(
> 
> 
> [  635.526580] BUG: unable to handle kernel NULL pointer dereference at
> 000000000000009c
> [  635.526692] IP: [<ffffffff803b26f1>] tty_open+0x1e6/0x411
> [  635.526775] PGD 13dd44067 PUD 1379d5067 PMD 0
> [  635.526905] Oops: 0000 [#1] SMP
> [  635.526999] last sysfs file: /sys/devices/virtual/net/lo/operstate
> [  635.527045] CPU 1
> [  635.527110] Modules linked in: netconsole configfs veth nvidia(P)
> binfmt_misc powernow_k8 cpufreq_powersave cpufreq_conservative
> cpufreq_stats cpufreq_userspace tun bridge stp ext2 mbcache it87
> hwmon_vid loop firewire_sbp2 snd_hda_codec_analog snd_hda_intel
> snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi
> snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usblp
> k8temp i2c_nforce2 pcspkr snd evdev asus_atk0110 i2c_core soundcore
> snd_page_alloc button processor xfs exportfs raid1 md_mod
> 
> 
> 
> So, I've copied the stacktrace on paper and typing it back,

Thanks for taking the time to do that !

> Process getty (pid:3940) <<<-- looks like a process in the container?
> 
> chrdev_open + 0x148/0x167
> chrdev_open + 0x0/0x167
> __dentry_open + 0x148/0x260
> do_flip_open + 0x468/0x85a
> alloc_fd + ....
> do_sys_gen + ...
> system_call_fastpath + ....
> 
> RIP tty_open

This stack trace reminds me something :)
Let me check if I can reproduce it with a small program.

Suka ? Are you aware of this bug ?

Thanks
   -- Daniel

Re: [lxc-devel] segfault on shutdown if containers running

[Daniel Lezcano]

Adam Majer wrote:
>>>>> chrdev_open + 0x148/0x167
>>>>> chrdev_open + 0x0/0x167
>>>>> __dentry_open + 0x148/0x260
>>>>> do_flip_open + 0x468/0x85a
>>>>> alloc_fd + ....
>>>>> do_sys_gen + ...
>>>>> system_call_fastpath + ....
>>>>>
>>>>> RIP tty_open
>>>> which kernel are you using ? could you run :
>>>>
>>>>     $ addr2line -e <vmlinux> ffffffff803b26f1
> 
>>> Would setting this to yes produce a backtrace with line numbers?
>> Yes and maybe CONFIG_FRAME_POINTER too.
> 
> Well, adding symbols didn't add line numbers to the backtrace. But now I
> can use addr2line with the vmlinux (all 78mb of it) to get you guys a
> backtrace with line numbers :)
> 
> 
> So,
> 
> OOPS (NULL pointer deference) at
> 
>   drivers/char/tty_io.c:1321  (tty_open)
> 
> Backtrace,
> 
>   fs/char_dev.c:397 (chrdev_open)
>   fs/char_dev.c:357 (chrdev_open)
>   fs/open.c:841     (__dentry_open)
>   arch/x86/include/asm/atomic_64.h:117 (do_flip_open)
>   fs/file.c:459     (alloc_fd)
>   ...

Excellent !

Suka ? Isn't this oops related to the newpts instance ?

Thanks
   -- Daniel

Re: [lxc-devel] segfault on shutdown if containers running

[Sukadev Bhattiprolu]

Daniel Lezcano [dlezcano-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
> Adam Majer wrote:
>>>>>> chrdev_open + 0x148/0x167
>>>>>> chrdev_open + 0x0/0x167
>>>>>> __dentry_open + 0x148/0x260
>>>>>> do_flip_open + 0x468/0x85a
>>>>>> alloc_fd + ....
>>>>>> do_sys_gen + ...
>>>>>> system_call_fastpath + ....
>>>>>>
>>>>>> RIP tty_open
>>>>> which kernel are you using ? could you run :
>>>>>
>>>>>     $ addr2line -e <vmlinux> ffffffff803b26f1
>>>> Would setting this to yes produce a backtrace with line numbers?
>>> Yes and maybe CONFIG_FRAME_POINTER too.
>> Well, adding symbols didn't add line numbers to the backtrace. But now I
>> can use addr2line with the vmlinux (all 78mb of it) to get you guys a
>> backtrace with line numbers :)
>> So,
>> OOPS (NULL pointer deference) at
>>   drivers/char/tty_io.c:1321  (tty_open)
>> Backtrace,
>>   fs/char_dev.c:397 (chrdev_open)
>>   fs/char_dev.c:357 (chrdev_open)
>>   fs/open.c:841     (__dentry_open)
>>   arch/x86/include/asm/atomic_64.h:117 (do_flip_open)
>>   fs/file.c:459     (alloc_fd)
>>   ...
>
> Excellent !
>
> Suka ? Isn't this oops related to the newpts instance ?
>


What version of the kernel are you running ? 

Could it be this bug: http://lkml.org/lkml/2009/1/26/274
It was fixed by following commit and should be in 2.6.29.

	commit 808ffa3d302257b9dc37b1412c1fcdf976fcddac
	Author: Eric Paris <eparis-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
	Date:   Tue Jan 27 11:50:37 2009 +0000

Sukadev

Re: [lxc-devel] segfault on shutdown if containers running

[Adam Majer]

Sukadev Bhattiprolu wrote:
> Daniel Lezcano [dlezcano-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org] wrote:
>> Adam Majer wrote:
>>>>>>> chrdev_open + 0x148/0x167
>>>>>>> chrdev_open + 0x0/0x167
>>>>>>> __dentry_open + 0x148/0x260
>>>>>>> do_flip_open + 0x468/0x85a
>>>>>>> alloc_fd + ....
>>>>>>> do_sys_gen + ...
>>>>>>> system_call_fastpath + ....
>>>>>>>
>>>>>>> RIP tty_open
>>>>>> which kernel are you using ? could you run :
>>>>>>
>>>>>>     $ addr2line -e <vmlinux> ffffffff803b26f1
>>>>> Would setting this to yes produce a backtrace with line numbers?
>>>> Yes and maybe CONFIG_FRAME_POINTER too.
>>> Well, adding symbols didn't add line numbers to the backtrace. But now I
>>> can use addr2line with the vmlinux (all 78mb of it) to get you guys a
>>> backtrace with line numbers :)
>>> So,
>>> OOPS (NULL pointer deference) at
>>>   drivers/char/tty_io.c:1321  (tty_open)
>>> Backtrace,
>>>   fs/char_dev.c:397 (chrdev_open)
>>>   fs/char_dev.c:357 (chrdev_open)
>>>   fs/open.c:841     (__dentry_open)
>>>   arch/x86/include/asm/atomic_64.h:117 (do_flip_open)
>>>   fs/file.c:459     (alloc_fd)
>>>   ...
>> Excellent !
>>
>> Suka ? Isn't this oops related to the newpts instance ?
>>
> 
> 
> What version of the kernel are you running ? 
> 
> Could it be this bug: http://lkml.org/lkml/2009/1/26/274
> It was fixed by following commit and should be in 2.6.29.
> 
> 	commit 808ffa3d302257b9dc37b1412c1fcdf976fcddac
> 	Author: Eric Paris <eparis-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> 	Date:   Tue Jan 27 11:50:37 2009 +0000
> 
> Sukadev

It's a Debian stock 2.6.30-1 kernel, so the 2.6.30. The original OOPS
was on the AMD64, but I've also just had a very similar OOPS in a 686
(32-bit) version of the same kernel.

The original oops (above) occurred when I did,

  lxc-start -n container &
  poweroff (in root system)

segfault occurred each time after the prompt "attempting to kill all
processes...." then oops then a "failed" from the cleanup process. If
the container is shut down prior to poweroff, no oops.


The following is all a 686 2.6.30 stock debian kernel,

Another OOPS (same function, slightly different backtrace),

  1. ssh to root machine
  2. lxc-start -n container &
  3. lxc-console -n container -t 1 &
  4. close ssh on originating machine
  5. re-ssh into the machine

at this point I had a console into the container and to the root machine
(where I ssh in) that were intermixed. Then the segfault occurred with
the following backtrace. Debugging info not compiled into it, but it is
similar symbols,

  IP: tty_open + 0x1a5
      chrdev_open
      dput
      chrdev_open
      __dentry_open
      ....

Again, I had a prompt from within the container and from the parent
machine when I re-ssh into the parent machine. So some ttys got intermixed.


I've also managed to generate a hard lockup of the kernel from within
the container, but that seems to be related to networking. The process
for that was,

  1. bridge + veth interface
  2. start container and networking up from within container
  3. ssh into container
  4. ifdown in cantainer
  5. at this point the ssh session remained in spite of container
networking down!!
  6. ifup container (new IP)
  7. hard lockup occurred with the backtrace scrolled way off screen


- Adam

PS. Running lxc-checkconfig on the Debian kernel yields,

Found kernel config file /boot/config-2.6.30-1-686
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup namespace: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: disabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled

I can provide you with the complete kernel config if you want it.