* Soft lockups during reading /proc/PID/smaps
@ 2014-07-31 7:22 Aleksei Besogonov
2014-07-31 7:43 ` David Rientjes
0 siblings, 1 reply; 6+ messages in thread
From: Aleksei Besogonov @ 2014-07-31 7:22 UTC (permalink / raw)
To: linux-kernel
I'm getting weird soft lockups while reading smaps on loaded systems with
some background cgroups usage. This issue can be reproduced with the most
recent kernel.
Here's the stack trace:
[ 1748.312052] BUG: soft lockup - CPU#6 stuck for 23s! [python2.7:1857]
[ 1748.312052] Modules linked in: xfs xt_addrtype xt_conntrack
iptable_filter ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc
dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nfsd
auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt psmouse serio_raw
ppdev parport_pc i2c_piix4 parport xen_fbfront fb_sys_fops syscopyarea
sysfillrect sysimgblt mac_hid isofs raid10 raid456 async_memcpy
async_raid6_recov async_pq async_xor async_tx xor raid6_pq raid1 raid0
multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd floppy
[ 1748.312052] CPU: 6 PID: 1857 Comm: python2.7 Not tainted
3.15.5-031505-generic #201407091543
[ 1748.312052] Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014
[ 1748.312052] task: ffff8800eab41930 ti: ffff8803b9a94000 task.ti:
ffff8803b9a94000
[ 1748.312052] RIP: 0010:[<ffffffff81013111>] [<ffffffff81013111>]
KSTK_ESP+0x11/0x40
[ 1748.312052] RSP: 0018:ffff8803b9a97c68 EFLAGS: 00000287
[ 1748.312052] RAX: 0000000000000000 RBX: ffff8803b60de3aa RCX: 00007f49ec000000
[ 1748.312052] RDX: 0000000000000001 RSI: ffff8800eba1c730 RDI: ffff880399434b90
[ 1748.312052] RBP: ffff8803b9a97c68 R08: 000000000000000a R09: 000000000000fffe
[ 1748.312052] R10: 0000000000000000 R11: 0000000000000007 R12: 0000000000000001
[ 1748.312052] R13: ffff8803b7a74108 R14: 00007f49ec021000 R15: ffff8803b9a9ffff
[ 1748.312052] FS: 00007fcc9562b740(0000) GS:ffff8803cfcc0000(0000)
knlGS:0000000000000000
[ 1748.312052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1748.312052] CR2: 00007fcc955ed180 CR3: 00000003b97a5000 CR4: 00000000001406e0
[ 1748.312052] Stack:
[ 1748.312052] ffff8803b9a97ca8 ffffffff8117bcc9 ffff8803b9a97df8
ffff880399435030
[ 1748.312052] ffff88003684d000 ffff8800eba1c730 0000000000000000
ffff8803b5dfc980
[ 1748.312052] ffff8803b9a97d38 ffffffff81236612 ffff88030000002d
00007f4900000070
[ 1748.312052] Call Trace:
[ 1748.312052] [<ffffffff8117bcc9>] vm_is_stack+0x59/0xe0
[ 1748.312052] [<ffffffff81236612>] show_map_vma+0x212/0x280
[ 1748.312052] [<ffffffff81236805>] show_smap+0x85/0x250
[ 1748.312052] [<ffffffff81237bc0>] ? smaps_pte_entry.isra.21+0x220/0x220
[ 1748.312052] [<ffffffff81236a03>] show_pid_smap+0x13/0x20
[ 1748.312052] [<ffffffff811f6016>] seq_read+0x256/0x3e0
[ 1748.312052] [<ffffffff811d30e1>] vfs_read+0xb1/0x180
[ 1748.312052] [<ffffffff811d335f>] SyS_read+0x4f/0xb0
[ 1748.312052] [<ffffffff8178527f>] tracesys+0xe1/0xe6
[ 1748.312052] Code: 89 f8 48 89 f2 89 c6 48 89 e5 65 48 8b 3c 25 c0 c7 00
00 e8 b2 f9 ff ff 5d c3 0f 1f 44 00 00 48 8b 47 08 55 48 89 e5 48 8b 40 10
<a9> 00 00 02 00 75 10 48 8b 87 10 06 00 00 5d c3 0f 1f 80 00 00
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Soft lockups during reading /proc/PID/smaps
2014-07-31 7:22 Soft lockups during reading /proc/PID/smaps Aleksei Besogonov
@ 2014-07-31 7:43 ` David Rientjes
2014-07-31 11:13 ` Aleksei Besogonov
0 siblings, 1 reply; 6+ messages in thread
From: David Rientjes @ 2014-07-31 7:43 UTC (permalink / raw)
To: Aleksei Besogonov, Oleg Nesterov; +Cc: linux-kernel
On Thu, 31 Jul 2014, Aleksei Besogonov wrote:
> I'm getting weird soft lockups while reading smaps on loaded systems with
> some background cgroups usage. This issue can be reproduced with the most
> recent kernel.
>
> Here's the stack trace:
> [ 1748.312052] BUG: soft lockup - CPU#6 stuck for 23s! [python2.7:1857]
> [ 1748.312052] Modules linked in: xfs xt_addrtype xt_conntrack
> iptable_filter ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc
> dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nfsd
> auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt psmouse serio_raw
> ppdev parport_pc i2c_piix4 parport xen_fbfront fb_sys_fops syscopyarea
> sysfillrect sysimgblt mac_hid isofs raid10 raid456 async_memcpy
> async_raid6_recov async_pq async_xor async_tx xor raid6_pq raid1 raid0
> multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
> aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd floppy
> [ 1748.312052] CPU: 6 PID: 1857 Comm: python2.7 Not tainted
> 3.15.5-031505-generic #201407091543
This isn't the most recent kernel, we're at 3.16-rc7 now, but I don't
think there are any changes that would prevent this.
> [ 1748.312052] Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014
> [ 1748.312052] task: ffff8800eab41930 ti: ffff8803b9a94000 task.ti:
> ffff8803b9a94000
> [ 1748.312052] RIP: 0010:[<ffffffff81013111>] [<ffffffff81013111>]
> KSTK_ESP+0x11/0x40
> [ 1748.312052] RSP: 0018:ffff8803b9a97c68 EFLAGS: 00000287
> [ 1748.312052] RAX: 0000000000000000 RBX: ffff8803b60de3aa RCX: 00007f49ec000000
> [ 1748.312052] RDX: 0000000000000001 RSI: ffff8800eba1c730 RDI: ffff880399434b90
> [ 1748.312052] RBP: ffff8803b9a97c68 R08: 000000000000000a R09: 000000000000fffe
> [ 1748.312052] R10: 0000000000000000 R11: 0000000000000007 R12: 0000000000000001
> [ 1748.312052] R13: ffff8803b7a74108 R14: 00007f49ec021000 R15: ffff8803b9a9ffff
> [ 1748.312052] FS: 00007fcc9562b740(0000) GS:ffff8803cfcc0000(0000)
> knlGS:0000000000000000
> [ 1748.312052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 1748.312052] CR2: 00007fcc955ed180 CR3: 00000003b97a5000 CR4: 00000000001406e0
> [ 1748.312052] Stack:
> [ 1748.312052] ffff8803b9a97ca8 ffffffff8117bcc9 ffff8803b9a97df8
> ffff880399435030
> [ 1748.312052] ffff88003684d000 ffff8800eba1c730 0000000000000000
> ffff8803b5dfc980
> [ 1748.312052] ffff8803b9a97d38 ffffffff81236612 ffff88030000002d
> 00007f4900000070
> [ 1748.312052] Call Trace:
> [ 1748.312052] [<ffffffff8117bcc9>] vm_is_stack+0x59/0xe0
> [ 1748.312052] [<ffffffff81236612>] show_map_vma+0x212/0x280
> [ 1748.312052] [<ffffffff81236805>] show_smap+0x85/0x250
> [ 1748.312052] [<ffffffff81237bc0>] ? smaps_pte_entry.isra.21+0x220/0x220
> [ 1748.312052] [<ffffffff81236a03>] show_pid_smap+0x13/0x20
> [ 1748.312052] [<ffffffff811f6016>] seq_read+0x256/0x3e0
> [ 1748.312052] [<ffffffff811d30e1>] vfs_read+0xb1/0x180
> [ 1748.312052] [<ffffffff811d335f>] SyS_read+0x4f/0xb0
> [ 1748.312052] [<ffffffff8178527f>] tracesys+0xe1/0xe6
The while_each_thread() in vm_is_stack() looks suspicious since the task
isn't current and rcu won't protect the iteration, and we also don't hold
sighand lock or a readlock on tasklist_lock.
I think Oleg will know how to proceed, cc'd.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Soft lockups during reading /proc/PID/smaps
2014-07-31 7:43 ` David Rientjes
@ 2014-07-31 11:13 ` Aleksei Besogonov
2014-08-02 18:19 ` Oleg Nesterov
0 siblings, 1 reply; 6+ messages in thread
From: Aleksei Besogonov @ 2014-07-31 11:13 UTC (permalink / raw)
To: David Rientjes; +Cc: Oleg Nesterov, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]
On 31 Jul 2014, at 00:43, David Rientjes <rientjes@google.com> wrote:
> On Thu, 31 Jul 2014, Aleksei Besogonov wrote:
>> I'm getting weird soft lockups while reading smaps on loaded systems with
>> some background cgroups usage. This issue can be reproduced with the most
>> recent kernel.
>>
>> Here's the stack trace:
>> [ 1748.312052] BUG: soft lockup - CPU#6 stuck for 23s! [python2.7:1857]
>> [ 1748.312052] Modules linked in: xfs xt_addrtype xt_conntrack
>> iptable_filter ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>> nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables bridge stp llc
>> dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nfsd
>> auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt psmouse serio_raw
>> ppdev parport_pc i2c_piix4 parport xen_fbfront fb_sys_fops syscopyarea
>> sysfillrect sysimgblt mac_hid isofs raid10 raid456 async_memcpy
>> async_raid6_recov async_pq async_xor async_tx xor raid6_pq raid1 raid0
>> multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
>> aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd floppy
>> [ 1748.312052] CPU: 6 PID: 1857 Comm: python2.7 Not tainted
>> 3.15.5-031505-generic #201407091543
> This isn't the most recent kernel, we're at 3.16-rc7 now, but I don't
> think there are any changes that would prevent this.
Yes, I tested it with the rc7, the error report is from a previous run with an Ubuntu 14.04 kernel.
> The while_each_thread() in vm_is_stack() looks suspicious since the task
> isn't current and rcu won't protect the iteration, and we also don't hold
> sighand lock or a readlock on tasklist_lock.
> I think Oleg will know how to proceed, cc'd.
I’m attaching a minimal test case that can reproduce the issue. Works in 100% cases on any system I’ve tried.
[-- Attachment #2: bug.py --]
[-- Type: text/x-python-script, Size: 1149 bytes --]
#!/usr/bin/env python2.7
from os import mkdir
from threading import Thread
from time import sleep
import os
__author__ = 'cyberax'
count = 0
def threadproc():
global count
count += 1
sleep(0.01)
count -= 1
def do_threads():
sleep(2)
while True:
while count > 200:
sleep(0.01)
th = Thread(target=threadproc)
th.start()
def do_reader(pid):
while True:
with open("/sys/fs/cgroup/memory/ck/1001/tasks", "r") as fl:
fl.readlines()
with open("/sys/fs/cgroup/memory/ck/1001/delegate/tasks", "r") as fl:
lines = fl.readlines()
for l in lines:
try:
with open("/proc/%s/smaps" % l.strip(), "r") as fl:
fl.readlines()
except:
pass
pid = os.fork()
if pid == 0:
do_threads()
exit(0)
try:
mkdir('/sys/fs/cgroup/memory/ck')
mkdir('/sys/fs/cgroup/memory/ck/1001')
mkdir('/sys/fs/cgroup/memory/ck/1001/delegate')
except:
pass
with open('/sys/fs/cgroup/memory/ck/1001/delegate/tasks', 'w') as fl:
fl.write('%d\n' % pid)
do_reader(pid)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Soft lockups during reading /proc/PID/smaps
2014-07-31 11:13 ` Aleksei Besogonov
@ 2014-08-02 18:19 ` Oleg Nesterov
2014-08-03 12:03 ` Aleksei Besogonov
0 siblings, 1 reply; 6+ messages in thread
From: Oleg Nesterov @ 2014-08-02 18:19 UTC (permalink / raw)
To: Aleksei Besogonov; +Cc: David Rientjes, linux-kernel
On 07/31, Aleksei Besogonov wrote:
>
> On 31 Jul 2014, at 00:43, David Rientjes <rientjes@google.com> wrote:
>
> > The while_each_thread() in vm_is_stack() looks suspicious since the task
> > isn't current and rcu won't protect the iteration, and we also don't hold
> > sighand lock or a readlock on tasklist_lock.
> > I think Oleg will know how to proceed, cc'd.
> I’m attaching a minimal test case that can reproduce the issue. Works in 100% cases on any system I’ve tried.
Thanks. I think David is right and we need the simple patch below.
This reminds me I should kill while_each_thread :/
Any chance you can test it? If not, I will do this later and send
the patch if it helps.
Oleg.
--- x/mm/util.c
+++ x/mm/util.c
@@ -277,17 +277,14 @@ pid_t vm_is_stack(struct task_struct *ta
if (in_group) {
struct task_struct *t;
- rcu_read_lock();
- if (!pid_alive(task))
- goto done;
- t = task;
- do {
+ rcu_read_lock();
+ for_each_thread(task, t) {
if (vm_is_stack_for_task(t, vma)) {
ret = t->pid;
goto done;
}
- } while_each_thread(task, t);
+ }
done:
rcu_read_unlock();
}
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Soft lockups during reading /proc/PID/smaps
2014-08-02 18:19 ` Oleg Nesterov
@ 2014-08-03 12:03 ` Aleksei Besogonov
2014-08-04 19:26 ` Oleg Nesterov
0 siblings, 1 reply; 6+ messages in thread
From: Aleksei Besogonov @ 2014-08-03 12:03 UTC (permalink / raw)
To: Oleg Nesterov; +Cc: David Rientjes, linux-kernel
On 02 Aug 2014, at 11:19, Oleg Nesterov <oleg@redhat.com> wrote:
> On 07/31, Aleksei Besogonov wrote:
>>
>> On 31 Jul 2014, at 00:43, David Rientjes <rientjes@google.com> wrote:
>>
>>> The while_each_thread() in vm_is_stack() looks suspicious since the task
>>> isn't current and rcu won't protect the iteration, and we also don't hold
>>> sighand lock or a readlock on tasklist_lock.
>>> I think Oleg will know how to proceed, cc'd.
>> I’m attaching a minimal test case that can reproduce the issue. Works in 100% cases on any system I’ve tried.
> Thanks. I think David is right and we need the simple patch below.
> This reminds me I should kill while_each_thread :/
> Any chance you can test it? If not, I will do this later and send
> the patch if it helps.
Thanks, it works on the rc7 kernel.
I can make a backported version for earlier kernels if nobody else is interested.
>
> Oleg.
>
> --- x/mm/util.c
> +++ x/mm/util.c
> @@ -277,17 +277,14 @@ pid_t vm_is_stack(struct task_struct *ta
>
> if (in_group) {
> struct task_struct *t;
> - rcu_read_lock();
> - if (!pid_alive(task))
> - goto done;
>
> - t = task;
> - do {
> + rcu_read_lock();
> + for_each_thread(task, t) {
> if (vm_is_stack_for_task(t, vma)) {
> ret = t->pid;
> goto done;
> }
> - } while_each_thread(task, t);
> + }
> done:
> rcu_read_unlock();
> }
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Soft lockups during reading /proc/PID/smaps
2014-08-03 12:03 ` Aleksei Besogonov
@ 2014-08-04 19:26 ` Oleg Nesterov
0 siblings, 0 replies; 6+ messages in thread
From: Oleg Nesterov @ 2014-08-04 19:26 UTC (permalink / raw)
To: Aleksei Besogonov; +Cc: David Rientjes, linux-kernel
On 08/03, Aleksei Besogonov wrote:
>
> Thanks, it works on the rc7 kernel.
Great!
I didn't have time today, will write the changelog and send the patch
tomorrow.
Thanks a lot Aleksei.
Oleg.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-08-04 19:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-07-31 7:22 Soft lockups during reading /proc/PID/smaps Aleksei Besogonov
2014-07-31 7:43 ` David Rientjes
2014-07-31 11:13 ` Aleksei Besogonov
2014-08-02 18:19 ` Oleg Nesterov
2014-08-03 12:03 ` Aleksei Besogonov
2014-08-04 19:26 ` Oleg Nesterov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox