* Re: [RFC][PATCH 0/7] nested sleeps, fixes and debug infra
[not found] <20140804103025.478913141@infradead.org>
@ 2014-08-05 8:33 ` Ilya Dryomov
2014-08-05 13:06 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Ilya Dryomov @ 2014-08-05 8:33 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Ingo Molnar, oleg, Linus Torvalds, tglx, Mike Galbraith,
Linux Kernel Mailing List, netdev, linux-mm
On Mon, Aug 4, 2014 at 2:30 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> Hi,
>
> Ilya recently tripped over a nested sleep which made Ingo suggest we should
> have debug checks for that. So I did some, see patch 7. Of course that
> triggered a whole bunch of fail the instant I tried to boot my machine.
>
> With this series I can boot my test box and build a kernel on it, I'm fairly
> sure that's far too limited a test to have found all, but its a start.
FWIW, I'm getting a lot of these during light rbd testing. CC'ed
netdev and linux-mm.
WARNING: CPU: 2 PID: 1978 at kernel/sched/core.c:7094 __might_sleep+0x5b/0x1e0()
do not call blocking ops when !TASK_RUNNING; state=1 set at
[<ffffffff81070640>] prepare_to_wait+0x50/0xa0
Modules linked in:
CPU: 2 PID: 1978 Comm: ceph-osd Not tainted 3.16.0-vm+ #109
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
0000000000001bb6 ffff8800126739e8 ffffffff8156ec1d 0000000000000000
ffff880012673a38 ffff880012673a28 ffffffff81032c27 ffff880012673a58
0000000000000200 ffff8800150fa060 00000000000007ad ffffffff817ed352
Call Trace:
[<ffffffff8156ec1d>] dump_stack+0x4f/0x7c
[<ffffffff81032c27>] warn_slowpath_common+0x87/0xb0
[<ffffffff81032cf1>] warn_slowpath_fmt+0x41/0x50
[<ffffffff814f23cf>] ? tcp_v4_do_rcv+0x10f/0x4a0
[<ffffffff81070640>] ? prepare_to_wait+0x50/0xa0
[<ffffffff81070640>] ? prepare_to_wait+0x50/0xa0
[<ffffffff8105b53b>] __might_sleep+0x5b/0x1e0
[<ffffffff8148d73d>] release_sock+0x13d/0x200
[<ffffffff81498223>] sk_stream_wait_memory+0x133/0x2d0
[<ffffffff810701d0>] ? woken_wake_function+0x10/0x10
[<ffffffff814dfdbf>] tcp_sendmsg+0xb6f/0xd70
[<ffffffff815096cf>] inet_sendmsg+0xdf/0x100
[<ffffffff815095f0>] ? inet_recvmsg+0x100/0x100
[<ffffffff814896d7>] sock_sendmsg+0x67/0x90
[<ffffffff810fd961>] ? might_fault+0x51/0xb0
[<ffffffff81489a22>] ___sys_sendmsg+0x2d2/0x2e0
[<ffffffff81095e58>] ? futex_wake+0x128/0x140
[<ffffffff81095d31>] ? futex_wake+0x1/0x140
[<ffffffff81141dd0>] ? do_dup2+0xd0/0xd0
[<ffffffff8105fa31>] ? get_parent_ip+0x11/0x50
[<ffffffff813cea27>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff813c33c5>] ? delay_tsc+0x85/0xb0
[<ffffffff81141ead>] ? __fget+0xdd/0xf0
[<ffffffff81141dd0>] ? do_dup2+0xd0/0xd0
[<ffffffff81141f05>] ? __fget_light+0x45/0x60
[<ffffffff81141f2e>] ? __fdget+0xe/0x10
[<ffffffff8148a4e4>] __sys_sendmsg+0x44/0x70
[<ffffffff8148a519>] SyS_sendmsg+0x9/0x10
[<ffffffff81575b92>] system_call_fastpath+0x16/0x1b
WARNING: CPU: 0 PID: 380 at kernel/sched/core.c:7094 __might_sleep+0x5b/0x1e0()
do not call blocking ops when !TASK_RUNNING; state=1 set at
[<ffffffff81070640>] prepare_to_wait+0x50/0xa0
Modules linked in:
CPU: 0 PID: 380 Comm: kswapd0 Tainted: G W 3.16.0-vm+ #109
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
0000000000001bb6 ffff88007b64bc68 ffffffff8156ec1d 0000000000000000
ffff88007b64bcb8 ffff88007b64bca8 ffffffff81032c27 0000000000000000
0000000000000000 ffff88007c062060 0000000000000065 ffffffff8179ca1f
Call Trace:
[<ffffffff8156ec1d>] dump_stack+0x4f/0x7c
[<ffffffff81032c27>] warn_slowpath_common+0x87/0xb0
[<ffffffff81032cf1>] warn_slowpath_fmt+0x41/0x50
[<ffffffff81070640>] ? prepare_to_wait+0x50/0xa0
[<ffffffff81070640>] ? prepare_to_wait+0x50/0xa0
[<ffffffff8105b53b>] __might_sleep+0x5b/0x1e0
[<ffffffff810f7fd3>] __reset_isolation_suitable+0x83/0x140
[<ffffffff810f83f3>] reset_isolation_suitable+0x33/0x50
[<ffffffff810eb717>] kswapd+0x2e7/0x4d0
[<ffffffff810701d0>] ? woken_wake_function+0x10/0x10
[<ffffffff810eb430>] ? balance_pgdat+0x5b0/0x5b0
[<ffffffff810539ab>] kthread+0xfb/0x110
[<ffffffff810538b0>] ? flush_kthread_worker+0x130/0x130
[<ffffffff81575aec>] ret_from_fork+0x7c/0xb0
[<ffffffff810538b0>] ? flush_kthread_worker+0x130/0x130
Thanks,
Ilya
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC][PATCH 0/7] nested sleeps, fixes and debug infra
2014-08-05 8:33 ` [RFC][PATCH 0/7] nested sleeps, fixes and debug infra Ilya Dryomov
@ 2014-08-05 13:06 ` Peter Zijlstra
2014-08-06 7:51 ` Ilya Dryomov
0 siblings, 1 reply; 6+ messages in thread
From: Peter Zijlstra @ 2014-08-05 13:06 UTC (permalink / raw)
To: Ilya Dryomov
Cc: Ingo Molnar, oleg, Linus Torvalds, tglx, Mike Galbraith,
Linux Kernel Mailing List, netdev, linux-mm
[-- Attachment #1: Type: text/plain, Size: 3776 bytes --]
On Tue, Aug 05, 2014 at 12:33:16PM +0400, Ilya Dryomov wrote:
> On Mon, Aug 4, 2014 at 2:30 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> > Hi,
> >
> > Ilya recently tripped over a nested sleep which made Ingo suggest we should
> > have debug checks for that. So I did some, see patch 7. Of course that
> > triggered a whole bunch of fail the instant I tried to boot my machine.
> >
> > With this series I can boot my test box and build a kernel on it, I'm fairly
> > sure that's far too limited a test to have found all, but its a start.
>
> FWIW, I'm getting a lot of these during light rbd testing. CC'ed
> netdev and linux-mm.
Both are cond_resched() calls, and that's not blocking as such, just a
preemption point, so lets exclude those.
From the school of '_' are free:
---
include/linux/kernel.h | 3 +++
include/linux/sched.h | 6 +++---
kernel/sched/core.c | 12 +++++++++---
3 files changed, 15 insertions(+), 6 deletions(-)
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -162,6 +162,7 @@ extern int _cond_resched(void);
#endif
#ifdef CONFIG_DEBUG_ATOMIC_SLEEP
+ void ___might_sleep(const char *file, int line, int preempt_offset);
void __might_sleep(const char *file, int line, int preempt_offset);
/**
* might_sleep - annotation for functions that can sleep
@@ -176,6 +177,8 @@ extern int _cond_resched(void);
# define might_sleep() \
do { __might_sleep(__FILE__, __LINE__, 0); might_resched(); } while (0)
#else
+ static inline void ___might_sleep(const char *file, int line,
+ int preempt_offset) { }
static inline void __might_sleep(const char *file, int line,
int preempt_offset) { }
# define might_sleep() do { might_resched(); } while (0)
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2754,7 +2754,7 @@ static inline int signal_pending_state(l
extern int _cond_resched(void);
#define cond_resched() ({ \
- __might_sleep(__FILE__, __LINE__, 0); \
+ ___might_sleep(__FILE__, __LINE__, 0); \
_cond_resched(); \
})
@@ -2767,14 +2767,14 @@ extern int __cond_resched_lock(spinlock_
#endif
#define cond_resched_lock(lock) ({ \
- __might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET); \
+ ___might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET);\
__cond_resched_lock(lock); \
})
extern int __cond_resched_softirq(void);
#define cond_resched_softirq() ({ \
- __might_sleep(__FILE__, __LINE__, SOFTIRQ_DISABLE_OFFSET); \
+ ___might_sleep(__FILE__, __LINE__, SOFTIRQ_DISABLE_OFFSET); \
__cond_resched_softirq(); \
})
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7078,8 +7078,6 @@ static inline int preempt_count_equals(i
void __might_sleep(const char *file, int line, int preempt_offset)
{
- static unsigned long prev_jiffy; /* ratelimiting */
-
/*
* Blocking primitives will set (and therefore destroy) current->state,
* since we will exit with TASK_RUNNING make sure we enter with it,
@@ -7093,6 +7091,14 @@ void __might_sleep(const char *file, int
(void *)current->task_state_change))
__set_current_state(TASK_RUNNING);
+ ___might_sleep(file, line, preempt_offset);
+}
+EXPORT_SYMBOL(__might_sleep);
+
+void ___might_sleep(const char *file, int line, int preempt_offset)
+{
+ static unsigned long prev_jiffy; /* ratelimiting */
+
rcu_sleep_check(); /* WARN_ON_ONCE() by default, no rate limit reqd. */
if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
!is_idle_task(current)) ||
@@ -7122,7 +7128,7 @@ void __might_sleep(const char *file, int
#endif
dump_stack();
}
-EXPORT_SYMBOL(__might_sleep);
+EXPORT_SYMBOL(___might_sleep);
#endif
#ifdef CONFIG_MAGIC_SYSRQ
[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC][PATCH 0/7] nested sleeps, fixes and debug infra
2014-08-05 13:06 ` Peter Zijlstra
@ 2014-08-06 7:51 ` Ilya Dryomov
2014-08-06 8:31 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: Ilya Dryomov @ 2014-08-06 7:51 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Ingo Molnar, oleg, Linus Torvalds, tglx, Mike Galbraith,
Linux Kernel Mailing List, netdev, linux-mm
On Tue, Aug 5, 2014 at 5:06 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Tue, Aug 05, 2014 at 12:33:16PM +0400, Ilya Dryomov wrote:
>> On Mon, Aug 4, 2014 at 2:30 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> > Hi,
>> >
>> > Ilya recently tripped over a nested sleep which made Ingo suggest we should
>> > have debug checks for that. So I did some, see patch 7. Of course that
>> > triggered a whole bunch of fail the instant I tried to boot my machine.
>> >
>> > With this series I can boot my test box and build a kernel on it, I'm fairly
>> > sure that's far too limited a test to have found all, but its a start.
>>
>> FWIW, I'm getting a lot of these during light rbd testing. CC'ed
>> netdev and linux-mm.
>
> Both are cond_resched() calls, and that's not blocking as such, just a
> preemption point, so lets exclude those.
OK, this one is a bit different.
WARNING: CPU: 1 PID: 1744 at kernel/sched/core.c:7104 __might_sleep+0x58/0x90()
do not call blocking ops when !TASK_RUNNING; state=1 set at
[<ffffffff81070e10>] prepare_to_wait+0x50 /0xa0
Modules linked in:
CPU: 1 PID: 1744 Comm: lt-ceph_test_li Not tainted 3.16.0-vm+ #113
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
0000000000001bc0 ffff88006c4479d8 ffffffff8156f455 0000000000000000
ffff88006c447a28 ffff88006c447a18 ffffffff81033357 0000000000000001
0000000000000000 0000000000000950 ffffffff817ee48a ffff88006dba6120
Call Trace:
[<ffffffff8156f455>] dump_stack+0x4f/0x7c
[<ffffffff81033357>] warn_slowpath_common+0x87/0xb0
[<ffffffff81033421>] warn_slowpath_fmt+0x41/0x50
[<ffffffff81078bb2>] ? trace_hardirqs_on_caller+0x182/0x1f0
[<ffffffff81070e10>] ? prepare_to_wait+0x50/0xa0
[<ffffffff81070e10>] ? prepare_to_wait+0x50/0xa0
[<ffffffff8105bc38>] __might_sleep+0x58/0x90
[<ffffffff8148c671>] lock_sock_nested+0x31/0xb0
[<ffffffff8148dfeb>] ? release_sock+0x1bb/0x200
[<ffffffff81498aaa>] sk_stream_wait_memory+0x18a/0x2d0
[<ffffffff810709a0>] ? woken_wake_function+0x10/0x10
[<ffffffff814e058f>] tcp_sendmsg+0xb6f/0xd70
[<ffffffff81509e9f>] inet_sendmsg+0xdf/0x100
[<ffffffff81509dc0>] ? inet_recvmsg+0x100/0x100
[<ffffffff81489f07>] sock_sendmsg+0x67/0x90
[<ffffffff810fe391>] ? might_fault+0x51/0xb0
[<ffffffff8148a252>] ___sys_sendmsg+0x2d2/0x2e0
[<ffffffff811428a0>] ? do_dup2+0xd0/0xd0
[<ffffffff811428a0>] ? do_dup2+0xd0/0xd0
[<ffffffff8105bfe0>] ? finish_task_switch+0x50/0x100
[<ffffffff811429d5>] ? __fget_light+0x45/0x60
[<ffffffff811429fe>] ? __fdget+0xe/0x10
[<ffffffff8148ad14>] __sys_sendmsg+0x44/0x70
[<ffffffff8148ad49>] SyS_sendmsg+0x9/0x10
[<ffffffff815764d2>] system_call_fastpath+0x16/0x1b
Thanks,
Ilya
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC][PATCH 0/7] nested sleeps, fixes and debug infra
2014-08-06 7:51 ` Ilya Dryomov
@ 2014-08-06 8:31 ` Peter Zijlstra
2014-08-06 21:16 ` David Miller
0 siblings, 1 reply; 6+ messages in thread
From: Peter Zijlstra @ 2014-08-06 8:31 UTC (permalink / raw)
To: Ilya Dryomov
Cc: Ingo Molnar, oleg, Linus Torvalds, tglx, Mike Galbraith,
Linux Kernel Mailing List, netdev, linux-mm
[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]
On Wed, Aug 06, 2014 at 11:51:29AM +0400, Ilya Dryomov wrote:
> OK, this one is a bit different.
>
> WARNING: CPU: 1 PID: 1744 at kernel/sched/core.c:7104 __might_sleep+0x58/0x90()
> do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff81070e10>] prepare_to_wait+0x50 /0xa0
> [<ffffffff8105bc38>] __might_sleep+0x58/0x90
> [<ffffffff8148c671>] lock_sock_nested+0x31/0xb0
> [<ffffffff81498aaa>] sk_stream_wait_memory+0x18a/0x2d0
Urgh, tedious. Its not an actual bug as is. Due to the condition check
in sk_wait_event() we can call lock_sock() with ->state != TASK_RUNNING.
I'm not entirely sure what the cleanest way is to make this go away.
Possibly something like so:
---
include/net/sock.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/include/net/sock.h b/include/net/sock.h
index 156350745700..37902176c5ab 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -886,6 +886,7 @@ static inline void sock_rps_reset_rxhash(struct sock *sk)
if (!__rc) { \
*(__timeo) = schedule_timeout(*(__timeo)); \
} \
+ __set_current_state(TASK_RUNNING); \
lock_sock(__sk); \
__rc = __condition; \
__rc; \
[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [RFC][PATCH 0/7] nested sleeps, fixes and debug infra
2014-08-06 8:31 ` Peter Zijlstra
@ 2014-08-06 21:16 ` David Miller
2014-08-07 8:10 ` Peter Zijlstra
0 siblings, 1 reply; 6+ messages in thread
From: David Miller @ 2014-08-06 21:16 UTC (permalink / raw)
To: peterz
Cc: ilya.dryomov, mingo, oleg, torvalds, tglx, umgwanakikbuti,
linux-kernel, netdev, linux-mm
From: Peter Zijlstra <peterz@infradead.org>
Date: Wed, 6 Aug 2014 10:31:34 +0200
> On Wed, Aug 06, 2014 at 11:51:29AM +0400, Ilya Dryomov wrote:
>
>> OK, this one is a bit different.
>>
>> WARNING: CPU: 1 PID: 1744 at kernel/sched/core.c:7104 __might_sleep+0x58/0x90()
>> do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff81070e10>] prepare_to_wait+0x50 /0xa0
>
>> [<ffffffff8105bc38>] __might_sleep+0x58/0x90
>> [<ffffffff8148c671>] lock_sock_nested+0x31/0xb0
>> [<ffffffff81498aaa>] sk_stream_wait_memory+0x18a/0x2d0
>
> Urgh, tedious. Its not an actual bug as is. Due to the condition check
> in sk_wait_event() we can call lock_sock() with ->state != TASK_RUNNING.
>
> I'm not entirely sure what the cleanest way is to make this go away.
> Possibly something like so:
If you submit this formally to netdev with a signoff I'm willing to apply
this if it helps the debug infrastructure.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC][PATCH 0/7] nested sleeps, fixes and debug infra
2014-08-06 21:16 ` David Miller
@ 2014-08-07 8:10 ` Peter Zijlstra
0 siblings, 0 replies; 6+ messages in thread
From: Peter Zijlstra @ 2014-08-07 8:10 UTC (permalink / raw)
To: David Miller
Cc: ilya.dryomov, mingo, oleg, torvalds, tglx, umgwanakikbuti,
linux-kernel, netdev, linux-mm
[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]
On Wed, Aug 06, 2014 at 02:16:03PM -0700, David Miller wrote:
> From: Peter Zijlstra <peterz@infradead.org>
> Date: Wed, 6 Aug 2014 10:31:34 +0200
>
> > On Wed, Aug 06, 2014 at 11:51:29AM +0400, Ilya Dryomov wrote:
> >
> >> OK, this one is a bit different.
> >>
> >> WARNING: CPU: 1 PID: 1744 at kernel/sched/core.c:7104 __might_sleep+0x58/0x90()
> >> do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff81070e10>] prepare_to_wait+0x50 /0xa0
> >
> >> [<ffffffff8105bc38>] __might_sleep+0x58/0x90
> >> [<ffffffff8148c671>] lock_sock_nested+0x31/0xb0
> >> [<ffffffff81498aaa>] sk_stream_wait_memory+0x18a/0x2d0
> >
> > Urgh, tedious. Its not an actual bug as is. Due to the condition check
> > in sk_wait_event() we can call lock_sock() with ->state != TASK_RUNNING.
> >
> > I'm not entirely sure what the cleanest way is to make this go away.
> > Possibly something like so:
>
> If you submit this formally to netdev with a signoff I'm willing to apply
> this if it helps the debug infrastructure.
Thanks, for now I'm just collecting things to see how far I can take
this. But I'll certainly include you and netdev on a next posting.
[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-08-07 8:10 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20140804103025.478913141@infradead.org>
2014-08-05 8:33 ` [RFC][PATCH 0/7] nested sleeps, fixes and debug infra Ilya Dryomov
2014-08-05 13:06 ` Peter Zijlstra
2014-08-06 7:51 ` Ilya Dryomov
2014-08-06 8:31 ` Peter Zijlstra
2014-08-06 21:16 ` David Miller
2014-08-07 8:10 ` Peter Zijlstra
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).