All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vineet Gupta <Vineet.Gupta1@synopsys.com>
To: Peter Hurley <peter@hurleysoftware.com>
Cc: lkml <linux-kernel@vger.kernel.org>,
	linux-serial@vger.kernel.org, Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Jiri Slaby <jslaby@suse.cz>
Subject: Re: n_tty_write() going into schedule but NOT coming out
Date: Tue, 2 Apr 2013 18:56:46 +0530	[thread overview]
Message-ID: <515ADC96.7090908@synopsys.com> (raw)
In-Reply-To: <1364829008.3617.21.camel@thor.lan>

On 04/01/2013 08:40 PM, Peter Hurley wrote:
> On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
>> Hi,
>>
>> Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
>> CONFIG_PREEMPT_COUNT
>>
>> -Vineet
>>
>> On 03/30/2013 06:05 PM, Vineet Gupta wrote:
>>> Hi,
>>>
>>> I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
>>> setup has 3 telnet sessions, each running find . -name "*" in a loop.
>>> The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
>>> has !SMP and PREEMPT_NONE.
>>>
>>> After ~10 mins of run, I see that one of the telnet session gets stuck (and later
>>> the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
>>>
>>> [ARCLinux]$ ps
>>> ....
>>>     7 root       0:00 inetd
>>>    62 root       0:00 -/bin/sh
>>>    64 root       1:34 telnetd -i -l /bin/sh
>>>    65 root       0:00 /bin/sh
>>>    75 root       1:47 telnetd -i -l /bin/sh
>>>    76 root       0:00 /bin/sh
>>>    79 root       0:53 telnetd -i -l /bin/sh
>>>    80 root       0:00 /bin/sh
>>>   281 root       0:00 find / -name *	<--- stuck
>>>   358 root       0:03 find / -name *	<--- stuck
>>>   377 root       0:00 find / -name *
>>>   378 root       0:00 ps
>>>
>>> Hung find task is sitting in the schedule() call in n_tty_write()
>>>
>>> [ARCLinux]$ cat /proc/281/stack
>>> [<8065945e>] n_tty_write+0x23a/0x424
>>> [<80655cd4>] tty_write+0x1ac/0x2d4
>>> [<805976ba>] vfs_write+0x92/0x110
>>> [<80597816>] sys_write+0x4e/0x88
>>> [<8050e780>] ret_from_system_call+0x0/0x4
> Likely the writer is stuck because the receive buffer is full and the
> reader is hung. What are the respective shells and telnetd doing?

I've added a couple of printk's around that schedule() call and in one pathetic
case all 3 find tasks never return:

S     0    58    47  1768   488 0:0   00:00 00:03:50 telnetd -i -l /bin/sh
S     0    59    58  1768   520 pts0  00:00 00:00:01 /bin/sh
S     0    61    47  1768   488 0:0   00:00 00:03:45 telnetd -i -l /bin/sh
S     0    62    61  1768   528 pts1  00:00 00:00:01 /bin/sh
S     0    65    47  1768   488 0:0   00:00 00:03:23 telnetd -i -l /bin/sh
S     0    66    65  1776   528 pts2  00:00 00:00:01 /bin/sh
S     0   833    66  1840   488 pts2  00:58 00:00:01 find / -name *        <--
S     0   871    62  1832   480 pts1  01:01 00:00:00 find / -name *        <--
S     0   881    59  1840   488 pts0  01:01 00:00:02 find / -name *        <--


---> 833 8e21c580
<--- 833 8e21c580
---> 833 8e21c580

---> 871 8e21c840
<--- 871 8e21c840
---> 871 8e21c840

---> 881 8e21c2c0
<--- 881 8e21c2c0
---> 881 8e21c2c0

I don't undersand how the receive buffer full/empty is coming into play - that
schedule() call is not a wait queue or something, it's an unconditional yield,
expecting sched to unconditionally return at some point - unless ofcourse - that
itself relies on some buffer r/w ISR semantics triggering an ISR, which makes the
task runnable again causing schdule() to return.

-Vineet

WARNING: multiple messages have this Message-ID (diff)
From: Vineet Gupta <Vineet.Gupta1@synopsys.com>
To: Peter Hurley <peter@hurleysoftware.com>
Cc: lkml <linux-kernel@vger.kernel.org>,
	<linux-serial@vger.kernel.org>, "Ingo Molnar" <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Jiri Slaby <jslaby@suse.cz>
Subject: Re: n_tty_write() going into schedule but NOT coming out
Date: Tue, 2 Apr 2013 18:56:46 +0530	[thread overview]
Message-ID: <515ADC96.7090908@synopsys.com> (raw)
In-Reply-To: <1364829008.3617.21.camel@thor.lan>

On 04/01/2013 08:40 PM, Peter Hurley wrote:
> On Mon, 2013-04-01 at 19:27 +0530, Vineet Gupta wrote:
>> Hi,
>>
>> Any thoughts: I observe the same issue even with CONFIG_PREEMPT and
>> CONFIG_PREEMPT_COUNT
>>
>> -Vineet
>>
>> On 03/30/2013 06:05 PM, Vineet Gupta wrote:
>>> Hi,
>>>
>>> I've been stress testing ARC Linux 3.8 (same happens for 3.9-rc3 as well). The
>>> setup has 3 telnet sessions, each running find . -name "*" in a loop.
>>> The platform is a FPGA @ 80 MHz, running a single core ARC700 so kernel .config
>>> has !SMP and PREEMPT_NONE.
>>>
>>> After ~10 mins of run, I see that one of the telnet session gets stuck (and later
>>> the 2nd one as well), while system is still alive, 3rd telnet is running find merrily.
>>>
>>> [ARCLinux]$ ps
>>> ....
>>>     7 root       0:00 inetd
>>>    62 root       0:00 -/bin/sh
>>>    64 root       1:34 telnetd -i -l /bin/sh
>>>    65 root       0:00 /bin/sh
>>>    75 root       1:47 telnetd -i -l /bin/sh
>>>    76 root       0:00 /bin/sh
>>>    79 root       0:53 telnetd -i -l /bin/sh
>>>    80 root       0:00 /bin/sh
>>>   281 root       0:00 find / -name *	<--- stuck
>>>   358 root       0:03 find / -name *	<--- stuck
>>>   377 root       0:00 find / -name *
>>>   378 root       0:00 ps
>>>
>>> Hung find task is sitting in the schedule() call in n_tty_write()
>>>
>>> [ARCLinux]$ cat /proc/281/stack
>>> [<8065945e>] n_tty_write+0x23a/0x424
>>> [<80655cd4>] tty_write+0x1ac/0x2d4
>>> [<805976ba>] vfs_write+0x92/0x110
>>> [<80597816>] sys_write+0x4e/0x88
>>> [<8050e780>] ret_from_system_call+0x0/0x4
> Likely the writer is stuck because the receive buffer is full and the
> reader is hung. What are the respective shells and telnetd doing?

I've added a couple of printk's around that schedule() call and in one pathetic
case all 3 find tasks never return:

S     0    58    47  1768   488 0:0   00:00 00:03:50 telnetd -i -l /bin/sh
S     0    59    58  1768   520 pts0  00:00 00:00:01 /bin/sh
S     0    61    47  1768   488 0:0   00:00 00:03:45 telnetd -i -l /bin/sh
S     0    62    61  1768   528 pts1  00:00 00:00:01 /bin/sh
S     0    65    47  1768   488 0:0   00:00 00:03:23 telnetd -i -l /bin/sh
S     0    66    65  1776   528 pts2  00:00 00:00:01 /bin/sh
S     0   833    66  1840   488 pts2  00:58 00:00:01 find / -name *        <--
S     0   871    62  1832   480 pts1  01:01 00:00:00 find / -name *        <--
S     0   881    59  1840   488 pts0  01:01 00:00:02 find / -name *        <--


---> 833 8e21c580
<--- 833 8e21c580
---> 833 8e21c580

---> 871 8e21c840
<--- 871 8e21c840
---> 871 8e21c840

---> 881 8e21c2c0
<--- 881 8e21c2c0
---> 881 8e21c2c0

I don't undersand how the receive buffer full/empty is coming into play - that
schedule() call is not a wait queue or something, it's an unconditional yield,
expecting sched to unconditionally return at some point - unless ofcourse - that
itself relies on some buffer r/w ISR semantics triggering an ISR, which makes the
task runnable again causing schdule() to return.

-Vineet

  parent reply	other threads:[~2013-04-02 13:27 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-03-30 12:35 n_tty_write() going into schedule but NOT coming out Vineet Gupta
2013-03-30 12:35 ` Vineet Gupta
2013-04-01 13:57 ` Vineet Gupta
2013-04-01 13:57   ` Vineet Gupta
2013-04-01 15:10   ` Peter Hurley
2013-04-02 11:09     ` Vineet Gupta
2013-04-02 11:09       ` Vineet Gupta
2013-04-02 13:26     ` Vineet Gupta [this message]
2013-04-02 13:26       ` Vineet Gupta
2013-04-02 13:38       ` Peter Hurley
2013-04-05 19:52       ` Peter Hurley
2013-04-06  9:32         ` Vineet Gupta
2013-04-06  9:32           ` Vineet Gupta
2013-04-03  5:44 ` Ilya Zykov
2013-04-03  7:42   ` Vineet Gupta
2013-04-03  7:42     ` Vineet Gupta

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=515ADC96.7090908@synopsys.com \
    --to=vineet.gupta1@synopsys.com \
    --cc=jslaby@suse.cz \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-serial@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peter@hurleysoftware.com \
    --cc=peterz@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.