tty: fix a possible hang on tty device

tty: fix a possible hang on tty device

[cael]

We have met a hang on pty device, the reader was blocking at epoll on
master side, the writer was sleeping at wait_woken inside n_tty_write
on slave side , and the write buffer on tty_port was full, we found
that the reader and writer would never be woken again and block
forever.

We thought the problem was caused as a race between reader and kworker
as follows:
n_tty_read(reader):                            |
  n_tty_receive_buf_common(kworker):
                                                           |
              room = N_TTY_BUF_SIZE - (ldata->read_head - tail);
                                                           |
              room <= 0
copy_from_read_buf(tty, &b, &nr);     |
n_tty_kick_worker(tty);                       |
                                                           |
              ldata->no_room = true

After writing to slave device, writer wakes up kworker to flush data
on tty_port to reader, and the kworker finds that reader has no room
to store data so room <= 0 is met. At this moment, reader consumes all
the data on reader buffer and call n_tty_kick_worker to check
ldata->no_room and finds that there is no need to call
tty_buffer_restart_work to flush data to reader and reader quits
reading. Then kworker sets  ldata->no_room = true and quits too.

If write buffer is not full, writer will wake kworker to flush data
again after following writes, but if writer buffer is full and writer
goes to sleep, kworker will never be woken again and tty device is
blocked.

We think this problem can be solved with a check for read buffer
inside function n_tty_receive_buf_common, if read buffer is empty and
ldata->no_room is true, this means that kworker has more data to flush
to read buffer, so a call to n_tty_kick_worker is necessary.

diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index f9c584244..4e65e2422 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -1760,6 +1760,8 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
        } else
                n_tty_check_throttle(tty);

+       if (!chars_in_buffer(tty))
+               n_tty_kick_worker(tty);
        up_read(&tty->termios_rwsem);

        return rcvd;

Re: tty: fix a possible hang on tty device

[Greg KH]

On Sat, May 07, 2022 at 05:11:35PM +0800, cael wrote:
> We have met a hang on pty device, the reader was blocking at epoll on
> master side, the writer was sleeping at wait_woken inside n_tty_write
> on slave side , and the write buffer on tty_port was full, we found
> that the reader and writer would never be woken again and block
> forever.
> 
> We thought the problem was caused as a race between reader and kworker
> as follows:
> n_tty_read(reader):                            |
>   n_tty_receive_buf_common(kworker):
>                                                            |
>               room = N_TTY_BUF_SIZE - (ldata->read_head - tail);
>                                                            |
>               room <= 0
> copy_from_read_buf(tty, &b, &nr);     |
> n_tty_kick_worker(tty);                       |
>                                                            |
>               ldata->no_room = true
> 
> After writing to slave device, writer wakes up kworker to flush data
> on tty_port to reader, and the kworker finds that reader has no room
> to store data so room <= 0 is met. At this moment, reader consumes all
> the data on reader buffer and call n_tty_kick_worker to check
> ldata->no_room and finds that there is no need to call
> tty_buffer_restart_work to flush data to reader and reader quits
> reading. Then kworker sets  ldata->no_room = true and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> We think this problem can be solved with a check for read buffer
> inside function n_tty_receive_buf_common, if read buffer is empty and
> ldata->no_room is true, this means that kworker has more data to flush
> to read buffer, so a call to n_tty_kick_worker is necessary.
> 
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index f9c584244..4e65e2422 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -1760,6 +1760,8 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (!chars_in_buffer(tty))
> +               n_tty_kick_worker(tty);
>         up_read(&tty->termios_rwsem);
> 
>         return rcvd;


Hi,

This is the friendly patch-bot of Greg Kroah-Hartman.  You have sent him
a patch that has triggered this response.  He used to manually respond
to these common problems, but in order to save his sanity (he kept
writing the same thing over and over, yet to different people), I was
created.  Hopefully you will not take offence and will fix the problem
in your patch and resubmit it so that it can be accepted into the Linux
kernel tree.

You are receiving this message because of the following common error(s)
as indicated below:

- Your patch is malformed (tabs converted to spaces, linewrapped, etc.)
  and can not be applied.  Please read the file,
  Documentation/email-clients.txt in order to fix this.

- Your patch does not have a Signed-off-by: line.  Please read the
  kernel file, Documentation/SubmittingPatches and resend it after
  adding that line.  Note, the line needs to be in the body of the
  email, before the patch, not at the bottom of the patch or in the
  email signature.

- You did not write a descriptive Subject: for the patch, allowing Greg,
  and everyone else, to know what this patch is all about.  Please read
  the section entitled "The canonical patch format" in the kernel file,
  Documentation/SubmittingPatches for what a proper Subject: line should
  look like.

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Greg will reply once he has dug out from the pending patches received
from other developers.

thanks,

greg k-h's patch email bot

tty: fix a possible hang on tty device

[cael]

We have met a hang on pty device, the reader was blocking at
 epoll on master side, the writer was sleeping at wait_woken inside
 n_tty_write on slave side ,and the write buffer on tty_port was full, we
 found that the reader and writer would never be woken again and block
 forever.

We thought the problem was caused as a race between reader and
kworker as follows:
n_tty_read(reader)| n_tty_receive_buf_common(kworker)
                  |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
                  |room <= 0
copy_from_read_buf|
n_tty_kick_worker |
                  |ldata->no_room = true

After writing to slave device, writer wakes up kworker to flush
data on tty_port to reader, and the kworker finds that reader
has no room to store data so room <= 0 is met. At this moment,
reader consumes all the data on reader buffer and call
n_tty_kick_worker to check ldata->no_room and finds that there
is no need to call tty_buffer_restart_work to flush data to reader
and reader quits reading. Then kworker sets ldata->no_room=true
and quits too.

If write buffer is not full, writer will wake kworker to flush data
again after following writes, but if writer buffer is full and writer
goes to sleep, kworker will never be woken again and tty device is
blocked.

We think this problem can be solved with a check for read buffer
inside function n_tty_receive_buf_common, if read buffer is empty and
ldata->no_room is true, this means that kworker has more data to flush
to read buffer, so a call to n_tty_kick_worker is necessary.

Signed-off-by: cael <juanfengpy@gmail.com>
---
diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index efc72104c840..36c7bc033c78 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
        } else
                n_tty_check_throttle(tty);

+       if (!chars_in_buffer(tty))
+               n_tty_kick_worker(tty);
+
        up_read(&tty->termios_rwsem);

        return rcvd;
-- 
2.27.0

Re: tty: fix a possible hang on tty device

[Ilpo Järvinen]

On Tue, 24 May 2022, cael wrote:

> We have met a hang on pty device, the reader was blocking at
>  epoll on master side, the writer was sleeping at wait_woken inside
>  n_tty_write on slave side ,and the write buffer on tty_port was full, we

Space after comma. It would be also useful to tone down usage of "we" in 
the changelog.

>  found that the reader and writer would never be woken again and block
>  forever.
> 
> We thought the problem was caused as a race between reader and
> kworker as follows:
> n_tty_read(reader)| n_tty_receive_buf_common(kworker)
>                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
>                   |room <= 0
> copy_from_read_buf|
> n_tty_kick_worker |
>                   |ldata->no_room = true
>
> After writing to slave device, writer wakes up kworker to flush
> data on tty_port to reader, and the kworker finds that reader
> has no room to store data so room <= 0 is met. At this moment,
> reader consumes all the data on reader buffer and call
> n_tty_kick_worker to check ldata->no_room and finds that there
> is no need to call tty_buffer_restart_work to flush data to reader
> and reader quits reading. Then kworker sets ldata->no_room=true
> and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> We think this problem can be solved with a check for read buffer
> inside function n_tty_receive_buf_common, if read buffer is empty and
> ldata->no_room is true, this means that kworker has more data to flush
> to read buffer, so a call to n_tty_kick_worker is necessary.
> 
> Signed-off-by: cael <juanfengpy@gmail.com>
> ---
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..36c7bc033c78 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (!chars_in_buffer(tty))
> +               n_tty_kick_worker(tty);
> +

chars_in_buffer() accesses ldata->read_tail in producer context so this 
probably just moves the race there?


-- 
 i.

Re: tty: fix a possible hang on tty device

[cael]

Thanks for the answer, yes, there exists a race between reader and kworker,
but it's OK. Before checking chars_in_buffer in kworker,
ldata->no_room is set true,
if reader changes ldata->read_tail in n_tty_read when kworker checks this value
which makes the check fail, then when reader reaches end of n_tty_read,
n_tty_kick_worker will also be called. Besides, kworker and reader may
call n_tty_kick_worker at the same time, this function only queues work
on workqueue, so it's harmless.

Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 17:11写道：
>
> On Tue, 24 May 2022, cael wrote:
>
> > We have met a hang on pty device, the reader was blocking at
> >  epoll on master side, the writer was sleeping at wait_woken inside
> >  n_tty_write on slave side ,and the write buffer on tty_port was full, we
>
> Space after comma. It would be also useful to tone down usage of "we" in
> the changelog.
>
> >  found that the reader and writer would never be woken again and block
> >  forever.
> >
> > We thought the problem was caused as a race between reader and
> > kworker as follows:
> > n_tty_read(reader)| n_tty_receive_buf_common(kworker)
> >                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
> >                   |room <= 0
> > copy_from_read_buf|
> > n_tty_kick_worker |
> >                   |ldata->no_room = true
> >
> > After writing to slave device, writer wakes up kworker to flush
> > data on tty_port to reader, and the kworker finds that reader
> > has no room to store data so room <= 0 is met. At this moment,
> > reader consumes all the data on reader buffer and call
> > n_tty_kick_worker to check ldata->no_room and finds that there
> > is no need to call tty_buffer_restart_work to flush data to reader
> > and reader quits reading. Then kworker sets ldata->no_room=true
> > and quits too.
> >
> > If write buffer is not full, writer will wake kworker to flush data
> > again after following writes, but if writer buffer is full and writer
> > goes to sleep, kworker will never be woken again and tty device is
> > blocked.
> >
> > We think this problem can be solved with a check for read buffer
> > inside function n_tty_receive_buf_common, if read buffer is empty and
> > ldata->no_room is true, this means that kworker has more data to flush
> > to read buffer, so a call to n_tty_kick_worker is necessary.
> >
> > Signed-off-by: cael <juanfengpy@gmail.com>
> > ---
> > diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> > index efc72104c840..36c7bc033c78 100644
> > --- a/drivers/tty/n_tty.c
> > +++ b/drivers/tty/n_tty.c
> > @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> > const unsigned char *cp,
> >         } else
> >                 n_tty_check_throttle(tty);
> >
> > +       if (!chars_in_buffer(tty))
> > +               n_tty_kick_worker(tty);
> > +
>
> chars_in_buffer() accesses ldata->read_tail in producer context so this
> probably just moves the race there?
>
>
> --
>  i.
>

Re: tty: fix a possible hang on tty device

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 3377 bytes --]

On Tue, 24 May 2022, cael wrote:

> Thanks for the answer, yes, there exists a race between reader and kworker,
> but it's OK. Before checking chars_in_buffer in kworker,
> ldata->no_room is set true,

Nothing seems to guarantee this.

> if reader changes ldata->read_tail in n_tty_read when kworker checks this value
> which makes the check fail, then when reader reaches end of n_tty_read,
> n_tty_kick_worker will also be called. Besides, kworker and reader may
> call n_tty_kick_worker at the same time, this function only queues work
> on workqueue, so it's harmless.

I'm not worried about the case where both cpus call n_tty_kick_worker but 
the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
!no_room.

-- 
 i.

> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 17:11写道：
> >
> > On Tue, 24 May 2022, cael wrote:
> >
> > > We have met a hang on pty device, the reader was blocking at
> > >  epoll on master side, the writer was sleeping at wait_woken inside
> > >  n_tty_write on slave side ,and the write buffer on tty_port was full, we
> >
> > Space after comma. It would be also useful to tone down usage of "we" in
> > the changelog.
> >
> > >  found that the reader and writer would never be woken again and block
> > >  forever.
> > >
> > > We thought the problem was caused as a race between reader and
> > > kworker as follows:
> > > n_tty_read(reader)| n_tty_receive_buf_common(kworker)
> > >                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
> > >                   |room <= 0
> > > copy_from_read_buf|
> > > n_tty_kick_worker |
> > >                   |ldata->no_room = true
> > >
> > > After writing to slave device, writer wakes up kworker to flush
> > > data on tty_port to reader, and the kworker finds that reader
> > > has no room to store data so room <= 0 is met. At this moment,
> > > reader consumes all the data on reader buffer and call
> > > n_tty_kick_worker to check ldata->no_room and finds that there
> > > is no need to call tty_buffer_restart_work to flush data to reader
> > > and reader quits reading. Then kworker sets ldata->no_room=true
> > > and quits too.
> > >
> > > If write buffer is not full, writer will wake kworker to flush data
> > > again after following writes, but if writer buffer is full and writer
> > > goes to sleep, kworker will never be woken again and tty device is
> > > blocked.
> > >
> > > We think this problem can be solved with a check for read buffer
> > > inside function n_tty_receive_buf_common, if read buffer is empty and
> > > ldata->no_room is true, this means that kworker has more data to flush
> > > to read buffer, so a call to n_tty_kick_worker is necessary.
> > >
> > > Signed-off-by: cael <juanfengpy@gmail.com>
> > > ---
> > > diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> > > index efc72104c840..36c7bc033c78 100644
> > > --- a/drivers/tty/n_tty.c
> > > +++ b/drivers/tty/n_tty.c
> > > @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> > > const unsigned char *cp,
> > >         } else
> > >                 n_tty_check_throttle(tty);
> > >
> > > +       if (!chars_in_buffer(tty))
> > > +               n_tty_kick_worker(tty);
> > > +
> >
> > chars_in_buffer() accesses ldata->read_tail in producer context so this
> > probably just moves the race there?
> >
> >
> > --
> >  i.
> >
> 

Re: tty: fix a possible hang on tty device

[cael]

if  ldata->no_room is not true, that means kworker has flushed
at least n characters to break the while loop, so return value of
n_tty_receive_buf_common is not zero, flush_to_ldisc will
continue to call this function to flush data to reader if write buffer
is not empty.

Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 19:40写道：
>
> On Tue, 24 May 2022, cael wrote:
>
> > Thanks for the answer, yes, there exists a race between reader and kworker,
> > but it's OK. Before checking chars_in_buffer in kworker,
> > ldata->no_room is set true,
>
> Nothing seems to guarantee this.
>
> > if reader changes ldata->read_tail in n_tty_read when kworker checks this value
> > which makes the check fail, then when reader reaches end of n_tty_read,
> > n_tty_kick_worker will also be called. Besides, kworker and reader may
> > call n_tty_kick_worker at the same time, this function only queues work
> > on workqueue, so it's harmless.
>
> I'm not worried about the case where both cpus call n_tty_kick_worker but
> the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
> !no_room.
>
> --
>  i.
>
> > Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 17:11写道：
> > >
> > > On Tue, 24 May 2022, cael wrote:
> > >
> > > > We have met a hang on pty device, the reader was blocking at
> > > >  epoll on master side, the writer was sleeping at wait_woken inside
> > > >  n_tty_write on slave side ,and the write buffer on tty_port was full, we
> > >
> > > Space after comma. It would be also useful to tone down usage of "we" in
> > > the changelog.
> > >
> > > >  found that the reader and writer would never be woken again and block
> > > >  forever.
> > > >
> > > > We thought the problem was caused as a race between reader and
> > > > kworker as follows:
> > > > n_tty_read(reader)| n_tty_receive_buf_common(kworker)
> > > >                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
> > > >                   |room <= 0
> > > > copy_from_read_buf|
> > > > n_tty_kick_worker |
> > > >                   |ldata->no_room = true
> > > >
> > > > After writing to slave device, writer wakes up kworker to flush
> > > > data on tty_port to reader, and the kworker finds that reader
> > > > has no room to store data so room <= 0 is met. At this moment,
> > > > reader consumes all the data on reader buffer and call
> > > > n_tty_kick_worker to check ldata->no_room and finds that there
> > > > is no need to call tty_buffer_restart_work to flush data to reader
> > > > and reader quits reading. Then kworker sets ldata->no_room=true
> > > > and quits too.
> > > >
> > > > If write buffer is not full, writer will wake kworker to flush data
> > > > again after following writes, but if writer buffer is full and writer
> > > > goes to sleep, kworker will never be woken again and tty device is
> > > > blocked.
> > > >
> > > > We think this problem can be solved with a check for read buffer
> > > > inside function n_tty_receive_buf_common, if read buffer is empty and
> > > > ldata->no_room is true, this means that kworker has more data to flush
> > > > to read buffer, so a call to n_tty_kick_worker is necessary.
> > > >
> > > > Signed-off-by: cael <juanfengpy@gmail.com>
> > > > ---
> > > > diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> > > > index efc72104c840..36c7bc033c78 100644
> > > > --- a/drivers/tty/n_tty.c
> > > > +++ b/drivers/tty/n_tty.c
> > > > @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> > > > const unsigned char *cp,
> > > >         } else
> > > >                 n_tty_check_throttle(tty);
> > > >
> > > > +       if (!chars_in_buffer(tty))
> > > > +               n_tty_kick_worker(tty);
> > > > +
> > >
> > > chars_in_buffer() accesses ldata->read_tail in producer context so this
> > > probably just moves the race there?
> > >
> > >
> > > --
> > >  i.
> > >
> >

Re: tty: fix a possible hang on tty device

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 4239 bytes --]

On Tue, 24 May 2022, cael wrote:

> if  ldata->no_room is not true, that means kworker has flushed
> at least n characters to break the while loop, so return value of
> n_tty_receive_buf_common is not zero, flush_to_ldisc will
> continue to call this function to flush data to reader if write buffer
> is not empty.

Now you switched to an entirely different case, not the one we were 
talking about. ...There is no ldisc->no_room = true race in the case
you now described.

-- 
 i.

> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 19:40写道：
> >
> > On Tue, 24 May 2022, cael wrote:
> >
> > > Thanks for the answer, yes, there exists a race between reader and kworker,
> > > but it's OK. Before checking chars_in_buffer in kworker,
> > > ldata->no_room is set true,
> >
> > Nothing seems to guarantee this.
> >
> > > if reader changes ldata->read_tail in n_tty_read when kworker checks this value
> > > which makes the check fail, then when reader reaches end of n_tty_read,
> > > n_tty_kick_worker will also be called. Besides, kworker and reader may
> > > call n_tty_kick_worker at the same time, this function only queues work
> > > on workqueue, so it's harmless.
> >
> > I'm not worried about the case where both cpus call n_tty_kick_worker but
> > the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
> > !no_room.
> >
> > --
> >  i.
> >
> > > Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 17:11写道：
> > > >
> > > > On Tue, 24 May 2022, cael wrote:
> > > >
> > > > > We have met a hang on pty device, the reader was blocking at
> > > > >  epoll on master side, the writer was sleeping at wait_woken inside
> > > > >  n_tty_write on slave side ,and the write buffer on tty_port was full, we
> > > >
> > > > Space after comma. It would be also useful to tone down usage of "we" in
> > > > the changelog.
> > > >
> > > > >  found that the reader and writer would never be woken again and block
> > > > >  forever.
> > > > >
> > > > > We thought the problem was caused as a race between reader and
> > > > > kworker as follows:
> > > > > n_tty_read(reader)| n_tty_receive_buf_common(kworker)
> > > > >                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
> > > > >                   |room <= 0
> > > > > copy_from_read_buf|
> > > > > n_tty_kick_worker |
> > > > >                   |ldata->no_room = true
> > > > >
> > > > > After writing to slave device, writer wakes up kworker to flush
> > > > > data on tty_port to reader, and the kworker finds that reader
> > > > > has no room to store data so room <= 0 is met. At this moment,
> > > > > reader consumes all the data on reader buffer and call
> > > > > n_tty_kick_worker to check ldata->no_room and finds that there
> > > > > is no need to call tty_buffer_restart_work to flush data to reader
> > > > > and reader quits reading. Then kworker sets ldata->no_room=true
> > > > > and quits too.
> > > > >
> > > > > If write buffer is not full, writer will wake kworker to flush data
> > > > > again after following writes, but if writer buffer is full and writer
> > > > > goes to sleep, kworker will never be woken again and tty device is
> > > > > blocked.
> > > > >
> > > > > We think this problem can be solved with a check for read buffer
> > > > > inside function n_tty_receive_buf_common, if read buffer is empty and
> > > > > ldata->no_room is true, this means that kworker has more data to flush
> > > > > to read buffer, so a call to n_tty_kick_worker is necessary.
> > > > >
> > > > > Signed-off-by: cael <juanfengpy@gmail.com>
> > > > > ---
> > > > > diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> > > > > index efc72104c840..36c7bc033c78 100644
> > > > > --- a/drivers/tty/n_tty.c
> > > > > +++ b/drivers/tty/n_tty.c
> > > > > @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> > > > > const unsigned char *cp,
> > > > >         } else
> > > > >                 n_tty_check_throttle(tty);
> > > > >
> > > > > +       if (!chars_in_buffer(tty))
> > > > > +               n_tty_kick_worker(tty);
> > > > > +
> > > >
> > > > chars_in_buffer() accesses ldata->read_tail in producer context so this
> > > > probably just moves the race there?

Re: tty: fix a possible hang on tty device

[cael]

>Now you switched to an entirely different case, not the one we were
>talking about. ...There is no ldisc->no_room = true race in the case
>you now described.
So, I think we should back to the case ldata->no_room=true as
ldata->no_room=false seems harmless.

>I'm not worried about the case where both cpus call n_tty_kick_worker but
>the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
>!no_room.
As ldata->no_room=true is set before checking chars_in_buffer(), if producer
finds chars_in_buffer() > 0, then if reader is currently in n_tty_read,
when reader quits n_tty_read, n_tty_kick_worker will be called. If reader
has already exited n_tty_read, which means that reader still has data to read,
next time reader will call n_tty_kick_worker inside n_tty_read too.

Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 21:25写道：
>
> On Tue, 24 May 2022, cael wrote:
>
> > if  ldata->no_room is not true, that means kworker has flushed
> > at least n characters to break the while loop, so return value of
> > n_tty_receive_buf_common is not zero, flush_to_ldisc will
> > continue to call this function to flush data to reader if write buffer
> > is not empty.
>
> Now you switched to an entirely different case, not the one we were
> talking about. ...There is no ldisc->no_room = true race in the case
> you now described.
>
> --
>  i.
>
> > Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 19:40写道：
> > >
> > > On Tue, 24 May 2022, cael wrote:
> > >
> > > > Thanks for the answer, yes, there exists a race between reader and kworker,
> > > > but it's OK. Before checking chars_in_buffer in kworker,
> > > > ldata->no_room is set true,
> > >
> > > Nothing seems to guarantee this.
> > >
> > > > if reader changes ldata->read_tail in n_tty_read when kworker checks this value
> > > > which makes the check fail, then when reader reaches end of n_tty_read,
> > > > n_tty_kick_worker will also be called. Besides, kworker and reader may
> > > > call n_tty_kick_worker at the same time, this function only queues work
> > > > on workqueue, so it's harmless.
> > >
> > > I'm not worried about the case where both cpus call n_tty_kick_worker but
> > > the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
> > > !no_room.
> > >
> > > --
> > >  i.
> > >
> > > > Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月24日周二 17:11写道：
> > > > >
> > > > > On Tue, 24 May 2022, cael wrote:
> > > > >
> > > > > > We have met a hang on pty device, the reader was blocking at
> > > > > >  epoll on master side, the writer was sleeping at wait_woken inside
> > > > > >  n_tty_write on slave side ,and the write buffer on tty_port was full, we
> > > > >
> > > > > Space after comma. It would be also useful to tone down usage of "we" in
> > > > > the changelog.
> > > > >
> > > > > >  found that the reader and writer would never be woken again and block
> > > > > >  forever.
> > > > > >
> > > > > > We thought the problem was caused as a race between reader and
> > > > > > kworker as follows:
> > > > > > n_tty_read(reader)| n_tty_receive_buf_common(kworker)
> > > > > >                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
> > > > > >                   |room <= 0
> > > > > > copy_from_read_buf|
> > > > > > n_tty_kick_worker |
> > > > > >                   |ldata->no_room = true
> > > > > >
> > > > > > After writing to slave device, writer wakes up kworker to flush
> > > > > > data on tty_port to reader, and the kworker finds that reader
> > > > > > has no room to store data so room <= 0 is met. At this moment,
> > > > > > reader consumes all the data on reader buffer and call
> > > > > > n_tty_kick_worker to check ldata->no_room and finds that there
> > > > > > is no need to call tty_buffer_restart_work to flush data to reader
> > > > > > and reader quits reading. Then kworker sets ldata->no_room=true
> > > > > > and quits too.
> > > > > >
> > > > > > If write buffer is not full, writer will wake kworker to flush data
> > > > > > again after following writes, but if writer buffer is full and writer
> > > > > > goes to sleep, kworker will never be woken again and tty device is
> > > > > > blocked.
> > > > > >
> > > > > > We think this problem can be solved with a check for read buffer
> > > > > > inside function n_tty_receive_buf_common, if read buffer is empty and
> > > > > > ldata->no_room is true, this means that kworker has more data to flush
> > > > > > to read buffer, so a call to n_tty_kick_worker is necessary.
> > > > > >
> > > > > > Signed-off-by: cael <juanfengpy@gmail.com>
> > > > > > ---
> > > > > > diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> > > > > > index efc72104c840..36c7bc033c78 100644
> > > > > > --- a/drivers/tty/n_tty.c
> > > > > > +++ b/drivers/tty/n_tty.c
> > > > > > @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> > > > > > const unsigned char *cp,
> > > > > >         } else
> > > > > >                 n_tty_check_throttle(tty);
> > > > > >
> > > > > > +       if (!chars_in_buffer(tty))
> > > > > > +               n_tty_kick_worker(tty);
> > > > > > +
> > > > >
> > > > > chars_in_buffer() accesses ldata->read_tail in producer context so this
> > > > > probably just moves the race there?
>
>

Re: tty: fix a possible hang on tty device

[Ilpo Järvinen]

On Wed, 25 May 2022, cael wrote:

> >Now you switched to an entirely different case, not the one we were
> >talking about. ...There is no ldisc->no_room = true race in the case
> >you now described.
> So, I think we should back to the case ldata->no_room=true as
> ldata->no_room=false seems harmless.
> 
> >I'm not worried about the case where both cpus call n_tty_kick_worker but
> >the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
> >!no_room.
>
> As ldata->no_room=true is set before checking chars_in_buffer()

Please take a brief look at Documentation/memory-barriers.txt and then 
tell me if you still find this claim to be true.

> if producer
> finds chars_in_buffer() > 0, then if reader is currently in n_tty_read,

...Then please do a similar analysis for ldata->read_tail. What guarantees 
its update is seen by the producer cpu when the reader is already past the 
point you think it still must be in?

> when reader quits n_tty_read, n_tty_kick_worker will be called. If reader
> has already exited n_tty_read, which means that reader still has data to read,
> next time reader will call n_tty_kick_worker inside n_tty_read too.

C-level analysis alone is not going to be very useful here given you're 
dealing with a concurrency challenge here.


-- 
 i.

Re: tty: fix a possible hang on tty device

[cael]

Thanks, You are right, barrier is needed here. I changed the patch as follows:
1) WRITE_ONCE and READ_ONCE is used to access ldata->no_room since
n_tty_kick_worker  would be called in kworker and reader cpu;
2) smp_mb added in chars_in_buffer as this function will be called in
reader and kworker, accessing commit_head and read_tail; and to make
sure that read_tail is not read before setting no_room in
n_tty_receive_buf_common;
3) smp_mb added in n_tty_read to make sure that no_room is not read
before setting read_tail.
---
diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index efc72104c840..3327687da0d3 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
        struct n_tty_data *ldata = tty->disc_data;

        /* Did the input worker stop? Restart it */
-       if (unlikely(ldata->no_room)) {
-               ldata->no_room = 0;
+       if (unlikely(READ_ONCE(ldata->no_room))) {
+               WRITE_ONCE(ldata->no_room, 0);

                WARN_RATELIMIT(tty->port->itty == NULL,
                                "scheduling with invalid itty\n");
@@ -221,6 +221,7 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
        struct n_tty_data *ldata = tty->disc_data;
        ssize_t n = 0;

+       smp_mb();
        if (!ldata->icanon)
                n = ldata->commit_head - ldata->read_tail;
        else
@@ -1632,7 +1633,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
                        if (overflow && room < 0)
                                ldata->read_head--;
                        room = overflow;
-                       ldata->no_room = flow && !room;
+                       WRITE_ONCE(ldata->no_room, flow && !room);
                } else
                        overflow = 0;

@@ -1663,6 +1664,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
        } else
                n_tty_check_throttle(tty);

+       if (!chars_in_buffer(tty))
+               n_tty_kick_worker(tty);
+
        up_read(&tty->termios_rwsem);

        return rcvd;
@@ -2180,8 +2184,10 @@ static ssize_t n_tty_read(struct tty_struct
*tty, struct file *file,
                if (time)
                        timeout = time;
        }
-       if (tail != ldata->read_tail)
+       if (tail != ldata->read_tail) {
+               smp_mb();
                n_tty_kick_worker(tty);
+       }
        up_read(&tty->termios_rwsem);

        remove_wait_queue(&tty->read_wait, &wait);
--
2.27.0

Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月25日周三 19:21写道：
>
> On Wed, 25 May 2022, cael wrote:
>
> > >Now you switched to an entirely different case, not the one we were
> > >talking about. ...There is no ldisc->no_room = true race in the case
> > >you now described.
> > So, I think we should back to the case ldata->no_room=true as
> > ldata->no_room=false seems harmless.
> >
> > >I'm not worried about the case where both cpus call n_tty_kick_worker but
> > >the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
> > >!no_room.
> >
> > As ldata->no_room=true is set before checking chars_in_buffer()
>
> Please take a brief look at Documentation/memory-barriers.txt and then
> tell me if you still find this claim to be true.
>
> > if producer
> > finds chars_in_buffer() > 0, then if reader is currently in n_tty_read,
>
> ...Then please do a similar analysis for ldata->read_tail. What guarantees
> its update is seen by the producer cpu when the reader is already past the
> point you think it still must be in?
>
> > when reader quits n_tty_read, n_tty_kick_worker will be called. If reader
> > has already exited n_tty_read, which means that reader still has data to read,
> > next time reader will call n_tty_kick_worker inside n_tty_read too.
>
> C-level analysis alone is not going to be very useful here given you're
> dealing with a concurrency challenge here.
>
>
> --
>  i.
>
>

Re: tty: fix a possible hang on tty device

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 4823 bytes --]

On Mon, 30 May 2022, cael wrote:

> Thanks, You are right, barrier is needed here. I changed the patch as follows:
> 1) WRITE_ONCE and READ_ONCE is used to access ldata->no_room since
> n_tty_kick_worker  would be called in kworker and reader cpu;
> 2) smp_mb added in chars_in_buffer as this function will be called in
> reader and kworker, accessing commit_head and read_tail; and to make
> sure that read_tail is not read before setting no_room in
> n_tty_receive_buf_common;
> 3) smp_mb added in n_tty_read to make sure that no_room is not read
> before setting read_tail.

Please include proper changelog to all revised patch submissions, not 
just list of changes you've made (and properly version the submissions 
with [PATCH v2] etc. in the subject).

> ---
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..3327687da0d3 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
>         struct n_tty_data *ldata = tty->disc_data;
> 
>         /* Did the input worker stop? Restart it */
> -       if (unlikely(ldata->no_room)) {
> -               ldata->no_room = 0;
> +       if (unlikely(READ_ONCE(ldata->no_room))) {
> +               WRITE_ONCE(ldata->no_room, 0);
>                 WARN_RATELIMIT(tty->port->itty == NULL,
>                                 "scheduling with invalid itty\n");
> @@ -221,6 +221,7 @@ static ssize_t chars_in_buffer(struct tty_struct *tty)
>         struct n_tty_data *ldata = tty->disc_data;
>         ssize_t n = 0;
> 
> +       smp_mb();

You should add the reason in comment for any barriers you add.

>         if (!ldata->icanon)
>                 n = ldata->commit_head - ldata->read_tail;
>         else
> @@ -1632,7 +1633,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>                         if (overflow && room < 0)
>                                 ldata->read_head--;
>                         room = overflow;
> -                       ldata->no_room = flow && !room;
> +                       WRITE_ONCE(ldata->no_room, flow && !room);
>                 } else
>                         overflow = 0;
> 
> @@ -1663,6 +1664,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (!chars_in_buffer(tty))
> +               n_tty_kick_worker(tty);
> +

Instead of having the barrier in chars_in_buffer() perhaps it would be 
more obvious what's going on here and also scope down to the cases where 
the barrier might be needed in the first place if you'd do:

	if (ldata->no_room) {
		/* ... */
		smp_mb();
		if (!chars_in_buffer(tty))
			n_tty_kick_worker(tty);
	}


-- 
 i.

>         up_read(&tty->termios_rwsem);
> 
>         return rcvd;
> @@ -2180,8 +2184,10 @@ static ssize_t n_tty_read(struct tty_struct
> *tty, struct file *file,
>                 if (time)
>                         timeout = time;
>         }
> -       if (tail != ldata->read_tail)
> +       if (tail != ldata->read_tail) {
> +               smp_mb();
>                 n_tty_kick_worker(tty);
> +       }
>         up_read(&tty->termios_rwsem);
> 
>         remove_wait_queue(&tty->read_wait, &wait);
> --
> 2.27.0
> 
> Ilpo Järvinen <ilpo.jarvinen@linux.intel.com> 于2022年5月25日周三 19:21写道：
> >
> > On Wed, 25 May 2022, cael wrote:
> >
> > > >Now you switched to an entirely different case, not the one we were
> > > >talking about. ...There is no ldisc->no_room = true race in the case
> > > >you now described.
> > > So, I think we should back to the case ldata->no_room=true as
> > > ldata->no_room=false seems harmless.
> > >
> > > >I'm not worried about the case where both cpus call n_tty_kick_worker but
> > > >the case where producer cpu sees chars_in_buffer() > 0 and consumer cpu
> > > >!no_room.
> > >
> > > As ldata->no_room=true is set before checking chars_in_buffer()
> >
> > Please take a brief look at Documentation/memory-barriers.txt and then
> > tell me if you still find this claim to be true.
> >
> > > if producer
> > > finds chars_in_buffer() > 0, then if reader is currently in n_tty_read,
> >
> > ...Then please do a similar analysis for ldata->read_tail. What guarantees
> > its update is seen by the producer cpu when the reader is already past the
> > point you think it still must be in?
> >
> > > when reader quits n_tty_read, n_tty_kick_worker will be called. If reader
> > > has already exited n_tty_read, which means that reader still has data to read,
> > > next time reader will call n_tty_kick_worker inside n_tty_read too.
> >
> > C-level analysis alone is not going to be very useful here given you're
> > dealing with a concurrency challenge here.
> >
> >
> > --
> >  i.
> >
> >
> 

Re: tty: fix a possible hang on tty device

[Greg KH]

On Tue, May 24, 2022 at 10:21:04AM +0800, cael wrote:
> We have met a hang on pty device, the reader was blocking at
>  epoll on master side, the writer was sleeping at wait_woken inside
>  n_tty_write on slave side ,and the write buffer on tty_port was full, we
>  found that the reader and writer would never be woken again and block
>  forever.
> 
> We thought the problem was caused as a race between reader and
> kworker as follows:
> n_tty_read(reader)| n_tty_receive_buf_common(kworker)
>                   |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
>                   |room <= 0
> copy_from_read_buf|
> n_tty_kick_worker |
>                   |ldata->no_room = true
> 
> After writing to slave device, writer wakes up kworker to flush
> data on tty_port to reader, and the kworker finds that reader
> has no room to store data so room <= 0 is met. At this moment,
> reader consumes all the data on reader buffer and call
> n_tty_kick_worker to check ldata->no_room and finds that there
> is no need to call tty_buffer_restart_work to flush data to reader
> and reader quits reading. Then kworker sets ldata->no_room=true
> and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> We think this problem can be solved with a check for read buffer
> inside function n_tty_receive_buf_common, if read buffer is empty and
> ldata->no_room is true, this means that kworker has more data to flush
> to read buffer, so a call to n_tty_kick_worker is necessary.
> 
> Signed-off-by: cael <juanfengpy@gmail.com>
> ---
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..36c7bc033c78 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -1663,6 +1663,9 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (!chars_in_buffer(tty))
> +               n_tty_kick_worker(tty);
> +
>         up_read(&tty->termios_rwsem);
> 
>         return rcvd;
> -- 
> 2.27.0

Hi,

This is the friendly patch-bot of Greg Kroah-Hartman.  You have sent him
a patch that has triggered this response.  He used to manually respond
to these common problems, but in order to save his sanity (he kept
writing the same thing over and over, yet to different people), I was
created.  Hopefully you will not take offence and will fix the problem
in your patch and resubmit it so that it can be accepted into the Linux
kernel tree.

You are receiving this message because of the following common error(s)
as indicated below:

- Your patch is malformed (tabs converted to spaces, linewrapped, etc.)
  and can not be applied.  Please read the file,
  Documentation/email-clients.txt in order to fix this.

- You did not specify a description of why the patch is needed, or
  possibly, any description at all, in the email body.  Please read the
  section entitled "The canonical patch format" in the kernel file,
  Documentation/SubmittingPatches for what is needed in order to
  properly describe the change.

- You did not write a descriptive Subject: for the patch, allowing Greg,
  and everyone else, to know what this patch is all about.  Please read
  the section entitled "The canonical patch format" in the kernel file,
  Documentation/SubmittingPatches for what a proper Subject: line should
  look like.

- It looks like you did not use your "real" name for the patch on either
  the Signed-off-by: line, or the From: line (both of which have to
  match).  Please read the kernel file, Documentation/SubmittingPatches
  for how to do this correctly.

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Greg will reply once he has dug out from the pending patches received
from other developers.

thanks,

greg k-h's patch email bot

Re: tty: fix a possible hang on tty device

[cael]

From: cael <juanfengpy@gmail.com>
Subject: [PATCH v2] tty: fix a possible hang on tty device

We have met a hang on pty device, the reader was blocking
at epoll on master side, the writer was sleeping at wait_woken
inside n_tty_write on slave side, and the write buffer on
tty_port was full, we found that the reader and writer would
never be woken again and block forever.

The problem was caused by a race between reader and kworker:
n_tty_read(reader):  n_tty_receive_buf_common(kworker):
                    |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
                    |room <= 0
copy_from_read_buf()|
n_tty_kick_worker() |
                    |ldata->no_room = true

After writing to slave device, writer wakes up kworker to flush
data on tty_port to reader, and the kworker finds that reader
has no room to store data so room <= 0 is met. At this moment,
reader consumes all the data on reader buffer and call
n_tty_kick_worker to check ldata->no_room which is false and
reader quits reading. Then kworker sets ldata->no_room=true
and quits too.

If write buffer is not full, writer will wake kworker to flush data
again after following writes, but if writer buffer is full and writer
goes to sleep, kworker will never be woken again and tty device is
blocked.

This problem can be solved with a check for read buffer size inside
n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
is true, a call to n_tty_kick_worker is necessary to keep flushing
data to reader.

Signed-off-by: cael <juanfengpy@gmail.com>
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>

---
diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index efc72104c840..21241ea7cdb9 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
        struct n_tty_data *ldata = tty->disc_data;

        /* Did the input worker stop? Restart it */
-       if (unlikely(ldata->no_room)) {
-               ldata->no_room = 0;
+       if (unlikely(READ_ONCE(ldata->no_room))) {
+               WRITE_ONCE(ldata->no_room, 0);

                WARN_RATELIMIT(tty->port->itty == NULL,
                                "scheduling with invalid itty\n");
@@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
                        if (overflow && room < 0)
                                ldata->read_head--;
                        room = overflow;
-                       ldata->no_room = flow && !room;
+                       WRITE_ONCE(ldata->no_room, flow && !room);
                } else
                        overflow = 0;

@@ -1663,6 +1663,21 @@ n_tty_receive_buf_common(struct tty_struct
*tty, const unsigned char *cp,
        } else
                n_tty_check_throttle(tty);

+       if (READ_ONCE(ldata->no_room)) {
+               /*
+                * Reader ensures that read_tail is updated before
checking no_room,
+                * make sure that no_room is set before reading read_tail here.
+                * Now no_room is visible by reader, the race needs to
be handled is
+                * that reader has passed checkpoint for no_room and
reader buffer
+                * is empty, if so n_tty_kick_worker will not be
called by reader,
+                * instead, this function is called here.
+                * barrier is paired with smp_mb() in n_tty_read()
+                */
+               smp_mb();
+               if (!chars_in_buffer(tty))
+                       n_tty_kick_worker(tty);
+       }
+
        up_read(&tty->termios_rwsem);

        return rcvd;
@@ -2180,8 +2195,14 @@ static ssize_t n_tty_read(struct tty_struct
*tty, struct file *file,
                if (time)
                        timeout = time;
        }
-       if (tail != ldata->read_tail)
+       if (tail != ldata->read_tail) {
+               /*
+                * Make sure no_room is not read before setting read_tail,
+                * paired with smp_mb() in n_tty_receive_buf_common()
+                */

+               smp_mb();
                n_tty_kick_worker(tty);
+       }
        up_read(&tty->termios_rwsem);

        remove_wait_queue(&tty->read_wait, &wait);
--
2.27.0

Re: tty: fix a possible hang on tty device

[Greg KH]

On Wed, Jun 01, 2022 at 09:39:27PM +0800, cael wrote:
> From: cael <juanfengpy@gmail.com>
> Subject: [PATCH v2] tty: fix a possible hang on tty device
> 
> We have met a hang on pty device, the reader was blocking
> at epoll on master side, the writer was sleeping at wait_woken
> inside n_tty_write on slave side, and the write buffer on
> tty_port was full, we found that the reader and writer would
> never be woken again and block forever.
> 
> The problem was caused by a race between reader and kworker:
> n_tty_read(reader):  n_tty_receive_buf_common(kworker):
>                     |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
>                     |room <= 0
> copy_from_read_buf()|
> n_tty_kick_worker() |
>                     |ldata->no_room = true
> 
> After writing to slave device, writer wakes up kworker to flush
> data on tty_port to reader, and the kworker finds that reader
> has no room to store data so room <= 0 is met. At this moment,
> reader consumes all the data on reader buffer and call
> n_tty_kick_worker to check ldata->no_room which is false and
> reader quits reading. Then kworker sets ldata->no_room=true
> and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> This problem can be solved with a check for read buffer size inside
> n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
> is true, a call to n_tty_kick_worker is necessary to keep flushing
> data to reader.
> 
> Signed-off-by: cael <juanfengpy@gmail.com>
> Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
> 
> ---
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..21241ea7cdb9 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
>         struct n_tty_data *ldata = tty->disc_data;
> 
>         /* Did the input worker stop? Restart it */
> -       if (unlikely(ldata->no_room)) {
> -               ldata->no_room = 0;
> +       if (unlikely(READ_ONCE(ldata->no_room))) {
> +               WRITE_ONCE(ldata->no_room, 0);
> 
>                 WARN_RATELIMIT(tty->port->itty == NULL,
>                                 "scheduling with invalid itty\n");
> @@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>                         if (overflow && room < 0)
>                                 ldata->read_head--;
>                         room = overflow;
> -                       ldata->no_room = flow && !room;
> +                       WRITE_ONCE(ldata->no_room, flow && !room);
>                 } else
>                         overflow = 0;
> 
> @@ -1663,6 +1663,21 @@ n_tty_receive_buf_common(struct tty_struct
> *tty, const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (READ_ONCE(ldata->no_room)) {
> +               /*
> +                * Reader ensures that read_tail is updated before
> checking no_room,
> +                * make sure that no_room is set before reading read_tail here.
> +                * Now no_room is visible by reader, the race needs to
> be handled is
> +                * that reader has passed checkpoint for no_room and
> reader buffer
> +                * is empty, if so n_tty_kick_worker will not be
> called by reader,
> +                * instead, this function is called here.
> +                * barrier is paired with smp_mb() in n_tty_read()
> +                */
> +               smp_mb();
> +               if (!chars_in_buffer(tty))
> +                       n_tty_kick_worker(tty);
> +       }
> +
>         up_read(&tty->termios_rwsem);
> 
>         return rcvd;
> @@ -2180,8 +2195,14 @@ static ssize_t n_tty_read(struct tty_struct
> *tty, struct file *file,
>                 if (time)
>                         timeout = time;
>         }
> -       if (tail != ldata->read_tail)
> +       if (tail != ldata->read_tail) {
> +               /*
> +                * Make sure no_room is not read before setting read_tail,
> +                * paired with smp_mb() in n_tty_receive_buf_common()
> +                */
> 
> +               smp_mb();
>                 n_tty_kick_worker(tty);
> +       }
>         up_read(&tty->termios_rwsem);
> 
>         remove_wait_queue(&tty->read_wait, &wait);
> --
> 2.27.0

Hi,

This is the friendly patch-bot of Greg Kroah-Hartman.  You have sent him
a patch that has triggered this response.  He used to manually respond
to these common problems, but in order to save his sanity (he kept
writing the same thing over and over, yet to different people), I was
created.  Hopefully you will not take offence and will fix the problem
in your patch and resubmit it so that it can be accepted into the Linux
kernel tree.

You are receiving this message because of the following common error(s)
as indicated below:

- Your patch is malformed (tabs converted to spaces, linewrapped, etc.)
  and can not be applied.  Please read the file,
  Documentation/email-clients.txt in order to fix this.

- You did not write a descriptive Subject: for the patch, allowing Greg,
  and everyone else, to know what this patch is all about.  Please read
  the section entitled "The canonical patch format" in the kernel file,
  Documentation/SubmittingPatches for what a proper Subject: line should
  look like.

- It looks like you did not use your "real" name for the patch on either
  the Signed-off-by: line, or the From: line (both of which have to
  match).  Please read the kernel file, Documentation/SubmittingPatches
  for how to do this correctly.

- This looks like a new version of a previously submitted patch, but you
  did not list below the --- line any changes from the previous version.
  Please read the section entitled "The canonical patch format" in the
  kernel file, Documentation/SubmittingPatches for what needs to be done
  here to properly describe this.

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Greg will reply once he has dug out from the pending patches received
from other developers.

thanks,

greg k-h's patch email bot

Re: tty: fix a possible hang on tty device

[Ilpo Järvinen]

[-- Attachment #1: Type: text/plain, Size: 4065 bytes --]

On Wed, 1 Jun 2022, cael wrote:

> From: cael <juanfengpy@gmail.com>
> Subject: [PATCH v2] tty: fix a possible hang on tty device
> 
> We have met a hang on pty device, the reader was blocking
> at epoll on master side, the writer was sleeping at wait_woken
> inside n_tty_write on slave side, and the write buffer on
> tty_port was full, we found that the reader and writer would
> never be woken again and block forever.
> 
> The problem was caused by a race between reader and kworker:
> n_tty_read(reader):  n_tty_receive_buf_common(kworker):
>                     |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
>                     |room <= 0
> copy_from_read_buf()|
> n_tty_kick_worker() |
>                     |ldata->no_room = true
> 
> After writing to slave device, writer wakes up kworker to flush
> data on tty_port to reader, and the kworker finds that reader
> has no room to store data so room <= 0 is met. At this moment,
> reader consumes all the data on reader buffer and call
> n_tty_kick_worker to check ldata->no_room which is false and
> reader quits reading. Then kworker sets ldata->no_room=true
> and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> This problem can be solved with a check for read buffer size inside
> n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
> is true, a call to n_tty_kick_worker is necessary to keep flushing
> data to reader.
> 
> Signed-off-by: cael <juanfengpy@gmail.com>
> Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>

You should not add Reviewed-by on your own. Only after the person 
himself/herself gives that tag for you, include it.

> 
> ---
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..21241ea7cdb9 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
>         struct n_tty_data *ldata = tty->disc_data;
> 
>         /* Did the input worker stop? Restart it */
> -       if (unlikely(ldata->no_room)) {
> -               ldata->no_room = 0;
> +       if (unlikely(READ_ONCE(ldata->no_room))) {
> +               WRITE_ONCE(ldata->no_room, 0);
> 
>                 WARN_RATELIMIT(tty->port->itty == NULL,
>                                 "scheduling with invalid itty\n");
> @@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>                         if (overflow && room < 0)
>                                 ldata->read_head--;
>                         room = overflow;
> -                       ldata->no_room = flow && !room;
> +                       WRITE_ONCE(ldata->no_room, flow && !room);
>                 } else
>                         overflow = 0;
> 
> @@ -1663,6 +1663,21 @@ n_tty_receive_buf_common(struct tty_struct
> *tty, const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (READ_ONCE(ldata->no_room)) {

Hmm, since this function is only one setting it to non-zero value, perhaps 
the information could be carried over here in a no_room local var (and 
maybe unlikely() would be useful too similar to n_tty_kick_worker). After 
all, this check is just an optimization for the common case where we know 
no_room is definitely zero.

> +               /*
> +                * Reader ensures that read_tail is updated before
> checking no_room,
> +                * make sure that no_room is set before reading read_tail here.



> +                * Now no_room is visible by reader, the race needs to
> be handled is
> +                * that reader has passed checkpoint for no_room and
> reader buffer
> +                * is empty, if so n_tty_kick_worker will not be
> called by reader,
> +                * instead, this function is called here.

This part is hard to parse/understand. Please try to rephrase.


-- 
 i.

Re: tty: fix a possible hang on tty device

[cael]

[-- Attachment #1: Type: text/plain, Size: 4903 bytes --]

From: cael <juanfengpy@gmail.com>
Subject:[PATCH v3] tty: fix a possible hang on tty device

We have met a hang on pty device, the reader was blocking
at epoll on master side, the writer was sleeping at wait_woken
inside n_tty_write on slave side, and the write buffer on
tty_port was full, we found that the reader and writer would
never be woken again and block forever.

The problem was caused by a race between reader and kworker:
n_tty_read(reader):  n_tty_receive_buf_common(kworker):
                    |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
                    |room <= 0
copy_from_read_buf()|
n_tty_kick_worker() |
                    |ldata->no_room = true

After writing to slave device, writer wakes up kworker to flush
data on tty_port to reader, and the kworker finds that reader
has no room to store data so room <= 0 is met. At this moment,
reader consumes all the data on reader buffer and call
n_tty_kick_worker to check ldata->no_room which is false and
reader quits reading. Then kworker sets ldata->no_room=true
and quits too.

If write buffer is not full, writer will wake kworker to flush data
again after following writes, but if writer buffer is full and writer
goes to sleep, kworker will never be woken again and tty device is
blocked.

This problem can be solved with a check for read buffer size inside
n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
is true, a call to n_tty_kick_worker is necessary to keep flushing
data to reader.

Signed-off-by: cael <juanfengpy@gmail.com>

diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index efc72104c840..544f782b9a11 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
        struct n_tty_data *ldata = tty->disc_data;

        /* Did the input worker stop? Restart it */
-       if (unlikely(ldata->no_room)) {
-               ldata->no_room = 0;
+       if (unlikely(READ_ONCE(ldata->no_room))) {
+               WRITE_ONCE(ldata->no_room, 0);

                WARN_RATELIMIT(tty->port->itty == NULL,
                                "scheduling with invalid itty\n");
@@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
                        if (overflow && room < 0)
                                ldata->read_head--;
                        room = overflow;
-                       ldata->no_room = flow && !room;
+                       WRITE_ONCE(ldata->no_room, flow && !room);
                } else
                        overflow = 0;

@@ -1663,6 +1663,24 @@ n_tty_receive_buf_common(struct tty_struct
*tty, const unsigned char *cp,
        } else
                n_tty_check_throttle(tty);

+       if (unlikely(ldata->no_room)) {
+               /*
+                * Barrier here is to ensure to read the latest read_tail in
+                * chars_in_buffer() and to make sure that read_tail
is not loaded
+                * before ldata->no_room is set, otherwise, following
race may occur:
+                * n_tty_receive_buf_common() |n_tty_read()
+                * chars_in_buffer() > 0      |
+                *
|copy_from_read_buf()->chars_in_buffer()==0
+                *                            |if (ldata->no_room)
+                * ldata->no_room = 1         |
+                * Then both kworker and reader will fail to kick
n_tty_kick_worker(),
+                * smp_mb is paired with smp_mb() in n_tty_read().
+                */
+               smp_mb();
+               if (!chars_in_buffer(tty))
+                       n_tty_kick_worker(tty);
+       }
+
        up_read(&tty->termios_rwsem);

        return rcvd;
@@ -2180,8 +2198,23 @@ static ssize_t n_tty_read(struct tty_struct
*tty, struct file *file,
                if (time)
                        timeout = time;
        }
-       if (tail != ldata->read_tail)
+       if (tail != ldata->read_tail) {
+               /*
+                * Make sure no_room is not read before setting read_tail,
+                * otherwise, following race may occur:
+                * n_tty_read()
|n_tty_receive_buf_common()
+                * if(ldata->no_room)->false            |
+                *                                      |ldata->no_room = 1
+                *                                      |char_in_buffer() > 0
+                * ldata->read_tail = ldata->commit_head|
+                * Then copy_from_read_buf() in reader consumes all the data
+                * in read buffer, both reader and kworker will fail to kick
+                * tty_buffer_restart_work().
+                * smp_mb is paired with smp_mb() in n_tty_receive_buf_common().
+                */
+               smp_mb();
                n_tty_kick_worker(tty);
+       }
        up_read(&tty->termios_rwsem);

        remove_wait_queue(&tty->read_wait, &wait);
-- 
2.27.0

[-- Attachment #2: 0001-PATCH-v3-tty-fix-a-possible-hang-on-tty-device.patch --]
[-- Type: application/octet-stream, Size: 4314 bytes --]

From 6d213bd916fce9140557221a7eff0d65bd33df57 Mon Sep 17 00:00:00 2001
From: cael <juanfengpy@gmail.com>
Date: Mon, 23 May 2022 20:53:55 +0800
Subject: [PATCH] [PATCH v3] tty: fix a possible hang on tty device

We have met a hang on pty device, the reader was blocking
at epoll on master side, the writer was sleeping at wait_woken
inside n_tty_write on slave side, and the write buffer on
tty_port was full, we found that the reader and writer would
never be woken again and block forever.

The problem was caused by a race between reader and kworker:
n_tty_read(reader):  n_tty_receive_buf_common(kworker):
                    |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
                    |room <= 0
copy_from_read_buf()|
n_tty_kick_worker() |
                    |ldata->no_room = true

After writing to slave device, writer wakes up kworker to flush
data on tty_port to reader, and the kworker finds that reader
has no room to store data so room <= 0 is met. At this moment,
reader consumes all the data on reader buffer and call
n_tty_kick_worker to check ldata->no_room which is false and
reader quits reading. Then kworker sets ldata->no_room=true
and quits too.

If write buffer is not full, writer will wake kworker to flush data
again after following writes, but if writer buffer is full and writer
goes to sleep, kworker will never be woken again and tty device is
blocked.

This problem can be solved with a check for read buffer size inside
n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
is true, a call to n_tty_kick_worker is necessary to keep flushing
data to reader.

Signed-off-by: cael <juanfengpy@gmail.com>

diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index efc72104c840..544f782b9a11 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
 	struct n_tty_data *ldata = tty->disc_data;
 
 	/* Did the input worker stop? Restart it */
-	if (unlikely(ldata->no_room)) {
-		ldata->no_room = 0;
+	if (unlikely(READ_ONCE(ldata->no_room))) {
+		WRITE_ONCE(ldata->no_room, 0);
 
 		WARN_RATELIMIT(tty->port->itty == NULL,
 				"scheduling with invalid itty\n");
@@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty, const unsigned char *cp,
 			if (overflow && room < 0)
 				ldata->read_head--;
 			room = overflow;
-			ldata->no_room = flow && !room;
+			WRITE_ONCE(ldata->no_room, flow && !room);
 		} else
 			overflow = 0;
 
@@ -1663,6 +1663,24 @@ n_tty_receive_buf_common(struct tty_struct *tty, const unsigned char *cp,
 	} else
 		n_tty_check_throttle(tty);
 
+	if (unlikely(ldata->no_room)) {
+		/*
+		 * Barrier here is to ensure to read the latest read_tail in
+		 * chars_in_buffer() and to make sure that read_tail is not loaded
+		 * before ldata->no_room is set, otherwise, following race may occur:
+		 * n_tty_receive_buf_common() |n_tty_read()
+		 * chars_in_buffer() > 0      |
+		 *                            |copy_from_read_buf()->chars_in_buffer()==0
+		 *                            |if (ldata->no_room)
+		 * ldata->no_room = 1         |
+		 * Then both kworker and reader will fail to kick n_tty_kick_worker(),
+		 * smp_mb is paired with smp_mb() in n_tty_read().
+		 */
+		smp_mb();
+		if (!chars_in_buffer(tty))
+			n_tty_kick_worker(tty);
+	}
+
 	up_read(&tty->termios_rwsem);
 
 	return rcvd;
@@ -2180,8 +2198,23 @@ static ssize_t n_tty_read(struct tty_struct *tty, struct file *file,
 		if (time)
 			timeout = time;
 	}
-	if (tail != ldata->read_tail)
+	if (tail != ldata->read_tail) {
+		/*
+		 * Make sure no_room is not read before setting read_tail,
+		 * otherwise, following race may occur:
+		 * n_tty_read()		                |n_tty_receive_buf_common()
+		 * if(ldata->no_room)->false            |
+		 *			                |ldata->no_room = 1
+		 *                                      |char_in_buffer() > 0
+		 * ldata->read_tail = ldata->commit_head|
+		 * Then copy_from_read_buf() in reader consumes all the data
+		 * in read buffer, both reader and kworker will fail to kick
+		 * tty_buffer_restart_work().
+		 * smp_mb is paired with smp_mb() in n_tty_receive_buf_common().
+		 */
+		smp_mb();
 		n_tty_kick_worker(tty);
+	}
 	up_read(&tty->termios_rwsem);
 
 	remove_wait_queue(&tty->read_wait, &wait);
-- 
2.27.0

Re: tty: fix a possible hang on tty device

[Greg KH]

On Mon, Jun 06, 2022 at 09:40:16PM +0800, cael wrote:
> From: cael <juanfengpy@gmail.com>
> Subject:[PATCH v3] tty: fix a possible hang on tty device
> 
> We have met a hang on pty device, the reader was blocking
> at epoll on master side, the writer was sleeping at wait_woken
> inside n_tty_write on slave side, and the write buffer on
> tty_port was full, we found that the reader and writer would
> never be woken again and block forever.
> 
> The problem was caused by a race between reader and kworker:
> n_tty_read(reader):  n_tty_receive_buf_common(kworker):
>                     |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
>                     |room <= 0
> copy_from_read_buf()|
> n_tty_kick_worker() |
>                     |ldata->no_room = true
> 
> After writing to slave device, writer wakes up kworker to flush
> data on tty_port to reader, and the kworker finds that reader
> has no room to store data so room <= 0 is met. At this moment,
> reader consumes all the data on reader buffer and call
> n_tty_kick_worker to check ldata->no_room which is false and
> reader quits reading. Then kworker sets ldata->no_room=true
> and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> This problem can be solved with a check for read buffer size inside
> n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
> is true, a call to n_tty_kick_worker is necessary to keep flushing
> data to reader.
> 
> Signed-off-by: cael <juanfengpy@gmail.com>
> 
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..544f782b9a11 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
>         struct n_tty_data *ldata = tty->disc_data;
> 
>         /* Did the input worker stop? Restart it */
> -       if (unlikely(ldata->no_room)) {
> -               ldata->no_room = 0;
> +       if (unlikely(READ_ONCE(ldata->no_room))) {
> +               WRITE_ONCE(ldata->no_room, 0);
> 
>                 WARN_RATELIMIT(tty->port->itty == NULL,
>                                 "scheduling with invalid itty\n");
> @@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>                         if (overflow && room < 0)
>                                 ldata->read_head--;
>                         room = overflow;
> -                       ldata->no_room = flow && !room;
> +                       WRITE_ONCE(ldata->no_room, flow && !room);
>                 } else
>                         overflow = 0;
> 
> @@ -1663,6 +1663,24 @@ n_tty_receive_buf_common(struct tty_struct
> *tty, const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (unlikely(ldata->no_room)) {
> +               /*
> +                * Barrier here is to ensure to read the latest read_tail in
> +                * chars_in_buffer() and to make sure that read_tail
> is not loaded
> +                * before ldata->no_room is set, otherwise, following
> race may occur:
> +                * n_tty_receive_buf_common() |n_tty_read()
> +                * chars_in_buffer() > 0      |
> +                *
> |copy_from_read_buf()->chars_in_buffer()==0
> +                *                            |if (ldata->no_room)
> +                * ldata->no_room = 1         |
> +                * Then both kworker and reader will fail to kick
> n_tty_kick_worker(),
> +                * smp_mb is paired with smp_mb() in n_tty_read().
> +                */
> +               smp_mb();
> +               if (!chars_in_buffer(tty))
> +                       n_tty_kick_worker(tty);
> +       }
> +
>         up_read(&tty->termios_rwsem);
> 
>         return rcvd;
> @@ -2180,8 +2198,23 @@ static ssize_t n_tty_read(struct tty_struct
> *tty, struct file *file,
>                 if (time)
>                         timeout = time;
>         }
> -       if (tail != ldata->read_tail)
> +       if (tail != ldata->read_tail) {
> +               /*
> +                * Make sure no_room is not read before setting read_tail,
> +                * otherwise, following race may occur:
> +                * n_tty_read()
> |n_tty_receive_buf_common()
> +                * if(ldata->no_room)->false            |
> +                *                                      |ldata->no_room = 1
> +                *                                      |char_in_buffer() > 0
> +                * ldata->read_tail = ldata->commit_head|
> +                * Then copy_from_read_buf() in reader consumes all the data
> +                * in read buffer, both reader and kworker will fail to kick
> +                * tty_buffer_restart_work().
> +                * smp_mb is paired with smp_mb() in n_tty_receive_buf_common().
> +                */
> +               smp_mb();
>                 n_tty_kick_worker(tty);
> +       }
>         up_read(&tty->termios_rwsem);
> 
>         remove_wait_queue(&tty->read_wait, &wait);
> -- 
> 2.27.0


Hi,

This is the friendly patch-bot of Greg Kroah-Hartman.  You have sent him
a patch that has triggered this response.  He used to manually respond
to these common problems, but in order to save his sanity (he kept
writing the same thing over and over, yet to different people), I was
created.  Hopefully you will not take offence and will fix the problem
in your patch and resubmit it so that it can be accepted into the Linux
kernel tree.

You are receiving this message because of the following common error(s)
as indicated below:

- Your patch is malformed (tabs converted to spaces, linewrapped, etc.)
  and can not be applied.  Please read the file,
  Documentation/email-clients.txt in order to fix this.

- Your patch was attached, please place it inline so that it can be
  applied directly from the email message itself.

- You did not write a descriptive Subject: for the patch, allowing Greg,
  and everyone else, to know what this patch is all about.  Please read
  the section entitled "The canonical patch format" in the kernel file,
  Documentation/SubmittingPatches for what a proper Subject: line should
  look like.

- It looks like you did not use your "real" name for the patch on either
  the Signed-off-by: line, or the From: line (both of which have to
  match).  Please read the kernel file, Documentation/SubmittingPatches
  for how to do this correctly.

- This looks like a new version of a previously submitted patch, but you
  did not list below the --- line any changes from the previous version.
  Please read the section entitled "The canonical patch format" in the
  kernel file, Documentation/SubmittingPatches for what needs to be done
  here to properly describe this.

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Greg will reply once he has dug out from the pending patches received
from other developers.

thanks,

greg k-h's patch email bot

Re: tty: fix a possible hang on tty device

[cael]

From: cael <juanfengpy@gmail.com>
Subject: [PATCH] [PATCH v3] tty: fix a possible hang on tty device

We have met a hang on pty device, the reader was blocking
at epoll on master side, the writer was sleeping at wait_woken
inside n_tty_write on slave side, and the write buffer on
tty_port was full, we found that the reader and writer would
never be woken again and block forever.

The problem was caused by a race between reader and kworker:
n_tty_read(reader):  n_tty_receive_buf_common(kworker):
                    |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
                    |room <= 0
copy_from_read_buf()|
n_tty_kick_worker() |
                    |ldata->no_room = true

After writing to slave device, writer wakes up kworker to flush
data on tty_port to reader, and the kworker finds that reader
has no room to store data so room <= 0 is met. At this moment,
reader consumes all the data on reader buffer and call
n_tty_kick_worker to check ldata->no_room which is false and
reader quits reading. Then kworker sets ldata->no_room=true
and quits too.

If write buffer is not full, writer will wake kworker to flush data
again after following writes, but if writer buffer is full and writer
goes to sleep, kworker will never be woken again and tty device is
blocked.

This problem can be solved with a check for read buffer size inside
n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
is true, a call to n_tty_kick_worker is necessary to keep flushing
data to reader.

Signed-off-by: cael <juanfengpy@gmail.com>

diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index efc72104c840..544f782b9a11 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
        struct n_tty_data *ldata = tty->disc_data;

        /* Did the input worker stop? Restart it */
-       if (unlikely(ldata->no_room)) {
-               ldata->no_room = 0;
+       if (unlikely(READ_ONCE(ldata->no_room))) {
+               WRITE_ONCE(ldata->no_room, 0);

                WARN_RATELIMIT(tty->port->itty == NULL,
                                "scheduling with invalid itty\n");
@@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
const unsigned char *cp,
                        if (overflow && room < 0)
                                ldata->read_head--;
                        room = overflow;
-                       ldata->no_room = flow && !room;
+                       WRITE_ONCE(ldata->no_room, flow && !room);
                } else
                        overflow = 0;

@@ -1663,6 +1663,24 @@ n_tty_receive_buf_common(struct tty_struct
*tty, const unsigned char *cp,
        } else
                n_tty_check_throttle(tty);

+       if (unlikely(ldata->no_room)) {
+               /*
+                * Barrier here is to ensure to read the latest read_tail in
+                * chars_in_buffer() and to make sure that read_tail
is not loaded
+                * before ldata->no_room is set, otherwise, following
race may occur:
+                * n_tty_receive_buf_common() |n_tty_read()
+                * chars_in_buffer() > 0      |
+                *
|copy_from_read_buf()->chars_in_buffer()==0
+                *                            |if (ldata->no_room)
+                * ldata->no_room = 1         |
+                * Then both kworker and reader will fail to kick
n_tty_kick_worker(),
+                * smp_mb is paired with smp_mb() in n_tty_read().
+                */
+               smp_mb();
+               if (!chars_in_buffer(tty))
+                       n_tty_kick_worker(tty);
+       }
+
        up_read(&tty->termios_rwsem);

        return rcvd;
@@ -2180,8 +2198,23 @@ static ssize_t n_tty_read(struct tty_struct
*tty, struct file *file,
                if (time)
                        timeout = time;
        }
-       if (tail != ldata->read_tail)
+       if (tail != ldata->read_tail) {
+               /*
+                * Make sure no_room is not read before setting read_tail,
+                * otherwise, following race may occur:
+                * n_tty_read()
|n_tty_receive_buf_common()
+                * if(ldata->no_room)->false            |
+                *                                      |ldata->no_room = 1
+                *                                      |char_in_buffer() > 0
+                * ldata->read_tail = ldata->commit_head|
+                * Then copy_from_read_buf() in reader consumes all the data
+                * in read buffer, both reader and kworker will fail to kick
+                * tty_buffer_restart_work().
+                * smp_mb is paired with smp_mb() in n_tty_receive_buf_common().
+                */
+               smp_mb();
                n_tty_kick_worker(tty);
+       }
        up_read(&tty->termios_rwsem);

        remove_wait_queue(&tty->read_wait, &wait);
-- 
2.27.0

Greg KH <gregkh@linuxfoundation.org> 于2022年6月6日周一 22:43写道：
>
> On Mon, Jun 06, 2022 at 09:40:16PM +0800, cael wrote:
> > From: cael <juanfengpy@gmail.com>
> > Subject:[PATCH v3] tty: fix a possible hang on tty device
> >
> > We have met a hang on pty device, the reader was blocking
> > at epoll on master side, the writer was sleeping at wait_woken
> > inside n_tty_write on slave side, and the write buffer on
> > tty_port was full, we found that the reader and writer would
> > never be woken again and block forever.
> >
> > The problem was caused by a race between reader and kworker:
> > n_tty_read(reader):  n_tty_receive_buf_common(kworker):
> >                     |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
> >                     |room <= 0
> > copy_from_read_buf()|
> > n_tty_kick_worker() |
> >                     |ldata->no_room = true
> >
> > After writing to slave device, writer wakes up kworker to flush
> > data on tty_port to reader, and the kworker finds that reader
> > has no room to store data so room <= 0 is met. At this moment,
> > reader consumes all the data on reader buffer and call
> > n_tty_kick_worker to check ldata->no_room which is false and
> > reader quits reading. Then kworker sets ldata->no_room=true
> > and quits too.
> >
> > If write buffer is not full, writer will wake kworker to flush data
> > again after following writes, but if writer buffer is full and writer
> > goes to sleep, kworker will never be woken again and tty device is
> > blocked.
> >
> > This problem can be solved with a check for read buffer size inside
> > n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
> > is true, a call to n_tty_kick_worker is necessary to keep flushing
> > data to reader.
> >
> > Signed-off-by: cael <juanfengpy@gmail.com>
> >
> > diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> > index efc72104c840..544f782b9a11 100644
> > --- a/drivers/tty/n_tty.c
> > +++ b/drivers/tty/n_tty.c
> > @@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
> >         struct n_tty_data *ldata = tty->disc_data;
> >
> >         /* Did the input worker stop? Restart it */
> > -       if (unlikely(ldata->no_room)) {
> > -               ldata->no_room = 0;
> > +       if (unlikely(READ_ONCE(ldata->no_room))) {
> > +               WRITE_ONCE(ldata->no_room, 0);
> >
> >                 WARN_RATELIMIT(tty->port->itty == NULL,
> >                                 "scheduling with invalid itty\n");
> > @@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> > const unsigned char *cp,
> >                         if (overflow && room < 0)
> >                                 ldata->read_head--;
> >                         room = overflow;
> > -                       ldata->no_room = flow && !room;
> > +                       WRITE_ONCE(ldata->no_room, flow && !room);
> >                 } else
> >                         overflow = 0;
> >
> > @@ -1663,6 +1663,24 @@ n_tty_receive_buf_common(struct tty_struct
> > *tty, const unsigned char *cp,
> >         } else
> >                 n_tty_check_throttle(tty);
> >
> > +       if (unlikely(ldata->no_room)) {
> > +               /*
> > +                * Barrier here is to ensure to read the latest read_tail in
> > +                * chars_in_buffer() and to make sure that read_tail
> > is not loaded
> > +                * before ldata->no_room is set, otherwise, following
> > race may occur:
> > +                * n_tty_receive_buf_common() |n_tty_read()
> > +                * chars_in_buffer() > 0      |
> > +                *
> > |copy_from_read_buf()->chars_in_buffer()==0
> > +                *                            |if (ldata->no_room)
> > +                * ldata->no_room = 1         |
> > +                * Then both kworker and reader will fail to kick
> > n_tty_kick_worker(),
> > +                * smp_mb is paired with smp_mb() in n_tty_read().
> > +                */
> > +               smp_mb();
> > +               if (!chars_in_buffer(tty))
> > +                       n_tty_kick_worker(tty);
> > +       }
> > +
> >         up_read(&tty->termios_rwsem);
> >
> >         return rcvd;
> > @@ -2180,8 +2198,23 @@ static ssize_t n_tty_read(struct tty_struct
> > *tty, struct file *file,
> >                 if (time)
> >                         timeout = time;
> >         }
> > -       if (tail != ldata->read_tail)
> > +       if (tail != ldata->read_tail) {
> > +               /*
> > +                * Make sure no_room is not read before setting read_tail,
> > +                * otherwise, following race may occur:
> > +                * n_tty_read()
> > |n_tty_receive_buf_common()
> > +                * if(ldata->no_room)->false            |
> > +                *                                      |ldata->no_room = 1
> > +                *                                      |char_in_buffer() > 0
> > +                * ldata->read_tail = ldata->commit_head|
> > +                * Then copy_from_read_buf() in reader consumes all the data
> > +                * in read buffer, both reader and kworker will fail to kick
> > +                * tty_buffer_restart_work().
> > +                * smp_mb is paired with smp_mb() in n_tty_receive_buf_common().
> > +                */
> > +               smp_mb();
> >                 n_tty_kick_worker(tty);
> > +       }
> >         up_read(&tty->termios_rwsem);
> >
> >         remove_wait_queue(&tty->read_wait, &wait);
> > --
> > 2.27.0
>
>
> Hi,
>
> This is the friendly patch-bot of Greg Kroah-Hartman.  You have sent him
> a patch that has triggered this response.  He used to manually respond
> to these common problems, but in order to save his sanity (he kept
> writing the same thing over and over, yet to different people), I was
> created.  Hopefully you will not take offence and will fix the problem
> in your patch and resubmit it so that it can be accepted into the Linux
> kernel tree.
>
> You are receiving this message because of the following common error(s)
> as indicated below:
>
> - Your patch is malformed (tabs converted to spaces, linewrapped, etc.)
>   and can not be applied.  Please read the file,
>   Documentation/email-clients.txt in order to fix this.
>
> - Your patch was attached, please place it inline so that it can be
>   applied directly from the email message itself.
>
> - You did not write a descriptive Subject: for the patch, allowing Greg,
>   and everyone else, to know what this patch is all about.  Please read
>   the section entitled "The canonical patch format" in the kernel file,
>   Documentation/SubmittingPatches for what a proper Subject: line should
>   look like.
>
> - It looks like you did not use your "real" name for the patch on either
>   the Signed-off-by: line, or the From: line (both of which have to
>   match).  Please read the kernel file, Documentation/SubmittingPatches
>   for how to do this correctly.
>
> - This looks like a new version of a previously submitted patch, but you
>   did not list below the --- line any changes from the previous version.
>   Please read the section entitled "The canonical patch format" in the
>   kernel file, Documentation/SubmittingPatches for what needs to be done
>   here to properly describe this.
>
> If you wish to discuss this problem further, or you have questions about
> how to resolve this issue, please feel free to respond to this email and
> Greg will reply once he has dug out from the pending patches received
> from other developers.
>
> thanks,
>
> greg k-h's patch email bot

Re: tty: fix a possible hang on tty device

[Greg KH]

On Sat, Jun 11, 2022 at 02:50:54PM +0800, cael wrote:
> From: cael <juanfengpy@gmail.com>
> Subject: [PATCH] [PATCH v3] tty: fix a possible hang on tty device
> 
> We have met a hang on pty device, the reader was blocking
> at epoll on master side, the writer was sleeping at wait_woken
> inside n_tty_write on slave side, and the write buffer on
> tty_port was full, we found that the reader and writer would
> never be woken again and block forever.
> 
> The problem was caused by a race between reader and kworker:
> n_tty_read(reader):  n_tty_receive_buf_common(kworker):
>                     |room = N_TTY_BUF_SIZE - (ldata->read_head - tail)
>                     |room <= 0
> copy_from_read_buf()|
> n_tty_kick_worker() |
>                     |ldata->no_room = true
> 
> After writing to slave device, writer wakes up kworker to flush
> data on tty_port to reader, and the kworker finds that reader
> has no room to store data so room <= 0 is met. At this moment,
> reader consumes all the data on reader buffer and call
> n_tty_kick_worker to check ldata->no_room which is false and
> reader quits reading. Then kworker sets ldata->no_room=true
> and quits too.
> 
> If write buffer is not full, writer will wake kworker to flush data
> again after following writes, but if writer buffer is full and writer
> goes to sleep, kworker will never be woken again and tty device is
> blocked.
> 
> This problem can be solved with a check for read buffer size inside
> n_tty_receive_buf_common, if read buffer is empty and ldata->no_room
> is true, a call to n_tty_kick_worker is necessary to keep flushing
> data to reader.
> 
> Signed-off-by: cael <juanfengpy@gmail.com>
> 
> diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
> index efc72104c840..544f782b9a11 100644
> --- a/drivers/tty/n_tty.c
> +++ b/drivers/tty/n_tty.c
> @@ -201,8 +201,8 @@ static void n_tty_kick_worker(struct tty_struct *tty)
>         struct n_tty_data *ldata = tty->disc_data;
> 
>         /* Did the input worker stop? Restart it */
> -       if (unlikely(ldata->no_room)) {
> -               ldata->no_room = 0;
> +       if (unlikely(READ_ONCE(ldata->no_room))) {
> +               WRITE_ONCE(ldata->no_room, 0);
> 
>                 WARN_RATELIMIT(tty->port->itty == NULL,
>                                 "scheduling with invalid itty\n");
> @@ -1632,7 +1632,7 @@ n_tty_receive_buf_common(struct tty_struct *tty,
> const unsigned char *cp,
>                         if (overflow && room < 0)
>                                 ldata->read_head--;
>                         room = overflow;
> -                       ldata->no_room = flow && !room;
> +                       WRITE_ONCE(ldata->no_room, flow && !room);
>                 } else
>                         overflow = 0;
> 
> @@ -1663,6 +1663,24 @@ n_tty_receive_buf_common(struct tty_struct
> *tty, const unsigned char *cp,
>         } else
>                 n_tty_check_throttle(tty);
> 
> +       if (unlikely(ldata->no_room)) {
> +               /*
> +                * Barrier here is to ensure to read the latest read_tail in
> +                * chars_in_buffer() and to make sure that read_tail
> is not loaded
> +                * before ldata->no_room is set, otherwise, following
> race may occur:
> +                * n_tty_receive_buf_common() |n_tty_read()
> +                * chars_in_buffer() > 0      |
> +                *
> |copy_from_read_buf()->chars_in_buffer()==0
> +                *                            |if (ldata->no_room)
> +                * ldata->no_room = 1         |
> +                * Then both kworker and reader will fail to kick
> n_tty_kick_worker(),
> +                * smp_mb is paired with smp_mb() in n_tty_read().
> +                */
> +               smp_mb();
> +               if (!chars_in_buffer(tty))
> +                       n_tty_kick_worker(tty);
> +       }
> +
>         up_read(&tty->termios_rwsem);
> 
>         return rcvd;
> @@ -2180,8 +2198,23 @@ static ssize_t n_tty_read(struct tty_struct
> *tty, struct file *file,
>                 if (time)
>                         timeout = time;
>         }
> -       if (tail != ldata->read_tail)
> +       if (tail != ldata->read_tail) {
> +               /*
> +                * Make sure no_room is not read before setting read_tail,
> +                * otherwise, following race may occur:
> +                * n_tty_read()
> |n_tty_receive_buf_common()
> +                * if(ldata->no_room)->false            |
> +                *                                      |ldata->no_room = 1
> +                *                                      |char_in_buffer() > 0
> +                * ldata->read_tail = ldata->commit_head|
> +                * Then copy_from_read_buf() in reader consumes all the data
> +                * in read buffer, both reader and kworker will fail to kick
> +                * tty_buffer_restart_work().
> +                * smp_mb is paired with smp_mb() in n_tty_receive_buf_common().
> +                */
> +               smp_mb();
>                 n_tty_kick_worker(tty);
> +       }
>         up_read(&tty->termios_rwsem);
> 
>         remove_wait_queue(&tty->read_wait, &wait);
> -- 
> 2.27.0

Is there any specific reason you ignored all of the recommendations from
my previous email as to what needs to be changed in order for this patch
to be accepted?  It doesn't make any sense for me to just keep sending
the same information again :(

thanks,

greg k-h