Potential data race in flush_to_ldisc

Potential data race in flush_to_ldisc

[Dmitry Vyukov]

Hello,

We are working on a dynamic data race detector for the Linux kernel,
KernelThreadSanitizer (ktsan):
https://github.com/google/ktsan/wiki

While booting kernel (upstream revision 21bdb584af8c) we got a report:

ThreadSanitizer: data-race in release_tty

Write of size 8 by thread T325 (K2579):
 [<ffffffff81655c43>] release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688
 [<ffffffff816563a8>] tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920
 [<ffffffff8126154f>] __fput+0x15f/0x310 fs/file_table.c:207
 [<ffffffff8126176d>] ____fput+0x1d/0x30 fs/file_table.c:243
 [<ffffffff810b9485>] task_work_run+0x115/0x130 kernel/task_work.c:123
(discriminator 1)
 [<     inlined    >] do_notify_resume+0x73/0x80
tracehook_notify_resume include/linux/tracehook.h:190
 [<ffffffff81006da3>] do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757
 [<ffffffff81ee25fc>] int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326

Previous read of size 8 by thread T19 (K16):
 [<ffffffff816624d9>] flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472
 [<ffffffff810b1fce>] process_one_work+0x47e/0x930 kernel/workqueue.c:2036
 [<ffffffff810b2530>] worker_thread+0xb0/0x900 kernel/workqueue.c:2170
 [<ffffffff810bbbd0>] kthread+0x150/0x170 kernel/kthread.c:207
 [<ffffffff81ee281f>] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:526


flush_to_ldisc accesses port->itty:

static void flush_to_ldisc(struct work_struct *work)
{
   ...
    tty = port->itty;
    if (tty == NULL)
        return;
    disc = tty_ldisc_ref(tty);

while release_tty concurrently sets itty to NULL:

static void release_tty(struct tty_struct *tty, int idx)
{
    ...
    tty->port->itty = NULL;
    if (tty->link)
        tty->link->port->itty = NULL;
    cancel_work_sync(&tty->port->buf.work);
    tty_kref_put(tty->link);
    tty_kref_put(tty);
}

It seems that read of port->itty requires to be at least READ_ONCE,
because otherwise flush_to_ldisc can check that itty is not NULL, then
re-read it again and crash with NULL deref.
I don't know what is ownership and locking story here. There can be
larger issue here: either a lock is missing, or itty can be deleted
under flush_to_ldisc feet.

Please confirm that this is real but. If so please fix it.

Thank you

Re: Potential data race in flush_to_ldisc

[Greg Kroah-Hartman]

On Fri, Aug 28, 2015 at 06:57:17PM +0200, Dmitry Vyukov wrote:
> Hello,
> 
> We are working on a dynamic data race detector for the Linux kernel,
> KernelThreadSanitizer (ktsan):
> https://github.com/google/ktsan/wiki
> 
> While booting kernel (upstream revision 21bdb584af8c) we got a report:
> 
> ThreadSanitizer: data-race in release_tty
> 
> Write of size 8 by thread T325 (K2579):
>  [<ffffffff81655c43>] release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688
>  [<ffffffff816563a8>] tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920
>  [<ffffffff8126154f>] __fput+0x15f/0x310 fs/file_table.c:207
>  [<ffffffff8126176d>] ____fput+0x1d/0x30 fs/file_table.c:243
>  [<ffffffff810b9485>] task_work_run+0x115/0x130 kernel/task_work.c:123
> (discriminator 1)
>  [<     inlined    >] do_notify_resume+0x73/0x80
> tracehook_notify_resume include/linux/tracehook.h:190
>  [<ffffffff81006da3>] do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757
>  [<ffffffff81ee25fc>] int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326
> 
> Previous read of size 8 by thread T19 (K16):
>  [<ffffffff816624d9>] flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472
>  [<ffffffff810b1fce>] process_one_work+0x47e/0x930 kernel/workqueue.c:2036
>  [<ffffffff810b2530>] worker_thread+0xb0/0x900 kernel/workqueue.c:2170
>  [<ffffffff810bbbd0>] kthread+0x150/0x170 kernel/kthread.c:207
>  [<ffffffff81ee281f>] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:526
> 
> 
> flush_to_ldisc accesses port->itty:
> 
> static void flush_to_ldisc(struct work_struct *work)
> {
>    ...
>     tty = port->itty;
>     if (tty == NULL)
>         return;
>     disc = tty_ldisc_ref(tty);
> 
> while release_tty concurrently sets itty to NULL:
> 
> static void release_tty(struct tty_struct *tty, int idx)
> {
>     ...
>     tty->port->itty = NULL;
>     if (tty->link)
>         tty->link->port->itty = NULL;
>     cancel_work_sync(&tty->port->buf.work);
>     tty_kref_put(tty->link);
>     tty_kref_put(tty);
> }
> 
> It seems that read of port->itty requires to be at least READ_ONCE,
> because otherwise flush_to_ldisc can check that itty is not NULL, then
> re-read it again and crash with NULL deref.
> I don't know what is ownership and locking story here. There can be
> larger issue here: either a lock is missing, or itty can be deleted
> under flush_to_ldisc feet.
> 
> Please confirm that this is real but. If so please fix it.

Patches are always gladly accepted.  Don't force us to try to determine
if your tool is finding false-positives or not.  That is your
responsibility, not ours :)

thanks,

greg k-h

Re: Potential data race in flush_to_ldisc

[Dmitry Vyukov]

On Fri, Aug 28, 2015 at 8:10 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Fri, Aug 28, 2015 at 06:57:17PM +0200, Dmitry Vyukov wrote:
>> Hello,
>>
>> We are working on a dynamic data race detector for the Linux kernel,
>> KernelThreadSanitizer (ktsan):
>> https://github.com/google/ktsan/wiki
>>
>> While booting kernel (upstream revision 21bdb584af8c) we got a report:
>>
>> ThreadSanitizer: data-race in release_tty
>>
>> Write of size 8 by thread T325 (K2579):
>>  [<ffffffff81655c43>] release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688
>>  [<ffffffff816563a8>] tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920
>>  [<ffffffff8126154f>] __fput+0x15f/0x310 fs/file_table.c:207
>>  [<ffffffff8126176d>] ____fput+0x1d/0x30 fs/file_table.c:243
>>  [<ffffffff810b9485>] task_work_run+0x115/0x130 kernel/task_work.c:123
>> (discriminator 1)
>>  [<     inlined    >] do_notify_resume+0x73/0x80
>> tracehook_notify_resume include/linux/tracehook.h:190
>>  [<ffffffff81006da3>] do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757
>>  [<ffffffff81ee25fc>] int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326
>>
>> Previous read of size 8 by thread T19 (K16):
>>  [<ffffffff816624d9>] flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472
>>  [<ffffffff810b1fce>] process_one_work+0x47e/0x930 kernel/workqueue.c:2036
>>  [<ffffffff810b2530>] worker_thread+0xb0/0x900 kernel/workqueue.c:2170
>>  [<ffffffff810bbbd0>] kthread+0x150/0x170 kernel/kthread.c:207
>>  [<ffffffff81ee281f>] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:526
>>
>>
>> flush_to_ldisc accesses port->itty:
>>
>> static void flush_to_ldisc(struct work_struct *work)
>> {
>>    ...
>>     tty = port->itty;
>>     if (tty == NULL)
>>         return;
>>     disc = tty_ldisc_ref(tty);
>>
>> while release_tty concurrently sets itty to NULL:
>>
>> static void release_tty(struct tty_struct *tty, int idx)
>> {
>>     ...
>>     tty->port->itty = NULL;
>>     if (tty->link)
>>         tty->link->port->itty = NULL;
>>     cancel_work_sync(&tty->port->buf.work);
>>     tty_kref_put(tty->link);
>>     tty_kref_put(tty);
>> }
>>
>> It seems that read of port->itty requires to be at least READ_ONCE,
>> because otherwise flush_to_ldisc can check that itty is not NULL, then
>> re-read it again and crash with NULL deref.
>> I don't know what is ownership and locking story here. There can be
>> larger issue here: either a lock is missing, or itty can be deleted
>> under flush_to_ldisc feet.
>>
>> Please confirm that this is real but. If so please fix it.
>
> Patches are always gladly accepted.  Don't force us to try to determine
> if your tool is finding false-positives or not.  That is your
> responsibility, not ours :)


Well, I did my homework of eliminating all known false positives from
the tool and also by looking at the code to ensure that the report
makes sense. But I have very little experience with kernel code, so
cannot be 100% sure that this is a real race. So I am asking
maintainers to confirm.
Regarding a patch, should I just take tty_mutex in flush_to_ldisc? If
so, should it be locked before buf->lock or after?


Thank you

Re: Potential data race in flush_to_ldisc

[Peter Hurley]

On 08/28/2015 12:57 PM, Dmitry Vyukov wrote:
> Hello,
> 
> We are working on a dynamic data race detector for the Linux kernel,
> KernelThreadSanitizer (ktsan):
> https://github.com/google/ktsan/wiki
> 
> While booting kernel (upstream revision 21bdb584af8c) we got a report:
> 
> ThreadSanitizer: data-race in release_tty
> 
> Write of size 8 by thread T325 (K2579):
>  [<ffffffff81655c43>] release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688
>  [<ffffffff816563a8>] tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920
>  [<ffffffff8126154f>] __fput+0x15f/0x310 fs/file_table.c:207
>  [<ffffffff8126176d>] ____fput+0x1d/0x30 fs/file_table.c:243
>  [<ffffffff810b9485>] task_work_run+0x115/0x130 kernel/task_work.c:123
> (discriminator 1)
>  [<     inlined    >] do_notify_resume+0x73/0x80
> tracehook_notify_resume include/linux/tracehook.h:190
>  [<ffffffff81006da3>] do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757
>  [<ffffffff81ee25fc>] int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326
> 
> Previous read of size 8 by thread T19 (K16):
>  [<ffffffff816624d9>] flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472
>  [<ffffffff810b1fce>] process_one_work+0x47e/0x930 kernel/workqueue.c:2036
>  [<ffffffff810b2530>] worker_thread+0xb0/0x900 kernel/workqueue.c:2170
>  [<ffffffff810bbbd0>] kthread+0x150/0x170 kernel/kthread.c:207
>  [<ffffffff81ee281f>] ret_from_fork+0x3f/0x70 arch/x86/entry/entry_64.S:526
> 
> 
> flush_to_ldisc accesses port->itty:
> 
> static void flush_to_ldisc(struct work_struct *work)
> {
>    ...
>     tty = port->itty;
>     if (tty == NULL)
>         return;
>     disc = tty_ldisc_ref(tty);
> 
> while release_tty concurrently sets itty to NULL:
> 
> static void release_tty(struct tty_struct *tty, int idx)
> {
>     ...
>     tty->port->itty = NULL;
>     if (tty->link)
>         tty->link->port->itty = NULL;
>     cancel_work_sync(&tty->port->buf.work);
>     tty_kref_put(tty->link);
>     tty_kref_put(tty);
> }
> 
> It seems that read of port->itty requires to be at least READ_ONCE,

Agree; it should be READ_ONCE.

> because otherwise flush_to_ldisc can check that itty is not NULL, then
> re-read it again and crash with NULL deref.
> I don't know what is ownership and locking story here. There can be
> larger issue here: either a lock is missing, or itty can be deleted
> under flush_to_ldisc feet.
> 
> Please confirm that this is real but. If so please fix it.

Not a race.

The cancel_work_sync() waits for flush_to_ldisc() to complete, if already
running. For example,

CPU 0                                   | CPU 1
                                        |
release_tty()                           | flush_to_ldisc()
                                        |   tty = port->itty;
                                        |   tty == NULL? no
                                        |   ...
  port->itty = NULL                     |
  cancel_work_sync()                    |
    sleep here since flush_to_ldisc()   |
         running on CPU1                |
                                        | worker ends
    woken   <===========================| wake waiters

If flush_to_ldisc() was scheduled but not yet running, it will be cancelled
and not run.

Also, if flush_to_ldisc() is scheduled from some other cpu after cancel_work_sync(),
flush_to_ldisc() is guaranteed to 'see' the NULL port->itty.

Regards,
Peter Hurley

PS - And what Greg said; analyzing what is and is not a race will rapidly
improve your kernel familiarity.