Success: tty_io flush_to_ldisc() error message triggered

Success: tty_io flush_to_ldisc() error message triggered

[Chuck Ebbert]

Using the patch below that you sent me, this message printed today on
my SMP system when terminating pppd:

        flush_to_ldisc - TTY_FLUSHING set, low_latency=0

And this bug is also in 2.6.17.4 - I forgot to reboot into 2.6.16.x
and the system locked up the same way 2.6.16 used to.  That was made
even more fun by the 'SysRq broken on SMP' bug fixed by a pending
2.6.17.7 patch...


--- 2.6.16.20-d4.orig/include/linux/tty.h
+++ 2.6.16.20-d4/include/linux/tty.h
@@ -266,6 +266,7 @@ struct tty_struct {
 #define TTY_PTY_LOCK 		16	/* pty private */
 #define TTY_NO_WRITE_SPLIT 	17	/* Preserve write boundaries to driver */
 #define TTY_HUPPED 		18	/* Post driver->hangup() */
+#define TTY_FLUSHING 		19	/* Flushing tty buffers to line discipline */
 
 #define TTY_WRITE_FLUSH(tty) tty_write_flush((tty))
 
--- 2.6.16.20-d4.orig/drivers/char/tty_io.c
+++ 2.6.16.20-d4/drivers/char/tty_io.c
@@ -2781,9 +2781,12 @@ static void flush_to_ldisc(void *private
 		return;
 
 	if (test_bit(TTY_DONT_FLIP, &tty->flags)) {
-		/*
-		 * Do it after the next timer tick:
-		 */
+		printk(KERN_ERR"flush_to_ldisc - TTY_DONT_FLIP set\n");
+		schedule_delayed_work(&tty->buf.work, 1);
+		goto out;
+	}
+	if (test_and_set_bit(TTY_FLUSHING, &tty->flags)) {
+		printk(KERN_ERR"flush_to_ldisc - TTY_FLUSHING set, low_latency=%d\n", tty->low_latency);
 		schedule_delayed_work(&tty->buf.work, 1);
 		goto out;
 	}
@@ -2805,6 +2808,7 @@ static void flush_to_ldisc(void *private
 		tty_buffer_free(tty, tbuf);
 	}
 	spin_unlock_irqrestore(&tty->buf.lock, flags);
+	clear_bit(TTY_FLUSHING, &tty->flags);
 out:
 	tty_ldisc_deref(disc);
 }
-- 
Chuck

Re: Success: tty_io flush_to_ldisc() error message triggered

[Paul Fulghum]

Chuck Ebbert wrote:
> Using the patch below that you sent me, this message printed today on
> my SMP system when terminating pppd:
> 
>         flush_to_ldisc - TTY_FLUSHING set, low_latency=0
> 
> And this bug is also in 2.6.17.4 - I forgot to reboot into 2.6.16.x
> and the system locked up the same way 2.6.16 used to.  That was made
> even more fun by the 'SysRq broken on SMP' bug fixed by a pending
> 2.6.17.7 patch...

That confirms my thoughts on what went wrong:
multiple copies of the queued work (flush_to_ldisc)
running in parallel and corrupting the free buffer list.

A cleaner fix for this is already
in the 2.6.18 rc series.

Thanks,
Paul

--
Paul Fulghum
Microgate Systems, Ltd.

Re: Success: tty_io flush_to_ldisc() error message triggered

[Paul Fulghum]

Chuck Ebbert wrote:
> Using the patch below that you sent me, this message printed today on
> my SMP system when terminating pppd:
> 
>         flush_to_ldisc - TTY_FLUSHING set, low_latency=0
> 
> And this bug is also in 2.6.17.4 - I forgot to reboot into 2.6.16.x
> and the system locked up the same way 2.6.16 used to.  That was made
> even more fun by the 'SysRq broken on SMP' bug fixed by a pending
> 2.6.17.7 patch...

That confirms my thoughts on what went wrong:
multiple copies of the queued work (flush_to_ldisc)
running in parallel and corrupting the free buffer list.

A cleaner fix for this is already
in the 2.6.18 rc series.

Thanks,
Paul

--
Paul Fulghum
Microgate Systems, Ltd.

Re: Success: tty_io flush_to_ldisc() error message triggered

[Chuck Ebbert]

In-Reply-To: <44C2307C.9060607@microgate.com>

On Sat, 22 Jul 2006 09:04:44 -0500, Paul Fulghum wrote:

> That confirms my thoughts on what went wrong:
> multiple copies of the queued work (flush_to_ldisc)
> running in parallel and corrupting the free buffer list.
> 
> A cleaner fix for this is already
> in the 2.6.18 rc series.

The cleaner fix looks more intrusive, though.

Is this simpler change (what I'm running but without the warning
messages) the preferred fix for -stable?


From: Paul Fulghum <paulkf@microgate.com>

Serialize flush_to_ldisc() per-device. Fixes free list corruption
that causes lockup on SMP systems.

Signed-off-by: Paul Fulghum <paulkf@microgate.com>
Acked-by: Chuck Ebbert <76306.1226@compuserve.com>

--- 2.6.16.20-d4.orig/include/linux/tty.h
+++ 2.6.16.20-d4/include/linux/tty.h
@@ -266,6 +266,7 @@ struct tty_struct {
 #define TTY_PTY_LOCK 		16	/* pty private */
 #define TTY_NO_WRITE_SPLIT 	17	/* Preserve write boundaries to driver */
 #define TTY_HUPPED 		18	/* Post driver->hangup() */
+#define TTY_FLUSHING 		19	/* Flushing tty buffers to line discipline */
 
 #define TTY_WRITE_FLUSH(tty) tty_write_flush((tty))
 
--- 2.6.16.20-d4.orig/drivers/char/tty_io.c
+++ 2.6.16.20-d4/drivers/char/tty_io.c
@@ -2780,10 +2780,8 @@ static void flush_to_ldisc(void *private
 	if (disc == NULL)	/*  !TTY_LDISC */
 		return;
 
-	if (test_bit(TTY_DONT_FLIP, &tty->flags)) {
-		/*
-		 * Do it after the next timer tick:
-		 */
+	if (test_bit(TTY_DONT_FLIP, &tty->flags) ||
+	    test_and_set_bit(TTY_FLUSHING, &tty->flags)) {
 		schedule_delayed_work(&tty->buf.work, 1);
 		goto out;
 	}
@@ -2805,6 +2803,7 @@ static void flush_to_ldisc(void *private
 		tty_buffer_free(tty, tbuf);
 	}
 	spin_unlock_irqrestore(&tty->buf.lock, flags);
+	clear_bit(TTY_FLUSHING, &tty->flags);
 out:
 	tty_ldisc_deref(disc);
 }
-- 
Chuck

Re: Success: tty_io flush_to_ldisc() error message triggered

[Paul Fulghum]

Chuck Ebbert wrote:
> The cleaner fix looks more intrusive, though.
> 
> Is this simpler change (what I'm running but without the warning
> messages) the preferred fix for -stable?

It fixes the problem.

The ugly thing about that patch is adding
another TTY_XXX macro that is not really necessary.
I created it as a quick and dirty way of testing
my hypothesis about the problem.

I don't see a problem using it as a simple
(albeit naive) approach to fix stable.

--
Paul Fulghum
Microgate Systems, Ltd