Re: linux-next: Tree for March 12 (tty)

[Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>]

Hi,

On Thursday 12 March 2009, Stephen Rothwell wrote:
> Hi all,
> 
> Changes since 20090311:

It fails to boot here (both with QEMU and the real hardware):

...
Warning: unable to open an initial console.
Kernel panic - not syncing: Attempted to kill init!
...

I bisected the problem to:

>From 31f8326f981117fc548d292656fa8dbed28bc42b Mon Sep 17 00:00:00 2001
From: Alan Cox <alan@lxorguk.ukuu.org.uk>
Date: Thu, 12 Mar 2009 10:00:15 +1100
Subject: [PATCH] tty-fix-mismatch-in-lookup-han

The lookup methods return NULL or a pointer. The caller expects an ERR_PTR.
It seems to make sense to just use NULL and return -ENODEV here as it keeps
the code simple and clean and the way it always used to work (effectively)
before the restructure

Problem noticed by Jiri Slaby.

Signed-off-by: Alan Cox <alan@lxorguk.ukuu.org.uk>
---
 drivers/char/tty_io.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/char/tty_io.c b/drivers/char/tty_io.c
index bc84e12..0cc0c60 100644
--- a/drivers/char/tty_io.c
+++ b/drivers/char/tty_io.c
@@ -1817,9 +1817,9 @@ got_driver:
 		/* check whether we're reopening an existing tty */
 		tty = tty_driver_lookup_tty(driver, inode, index);
 
-		if (IS_ERR(tty)) {
+		if (tty == NULL) {
 			mutex_unlock(&tty_mutex);
-			return PTR_ERR(tty);
+			return -ENODEV;
 		}
 	}
 
-- 

It seems that fixing IS_ERR() check uncovered some underlying problem
since we are later checking for tty == NULL in __tty_open().

The following patch fixes the issue for me (Alan, feel free to merge
it back into your tty-fix-mismatch-in-lookup-han patch or replace by
a more complete __tty_open() reorganization):

From: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
Subject: [PATCH] tty: fix __tty_open()

On tty_driver_lookup_tty() failure __tty_open() should continue
instead of returning an error, fix it.

Also update tty_driver_lookup_tty() documentation while at it.

Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
---
 drivers/char/tty_io.c |   10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

Index: b/drivers/char/tty_io.c
===================================================================
--- a/drivers/char/tty_io.c
+++ b/drivers/char/tty_io.c
@@ -1205,7 +1205,7 @@ static void tty_line_name(struct tty_dri
  *	@driver: the driver for the tty
  *	@idx:	 the minor number
  *
- *	Return the tty, if found or ERR_PTR() otherwise.
+ *	Return the tty, if found or NULL otherwise.
  *
  *	Locking: tty_mutex must be held. If tty is found, the mutex must
  *	be held until the 'fast-open' is also done. Will change once we
@@ -1813,16 +1813,10 @@ retry_open:
 		return -ENODEV;
 	}
 got_driver:
-	if (!tty) {
+	if (!tty)
 		/* check whether we're reopening an existing tty */
 		tty = tty_driver_lookup_tty(driver, inode, index);
 
-		if (tty == NULL) {
-			mutex_unlock(&tty_mutex);
-			return -ENODEV;
-		}
-	}
-
 	if (tty) {
 		retval = tty_reopen(tty);
 		if (retval)