RE: 2.6.9: serial_core: uart_open

RE: 2.6.9: serial_core: uart_open

[karl malbrain]

Thanks for the suggestion of setting the modem termio to a copy of the xterm
console state.  Unfortunately, it doesn't help, the system still goes
out-to-lunch on the non-blocking open, and becomes nearly completely
unresponsive at the console and to telnet sessions.

Yesterday evening after testing via a telnet connection, the problem finally
cleared itself up and everything started to work as expected (even from the
console).  This morning, after a re-boot, the problem at the console
reoccurs.  Note that this test is being done through an xterm session.

I've also noticed that the boot sequence probes for modems on the serial
ports.  Is it possible that 8250.c is having a problem servicing an
interrupt from a character/state-change left over from this initialization?

Thanks, karl m

-----Original Message-----
From: Richard B. Johnson
Sent: Wednesday, July 13, 2005 6:04 AM
To: karl malbrain
Cc: Linux-Kernel@Vger. Kernel. Org
Subject: RE: 2.6.9: serial_core: uart_open



The attached code will set the UART to a sane state, then
clear the local flag, then open, waiting for modem control.
It clearly works, executing `ps` from another terminal will
clearly show that the task waiting for modem-control to open,
will be sleeping.

There is nothing wrong with kernel code that calls schedule().
That's how unix-machines work. When they are waiting for something
to happen, they execute schedule() which gives the CPU to other
runable tasks. The call to schedule() returns each time the
run queue is traversed. The driver code again checks for whatever
it was waiting for, then if it hasn't happened, the cycle repeats.
This is called "sleeping". That's what sleeping is. There are some
macros that do the same thing. They have names like "wait_for...".

Re: 2.6.9: serial_core: uart_open

[Russell King]

On Wed, Jul 13, 2005 at 10:53:19AM -0700, karl malbrain wrote:
> I've also noticed that the boot sequence probes for modems on the serial
> ports.  Is it possible that 8250.c is having a problem servicing an
> interrupt from a character/state-change left over from this initialization?

I did ask for a process listing a while back.  I don't want to
speculate on possible causes until we have some real information
from the system as to what's going on.

Please run up your test program and get the machine into the
problematic state.  Let it remain like that for about 2 minutes,
and then run via a telnet session or other window:

ps aux > /tmp/ps-forrmk.txt

and send me that file.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

RE: 2.6.9: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Thursday, July 14, 2005 1:27 AM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9: serial_core: uart_open
>
>
> On Wed, Jul 13, 2005 at 10:53:19AM -0700, karl malbrain wrote:
> > I've also noticed that the boot sequence probes for modems on the serial
> > ports.  Is it possible that 8250.c is having a problem servicing an
> > interrupt from a character/state-change left over from this
> initialization?
>
> I did ask for a process listing a while back.  I don't want to
> speculate on possible causes until we have some real information
> from the system as to what's going on.
>
> Please run up your test program and get the machine into the
> problematic state.  Let it remain like that for about 2 minutes,
> and then run via a telnet session or other window:
>
> ps aux > /tmp/ps-forrmk.txt
>
> and send me that file.

I'd love to do a ps listing for you, but, except for the mouse, the system
is completely unresponsive after issuing the blocking open("/dev/ttyS1",
O_RDRW).

Telnet is dead; the console will respond to the mouse, but the only thing I
can do is close the xterm window and thereby kill the process. I can launch
a new xterm window from the menu using the mouse, but the new window is dead
and has no cursor nor bash prompt.

The clock on the display is being updated.  After several hours the system
reboots on its own.

I recall from my DOS days that 8250/16550 programming requires that the
specific IIR source be responded to, or the chip will immediately
turn-around with another interrupt.  It doesn't look like 8250.c is
organized to respond directly to the modem-status-change value in IIR which
requires reading MSR to reset.

I wish I could be of more assistance.  karl m

Re: 2.6.9: serial_core: uart_open

[Russell King]

On Thu, Jul 14, 2005 at 10:16:23AM -0700, karl malbrain wrote:
> I'd love to do a ps listing for you, but, except for the mouse, the system
> is completely unresponsive after issuing the blocking open("/dev/ttyS1",
> O_RDRW).
> 
> Telnet is dead; the console will respond to the mouse, but the only thing I
> can do is close the xterm window and thereby kill the process. I can launch
> a new xterm window from the menu using the mouse, but the new window is dead
> and has no cursor nor bash prompt.
> 
> The clock on the display is being updated.  After several hours the system
> reboots on its own.
> 
> I recall from my DOS days that 8250/16550 programming requires that the
> specific IIR source be responded to, or the chip will immediately
> turn-around with another interrupt.  It doesn't look like 8250.c is
> organized to respond directly to the modem-status-change value in IIR which
> requires reading MSR to reset.

Well, at this point interrupts are enabled, and _are_ handled.  The
only thing we use the IIR for is to answer the question "did this
device say it had an interrupt?"

If it did, we unconditionally read the MSR without fail.

So, I've no idea what so ever about what's going on here.  I don't
understand why your system is behaving the way it is.  Therefore,
I don't think we can progress this any further, sorry.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

RE: 2.6.9: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King [mailto:rmk@arm.linux.org.uk]On Behalf Of Russell
> King
> Sent: Thursday, July 14, 2005 11:57 AM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9: serial_core: uart_open
>
>
> On Thu, Jul 14, 2005 at 10:16:23AM -0700, karl malbrain wrote:
> > I'd love to do a ps listing for you, but, except for the mouse,
> the system
> > is completely unresponsive after issuing the blocking open("/dev/ttyS1",
> > O_RDRW).
> >
> > Telnet is dead; the console will respond to the mouse, but the
> only thing I
> > can do is close the xterm window and thereby kill the process.
> I can launch
> > a new xterm window from the menu using the mouse, but the new
> window is dead
> > and has no cursor nor bash prompt.
> >
> > The clock on the display is being updated.  After several hours
> the system
> > reboots on its own.
> >
> > I recall from my DOS days that 8250/16550 programming requires that the
> > specific IIR source be responded to, or the chip will immediately
> > turn-around with another interrupt.  It doesn't look like 8250.c is
> > organized to respond directly to the modem-status-change value
> in IIR which
> > requires reading MSR to reset.
>
> Well, at this point interrupts are enabled, and _are_ handled.  The
> only thing we use the IIR for is to answer the question "did this
> device say it had an interrupt?"
>
> If it did, we unconditionally read the MSR without fail.
>
> So, I've no idea what so ever about what's going on here.  I don't
> understand why your system is behaving the way it is.  Therefore,
> I don't think we can progress this any further, sorry.

There is some code inserted at the top of the main receive_chars loop in
8250.c that examines  tty->flip.count and returns without reading UART_RX
under the condition that tty->flip.count is not reset after a call to
tty->flip.work.func.  This would leave the chip RX IIR unserviced and
subject another interrupt request.  Is it possible that this is the cause of
the problem?

karl m

Thanks, karl m

RE: 2.6.9: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Thursday, July 14, 2005 11:57 AM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9: serial_core: uart_open
>
>
> On Thu, Jul 14, 2005 at 10:16:23AM -0700, karl malbrain wrote:
> > I'd love to do a ps listing for you, but, except for the mouse,
> the system
> > is completely unresponsive after issuing the blocking open("/dev/ttyS1",
> > O_RDRW).
> >
> > Telnet is dead; the console will respond to the mouse, but the
> only thing I
> > can do is close the xterm window and thereby kill the process.
> I can launch
> > a new xterm window from the menu using the mouse, but the new
> window is dead
> > and has no cursor nor bash prompt.
> >
> > The clock on the display is being updated.  After several hours
> the system
> > reboots on its own.
> >
> > I recall from my DOS days that 8250/16550 programming requires that the
> > specific IIR source be responded to, or the chip will immediately
> > turn-around with another interrupt.  It doesn't look like 8250.c is
> > organized to respond directly to the modem-status-change value
> in IIR which
> > requires reading MSR to reset.
>
> Well, at this point interrupts are enabled, and _are_ handled.  The
> only thing we use the IIR for is to answer the question "did this
> device say it had an interrupt?"
>
> If it did, we unconditionally read the MSR without fail.
>
> So, I've no idea what so ever about what's going on here.  I don't
> understand why your system is behaving the way it is.  Therefore,
> I don't think we can progress this any further, sorry.

AT LAST I HAVE SOME DATA!!!

The problem is that ALL SYSTEM CALLS to open "/dev/tty" are blocking!! even
with O_NDELAY set and even from completely disjoint sessions.  I discovered
this via issuing "strace sh".  That's why the new xterm windows froze.

The original process doing the open("/dev/ttyS1", O_RDWR) is listed in the
ps aux listing as status S+.

Hope this helps.... karl m

RE: 2.6.9 chrdev_open: serial_core: uart_open

[karl malbrain]

chrdev_open issues a lock_kernel() before calling uart_open.

It would appear that servicing the blocking open request uart_open goes to
sleep with the kernel locked.  Would this shut down subsequent access to
opening "/dev/tty"???

karl m

Re: 2.6.9 chrdev_open: serial_core: uart_open

[Russell King]

On Thu, Jul 14, 2005 at 04:50:00PM -0700, karl malbrain wrote:
> chrdev_open issues a lock_kernel() before calling uart_open.
> 
> It would appear that servicing the blocking open request uart_open goes to
> sleep with the kernel locked.  Would this shut down subsequent access to
> opening "/dev/tty"???

No.  lock_kernel() is automatically released when a process sleeps.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: 2.6.9: serial_core: uart_open

[Russell King]

On Thu, Jul 14, 2005 at 03:35:07PM -0700, karl malbrain wrote:
> AT LAST I HAVE SOME DATA!!!
> 
> The problem is that ALL SYSTEM CALLS to open "/dev/tty" are blocking!! even
> with O_NDELAY set and even from completely disjoint sessions.  I discovered
> this via issuing "strace sh".  That's why the new xterm windows froze.
> 
> The original process doing the open("/dev/ttyS1", O_RDWR) is listed in the
> ps aux listing as status S+.

Ok, 'S' means it's sleeping.

Can you enable Magic SYSRQ, and ensure that you have a large kernel
log buffer (the LOG_BUF_SHIFT configuration symbol).  Ensure that
/proc/sys/kernel/sysrq is 1, and re-run your test such that you have
something else waiting (eg, the strace sh).  Then hit Alt-SysRQ-T.

You can then read the kernel messages with dmesg - you may need the
-s argument to capture the entire kernel buffer.

This will tell us where all processes are sleeping.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

RE: 2.6.9: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 12:29 AM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9: serial_core: uart_open
>
>
> On Thu, Jul 14, 2005 at 03:35:07PM -0700, karl malbrain wrote:
> > AT LAST I HAVE SOME DATA!!!
> >
> > The problem is that ALL SYSTEM CALLS to open "/dev/tty" are
> blocking!! even
> > with O_NDELAY set and even from completely disjoint sessions.
> I discovered
> > this via issuing "strace sh".  That's why the new xterm windows froze.
> >
> > The original process doing the open("/dev/ttyS1", O_RDWR) is
> listed in the
> > ps aux listing as status S+.
>
> Ok, 'S' means it's sleeping.
>
> Can you enable Magic SYSRQ, and ensure that you have a large kernel
> log buffer (the LOG_BUF_SHIFT configuration symbol).  Ensure that
> /proc/sys/kernel/sysrq is 1, and re-run your test such that you have
> something else waiting (eg, the strace sh).  Then hit Alt-SysRQ-T.
>
> You can then read the kernel messages with dmesg - you may need the
> -s argument to capture the entire kernel buffer.
>
> This will tell us where all processes are sleeping.


 sh            D 00000006  3036  5341   5252                     (NOTLB)
d0408eb0 00000086 c01c14d7 00000006 d0408e94 000f4fa5 c0d38f81 000039a0
       df461240 df4613cc c035ff00 00000246 d0408ecc df461240 c0300e33
00000001
       df461240 c011c856 c035ff20 c035ff20 d0408000 00000001 c035abe0
d0408000
Call Trace:
 [<c01c14d7>] inode_has_perm+0x4c/0x54
 [<c0300e33>] __down+0x103/0x1fe
 [<c011c856>] default_wake_function+0x0/0xc
 [<c0301180>] __down_failed+0x8/0xc
 [<c021a4d0>] .text.lock.tty_io+0x87/0x10f
 [<c016d78c>] chrdev_open+0x325/0x3b9
 [<c016256f>] dentry_open+0xbd/0x180
 [<c01624ac>] filp_open+0x36/0x3c
 [<c01da502>] direct_strncpy_from_user+0x46/0x5d
 [<c0162970>] sys_open+0x31/0x7d
 [<c03036f3>] syscall_call+0x7/0xb

I believe that this is the sh process that's opening "/dev/tty"
karl m

RE: 2.6.9: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 12:29 AM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9: serial_core: uart_open
>
>
> On Thu, Jul 14, 2005 at 03:35:07PM -0700, karl malbrain wrote:
> > AT LAST I HAVE SOME DATA!!!
> >
> > The problem is that ALL SYSTEM CALLS to open "/dev/tty" are
> blocking!! even
> > with O_NDELAY set and even from completely disjoint sessions.
> I discovered
> > this via issuing "strace sh".  That's why the new xterm windows froze.
> >
> > The original process doing the open("/dev/ttyS1", O_RDWR) is
> listed in the
> > ps aux listing as status S+.
>
> Ok, 'S' means it's sleeping.
>
> Can you enable Magic SYSRQ, and ensure that you have a large kernel
> log buffer (the LOG_BUF_SHIFT configuration symbol).  Ensure that
> /proc/sys/kernel/sysrq is 1, and re-run your test such that you have
> something else waiting (eg, the strace sh).  Then hit Alt-SysRQ-T.
>
> You can then read the kernel messages with dmesg - you may need the
> -s argument to capture the entire kernel buffer.
>
> This will tell us where all processes are sleeping.

Here is the dump of the original process that's waiting for the
open("/dev/ttyS1", O_RDWR) to return:

test5         S C023B673  3036  5399   5224                     (NOTLB)
d0fb6e88 00000086 d3462ea0 c023b673 04000000 001123f9 3949e4f7 00003b4e
       d204e170 d204e2fc dfee9658 dfd4ec60 c0422448 dfee9640 c02390c0
d1f22940
       00000000 d204e170 c011c856 00000000 00000000 c0236d4c 00000000
dfee9640
Call Trace:
 [<c023b673>] serial8250_interrupt+0x0/0x200
 [<c02390c0>] uart_block_til_ready+0x198/0x224
 [<c011c856>] default_wake_function+0x0/0xc
 [<c0236d4c>] uart_startup+0xb6/0x1d8
 [<c011c856>] default_wake_function+0x0/0xc
 [<c023980b>] uart_change_pm+0x1c/0x24
 [<c023938a>] uart_open+0xd1/0x105
 [<c0218226>] tty_open+0x18f/0x3b8
 [<c016d78c>] chrdev_open+0x325/0x3b9
 [<c016256f>] dentry_open+0xbd/0x180
 [<c01624ac>] filp_open+0x36/0x3c
 [<c0301e98>] __cond_resched+0x14/0x3b
 [<c01da4fa>] direct_strncpy_from_user+0x3e/0x5d
 [<c0162970>] sys_open+0x31/0x7d
 [<c03036f3>] syscall_call+0x7/0xb

Hope this helps, karl m

RE: 2.6.9 chrdev_open: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 12:23 AM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
>
>
> On Thu, Jul 14, 2005 at 04:50:00PM -0700, karl malbrain wrote:
> > chrdev_open issues a lock_kernel() before calling uart_open.
> >
> > It would appear that servicing the blocking open request
> uart_open goes to
> > sleep with the kernel locked.  Would this shut down subsequent access to
> > opening "/dev/tty"???
>
> No.  lock_kernel() is automatically released when a process sleeps.

Drilling down between the uart_open and chrdev_open into tty_open is a
semaphore tty_sem that is being held during the sleep cycle in uart_open.

This would appear to be the problem!!  Is this a new semaphore in 2.6? How
could this have ever worked with tty blocking mode?  It would appear that
tty_sem is going to have to be released before sleeping in uart_open.  What
a mess.

N.b. I don't pretend to understand how uart_change_pm, uart_startup, and
uart_block_til_ready could ALL be on the call stack.  Uart_open calls them
sequentially.  Perhaps you might explain how this works?  Thanks, karl m

Re: 2.6.9 chrdev_open: serial_core: uart_open

[Russell King]

On Fri, Jul 15, 2005 at 01:11:33PM -0700, karl malbrain wrote:
> > -----Original Message-----
> > From: Russell King
> > Sent: Friday, July 15, 2005 12:23 AM
> > To: karl malbrain
> > Cc: Linux-Kernel@Vger. Kernel. Org
> > Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
> >
> >
> > On Thu, Jul 14, 2005 at 04:50:00PM -0700, karl malbrain wrote:
> > > chrdev_open issues a lock_kernel() before calling uart_open.
> > >
> > > It would appear that servicing the blocking open request
> > uart_open goes to
> > > sleep with the kernel locked.  Would this shut down subsequent access to
> > > opening "/dev/tty"???
> >
> > No.  lock_kernel() is automatically released when a process sleeps.
> 
> Drilling down between the uart_open and chrdev_open into tty_open is a
> semaphore tty_sem that is being held during the sleep cycle in uart_open.

chrdev_open() calls tty_open(), which then calls init_dev().  init_dev()
takes tty_sem, does its stuff, and then releases tty_sem.  A little
later on, tty_open() calls the uart driver's uart_open() function.

So it does this with tty_sem unlocked.

> N.b. I don't pretend to understand how uart_change_pm, uart_startup, and
> uart_block_til_ready could ALL be on the call stack.  Uart_open calls them
> sequentially.  Perhaps you might explain how this works?  Thanks, karl m

The stack traces on x86 are very hap-hazard - the code just scans the
stack for anything which looks like it may be in kernel text and dumps
the result.  This means that stale words on the stack which may have
been return addresses at one time may still look like return addresses.

Don't believe everything you see in an x86 stacktrace!

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: 2.6.9: serial_core: uart_open

[Russell King]

On Fri, Jul 15, 2005 at 09:02:48AM -0700, karl malbrain wrote:
> > -----Original Message-----
> > From: Russell King
> > Sent: Friday, July 15, 2005 12:29 AM
> > To: karl malbrain
> > Cc: Linux-Kernel@Vger. Kernel. Org
> > Subject: Re: 2.6.9: serial_core: uart_open
> >
> >
> > On Thu, Jul 14, 2005 at 03:35:07PM -0700, karl malbrain wrote:
> > > AT LAST I HAVE SOME DATA!!!
> > >
> > > The problem is that ALL SYSTEM CALLS to open "/dev/tty" are
> > blocking!! even
> > > with O_NDELAY set and even from completely disjoint sessions.
> > I discovered
> > > this via issuing "strace sh".  That's why the new xterm windows froze.
> > >
> > > The original process doing the open("/dev/ttyS1", O_RDWR) is
> > listed in the
> > > ps aux listing as status S+.
> >
> > Ok, 'S' means it's sleeping.
> >
> > Can you enable Magic SYSRQ, and ensure that you have a large kernel
> > log buffer (the LOG_BUF_SHIFT configuration symbol).  Ensure that
> > /proc/sys/kernel/sysrq is 1, and re-run your test such that you have
> > something else waiting (eg, the strace sh).  Then hit Alt-SysRQ-T.
> >
> > You can then read the kernel messages with dmesg - you may need the
> > -s argument to capture the entire kernel buffer.
> >
> > This will tell us where all processes are sleeping.
> 
> 
>  sh            D 00000006  3036  5341   5252                     (NOTLB)
> d0408eb0 00000086 c01c14d7 00000006 d0408e94 000f4fa5 c0d38f81 000039a0
>        df461240 df4613cc c035ff00 00000246 d0408ecc df461240 c0300e33
> 00000001
>        df461240 c011c856 c035ff20 c035ff20 d0408000 00000001 c035abe0
> d0408000
> Call Trace:
>  [<c01c14d7>] inode_has_perm+0x4c/0x54
>  [<c0300e33>] __down+0x103/0x1fe
>  [<c011c856>] default_wake_function+0x0/0xc
>  [<c0301180>] __down_failed+0x8/0xc
>  [<c021a4d0>] .text.lock.tty_io+0x87/0x10f
>  [<c016d78c>] chrdev_open+0x325/0x3b9

This seems to imply that there's a lock being taken in tty_open().  The
2.6.9 source contains no such thing.  Are you sure you're using an
unpatched 2.6.9 kernel?

>  [<c016256f>] dentry_open+0xbd/0x180
>  [<c01624ac>] filp_open+0x36/0x3c
>  [<c01da502>] direct_strncpy_from_user+0x46/0x5d
>  [<c0162970>] sys_open+0x31/0x7d
>  [<c03036f3>] syscall_call+0x7/0xb

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: 2.6.9: serial_core: uart_open

[karl malbrain]

----- Original Message ----- 
From: "Russell King" <rmk+lkml@arm.linux.org.uk>
To: "karl malbrain" <karl@petzent.com>
Cc: "Linux-Kernel@Vger. Kernel. Org" <linux-kernel@vger.kernel.org>
Sent: Friday, July 15, 2005 1:32 PM
Subject: Re: 2.6.9: serial_core: uart_open


> On Fri, Jul 15, 2005 at 09:02:48AM -0700, karl malbrain wrote:
> > > -----Original Message-----
> > > From: Russell King
> > > Sent: Friday, July 15, 2005 12:29 AM
> > > To: karl malbrain
> > > Cc: Linux-Kernel@Vger. Kernel. Org
> > > Subject: Re: 2.6.9: serial_core: uart_open
> > >
> > >
> > > On Thu, Jul 14, 2005 at 03:35:07PM -0700, karl malbrain wrote:
> > > > AT LAST I HAVE SOME DATA!!!
> > > >
> > > > The problem is that ALL SYSTEM CALLS to open "/dev/tty" are
> > > blocking!! even
> > > > with O_NDELAY set and even from completely disjoint sessions.
> > > I discovered
> > > > this via issuing "strace sh".  That's why the new xterm windows
froze.
> > > >
> > > > The original process doing the open("/dev/ttyS1", O_RDWR) is
> > > listed in the
> > > > ps aux listing as status S+.
> > >
> > > Ok, 'S' means it's sleeping.
> > >
> > > Can you enable Magic SYSRQ, and ensure that you have a large kernel
> > > log buffer (the LOG_BUF_SHIFT configuration symbol).  Ensure that
> > > /proc/sys/kernel/sysrq is 1, and re-run your test such that you have
> > > something else waiting (eg, the strace sh).  Then hit Alt-SysRQ-T.
> > >
> > > You can then read the kernel messages with dmesg - you may need the
> > > -s argument to capture the entire kernel buffer.
> > >
> > > This will tell us where all processes are sleeping.
> >
> >
> >  sh            D 00000006  3036  5341   5252                     (NOTLB)
> > d0408eb0 00000086 c01c14d7 00000006 d0408e94 000f4fa5 c0d38f81 000039a0
> >        df461240 df4613cc c035ff00 00000246 d0408ecc df461240 c0300e33
> > 00000001
> >        df461240 c011c856 c035ff20 c035ff20 d0408000 00000001 c035abe0
> > d0408000
> > Call Trace:
> >  [<c01c14d7>] inode_has_perm+0x4c/0x54
> >  [<c0300e33>] __down+0x103/0x1fe
> >  [<c011c856>] default_wake_function+0x0/0xc
> >  [<c0301180>] __down_failed+0x8/0xc
> >  [<c021a4d0>] .text.lock.tty_io+0x87/0x10f
> >  [<c016d78c>] chrdev_open+0x325/0x3b9
>
> This seems to imply that there's a lock being taken in tty_open().  The
> 2.6.9 source contains no such thing.  Are you sure you're using an
> unpatched 2.6.9 kernel?
>
> >  [<c016256f>] dentry_open+0xbd/0x180
> >  [<c01624ac>] filp_open+0x36/0x3c
> >  [<c01da502>] direct_strncpy_from_user+0x46/0x5d
> >  [<c0162970>] sys_open+0x31/0x7d
> >  [<c03036f3>] syscall_call+0x7/0xb
>

The system is red-hat 4.6.9-11EL.  There is a patch to tty_io but it doesn't
mention locking anything.  karl m

RE: 2.6.9 chrdev_open: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 1:31 PM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
>
>
> On Fri, Jul 15, 2005 at 01:11:33PM -0700, karl malbrain wrote:
> > > -----Original Message-----
> > > From: Russell King
> > > Sent: Friday, July 15, 2005 12:23 AM
> > > To: karl malbrain
> > > Cc: Linux-Kernel@Vger. Kernel. Org
> > > Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
> > >
> > >
> > > On Thu, Jul 14, 2005 at 04:50:00PM -0700, karl malbrain wrote:
> > > > chrdev_open issues a lock_kernel() before calling uart_open.
> > > >
> > > > It would appear that servicing the blocking open request
> > > uart_open goes to
> > > > sleep with the kernel locked.  Would this shut down
> subsequent access to
> > > > opening "/dev/tty"???
> > >
> > > No.  lock_kernel() is automatically released when a process sleeps.
> >
> > Drilling down between the uart_open and chrdev_open into tty_open is a
> > semaphore tty_sem that is being held during the sleep cycle in
> uart_open.
>
> chrdev_open() calls tty_open(), which then calls init_dev().  init_dev()
> takes tty_sem, does its stuff, and then releases tty_sem.  A little
> later on, tty_open() calls the uart driver's uart_open() function.
>
> So it does this with tty_sem unlocked.

On my 2.6.9-11EL source it clearly shows the up(&tty_sem) after the call to
uart_open. Init_dev never touches tty_sem.

karl m

Re: 2.6.9 chrdev_open: serial_core: uart_open

[Russell King]

On Fri, Jul 15, 2005 at 01:52:15PM -0700, karl malbrain wrote:
> On my 2.6.9-11EL source it clearly shows the up(&tty_sem) after the call to
> uart_open. Init_dev never touches tty_sem.

In which case, I have to say...

Congratulations!  You've found a bug with Red Hat's Enterprise Linux
kernel!  Go straight to Red Hat's bugzilla!  Do not collect 200$.  Do
not pass go.

Seriously though, this bug is not present in mainline kernels, so I
can't resolve this issue for you.  Mainline kernels appear to work
properly.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

RE: 2.6.9 chrdev_open: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 1:59 PM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
>
>
> On Fri, Jul 15, 2005 at 01:52:15PM -0700, karl malbrain wrote:
> > On my 2.6.9-11EL source it clearly shows the up(&tty_sem) after
> the call to
> > uart_open. Init_dev never touches tty_sem.
>
> In which case, I have to say...
>
> Congratulations!  You've found a bug with Red Hat's Enterprise Linux
> kernel!  Go straight to Red Hat's bugzilla!  Do not collect 200$.  Do
> not pass go.
>
> Seriously though, this bug is not present in mainline kernels, so I
> can't resolve this issue for you.  Mainline kernels appear to work
> properly.

Could tty_io.c be all that changed by a small set of red-hat patches to
2.6.9?  Why would they need to go in there to make so many changes in the
first place?  Which 2.6 release changed tty_io.c's use of tty_sem so
heavily?

This conflict between main-line/redhat looks much worse than the unix sysV
r4.0 divergence after 3.2. Ouch.

Thanks for your help, though.  karl m

Re: 2.6.9 chrdev_open: serial_core: uart_open

[Russell King]

On Fri, Jul 15, 2005 at 02:17:01PM -0700, karl malbrain wrote:
> > -----Original Message-----
> > From: Russell King
> > Sent: Friday, July 15, 2005 1:59 PM
> > To: karl malbrain
> > Cc: Linux-Kernel@Vger. Kernel. Org
> > Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
> >
> >
> > On Fri, Jul 15, 2005 at 01:52:15PM -0700, karl malbrain wrote:
> > > On my 2.6.9-11EL source it clearly shows the up(&tty_sem) after
> > the call to
> > > uart_open. Init_dev never touches tty_sem.
> >
> > In which case, I have to say...
> >
> > Congratulations!  You've found a bug with Red Hat's Enterprise Linux
> > kernel!  Go straight to Red Hat's bugzilla!  Do not collect 200$.  Do
> > not pass go.
> >
> > Seriously though, this bug is not present in mainline kernels, so I
> > can't resolve this issue for you.  Mainline kernels appear to work
> > properly.
> 
> Could tty_io.c be all that changed by a small set of red-hat patches to
> 2.6.9?  Why would they need to go in there to make so many changes in the
> first place?  Which 2.6 release changed tty_io.c's use of tty_sem so
> heavily?

These are questions to ask of Red Hat, and can only be answered by
their representatives.

Thanks anyway, and I'm sorry that this hasn't been resolved given
the amount of time put into it by both of us.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

RE: 2.6.9 chrdev_open: serial_core: uart_open

[karl malbrain]

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 2:54 PM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
>
>
> On Fri, Jul 15, 2005 at 02:17:01PM -0700, karl malbrain wrote:
> > > -----Original Message-----
> > > From: Russell King
> > > Sent: Friday, July 15, 2005 1:59 PM
> > > To: karl malbrain
> > > Cc: Linux-Kernel@Vger. Kernel. Org
> > > Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
> > >
> > >
> > > On Fri, Jul 15, 2005 at 01:52:15PM -0700, karl malbrain wrote:
> > > > On my 2.6.9-11EL source it clearly shows the up(&tty_sem) after
> > > the call to
> > > > uart_open. Init_dev never touches tty_sem.
> > >
> > > In which case, I have to say...
> > >
> > > Congratulations!  You've found a bug with Red Hat's Enterprise Linux
> > > kernel!  Go straight to Red Hat's bugzilla!  Do not collect 200$.  Do
> > > not pass go.
> > >
> > > Seriously though, this bug is not present in mainline kernels, so I
> > > can't resolve this issue for you.  Mainline kernels appear to work
> > > properly.
> >
> > Could tty_io.c be all that changed by a small set of red-hat patches to
> > 2.6.9?  Why would they need to go in there to make so many
> changes in the
> > first place?  Which 2.6 release changed tty_io.c's use of tty_sem so
> > heavily?
>
> These are questions to ask of Red Hat, and can only be answered by
> their representatives.

I've since answered part of my question.  Red Hat pulled some code-changes
from 2.6.10 tty_io.c with the somewhat cryptic comment "fix the trivial
exploits caused by Rolands controlling tty changes (part 1)" and moved the
tty_sem ops.

Do you know if this would be Roland at Red Hat, or a Roland at lkml?

Thanks, karl m

RE: 2.6.9 chrdev_open: serial_core: uart_open

[Alan Cox]

On Gwe, 2005-07-15 at 13:11 -0700, karl malbrain wrote:
> N.b. I don't pretend to understand how uart_change_pm, uart_startup, and
> uart_block_til_ready could ALL be on the call stack.  Uart_open calls them
> sequentially.  Perhaps you might explain how this works?  Thanks, karl m

gcc does smart things including deferring stack cleanup so that it can
turn

	push, push, call, adjust stack, push, push call .. etc

into a sequence with less stack pointer adjustment for performance
reasons. That sometimes fools the traceback code. A good rule of thumb
is to trace the sequence of calls and assume that the last sane sequence
is the one that occurred before the failure.

Alan

RE: 2.6.9 chrdev_open: serial_core: uart_open

[Alan Cox]

On Gwe, 2005-07-15 at 15:02 -0700, karl malbrain wrote:
> I've since answered part of my question.  Red Hat pulled some code-changes
> from 2.6.10 tty_io.c with the somewhat cryptic comment "fix the trivial
> exploits caused by Rolands controlling tty changes (part 1)" and moved the
> tty_sem ops.
> 
> Do you know if this would be Roland at Red Hat, or a Roland at lkml?

Roland at Red Hat. See the kernel list but 2.6.9 had an interesting hole
where you could crash the system by playing games with setsid and
threaded applications.

RHEL bugs really are best in the RHEL bugzilla, or through your support
contact as a customer. That ensures that the bug is processed promptly
and chased up for you while bugzilla is just for bug collection.

Alan

RE: 2.6.9 chrdev_open: serial_core: uart_open

[Matthias Urlichs]

Hi, Alan Cox wrote:

> A good rule of thumb
> is to trace the sequence of calls and assume that the last sane sequence
> is the one that occurred before the failure.

Note also that gcc does sibling optimization, i.e. it will happily
reduce the code at the end of
	int bar(a,b) { [...] return baz(x,y); }
into something like
	overwrite 'a' with 'x', and 'b' with 'y'
	pop local stack frame, if present
	jump to baz

which saves some stack space and is faster, but makes you wonder
how in hell the
	baz
	foo
stack dump you're seeing in your crash dump came about.

(2.6.13 will turn that off when debugging.)

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
There is a vast difference between putting your nose in other people's
business and putting your heart in other people's problems.

RE: 2.6.9 chrdev_open: serial_core: uart_open

[karl malbrain]

Successful resolution!! The red-hat engineers monitoring their bugzilla list
posted a fix for tty_io.c Friday that works.  Thanks again for your help.
karl m

> -----Original Message-----
> From: Russell King
> Sent: Friday, July 15, 2005 2:54 PM
> To: karl malbrain
> Cc: Linux-Kernel@Vger. Kernel. Org
> Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
>
>
> On Fri, Jul 15, 2005 at 02:17:01PM -0700, karl malbrain wrote:
> > > -----Original Message-----
> > > From: Russell King
> > > Sent: Friday, July 15, 2005 1:59 PM
> > > To: karl malbrain
> > > Cc: Linux-Kernel@Vger. Kernel. Org
> > > Subject: Re: 2.6.9 chrdev_open: serial_core: uart_open
> > >
> > >
> > > On Fri, Jul 15, 2005 at 01:52:15PM -0700, karl malbrain wrote:
> > > > On my 2.6.9-11EL source it clearly shows the up(&tty_sem) after
> > > the call to
> > > > uart_open. Init_dev never touches tty_sem.
> > >
> > > In which case, I have to say...
> > >
> > > Congratulations!  You've found a bug with Red Hat's Enterprise Linux
> > > kernel!  Go straight to Red Hat's bugzilla!  Do not collect 200$.  Do
> > > not pass go.
> > >
> > > Seriously though, this bug is not present in mainline kernels, so I
> > > can't resolve this issue for you.  Mainline kernels appear to work
> > > properly.
> >
> > Could tty_io.c be all that changed by a small set of red-hat patches to
> > 2.6.9?  Why would they need to go in there to make so many
> changes in the
> > first place?  Which 2.6 release changed tty_io.c's use of tty_sem so
> > heavily?
>
> These are questions to ask of Red Hat, and can only be answered by
> their representatives.
>
> Thanks anyway, and I'm sorry that this hasn't been resolved given
> the amount of time put into it by both of us.
>
> --
> Russell King
>  Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
>  maintainer of:  2.6 Serial core