Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Ben Thomas]

This may not be new with the latest source. I've been seeing the same
thing for some time, on certain hardware. It's right after boot,
generally while printing the E820 map.  I've noticed some patterns
to it, but haven't spent the time to track it down. There's always
been something else more pressing to do.  One slight difference is
between your example and what I've been doing is that I only use
com1 as the console for Xen.  I've never seen it hang with only
vga as the console, but only with com1.

And, while it may seem unlikely that a simple serial UART is
having problems, it's quite possible.

-b


Date: Mon, 20 Mar 2006 11:04:05 +0000
From: Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
Subject: Re: [Xen-devel] Re: [Xen-users] xen hg tree from yesterday
	hangs,	when kernel console is directed to ttyS0


On 19 Mar 2006, at 15:10, Jussi Hamalainen wrote:


 >>>> When I reboot my system with xen with the following options to xen
 >>>> and vmlinuz,
 >>>>
 >>>> xen.gz com1=115200,8n1 console=com1,vga
 >>>> vmlinuz console=ttyS0,115200 console=tty0
 >>>>
 >>>> the system hangs up while booting the dom0 kernel.
 >
 >>
 >> I think Xen has a major buffering/blocking IO issue with serial
 >> console. It works for a while but when loads of lines get
 >> flooded onto the console, the system freezes up. Adding a
 >> "sync_console" to your Xen parameters might solve the problem, or at
 >> least it worked for me.


Maybe it's hardware specific. I can echo a long string to the serial
console in a tight loop from dom0 shell and I see no problems. My setup
is similar to the OP's (115200 baud).

   -- Keir


-- 
------------------------------------------------------------------------
Ben Thomas                                         Virtual Iron Software
bthomas@virtualiron.com                            Tower 1, Floor 2
978-849-1214                                       900 Chelmsford Street
                                                    Lowell, MA 01851

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Stephen C. Tweedie]

Hi,

On Mon, 2006-03-20 at 08:24 -0500, Ben Thomas wrote:
> This may not be new with the latest source. I've been seeing the same
> thing for some time, on certain hardware. It's right after boot,
> generally while printing the E820 map.

I see this a lot too.  In every case I've come across myself, using the
HV "sync_console" option seems to cure it, but I have no idea why.

--Stephen

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Ben Thomas]

Oh, I wish, I wish...

When I see this, I generally never get very far at all.  Often, I'm
at the "no output at all past grub" stage.  Even with vga.  Luckily,
it's confined to specific machines.

At some point, I'll slow down to warp 8 and be able to look more
at it.  I have some ideas on how to find it.

-b


Stephen C. Tweedie wrote:
> Hi,
> 
> On Mon, 2006-03-20 at 08:24 -0500, Ben Thomas wrote:
> 
>>This may not be new with the latest source. I've been seeing the same
>>thing for some time, on certain hardware. It's right after boot,
>>generally while printing the E820 map.
> 
> 
> I see this a lot too.  In every case I've come across myself, using the
> HV "sync_console" option seems to cure it, but I have no idea why.
> 
> --Stephen
> 
> 


-- 
------------------------------------------------------------------------
Ben Thomas                                         Virtual Iron Software
bthomas@virtualiron.com                            Tower 1, Floor 2
978-849-1214                                       900 Chelmsford Street
                                                    Lowell, MA 01851

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Stephen C. Tweedie]

Hi,

On Wed, 2006-03-22 at 15:22 -0500, Ben Thomas wrote:

> When I see this, I generally never get very far at all.  Often, I'm
> at the "no output at all past grub" stage.  Even with vga.  Luckily,
> it's confined to specific machines.

Even with *only* VGA?  I usually see exactly this with serial console
enabled: if I attempt it without sync_console, dom0 always hangs solid
very early in the boot process (though rarely at exactly the same
location.)  But I've never seen that sort of sporadic hang during boot
using just VGA console exclusively; the bugs I've seen have always been
either oopses or very predictable hangs at a specific location.

--Stephen

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Ben Thomas]

No, sorry, implicit context here.  It really feels like a
serial line issue.  When I talk about using vga, it's
concurrent usage with com1.  I've never had a problem
with using only vga.  Although, that's a very small
percentage of my usage.

-b

Stephen C. Tweedie wrote:
> Hi,
> 
> On Wed, 2006-03-22 at 15:22 -0500, Ben Thomas wrote:
> 
> 
>>When I see this, I generally never get very far at all.  Often, I'm
>>at the "no output at all past grub" stage.  Even with vga.  Luckily,
>>it's confined to specific machines.
> 
> 
> Even with *only* VGA?  I usually see exactly this with serial console
> enabled: if I attempt it without sync_console, dom0 always hangs solid
> very early in the boot process (though rarely at exactly the same
> location.)  But I've never seen that sort of sporadic hang during boot
> using just VGA console exclusively; the bugs I've seen have always been
> either oopses or very predictable hangs at a specific location.
> 
> --Stephen
> 
> 


-- 
------------------------------------------------------------------------
Ben Thomas                                         Virtual Iron Software
bthomas@virtualiron.com                            Tower 1, Floor 2
978-849-1214                                       900 Chelmsford Street
                                                    Lowell, MA 01851

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Stephen C. Tweedie]

Hi,

On Wed, 2006-03-22 at 16:37 -0500, Ben Thomas wrote:
> No, sorry, implicit context here.  It really feels like a
> serial line issue.  When I talk about using vga, it's
> concurrent usage with com1.  I've never had a problem
> with using only vga.

OK, so it sounds very similar to what I'm seeing; does sync_console
really not help?

--Stephen

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Ben Thomas]

Not one bit (pun intended).

-b

Stephen C. Tweedie wrote:
> Hi,
> 
> On Wed, 2006-03-22 at 16:37 -0500, Ben Thomas wrote:
> 
>>No, sorry, implicit context here.  It really feels like a
>>serial line issue.  When I talk about using vga, it's
>>concurrent usage with com1.  I've never had a problem
>>with using only vga.
> 
> 
> OK, so it sounds very similar to what I'm seeing; does sync_console
> really not help?
> 
> --Stephen
> 
> 


-- 
------------------------------------------------------------------------
Ben Thomas                                         Virtual Iron Software
bthomas@virtualiron.com                            Tower 1, Floor 2
978-849-1214                                       900 Chelmsford Street
                                                    Lowell, MA 01851

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Keir Fraser]

In unstable changeset 9419 I've reduced contention for the serial-line
spinlock, particularly in the serial ISR. It certainly has cleaned up
some weird behaviour that I was able to reproduce when I hit the
serial line hard from within Xen itself. It's not clear it'll fix your
problems, but worth testing.

 -- Keir

> Not one bit (pun intended).
> 
> -b
> 
> Stephen C. Tweedie wrote:
> > Hi,
> > 
> > On Wed, 2006-03-22 at 16:37 -0500, Ben Thomas wrote:
> > 
> >>No, sorry, implicit context here.  It really feels like a
> >>serial line issue.  When I talk about using vga, it's
> >>concurrent usage with com1.  I've never had a problem
> >>with using only vga.
> > 
> > 
> > OK, so it sounds very similar to what I'm seeing; does sync_console
> > really not help?
> > 
> > --Stephen
> > 
> > 
> 
> 
> -- 
> ------------------------------------------------------------------------
> Ben Thomas                                         Virtual Iron Software
> bthomas@virtualiron.com                            Tower 1, Floor 2
> 978-849-1214                                       900 Chelmsford Street
>                                                     Lowell, MA 01851
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Ben Thomas]

Hi Keir,

Thanks for the heads-up.  I'll certainly give it a try.

As this hang appears during boot, it's not clear how much reducing
spinlock contention will help.  At times, it's not even clear that
I make it into __start_xen.  Certainly, when I get a hang, I
never make it out of __start_xen/init_e820/print_e820_memory_map.
As it often stops in the middle of print_e820_memory_map
output, that's what lead to the suspicion about serial lines.
Clearly, though, it's merely a suspicion/guess.

Thanks,
-b

Keir Fraser wrote:
> In unstable changeset 9419 I've reduced contention for the serial-line
> spinlock, particularly in the serial ISR. It certainly has cleaned up
> some weird behaviour that I was able to reproduce when I hit the
> serial line hard from within Xen itself. It's not clear it'll fix your
> problems, but worth testing.
> 
>  -- Keir


-- 
------------------------------------------------------------------------
Ben Thomas                                         Virtual Iron Software
bthomas@virtualiron.com                            Tower 1, Floor 2
978-849-1214                                       900 Chelmsford Street
                                                    Lowell, MA 01851

Re: Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0

[Keir Fraser]

> Hi Keir,
> 
> Thanks for the heads-up.  I'll certainly give it a try.
> 
> As this hang appears during boot, it's not clear how much reducing
> spinlock contention will help.  At times, it's not even clear that
> I make it into __start_xen.  Certainly, when I get a hang, I
> never make it out of __start_xen/init_e820/print_e820_memory_map.
> As it often stops in the middle of print_e820_memory_map
> output, that's what lead to the suspicion about serial lines.
> Clearly, though, it's merely a suspicion/guess.

That's a very interesting data point. It means that the problem you
are seeing has nothing to do with asynchronous interrupt-driven
transmit. At that point during boot characters are synchronously
written to the transmit holding register when the status register
indicates it is empty. It's all amazingly straightforward. And of
course there can be no SMP issues either.

 -- Keir