Xen hypervisor hangs on booting against c/s 16925

Xen hypervisor hangs on booting against c/s 16925

[Li, Haicheng]

Xen hangs on booting with following log printed: 

(XEN) Initializing CPU#0 
(XEN) Detected 2992.574 MHz processor. 
(XEN) (XEN) (XEN) (XEN) (XEN) 

This issue was introduced by c/s 16889.

Here is the bugzilla entry:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157

-- haicheng

RE: Xen hypervisor hangs on booting against c/s 16925

[Li, Haicheng]

Li, Haicheng wrote:
> Xen hangs on booting with following log printed:
> 
> (XEN) Initializing CPU#0
> (XEN) Detected 2992.574 MHz processor.
> (XEN) (XEN) (XEN) (XEN) (XEN)
> 
> This issue was introduced by c/s 16889.
> 
> Here is the bugzilla entry:
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157
> 

This issue only happens when we pass parameter "console=com1,vga" to
xen.
If just using "console=com1", we can bypass this issue.

-- haicheng

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

On 29/1/08 07:40, "Li, Haicheng" <haicheng.li@intel.com> wrote:

>> Xen hangs on booting with following log printed:
>> 
>> (XEN) Initializing CPU#0
>> (XEN) Detected 2992.574 MHz processor.
>> (XEN) (XEN) (XEN) (XEN) (XEN)
>> 
>> This issue was introduced by c/s 16889.
>> 
>> Here is the bugzilla entry:
>> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157
> 
> This issue only happens when we pass parameter "console=com1,vga" to
> xen.
> If just using "console=com1", we can bypass this issue.

Ah, easily fixed. Most of our internal testing is serial only. :-)

 -- Keir

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

Fixed by c/s 16926.

 -- Keir

On 29/1/08 05:58, "Li, Haicheng" <haicheng.li@intel.com> wrote:

> Xen hangs on booting with following log printed:
> 
> (XEN) Initializing CPU#0
> (XEN) Detected 2992.574 MHz processor.
> (XEN) (XEN) (XEN) (XEN) (XEN)
> 
> This issue was introduced by c/s 16889.
> 
> Here is the bugzilla entry:
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157
> 
> -- haicheng
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen hypervisor hangs on booting against c/s 16925

[Jan Beulich]

Don't you think that it would also be reasonable to map all of the RAM
holes below 1M uncacheable, not just the VGA space? Since early boot
code can't know this, I added a map_pages_to_xen(...,
PAGE_HYPERVISOR_NOCACHE) for the below-1Mb case of the code
that you just fixed.

Jan

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 10:37 >>>
Fixed by c/s 16926.

 -- Keir

On 29/1/08 05:58, "Li, Haicheng" <haicheng.li@intel.com> wrote:

> Xen hangs on booting with following log printed:
> 
> (XEN) Initializing CPU#0
> (XEN) Detected 2992.574 MHz processor.
> (XEN) (XEN) (XEN) (XEN) (XEN)
> 
> This issue was introduced by c/s 16889.
> 
> Here is the bugzilla entry:
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157 
> 
> -- haicheng
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com 
http://lists.xensource.com/xen-devel

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

Isn't the memory map below 1MB well known, and no I/O exists down there
except for VGA? Below that is conventional RAM; above that is ROM space.

 -- Keir

On 29/1/08 10:05, "Jan Beulich" <jbeulich@novell.com> wrote:

> Don't you think that it would also be reasonable to map all of the RAM
> holes below 1M uncacheable, not just the VGA space? Since early boot
> code can't know this, I added a map_pages_to_xen(...,
> PAGE_HYPERVISOR_NOCACHE) for the below-1Mb case of the code
> that you just fixed.
> 
> Jan
> 
>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 10:37 >>>
> Fixed by c/s 16926.
> 
>  -- Keir
> 
> On 29/1/08 05:58, "Li, Haicheng" <haicheng.li@intel.com> wrote:
> 
>> Xen hangs on booting with following log printed:
>> 
>> (XEN) Initializing CPU#0
>> (XEN) Detected 2992.574 MHz processor.
>> (XEN) (XEN) (XEN) (XEN) (XEN)
>> 
>> This issue was introduced by c/s 16889.
>> 
>> Here is the bugzilla entry:
>> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157
>> 
>> -- haicheng
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Xen hypervisor hangs on booting against c/s 16925

[Christoph Egger]

On Tuesday 29 January 2008 11:20:24 Keir Fraser wrote:
> Isn't the memory map below 1MB well known, and no I/O exists down there
> except for VGA? Below that is conventional RAM; above that is ROM space.
>  -- Keir
>

You remind me on the old days where everyone played with config.sys and 
autoexec.bat. ;-)

Christoph


> On 29/1/08 10:05, "Jan Beulich" <jbeulich@novell.com> wrote:
> > Don't you think that it would also be reasonable to map all of the RAM
> > holes below 1M uncacheable, not just the VGA space? Since early boot
> > code can't know this, I added a map_pages_to_xen(...,
> > PAGE_HYPERVISOR_NOCACHE) for the below-1Mb case of the code
> > that you just fixed.
> >
> > Jan
> >
> >>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 10:37 >>>
> >
> > Fixed by c/s 16926.
> >
> >  -- Keir
> >
> > On 29/1/08 05:58, "Li, Haicheng" <haicheng.li@intel.com> wrote:
> >> Xen hangs on booting with following log printed:
> >>
> >> (XEN) Initializing CPU#0
> >> (XEN) Detected 2992.574 MHz processor.
> >> (XEN) (XEN) (XEN) (XEN) (XEN)
> >>
> >> This issue was introduced by c/s 16889.
> >>
> >> Here is the bugzilla entry:
> >> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157
> >>
> >> -- haicheng




-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: Xen hypervisor hangs on booting against c/s 16925

[Jan Beulich]

I/O can at least theoretically exist anywhere in that space, not just in
the VGA range. Also, mapping ROM WB seems suspicious at least - you're
relying on MTRR settings then without really needing to. Otoh, with
PAGE_HYPERVISOR_NOCACHE encoding UC- rather than UC, mapping
WP MTRR regions this way would yield (model specific) WC, which again
seems suspicious, so maybe the VGA hole should simply be kept (from
boot, at UC-), but everything else should be forced to UC (or MTRRs
should be inquired so that at least WP regions can be kept WP).

Jan

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 11:20 >>>
Isn't the memory map below 1MB well known, and no I/O exists down there
except for VGA? Below that is conventional RAM; above that is ROM space.

 -- Keir

On 29/1/08 10:05, "Jan Beulich" <jbeulich@novell.com> wrote:

> Don't you think that it would also be reasonable to map all of the RAM
> holes below 1M uncacheable, not just the VGA space? Since early boot
> code can't know this, I added a map_pages_to_xen(...,
> PAGE_HYPERVISOR_NOCACHE) for the below-1Mb case of the code
> that you just fixed.
> 
> Jan
> 
>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 10:37 >>>
> Fixed by c/s 16926.
> 
>  -- Keir
> 
> On 29/1/08 05:58, "Li, Haicheng" <haicheng.li@intel.com> wrote:
> 
>> Xen hangs on booting with following log printed:
>> 
>> (XEN) Initializing CPU#0
>> (XEN) Detected 2992.574 MHz processor.
>> (XEN) (XEN) (XEN) (XEN) (XEN)
>> 
>> This issue was introduced by c/s 16889.
>> 
>> Here is the bugzilla entry:
>> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157 
>> 
>> -- haicheng
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com 
>> http://lists.xensource.com/xen-devel 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com 
> http://lists.xensource.com/xen-devel 
> 

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

How about we just map the whole megabyte UC- at start of day? We rarely
touch anything down there and when we do it'll mostly be BIOS tables (hence
in ROM and hence not marked as RAM in e820 and hence your approach would
have them also marked UC-). That at least has the tangible benefit of
removing a few lines of (assembly) code. :-)

 -- Keir

On 29/1/08 10:34, "Jan Beulich" <jbeulich@novell.com> wrote:

> I/O can at least theoretically exist anywhere in that space, not just in
> the VGA range. Also, mapping ROM WB seems suspicious at least - you're
> relying on MTRR settings then without really needing to. Otoh, with
> PAGE_HYPERVISOR_NOCACHE encoding UC- rather than UC, mapping
> WP MTRR regions this way would yield (model specific) WC, which again
> seems suspicious, so maybe the VGA hole should simply be kept (from
> boot, at UC-), but everything else should be forced to UC (or MTRRs
> should be inquired so that at least WP regions can be kept WP).
> 
> Jan
> 
>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 11:20 >>>
> Isn't the memory map below 1MB well known, and no I/O exists down there
> except for VGA? Below that is conventional RAM; above that is ROM space.
> 
>  -- Keir
> 
> On 29/1/08 10:05, "Jan Beulich" <jbeulich@novell.com> wrote:
> 
>> Don't you think that it would also be reasonable to map all of the RAM
>> holes below 1M uncacheable, not just the VGA space? Since early boot
>> code can't know this, I added a map_pages_to_xen(...,
>> PAGE_HYPERVISOR_NOCACHE) for the below-1Mb case of the code
>> that you just fixed.
>> 
>> Jan
>> 
>>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 10:37 >>>
>> Fixed by c/s 16926.
>> 
>>  -- Keir
>> 
>> On 29/1/08 05:58, "Li, Haicheng" <haicheng.li@intel.com> wrote:
>> 
>>> Xen hangs on booting with following log printed:
>>> 
>>> (XEN) Initializing CPU#0
>>> (XEN) Detected 2992.574 MHz processor.
>>> (XEN) (XEN) (XEN) (XEN) (XEN)
>>> 
>>> This issue was introduced by c/s 16889.
>>> 
>>> Here is the bugzilla entry:
>>> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1157
>>> 
>>> -- haicheng
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 
> 
> 
> 

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

Also, bear in mind that dom0 permanently has a WB mapping of the whole
bottom megabyte.

 -- Keir

On 29/1/08 10:36, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> How about we just map the whole megabyte UC- at start of day? We rarely touch
> anything down there and when we do it'll mostly be BIOS tables (hence in ROM
> and hence not marked as RAM in e820 and hence your approach would have them
> also marked UC-). That at least has the tangible benefit of removing a few
> lines of (assembly) code. :-)
> 
>  -- Keir

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

On 29/1/08 10:34, "Jan Beulich" <jbeulich@novell.com> wrote:

> Otoh, with
> PAGE_HYPERVISOR_NOCACHE encoding UC- rather than UC, mapping
> WP MTRR regions this way would yield (model specific) WC, which again
> seems suspicious, so maybe the VGA hole should simply be kept (from
> boot, at UC-), but everything else should be forced to UC (or MTRRs
> should be inquired so that at least WP regions can be kept WP).

If we don't write to a MTRR WP region, then actually mapping as WC does not
matter. We never write to the range 0xc0000-0xfffff. Anything we write to
below 0xa0000 had better be RAM, or we've got bigger problems than the WP
attribute!

 -- Keir

Re: Xen hypervisor hangs on booting against c/s 16925

[Jan Beulich]

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 11:45 >>>
>On 29/1/08 10:34, "Jan Beulich" <jbeulich@novell.com> wrote:
>
>> Otoh, with
>> PAGE_HYPERVISOR_NOCACHE encoding UC- rather than UC, mapping
>> WP MTRR regions this way would yield (model specific) WC, which again
>> seems suspicious, so maybe the VGA hole should simply be kept (from
>> boot, at UC-), but everything else should be forced to UC (or MTRRs
>> should be inquired so that at least WP regions can be kept WP).
>
>If we don't write to a MTRR WP region, then actually mapping as WC does not
>matter. We never write to the range 0xc0000-0xfffff. Anything we write to
>below 0xa0000 had better be RAM, or we've got bigger problems than the WP
>attribute!

Unfortunately it does matter - WC does not permit read caching, but
does permit speculative reads. That's both not what we want.

Jan

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

On 29/1/08 10:52, "Jan Beulich" <jbeulich@novell.com> wrote:

>> If we don't write to a MTRR WP region, then actually mapping as WC does not
>> matter. We never write to the range 0xc0000-0xfffff. Anything we write to
>> below 0xa0000 had better be RAM, or we've got bigger problems than the WP
>> attribute!
> 
> Unfortunately it does matter - WC does not permit read caching, but
> does permit speculative reads. That's both not what we want.

Is your concern that something with read side effects (i.e., I/O) would be
mapped with WP type in the MTRR? Does that not seem rather unlikely?

Anyhow, WP attribute also allows read speculation, just like WC, so in
respect of read speculation it does not matter which caching attribute we
actually use.

 -- Keir

Re: Xen hypervisor hangs on booting against c/s 16925

[Jan Beulich]

.. which is bad, and should probably be changed, too. Jan

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 11:40 >>>
Also, bear in mind that dom0 permanently has a WB mapping of the whole
bottom megabyte.

 -- Keir

On 29/1/08 10:36, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> How about we just map the whole megabyte UC- at start of day? We rarely touch
> anything down there and when we do it'll mostly be BIOS tables (hence in ROM
> and hence not marked as RAM in e820 and hence your approach would have them
> also marked UC-). That at least has the tangible benefit of removing a few
> lines of (assembly) code. :-)
> 
>  -- Keir

Re: Xen hypervisor hangs on booting against c/s 16925

[Keir Fraser]

Does Linux have a permanent mapping, indeed a superpage mapping, of the
bottom megabyte? I think it does. The fact that this persists in the Linux
kernel, which is thoroughly tested on many many platforms, I think indicates
we are over-worrying about mappings of the bottom megabyte.

 -- Keir

On 29/1/08 10:54, "Jan Beulich" <jbeulich@novell.com> wrote:

> .. which is bad, and should probably be changed, too. Jan
> 
>>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 11:40 >>>
> Also, bear in mind that dom0 permanently has a WB mapping of the whole
> bottom megabyte.
> 
>  -- Keir
> 
> On 29/1/08 10:36, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> How about we just map the whole megabyte UC- at start of day? We rarely touch
>> anything down there and when we do it'll mostly be BIOS tables (hence in ROM
>> and hence not marked as RAM in e820 and hence your approach would have them
>> also marked UC-). That at least has the tangible benefit of removing a few
>> lines of (assembly) code. :-)
>> 
>>  -- Keir
> 
> 
> 

Re: Xen hypervisor hangs on booting against c/s 16925

[Jan Beulich]

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 29.01.08 11:54 >>>
>On 29/1/08 10:52, "Jan Beulich" <jbeulich@novell.com> wrote:
>
>>> If we don't write to a MTRR WP region, then actually mapping as WC does not
>>> matter. We never write to the range 0xc0000-0xfffff. Anything we write to
>>> below 0xa0000 had better be RAM, or we've got bigger problems than the WP
>>> attribute!
>> 
>> Unfortunately it does matter - WC does not permit read caching, but
>> does permit speculative reads. That's both not what we want.
>
>Is your concern that something with read side effects (i.e., I/O) would be
>mapped with WP type in the MTRR? Does that not seem rather unlikely?

No, it was rather the ordering of reads that I was concerned about, but ...

>Anyhow, WP attribute also allows read speculation, just like WC, so in
>respect of read speculation it does not matter which caching attribute we
>actually use.

... yes, I didn't look closely enough at the WP characteristics. So we
would only inhibit read caching then.

Jan

Re: Xen hypervisor hangs on booting against c/s 16925

[Alan Cox]

On Tue, 29 Jan 2008 10:20:24 +0000
Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:

> Isn't the memory map below 1MB well known, and no I/O exists down there
> except for VGA? Below that is conventional RAM; above that is ROM space.

Low 640K RAM
Above that ISA hole
Above that ROM (often shadowed into RAM)

Of the 640K RAM the low page is owned by the BIOS as are pages above the
EBDA. Some systems also use a 1 page EBDA to hide hardware bugs with
DMA prefetches running off the end of the 640K of RAM into I/O space.

The E820 map then tells you which bits of the 640K are owned by the BIOS
ACPI layer.

The ISA window is all non-cachable device mmio but rarely used for
anything but the PCI routed legacy VGA and MDA.

Alan

Re: Xen hypervisor hangs on booting against c/s 16925

[Alan Cox]

On Tue, 29 Jan 2008 10:36:57 +0000
Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:

> How about we just map the whole megabyte UC- at start of day? We rarely
> touch anything down there and when we do it'll mostly be BIOS tables (hence
> in ROM and hence not marked as RAM in e820 

Most BIOS tables are in RAM because they are built each boot. Mapping the
low 640K UC is also going to suprise the BIOS somewhat and seems to be
asking for trouble.

The MTRR mappings for the low 1Mb are special anyway and aligned with the
need for 640K RAM and a following ISA hole.

Alan