Daily Xen Build

Daily Xen Build

[David F Barrera]

August 24, 2005, using HG source as of:

changeset:   6375:603f55eaa690ef8d47e54bdb57e20fb3266d8f56
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Wed Aug 24 05:48:24 2005
summary:     Initialise syscall32 vsyscall page early, as it may be
needed


x86_32 (no PAE support)
=======================   

SLES 9 SP2 on IBM xSeries 235 (512M RAM), 335 (2GB RAM) and FC3 on IBM
ThinkCentre (1GB RAM)

ISSUE: 
              
Creating DomU causes Dom0 to crash 
* SLES 9, RHEL 4 and FC3 boxen, PAE and non-PAE.
* Creating DomU without networking does NOT cause crash.
* Bugzilla #188
                  
changeset:   6329:3889ca17ff5867d5efa19eaf25463d59dd6c8c7d
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Tue Aug 23 07:30:35 2005
summary:     phys_to_machine_mapping array is not an array of longs.
     

x86_32 (PAE)
============

SLES 9 SP2, FC4, and RHEL 4 IBM xSeries 305, 335, and IBM ThinkCentre

    
ISSUES:
Creating DomU with networking crashes Dom0, Bugzilla #188

DomU boot crashes Dom0, Bugzilla #168 
* Workaround: use swiotlb=force as a kernel parameter when
                booting dom0
* DomU boots up normally if using swiotlb=force

X server will not come up on Dom0 on a FC3 box when Xen built with PAE
'on', Bugzilla #137

Networking issue on FC3 when Xen built with PAE, Bugzilla #173
* DomU unable to get dhcp address
* tcpdump shows lots of networking activity being 'heard'
* ping from external box shows corrupt packets
* DomU has no trouble getting IP address when Xen built non-PAE

X server does not come up on FC3 box with PAE support, Bugzilla #137
* X server comes up normally when xen built without PAE support
                
Last good changeset for x86_32,  PAE:
        
changeset:   6329:3889ca17ff5867d5efa19eaf25463d59dd6c8c7d
tag:         tip
user:        kaf24@firebug.cl.cam.ac.uk
date:        Tue Aug 23 07:30:35 2005
summary:     phys_to_machine_mapping array is not an array of longs.
        


x86_64 (SLES 9 SP2 and FC4 on IBM HS20 Blades)
==============================================

Problems (severe) booting up Dom0 on x86_64 platforms (SLES 9 and FC4)

ISSUES:

Bugzilla Bug 187- Dom0 will not boot on x86_64 machines, SLES 9 SP2 and
FC4.
* Unable to handle kernel paging request at ffff800000373fc0 RIP:
<ffffffff80120499>{__ioremap+105}

         
Bugzilla #176  - x86_64 - *UNABLE TO BOOT DOMU*
* VFS: Cannot open root device "sdb2" or unknown-block (0,0)
* Please append a correct "root=" boot option
* Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-
block(0,0)
              * 
Bugzilla #169 - x86_64 - Unable to handle kernel paging request at
ffff8000003e0000 RIP: <ffffffff8015d16c>{unmap_vmas+1084}
        
Last good changeset for x86_64:
        
changeset: 6228:1a94949348ff52863b9fbef4eaf102c7e46a345d
tag: tip
user: kaf24@firebug.cl.cam.ac.uk
date: Thu Aug 18 05:41:55 2005
Summary: Fix range_straddles_boundary() check to exclude regions that


-- 
Regards,

David F Barrera
Linux Technology Center
Systems and Technology Group, IBM

"The wisest men follow their own direction. "
                                                        Euripides

RE: Daily Xen Build

[Puthiyaparambil, Aravindh]

David,

I am able to boot Xen and Dom0 on my x86_64 Dell and ES7000 box running
SLES9 SP2. When I try to bring up DomUs I run into Bug 176. Only
difference is that now I see a lot of the following messages during the
DomU boot.

xen_net: Initialising virtual ethernet driver.
xen_net (probe_interfaces:1304) >
xen_net (probe_interfaces:1307) > wait_i=0
xen_net (probe_interfaces:1310) > schedule_timeout...
xen_net (probe_interfaces:1307) > wait_i=1
xen_net (probe_interfaces:1310) > schedule_timeout...
xen_net (probe_interfaces:1307) > wait_i=2

Aravindh

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of David F Barrera
> Sent: Wednesday, August 24, 2005 11:16 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Daily Xen Build
> 
> 
> August 24, 2005, using HG source as of:
> 
> changeset:   6375:603f55eaa690ef8d47e54bdb57e20fb3266d8f56
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Wed Aug 24 05:48:24 2005
> summary:     Initialise syscall32 vsyscall page early, as it may be
> needed
> 
> 
> x86_32 (no PAE support)
> =======================
> 
> SLES 9 SP2 on IBM xSeries 235 (512M RAM), 335 (2GB RAM) and FC3 on IBM
> ThinkCentre (1GB RAM)
> 
> ISSUE:
> 
> Creating DomU causes Dom0 to crash
> * SLES 9, RHEL 4 and FC3 boxen, PAE and non-PAE.
> * Creating DomU without networking does NOT cause crash.
> * Bugzilla #188
> 
> changeset:   6329:3889ca17ff5867d5efa19eaf25463d59dd6c8c7d
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Tue Aug 23 07:30:35 2005
> summary:     phys_to_machine_mapping array is not an array of longs.
> 
> 
> x86_32 (PAE)
> ============
> 
> SLES 9 SP2, FC4, and RHEL 4 IBM xSeries 305, 335, and IBM ThinkCentre
> 
> 
> ISSUES:
> Creating DomU with networking crashes Dom0, Bugzilla #188
> 
> DomU boot crashes Dom0, Bugzilla #168
> * Workaround: use swiotlb=force as a kernel parameter when
>                 booting dom0
> * DomU boots up normally if using swiotlb=force
> 
> X server will not come up on Dom0 on a FC3 box when Xen built with PAE
> 'on', Bugzilla #137
> 
> Networking issue on FC3 when Xen built with PAE, Bugzilla #173
> * DomU unable to get dhcp address
> * tcpdump shows lots of networking activity being 'heard'
> * ping from external box shows corrupt packets
> * DomU has no trouble getting IP address when Xen built non-PAE
> 
> X server does not come up on FC3 box with PAE support, Bugzilla #137
> * X server comes up normally when xen built without PAE support
> 
> Last good changeset for x86_32,  PAE:
> 
> changeset:   6329:3889ca17ff5867d5efa19eaf25463d59dd6c8c7d
> tag:         tip
> user:        kaf24@firebug.cl.cam.ac.uk
> date:        Tue Aug 23 07:30:35 2005
> summary:     phys_to_machine_mapping array is not an array of longs.
> 
> 
> 
> x86_64 (SLES 9 SP2 and FC4 on IBM HS20 Blades)
> ==============================================
> 
> Problems (severe) booting up Dom0 on x86_64 platforms (SLES 9 and FC4)
> 
> ISSUES:
> 
> Bugzilla Bug 187- Dom0 will not boot on x86_64 machines, SLES 9 SP2
and
> FC4.
> * Unable to handle kernel paging request at ffff800000373fc0 RIP:
> <ffffffff80120499>{__ioremap+105}
> 
> 
> Bugzilla #176  - x86_64 - *UNABLE TO BOOT DOMU*
> * VFS: Cannot open root device "sdb2" or unknown-block (0,0)
> * Please append a correct "root=" boot option
> * Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-
> block(0,0)
>               *
> Bugzilla #169 - x86_64 - Unable to handle kernel paging request at
> ffff8000003e0000 RIP: <ffffffff8015d16c>{unmap_vmas+1084}
> 
> Last good changeset for x86_64:
> 
> changeset: 6228:1a94949348ff52863b9fbef4eaf102c7e46a345d
> tag: tip
> user: kaf24@firebug.cl.cam.ac.uk
> date: Thu Aug 18 05:41:55 2005
> Summary: Fix range_straddles_boundary() check to exclude regions that
> 
> 
> --
> Regards,
> 
> David F Barrera
> Linux Technology Center
> Systems and Technology Group, IBM
> 
> "The wisest men follow their own direction. "
>                                                         Euripides
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Daily Xen Build

[Keir Fraser]

On 24 Aug 2005, at 16:15, David F Barrera wrote:

> August 24, 2005, using HG source as of:

Most of the bugs in this report are fixed in the repository versions 
you mention. With current tip I would expect only the domU-crashes-dom0 
network bug to be reproducible (the one that's fixed by disabling 
network grant refs).

Bugs fixed include the one that needed you to specify swiotlb=force, 
the one that prevented you running X on PAE, and the one that prevented 
you booting x86/64 domUs.

  -- Keir

Re: Daily Xen Build

[David F Barrera]

On Wed, 2005-08-24 at 16:34 +0100, Keir Fraser wrote:
> On 24 Aug 2005, at 16:15, David F Barrera wrote:
> 
> > August 24, 2005, using HG source as of:
> 
> Most of the bugs in this report are fixed in the repository versions 
> you mention. With current tip I would expect only the domU-crashes-dom0 
> network bug to be reproducible (the one that's fixed by disabling 
> network grant refs).
> 
> Bugs fixed include the one that needed you to specify swiotlb=force, 
> the one that prevented you running X on PAE,
You are absolutely right. I have closed those defects in bugzilla
>  and the one that prevented you booting x86/64 domUs.
I can't verify this since I can't boot Dom0 on x86_64, bugzilla #187
(new).
> 
>   -- Keir
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
-- 
Regards,

David F Barrera
Linux Technology Center
Systems and Technology Group, IBM

"The wisest men follow their own direction. "
                                                        Euripides

Re: Daily Xen Build

[Keir Fraser]

On 24 Aug 2005, at 17:51, David F Barrera wrote:

>>  and the one that prevented you booting x86/64 domUs.
> I can't verify this since I can't boot Dom0 on x86_64, bugzilla #187
> (new).

Weird - I can't reporoduce this myself, even though my test box has a 
tg3 (and it looks like tg3 initialisation is what's causing the crash).

I'll see if I can kick the kernel around some more tomorrow and get the 
behaviour you observe.

  -- Keir

RE: Daily Xen Build

[Nakajima, Jun]

Keir Fraser wrote:
> On 24 Aug 2005, at 17:51, David F Barrera wrote:
> 
>>>  and the one that prevented you booting x86/64 domUs.
>> I can't verify this since I can't boot Dom0 on x86_64, bugzilla #187
>> (new).
> 
> Weird - I can't reporoduce this myself, even though my test box has a
> tg3 (and it looks like tg3 initialisation is what's causing the
> crash). 
> 
> I'll see if I can kick the kernel around some more tomorrow and get
> the behaviour you observe.
> 
>   -- Keir
> 

I'm also seeing SMP dom0 broken, and only 1 CPU is up:

Booting processor 1/1 rip ffffffff80100014 rsp ffff8801ee715f58
(XEN) DOM0: (file=mm.c, line=1455) Bad type (saw 00000000f0000001!= exp
00000000a0000000) for mfn 9d000 (pfn 0)
Booting processor 2/2 rip ffffffff80100014 rsp ffff8801ee71ff58
(XEN) DOM0: (file=mm.c, line=1455) Bad type (saw 00000000f0000001!= exp
00000000a0000000) for mfn 9d002 (pfn 2)
...


Jun
---
Intel Open Source Technology Center

RE: Daily Xen Build

[Nakajima, Jun]

Nakajima, Jun wrote:
> Keir Fraser wrote:
>> On 24 Aug 2005, at 17:51, David F Barrera wrote:
>> 
>>>>  and the one that prevented you booting x86/64 domUs.
>>> I can't verify this since I can't boot Dom0 on x86_64, bugzilla #187
>>> (new).
>> 
>> Weird - I can't reporoduce this myself, even though my test box has a
>> tg3 (and it looks like tg3 initialisation is what's causing the
>> crash). 
>> 
>> I'll see if I can kick the kernel around some more tomorrow and get
>> the behaviour you observe. 
>> 
>>   -- Keir
>> 
> 
> I'm also seeing SMP dom0 broken, and only 1 CPU is up:
> 
> Booting processor 1/1 rip ffffffff80100014 rsp ffff8801ee715f58
> (XEN) DOM0: (file=mm.c, line=1455) Bad type (saw 00000000f0000001!=
> exp 00000000a0000000) for mfn 9d000 (pfn 0)
> Booting processor 2/2 rip ffffffff80100014 rsp ffff8801ee71ff58
> (XEN) DOM0: (file=mm.c, line=1455) Bad type (saw 00000000f0000001!=
> exp 00000000a0000000) for mfn 9d002 (pfn 2)
> ...
> 
> 
> Jun
> ---
> Intel Open Source Technology Center
> 
If I revert the changeset 6379, they are back.

--- a/linux-2.6-xen-sparse/arch/xen/x86_64/kernel/setup.c	Wed Aug
24 15:22:44 2005
+++ b/linux-2.6-xen-sparse/arch/xen/x86_64/kernel/setup.c	Wed Aug
24 15:46:33 2005
@@ -428,8 +428,9 @@
 {
         unsigned long bootmap_size = init_bootmem(start_pfn, end_pfn);
         free_bootmem(0, end_pfn << PAGE_SHIFT);   
-        /* XXX KAF: Why can't we leave low 1MB of memory free? */
-        reserve_bootmem(0, (PFN_PHYS(start_pfn) + bootmap_size +
PAGE_SIZE-1));
+        reserve_bootmem(HIGH_MEMORY,
+                        (PFN_PHYS(start_pfn) + bootmap_size +
PAGE_SIZE-1)
+                        - HIGH_MEMORY);
 }
 #else
 static void __init contig_initmem_init(void)


Jun
---
Intel Open Source Technology Center

Re: Daily Xen Build

[Keir Fraser]

> 
> On 24 Aug 2005, at 17:51, David F Barrera wrote:
> 
> >>  and the one that prevented you booting x86/64 domUs.
> > I can't verify this since I can't boot Dom0 on x86_64, bugzilla #187
> > (new).
> 
> Weird - I can't reporoduce this myself, even though my test box has a 
> tg3 (and it looks like tg3 initialisation is what's causing the crash).
> 
> I'll see if I can kick the kernel around some more tomorrow and get the 
> behaviour you observe.

Can you try adding some tracing to __ioremap() and is_local_lowmem()
in arch/xen/i386/mm/ioremap.c.

In __ioremap() there is a section in the middle that is conditional on
is_local_lowmem(phys_addr). Add a printk in there to see if we execute
that conditional code.

If we do, it would be interesting to instrument
is_local_lowmem(). Add something like this:
 printk(KERN_ALERT " *** %lx %lx %lx %x\n",
        mfn, pfn, max_low_pfn, phys_to_machine_mapping[pfn]);

 -- Keir

Re: Daily Xen Build

[David F Barrera]

Keir Fraser wrote:

>>On 24 Aug 2005, at 17:51, David F Barrera wrote:
>>
>>    
>>
>>>> and the one that prevented you booting x86/64 domUs.
>>>>        
>>>>
>>>I can't verify this since I can't boot Dom0 on x86_64, bugzilla #187
>>>(new).
>>>      
>>>
>>Weird - I can't reporoduce this myself, even though my test box has a 
>>tg3 (and it looks like tg3 initialisation is what's causing the crash).
>>
>>I'll see if I can kick the kernel around some more tomorrow and get the 
>>behaviour you observe.
>>    
>>
>
>Can you try adding some tracing to __ioremap() and is_local_lowmem()
>in arch/xen/i386/mm/ioremap.c.
>  
>
Should this be in arch/xen/*x86_64*/mm/ioremap.c. ?

>In __ioremap() there is a section in the middle that is conditional on
>is_local_lowmem(phys_addr). Add a printk in there to see if we execute
>that conditional code.
>
>If we do, it would be interesting to instrument
>is_local_lowmem(). Add something like this:
> printk(KERN_ALERT " *** %lx %lx %lx %x\n",
>        mfn, pfn, max_low_pfn, phys_to_machine_mapping[pfn]);
>
> -- Keir
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

Re: Daily Xen Build

[Keir Fraser]

On 25 Aug 2005, at 16:39, David F Barrera wrote:

> Should this be in arch/xen/*x86_64*/mm/ioremap.c. ?

No, the 64-bit build shares the file from xen/i386.

  -- Keir

RE: Daily Xen Build

[Nakajima, Jun]

This problem has been fixed by the changeset 6395, and all the
processors are up on x86_64 SMP xenlinux. Thanks a lot for the cleanups,
Keir.

Jun
---
Intel Open Source Technology Center


Nakajima, Jun wrote:
>> 
>> I'm also seeing SMP dom0 broken, and only 1 CPU is up:
>> 
>> Booting processor 1/1 rip ffffffff80100014 rsp ffff8801ee715f58
>> (XEN) DOM0: (file=mm.c, line=1455) Bad type (saw 00000000f0000001!=
>> exp 00000000a0000000) for mfn 9d000 (pfn 0)
>> Booting processor 2/2 rip ffffffff80100014 rsp ffff8801ee71ff58
>> (XEN) DOM0: (file=mm.c, line=1455) Bad type (saw 00000000f0000001!=
>> exp 00000000a0000000) for mfn 9d002 (pfn 2)
>> ...
>> 
>> 
>> Jun
>> ---
>> Intel Open Source Technology Center
>> 
> If I revert the changeset 6379, they are back.
> 
> --- a/linux-2.6-xen-sparse/arch/xen/x86_64/kernel/setup.c	Wed Aug
> 24 15:22:44 2005
> +++ b/linux-2.6-xen-sparse/arch/xen/x86_64/kernel/setup.c	Wed Aug
> 24 15:46:33 2005
> @@ -428,8 +428,9 @@
>  {
>          unsigned long bootmap_size = init_bootmem(start_pfn,
>          end_pfn); free_bootmem(0, end_pfn << PAGE_SHIFT);
> -        /* XXX KAF: Why can't we leave low 1MB of memory free? */
> -        reserve_bootmem(0, (PFN_PHYS(start_pfn) + bootmap_size +
> PAGE_SIZE-1));
> +        reserve_bootmem(HIGH_MEMORY,
> +                        (PFN_PHYS(start_pfn) + bootmap_size +
> PAGE_SIZE-1)
> +                        - HIGH_MEMORY);
>  }
>  #else
>  static void __init contig_initmem_init(void)
> 
> 
> Jun
> ---
> Intel Open Source Technology Center
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Daily Xen Build

[David F Barrera]

Keir Fraser wrote:

>Can you try adding some tracing to __ioremap() and is_local_lowmem()
>in arch/xen/i386/mm/ioremap.c.
>
>In __ioremap() there is a section in the middle that is conditional on
>is_local_lowmem(phys_addr). Add a printk in there to see if we execute
>that conditional code.
>
>  
>
Keir,

I doesn't seem to have executed it. Here's the trace:
[...]
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80100000->ffffffff80647086
(XEN)  Init. ramdisk: ffffffff80648000->ffffffff80648000
(XEN)  Phys-Mach map: ffffffff80648000->ffffffff80686800
(X...done.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch 
input to Xen).
Linux version 2.6.12-xen0 (root@bl2-1) (gcc version 3.3.3 (SuSE Linux)) 
#3 Fri Aug 26 00:04:19 CDT 2005
kernel direct mapping tables upto ffff88000fa00000 @ 800000-87f000
No mptable found.
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x07] enabled)
QIÝ¡¥±¥¹¥Ñ¥±¥é¥¹u%IEͥͱá¹}µµ%¹¥Ñ¥±¥Í¥¹M%ÍÕÍåÍѵ¥¹¥Ñ¥±¥éÕͽɥ±¥é¥¹
                RAMDISK driver initialized: 16 RAM disks of 16384K size 
1024 blocksize
loop: loaded (max 8 devices)
HP CISS Driver (v 2.6.6)
Intel(R) PRO/1000 Network Driver - version 6.0.54-k2
Copyright (c) 1999-2004 Intel Corporation.
pcnet32.c:v1.30j 29.04.2005 tsbogend@alpha.franken.de
e100: Intel(R) PRO/100 Network Driver, 3.4.8-k2-NAPI
e100: Copyright(c) 1999-2005 Intel Corporation
tg3.c:v3.31 (June 8, 2005)
ACPI: PCI Interrupt 0000:05:01.0[A] -> GSI 77 (level, low) -> IRQ 77
Unable to handle kernel paging request at ffff800000373fc0 RIP:
<ffffffff80120436>{__ioremap+118}
PGD 55555555067 BAD
Oops: 0000 [1]
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.12-xen0
RIP: e030:[<ffffffff80120436>] <ffffffff80120436>{__ioremap+118}
RSP: e02b:ffff880000a43c58  EFLAGS: 00010206
RAX: 00000000000dcff0 RBX: 0000000000010000 RCX: 00000000000dcff0
RDX: 00000000dc463000 RSI: 0000000000010000 RDI: 00000000dcff0000
RBP: 0000000} <ffffffff802adec4>{tg3_init_one+644}
       <ffffffff8012d990>{default_wake_function+0} 
<ffffffff8012d990>{default_wake_function+0}
       <ffffffff80140579>{__queue_work+73} 
<ffffffff80140119>{call_usermodehelper+217}
       <ffffffff80140140>{__call_usermodehelper+0} 
<ffffffff80229269>{pci_device_probe+121}
       <ffffffff8026371d>{driver_probe_device+77} 
<ffffffff80263796>{driver_attach+70}
       <ffffffff80263878>{bus_add_driver+152} 
<ffffffff80229545>{pci_register_driver+117}

>If we do, it would be interesting to instrument
>is_local_lowmem(). Add something like this:
> printk(KERN_ALERT " *** %lx %lx %lx %x\n",
>        mfn, pfn, max_low_pfn, phys_to_machine_mapping[pfn]);
>
> -- Keir
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>