2.6.27-rc1 >4096MB issue

2.6.27-rc1 >4096MB issue

[Carl Jones]

Hi,

I'm testing 2.6.27-rc1 and seem to have an issue when booting the domU
with >4096MB.

Xen: 3.2.1 x86_64 2.6.18.8 xen.org kernel
domU: 2.6.27-rc1 32bit PAE

# xm list test1
Name                                        ID   Mem VCPUs      State   Time(s)
test1                                       24  6000     2     -b----      5.6
# xm console test1
free -m
             total       used       free     shared    buffers     cached
Mem:          1904         75       1828          0         19         15
-/+ buffers/cache:         40       1864
Swap:         2047          0       2047
testing:~# uname -a
Linux testing 2.6.27-rc1-xenU #2 SMP Thu Jul 31 22:57:42 EDT 2008 i686 GNU/Linux

(Minimal) Config: http://radium.outervoid.net/~carl/2.6.27-rc1

I see the same when using .26. Am I perhaps missing an option somewhere?

Regards,
Carl

Re: 2.6.27-rc1 >4096MB issue

[Jeremy Fitzhardinge]

Carl Jones wrote:
> Hi,
>
> I'm testing 2.6.27-rc1 and seem to have an issue when booting the domU
> with >4096MB.
>
> Xen: 3.2.1 x86_64 2.6.18.8 xen.org kernel
> domU: 2.6.27-rc1 32bit PAE
>
> # xm list test1
> Name                                        ID   Mem VCPUs      State   Time(s)
> test1                                       24  6000     2     -b----      5.6
> # xm console test1
> free -m
>              total       used       free     shared    buffers     cached
> Mem:          1904         75       1828          0         19         15
> -/+ buffers/cache:         40       1864
> Swap:         2047          0       2047
> testing:~# uname -a
> Linux testing 2.6.27-rc1-xenU #2 SMP Thu Jul 31 22:57:42 EDT 2008 i686 GNU/Linux
>
> (Minimal) Config: http://radium.outervoid.net/~carl/2.6.27-rc1
>
> I see the same when using .26. Am I perhaps missing an option somewhere?
The config has all the obvious things (namely, HIGHMEM64G).

Hm, it does look like something is truncating at 4G and wrapping.  The 
question is whether the system really only has 2G available to it, or if 
it's just mis-reporting.

Can you post the full boot log, and the contents of /proc/meminfo?

Thanks,
    J

Re: 2.6.27-rc1 >4096MB issue

[Carl Jones]

On Fri, Aug 1, 2008 at 5:13 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Carl Jones wrote:
>>
>> Hi,
>>
>> I'm testing 2.6.27-rc1 and seem to have an issue when booting the domU
>> with >4096MB.
>>
>> Xen: 3.2.1 x86_64 2.6.18.8 xen.org kernel
>> domU: 2.6.27-rc1 32bit PAE
>>
>> # xm list test1
>> Name                                        ID   Mem VCPUs      State
>> Time(s)
>> test1                                       24  6000     2     -b----
>>  5.6
>> # xm console test1
>> free -m
>>             total       used       free     shared    buffers     cached
>> Mem:          1904         75       1828          0         19         15
>> -/+ buffers/cache:         40       1864
>> Swap:         2047          0       2047
>> testing:~# uname -a
>> Linux testing 2.6.27-rc1-xenU #2 SMP Thu Jul 31 22:57:42 EDT 2008 i686
>> GNU/Linux
>>
>> (Minimal) Config: http://radium.outervoid.net/~carl/2.6.27-rc1
>>
>> I see the same when using .26. Am I perhaps missing an option somewhere?
>
> The config has all the obvious things (namely, HIGHMEM64G).
>
> Hm, it does look like something is truncating at 4G and wrapping.  The
> question is whether the system really only has 2G available to it, or if
> it's just mis-reporting.
>
> Can you post the full boot log, and the contents of /proc/meminfo?
>
> Thanks,
>   J

Sure.

testing:~# cat /proc/meminfo
MemTotal:      1949928 kB
MemFree:       1887372 kB
Buffers:          6068 kB
Cached:          15968 kB
SwapCached:          0 kB
Active:          17008 kB
Inactive:        10184 kB
HighTotal:     1204232 kB
HighFree:      1181512 kB
LowTotal:       745696 kB
LowFree:        705860 kB
SwapTotal:     2097144 kB
SwapFree:      2097144 kB
Dirty:              56 kB
Writeback:           0 kB
AnonPages:        5168 kB
Mapped:           4256 kB
Slab:             2872 kB
SReclaimable:      804 kB
SUnreclaim:       2068 kB
PageTables:        500 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:   3072108 kB
Committed_AS:    27176 kB
VmallocTotal:   116728 kB
VmallocUsed:      1208 kB
VmallocChunk:   115520 kB
DirectMap4k:    186366
DirectMap2M:         0

http://radium.outervoid.net/~carl/boot.log
http://radium.outervoid.net/~carl/xmlist.log ('xm list test1 --long'
output, in case that is helpful)

Regards,
Carl

Re: 2.6.27-rc1 >4096MB issue

[Jeremy Fitzhardinge]

Carl Jones wrote:
> http://radium.outervoid.net/~carl/boot.log
> http://radium.outervoid.net/~carl/xmlist.log ('xm list test1 --long'
> output, in case that is helpful)
>   

Thanks.  Does this fix it?

iff -r 25bf2d9a2e4c arch/x86/xen/setup.c
--- a/arch/x86/xen/setup.c	Fri Aug 01 17:12:18 2008 -0700
+++ b/arch/x86/xen/setup.c	Sat Aug 02 09:00:02 2008 -0700
@@ -42,7 +42,7 @@
 
 	e820.nr_map = 0;
 
-	e820_add_region(0, PFN_PHYS(max_pfn), E820_RAM);
+	e820_add_region(0, PFN_PHYS((u64)max_pfn), E820_RAM);
 
 	/*
 	 * Even though this is normal, usable memory under Xen, reserve


    J

Re: 2.6.27-rc1 >4096MB issue

[Carl Jones]

On Sun, Aug 3, 2008 at 4:00 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Carl Jones wrote:
>>
>> http://radium.outervoid.net/~carl/boot.log
>> http://radium.outervoid.net/~carl/xmlist.log ('xm list test1 --long'
>> output, in case that is helpful)
>>
>
> Thanks.  Does this fix it?
>
> iff -r 25bf2d9a2e4c arch/x86/xen/setup.c
> --- a/arch/x86/xen/setup.c      Fri Aug 01 17:12:18 2008 -0700
> +++ b/arch/x86/xen/setup.c      Sat Aug 02 09:00:02 2008 -0700
> @@ -42,7 +42,7 @@
>
>        e820.nr_map = 0;
>
> -       e820_add_region(0, PFN_PHYS(max_pfn), E820_RAM);
> +       e820_add_region(0, PFN_PHYS((u64)max_pfn), E820_RAM);
>
>        /*
>         * Even though this is normal, usable memory under Xen, reserve
>
>
>   J
>

Yep works nicely now. I tested up to 15GB or so with with that patch
applied and CONFIG_XEN_MAX_DOMAIN_MEMORY=32 set:

testing:~# cat /proc/meminfo
MemTotal:     15769832 kB

Regards,
Carl

Re: 2.6.27-rc1 >4096MB issue

[Jeremy Fitzhardinge]

Carl Jones wrote:
> On Sun, Aug 3, 2008 at 4:00 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>   
>> Carl Jones wrote:
>>     
>>> http://radium.outervoid.net/~carl/boot.log
>>> http://radium.outervoid.net/~carl/xmlist.log ('xm list test1 --long'
>>> output, in case that is helpful)
>>>
>>>       
>> Thanks.  Does this fix it?
>>
>> iff -r 25bf2d9a2e4c arch/x86/xen/setup.c
>> --- a/arch/x86/xen/setup.c      Fri Aug 01 17:12:18 2008 -0700
>> +++ b/arch/x86/xen/setup.c      Sat Aug 02 09:00:02 2008 -0700
>> @@ -42,7 +42,7 @@
>>
>>        e820.nr_map = 0;
>>
>> -       e820_add_region(0, PFN_PHYS(max_pfn), E820_RAM);
>> +       e820_add_region(0, PFN_PHYS((u64)max_pfn), E820_RAM);
>>
>>        /*
>>         * Even though this is normal, usable memory under Xen, reserve
>>
>>
>>   J
>>
>>     
>
> Yep works nicely now. I tested up to 15GB or so with with that patch
> applied and CONFIG_XEN_MAX_DOMAIN_MEMORY=32 set:
>
> testing:~# cat /proc/meminfo
> MemTotal:     15769832 kB
>   

Excellent, thanks.  That's 32-bit?  (That's a pretty silly amount of 
memory to give to a 32-bit system, but it's nice to know it works.)

Could you try this patch instead to see if it works?  It's a more 
general fix.

Thanks,
    J


Subject: make PFN_PHYS explicitly return 64-bit result

PFN_PHYS, as its name suggests, turns a pfn into a physical address.
However, it is a macro which just operates on its argument without
modifying its type.  pfns are typed unsigned long, but an unsigned
long may not be long enough to hold a physical address (32-bit systems
with more than 32 bits of physcial address).  This means that the
resulting address could be truncated if it doesn't fit within an
unsigned long.  This isn't generally a problem because most users end
up using it for "low" memory, but there's no reason why PFN_PHYS
couldn't be used for any possible pfn.

Unfortunately there's no univerally recognized type for holding a full
physical address, so this patch makes it always return a u64 result.
In theory this could generate worse code, but in practice it will make
no difference:
- most users end up casting the result to a pointer or unsigned long,
  so on 32-bit systems the compiler should be able to eliminate the 64
  bit part of the expression.
- most users are in init code, which is neither size or performance
  critical

This patch also introduces PFN_LOW_PHYS which explicitly operates on
an unsigned long.  It is currently unused, but it could be used to
document the fact that the caller expects that the result will fit
into an unsigned long.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Cc: Eduardo Habkost <ehabkost@redhat.com>
---
 arch/m32r/mm/discontig.c   |    2 +-
 include/asm-x86/xen/page.h |    4 ++--
 include/linux/pfn.h        |    3 ++-
 3 files changed, 5 insertions(+), 4 deletions(-)

===================================================================
--- a/arch/m32r/mm/discontig.c
+++ b/arch/m32r/mm/discontig.c
@@ -112,7 +112,7 @@
 				initrd_start, INITRD_SIZE);
 		} else {
 			printk("initrd extends beyond end of memory "
-				"(0x%08lx > 0x%08lx)\ndisabling initrd\n",
+				"(0x%08lx > 0x%08llx)\ndisabling initrd\n",
 				INITRD_START + INITRD_SIZE,
 				PFN_PHYS(max_low_pfn));
 
===================================================================
--- a/include/asm-x86/xen/page.h
+++ b/include/asm-x86/xen/page.h
@@ -76,13 +76,13 @@
 static inline xmaddr_t phys_to_machine(xpaddr_t phys)
 {
 	unsigned offset = phys.paddr & ~PAGE_MASK;
-	return XMADDR(PFN_PHYS((u64)pfn_to_mfn(PFN_DOWN(phys.paddr))) | offset);
+	return XMADDR(PFN_PHYS(pfn_to_mfn(PFN_DOWN(phys.paddr))) | offset);
 }
 
 static inline xpaddr_t machine_to_phys(xmaddr_t machine)
 {
 	unsigned offset = machine.maddr & ~PAGE_MASK;
-	return XPADDR(PFN_PHYS((u64)mfn_to_pfn(PFN_DOWN(machine.maddr))) | offset);
+	return XPADDR(PFN_PHYS(mfn_to_pfn(PFN_DOWN(machine.maddr))) | offset);
 }
 
 /*
===================================================================
--- a/include/linux/pfn.h
+++ b/include/linux/pfn.h
@@ -4,6 +4,7 @@
 #define PFN_ALIGN(x)	(((unsigned long)(x) + (PAGE_SIZE - 1)) & PAGE_MASK)
 #define PFN_UP(x)	(((x) + PAGE_SIZE-1) >> PAGE_SHIFT)
 #define PFN_DOWN(x)	((x) >> PAGE_SHIFT)
-#define PFN_PHYS(x)	((x) << PAGE_SHIFT)
+#define PFN_PHYS(x)	((u64)(x) << PAGE_SHIFT)
+#define PFN_LOW_PHYS(x)	((unsigned long)(x) << PAGE_SHIFT)
 
 #endif

Re: 2.6.27-rc1 >4096MB issue

[Carl Jones]

On Tue, Aug 5, 2008 at 3:07 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Carl Jones wrote:
>>
>> On Sun, Aug 3, 2008 at 4:00 AM, Jeremy Fitzhardinge <jeremy@goop.org>
>> wrote:
>>
>>>
>>> Carl Jones wrote:
>>>
>>>>
>>>> http://radium.outervoid.net/~carl/boot.log
>>>> http://radium.outervoid.net/~carl/xmlist.log ('xm list test1 --long'
>>>> output, in case that is helpful)
>>>>
>>>>
>>>
>>> Thanks.  Does this fix it?
>>>
>>> iff -r 25bf2d9a2e4c arch/x86/xen/setup.c
>>> --- a/arch/x86/xen/setup.c      Fri Aug 01 17:12:18 2008 -0700
>>> +++ b/arch/x86/xen/setup.c      Sat Aug 02 09:00:02 2008 -0700
>>> @@ -42,7 +42,7 @@
>>>
>>>       e820.nr_map = 0;
>>>
>>> -       e820_add_region(0, PFN_PHYS(max_pfn), E820_RAM);
>>> +       e820_add_region(0, PFN_PHYS((u64)max_pfn), E820_RAM);
>>>
>>>       /*
>>>        * Even though this is normal, usable memory under Xen, reserve
>>>
>>>
>>>  J
>>>
>>>
>>
>> Yep works nicely now. I tested up to 15GB or so with with that patch
>> applied and CONFIG_XEN_MAX_DOMAIN_MEMORY=32 set:
>>
>> testing:~# cat /proc/meminfo
>> MemTotal:     15769832 kB
>>
>
> Excellent, thanks.  That's 32-bit?  (That's a pretty silly amount of memory
> to give to a 32-bit system, but it's nice to know it works.)
>
> Could you try this patch instead to see if it works?  It's a more general
> fix.
>
> Thanks,
>   J
>
>
> Subject: make PFN_PHYS explicitly return 64-bit result

Hi,

Guest is 32bit. That patch works fine too.

Regards,
Carl

Re: 2.6.27-rc1 >4096MB issue

[Jan Beulich]

>>> Jeremy Fitzhardinge <jeremy@goop.org> 05.08.08 05:07 >>>
>Subject: make PFN_PHYS explicitly return 64-bit result
>
>PFN_PHYS, as its name suggests, turns a pfn into a physical address.
>However, it is a macro which just operates on its argument without
>modifying its type.  pfns are typed unsigned long, but an unsigned
>long may not be long enough to hold a physical address (32-bit systems
>with more than 32 bits of physcial address).  This means that the
>resulting address could be truncated if it doesn't fit within an
>unsigned long.  This isn't generally a problem because most users end
>up using it for "low" memory, but there's no reason why PFN_PHYS
>couldn't be used for any possible pfn.
>
>Unfortunately there's no univerally recognized type for holding a full
>physical address, so this patch makes it always return a u64 result.

Couldn't you use resource_size_t here?

Jan

Re: 2.6.27-rc1 >4096MB issue

[Jeremy Fitzhardinge]

Jan Beulich wrote:
>>>> Jeremy Fitzhardinge <jeremy@goop.org> 05.08.08 05:07 >>>
>>>>         
>> Subject: make PFN_PHYS explicitly return 64-bit result
>>
>> PFN_PHYS, as its name suggests, turns a pfn into a physical address.
>> However, it is a macro which just operates on its argument without
>> modifying its type.  pfns are typed unsigned long, but an unsigned
>> long may not be long enough to hold a physical address (32-bit systems
>> with more than 32 bits of physcial address).  This means that the
>> resulting address could be truncated if it doesn't fit within an
>> unsigned long.  This isn't generally a problem because most users end
>> up using it for "low" memory, but there's no reason why PFN_PHYS
>> couldn't be used for any possible pfn.
>>
>> Unfortunately there's no univerally recognized type for holding a full
>> physical address, so this patch makes it always return a u64 result.
>>     
>
> Couldn't you use resource_size_t here?

Yeah, looks like I can.  It had crossed my mind, but I'd vaguely 
remembered that it might no be set if you don't have 64-bit IO devices.  
But it looks like it's set for all interesting cases (64 bit machines, 
and 32-bit x86 PAE).

    J