X86_64 and 4GB RAM using Flat Memory Model?

X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

Hello,
I have an odd problem on a dual processor, dual core Opteron system.
Obviosously it is x86_64 so should have no problem with large amounts of
RAM. The system has 4 GB installed (2GB on each processor).

If I boot the system with a fresh install of Debian Etch it sees all the memory
fine. dmesg reports:

Memory: 4107008k/5242880k available (1929k kernel code, 86836k
reserved, 864k data, 176k init)

However if I boot a xen-3.0.3  kernel, it only sees about half of this:
The xen dmesg reports:

Memory: 183852k/270336k available (3531k kernel code, 77948k reserved,
1250k data, 208k init)

However  'xm info'  shows:

total_memory           : 3071


This is running a source install of Xen 3.0.3  with kernel 2.6.16.29
I have checked the kernel configuration that Etch uses, compared to my
Xen kernel,
and see that Etch is using "Discontiguos Memory" model, where as the only option
in the xen compile is "Flat Memory" model.  Also Etch is using NUMA,
which doesn't
seem to be an option during the xen kernel compile.

Surely, xen can access the full memory of this system?
Can anyone recommend any things to try?

My kernel config for "processor type and features" is below.
Etch kernel uses:

CONFIG_ARCH_DISCONTIGMEM_ENABLE=y
CONFIG_ARCH_DISCONTIGMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
CONFIG_DISCONTIGMEM_MANUAL=y
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_DISCONTIGMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y

Whereas my Xen kernel config uses:

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_VSMP is not set
CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_64_XEN=y
CONFIG_X86_NO_TSS=y
CONFIG_X86_NO_IDT=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_GOOD_APIC=y
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
CONFIG_X86_IO_APIC=y
CONFIG_X86_XEN_GENAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_MTRR=y
CONFIG_SMP=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_BKL is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPLIT_PTLOCK_CPUS=4096
CONFIG_NR_CPUS=32
# CONFIG_HOTPLUG_CPU is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_SWIOTLB=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_ISA_DMA_API=y
CONFIG_GENERIC_PENDING_IRQ=y

-- 

John


-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 28/4/07 14:14, "John Hannfield" <hal9020@gmail.com> wrote:

> I have an odd problem on a dual processor, dual core Opteron system.
> Obviosously it is x86_64 so should have no problem with large amounts of
> RAM. The system has 4 GB installed (2GB on each processor).

Post the output of 'xm dmesg'.

 -- Keir

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

Hi Kier,

Thanks for the quick reply.

> Post the output of 'xm dmesg'.

Here it is below.
It reports 3GB of RAM, though BIOS says 4 GB, and Debian etch boots
with 4 GB.


-------
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0.3-0 (root@mydomain.com) (gcc version 4.1.2 20061115
(prerelease) (Debian 4.1.1-21)) Sun Mar  4 19:23:49 GMT 2007
 Latest ChangeSet: unavailable

(XEN) Command line: /boot/xen-3.0.3-0.gz dom0_mem=262144
com1=115200,8n1 console=vga,com1
(XEN) Physical RAM map:
(XEN)  0000000000000000 - 000000000009dc00 (usable)
(XEN)  0000000000100000 - 00000000bfff0000 (usable)
(XEN) System RAM: 3071MB (3145268kB)
(XEN) Xen heap: 14MB (14384kB)
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: RSDP (v002 ACPIAM                                ) @
0x00000000000f9480
(XEN) ACPI: XSDT (v001 A M I  OEMXSDT  0x12000614 MSFT 0x00000097) @
0x00000000bfff0100
(XEN) ACPI: FADT (v003 A M I  OEMFACP  0x12000614 MSFT 0x00000097) @
0x00000000bfff0290
(XEN) ACPI: MADT (v001 A M I  OEMAPIC  0x12000614 MSFT 0x00000097) @
0x00000000bfff0390
(XEN) ACPI: SPCR (v001 A M I  OEMSPCR  0x12000614 MSFT 0x00000097) @
0x00000000bfff0410
(XEN) ACPI: SLIT (v001 A M I  OEMSLIT  0x12000614 MSFT 0x00000097) @
0x00000000bfff04a0
(XEN) ACPI: OEMB (v001 A M I  AMI_OEM  0x12000614 MSFT 0x00000097) @
0x00000000bfffe040
(XEN) ACPI: HPET (v001 A M I  OEMHPET0 0x12000614 MSFT 0x00000097) @
0x00000000bfff54d0
(XEN) ACPI: SRAT (v001 AMD    HAMMER   0x00000001 AMD  0x00000001) @
0x00000000bfff5510
(XEN) ACPI: SSDT (v001 A M I  POWERNOW 0x00000001 AMD  0x00000001) @
0x00000000bfff5620
(XEN) ACPI: DSDT (v001  DATER DATER111 0x00000111 INTL 0x20051117) @
0x0000000000000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 15:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
(XEN) Processor #1 15:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
(XEN) Processor #2 15:1 APIC version 16
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
(XEN) Processor #3 15:1 APIC version 16
(XEN) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) ACPI: IRQ14 used by override.
(XEN) ACPI: IRQ15 used by override.
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x10de8201 base: 0xfed00000
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 1800.093 MHz processor.
(XEN) CPU0: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 1024K (64 bytes/line)
(XEN) CPU 0(2) -> Core 0
(XEN) AMD SVM Extension is enabled for cpu 0.
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: AMD Dual-Core AMD Opteron(tm) Processor 2210 stepping 02
(XEN) Booting processor 1/1 eip 90000
(XEN) Initializing CPU#1
(XEN) CPU1: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 1024K (64 bytes/line)
(XEN) CPU 1(2) -> Core 1
(XEN) AMD: Disabling C1 Clock Ramping Node #0
(XEN) AMD: Disabling C1 Clock Ramping Node #1
(XEN) AMD SVM Extension is enabled for cpu 1.
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: AMD Dual-Core AMD Opteron(tm) Processor 2210 stepping 02
(XEN) Booting processor 2/2 eip 90000
(XEN) Initializing CPU#2
(XEN) CPU2: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 1024K (64 bytes/line)
(XEN) CPU 2(2) -> Core 0
(XEN) AMD SVM Extension is enabled for cpu 2.
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: AMD Dual-Core AMD Opteron(tm) Processor 2210 stepping 02
(XEN) Booting processor 3/3 eip 90000
(XEN) Initializing CPU#3
(XEN) CPU3: AMD Flush Filter disabled
(XEN) CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
(XEN) CPU: L2 Cache: 1024K (64 bytes/line)
(XEN) CPU 3(2) -> Core 1
(XEN) AMD SVM Extension is enabled for cpu 3.
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: AMD Dual-Core AMD Opteron(tm) Processor 2210 stepping 02
(XEN) Total of 4 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 4 CPUs: passed.
(XEN) Platform timer is 25.000MHz HPET
(XEN) Brought up 4 CPUs
(XEN) Machine check exception polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Domain 0 kernel supports features = { 0000001f }.
(XEN) Domain 0 kernel requires features = { 00000000 }.
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000003800000->0000000004000000 (63488 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80100000->ffffffff806cf3b0
(XEN)  Init. ramdisk: ffffffff806d0000->ffffffff806d0000
(XEN)  Phys-Mach map: ffffffff806d0000->ffffffff80750000
(XEN)  Start info:    ffffffff80750000->ffffffff8075049c
(XEN)  Page tables:   ffffffff80751000->ffffffff80758000
(XEN)  Boot stack:    ffffffff80758000->ffffffff80759000
(XEN)  TOTAL:         ffffffff80000000->ffffffff80800000
(XEN)  ENTRY ADDRESS: ffffffff80100000
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM: ...............................done.
(XEN) Xen trace buffers: disabled
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen).

-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 28/4/07 14:33, "John Hannfield" <hal9020@gmail.com> wrote:

> Here it is below.
> It reports 3GB of RAM, though BIOS says 4 GB, and Debian etch boots
> with 4 GB.

What does the start of a native Linux dmesg output look like (i.e, the bits
where it prints out the memory map)?

 -- Keir

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

> What does the start of a native Linux dmesg output look like (i.e, the bits
> where it prints out the memory map)?

Yes, it's quite different.
Now that you mention it, xen sees the memory which are marked as (usable),
but not the ones marked as (reserved). Whereas debian etch kernel sees
the (reserved) ones as well. Below if the dmesg output from the debian etch
bootup.

Bootdata ok (command line is root=/dev/md0 ro )
Linux version 2.6.18-3-amd64 (Debian 2.6.18-7) (waldi@debian.org) (gcc
version 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)) #1 SMP Mon Dec
4 17:04:37 CET 2006
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
 BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000bfff0000 (usable)
 BIOS-e820: 00000000bfff0000 - 00000000bfffe000 (ACPI data)
 BIOS-e820: 00000000bfffe000 - 00000000c0000000 (ACPI NVS)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
 BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
DMI present.
ACPI: RSDP (v002 ACPIAM                                ) @ 0x00000000000f9480
ACPI: XSDT (v001 A M I  OEMXSDT  0x12000614 MSFT 0x00000097) @
0x00000000bfff0100
ACPI: FADT (v003 A M I  OEMFACP  0x12000614 MSFT 0x00000097) @
0x00000000bfff0290
ACPI: MADT (v001 A M I  OEMAPIC  0x12000614 MSFT 0x00000097) @
0x00000000bfff0390
ACPI: SPCR (v001 A M I  OEMSPCR  0x12000614 MSFT 0x00000097) @
0x00000000bfff0410
ACPI: SLIT (v001 A M I  OEMSLIT  0x12000614 MSFT 0x00000097) @
0x00000000bfff04a0
ACPI: OEMB (v001 A M I  AMI_OEM  0x12000614 MSFT 0x00000097) @
0x00000000bfffe040
ACPI: HPET (v001 A M I  OEMHPET0 0x12000614 MSFT 0x00000097) @
0x00000000bfff54d0
ACPI: SRAT (v001 AMD    HAMMER   0x00000001 AMD  0x00000001) @
0x00000000bfff5510
ACPI: SSDT (v001 A M I  POWERNOW 0x00000001 AMD  0x00000001) @
0x00000000bfff5620
ACPI: DSDT (v001  DATER DATER111 0x00000111 INTL 0x20051117) @
0x0000000000000000
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 0 -> APIC 1 -> Node 0
SRAT: PXM 1 -> APIC 2 -> Node 1
SRAT: PXM 1 -> APIC 3 -> Node 1
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 0-80000000
SRAT: Node 1 PXM 1 80000000-c0000000
SRAT: Node 1 PXM 1 80000000-140000000
NUMA: Using 31 for the hash shift.
Bootmem setup node 0 0000000000000000-0000000080000000
Bootmem setup node 1 0000000080000000-0000000140000000
On node 0 totalpages: 516134
  DMA zone: 3054 pages, LIFO batch:0
  DMA32 zone: 513080 pages, LIFO batch:31
On node 1 totalpages: 513520
  DMA32 zone: 254960 pages, LIFO batch:31
  Normal zone: 258560 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x2008
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:1 APIC version 16
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:1 APIC version 16
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
Processor #2 15:1 APIC version 16
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
Processor #3 15:1 APIC version 16
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
ACPI: IRQ14 used by override.
ACPI: IRQ15 used by override.
Setting APIC routing to physical flat
ACPI: HPET id: 0x10de8201 base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)
SMP: Allowing 4 CPUs, 0 hotplug CPUs
Built 2 zonelists.  Total pages: 1029654
Kernel command line: root=/dev/md0 ro
Initializing CPU#0
-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 28/4/07 15:30, "John Hannfield" <hal9020@gmail.com> wrote:

>> What does the start of a native Linux dmesg output look like (i.e, the bits
>> where it prints out the memory map)?
> 
> Yes, it's quite different.
> Now that you mention it, xen sees the memory which are marked as (usable),
> but not the ones marked as (reserved). Whereas debian etch kernel sees
> the (reserved) ones as well. Below if the dmesg output from the debian etch
> bootup.

Which bootloader are you using to boot Xen? It's the bootloader which
provides the memory map.

 -- Keir

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

> Which bootloader are you using to boot Xen? It's the bootloader which
> provides the memory map.

I'm using Grub.

grub --version
grub (GNU GRUB 0.97)

Someone suggested getting grub to display the memory map, which shows this.

grub> displaymem
displaymem
 EISA Memory BIOS Interface is present
 Address Map BIOS Interface is present
 Lower memory: 640K, Upper memory (to first chipset hole): 3072K
 [Address Range Descriptor entries immediately follow (values are 64-bit)]
   Usable RAM:  Base Address:  0x0 X 4GB + 0x0,
      Length:   0x0 X 4GB + 0xa0000 bytes
   Reserved:  Base Address:  0x0 X 4GB + 0xa0000,
      Length:   0x0 X 4GB + 0x60000 bytes
   Usable RAM:  Base Address:  0x0 X 4GB + 0x100000,
      Length:   0x0 X 4GB + 0x300000 bytes

Does that look right ?
My boot entry in menu.lst is:

title           Xen 3.0.3 (kernel 2.6.16 + xensource)  [*** Option B ***]
kernel          /boot/xen-3.0.3-0.gz dom0_mem=262144 com1=115200,8n1
console=vga,com1
module          /boot/vmlinuz-2.6.16.29-xen0 root=/dev/md0 ro
console=tty0 console=ttyS0


-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 28/4/07 15:39, "John Hannfield" <hal9020@gmail.com> wrote:

>  [Address Range Descriptor entries immediately follow (values are 64-bit)]
>    Usable RAM:  Base Address:  0x0 X 4GB + 0x0,
>       Length:   0x0 X 4GB + 0xa0000 bytes
>    Reserved:  Base Address:  0x0 X 4GB + 0xa0000,
>       Length:   0x0 X 4GB + 0x60000 bytes
>    Usable RAM:  Base Address:  0x0 X 4GB + 0x100000,
>       Length:   0x0 X 4GB + 0x300000 bytes

Here it looks like GRUB is claiming you have 4MB of RAM, which obviously
isn't right.

 -- Keir

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

On 4/28/07, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> On 28/4/07 15:39, "John Hannfield" <hal9020@gmail.com> wrote:
>
> >  [Address Range Descriptor entries immediately follow (values are 64-bit)]
> >    Usable RAM:  Base Address:  0x0 X 4GB + 0x0,
> >       Length:   0x0 X 4GB + 0xa0000 bytes
> >    Reserved:  Base Address:  0x0 X 4GB + 0xa0000,
> >       Length:   0x0 X 4GB + 0x60000 bytes
> >    Usable RAM:  Base Address:  0x0 X 4GB + 0x100000,
> >       Length:   0x0 X 4GB + 0x300000 bytes
>
> Here it looks like GRUB is claiming you have 4MB of RAM, which obviously
> isn't right.

You mean this bit about 3072K ?

grub> displaymem
displaymem
 EISA Memory BIOS Interface is present
 Address Map BIOS Interface is present
 Lower memory: 640K, Upper memory (to first chipset hole): 3072K

I'll try updating grub. I am using debian grub 0.9.7-23   but there is
a more recent
version available.

Thanks Kier,

-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 28/4/07 16:41, "John Hannfield" <hal9020@gmail.com> wrote:

>  Here it looks like GRUB is claiming you have 4MB of RAM, which obviously
>> isn't right.
> 
> You mean this bit about 3072K ?

Yes, and the 'Address Range Descriptors' said the same thing. However, I
think the problem is that you ran grub from the command line. If you did,
displaymem just lies to you (making it completely pointless!). To get the
right info you have to break to the command line at the grub boot menu (by
pressing 'c') and type the command there. And then write down the output
since you won't be able to cut-and-paste. :-)

> I'll try updating grub. I am using debian grub 0.9.7-23   but there is
> a more recent
> version available.

Worth a shot...

 -- Keir

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

> Yes, and the 'Address Range Descriptors' said the same thing. However, I
> think the problem is that you ran grub from the command line. If you did,
> displaymem just lies to you (making it completely pointless!).

Ahh, I see. Not very useful then!
I'll have access to the server on Tuesday, so will reboot at the grub prompt
and take a photo of the displaymem and report back.

> Worth a shot...
>
Hopefully it's a bug, that the update will fix.
I'll let you know.
Thanks

-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

Hi Kier,

OK, I have updated my grub to the latest available from Debian stable/etch.
When I access grub at the boot menu, it displays the following for
'displaymem'

grub> displaymem
EISA Memory BIOS Interface is present
Address Map BIOS Interface is present
Lower memory: 631K, Upper memory (to first chipset hole): 3144640K

So this is a bit better. More than 4 MB.
However it does not show any address ranges below that, like it does on
when booted in to the OS.

Any further ideas?

Normal debian etch booting on same machine sees all 4GB, using the same
grub. But I guess it doesn't get the memory map from grub?  If I know the memory
map from the vanilla linux boot, can I provide this another way to xen?

-- 

John

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 1/5/07 11:24, "John Hannfield" <hal9020@gmail.com> wrote:

> OK, I have updated my grub to the latest available from Debian stable/etch.
> When I access grub at the boot menu, it displays the following for
> 'displaymem'
> 
> grub> displaymem
> EISA Memory BIOS Interface is present
> Address Map BIOS Interface is present
> Lower memory: 631K, Upper memory (to first chipset hole): 3144640K

So the issue here is that grub has for some reason been upset by the values
returned to it by the e820 bios command, and has decided to fall back to a
simpler bios command which only tells you amount of memory up to the first
memory hole. This wouldn't necessarily affect Linux because it interrogates
the bios itself, using different code which isn't affected by this bug
(either it's a bios bug which Linux explicitly works around or is simply not
susceptible to; or it's a grub bug which Linux doesn't have).

> Any further ideas?

It's grim news I'm afraid.

 1. Dig into why GRUB is bailing on the e820 map, and come up with a grub
patch to fix (or work around) the bug. This will require modifying,
building, installing grub, rebooting the machine, etc etc. It's a pain in
the arse.

 2. Boot via a different method (basically that would mean pxe). This might
be convenient or impossible, depending on your machine's local network
environment.

 3. There is no command-line option for overrding the e820 map. We could
lash one up, or I can give you a basic patch for you to fill in the blanks
with your memory map. You can then build special Xen with hardcoded map for
that box.

 -- Keir

> Normal debian etch booting on same machine sees all 4GB, using the same
> grub. But I guess it doesn't get the memory map from grub?  If I know the
> memory
> map from the vanilla linux boot, can I provide this another way to xen?

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 1/5/07 14:15, "Keir Fraser" <keir@xensource.com> wrote:

>  3. There is no command-line option for overrding the e820 map. We could
> lash one up, or I can give you a basic patch for you to fill in the blanks
> with your memory map. You can then build special Xen with hardcoded map for
> that box.

I should add the slightly longer term (post 3.0.5) solution, which will be
that Xen can return to real mode and interrogate e820 itself. We could at
least provide this as a command-line option, even if it's not the default.

 -- Keir

Re: X86_64 and 4GB RAM using Flat Memory Model?

[John Hannfield]

Hi Kier,

> So the issue here is that grub has for some reason been upset by the values
> returned to it by the e820 bios command, and has decided to fall back to a
> simpler bios command which only tells you amount of memory up to the first
> memory hole. This wouldn't necessarily affect Linux because it interrogates
> the bios itself, using different code which isn't affected by this bug
> (either it's a bios bug which Linux explicitly works around or is simply not
> susceptible to; or it's a grub bug which Linux doesn't have).


OK, I understand now.
Thanks for the explanation.

>  1. Dig into why GRUB is bailing on the e820 map, and come up with a grub
> patch to fix (or work around) the bug. This will require modifying,
> building, installing grub, rebooting the machine, etc etc. It's a pain in
> the arse.

OK, I will look in to that. I have tested the latest debian grub 0.97-27
with some patches which fix BIOS e820 fixes people recommended, but
still the same.

>  2. Boot via a different method (basically that would mean pxe). This might
> be convenient or impossible, depending on your machine's local network
> environment.

Hmmm. Maybe.

>  3. There is no command-line option for overrding the e820 map. We could
> lash one up, or I can give you a basic patch for you to fill in the blanks
> with your memory map. You can then build special Xen with hardcoded map for
> that box.

That would be great. I have the memory map from the vanilla linux boot, so
if you can show me where to hardcode that in, I will give it a shot. I always
compile xen from source anyway for each server.

> I should add the slightly longer term (post 3.0.5) solution, which will be
> that Xen can return to real mode and interrogate e820 itself. We could at
> least provide this as a command-line option, even if it's not the default.

Ok, that would be useful.

Thanks

John

RE: X86_64 and 4GB RAM using Flat Memory Model?

[James Harper]

Just a suggestion, I had all sorts of problems with funny memory maps
when I was trying to get gPXE (the next version of Etherboot) running on
a particular machine. I can't remember exactly what the problem was but
it was something about overlapping memory regions, or a zero sized
memory region.

Anyway, the way I tracked down that problem was to hack the qemu bios a
bit so that it returned a memory map with the same problems, and then
hack away at the gPXE code until it worked (actually... I think I might
have given the modified qemu bios to the gPXE developers and they got it
working... can't quite remember).

Either way, gPXE with the appropriate debugging turned on might be able
to give you a verbose enough memory map dump to see if you can see what
is wrong. And if you can modify the memory map in the qemu bios to
duplicate that error, it gives you a nice sandbox to test your grub
fixes in!

James

Re: X86_64 and 4GB RAM using Flat Memory Model?

[Keir Fraser]

On 2/5/07 01:07, "James Harper" <james.harper@bendigoit.com.au> wrote:

> Just a suggestion, I had all sorts of problems with funny memory maps
> when I was trying to get gPXE (the next version of Etherboot) running on
> a particular machine. I can't remember exactly what the problem was but
> it was something about overlapping memory regions, or a zero sized
> memory region.

That might be worth looking into, although I don't think GRUB actually does
much processing of the e820 entries. Afaics it just stores them away in
order, prints them on request, and feeds them to multiboot kernels that ask
for them.

 -- Keir