[parisc-linux] Linux only see 2Gb of ram of N4k

[parisc-linux] Linux only see 2Gb of ram of N4k

[Joel Soete]

Hi all,

My collegue make me notice that the N4k on which I can test Linux was supplied
with 4Gb of ram (which if confirm at boot prompt) but Linux only see 2Gb.

It is a dual cpu machine, is that 1/2 ram dedicated to 1 cpu and the other
1/2 to the 2d cpu?

Thanks for advise,
    Joel

----------------------------------------------------------------------------------------
Tiscali ADSL: 19,50 €/mois, pendant 3 mois! L'Internet rapide, c'est pour
tout le monde.
http://reg.tiscali.be/default.asp?lg=fr

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Matthew Wilcox]

On Fri, Feb 20, 2004 at 07:04:26PM +0100, Joel Soete wrote:
> Hi all,
> 
> My collegue make me notice that the N4k on which I can test Linux was supplied
> with 4Gb of ram (which if confirm at boot prompt) but Linux only see 2Gb.
> 
> It is a dual cpu machine, is that 1/2 ram dedicated to 1 cpu and the other
> 1/2 to the 2d cpu?

Unlikely.  As I recall, the N4k diagram looks like this:

<-PCI-> Elroy <-Ropes\      PA <-+-> PA        PA <-+-> PA       /Ropes-> Elroy
<-PCI-> Elroy <-Ropes-\         DEW       RAM      DEW          /-Ropes-> Elroy
<-PCI-> Elroy <-Ropes--> IKE <-Merced-> Stretch <-Merced-> IKE <--Ropes-> Elroy
<-PCI-> Elroy <-Ropes-/         DEW       RAM      DEW          \-Ropes-> Elroy
<-PCI-> Elroy <-Ropes/      PA <-+-> PA        PA <-+-> PA       \Ropes-> Elroy

So the RAM is uniformly-accessible from all CPUs, but the IO is not.

If you look at pat_memconfig() in arch/parisc/kernel/inventory.c, you'll
see how we try to figure out what memory ranges are in the machine.
Want to try debugging that, see what's being reported by firmware?

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Joel Soete]

Matthew Wilcox wrote:
> On Fri, Feb 20, 2004 at 07:04:26PM +0100, Joel Soete wrote:
> 
>>Hi all,
>>
>>My collegue make me notice that the N4k on which I can test Linux was supplied
>>with 4Gb of ram (which if confirm at boot prompt) but Linux only see 2Gb.
>>
>>It is a dual cpu machine, is that 1/2 ram dedicated to 1 cpu and the other
>>1/2 to the 2d cpu?
> 
> 
> Unlikely.  As I recall, the N4k diagram looks like this:
> 
> <-PCI-> Elroy <-Ropes\      PA <-+-> PA        PA <-+-> PA       /Ropes-> Elroy
> <-PCI-> Elroy <-Ropes-\         DEW       RAM      DEW          /-Ropes-> Elroy
> <-PCI-> Elroy <-Ropes--> IKE <-Merced-> Stretch <-Merced-> IKE <--Ropes-> Elroy
> <-PCI-> Elroy <-Ropes-/         DEW       RAM      DEW          \-Ropes-> Elroy
> <-PCI-> Elroy <-Ropes/      PA <-+-> PA        PA <-+-> PA       \Ropes-> Elroy
> 
> So the RAM is uniformly-accessible from all CPUs, but the IO is not.

That is also what I believe to have understood from our previous talk about N Stretch mmu
(but I am never sure to well undertand, thanks to confirm)

> 
> If you look at pat_memconfig() in arch/parisc/kernel/inventory.c, you'll
> see how we try to figure out what memory ranges are in the machine.
> Want to try debugging that, see what's being reported by firmware?
> 
I will try.

Many thanks,
	Joel

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Grant Grundler]

On Fri, Feb 20, 2004 at 06:21:40PM +0000, Matthew Wilcox wrote:
> Unlikely.  As I recall, the N4k diagram looks like this:

Beautiful diagram! (I'm saving that one)
And I confirmed it is correct in case anyone had doubts.
(I did)

> So the RAM is uniformly-accessible from all CPUs, but the IO is not.

I don't think there is much penalty for CPU's to access MMIO space
on "the other side". MMIO reads are so expensive anyway, I'd doubt
it would make that much difference.

> If you look at pat_memconfig() in arch/parisc/kernel/inventory.c, you'll
> see how we try to figure out what memory ranges are in the machine.
> Want to try debugging that, see what's being reported by firmware?

I'm pretty sure it will look like this:
RAM      Phys Address
0-2GB -> 0-2GB
      -> 2-4GB I/O Space (mostly PCI MMIO)
2-4GB -> 4-6GB (courtesy of the memory controller)

The N-class was the first box which has an I/O hole from 2-4GB
physical address range. I hope Joel will be able to confirm this.
If not, I likely have documentation to confirm this.

grant

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Joel Soete]

>Beautiful diagram! (I'm saving that one)
Yes, Nice
>And I confirmed it is correct in case anyone had doubts.
(I did)

>> So the RAM is uniformly-accessible from all CPUs, but the IO is not.
>
>I don't think there is much penalty for CPU's to access MMIO space
>on "the other side". MMIO reads are so expensive anyway, I'd doubt
>it would make that much difference.
>
>> If you look at pat_memconfig() in arch/parisc/kernel/inventory.c, you'll
>> see how we try to figure out what memory ranges are in the machine.
>> Want to try debugging that, see what's being reported by firmware?
>
>I'm pretty sure it will look like this:
>RAM      Phys Address
>0-2GB -> 0-2GB
>      -> 2-4GB I/O Space (mostly PCI MMIO)
>2-4GB -> 4-6GB (courtesy of the memory controller)
>
>The N-class was the first box which has an I/O hole from 2-4GB
>physical address range. I hope Joel will be able to confirm this.
>If not, I likely have documentation to confirm this.
>
Awaiting I have a more detail look into code, I already have a cat of /proc/iomem:

palx4000:/Sources/Debian4hppa# cat /proc/iomem 
00000000-7fffffff : System RAM
  00000000-000009ff : PDC data (Page Zero)
  00100000-00451387 : Kernel code
  00451388-00605567 : Kernel data
fffffff004000000-fffffff07fffffff : LBA GMMIO
fffffff084000000-fffffff0ffffffff : LBA GMMIO
fffffff104000000-fffffff17fffffff : LBA GMMIO
fffffff204000000-fffffff27fffffff : LBA GMMIO
fffffff284000000-fffffff2ffffffff : LBA GMMIO
fffffff404000000-fffffff47fffffff : LBA GMMIO
fffffff504000000-fffffff57fffffff : LBA GMMIO
fffffff604000000-fffffff67fffffff : LBA GMMIO
fffffff804000000-fffffff87fffffff : LBA GMMIO
fffffff904000000-fffffff97fffffff : LBA GMMIO
fffffffa04000000-fffffffa7fffffff : LBA GMMIO
fffffffc04000000-fffffffc7fffffff : LBA GMMIO
fffffffd04000000-fffffffd7fffffff : LBA GMMIO
fffffffe04000000-fffffffe7fffffff : LBA GMMIO
ffffffff80000000-ffffffff81ffffff : LBA LMMIO
  ffffffff80000000-ffffffff80000fff : 0000:00:04.1
    ffffffff80000000-ffffffff80000007 : serial
    ffffffff80000008-ffffffff8000000f : serial
    ffffffff80000010-ffffffff80000017 : serial
    ffffffff80000038-ffffffff8000003f : serial
  ffffffff80001000-ffffffff80001fff : 0000:00:02.0
    ffffffff80001000-ffffffff80001fff : sym53c8xx
  ffffffff80002000-ffffffff80002fff : 0000:00:02.1
    ffffffff80002000-ffffffff80002fff : sym53c8xx
  ffffffff80003000-ffffffff800033ff : 0000:00:00.0
    ffffffff80003000-ffffffff800033ff : tulip
  ffffffff80004000-ffffffff800040ff : 0000:00:02.0
    ffffffff80004000-ffffffff800040ff : sym53c8xx
  ffffffff80005000-ffffffff800050ff : 0000:00:02.1
    ffffffff80005000-ffffffff800050ff : sym53c8xx
  ffffffff80040000-ffffffff8007ffff : 0000:00:00.0
  ffffffff80080000-ffffffff800800ff : 0000:00:01.0
    ffffffff80080000-ffffffff800800ff : sym53c8xx
  ffffffff80100000-ffffffff80100fff : 0000:00:01.0
    ffffffff80100000-ffffffff80100fff : sym53c8xx
ffffffff82000000-ffffffff83ffffff : LBA LMMIO
ffffffff84000000-ffffffff85ffffff : LBA LMMIO
ffffffff88000000-ffffffff89ffffff : LBA LMMIO
  ffffffff88000000-ffffffff8801ffff : 0000:20:00.0
  ffffffff88020000-ffffffff8803ffff : 0000:20:00.0
  ffffffff88040000-ffffffff880401ff : 0000:20:00.0
ffffffff8a000000-ffffffff8bffffff : LBA LMMIO
  ffffffff8a000000-ffffffff8a0003ff : 0000:28:00.0
    ffffffff8a000000-ffffffff8a0003ff : tulip
ffffffff90000000-ffffffff91ffffff : LBA LMMIO
ffffffff94000000-ffffffff95ffffff : LBA LMMIO
ffffffff98000000-ffffffff99ffffff : LBA LMMIO
ffffffffbffe0000-ffffffffbffe0fff : lba
ffffffffbffe2000-ffffffffbffe2fff : lba
ffffffffbffe4000-ffffffffbffe4fff : lba
ffffffffbffe8000-ffffffffbffe8fff : lba
ffffffffbffea000-ffffffffbffeafff : lba
ffffffffbfff0000-ffffffffbfff0fff : lba
ffffffffbfff4000-ffffffffbfff4fff : lba
ffffffffbfff8000-ffffffffbfff8fff : lba
ffffffffc0000000-ffffffffc1ffffff : LBA LMMIO
ffffffffc4000000-ffffffffc5ffffff : LBA LMMIO
ffffffffc8000000-ffffffffc9ffffff : LBA LMMIO
ffffffffd0000000-ffffffffd1ffffff : LBA LMMIO
ffffffffd4000000-ffffffffd5ffffff : LBA LMMIO
  ffffffffd4000000-ffffffffd401ffff : 0000:d0:00.0
  ffffffffd4020000-ffffffffd403ffff : 0000:d0:00.0
  ffffffffd4040000-ffffffffd40401ff : 0000:d0:00.0
ffffffffd8000000-ffffffffd9ffffff : LBA LMMIO
  ffffffffd8000000-ffffffffd801ffff : 0000:e0:00.0
  ffffffffd8020000-ffffffffd803ffff : 0000:e0:00.0
  ffffffffd8040000-ffffffffd80401ff : 0000:e0:00.0
fffffffffece0000-fffffffffece0fff : lba
fffffffffece4000-fffffffffece4fff : lba
fffffffffece8000-fffffffffece8fff : lba
fffffffffecf0000-fffffffffecf0fff : lba
fffffffffecf4000-fffffffffecf4fff : lba
fffffffffecf8000-fffffffffecf8fff : lba
fffffffffed00000-fffffffffed00fff : SBA
fffffffffed25000-fffffffffed25fff : CPU
fffffffffed40000-fffffffffed40fff : SBA
fffffffffff80000-fffffffffffaffff : Central Bus
fffffffffffb0000-fffffffffffdffff : Local Broadcast
fffffffffffe0000-ffffffffffffffff : Global Broadcast

Does it help?

Joel

----------------------------------------------------------------------------------------
Tiscali ADSL: 19,50 €/mois, pendant 3 mois! L'Internet rapide, c'est pour
tout le monde.
http://reg.tiscali.be/default.asp?lg=fr

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Joel Soete]

Hi Grant,

I come back to you with more relevant info (I hope)
>I'm pretty sure it will look like this:
>RAM      Phys Address
>0-2GB -> 0-2GB
>      -> 2-4GB I/O Space (mostly PCI MMIO)
>2-4GB -> 4-6GB (courtesy of the memory controller)
>
>The N-class was the first box which has an I/O hole from 2-4GB
>physical address range. I hope Joel will be able to confirm this.
>If not, I likely have documentation to confirm this.
>
I added some more printk as follow:
in arch/parisc/kernel/inventory.c

static void __init pat_memconfig(void)
{
        unsigned long actual_len;
        struct pdc_pat_pd_addr_map_entry mem_table[PAT_MAX_RANGES+1];
        struct pdc_pat_pd_addr_map_entry *mtbl_ptr;
        physmem_range_t *pmem_ptr;
        long status;
        int entries;
        unsigned long length;
        int i;

        length = (PAT_MAX_RANGES + 1) * sizeof(struct pdc_pat_pd_addr_map_entry);

        status = pdc_pat_pd_get_addr_map(&actual_len, mem_table, length,
0L);

/* JSO S */
        if ((status == PDC_OK)) {
                printk("Status == PDC_OK.\n");
                printk("Sizeof(struct pdc_pat_pd_addr_map_entry) = %d.\n",
sizeof(struct pdc_pat_pd_addr_map_entry));
                printk("actual_len = %ld.\n", actual_len);
                printk("length = %ld.\n", length);
        }
/* JSO E */
        if ((status != PDC_OK)
            || ((actual_len % sizeof(struct pdc_pat_pd_addr_map_entry)) !=
0)) {

                /* The above pdc call shouldn't fail, but, just in
                 * case, just use the PAGE0 info.
                 */

                printk("\n\n\n");
                printk(KERN_WARNING "WARNING! Could not get full memory configuration.
"
                        "All memory may not be used!\n\n\n");
                pagezero_memconfig();
                return;
        }

        entries = actual_len / sizeof(struct pdc_pat_pd_addr_map_entry);
        printk("entries = %d.\n", entries);

        if (entries > PAT_MAX_RANGES) {
                printk(KERN_WARNING "This Machine has more memory ranges
than we support!\n");
                printk(KERN_WARNING "Some memory may not be used!\n");
        }

        /* Copy information into the firmware independent pmem_ranges
         * array, skipping types we don't care about. Notice we said
         * "may" above. We'll use all the entries that were returned.
         */

        npmem_ranges = 0;
        mtbl_ptr = mem_table;
        pmem_ptr = pmem_ranges; /* Global firmware independent table */
        for (i = 0; i < entries; i++,mtbl_ptr++) {
                if (   (mtbl_ptr->entry_type != PAT_MEMORY_DESCRIPTOR)
                    || (mtbl_ptr->memory_type != PAT_MEMTYPE_MEMORY)
                    || (mtbl_ptr->pages == 0)
                    || (   (mtbl_ptr->memory_usage != PAT_MEMUSE_GENERAL)
                        && (mtbl_ptr->memory_usage != PAT_MEMUSE_GI)
                        && (mtbl_ptr->memory_usage != PAT_MEMUSE_GNI) ) )
{

                        continue;
                }

                if (npmem_ranges == MAX_PHYSMEM_RANGES) {
                        printk(KERN_WARNING "This Machine has more memory
ranges than we support!\n");
                        printk(KERN_WARNING "Some memory will not be used!\n");
                        break;
                }

                set_pmem_entry(pmem_ptr++,mtbl_ptr->paddr,mtbl_ptr->pages);
                npmem_ranges++;
        }
        for (i = 0; i < entries; i++)
                printk("pmem_ranges[%d].pages = %ld.\n", i, pmem_ranges[i].pages);
        printk("npmem_ranges = %d.\n", npmem_ranges);
}

arch/parisc/mm/init.c
static void __init setup_bootmem(void)
{
[snip]
#ifndef CONFIG_DISCONTIGMEM
        /*
         * Sort the ranges. Since the number of ranges is typically
         * small, and performance is not an issue here, just do
         * a simple insertion sort.
         */

        printk("npmem_ranges = %d.(line 143)\n", npmem_ranges);
        for (i = 1; i < npmem_ranges; i++) {
                int j;

                for (j = i; j > 0; j--) {
                        unsigned long tmp;

                        if (pmem_ranges[j-1].start_pfn <
                            pmem_ranges[j].start_pfn) {

                                break;
                        }
                        tmp = pmem_ranges[j-1].start_pfn;
                        pmem_ranges[j-1].start_pfn = pmem_ranges[j].start_pfn;
                        pmem_ranges[j].start_pfn = tmp;
                        tmp = pmem_ranges[j-1].pages;
                        pmem_ranges[j-1].pages = pmem_ranges[j].pages;
                        pmem_ranges[j].pages = tmp;
                }
        }

        /*
         * Throw out ranges that are too far apart (controlled by
         * MAX_GAP). If CONFIG_DISCONTIGMEM wasn't implemented so
         * poorly, we would recommend enabling that option, but,
         * until it is fixed, this is the best way to go.
         */

        printk("npmem_ranges = %d.(line 171)\n", npmem_ranges);
        for (i = 1; i < npmem_ranges; i++) {
                printk("pmem_ranges[%d].start_pfn = %ld.\n", i-1, pmem_ranges[i-1].pages);
                printk("pmem_ranges[%d].pages = %ld.\n", i-1, pmem_ranges[i-1].pages);
                printk("pmem_ranges[%d].start_pfn = %ld.\n", i, pmem_ranges[i].pages);
                printk("MAX_GAP = %ld.\n", MAX_GAP);
                if (pmem_ranges[i].start_pfn -
                        (pmem_ranges[i-1].start_pfn +
                         pmem_ranges[i-1].pages) > MAX_GAP) {
                        npmem_ranges = i;<=====================[ref1.]
                        break;
                }
        }
#endif

        printk("npmem_ranges = %d.(line 186)\n", npmem_ranges);
        if (npmem_ranges > 1) {
[snip]

<==== return by pat_memconfig() ====>
Status == PDC_OK.
Sizeof(struct pdc_pat_pd_addr_map_entry) = 32.
actual_len = 64.
length = 1056.
entries = 2.
pmem_ranges[0].pages = 524288.
pmem_ranges[1].pages = 524288.
npmem_ranges = 2.

So pat_memconfig() returns well 2 entries of 2Gb Ok

<==== return by setup_bootmem() ====>
npmem_ranges = 2.(line 143)
npmem_ranges = 2.(line 171)
pmem_ranges[0].start_pfn = 524288.
pmem_ranges[0].pages = 524288.
pmem_ranges[1].start_pfn = 524288.
MAX_GAP = 262144.
npmem_ranges = 1.(line 186)
pmem_ranges[0].pages = 524288.

So here only one entry is scan because of condition ([ref1]) but I have no
clue of the actual meaning of this condition. Thanks in advance for info
:)

RSize = 2147483648.
Total Memory: 2048 Mb
On node 0 totalpages: 524288

Thanks,
    Joel


----------------------------------------------------------------------------------------
Tiscali ADSL: 35 €/mois, la meilleure offre du marché!
http://reg.tiscali.be/default.asp?lg=fr

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Joel Soete]

Hi Grant,

Sorry but I make a typo in
arch/parisc/mm/init.c
static void __init setup_bootmem(void)
{
[snip]
        /*
         * Throw out ranges that are too far apart (controlled by
         * MAX_GAP). If CONFIG_DISCONTIGMEM wasn't implemented so
         * poorly, we would recommend enabling that option, but,
         * until it is fixed, this is the best way to go.
         */

        printk("npmem_ranges = %d.(line 171)\n", npmem_ranges);
        for (i = 1; i < npmem_ranges; i++) {
              printk("pmem_ranges[%d].start_pfn = %ld.\n", i-1, 
pmem_ranges[i-1].pages);  <=== should be start_pfn also

                printk("pmem_ranges[%d].pages = %ld.\n", i-1, 
pmem_ranges[i-1].pages);
                printk("pmem_ranges[%d].start_pfn = %ld.\n", i, 
pmem_ranges[i].pages);    <==== here the same
                printk("MAX_GAP = %ld.\n", MAX_GAP);
[snip]
           
And actual results would be:
<==== return by setup_bootmem() ====>
pmem_ranges[0].start_pfn = 0.
pmem_ranges[0].pages = 524288.
pmem_ranges[1].start_pfn = 1572864.

Is it much more in accordance to your knowledge?

Thanks for advise,
    Joel

PS: May I risk to play with CONFIG_DISCONTIGMEM?

----------------------------------------------------------------------------------------
Tiscali ADSL: 35 €/mois, la meilleure offre du marché!
http://reg.tiscali.be/default.asp?lg=fr

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Grant Grundler]

On Wed, Mar 24, 2004 at 05:42:06PM +0100, Joel Soete wrote:
...
> And actual results would be:
> <==== return by setup_bootmem() ====>
> pmem_ranges[0].start_pfn = 0.
> pmem_ranges[0].pages = 524288.
> pmem_ranges[1].start_pfn = 1572864.
> 
> Is it much more in accordance to your knowledge?

It's more or less what I was expecting, yes.

> PS: May I risk to play with CONFIG_DISCONTIGMEM?

Isn't this open source? of course you can.

thanks,
grant

Re: [parisc-linux] Linux only see 2Gb of ram of N4k

[Joel Soete]

> > PS: May I risk to play with CONFIG_DISCONTIGMEM?
>
> Isn't this open source? of course you can.

:))

I just didn't see any way to select it with 'make menuconfig' as there are
no Kconfig entry for parisc. I so conclude that anybody had never tested
this code (even in 2.4).

But ok when I will recover the availability of this server for my linux test,
I will try to build a kernel with this stuff.

Thanks a lot,
    Joel


----------------------------------------------------------------------------------------
Tiscali ADSL: 35 €/mois, la meilleure offre du marché!
http://reg.tiscali.be/default.asp?lg=fr