Re: Kernel writes to RAM it doesn't own on 2.4.24

Re: Kernel writes to RAM it doesn't own on 2.4.24

[Ross Dickson]

On Fri, 16 Apr 2004, Ross Biro wrote: 

> On Fri, 16 Apr 2004, Richard B. Johnson wrote: 
> > 
> > On Fri, 16 Apr 2004, Ross Biro wrote: 
> > 
> > > mem= isn't there to tell the kernel what ram it owns and what ram it 
> > > doesn't own. It's there to tell the kernel what ram is in the system. 
> > > Since you told the system it only has 500m, it assumes the rest of 
> > > the 3.5G of address space is available for things like memory mapped 
> > > i/o. If you cat /proc/iomem, you'll probably see something has 
> > > reserved the memory range in question. 
> > > 
> > 
> > No! This is address space, not RAM. Whether or not a PCI device 
> > or whatever has internal RAM that's mapped makes no difference. 
> > 
> > I told the kernel that it has 500m of RAM. It better not assume 
> > I don't know what I'm talking about. I might have reserved that 
> > RAM because it's bad or I may have something else important to 
> > do with that RAM (which I do). 
 


> The problem is that the kernel does assume you know what you are 
> talking about, and you don't. You are abusing the mem= parameter. 
> That's fine, but then you have to tell the kernel what you really 
> mean. What you really want to say is there is memory above 500M and I 
> don't want you to touch it. There may be a way to do that via the 
> fancy mem=@ parameters. 
 


> What mem= tells the kernel is that there is RAM in a certain spot an 
>  no where else. Since you told the kernel there is no ram about 500M, 
> that means that address space is free to be used for memory mapped 
> I/O. Since the kernel trusts you, it started using the memory above 
> 500m for memory mapped i/o. Since you LIED to the kernel, you are 
> getting results you do not like. The solution I settled on was to 
> tell the kernel that people LIE to it and only use memory for I/O if 
> both the BIOS and the USER agree that it's available. You have to 
> find a way to tell the kernel the TRUTH, or you will never get the 
> results you want. 
> - 


This is all most enlightening. If I am understanding correctly then every
device driver that the author specifies to use a "mem=" command to 
reserve some memory for said drivers use at the upper part of physical
memory is stuffed by design. 

I thought it was a valid technique? I never questioned it because there is
a history of its use -I think the early bttv driver was written this way.

I have been debugging an oops on a system which uses the open source
driver for the Matrox MeteorII multichannel available from,
http://www.emlix.com/index.php?id=158
This driver uses the technique and I am getting a corrupted slab free list. 

Ross B, could I please have details of your mem bios hack please so I can try
it as a workaround.

Regards
Ross Dickson

Re: Kernel writes to RAM it doesn't own on 2.4.24

[Ross Biro]

On Sat, 17 Apr 2004 14:40:18 +1000, Ross Dickson
<ross@datscreative.com.au> wrote:
> This is all most enlightening. If I am understanding correctly then every
> device driver that the author specifies to use a "mem=" command to
> reserve some memory for said drivers use at the upper part of physical
> memory is stuffed by design.

The problem is really that Linux doesn't trust the BARs assigned by
the PCI bios because some BIOSes do it incorrectly.  So it reprograms
them based on the memory map it got from the BIOS.  However, before it
does that the mem= parameter overrides the memory map from the BIOS.

I believe what I did was to save a copy of the e820 maps for later,
and then take then take the first free address as the max of the first
free address from the user supplied map and the bios supplied map. 
I'll send out a patch on Tuesday.

  Ross

Re: Kernel writes to RAM it doesn't own on 2.4.24

[Ross Biro]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

> I believe what I did was to save a copy of the e820 maps for later,
> and then take then take the first free address as the max of the first
> free address from the user supplied map and the bios supplied map.
> I'll send out a patch on Tuesday.

Here's the changes to setup.c.  I haven't check to see if it's
complete, but I did the diff from a working 2.4.18-kernel.  You need
to apply this patch in arch/i386/kernel.

[-- Attachment #2: stuff --]
[-- Type: application/octet-stream, Size: 3418 bytes --]

--- /home/rossb/local/linux-2.4.18/arch/i386/kernel/setup.c	2003-09-23 19:55:13.000000000 -0700
+++ setup.c	2004-03-24 10:20:56.000000000 -0800
@@ -113,6 +113,17 @@
 #include <asm/dma.h>
 #include <asm/mpspec.h>
 #include <asm/mmu_context.h>
+
+#define PFN_UP(x)	(((x) + PAGE_SIZE-1) >> PAGE_SHIFT)
+#define PFN_DOWN(x)	((x) >> PAGE_SHIFT)
+#define PFN_PHYS(x)	((x) << PAGE_SHIFT)
+
+/*
+ * Reserved space for vmalloc and iomap - defined in asm/page.h
+ */
+#define MAXMEM_PFN	PFN_DOWN(MAXMEM)
+#define MAX_NONPAE_PFN	(1 << 20)
+
 /*
  * Machine setup..
  */
@@ -121,6 +132,8 @@
 struct cpuinfo_x86 boot_cpu_data = { 0, 0, 0, 0, -1, 1, 0, 0, -1 };
 
 unsigned long mmu_cr4_features;
+unsigned long num_e820physpages;
+unsigned long num_e820_nonpae_physpages;
 
 /*
  * Bus types ..
@@ -151,6 +164,7 @@
 };
 
 struct e820map e820;
+struct e820map bios_e820;
 
 unsigned char aux_device_present;
 
@@ -710,11 +724,44 @@
 	char c = ' ', *to = command_line, *from = COMMAND_LINE;
 	int len = 0;
 	int usermem = 0;
+        int i;
+        unsigned long max_pfn;
 
 	/* Save unparsed command line copy for /proc/cmdline */
 	memcpy(saved_command_line, COMMAND_LINE, COMMAND_LINE_SIZE);
 	saved_command_line[COMMAND_LINE_SIZE-1] = '\0';
 
+        /* Compute this here in case the user wants to know it later. */
+	max_pfn = 0;
+        max_low_pfn = 0;
+	for (i = 0; i < e820.nr_map; i++) {
+		unsigned long start, end;
+		/* RAM? */
+		if (e820.map[i].type != E820_RAM &&
+                    e820.map[i].type != E820_ACPI)
+			continue;
+
+		start = PFN_UP(e820.map[i].addr);
+		end = PFN_DOWN(e820.map[i].addr + e820.map[i].size);
+		if (start >= end)
+			continue;
+                if (end > MAX_NONPAE_PFN) {
+                     if (start < MAX_NONPAE_PFN) {
+                          max_low_pfn = MAX_NONPAE_PFN;
+                     }
+                } else {
+                     if (end > max_low_pfn) {
+                          max_low_pfn = end;
+                     }
+                }
+		if (end > max_pfn)
+			max_pfn = end;
+	}
+
+        num_e820physpages = max_pfn;
+        num_e820_nonpae_physpages = max_low_pfn;
+        memcpy(&bios_e820, &e820, sizeof(e820));
+
 	for (;;) {
 		/*
 		 * "mem=nopentium" disables the 4MB page tables.
@@ -783,6 +830,7 @@
 {
 	unsigned long bootmap_size, low_mem_size;
 	unsigned long start_pfn, max_pfn, max_low_pfn;
+        unsigned long e820_low_mem_size;
 	int i;
 
 #ifdef CONFIG_VISWS
@@ -822,16 +870,6 @@
 
 	parse_mem_cmdline(cmdline_p);
 
-#define PFN_UP(x)	(((x) + PAGE_SIZE-1) >> PAGE_SHIFT)
-#define PFN_DOWN(x)	((x) >> PAGE_SHIFT)
-#define PFN_PHYS(x)	((x) << PAGE_SHIFT)
-
-/*
- * Reserved space for vmalloc and iomap - defined in asm/page.h
- */
-#define MAXMEM_PFN	PFN_DOWN(MAXMEM)
-#define MAX_NONPAE_PFN	(1 << 20)
-
 	/*
 	 * partially used pages are not usable - thus
 	 * we are rounding upwards:
@@ -1032,8 +1070,16 @@
 
 	/* Tell the PCI layer not to allocate too close to the RAM area.. */
 	low_mem_size = ((max_low_pfn << PAGE_SHIFT) + 0xfffff) & ~0xfffff;
+
+        e820_low_mem_size = ((num_e820_nonpae_physpages << PAGE_SHIFT) + 
+                             0xfffff) & ~0xfffff;
+
+	if (e820_low_mem_size > pci_mem_start)
+             pci_mem_start = e820_low_mem_size;
+        
 	if (low_mem_size > pci_mem_start)
 		pci_mem_start = low_mem_size;
+                     
 
 #ifdef CONFIG_VT
 #if defined(CONFIG_VGA_CONSOLE)

Re: Kernel writes to RAM it doesn't own on 2.4.24

[H. Peter Anvin]

Followup to:  <200404171440.18829.ross@datscreative.com.au>
By author:    Ross Dickson <ross@datscreative.com.au>
In newsgroup: linux.dev.kernel
> 
> This is all most enlightening. If I am understanding correctly then every
> device driver that the author specifies to use a "mem=" command to 
> reserve some memory for said drivers use at the upper part of physical
> memory is stuffed by design. 
> 

Yup.

	-hpa

Kernel writes to RAM it doesn't own on 2.4.24

[Richard B. Johnson]

Hello again,

If I start a system that has 1 Gb of memory with mem=500m,
the value of the kernel's num_physpages is 0x20000 as would
be expected. If I multiply that by PAGE_SIZE, I get 0x20000000,
also as expected. If I observe that memory region, I note
that somebody has written something there!

This is not good. The kernel touches RAM it doesn't own. I have
booted the system with only the internal floppy controller
and no other modules installed. I see the same thing.

Script started on Fri Apr 16 11:33:39 2004
# monitor
                  TMD Platinum(tm) Control System Version 2.0
                 Copyright(c) 1999-2003, Analogic Corporation

  Enter "help" for commands

PLATINUM> dump=20000000
20000000  78 56 34 12 21 43 65 87-FF FF FF FF FF FF FF FF   xV4.!Ce.........
20000010  FF FF FF FF FF FF FF FD-FF FF FF FF FF FF FF FF   ................
20000020  FF FF FF FF FF FE FF FF-FF FF FE FF FF FF FF FF   ................
[SNIPPED...]

My temporary work around for the kernel's destroying a
precious DMA buffer is to start one page higher. However,
whomever is writing to that RAM is likely writing other
places it doesn't belong also. This could lead to some
very interesting bugs.

Note that the value written there is 0x12345678, twice, once
in little endian and another in swap-nibble big endian, like
a mirror. This is evil.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (5596.77 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Kernel writes to RAM it doesn't own on 2.4.24

[Ross Biro]

mem= isn't there to tell the kernel what ram it owns and what ram it
doesn't own.  It's there to tell the kernel what ram is in the system.
 Since you told the system it only has 500m, it assumes the rest of
the 3.5G of address space is available for things like memory mapped
i/o.  If you cat /proc/iomem, you'll probably see something has
reserved the memory range in question.

I added a hack to make the kernel assume the greater of the mem= and
what is passed to in from the BIOS via the e820 maps is where the
unused address space starts.  It seems to eliminate such problems.

    Ross

On Fri, 16 Apr 2004 11:55:28 -0400 (EDT), Richard B. Johnson
<root@chaos.analogic.com> wrote:
> 
> 
> Hello again,
> 
> If I start a system that has 1 Gb of memory with mem=500m,
> the value of the kernel's num_physpages is 0x20000 as would
> be expected. If I multiply that by PAGE_SIZE, I get 0x20000000,
> also as expected. If I observe that memory region, I note
> that somebody has written something there!
> 
> This is not good. The kernel touches RAM it doesn't own. I have
> booted the system with only the internal floppy controller
> and no other modules installed. I see the same thing.
> 
> Script started on Fri Apr 16 11:33:39 2004
> # monitor
>                   TMD Platinum(tm) Control System Version 2.0
>                  Copyright(c) 1999-2003, Analogic Corporation
> 
>   Enter "help" for commands
> 
> PLATINUM> dump=20000000
> 20000000  78 56 34 12 21 43 65 87-FF FF FF FF FF FF FF FF   xV4.!Ce.........
> 20000010  FF FF FF FF FF FF FF FD-FF FF FF FF FF FF FF FF   ................
> 20000020  FF FF FF FF FF FE FF FF-FF FF FE FF FF FF FF FF   ................
> [SNIPPED...]
> 
> My temporary work around for the kernel's destroying a
> precious DMA buffer is to start one page higher. However,
> whomever is writing to that RAM is likely writing other
> places it doesn't belong also. This could lead to some
> very interesting bugs.
> 
> Note that the value written there is 0x12345678, twice, once
> in little endian and another in swap-nibble big endian, like
> a mirror. This is evil.
> 
> Cheers,
> Dick Johnson
> Penguin : Linux version 2.4.24 on an i686 machine (5596.77 BogoMips).
>             Note 96.31% of all statistics are fiction.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Kernel writes to RAM it doesn't own on 2.4.24

[Richard B. Johnson]

On Fri, 16 Apr 2004, Ross Biro wrote:

> mem= isn't there to tell the kernel what ram it owns and what ram it
> doesn't own.  It's there to tell the kernel what ram is in the system.
>  Since you told the system it only has 500m, it assumes the rest of
> the 3.5G of address space is available for things like memory mapped
> i/o.  If you cat /proc/iomem, you'll probably see something has
> reserved the memory range in question.
>

No!  This is address space, not RAM. Whether or not a PCI device
or whatever has internal RAM that's mapped makes no difference.

I told the kernel that it has 500m of RAM. It better not assume
I don't know what I'm talking about. I might have reserved that
RAM because it's bad or I may have something else important to
do with that RAM (which I do).

> I added a hack to make the kernel assume the greater of the mem= and
> what is passed to in from the BIOS via the e820 maps is where the
> unused address space starts.  It seems to eliminate such problems.
>
>     Ross
>
> On Fri, 16 Apr 2004 11:55:28 -0400 (EDT), Richard B. Johnson
> <root@chaos.analogic.com> wrote:


Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (5596.77 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Kernel writes to RAM it doesn't own on 2.4.24

[Ross Biro]

On Fri, 16 Apr 2004 12:55:33 -0400 (EDT), Richard B. Johnson
<root@chaos.analogic.com> wrote:
> 
> On Fri, 16 Apr 2004, Ross Biro wrote:
> 
> > mem= isn't there to tell the kernel what ram it owns and what ram it
> > doesn't own.  It's there to tell the kernel what ram is in the system.
> >  Since you told the system it only has 500m, it assumes the rest of
> > the 3.5G of address space is available for things like memory mapped
> > i/o.  If you cat /proc/iomem, you'll probably see something has
> > reserved the memory range in question.
> >
> 
> No!  This is address space, not RAM. Whether or not a PCI device
> or whatever has internal RAM that's mapped makes no difference.
> 
> I told the kernel that it has 500m of RAM. It better not assume
> I don't know what I'm talking about. I might have reserved that
> RAM because it's bad or I may have something else important to
> do with that RAM (which I do).

The problem is that the kernel does assume you know what you are
talking about, and you don't.  You are abusing the mem= parameter. 
That's fine, but then you have to tell the kernel what you really
mean.  What you really want to say is there is memory above 500M and I
don't want you to touch it.  There may be a way to do that via the
fancy mem=@ parameters.

What mem= tells the kernel is that there is RAM in a certain spot an
no where else.  Since you told the kernel there is no ram about 500M,
that means that address space is free to be used for memory mapped
I/O.  Since the kernel trusts you, it started using the memory above
500m for memory mapped i/o.  Since you LIED to the kernel, you are
getting results you do not like.  The solution I settled on was to
tell the kernel that people LIE to it and only use memory for I/O if
both the BIOS and the USER agree that it's available.   You have to
find a way to tell the kernel the TRUTH, or you will never get the
results you want.