srmmu: out of nocache

srmmu: out of nocache

[lists]

Hi!

I have an old SS5 with 128 MB memory. Sometimes, the kernel seems to have some 
memory problems:
Mar 24 07:03:05 ferdi kernel: bitmap out: size 8800 used 6242 off 1248 len 16 
align 16 count 8816
Mar 24 07:03:05 ferdi kernel: srmmu: out of nocache 4096: 2252800/1597952
Mar 24 07:03:05 ferdi kernel: VM: killing process sendmail

Here is an actual memory overview: 

ferdi:~# cat /proc/meminfo
MemTotal:       125256 kB
MemFree:          6744 kB
Buffers:         17836 kB
Cached:          51156 kB
SwapCached:      13556 kB
Active:          73420 kB
Inactive:        33572 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       125256 kB
LowFree:          6744 kB
SwapTotal:      249652 kB
SwapFree:       228840 kB
Dirty:             188 kB
Writeback:           0 kB
Mapped:          40640 kB
Slab:             7736 kB
Committed_AS:   141536 kB
PageTables:        856 kB
VmallocTotal:    25600 kB
VmallocUsed:       500 kB
VmallocChunk:    25060 kB

These problems are reproducible under high load (e. g. updating spamassassins 
bayes-database while receiving a lot of mails). What can I do to fix this 
behaviour? Are 128 MB memory not enough? My x86-bases systems have no 
problems with less memory...

regards,
Jörg

-- 
Hi! I'm a .signature virus! Copy me into your signature to help me spread!.-.
PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D    oo|
PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605                  _ // /`'\
I am Ohm of Borg. Resistance is voltage divided by current.         \X/ (\_;/)

Re: srmmu: out of nocache

[Keith M Wesolowski]

On Sun, Mar 28, 2004 at 09:53:19PM +0200, lists@naasa.net wrote:

> I have an old SS5 with 128 MB memory. Sometimes, the kernel seems to have some 
> memory problems:
> Mar 24 07:03:05 ferdi kernel: bitmap out: size 8800 used 6242 off 1248 len 16 
> align 16 count 8816
> Mar 24 07:03:05 ferdi kernel: srmmu: out of nocache 4096: 2252800/1597952
> Mar 24 07:03:05 ferdi kernel: VM: killing process sendmail

You didn't even say what kernel version you use.  This was fairly
common in early 2.6; I haven't had any reports of it since 2.6.4 or
so.

-- 
Keith M Wesolowski

Re: srmmu: out of nocache

[Joerg Platte]

Am Montag, 29. März 2004 02:10 schrieben Sie:

> You didn't even say what kernel version you use.  This was fairly

Ooops, it's a vanilla 2.6.4.

> common in early 2.6; I haven't had any reports of it since 2.6.4 or
> so.

OK, here's the first report for a 2.6.4... :-) I had more problems with a 
2.6.3, but the problem remains... Is it possible to increase the nocached 
part of the memory?

regards,
Jörg

Re: srmmu: out of nocache

[Keith M Wesolowski]

On Mon, Mar 29, 2004 at 08:25:09AM +0200, Joerg Platte wrote:

> OK, here's the first report for a 2.6.4... :-) I had more problems with a 
> 2.6.3, but the problem remains... Is it possible to increase the nocached 
> part of the memory?

Yes, but I don't recommend it.  You're using only about 2/3 of the
pagetable memory when you run out because of fragmentation.

This patch changes the next-fit allocator to a next/first to possibly
reduce fragmentation.  There is some performance cost associated.
Please try it and report your results.  If it does fail, the log
messages are important.

Thanks.

=== arch/sparc/lib/bitext.c 1.1 vs edited ==--- 1.1/arch/sparc/lib/bitext.c	Wed Apr 30 01:05:41 2003
+++ edited/arch/sparc/lib/bitext.c	Mon Mar 29 01:57:44 2004
@@ -42,7 +42,10 @@
 		BUG();
 
 	spin_lock(&t->lock);
-	offset = t->last_off & ~align1;
+	if (len < t->last_size)
+		offset = t->first_free;
+	else
+		offset = t->last_off & ~align1;
 	count = 0;
 	for (;;) {
 		off_new = find_next_zero_bit(t->map, t->size, offset);
@@ -71,9 +74,14 @@
 			if (i = len) {
 				for (i = 0; i < len; i++)
 					__set_bit(offset + i, t->map);
+				if (offset = t->first_free)
+					t->first_free = find_next_zero_bit
+							(t->map, t->size,
+							 t->first_free + len);
 				if ((t->last_off = offset + len) >= t->size)
 					t->last_off = 0;
 				t->used += len;
+				t->last_size = len;
 				spin_unlock(&t->lock);
 				return offset;
 			}
@@ -96,6 +104,8 @@
 			BUG();
 		__clear_bit(offset + i, t->map);
 	}
+	if (offset < t->first_free)
+		t->first_free = offset;
 	t->used -= len;
 	spin_unlock(&t->lock);
 }
@@ -111,4 +121,6 @@
 	spin_lock_init(&t->lock);
 	t->map = map;
 	t->size = size;
+	t->last_size = 0;
+	t->first_free = 0;
 }
=== include/asm-sparc/bitext.h 1.2 vs edited ==--- 1.2/include/asm-sparc/bitext.h	Thu Oct 30 12:50:10 2003
+++ edited/include/asm-sparc/bitext.h	Mon Mar 29 01:39:07 2004
@@ -15,6 +15,8 @@
 	int size;
 	int used;
 	int last_off;
+	int last_size;
+	int first_free;
 };
 
 extern int bit_map_string_get(struct bit_map *t, int len, int align);


-- 
Keith M Wesolowski

Re: srmmu: out of nocache

[Cajus Pollmeier]

On Montag, 29. März 2004 12:21, Keith M Wesolowski wrote:
> On Mon, Mar 29, 2004 at 08:25:09AM +0200, Joerg Platte wrote:
> 
> > OK, here's the first report for a 2.6.4... :-) I had more problems with a 
> > 2.6.3, but the problem remains... Is it possible to increase the nocached 
> > part of the memory?
> 
> Yes, but I don't recommend it.  You're using only about 2/3 of the
> pagetable memory when you run out because of fragmentation.
> 
> This patch changes the next-fit allocator to a next/first to possibly
> reduce fragmentation.  There is some performance cost associated.
> Please try it and report your results.  If it does fail, the log
> messages are important.

Ok. I've recompiled the kernel with your patch (hell - I guess we've to
make use of cross compiling...) The machine boots cleanly and
everything works fine till this moment. Cronjobs, etc. fire up tomorrow
and we'll inspect the system mails for "out of nocache" messages and
other anomalies.

Thanks,
Cajus

Re: srmmu: out of nocache

[Joerg Platte]

Am Montag, 29. März 2004 12:21 schrieben Sie:

> This patch changes the next-fit allocator to a next/first to possibly
> reduce fragmentation.  There is some performance cost associated.
> Please try it and report your results.  If it does fail, the log
> messages are important.

It works without problems. There were no more srmmu nocache messages in syslog 
and no killed programs.

regards,
Jörg

srmmu: out of nocache

[Joerg Platte]

Hi!

This happens with a 2.6.5er kernel with the following patch applied:

=== arch/sparc/lib/bitext.c 1.1 vs edited ==--- 1.1/arch/sparc/lib/bitext.c Wed Apr 30 01:05:41 2003
+++ edited/arch/sparc/lib/bitext.c      Thu Apr  1 01:57:22 2004
@@ -60,8 +60,8 @@
                }

                if (offset + len > t->size) {
-                       offset = 0;
                        count += t->size - offset;
+                       offset = 0;
                        continue;
                }


Apr 11 15:05:10 ferdi kernel: bitmap out: size 8800 used 6268 off 8016 len 16 
align 16 count 8816
Apr 11 15:05:10 ferdi kernel: srmmu: out of nocache 4096: 2252800/1604608
Apr 11 15:05:14 ferdi kernel: bitmap out: size 8800 used 6305 off 7456 len 16 
align 16 count 8816
Apr 11 15:05:14 ferdi postfix/master[424]: warning: master_spawn: fork: Cannot 
allocate memory -- throttling
Apr 11 15:05:14 ferdi kernel: srmmu: out of nocache 4096: 2252800/1614080
Apr 11 15:05:15 ferdi kernel: bitmap out: size 8800 used 6310 off 2400 len 16 
align 16 count 8816
Apr 11 15:05:15 ferdi kernel: srmmu: out of nocache 4096: 2252800/1615360
Apr 11 15:05:15 ferdi kernel: bitmap out: size 8800 used 6315 off 2432 len 16 
align 16 count 8816
Apr 11 15:05:15 ferdi kernel: srmmu: out of nocache 4096: 2252800/1616640
Apr 11 15:05:15 ferdi spamd[279]: cannot fork: Cannot allocate memory 
Apr 11 15:05:29 ferdi kernel: bitmap out: size 8800 used 6285 off 7632 len 16 
align 16 count 8816

ferdi has 128MB memory... 

regards,
Jörg

Re: srmmu: out of nocache

[Keith M Wesolowski]

On Sun, Apr 11, 2004 at 05:36:06PM +0200, Joerg Platte wrote:

> This happens with a 2.6.5er kernel with the following patch applied:

> Apr 11 15:05:10 ferdi kernel: bitmap out: size 8800 used 6268 off
> 8016 len 16 align 16 count 8816
> Apr 11 15:05:10 ferdi kernel: srmmu: out of nocache 4096: 2252800/1604608

It would appear that the allocation sequence patch is also needed,
then (below).  Please try with both patches while I brainstorm ways to
fix the problem for good.

=== arch/sparc/lib/bitext.c 1.1 vs edited ==--- 1.1/arch/sparc/lib/bitext.c	Wed Apr 30 01:05:41 2003
+++ edited/arch/sparc/lib/bitext.c	Mon Mar 29 01:57:44 2004
@@ -42,7 +42,10 @@
 		BUG();
 
 	spin_lock(&t->lock);
-	offset = t->last_off & ~align1;
+	if (len < t->last_size)
+		offset = t->first_free;
+	else
+		offset = t->last_off & ~align1;
 	count = 0;
 	for (;;) {
 		off_new = find_next_zero_bit(t->map, t->size, offset);
@@ -71,9 +74,14 @@
 			if (i = len) {
 				for (i = 0; i < len; i++)
 					__set_bit(offset + i, t->map);
+				if (offset = t->first_free)
+					t->first_free = find_next_zero_bit
+							(t->map, t->size,
+							 t->first_free + len);
 				if ((t->last_off = offset + len) >= t->size)
 					t->last_off = 0;
 				t->used += len;
+				t->last_size = len;
 				spin_unlock(&t->lock);
 				return offset;
 			}
@@ -96,6 +104,8 @@
 			BUG();
 		__clear_bit(offset + i, t->map);
 	}
+	if (offset < t->first_free)
+		t->first_free = offset;
 	t->used -= len;
 	spin_unlock(&t->lock);
 }
@@ -111,4 +121,6 @@
 	spin_lock_init(&t->lock);
 	t->map = map;
 	t->size = size;
+	t->last_size = 0;
+	t->first_free = 0;
 }
=== include/asm-sparc/bitext.h 1.2 vs edited ==--- 1.2/include/asm-sparc/bitext.h	Thu Oct 30 12:50:10 2003
+++ edited/include/asm-sparc/bitext.h	Mon Mar 29 01:39:07 2004
@@ -15,6 +15,8 @@
 	int size;
 	int used;
 	int last_off;
+	int last_size;
+	int first_free;
 };
 
 extern int bit_map_string_get(struct bit_map *t, int len, int align);


-- 
Keith M Wesolowski