VM problems still in 2.2.18

VM problems still in 2.2.18

[Mark Symonds]

Hi, 

I upgraded from 2.2.14 to 2.2.16 after the security
bug was discovered.  Ever since, I have two boxes here
that keep falling over.  Box A will randomly lock without 
warning and box B will die and start printing this message 
repeatedly on the screen until I physically hit reset:

VM: do_try_to_free_pages failed for cron...

...or something similar (could be apache etc.)

I tried upgrading to 2.2.17 for a week and the problem 
didn't go away, installed 2.2.18 last night and the problem 
is still there.  Don't know what to do since anything 
> 2.2.14 is very unstable on these machines, and anything
< 2.2.14 has the security bug.

Both machines have lots of these in the logs, just as others
have reported it happens in spurts:

Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for kupdate...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for eggdrop...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for mysqld...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for apache...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for eggdrop...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for klogd...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for eggdrop...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for init...
Dec 13 08:00:12 symonds kernel: VM: do_try_to_free_pages failed for traceroute...

...etc.

Sometimes this will happen 10 minutes after a reboot,
or 45 mins, but neither machine will stay up for more 
than 30 hours or so (usually *much* less).  

Something else I noticed is that the Load average 
is usually around 0.08, but when I let it idle for a 
few mins, just tapping the spacebar in a terminal will 
cause it to stop responding for 10 or so seconds with 
the load average skyrocketing to over 6.  After that the 
system sometimes recovers and starts responding normally, 
other times it will die.

Is there a patch out there that I can apply to 2.2.14
against the security bug?  The machines were very stable
on that kernel.

Anything I can do from userland to help, let me know.

--
Mark

grep me no patterns and I'll tell you no lines.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: VM problems still in 2.2.18

[Alan Cox]

> bug was discovered.  Ever since, I have two boxes here
> that keep falling over.  Box A will randomly lock without 
> warning and box B will die and start printing this message 
> repeatedly on the screen until I physically hit reset:

What are these two boxes doing ?

> Is there a patch out there that I can apply to 2.2.14
> against the security bug?  The machines were very stable
> on that kernel.

Andrea's VM-global patch seems to be a wonder cure for those who have tried
it. Give it a shot and let folks know.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[thunder7]

On Thu, Dec 14, 2000 at 09:57:28AM +0000, Alan Cox wrote:
> > bug was discovered.  Ever since, I have two boxes here
> > that keep falling over.  Box A will randomly lock without 
> > warning and box B will die and start printing this message 
> > repeatedly on the screen until I physically hit reset:
> 
> What are these two boxes doing ?
> 
> > Is there a patch out there that I can apply to 2.2.14
> > against the security bug?  The machines were very stable
> > on that kernel.
> 
> Andrea's VM-global patch seems to be a wonder cure for those who have tried
> it. Give it a shot and let folks know.
My experience:

2.2.18pre25 erroneously booted with mem=64M:

slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
a lot of 'trying_to_free..' errors.

2.2.18 + VM-global, booted with mem=32M:

slrnpull --expire on the same spool worked fine.

Good luck,
Jurriaan
-- 
proof by reference to inaccessible literature:
	The author cites a simple corollary of a theorem to be found 
	in a privately circulated memoir of the Slovenian 
	Philological Society, 1883 (second edition).
GNU/Linux 2.2.18 SMP 2x1117 bogomips load av: 0.56 0.15 0.05
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Alan Cox]

> slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> a lot of 'trying_to_free..' errors.
> 
> 2.2.18 + VM-global, booted with mem=32M:
> 
> slrnpull --expire on the same spool worked fine.

I think Andrea just earned his official God status ;)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[J Sloan]

Alan Cox wrote:

> > slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> > a lot of 'trying_to_free..' errors.
> >
> > 2.2.18 + VM-global, booted with mem=32M:
> >
> > slrnpull --expire on the same spool worked fine.
>
> I think Andrea just earned his official God status ;)

So, maybe his divine VM patches will make it into 2.2.19?

jjs

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Alan Cox]

> > I think Andrea just earned his official God status ;)
> So, maybe his divine VM patches will make it into 2.2.19?

The question is merely 'in what form' . I wanted to keep them seperate from
the other large changes in 2.2.18 for obvious reasons.

Andrea - can we have the core VM changes you did without adopting the
change in semaphore semantics for file system locking which will give third 
party fs maintainers headaches and doesnt match 2.4 behaviour either ?

Alan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Andrea Arcangeli]

On Thu, Dec 14, 2000 at 11:17:11PM +0000, Alan Cox wrote:
> Andrea - can we have the core VM changes you did without adopting the
> change in semaphore semantics for file system locking which will give third 
> party fs maintainers headaches and doesnt match 2.4 behaviour either ?

The changes in semaphore semantics are necessary to fix the spurious out of
memory with MAP_SHARED mappings and they came together with the removal of the
always-asynchronous kpiod. While it's certainly possible to remove it I don't
think removing the fix for MAP_SHARED stuff is a good idea.

Basically it's always safe to replace:

	down(&inode->i_sem);
	/* critical section */
	up(&inode->isem);

with the new fs-semaphore:

	fs_down(&inode->i_sem);
	/* critical section */
	fs_up(&inode->i_sem);

While it's always safe to use fs_down() in place of down(), it should be done
only with inodes that can be memory mapped using MAP_SHARED if in the
/* critical section */ there is any kind of not-GFP_ATOMIC or not-GFP_BUFFER
allocation like any kind of user-copy (as in every f_ops->write callback).
Most of those down are handled in the VFS and 99% of the time no changes are
necessary in the lowlevel fs so it should not even generate headaches to third
party fs maintainers (I think infact reiserfs patch didn't need any change).

Note: directory i_sem doesn't need the fs_down since a directory can't be
mapped in memory.

2.4.x doesn't need this per-process fs_locks information to avoid recursing on
the i_sem while flushing MAP_SHARED mappings to disk because the lowlevel fs
is required to implement a_ops->writepage to support MAP_SHARED, and the
page-flush gets synchronized only via the per-page lock (while in 2.2.x we need
the i_sem on the inode to call the non-zero-copy f_ops->write to flush the
MAP_SHARED dirty pages).

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Alan Cox]

> The changes in semaphore semantics are necessary to fix the spurious out of
> memory with MAP_SHARED mappings and they came together with the removal of the
> always-asynchronous kpiod. While it's certainly possible to remove it I don't
> think removing the fix for MAP_SHARED stuff is a good idea.

How hard is it to seperate losing kpiod (optimisation) from the MAP_SHARED 
changes ? I am assuming they are two seperate issues, possibly wrongly

> Basically it's always safe to replace:
> 
> 	down(&inode->i_sem);
> 	/* critical section */
> 	up(&inode->isem);
> 
> with the new fs-semaphore:
> 
> 	fs_down(&inode->i_sem);
> 	/* critical section */
> 	fs_up(&inode->i_sem);

Providing no inode semaphore is upped from a different task , which seems
currently quite a valid legal thing to do (ditto doing the up on completion of
something in bh or irq context)

Alan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Andrea Arcangeli]

On Fri, Dec 15, 2000 at 05:57:18PM +0000, Alan Cox wrote:
> How hard is it to seperate losing kpiod (optimisation) from the MAP_SHARED 
> changes ? I am assuming they are two seperate issues, possibly wrongly

Losing kpiod isn't an optimization ;(. Losing kpiod is the MAP_SHARED bugfix.

The problem was:

o	swap_out
o	wants to flush a MAP_SHARED dirty page to disk
o	so allocate kpiod-struct
o	sumbit the page-flush request to kpiod
o	don't wait I/O completion to avoid deadlocking on the i_sem
o	swap_out returns 1 and memory balancing code so thinks we did progress
	in freeing memory and goes to allocate memory from the freelist
	without waiting I/O completion
o	repeat N times the above

o	in the meantime kpiod has a big queue but it's blocked slowly writing
	those pages to disk
o	while it writes a few pages swap_out floods again the queue
	without waiting and it empties the freelist (task killed)

The problem was the lack of write throttling due the kpiod async-only nature.

> Providing no inode semaphore is upped from a different task , which seems
> currently quite a valid legal thing to do (ditto doing the up on completion of
> something in bh or irq context)

Yes, the same `current' context must run the down/up pair of calls and as you
said it is legal to rely on it on all the places it's used.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Alan Cox]

> o	don't wait I/O completion to avoid deadlocking on the i_sem
> o	swap_out returns 1 and memory balancing code so thinks we did progress
> 	in freeing memory and goes to allocate memory from the freelist
> 	without waiting I/O completion
> o	repeat N times the above

so the actual problem is either - the returning 1 when it is the wrong answer
- or the failure to block somewhere else (where its safe) based on a kpiod
maintained semaphore ?

> Yes, the same `current' context must run the down/up pair of calls and as you
> said it is legal to rely on it on all the places it's used.

I assume thats not an issue to reiserfs ?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Andrea Arcangeli]

On Fri, Dec 15, 2000 at 06:46:32PM +0000, Alan Cox wrote:
> so the actual problem is either - the returning 1 when it is the wrong answer
> - or the failure to block somewhere else (where its safe) based on a kpiod
> maintained semaphore ?

The problem is not to find a safe place where to wait, the problem is to know
"when" we can block. That's the only thing current->fs_locks tells us. Sometime
we simply can't wait because I/O can't start until we return (it would deadlock
regardless we wait on a kpiod semaphore or in down(i_sem) ourselfs).

Once we know "if" we can't wait or not the whole point of kpiod is lost
as kpiod exists only because we didn't know that and so we were not able
to wait.

Now we know when we can block so we can run f_ops->write ourselfs that's also
more efficient in terms of both performance and also memory pressure during
swap of course.

> I assume thats not an issue to reiserfs ?

As said reiserfs AFIK didn't need any change, so only VFS is using
fs_down/fs_up from the point of view of reiserfs.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Alan Cox]

> Now we know when we can block so we can run f_ops->write ourselfs that's also
> more efficient in terms of both performance and also memory pressure during
> swap of course.

Yep

> As said reiserfs AFIK didn't need any change, so only VFS is using
> fs_down/fs_up from the point of view of reiserfs.

Ok. Im not keen on adding fs_down but it does look like the right bandage
(and a nice speed up) for 2.2. 

I hate that kind of bug

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Chris Mason]

On Fri, 15 Dec 2000, Alan Cox wrote:

> > Yes, the same `current' context must run the down/up pair of calls and as you
> > said it is legal to rely on it on all the places it's used.
> 
> I assume thats not an issue to reiserfs ?
> 
I don't think so.  There are two places reiserfs calls down/up.  In our
file->release() callback, and in the ioctl we added for lilo so it can
upack files with tails.

Neither calls copy_from/to_user or does an allocation other than
GFP_BUFFER.  As far as I know that should be safe, but the change is
trivial if we need to make it.

-chris


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Andrea Arcangeli]

On Sat, Dec 16, 2000 at 02:49:40AM -0800, Chris Mason wrote:
> GFP_BUFFER.  As far as I know that should be safe, but the change is

Yes that's ok.

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[J . A . Magallon]

On 2000/12/14 Alan Cox wrote:
> > slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> > a lot of 'trying_to_free..' errors.
> > 
> > 2.2.18 + VM-global, booted with mem=32M:
> > 
> > slrnpull --expire on the same spool worked fine.
> 
> I think Andrea just earned his official God status ;)
> 

How about a 2.2.19-pre1 == 2.2.18-aa2 ?

-- 
Juan Antonio Magallon Lacarta                                 #> cd /pub
mailto:jamagallon@able.es                                     #> more beer

Linux werewolf 2.2.18-aa2 #1 SMP Thu Dec 14 21:22:40 CET 2000 i686

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Mark Symonds]

----- Original Message ----- 
From: "Alan Cox" <alan@lxorguk.ukuu.org.uk>
To: <thunder7@xs4all.nl>
Cc: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 14, 2000 2:38 PM
Subject: Re: [lkml]Re: VM problems still in 2.2.18


> 
> I think Andrea just earned his official God status ;)

Just wanted to quickly report that after applying
Andreas patch, the problem has been fixed here as
well.  I've got two machines here that were previously
unable to stay up more than a day, been pounding on 
them since Friday without any problems.  

Great!  :)

--
Mark

grep me no patterns and I'll tell you no lines.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: VM problems still in 2.2.18

[Mark Symonds]

----- Original Message ----- 
From: "Alan Cox" <alan@lxorguk.ukuu.org.uk>
To: "Mark Symonds" <mark@symonds.net>
Cc: <linux-kernel@vger.kernel.org>
Sent: Thursday, December 14, 2000 1:57 AM
Subject: Re: VM problems still in 2.2.18


Hihi,

>> Box A will randomly lock without 
>> warning and box B will die and start printing this message 
>> repeatedly on the screen until I physically hit reset:
>
>What are these two boxes doing ?

One is a P233 32MB and the other is a P100 24MB.  We do 
run *alot* of different things on it and the load has
grown substantially since the switch from 2.2.14.  For
example one of the web developers did a webmail which uses
php-mysql so we have apache-ssl's and mysqld's sitting
there, then there is the guy who likes jserv, then the 
mailserver (around 2000 a day) etc. etc.

> Andrea's VM-global patch seems to be a wonder cure for 
> those who have tried it. Give it a shot and let folks know.

OK, will do.  Just booted the new patched kernel about 
five minutes ago so should know before the end of the 
weekend if that does it.  

Thanks much.  :)

--
Mark

grep me no patterns and I'll tell you no lines.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

[lkml]Re: VM problems still in 2.2.18

[Ed Tomlinson]

Alan Cox wrote:

> > slrnpull --expire on a news-spool of about 600 Mb in 200,000 files gave
> > a lot of 'trying_to_free..' errors.
> > 
> > 2.2.18 + VM-global, booted with mem=32M:
> > 
> > slrnpull --expire on the same spool worked fine.
> 
> I think Andrea just earned his official God status ;)

I have been using the aa series kernels through out the 18pre series (with 
reiserfs).  They have worked very well.   Suggest you and Andrea talk and 
figure out what else from this series can be put into 19pre.  Believe the 
major changes left in the aa series are bigmem and lvm.  I would love to see 
lvm officially in 2.2...

Luck,

Ed Tomlinson
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: [lkml]Re: VM problems still in 2.2.18

[Alan Cox]

> figure out what else from this series can be put into 19pre.  Believe the 
> major changes left in the aa series are bigmem and lvm.  I would love to see 
> lvm officially in 2.2...

lvm, 4Gb support, raid 0.90... to be honest by the time that sort of stuff
would get integrated (except maybe the 4Gig support) 2.4 will probable be
stable and a better option for such boxes
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/