Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Robert Macaulay]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 634 bytes --]

On Wed, 26 Sep 2001, Andrea Arcangeli wrote:

>
> the best/first info in this case would be sysrq+T along with the
> system.map.
>
> You may want to give a spin also to the patch in the attached email.
>
> thanks,
> Andrea
>
>

Sent once more. This time without the huge attachment. Hope this one makes 
it through.

The system.map file is available through 
http://lists.us.dell.com/System

I've tried it again with the 2.4.10 kernel with vmtweaks2, capturing the
information you wanted. I've also tried with the patch you sent(both
chunks). It didn't seem to freeze once I applied them. I'll keep pounding
it for a while.

Robert



[-- Attachment #2: Type: TEXT/PLAIN, Size: 8425 bytes --]

SysRq : Show State

                         free                        sibling
  task             PC    stack   pid father child younger older
init          D F7108AA0     0     1      0   754       3       (NOTLB)
Call Trace: [<c0139df6>] [<c013d4bb>] [<c013d693>] [<c01ddf79>] [<c0131751>] 
   [<c0131b29>] [<c0131b7e>] [<c013257e>] [<c013284c>] [<c0132950>] [<c01494b3>] 
   [<c01440e9>] [<c0149717>] [<c0149b99>] [<c010710b>] 
keventd       S CC412000     0     2      1            14       (L-TLB)
Call Trace: [<c0125321>] [<c0125200>] [<c0105000>] [<c0105806>] [<c0125200>] 
ksoftirqd_CPU S C011CE7B     0     3      0             4     1 (L-TLB)
Call Trace: [<c011ce7b>] [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S F7236800     0     4      0             5     3 (L-TLB)
Call Trace: [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S F7236800     0     5      0             6     4 (L-TLB)
Call Trace: [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S 00000011     0     6      0             7     5 (L-TLB)
Call Trace: [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S C011CE7B     0     7      0             8     6 (L-TLB)
Call Trace: [<c011ce7b>] [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S C011CE7B     0     8      0             9     7 (L-TLB)
Call Trace: [<c011ce7b>] [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S F6AEEE60     0     9      0            10     8 (L-TLB)
Call Trace: [<c011d162>] [<c0105806>] [<c011d0b0>] 
ksoftirqd_CPU S F7236800     0    10      0            11     9 (L-TLB)
Call Trace: [<c011d162>] [<c0105806>] [<c011d0b0>] 
kswapd        D D5E97A00     0    11      0            12    10 (L-TLB)
Call Trace: [<c0139df6>] [<c01e2732>] [<c013d4bb>] [<c013d693>] [<c0131751>] 
   [<c0131b29>] [<c0131b7e>] [<c0131c61>] [<c0131cd6>] [<c0131e31>] [<c0131d90>] 
   [<c0105000>] [<c0105806>] [<c0131d90>] 
bdflush       S 00000008     0    12      0            13    11 (L-TLB)
Call Trace: [<c011615a>] [<c013da4a>] [<c0105000>] [<c0105806>] [<c013d970>] 
kupdated      D D65B0EC0     0    13      0                  12 (L-TLB)
Call Trace: [<c0205205>] [<c013b664>] [<c0139df6>] [<c01df5f4>] [<c013d4bb>] 
   [<c013d693>] [<c0131751>] [<c0131b29>] [<c0131b7e>] [<c013257e>] [<c013284c>] 
   [<c0137160>] [<c01372dc>] [<c01859d6>] [<c01f317a>] [<c01860ba>] [<c018610d>] 
   [<c0139ea4>] [<c0139f57>] [<c013d84e>] [<c013dba1>] [<c0105000>] [<c0105806>] 
   [<c013da50>] 
mdrecoveryd   S 00000010     0    14      1           349     2 (L-TLB)
Call Trace: [<c01d3927>] [<c0105806>] [<c01d3810>] 
syslogd       S F71D3F54     0   349      1           354    14 (NOTLB)
Call Trace: [<c0132950>] [<c0115677>] [<c01db54f>] [<c01497ef>] [<c0149b99>] 
   [<c015f70c>] [<c013a3c3>] [<c010710b>] 
klogd         R 00000000     0   354      1           368   349 (NOTLB)
Call Trace: [<c011866d>] [<c0138e86>] [<c011321c>] [<c010710b>] 
portmap       S C01E0BC7     0   368      1           383   354 (NOTLB)
Call Trace: [<c01e0bc7>] [<c0115677>] [<c0149d39>] [<c0149de6>] [<c0149e1e>] 
   [<c014a092>] [<c01dc88c>] [<c010710b>] 
rpc.statd     S 00000246    16   383      1           472   368 (NOTLB)
Call Trace: [<c0115677>] [<c0214ee3>] [<c01db54f>] [<c01497ef>] [<c0149b99>] 
   [<c010710b>] 
ntpd          S F719BF54     0   472      1           520   383 (NOTLB)
Call Trace: [<c0132950>] [<c0115677>] [<c0106fd3>] [<c01db54f>] [<c01497ef>] 
   [<c0149b99>] [<c010710b>] 
automount     S 00000000     0   520      1           535   472 (NOTLB)
Call Trace: [<c0143b8c>] [<c0143c74>] [<c0138e86>] [<c0120f02>] [<c010710b>] 
atd           S CC598FA0     0   535      1           547   520 (NOTLB)
Call Trace: [<c014ce7c>] [<c01156de>] [<c0115600>] [<c01389cd>] [<c01210bb>] 
   [<c010710b>] 
sshd          S 00000246     0   547      1   837     654   535 (NOTLB)
Call Trace: [<c0115677>] [<c01db54f>] [<c01497ef>] [<c0149b99>] [<c010710b>] 
xinetd        S 00000246     0   654      1           674   547 (NOTLB)
Call Trace: [<c0115677>] [<c01300db>] [<c01db54f>] [<c01497ef>] [<c0149b99>] 
   [<c010710b>] 
sshd          S 00000246     0   664    547   708     755       (NOTLB)
Call Trace: [<c0115677>] [<c01db54f>] [<c01497ef>] [<c0149b99>] [<c0108bad>] 
   [<c010710b>] 
gpm           S C03140C0  5540   674      1           686   654 (NOTLB)
Call Trace: [<c01156de>] [<c0115600>] [<c01db54f>] [<c01497ef>] [<c0149b99>] 
   [<c010710b>] 
crond         S 00000000     0   686      1           699   674 (NOTLB)
Call Trace: [<c01156de>] [<c0115600>] [<c0142444>] [<c01210bb>] [<c010710b>] 
mingetty      S 00007473     0   699      1           700   686 (NOTLB)
Call Trace: [<c0115677>] [<c0177829>] [<c016f23a>] [<c016b2ff>] [<c0129022>] 
   [<c0138e86>] [<c0129063>] [<c010710b>] 
mingetty      S 00000011  4980   700      1           701   699 (NOTLB)
Call Trace: [<c0115677>] [<c0128294>] [<c016f23a>] [<c016b2ff>] [<c0129022>] 
   [<c0138e86>] [<c0129063>] [<c010710b>] 
mingetty      S 00000011  4980   701      1           702   700 (NOTLB)
Call Trace: [<c0115677>] [<c0128294>] [<c016f23a>] [<c016b2ff>] [<c0129022>] 
   [<c0138e86>] [<c0129063>] [<c010710b>] 
mingetty      S 00007473  4980   702      1           703   701 (NOTLB)
Call Trace: [<c0115677>] [<c0128294>] [<c016f23a>] [<c016b2ff>] [<c0129022>] 
   [<c0138e86>] [<c0129063>] [<c010710b>] 
mingetty      S 00007473     0   703      1           704   702 (NOTLB)
Call Trace: [<c0115677>] [<c0128294>] [<c016f23a>] [<c016b2ff>] [<c0129022>] 
   [<c0138e86>] [<c0129063>] [<c010710b>] 
mingetty      S 00007473  4980   704      1           705   703 (NOTLB)
Call Trace: [<c0115677>] [<c0128294>] [<c016f23a>] [<c016b2ff>] [<c0129022>] 
   [<c0138e86>] [<c0129063>] [<c010710b>] 
login         S 00000246     0   705      1   798     754   704 (NOTLB)
Call Trace: [<c011be78>] [<c010710b>] 
bash          S C016CA06     0   708    664   910               (NOTLB)
Call Trace: [<c016ca06>] [<c011be78>] [<c010710b>] 
qla2100_dpc_1 S 00000001     0   754      1                 705 (L-TLB)
Call Trace: [<c0105e54>] [<c0105f2f>] [<f893c6b5>] [<c0105806>] [<f89344b0>] 
sshd          S 00000246     0   755    547   756     837   664 (NOTLB)
Call Trace: [<c0115677>] [<c01db54f>] [<c01497ef>] [<c0149b99>] [<c010710b>] 
bash          S C016CA06     0   756    755   794               (NOTLB)
Call Trace: [<c016ca06>] [<c011be78>] [<c010710b>] 
vmstat        S C016B51C     0   794    756                     (NOTLB)
Call Trace: [<c016b51c>] [<c01156de>] [<c0115600>] [<c0138f82>] [<c01210bb>] 
   [<c010710b>] 
bash          S 6C656E72     0   798    705                     (NOTLB)
Call Trace: [<c0115677>] [<c017e282>] [<c016f23a>] [<c016b2ff>] [<c0122b53>] 
   [<c0138e86>] [<c010710b>] 
sshd          S 00000246     0   837    547   838           755 (NOTLB)
Call Trace: [<c0115677>] [<c01db54f>] [<c01497ef>] [<c0149b99>] [<c0148d62>] 
   [<c010710b>] 
bash          S C016CA06     0   838    837   918               (NOTLB)
Call Trace: [<c016ca06>] [<c011be78>] [<c010710b>] 
ncftp         D ED98E000     0   910    708   917               (NOTLB)
Call Trace: [<c0120ae0>] [<c0139df6>] [<c013d4bb>] [<c013d693>] [<c0105f48>] 
   [<c0131751>] [<c0131b29>] [<c0131b7e>] [<c013257e>] [<c013284c>] [<c0132950>] 
   [<c01494b3>] [<c01f930e>] [<c01db54f>] [<c0149717>] [<c0149b99>] [<c010710b>] 
tar           D C015E0E0  4800   917    910                     (NOTLB)
Call Trace: [<c015e0e0>] [<c0139df6>] [<c0115b59>] [<c013d4bb>] [<c0105dcc>] 
   [<c013d693>] [<c0105f24>] [<c0131751>] [<c0131b29>] [<c0131b7e>] [<c013257e>] 
   [<c013284c>] [<c012cfb7>] [<c0143d99>] [<c0138f56>] [<c0108bad>] [<c010710b>] 
dd            D DEAABD68     0   918    838                     (NOTLB)
Call Trace: [<c0139df6>] [<c013d4bb>] [<c013d693>] [<c0105f48>] [<c0131751>] 
   [<c0131b29>] [<c0131b7e>] [<c013257e>] [<c013284c>] [<c012a1e6>] [<c012a626>] 
   [<c012a8d3>] [<c012af2a>] [<c012adc0>] [<c0138e86>] [<c0108bad>] [<c010710b>] 




SysRq : Show Regs

Pid: 0, comm:              swapper
EIP: 0010:[<c01053fc>] CPU: 1 EFLAGS: 00000246
EAX: 00000000 EBX: c01053d0 ECX: 00000032 EDX: cc40e000
ESI: cc40e000 EDI: cc40e000 EBP: c01053d0 DS: 0018 ES: 0018
CR0: 8005003b CR2: bfffeffc CR3: 0c5cbb20 CR4: 000006f0
Call Trace: [<c0105482>] [<c0118a16>] [<c0118b7b>] 

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Keith Owens]

On Wed, 26 Sep 2001 17:54:52 -0500 (CDT), 
Robert Macaulay <robert_macaulay@dell.com> wrote:

SysRq : Show State

                         free                        sibling
  task             PC    stack   pid father child younger older
init          D F7108AA0     0     1      0   754       3       (NOTLB)
Call Trace: [<c0139df6>] [<c013d4bb>] [<c013d693>] [<c01ddf79>] [<c0131751>] 
   [<c0131b29>] [<c0131b7e>] [<c013257e>] [<c013284c>] [<c0132950>] [<c01494b3>] 
   [<c01440e9>] [<c0149717>] [<c0149b99>] [<c010710b>] 

Run that through ksymoops version >= 2.4.2, it decodes sysrq-T output.

VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Craig Kulesa]

As requested, here are a number of tests of the latest VM patches.  Tests
are described in a previous post, archived here:

    http://www.uwsg.indiana.edu/hypermail/linux/kernel/0109.3/0033.html


Results:

2.4.10 performance is great compared to 2.4.[7-9], but these tests
still seem to point out some room for improvement in the 2.4.10 VM tree.
2.4.10 and 2.4.10(+00_vm-tweaks-1) performed similarly.  The vm-tweaks
patch improved the swap smoothness, but the number of pages swapped out
didn't change measurably, nor did the large number of swap-ins.  Clogging
the system with dirty pages via 'dd' still causes XMMS to skip badly.

Let's push the aging/list-order code more by driving the system a bit
harder in step d), namely adding mozilla to the common user application
test.  We will also stream mp3 audio throughout the entire test.

2.4.10(+00_vm-tweaks-1)
	48 sec StarOffice load time
	28 sec 2560x2560 GIMP image rotation
	82400 KB swapped out, 92148 KB swapped back in

2.4.9-ac14 + aging
	33 sec StarOffice load time
	25 sec GIMP image rotation
	30072 KB swapped out, 22252 KB swapped back in

2.4.9-ac15 + aging + launder
	33 sec StarOffice load time
	24 sec GIMP image rotation
	57556 KB swapped out, 25900 KB swapped back in

'vmstat 1' sessions for these three cases are available at:
	http://loke.as.arizona.edu/~ckulesa/kernel/


2.4.10+ is clearly working a LOT harder to keep dentry and inode caches
in memory, and is swapping out harder to compensate.  The ac14/ac15 tree
frees those caches more freely, and don't page application working sets
out so readily.

Let's test this statement by not pre-filling the inode and dentry caches
with 'slocate' and performing the same test:

2.4.10(+00_vm-tweaks)
	26 sec StarOffice load time
	24 sec GIMP image rotation
	48332 KB swapped out, 33521 KB swapped back in

2.4.9-ac14 + aging
	32 sec StarOffice load time
	26 sec GIMP image rotation
	37392 KB swapped out, 11952 KB swapped back in

2.4.9-ac15 + aging + launder
	32 sec StarOffice load time
	22 second GIMP image rotation
	23884 KB swapped out, 10828 KB swapped back in

2.4.10 does much better this time; in particular the StarOffice loading
that was so plagued by swapouts, pressured by dentry/inode caching last
time, went smoothly.  But there's still more paging than with
2.4.9-ac1[4-5].


Let's try one more aging/list-order experiment.  Instead of creating a
2560x2560 GIMP image first, then loading StarOffice and many other
applications after (to start swapping, and cause GIMP pages to be
candidates for reaping) -- this time let's load StarOffice first and then
create the GIMP image.  This should keep the GIMP image at a 'younger' age
and presumably shouldn't page back into memory (rotation should be
faster).  StarOffice may swap itself entirely out however.

2.4.10(+00_vm-tweaks)
	25 sec StarOffice load time
	29 sec GIMP image rotation
	64427 KB swapped out, 77422 KB swapped back in

2.4.9-ac14 + aging
	30 sec StarOffice load time
	24 sec GIMP image rotation
	22147 KB swapped out, 8922 swapped back in

2.4.9-ac15 + aging + launder
	31 sec StarOffice load time
	21 second GIMP image rotation
	17204 KB swapped out, 8224 swapped back in

The 2.4.10 behavior surprised me.  The GIMP pages are younger in memory,
yet the rotation was slowed by swapin & swapout activity --  slower than
before. Plus more StarOffice pages were swapped out, so it had to be paged
back in order to close the application.  I'm puzzled.  The ac14/ac15
behavior was closer to what I expected; the GIMP pages were young and
unswapped, only the earliest StarOffice pages had to be recalled.

These are samples of rather 'ordinary' loads which 2.4.10 needs some work
handling; the ac15 tree is doing a better job with this particular set
right now (ac15 tree also doesn't skip XMMS with the creation of lots of
dirty pages via 'dd').  But all three kernels tested kept the user
interface relatively responsive, which is an improvement over previous
2.4 releases.  Very cool.

A note on page_launder().  ac14 has the smoothest swapping, with small
chunks laundered at a time.  ac14+aging and ac15+aging+launder both swap
out huge (10-20 MB) chunks at a time.  Admittedly, the user interface is
responsive and XMMS doesn't skip a beat, but most of the 60 MB of
actual swapout in the first test in ac15+stuff came from only THREE
lines of 'vmstat 1' output.  Otherwise there was no swapout activity.

Best regards, and thanks for the excellent work!


Craig Kulesa
Steward Observatory, Univ. of Arizona

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Andrea Arcangeli]

On Wed, Sep 26, 2001 at 06:38:48AM -0700, Craig Kulesa wrote:
> in memory, and is swapping out harder to compensate.  The ac14/ac15 tree

2.4.10 is swapping out more also because I don't keep track of which
pages are just uptodate on the swap space. This will be fixed as soon as
I teach get_swap_page to collect away from the swapcache mapped
exclusive swap pages.

Andrea

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Rik van Riel]

On Wed, 26 Sep 2001, Andrea Arcangeli wrote:
> On Wed, Sep 26, 2001 at 06:38:48AM -0700, Craig Kulesa wrote:
> > in memory, and is swapping out harder to compensate.  The ac14/ac15 tree
>
> 2.4.10 is swapping out more also because I don't keep track of which
> pages are just uptodate on the swap space. This will be fixed as soon
> as I teach get_swap_page to collect away from the swapcache mapped
> exclusive swap pages.

Wouldn't that be easier to do from try_to_swap_out() ?

cheers,

Rik
-- 
IA64: a worthy successor to i860.

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Andrea Arcangeli]

On Wed, Sep 26, 2001 at 11:23:44AM -0300, Rik van Riel wrote:
> On Wed, 26 Sep 2001, Andrea Arcangeli wrote:
> > On Wed, Sep 26, 2001 at 06:38:48AM -0700, Craig Kulesa wrote:
> > > in memory, and is swapping out harder to compensate.  The ac14/ac15 tree
> >
> > 2.4.10 is swapping out more also because I don't keep track of which
> > pages are just uptodate on the swap space. This will be fixed as soon
> > as I teach get_swap_page to collect away from the swapcache mapped
> > exclusive swap pages.
> 
> Wouldn't that be easier to do from try_to_swap_out() ?

Of course that's a possibility but then we'd have to duplicate it in all
other get_swap_page callers, see?

And I think it much better fits hided in get_swap_page: the semantics of
get_swap_page() are "give to the caller a newly allocated swap entry".
So IMHO it is its own business to discard our "optimizations" to
generate a free swap entry in case all swap was just allocated.

Andrea

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Robert Macaulay]

We've tried the 2.4.10 with vmtweaks2 on out machine with 8GB RAM. It was 
looking good for a while, until it just stopped. Here is what was 
happening on the machine.

I was ftping files into the box at a rate of about 8MB/sec. This continued 
until all the RAM was in the  cache column. This was earlier in the 
included vmstat output. The I started a dd if=/dev/sde of=/dev/null in a 
new window.

All was looking good until it just stopped. I captured the vmstat below. 
vmstat continued running for about 1 minute, then it died too. What other 
info can I provide?

 2  0  0   4148   3612  36088 7946652   0   0 15488    64 10216 23346   0  
11  88
 2  0  1   4148   6424  36100 7944288   0   0 11526    40 7107 15848   0  
18  82
 1  1  1   4132   5452  36112 7945444   0   0 11642  6208 7531 16983   0  
17  83
 2  1  1   4132   4972  36128 7946100   0   0 14272 11904 10651 24330   0  
13  87
 3  0  0   4132   4480  36144 7946588   0   0 13120  6760 11007 25144   0  
12  88
 0  1  0   4132   5312  36160 7944964   0   0 15616     0 9935 22793   0  
10  89
 0  3  1   4132   2924  36168 7947052   0   0  6727 11010 5049 11226   0  
26  74
 0  2  2   4132   2668  36168 7946396   0   0  1666  8598 2230  4598   0  
11  89
 0  2  2   4132   3776  36168 7946396   0   0     0     0  159     5   0   
0 100
 0  2  2   4132   3768  36168 7946396   0   0     0     0  121     5   0   
0 100
 0  2  2   4132   3760  36168 7946396   0   0     0     0  126     4   0   
0 100
 0  2  2   4132   3756  36168 7946396   0   0     0     0  139     4   0   
0 100
 0  2  2   4132   3756  36168 7946396   0   0     0     0  148     5   0   
0 100

Re: VM in 2.4.10(+tweaks) vs. 2.4.9-ac14/15(+stuff)

[Andrea Arcangeli]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

On Wed, Sep 26, 2001 at 01:17:29PM -0500, Robert Macaulay wrote:
> We've tried the 2.4.10 with vmtweaks2 on out machine with 8GB RAM. It was 
> looking good for a while, until it just stopped. Here is what was 
> happening on the machine.
> 
> I was ftping files into the box at a rate of about 8MB/sec. This continued 
> until all the RAM was in the  cache column. This was earlier in the 
> included vmstat output. The I started a dd if=/dev/sde of=/dev/null in a 
> new window.
> 
> All was looking good until it just stopped. I captured the vmstat below. 
> vmstat continued running for about 1 minute, then it died too. What other 
> info can I provide?

the best/first info in this case would be sysrq+T along with the system.map.

You may want to give a spin also to the patch in the attached email.

thanks,
Andrea

[-- Attachment #2: Type: message/rfc822, Size: 6144 bytes --]

From: Andrea Arcangeli <andrea@suse.de>
To: "Oleg A. Yurlov" <kris@spylog.com>
Cc: linux-kernel@vger.kernel.org, Bob Matthews <bmatthews@redhat.com>, Linus Torvalds <torvalds@transmeta.com>, Marcelo Tosatti <marcelo@conectiva.com.br>, Rik van Riel <riel@conectiva.com.br>
Subject: Re: 2.4.10aa1 - 0-order allocation failed.
Date: Wed, 26 Sep 2001 16:45:42 +0200
Message-ID: <20010926164542.I27945@athlon.random>

On Wed, Sep 26, 2001 at 06:07:48PM +0400, Oleg A. Yurlov wrote:
> 
>         Hi, Andrea,
> 
>         We have next problem on our servers:
> 
> Sep 26 11:22:39 sol kernel: __alloc_pages: 0-order allocation failed (gfp=0x20/0)
> Sep 26 11:22:39 sol kernel: f048dd94 e02ab000 00000000 00000020 00000000 00000020 00000020 e298f820 
> Sep 26 11:22:39 sol kernel:        e298f844 00000001 e030a56c e030a6c4 00000020 00000000 e01382be 00000000 
> Sep 26 11:22:39 sol kernel:        e013874a e013488c 00000000 e298f820 00000202 e298f898 00000202 00000246 
> Sep 26 11:22:39 sol kernel: Call Trace: [put_dirty_page+122/132] [flush_old_exec+234/572] [sys_ustat+212/268] [kill_super+232/352] [unix_gc+394/748] 
> Sep 26 11:22:39 sol kernel:    [Unused_offset+27374/99203] [Unused_offset+12842/99203] [call_spurious_interrupt+14521/27705] [Unused_offset+43342/99203] [call_spurious_interrupt+14615/27705] [call_spurious_interrupt+16483/27705] 
> Sep 26 11:22:39 sol kernel:    [Unused_offset+90704/99203] [ipgre_rcv+233/636] [ipgre_rcv+503/636] [fcntl_getlk+327/624] [do_invalid_TSS+43/96] 
> Sep 26 11:22:39 sol kernel: __alloc_pages: 0-order allocation failed (gfp=0x20/0)
> Sep 26 11:22:39 sol kernel: f048ddd4 e02ab000 00000000 00000020 00000000 00000020 00000020 e298f820 
> Sep 26 11:22:39 sol kernel:        e298f844 00000001 e030a56c e030a6c4 00000020 00000000 e01382be 00000000 
> Sep 26 11:22:39 sol kernel:        e013874a e013488c 00000000 e298f820 00000202 e298f898 00000202 00000246 
> Sep 26 11:22:39 sol kernel: Call Trace: [put_dirty_page+122/132] [flush_old_exec+234/572] [sys_ustat+212/268] [kill_super+232/352] [unix_gc+394/748] 
> Sep 26 11:22:39 sol kernel:    [Unused_offset+27374/99203] [call_spurious_interrupt+13905/27705] [call_spurious_interrupt+17048/27705] [Unused_offset+90704/99203] [ipgre_rcv+233/636] [ipgre_rcv+503/636] 
> Sep 26 11:22:39 sol kernel:    [fcntl_getlk+327/624] [do_invalid_TSS+43/96] 

the system.map is wrong but this should be harmless, just a notice (if
you do the reverse lookup to find the address and you resolve the right
symbols we could make sure of that).

For driver writers (since it could be on topic with those GFP_ATOMIC
faliures): as I suggested to the SG folks make sure to never use
GFP_ATOMIC in normal kernel context, if you want lowlatency use GFP_NOIO
instead. GFP_NOIO can schedule (so you must release all the spinlocks
first) but it will never block on I/O so it will provide a small latency
too _but_ it will be able to shrink the clean cache so it is very unlikely
it will fail unless you have lots of dirty or mapped cache in ram.

>         Also, we see next in process status:
> 
> USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
> vz         927  0.0 625.1 43900 4267034752 ? S    08:10   0:00 hits
> vz        1030  0.0 625.1 43900 4267034752 ? S    08:11   0:00 hits
> vz        4561  1.3 625.1 45948 4267034724 ? S    10:48   0:00 hits
> root      4564  0.0  0.0  1460  548 pts/2    S    10:48   0:00 grep hits
> vz        4566  0.0 625.1 45948 4267034724 ? S    10:48   0:00 hits

Ben sent the fix for this one [Linus, you can find it on l-k if you
weren't cc'ed] (was a missing check in the tlb shootdown smp fixes) but
it's only a beauty issue, so really don't worry about it :)

>         After these errors we see some uninterruptable processes (with flag D in
> process  status),  gdb  say  that function "fdatasync" called and no returned...
> Soft reboot not work.
> 
>         Server   has   2  CPUs (Pentium III Katmai), 2Gb RAM, 2Gb swap, Hardware
> RAID (Mylex DAC960PTL1 PCI RAID Controller).
> 
>         Any ideas ?

Yes you have highmem.

Last night I spent one hour on the traces from Bob (btw, many thanks for
the helpful report Bob!) and the first suspect is the recent
GFP_NOHIGHIO logic.

Despite Bob's traces not obviously showing this, I think I can see a
potential problem with writepage with regard to the GFP_NOHIGHIO logic
(I just checked 2.4.9ac15 has the same issue too, see the CAN_DO_FS
definition so this shouldn't been introduced recently).

This should fix it, and please also apply vm-tweaks-2 posted to l-k a
few minutes ago.

--- 2.4.10aa1/mm/vmscan.c	Sun Sep 23 22:16:22 2001
+++ vm/mm/vmscan.c	Wed Sep 26 16:34:30 2001
@@ -392,7 +384,7 @@
 			int (*writepage)(struct page *);
 
 			writepage = page->mapping->a_ops->writepage;
-			if ((gfp_mask & __GFP_FS) && writepage) {
+			if ((gfp_mask & __GFP_FS) && ((gfp_mask & __GFP_HIGHIO) || !PageHighMem(page)) && writepage) {
 				ClearPageDirty(page);
 				page_cache_get(page);
 				spin_unlock(&pagemap_lru_lock);


And if the above patch still doesn't help can you just apply this below
patch to disable the NOHIGHIO logic all together, just to make sure
we're looking in the right place?

--- 2.4.10aa1/mm/highmem.c.~1~	Sun Sep 23 21:11:43 2001
+++ 2.4.10aa1/mm/highmem.c	Wed Sep 26 16:38:34 2001
@@ -328,7 +328,7 @@
 	struct page *page;
 
 repeat_alloc:
-	page = alloc_page(GFP_NOHIGHIO);
+	page = alloc_page(GFP_NOIO);
 	if (page)
 		return page;
 	/*
@@ -366,7 +366,7 @@
 	struct buffer_head *bh;
 
 repeat_alloc:
-	bh = kmem_cache_alloc(bh_cachep, SLAB_NOHIGHIO);
+	bh = kmem_cache_alloc(bh_cachep, SLAB_NOIO);
 	if (bh)
 		return bh;
 	/*

Of course also make sure that a SYSRQ+e or SYSRQ+i doesn't relieve the
machine and allows to kill the D tasks :).

thanks!

Andrea