RE: 2.4.0-test10 Sluggish After Load

RE: 2.4.0-test10 Sluggish After Load

[Jonathan George]

Matt,

It might be helpful to show the current (post crippled) results of top.
Futhermore, a list of allocated ipc resources (share memory, etc.) and open
files (lsof) would be nice.

--Jonathan--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On Wed, 1 Nov 2000, Jonathan George wrote:

> It might be helpful to show the current (post crippled) results
> of top. Futhermore, a list of allocated ipc resources (share
> memory, etc.) and open files (lsof) would be nice.

The problem may well be that Oracle wants to clean up
all memory at once, accessing much more memory than
it did while under stress with more tricky access
patterns.

The 2.4 VM is basically pretty good when you're not
thrashing and has efficient management of the VM until
your working set reaches the size of physical memory.

But once you hit the thrashing point, the VM falls
flat on its face. This is a nasty surprise to many
people and I am working on (trivial) thrashing control,
but it's not there yet (and not all that important).

If this looks bad to you, compare the points where 2.2
starts thrashing and where 2.4 starts thrashing. You'll
most likely (there must be a few corner cases where 2.2
does better ;)) see that 2.4 still runs fine where 2.2
performance has already "degraded heavily" and that 2.2
has "hit the wall" before 2.4 does so ... the difference
just is that 2.4 hits the wall more suddenly ;)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Christoph Rohland]

Hi Rik,

On Wed, 1 Nov 2000, Rik van Riel wrote:
> The 2.4 VM is basically pretty good when you're not
> thrashing and has efficient management of the VM until
> your working set reaches the size of physical memory.
> 
> But once you hit the thrashing point, the VM falls
> flat on its face. This is a nasty surprise to many
> people and I am working on (trivial) thrashing control,
> but it's not there yet (and not all that important).

I looked into this argument a little bit further: 
In my usual stress tests 12 processes select a random memory object
out of 15 to mmap() or shmat() it and then access it serially. Each
segment is 666000000 bytes and I have 8GB of memory. So at one time
there are at most 666000000*12 bytes = 7.45GB memory attached and in
use. So I do not see that the machine qualifies as thrashing. Of
course the memory pressure is very high all the time since we have to
swap out unused segments.

But the current VM does not behave good at all on that load.

Greetings
		Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On 3 Nov 2000, Christoph Rohland wrote:
> On Wed, 1 Nov 2000, Rik van Riel wrote:
> > The 2.4 VM is basically pretty good when you're not
> > thrashing and has efficient management of the VM until
> > your working set reaches the size of physical memory.
> > 
> > But once you hit the thrashing point, the VM falls
> > flat on its face. This is a nasty surprise to many
> > people and I am working on (trivial) thrashing control,
> > but it's not there yet (and not all that important).
> 
> I looked into this argument a little bit further:  In my usual stress
> tests 12 processes select a random memory object out of 15 to mmap()
> or shmat() it and then access it serially. Each segment is 666000000
> bytes and I have 8GB of memory. So at one time there are at most
> 666000000*12 bytes = 7.45GB memory attached and in use. So I do not
> see that the machine qualifies as thrashing. Of course the memory
> pressure is very high all the time since we have to swap out unused
> segments.
> 
> But the current VM does not behave good at all on that load.

Indeed, shared memory performance still sucks rocks.

I have not had time to fix this and I'm afraid I probably
won't have time to fix this in the near future. I'm willing
to give some advice to people who do want to take on the job
of fixing SHM performance, though..

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Christoph Rohland]

Hi Rik,

Rik van Riel <riel@conectiva.com.br> writes:
> Indeed, shared memory performance still sucks rocks.

No, it's not a performance problem. It is a hard lockup problem on
highmem machines.

I do see two problems here:
1) shm_swap_core does not handle the failure of prepare_higmem_swapout
   right and basically cannot do so. It gets called zone independant
   and should probably get called per zone. At least it has to react:
   If the lowmem zone is full it should not try to swap out highmem
   pages. I have a little workaround for that. Unfortunately this is
   not the whole thing:
2) Apparently the vm does not handle the case, where it has hardly any
   pages in the queues. My machine locks up with the following memory
   numbers:

SysRq: Show Memory
Mem-info:
Free pages:        2560kB (  1028kB HighMem)
( Active: 5, inactive_dirty: 27, inactive_clean: 27, free: 640 (638 1276 1914) )0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB)
7*4kB 14*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1020kB)
3*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1028kB)
Swap cache: add 31231, delete 31218, find 21433/21531
Free swap:       1961028kB
2162688 pages of RAM
1867776 pages of HIGHMEM
102296 reserved pages
1204551 pages shared
13 pages swap cached
0 pages in page table cache
Buffer memory:      100kB
    CLEAN: 2 buffers, 8 kbyte, 2 used (last=2), 0 locked, 0 protected, 0 dirty
   LOCKED: 25 buffers, 100 kbyte, 25 used (last=25), 2 locked, 0
   protected, 0 dirty  

You see: you only have 5+27+27=59 pages under your control...

> I have not had time to fix this and I'm afraid I probably
> won't have time to fix this in the near future. I'm willing
> to give some advice to people who do want to take on the job
> of fixing SHM performance, though..

As you know I started this some time before, but nobody helped me in
integrating it into the rest of the VM :-( 

Unfortunately my time is very limited, but I probably would give it a
second try since I think shm swapping should work. But I think a clean
integration is a 2.5 issue.

Greetings
                Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On 4 Nov 2000, Christoph Rohland wrote:
> Rik van Riel <riel@conectiva.com.br> writes:
> > Indeed, shared memory performance still sucks rocks.
> 
> No, it's not a performance problem. It is a hard lockup problem on
> highmem machines.
> 
> I do see two problems here:
> 1) shm_swap_core does not handle the failure of prepare_higmem_swapout
>    right and basically cannot do so. It gets called zone independant
>    and should probably get called per zone. At least it has to react:

AFAIC try_to_swap_out can handle this situation fine, it
shouldn't be very difficult to get shm_swap to handle it
too...

Unfortunately I don't have a really big memory machine so
I cannot test this stuff :(

> You see: you only have 5+27+27=59 pages under your control...

Ughhhh. Maybe we need some rebalancing there as well.
That's a maximum of 5 pages of executable text mapped
into all processes...

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Christoph Rohland]

Hi Rik,

Rik van Riel <riel@conectiva.com.br> writes:
> On 4 Nov 2000, Christoph Rohland wrote:
> > I do see two problems here:
> > 1) shm_swap_core does not handle the failure of prepare_higmem_swapout
> >    right and basically cannot do so. It gets called zone independant
> >    and should probably get called per zone. At least it has to react:
> 
> AFAIC try_to_swap_out can handle this situation fine, it
> shouldn't be very difficult to get shm_swap to handle it
> too...

No I do not think that try_to_swap_out does handle this. It also
simply fails on this. I have seen lockups after try_to_swap_out
failing on prepare_higmem_swapout.

> > You see: you only have 5+27+27=59 pages under your control...
> 
> Ughhhh. Maybe we need some rebalancing there as well.
> That's a maximum of 5 pages of executable text mapped
> into all processes...

Yes, that's reasonable in my test case...

Greetings
                Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 2.4.0-test10 Sluggish After Load

[Jonathan George]

-----Original Message-----
From: Rik van Riel [mailto:riel@conectiva.com.br]
Sent: Wednesday, November 01, 2000 11:06 AM
To: Jonathan George
Cc: 'matthew@mattshouse.com'; 'linux-kernel@vger.kernel.org'
Subject: RE: 2.4.0-test10 Sluggish After Load


On Wed, 1 Nov 2000, Jonathan George wrote:

> It might be helpful to show the current (post crippled) results
> of top. Futhermore, a list of allocated ipc resources (share
> memory, etc.) and open files (lsof) would be nice.

The problem may well be that Oracle wants to clean up
all memory at once, accessing much more memory than
it did while under stress with more tricky access
patterns.
<SNIP>
If this looks bad to you, compare the points where 2.2
starts thrashing and where 2.4 starts thrashing. You'll
most likely (there must be a few corner cases where 2.2
does better ;)) see that 2.4 still runs fine where 2.2
performance has already "degraded heavily" and that 2.2
has "hit the wall" before 2.4 does so ... the difference
just is that 2.4 hits the wall more suddenly ;)

regards,

Rik
---------------

It sounded to me as if his machine never actually recovered from thrashing.
Futhermore, even a thrashing case on a machine like that shouldn't last for
more than about 10 minutes.  It would be interesting to contrast FreeBSD's
behavior if "simple" cleanup was the problem.  BTW, I think that everyone is
happy with the direction of the new VM.  I'm looking forward to your
upcoming enhancements which I hope will make it in to a later 2.4 release.

--Jonathan--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On Wed, 1 Nov 2000, Jonathan George wrote:

> It sounded to me as if his machine never actually recovered from
> thrashing.

That's the nature of thrashing ... nothing in the system
is able to make any progress, hence it takes ages until
the situation changes...

> Futhermore, even a thrashing case on a machine like that
> shouldn't last for more than about 10 minutes.  It would be
> interesting to contrast FreeBSD's behavior if "simple" cleanup
> was the problem.

FreeBSD has 2 methods of thrashing control. Current Linux 2.4
VM has none. I have been experimenting with some thrashing
control stuff here, but haven't gotten anything clean and
obviously right yet...

> BTW, I think that everyone is happy with the direction of the
> new VM.  I'm looking forward to your upcoming enhancements which
> I hope will make it in to a later 2.4 release.

I'm working on it. I have no idea if it'll be ready in time
for other 2.4 releases ... maybe stuff will be there, maybe
it'll be 2.5 work.

Of course, this also depends on the amount of people willing
to test out new VM patches and/or help with development.

Now that the 2.4 tree is frozen, I'll periodically upload new
patches to my home page for people to test. I don't want to
touch 2.4 except with the most trivial fixes, and accumulate
a big set of more invasive improvements for 2.5.

The URL where I (after I return from .nl in 2 weeks) will put
my VM patches:

	http://www.surriel.com/patches/

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Christoph Rohland]

Hi Rik,

On Wed, 1 Nov 2000, Rik van Riel wrote:
>> BTW, I think that everyone is happy with the direction of the
>> new VM.  I'm looking forward to your upcoming enhancements which
>> I hope will make it in to a later 2.4 release.
> 
> I'm working on it. I have no idea if it'll be ready in time
> for other 2.4 releases ... maybe stuff will be there, maybe
> it'll be 2.5 work.
> 
> Of course, this also depends on the amount of people willing
> to test out new VM patches and/or help with development.

As you know I am doing regular thrash tests and I am willing to do
this further. I would hate to see a customer go down because his
machine becomes unusable. IMHO we should try to fix this during 2.4.

Greetings
		Christoph

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On 2 Nov 2000, Christoph Rohland wrote:
> On Wed, 1 Nov 2000, Rik van Riel wrote:
> >> BTW, I think that everyone is happy with the direction of the
> >> new VM.  I'm looking forward to your upcoming enhancements which
> >> I hope will make it in to a later 2.4 release.
> > 
> > I'm working on it. I have no idea if it'll be ready in time
> > for other 2.4 releases ... maybe stuff will be there, maybe
> > it'll be 2.5 work.
> > 
> > Of course, this also depends on the amount of people willing
> > to test out new VM patches and/or help with development.
> 
> As you know I am doing regular thrash tests and I am willing to do
> this further. I would hate to see a customer go down because his
> machine becomes unusable. IMHO we should try to fix this during 2.4.

Agreed. Also, we don't have to have the thrashing control
be too friendly, as long as it is effective and simple ;)

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[matthew]

On Thu, 2 Nov 2000, Rik van Riel wrote:

> > > Of course, this also depends on the amount of people willing
> > > to test out new VM patches and/or help with development.
> > 
> > As you know I am doing regular thrash tests and I am willing to do
> > this further. I would hate to see a customer go down because his
> > machine becomes unusable. IMHO we should try to fix this during 2.4.
> 
> Agreed. Also, we don't have to have the thrashing control
> be too friendly, as long as it is effective and simple ;)


Here's an update.  I never lost control of the machine, so after about 24
hours I decided to try to fix it without the use of the Big Red
Switch.  There were still > 1000 connections showing, and pstools were
unresponsive, so I did:

ls /proc > killscript
added "kill -9" to the beginning and "\" to the end of each line,
ran it as the database user.  It worked pretty well.  After about 5
minutes it killed all of the oracle processes and the machine appeared to
have returned to normal.  I've since installed Oracle 81610 and everything
looks good.

Matthew


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Peter Samuelson]

[matthew]
> ls /proc > killscript
> added "kill -9" to the beginning and "\" to the end of each line,
> ran it as the database user.  It worked pretty well.

Sounds like a lot of trouble.

  su {oracle} -c 'kill -9 -1'

Or is there some reason that wouldn't have worked in your case?

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 2.4.0-test10 Sluggish After Load

[matthew]

>>Rik van Riel said:
>> The problem may well be that Oracle wants to clean up
>> all memory at once, accessing much more memory than
>> it did while under stress with more tricky access
>> patterns.
>> <SNIP>
>> If this looks bad to you, compare the points where 2.2
>> starts thrashing and where 2.4 starts thrashing. You'll
>> most likely (there must be a few corner cases where 2.2
>> does better ;)) see that 2.4 still runs fine where 2.2
>> performance has already "degraded heavily" and that 2.2
>> has "hit the wall" before 2.4 does so ... the difference
>> just is that 2.4 hits the wall more suddenly ;)
>>
>Jonathan George said: 
> It sounded to me as if his machine never actually recovered from thrashing.
> Futhermore, even a thrashing case on a machine like that shouldn't last for
> more than about 10 minutes.  It would be interesting to contrast FreeBSD's
> behavior if "simple" cleanup was the problem.  BTW, I think that everyone is
> happy with the direction of the new VM.  I'm looking forward to your
> upcoming enhancements which I hope will make it in to a later 2.4 release.


The "thrashing" has been going on for roughly 10 hours now.  Is there a
point at which I can expect it to stop?  The load average is at 441 (down 
from > 700 last night), and the stress program was killed at 1:00AM CST
last night.  This (obviously) isn't an important machine so if you want me
to ride it out I will.

Matthew


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On Wed, 1 Nov 2000, matthew wrote:

> The "thrashing" has been going on for roughly 10 hours now.  Is
> there a point at which I can expect it to stop?  The load
> average is at 441 (down from > 700 last night), and the stress
> program was killed at 1:00AM CST last night.  This (obviously)
> isn't an important machine so if you want me to ride it out I
> will.

Interpolating from those load figures, you'll probably have
to wait for 5 to 10 more hours ;)

It looks like you were testing the machine /very/ close to
the thrashing point and the garbage collection afterwards
pushed it right over the edge.

One possible solution to speed things up would be to send
-SIGSTOP to 90% of the oracle processes and wake them up
again slowly later on...  (this is basically what thrashing
control in an OS does, suspend processes and wake them up
again later)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

2.4.0-test10 Sluggish After Load

[matthew]

I'm evaluating my company's needs for a new database server which means
that I get to beat the hell out of the contenders.  I'm a long time Linux
user, and the founder of the Orasoft Project (Oracle Apps for
Linux).  There, I'm no Linux spring chicken. :-)

I wrote stress testing apps that simulates extreme database load.  I ran
it against 2.4.0-test9 and locked it up solid on two occasions.  I grabbed
2.0.4-test10 and reran.

I was impressed.  The distributed stress tester (DDOS) created 1800 Oracle
connections, and then proceeded to bang on the server by selecting random
connections and executing SELECT statements.

Load average got to 5.48.
Memory was depleted, and 200M remained of the 800M swap.

After being pleased by the results, I killed the stress tester.  That's
when things went to hell.  After the stresser was dead the server's load
average spiked to 795!  I still had a remote console open so I tried to
run some things.  The pstools would hang for about 20 minutes before
spitting out data.  Just hitting ENTER in my ssh session would take 10
minutes to register with the server.

The machine shouldn't be under any kind of load right now, yet it's acting
as though it's getting killed.

I've left it in this condition.  It's still accepting SSH connections,
albeit very slowly.  If anyone want to take a look I'll create an account
and allow SSH.  I'd prefer it to be someone that I know, or someone with
an @redhat||suse||etc e-mail address.

Here's the hardware:
2x 500mhz PIII
256M RAM
100% SCSI
HP Kayak
Kernel 2.4.0-test10

Here's some stats just before I killed the stress tester:

[root@oserv02 /root]# pstree 
init-+-atd
     |-crond
     |-gpm
     |-kflushd
     |-klogd
     |-kreclaimd
     |-kswapd
     |-kupdate
     |-6*[mingetty]
     |-1803*[oracle]
     |-portmap
     |-sendmail
     |-sshd---sshd---bash---pstree
     |-syslogd
     |-tnslsnr
     |-wu.ftpd
     `-xfs

[root@oserv02 /root]# w
  9:40pm  up 26 min,  1 user,  load average: 4.40, 5.48, 4.10
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
root     pts/1    matthome.godfrey  9:29pm  1:38   5.94s  4.57s  w


[root@oserv02 /root]# cat /proc/meminfo 
        total:    used:    free:  shared: buffers:  cached:
Mem:  261881856 259772416  2109440        0   212992 24477696
Swap: 814178304 524578816 289599488
MemTotal:       255744 kB
MemFree:          2060 kB
MemShared:           0 kB
Buffers:           208 kB
Cached:          23904 kB
Active:          15268 kB
Inact_dirty:       180 kB
Inact_clean:      8664 kB
Inact_target:    12036 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       255744 kB
LowFree:          2060 kB
SwapTotal:      795096 kB
SwapFree:       282812 kB

I'm not on the list, so direct your reply to matthew@mattshouse.com.

Matthew


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Sean Hunter]

Pardon my speculations (if I am wrong), but isn't this an oracle question?  

Isn't oracle killing the server by trying to clean up 1800 connections all at
once?  When they're all connected, most of the work is done by one or two
oracle processes, but when you kill your ddos thing, all of the oracle
listeners (of which there is one per connection), steam in and try to clean up.

I thought oracle had an internal connection limit (on our servers it is set to
440 connections), anyways.

Sean

On Wed, Nov 01, 2000 at 09:00:29AM -0600, matthew wrote:
> 
> 
> I'm evaluating my company's needs for a new database server which means
> that I get to beat the hell out of the contenders.  I'm a long time Linux
> user, and the founder of the Orasoft Project (Oracle Apps for
> Linux).  There, I'm no Linux spring chicken. :-)
> 
> I wrote stress testing apps that simulates extreme database load.  I ran
> it against 2.4.0-test9 and locked it up solid on two occasions.  I grabbed
> 2.0.4-test10 and reran.
> 
> I was impressed.  The distributed stress tester (DDOS) created 1800 Oracle
> connections, and then proceeded to bang on the server by selecting random
> connections and executing SELECT statements.
> 
> Load average got to 5.48.
> Memory was depleted, and 200M remained of the 800M swap.
> 
> After being pleased by the results, I killed the stress tester.  That's
> when things went to hell.  After the stresser was dead the server's load
> average spiked to 795!  I still had a remote console open so I tried to
> run some things.  The pstools would hang for about 20 minutes before
> spitting out data.  Just hitting ENTER in my ssh session would take 10
> minutes to register with the server.
> 
> The machine shouldn't be under any kind of load right now, yet it's acting
> as though it's getting killed.
> 
> I've left it in this condition.  It's still accepting SSH connections,
> albeit very slowly.  If anyone want to take a look I'll create an account
> and allow SSH.  I'd prefer it to be someone that I know, or someone with
> an @redhat||suse||etc e-mail address.
> 
> Here's the hardware:
> 2x 500mhz PIII
> 256M RAM
> 100% SCSI
> HP Kayak
> Kernel 2.4.0-test10
> 
> Here's some stats just before I killed the stress tester:
> 
> [root@oserv02 /root]# pstree 
> init-+-atd
>      |-crond
>      |-gpm
>      |-kflushd
>      |-klogd
>      |-kreclaimd
>      |-kswapd
>      |-kupdate
>      |-6*[mingetty]
>      |-1803*[oracle]
>      |-portmap
>      |-sendmail
>      |-sshd---sshd---bash---pstree
>      |-syslogd
>      |-tnslsnr
>      |-wu.ftpd
>      `-xfs
> 
> [root@oserv02 /root]# w
>   9:40pm  up 26 min,  1 user,  load average: 4.40, 5.48, 4.10
> USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
> root     pts/1    matthome.godfrey  9:29pm  1:38   5.94s  4.57s  w
> 
> 
> [root@oserv02 /root]# cat /proc/meminfo 
>         total:    used:    free:  shared: buffers:  cached:
> Mem:  261881856 259772416  2109440        0   212992 24477696
> Swap: 814178304 524578816 289599488
> MemTotal:       255744 kB
> MemFree:          2060 kB
> MemShared:           0 kB
> Buffers:           208 kB
> Cached:          23904 kB
> Active:          15268 kB
> Inact_dirty:       180 kB
> Inact_clean:      8664 kB
> Inact_target:    12036 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:       255744 kB
> LowFree:          2060 kB
> SwapTotal:      795096 kB
> SwapFree:       282812 kB
> 
> I'm not on the list, so direct your reply to matthew@mattshouse.com.
> 
> Matthew
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
> 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[matthew]

On Wed, 1 Nov 2000, Sean Hunter wrote:

> Pardon my speculations (if I am wrong), but isn't this an oracle question?  


It could be.


> Isn't oracle killing the server by trying to clean up 1800 connections all at
> once?  When they're all connected, most of the work is done by one or two
> oracle processes, but when you kill your ddos thing, all of the oracle
> listeners (of which there is one per connection), steam in and try to clean up.


Yes, but the factor that drove me to the list was that it's been > 400
load average for 10 hours now.  Even if Oracle tried to clean up 1800
connections at once, would it take this long?  That's not rhetorical, as
the answer may well be "yes".


> I thought oracle had an internal connection limit (on our servers it is set to
> 440 connections), anyways.


This is set in the init.ora.  I jacked it up to allow > 2000 connections.

Matthew


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Sean Hunter]

On Wed, Nov 01, 2000 at 11:10:46AM -0600, matthew wrote:
> On Wed, 1 Nov 2000, Sean Hunter wrote:
> 
> > Pardon my speculations (if I am wrong), but isn't this an oracle question?  
> 
> 
> It could be.
> 
> 
> > Isn't oracle killing the server by trying to clean up 1800 connections all at
> > once?  When they're all connected, most of the work is done by one or two
> > oracle processes, but when you kill your ddos thing, all of the oracle
> > listeners (of which there is one per connection), steam in and try to clean up.
> 
> 
> Yes, but the factor that drove me to the list was that it's been > 400
> load average for 10 hours now.  Even if Oracle tried to clean up 1800
> connections at once, would it take this long?  That's not rhetorical, as
> the answer may well be "yes".
> 

Yup.  What seems to have happened is that waking up 1800 processes at once has
caused the box to thrash so hard it is taking ages for any one process to get
enough scheduler time to clean itself up and exit.

I guess we may need a thrash preventer that slows things down enough for each
process to get a healthy bite of the cherry.

Sean

> 
> > I thought oracle had an internal connection limit (on our servers it is set to
> > 440 connections), anyways.
> 
> 
> This is set in the init.ora.  I jacked it up to allow > 2000 connections.
> 
> Matthew
> 
> 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Rik van Riel]

On Wed, 1 Nov 2000, Sean Hunter wrote:

> Yup.  What seems to have happened is that waking up 1800
> processes at once has caused the box to thrash so hard it is
> taking ages for any one process to get enough scheduler time to
> clean itself up and exit.
> 
> I guess we may need a thrash preventer that slows things down
> enough for each process to get a healthy bite of the cherry.

That's the idea, yes. The current (highly experimental) code
to do this is very ugly, though, and it doesn't quite work
the way it should either ;)

I hope to have it cleaned up and available as an add-on patch
soon.

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
       -- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/		http://www.surriel.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: 2.4.0-test10 Sluggish After Load

[Alessandro Suardi]

Sean Hunter wrote:
> 
> Pardon my speculations (if I am wrong), but isn't this an oracle question?

Maybe, maybe not... what Oracle version ?

> Isn't oracle killing the server by trying to clean up 1800 connections all at
> once?  When they're all connected, most of the work is done by one or two
> oracle processes, but when you kill your ddos thing, all of the oracle
> listeners (of which there is one per connection), steam in and try to clean up.

PMON does the cleanup, so you may have 1 process pegging your CPU at 100%,
 which is not what you see. How does your stress tester work exactly and
 what happens when you stop it ? (my guess is that shadow processes may
 have gone into a CPU loop if detached from the calling process and yes
 that would be abnormal behavior on Oracle's side).

> I thought oracle had an internal connection limit (on our servers it is set to
> 440 connections), anyways.

If Oracle couldn't manage 1800 connections that would be bad news :)


Thanks & ciao,

--alessandro      <alessandro.suardi@oracle.com> <asuardi@uninetcom.it>

Linux:  kernel 2.2.18p18/2.4.0-t10p7 glibc-2.1.94 gcc-2.95.2 binutils-2.10.0.33
Oracle: Oracle8i 8.1.6.1.0 Enterprise Edition for Linux
motto:  Tell the truth, there's less to remember.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/