2.4.10 still slow compared to 2.4.5pre1

2.4.10 still slow compared to 2.4.5pre1

[DICKENS,CARY (HP-Loveland,ex2)]

We have run 2.4.10 under a heavy nfs load and kswapd now appears to be under
control ( never went above 88.5%cpu and then only for a short time), but the
nfs performance is about 45% of what it had been for the 2.4.5pre1 kernel.
The response time grows steadily throughout the test until the test goes
invalid.

Hardware:
4 processors, 4GB ram
45 fibre channel drives, set up in hardware RAID 0/1
2 direct Gigabit Ethernet connections between SPEC SFS prime client and
system under test
reiserfs
all NFS filesystems exported with sync,no_wdelay to insure O_SYNC writes to
storage
NFS v3 UDP

I can provide top logs if anyone would like to see what is happening at any
particular time.  Also, if you would like to see some results from a
particular test, please let me know what test it would be.

We tried the 00_vmtweaks patch from Andrea and it failed to boot.  There was
an issue starting kswapd and the kernel would oops.

Cary Dickens
Hewlett-Packard

Re: 2.4.10 still slow compared to 2.4.5pre1

[Andrew Morton]

"DICKENS,CARY (HP-Loveland,ex2)" wrote:
> 
> We have run 2.4.10 under a heavy nfs load and kswapd now appears to be under
> control ( never went above 88.5%cpu and then only for a short time), but the
> nfs performance is about 45% of what it had been for the 2.4.5pre1 kernel.
> The response time grows steadily throughout the test until the test goes
> invalid.
> 
> Hardware:
> 4 processors, 4GB ram
> 45 fibre channel drives, set up in hardware RAID 0/1
> 2 direct Gigabit Ethernet connections between SPEC SFS prime client and
> system under test
> reiserfs
> all NFS filesystems exported with sync,no_wdelay to insure O_SYNC writes to
> storage
> NFS v3 UDP
> 
> I can provide top logs if anyone would like to see what is happening at any
> particular time.  Also, if you would like to see some results from a
> particular test, please let me know what test it would be.
> 

With a synchronous NFS export, I'd expect the disk throughput
to be lowered to such an extent that VM issues were not
significant in throughput.  But you have been seeing kswapd
problems so hmmm...

Conceivably this is a networking problem, and not an FS/VM
problem.  There were significant changes to the softirq
handling between 2.4.5 and 2.4.10, for example.

Could I suggest that you split these variables apart?  Perform
some comparative FS/VM testing between the kernels, and then
some comparative network testing?

Is it possible to run the SFS clients on the same machine,
over loopback?

Re: 2.4.10 still slow compared to 2.4.5pre1

[Andrea Arcangeli]

On Tue, Sep 25, 2001 at 05:22:41PM -0400, DICKENS,CARY (HP-Loveland,ex2) wrote:
> We tried the 00_vmtweaks patch from Andrea and it failed to boot.  There was
> an issue starting kswapd and the kernel would oops.

You did something wrong then, please try it again.

Andrea

RE: 2.4.10 still slow compared to 2.4.5pre1

[DICKENS,CARY (HP-Loveland,ex2)]

> 
> With a synchronous NFS export, I'd expect the disk throughput
> to be lowered to such an extent that VM issues were not
> significant in throughput.  But you have been seeing kswapd
> problems so hmmm...

We are comparing synchronous to synchronous between 2.4.5pre1 and 2.4.10 so
I wouldn't expect such a difference.
  
> Conceivably this is a networking problem, and not an FS/VM
> problem.  There were significant changes to the softirq
> handling between 2.4.5 and 2.4.10, for example.

I don't understand what the softirq is or how that could effect performance.
If you could point me in a direction to look, I'll check that out.
 
> Could I suggest that you split these variables apart?  Perform
> some comparative FS/VM testing between the kernels, and then
> some comparative network testing?

This was on my list of things to do, but I haven't gotten there yet. ;)
Working on it though.

> Is it possible to run the SFS clients on the same machine,
> over loopback?
>

I don't see me getting to this anytime in the near future.  If it will tell
me what I need to know, I'll add it to my to do list.

Cary 

RE: 2.4.10 still slow compared to 2.4.5pre1

[DICKENS,CARY (HP-Loveland,ex2)]

> -----Original Message-----
> From: Andrea Arcangeli [mailto:andrea@suse.de]
> Sent: Tuesday, September 25, 2001 3:57 PM
> To: DICKENS,CARY (HP-Loveland,ex2)
> Cc: 'linux-kernel@vger.kernel.org'; HABBINGA,ERIK (HP-Loveland,ex1)
> Subject: Re: 2.4.10 still slow compared to 2.4.5pre1
> 
> 
> On Tue, Sep 25, 2001 at 05:22:41PM -0400, DICKENS,CARY 
> (HP-Loveland,ex2) wrote:
> > We tried the 00_vmtweaks patch from Andrea and it failed to 
> boot.  There was
> > an issue starting kswapd and the kernel would oops.
> 
> You did something wrong then, please try it again.
> 
> Andrea
>
Andrea,

I hate to inform you that we tracked this down and nr_inactive_pages can be
zero.  This causes divide by zero in shrink_caches.

This is from the 00_vm-tweaks-1 patch:
 static int shrink_caches(int priority, zone_t * classzone, unsigned int
gfp_mask, int nr_pages)
 {
-	int max_scan = nr_inactive_pages / priority;
+	int max_scan;
+	int chunk_size = nr_pages;
+	unsigned long ratio;
 
 	nr_pages -= kmem_cache_reap(gfp_mask);
 	if (nr_pages <= 0)
 		return 0;
 
-	/* Do we want to age the active list? */
-	if (nr_inactive_pages < nr_active_pages*2)
-		refill_inactive(nr_pages);
+	spin_lock(&pagemap_lru_lock);
+	nr_pages = chunk_size;
+	/* try to keep the active list 2/3 of the size of the cache */
+	ratio = (unsigned long) nr_pages * nr_active_pages /
(nr_inactive_pages * 2);
	
^^^^^^^^^^^^^^^^^^^^^^
+	refill_inactive(ratio);
 
+	max_scan = nr_inactive_pages / priority;
 	nr_pages = shrink_cache(nr_pages, max_scan, classzone, gfp_mask);
 	if (nr_pages <= 0)
 		return 0; 

Hope this helps,
Cary

Re: 2.4.10 still slow compared to 2.4.5pre1

[Andrea Arcangeli]

On Tue, Sep 25, 2001 at 08:44:35PM -0400, DICKENS,CARY (HP-Loveland,ex2) wrote:
> Andrea,
> 
> I hate to inform you that we tracked this down and nr_inactive_pages can be
> zero.  This causes divide by zero in shrink_caches.
> 
> This is from the 00_vm-tweaks-1 patch:
>  static int shrink_caches(int priority, zone_t * classzone, unsigned int
> gfp_mask, int nr_pages)
>  {
> -	int max_scan = nr_inactive_pages / priority;
> +	int max_scan;
> +	int chunk_size = nr_pages;
> +	unsigned long ratio;
>  
>  	nr_pages -= kmem_cache_reap(gfp_mask);
>  	if (nr_pages <= 0)
>  		return 0;
>  
> -	/* Do we want to age the active list? */
> -	if (nr_inactive_pages < nr_active_pages*2)
> -		refill_inactive(nr_pages);
> +	spin_lock(&pagemap_lru_lock);
> +	nr_pages = chunk_size;
> +	/* try to keep the active list 2/3 of the size of the cache */
> +	ratio = (unsigned long) nr_pages * nr_active_pages /
> (nr_inactive_pages * 2);

how can you ever trigger it during boot?

anyways that is a real bug, thanks for spotting it, you can just add 1
to nr_inactive_pages to fix it.

	ratio = (unsigned long) nr_pages * nr_active_pages / ((nr_inactive_pages+1) * 2);

it will be fixed in the next update of course.

Andrea