2.6.29.1 vs 2.6.27.21 ext4 performance problem

2.6.29.1 vs 2.6.27.21 ext4 performance problem

[Alexander V. Lukyanov]

I have a server that is busy reading and writing several disks with ext4 fs.
After booting 2.6.29.1 kernel I have noticed significant increase in read rate.
I measure the read rate using a script which reads sysfs byte and access counts
and calculates average per second.

Here is the output of the script from 2.6.29.1 kernel:

sda: read-rate=58932   (r=4163)
sdb: read-rate=2570440 (r=40763)
sdc: read-rate=3320403 (r=52756)
sdd: read-rate=3199639 (r=52885)
SUM: read-rate=9149414 (r=150567)

And this is output from 2.6.27.21:

sda: read-rate=32768   (r=4096)
sdb: read-rate=204800  (r=24576)
sdc: read-rate=204800  (r=22528)
sdd: read-rate=704512  (r=45056)
SUM: read-rate=1146880 (r=96256)

The rate is in bytes per second, "r" is accesses per second.
One can notice that the access rate is similar (the work load on the server is
basically the same), but byte rate is much higher.

I don't yet know what causes this reading increase. Any ideas?

-- 
   Alexander.

Re: 2.6.29.1 vs 2.6.27.21 ext4 performance problem

[Theodore Tso]

On Fri, Apr 10, 2009 at 07:12:35PM +0400, Alexander V. Lukyanov wrote:
> I have a server that is busy reading and writing several disks with
> ext4 fs.  After booting 2.6.29.1 kernel I have noticed significant
> increase in read rate.  I measure the read rate using a script which
> reads sysfs byte and access counts and calculates average per
> second.

Without knowing exactly what your workload is, it's hard to say for
sure.  One of the things which did go in after 2.6.27 was inode table
readahead.  The intent is to actually improve performance, although at
least initially it might mean a bit more I/O activity.

Does the differences go away if you do echo 0 into
/sys/fs/ext4/<dev>/inode_readahead_blks?

Are you seeing an increase in time, or decrease in transactions per
second, or just that there is is an increase in read rates?

	   	     	      	    	     - Ted

Re: 2.6.29.1 vs 2.6.27.21 ext4 performance problem

[Alexander V. Lukyanov]

On Fri, Apr 10, 2009 at 12:36:20PM -0400, Theodore Tso wrote:
> Does the differences go away if you do echo 0 into
> /sys/fs/ext4/<dev>/inode_readahead_blks?

Yes! Thanks for the tip. I think the problem here is that there
are lots of files (2M per file system) and the access is random.

> Are you seeing an increase in time, or decrease in transactions per
> second, or just that there is is an increase in read rates?

Load average goes up from 6 to 30, and request latency increases.

-- 
   Alexander..