Disk I/O degraded performance

Disk I/O degraded performance

[Ramon Chimeno]

Hi all

I migrated one of my server from kernel 2.6.18 to the latest 2.6.22
and I experienced lower disk performance for processes that open file
with the O_DIRECT flag.

I did a very simple test program that opens two files with O_DIRECT
flag and reads the files to end. I monitored the time spent to read
the files and I have ~ 40% of difference between 2.6.18 and 2.6.22.

For information, the files are stored on a XFS partition which is part
of a software raid-5 block device (the raid-5 is made with 3 SATA
drives).

Does that make sense to you gurus ? Of course, I didn't change the
configuration (task scheduler, HZ, default I/O scheduler, ...) between
the two kernels.

For information, the code of my test program is pasted below (I
snipped the #include part):
----------------------8<----------------------8<----------------------
#define BSIZE (512*512)
int main(int ac, char * av[])
{
 int   fd[2];
 char* buffer;

 if (ac != 3)  {
   printf("USAGE: %s <FILE-1> <FILE-2>\n", av[0]);
   exit(EXIT_FAILURE);
 }

 if ((fd[0] = open(av[1], O_RDONLY|O_DIRECT)) == -1)
   exit(EXIT_FAILURE);
 if ((fd[1] = open(av[2], O_RDONLY|O_DIRECT)) == -1)
   exit(EXIT_FAILURE);
 if ((buffer = (char *)memalign(4096, BSIZE)) == 0)
   exit(EXIT_FAILURE);

 for (;;)  {
   if (read(fd[0], buffer, BSIZE) != BSIZE)
     break;
   if (read(fd[1], buffer, BSIZE) != BSIZE)
     break;
 }
 close(fd[0]);
 close(fd[1]);
 free(buffer);
 return EXIT_SUCCESS;
}
----------------------8<----------------------8<----------------------

Any help/hint would be appreciate.

-- R.C

Re: Disk I/O degraded performance

[David Chinner]

On Wed, Sep 19, 2007 at 10:09:12PM +0200, Ramon Chimeno wrote:
> Hi all
> 
> I migrated one of my server from kernel 2.6.18 to the latest 2.6.22
> and I experienced lower disk performance for processes that open file
> with the O_DIRECT flag.
> 
> I did a very simple test program that opens two files with O_DIRECT
> flag and reads the files to end. I monitored the time spent to read
> the files and I have ~ 40% of difference between 2.6.18 and 2.6.22.
> 
> For information, the files are stored on a XFS partition which is part
> of a software raid-5 block device (the raid-5 is made with 3 SATA
> drives).

Start by eliminating the filesystem. i.e. run the same test using
different offsets on the raw device (e.g. seek one fd a few gigabytes
further into the disk than the other then start reading).

Also, you might want to check that you are comparing apples to apples;
are the two files you used in each test the same? If not, are
they all unfragmented, in the same AGs (i.e. all on the same area of
disk as there's a 2x speed difference betweem the inner and outer edges),
etc....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group