rewrite very slow

rewrite very slow

[DS]

Hello,

I need some help to tunning my storage.

Very simple test at begin (perl script test.pl):

#!/usr/bin/perl

$time=time();
for ($i=1;$i<100;$i++)
{
open (SUBOR,">$i.txt");
print SUBOR "aaaaaaaaaaaaaaaaaaa\n";
close (SUBOR);
print "WRITE $i. FILE\n";
}
$time2=time();
$rozdiel_casov=$time2-$time;

print "TIME ".$rozdiel_casov." sekund\n";

First run:
file1:/mnt/hosting/test# ./test.pl
WRITE 1. FILE
WRITE 2. FILE
...
WRITE 98. FILE
WRITE 99. FILE
TIME 0 sekund


Every next run (rewrite existing files) is very very very slow:

file1:/mnt/hosting/test# ./test.pl
WRITE 1. FILE
WRITE 2. FILE
...

WRITE 98. FILE
WRITE 99. FILE
TIME 43 sekund


I try it on ext3. Every run is only 1-3seconds.

I tried tunning my storage.
My last mount specification: 

(rw,noatime,nodiratime,nobarrier,osyncisosync,logbsize=256k,logbufs=8,quota)
I tried all possible configuration (barrier, nobarrier, osyncisosync,
logbssize=XX,logbufs=2-8) nothing help me.

file1:/mnt/hosting/test# xfs_info /mnt/hosting/
meta-data=/dev/mapper/vghosting-hosting isize=256    agcount=32,
agsize=7626752 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=244056064,
imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

I test it on 4 other xfs storages and results was similar.


Any ideas?

Thanks.

Dusan

Re: rewrite very slow

[Eric Sandeen]

DS wrote:
> Hello,
> 
> I need some help to tunning my storage.

...
> TIME 43 sekund


What kernel?  when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I
see 2 and 7 seconds for rewrite, respectively.

but granted, on ext3 I get 0 seconds for every run.

Also the difference appears to be O_TRUNC (which the perl script does);
if I code it in c:

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

void main(void)
{
        int i;
        int fd;
        char file[4];

        for (i = 0; i < 100; i++) {
                sprintf(file, "%d.txt", i);
                fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
                write(fd, "aaaaaaaaaaaaaaaaaaa\n");
                close(fd);
        }
}

rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it.  Not
sure about the xfs/ext3 difference... this is probably a side-effect of
flushes xfs put into place on truncate (IIRC?)

-Eric

Re: rewrite very slow

[David Chinner]

On Fri, Feb 15, 2008 at 06:36:06PM -0600, Eric Sandeen wrote:
> DS wrote:
> > Hello,
> > 
> > I need some help to tunning my storage.
> 
> ...
> > TIME 43 sekund
> 
> 
> What kernel?  when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I
> see 2 and 7 seconds for rewrite, respectively.
> 
> but granted, on ext3 I get 0 seconds for every run.
> 
> Also the difference appears to be O_TRUNC (which the perl script does);
> if I code it in c:
> 
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> 
> void main(void)
> {
>         int i;
>         int fd;
>         char file[4];
> 
>         for (i = 0; i < 100; i++) {
>                 sprintf(file, "%d.txt", i);
>                 fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
>                 write(fd, "aaaaaaaaaaaaaaaaaaa\n");
>                 close(fd);
>         }
> }
> 
> rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it.  Not
> sure about the xfs/ext3 difference... this is probably a side-effect of
> flushes xfs put into place on truncate (IIRC?)

Yup - after a truncate we use flush-on-close semantics if the file
is closed before pdflush does writeback. yes, it has a measurable
impact on silly microbenchmarks like this, but nobody even noticed
it when we introduced this code 2-3 years ago...

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: rewrite very slow

[DS]

Hmm, and what cat I do to get it work now?

DS

On Sat, Feb 16, 2008 at 01:46:34PM +1100, David Chinner wrote:
> On Fri, Feb 15, 2008 at 06:36:06PM -0600, Eric Sandeen wrote:
> > DS wrote:
> > > Hello,
> > > 
> > > I need some help to tunning my storage.
> > 
> > ...
> > > TIME 43 sekund
> > 
> > 
> > What kernel?  when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I
> > see 2 and 7 seconds for rewrite, respectively.
> > 
> > but granted, on ext3 I get 0 seconds for every run.
> > 
> > Also the difference appears to be O_TRUNC (which the perl script does);
> > if I code it in c:
> > 
> > #include <sys/types.h>
> > #include <sys/stat.h>
> > #include <fcntl.h>
> > 
> > void main(void)
> > {
> >         int i;
> >         int fd;
> >         char file[4];
> > 
> >         for (i = 0; i < 100; i++) {
> >                 sprintf(file, "%d.txt", i);
> >                 fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
> >                 write(fd, "aaaaaaaaaaaaaaaaaaa\n");
> >                 close(fd);
> >         }
> > }
> > 
> > rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it.  Not
> > sure about the xfs/ext3 difference... this is probably a side-effect of
> > flushes xfs put into place on truncate (IIRC?)
> 
> Yup - after a truncate we use flush-on-close semantics if the file
> is closed before pdflush does writeback. yes, it has a measurable
> impact on silly microbenchmarks like this, but nobody even noticed
> it when we introduced this code 2-3 years ago...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group

Re: rewrite very slow

[DS]

Test configuration:
Linux kernel 2.6.23.1 #1 SMP
2x Intel(R) Xeon(TM) CPU 2.40GHz with HT

iSCSI storage (1TB - 7 sata disks in RAID6, 2GB cache controler)


Yes, your "test" works fine:
file1:/mnt/hosting/test# time ./test

real    0m0.334s
user    0m0.000s
sys     0m0.000s


Is there any way to get it work for perl/php/other scripts/programs?

DS


On Fri, Feb 15, 2008 at 06:36:06PM -0600, Eric Sandeen wrote:
> DS wrote:
> > Hello,
> > 
> > I need some help to tunning my storage.
> 
> ...
> > TIME 43 sekund
> 
> 
> What kernel?  when I test on my 2.6.23.9-85.fc8 and 2.6.22.5 boxes, I
> see 2 and 7 seconds for rewrite, respectively.
> 
> but granted, on ext3 I get 0 seconds for every run.
> 
> Also the difference appears to be O_TRUNC (which the perl script does);
> if I code it in c:
> 
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> 
> void main(void)
> {
>         int i;
>         int fd;
>         char file[4];
> 
>         for (i = 0; i < 100; i++) {
>                 sprintf(file, "%d.txt", i);
>                 fd = open(file, O_CREAT|O_RDWR|O_TRUNC, 0644);
>                 write(fd, "aaaaaaaaaaaaaaaaaaa\n");
>                 close(fd);
>         }
> }
> 
> rewrite is a bit slower w/ O_TRUNC in place, plenty fast w/o it.  Not
> sure about the xfs/ext3 difference... this is probably a side-effect of
> flushes xfs put into place on truncate (IIRC?)
> 
> -Eric
> 
>