* Weird performance decrease
@ 2006-11-06 9:28 Sascha Nitsch
2006-11-06 11:05 ` Ruben Rubio
2006-11-06 23:31 ` Shailendra Tripathi
0 siblings, 2 replies; 6+ messages in thread
From: Sascha Nitsch @ 2006-11-06 9:28 UTC (permalink / raw)
To: xfs
Hi,
I'm observing a rather strange behaviour of the filesystem cache algorithm.
I have a server running the following app scenario:
A filesystem tree with a depth of 7 directories and 4 character directory
names.
In the deepest directories are files.
filesize from 100 bytes to 5kb.
Filesystem is XFS.
The app creates dirs in the tree and reads/writes files into the deepest dirs
in the tree.
CPU: Dual Xeon 3.0 Ghz w/HT 512KB cache each, 2GB RAM, SCSI-HDD 15k RPM
The first while, all is fine and extremely fast. After a while the buffer size
is about 3.5 MB
and cache size about 618 MB.
Until that moment ~445000 directories and ~106000 files have been created
Thats where the weird behaviour starts.
The buffer size drops to ~200 kb and cache size starts decreasing fast.
This results in a drastic performace drop in my app.
(avg. read/write times increase from 0.3ms to 4ms)
not a constant increase, a jumping increase. During the next while it
constantly gets slower (19ms and more).
After running a while (with still reducing cache size) the buffer size stays
at
~700kb and cache about 400 MB. Performane is terrible. Way slower than
starting up with no cache.
restarting the app makes no change, neither remounting the partition.
cmd to create the fs:
mkfs.xfs -b size=512 -i maxpct=0 -l version=2 -n size=16k /dev/sdc
mounting with
mount /dev/sdc /data
I'm open for suggestion on mkfs calls, mount options and kernel tuning via
procfs.
I have a testcase to reproduce the problem. It happens after ~45 minutes.
xfs_info /data/
meta-data=/data isize=256 agcount=16, agsize=8960921 blks
= sectsz=512
data = bsize=512 blocks=143374736, imaxpct=0
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=16384
log =internal bsize=512 blocks=65536, version=2
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0
kernel:
a 2.6.9-34.0.2.ELsmp #1 SMP Mon Jul 17 21:41:41 CDT 2006 i686 i686 i386
GNU/Linux
filesystem usage is < 1%
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird performance decrease
2006-11-06 9:28 Weird performance decrease Sascha Nitsch
@ 2006-11-06 11:05 ` Ruben Rubio
2006-11-06 11:31 ` Sascha Nitsch
2006-11-06 23:31 ` Shailendra Tripathi
1 sibling, 1 reply; 6+ messages in thread
From: Ruben Rubio @ 2006-11-06 11:05 UTC (permalink / raw)
To: Sascha Nitsch; +Cc: xfs
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I have seen that there is performance problems when there is some files
in a directory and data is being added in the files.
Are the files franmented?
Go to a directory where the files are listed, and
xfs_bmap -v * | less
Check out the results.
Sascha Nitsch escribió:
> Hi,
>
> I'm observing a rather strange behaviour of the filesystem cache algorithm.
>
> I have a server running the following app scenario:
>
> A filesystem tree with a depth of 7 directories and 4 character directory
> names.
> In the deepest directories are files.
> filesize from 100 bytes to 5kb.
> Filesystem is XFS.
>
> The app creates dirs in the tree and reads/writes files into the deepest dirs
> in the tree.
>
> CPU: Dual Xeon 3.0 Ghz w/HT 512KB cache each, 2GB RAM, SCSI-HDD 15k RPM
>
> The first while, all is fine and extremely fast. After a while the buffer size
> is about 3.5 MB
> and cache size about 618 MB.
> Until that moment ~445000 directories and ~106000 files have been created
>
> Thats where the weird behaviour starts.
>
> The buffer size drops to ~200 kb and cache size starts decreasing fast.
> This results in a drastic performace drop in my app.
> (avg. read/write times increase from 0.3ms to 4ms)
> not a constant increase, a jumping increase. During the next while it
> constantly gets slower (19ms and more).
>
> After running a while (with still reducing cache size) the buffer size stays
> at
> ~700kb and cache about 400 MB. Performane is terrible. Way slower than
> starting up with no cache.
>
> restarting the app makes no change, neither remounting the partition.
>
> cmd to create the fs:
> mkfs.xfs -b size=512 -i maxpct=0 -l version=2 -n size=16k /dev/sdc
> mounting with
> mount /dev/sdc /data
>
> I'm open for suggestion on mkfs calls, mount options and kernel tuning via
> procfs.
> I have a testcase to reproduce the problem. It happens after ~45 minutes.
>
> xfs_info /data/
> meta-data=/data isize=256 agcount=16, agsize=8960921 blks
> = sectsz=512
> data = bsize=512 blocks=143374736, imaxpct=0
> = sunit=0 swidth=0 blks, unwritten=1
> naming =version 2 bsize=16384
> log =internal bsize=512 blocks=65536, version=2
> = sectsz=512 sunit=0 blks
> realtime =none extsz=65536 blocks=0, rtextents=0
>
> kernel:
> a 2.6.9-34.0.2.ELsmp #1 SMP Mon Jul 17 21:41:41 CDT 2006 i686 i686 i386
> GNU/Linux
>
> filesystem usage is < 1%
>
>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFTxbjIo1XmbAXRboRAuFBAKCDFC+FKGmIPEC7m2qPwntgAQO2pgCeJvZ1
fC5bypzpHkU7KMOwtwxObQI=
=mIEc
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird performance decrease
2006-11-06 11:05 ` Ruben Rubio
@ 2006-11-06 11:31 ` Sascha Nitsch
0 siblings, 0 replies; 6+ messages in thread
From: Sascha Nitsch @ 2006-11-06 11:31 UTC (permalink / raw)
To: Ruben Rubio; +Cc: xfs
On Monday 06 November 2006 12:05, you wrote:
> I have seen that there is performance problems when there is some files
> in a directory and data is being added in the files.
>
> Are the files franmented?
> Go to a directory where the files are listed, and
> xfs_bmap -v * | less
>
> Check out the results.
The files themself are not fragmentented. They only get (re)written at once.
No appends.
But a couple of the directories have extends.
Examples:
0354:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..7]: 8964012..8964019 1 (3091..3098) 8
1: [8..15]: 9013089..9013096 1 (52168..52175) 8
00f0:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL
0: [0..7]: 80648721..80648728 9 (432..439) 8
1: [8..15]: 80654561..80654568 9 (6272..6279) 8
2: [16..23]: 80662073..80662080 9 (13784..13791) 8
3: [24..31]: 80669473..80669480 9 (21184..21191) 8
4: [32..39]: 80677185..80677192 9 (28896..28903) 8
5: [40..47]: 80685105..80685112 9 (36816..36823) 8
6: [48..55]: 80692545..80692552 9 (44256..44263) 8
7: [56..63]: 80700001..80700008 9 (51712..51719) 8
8: [64..71]: 80708272..80708279 9 (59983..59990) 8
9: [72..79]: 80716819..80716826 9 (68530..68537) 8
some up to 123 (tested via some checks in random picked directories).
Would increasing the directory size help to avoid those extends?
I'm quite new when it comes to internal stuff of xfs, just used it "as is"
and was happy.
Sascha
> Sascha Nitsch escribió:
> > Hi,
> >
> > I'm observing a rather strange behaviour of the filesystem cache
> > algorithm.
> >
> > I have a server running the following app scenario:
> >
> > A filesystem tree with a depth of 7 directories and 4 character directory
> > names.
> > In the deepest directories are files.
> > filesize from 100 bytes to 5kb.
> > Filesystem is XFS.
> >
> > The app creates dirs in the tree and reads/writes files into the deepest
> > dirs in the tree.
> >
> > CPU: Dual Xeon 3.0 Ghz w/HT 512KB cache each, 2GB RAM, SCSI-HDD 15k RPM
> >
> > The first while, all is fine and extremely fast. After a while the buffer
> > size is about 3.5 MB
> > and cache size about 618 MB.
> > Until that moment ~445000 directories and ~106000 files have been created
> >
> > Thats where the weird behaviour starts.
> >
> > The buffer size drops to ~200 kb and cache size starts decreasing fast.
> > This results in a drastic performace drop in my app.
> > (avg. read/write times increase from 0.3ms to 4ms)
> > not a constant increase, a jumping increase. During the next while it
> > constantly gets slower (19ms and more).
> >
> > After running a while (with still reducing cache size) the buffer size
> > stays at
> > ~700kb and cache about 400 MB. Performane is terrible. Way slower than
> > starting up with no cache.
> >
> > restarting the app makes no change, neither remounting the partition.
> >
> > cmd to create the fs:
> > mkfs.xfs -b size=512 -i maxpct=0 -l version=2 -n size=16k /dev/sdc
> > mounting with
> > mount /dev/sdc /data
> >
> > I'm open for suggestion on mkfs calls, mount options and kernel tuning
> > via procfs.
> > I have a testcase to reproduce the problem. It happens after ~45 minutes.
> >
> > xfs_info /data/
> > meta-data=/data isize=256 agcount=16, agsize=8960921
> > blks = sectsz=512
> > data = bsize=512 blocks=143374736, imaxpct=0
> > = sunit=0 swidth=0 blks, unwritten=1
> > naming =version 2 bsize=16384
> > log =internal bsize=512 blocks=65536, version=2
> > = sectsz=512 sunit=0 blks
> > realtime =none extsz=65536 blocks=0, rtextents=0
> >
> > kernel:
> > a 2.6.9-34.0.2.ELsmp #1 SMP Mon Jul 17 21:41:41 CDT 2006 i686 i686 i386
> > GNU/Linux
> >
> > filesystem usage is < 1%
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird performance decrease
2006-11-06 9:28 Weird performance decrease Sascha Nitsch
2006-11-06 11:05 ` Ruben Rubio
@ 2006-11-06 23:31 ` Shailendra Tripathi
2006-11-07 10:44 ` Sascha Nitsch
1 sibling, 1 reply; 6+ messages in thread
From: Shailendra Tripathi @ 2006-11-06 23:31 UTC (permalink / raw)
To: Sascha Nitsch; +Cc: xfs
Hi Sascha,
Did you notice the iostat -x on the device ? Please
verify the turnaround time of the device when you are getting the slowdown.
For example, if %b is closer towards 100, perhaps you are maxing out on
the disk I/O ops per sec.
Since you have only one disk, once I/O becomes random, the disk wouldn't
be able to do more than 200-250 disk ops per sec.
# iostat -x sda 1
extended device statistics
device mgr/s mgw/s r/s w/s kr/s kw/s size queue wait
svc_t %b
sda 3 7 3.5 18.5 157.0 112.9 12.3 0.2 9.7
1.7 4
Regards,
Shailendra
Sascha Nitsch wrote:
> Hi,
>
> I'm observing a rather strange behaviour of the filesystem cache algorithm.
>
> I have a server running the following app scenario:
>
> A filesystem tree with a depth of 7 directories and 4 character directory
> names.
> In the deepest directories are files.
> filesize from 100 bytes to 5kb.
> Filesystem is XFS.
>
> The app creates dirs in the tree and reads/writes files into the deepest dirs
> in the tree.
>
> CPU: Dual Xeon 3.0 Ghz w/HT 512KB cache each, 2GB RAM, SCSI-HDD 15k RPM
>
> The first while, all is fine and extremely fast. After a while the buffer size
> is about 3.5 MB
> and cache size about 618 MB.
> Until that moment ~445000 directories and ~106000 files have been created
>
> Thats where the weird behaviour starts.
>
> The buffer size drops to ~200 kb and cache size starts decreasing fast.
> This results in a drastic performace drop in my app.
> (avg. read/write times increase from 0.3ms to 4ms)
> not a constant increase, a jumping increase. During the next while it
> constantly gets slower (19ms and more).
>
> After running a while (with still reducing cache size) the buffer size stays
> at
> ~700kb and cache about 400 MB. Performane is terrible. Way slower than
> starting up with no cache.
>
> restarting the app makes no change, neither remounting the partition.
>
> cmd to create the fs:
> mkfs.xfs -b size=512 -i maxpct=0 -l version=2 -n size=16k /dev/sdc
> mounting with
> mount /dev/sdc /data
>
> I'm open for suggestion on mkfs calls, mount options and kernel tuning via
> procfs.
> I have a testcase to reproduce the problem. It happens after ~45 minutes.
>
> xfs_info /data/
> meta-data=/data isize=256 agcount=16, agsize=8960921 blks
> = sectsz=512
> data = bsize=512 blocks=143374736, imaxpct=0
> = sunit=0 swidth=0 blks, unwritten=1
> naming =version 2 bsize=16384
> log =internal bsize=512 blocks=65536, version=2
> = sectsz=512 sunit=0 blks
> realtime =none extsz=65536 blocks=0, rtextents=0
>
> kernel:
> a 2.6.9-34.0.2.ELsmp #1 SMP Mon Jul 17 21:41:41 CDT 2006 i686 i686 i386
> GNU/Linux
>
> filesystem usage is < 1%
>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird performance decrease
2006-11-06 23:31 ` Shailendra Tripathi
@ 2006-11-07 10:44 ` Sascha Nitsch
2006-11-07 18:29 ` Shailendra Tripathi
0 siblings, 1 reply; 6+ messages in thread
From: Sascha Nitsch @ 2006-11-07 10:44 UTC (permalink / raw)
To: Shailendra Tripathi; +Cc: xfs
On Tuesday 07 November 2006 00:31, you wrote:
> Hi Sascha,
> Did you notice the iostat -x on the device ? Please
> verify the turnaround time of the device when you are getting the slowdown.
>
> For example, if %b is closer towards 100, perhaps you are maxing out on
> the disk I/O ops per sec.
> Since you have only one disk, once I/O becomes random, the disk wouldn't
> be able to do more than 200-250 disk ops per sec.
>
> # iostat -x sda 1
> extended device statistics
> device mgr/s mgw/s r/s w/s kr/s kw/s size queue wait
> svc_t %b
> sda 3 7 3.5 18.5 157.0 112.9 12.3 0.2 9.7
> 1.7 4
>
>
> Regards,
> Shailendra
Hi Shailendra,
here are some measurements:
== startup (very high performance) ==
top:
Cpu0 : 0.2% us, 0.0% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu1 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2074432k total, 133784k used, 1940648k free, 15652k buffers
Swap: 2618488k total, 160k used, 2618328k free, 53508k cached
iostat -x
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sdc 0.08 1.77 0.47 2.48 9.37 58.29 4.68 29.14
22.94 0.32 107.23 3.07 0.91
== shortly before performance drops ==
top:
Cpu0 : 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu1 : 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2074432k total, 1342464k used, 731968k free, 17300k buffers
Swap: 2618488k total, 160k used, 2618328k free, 645180k cached
iostat -x
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sdc 0.08 1.96 0.47 2.55 9.35 63.53 4.68 31.77
24.11 0.35 115.27 3.05 0.92
== directly after drop ==
top:
Cpu0 : 0.0% us, 0.6% sy, 0.0% ni, 98.6% id, 0.6% wa, 0.2% hi, 0.0% si
Cpu1 : 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2074432k total, 1355704k used, 718728k free, 532k buffers
Swap: 2618488k total, 160k used, 2618328k free, 656548k cached
iostat -x
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sdc 0.08 1.96 0.47 2.56 9.36 63.87 4.68 31.93
24.16 0.35 115.32 3.05 0.93
notice the buffer size drop
== after running for a while with slow performance ==
top:
Cpu0 : 0.0% us, 1.0% sy, 0.0% ni, 85.0% id, 14.0% wa, 0.0% hi, 0.0% si
Cpu1 : 0.0% us, 0.2% sy, 0.0% ni, 97.0% id, 2.8% wa, 0.0% hi, 0.0% si
Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2074432k total, 1065216k used, 1009216k free, 292k buffers
Swap: 2618488k total, 160k used, 2618328k free, 296888k cached
iostat -x
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
sdc 0.08 1.97 0.50 2.69 10.16 67.99 5.08 34.00
24.52 0.36 111.35 3.10 0.99
without buffers and low cache it's no wonder that the io wait increases.
But why get the buffers and and cache disabled and not rebuild?
A Note: the workload and types of io operations are the same from the first to
the last second, nothing is changing.
what iostat fails to detect ist that on average, there are ~60 reads/s and ~60
writes/s.
Average read time is starting at 30ns/read attempt (on a non-existig file;
put still pretty impressive)
write time (including average creation of 4.3 directories/write) starts
at .3ms and it stays at that speed until the drop.
After that it start to increase to more than 19ms for read ops and 4ms for
write ops.
I'm absolutely running out of possible ideas and workarounds.
Regards,
Sascha
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Weird performance decrease
2006-11-07 10:44 ` Sascha Nitsch
@ 2006-11-07 18:29 ` Shailendra Tripathi
0 siblings, 0 replies; 6+ messages in thread
From: Shailendra Tripathi @ 2006-11-07 18:29 UTC (permalink / raw)
To: Sascha Nitsch; +Cc: xfs
Hi Sascha,
Please run the iostat continuously to monitor the disk
performance. Please monitor the await field and %util field on the
device. Looking at this, it does look like there is some bottleneck in
I/O path (or number of requests generated are too high). Please note
that when you take iostat just once, it gives the average stats on the
device (accumulative). So, you are not getting the real picture.
However, I can see that the average I/O response time is way too high.
(await=115.27). This means that an I/O has been spending average 115 ms
(too bad). It includes the time it spent in the disk I/O queue (called
elevator queue) and actual service time. Your disk is performing good as
it is service time is 3.05 ms This time has less to do with
caching/buffers availability.
Again, it appears to me the number of requests generated are
overwhelming the device. That is. the deivce has seen overwhelming I/O
in recent past.
For example, when I do this:
dd if=/dev/zero of=/tmp/1 bs=32k count=100000
I see this (below). Note that I see await of 218 ms.
$ iostat -x hda6
avg-cpu: %user %nice %sys %iowait %idle
1.75 0.00 0.41 0.21 97.63
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
hda6 0.00 0.77 0.00 0.06 0.00 6.59 0.00 3.30
114.62 0.01 218.10 2.22 0.01
> here are some measurements:
>
> == startup (very high performance) ==
>
> top:
> Cpu0 : 0.2% us, 0.0% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu1 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Mem: 2074432k total, 133784k used, 1940648k free, 15652k buffers
> Swap: 2618488k total, 160k used, 2618328k free, 53508k cached
>
> iostat -x
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc 0.08 1.77 0.47 2.48 9.37 58.29 4.68 29.14
> 22.94 0.32 107.23 3.07 0.91
>
> == shortly before performance drops ==
>
> top:
> Cpu0 : 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu1 : 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Mem: 2074432k total, 1342464k used, 731968k free, 17300k buffers
> Swap: 2618488k total, 160k used, 2618328k free, 645180k cached
>
> iostat -x
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc 0.08 1.96 0.47 2.55 9.35 63.53 4.68 31.77
> 24.11 0.35 115.27 3.05 0.92
>
> == directly after drop ==
> top:
> Cpu0 : 0.0% us, 0.6% sy, 0.0% ni, 98.6% id, 0.6% wa, 0.2% hi, 0.0% si
> Cpu1 : 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Mem: 2074432k total, 1355704k used, 718728k free, 532k buffers
> Swap: 2618488k total, 160k used, 2618328k free, 656548k cached
>
> iostat -x
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc 0.08 1.96 0.47 2.56 9.36 63.87 4.68 31.93
> 24.16 0.35 115.32 3.05 0.93
>
> notice the buffer size drop
>
> == after running for a while with slow performance ==
> top:
> Cpu0 : 0.0% us, 1.0% sy, 0.0% ni, 85.0% id, 14.0% wa, 0.0% hi, 0.0% si
> Cpu1 : 0.0% us, 0.2% sy, 0.0% ni, 97.0% id, 2.8% wa, 0.0% hi, 0.0% si
> Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Cpu3 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si
> Mem: 2074432k total, 1065216k used, 1009216k free, 292k buffers
> Swap: 2618488k total, 160k used, 2618328k free, 296888k cached
>
> iostat -x
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> avgrq-sz avgqu-sz await svctm %util
> sdc 0.08 1.97 0.50 2.69 10.16 67.99 5.08 34.00
> 24.52 0.36 111.35 3.10 0.99
>
> without buffers and low cache it's no wonder that the io wait increases.
> But why get the buffers and and cache disabled and not rebuild?
>
>
> A Note: the workload and types of io operations are the same from the first to
> the last second, nothing is changing.
> what iostat fails to detect ist that on average, there are ~60 reads/s and ~60
> writes/s.
>
> Average read time is starting at 30ns/read attempt (on a non-existig file;
> put still pretty impressive)
> write time (including average creation of 4.3 directories/write) starts
> at .3ms and it stays at that speed until the drop.
>
> After that it start to increase to more than 19ms for read ops and 4ms for
> write ops.
>
> I'm absolutely running out of possible ideas and workarounds.
>
> Regards,
> Sascha
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2006-11-07 18:38 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-11-06 9:28 Weird performance decrease Sascha Nitsch
2006-11-06 11:05 ` Ruben Rubio
2006-11-06 11:31 ` Sascha Nitsch
2006-11-06 23:31 ` Shailendra Tripathi
2006-11-07 10:44 ` Sascha Nitsch
2006-11-07 18:29 ` Shailendra Tripathi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox