Bonnie getc reading hitting exact 65536 kb/s

Bonnie getc reading hitting exact 65536 kb/s

[A. Liemen]

Hi,

during some xfs tuning tests I discovered the following phenomenon. 
Maybe someone can explain it to me:

Bonnie++ Reading with getc()... hits exactly an almost constant
65536 kbyte/s with only ~15% IO usage.

iostat and bonnie output can be found under:

http://alexander.liemen.net/bonnie.txt

sdb is a 14 disk RAID10 (128kb stripe) 2.8Tbyte XFS Volume with an 
external log (128Mb) on sda7 (Raid1)

mount options are:

noatime,nodiratime,logbufs=8,ihashsize=65536,logdev=/dev/sda7

mkfs:

mkfs.xfs -f -d sunit=256,swidth=1792 -l 
version=2,size=128m,logdev=/dev/sda7  /dev/sdb

scheduler: deadline

system:
- debian testing 2.6.15-1-amd64-k8-smp
- dual opteron 275-285 (both tested)
- 16gb ram
- areca pci-x 1160 controller 1gb cache v1.41 firmware
- WD400YR disks

CPU usage is 100% (on one of the cpus) but i don't think that's the 
limiting factor since it hits that 16bit value absolut constantly.

I have also tested it with exactly the same server but faster CPUs and 
it's hitting the same barrier + tested it with no mkfs.xfs options + no 
mount options on a second identical server. Same problem.

Intelligent reading is way above with 350+ Mb/s (check bonnie.txt)


Anyone got an idea why it is hitting that 16bit value / barrier?

Also of interest would be why the random seeks are so bad.


Thanks a lot
Alex

Re: Bonnie getc reading hitting exact 65536 kb/s

[David Chinner]

On Sat, Jul 15, 2006 at 07:13:10PM +0200, A. Liemen wrote:
> Hi,
> 
> during some xfs tuning tests I discovered the following phenomenon. 
> Maybe someone can explain it to me:
> 
> Bonnie++ Reading with getc()... hits exactly an almost constant
> 65536 kbyte/s with only ~15% IO usage.
> 
> iostat and bonnie output can be found under:
> 
> http://alexander.liemen.net/bonnie.txt

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
storage4     40000M 64890  99 229195  39 105826  24 63895  98 350285  38 361.1   0
                    ^^^^^^^^^                       ^^^^^^^^^

It's cpu bound, not I/O bound. It can't go any faster because it's a
single threaded test....

> CPU usage is 100% (on one of the cpus) but i don't think that's the 
> limiting factor since it hits that 16bit value absolut constantly.

What you are seeing is XFS issuing regular sized I/Os at a constant
throughput.

If you really think this is a 16bit ceiling, see what the block I/O tests
report in iostat - you're getting ~230MB/s for write and 350MB/s for read, so
if iostat is has a bug then you'll see it there.

> Also of interest would be why the random seeks are so bad.

Try creating more than 16 files for youre create/read/delete so
that the runtime is somewhat greater than the timer resolution....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group