howto boost write(2) performance?

howto boost write(2) performance?

[Michael Stiller]

Hi list,

i'm developing an application (in C) which needs to write about 
1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. 
It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. 

I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
dedicated thread to write them to the raw disk (no fs). 

The write(2) performance is not good enough, the writer threads take to
much time, and i ask you for ideas, howto to boost the write
performance. 

Maybe mmaping the disk would work? 

Cheers,

-Michael

PS. I would like to be cc'd as i usually don't read the list due to high
traffic.

Re: howto boost write(2) performance?

[Nick Piggin]

On Tuesday 09 October 2007 23:50, Michael Stiller wrote:
> Hi list,
>
> i'm developing an application (in C) which needs to write about
> 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI.
> It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
>
> I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
> dedicated thread to write them to the raw disk (no fs).
>
> The write(2) performance is not good enough, the writer threads take to
> much time, and i ask you for ideas, howto to boost the write
> performance.

The kernel really cannot sustain 125MB/s? I assume the disk
array is capable?

Where is the bottleneck? Does it keep all disks busy, or are
the CPUs overloaded?

Re: howto boost write(2) performance?

[Michael Stiller]

> 
> The kernel really cannot sustain 125MB/s? I assume the disk
> array is capable?

Yes, the array should be capable, LSI controller U320, SATA disks inside
the array. 

> Where is the bottleneck? Does it keep all disks busy, or are
> the CPUs overloaded?

I'm not sure where the bottleneck is. CPU Load goes up with 
many pdflush processes in D state.

Thanks for the suggestions with sg and O_DIRECT so far, 
will try this.

-Michael

Re: howto boost write(2) performance?

[Gustavo Chain]

El Tue, 09 Oct 2007 15:50:17 +0200
Michael Stiller <ms@2scale.net> escribió:

> Hi list,
> 
> i'm developing an application (in C) which needs to write about 
> 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. 
> It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. 
> 
> I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
> dedicated thread to write them to the raw disk (no fs). 
> 
> The write(2) performance is not good enough, the writer threads take
> to much time, and i ask you for ideas, howto to boost the write
> performance. 
> 
> Maybe mmaping the disk would work? 
> 
> Cheers,
> 
> -Michael
> 
> PS. I would like to be cc'd as i usually don't read the list due to
> high traffic.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Create a 1GB ram disk, write data there, and then backup into a hard
disk

-- 
Gustavo Chaín Dumit

Re: howto boost write(2) performance?

[Andi Kleen]

Michael Stiller <ms@2scale.net> writes:
> 
> The write(2) performance is not good enough, the writer threads take to
> much time, and i ask you for ideas, howto to boost the write
> performance. 

You could use an O_DIRECT write if the data is suitably aligned
and your IO sizes are big enough (O_DIRECT is usually a loss
on small IOs). It will also be synchronous, but if you do it 
from a separate thread anyways that should be fine.

-Andi

Re: howto boost write(2) performance?

[Boaz Harrosh]

On Tue, Oct 09 2007 at 16:56 +0200, Andi Kleen <andi@firstfloor.org> wrote:
> Michael Stiller <ms@2scale.net> writes:
>> The write(2) performance is not good enough, the writer threads take to
>> much time, and i ask you for ideas, howto to boost the write
>> performance. 
> 
> You could use an O_DIRECT write if the data is suitably aligned
> and your IO sizes are big enough (O_DIRECT is usually a loss
> on small IOs). It will also be synchronous, but if you do it 
> from a separate thread anyways that should be fine.
> 
> -Andi

If your target is a SCSI target you can gain up to 15% by using
sg. Search on the net for the "sg utils" package. The source code
of sg_dd and others are a grate example of how to do it.

Other wise O_DIRECT is your friend. Also look for asynchronous
I/O so you have a few pending IO buffers to keep the pipes full.

Boaz

Re: howto boost write(2) performance?

[Michael Tokarev]

Boaz Harrosh wrote:
[]
> If your target is a SCSI target you can gain up to 15% by using
> sg. Search on the net for the "sg utils" package. The source code
> of sg_dd and others are a grate example of how to do it.
> 
> Other wise O_DIRECT is your friend. Also look for asynchronous
> I/O so you have a few pending IO buffers to keep the pipes full.

What's the advantage of sg_io over O_DIRECT to the block device?
I think sg devices are being (slowly?) obsoleted since most stuff
which has been in sg now works over normal block device, no?
At least, I haven't seen any significant difference between
sg_dd and dd with oflag=direct.

Thanks.

/mjt

Re: howto boost write(2) performance?

[Michael Stiller]

On Tue, 2007-10-09 at 16:56 +0200, Andi Kleen wrote:
> Michael Stiller <ms@2scale.net> writes:
> > 
> > The write(2) performance is not good enough, the writer threads take to
> > much time, and i ask you for ideas, howto to boost the write
> > performance. 
> 
> You could use an O_DIRECT write if the data is suitably aligned
> and your IO sizes are big enough (O_DIRECT is usually a loss
> on small IOs). It will also be synchronous, but if you do it 
> from a separate thread anyways that should be fine.

Thanks to all who answered and especially to Andi. Using O_DIRECT did
the trick.

Cheers,

Michael

Re: howto boost write(2) performance?

[Bill Davidsen]

Michael Stiller wrote:
> Hi list,
> 
> i'm developing an application (in C) which needs to write about 
> 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. 
> It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. 

People regularly report speeds higher than that on the RAID list, and I 
can get that order of magnitude speed using dd with 1MB buffers to a 
software RAID-0 array or cheap SATA drives. Are you using a decent 
controller? Many "RAID" controllers have bandwidth limitations, buffer 
size issues, etc. I had some chea "SCSI" arrays which were just SCSI 
controllers in from of a bunch of cheap, slow, non-SCSI drives.
> 
> I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
> dedicated thread to write them to the raw disk (no fs).

I would limit the write size to a MB and see if that helps, regardless 
of the buffer size. A circular queue of smaller buffers, like ptbuf, may 
perform better.

> 
> The write(2) performance is not good enough, the writer threads take to
> much time, and i ask you for ideas, howto to boost the write
> performance. 
> 
> Maybe mmaping the disk would work? 

I don't think it would help, I'd really try limiting the size of the 
write() calls first, assuming your hardware is adequate.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot