weird throughput on write to SATA disk

weird throughput on write to SATA disk

[kenneth johansson]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

When writing data to /dev/sdb (that is the whole disk and no filesystem)
it starts out ok around 100MB/sec but soon end up doing considerably
worse. For prolonged times lasting several minutes it hovers in the
20-30MB/sec. 

This happens on two different SATA disk one samsung and one seagate and
on both 2.6.26 and 2.6.27 kernel. The disk are connected to the external
esata connector. Controller is 00:1f.2 SATA controller: Intel
Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)

What could be the reason for this ?? reading is no problem.



[-- Attachment #2: log.png --]
[-- Type: image/png, Size: 5365 bytes --]

Re: weird throughput on write to SATA disk

[Grant Grundler]

On Mon, Oct 13, 2008 at 12:19 PM, kenneth johansson <ken@kenjo.org> wrote:
> When writing data to /dev/sdb (that is the whole disk and no filesystem)
> it starts out ok around 100MB/sec but soon end up doing considerably
> worse. For prolonged times lasting several minutes it hovers in the
> 20-30MB/sec.

Is the application using "O_DIRECT"?
If not, could be issue with how VM writeback is (not) working.

Can you reproduce this using fio or even "dd"?
(See git://git.kernel.dk/fio)

grant

>
> This happens on two different SATA disk one samsung and one seagate and
> on both 2.6.26 and 2.6.27 kernel. The disk are connected to the external
> esata connector. Controller is 00:1f.2 SATA controller: Intel
> Corporation 82801HB (ICH8) 4 port SATA AHCI Controller (rev 02)
>
> What could be the reason for this ?? reading is no problem.
>
>
>

Re: weird throughput on write to SATA disk

[kenneth johansson]

On Mon, 2008-10-13 at 14:07 -0700, Grant Grundler wrote:
> On Mon, Oct 13, 2008 at 12:19 PM, kenneth johansson <ken@kenjo.org> wrote:
> > When writing data to /dev/sdb (that is the whole disk and no filesystem)
> > it starts out ok around 100MB/sec but soon end up doing considerably
> > worse. For prolonged times lasting several minutes it hovers in the
> > 20-30MB/sec.
> 
> Is the application using "O_DIRECT"?

no just open then write no fancy stuff

> If not, could be issue with how VM writeback is (not) working.

the access pattern is the simplest possible. Is there any knobs I could
try to adjust.

> Can you reproduce this using fio or even "dd"?
> (See git://git.kernel.dk/fio)

dd is having the same effect. 

Re: weird throughput on write to SATA disk

[Grant Grundler]

On Mon, Oct 13, 2008 at 2:22 PM, kenneth johansson <ken@kenjo.org> wrote:
> On Mon, 2008-10-13 at 14:07 -0700, Grant Grundler wrote:
>> On Mon, Oct 13, 2008 at 12:19 PM, kenneth johansson <ken@kenjo.org> wrote:
>> > When writing data to /dev/sdb (that is the whole disk and no filesystem)
>> > it starts out ok around 100MB/sec but soon end up doing considerably
>> > worse. For prolonged times lasting several minutes it hovers in the
>> > 20-30MB/sec.
>>
>> Is the application using "O_DIRECT"?
>
> no just open then write no fancy stuff
>
>> If not, could be issue with how VM writeback is (not) working.
>
> the access pattern is the simplest possible. Is there any knobs I could
> try to adjust.
>
>> Can you reproduce this using fio or even "dd"?
>> (See git://git.kernel.dk/fio)
>
> dd is having the same effect.

Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

Or try with "sgp_dd"? It's part of sg3-utils package.

thanks,
grant

Re: weird throughput on write to SATA disk

[Grant Grundler]

On Mon, Oct 13, 2008 at 2:34 PM, Grant Grundler <grundler@google.com> wrote:
...
> Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

sorry...meant to type "if=/dev/zero"...I don't recall if /dev/null
will work the same.

grant

Re: weird throughput on write to SATA disk

[kenneth johansson]

On Mon, 2008-10-13 at 14:34 -0700, Grant Grundler wrote:
> On Mon, Oct 13, 2008 at 2:22 PM, kenneth johansson <ken@kenjo.org> wrote:
> > On Mon, 2008-10-13 at 14:07 -0700, Grant Grundler wrote:
> >> On Mon, Oct 13, 2008 at 12:19 PM, kenneth johansson <ken@kenjo.org> wrote:
> >> > When writing data to /dev/sdb (that is the whole disk and no filesystem)
> >> > it starts out ok around 100MB/sec but soon end up doing considerably
> >> > worse. For prolonged times lasting several minutes it hovers in the
> >> > 20-30MB/sec.
> >>
> >> Is the application using "O_DIRECT"?
> >
> > no just open then write no fancy stuff
> >
> >> If not, could be issue with how VM writeback is (not) working.
> >
> > the access pattern is the simplest possible. Is there any knobs I could
> > try to adjust.
> >
> >> Can you reproduce this using fio or even "dd"?
> >> (See git://git.kernel.dk/fio)
> >
> > dd is having the same effect.
> 
> Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?

I changed to use O_DIRECT and it's much more consistent now. 69-75 with
74 about 95% of the time. the disk is supposed to have 105 115 sustained
data rate. 

Re: weird throughput on write to SATA disk

[Grant Grundler]

On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <ken@kenjo.org> wrote:
...
>> Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
>
> I changed to use O_DIRECT and it's much more consistent now. 69-75 with
> 74 about 95% of the time.

That's low but it could be worse. Many things can contribute to slow
disks. Favorites are overtemp (See SMART field 194) and vibration (no
measurement possible w/o special equipment). "dd" isn't exactly a
performance application until one uses really big block sizes (1MB or
larger). sgp_dd is better since it is multi-threaded. fio has all the
right knobs to test disk perf and measure it properly.

> the disk is supposed to have 105 115 sustained
> data rate.

Where did 105-115 number come from?

The best I've seen to date with 7200 rpm drives was 108MB/s on the
outside diameter. That degrades slowly until about 60-70% towards the
inside diameter and then drops off dramatically (down to something
like 50MB/s).

hth,
grant

Re: weird throughput on write to SATA disk

[kenneth johansson]

On Mon, 2008-10-13 at 18:33 -0700, Grant Grundler wrote:
> On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <ken@kenjo.org> wrote:
> ...
> >> Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
> >
> > I changed to use O_DIRECT and it's much more consistent now. 69-75 with
> > 74 about 95% of the time.
> 
> That's low but it could be worse. Many things can contribute to slow
> disks. 
Well my main problem was that it was fluctuating so much that can't be
right. 

> Favorites are overtemp (See SMART field 194) and vibration (no
> measurement possible w/o special equipment). "dd" isn't exactly a
> performance application until one uses really big block sizes (1MB or
> larger).
All my numbers comes from using 1MB blocks. but I'm not using dd.
After removing the logic that actually put some data into the buffers I
do get about 105 MB/sec using O_DIRECT. I'm doing a full disk write now
to get a reference plot. Not using O_DIRECT should be almost identical. 


> > the disk is supposed to have 105 115 sustained
> > data rate.
> 
> Where did 105-115 number come from?
datasheet . they listed two drives in the same column so that was not
the range it was sustained OD and the model I have max out at 105. 

Re: weird throughput on write to SATA disk

[Tejun Heo]

kenneth johansson wrote:
> On Mon, 2008-10-13 at 18:33 -0700, Grant Grundler wrote:
>> On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <ken@kenjo.org> wrote:
>> ...
>>>> Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
>>> I changed to use O_DIRECT and it's much more consistent now. 69-75 with
>>> 74 about 95% of the time.
>> That's low but it could be worse. Many things can contribute to slow
>> disks. 
> Well my main problem was that it was fluctuating so much that can't be
> right. 
> 
>> Favorites are overtemp (See SMART field 194) and vibration (no
>> measurement possible w/o special equipment). "dd" isn't exactly a
>> performance application until one uses really big block sizes (1MB or
>> larger).
> All my numbers comes from using 1MB blocks. but I'm not using dd.
> After removing the logic that actually put some data into the buffers I
> do get about 105 MB/sec using O_DIRECT. I'm doing a full disk write now
> to get a reference plot. Not using O_DIRECT should be almost identical. 
> 
> 
>>> the disk is supposed to have 105 115 sustained
>>> data rate.
>> Where did 105-115 number come from?
> datasheet . they listed two drives in the same column so that was not
> the range it was sustained OD and the model I have max out at 105. 

Can you try deadline scheduler?

-- 
tejun

Re: weird throughput on write to SATA disk

[kenneth johansson]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On Tue, 2008-10-14 at 18:17 +0900, Tejun Heo wrote:
> kenneth johansson wrote:
> > On Mon, 2008-10-13 at 18:33 -0700, Grant Grundler wrote:
> >> On Mon, Oct 13, 2008 at 3:19 PM, kenneth johansson <ken@kenjo.org> wrote:
> >> ...
> >>>> Can you try "dd oflag=direct if=/dev/null of=/dev/sdb bs=64k"?
> >>> I changed to use O_DIRECT and it's much more consistent now. 69-75 with
> >>> 74 about 95% of the time.
> >> That's low but it could be worse. Many things can contribute to slow
> >> disks. 
> > Well my main problem was that it was fluctuating so much that can't be
> > right. 
> Can you try deadline scheduler?
> 

I did but it was so bad I restarted my computer and did it again. Same
result complete disaster. so fas O_DIRECT is the only one that is as it
should be.

Alan mentioned a patch by Arjan van de Ven anyone knows how to find that
one?



[-- Attachment #2: deadline2.png --]
[-- Type: image/png, Size: 5526 bytes --]

[-- Attachment #3: direct.png --]
[-- Type: image/png, Size: 3778 bytes --]

Re: weird throughput on write to SATA disk

[Grant Grundler]

On Tue, Oct 14, 2008 at 2:14 AM, kenneth johansson <ken@kenjo.org> wrote:
...
> All my numbers comes from using 1MB blocks. but I'm not using dd.
> After removing the logic that actually put some data into the buffers I
> do get about 105 MB/sec using O_DIRECT.

Ok. Good.

> I'm doing a full disk write now to get a reference plot.

See http://hdperf.sourceforge.net/  and http://nathan.laredo.name/hdperf/
Simple tool to (relatively) quickly measure the perf across the disk platter.

> Not using O_DIRECT should be almost identical.

*should* - but there are known problems with writeback.
Some of them are described nicely by Dave Chinner as "Random Thought #3":
    http://iou.parisc-linux.org/lsf2008/linux_storage_scalability-Dave_Chinner.odp

Look for "writeback" for a summary of the discussion:
    http://iou.parisc-linux.org/lsf2008/SUMMARY-Storage.txt

hth,
grant

Re: weird throughput on write to SATA disk

[Alan Cox]

On Mon, 13 Oct 2008 14:07:15 -0700
"Grant Grundler" <grundler@google.com> wrote:

> On Mon, Oct 13, 2008 at 12:19 PM, kenneth johansson <ken@kenjo.org> wrote:
> > When writing data to /dev/sdb (that is the whole disk and no filesystem)
> > it starts out ok around 100MB/sec but soon end up doing considerably
> > worse. For prolonged times lasting several minutes it hovers in the
> > 20-30MB/sec.
> 
> Is the application using "O_DIRECT"?
> If not, could be issue with how VM writeback is (not) working.

Or just mashing up the I/O horribly. Would also be worth turning the CFQ
scheduler off or applying Arjan van de Ven's patches to fix the ioprio of
the kernel writeout threads. That last patch makes a huge difference here.

Alan