Lower HD transfer rate with NCQ enabled?

Lower HD transfer rate with NCQ enabled?

[Paa Paa]

I'm using Linux 2.6.20.4. I noticed that I get lower SATA hard drive 
throughput with 2.6.20.4 than with 2.6.19. The reason was that 2.6.20 
enables NCQ by defauly (queue_depth = 31/32 instead of 0/32). Transfer rate 
was measured using "hdparm -t":

With NCQ (queue_depth == 31): 50MB/s.
Without NCQ (queue_depth == 0): 60MB/s.

20% difference is quite a lot. This is with Intel ICH8R controller and 
Western Digital WD1600YS hard disk in AHCI mode. I also used the next 
command to cat-copy a biggish (540MB) file and time it:

rm temp && sync && time sh -c 'cat quite_big_file > temp && sync'

Here I noticed no differences at all with and without NCQ. The times (real 
time) were basically the same in many successive runs. Around 19s.

Q: What conclusion can I make on "hdparm -t" results or can I make any 
conclusions? Do I really have lower performance with NCQ or not? If I do, is 
this because of my HD or because of kernel?

Best regards,
Paapaa

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Lower HD transfer rate with NCQ enabled?

[Nick Piggin]

Paa Paa wrote:
> I'm using Linux 2.6.20.4. I noticed that I get lower SATA hard drive 
> throughput with 2.6.20.4 than with 2.6.19. The reason was that 2.6.20 
> enables NCQ by defauly (queue_depth = 31/32 instead of 0/32). Transfer 
> rate was measured using "hdparm -t":
> 
> With NCQ (queue_depth == 31): 50MB/s.
> Without NCQ (queue_depth == 0): 60MB/s.
> 
> 20% difference is quite a lot. This is with Intel ICH8R controller and 
> Western Digital WD1600YS hard disk in AHCI mode. I also used the next 
> command to cat-copy a biggish (540MB) file and time it:
> 
> rm temp && sync && time sh -c 'cat quite_big_file > temp && sync'
> 
> Here I noticed no differences at all with and without NCQ. The times 
> (real time) were basically the same in many successive runs. Around 19s.
> 
> Q: What conclusion can I make on "hdparm -t" results or can I make any 
> conclusions? Do I really have lower performance with NCQ or not? If I 
> do, is this because of my HD or because of kernel?

What IO scheduler are you using? If AS or CFQ, could you try with deadline?

-- 
SUSE Labs, Novell Inc.

Re: Lower HD transfer rate with NCQ enabled?

[Paa Paa]

>>Q: What conclusion can I make on "hdparm -t" results or can I make any 
>>conclusions? Do I really have lower performance with NCQ or not? If I do, 
>>is this because of my HD or because of kernel?
>
>What IO scheduler are you using? If AS or CFQ, could you try with deadline?

I was using CFQ. I now tried with Deadline and that doesn't seem to degrade 
the performance at all! With Deadline I got 60MB/s both with and without 
NCQ. This was with "hdparm -t".

So what does this tell us?

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: Lower HD transfer rate with NCQ enabled?

[Nick Piggin]

Paa Paa wrote:
>>> Q: What conclusion can I make on "hdparm -t" results or can I make 
>>> any conclusions? Do I really have lower performance with NCQ or not? 
>>> If I do, is this because of my HD or because of kernel?
>>
>>
>> What IO scheduler are you using? If AS or CFQ, could you try with 
>> deadline?
> 
> 
> I was using CFQ. I now tried with Deadline and that doesn't seem to 
> degrade the performance at all! With Deadline I got 60MB/s both with and 
> without NCQ. This was with "hdparm -t".
> 
> So what does this tell us?

Thanks. I believe CFQ contains some code to keep NCQ depths managable,
which might be causing the problem. I've cc'ed the CFQ author (Jens)
who might be able to give some more ideas.

Thanks for reporting!

-- 
SUSE Labs, Novell Inc.

Re: Lower HD transfer rate with NCQ enabled?

[Bill Davidsen]

Paa Paa wrote:
>>> Q: What conclusion can I make on "hdparm -t" results or can I make 
>>> any conclusions? Do I really have lower performance with NCQ or not? 
>>> If I do, is this because of my HD or because of kernel?
>>
>> What IO scheduler are you using? If AS or CFQ, could you try with 
>> deadline?
> 
> I was using CFQ. I now tried with Deadline and that doesn't seem to 
> degrade the performance at all! With Deadline I got 60MB/s both with and 
> without NCQ. This was with "hdparm -t".
> 
> So what does this tell us?
> 
It suggests that it's time to test with real load and see if deadline 
works well for you in the general case.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Lower HD transfer rate with NCQ enabled?

[Chris Snook]

Paa Paa wrote:
> I'm using Linux 2.6.20.4. I noticed that I get lower SATA hard drive 
> throughput with 2.6.20.4 than with 2.6.19. The reason was that 2.6.20 
> enables NCQ by defauly (queue_depth = 31/32 instead of 0/32). Transfer 
> rate was measured using "hdparm -t":
> 
> With NCQ (queue_depth == 31): 50MB/s.
> Without NCQ (queue_depth == 0): 60MB/s.
> 
> 20% difference is quite a lot. This is with Intel ICH8R controller and 
> Western Digital WD1600YS hard disk in AHCI mode. I also used the next 
> command to cat-copy a biggish (540MB) file and time it:
> 
> rm temp && sync && time sh -c 'cat quite_big_file > temp && sync'
> 
> Here I noticed no differences at all with and without NCQ. The times 
> (real time) were basically the same in many successive runs. Around 19s.
> 
> Q: What conclusion can I make on "hdparm -t" results or can I make any 
> conclusions? Do I really have lower performance with NCQ or not? If I 
> do, is this because of my HD or because of kernel?

hdparm -t is a perfect example of a synthetic benchmark.  NCQ was 
designed to optimize real-world workloads.  The overhead gets hidden 
pretty well when there are multiple requests in flight simultaneously, 
as tends to be the case when you have a user thread reading data while a 
kernel thread is asynchronously flushing the user thread's buffered 
writes.  Given that you're breaking even with one user thread and one 
kernel thread doing I/O, you'll probably get performance improvements 
with higher thread counts.

	-- Chris

Re: Lower HD transfer rate with NCQ enabled?

[Mark Lord]

Chris Snook wrote:
> Paa Paa wrote:
>> I'm using Linux 2.6.20.4. I noticed that I get lower SATA hard drive 
>> throughput with 2.6.20.4 than with 2.6.19. The reason was that 2.6.20 
>> enables NCQ by defauly (queue_depth = 31/32 instead of 0/32). Transfer 
>> rate was measured using "hdparm -t":
>>
>> With NCQ (queue_depth == 31): 50MB/s.
>> Without NCQ (queue_depth == 0): 60MB/s.
>>
>> 20% difference is quite a lot. This is with Intel ICH8R controller and 
>> Western Digital WD1600YS hard disk in AHCI mode. I also used the next 
>> command to cat-copy a biggish (540MB) file and time it:
>>
>> rm temp && sync && time sh -c 'cat quite_big_file > temp && sync'
>>
>> Here I noticed no differences at all with and without NCQ. The times 
>> (real time) were basically the same in many successive runs. Around 19s.
>>
>> Q: What conclusion can I make on "hdparm -t" results or can I make any 
>> conclusions? Do I really have lower performance with NCQ or not? If I 
>> do, is this because of my HD or because of kernel?
> 
> hdparm -t is a perfect example of a synthetic benchmark.  NCQ was 
> designed to optimize real-world workloads. 

No, NCQ was designed to optimize *server* workloads:  lots of *small*,
random I/O's.

But WD drives, in particular the Raptor series, have a firmware "feature"
that disables "drive readahead" whenever NCQ is in use.

So they will perform poorly only any medium/large sequential access
if NCQ is employed.  Which is why the custom MS drivers avoid NCQ when
doing large sequential accesses.  Ours don't do this, yet.

-ml

Re: Lower HD transfer rate with NCQ enabled?

[Phillip Susi]

Mark Lord wrote:
> But WD drives, in particular the Raptor series, have a firmware "feature"
> that disables "drive readahead" whenever NCQ is in use.

Why is this an issue?  Shouldn't the kernel be sending down its own 
readahead requests to keep the disk busy?

Re: Lower HD transfer rate with NCQ enabled?

[Mark Lord]

Phillip Susi wrote:
> Mark Lord wrote:
>> But WD drives, in particular the Raptor series, have a firmware "feature"
>> that disables "drive readahead" whenever NCQ is in use.
> 
> Why is this an issue?  Shouldn't the kernel be sending down its own 
> readahead requests to keep the disk busy?

The drive firmware readahead is inherently *way* more effective than
other forms, and without it, sequential read performance really suffers.
Regardless of how software tries to compensate.

This is mostly a problem with the WD Raptor drive, and some other WD drives.
I have not yet encountered/noticed the problem with other brands.

-ml

Re: Lower HD transfer rate with NCQ enabled?

[Phillip Susi]

Mark Lord wrote:
> The drive firmware readahead is inherently *way* more effective than
> other forms, and without it, sequential read performance really suffers.
> Regardless of how software tries to compensate.

Why?  As the platter spins under the head, the drive can either read or 
ignore the data.  As long as it keeps reading the track, it can't go any 
faster.  Whether in response to its own readahead or to the request 
queue from the host, either way it should read the data.  If its own 
readahead is faster, then either the host must not be properly 
performing readahead and keeping the drive request queue filled with 
sequential requests, or the drive firmware must be broken and decides to 
ignore the data as it passes under the head, and instead go back to pick 
it up on the next rotation.

> This is mostly a problem with the WD Raptor drive, and some other WD 
> drives.
> I have not yet encountered/noticed the problem with other brands.

Sounds like this is a serious bug in the WD firmware.

Re: Lower HD transfer rate with NCQ enabled?

[Mark Lord]

Phillip Susi wrote:
> Mark Lord wrote:
>> This is mostly a problem with the WD Raptor drive, and some other WD 
>> drives.
>> I have not yet encountered/noticed the problem with other brands.
> 
> Sounds like this is a serious bug in the WD firmware.

For personal systems, yes.  For servers, probably not a bug.

Disabling readahead means faster execution queued commands,
since it doesn't have to "linger" and do unwanted read-ahead.
So this bug is a "feature" for random access servers.
And a big nuisance for everything else.

Cheers

Re: Lower HD transfer rate with NCQ enabled?

[Lennart Sorensen]

On Thu, Apr 05, 2007 at 12:11:57PM -0400, Mark Lord wrote:
> For personal systems, yes.  For servers, probably not a bug.
> 
> Disabling readahead means faster execution queued commands,
> since it doesn't have to "linger" and do unwanted read-ahead.
> So this bug is a "feature" for random access servers.
> And a big nuisance for everything else.

Sounds like a trade off between doing read ahead on the current track,
or immediately starting to seek to the track for the next request in the
queue.  After all reading ahead on the current track means you can't
start seeking to the next request yet.

I agree it does sound like a feature.  Let the user/OS decide if read
aheads should be done by queueing them if wanted, and not queueing them
if not wanted.  Seems simple enough.  Also avoids poluting the cache
with unwanted data at the expense of potentially wanted data that was
already in the cache.

--
Len Sorensen

Re: Lower HD transfer rate with NCQ enabled?

[Phillip Susi]

Mark Lord wrote:
> Phillip Susi wrote:
>> Sounds like this is a serious bug in the WD firmware.
> 
> For personal systems, yes.  For servers, probably not a bug.
> 
> Disabling readahead means faster execution queued commands,
> since it doesn't have to "linger" and do unwanted read-ahead.
> So this bug is a "feature" for random access servers.
> And a big nuisance for everything else.

I think you misunderstand the bug.  The bug is not that the drive 
disables internal readahead; the bug is that host supplied readahead 
requests work so horribly.  It is a good thing that the drive allows the 
host to control the readahead, but something is wrong if the drive's 
readahead is WAY better than any the host can perform.


In other words, if the host is sending down multiple sequential reads 
such that the queue always has a request for the next sector, the drive 
should be hitting its maximum physical transfer rate.  If it doesn't, 
then something is very broken with its firmware.

It is still possible though, that the kernel is simply not issuing its 
own readahead effectively so the queue ends up empty at times and the 
disk goes idle.  This would be worth investigating before letting WD 
know they have a serious firmware problem.

Re: Lower HD transfer rate with NCQ enabled?

[Mark Lord]

Phillip Susi wrote:
> Mark Lord wrote:
>> Phillip Susi wrote:
>>> Sounds like this is a serious bug in the WD firmware.
>>
>> For personal systems, yes.  For servers, probably not a bug.
>>
>> Disabling readahead means faster execution queued commands,
>> since it doesn't have to "linger" and do unwanted read-ahead.
>> So this bug is a "feature" for random access servers.
>> And a big nuisance for everything else.
> 
> I think you misunderstand the bug.  The bug is not that the drive 
> disables internal readahead; the bug is that host supplied readahead 
> requests work so horribly.  It is a good thing that the drive allows the 
> host to control the readahead, but something is wrong if the drive's 
> readahead is WAY better than any the host can perform.

Well, in this case, it has already been determined that switching
to a different Linux I/O scheduler gives back most of the performance.

But the drive can do readahead better than the OS:  With the OS,
everything is broken up into discrete requests, whereas with the 
drive firmware, it can continuously update it's readahead projections,
even in the midst of a command.  So it does have an advantage.

But again, only the WD Raptor seems to have serious problems here.
Other drives cope well with readahead + NCQ just fine.

Cheers

Re: Lower HD transfer rate with NCQ enabled?

[Paa Paa]

Mark Lord wrote:
>>>This is mostly a problem with the WD Raptor drive, and some other WD 
>>>drives.
>>>I have not yet encountered/noticed the problem with other brands.
>>
>>Sounds like this is a serious bug in the WD firmware.
>
>For personal systems, yes.  For servers, probably not a bug.

In my case the choice of IO scheduler affected the result of "hdparm -t" 
significantly. With Deadline I got identical result with NCQ and wihout NCQ. 
It was only with CFQ when I got worse results with NCQ than without it.

What might be the reason CFQ+NCQ gived so much worse throughput with "hdparm 
-t"?

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/