Performance issue with port multiplier

Performance issue with port multiplier

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I recently picked up a Thermaltake BlacX Duet dual drive esata dock
and I am having some performance problems trying to use both drives
concurrently.  The link is running at 3 Gbps both to the PMP and from
there to each drive.  An individual drive pulls ~175 MB/s, but trying
to use both drops to ~60 MB/s each.  This is with a simple sequential
dd to /dev/null with a 1 MiB block size.  Oddly, disabling NCQ results
in an increase to ~80 MB/s per drive.  I wouldn't think that NCQ would
really help for a sequential read, but it certainly shouldn't hurt.

Now it does seem that my intel ahci chipset does not support fis based
switching, but it doesn't seem like that should make a whole lot of
difference for sequential IO since while you are busy sending commands
to one drive, the other should still be performing its own readahead
into its cache and be ready to satisfy the request quickly once the
first drive finishes.

How can this behavior make sense?  Any ideas on other things I could try?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTWHoaAAoJEI5FoCIzSKrwgY4H/jIJw7EC8Oh6eXqNma67tVfn
eNYgjKOKv1qsn4QDAva1F6TyUdmTzeKVoym4yxOgu51hPzcnPCoeOyOg0ZNNbOSa
QSUxWHfgeGVHQvn5j30AZDqbfyIXd062mAjwDndFl4QSYRmAsZmijRavR33bZ8LP
S81myeHoRSSIJNJDkRAXtiJckUxjU8HdDkLzvowGCcvgn6MNi6uMtaNepdCOGD5V
9u91uUB7HcsrfBCPutSW3Yv1GoSn4wEPL0/lIzqfP0q81qErsWjXkjqi8VEYXPys
FcNT4lSGH0isjSCBtatDYMaWolLsOdkeZW4gwx07WyvsOP4H5fKFpVNrkFoAjoE=
=X8IN
-----END PGP SIGNATURE-----

Re: Performance issue with port multiplier

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 04/23/2014 10:42 PM, Phillip Susi wrote:
> I recently picked up a Thermaltake BlacX Duet dual drive esata dock
> and I am having some performance problems trying to use both drives
> concurrently.  The link is running at 3 Gbps both to the PMP and from
> there to each drive.  An individual drive pulls ~175 MB/s, but trying
> to use both drops to ~60 MB/s each.  This is with a simple sequential
> dd to /dev/null with a 1 MiB block size.  Oddly, disabling NCQ results
> in an increase to ~80 MB/s per drive.  I wouldn't think that NCQ would
> really help for a sequential read, but it certainly shouldn't hurt.
> 
> Now it does seem that my intel ahci chipset does not support fis based
> switching, but it doesn't seem like that should make a whole lot of
> difference for sequential IO since while you are busy sending commands
> to one drive, the other should still be performing its own readahead
> into its cache and be ready to satisfy the request quickly once the
> first drive finishes.
> 
> How can this behavior make sense?  Any ideas on other things I could try?

So I would have thought that if you asked a disk drive with 64 mb of
cache to read a few mb, then waited a second before issuing more
commands, it would continue to read the next several mb into cache while
waiting for new commands.  I wrote a simple benchmark to test this by
reading 16 256k blocks ( with O_DIRECT from a random starting offset to
start with cold cache ), sleeping for 1 second, then reading the next
16, and the results seem to show that the drive stops reading ahead just
after 1 MB.  Do drives really have such terrible readahead algorithms in
general?  If so, this would seem to explain the terrible port multiplier
performance in the absence of FBS.  If this is really the norm, then
perhaps libata needs to do a better job of balancing requests between
the drives?

My test results:

read took 6598 us
read took 1157 us
read took 1207 us
read took 1178 us
read took 1187 us
read took 2373 us << odd delay
read took 1172 us
read took 1180 us
read took 1163 us
read took 1171 us
read took 1172 us
read took 1136 us
read took 2373 us
read took 1129 us
read took 1170 us
read took 1171 us
- ---
read took 636 us
read took 628 us
read took 625 us
read took 627 us
read took 626 us
read took 626 us
read took 580 us
read took 563 us
read took 6733 us << why is this even longer than the first seek
read took 1237 us << cache is now empty, waiting on actual reads
read took 2326 us << odd delay
read took 1165 us
read took 1169 us
read took 1171 us
read took 1173 us
read took 1169 us
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTcuWsAAoJEI5FoCIzSKrwMEEH/03Qne3kWo3UdW62OZVTcD0Z
DgCtOSIx//feHDgXOIu46f8QO6BnvAFG+RyrplVjMvNTKio4M4cgCHJU7EOUaka0
tN2eturZFnzCo5ZQLgaQ8ZWWihbxVdT9BVjt0ixn+lUNjbhLfLf+qvagzOeLenOa
RtMV2RbywK/G0DTEurNocQgJq/bVtoAtHGwv24WADL4ITd+5oRQHqBB5JupnUCD7
Nr8/uwLKkJZ7aVAX0zYAvFIUu3O0AucX/fPl8W63rm3wuB+E8BlLpHkMoF9hV8g0
qJiobIvUnbJ3y8luRlZjO5uH39osGfhFWH4r+cBDnI8eFoxdfcigyk3KEMeI48w=
=ROo0
-----END PGP SIGNATURE-----

Re: understanding non fis based switching

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 5/13/2014 11:40 PM, Phillip Susi wrote:
> So I would have thought that if you asked a disk drive with 64 mb
> of cache to read a few mb, then waited a second before issuing
> more commands, it would continue to read the next several mb into
> cache while waiting for new commands.  I wrote a simple benchmark
> to test this by reading 16 256k blocks ( with O_DIRECT from a
> random starting offset to start with cold cache ), sleeping for 1
> second, then reading the next 16, and the results seem to show that
> the drive stops reading ahead just after 1 MB.  Do drives really
> have such terrible readahead algorithms in general?  If so, this
> would seem to explain the terrible port multiplier performance in
> the absence of FBS.  If this is really the norm, then perhaps
> libata needs to do a better job of balancing requests between the
> drives?

I have been trying to follow the code to see how it switches back and
forth between the two drives, and what, if anything, is done to keep
that switching smooth and fair.  It looks like the libata layer just
returns an error to the scsi layer telling it to try the request again
later whenever the other disk has outstanding commands and holds the
port exclusively.  I can't find where in the scsi code it decides it
is time to retry that command, or if it does anything to stop any more
commands being issued to the first drive so it can free up the port so
the second drive can get a crack at it.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTc8svAAoJEI5FoCIzSKrw9/EIAJSV6wvL/BFeOpuv7M8Q6dQz
MExvhLe4dO/Nlc6bHr7I9bK6Pa1UL9vOmqpIbTDpmL2o8fQVri/0SVPrZ2fUw3BS
rmkq7md/eM875/NQAdExnZ8EQYzpevAOTobUnJSRkdtNvLFSmqK1HhZAipHmI1Bu
TW2Xy7pp7qj8qs/6Cc1Z+Ny60GY9yB6fVR6hlltL5M56wrBCT+q6vTeJatBvf8Ra
+E3Sxa/7X8Tv8/MQJ0oE1q5sMA3Vwz3Mrg+8cqr064SGD0JhDn7WwCOkxj3kQBIO
+K4wnQdkB3wUzQsm5CAnTztJTH2hrx6eR1RTac+9HncPqo0eMuXubupkhtb6KVU=
=kQSu
-----END PGP SIGNATURE-----