Re: Linux I/O subsystem performance (was: linuxcon 2010...)

[Vladislav Bolkhovitin <vst@vlnb.net>]

Pasi Kärkkäinen, on 08/24/2010 11:25 AM wrote:
> On Mon, Aug 23, 2010 at 02:03:26PM -0400, Chetan Loke wrote:
>> I actually received 3+ off-post emails asking whether I was talking
>> about initiator or target in the 100K IOPS case below and what did I
>> mean by the ACKs.
>> I was referring to the 'Initiator' side.
>> ACKs == When scsi-ML down-calls the LLD via the queue-command, process
>> the sgl's(if you like) and then trigger the scsi_done up-call path.
>>
>
> Uhm, Intel and Microsoft demonstrated over 1 million IOPS
> using software iSCSI and a single 10 Gbit Ethernet NIC (Intel 82599).
>
> How come there is such a huge difference? What are we lacking in Linux?

I also have an impression that Linux I/O subsystem has some performance 
problems. For instance, in one recent SCST performance test only 8 Linux 
initiators with fio as a load generator were able to saturate a single 
SCST target with dual IB cards (SRP) on 4K AIO direct accesses over an 
SSD backend. This rawly means that any initiator took several times (8?) 
more processing time than the target. Hardware used for that target and 
initiators was the same. I can't see on this load why the initiators 
would need to do something more than the target. Well, I know we in SCST 
did an excellent work to maximize performance, but such a difference 
looks too much ;)

Also it looks very suspicious why nobody even tried to match that 
Microsoft/Intel record, even Intel itself who closely works with Linux 
community in the storage area and could do it using the same hardware.

Vlad