linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Performance degradation in IO writes vs. reads (was scsi-mq V2)
       [not found]   ` <53C39F36.9010003@dev.mellanox.co.il>
@ 2014-08-21 12:32     ` Sagi Grimberg
  2014-08-21 13:03       ` Christoph Hellwig
  0 siblings, 1 reply; 4+ messages in thread
From: Sagi Grimberg @ 2014-08-21 12:32 UTC (permalink / raw)
  To: Christoph Hellwig, James Bottomley, Jens Axboe, Bart Van Assche,
	Robert Elliott, linux-scsi, linux-kernel, viro
  Cc: Or Gerlitz, Oren Duer, Nicholas A. Bellinger, Mike Christie,
	linux-fsdevel, linux-kernel, Max Gurtovoy

On 7/14/2014 12:13 PM, Sagi Grimberg wrote:

<SNIP>

> I'd like to share some benchmarks I took on this patch set using iSER
> initiator (+2 pre-submitted performance improvements) vs LIO iSER target.
> I ran workloads I think are interesting use-cases (single LUN with 1,2,4
> IO threads up to a fully occupied system doing IO to multiple LUNs).
> Overall (except 2 strange anomalies) seems that scsi-mq patches
> (use_blk_mq=N) roughly sustains traditional scsi performance.
> On the other hand scsi-mq code path (use_blk_mq=Y) on its own clearly
> shows better performance (tables below).
>
> At first I too hit the aio issues discussed in this thread and converted
> to scsi-mq.3-no-rebase for testing (thanks Doug & Rob for raising it).
> I must say that for some reason I get very low numbers for writes vs.
> reads (writes perf stuck at ~20K IOPs per thread), this happens
> on 3.16-rc2 even before scsi-mq patches. Did anyone step on this as well
> or is it just a weird problem I'm having in my setup?
> Anyway this is why my benchmarks shows only randread IO pattern (getting
> familiar numbers). I need to figure out whats wrong
> with IO writes - I'll start bisecting on this.
>

Hi,

So I just got back to checking this issue of *extremely low* IO write
performance I got in 3.16-rc2.

Reminder:
I used iSER to benchmark Christoph's scsi-mq patches performance and
noticed that direct-IO writes are stuck at 20-50K IOPs instead of 350K
IOPs I was used to see for a single device. This issue existed also when
I removed the scsi-mq patches, so I started bisecting to see what broke
stuff.

Finally I got to the completely unbisectable bulk that seems to yield
the issue:

/* IO write poor performance*/
2b777c9 ceph_sync_read: stop poking into iov_iter guts
f0d1bec new helper: copy_page_from_iter()
84c3d55 fuse: switch to ->write_iter()
b30ac0f btrfs: switch to ->write_iter()
3ef045c ocfs2: switch to ->write_iter()
bf97f3b xfs: switch to ->write_iter()
50b5551 afs: switch to ->write_iter()
da56e45 gfs2: switch to ->write_iter()
edaf436 nfs: switch to ->write_iter()
f5674c3 ubifs: switch to ->write_iter()
a8f3550 bury __generic_file_aio_write()
3dae875 cifs: switch to ->write_iter()
d4637bc udf: switch to ->write_iter()
9b88416 convert ext4 to ->write_iter()
a832475 Merge ext4 changes in ext4_file_write() into for-next
1456c0a blkdev_aio_write() - turn into blkdev_write_iter()
8174202 write_iter variants of {__,}generic_file_aio_write()
3644424 ceph: switch to ->read_iter()
3aa2d19 nfs: switch to ->read_iter()
a886038 fs/block_dev.c: switch to ->read_iter()
2ba5bbe shmem: switch to ->read_iter()
fb9096a pipe: switch to ->read_iter()
e6a7bcb cifs: switch to ->read_iter()
37c20f1 fuse_file_aio_read(): convert to ->read_iter()
3cd9ad5 ocfs2: switch to ->read_iter()
0279782 ecryptfs: switch to ->read_iter()
b4f5d2c xfs: switch to ->read_iter()
aad4f8b switch simple generic_file_aio_read() users to ->read_iter()
293bc98 new methods: ->read_iter() and ->write_iter()
7f7f25e replace checking for ->read/->aio_read presence with check in 
->f_mode
b318891 xfs: trim the argument lists of xfs_file_{dio,buffered}_aio_write()
37938463 blkdev_aio_read(): switch to generic_file_read_iter(), get rid 
of iov_shorten()
0c94933 iov_iter_truncate()
28060d5 btrfs: switch check_direct_IO() to iov_iter
91f79c4 new helper: iov_iter_get_pages_alloc()
f67da30 new helper: iov_iter_npages()
5b46f25 f2fs: switch to iov_iter_alignment()
c9c37e2 fuse: switch to iov_iter_get_pages()
d22a943 fuse: pull iov_iter initializations up
7b2c99d new helper: iov_iter_get_pages()
3320c60 dio: take updating ->result into do_direct_IO(
71d8e53 start adding the tag to iov_iter
ed978a8 new helper: generic_file_read_iter()
23faa7b fuse_file_aio_write(): merge initializations of iov_iter
05bb2e0 ceph_aio_read(): keep iov_iter across retries
886a391 new primitive: iov_iter_alignment()
26978b8 give ->direct_IO() a copy of iov_iter
31b1403 switch {__,}blockdev_direct_IO() to iov_iter
a6cbcd4 get rid of pointless iov_length() in ->direct_IO()
16b1f05 ext4: switch the guts of ->direct_IO() to iov_iter
619d30b convert the guts of nfs_direct_IO() to iov_iter
d8d3d94 pass iov_iter to ->direct_IO()
cb66a7a kill generic_segment_checks()
0ae5e4d __btrfs_direct_write(): switch to iov_iter
f8579f8 generic_file_direct_write(): switch to iov_iter
e7c2460 kill iov_iter_copy_from_user()
f6c0a19 fs/file.c: don't open-code kvfree()
/* IO write performance is OK*/

I tried to isolate the issue by running fio on a null_blk
device and also got performance degradation although it wasn't
as severe as with iSER/iSCSI. IO write performance decreased
from 360K IOPs to 280 KIOPs.

So at the moment I can't pin point the problem, but I figured
I'd raise the issue in case anyone else had stepped on this one
(hard to imagine that no one saw this...)

I'll run perf comparison to see if I get anything interesting.

CC'ing Al Viro, the author of all the above commits.

Cheers,
Sagi.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Performance degradation in IO writes vs. reads (was scsi-mq V2)
  2014-08-21 12:32     ` Performance degradation in IO writes vs. reads (was scsi-mq V2) Sagi Grimberg
@ 2014-08-21 13:03       ` Christoph Hellwig
  2014-08-21 14:02         ` Sagi Grimberg
  0 siblings, 1 reply; 4+ messages in thread
From: Christoph Hellwig @ 2014-08-21 13:03 UTC (permalink / raw)
  To: Sagi Grimberg
  Cc: Christoph Hellwig, James Bottomley, Jens Axboe, Bart Van Assche,
	Robert Elliott, linux-scsi, linux-kernel, viro, Or Gerlitz,
	Oren Duer, Nicholas A. Bellinger, Mike Christie, linux-fsdevel,
	Max Gurtovoy

On Thu, Aug 21, 2014 at 03:32:09PM +0300, Sagi Grimberg wrote:
> So I just got back to checking this issue of *extremely low* IO write
> performance I got in 3.16-rc2.

Please test with 3.16 final.  There once issue each in aio and dio
that caused bad I/O performance regression that were only fixed last
minute in the 3.16 cycle.


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Performance degradation in IO writes vs. reads (was scsi-mq V2)
  2014-08-21 13:03       ` Christoph Hellwig
@ 2014-08-21 14:02         ` Sagi Grimberg
  2014-08-24 16:41           ` Sagi Grimberg
  0 siblings, 1 reply; 4+ messages in thread
From: Sagi Grimberg @ 2014-08-21 14:02 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: James Bottomley, Jens Axboe, Bart Van Assche, Robert Elliott,
	linux-scsi, linux-kernel, viro, Or Gerlitz, Oren Duer,
	Nicholas A. Bellinger, Mike Christie, linux-fsdevel, Max Gurtovoy

On 8/21/2014 4:03 PM, Christoph Hellwig wrote:
> On Thu, Aug 21, 2014 at 03:32:09PM +0300, Sagi Grimberg wrote:
>> So I just got back to checking this issue of *extremely low* IO write
>> performance I got in 3.16-rc2.
>
> Please test with 3.16 final.  There once issue each in aio and dio
> that caused bad I/O performance regression that were only fixed last
> minute in the 3.16 cycle.
>

I'll do that now.

Thanks,
Sagi.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Performance degradation in IO writes vs. reads (was scsi-mq V2)
  2014-08-21 14:02         ` Sagi Grimberg
@ 2014-08-24 16:41           ` Sagi Grimberg
  0 siblings, 0 replies; 4+ messages in thread
From: Sagi Grimberg @ 2014-08-24 16:41 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: James Bottomley, Jens Axboe, Bart Van Assche, Robert Elliott,
	linux-scsi, linux-kernel, viro, Or Gerlitz, Oren Duer,
	Nicholas A. Bellinger, Mike Christie, linux-fsdevel, Max Gurtovoy

On 8/21/2014 5:02 PM, Sagi Grimberg wrote:
> On 8/21/2014 4:03 PM, Christoph Hellwig wrote:
>> On Thu, Aug 21, 2014 at 03:32:09PM +0300, Sagi Grimberg wrote:
>>> So I just got back to checking this issue of *extremely low* IO write
>>> performance I got in 3.16-rc2.
>>
>> Please test with 3.16 final.  There once issue each in aio and dio
>> that caused bad I/O performance regression that were only fixed last
>> minute in the 3.16 cycle.
>>
>
> I'll do that now.
>

Indeed this issue is resolved in 3.16. Sorry for the false alarm...
Thanks Christoph.

Sagi.

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2014-08-24 16:42 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <1403715121-1201-1-git-send-email-hch@lst.de>
     [not found] ` <20140708144829.GA5539@infradead.org>
     [not found]   ` <53C39F36.9010003@dev.mellanox.co.il>
2014-08-21 12:32     ` Performance degradation in IO writes vs. reads (was scsi-mq V2) Sagi Grimberg
2014-08-21 13:03       ` Christoph Hellwig
2014-08-21 14:02         ` Sagi Grimberg
2014-08-24 16:41           ` Sagi Grimberg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).