40% performance loss with multipath since kernel 3.12

40% performance loss with multipath since kernel 3.12

[Derick Swanepoel]

Hi

We have an iSCSI-based SAN that uses dm-multipath to aggregate two 10 Gb links in multibus mode. With kernel 3.11.6 we got very good performance, achieving up to 1550 MB/s read on the multipath device.

After upgrading to kernel 4.0.5, performance has dropped by 40%. Read speed is now limited to 960 MB/s, which is slower than the 975 MB/s that the individual paths can achieve.

We did a lot of testing on 3.11.6 to find an optimal rr_min_io_rq value and settled on 9 using the round-robin path selector. With 4.0.5, we’ve tried all the path selectors with rr_min_io_rq from 1 to 100, but can’t seem to beat the performance of a single path any more. For interest’s sake, the default rr_min_io_rq of 1 was actually one of the worst performing values.

We also tried kernel 3.12 and performance was also degraded but not quite as bad as 4.0.5. I am wondering if there was some significant change since 3.11 (dm-multipath 1.5.1) and 3.12 (dm-multipath 1.6.0).

Regards,
Derick

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: 40% performance loss with multipath since kernel 3.12

[Mike Snitzer]

On Thu, Jun 25 2015 at 12:07pm -0400,
Derick Swanepoel <dswanepoel@gmail.com> wrote:

> Hi
> 
> We have an iSCSI-based SAN that uses dm-multipath to aggregate two 10 Gb links in multibus mode. With kernel 3.11.6 we got very good performance, achieving up to 1550 MB/s read on the multipath device.
> 
> After upgrading to kernel 4.0.5, performance has dropped by 40%. Read speed is now limited to 960 MB/s, which is slower than the 975 MB/s that the individual paths can achieve.
> 
> We did a lot of testing on 3.11.6 to find an optimal rr_min_io_rq value and settled on 9 using the round-robin path selector. With 4.0.5, we’ve tried all the path selectors with rr_min_io_rq from 1 to 100, but can’t seem to beat the performance of a single path any more. For interest’s sake, the default rr_min_io_rq of 1 was actually one of the worst performing values.
> 
> We also tried kernel 3.12 and performance was also degraded but not quite as bad as 4.0.5. I am wondering if there was some significant change since 3.11 (dm-multipath 1.5.1) and 3.12 (dm-multipath 1.6.0).

Nothing stands out to me:

# git log --oneline --tags --source drivers/md/dm.c drivers/md/dm-mpath.c | grep 3.12- | tac
670368a dm-3.12-changes dm: stop using WQ_NON_REENTRANT
7e782af v3.12-rc1 [SCSI] Return ENODATA on medium error
00c4fc3 dm-3.12-changes dm ioctl: increase granularity of type_lock when loading table
fd2ed4d dm-3.12-changes dm: add statistics support
7426d62 v3.12-rc1 Merge tag 'dm-3.12-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm
cc9d3c3 dm-3.12-fixes dm mpath: do not fail path on -ENOSPC
f84cb8a dm-3.12-fixes dm mpath: disable WRITE SAME if it fails
6cfa585 dm-3.12-fixes dm: lower bio-based mempool reservation
f479082 dm-3.12-fixes dm: add reserved_rq_based_ios module parameter
e860313 dm-3.12-fixes dm: add reserved_bio_based_ios module parameter

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel