Some throughput tests with MQ and BFQ on MMC/SD

Some throughput tests with MQ and BFQ on MMC/SD

[Linus Walleij]

This week I tested the following:

- Merge the in-flight BFQ work from Paolo with my MMC MQ patch set
- Enable BFQ
- Run a few iterations of classic throughput tests

dd on whole internal eMMC, 7.38 GiB:

sync
echo 3 > /proc/sys/vm/drop_caches
sync
time dd if=/dev/mmcblk3 of=/dev/null
time dd if=/dev/mmcblk3 of=/dev/null bs=1M

iozone on a Noname SD card 2GB

mount /dev/mmcblk0p1 /mnt
sync
echo 3 > /proc/sys/vm/drop_caches
sync
iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test

The results:

Before patches (v4.10-rc8):

7918845952 bytes (7.4GB) copied, 194.504059 seconds, 38.8MB/s
real    3m 14.51s
user    0m 7.41s
sys     1m 10.34s

7918845952 bytes (7.4GB) copied, 176.519531 seconds, 42.8MB/s
real    2m 56.53s
user    0m 0.06s
sys     0m 36.57s

Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
Output is in kBytes/sec

                                                   random    random
   kB  reclen    write  rewrite    read    reread    read     write
20480       4     1960     2105     5991     6023     5202       40
20480       8     4636     4901     9087     9103     9066       80
20480      16     5522     5663    12237    12242    12206      163
20480      32     5976     6031    14915    14917    14901      333
20480      64     6286     6387    16737    16763    16738      678
20480     128     6720     6757    17876    17857    17865     1403
20480     256     6846     6909    18230    17568    16719     3039
20480     512     7204     7229    18471    18751    18834     7209
20480    1024     7257     7315    18684    18044    18095     7337
20480    2048     7322     7388    18605    18802    19437     7401
20480    4096     7553     7652    21510    21108    21503     7688
20480    8192     7534     7745    22164    22300    22490     7758
20480   16384     7357     7818    23053    23048    23056     7834


After MMC MQ patches:

7918845952 bytes (7.4GB) copied, 196.907776 seconds, 38.4MB/s
real    3m 16.91s
user    0m 7.17s
sys     1m 8.03s

7918845952 bytes (7.4GB) copied, 192.595734 seconds, 39.2MB/s
real    3m 12.60s
user    0m 0.12s
sys     0m 33.11s

Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
                                                   random    random
   kB  reclen    write  rewrite    read    reread    read     write
20480       4     2049     2154     5991     5998     5934       40
20480       8     4654     4921     9081     9075     9028       81
20480      16     5572     5747    12250    12252    12177      164
20480      32     6040     6084    14858    14895    14833      335
20480      64     6370     6449    16759    16770    16715      682
20480     128     6834     6814    17882    17843    17878     1411
20480     256     6892     6900    18526    18105    18430     3066
20480     512     7239     7254    18839    18864    18837     7258
20480    1024     7342     6453    18787    18161    17522     7343
20480    2048     7408     7439    17891    18211    19029     7472
20480    4096     7641     7703    20950    21044    20900     7705
20480    8192     7584     7811    22261    22170    22385     7809
20480   16384     7407     7873    23033    23050    23048     7905


After MMC MQ+BFQ patches:

7918845952 bytes (7.4GB) copied, 197.097717 seconds, 38.3MB/s
real    3m 17.10s
user    0m 7.67s
sys     1m 7.33s

7552+0 records in
7552+0 records out
7918845952 bytes (7.4GB) copied, 187.119538 seconds, 40.4MB/s
real    3m 7.12s
user    0m 0.11s
sys     0m 34.61s

Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
Output is in kBytes/sec
                                                   random    random
   kB  reclen    write  rewrite    read    reread    read     write
20480       4     1734     1786     5923     5166     5894       40
20480       8     4614     4853     8950     8949     8909       80
20480      16     5525     5705    12086    12098    12040      164
20480      32     6027     6040    14765    14793    14755      334
20480      64     6341     6404    16696    16697    16670      680
20480     128     6799     6842    17830    17833    17814     1407
20480     256     6848     6849    17394    18251    17537     3054
20480     512     7191     7229    18545    18628    18801     7224
20480    1024     7241     7331    17845    17909    18206     7302
20480    2048     7375     7433    18794    19288    19675     7426
20480    4096     7583     7696    21024    21194    21082     7659
20480    8192     7555     7767    22068    22170    22168     7808
20480   16384     7350     7831    23021    23032    23050     7870


As you can see there are no huge performance regressions with these
kinds of "raw" throughput tests.

These iozone figures are unintuitive unless your head can
plot logarithmic, look at the charts here for a more visual presentation
of the iozone results:
https://docs.google.com/spreadsheets/d/1rm72TiGlTnzDeGLR__aqvjcJ2UkA-Ro3-XyKA8r1M-c

Compare this to the performance change we got when first introducing
the asynchronous requests:
https://wiki.linaro.org/WorkingGroups/KernelArchived/Specs/StoragePerfMMC-async-req

The patches need some issues fixed from the build server
complaints and some robustness hammering, but after that I
think they will be ripe for merging for v4.12.

Yours,
Linus Walleij

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Ziji Hu]

Hi Linus,

On 2017/2/17 17:33, Linus Walleij wrote:
> This week I tested the following:
> 
> - Merge the in-flight BFQ work from Paolo with my MMC MQ patch set
> - Enable BFQ
> - Run a few iterations of classic throughput tests
> 
> dd on whole internal eMMC, 7.38 GiB:
> 
> sync
> echo 3 > /proc/sys/vm/drop_caches
> sync
> time dd if=/dev/mmcblk3 of=/dev/null
> time dd if=/dev/mmcblk3 of=/dev/null bs=1M
> 
> iozone on a Noname SD card 2GB
> 
> mount /dev/mmcblk0p1 /mnt
> sync
> echo 3 > /proc/sys/vm/drop_caches
> sync
> iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
> 
> The results:
> 
> Before patches (v4.10-rc8):
> 
> 7918845952 bytes (7.4GB) copied, 194.504059 seconds, 38.8MB/s
> real    3m 14.51s
> user    0m 7.41s
> sys     1m 10.34s
> 
> 7918845952 bytes (7.4GB) copied, 176.519531 seconds, 42.8MB/s
> real    2m 56.53s
> user    0m 0.06s
> sys     0m 36.57s
> 
> Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
> Output is in kBytes/sec
> 
>                                                    random    random
>    kB  reclen    write  rewrite    read    reread    read     write
> 20480       4     1960     2105     5991     6023     5202       40
> 20480       8     4636     4901     9087     9103     9066       80
> 20480      16     5522     5663    12237    12242    12206      163
> 20480      32     5976     6031    14915    14917    14901      333
> 20480      64     6286     6387    16737    16763    16738      678
> 20480     128     6720     6757    17876    17857    17865     1403
> 20480     256     6846     6909    18230    17568    16719     3039
> 20480     512     7204     7229    18471    18751    18834     7209
> 20480    1024     7257     7315    18684    18044    18095     7337
> 20480    2048     7322     7388    18605    18802    19437     7401
> 20480    4096     7553     7652    21510    21108    21503     7688
> 20480    8192     7534     7745    22164    22300    22490     7758
> 20480   16384     7357     7818    23053    23048    23056     7834
> 
> 
> After MMC MQ patches:
> 
> 7918845952 bytes (7.4GB) copied, 196.907776 seconds, 38.4MB/s
> real    3m 16.91s
> user    0m 7.17s
> sys     1m 8.03s
> 
> 7918845952 bytes (7.4GB) copied, 192.595734 seconds, 39.2MB/s
> real    3m 12.60s
> user    0m 0.12s
> sys     0m 33.11s
> 
> Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
>                                                    random    random
>    kB  reclen    write  rewrite    read    reread    read     write
> 20480       4     2049     2154     5991     5998     5934       40
> 20480       8     4654     4921     9081     9075     9028       81
> 20480      16     5572     5747    12250    12252    12177      164
> 20480      32     6040     6084    14858    14895    14833      335
> 20480      64     6370     6449    16759    16770    16715      682
> 20480     128     6834     6814    17882    17843    17878     1411
> 20480     256     6892     6900    18526    18105    18430     3066
> 20480     512     7239     7254    18839    18864    18837     7258
> 20480    1024     7342     6453    18787    18161    17522     7343
> 20480    2048     7408     7439    17891    18211    19029     7472
> 20480    4096     7641     7703    20950    21044    20900     7705
> 20480    8192     7584     7811    22261    22170    22385     7809
> 20480   16384     7407     7873    23033    23050    23048     7905
> 
> 
> After MMC MQ+BFQ patches:
> 
> 7918845952 bytes (7.4GB) copied, 197.097717 seconds, 38.3MB/s
> real    3m 17.10s
> user    0m 7.67s
> sys     1m 7.33s
> 
> 7552+0 records in
> 7552+0 records out
> 7918845952 bytes (7.4GB) copied, 187.119538 seconds, 40.4MB/s
> real    3m 7.12s
> user    0m 0.11s
> sys     0m 34.61s
> 
> Command line used: iozone -az -i0 -i1 -i2 -s 20m -I -f /mnt/foo.test
> Output is in kBytes/sec
>                                                    random    random
>    kB  reclen    write  rewrite    read    reread    read     write
> 20480       4     1734     1786     5923     5166     5894       40
> 20480       8     4614     4853     8950     8949     8909       80
> 20480      16     5525     5705    12086    12098    12040      164
> 20480      32     6027     6040    14765    14793    14755      334
> 20480      64     6341     6404    16696    16697    16670      680
> 20480     128     6799     6842    17830    17833    17814     1407
> 20480     256     6848     6849    17394    18251    17537     3054
> 20480     512     7191     7229    18545    18628    18801     7224
> 20480    1024     7241     7331    17845    17909    18206     7302
> 20480    2048     7375     7433    18794    19288    19675     7426
> 20480    4096     7583     7696    21024    21194    21082     7659
> 20480    8192     7555     7767    22068    22170    22168     7808
> 20480   16384     7350     7831    23021    23032    23050     7870
> 

	I would like to suggest that you should try the multiple thread
	test mode of iozone, since you are testing *Multi* Queue.

	Besides, it seems that your eMMC transfer speed is quite low.
	It is normal that read speed can reach more than 100MB/s in HS400.
	Could you try a higher speed mode? The test result might be
	limited by the bus clock frequency.	

> 
> As you can see there are no huge performance regressions with these
> kinds of "raw" throughput tests.
> 
> These iozone figures are unintuitive unless your head can
> plot logarithmic, look at the charts here for a more visual presentation
> of the iozone results:
> https://docs.google.com/spreadsheets/d/1rm72TiGlTnzDeGLR__aqvjcJ2UkA-Ro3-XyKA8r1M-c
> 
> Compare this to the performance change we got when first introducing
> the asynchronous requests:
> https://wiki.linaro.org/WorkingGroups/KernelArchived/Specs/StoragePerfMMC-async-req
> 
> The patches need some issues fixed from the build server
> complaints and some robustness hammering, but after that I
> think they will be ripe for merging for v4.12.
> 

	Actually I have been following your thread for some time.
	But currently I'm a little confused.
	May I know the purpose of your patch?

	Thank you.

Best regards,
Hu Ziji

> Yours,
> Linus Walleij
> --
> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Ulf Hansson]

[...]

>
>         I would like to suggest that you should try the multiple thread
>         test mode of iozone, since you are testing *Multi* Queue.

Yes. That seems reasonable.

However, the most important part here is the comparison between the
different code bases.

>
>         Besides, it seems that your eMMC transfer speed is quite low.
>         It is normal that read speed can reach more than 100MB/s in HS400.
>         Could you try a higher speed mode? The test result might be
>         limited by the bus clock frequency.

Perhaps if Linus can share a branch of the code integrated for the
different tests, we all can help out running them on those HW we have
at hand. Would you be willing to help out here?

>
>>
>> As you can see there are no huge performance regressions with these
>> kinds of "raw" throughput tests.
>>
>> These iozone figures are unintuitive unless your head can
>> plot logarithmic, look at the charts here for a more visual presentation
>> of the iozone results:
>> https://docs.google.com/spreadsheets/d/1rm72TiGlTnzDeGLR__aqvjcJ2UkA-Ro3-XyKA8r1M-c
>>
>> Compare this to the performance change we got when first introducing
>> the asynchronous requests:
>> https://wiki.linaro.org/WorkingGroups/KernelArchived/Specs/StoragePerfMMC-async-req
>>
>> The patches need some issues fixed from the build server
>> complaints and some robustness hammering, but after that I
>> think they will be ripe for merging for v4.12.
>>
>
>         Actually I have been following your thread for some time.
>         But currently I'm a little confused.
>         May I know the purpose of your patch?

I want MMC to move to the new BLKMQ interface and I want that because
of several reasons, see below.

1. It's new blk interface, all new development happens here. We should
use it to benefit from that.
2. The BLKMQ interface allow the MMC block device driver to be
significantly cleaner implemented - and I need that to be able to
maintain the code.
3. We want to make use of Paolo's BFQ-MK I/O scheduler, which
addresses provides guaranteed low latency. For example being able to
play a video clip, while doing a disc backup without getting frame
drops.

Kind regards
Uffe

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Linus Walleij]

On Fri, Feb 17, 2017 at 12:53 PM, Ziji Hu <huziji@marvell.com> wrote:

>         I would like to suggest that you should try the multiple thread
>         test mode of iozone, since you are testing *Multi* Queue.

Good point. This target has only 2 CPUs but still, maybe it performs!

>         Besides, it seems that your eMMC transfer speed is quite low.
>         It is normal that read speed can reach more than 100MB/s in HS400.
>         Could you try a higher speed mode? The test result might be
>         limited by the bus clock frequency.

The iozone tests are done on an SDcard. And I only did read tests on
the eMMC I have.

It's because I'm afriad of wearing out my eMMC :(

But OK I'll just take the risk and run iozone on the eMMC.

>         Actually I have been following your thread for some time.
>         But currently I'm a little confused.
>         May I know the purpose of your patch?

Ulf describes it: we want to switch MMC/SD to MQ.

To me, there are two reasons for that (no secret agendas...)

1. To get away from the legacy codebase in the old block layer.
   Christoph and Jens have been very clear stating that the old block
   layer is in maintenance mode and should be phased out, and they
   asked explicitly for out help to do so. Currently
   MMC/SD is a big fat roadblock to these plans so it is win-win for
   MMC/SD and the block layer if we can just switch over to MQ.

2. My colleague Paolo Valente is working on the next generation
  block scheduler BFQ which has very promising potential for
  interactive loads. (Like taking a backup of your harddrive while
  playing 1080p video let's say.) Since the
  old block layer is no longer maintained, this scheduler will only
  be merged and made available for systems deploying MQ. He's
  already working full steam on that.

I would like to make 1+2 happen in the next merge window
ultimately, but yeah, maybe I'm overly optimistic. But I will sure
try.

Maybe I should add:

3. MQ is a better and future-proof fit for command queueing.

Yours,
Linus Walleij

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Ziji Hu]

Hi Ulf,

On 2017/2/17 20:09, Ulf Hansson wrote:
> [...]
> 
>>
>>         I would like to suggest that you should try the multiple thread
>>         test mode of iozone, since you are testing *Multi* Queue.
> 
> Yes. That seems reasonable.
> 
> However, the most important part here is the comparison between the
> different code bases.
> 
>>
>>         Besides, it seems that your eMMC transfer speed is quite low.
>>         It is normal that read speed can reach more than 100MB/s in HS400.
>>         Could you try a higher speed mode? The test result might be
>>         limited by the bus clock frequency.
> 
> Perhaps if Linus can share a branch of the code integrated for the
> different tests, we all can help out running them on those HW we have
> at hand. Would you be willing to help out here?
> 
	I'm glad to.
	But my available platforms all stay in Linux v4.4/v4.1.
	I can help when my platforms can upgrade.

>>
>>>
>>> As you can see there are no huge performance regressions with these
>>> kinds of "raw" throughput tests.
>>>
>>> These iozone figures are unintuitive unless your head can
>>> plot logarithmic, look at the charts here for a more visual presentation
>>> of the iozone results:
>>> https://docs.google.com/spreadsheets/d/1rm72TiGlTnzDeGLR__aqvjcJ2UkA-Ro3-XyKA8r1M-c
>>>
>>> Compare this to the performance change we got when first introducing
>>> the asynchronous requests:
>>> https://wiki.linaro.org/WorkingGroups/KernelArchived/Specs/StoragePerfMMC-async-req
>>>
>>> The patches need some issues fixed from the build server
>>> complaints and some robustness hammering, but after that I
>>> think they will be ripe for merging for v4.12.
>>>
>>
>>         Actually I have been following your thread for some time.
>>         But currently I'm a little confused.
>>         May I know the purpose of your patch?
> 
> I want MMC to move to the new BLKMQ interface and I want that because
> of several reasons, see below.
> 
> 1. It's new blk interface, all new development happens here. We should
> use it to benefit from that.
> 2. The BLKMQ interface allow the MMC block device driver to be
> significantly cleaner implemented - and I need that to be able to
> maintain the code.
> 3. We want to make use of Paolo's BFQ-MK I/O scheduler, which
> addresses provides guaranteed low latency. For example being able to
> play a video clip, while doing a disc backup without getting frame
> drops.
> 
	Got it. Thanks a lot for your detailed explanation.
	If we focus on the low latency, we shall get a cleaner result
	of IOPS in tests.
	IOPS is a more common performance index in eMMC/SD.

	Thank you.

Best regards,
Hu Ziji
> Kind regards
> Uffe
> 

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Ziji Hu]

Hi Linus,

On 2017/2/17 21:22, Linus Walleij wrote:
> On Fri, Feb 17, 2017 at 12:53 PM, Ziji Hu <huziji@marvell.com> wrote:
> 
>>         I would like to suggest that you should try the multiple thread
>>         test mode of iozone, since you are testing *Multi* Queue.
> 
> Good point. This target has only 2 CPUs but still, maybe it performs!
>
	In my very own opinion, (although I'm not the expert of marketing),
	quad-core platforms will be more and more popular.
	Quad-core might have a different result, especially when we are
	testing multiple threads. 

>>         Besides, it seems that your eMMC transfer speed is quite low.
>>         It is normal that read speed can reach more than 100MB/s in HS400.
>>         Could you try a higher speed mode? The test result might be
>>         limited by the bus clock frequency.
> 
> The iozone tests are done on an SDcard. And I only did read tests on
> the eMMC I have.
> 
> It's because I'm afriad of wearing out my eMMC :(
> 
> But OK I'll just take the risk and run iozone on the eMMC.
> 
	Actually, there is a parameter to limit the size of test file in iozone.
	I'm not sure why you need to scan the whole eMMC. But I personally
	believe it is unnecessary.

	Sorry for delay reply. Hope your eMMC is not broken yet. :p
	eMMC usually contains much more physical pages than the capacity
	it shows. Thus I guess your eMMC should be fine unless you torture
	it by entirely writing it again and again.

>>         Actually I have been following your thread for some time.
>>         But currently I'm a little confused.
>>         May I know the purpose of your patch?
> 
> Ulf describes it: we want to switch MMC/SD to MQ.
> 
> To me, there are two reasons for that (no secret agendas...)
> 
> 1. To get away from the legacy codebase in the old block layer.
>    Christoph and Jens have been very clear stating that the old block
>    layer is in maintenance mode and should be phased out, and they
>    asked explicitly for out help to do so. Currently
>    MMC/SD is a big fat roadblock to these plans so it is win-win for
>    MMC/SD and the block layer if we can just switch over to MQ.
> 
> 2. My colleague Paolo Valente is working on the next generation
>   block scheduler BFQ which has very promising potential for
>   interactive loads. (Like taking a backup of your harddrive while
>   playing 1080p video let's say.) Since the
>   old block layer is no longer maintained, this scheduler will only
>   be merged and made available for systems deploying MQ. He's
>   already working full steam on that.
> 
> I would like to make 1+2 happen in the next merge window
> ultimately, but yeah, maybe I'm overly optimistic. But I will sure
> try.
	I see. Thank you for the details.
> 
> Maybe I should add:
> 
> 3. MQ is a better and future-proof fit for command queueing.
	I strongly agree with you on it.

	Thank you.

Best regards,
Hu Ziji


> 
> Yours,
> Linus Walleij
> 

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Adrian Hunter]

On 17/02/17 15:22, Linus Walleij wrote:
> On Fri, Feb 17, 2017 at 12:53 PM, Ziji Hu <huziji@marvell.com> wrote:
> 
>>         I would like to suggest that you should try the multiple thread
>>         test mode of iozone, since you are testing *Multi* Queue.
> 
> Good point. This target has only 2 CPUs but still, maybe it performs!
> 
>>         Besides, it seems that your eMMC transfer speed is quite low.
>>         It is normal that read speed can reach more than 100MB/s in HS400.
>>         Could you try a higher speed mode? The test result might be
>>         limited by the bus clock frequency.
> 
> The iozone tests are done on an SDcard. And I only did read tests on
> the eMMC I have.
> 
> It's because I'm afriad of wearing out my eMMC :(
> 
> But OK I'll just take the risk and run iozone on the eMMC.
> 
>>         Actually I have been following your thread for some time.
>>         But currently I'm a little confused.
>>         May I know the purpose of your patch?
> 
> Ulf describes it: we want to switch MMC/SD to MQ.
> 
> To me, there are two reasons for that (no secret agendas...)
> 
> 1. To get away from the legacy codebase in the old block layer.
>    Christoph and Jens have been very clear stating that the old block
>    layer is in maintenance mode and should be phased out, and they
>    asked explicitly for out help to do so. Currently
>    MMC/SD is a big fat roadblock to these plans so it is win-win for
>    MMC/SD and the block layer if we can just switch over to MQ.
> 
> 2. My colleague Paolo Valente is working on the next generation
>   block scheduler BFQ which has very promising potential for
>   interactive loads. (Like taking a backup of your harddrive while
>   playing 1080p video let's say.) Since the
>   old block layer is no longer maintained, this scheduler will only
>   be merged and made available for systems deploying MQ. He's
>   already working full steam on that.
> 
> I would like to make 1+2 happen in the next merge window
> ultimately, but yeah, maybe I'm overly optimistic. But I will sure
> try.
> 
> Maybe I should add:
> 
> 3. MQ is a better and future-proof fit for command queueing.

MQ is not better - it is just different.  Because mmc devices do not have
multiple hardware queues, blk-mq essentially offers nothing but a different
way of doing the same thing.  And there are problems, such as blk-mq assumes
that the primary arbiter of whether a request can be issued is the queue
depth.  As I wrote here:
  https://marc.info/?l=linux-mmc&m=148336571720463&w=2
that is not the case for mmc, even with command queuing.

Also I wouldn't be surprise if BFQ needs some changes to work well with
command queuing.

It would be better if blk-mq support was experimental until we can see how
well it works in practice.

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Ziji Hu]

Hi Adrian,

On 2017/2/20 16:03, Adrian Hunter wrote:
> On 17/02/17 15:22, Linus Walleij wrote:
>> On Fri, Feb 17, 2017 at 12:53 PM, Ziji Hu <huziji@marvell.com> wrote:
<snip>
>> Ulf describes it: we want to switch MMC/SD to MQ.
>>
>> To me, there are two reasons for that (no secret agendas...)
>>
>> 1. To get away from the legacy codebase in the old block layer.
>>    Christoph and Jens have been very clear stating that the old block
>>    layer is in maintenance mode and should be phased out, and they
>>    asked explicitly for out help to do so. Currently
>>    MMC/SD is a big fat roadblock to these plans so it is win-win for
>>    MMC/SD and the block layer if we can just switch over to MQ.
>>
>> 2. My colleague Paolo Valente is working on the next generation
>>   block scheduler BFQ which has very promising potential for
>>   interactive loads. (Like taking a backup of your harddrive while
>>   playing 1080p video let's say.) Since the
>>   old block layer is no longer maintained, this scheduler will only
>>   be merged and made available for systems deploying MQ. He's
>>   already working full steam on that.
>>
>> I would like to make 1+2 happen in the next merge window
>> ultimately, but yeah, maybe I'm overly optimistic. But I will sure
>> try.
>>
>> Maybe I should add:
>>
>> 3. MQ is a better and future-proof fit for command queueing.
> 
> MQ is not better - it is just different.  Because mmc devices do not have
> multiple hardware queues, blk-mq essentially offers nothing but a different
> way of doing the same thing.  And there are problems, such as blk-mq assumes
> that the primary arbiter of whether a request can be issued is the queue
> depth.  As I wrote here:
>   https://marc.info/?l=linux-mmc&m=148336571720463&w=2
> that is not the case for mmc, even with command queuing.
> 
	I guess it might benefit CMDQ since multi-thread can more easily let
	CMDQ achieve full performance. It is just my guess. We shall wait for
	the test result.

	Based on my own experience on developing CMDQ driver, some of the issues, like different
	partition and ioctl, can be solved by getting and putting mmc_host (mmc_claim_host/mmc_release_host).
	The key issue is Direct-CMD, It will be more complex to handle Direct-CMD with blk-mq,
	than doing it with current block layer. With current block layer, I just let CMDQ driver
	borrow existing non-CMDQ transfer routine, and turn back to CMDQ transfer routine
	after DCMD completes. But I'm not the effort we need with blk-mq.

	My key point is, please correct me if I'm wrong, shall we avoid binding CMDQ and
	non-CMDQ card layer together?
	To be honest, I don't think CMDQ is a good design. It will bring a lot of troubles.
	In my very own opinion, if we try to develop a single routine to support both CMDQ
	and non-CMDQ, it will be painful to both sides.

	Thank you.

Best regards,
Hu Ziji

> Also I wouldn't be surprise if BFQ needs some changes to work well with
> command queuing.
> 
> It would be better if blk-mq support was experimental until we can see how
> well it works in practice.
> 

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Adrian Hunter]

On 20/02/17 13:04, Ziji Hu wrote:
> Hi Adrian,
> 
> On 2017/2/20 16:03, Adrian Hunter wrote:
>> On 17/02/17 15:22, Linus Walleij wrote:
>>> On Fri, Feb 17, 2017 at 12:53 PM, Ziji Hu <huziji@marvell.com> wrote:
> <snip>
>>> Ulf describes it: we want to switch MMC/SD to MQ.
>>>
>>> To me, there are two reasons for that (no secret agendas...)
>>>
>>> 1. To get away from the legacy codebase in the old block layer.
>>>    Christoph and Jens have been very clear stating that the old block
>>>    layer is in maintenance mode and should be phased out, and they
>>>    asked explicitly for out help to do so. Currently
>>>    MMC/SD is a big fat roadblock to these plans so it is win-win for
>>>    MMC/SD and the block layer if we can just switch over to MQ.
>>>
>>> 2. My colleague Paolo Valente is working on the next generation
>>>   block scheduler BFQ which has very promising potential for
>>>   interactive loads. (Like taking a backup of your harddrive while
>>>   playing 1080p video let's say.) Since the
>>>   old block layer is no longer maintained, this scheduler will only
>>>   be merged and made available for systems deploying MQ. He's
>>>   already working full steam on that.
>>>
>>> I would like to make 1+2 happen in the next merge window
>>> ultimately, but yeah, maybe I'm overly optimistic. But I will sure
>>> try.
>>>
>>> Maybe I should add:
>>>
>>> 3. MQ is a better and future-proof fit for command queueing.
>>
>> MQ is not better - it is just different.  Because mmc devices do not have
>> multiple hardware queues, blk-mq essentially offers nothing but a different
>> way of doing the same thing.  And there are problems, such as blk-mq assumes
>> that the primary arbiter of whether a request can be issued is the queue
>> depth.  As I wrote here:
>>   https://marc.info/?l=linux-mmc&m=148336571720463&w=2
>> that is not the case for mmc, even with command queuing.
>>
> 	I guess it might benefit CMDQ since multi-thread can more easily let
> 	CMDQ achieve full performance. It is just my guess. We shall wait for
> 	the test result.
> 
> 	Based on my own experience on developing CMDQ driver, some of the issues, like different
> 	partition and ioctl, can be solved by getting and putting mmc_host (mmc_claim_host/mmc_release_host).
> 	The key issue is Direct-CMD, It will be more complex to handle Direct-CMD with blk-mq,
> 	than doing it with current block layer. With current block layer, I just let CMDQ driver
> 	borrow existing non-CMDQ transfer routine, and turn back to CMDQ transfer routine
> 	after DCMD completes. But I'm not the effort we need with blk-mq.
> 
> 	My key point is, please correct me if I'm wrong, shall we avoid binding CMDQ and
> 	non-CMDQ card layer together?
> 	To be honest, I don't think CMDQ is a good design. It will bring a lot of troubles.
> 	In my very own opinion, if we try to develop a single routine to support both CMDQ
> 	and non-CMDQ, it will be painful to both sides.

Not sure what you mean, but the patches linked below support DCMD and
non-DCMD for flush, and the regular approach for discards:

	https://marc.info/?l=linux-mmc&m=148673270920906

CQE can be used for reads/writes only, or reads/writes/flushes.  Everything
else is done the regular way.


> 
> 	Thank you.
> 
> Best regards,
> Hu Ziji
> 
>> Also I wouldn't be surprise if BFQ needs some changes to work well with
>> command queuing.
>>
>> It would be better if blk-mq support was experimental until we can see how
>> well it works in practice.
>>
> 

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Linus Walleij]

On Mon, Feb 20, 2017 at 9:03 AM, Adrian Hunter <adrian.hunter@intel.com> wrote:

> MQ is not better - it is just different.

Well it is better in the sense that it has active maintainers and is
not scheduled
for depreciation.

> Because mmc devices do not have
> multiple hardware queues, blk-mq essentially offers nothing but a different
> way of doing the same thing.

I think what Ziji is pointing out is the hourglass-shaped structure of MQ.
It has multiple *issue* queues as well, not just multiple *hardware*
queues. That means that processes can have issue queues on different
CPUs and not all requests end up in a single nexus like with the old blk
layer.

Whether it benefits MMC/SD in the end is a good question. It might,
testing on multicores with multiple issue threads is needed.

> It would be better if blk-mq support was experimental until we can see how
> well it works in practice.

Do you mean experimental in the MMC/SD stack, such that we should
merge it as an additional scheduler instead of as the only scheduler
replacement?

I think SCSI did/still does things like that. On the other hand, UBI
just replaced the old block layer with MQ in commit
ff1f48ee3bb3, and it is also very widely
used, so there are example of both approaches. (How typical.)

Yours,
Linus Walleij

Re: Some throughput tests with MQ and BFQ on MMC/SD

[Adrian Hunter]

On 20/02/17 15:46, Linus Walleij wrote:
> On Mon, Feb 20, 2017 at 9:03 AM, Adrian Hunter <adrian.hunter@intel.com> wrote:
> 
>> MQ is not better - it is just different.
> 
> Well it is better in the sense that it has active maintainers and is
> not scheduled
> for depreciation.
> 
>> Because mmc devices do not have
>> multiple hardware queues, blk-mq essentially offers nothing but a different
>> way of doing the same thing.
> 
> I think what Ziji is pointing out is the hourglass-shaped structure of MQ.
> It has multiple *issue* queues as well, not just multiple *hardware*
> queues. That means that processes can have issue queues on different
> CPUs and not all requests end up in a single nexus like with the old blk
> layer.

blk-mq has a lighter touch, but if you use BLK_MQ_F_BLOCKING and only have
one hardware context, then you are still putting everything through a single
work item.

> 
> Whether it benefits MMC/SD in the end is a good question. It might,
> testing on multicores with multiple issue threads is needed.
> 
>> It would be better if blk-mq support was experimental until we can see how
>> well it works in practice.
> 
> Do you mean experimental in the MMC/SD stack, such that we should
> merge it as an additional scheduler instead of as the only scheduler
> replacement?

Not sure what you mean by "scheduler".

> 
> I think SCSI did/still does things like that.

A quick look at SCSI shows a module parameter use_blk_mq that defaults based
on a config option CONFIG_SCSI_MQ_DEFAULT i.e. defaults to false unless
selected.

Yes, something like that.

>                                               On the other hand, UBI
> just replaced the old block layer with MQ in commit
> ff1f48ee3bb3, and it is also very widely
> used, so there are example of both approaches. (How typical.)

UBI block is read-only and does not benefit from an I/O scheduler, and has
nothing like a command queue.   It doesn't seem unreasonable that very
different kinds of devices would take different approaches.

Unfortunately, ff1f48ee3bb3 mixes together switching the underlying I/O to
using a scatter gather list and removing a serializing mutex, so it is hard
to tell if improvements came from that or from blk-mq.