Re: slow eMMC write speed

Praveen G K
2011-09-28T22:24:03Z

On Wed, Sep 28, 2011 at 2:34 PM, J Freyensee
<james_p_freyensee@linux.intel.com> wrote:
> On 09/28/2011 02:03 PM, Praveen G K wrote:
>>
>> On Wed, Sep 28, 2011 at 2:01 PM, J Freyensee
>> <james_p_freyensee@linux.intel.com>  wrote:
>>>
>>> On 09/28/2011 01:34 PM, Praveen G K wrote:
>>>>
>>>> On Wed, Sep 28, 2011 at 12:59 PM, J Freyensee
>>>> <james_p_freyensee@linux.intel.com>    wrote:
>>>>>
>>>>> On 09/28/2011 12:06 PM, Praveen G K wrote:
>>>>>>
>>>>>> On Tue, Sep 27, 2011 at 10:42 PM, Linus Walleij
>>>>>> <linus.walleij@linaro.org>      wrote:
>>>>>>>
>>>>>>> On Fri, Sep 23, 2011 at 7:05 AM, Praveen G K<praveen.gk@gmail.com>
>>>>>>>  wrote:
>>>>>>>
>>>>>>>> I am working on the block driver module of the eMMC driver (SDIO 3.0
>>>>>>>> controller).  I am seeing very low write speed for eMMC transfers.
>>>>>>>>  On
>>>>>>>> further debugging, I observed that every 63rd and 64th transfer
>>>>>>>> takes
>>>>>>>> a long time.
>>>>>>>
>>>>>>> Are you not just seeing the card-internal garbage collection?
>>>>>>> http://lwn.net/Articles/428584/
>>>>>>
>>>>>> Does this mean, theoretically, I should be able to achieve larger
>>>>>> speeds if I am not using linux?
>>>>>
>>>>> In theory in a fairy-tale world, maybe, in reality, not really.  In R/W
>>>>> performance measurements we have done, eMMC performance in products
>>>>> users
>>>>> would buy falls well, well short of any theoretical numbers.  We
>>>>> believe
>>>>> in
>>>>> theory, the eMMC interface should be able to support up to 100MB/s, but
>>>>> in
>>>>> reality on real customer platforms write bandwidths (for example)
>>>>> barely
>>>>> approach 20MB/s, regardless if it's a Microsoft Windows environment or
>>>>> Android (Linux OS environment we care about).  So maybe it is software
>>>>> implementation issues of multiple OSs preventing higher eMMC
>>>>> performance
>>>>> numbers (hence the reason why I sometimes ask basic coding questions of
>>>>> the
>>>>> MMC subsystem- the code isn't the easiest to follow); however, one
>>>>> looks
>>>>> no
>>>>> further than what Apple has done with the iPad2 to see that eMMC
>>>>> probably
>>>>> just is not a good solution to use in the first place.  We have
>>>>> measured
>>>>> Apple's iPad2 write performance on *WHAT A USER WOULD SEE* being double
>>>>> what
>>>>> we see with products using eMMC solutions. The big difference?  Apple
>>>>> doesn't use eMMC at all for the iPad2.
>>>>
>>>> Thanks for all the clarification.  The problem is I am seeing write
>>>> speeds of about 5MBps on a Sandisk eMMC product and I can clearly see
>>>> the time lost when measured between sending a command and receiving a
>>>> data irq.  I am not sure what kind of an issue this is.  5MBps feels
>>>> really slow but can the internal housekeeping of the card take so much
>>>> time?
>>>
>>> Have you tried to trace through all structs used for an MMC operation??!
>>>  Good gravy, there are request, mmc_queue, mmc_card, mmc_host,
>>> mmc_blk_request, mmc_request, multiple mmc_command and multiple
>>> scatterlists
>>> that these other structs use...I've been playing around on trying to
>>> cache
>>> some things to try and improve performance and it blows me away how many
>>> variables and pointers I have to keep track of for one operation going to
>>> an
>>> LBA on an MMC.  I keep wondering if more of the 'struct request' could
>>> have
>>> been used, and 1/3 of these structures could be eliminated.  And another
>>> thing I wonder too is how much of this infrastructure is really needed,
>>> that
>>> when I do ask "what is this for?" question on the list and no one
>>> responds,
>>> if anyone else understands if it's needed either.
>>
>> I know I am not using the scatterlists, since the scatterlists are
>> aggregated into a 64k bounce buffer.  Regarding the different structs,
>> I am just taking them on face value assuming everything works "well".
>> But, my concern is why does it take such a long time (250 ms) to
>> return a transfer complete interrupt on occasional cases.  During this
>> time, the kernel is just waiting for the txfer_complete interrupt.
>> That's it.
>
> I think one fundamental problem with execution of the MMC commands is even
> though the MMC has it's own structures and own DMA/Host-controller, the OS's
> block subsystem and MMC subsystem do not really run independent of either
> other and each are still tied to each others' fate, holding up performance
> of the kernel in general.
>
> In particular, I have found that in the 2.6.36+ kernels that the sooner you
> can retire the 'struct request *req' (ie using __blk_end_request()) with
> respect to when the mmc_wait_for_req() call is made, the higher performance
> you are going to get out of the OS in terms of reads/writes using an MMC.
>  mmc_wait_for_req() is a blocking call, so that OS 'struct request req' will
> just sit around and do nothing until mmc_wait_for_req() is done.  I have
> been able to do some caching of some commands, calling __blk_end_request()
> before mmc_wait_for_req(), and getting much higher performance in a few
> experiments (but the work certainly is not ready for prime-time).
>
> Now in the 3.0 kernel I know mmc_wait_for_req() has changed and the goal was
> to try and make that function a bit more non-blocking, but I have not played
> with it too much because my current focus is on existing products and no
> handheld product uses a 3.0 kernel yet (that I am aware of at least).
>  However, I still see the fundamental problem is that the MMC stack, which
> was probably written with the intended purpose to be independent of the OS
> block subsystem (struct request and other stuff), really isn't independent
> of the OS block subsystem and will cause holdups between one another,
> thereby dragging down read/write performance of the MMC.
>
> The other fundamental problem is the writes themselves.  Way, WAY more
> writes occur on a handheld system in an end-user's hands than reads.
> Fundamental computer principle states "you make the common case fast". So
> focus should be on how to complete a write operation the fastest way
> possible.

Thanks for the detailed explanation.
Please let me know if there is a fundamental issue with the way I am
inserting the high res timers. In the block.c file, I am timing the
transfers as follows

1. Start timer
mmc_queue_bounce_pre()
mmc_wait_for_req()
mmc_queue_bounce_post()
End timer

So, I don't really have to worry about the blk_end_request right.
Like you said, wait_for_req is a blocking wait. I don't see what is
wrong with that being a blocking wait, because until you get the data
xfer complete irq, there is no point in going ahead. The
blk_end_request comes later in the picture only when all the data is
transferred to the card.
My line of thought is that the card is taking a lot of time for its
internal housekeeping. But, I want to be absolutely sure of my
analysis before I can pass that judgement.

I have also used another Toshiba card that gives me about 12 MBps
write speed for the same code, but I am worried is whether I am
masking some issue by blaming it on the card. What if the Toshiba
card can give a throughput more than 12MBps ideally?

Or could there be an issue that the irq handler(sdhci_irq) is called
with some kind of a delay and is there a possibility that we are not
capturing the transfer complete interrupt immediately?

>>
>>> I mean, for the usual transfers it takes about 3ms to transfer
>>>>
>>>> 64kB of data, but for the 63rd and 64th transfers, it takes 250 ms.
>>>> The thing is this is not on a file system.  I am measuring the speed
>>>> using basic "dd" command to write directly to the block device.
>>>>
>>>>> So, is this a software issue? or if
>>>>>>
>>>>>> there is a way to increase the size of bounce buffers to 4MB?
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>> Yours,
>>>>>>> Linus Walleij
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-mmc"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>> --
>>>>> J (James/Jay) Freyensee
>>>>> Storage Technology Group
>>>>> Intel Corporation
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>> --
>>> J (James/Jay) Freyensee
>>> Storage Technology Group
>>> Intel Corporation
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-mmc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> --
> J (James/Jay) Freyensee
> Storage Technology Group
> Intel Corporation
>