USB Mass Storage write problem

USB Mass Storage write problem

[Morgan Howe]

Greetings,

We're working on an mx27 based board with 2 SD cards mounted as USB
mass storage devices.  The test we are doing is basically as follows:

1) Copy 1MB file to SD1
2) Copy same file from SD1 to SD2
3) sync
.. Repeat until disk full

During the test process nothing will ever report failure, however,
after the test completes, doing an MD5 check of the files will show
that some files were not written properly.  The file sizes are all
correct, but checking the contents we can see that in the bad files, at
some arbitrary point in the file it will stop writing the actual data
and the rest of the file will just be zero-filled.  The subsequent files
after the bad one will be perfectly fine for a while (usually >1GB
worth of files) and then randomly another file will get corrupted in
this manner.  Using 2 16GB SD cards we usually see around 2-6 bad files
for a full test run (until disk full).  We have tested this in both the
2.6.22.6 kernel and 2.6.35.

Has anyone experienced anything like this or have some idea where to
look?  I found this thread[1] which seemed related, but I'm running the
test program for that issue and so far no errors reported.  Any
suggestions would be appreciated.

Regards,
Morgan

[1] http://lkml.org/lkml/2010/3/4/220

USB Mass Storage write problem

[Baurzhan Ismagulov]

On Fri, Oct 01, 2010 at 04:00:35PM +0800, Morgan Howe wrote:
> During the test process nothing will ever report failure, however,
> after the test completes, doing an MD5 check of the files will show
> that some files were not written properly.  The file sizes are all
> correct, but checking the contents we can see that in the bad files, at
> some arbitrary point in the file it will stop writing the actual data
> and the rest of the file will just be zero-filled.  The subsequent files
> after the bad one will be perfectly fine for a while (usually >1GB
> worth of files) and then randomly another file will get corrupted in
> this manner.  Using 2 16GB SD cards we usually see around 2-6 bad files
> for a full test run (until disk full).  We have tested this in both the
> 2.6.22.6 kernel and 2.6.35.
> 
> Has anyone experienced anything like this or have some idea where to
> look?

I had a somewhat similar problem which turned out to be floating DMA
control lines of the USB controller, which resulted in incorrect reads
from and writes to the flash on the CPU bus.

In your case, I'd disable all drivers that are not used in the test case
and repeat the test.

The next step could be tracing the lowest-level writing routine in the
kernel in order to see whether it is called at all, gets the right data
(e.g., you could generate a file that has the same checksum for every
block written, etc.) and correctly handles status information from the
hardware.

If this is also doesn't reveal the problem, logic analyzer could help to
see what happens in the hardware and to compare good and bad writes.

That said, such problems are often hard to find systematically.
Sometimes testing different configurations, hardware, etc. helps better;
for example, do you have a configuration that passes your test?

With kind regards,
-- 
Baurzhan Ismagulov
http://www.kz-easy.com/

USB Mass Storage write problem

[Morgan Howe]

Dear Baurzhan,

On Fri, Oct 1, 2010 at 4:49 PM, Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Fri, Oct 01, 2010 at 04:00:35PM +0800, Morgan Howe wrote:
> > Has anyone experienced anything like this or have some idea where to
> > look?
>
> I had a somewhat similar problem which turned out to be floating DMA
> control lines of the USB controller, which resulted in incorrect reads
> from and writes to the flash on the CPU bus.
>
> In your case, I'd disable all drivers that are not used in the test case
> and repeat the test.
>
> The next step could be tracing the lowest-level writing routine in the
> kernel in order to see whether it is called at all, gets the right data
> (e.g., you could generate a file that has the same checksum for every
> block written, etc.) and correctly handles status information from the
> hardware.
>

I appreciate you taking the time to provide such a helpful response.  It
definitely gave us some direction to start debugging this problem.  As it
turns out, we seem to have been able to resolve the problem with some
hardware rework.

Thanks and regards,
Morgan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20101007/2848d009/attachment-0001.html>