Houston, we have a problem: big file copying, USB, external disks

Houston, we have a problem: big file copying, USB, external disks

[John Zupfel]

Are recent kernels (since somewhere around 2.6.27 - 2.6.28)
choking on big file copying?


https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/197762?comments=all

Quoting the latest post:
> No, it's not Ubuntu specific and Canonical didn't cause it.
> 
> Right now I'm copying between two SATA drives, connected to my PC via
> eSATA.  So far a lousy 600MB has been copied and I'm already down to
> 5MB/s and falling.
> 
> Here is what I know about the problem:
> 
> It affects ALL hardware.
> 
> It affects ALL file copying connections to varying degrees: USB,
> network, SATA/eSATA.
> 
> In some instances, it also affects multitasking, all but locking the
> computer up despite little to no CPU use in System Monitor.
> 
> There are no error logs, messages or anything else that occur when the
> problem is happening.
> 
> In my experience, if you test for the problem and nothing seems to be
> wrong, then that's because you aren't copying enough data, you don't
> know how fast your hardware should be copying files, or you are lucky
> enough to only be subtly affected that time.  Try copying more multi-
> gigabytes of data, several times and see if you can't get wildly
> different durations for the same data size.  As I state above, all
> hardware I have tested is affected to varying degrees, some hardware
> shows minor performance degradation over time, some hardware is all but
> useless for file copying.
> 
> If you ask me, it's either the scheduler (GUI lockups with no CPU usage
> makes me think this) or whatever performs the actual file copy process.
> It's definitely not a USB problem because it affects network copying
> too.



      

Re: Houston, we have a problem: big file copying, USB, external disks

[Ralf Hildebrandt]

* John Zupfel <jzupfel@yahoo.co.uk>:
> 
> Are recent kernels (since somewhere around 2.6.27 - 2.6.28)
> choking on big file copying?
> 
> 
> https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/197762?comments=all

Does this happen AFTER a suspend/hibernate or ALL THE TIME?

> > No, it's not Ubuntu specific and Canonical didn't cause it.
> > 
> > Right now I'm copying between two SATA drives, connected to my PC via
> > eSATA.  So far a lousy 600MB has been copied and I'm already down to
> > 5MB/s and falling.
> > 
> > Here is what I know about the problem:
> > 
> > It affects ALL hardware.
> > 
> > It affects ALL file copying connections to varying degrees: USB,
> > network, SATA/eSATA.
> > 
> > In some instances, it also affects multitasking, all but locking the
> > computer up despite little to no CPU use in System Monitor.
> > 
> > There are no error logs, messages or anything else that occur when the
> > problem is happening.
> > 
> > In my experience, if you test for the problem and nothing seems to be
> > wrong, then that's because you aren't copying enough data, you don't
> > know how fast your hardware should be copying files, or you are lucky
> > enough to only be subtly affected that time.  Try copying more multi-
> > gigabytes of data, several times and see if you can't get wildly
> > different durations for the same data size.  As I state above, all
> > hardware I have tested is affected to varying degrees, some hardware
> > shows minor performance degradation over time, some hardware is all but
> > useless for file copying.
> > 
> > If you ask me, it's either the scheduler (GUI lockups with no CPU usage
> > makes me think this) or whatever performs the actual file copy process.
> > It's definitely not a USB problem because it affects network copying
> > too.
> 
> 
> 
>       
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12200 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  Ralf.Hildebrandt@charite.de | http://www.charite.de

Re: Houston, we have a problem: big file copying, USB, external disks

[John Zupfel]

--- On Mon, 18/5/09, Ralf Hildebrandt <Ralf.Hildebrandt@charite.de> wrote:
> Does this happen AFTER a suspend/hibernate or ALL THE TIME?

All the time. And according to most of the reports, the slowdown usually starts after the first several hundred MiBs have been copied.


      

Re: Houston, we have a problem: big file copying, USB, external disks

[Simon Holm Thøgersen]

man, 18 05 2009 kl. 13:47 +0000, skrev John Zupfel:
> --- On Mon, 18/5/09, Ralf Hildebrandt <Ralf.Hildebrandt@charite.de> wrote:
> > Does this happen AFTER a suspend/hibernate or ALL THE TIME?
> 
> All the time. And according to most of the reports, the slowdown
> usually starts after the first several hundred MiBs have been copied.

Please use the reply button in your email client to avoid breaking
threading.

The bug you link to

> https://bugs.launchpad.net/ubuntu/+source/gvfs/+bug/197762?comments=all

is unfortunately a complete mess and not really helpful. There are 20
different people involved and they mostly pop in with a message or two
with next to nothing of real substance with regard to solving the
problem. As far as I can tell there are people experiencing all of the
following "problems" (indiviually, not at the same time)
 - their hardware is in USB 1.1 and not 2.0 mode,
 - the file copy writes into the page cache that is in memory and the
   writing to disk does not start until the cache is full or
   pdflush initiates writing to disk (the GUI shows a fast transfer rate
   at the beginning of the transfer and then slows down to the real
   bandwidth of the underlying device)
 - performance is slower when disk is mounted sync and not async,
 - the underlying device cannot sustain an even write rate over longer
   periods (possible for flash based devices), or
 - excessive dmesg output (to the point it could even slow the copy
   down?)

Then there is the genuine possibility of a bug in
 - mmc/sdhci driver or layer,
 - usb driver or layer, or
 - somewhere else.

It is also unclear how the users determine that a given transfer rate is
bad. Again, everything along the lines of the following
 - Linux does worse than Windows XP,
 - version 2.6.27.9 is worse than 2.6.27.7,
 - the 64 bit version is worse than the 32 bit version of the same
   kernel version, 
 - the transfer rate is ridiculous slow (< 100 kB/s), or
 - using the sync mount option is worse than using async.

Now if some users can narrow it down to somewhere between 2.6.27.7
and .9 then get them to bisect the patches that went in between the two
minor releases. Or possibly inspect the patches to see if there is an
obvious candidate that can be tried out right away.

Generally, if they can provide another version of Ubuntu (and thus the
kernel) that works better it should be relatively easy to make some
progress.

If some users can tell a significant difference between a 32 vs. 64 bit
version go investigate further, starting with the output of dmesg, lspci
and lsusb for both versions.

If transfer rate is ridiculous slow then check dmesg, lspci and lsusb
with the driver or subsystem author of the device (after checking for
obvious signs of something odd in dmesg and having tested the latest
release candidate of the upstream kernel).

If transfer rate is in the ~1 MB/s area then check that the device is
really running in 2.0 mode. Then check that another operating system
really have better performance (remembering to properly wait for all
data to hit disk before stopping the clock). Or better yet, find another
version of Linux that have better performance.

If sync is somewhat slower than async it is probably what can be
expected. A large difference for big files should probably taken up with
the kernel developers. Again, check with the latest upstream version
before reporting though.

And how come the Ubuntu folks mark bugs as duplicates without having any
evidence that the cause is the same. How on earth can a bug be marked as
a duplicate of another when there is absolute no idea what the course of
the latter is?

Now, Leann Ogasawara actually already told people to report in separate
bugs and provide various info. Few others actually did. How about you do
something useful and make it happen that the people provide useful
information in separate bugs such that it is parsable without having to
spent literally hours trying to figure how a report from X is (possibly
not) related to the one from Y?



Simon Holm Thøgersen