Extremely slow metadata performance

[John Goerzen <jgoerzen@complete.org>]

Hello,

I have observed extremely slow metadata performance with btrfs. This may 
be a bit of a nightmare scenario; it involves untarring a backup of
1.6TB of backuppc data, which contains millions of hardlinks and much
data, onto USB 2.0 disks.

I have run disk monitoring tools such as dstat while performing these
operations to see what's going on.

The behavior I notice is this:

   * When unpacking large files, the USB drives sustain activity in the
     20-40 MB/s range, as expected.
   * When creating vast numbers of hardlinks instead, the activity is
     roughly this:
       o Bursts of output from tar due to -v, sometimes corresponding to
         reads in the 300KB/s range (I suspect this has
         to do with caching)
       o Tar blocked for minutes while writes to the disk occur, in the
         300-600KB/s range.

This occurs even when nobarrier,noatime are specified as mount options. 
  I know the disk is capable of far more, because btrfs gets
far more from it when writing large files.

There are two USB drives in this btrfs filesystem: a 1TB and a 2TB
drive.  I have tried the raid1, raid0, and single metadata profiles.
Anecdotal evidence suggests that raid1 performs the worst, raid0 the
best, and single somewhere in between.  The data is in single mode.

Is this behavior known and expected?

Thanks,

John