dump ext4 performance degrades linearly as disk fills

["Joseph D. Wagner" <joe@josephdwagner.info>]

Background:
- I use lvm snapshots on ext4 for backup.  I use dump to backup the 
snapshots.  The backup goes to an external hard drive over usb 3.0. The 
external hard drive has 1 partition formatted with ext4.

Problem:
- I've noticed the performance of dump to my ext4 backup degrades 
linearly as the disk fills.  In the output, the kbps in the status line 
keeps getting lower as the percent complete keeps getting higher.

My Thoughts So Far:
- I suspect that either 1) dump is doing something which lowers 
performance as the backup progresses, or 2) the ext4 algorithm for 
finding and allocating free blocks is vulnerable to performance 
degradation as the volume fills.

- I haven't tested this thoroughly.  However, performance appears to 
improve when I clear out the external drive and do a fresh, full dump 
(-0), and performance appears to remain degraded on incremental backups 
on a nearly full volume.  This leads me to suspect #2.

Help from you:
- What steps can I take to isolate the cause of the problem?  If there's 
any information I can provide, please let me know.

What I've Done:
- I've run both the short and long SMART tests on the external drive, 
which came back clean.  They also told me the relocated sector count was 
zero.

- I also made sure the disk was connecting in usb 3.0 mode.

- I changed the external drive ext4 mount from journal=ordered to 
journal=writeback.  The performance difference was nominal.

Any insight would be appreciated.

Thanks.

Joseph D. Wagner