Weird behavior (no commit?) when writing to a really slow block device

[Roman Mamedov <rm@romanrm.net>]

Hello,

I am rsyncing some files from fast devices (100-200 MB/s) to a slow remote one
(15 MB/s) over nbd.

There is some strange behavior, at least on my current kernel 5.10.161:

  * free space shown in "df" does NOT decrease even after many hours and
    hundreds of GBs already written;

  * freshly written files (already fully copied by rsync log) are shown with
    their real size in "ls -la", but all have zero size in "du".

I keep an eye on: watch "grep -e Dirty -e Writeback: /proc/meminfo"
This currently shows:

  Dirty:            554936 kB
  Writeback:       3388616 kB

The latter figure hovers around 3.5-4 GB all the time. For these syncing
setups, I have these settings tweaked on this machine (so I guess it's not as
large as it could have been with 64 GB of RAM):

  vm.dirty_background_ratio=5
  vm.dirty_ratio=10

As I understand the described weird symptoms would have cleared up if there
was a complete flush of data and metadata. And I suspect issuing a "sync" on
command-line would cause that to occur. But the question is why this didn't
occur on its own, as said above, over many hours of copying? If I disconnect
the block device now, wouldn't everything copied so far be actually lost?

I know there are the "commit" and "flushoncommit" options, but in this case I
haven't overridden either. Mount options are:

  rw,noatime,nodiratime,compress=zstd:3,nossd,space_cache=v2,skip_balance,subvolid=5,subvol=/

So if I had to guess, given a slow block device and the constant ongoing write
load trying to stream at a much faster rate, the "commit" doesn't actually
happen, neither in its predefined default interval of 30 seconds, nor during
much much longer (hours).

-- 
With respect,
Roman