Re: bcachefs as a caching filesystem

[Brian Foster <bfoster@redhat.com>]

On Mon, Apr 10, 2023 at 01:51:07PM +0200, Dr.-Ing. Heiko Münkel wrote:
> Hello,
> 

FYI, there's no need to send multiple mails to the list. If you aren't
sure whether your mail was delivered, you can always reference the
archive at: https://lore.kernel.org/linux-bcachefs/.

> I am in the process of testing bcachefs. I have two partitions
> /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
> for the second (is an SSD on a USB3 port).
> 
> First I created the bcachefs file system with:
>   sudo bcachefs format \
>        --label=Shotwell-3 /dev/sde1 \
>        --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
>        --foreground_target /dev/nvme0n1p1 \
>        --promote_target /dev/nvme0n1p1 \
> --background_target /dev/sde1
> and then mount the disks with
>  sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
> 
> This worked fine so far. I then used dd to create a 100GB file on /mnt and
> found, as expected, that the file was written much faster.
> 

Hm. I don't have that amount of storage on my test box and I've not
played around that much with multi-device support yet, but I gave your
configuration a quick test with a couple smaller devices in my test
environment.

The first thing I see is that writes seem to hit the foreground target
first, but very quickly move out to the background target. I.e., within
a few seconds of completing a 10GB write with dd, the entire content
seems to have been moved out to the background device.

Is that consistent with what you observe, or is there a longer tail
background copy going on? 'bcachefs fs usage <mnt>' should show how much
data resides on each device, and you can watch it to see if data is
still migrating around.

> Since I occasionally want to use the external disk on another machine, I
> then tried disconnecting the cache from the array.
> 
> Since I didn't really find anything in the documentation about this, I tried
> the following command (after the external disk had come to rest and thus the
> 100GB file was presumably also on the disk):
>     bcachefs device evacuate /dev/nvme0n1p1
> 
> After this command, it took what felt like hours for the external disk to
> come to rest again. Is this normal?
> 

It's not clear to me if the use case is to move the background disk
(sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.
In any event, in my test a 'bcachefs device evacuate' of my foreground
drive completes almost instantly (since data was quickly moved off). If
I repeat the process with my background drive, the userspace tool shows
a progress meter and data migrates off the background device in ~30s or
so. From there, I'm able to 'bcachefs device remove' the empty device
without disruption [1], so I assume that's the appropriate process (if
not, then I'm sure Kent can chime in..).

I'm wondering if the background device is slow enough such the initial
migrate is still going on when the evacuate starts. Do you see fs usage
still adjusting after your copy completes? If not, what does the
evacuate command show? Is it making progress or does it appear stuck? If
the latter, does top show any activity, or is there a consistent stack
trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?

[1] FWIW, I have managed to reproduce a locked up evacuate command in my
attempts to repeat the test a couple times by removing/readding a
device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
report in the usage reporting path:

BUG: unable to handle page fault for address: ffff99e55343b000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0003) - permissions violation
PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
Oops: 0003 [#1] PREEMPT SMP PTI
CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
RIP: 0010:memcpy_erms+0x6/0x10
...
Call Trace:
 <TASK>
 bch2_fs_usage_read+0x12d/0x270 [bcachefs]
...

... so I'll have to see if/how to reproduce that one. It might be wise
to check your dmesg for unexpected events as well.

Brian

> Is this the right command for this case, or do I have to use the command
>   bcachefs device offline /dev/nvme0n1p1
> to disconnect the cache and
>   bcachefs device online /dev/nvme0n1p1
> to reconnect?
> 
> 
> Thanks for your help,
> 
> Heiko
> 
> 
> 
> 
> 
>