bcachefs as a caching filesystem

bcachefs as a caching filesystem

[Dr.-Ing. Heiko Münkel]

Hello,

I am in the process of testing bcachefs. I have two partitions 
/dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as 
cache for the second (is an SSD on a USB3 port).

First I created the bcachefs file system with:
   sudo bcachefs format \
        --label=Shotwell-3 /dev/sde1 \
        --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
        --foreground_target /dev/nvme0n1p1 \
        --promote_target /dev/nvme0n1p1 \
--background_target /dev/sde1
and then mount the disks with
  sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt

This worked fine so far. I then used dd to create a 100GB file on /mnt 
and found, as expected, that the file was written much faster.

Since I occasionally want to use the external disk on another machine, I 
then tried disconnecting the cache from the array.

Since I didn't really find anything in the documentation about this, I 
tried the following command (after the external disk had come to rest 
and thus the 100GB file was presumably also on the disk):
     bcachefs device evacuate /dev/nvme0n1p1

After this command, it took what felt like hours for the external disk 
to come to rest again. Is this normal?

Is this the right command for this case, or do I have to use the command
   bcachefs device offline /dev/nvme0n1p1
to disconnect the cache and
   bcachefs device online /dev/nvme0n1p1
to reconnect?


Thanks for your help,

Heiko

Re: bcachefs as a caching filesystem

[Brian Foster]

On Mon, Apr 10, 2023 at 01:51:07PM +0200, Dr.-Ing. Heiko Münkel wrote:
> Hello,
> 

FYI, there's no need to send multiple mails to the list. If you aren't
sure whether your mail was delivered, you can always reference the
archive at: https://lore.kernel.org/linux-bcachefs/.

> I am in the process of testing bcachefs. I have two partitions
> /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
> for the second (is an SSD on a USB3 port).
> 
> First I created the bcachefs file system with:
>   sudo bcachefs format \
>        --label=Shotwell-3 /dev/sde1 \
>        --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
>        --foreground_target /dev/nvme0n1p1 \
>        --promote_target /dev/nvme0n1p1 \
> --background_target /dev/sde1
> and then mount the disks with
>  sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
> 
> This worked fine so far. I then used dd to create a 100GB file on /mnt and
> found, as expected, that the file was written much faster.
> 

Hm. I don't have that amount of storage on my test box and I've not
played around that much with multi-device support yet, but I gave your
configuration a quick test with a couple smaller devices in my test
environment.

The first thing I see is that writes seem to hit the foreground target
first, but very quickly move out to the background target. I.e., within
a few seconds of completing a 10GB write with dd, the entire content
seems to have been moved out to the background device.

Is that consistent with what you observe, or is there a longer tail
background copy going on? 'bcachefs fs usage <mnt>' should show how much
data resides on each device, and you can watch it to see if data is
still migrating around.

> Since I occasionally want to use the external disk on another machine, I
> then tried disconnecting the cache from the array.
> 
> Since I didn't really find anything in the documentation about this, I tried
> the following command (after the external disk had come to rest and thus the
> 100GB file was presumably also on the disk):
>     bcachefs device evacuate /dev/nvme0n1p1
> 
> After this command, it took what felt like hours for the external disk to
> come to rest again. Is this normal?
> 

It's not clear to me if the use case is to move the background disk
(sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.
In any event, in my test a 'bcachefs device evacuate' of my foreground
drive completes almost instantly (since data was quickly moved off). If
I repeat the process with my background drive, the userspace tool shows
a progress meter and data migrates off the background device in ~30s or
so. From there, I'm able to 'bcachefs device remove' the empty device
without disruption [1], so I assume that's the appropriate process (if
not, then I'm sure Kent can chime in..).

I'm wondering if the background device is slow enough such the initial
migrate is still going on when the evacuate starts. Do you see fs usage
still adjusting after your copy completes? If not, what does the
evacuate command show? Is it making progress or does it appear stuck? If
the latter, does top show any activity, or is there a consistent stack
trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?

[1] FWIW, I have managed to reproduce a locked up evacuate command in my
attempts to repeat the test a couple times by removing/readding a
device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
report in the usage reporting path:

BUG: unable to handle page fault for address: ffff99e55343b000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0003) - permissions violation
PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
Oops: 0003 [#1] PREEMPT SMP PTI
CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
RIP: 0010:memcpy_erms+0x6/0x10
...
Call Trace:
 <TASK>
 bch2_fs_usage_read+0x12d/0x270 [bcachefs]
...

... so I'll have to see if/how to reproduce that one. It might be wise
to check your dmesg for unexpected events as well.

Brian

> Is this the right command for this case, or do I have to use the command
>   bcachefs device offline /dev/nvme0n1p1
> to disconnect the cache and
>   bcachefs device online /dev/nvme0n1p1
> to reconnect?
> 
> 
> Thanks for your help,
> 
> Heiko
> 
> 
> 
> 
> 
> 

Re: bcachefs as a caching filesystem

[Dr.-Ing. Heiko Münkel]

Hello,

On 10.04.23 15:46, Brian Foster wrote:
> FYI, there's no need to send multiple mails to the list. If you aren't
> sure whether your mail was delivered, you can always reference the
> archive at: https://lore.kernel.org/linux-bcachefs/.

Thanks for the hint. I had assumed that I would get the emails from the 
list including my own emails as well. Is that possible?


>> I am in the process of testing bcachefs. I have two partitions
>> /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
>> for the second (is an SSD on a USB3 port).
>>
>> First I created the bcachefs file system with:
>>    sudo bcachefs format \
>>         --label=Shotwell-3 /dev/sde1 \
>>         --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
>>         --foreground_target /dev/nvme0n1p1 \
>>         --promote_target /dev/nvme0n1p1 \
>> --background_target /dev/sde1
>> and then mount the disks with
>>   sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
>>
>> This worked fine so far. I then used dd to create a 100GB file on /mnt and
>> found, as expected, that the file was written much faster.
>>
> Hm. I don't have that amount of storage on my test box and I've not
> played around that much with multi-device support yet, but I gave your
> configuration a quick test with a couple smaller devices in my test
> environment.
>
> The first thing I see is that writes seem to hit the foreground target
> first, but very quickly move out to the background target. I.e., within
> a few seconds of completing a 10GB write with dd, the entire content
> seems to have been moved out to the background device.
>
> Is that consistent with what you observe, or is there a longer tail
> background copy going on? 'bcachefs fs usage <mnt>' should show how much
> data resides on each device, and you can watch it to see if data is
> still migrating around.

Yes, it's the same here.

>> Since I occasionally want to use the external disk on another machine, I
>> then tried disconnecting the cache from the array.
>>
>> Since I didn't really find anything in the documentation about this, I tried
>> the following command (after the external disk had come to rest and thus the
>> 100GB file was presumably also on the disk):
>>      bcachefs device evacuate /dev/nvme0n1p1
>>
>> After this command, it took what felt like hours for the external disk to
>> come to rest again. Is this normal?
>>
> It's not clear to me if the use case is to move the background disk
> (sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.

The goal is to occasionally use the external (background) disk on 
another PC. For this I need to be able to disconnect it from the cache 
and add it again later.


> In any event, in my test a 'bcachefs device evacuate' of my foreground
> drive completes almost instantly (since data was quickly moved off). If
> I repeat the process with my background drive, the userspace tool shows
> a progress meter and data migrates off the background device in ~30s or
> so. From there, I'm able to 'bcachefs device remove' the empty device
> without disruption [1], so I assume that's the appropriate process (if
> not, then I'm sure Kent can chime in..).

I thought that it is enough to evacuate the disk with the cache 
(foreground disk). Isn't it?

> I'm wondering if the background device is slow enough such the initial
> migrate is still going on when the evacuate starts. Do you see fs usage
> still adjusting after your copy completes? If not, what does the
> evacuate command show? Is it making progress or does it appear stuck? If
> the latter, does top show any activity, or is there a consistent stack
> trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?

I've repeated the test, but now with only 10GB. The evacuate command 
needed only a few second to evacuate the foreground disk, but the 
background disk was again still working after the command finished (it's 
LED was still flickering for at least 36 minutes).

The background disk is also a SSD, so it's not realy slow.

The fs usage was still adjusting after the evacuate command ended. The 
number of buckets increased very slowly and the fragmented btree size 
droped also very slowly.

>
> [1] FWIW, I have managed to reproduce a locked up evacuate command in my
> attempts to repeat the test a couple times by removing/readding a
> device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
> report in the usage reporting path:
>
> BUG: unable to handle page fault for address: ffff99e55343b000
> #PF: supervisor write access in kernel mode
> #PF: error_code(0x0003) - permissions violation
> PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
> Oops: 0003 [#1] PREEMPT SMP PTI
> CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
> RIP: 0010:memcpy_erms+0x6/0x10
> ...
> Call Trace:
>   <TASK>
>   bch2_fs_usage_read+0x12d/0x270 [bcachefs]
> ...
>
> ... so I'll have to see if/how to reproduce that one. It might be wise
> to check your dmesg for unexpected events as well.
I didn't see this bug entry.
>
> Brian
>
>> Is this the right command for this case, or do I have to use the command
>>    bcachefs device offline /dev/nvme0n1p1
>> to disconnect the cache and
>>    bcachefs device online /dev/nvme0n1p1
>> to reconnect?
>>
>>
>> Thanks for your help,
>>
>> Heiko
>>
>>
>>
>>
>>
>>

Re: bcachefs as a caching filesystem

[Brian Foster]

On Tue, Apr 11, 2023 at 11:20:31PM +0200, Dr.-Ing. Heiko Münkel wrote:
> Hello,
> 
> On 10.04.23 15:46, Brian Foster wrote:
> > FYI, there's no need to send multiple mails to the list. If you aren't
> > sure whether your mail was delivered, you can always reference the
> > archive at: https://lore.kernel.org/linux-bcachefs/.
> 
> Thanks for the hint. I had assumed that I would get the emails from the list
> including my own emails as well. Is that possible?
> 
> 
> > > I am in the process of testing bcachefs. I have two partitions
> > > /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
> > > for the second (is an SSD on a USB3 port).
> > > 
> > > First I created the bcachefs file system with:
> > >    sudo bcachefs format \
> > >         --label=Shotwell-3 /dev/sde1 \
> > >         --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
> > >         --foreground_target /dev/nvme0n1p1 \
> > >         --promote_target /dev/nvme0n1p1 \
> > > --background_target /dev/sde1
> > > and then mount the disks with
> > >   sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
> > > 
> > > This worked fine so far. I then used dd to create a 100GB file on /mnt and
> > > found, as expected, that the file was written much faster.
> > > 
> > Hm. I don't have that amount of storage on my test box and I've not
> > played around that much with multi-device support yet, but I gave your
> > configuration a quick test with a couple smaller devices in my test
> > environment.
> > 
> > The first thing I see is that writes seem to hit the foreground target
> > first, but very quickly move out to the background target. I.e., within
> > a few seconds of completing a 10GB write with dd, the entire content
> > seems to have been moved out to the background device.
> > 
> > Is that consistent with what you observe, or is there a longer tail
> > background copy going on? 'bcachefs fs usage <mnt>' should show how much
> > data resides on each device, and you can watch it to see if data is
> > still migrating around.
> 
> Yes, it's the same here.
> 
> > > Since I occasionally want to use the external disk on another machine, I
> > > then tried disconnecting the cache from the array.
> > > 
> > > Since I didn't really find anything in the documentation about this, I tried
> > > the following command (after the external disk had come to rest and thus the
> > > 100GB file was presumably also on the disk):
> > >      bcachefs device evacuate /dev/nvme0n1p1
> > > 
> > > After this command, it took what felt like hours for the external disk to
> > > come to rest again. Is this normal?
> > > 
> > It's not clear to me if the use case is to move the background disk
> > (sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.
> 
> The goal is to occasionally use the external (background) disk on another
> PC. For this I need to be able to disconnect it from the cache and add it
> again later.
> 

Hmm. IIUC, the foreground and promote targets specified above
essentially mark nvme0n1p1 as a fast/cache device and sde1 as the slower
background device. I'm a little confused why you would want to use sde1
in this sort of configuration. Assuming data is always going to flow
back into sde1 when it's part of the bcachefs volume, you'd end up
having to migrate data off and back on during every disconnect cycle.

That means that the foreground (nvme) device still has to be big enough
to hold all of the data in the fs, so what benefit is there to using
sde1 as a background device like this? Am I misunderstanding something
about the configuration or use case?

> 
> > In any event, in my test a 'bcachefs device evacuate' of my foreground
> > drive completes almost instantly (since data was quickly moved off). If
> > I repeat the process with my background drive, the userspace tool shows
> > a progress meter and data migrates off the background device in ~30s or
> > so. From there, I'm able to 'bcachefs device remove' the empty device
> > without disruption [1], so I assume that's the appropriate process (if
> > not, then I'm sure Kent can chime in..).
> 
> I thought that it is enough to evacuate the disk with the cache (foreground
> disk). Isn't it?
> 

That seems to be the case from my tests. Evacuate should mark the device
read-only and migrate data off so when complete, the device can be
explicitly removed from the volume.

> > I'm wondering if the background device is slow enough such the initial
> > migrate is still going on when the evacuate starts. Do you see fs usage
> > still adjusting after your copy completes? If not, what does the
> > evacuate command show? Is it making progress or does it appear stuck? If
> > the latter, does top show any activity, or is there a consistent stack
> > trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?
> 
> I've repeated the test, but now with only 10GB. The evacuate command needed
> only a few second to evacuate the foreground disk, but the background disk
> was again still working after the command finished (it's LED was still
> flickering for at least 36 minutes).
> 
> The background disk is also a SSD, so it's not realy slow.
> 
> The fs usage was still adjusting after the evacuate command ended. The
> number of buckets increased very slowly and the fragmented btree size droped
> also very slowly.
> 

I've been playing around with some evac-remove-add tests the past few
days and have reproduced some varying potentially problematic behaviors
(that I haven't had a chance to dig into yet). I've seen some cases
where data copies, but it seems like there's some background discard
work going on keeping the device busy. In other cases it seems like data
partially copies, then grinds down to a halt for some unknown reason.

FWIW, I do seem to be able to evac data back and forth between the two
devices without any issues at all if I don't actually remove, but once I
remove and re-add devices, things start to behave a bit odd, including
fsck starting to complain about various things. I'll probably have to
look into some of this before I can comment further...

Brian

> > 
> > [1] FWIW, I have managed to reproduce a locked up evacuate command in my
> > attempts to repeat the test a couple times by removing/readding a
> > device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
> > report in the usage reporting path:
> > 
> > BUG: unable to handle page fault for address: ffff99e55343b000
> > #PF: supervisor write access in kernel mode
> > #PF: error_code(0x0003) - permissions violation
> > PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
> > Oops: 0003 [#1] PREEMPT SMP PTI
> > CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
> > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
> > RIP: 0010:memcpy_erms+0x6/0x10
> > ...
> > Call Trace:
> >   <TASK>
> >   bch2_fs_usage_read+0x12d/0x270 [bcachefs]
> > ...
> > 
> > ... so I'll have to see if/how to reproduce that one. It might be wise
> > to check your dmesg for unexpected events as well.
> I didn't see this bug entry.
> > 
> > Brian
> > 
> > > Is this the right command for this case, or do I have to use the command
> > >    bcachefs device offline /dev/nvme0n1p1
> > > to disconnect the cache and
> > >    bcachefs device online /dev/nvme0n1p1
> > > to reconnect?
> > > 
> > > 
> > > Thanks for your help,
> > > 
> > > Heiko
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> 

Re: bcachefs as a caching filesystem

[Dr.-Ing. Heiko Münkel]

On 13.04.23 20:37, Brian Foster wrote:
> On Tue, Apr 11, 2023 at 11:20:31PM +0200, Dr.-Ing. Heiko Münkel wrote:
>> Hello,
>>
>> On 10.04.23 15:46, Brian Foster wrote:
>>> FYI, there's no need to send multiple mails to the list. If you aren't
>>> sure whether your mail was delivered, you can always reference the
>>> archive at: https://lore.kernel.org/linux-bcachefs/.
>> Thanks for the hint. I had assumed that I would get the emails from the list
>> including my own emails as well. Is that possible?
>>
>>
>>>> I am in the process of testing bcachefs. I have two partitions
>>>> /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
>>>> for the second (is an SSD on a USB3 port).
>>>>
>>>> First I created the bcachefs file system with:
>>>>     sudo bcachefs format \
>>>>          --label=Shotwell-3 /dev/sde1 \
>>>>          --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
>>>>          --foreground_target /dev/nvme0n1p1 \
>>>>          --promote_target /dev/nvme0n1p1 \
>>>> --background_target /dev/sde1
>>>> and then mount the disks with
>>>>    sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
>>>>
>>>> This worked fine so far. I then used dd to create a 100GB file on /mnt and
>>>> found, as expected, that the file was written much faster.
>>>>
>>> Hm. I don't have that amount of storage on my test box and I've not
>>> played around that much with multi-device support yet, but I gave your
>>> configuration a quick test with a couple smaller devices in my test
>>> environment.
>>>
>>> The first thing I see is that writes seem to hit the foreground target
>>> first, but very quickly move out to the background target. I.e., within
>>> a few seconds of completing a 10GB write with dd, the entire content
>>> seems to have been moved out to the background device.
>>>
>>> Is that consistent with what you observe, or is there a longer tail
>>> background copy going on? 'bcachefs fs usage <mnt>' should show how much
>>> data resides on each device, and you can watch it to see if data is
>>> still migrating around.
>> Yes, it's the same here.
>>
>>>> Since I occasionally want to use the external disk on another machine, I
>>>> then tried disconnecting the cache from the array.
>>>>
>>>> Since I didn't really find anything in the documentation about this, I tried
>>>> the following command (after the external disk had come to rest and thus the
>>>> 100GB file was presumably also on the disk):
>>>>       bcachefs device evacuate /dev/nvme0n1p1
>>>>
>>>> After this command, it took what felt like hours for the external disk to
>>>> come to rest again. Is this normal?
>>>>
>>> It's not clear to me if the use case is to move the background disk
>>> (sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.
>> The goal is to occasionally use the external (background) disk on another
>> PC. For this I need to be able to disconnect it from the cache and add it
>> again later.
>>
> Hmm. IIUC, the foreground and promote targets specified above
> essentially mark nvme0n1p1 as a fast/cache device and sde1 as the slower
> background device. I'm a little confused why you would want to use sde1
> in this sort of configuration. Assuming data is always going to flow
> back into sde1 when it's part of the bcachefs volume, you'd end up
> having to migrate data off and back on during every disconnect cycle.
>
> That means that the foreground (nvme) device still has to be big enough
> to hold all of the data in the fs, so what benefit is there to using
> sde1 as a background device like this? Am I misunderstanding something
> about the configuration or use case?

My idea was to take the faster internal (500GB size) disk nvme0n1p1 as a 
cache for the slower external (8TB size) disk sde1. This way I wanted to 
make it faster to work with the data on the external disk when it is 
attached to my PC. Occasionally I want to attach the external disk to my 
laptop, which of course then lacks the cache disk.
I assumed that the cache disk can be smaller than the "actual" data 
disk, similar to the internal cache of a magnetic disk, for example.
If I understood you correctly, the cache disk would have to be the same 
size as the main disk. Then the whole thing probably doesn't make sense.

>
>>> In any event, in my test a 'bcachefs device evacuate' of my foreground
>>> drive completes almost instantly (since data was quickly moved off). If
>>> I repeat the process with my background drive, the userspace tool shows
>>> a progress meter and data migrates off the background device in ~30s or
>>> so. From there, I'm able to 'bcachefs device remove' the empty device
>>> without disruption [1], so I assume that's the appropriate process (if
>>> not, then I'm sure Kent can chime in..).
>> I thought that it is enough to evacuate the disk with the cache (foreground
>> disk). Isn't it?
>>
> That seems to be the case from my tests. Evacuate should mark the device
> read-only and migrate data off so when complete, the device can be
> explicitly removed from the volume.
>
>>> I'm wondering if the background device is slow enough such the initial
>>> migrate is still going on when the evacuate starts. Do you see fs usage
>>> still adjusting after your copy completes? If not, what does the
>>> evacuate command show? Is it making progress or does it appear stuck? If
>>> the latter, does top show any activity, or is there a consistent stack
>>> trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?
>> I've repeated the test, but now with only 10GB. The evacuate command needed
>> only a few second to evacuate the foreground disk, but the background disk
>> was again still working after the command finished (it's LED was still
>> flickering for at least 36 minutes).
>>
>> The background disk is also a SSD, so it's not realy slow.
>>
>> The fs usage was still adjusting after the evacuate command ended. The
>> number of buckets increased very slowly and the fragmented btree size droped
>> also very slowly.
>>
> I've been playing around with some evac-remove-add tests the past few
> days and have reproduced some varying potentially problematic behaviors
> (that I haven't had a chance to dig into yet). I've seen some cases
> where data copies, but it seems like there's some background discard
> work going on keeping the device busy. In other cases it seems like data
> partially copies, then grinds down to a halt for some unknown reason.

I've seen in the output of the fs usage command, that some values were 
changing after the end of the evacuate command and before the external 
disk stops working. For me it seems, that the bcachefs driver tried to 
defragment the disk, although I don't understand, why the disk should be 
fragmented in that situation. I had only copied one big file on the disk 
and ran the evacute after it.

>
> FWIW, I do seem to be able to evac data back and forth between the two
> devices without any issues at all if I don't actually remove, but once I
> remove and re-add devices, things start to behave a bit odd, including
> fsck starting to complain about various things. I'll probably have to
> look into some of this before I can comment further...
>
> Brian
>
>>> [1] FWIW, I have managed to reproduce a locked up evacuate command in my
>>> attempts to repeat the test a couple times by removing/readding a
>>> device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
>>> report in the usage reporting path:
>>>
>>> BUG: unable to handle page fault for address: ffff99e55343b000
>>> #PF: supervisor write access in kernel mode
>>> #PF: error_code(0x0003) - permissions violation
>>> PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
>>> Oops: 0003 [#1] PREEMPT SMP PTI
>>> CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
>>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
>>> RIP: 0010:memcpy_erms+0x6/0x10
>>> ...
>>> Call Trace:
>>>    <TASK>
>>>    bch2_fs_usage_read+0x12d/0x270 [bcachefs]
>>> ...
>>>
>>> ... so I'll have to see if/how to reproduce that one. It might be wise
>>> to check your dmesg for unexpected events as well.
>> I didn't see this bug entry.
>>> Brian
>>>
>>>> Is this the right command for this case, or do I have to use the command
>>>>     bcachefs device offline /dev/nvme0n1p1
>>>> to disconnect the cache and
>>>>     bcachefs device online /dev/nvme0n1p1
>>>> to reconnect?
>>>>
>>>>
>>>> Thanks for your help,
>>>>
>>>> Heiko
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>

Re: bcachefs as a caching filesystem

[Brian Foster]

On Sat, Apr 15, 2023 at 06:54:42PM +0200, Dr.-Ing. Heiko Münkel wrote:
> 
> On 13.04.23 20:37, Brian Foster wrote:
> > On Tue, Apr 11, 2023 at 11:20:31PM +0200, Dr.-Ing. Heiko Münkel wrote:
> > > Hello,
> > > 
> > > On 10.04.23 15:46, Brian Foster wrote:
> > > > FYI, there's no need to send multiple mails to the list. If you aren't
> > > > sure whether your mail was delivered, you can always reference the
> > > > archive at: https://lore.kernel.org/linux-bcachefs/.
> > > Thanks for the hint. I had assumed that I would get the emails from the list
> > > including my own emails as well. Is that possible?
> > > 
> > > 
> > > > > I am in the process of testing bcachefs. I have two partitions
> > > > > /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
> > > > > for the second (is an SSD on a USB3 port).
> > > > > 
> > > > > First I created the bcachefs file system with:
> > > > >     sudo bcachefs format \
> > > > >          --label=Shotwell-3 /dev/sde1 \
> > > > >          --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
> > > > >          --foreground_target /dev/nvme0n1p1 \
> > > > >          --promote_target /dev/nvme0n1p1 \
> > > > > --background_target /dev/sde1
> > > > > and then mount the disks with
> > > > >    sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
> > > > > 
> > > > > This worked fine so far. I then used dd to create a 100GB file on /mnt and
> > > > > found, as expected, that the file was written much faster.
> > > > > 
> > > > Hm. I don't have that amount of storage on my test box and I've not
> > > > played around that much with multi-device support yet, but I gave your
> > > > configuration a quick test with a couple smaller devices in my test
> > > > environment.
> > > > 
> > > > The first thing I see is that writes seem to hit the foreground target
> > > > first, but very quickly move out to the background target. I.e., within
> > > > a few seconds of completing a 10GB write with dd, the entire content
> > > > seems to have been moved out to the background device.
> > > > 
> > > > Is that consistent with what you observe, or is there a longer tail
> > > > background copy going on? 'bcachefs fs usage <mnt>' should show how much
> > > > data resides on each device, and you can watch it to see if data is
> > > > still migrating around.
> > > Yes, it's the same here.
> > > 
> > > > > Since I occasionally want to use the external disk on another machine, I
> > > > > then tried disconnecting the cache from the array.
> > > > > 
> > > > > Since I didn't really find anything in the documentation about this, I tried
> > > > > the following command (after the external disk had come to rest and thus the
> > > > > 100GB file was presumably also on the disk):
> > > > >       bcachefs device evacuate /dev/nvme0n1p1
> > > > > 
> > > > > After this command, it took what felt like hours for the external disk to
> > > > > come to rest again. Is this normal?
> > > > > 
> > > > It's not clear to me if the use case is to move the background disk
> > > > (sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.
> > > The goal is to occasionally use the external (background) disk on another
> > > PC. For this I need to be able to disconnect it from the cache and add it
> > > again later.
> > > 
> > Hmm. IIUC, the foreground and promote targets specified above
> > essentially mark nvme0n1p1 as a fast/cache device and sde1 as the slower
> > background device. I'm a little confused why you would want to use sde1
> > in this sort of configuration. Assuming data is always going to flow
> > back into sde1 when it's part of the bcachefs volume, you'd end up
> > having to migrate data off and back on during every disconnect cycle.
> > 
> > That means that the foreground (nvme) device still has to be big enough
> > to hold all of the data in the fs, so what benefit is there to using
> > sde1 as a background device like this? Am I misunderstanding something
> > about the configuration or use case?
> 
> My idea was to take the faster internal (500GB size) disk nvme0n1p1 as a
> cache for the slower external (8TB size) disk sde1. This way I wanted to
> make it faster to work with the data on the external disk when it is
> attached to my PC. Occasionally I want to attach the external disk to my
> laptop, which of course then lacks the cache disk.
> I assumed that the cache disk can be smaller than the "actual" data disk,
> similar to the internal cache of a magnetic disk, for example.
> If I understood you correctly, the cache disk would have to be the same size
> as the main disk. Then the whole thing probably doesn't make sense.
> 

Ok. I was confused about whether you're trying to repurpose this device
to be used for something else or just to access the bcachefs data from
another system. It sounds more like the latter. Therefore, you
remove/drain the cache device from the volume such that everything
flushes to the slower backing dev, then can relocate that back and
forth. Sorry for the confusion.

With my limited knowledge of the interface and mechanism, I think your
use case makes sense and doesn't have any special size requirements for
the cache device. It just seems that frequent device
migrate/remove/re-add sequences might have some bugs that need attention
so this can be done reliably.

> > 
> > > > In any event, in my test a 'bcachefs device evacuate' of my foreground
> > > > drive completes almost instantly (since data was quickly moved off). If
> > > > I repeat the process with my background drive, the userspace tool shows
> > > > a progress meter and data migrates off the background device in ~30s or
> > > > so. From there, I'm able to 'bcachefs device remove' the empty device
> > > > without disruption [1], so I assume that's the appropriate process (if
> > > > not, then I'm sure Kent can chime in..).
> > > I thought that it is enough to evacuate the disk with the cache (foreground
> > > disk). Isn't it?
> > > 
> > That seems to be the case from my tests. Evacuate should mark the device
> > read-only and migrate data off so when complete, the device can be
> > explicitly removed from the volume.
> > 
> > > > I'm wondering if the background device is slow enough such the initial
> > > > migrate is still going on when the evacuate starts. Do you see fs usage
> > > > still adjusting after your copy completes? If not, what does the
> > > > evacuate command show? Is it making progress or does it appear stuck? If
> > > > the latter, does top show any activity, or is there a consistent stack
> > > > trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?
> > > I've repeated the test, but now with only 10GB. The evacuate command needed
> > > only a few second to evacuate the foreground disk, but the background disk
> > > was again still working after the command finished (it's LED was still
> > > flickering for at least 36 minutes).
> > > 
> > > The background disk is also a SSD, so it's not realy slow.
> > > 
> > > The fs usage was still adjusting after the evacuate command ended. The
> > > number of buckets increased very slowly and the fragmented btree size droped
> > > also very slowly.
> > > 
> > I've been playing around with some evac-remove-add tests the past few
> > days and have reproduced some varying potentially problematic behaviors
> > (that I haven't had a chance to dig into yet). I've seen some cases
> > where data copies, but it seems like there's some background discard
> > work going on keeping the device busy. In other cases it seems like data
> > partially copies, then grinds down to a halt for some unknown reason.
> 
> I've seen in the output of the fs usage command, that some values were
> changing after the end of the evacuate command and before the external disk
> stops working. For me it seems, that the bcachefs driver tried to defragment
> the disk, although I don't understand, why the disk should be fragmented in
> that situation. I had only copied one big file on the disk and ran the
> evacute after it.
> 

I suppose it's possible some background activity is going on. I couldn't
say off the top of my head. I want to say I saw some discard activity on
some of my tests, for example. The more interesting question is that if
the evacuate completes, does any of this prevent device removal?

Brian

> > 
> > FWIW, I do seem to be able to evac data back and forth between the two
> > devices without any issues at all if I don't actually remove, but once I
> > remove and re-add devices, things start to behave a bit odd, including
> > fsck starting to complain about various things. I'll probably have to
> > look into some of this before I can comment further...
> > 
> > Brian
> > 
> > > > [1] FWIW, I have managed to reproduce a locked up evacuate command in my
> > > > attempts to repeat the test a couple times by removing/readding a
> > > > device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
> > > > report in the usage reporting path:
> > > > 
> > > > BUG: unable to handle page fault for address: ffff99e55343b000
> > > > #PF: supervisor write access in kernel mode
> > > > #PF: error_code(0x0003) - permissions violation
> > > > PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
> > > > Oops: 0003 [#1] PREEMPT SMP PTI
> > > > CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
> > > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
> > > > RIP: 0010:memcpy_erms+0x6/0x10
> > > > ...
> > > > Call Trace:
> > > >    <TASK>
> > > >    bch2_fs_usage_read+0x12d/0x270 [bcachefs]
> > > > ...
> > > > 
> > > > ... so I'll have to see if/how to reproduce that one. It might be wise
> > > > to check your dmesg for unexpected events as well.
> > > I didn't see this bug entry.
> > > > Brian
> > > > 
> > > > > Is this the right command for this case, or do I have to use the command
> > > > >     bcachefs device offline /dev/nvme0n1p1
> > > > > to disconnect the cache and
> > > > >     bcachefs device online /dev/nvme0n1p1
> > > > > to reconnect?
> > > > > 
> > > > > 
> > > > > Thanks for your help,
> > > > > 
> > > > > Heiko
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> 

Re: bcachefs as a caching filesystem

[Dr.-Ing. Heiko Münkel]

On 17.04.23 17:52, Brian Foster wrote:
> On Sat, Apr 15, 2023 at 06:54:42PM +0200, Dr.-Ing. Heiko Münkel wrote:
>> On 13.04.23 20:37, Brian Foster wrote:
>>> On Tue, Apr 11, 2023 at 11:20:31PM +0200, Dr.-Ing. Heiko Münkel wrote:
>>>> Hello,
>>>>
>>>> On 10.04.23 15:46, Brian Foster wrote:
>>>>> FYI, there's no need to send multiple mails to the list. If you aren't
>>>>> sure whether your mail was delivered, you can always reference the
>>>>> archive at: https://lore.kernel.org/linux-bcachefs/.
>>>> Thanks for the hint. I had assumed that I would get the emails from the list
>>>> including my own emails as well. Is that possible?
>>>>
>>>>
>>>>>> I am in the process of testing bcachefs. I have two partitions
>>>>>> /dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as cache
>>>>>> for the second (is an SSD on a USB3 port).
>>>>>>
>>>>>> First I created the bcachefs file system with:
>>>>>>      sudo bcachefs format \
>>>>>>           --label=Shotwell-3 /dev/sde1 \
>>>>>>           --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
>>>>>>           --foreground_target /dev/nvme0n1p1 \
>>>>>>           --promote_target /dev/nvme0n1p1 \
>>>>>> --background_target /dev/sde1
>>>>>> and then mount the disks with
>>>>>>     sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt
>>>>>>
>>>>>> This worked fine so far. I then used dd to create a 100GB file on /mnt and
>>>>>> found, as expected, that the file was written much faster.
>>>>>>
>>>>> Hm. I don't have that amount of storage on my test box and I've not
>>>>> played around that much with multi-device support yet, but I gave your
>>>>> configuration a quick test with a couple smaller devices in my test
>>>>> environment.
>>>>>
>>>>> The first thing I see is that writes seem to hit the foreground target
>>>>> first, but very quickly move out to the background target. I.e., within
>>>>> a few seconds of completing a 10GB write with dd, the entire content
>>>>> seems to have been moved out to the background device.
>>>>>
>>>>> Is that consistent with what you observe, or is there a longer tail
>>>>> background copy going on? 'bcachefs fs usage <mnt>' should show how much
>>>>> data resides on each device, and you can watch it to see if data is
>>>>> still migrating around.
>>>> Yes, it's the same here.
>>>>
>>>>>> Since I occasionally want to use the external disk on another machine, I
>>>>>> then tried disconnecting the cache from the array.
>>>>>>
>>>>>> Since I didn't really find anything in the documentation about this, I tried
>>>>>> the following command (after the external disk had come to rest and thus the
>>>>>> 100GB file was presumably also on the disk):
>>>>>>        bcachefs device evacuate /dev/nvme0n1p1
>>>>>>
>>>>>> After this command, it took what felt like hours for the external disk to
>>>>>> come to rest again. Is this normal?
>>>>>>
>>>>> It's not clear to me if the use case is to move the background disk
>>>>> (sde1 w/ bcachefs+data) or to clear out the nvme drive to be repurposed.
>>>> The goal is to occasionally use the external (background) disk on another
>>>> PC. For this I need to be able to disconnect it from the cache and add it
>>>> again later.
>>>>
>>> Hmm. IIUC, the foreground and promote targets specified above
>>> essentially mark nvme0n1p1 as a fast/cache device and sde1 as the slower
>>> background device. I'm a little confused why you would want to use sde1
>>> in this sort of configuration. Assuming data is always going to flow
>>> back into sde1 when it's part of the bcachefs volume, you'd end up
>>> having to migrate data off and back on during every disconnect cycle.
>>>
>>> That means that the foreground (nvme) device still has to be big enough
>>> to hold all of the data in the fs, so what benefit is there to using
>>> sde1 as a background device like this? Am I misunderstanding something
>>> about the configuration or use case?
>> My idea was to take the faster internal (500GB size) disk nvme0n1p1 as a
>> cache for the slower external (8TB size) disk sde1. This way I wanted to
>> make it faster to work with the data on the external disk when it is
>> attached to my PC. Occasionally I want to attach the external disk to my
>> laptop, which of course then lacks the cache disk.
>> I assumed that the cache disk can be smaller than the "actual" data disk,
>> similar to the internal cache of a magnetic disk, for example.
>> If I understood you correctly, the cache disk would have to be the same size
>> as the main disk. Then the whole thing probably doesn't make sense.
>>
> Ok. I was confused about whether you're trying to repurpose this device
> to be used for something else or just to access the bcachefs data from
> another system. It sounds more like the latter. Therefore, you
> remove/drain the cache device from the volume such that everything
> flushes to the slower backing dev, then can relocate that back and
> forth. Sorry for the confusion.
>
> With my limited knowledge of the interface and mechanism, I think your
> use case makes sense and doesn't have any special size requirements for
> the cache device. It just seems that frequent device
> migrate/remove/re-add sequences might have some bugs that need attention
> so this can be done reliably.
>
>>>>> In any event, in my test a 'bcachefs device evacuate' of my foreground
>>>>> drive completes almost instantly (since data was quickly moved off). If
>>>>> I repeat the process with my background drive, the userspace tool shows
>>>>> a progress meter and data migrates off the background device in ~30s or
>>>>> so. From there, I'm able to 'bcachefs device remove' the empty device
>>>>> without disruption [1], so I assume that's the appropriate process (if
>>>>> not, then I'm sure Kent can chime in..).
>>>> I thought that it is enough to evacuate the disk with the cache (foreground
>>>> disk). Isn't it?
>>>>
>>> That seems to be the case from my tests. Evacuate should mark the device
>>> read-only and migrate data off so when complete, the device can be
>>> explicitly removed from the volume.
>>>
>>>>> I'm wondering if the background device is slow enough such the initial
>>>>> migrate is still going on when the evacuate starts. Do you see fs usage
>>>>> still adjusting after your copy completes? If not, what does the
>>>>> evacuate command show? Is it making progress or does it appear stuck? If
>>>>> the latter, does top show any activity, or is there a consistent stack
>>>>> trace shown in /proc/<pid of bcachefs evacuate cmd>/stack?
>>>> I've repeated the test, but now with only 10GB. The evacuate command needed
>>>> only a few second to evacuate the foreground disk, but the background disk
>>>> was again still working after the command finished (it's LED was still
>>>> flickering for at least 36 minutes).
>>>>
>>>> The background disk is also a SSD, so it's not realy slow.
>>>>
>>>> The fs usage was still adjusting after the evacuate command ended. The
>>>> number of buckets increased very slowly and the fragmented btree size droped
>>>> also very slowly.
>>>>
>>> I've been playing around with some evac-remove-add tests the past few
>>> days and have reproduced some varying potentially problematic behaviors
>>> (that I haven't had a chance to dig into yet). I've seen some cases
>>> where data copies, but it seems like there's some background discard
>>> work going on keeping the device busy. In other cases it seems like data
>>> partially copies, then grinds down to a halt for some unknown reason.
>> I've seen in the output of the fs usage command, that some values were
>> changing after the end of the evacuate command and before the external disk
>> stops working. For me it seems, that the bcachefs driver tried to defragment
>> the disk, although I don't understand, why the disk should be fragmented in
>> that situation. I had only copied one big file on the disk and ran the
>> evacute after it.
>>
> I suppose it's possible some background activity is going on. I couldn't
> say off the top of my head. I want to say I saw some discard activity on
> some of my tests, for example. The more interesting question is that if
> the evacuate completes, does any of this prevent device removal?

No, I was able to remove the device after the evacuate stopped and 
before the drive activity stopped.


Heiko


>
> Brian
>
>>> FWIW, I do seem to be able to evac data back and forth between the two
>>> devices without any issues at all if I don't actually remove, but once I
>>> remove and re-add devices, things start to behave a bit odd, including
>>> fsck starting to complain about various things. I'll probably have to
>>> look into some of this before I can comment further...
>>>
>>> Brian
>>>
>>>>> [1] FWIW, I have managed to reproduce a locked up evacuate command in my
>>>>> attempts to repeat the test a couple times by removing/readding a
>>>>> device, monitoring usage, etc. Looking at my syslogs, I've hit a BUG
>>>>> report in the usage reporting path:
>>>>>
>>>>> BUG: unable to handle page fault for address: ffff99e55343b000
>>>>> #PF: supervisor write access in kernel mode
>>>>> #PF: error_code(0x0003) - permissions violation
>>>>> PGD 213e01067 P4D 213e01067 PUD 101c33063 PMD 113408063 PTE 800000011343b061
>>>>> Oops: 0003 [#1] PREEMPT SMP PTI
>>>>> CPU: 60 PID: 7440 Comm: bcachefs Tainted: G            E      6.2.0+ #30
>>>>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.1-2.fc36 04/01/2014
>>>>> RIP: 0010:memcpy_erms+0x6/0x10
>>>>> ...
>>>>> Call Trace:
>>>>>     <TASK>
>>>>>     bch2_fs_usage_read+0x12d/0x270 [bcachefs]
>>>>> ...
>>>>>
>>>>> ... so I'll have to see if/how to reproduce that one. It might be wise
>>>>> to check your dmesg for unexpected events as well.
>>>> I didn't see this bug entry.
>>>>> Brian
>>>>>
>>>>>> Is this the right command for this case, or do I have to use the command
>>>>>>      bcachefs device offline /dev/nvme0n1p1
>>>>>> to disconnect the cache and
>>>>>>      bcachefs device online /dev/nvme0n1p1
>>>>>> to reconnect?
>>>>>>
>>>>>>
>>>>>> Thanks for your help,
>>>>>>
>>>>>> Heiko
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>

bcachefs as a caching filesystem

[Dr.-Ing. Heiko Münkel]

Hello,

I am in the process of testing bcachefs. I have two partitions 
/dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as 
cache for the second (is an SSD on a USB3 port).

First I created the bcachefs file system with:
   sudo bcachefs format \
        --label=Shotwell-3 /dev/sde1 \
        --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
        --foreground_target /dev/nvme0n1p1 \
        --promote_target /dev/nvme0n1p1 \
--background_target /dev/sde1
and then mount the disks with
  sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt

This worked fine so far. I then used dd to create a 100GB file on /mnt 
and found, as expected, that the file was written much faster.

Since I occasionally want to use the external disk on another machine, I 
then tried disconnecting the cache from the array.

Since I didn't really find anything in the documentation about this, I 
tried the following command (after the external disk had come to rest 
and thus the 100GB file was presumably also on the disk):
     bcachefs device evacuate /dev/nvme0n1p1

After this command, it took what felt like hours for the external disk 
to come to rest again. Is this normal?

Is this the right command for this case, or do I have to use the command
   bcachefs device offline /dev/nvme0n1p1
to disconnect the cache and
   bcachefs device online /dev/nvme0n1p1
to reconnect?


Thanks for your help,

Heiko

bcachefs as a caching filesystem

[Dr.-Ing. Heiko Münkel]

Hello,

I am in the process of testing bcachefs. I have two partitions 
/dev/nvme0n1p1 and /dev/sde1. The first (internal SSD) should be as 
cache for the second (is an SSD on a USB3 port).

First I created the bcachefs file system with:
   sudo bcachefs format \
        --label=Shotwell-3 /dev/sde1 \
        --label=Shotwell-Cache-1 /dev/nvme0n1p1 \
        --foreground_target /dev/nvme0n1p1 \
        --promote_target /dev/nvme0n1p1 \
--background_target /dev/sde1
and then mount the disks with
  sudo mount -t bcachefs /dev/nvme0n1p1:/dev/sde1 /mnt

This worked fine so far. I then used dd to create a 100GB file on /mnt 
and found, as expected, that the file was written much faster.

Since I occasionally want to use the external disk on another machine, I 
then tried disconnecting the cache from the array.

Since I didn't really find anything in the documentation about this, I 
tried the following command (after the external disk had come to rest 
and thus the 100GB file was presumably also on the disk):
     bcachefs device evacuate /dev/nvme0n1p1

After this command, it took what felt like hours for the external disk 
to come to rest again. Is this normal?

Is this the right command for this case, or do I have to use the command
   bcachefs device offline /dev/nvme0n1p1
to disconnect the cache and
   bcachefs device online /dev/nvme0n1p1
to reconnect?


Thanks for your help,

Heiko