Question about 32 bit limitations

Question about 32 bit limitations

[David Schiller]

Hi!

I'm running a four disk "raid1" array on old Atom-based NAS hardware (no
64 bit compatibility). It's a mixed set of disks (2x 4T, 2x 6T) and
"btrfs filesystem usage" reports around 18 TiB under device size. I'm
aware of the logical address limit and was regularly seeing the 5/8
warning in the kernel log (the FS was created before this warning was
introduced). The system is non-critical and only sees I/O once a day
when an rsync job is running.

Everything ran fine until the FS was remounted read-only during a scrub.
The scrub finished without errors and a reboot got it back to read-write
immediately. At this point I thought to delete a subvolume and resize
the FS on the two larger disks to come in at under 16 TiB device size
total. Unfortunately the delete and subsequent background clean task
kept OOM'ing on this ancient machine and I had to temporarily move the
disks to a different (64 bit) machine with more memory. There the
deletion worked fine and I did a check, two resizes and a balance (in
hindsight the last one was probably a mistake). Now device size is 15.99
TiB, but when I move the disks back, I can't mount the FS anymore.

[  604.404057] BTRFS error (device sdb1): reached 32bit limit for logical addresses                     
[  604.404110] BTRFS error (device sdb1): due to page cache limit on 32bit systems, metadata beyond 16T can't be accessed
[  604.404145] BTRFS error (device sdb1): please consider upgrading to 64bit kernel/hardware
[  604.404175] BTRFS error (device sdb1): failed to read chunk tree: -75
[  604.441574] BTRFS error (device sdb1): open_ctree failed

I guess since it talks about "logical" addresses and not physical ones,
the resize was pointless to begin with. Is there anything I can do to
make this FS mountable on 32 bit again?

I don't want to decommission this box, so the alternative would be to
recreate the file system from scratch and restore from backup. This is a
very low priority system, that's why I ignored the warning initially.
This is by no means the fault of BTRFS.

Best regards
Dave

Re: Question about 32 bit limitations

[Roman Mamedov]

On Thu, 18 Jul 2024 13:02:25 +0200
David Schiller <david.schiller@jku.at> wrote:

> I don't want to decommission this box, so the alternative would be to
> recreate the file system from scratch and restore from backup. This is a
> very low priority system, that's why I ignored the warning initially.
> This is by no means the fault of BTRFS.

As an option, you can avoid mounting the disks on the box itself, and instead
export them via NBD, AoE or iSCSI to be mounted on another box.

I run a very old DNS-323 armv5 NAS with just 64 MB of RAM, exporting a 6 TB
disk over NBD. (qemu-nbd was the only NBD server that works well given the
limitations). Every day my amd64 server temporarily mounts the NBD device,
runs rsync to it, then unmounts and disconnects until the next backup.

In your case it would be a bit more involved to export 4 devices and then
ensure all are connected remotely on the other machine before mounting. But
not impossible with a bit of scripting.

-- 
With respect,
Roman

Re: Question about 32 bit limitations

[David Schiller]

On Thu, 2024-07-18 at 18:50 +0500, Roman Mamedov wrote:
> As an option, you can avoid mounting the disks on the box itself, and
> instead export them via NBD, AoE or iSCSI to be mounted on another
> box.

That's actually a good idea. Thank you!
I initially considered block-level network protocols, when I ran out of
memory, but concluded it was easier to just physically move the disks to
the other machine.
Never thought of it as a more permanent solution.

Still hoping that there is some internal debug "magic" to rewrite the
offending logical addresses to be smaller than 2^32.