2 PB filesystem ok?

2 PB filesystem ok?

[Ulli Horlacher]

We are planing to use a Seagate Exos Corvault System with 2 PB as the
storage backend for restic backup. It has some hundreds hard disks and
exports them as a single virtual disk (via SAS). Linux sees only a big
SCSI disk. This special RAID is all in hardware, no need for BTRFS
software RAID.

So far my biggest btrfs filesystems have only about 100 TB.

Is 2 PB ok for one btrfs filesystem?


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<20240428233134.GA355040@tik.uni-stuttgart.de>

Re: 2 PB filesystem ok?

[Andrea Gelmini]

Il giorno lun 29 apr 2024 alle ore 01:40 Ulli Horlacher
<framstag@rus.uni-stuttgart.de> ha scritto:
> Is 2 PB ok for one btrfs filesystem?

Well, it works (you can play by yourself with loopback just for fun
with a test file!!!).

But, I dunno about RAM request if you are in trouble and need to run
fsck or repair. Also about the RAM requested to have all extents
mapped.
You could be in sad situation to not be able to mount it.
Also, better if you can update the kernel whenever you want, and you
don't have third party constraints.

In my experience (max 200TB volume),  I use BTRFS for big volumes, but
they need more extra care than the usual EXT4/XFS.

Re: 2 PB filesystem ok?

[Johannes Thumshirn]

On 29.04.24 01:41, Ulli Horlacher wrote:
> 
> We are planing to use a Seagate Exos Corvault System with 2 PB as the
> storage backend for restic backup. It has some hundreds hard disks and
> exports them as a single virtual disk (via SAS). Linux sees only a big
> SCSI disk. This special RAID is all in hardware, no need for BTRFS
> software RAID.
> 
> So far my biggest btrfs filesystems have only about 100 TB.
> 
> Is 2 PB ok for one btrfs filesystem?
> 
> 

I have used 2PB filesystems in my RAID Stripe Tree test environment, but 
for practical uses, I suggest you to enable the block-group tree feature 
during mkfs time.

This stores block-group items in a separate tree instead of the extent 
tree, drastically improving mount times on huge FSes.

Re: 2 PB filesystem ok?

[Ulli Horlacher]

On Mon 2024-04-29 (12:34), Johannes Thumshirn wrote:

> I have used 2PB filesystems in my RAID Stripe Tree test environment, but
> for practical uses, I suggest you to enable the block-group tree feature
> during mkfs time.

I cannot find such an option in man-page for mkfs.btrfs


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<6609017e-8931-4559-b613-4b3e28d9fb48@wdc.com>

Re: 2 PB filesystem ok?

[Tomas Volf]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

On 2024-05-01 10:50:12 +0200, Ulli Horlacher wrote:
> On Mon 2024-04-29 (12:34), Johannes Thumshirn wrote:
>
> > I have used 2PB filesystems in my RAID Stripe Tree test environment, but
> > for practical uses, I suggest you to enable the block-group tree feature
> > during mkfs time.
>
> I cannot find such an option in man-page for mkfs.btrfs

    $ mkfs.btrfs  -O list-all
    Filesystem features available:
    mixed-bg            - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37)
    quota               - quota support (qgroups) (compat=3.4)
    extref              - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12)
    raid56              - raid56 extended format (compat=3.9)
    skinny-metadata     - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18)
    no-holes            - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15)
    free-space-tree     - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15)
    raid1c34            - RAID1 with 3 or 4 copies (compat=5.5)
    zoned               - support zoned devices (compat=5.12)
    block-group-tree    - block group tree to reduce mount time (compat=6.1)

So I believe it would be `mkfs.btrfs -O block-group-tree ...'.

>
>
> --
> Ullrich Horlacher              Server und Virtualisierung
> Rechenzentrum TIK
> Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
> Allmandring 30a                Tel:    ++49-711-68565868
> 70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
> REF:<6609017e-8931-4559-b613-4b3e28d9fb48@wdc.com>
>

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: 2 PB filesystem ok?

[Ulli Horlacher]

On Wed 2024-05-01 (16:16), Tomas Volf wrote:
> On 2024-05-01 10:50:12 +0200, Ulli Horlacher wrote:
> 
> > On Mon 2024-04-29 (12:34), Johannes Thumshirn wrote:
> >
> > > I have used 2PB filesystems in my RAID Stripe Tree test environment, but
> > > for practical uses, I suggest you to enable the block-group tree feature
> > > during mkfs time.
> >
> > I cannot find such an option in man-page for mkfs.btrfs
> 
>     $ mkfs.btrfs  -O list-all
>     Filesystem features available:
>     mixed-bg            - mixed data and metadata block groups (compat=2.6.37, safe=2.6.37)
>     quota               - quota support (qgroups) (compat=3.4)
>     extref              - increased hardlink limit per file to 65536 (compat=3.7, safe=3.12, default=3.12)
>     raid56              - raid56 extended format (compat=3.9)
>     skinny-metadata     - reduced-size metadata extent refs (compat=3.10, safe=3.18, default=3.18)
>     no-holes            - no explicit hole extents for files (compat=3.14, safe=4.0, default=5.15)
>     free-space-tree     - free space tree (space_cache=v2) (compat=4.5, safe=4.9, default=5.15)
>     raid1c34            - RAID1 with 3 or 4 copies (compat=5.5)
>     zoned               - support zoned devices (compat=5.12)
>     block-group-tree    - block group tree to reduce mount time (compat=6.1)
> 
> So I believe it would be `mkfs.btrfs -O block-group-tree ...'.

root@fex:~# mkfs.btrfs  -O list-all
Filesystem features available:
mixed-bg            - mixed data and metadata block groups (0x4, compat=2.6.37, safe=2.6.37)
extref              - increased hardlink limit per file to 65536 (0x40, compat=3.7, safe=3.12, default=3.12)
raid56              - raid56 extended format (0x80, compat=3.9)
skinny-metadata     - reduced-size metadata extent refs (0x100, compat=3.10, safe=3.18, default=3.18)
no-holes            - no explicit hole extents for files (0x200, compat=3.14, safe=4.0, default=5.15)
raid1c34            - RAID1 with 3 or 4 copies (0x800, compat=5.5)
zoned               - support zoned devices (0x1000, compat=5.12)

root@fex:~# btrfs --version
btrfs-progs v5.16.2


This is Ubuntu 22.04



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<ZjJOuiu0o1PqV3jA@ws>

Re: 2 PB filesystem ok?

[Martin Steigerwald]

Ulli Horlacher - 01.05.24, 20:19:03 CEST:
> On Wed 2024-05-01 (16:16), Tomas Volf wrote:
> > On 2024-05-01 10:50:12 +0200, Ulli Horlacher wrote:
> > > On Mon 2024-04-29 (12:34), Johannes Thumshirn wrote:
> > > > I have used 2PB filesystems in my RAID Stripe Tree test
> > > > environment, but for practical uses, I suggest you to enable the
> > > > block-group tree feature during mkfs time.
> > > 
> > > I cannot find such an option in man-page for mkfs.btrfs
> > > 
> >     $ mkfs.btrfs  -O list-all
[…]
> > So I believe it would be `mkfs.btrfs -O block-group-tree ...'.
> 
> root@fex:~# mkfs.btrfs  -O list-all
[…]
> root@fex:~# btrfs --version
> btrfs-progs v5.16.2
> 
> 
> This is Ubuntu 22.04

Too old.

See here for minimum requirements:

https://btrfs.readthedocs.io/en/latest/Feature-by-version.html

Kernel (and I bet) btrfs progs 6.1.

I bet there is a PPA for newer kernels for Ubuntu. Whether there is a PPA 
for never btrfs progs, I don't know.

Best,
-- 
Martin

Re: 2 PB filesystem ok?

[Ulli Horlacher]

On Wed 2024-05-01 (21:47), Martin Steigerwald wrote:

> > > So I believe it would be `mkfs.btrfs -O block-group-tree ...'.
> >
> > root@fex:~# mkfs.btrfs  -O list-all
> [...]
> > root@fex:~# btrfs --version
> > btrfs-progs v5.16.2
> >
> >
> > This is Ubuntu 22.04
> 
> Too old.

root@zentux:~# mkfs.btrfs  -O list-all 2>&1 | grep block-group-tree
bgt                 - block-group-tree alias
block-group-tree    - block group tree to reduce mount time (compat=6.1)

root@zentux:~# sysinfo
System:        Linux zentux 6.8.0-62-generic x86_64
Distribution:  Ubuntu 24.04.2 LTS /sw

root@zentux:~# btrfs --version
btrfs-progs v6.6.3

Got it! :-)

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<2183753.irdbgypaU6@lichtvoll.de>