experience with very large filesystems

experience with very large filesystems

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

Hi everyone,

I have a large filesystem (1.2 PiB) that's been working flawlessly for
4 years (an LV aggregating 4 300~ish TB RAID-6 arrays). However it's
getting really full (more than 1.1 PiB full right now), and I'm
considering expanding it to 1.8 PiB or more.

Any thoughts, caveats or else?

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: experience with very large filesystems

[Dave Chinner]

On Wed, Feb 23, 2022 at 04:35:13PM +0100, Emmanuel Florac wrote:
> Hi everyone,
> 
> I have a large filesystem (1.2 PiB) that's been working flawlessly for
> 4 years (an LV aggregating 4 300~ish TB RAID-6 arrays). However it's
> getting really full (more than 1.1 PiB full right now), and I'm
> considering expanding it to 1.8 PiB or more.
> 
> Any thoughts, caveats or else?

From the filesystem side, that should not be an issue at all.

From the storage side, you really want to expand the storage with
chunks that have the same geometry (stripe unit and stripe width)
so that it doesn't screw up the alignment of the filesystem to the
new storage.

And that's where the difficultly may lie. If the existing storage
volume the filesystem sits on doesn't end exactly on a stripe width
boundary, you're going to have to offset the start of the new
storage volumes part way into the first stripe width in the volumes
to ensure that when the filesystem expands, then end of the first
stripe width in the new volume is exactly where the filesystem
expects it to be.

Other than that, there shouldn't be any filesystem level concerns
about doubling the size of the filesystem capacity via growfs.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: experience with very large filesystems

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1362 bytes --]

Le Thu, 24 Feb 2022 08:10:06 +1100
Dave Chinner <david@fromorbit.com> écrivait:

> From the storage side, you really want to expand the storage with
> chunks that have the same geometry (stripe unit and stripe width)
> so that it doesn't screw up the alignment of the filesystem to the
> new storage.

That part should be easy, I plan to use the same type of hardware anyway
(12 TB Ultrastar disks).
 
> And that's where the difficultly may lie. If the existing storage
> volume the filesystem sits on doesn't end exactly on a stripe width
> boundary, you're going to have to offset the start of the new
> storage volumes part way into the first stripe width in the volumes
> to ensure that when the filesystem expands, then end of the first
> stripe width in the new volume is exactly where the filesystem
> expects it to be.

That should be somewhat more manageable given that I've setup
everything as an LVM volume. I may use partitions on the new devices to
precisely align the PVs before expanding the LV. 

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

Re: experience with very large filesystems

[Dave Chinner]

On Thu, Feb 24, 2022 at 01:09:53PM +0100, Emmanuel Florac wrote:
> Le Thu, 24 Feb 2022 08:10:06 +1100
> Dave Chinner <david@fromorbit.com> écrivait:
> 
> > From the storage side, you really want to expand the storage with
> > chunks that have the same geometry (stripe unit and stripe width)
> > so that it doesn't screw up the alignment of the filesystem to the
> > new storage.
> 
> That part should be easy, I plan to use the same type of hardware anyway
> (12 TB Ultrastar disks).
>  
> > And that's where the difficultly may lie. If the existing storage
> > volume the filesystem sits on doesn't end exactly on a stripe width
> > boundary, you're going to have to offset the start of the new
> > storage volumes part way into the first stripe width in the volumes
> > to ensure that when the filesystem expands, then end of the first
> > stripe width in the new volume is exactly where the filesystem
> > expects it to be.
> 
> That should be somewhat more manageable given that I've setup
> everything as an LVM volume. I may use partitions on the new devices to
> precisely align the PVs before expanding the LV. 

You can use the PV/LV header size options to align the contents of
the LV.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com