Question: Is growing log size when filesystem is grown a feasible solution

Question: Is growing log size when filesystem is grown a feasible solution

[Srikanth C S]

Hi all,

We had earlier seen a few instances where there was an impact
in the read/write performance with processes stuck in
xlog_grant_head_wait(). And our investigation brought us to the
conclusion that the limitation in log space was causing this.
This was when we have a lot of read/write going through and the
log space was limited.

The problem was seen especially when the file system was small
initially and later grown to a larger size using xfs_growfs.
The log size does not grow when the FS is grown. In these cases,
we are stuck with the same log size calculated for the smaller
file system size (which was 10MB, the earlier default value).

The problem was partially addressed by the commit:
cdfa467 mkfs: increase the minimum log size to 64MB when possible

This commit make the default log size to 64MB for new filesystems,
but does not address the issue for existing filesystems with a
small log size. The only solution for such Fs is to recreate it
from scratch, which is not feasible for production systems. Hence,
there is a need to provide a solution - and we want to explore
possibility of growing the log size using xfs_growfs.

It would be great if I can get some comments from the community
if the change where xfs_growfs can grow the log is encouraged.
If it is, then any advice or comments on how to proceed with the
problem would be great too.

Thanks,
Srikanth

Re: Question: Is growing log size when filesystem is grown a feasible solution

[Dave Chinner]

On Thu, Dec 07, 2023 at 09:17:09AM +0000, Srikanth C S wrote:
> Hi all,
> 
> We had earlier seen a few instances where there was an impact
> in the read/write performance with processes stuck in
> xlog_grant_head_wait(). And our investigation brought us to the
> conclusion that the limitation in log space was causing this.
> This was when we have a lot of read/write going through and the
> log space was limited.
> 
> The problem was seen especially when the file system was small
> initially and later grown to a larger size using xfs_growfs.
> The log size does not grow when the FS is grown. In these cases,
> we are stuck with the same log size calculated for the smaller
> file system size (which was 10MB, the earlier default value).
> 
> The problem was partially addressed by the commit:
> cdfa467 mkfs: increase the minimum log size to 64MB when possible
> 
> This commit make the default log size to 64MB for new filesystems,
> but does not address the issue for existing filesystems with a
> small log size. The only solution for such Fs is to recreate it
> from scratch, which is not feasible for production systems. Hence,
> there is a need to provide a solution - and we want to explore
> possibility of growing the log size using xfs_growfs.
> 
> It would be great if I can get some comments from the community
> if the change where xfs_growfs can grow the log is encouraged.
> If it is, then any advice or comments on how to proceed with the
> problem would be great too.

It's possible to implement an online log resize. That doesn't mean
it is easy to do. Start thinking about all the failure modes and how
to reliably avoid and/or recover from such failures.

e.g. how do you do an atomic swap from the old log to the new log in
a way that if we crash in the middle of the operation then log
recovery on the next mount will result in either the original state
on the old journal (nothing has changed) or that full operation
completion occurs (running on a fully initialised new journal).

Doing it offline is simpler, but there's still a bunch of metadata
modifications that have to be done correctly. i.e. we still have to
handle the case that the system could crash half way through the
offline log move operation in a manner that doesn't trash the
filesystem or user data.

If you've got 6 months to a year to spend getting all this right,
then I don't think anyone if going to say not to being able to
resize logs and/or convert internal/external logs dynamically. It's
just a lot of work to make it robust and reliable....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com