Re: [PATCH v2] btrfs: use preallocated pages for super block write

[Filipe Manana <fdmanana@kernel.org>]

On Fri, Jun 10, 2022 at 03:33:47PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/6/10 15:23, Nikolay Borisov wrote:
> > 
> > 
> > On 10.06.22 г. 3:07 ч., Qu Wenruo wrote:
> > > 
> > > 
> > > On 2022/6/10 00:46, David Sterba wrote:
> > > > Currently the super block page is from the mapping of the block device,
> > > > this is result of direct conversion from the previous buffer_head to bio
> > > > API.  We don't use the page cache or the mapping anywhere else, the page
> > > > is a temporary space for the associated bio.
> > > > 
> > > > Allocate pages for all super block copies at device allocation time,
> > > > also to avoid any later allocation problems when writing the super
> > > > block. This simplifies the page reference tracking, but the page lock is
> > > > still used as waiting mechanism for the write and write error is tracked
> > > > in the page.
> > > > 
> > > > As there is a separate page for each super block copy all can be
> > > > submitted in parallel, as before.
> > > 
> > > Is there any history on why we want parallel super block submission?
> > 
> > Because it's in the transaction critical path so it affects latency of
> > operations.
> 
> Not exactly.
> 
> Super block writeback happens with TRANS_STATE_UNBLOCKED status, which
> means new transaction can already be started.
> 
> Thus even if we don't do any parallel submission, there is no
> performance impact on transaction path.

Hell, no. There's more than transaction states to consider.

Taking longer to write super blocks can have a performance impact on fsync
for example. And fsync always has to write super blocks and wait for them
to complete. In fact, a large part of time spent on fsync is precisely on
writing super blocks.

In some cases fsync has to fallback to a transaction commit and wait for
it to complete before returning to user space - which again requires writing
super blocks and waiting for their completion.

Similarly, there are ioctls like snapshot and subvolume creation which
commit a transaction, and any changes to the way super blocks are written
can also affect their latency and impact applications.

> 
> Thanks,
> Qu
> > 
> > > 
> > > In fact, for the 3 super blocks, the primary one has FUA flag, while the
> > > other backup ones doesn't.
> > > 
> > > This means, even we wait for the super block write, only the first one
> > > would take some real time, while the other two can return almost
> > > immediate for devices with write cache.
> > > 
> > > In fact, waiting for the super block write back can tell us if our
> > > primary super block did really reach disk, allowing us to do better
> > > error handling, other than the almost non-exist check in endio.
> > > 
> > > And such synchronous submission allows us to have only one copy of the
> > > sb.
> > > 
> > > 
> > > Furthermore, if we really go parallel, I don't think the current way is
> > > the correct way.
> > > 
> > > One device can have at most 3 superblocks, but a btrfs can easily have
> > > more than 4 devices.
> > > 
> > > Shouldn't we parallel based on device, other than each superblock?
> > 
> > I agree that instead of having 3 pages per-device we can go the 1 page
> > per device, and parallelize based on the device, rather than the super
> > block copies.
> > 
> > <snip>