From: Dave Chinner <david@fromorbit.com>
To: Amir Goldstein <amir73il@gmail.com>
Cc: Allison Collins <allison.henderson@oracle.com>,
linux-xfs <linux-xfs@vger.kernel.org>
Subject: Re: [PATCH v7 00/19] xfs: Delayed Ready Attrs
Date: Tue, 25 Feb 2020 20:52:12 +1100 [thread overview]
Message-ID: <20200225095212.GM10776@dread.disaster.area> (raw)
In-Reply-To: <CAOQ4uxgvJOF6+jd9BuJfxxGQbiit6J7zVOVnigwLb-RWizRqfg@mail.gmail.com>
On Sun, Feb 23, 2020 at 09:55:48AM +0200, Amir Goldstein wrote:
> On Sun, Feb 23, 2020 at 4:06 AM Allison Collins
> <allison.henderson@oracle.com> wrote:
> >
> > Hi all,
> >
> > This set is a subset of a larger series for delayed attributes. Which is
> > a subset of an even larger series, parent pointers. Delayed attributes
> > allow attribute operations (set and remove) to be logged and committed
> > in the same way that other delayed operations do. This allows more
> > complex operations (like parent pointers) to be broken up into multiple
> > smaller transactions. To do this, the existing attr operations must be
> > modified to operate as either a delayed operation or a inline operation
> > since older filesystems will not be able to use the new log entries.
>
> High level question, before I dive into the series:
>
> Which other "delayed operations" already exist?
See Chandan's answer :P
> I think delayed operations were added by Darrick to handle the growth of
> translation size due to reflink. Right? So I assume the existing delayed
> operations deal with block accounting.
No, they are intended to allow atomic, recoverable multi-transaction
operations. They grew out of this:
https://xfs.org/index.php/Improving_Metadata_Performance_By_Reducing_Journal_Overhead#Atomic_Multi-Transaction_Operations
which was essentially an generalisation of the EFI/EFD intent
logging that has existed in XFS for 20 years.
Essentially, it is a mechanism of chaining intent operations to
ensure that recover will restart the operation at the point the
system failed so that once the operation is started (i.e. first
intent is logged to the journal) the entire operation is always
completed regardless of whether the system crashes or not.
> When speaking of parent pointers, without having looked into the details yet,
> it seem the delayed operations we would want to log are operations that deal
> with namespace changes, i.e.: link,unlink,rename.
> The information needed to be logged for these ops is minimal.
Not really. the parent pointers are held in attributes, so parent
pointers are effectively adding an attribute creation to every inode
allocation and an attribute modification to every directory
modification. And, well, when an inode has 100 million hard links,
it's going to have 100 million parent pointer attributes. Modifying
a link is then a major operation, and Chandan has done a great job
in analysing the attr btree to see if there are scalability issues
that will be exposed by this sort of attribute usage....
> Why do we need a general infrastructure for delayed attr operations?
These have to be done atomically with the create/unlink/rename/etc
and to include attribute modification in those transaction
reservations blows the size of them out massively (especially
rename!). By converting these operations to use defered operations
to add the parent pointer to the inode, we no longer need to
increase the log reservation for the operations (because the attr
reservation is usually smaller than the directory reservation), and
it is guaranteed to be atomic with the directory modification. i.e.
parent pointers never get out of sync, even when the system crashes.
Hence having attributes modified as a series of individual
operations chained together into an atomic whole via intents is a
pre-requisite for updating attributes atomically within directory
modification operations.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
prev parent reply other threads:[~2020-02-25 9:52 UTC|newest]
Thread overview: 135+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-23 2:05 [PATCH v7 00/19] xfs: Delayed Ready Attrs Allison Collins
2020-02-23 2:05 ` [PATCH v7 01/19] xfs: Replace attribute parameters with struct xfs_name Allison Collins
2020-02-23 9:34 ` Amir Goldstein
2020-02-23 16:03 ` Allison Collins
2020-02-25 0:49 ` Dave Chinner
2020-02-24 12:08 ` Chandan Rajendra
2020-02-24 16:25 ` Allison Collins
2020-02-24 13:06 ` Brian Foster
2020-02-24 16:25 ` Allison Collins
2020-02-23 2:05 ` [PATCH v7 02/19] xfs: Embed struct xfs_name in xfs_da_args Allison Collins
2020-02-23 11:54 ` Amir Goldstein
2020-02-23 16:51 ` Allison Collins
2020-02-24 6:50 ` Amir Goldstein
2020-02-24 7:36 ` Allison Collins
2020-02-24 7:43 ` Amir Goldstein
2020-02-25 0:57 ` Dave Chinner
2020-02-25 2:00 ` Allison Collins
2020-02-25 4:06 ` Dave Chinner
2020-02-25 4:19 ` Allison Collins
2020-02-25 4:27 ` Darrick J. Wong
2020-02-25 6:07 ` Allison Collins
2020-02-25 6:30 ` Dave Chinner
2020-02-25 17:21 ` Christoph Hellwig
2020-02-25 6:56 ` Chandan Rajendra
2020-02-25 23:26 ` Allison Collins
2020-02-23 2:05 ` [PATCH v7 03/19] xfs: Add xfs_has_attr and subroutines Allison Collins
2020-02-23 12:20 ` Amir Goldstein
2020-02-23 17:28 ` Allison Collins
2020-02-24 6:58 ` Amir Goldstein
2020-02-25 6:26 ` Dave Chinner
2020-02-25 6:43 ` Amir Goldstein
2020-02-25 22:27 ` Dave Chinner
2020-02-24 13:08 ` Brian Foster
2020-02-24 21:18 ` Allison Collins
2020-02-25 13:25 ` Brian Foster
2020-02-26 2:31 ` Allison Collins
2020-02-25 9:49 ` Chandan Rajendra
2020-02-25 10:15 ` Chandan Rajendra
2020-02-26 2:19 ` Allison Collins
2020-02-26 2:18 ` Allison Collins
2020-02-23 2:05 ` [PATCH v7 04/19] xfs: Check for -ENOATTR or -EEXIST Allison Collins
2020-02-23 12:25 ` Amir Goldstein
2020-02-23 17:33 ` Allison Collins
2020-02-24 13:08 ` Brian Foster
2020-02-24 21:18 ` Allison Collins
2020-02-23 2:05 ` [PATCH v7 05/19] xfs: Factor out new helper functions xfs_attr_rmtval_set Allison Collins
2020-02-25 12:53 ` Chandan Rajendra
2020-02-26 2:20 ` Allison Collins
2020-02-23 2:05 ` [PATCH v7 06/19] xfs: Factor out trans handling in xfs_attr3_leaf_flipflags Allison Collins
2020-02-23 12:30 ` Amir Goldstein
2020-02-23 17:36 ` Allison Collins
2020-02-28 4:56 ` Chandan Rajendra
2020-02-23 2:05 ` [PATCH v7 07/19] xfs: Factor out xfs_attr_leaf_addname helper Allison Collins
2020-02-23 12:42 ` Amir Goldstein
2020-02-23 18:38 ` Allison Collins
2020-02-24 6:38 ` Amir Goldstein
2020-02-24 7:09 ` Allison Collins
2020-02-24 7:27 ` Amir Goldstein
2020-02-25 6:42 ` Dave Chinner
2020-02-25 13:26 ` Brian Foster
2020-02-25 23:26 ` Allison Collins
2020-02-28 6:51 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 08/19] xfs: Refactor xfs_attr_try_sf_addname Allison Collins
2020-02-23 13:04 ` Amir Goldstein
2020-02-23 17:51 ` Allison Collins
2020-02-24 13:08 ` Brian Foster
2020-02-24 21:19 ` Allison Collins
2020-02-28 7:42 ` Chandan Rajendra
2020-02-28 18:14 ` Allison Collins
2020-02-23 2:06 ` [PATCH v7 09/19] xfs: Factor out trans roll from xfs_attr3_leaf_setflag Allison Collins
2020-02-28 7:59 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 10/19] xfs: Factor out xfs_attr_rmtval_invalidate Allison Collins
2020-02-28 10:42 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 11/19] xfs: Factor out trans roll in xfs_attr3_leaf_clearflag Allison Collins
2020-02-28 10:56 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 12/19] xfs: Add helper function xfs_attr_rmtval_unmap Allison Collins
2020-02-24 13:40 ` Brian Foster
2020-02-24 21:44 ` Allison Collins
2020-02-25 13:27 ` Brian Foster
2020-02-26 3:29 ` Allison Collins
2020-02-26 13:47 ` Brian Foster
2020-02-25 7:21 ` Dave Chinner
2020-02-25 23:27 ` Allison Collins
2020-02-28 14:22 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 13/19] xfs: Add delay ready attr remove routines Allison Collins
2020-02-24 15:25 ` Brian Foster
2020-02-24 17:03 ` Brian Foster
2020-02-24 23:14 ` Allison Collins
2020-02-24 23:56 ` Darrick J. Wong
2020-02-25 13:34 ` Brian Foster
2020-02-26 5:36 ` Allison Collins
2020-02-26 13:48 ` Brian Foster
2020-02-26 19:23 ` Allison Collins
2020-02-25 8:57 ` Dave Chinner
2020-02-26 0:57 ` Allison Collins
2020-02-26 22:34 ` Dave Chinner
2020-02-27 4:18 ` Allison Collins
2020-03-03 5:03 ` Chandan Rajendra
2020-03-03 5:40 ` Allison Collins
2020-02-23 2:06 ` [PATCH v7 14/19] xfs: Add delay ready attr set routines Allison Collins
2020-03-03 13:41 ` Chandan Rajendra
2020-03-03 17:07 ` Allison Collins
2020-02-23 2:06 ` [PATCH v7 15/19] xfs: Add helper function xfs_attr_node_shrink Allison Collins
2020-02-23 13:22 ` Amir Goldstein
2020-02-23 18:41 ` Allison Collins
2020-02-25 9:05 ` Dave Chinner
2020-02-26 1:48 ` Allison Collins
2020-02-23 2:06 ` [PATCH v7 16/19] xfs: Simplify xfs_attr_set_iter Allison Collins
2020-02-23 13:26 ` Amir Goldstein
2020-02-23 18:42 ` Allison Collins
2020-02-25 9:21 ` Dave Chinner
2020-02-26 2:13 ` Allison Collins
2020-02-26 22:39 ` Dave Chinner
2020-03-04 4:30 ` Chandan Rajendra
2020-03-04 17:04 ` Allison Collins
2020-03-05 3:39 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 17/19] xfs: Add helper function xfs_attr_leaf_mark_incomplete Allison Collins
2020-02-23 13:47 ` Amir Goldstein
2020-02-23 18:43 ` Allison Collins
2020-02-25 9:31 ` Dave Chinner
2020-02-26 2:17 ` Allison Collins
2020-03-04 4:37 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 18/19] xfs: Add remote block helper functions Allison Collins
2020-02-23 13:45 ` Amir Goldstein
2020-03-04 4:59 ` Chandan Rajendra
2020-02-23 2:06 ` [PATCH v7 19/19] xfs: Remove xfs_attr_rmtval_remove Allison Collins
2020-02-23 13:54 ` Amir Goldstein
2020-02-23 18:50 ` Allison Collins
2020-02-23 7:55 ` [PATCH v7 00/19] xfs: Delayed Ready Attrs Amir Goldstein
2020-02-23 16:02 ` Allison Collins
2020-02-24 6:30 ` Amir Goldstein
2020-02-24 16:23 ` Allison Collins
2020-02-25 5:53 ` Amir Goldstein
2020-02-24 8:31 ` Chandan Rajendra
2020-02-25 9:52 ` Dave Chinner [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200225095212.GM10776@dread.disaster.area \
--to=david@fromorbit.com \
--cc=allison.henderson@oracle.com \
--cc=amir73il@gmail.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox