From: "Darrick J. Wong" <djwong@kernel.org>
To: Allison Henderson <allison.henderson@oracle.com>
Cc: "linux-xfs@vger.kernel.org" <linux-xfs@vger.kernel.org>
Subject: Re: [PATCH v8 00/27] Parent Pointers
Date: Wed, 25 Jan 2023 09:10:21 -0800 [thread overview]
Message-ID: <Y9FifTUeIW+Mj1+B@magnolia> (raw)
In-Reply-To: <Y9CCrQ7IL1/R3ECD@magnolia>
[-- Attachment #1: Type: text/plain, Size: 10060 bytes --]
On Tue, Jan 24, 2023 at 05:15:25PM -0800, Darrick J. Wong wrote:
> On Tue, Jan 24, 2023 at 07:38:57AM +0000, Allison Henderson wrote:
> > On Mon, 2023-01-23 at 20:38 -0800, Darrick J. Wong wrote:
> > > On Mon, Jan 23, 2023 at 06:35:53PM -0700,
> > > allison.henderson@oracle.com wrote:
> > > > From: Allison Henderson <allison.henderson@oracle.com>
> > > >
> > > > Hi all,
> > > >
> > > > This is the latest parent pointer attributes for xfs.
> > > > The goal of this patch set is to add a parent pointer attribute to
> > > > each inode.
> > > > The attribute name containing the parent inode, generation, and
> > > > directory
> > > > offset, while the attribute value contains the file name. This
> > > > feature will
> > > > enable future optimizations for online scrub, shrink, nfs handles,
> > > > verity, or
> > > > any other feature that could make use of quickly deriving an inodes
> > > > path from
> > > > the mount point.
> > > >
> > > > This set can be viewed on github here
> > > > https://urldefense.com/v3/__https://github.com/allisonhenderson/xfs/tree/xfs_new_pptrsv8_r1__;!!ACWV5N9M2RV99hQ!N2nNiERMG5LB2c5JRplSXXpgpp6yVFE8n22VEStqjIgbSm7yY0m92DEvfpPkD7XfwguuE49Sclb-8uSz6Ss5$
> > > >
> > >
> > > I ran the kernel v8r1 branch through fstests and it came up with
> > > this:
> > > https://urldefense.com/v3/__https://djwong.org/fstests/output/.e2ecf3cd98a7b55bfe8b9d7f33d2ef9549ccb6526765421fd929cf6b1fa82265/.238f7848578c98c24e6347e59963548102fe83037127e44802014e48281a8ccc/?C=M;O=A__;!!ACWV5N9M2RV99hQ!N2nNiERMG5LB2c5JRplSXXpgpp6yVFE8n22VEStqjIgbSm7yY0m92DEvfpPkD7XfwguuE49Sclb-8vEH_S-U$
> >
> > Alrighty, I'll take a look, I had run it through a night or so ago, and
> > hadnt noticed the same failures you list here. Some of these fail out
> > of the box for me, so I didnt think them associated with pptr changes.
>
> <nod> Some of those are just problems resulting from hardcoding xfs_db
> and mkfs output.
>
> I noticed that the shutdowns are a result of reservationless link/rename
> operations; it's easier just to get rid of that nonfeature.
>
> Also, there's a huge memory leak of xfs_parent_defer objects.
>
> That's at least what I've found so far. Will send the whole mess
> through fstests tonight.
No disasters reported, which means that the rest of the test failures
are (I think) either the result of mkfs failures (x/306) or different
xattrs (x/021). I don't know why generic/050 fails, and ignore the
xfs/060 failure because the xfsdump tests fail randomly and nobody knows
why.
https://djwong.org/fstests/output/.e2ecf3cd98a7b55bfe8b9d7f33d2ef9549ccb6526765421fd929cf6b1fa82265/.27de2687d36a762c08d51a0f4a90e89b8ec852e883834bea47edffbfb03428eb/?C=M;O=A
I've attached a tarball of the patches I applied to your kernel,
xfsprogs, and fstests trees to generate the fstests results. Most of
the problems I found were a result of turning on KASAN, kmemcheck, or
lockdep. It's up to you if you want to rebase the patch changes into
your branch or simply tack them on the end.
--D
> > >
> > >
> > > Looks better than v7, though I haven't tracked down why the fs goes
> > > down
> > > in generic/083 yet. I think it's the same "rename doesn't reserve
> > > enough blocks" problem I was muttering about last time. I think I
> > > need
> > > to look through the block reservation calculations again.
> > >
> > > That said, I *did* finally write the code that scans the parent
> > > pointers
> > > to generate new directories. It works for simple stupid cases where
> > > fsstress isn't running, but the live hooks part doesn't work because
> > > I
> > > haven't though through the locking model yet! :)
> > >
> > > > And the corresponding xfsprogs code is here
> > > > https://urldefense.com/v3/__https://github.com/allisonhenderson/xfsprogs/tree/xfsprogs_new_pptrs_v8_r1__;!!ACWV5N9M2RV99hQ!N2nNiERMG5LB2c5JRplSXXpgpp6yVFE8n22VEStqjIgbSm7yY0m92DEvfpPkD7XfwguuE49Sclb-8nOCLP-H$
> > > >
> > >
> > > Will rebase xfsprogs against v8r1 tomorrow.
> > I think you had a scrub patch for this I forgot to add. will do
> > that... Aside from that, not much change there tho
>
> <nod>
>
> > >
> > > > This set has been tested with the below parent pointers tests
> > > > https://urldefense.com/v3/__https://lore.kernel.org/fstests/20221012013812.82161-1-catherine.hoang@oracle.com/T/*t__;Iw!!ACWV5N9M2RV99hQ!N2nNiERMG5LB2c5JRplSXXpgpp6yVFE8n22VEStqjIgbSm7yY0m92DEvfpPkD7XfwguuE49Sclb-8ilcWlei$
> > > >
> > >
> > > And fix fstests after that.
> > The testcase i had saved as something Catherine could work on, but
> > there's no rush on it. The testcase tends to get tossed around any
> > time there are api changes so it made sense to land at least kernel
> > side first, though hopefully things should be decently firm at this
> > point.
>
> <nod> I'll not pay too much attention to xfs/018 then.
>
> --D
>
> > Allison
> >
> > >
> > > --D
> > >
> > > > Updates since v7:
> > > >
> > > > xfs: Increase XFS_QM_TRANS_MAXDQS to 5
> > > > Modified xfs_dqlockn to sort dquotes before locking
> > > >
> > > > xfs: Hold inode locks in xfs_trans_alloc_dir
> > > > Modified xfs_trans_alloc_dir to release locks before retrying
> > > > trans allocation
> > > >
> > > > xfs: Hold inode locks in xfs_rename
> > > > Modified xfs_rename to release locks before retrying trans
> > > > allocation
> > > >
> > > > xfs: Expose init_xattrs in xfs_create_tmpfile
> > > > Fixed xfs_generic_create to init attr tree
> > > >
> > > > xfs: add parent pointer support to attribute code
> > > > Updated xchk_xattr_rec with new XFS_ATTR_PARENT flag
> > > >
> > > > xfs: Add parent pointer ioctl
> > > > Include xfs_parent_utils.h in xfs_parent_utils.c to quiet
> > > > compiler warnings
> > > >
> > > > Questions comments and feedback appreciated!
> > > >
> > > > Thanks all!
> > > > Allison
> > > >
> > > > Allison Henderson (27):
> > > > xfs: Add new name to attri/d
> > > > xfs: Increase XFS_DEFER_OPS_NR_INODES to 5
> > > > xfs: Increase XFS_QM_TRANS_MAXDQS to 5
> > > > xfs: Hold inode locks in xfs_ialloc
> > > > xfs: Hold inode locks in xfs_trans_alloc_dir
> > > > xfs: Hold inode locks in xfs_rename
> > > > xfs: Expose init_xattrs in xfs_create_tmpfile
> > > > xfs: get directory offset when adding directory name
> > > > xfs: get directory offset when removing directory name
> > > > xfs: get directory offset when replacing a directory name
> > > > xfs: add parent pointer support to attribute code
> > > > xfs: define parent pointer xattr format
> > > > xfs: Add xfs_verify_pptr
> > > > xfs: extend transaction reservations for parent attributes
> > > > xfs: parent pointer attribute creation
> > > > xfs: add parent attributes to link
> > > > xfs: add parent attributes to symlink
> > > > xfs: remove parent pointers in unlink
> > > > xfs: Indent xfs_rename
> > > > xfs: Add parent pointers to rename
> > > > xfs: Add parent pointers to xfs_cross_rename
> > > > xfs: Add the parent pointer support to the superblock version 5.
> > > > xfs: Add helper function xfs_attr_list_context_init
> > > > xfs: Filter XFS_ATTR_PARENT for getfattr
> > > > xfs: Add parent pointer ioctl
> > > > xfs: fix unit conversion error in xfs_log_calc_max_attrsetm_res
> > > > xfs: drop compatibility minimum log size computations for reflink
> > > >
> > > > fs/xfs/Makefile | 2 +
> > > > fs/xfs/libxfs/xfs_attr.c | 71 +++++-
> > > > fs/xfs/libxfs/xfs_attr.h | 13 +-
> > > > fs/xfs/libxfs/xfs_da_btree.h | 3 +
> > > > fs/xfs/libxfs/xfs_da_format.h | 38 ++-
> > > > fs/xfs/libxfs/xfs_defer.c | 28 ++-
> > > > fs/xfs/libxfs/xfs_defer.h | 8 +-
> > > > fs/xfs/libxfs/xfs_dir2.c | 21 +-
> > > > fs/xfs/libxfs/xfs_dir2.h | 7 +-
> > > > fs/xfs/libxfs/xfs_dir2_block.c | 9 +-
> > > > fs/xfs/libxfs/xfs_dir2_leaf.c | 8 +-
> > > > fs/xfs/libxfs/xfs_dir2_node.c | 8 +-
> > > > fs/xfs/libxfs/xfs_dir2_sf.c | 6 +
> > > > fs/xfs/libxfs/xfs_format.h | 4 +-
> > > > fs/xfs/libxfs/xfs_fs.h | 75 ++++++
> > > > fs/xfs/libxfs/xfs_log_format.h | 7 +-
> > > > fs/xfs/libxfs/xfs_log_rlimit.c | 53 ++++
> > > > fs/xfs/libxfs/xfs_parent.c | 207 +++++++++++++++
> > > > fs/xfs/libxfs/xfs_parent.h | 46 ++++
> > > > fs/xfs/libxfs/xfs_sb.c | 4 +
> > > > fs/xfs/libxfs/xfs_trans_resv.c | 324 ++++++++++++++++++++----
> > > > fs/xfs/libxfs/xfs_trans_space.h | 8 -
> > > > fs/xfs/scrub/attr.c | 4 +-
> > > > fs/xfs/xfs_attr_item.c | 142 +++++++++--
> > > > fs/xfs/xfs_attr_item.h | 1 +
> > > > fs/xfs/xfs_attr_list.c | 17 +-
> > > > fs/xfs/xfs_dquot.c | 38 +++
> > > > fs/xfs/xfs_dquot.h | 1 +
> > > > fs/xfs/xfs_file.c | 1 +
> > > > fs/xfs/xfs_inode.c | 428 +++++++++++++++++++++++++---
> > > > ----
> > > > fs/xfs/xfs_inode.h | 3 +-
> > > > fs/xfs/xfs_ioctl.c | 148 +++++++++--
> > > > fs/xfs/xfs_ioctl.h | 2 +
> > > > fs/xfs/xfs_iops.c | 3 +-
> > > > fs/xfs/xfs_ondisk.h | 4 +
> > > > fs/xfs/xfs_parent_utils.c | 126 ++++++++++
> > > > fs/xfs/xfs_parent_utils.h | 11 +
> > > > fs/xfs/xfs_qm.c | 4 +-
> > > > fs/xfs/xfs_qm.h | 2 +-
> > > > fs/xfs/xfs_super.c | 4 +
> > > > fs/xfs/xfs_symlink.c | 58 ++++-
> > > > fs/xfs/xfs_trans.c | 9 +-
> > > > fs/xfs/xfs_trans_dquot.c | 15 +-
> > > > fs/xfs/xfs_xattr.c | 5 +-
> > > > fs/xfs/xfs_xattr.h | 1 +
> > > > 45 files changed, 1731 insertions(+), 246 deletions(-)
> > > > create mode 100644 fs/xfs/libxfs/xfs_parent.c
> > > > create mode 100644 fs/xfs/libxfs/xfs_parent.h
> > > > create mode 100644 fs/xfs/xfs_parent_utils.c
> > > > create mode 100644 fs/xfs/xfs_parent_utils.h
> > > >
> > > > --
> > > > 2.25.1
> > > >
> >
[-- Attachment #2: ach.tar.xz --]
[-- Type: application/x-xz, Size: 7832 bytes --]
next prev parent reply other threads:[~2023-01-25 17:11 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-24 1:35 [PATCH v8 00/27] Parent Pointers allison.henderson
2023-01-24 1:35 ` [PATCH v8 01/27] xfs: Add new name to attri/d allison.henderson
2023-01-24 1:35 ` [PATCH v8 02/27] xfs: Increase XFS_DEFER_OPS_NR_INODES to 5 allison.henderson
2023-01-24 1:35 ` [PATCH v8 03/27] xfs: Increase XFS_QM_TRANS_MAXDQS " allison.henderson
2023-01-24 1:35 ` [PATCH v8 04/27] xfs: Hold inode locks in xfs_ialloc allison.henderson
2023-01-24 1:35 ` [PATCH v8 05/27] xfs: Hold inode locks in xfs_trans_alloc_dir allison.henderson
2023-01-24 1:35 ` [PATCH v8 06/27] xfs: Hold inode locks in xfs_rename allison.henderson
2023-01-24 1:36 ` [PATCH v8 07/27] xfs: Expose init_xattrs in xfs_create_tmpfile allison.henderson
2023-01-24 1:36 ` [PATCH v8 08/27] xfs: get directory offset when adding directory name allison.henderson
2023-01-24 1:36 ` [PATCH v8 09/27] xfs: get directory offset when removing " allison.henderson
2023-01-24 1:36 ` [PATCH v8 10/27] xfs: get directory offset when replacing a " allison.henderson
2023-01-24 1:36 ` [PATCH v8 11/27] xfs: add parent pointer support to attribute code allison.henderson
2023-01-24 1:36 ` [PATCH v8 12/27] xfs: define parent pointer xattr format allison.henderson
2023-01-24 1:36 ` [PATCH v8 13/27] xfs: Add xfs_verify_pptr allison.henderson
2023-01-24 1:36 ` [PATCH v8 14/27] xfs: extend transaction reservations for parent attributes allison.henderson
2023-01-24 1:36 ` [PATCH v8 15/27] xfs: parent pointer attribute creation allison.henderson
2023-01-24 1:36 ` [PATCH v8 16/27] xfs: add parent attributes to link allison.henderson
2023-01-24 1:36 ` [PATCH v8 17/27] xfs: add parent attributes to symlink allison.henderson
2023-01-24 1:36 ` [PATCH v8 18/27] xfs: remove parent pointers in unlink allison.henderson
2023-01-24 1:36 ` [PATCH v8 19/27] xfs: Indent xfs_rename allison.henderson
2023-01-24 1:36 ` [PATCH v8 20/27] xfs: Add parent pointers to rename allison.henderson
2023-01-24 1:36 ` [PATCH v8 21/27] xfs: Add parent pointers to xfs_cross_rename allison.henderson
2023-01-24 1:36 ` [PATCH v8 22/27] xfs: Add the parent pointer support to the superblock version 5 allison.henderson
2023-01-24 1:36 ` [PATCH v8 23/27] xfs: Add helper function xfs_attr_list_context_init allison.henderson
2023-01-24 1:36 ` [PATCH v8 24/27] xfs: Filter XFS_ATTR_PARENT for getfattr allison.henderson
2023-01-24 1:36 ` [PATCH v8 25/27] xfs: Add parent pointer ioctl allison.henderson
2023-01-24 1:36 ` [PATCH v8 26/27] xfs: fix unit conversion error in xfs_log_calc_max_attrsetm_res allison.henderson
2023-01-24 1:36 ` [PATCH v8 27/27] xfs: drop compatibility minimum log size computations for reflink allison.henderson
2023-01-24 4:38 ` [PATCH v8 00/27] Parent Pointers Darrick J. Wong
2023-01-24 7:38 ` Allison Henderson
2023-01-25 1:15 ` Darrick J. Wong
2023-01-25 17:10 ` Darrick J. Wong [this message]
2023-01-25 20:54 ` Allison Henderson
2023-02-01 1:12 ` Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y9FifTUeIW+Mj1+B@magnolia \
--to=djwong@kernel.org \
--cc=allison.henderson@oracle.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.