questions about hybird xfs wih ssd/hdd by realtime subvol

questions about hybird xfs wih ssd/hdd by realtime subvol

[Wang Yugui]

Hi,

I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.

Hybrid XFS―Using SSDs to Supercharge HDDs at Facebook
https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

There are some questions about how to control the data to save into
normal vol or realtime subvol firstly.

1, man xfsctl
here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 , 
but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19.
xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

3, we build a xfs filesystem with 1G device and 1G rtdev device. and
then we can save 2G data into this xfs filesystem.

Is there any tool/kernel option/kernel patch to control the data to save
into normal vol or realtime subvol firstly?

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/08/29

Re: questions about hybird xfs wih ssd/hdd by realtime subvol

[Carlos Maiolino]

On Mon, Aug 29, 2022 at 10:26:20AM +0800, Wang Yugui wrote:
> Hi,
> 
> I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.
> 
> Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
> https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
> 
> There are some questions about how to control the data to save into
> normal vol or realtime subvol firstly.
> 
> 1, man xfsctl
> here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 ,
> but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19.
> xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?

It's been a while since XFS uses FS_XFLAG features directly, so, what you're
specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a
preprocessor define:

#define XFS_XFLAG_REALTIME	FS_XFLAG_REALTIME

FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without
the realtime filesystems going first, so, unlikely it's gonna happen.

> 
> 2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?

You can use xfs_io's chattr command to add/remote the REALTIME attribute of a
file.


> 
> 3, we build a xfs filesystem with 1G device and 1G rtdev device. and
> then we can save 2G data into this xfs filesystem.
> 
> Is there any tool/kernel option/kernel patch to control the data to save
> into normal vol or realtime subvol firstly?

I didn't watch the talk you mentioned above, but when use an rt device, you
don't use the 'normal' one then the rt later, or vice-versa, the rt-device is
used to store data blocks for those files marked with the xattr above. For those
files you want to store in the realtime device, you should add the above xattr
to them.

Cheers.

-- 
Carlos Maiolino

Re: questions about hybird xfs wih ssd/hdd by realtime subvol

[Wang Yugui]

Hi,

> On Mon, Aug 29, 2022 at 10:26:20AM +0800, Wang Yugui wrote:
> > Hi,
> > 
> > I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.
> > 
> > Hybrid XFS―Using SSDs to Supercharge HDDs at Facebook
> > https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
> > 
> > There are some questions about how to control the data to save into
> > normal vol or realtime subvol firstly.
> > 
> > 1, man xfsctl
> > here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 ,
> > but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19.
> > xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?
> 
> It's been a while since XFS uses FS_XFLAG features directly, so, what you're
> specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a
> preprocessor define:
> 
> #define XFS_XFLAG_REALTIME	FS_XFLAG_REALTIME
> 
> FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without
> the realtime filesystems going first, so, unlikely it's gonna happen.
> 
> > 
> > 2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?
> 
> You can use xfs_io's chattr command to add/remote the REALTIME attribute of a
> file.
> 
> 
> > 
> > 3, we build a xfs filesystem with 1G device and 1G rtdev device. and
> > then we can save 2G data into this xfs filesystem.

Sorry, I cheched again.
This is a xfs filesystem with 2G device and 2G rtdev device

> > Is there any tool/kernel option/kernel patch to control the data to save
> > into normal vol or realtime subvol firstly?
> 
> I didn't watch the talk you mentioned above, but when use an rt device, you
> don't use the 'normal' one then the rt later, or vice-versa, the rt-device is
> used to store data blocks for those files marked with the xattr above. For those
> files you want to store in the realtime device, you should add the above xattr
> to them.

Although I still fail to check/set the attr by 'lsattr/chattr', but I
can check the free space of 'normal' and realtime subvol now.

# xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
typedef struct xfs_sb {
...
	uint64_t	sb_fdblocks;	/* free data blocks */
	uint64_t	sb_frextents;	/* free realtime extents */
...
}

And based the info from Carlos Maiolino

FB were running a modified kernel that selected the rt dev based on
the initial allocation size. Behaviour for them was predictable
because they also controlled the application that was storing the
data. See:

https://lore.kernel.org/linux-xfs/20171128215527.2510350-1-rwareing@fb.com/

With a dirty patch below for test only , Now realtime subvol will be used
as I expected, and that can be confirmed by
#xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.

diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index f4cc8a1aaeb4..d19e0fa34c1a 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -868,6 +868,9 @@ xfs_init_new_inode(
                flags |= XFS_ILOG_DEV;
                break;
        case S_IFREG:
+               if (xfs_has_realtime(ip->i_mount))
+                       ip->i_diflags |= XFS_DIFLAG_REALTIME;
+               fallthrough;
        case S_IFDIR:
                if (pip && (pip->i_diflags & XFS_DIFLAG_ANY))
                        xfs_inode_inherit_flags(ip, pip);

Thanks a lot.

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/08/30

Re: questions about hybird xfs wih ssd/hdd by realtime subvol

[Darrick J. Wong]

On Tue, Aug 30, 2022 at 08:57:21AM +0800, Wang Yugui wrote:
> Hi,
> 
> > On Mon, Aug 29, 2022 at 10:26:20AM +0800, Wang Yugui wrote:
> > > Hi,
> > > 
> > > I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.
> > > 
> > > Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
> > > https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
> > > 
> > > There are some questions about how to control the data to save into
> > > normal vol or realtime subvol firstly.
> > > 
> > > 1, man xfsctl
> > > here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 ,
> > > but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19.
> > > xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?
> > 
> > It's been a while since XFS uses FS_XFLAG features directly, so, what you're
> > specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a
> > preprocessor define:
> > 
> > #define XFS_XFLAG_REALTIME	FS_XFLAG_REALTIME
> > 
> > FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without
> > the realtime filesystems going first, so, unlikely it's gonna happen.
> > 
> > > 
> > > 2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?
> > 
> > You can use xfs_io's chattr command to add/remote the REALTIME attribute of a
> > file.
> > 
> > 
> > > 
> > > 3, we build a xfs filesystem with 1G device and 1G rtdev device. and
> > > then we can save 2G data into this xfs filesystem.
> 
> Sorry, I cheched again.
> This is a xfs filesystem with 2G device and 2G rtdev device
> 
> > > Is there any tool/kernel option/kernel patch to control the data to save
> > > into normal vol or realtime subvol firstly?
> > 
> > I didn't watch the talk you mentioned above, but when use an rt device, you
> > don't use the 'normal' one then the rt later, or vice-versa, the rt-device is
> > used to store data blocks for those files marked with the xattr above. For those
> > files you want to store in the realtime device, you should add the above xattr
> > to them.
> 
> Although I still fail to check/set the attr by 'lsattr/chattr', but I
> can check the free space of 'normal' and realtime subvol now.
> 
> # xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
> typedef struct xfs_sb {
> ...
> 	uint64_t	sb_fdblocks;	/* free data blocks */
> 	uint64_t	sb_frextents;	/* free realtime extents */
> ...
> }
> 
> And based the info from Carlos Maiolino
> 
> FB were running a modified kernel that selected the rt dev based on
> the initial allocation size. Behaviour for them was predictable
> because they also controlled the application that was storing the
> data. See:
> 
> https://lore.kernel.org/linux-xfs/20171128215527.2510350-1-rwareing@fb.com/
> 
> With a dirty patch below for test only , Now realtime subvol will be used
> as I expected, and that can be confirmed by
> #xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.

mkfs.xfs -d rtinherit=1...

--D

> diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
> index f4cc8a1aaeb4..d19e0fa34c1a 100644
> --- a/fs/xfs/xfs_inode.c
> +++ b/fs/xfs/xfs_inode.c
> @@ -868,6 +868,9 @@ xfs_init_new_inode(
>                 flags |= XFS_ILOG_DEV;
>                 break;
>         case S_IFREG:
> +               if (xfs_has_realtime(ip->i_mount))
> +                       ip->i_diflags |= XFS_DIFLAG_REALTIME;
> +               fallthrough;
>         case S_IFDIR:
>                 if (pip && (pip->i_diflags & XFS_DIFLAG_ANY))
>                         xfs_inode_inherit_flags(ip, pip);
> 
> Thanks a lot.
> 
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2022/08/30
> 

Re: questions about hybird xfs wih ssd/hdd by realtime subvol

[Wang Yugui]

Hi,

> On Tue, Aug 30, 2022 at 08:57:21AM +0800, Wang Yugui wrote:
> > Hi,
> > 
> > > On Mon, Aug 29, 2022 at 10:26:20AM +0800, Wang Yugui wrote:
> > > > Hi,
> > > > 
> > > > I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.
> > > > 
> > > > Hybrid XFS―Using SSDs to Supercharge HDDs at Facebook
> > > > https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
> > > > 
> > > > There are some questions about how to control the data to save into
> > > > normal vol or realtime subvol firstly.
> > > > 
> > > > 1, man xfsctl
> > > > here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 ,
> > > > but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19.
> > > > xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?
> > > 
> > > It's been a while since XFS uses FS_XFLAG features directly, so, what you're
> > > specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a
> > > preprocessor define:
> > > 
> > > #define XFS_XFLAG_REALTIME	FS_XFLAG_REALTIME
> > > 
> > > FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without
> > > the realtime filesystems going first, so, unlikely it's gonna happen.
> > > 
> > > > 
> > > > 2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?
> > > 
> > > You can use xfs_io's chattr command to add/remote the REALTIME attribute of a
> > > file.
> > > 
> > > 
> > > > 
> > > > 3, we build a xfs filesystem with 1G device and 1G rtdev device. and
> > > > then we can save 2G data into this xfs filesystem.
> > 
> > Sorry, I cheched again.
> > This is a xfs filesystem with 2G device and 2G rtdev device
> > 
> > > > Is there any tool/kernel option/kernel patch to control the data to save
> > > > into normal vol or realtime subvol firstly?
> > > 
> > > I didn't watch the talk you mentioned above, but when use an rt device, you
> > > don't use the 'normal' one then the rt later, or vice-versa, the rt-device is
> > > used to store data blocks for those files marked with the xattr above. For those
> > > files you want to store in the realtime device, you should add the above xattr
> > > to them.
> > 
> > Although I still fail to check/set the attr by 'lsattr/chattr', but I
> > can check the free space of 'normal' and realtime subvol now.
> > 
> > # xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
> > typedef struct xfs_sb {
> > ...
> > 	uint64_t	sb_fdblocks;	/* free data blocks */
> > 	uint64_t	sb_frextents;	/* free realtime extents */
> > ...
> > }
> > 
> > And based the info from Carlos Maiolino
> > 
> > FB were running a modified kernel that selected the rt dev based on
> > the initial allocation size. Behaviour for them was predictable
> > because they also controlled the application that was storing the
> > data. See:
> > 
> > https://lore.kernel.org/linux-xfs/20171128215527.2510350-1-rwareing@fb.com/
> > 
> > With a dirty patch below for test only , Now realtime subvol will be used
> > as I expected, and that can be confirmed by
> > #xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.
> 
> mkfs.xfs -d rtinherit=1...

'mkfs.xfs -d rtinherit=1' works very well. Thanks a lot.

I noticed that 'reflink not supported with realtime devices' both in
mkfs.xfs and in kernel/xfs(c14632ddac98 xfs: don't allow reflink +
realtime filesystems).

Is this a limit of current design?
or we just disable reflink when realtime devices  just for QoS/realtime?

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2022/08/31

Re: questions about hybird xfs wih ssd/hdd by realtime subvol

[Darrick J. Wong]

On Wed, Aug 31, 2022 at 08:50:50AM +0800, Wang Yugui wrote:
> Hi,
> 
> > On Tue, Aug 30, 2022 at 08:57:21AM +0800, Wang Yugui wrote:
> > > Hi,
> > > 
> > > > On Mon, Aug 29, 2022 at 10:26:20AM +0800, Wang Yugui wrote:
> > > > > Hi,
> > > > > 
> > > > > I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.
> > > > > 
> > > > > Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
> > > > > https://www.usenix.org/conference/srecon19asia/presentation/shamasunder
> > > > > 
> > > > > There are some questions about how to control the data to save into
> > > > > normal vol or realtime subvol firstly.
> > > > > 
> > > > > 1, man xfsctl
> > > > > here is XFS_XFLAG_REALTIME in man xfsctl of xfsprogs 5.0 ,
> > > > > but there is no XFS_XFLAG_REALTIME in xfsprogs 5.14/5.19.
> > > > > xfsctl(XFS_XFLAG_REALTIME) will be removed in the further?
> > > > 
> > > > It's been a while since XFS uses FS_XFLAG features directly, so, what you're
> > > > specifically looking for is FS_XFLAG_REALTIME. xfsprogs today only has a
> > > > preprocessor define:
> > > > 
> > > > #define XFS_XFLAG_REALTIME	FS_XFLAG_REALTIME
> > > > 
> > > > FS_XFLAG_REALTIME is part of the xfs realtime, unlikely it's going away without
> > > > the realtime filesystems going first, so, unlikely it's gonna happen.
> > > > 
> > > > > 
> > > > > 2, Is there some tool to do xfsctl(XFS_XFLAG_REALTIME)?
> > > > 
> > > > You can use xfs_io's chattr command to add/remote the REALTIME attribute of a
> > > > file.
> > > > 
> > > > 
> > > > > 
> > > > > 3, we build a xfs filesystem with 1G device and 1G rtdev device. and
> > > > > then we can save 2G data into this xfs filesystem.
> > > 
> > > Sorry, I cheched again.
> > > This is a xfs filesystem with 2G device and 2G rtdev device
> > > 
> > > > > Is there any tool/kernel option/kernel patch to control the data to save
> > > > > into normal vol or realtime subvol firstly?
> > > > 
> > > > I didn't watch the talk you mentioned above, but when use an rt device, you
> > > > don't use the 'normal' one then the rt later, or vice-versa, the rt-device is
> > > > used to store data blocks for those files marked with the xattr above. For those
> > > > files you want to store in the realtime device, you should add the above xattr
> > > > to them.
> > > 
> > > Although I still fail to check/set the attr by 'lsattr/chattr', but I
> > > can check the free space of 'normal' and realtime subvol now.
> > > 
> > > # xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'
> > > typedef struct xfs_sb {
> > > ...
> > > 	uint64_t	sb_fdblocks;	/* free data blocks */
> > > 	uint64_t	sb_frextents;	/* free realtime extents */
> > > ...
> > > }
> > > 
> > > And based the info from Carlos Maiolino
> > > 
> > > FB were running a modified kernel that selected the rt dev based on
> > > the initial allocation size. Behaviour for them was predictable
> > > because they also controlled the application that was storing the
> > > data. See:
> > > 
> > > https://lore.kernel.org/linux-xfs/20171128215527.2510350-1-rwareing@fb.com/
> > > 
> > > With a dirty patch below for test only , Now realtime subvol will be used
> > > as I expected, and that can be confirmed by
> > > #xfs_db -c sb -c p /dev/sdb8 |grep 'fdblocks\|frextents'.
> > 
> > mkfs.xfs -d rtinherit=1...
> 
> 'mkfs.xfs -d rtinherit=1' works very well. Thanks a lot.
> 
> I noticed that 'reflink not supported with realtime devices' both in
> mkfs.xfs and in kernel/xfs(c14632ddac98 xfs: don't allow reflink +
> realtime filesystems).
> 
> Is this a limit of current design?
> or we just disable reflink when realtime devices  just for QoS/realtime?

No, just unfinished features. :/

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink

--D

> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2022/08/31
> 
> 

Re: questions about hybird xfs wih ssd/hdd by realtime subvol

[Dave Chinner]

On Mon, Aug 29, 2022 at 10:26:20AM +0800, Wang Yugui wrote:
> Hi,
> 
> I saw some info about hybird xfs wih ssd/hdd  by realtime subvol.
> 
> Hybrid XFS—Using SSDs to Supercharge HDDs at Facebook
> https://www.usenix.org/conference/srecon19asia/presentation/shamasunder

....

> Is there any tool/kernel option/kernel patch to control the data to save
> into normal vol or realtime subvol firstly?

FB were running a modified kernel that selected the rt dev based on
the initial allocation size. Behaviour for them was predictable
because they also controlled the application that was storing the
data. See:

https://lore.kernel.org/linux-xfs/20171128215527.2510350-1-rwareing@fb.com/

-Dave.
-- 
Dave Chinner
david@fromorbit.com