[LSF/MM/BPF TOPIC] statx attributes

[LSF/MM/BPF TOPIC] statx attributes

[Steve French]

Following up on a discussion a few years ago about missing STATX
attributes, I noticed a case recently where some tools on other OS
have an option to skip offline files (e.g. the Windows equivalent of
grep, "findstr", and some Mac tools also seem to do this).

This reminded me that there are a few additional STATX attribute flags
that could be helpful beyond the 8 that are currently defined (e.g.
STATX_ATTR_COMPRESSED, STATX_ATTR_ENCRYPTED, STATX_ATTR_NO_DUMP,
STATX_ATTR_VERITY) and that it be worthwhile revisiting which
additional STATX attribute flags would be most useful.

"offline" could be helpful for fuse and cifs.ko and probably multiple
fs to be able to report, but there are likely other examples that
could help various filesystems.
-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] statx attributes

[Amir Goldstein]

On Thu, Mar 7, 2024 at 7:36 AM Steve French <smfrench@gmail.com> wrote:
>
> Following up on a discussion a few years ago about missing STATX
> attributes, I noticed a case recently where some tools on other OS
> have an option to skip offline files (e.g. the Windows equivalent of
> grep, "findstr", and some Mac tools also seem to do this).
>

Which API is used in other OS to query the offline bit?
Do they use SMB specific API, as Windows does?

> This reminded me that there are a few additional STATX attribute flags
> that could be helpful beyond the 8 that are currently defined (e.g.
> STATX_ATTR_COMPRESSED, STATX_ATTR_ENCRYPTED, STATX_ATTR_NO_DUMP,
> STATX_ATTR_VERITY) and that it be worthwhile revisiting which
> additional STATX attribute flags would be most useful.

I agree that it would be interesting to talk about new STATX_ attributes,
but it should already be covered by this talk:
https://lore.kernel.org/linux-fsdevel/2uvhm6gweyl7iyyp2xpfryvcu2g3padagaeqcbiavjyiis6prl@yjm725bizncq/

We have a recent example of what I see as a good process of
introducing new STATX_ attributes:
https://lore.kernel.org/linux-fsdevel/20240302220203.623614-1-kent.overstreet@linux.dev/
1. Kent needed stx_subvol_id for bcachefs, so he proposed a patch
2. The minimum required bikeshedding on the name ;)
3. Buy in by at least one other filesystem (btrfs)

w.r.t attributes that only serve one filesystem, certainly a requirement from
general purpose userspace tools will go a long way to help when introducing
new attributes such as STATX_ATTR_OFFLINE, so if you get userspace
projects to request this functionality I think you should be good to go.

>
> "offline" could be helpful for fuse and cifs.ko and probably multiple
> fs to be able to report,

I am not sure why you think that "offline" will be useful to fuse?
Is there any other network fs that already has the concept of "offline"
attribute?

> but there are likely other examples that could help various filesystems.

Maybe interesting for network fs that are integrated with fscache/netfs?
It may be useful for netfs to be able to raise the STATX_ATTR_OFFLINE
attribute for a certain cached file in some scenarios?

As a developer of HSM API [1], where files on any fs could have an
"offline" status,
STATX_ATTR_OFFLINE is interesting to me, but only if local disk fs
will map it to
persistent inode flags.

When I get to it, I may pick a victim local fs and write a patch for it.

Thanks,
Amir.

[1] https://github.com/amir73il/fsnotify-utils/wiki/Hierarchical-Storage-Management-API

Re: [LSF/MM/BPF TOPIC] statx attributes

[Steve French]

> Which API is used in other OS to query the offline bit?
> Do they use SMB specific API, as Windows does?

No it is not smb specific - a local fs can also report this.  It is
included in the attribute bits for files and directories, it also
includes a few additional bits that are used by HSM software on local
drives (e.g. FILE_ATTRIBUTE_PINNED when the file may not be taken
offline by HSM software)
ATTRIBUTE_HIDDEN
ATTRIBUTE_SYSTEM
ATTRIBUTE_DIRECTORY
ATTRIGBUTE_ARCHIVE
ATTRIBUTE_TEMPORARY
ATTRIBUTE_SPARSE_FILE
ATTRIBUTE_REPARE_POINT
ATTRIBUTE_COMPRESSED
ATTRIBUTE_NOT_CONTENT_INDEXED
ATTRIBUTE_ENCRYPTED
ATTRIBUTE_OFFLINE

On Thu, Mar 7, 2024 at 2:54 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Thu, Mar 7, 2024 at 7:36 AM Steve French <smfrench@gmail.com> wrote:
> >
> > Following up on a discussion a few years ago about missing STATX
> > attributes, I noticed a case recently where some tools on other OS
> > have an option to skip offline files (e.g. the Windows equivalent of
> > grep, "findstr", and some Mac tools also seem to do this).
> >
>
> Which API is used in other OS to query the offline bit?
> Do they use SMB specific API, as Windows does?
>
> > This reminded me that there are a few additional STATX attribute flags
> > that could be helpful beyond the 8 that are currently defined (e.g.
> > STATX_ATTR_COMPRESSED, STATX_ATTR_ENCRYPTED, STATX_ATTR_NO_DUMP,
> > STATX_ATTR_VERITY) and that it be worthwhile revisiting which
> > additional STATX attribute flags would be most useful.
>
> I agree that it would be interesting to talk about new STATX_ attributes,
> but it should already be covered by this talk:
> https://lore.kernel.org/linux-fsdevel/2uvhm6gweyl7iyyp2xpfryvcu2g3padagaeqcbiavjyiis6prl@yjm725bizncq/
>
> We have a recent example of what I see as a good process of
> introducing new STATX_ attributes:
> https://lore.kernel.org/linux-fsdevel/20240302220203.623614-1-kent.overstreet@linux.dev/
> 1. Kent needed stx_subvol_id for bcachefs, so he proposed a patch
> 2. The minimum required bikeshedding on the name ;)
> 3. Buy in by at least one other filesystem (btrfs)
>
> w.r.t attributes that only serve one filesystem, certainly a requirement from
> general purpose userspace tools will go a long way to help when introducing
> new attributes such as STATX_ATTR_OFFLINE, so if you get userspace
> projects to request this functionality I think you should be good to go.
>
> >
> > "offline" could be helpful for fuse and cifs.ko and probably multiple
> > fs to be able to report,
>
> I am not sure why you think that "offline" will be useful to fuse?
> Is there any other network fs that already has the concept of "offline"
> attribute?
>
> > but there are likely other examples that could help various filesystems.
>
> Maybe interesting for network fs that are integrated with fscache/netfs?
> It may be useful for netfs to be able to raise the STATX_ATTR_OFFLINE
> attribute for a certain cached file in some scenarios?
>
> As a developer of HSM API [1], where files on any fs could have an
> "offline" status,
> STATX_ATTR_OFFLINE is interesting to me, but only if local disk fs
> will map it to
> persistent inode flags.
>
> When I get to it, I may pick a victim local fs and write a patch for it.
>
> Thanks,
> Amir.
>
> [1] https://github.com/amir73il/fsnotify-utils/wiki/Hierarchical-Storage-Management-API



-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] statx attributes

[Kent Overstreet]

On Thu, Mar 07, 2024 at 10:37:13AM -0600, Steve French wrote:
> > Which API is used in other OS to query the offline bit?
> > Do they use SMB specific API, as Windows does?
> 
> No it is not smb specific - a local fs can also report this.  It is
> included in the attribute bits for files and directories, it also
> includes a few additional bits that are used by HSM software on local
> drives (e.g. FILE_ATTRIBUTE_PINNED when the file may not be taken
> offline by HSM software)
> ATTRIBUTE_HIDDEN
> ATTRIBUTE_SYSTEM
> ATTRIBUTE_DIRECTORY
> ATTRIGBUTE_ARCHIVE
> ATTRIBUTE_TEMPORARY
> ATTRIBUTE_SPARSE_FILE
> ATTRIBUTE_REPARE_POINT
> ATTRIBUTE_COMPRESSED
> ATTRIBUTE_NOT_CONTENT_INDEXED
> ATTRIBUTE_ENCRYPTED
> ATTRIBUTE_OFFLINE

we've already got some of these as inode flags available with the
getflags ioctl (compressed, also perhaps encrypted?) - but statx does
seem a better place for them.

statx can also report when they're supported, which does make sense for
these.

ATTRIBUTE_DIRECTORY, though?

we also need to try to define the semantics for these and not just dump
them in as just a bunch of identifiers if we want them to be used by
other things - and we do.

ATTRIBUTE_TEMPORARY is the one I'm eyeing; I've been planning tmpfile
support in bcachefs, it'll turn fsyncs into noops and also ensure files
are deleted on unmount/remount.

Re: [LSF/MM/BPF TOPIC] statx attributes

[Steve French]

On Thu, Mar 7, 2024 at 11:45 AM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Thu, Mar 07, 2024 at 10:37:13AM -0600, Steve French wrote:
> > > Which API is used in other OS to query the offline bit?
> > > Do they use SMB specific API, as Windows does?
> >
> > No it is not smb specific - a local fs can also report this.  It is
> > included in the attribute bits for files and directories, it also
> > includes a few additional bits that are used by HSM software on local
> > drives (e.g. FILE_ATTRIBUTE_PINNED when the file may not be taken
> > offline by HSM software)
> > ATTRIBUTE_HIDDEN
> > ATTRIBUTE_SYSTEM
> > ATTRIBUTE_DIRECTORY
> > ATTRIGBUTE_ARCHIVE
> > ATTRIBUTE_TEMPORARY
> > ATTRIBUTE_SPARSE_FILE
> > ATTRIBUTE_REPARE_POINT
> > ATTRIBUTE_COMPRESSED
> > ATTRIBUTE_NOT_CONTENT_INDEXED
> > ATTRIBUTE_ENCRYPTED
> > ATTRIBUTE_OFFLINE
>
> we've already got some of these as inode flags available with the
> getflags ioctl (compressed, also perhaps encrypted?) - but statx does
> seem a better place for them.
>
> statx can also report when they're supported, which does make sense for
> these.
>
> ATTRIBUTE_DIRECTORY, though?
>
> we also need to try to define the semantics for these and not just dump
> them in as just a bunch of identifiers if we want them to be used by
> other things - and we do.

They are all pretty clearly defined, but many are already in Linux,
and a few are not relevant (e.g. ATTRIBUTE_DIRECTORY is handled in
mode bits).  I suspect that Macs have equivalents of most of these
too.


-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] statx attributes

[Andrew Walker]

On Thu, Mar 7, 2024 at 2:04 PM Steve French <smfrench@gmail.com> wrote:
>
> On Thu, Mar 7, 2024 at 11:45 AM Kent Overstreet
> <kent.overstreet@linux.dev> wrote:
> >
> > On Thu, Mar 07, 2024 at 10:37:13AM -0600, Steve French wrote:
> > > > Which API is used in other OS to query the offline bit?
> > > > Do they use SMB specific API, as Windows does?
> > >
> > > No it is not smb specific - a local fs can also report this.  It is
> > > included in the attribute bits for files and directories, it also
> > > includes a few additional bits that are used by HSM software on local
> > > drives (e.g. FILE_ATTRIBUTE_PINNED when the file may not be taken
> > > offline by HSM software)
> > > ATTRIBUTE_HIDDEN
> > > ATTRIBUTE_SYSTEM
> > > ATTRIBUTE_DIRECTORY
> > > ATTRIGBUTE_ARCHIVE
> > > ATTRIBUTE_TEMPORARY
> > > ATTRIBUTE_SPARSE_FILE
> > > ATTRIBUTE_REPARE_POINT
> > > ATTRIBUTE_COMPRESSED
> > > ATTRIBUTE_NOT_CONTENT_INDEXED
> > > ATTRIBUTE_ENCRYPTED
> > > ATTRIBUTE_OFFLINE
> >
> > we've already got some of these as inode flags available with the
> > getflags ioctl (compressed, also perhaps encrypted?) - but statx does
> > seem a better place for them.
> >
> > statx can also report when they're supported, which does make sense for
> > these.
> >
> > ATTRIBUTE_DIRECTORY, though?
> >
> > we also need to try to define the semantics for these and not just dump
> > them in as just a bunch of identifiers if we want them to be used by
> > other things - and we do.
>
> They are all pretty clearly defined, but many are already in Linux,
> and a few are not relevant (e.g. ATTRIBUTE_DIRECTORY is handled in
> mode bits).  I suspect that Macs have equivalents of most of these
> too.

MacOS and FreeBSD return many of these in stat(2) output via st_flags.
Current set of supported flags are documented in chflags(2) manpage on both
platforms.