Re: Immutable vs read-only for Windows compatibility

["Pali Rohár" <pali@kernel.org>]

Some updates...

On Sunday 02 February 2025 16:23:43 Pali Rohár wrote:
> And how many bit flags are needed? I have done some investigation. Lets
> start with table which describes all 32 possible bit flags which are
> used by Windows system and also by filesystems FAT / exFAT / NTFS / ReFS
> and also by SMB over network:
> 
> bit / attrib.exe flag / SDK constant / description
> 
>  0 - R - FILE_ATTRIBUTE_READONLY              - writing to file or deleting it is disallowed
>  1 - H - FILE_ATTRIBUTE_HIDDEN                - inode is hidden
>  2 - S - FILE_ATTRIBUTE_SYSTEM                - inode is part of operating system
>  3 -   - FILE_ATTRIBUTE_VOLUME                - inode is the disk volume label entry
>  4 -   - FILE_ATTRIBUTE_DIRECTORY             - inode is directory
>  5 - A - FILE_ATTRIBUTE_ARCHIVE               - inode was not archived yet (when set)
>  6 -   - FILE_ATTRIBUTE_DEVICE                - inode represents  in-memory device (e.g. C:\), flag not stored on filesystem
>  7 -   - FILE_ATTRIBUTE_NORMAL                - no other flag is set (value 0 means to not change flags, bit 7 means to clear all flags)
>  8 -   - FILE_ATTRIBUTE_TEMPORARY             - inode data do not have to be flushed to disk
>  9 -   - FILE_ATTRIBUTE_SPARSE_FILE           - file is sparse with holes
> 10 -   - FILE_ATTRIBUTE_REPARSE_POINT         - inode has attached reparse point (symlink is also reparse point)
> 11 -   - FILE_ATTRIBUTE_COMPRESSED            - file is compressed, for directories it means that newly created inodes would have this flag set
> 12 - O - FILE_ATTRIBUTE_OFFLINE               - HSM - inode is used by HSM
> 13 - I - FILE_ATTRIBUTE_NOT_CONTENT_INDEXED   - inode will not be indexed by content indexing service
> 14 -   - FILE_ATTRIBUTE_ENCRYPTED             - file is encrypted, for directories it means that newly created inodes would have this flag set
> 15 - V - FILE_ATTRIBUTE_INTEGRITY_STREAM      - fs does checksumming of data and metadata when reading inode, read-only

FILE_ATTRIBUTE_INTEGRITY_STREAM can be enabled for individual inode via
FSCTL_SET_INTEGRITY_INFORMATION or FSCTL_SET_INTEGRITY_INFORMATION_EX
fs ioctl call, available on Windows and also via SMB protocol. So
de-facto it is read-write attribute, just over SMB requires separate
operation for changing it.

In similar way can be modified also FILE_ATTRIBUTE_COMPRESSED and
FILE_ATTRIBUTE_ENCRYPTED attributes.

> 16 -   - FILE_ATTRIBUTE_VIRTUAL               - inode is in %LocalAppData%\VirtualStore, flag not stored on filesystem
> 17 - X - FILE_ATTRIBUTE_NO_SCRUB_DATA         - do not use scrubber (proactive background data integrity scanner) on this file, for directories it means that newly created inodes would have this flag set
> 18 -   - FILE_ATTRIBUTE_EA                    - inode has xattrs, (not in readdir output, shares same bit with FILE_ATTRIBUTE_RECALL_ON_OPEN)
> 18 -   - FILE_ATTRIBUTE_RECALL_ON_OPEN        - HSM - inode is not stored locally (only in readdir output, shares same bit with FILE_ATTRIBUTE_EA)
> 19 - P - FILE_ATTRIBUTE_PINNED                - HSM - inode data content must be always stored on locally
> 20 - U - FILE_ATTRIBUTE_UNPINNED              - HSM - inode data content can be removed from local storage
> 21 -   -                                      - reserved
> 22 -   - FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS - HSM - inode data content is not stored locally
> 23 -   -                                      - reserved
> 24 -   -                                      - reserved
> 25 -   -                                      - reserved
> 26 -   -                                      - reserved
> 27 -   -                                      - reserved
> 28 -   -                                      - reserved
> 29 - B - FILE_ATTRIBUTE_STRICTLY_SEQUENTIAL   - SMR Blob, unknown meaning, read-only
> 30 -   -                                      - reserved
> 31 -   -                                      - reserved
> 
> (HSM means Hierarchical Storage Management software, which uses reparse
> points to make some remote file/folder available on the local
> filesystem, for example OneDrive or DropBox)
> 
> From above list only following bit flags are suitable for modification
> over some Linux API:
> - FILE_ATTRIBUTE_READONLY
> - FILE_ATTRIBUTE_HIDDEN
> - FILE_ATTRIBUTE_SYSTEM
> - FILE_ATTRIBUTE_ARCHIVE
> - FILE_ATTRIBUTE_TEMPORARY
> - FILE_ATTRIBUTE_COMPRESSED
> - FILE_ATTRIBUTE_OFFLINE
> - FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
> - FILE_ATTRIBUTE_ENCRYPTED
> - FILE_ATTRIBUTE_NO_SCRUB_DATA
> - FILE_ATTRIBUTE_PINNED
> - FILE_ATTRIBUTE_UNPINNED

Hence this list needs to be extended by FILE_ATTRIBUTE_INTEGRITY_STREAM
attribute.

FILE_ATTRIBUTE_INTEGRITY_STREAM is interesting attribute as it allows to
enable checksumming of file content.

> And if I'm looking correctly the FILE_ATTRIBUTE_COMPRESSED can be
> already mapped to Linux FS_COMPR_FL / STATX_ATTR_COMPRESSED, which has
> same meaning. Also FILE_ATTRIBUTE_ENCRYPTED can be mapped to
> FS_ENCRYPT_FL / STATX_ATTR_ENCRYPTED. Note that these two flags cannot
> be set over WinAPI or SMB directly and it is required to use special
> WinAPI or SMB ioctl.
> 
> So totally are needed 10 new bit flags. And for future there are 9
> reserved bits which could be introduced by MS in future.
> 
> Additionally there are get-only attributes which can be useful for statx
> purposes (for example exported by cifs.ko SMB client):
> - FILE_ATTRIBUTE_REPARSE_POINT
> - FILE_ATTRIBUTE_INTEGRITY_STREAM
> - FILE_ATTRIBUTE_RECALL_ON_OPEN
> - FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS
> - FILE_ATTRIBUTE_STRICTLY_SEQUENTIAL
> 
> From the above list of flags suitable for modification, following bit
> flags have no meaning for kernel and it is up to userspace how will use
> them. What is needed from kernel and/or filesystem driver is to preserve
> those bit flags.
> - FILE_ATTRIBUTE_HIDDEN
> - FILE_ATTRIBUTE_SYSTEM
> - FILE_ATTRIBUTE_ARCHIVE
> - FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
> 
> Following are bit flags which kernel / VFS / fsdriver would have to
> handle specially, to provide enforcement or correct behavior of them:
> - FILE_ATTRIBUTE_READONLY - enforce that data modification or unlink is disallowed when set
> - FILE_ATTRIBUTE_COMPRESSED - enforce compression on filesystem when set
> - FILE_ATTRIBUTE_ENCRYPTED - enforce encryption on filesystem when set
> 
> Then there are HSM flags which for local filesystem would need some
> cooperation with userspace synchronization software. For network
> filesystems (SMB / NFS4) they need nothing special, just properly
> propagating them over network:
> - FILE_ATTRIBUTE_OFFLINE
> - FILE_ATTRIBUTE_PINNED
> - FILE_ATTRIBUTE_UNPINNED
> 
> About following 2 flags, I'm not sure if the kernel / VFS / fs driver
> has to do something or it can just store bits to fs:
> - FILE_ATTRIBUTE_TEMPORARY
> - FILE_ATTRIBUTE_NO_SCRUB_DATA