FAT, NTFS, CIFS and DOS attributes

FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

Hello all,

I recently posted to LKML a patch to get or set DOS attribute flags for 
fatfs.  That patch used ioctl().  It was suggested that a better way 
would be using xattrs, although the xattr mechanism seems clumsy to me, 
and has namespace issues.

I also think it would be good to have a unified interface for FAT, NTFS 
and CIFS for these attributes.

I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
although it doesn't seem to be implemented.

Questions:

a) is xattr the right thing?  It seems to be a fairly complex and 
ill-thought-out mechanism all along, especially the whole namespace 
business (what is a system attribute to one filesystem is a user 
attribute to another, for example.)

b) if xattr is the right thing, shouldn't this be in the system 
namespace rather than the user namespace?

c) What should the representation be?  Binary byte?  String containing a 
subset of "rhsvda67" (barf)?

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[Michael B Allen]

H. Peter Anvin said:
> I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c,
> although it doesn't seem to be implemented.
>
> Questions:
>
> a) is xattr the right thing?  It seems to be a fairly complex and
> ill-thought-out mechanism all along, especially the whole namespace
> business (what is a system attribute to one filesystem is a user
> attribute to another, for example.)

Ahh, just go with xattrs Pete :-> I don't see the namespace issue to be a
big deal. The interface does seem a *little* overdesigned. It would have
been adequate to just use the dev:ino pair from stat(2) and dump
namespaces altogether since the real performance critical apps will have
stat'd the living daylights out of the path trying to canonicalize the
case so the last thing you want to do is a path lookup.

> b) if xattr is the right thing, shouldn't this be in the system
> namespace rather than the user namespace?

If we're just thinking about MS-oriented discretionary access control then
I think the owner of the file is basically king and should be the only
normal user to that can read and write it's xattrs. So whatever namespace
that is (not system).

> c) What should the representation be?  Binary byte?  String containing a
> subset of "rhsvda67" (barf)?

Definitely binary.

Mike

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

Michael B Allen wrote:
> 
>>b) if xattr is the right thing, shouldn't this be in the system
>>namespace rather than the user namespace?
> 
> If we're just thinking about MS-oriented discretionary access control then
> I think the owner of the file is basically king and should be the only
> normal user to that can read and write it's xattrs. So whatever namespace
> that is (not system).
> 

system namespace means that it's a name defined by the kernel as opposed 
to a name defined by the user.  One of the most glaring design errors in 
this whole thing, in my opinion, but if we're going to use xattrs we 
probably should stick with it.

Thus, I'd propose:

	system.dosattrib	- DOS attributes (single byte)
	system.dosshortname	- DOS short name (e.g. for VFAT)

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[Michael B Allen]

H. Peter Anvin said:
> Michael B Allen wrote:
>>
>>>b) if xattr is the right thing, shouldn't this be in the system
>>>namespace rather than the user namespace?
>>
>> If we're just thinking about MS-oriented discretionary access control
>> then
>> I think the owner of the file is basically king and should be the only
>> normal user to that can read and write it's xattrs. So whatever
>> namespace
>> that is (not system).
>>
>
> system namespace means that it's a name defined by the kernel as opposed
> to a name defined by the user.  One of the most glaring design errors in
> this whole thing, in my opinion, but if we're going to use xattrs we
> probably should stick with it.
>
> Thus, I'd propose:
>
> 	system.dosattrib	- DOS attributes (single byte)
> 	system.dosshortname	- DOS short name (e.g. for VFAT)

Oh, from the man page I thought 'system' attributes were "extended
attributes to which ordinary processes should not have access". Is that
not true? If it is true, how would an ordinary user change these attrs?

Mike

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

Michael B Allen wrote:
>>
>>Thus, I'd propose:
>>
>>	system.dosattrib	- DOS attributes (single byte)
>>	system.dosshortname	- DOS short name (e.g. for VFAT)
> 
> 
> Oh, from the man page I thought 'system' attributes were "extended
> attributes to which ordinary processes should not have access". Is that
> not true? If it is true, how would an ordinary user change these attrs?
> 

Nah:

    Extended system attributes
        Extended system attributes are used by the kernel to store  sys-
        tem objects such as Access Control Lists and Capabilities.  Read
        and write access permissions to system attributes depend on  the
        policy  implemented  for  each  system  attribute implemented by
        filesystems in the kernel.

It's not *that* broken...

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > 	system.dosattrib	- DOS attributes (single byte)
 > 	system.dosshortname	- DOS short name (e.g. for VFAT)

you need more than one byte for DOS attrib. These are the bits Samba4
defines:

/* FileAttributes (search attributes) field */
#define FILE_ATTRIBUTE_READONLY		0x0001
#define FILE_ATTRIBUTE_HIDDEN		0x0002
#define FILE_ATTRIBUTE_SYSTEM		0x0004
#define FILE_ATTRIBUTE_VOLUME		0x0008
#define FILE_ATTRIBUTE_DIRECTORY	0x0010
#define FILE_ATTRIBUTE_ARCHIVE		0x0020
#define FILE_ATTRIBUTE_DEVICE		0x0040
#define FILE_ATTRIBUTE_NORMAL		0x0080
#define FILE_ATTRIBUTE_TEMPORARY	0x0100
#define FILE_ATTRIBUTE_SPARSE		0x0200
#define FILE_ATTRIBUTE_REPARSE_POINT	0x0400
#define FILE_ATTRIBUTE_COMPRESSED	0x0800
#define FILE_ATTRIBUTE_OFFLINE		0x1000
#define FILE_ATTRIBUTE_NONINDEXED	0x2000
#define FILE_ATTRIBUTE_ENCRYPTED	0x4000

while most apps don't care about the bits beyond 0xFF at the moment, I
think that might change, especially for win32 clients accessing linux
filesystems via wine and Samba.

Also, there are many other bits of windows meta-data that matter for
apps that care about dos attributes, including the extra 1.5 time
fields (windows has 4 settable time fields, posix has 2 settable and 3
readable time fields), the 8.3 name, the allocation size and all the
DOS EAs (important for OS/2 clients).

We use the following xattrs in Samba4:

 user.DosAttrib     : structure holding basic non-privileged attribute information
 user.DosEAs        : all the DOS (OS/2) style EAs
 user.DosStreams    : list of alternate data steams, flagged as internal or external
 user.DosStream.name: the stream data itself for internal streams
 security.NTACL     : the NT ACL

the rationale for making most of them in the user namespace is that it
is 'mostly harmless' to allow the owner of the file to change those
ones. The NTACL needs to be in the security namespace as it contains
elements that the user must not be able to control without system
permission (everyone can read it, but only root can write).

see xattr.idl for the full definitions of the above.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
> 
> We use the following xattrs in Samba4:
> 
>  user.DosAttrib     : structure holding basic non-privileged attribute information
>  user.DosEAs        : all the DOS (OS/2) style EAs
>  user.DosStreams    : list of alternate data steams, flagged as internal or external
>  user.DosStream.name: the stream data itself for internal streams
>  security.NTACL     : the NT ACL
> 
> the rationale for making most of them in the user namespace is that it
> is 'mostly harmless' to allow the owner of the file to change those
> ones.
 >

Right, it's the "design is broken so everything ends up in user.*". 
Now, I clearly dislike the StudlyCaps used here, but if it's already 
deployed it's probably too late to fix this :(

Does Samba have any way do deal with VFAT short names?

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > Right, it's the "design is broken so everything ends up in user.*". 
 > Now, I clearly dislike the StudlyCaps used here, but if it's already 
 > deployed it's probably too late to fix this :(

Samba4 is only deployed by a very few brave sites (such as my wifes
server) who all know that things might change in non-compatible
ways. Still, I'd want a slightly stronger reason than dislike of
studly caps to change it :-)

 > Does Samba have any way do deal with VFAT short names?

Samba doesn't take advantage of the fact that VFAT can store short
names directly in the filesystem, and instead deals with short names
completely in userspace using a hash based name mangling scheme. It
treats VFAT as just another unix filesystem. People who want to deploy
a serious Samba server tend to want journaling, ACLs etc, so VFAT
isn't a candidate.

I currently don't store short file names in xattrs for Samba4 as there
has just no advantage to doing so. Without a "open by xattr contents"
call that doesn't have to scan the entire directory it is much more
efficient to store the 8.3 names in user-space where we can look them
up more efficiently. The scheme we use is to store a cache of
8.3->longname mappings, and when we get a 8.3 name that isn't in cache
we fall back to a directory scan, re-forming the 8.3 name using a hash
for each directory entry.

I'm also much less concerned about 8.3 names these days than I was a
few years ago, as the number of applications that need them is rapidly
dropping. There are some obvious exceptions (such as the idiotic API calls
that cmd.exe uses to implement "del *.*"), but we have worked out ways
to cope with those in a reasonable manner.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
>  > Right, it's the "design is broken so everything ends up in user.*". 
>  > Now, I clearly dislike the StudlyCaps used here, but if it's already 
>  > deployed it's probably too late to fix this :(
> 
> Samba4 is only deployed by a very few brave sites (such as my wifes
> server) who all know that things might change in non-compatible
> ways. Still, I'd want a slightly stronger reason than dislike of
> studly caps to change it :-)
> 

The slightly stronger reason is basically the same reason why we don't 
stuff a bunch of things into a struct stat and call a single system call 
to change a bunch of attributes; you don't want to have to change them 
all every time, and by putting them all in the same structure that's 
your only option, since setxattr() doesn't allow you to mask and merge.

Incidentally, the document you pointed me to wasn't clear on the 
endianness convention.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > The slightly stronger reason is basically the same reason why we don't 
 > stuff a bunch of things into a struct stat and call a single system call 
 > to change a bunch of attributes

yes, and if we want this to be a good API then we'd use something like
a bitmask to indicate what fields to update so we can update them in
groups in a raceless fashion, but that would require that the kernel
understand the internals of these structures. I didn't have that
luxury, so I grouped them in the way that best matched the common use
of the attributes.

 > you don't want to have to change them all every time, and by
 > putting them all in the same structure that's your only option,
 > since setxattr() doesn't allow you to mask and merge.

can you tell me who you imagining will be using these attributes apart
from Samba, Wine and backup/restore apps? 

 > 
 > Incidentally, the document you pointed me to wasn't clear on the 
 > endianness convention.

It's little-endian NDR. For a full description of NDR in all its gory
details see http://www.opengroup.org/onlinepubs/9629399/chap14.htm

NDR seems like overkill at first, until you start to look at security
descriptors. They are very complex beasts, and using IDL/NDR makes it
much easier (in fact, security descriptors need some minor
enhancements to NDR to encode them the same way windows does).

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[Nicholas Miell]

On Mon, 2005-01-03 at 16:24 -0800, H. Peter Anvin wrote:
> Right, it's the "design is broken so everything ends up in user.*". 

The design isn't broken, you're just missing an important detail of what
the system namespace entails:

xattrs in the system namespace have a format defined by the kernel and
(more importantly -- this is the important detail) modify kernel
behavior.

If the xattr namespace was flat, I would have no way of knowing whether
or not the kernel will set the Archived bit in fatattrs (or DosAttrib)
xattr when I write to a file that has that xattr or whether or not the
kernel will choose to enforce the ACL I store in the posix_acl_access
xattr.

With the system namespace, I can rely on the fact that xattrs in that
namespace actually have a meaning and are in sync with what the kernel
believes to be true about the file.

If I cannot rely on this to be true, than at best that's a bug and at
worst it's a gaping security hole. (NT ACLs can specify Allow and Deny,
if I create a NT ACL xattr that denies somebody access, the kernel damn
well better enforce it.)

Earlier you mentioned a desire to be able to backup the various pieces
of metadata that a filesystem exports via xattrs simply by copying the
files to another filesystem, and the fact that the destination
filesystem may not allow you to store the same attributes in the system
namespace as the source prevented you from being able to do this.

This is akin to complaining that you cannot make an accurate backup of
an ext3 filesystem simply by copying it's files to a vfat filesystem,
because vfat doesn't support the same notions of timestamps, ownership
or permissions that ext3 does.

Tools such as tar or cpio exist to store Unix files and their metadata
in a flat format, and they can be extended to understand the extra
metadata made available in Linux xattrs.

-- 
Nicholas Miell <nmiell@comcast.net>

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

Nicholas Miell wrote:
> 
> The design isn't broken, you're just missing an important detail of what
> the system namespace entails:
> 
> xattrs in the system namespace have a format defined by the kernel and
> (more importantly -- this is the important detail) modify kernel
> behavior.
> 
> If the xattr namespace was flat, I would have no way of knowing whether
> or not the kernel will set the Archived bit in fatattrs (or DosAttrib)
> xattr when I write to a file that has that xattr or whether or not the
> kernel will choose to enforce the ACL I store in the posix_acl_access
> xattr.
> 
> With the system namespace, I can rely on the fact that xattrs in that
> namespace actually have a meaning and are in sync with what the kernel
> believes to be true about the file.
> 

What you're neglecting is that there is a LARGE class of metadata where 
the important thing is that you store them; if you don't know what they 
are you merely ignore them and keep them as-is.

There is no place for those in the current design.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[Nicholas Miell]

On Mon, 2005-01-03 at 17:48 -0800, H. Peter Anvin wrote:
> What you're neglecting is that there is a LARGE class of metadata where 
> the important thing is that you store them; if you don't know what they 
> are you merely ignore them and keep them as-is.
> 
> There is no place for those in the current design.

That's what the user namespace is for -- the kernel has no interest in
its contents, it just stores it as-is for apps that do.

-- 
Nicholas Miell <nmiell@comcast.net>

Re: [Linux-NTFS-Dev] Re: FAT, NTFS, CIFS and DOS attributes

[Szakacsits Szabolcs]

On Tue, 4 Jan 2005 tridge@samba.org wrote:

> you need more than one byte for DOS attrib. These are the bits Samba4
> defines:
> 
> /* FileAttributes (search attributes) field */
 [ ... ]
> #define FILE_ATTRIBUTE_SPARSE		0x0200
 [ ... ]
> #define FILE_ATTRIBUTE_COMPRESSED	0x0800
 [ ... ] 
>
> while most apps don't care about the bits beyond 0xFF at the moment, I
> think that might change, especially for win32 clients accessing linux
> filesystems via wine and Samba.

Setting the above two attributes from a Linux client accessing NTFS via
cifsfs pops up more often too. One scenario would be

  ntfsclone --fs-compression --output /mnt/cifsfs/backup.img /dev/foo

and the transparently compressed backup.img should be loopback mountable
by whatever preferred NTFS driver for whatever reason.

	Szaka

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

Mike,

 > If we're just thinking about MS-oriented discretionary access control then
 > I think the owner of the file is basically king and should be the only
 > normal user to that can read and write it's xattrs. So whatever namespace
 > that is (not system).

for the DACL the owner is king (the owner gets the WRITE_DAC,
READ_CONTROL and STD_DELETE access bits forced on), but for the other
parts of the full security descriptor this is not true. The owner
doesn't get to arbitrarily write to the owner_sid or SACL. Thats why I
used security.NTACL not user.NTACL.

I suppose we could have a separate user.DACL attribute, but given that
there is just one API that sets all 4 elements of the SD (with a
bitmask to say which bits to set), it made more sense to me to group
them all together. The disadvantage is that Samba needs to gain/lose
root privileges for the "set SD" call even if the client is only
asking to set the DACL.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
> Mike,
> 
>  > If we're just thinking about MS-oriented discretionary access control then
>  > I think the owner of the file is basically king and should be the only
>  > normal user to that can read and write it's xattrs. So whatever namespace
>  > that is (not system).
> 
> for the DACL the owner is king (the owner gets the WRITE_DAC,
> READ_CONTROL and STD_DELETE access bits forced on), but for the other
> parts of the full security descriptor this is not true. The owner
> doesn't get to arbitrarily write to the owner_sid or SACL. Thats why I
> used security.NTACL not user.NTACL.
> 
> I suppose we could have a separate user.DACL attribute, but given that
> there is just one API that sets all 4 elements of the SD (with a
> bitmask to say which bits to set), it made more sense to me to group
> them all together. The disadvantage is that Samba needs to gain/lose
> root privileges for the "set SD" call even if the client is only
> asking to set the DACL.
> 

Even more so a reason for this not to be a general API.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[Nicholas Miell]

On Mon, 2005-01-03 at 14:24 -0800, H. Peter Anvin wrote:
> Hello all,
> 
> I recently posted to LKML a patch to get or set DOS attribute flags for 
> fatfs.  That patch used ioctl().  It was suggested that a better way 
> would be using xattrs, although the xattr mechanism seems clumsy to me, 
> and has namespace issues.
> 
> I also think it would be good to have a unified interface for FAT, NTFS 
> and CIFS for these attributes.
> 
> I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
> although it doesn't seem to be implemented.
> 
> Questions:
> 
> a) is xattr the right thing?  It seems to be a fairly complex and 
> ill-thought-out mechanism all along, especially the whole namespace 
> business (what is a system attribute to one filesystem is a user 
> attribute to another, for example.)

More importantly, what has a defined meaning for one filesystem and is
interpreted and generated on demand by the kernel is irrelevant or
unsupported on other filesystems.

So, yes, you can't just copy a bunch of files from vfat to ext3 and
preserve the vfat attributes, but you should be able to stuff a bunch of
vfat files into a tar file and then restore them to a vfat filesystem
with all their original attributes intact.

> b) if xattr is the right thing, shouldn't this be in the system 
> namespace rather than the user namespace?

Yes.

> c) What should the representation be?  Binary byte?  String containing a 
> subset of "rhsvda67" (barf)?

ASCII strings require no special tools to manipulate from shell scripts
(or even for the end user to interpret).

-- 
Nicholas Miell <nmiell@comcast.net>

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
 > although it doesn't seem to be implemented.

Thats taken from Samba4, where it is fully implemented. I guess Steve
is planning on integrating cifsfs with the Samba4 way of handling EAs,
NT ACLs, attribs, streams etc at some stage.

See 
  http://samba.org/ftp/unpacked/samba4/source/librpc/idl/xattr.idl 
for a full definition of the structures we use. 

I used a NDR encoding in each of the xattrs to provide a well defined
architecture independent encoding, and an easy way to extend the
structure in the future (thats why DosAttrib is a union with a version
switch). 

The place where this will really interact a lot with the kernel is in
the Samba LSM module that tpot and myself have been looking at
writing. That module will provide the in-kernel implementation of
these attributes that is needed to make them raceless (especially for
NT ACLs).

These xattr structures are also the key to solving the
case-insensitivity problem that has been plaguing Samba for so
long. We have come up with a very simple scheme for making
case-insensiitive filenames work very efficiently on any filesystem
that supports xattrs. It requires no additional kernel support beyond
xattrs, and gets rid of the need for a large user-space name
cache.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
> 
> Thats taken from Samba4, where it is fully implemented. I guess Steve
> is planning on integrating cifsfs with the Samba4 way of handling EAs,
> NT ACLs, attribs, streams etc at some stage.
> 
> See 
>   http://samba.org/ftp/unpacked/samba4/source/librpc/idl/xattr.idl 
> for a full definition of the structures we use. 
> 
> I used a NDR encoding in each of the xattrs to provide a well defined
> architecture independent encoding, and an easy way to extend the
> structure in the future (thats why DosAttrib is a union with a version
> switch). 
> 

Oh geez.  Couldn't you have split out the various data items into 
separate xattrs?  This seems to be a really bad user interface, 
especially for writing (can't chmod the file without poking at all the 
other data items), except for a non-DOS-based filesystem to keep data 
for Samba itself.  Samba clearly has other needs than other users, 
although of course it would be unfortunate if Samba then can't export 
this information.

In other words, I'm inclined to define simple system attributes or just 
go back to the original ioctl() patch for the DOS filesystems as seen by 
the kernel.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > Oh geez.  Couldn't you have split out the various data items into 
 > separate xattrs?

fetching and setting this stuff is _really_ common, so I have arranged
to store them in a way to make it as efficient as possible. For
example, when a remote client asks for a directory listing we need to
fetch just one xattr from the kernel per file (directory listings
in the windows world usually return the equivalent of a full stat
structure for every name).

Similarly, when these are set the client tends to set more than one at
a time. The most commonly used set call takes this structure:

	/* RAW_SFILEINFO_BASIC_INFO and
	   RAW_SFILEINFO_BASIC_INFORMATION interfaces */
	struct {
		enum smb_setfileinfo_level level;
		union setfileinfo_file file;

		struct {
			NTTIME create_time;
			NTTIME access_time;
			NTTIME write_time;
			NTTIME change_time;
			uint32_t attrib;
		} in;
	} basic_info;

Having the items that tend to get read/written together grouped
together allowed me to make it all quite efficient, while still having
a reasonable chance of the EAs all fitting in-inode on filesystems
that support that.

Obviously it is racy when dealt with from user space, but there really
is no way to avoid all these races without a user space accessible
"lock the files meta-data" call and that is why I'm looking forward to
having a Samba LSM module to avoid these races.

 > Samba clearly has other needs than other users, although of course
 > it would be unfortunate if Samba then can't export this
 > information.

I think you'll find that all users of dos attributes on Linux will
have very similar needs to Samba, and will want these things grouped
together. For example:

 - backup/restore apps will want to backup/restore these attributes as
   lumps
 - wine implements essentially the same APIs as Samba, just in a
   different form, and so tends to get the same groupings of
   attributes get/set calls that Samba does (the SMB protocol is to a
   large degree a on-the-wire version of Win32).

Are there any other significant users of DOS attributes on Linux that
want something different?

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
> 
> I think you'll find that all users of dos attributes on Linux will
> have very similar needs to Samba, and will want these things grouped
> together. For example:
> 
>  - backup/restore apps will want to backup/restore these attributes as
>    lumps
>  - wine implements essentially the same APIs as Samba, just in a
>    different form, and so tends to get the same groupings of
>    attributes get/set calls that Samba does (the SMB protocol is to a
>    large degree a on-the-wire version of Win32).
> 
> Are there any other significant users of DOS attributes on Linux that
> want something different?
> 

I don't know, but it certainly doesn't match my application (which is 
pretty simple... needing to frob DOS attribute bits while writing DOS 
files) or expectation thereof... plus it's not very unixy :-/

More or less what you seem to want is an ioctl() that takes a mask of 
what to write, similar to the way notify_change() works inside the 
kernel.  This is a legitimate API, but it requires knowledge of the 
internals, and isn't setxattr().  The big thing here is the need for a mask.

Of course, one way one can do this is to expose user.dos.attrib or 
something like that as a synthesized block form of these, and still have 
separate system.* xattrs that control the individual options on the 
filesystems which actually store this stuff.

I certainly see why *you* want it, but it still seems to me to be bad 
interface design for anything other than backup and file repository 
programs.  Also see my previous note about endianness of structures 
carried from place to place.

At this point I think I'm going to let my ioctl() patch stand as-is, 
solving the immediate problem, and let people who are more directly 
affected delve into this particular morass.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > More or less what you seem to want is an ioctl() that takes a mask of 
 > what to write, similar to the way notify_change() works inside the 
 > kernel.  This is a legitimate API, but it requires knowledge of the 
 > internals, and isn't setxattr().  The big thing here is the need for a mask.

That API would make sense, but I didn't really expect the kernel to
provide it. What I expected to happen was for Samba4 to use the xattr
blobs like it does now, hopefully for Wine to learn to interpret those
same blobs, for backup/restore apps to learn to backup/restore them
(as blobs, with no interpretation) and for the proposed Samba LSM
module to do the dirty work of interpreting the contents of these
blobs in-kernel to provide raceless windows file serving. 

The LSM module would then expose a richer API to a user space library
via some yet to be determined mechanism (netlink? ioctl? sysfs? proc?
dunno yet). That API would include the ability to tell the LSM module
what "nt token" (the windows equivalent of euid, egid and
supplementary groups) to use for operations, thus allowing the module
to correctly interpret the NT ACLs for read/write access to each of
these attributes. The module would also cache the xattr blob contents,
in unpacked form, to allow access decisions to be made very fast.

The whole design was based on the idea that proposing anything more
intrusive would (quite rightly) get smacked down as "this is not NT,
go away".

 > Also see my previous note about endianness of structures carried
 > from place to place.

I specifically chose NDR (and specifically little-endian NDR) as it
solves the endianness and 32/64 bit problems. You can take these blobs
and put them on any platform and they will be interpreted the same.

We also have a mechanism (an external tdb) for storing these xattrs on
filesystems that have no xattr support. That allows Samba4 to be fully
functional on any platform, but just much more efficient and scalable
on platforms that do have xattrs.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
> 
> That API would make sense, but I didn't really expect the kernel to
> provide it. What I expected to happen was for Samba4 to use the xattr
> blobs like it does now, hopefully for Wine to learn to interpret those
> same blobs, for backup/restore apps to learn to backup/restore them
> (as blobs, with no interpretation) and for the proposed Samba LSM
> module to do the dirty work of interpreting the contents of these
> blobs in-kernel to provide raceless windows file serving. 
> 

Right, but here we're talking about exposing the guts of a DOS-based 
filesystem so they can be manipulated by an application which isn't 
necessarily a bulk file handler.  The API doesn't really work for that, 
especially since some of the attributes have different access properties 
from the others.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

 > Right, but here we're talking about exposing the guts of a DOS-based 
 > filesystem so they can be manipulated by an application which isn't 
 > necessarily a bulk file handler.  The API doesn't really work for that, 
 > especially since some of the attributes have different access properties 
 > from the others.

right. Samba doesn't care much about VFAT, and you don't care about
all the other attributes, so we should get along fine without treading
on each others toes too much.

I explained what Samba4 does as you asked about the user.DosAttrib
xattr that Steve put a placeholder for in cifsfs. That came from
Samba4, so if you suddenly started using it in a different way I would
get a sore toe :-)

Once the Samba LSM module is done and Wine and Samba start working
more together on all these extra bits of meta-data then we could
consider making your ioctl work on all filesystems when the LSM module
is loaded.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[H. Peter Anvin]

tridge@samba.org wrote:
> 
> right. Samba doesn't care much about VFAT, and you don't care about
> all the other attributes, so we should get along fine without treading
> on each others toes too much.
> 
> I explained what Samba4 does as you asked about the user.DosAttrib
> xattr that Steve put a placeholder for in cifsfs. That came from
> Samba4, so if you suddenly started using it in a different way I would
> get a sore toe :-)
> 
> Once the Samba LSM module is done and Wine and Samba start working
> more together on all these extra bits of meta-data then we could
> consider making your ioctl work on all filesystems when the LSM module
> is loaded.
> 

Sounds reasonable.

	-hpa

Re: FAT, NTFS, CIFS and DOS attributes

[Kyle Moffett]

On Jan 03, 2005, at 21:05, tridge@samba.org wrote:
> Once the Samba LSM module is done and Wine and Samba start working
> more together on all these extra bits of meta-data then we could
> consider making your ioctl work on all filesystems when the LSM module
> is loaded.

IMHO, it would be really nice (Although I realize this would be a 
humongous
task) if there was a "Linux" ACL system supported by the VFS that is 
backwards
compatible with UNIX perms, POSIX ACLs, and NTFS ACLs (One of the few
things that Microsoft got right).  The fact stands that NT ACLs are 
much more
powerful and useful than their POSIX counterparts (Although some of the
subdirectory inheritance stuff is rather uselessly algorithmically 
complicated.)

I think it would be a major achievement if some kind of design and API 
could
be decided upon, because at least that way there would be some kind of
foundation upon which to build patches.

I'd like to work on a patch for this, but I'm afraid my kernel coding 
skills are a
bit rusty at the moment. Sorry :-D

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

Kyle,

 > IMHO, it would be really nice (Although I realize this would be a
 > humongous task) if there was a "Linux" ACL system supported by the
 > VFS that is backwards compatible with UNIX perms, POSIX ACLs, and
 > NTFS ACLs

The problem is that apps/users that care about posix ACLs and NT ACLs
tend to care about them being _exactly_ right, so half baked mappings
between the different systems don't really help much. We have learnt
this the hard way in Samba3, where we attempt to map NT ACLs to posix
ACLs on the fly, and we get a never ending stream of bug reports that
NT ACLs aren't working as expected for windows users. Thats why I
stuck the full NT ACL in an xattr for Samba4.

The plan for mapping between the systems in Samba4 goes like this:

 - when a NT ACL is needed for access control, or the client asks for
   the ACL via a query file info call, smbd first looks in the NTACL
   xattr. If it finds it then all is well with the world.

 - if the NTACL xattr isn't there, then it produces one using an
   approximate mapping. The mapping for mode_t is in the function
   pvfs_default_acl() in pvfs_acl.c. The mapping when there is an
   existing posix ACL isn't done yet (we have a mapping for that in
   Samba3, but it will be a bit different in Samba4).

 - if its a file/directory create, then it also obeys the inheritance
   rules, pulling the NTACL from the parent using the above method
   then applying to the child.

I also plan to catch the posix ACL set call in the Samba LSM module,
and wipe the NTACL when that happens. In that case the NTACL will be
overridden by the new posix ACL.

Similarly, when a posix application asks to read the posix ACL, then a
reverse mapping would be supplied if there is a current NT ACL on the
file.

The important thing is that if your file access is either only from
unix apps that know about posix ACLs or only from windows apps that
know about NT ACLs, then you need to provide perfect lossless storage
of those native ACLs for those applications. It is only when you get a
transition from one scheme to the other on the one file that a mapping
should happen.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[Kyle Moffett]

On Jan 03, 2005, at 21:49, tridge@samba.org wrote:
> The important thing is that if your file access is either only from
> unix apps that know about posix ACLs or only from windows apps that
> know about NT ACLs, then you need to provide perfect lossless storage
> of those native ACLs for those applications. It is only when you get a
> transition from one scheme to the other on the one file that a mapping
> should happen.

I was thinking something more along the lines of a more complex and
detailed scheme that is a superset of both NT ACLs and POSIX ACLs.
Then that scheme could be used for DAC within the kernel, although it
would still continue to support operations using the POSIX API by
translating losslessly from POSIX to "Linux", although it might not work
the other way around.  Such an ACL would need to be well designed
internally to be compatible with future ACL structures and such, but it
would make your work a lot easier.  There will always be data loss
when translating between incompatible formats, but if we can come up
with some kind of system that can handle both data-sets losslessly,
then we could have the VFS use that, even if the only programs that
understand it are the ACL tools.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: FAT, NTFS, CIFS and DOS attributes

[tridge]

Kyle,

 > I was thinking something more along the lines of a more complex and
 > detailed scheme that is a superset of both NT ACLs and POSIX ACLs.

superset is hard, as a uid_t/gid_t is only superfically similar to a
windows SID. Samba has to do quite a lot of complex stuff to map
between general SIDs and posix IDs. It can't be done in any reasonable
fashion without being able to talk MSRPC to domain controllers, or at
least having a (potentially quite large) persistent database of
mappings.

The schemes that attempt to do general SID -> uid/gid mappings via
fixed algorithmic mappings are hopeless. They are great for toy demos,
but useless for real deployments.

Cheers, Tridge

Re: FAT, NTFS, CIFS and DOS attributes

[Kyle Moffett]

On Jan 03, 2005, at 22:56, tridge@samba.org wrote:
> superset is hard, as a uid_t/gid_t is only superfically similar to a
> windows SID. Samba has to do quite a lot of complex stuff to map
> between general SIDs and posix IDs. It can't be done in any reasonable
> fashion without being able to talk MSRPC to domain controllers, or at
> least having a (potentially quite large) persistent database of
> mappings.

I only have a couple points of response here, really.  Primarily, I 
didn't
say that I thought this was possible, let alone easy, only that it 
would be
closer to the mythical "ideal" than the system what we currently have.

My personal opinion is that perhaps this (ideally) should be integrated
somehow with an additional "credentials" system, perhaps like that
which dhowells added.  An "SID" for a particular local NTFS (or maybe
even remote via CIFS too) filesystem would be another "credential"
given to users.  Heck, if we _really_ wanted to go far, the "user" and
"group" identifiers originating in UNIX could be "credentials" too, and
passed on from parent to child processes.  In such a system, many of
the core process identity syscalls would need to be changed, but it
would be the framework for a much better network environment
integration, because the UID and GIDs would be local to one specific
computer, not global like NFS pretends they are.  Someone could
even have differing UIDs for local ext3 filesystems and remote NFSv3
and CIFS filesystems, like this:

/		"bobdoe@localhost" (32)
/mnt/nfs	"bdoe@cronos" (1054)
/mnt/cifs	"Bob Doe@ATLAS" (S-1-3-21-123123-456456-789789-32)

Of course, the standard disclaimer is that all this is far far down the
road, mainly because of how extensive such changes would be :-\.
It's always nice to dream, though :-D.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: FAT, NTFS, CIFS and DOS attributes

[Michael B Allen]

H. Peter Anvin said:
> In other words, I'm inclined to define simple system attributes or just
> go back to the original ioctl() patch for the DOS filesystems as seen by
> the kernel.

Well if you're just interested in DOS attributes you don't care about how
Samba handles ACLs. You're only interested in about 3 bits. Anyway I have
an idea...

If the attrib member in xattr_DosInfo2 was moved to the beginning of the
structure the all attrib members would always be encoded at the same
offset within the data returned by getxattr regardless of what info
strucutre was used. So you could get and set these attributes as
user.DosAttrib in a fairly compatible way.

Mike

Re: FAT, NTFS, CIFS and DOS attributes

[Anton Altaparmakov]

On Mon, 2005-01-03 at 14:24 -0800, H. Peter Anvin wrote:
> I recently posted to LKML a patch to get or set DOS attribute flags for 
> fatfs.  That patch used ioctl().  It was suggested that a better way 
> would be using xattrs, although the xattr mechanism seems clumsy to me, 
> and has namespace issues.
> 
> I also think it would be good to have a unified interface for FAT, NTFS 
> and CIFS for these attributes.
> 
> I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
> although it doesn't seem to be implemented.
> 
> Questions:
> 
> a) is xattr the right thing?  It seems to be a fairly complex and 
> ill-thought-out mechanism all along, especially the whole namespace 
> business (what is a system attribute to one filesystem is a user 
> attribute to another, for example.)
> 
> b) if xattr is the right thing, shouldn't this be in the system 
> namespace rather than the user namespace?
> 
> c) What should the representation be?  Binary byte?  String containing a 
> subset of "rhsvda67" (barf)?

Definitely not c!

In NTFS, the "dos attribute flags" are part of the system information
attribute which is an entity in its own right, totally separate from
extended attributes (and named streams for that matter).  So if I were
to be thinking in an NTFS-only world I would be inclined to use an
ioctl() to access/modify them (i.e. not b either).  So if you implement
an ioctl() for vfat I will probably be able to provide the same in NTFS
with almost zero effort (we already have the code to read and write the
attribute flags in the kernel ntfs driver, we just do not provide an
interface for it).

But please note that it would be best if you could use 32-bits for the
flags.  At the very least 16-bits though as on NTFS there are currently
in use 16-bits in the standard information but the field is u32 sized on
disk (little endian) and two of the higher bits are in use in the file
name attribute as well and I would not be surprised if more bits get
used in future NTFS releases.  Tridge already gave you a list of all the
Samba dos attributes so here is the full list for NTFS (note they are
100% compatible to the Samba ones and also note in NTFS we always keep
the flags in little endian and just define all the constants to be
little endian as well - makes life much easier):

/*
 * File attribute flags (32-bit).
 */
enum {
        /*
         * The following flags are only present in the
STANDARD_INFORMATION
         * attribute (in the field file_attributes).
         */
        FILE_ATTR_READONLY              = const_cpu_to_le32(0x00000001),
        FILE_ATTR_HIDDEN                = const_cpu_to_le32(0x00000002),
        FILE_ATTR_SYSTEM                = const_cpu_to_le32(0x00000004),
        /* Old DOS volid. Unused in NT. = const_cpu_to_le32(0x00000008),
*/
                                                                                
        FILE_ATTR_DIRECTORY             = const_cpu_to_le32(0x00000010),
        /* Note, FILE_ATTR_DIRECTORY is not considered valid in NT.  It
is
           reserved for the DOS SUBDIRECTORY flag. */
        FILE_ATTR_ARCHIVE               = const_cpu_to_le32(0x00000020),
        FILE_ATTR_DEVICE                = const_cpu_to_le32(0x00000040),
        FILE_ATTR_NORMAL                = const_cpu_to_le32(0x00000080),
                                                                                
        FILE_ATTR_TEMPORARY             = const_cpu_to_le32(0x00000100),
        FILE_ATTR_SPARSE_FILE           = const_cpu_to_le32(0x00000200),
        FILE_ATTR_REPARSE_POINT         = const_cpu_to_le32(0x00000400),
        FILE_ATTR_COMPRESSED            = const_cpu_to_le32(0x00000800),
                                                                                
        FILE_ATTR_OFFLINE               = const_cpu_to_le32(0x00001000),
        FILE_ATTR_NOT_CONTENT_INDEXED   = const_cpu_to_le32(0x00002000),
        FILE_ATTR_ENCRYPTED             = const_cpu_to_le32(0x00004000),
                                                                                
        FILE_ATTR_VALID_FLAGS           = const_cpu_to_le32(0x00007fb7),
        /* Note, FILE_ATTR_VALID_FLAGS masks out the old DOS VolId and
the
           FILE_ATTR_DEVICE and preserves everything else.  This mask is
used
           to obtain all flags that are valid for reading. */
        FILE_ATTR_VALID_SET_FLAGS       = const_cpu_to_le32(0x000031a7),
        /* Note, FILE_ATTR_VALID_SET_FLAGS masks out the old DOS VolId,
the
           F_A_DEVICE, F_A_DIRECTORY, F_A_SPARSE_FILE,
F_A_REPARSE_POINT,
           F_A_COMPRESSED, and F_A_ENCRYPTED and preserves the rest.
This mask
           is used to to obtain all flags that are valid for setting. */
                                                                                
        /*
         * The following flags are only present in the FILE_NAME
attribute (in
         * the field file_attributes).
         */
        FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT   =
const_cpu_to_le32(0x10000000),        /* Note, this is a copy of the
corresponding bit from the mft record,
           telling us whether this is a directory or not, i.e. whether
it has
           an index root attribute or not. */
        FILE_ATTR_DUP_VIEW_INDEX_PRESENT        =
const_cpu_to_le32(0x20000000),        /* Note, this is a copy of the
corresponding bit from the mft record,
           telling us whether this file has a view index present (eg.
object id
           index, quota index, one of the security indexes or the
encrypting
           file system related indexes). */
};
                                                                                
typedef le32 FILE_ATTR_FLAGS;

Best regards,

        Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: FAT, NTFS, CIFS and DOS attributes

[Anton Altaparmakov]

On Tue, 2005-01-04 at 10:34 +0000, Anton Altaparmakov wrote:
> On Mon, 2005-01-03 at 14:24 -0800, H. Peter Anvin wrote:
> > I recently posted to LKML a patch to get or set DOS attribute flags for 
> > fatfs.  That patch used ioctl().  It was suggested that a better way 
> > would be using xattrs, although the xattr mechanism seems clumsy to me, 
> > and has namespace issues.
> > 
> > I also think it would be good to have a unified interface for FAT, NTFS 
> > and CIFS for these attributes.
> > 
> > I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
> > although it doesn't seem to be implemented.
> > 
> > Questions:
> > 
> > a) is xattr the right thing?  It seems to be a fairly complex and 
> > ill-thought-out mechanism all along, especially the whole namespace 
> > business (what is a system attribute to one filesystem is a user 
> > attribute to another, for example.)
> > 
> > b) if xattr is the right thing, shouldn't this be in the system 
> > namespace rather than the user namespace?
> > 
> > c) What should the representation be?  Binary byte?  String containing a 
> > subset of "rhsvda67" (barf)?
> 
> Definitely not c!

I meant definitely not the second suggestion of c (i.e. the string
subset).  That is just horrible...

> In NTFS, the "dos attribute flags" are part of the system information
> attribute which is an entity in its own right, totally separate from
> extended attributes (and named streams for that matter).  So if I were
> to be thinking in an NTFS-only world I would be inclined to use an
> ioctl() to access/modify them (i.e. not b either).  So if you implement
> an ioctl() for vfat I will probably be able to provide the same in NTFS
> with almost zero effort (we already have the code to read and write the
> attribute flags in the kernel ntfs driver, we just do not provide an
> interface for it).
> 
> But please note that it would be best if you could use 32-bits for the
> flags.  At the very least 16-bits though as on NTFS there are currently
> in use 16-bits in the standard information but the field is u32 sized on
> disk (little endian) and two of the higher bits are in use in the file
> name attribute as well and I would not be surprised if more bits get
> used in future NTFS releases.  Tridge already gave you a list of all the
> Samba dos attributes so here is the full list for NTFS (note they are
> 100% compatible to the Samba ones and also note in NTFS we always keep
> the flags in little endian and just define all the constants to be
> little endian as well - makes life much easier):

Another reason to want 32-bits is that this could be used to add more
Linux specific bits like "FILE_ATTR_CHAR_DEV" for character device
special files and use the existing "FILE_ATTR_DEVICE" for block devices.
Windows unfortunately does not distinguish between the two in the
flags.  )-:

> /*
>  * File attribute flags (32-bit).
>  */
> enum {
>         /*
>          * The following flags are only present in the
> STANDARD_INFORMATION
>          * attribute (in the field file_attributes).
>          */
>         FILE_ATTR_READONLY              = const_cpu_to_le32(0x00000001),
>         FILE_ATTR_HIDDEN                = const_cpu_to_le32(0x00000002),
>         FILE_ATTR_SYSTEM                = const_cpu_to_le32(0x00000004),
>         /* Old DOS volid. Unused in NT. = const_cpu_to_le32(0x00000008),
> */
>                                                                                 
>         FILE_ATTR_DIRECTORY             = const_cpu_to_le32(0x00000010),
>         /* Note, FILE_ATTR_DIRECTORY is not considered valid in NT.  It
> is
>            reserved for the DOS SUBDIRECTORY flag. */
>         FILE_ATTR_ARCHIVE               = const_cpu_to_le32(0x00000020),
>         FILE_ATTR_DEVICE                = const_cpu_to_le32(0x00000040),
>         FILE_ATTR_NORMAL                = const_cpu_to_le32(0x00000080),
>                                                                                 
>         FILE_ATTR_TEMPORARY             = const_cpu_to_le32(0x00000100),
>         FILE_ATTR_SPARSE_FILE           = const_cpu_to_le32(0x00000200),
>         FILE_ATTR_REPARSE_POINT         = const_cpu_to_le32(0x00000400),
>         FILE_ATTR_COMPRESSED            = const_cpu_to_le32(0x00000800),
>                                                                                 
>         FILE_ATTR_OFFLINE               = const_cpu_to_le32(0x00001000),
>         FILE_ATTR_NOT_CONTENT_INDEXED   = const_cpu_to_le32(0x00002000),
>         FILE_ATTR_ENCRYPTED             = const_cpu_to_le32(0x00004000),
>                                                                                 
>         FILE_ATTR_VALID_FLAGS           = const_cpu_to_le32(0x00007fb7),
>         /* Note, FILE_ATTR_VALID_FLAGS masks out the old DOS VolId and
> the
>            FILE_ATTR_DEVICE and preserves everything else.  This mask is
> used
>            to obtain all flags that are valid for reading. */
>         FILE_ATTR_VALID_SET_FLAGS       = const_cpu_to_le32(0x000031a7),
>         /* Note, FILE_ATTR_VALID_SET_FLAGS masks out the old DOS VolId,
> the
>            F_A_DEVICE, F_A_DIRECTORY, F_A_SPARSE_FILE,
> F_A_REPARSE_POINT,
>            F_A_COMPRESSED, and F_A_ENCRYPTED and preserves the rest.
> This mask
>            is used to to obtain all flags that are valid for setting. */
>                                                                                 
>         /*
>          * The following flags are only present in the FILE_NAME
> attribute (in
>          * the field file_attributes).
>          */
>         FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT   =
> const_cpu_to_le32(0x10000000),        /* Note, this is a copy of the
> corresponding bit from the mft record,
>            telling us whether this is a directory or not, i.e. whether
> it has
>            an index root attribute or not. */
>         FILE_ATTR_DUP_VIEW_INDEX_PRESENT        =
> const_cpu_to_le32(0x20000000),        /* Note, this is a copy of the
> corresponding bit from the mft record,
>            telling us whether this file has a view index present (eg.
> object id
>            index, quota index, one of the security indexes or the
> encrypting
>            file system related indexes). */
> };
>                                                                                 
> typedef le32 FILE_ATTR_FLAGS;

Best regards,

        Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: FAT, NTFS, CIFS and DOS attributes

[Nicholas Miell]

On Tue, 2005-01-04 at 10:34 +0000, Anton Altaparmakov wrote:
> On Mon, 2005-01-03 at 14:24 -0800, H. Peter Anvin wrote:
> > I recently posted to LKML a patch to get or set DOS attribute flags for 
> > fatfs.  That patch used ioctl().  It was suggested that a better way 
> > would be using xattrs, although the xattr mechanism seems clumsy to me, 
> > and has namespace issues.
> > 
> > I also think it would be good to have a unified interface for FAT, NTFS 
> > and CIFS for these attributes.
> > 
> > I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
> > although it doesn't seem to be implemented.
> > 
> > Questions:
> > 
> > a) is xattr the right thing?  It seems to be a fairly complex and 
> > ill-thought-out mechanism all along, especially the whole namespace 
> > business (what is a system attribute to one filesystem is a user 
> > attribute to another, for example.)
> > 
> > b) if xattr is the right thing, shouldn't this be in the system 
> > namespace rather than the user namespace?
> > 
> > c) What should the representation be?  Binary byte?  String containing a 
> > subset of "rhsvda67" (barf)?
> 
> Definitely not [a string]!

Why not? It makes special setfatattr and getfatattr (or setntfsattr and
getnetfsattr) tools completely unnecessary. All you need is setfattr and
getfattr, which already exist.

> 
> In NTFS, the "dos attribute flags" are part of the system information
> attribute which is an entity in its own right, totally separate from
> extended attributes (and named streams for that matter).  So if I were
> to be thinking in an NTFS-only world I would be inclined to use an
> ioctl() to access/modify them (i.e. not b either).  So if you implement
> an ioctl() for vfat I will probably be able to provide the same in NTFS
> with almost zero effort (we already have the code to read and write the
> attribute flags in the kernel ntfs driver, we just do not provide an
> interface for it).
> 

An ioctl would be wrong. Remember, the point of this exercise is to
expose these attributes in such a way that tools don't have to have any
special knowledge to correctly preserve them.

If I want to be able to copy files from one NTFS volume to another
(preserving all their NTFS attributes), I don't want to have to teach cp
to run a Linux-specific and NTFS-specific ioctl on each file on the
source and destination for it to work, it should be able to see the
xattrs and just do the right thing.

The fact that the NTFS "dos attribute flags" are seperate from real
extended attributes isn't a problem, either. Real extended attributes
can be exported in the user namespace, just like ext2/3 does. (Or are
the real extended attributes something other than inert blobs of data --
does Windows care about their contents at all, or does it just store
them for users who do?)

> But please note that it would be best if you could use 32-bits for the
> flags.  At the very least 16-bits though as on NTFS there are currently
> in use 16-bits in the standard information but the field is u32 sized on
> disk (little endian) and two of the higher bits are in use in the file
> name attribute as well and I would not be surprised if more bits get
> used in future NTFS releases.  Tridge already gave you a list of all the
> Samba dos attributes so here is the full list for NTFS (note they are
> 100% compatible to the Samba ones and also note in NTFS we always keep
> the flags in little endian and just define all the constants to be
> little endian as well - makes life much easier):
> 

Using anything other than 8-bits to represent the FAT attributes would
be wrong, too. It's better to separate the xattr holding NTFS DOS
attributes (what genius at Microsoft named this...) from the xattr
holding FAT attributes so that, again, cp can do the right thing without
any special knowledge about filesystems.

If the attributes aren't separated into different xattrs, what would
happen when I copy files from an NTFS volume to a VFAT volume?

cp will attempt to copy xattrs containing NTFS-specific bits that VFAT
can't store, now what? VFAT could either silently discard the bits that
it doesn't support, or it could fail the entire operation. Either way,
it has to do the wrong thing.

Now, if the attributes have been seperated into system.fatattrs and
system.ntfsattrs, cp will be able to completely preserve system.fatattrs
with no trouble at all and inform the user that it could not create
system.ntfsattrs (similar to what it does now when it cannot preserve
ownership or permissions when copying from ext3 to VFAT).

>
> Best regards,
> 
>         Anton
-- 
Nicholas Miell <nmiell@comcast.net>

Re: FAT, NTFS, CIFS and DOS attributes

[Anton Altaparmakov]

On Tue, 4 Jan 2005, Nicholas Miell wrote:
> On Tue, 2005-01-04 at 10:34 +0000, Anton Altaparmakov wrote:
> > On Mon, 2005-01-03 at 14:24 -0800, H. Peter Anvin wrote:
> > > I recently posted to LKML a patch to get or set DOS attribute flags for 
> > > fatfs.  That patch used ioctl().  It was suggested that a better way 
> > > would be using xattrs, although the xattr mechanism seems clumsy to me, 
> > > and has namespace issues.
> > > 
> > > I also think it would be good to have a unified interface for FAT, NTFS 
> > > and CIFS for these attributes.
> > > 
> > > I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, 
> > > although it doesn't seem to be implemented.
> > > 
> > > Questions:
> > > 
> > > a) is xattr the right thing?  It seems to be a fairly complex and 
> > > ill-thought-out mechanism all along, especially the whole namespace 
> > > business (what is a system attribute to one filesystem is a user 
> > > attribute to another, for example.)
> > > 
> > > b) if xattr is the right thing, shouldn't this be in the system 
> > > namespace rather than the user namespace?
> > > 
> > > c) What should the representation be?  Binary byte?  String containing a 
> > > subset of "rhsvda67" (barf)?
> > 
> > Definitely not [a string]!
> 
> Why not? It makes special setfatattr and getfatattr (or setntfsattr and
> getnetfsattr) tools completely unnecessary. All you need is setfattr and
> getfattr, which already exist.

For one because seeing things like "rhsvda67" makes me want to throw up.  
(-;  Also strings are no use for internationalisation.  Binary is much 
shorter and already well defined.  If you want text, just have a library 
or even do it in the application.  The lib or app can decode the binary to 
text and output it in any language it desires and in any shape it desires.  
Also, binary bit testing is a hell of a lot faster than strncmp() on each 
and every string...

> > In NTFS, the "dos attribute flags" are part of the system information
> > attribute which is an entity in its own right, totally separate from
> > extended attributes (and named streams for that matter).  So if I were
> > to be thinking in an NTFS-only world I would be inclined to use an
> > ioctl() to access/modify them (i.e. not b either).  So if you implement
> > an ioctl() for vfat I will probably be able to provide the same in NTFS
> > with almost zero effort (we already have the code to read and write the
> > attribute flags in the kernel ntfs driver, we just do not provide an
> > interface for it).
> 
> An ioctl would be wrong. Remember, the point of this exercise is to
> expose these attributes in such a way that tools don't have to have any
> special knowledge to correctly preserve them.

Certainly not for me.  I don't care about backup programs.  We have 
ntfsclone for that.  You can never use xattrs to do full backups of ntfs 
anyway so no point in trying.  How do you deal with compression or 
encryption for example?  Do you just allow everyone to read the crypto 
keys so they can back them up?  What about ACLs?  And quotas?  And reparse 
points (sym links and mount points and more complex stuff like DFS/HFS).  
The only way you can really backup ntfs IMO is using ntfsclone or if the 
ntfs driver and/or libntfs were to provide it, the MS BackupAPI or 
something equivalent.  Perhaps the "Linux Backup API"?  That could be 
defined and then exported by the VFS with appropriate hooks for each fs 
that does the nitty gritty work.  (-;

> If I want to be able to copy files from one NTFS volume to another
> (preserving all their NTFS attributes), I don't want to have to teach cp
> to run a Linux-specific and NTFS-specific ioctl on each file on the
> source and destination for it to work, it should be able to see the
> xattrs and just do the right thing.
> 
> The fact that the NTFS "dos attribute flags" are seperate from real
> extended attributes isn't a problem, either. Real extended attributes
> can be exported in the user namespace, just like ext2/3 does. (Or are
> the real extended attributes something other than inert blobs of data --
> does Windows care about their contents at all, or does it just store
> them for users who do?)

I do not believe windows cares about the EAs at all.

> > But please note that it would be best if you could use 32-bits for the
> > flags.  At the very least 16-bits though as on NTFS there are currently
> > in use 16-bits in the standard information but the field is u32 sized on
> > disk (little endian) and two of the higher bits are in use in the file
> > name attribute as well and I would not be surprised if more bits get
> > used in future NTFS releases.  Tridge already gave you a list of all the
> > Samba dos attributes so here is the full list for NTFS (note they are
> > 100% compatible to the Samba ones and also note in NTFS we always keep
> > the flags in little endian and just define all the constants to be
> > little endian as well - makes life much easier):
> 
> Using anything other than 8-bits to represent the FAT attributes would
> be wrong, too. It's better to separate the xattr holding NTFS DOS
> attributes (what genius at Microsoft named this...) from the xattr
> holding FAT attributes so that, again, cp can do the right thing without
> any special knowledge about filesystems.
> 
> If the attributes aren't separated into different xattrs, what would
> happen when I copy files from an NTFS volume to a VFAT volume?

Simply loose the bits that cannot be preserved.  It makes no sense to try 
to preserve them on VFAT as they have no meaning there.  It is just 
pointless.

> cp will attempt to copy xattrs containing NTFS-specific bits that VFAT
> can't store, now what? VFAT could either silently discard the bits that
> it doesn't support, or it could fail the entire operation. Either way,
> it has to do the wrong thing.

So according to your arguments all FS now need to support NTFS based 
compression and encryption as well then?  So no information gets lots when 
copying from one to the other?  Unlikely...  Far more likely is that you 
just loose this information and you copy the file uncompressed / 
unencrypted.  You then also want to loose the compressed/encrypted bits on 
the file attributes anyway as the file is no longer compressed/encrypted.  
That would certainly be the element of least surprise.  You simply loose 
unsupported bits since you also loose all the features that go with their 
meaning.  Makes perfect sense to me anyway.  YMMV  (-;

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: FAT, NTFS, CIFS and DOS attributes

[Nicholas Miell]

On Tue, 2005-01-04 at 23:04 +0000, Anton Altaparmakov wrote:
> On Tue, 4 Jan 2005, Nicholas Miell wrote:
> > On Tue, 2005-01-04 at 10:34 +0000, Anton Altaparmakov wrote:
> > > On Mon, 2005-01-03 at 14:24 -0800, H. Peter Anvin wrote:
> > > > c) What should the representation be?  Binary byte?  String containing a 
> > > > subset of "rhsvda67" (barf)?
> > > 
> > > Definitely not [a string]!
> > 
> > Why not? It makes special setfatattr and getfatattr (or setntfsattr and
> > getntfsattr) tools completely unnecessary. All you need is setfattr and
> > getfattr, which already exist.
> 
> For one because seeing things like "rhsvda67" makes me want to throw up.  
> (-;  Also strings are no use for internationalisation.  Binary is much 
> shorter and already well defined.  If you want text, just have a library 
> or even do it in the application.  The lib or app can decode the binary to 
> text and output it in any language it desires and in any shape it desires.  
> Also, binary bit testing is a hell of a lot faster than strncmp() on each 
> and every string...

Eh. I only suggested stringifying it in the first place to make the "use
text in /proc and /sys for shell scripts!! and people!!" faction happy,
I'm not particularly attached to the idea. (Although, I have become a
bit enamored with how the string representation needs no special tools
to use.)

> 
> > > In NTFS, the "dos attribute flags" are part of the system information
> > > attribute which is an entity in its own right, totally separate from
> > > extended attributes (and named streams for that matter).  So if I were
> > > to be thinking in an NTFS-only world I would be inclined to use an
> > > ioctl() to access/modify them (i.e. not b either).  So if you implement
> > > an ioctl() for vfat I will probably be able to provide the same in NTFS
> > > with almost zero effort (we already have the code to read and write the
> > > attribute flags in the kernel ntfs driver, we just do not provide an
> > > interface for it).
> > 
> > An ioctl would be wrong. Remember, the point of this exercise is to
> > expose these attributes in such a way that tools don't have to have any
> > special knowledge to correctly preserve them.
> 
> Certainly not for me.  I don't care about backup programs.  We have 
> ntfsclone for that.  You can never use xattrs to do full backups of ntfs 
> anyway so no point in trying.  How do you deal with compression or 
> encryption for example?  Do you just allow everyone to read the crypto 
> keys so they can back them up?  What about ACLs?  And quotas?  And reparse 
> points (sym links and mount points and more complex stuff like DFS/HFS).  
> The only way you can really backup ntfs IMO is using ntfsclone or if the 
> ntfs driver and/or libntfs were to provide it, the MS BackupAPI or 
> something equivalent.  Perhaps the "Linux Backup API"?  That could be 
> defined and then exported by the VFS with appropriate hooks for each fs 
> that does the nitty gritty work.  (-;

I'm not talking about backing up NTFS, I'm talking about preserving data
that is trivial to preserve. If it can be done, it should be done.
 
> > The fact that the NTFS "dos attribute flags" are seperate from real
> > extended attributes isn't a problem, either. Real extended attributes
> > can be exported in the user namespace, just like ext2/3 does. (Or are
> > the real extended attributes something other than inert blobs of data --
> > does Windows care about their contents at all, or does it just store
> > them for users who do?)
> 
> I do not believe windows cares about the EAs at all.

That's good, then they can just be exported in the user namespace.

> > Using anything other than 8-bits to represent the FAT attributes would
> > be wrong, too. It's better to separate the xattr holding NTFS DOS
> > attributes (what genius at Microsoft named this...) from the xattr
> > holding FAT attributes so that, again, cp can do the right thing without
> > any special knowledge about filesystems.
> > 
> > If the attributes aren't separated into different xattrs, what would
> > happen when I copy files from an NTFS volume to a VFAT volume?
> 
> Simply loose the bits that cannot be preserved.  It makes no sense to try 
> to preserve them on VFAT as they have no meaning there.  It is just 
> pointless.

Wouldn't you want the user to know that the bits are being thrown away? 
Silently discarding data is bound to piss somebody off.

> > cp will attempt to copy xattrs containing NTFS-specific bits that VFAT
> > can't store, now what? VFAT could either silently discard the bits that
> > it doesn't support, or it could fail the entire operation. Either way,
> > it has to do the wrong thing.
> 
> So according to your arguments all FS now need to support NTFS based 
> compression and encryption as well then?  So no information gets lots when 
> copying from one to the other?  Unlikely...  Far more likely is that you 
> just loose this information and you copy the file uncompressed / 
> unencrypted.  You then also want to loose the compressed/encrypted bits on 
> the file attributes anyway as the file is no longer compressed/encrypted.  
> That would certainly be the element of least surprise.  You simply loose 
> unsupported bits since you also loose all the features that go with their 
> meaning.  Makes perfect sense to me anyway.  YMMV  (-;

I never suggested that all filesystems should support every feature that
every filesystem has ever supported, and I don't know where you got such
a crazy idea.

I am advocating two things:

1) If information can be accurately preserved, it should be.
2) If information cannot be accurately preserved, the user should be
notified of this fact.

My suggestion of creating two seperate xattrs meets both of these
requirements nicely.

The system.fatattrs xattr is the interface to the bits supported by
every Microsoft filesystem, whether it be msdos, vfat, CIFS, or NTFS.

The system.ntfsattrs xattr is the interface to the bits only supported
by NTFS (and CIFS, too, I suppose).

Thus, I know for a fact that if I am able to use the system.fatattr
xattr with a file, that that filesystem accurately supports FAT
attributes. I also know that if I am able to use system.ntfsattr, that
the filesystem accurately supports NTFS DOS attributes. I can can use
neither, than I know that I will lose information, and if I can use one
and not the other, I know which information will be lost.


[ Note to audience: the following is a long (and largely irrelevant to
the subject) discussion of how NTFS could implement reparse points and
encryption on Linux. Feel free to ignore it. ]

Of course, reparse points and encrypted files make this difficult
because those are NTFS DOS attribute bits that require extra information
and cannot be explicitly set. I'm not sure how you'd want to implement
that, but here's my suggestion (although, you're the NTFS expert, so
these may be completely wrong).

Reparse points fall into two categories: those that Linux supports
(afaik, only symlinks) and those that Linux doesn't support.

Reparse points that Linux doesn't support are (conceptually) easy: just
add a system.ntfs_reparse_point xattr that exports the reparse tag,
reparse GUID, and the data buffer.

If something attempts set the reparse bit in the NTFS DOS attr on a file
without the system.ntfs_reparse_point xattr, silently clear that bit, if
something attempts to clear that bit on a file with the xattr, silently
set the bit, when something creates the xattr (i.e. when something turns
a file/directory into a reparse point), set the bit, when something
removes the xattr, clear the bit.

Reparse points that Linux does support (symlinks) are more difficult.
You could pretend that they aren't actually reparse points and just make
them look like symlinks to the Linux VFS with no bit set in the NTFS DOS
attributes and no system.ntfs_reparse_point xattr.

Or you could include the bit in the attributes and the
system.ntfs_reparse_point xattr, but silently resist any attempt to
change them via the xattr interface.

Finally, you could allow arbitrary modification of the
system.ntfs_reparse_point xattr, but then you have to deal with the
possibility that something could come along and turn a normal file into
a symlink right under the VFS's nose, which is bound to cause problems.

Encryption doesn't look to be all that hard, actually. When an app sets
the Encrypted bit, you can grab the appropriate NTFS-encrypting-key from
their keyring and use that to encrypt the file.

Arranging for copying files to use the right key is harder. If you're
happy with Win2k semantics, you can live with the fact that setting the
Encrypted bit just causes the file to be encrypted with the current key.
If you want XP semantics, where copying a file causes the copy to be
encrypted with the same key if possible (or the default key, if not),
you could add a system.ntfs_key_ids xattr that lists the ids of the keys
used and require that apps create all xattrs before writing to the file
for copying to work.

Those are just my ideas, I've no intention of implementing them and you
can probably come up with better ones.

Of course, I don't use NTFS (or, for that matter, FAT and CIFS), so I
don't care all that much what you do, beyond my general desire for Linux
to be stellar instead of merely adequate.

> Best regards,
> 
> 	Anton
-- 
Nicholas Miell <nmiell@comcast.net>

Re: FAT, NTFS, CIFS and DOS attributes

[Nicholas Miell]

On Tue, 2005-01-04 at 16:48 -0800, Nicholas Miell wrote:

> [ Note to audience: the following is a long (and largely irrelevant to
> the subject) discussion of how NTFS could implement reparse points and
> encryption on Linux. Feel free to ignore it. ]

Sorry for the reply to my own message, but I forget to mention in the
original that the right thing to do for the Encrypted and Reparse bits
may be just be to silently discard them and not support the manipulation
of these rather esoteric NTFS features through a general API at all.

-- 
Nicholas Miell <nmiell@comcast.net>