viewprinting: what format should views be stored in?

viewprinting: what format should views be stored in?

[Hans Reiser]

It is very important that we have something simple that reuses code for 
this purpose, and it also needs to be scalable.  That is, we need to be 
able to determine in insignificant time whether a file passes through a 
mask consisting of a million files.

One approach would be to use a format similar to that for chroot, that 
is, to keep the format we use for directory trees now, and use it for 
storing the mask.  This raises the question: how would the format for 
the mask differ from that of the underlying filesystem.

Another approach is to use stem compressed names, or some other unique 
within the fs format for the mask.

We need to be able to support specifying a mask that allows any file 
within a given directory to be visible.  That is, we need to support 
dirname/* but we don't yet need to support dirname/foo.*

We probably don't want to allow symbolic links to be followed unless 
where they point to is within the mask.

I want minimal code but I want it to be treelike in its performance 
scalability.

Ideas?

Re: viewprinting: what format should views be stored in?

[George Beshers]

[-- Attachment #1: Type: text/plain, Size: 3186 bytes --]


First a little (*/tentatively/*) suggested terminology---translation of 
(/tentatively/) I stand
behind it until I begin to suspect I've made a mistake and then I change 
sides ;-) :

1) A *mask *consists of a set of *prescriptions *(trying to avoid rule) 
which defines a
   / subset/ of permissible functionality for a file system.

2) The *permissible functionality *of a file system is roughly speaking, 
the ACLs
    for all the objects in the file system.

    Actually, I think this is a good place to start, because the type 
"permissible
    functionality" is in fact what the mask is applied to and what is 
returned.
   Grant you evaluation is done lazily as the execution of an 
application requires it.

3)  A user and group instantiated mask forms an *operational set of 
functionality*.

     What is important here is to recognize that a given executable may have
     different apparent functionality based on who is running it.

     To make this concept clearer, consider the possibility that group 
write access
     to a file might be elided (masked away) by the some prescription 
for a user
     who otherwise might have add access---but the user's privileges are 
limited
     by the executable.



Hans Reiser wrote:

> It is very important that we have something simple that reuses code 
> for this purpose, and it also needs to be scalable.  That is, we need 
> to be able to determine in insignificant time whether a file passes 
> through a mask consisting of a million files.

A million files or a million rules?  A single rule may give access to a 
million files and
    require only constant time to evaluate.

> One approach would be to use a format similar to that for chroot, that 
> is, to keep the format we use for directory trees now, and use it for 
> storing the mask.  This raises the question: how would the format for 
> the mask differ from that of the underlying filesystem.
>
> Another approach is to use stem compressed names, or some other unique 
> within the fs format for the mask.

Can you elaborate on this for a newbie

>
> We need to be able to support specifying a mask that allows any file 
> within a given directory to be visible.  That is, we need to support 
> dirname/* but we don't yet need to support dirname/foo.*

NB.  Hans and I had discussed limiting access to 'hidden files' 
informally:  '..' in particular needs some
special attention although it would be best not to have a special case 
if we can avoid it.

>
> We probably don't want to allow symbolic links to be followed unless 
> where they point to is within the mask.

Agreed.

> I want minimal code but I want it to be treelike in its performance 
> scalability.
>
> Ideas?

Well, for the moment only more questions:

Suppose we have a file system and a mask.  If we were to create a chroot 
by copying just
the file system accessible through the mask and run the application in 
that environment
would the semantics of running the application on the original file 
system with the mask
by equivalent.  By equivalent I mean no observable difference in the 
instructions executed
at the user level or the output generated.

Does the question make sense?


[-- Attachment #2: Type: text/html, Size: 4297 bytes --]

Re: viewprinting: what format should views be stored in?

[Hans Reiser]

George Beshers wrote:

>
> First a little (*/tentatively/*) suggested terminology---translation 
> of (/tentatively/) I stand
> behind it until I begin to suspect I've made a mistake and then I 
> change sides ;-) :
>
> 1) A *mask *consists of a set of *prescriptions *(trying to avoid 
> rule) which defines a
>    / subset/ of permissible functionality for a file system.
>
> 2) The *permissible functionality *of a file system is roughly 
> speaking, the ACLs
>     for all the objects in the file system.
>
>     Actually, I think this is a good place to start, because the type 
> "permissible
>     functionality" is in fact what the mask is applied to and what is 
> returned.
>    Grant you evaluation is done lazily as the execution of an 
> application requires it.
>
Yes.

> 3)  A user and group instantiated mask forms an *operational set of 
> functionality*.
>
>      What is important here is to recognize that a given executable 
> may have
>      different apparent functionality based on who is running it.

I think not in our particular niche.  We do process oriented security, 
and that is all for now.  Later we can make it more complex.

>
>      To make this concept clearer, consider the possibility that group 
> write access
>      to a file might be elided (masked away) by the some prescription 
> for a user
>      who otherwise might have add access---but the user's privileges 
> are limited
>      by the executable.
>
>
>
> Hans Reiser wrote:
>
>> It is very important that we have something simple that reuses code 
>> for this purpose, and it also needs to be scalable.  That is, we need 
>> to be able to determine in insignificant time whether a file passes 
>> through a mask consisting of a million files.
>
> A million files or a million rules

filenames actually.

>   A single rule may give access to a million files and
>     require only constant time to evaluate.
>
>> One approach would be to use a format similar to that for chroot, 
>> that is, to keep the format we use for directory trees now, and use 
>> it for storing the mask.  This raises the question: how would the 
>> format for the mask differ from that of the underlying filesystem.
>>
>> Another approach is to use stem compressed names, or some other 
>> unique within the fs format for the mask.
>
> Can you elaborate on this for a newbie

fired
fireman
fireplace
get stored as
fired
!4man
!4place
if we use classic stem compression, if we use a tree that branches where 
the names diverge (I forget, is that a radix tree or?) then we have 
another structure.

>>
>> We need to be able to support specifying a mask that allows any file 
>> within a given directory to be visible.  That is, we need to support 
>> dirname/* but we don't yet need to support dirname/foo.*
>
> NB.  Hans and I had discussed limiting access to 'hidden files' 
> informally:  '..' in particular needs some
> special attention although it would be best not to have a special case 
> if we can avoid it.

We can treat .. the same as we treat symboplic links.

>>
>> We probably don't want to allow symbolic links to be followed unless 
>> where they point to is within the mask.
>
> Agreed.
>
>> I want minimal code but I want it to be treelike in its performance 
>> scalability.
>>
>> Ideas?
>
> Well, for the moment only more questions:
>
> Suppose we have a file system and a mask.  If we were to create a 
> chroot by copying just
> the file system 

semantic tree (not files, just filenames)

> accessible through the mask and run the application in that environment
> would the semantics of running the application on the original file 
> system with the mask
> by equivalent.  By equivalent I mean no observable difference in the 
> instructions executed
> at the user level or the output generated.

No, because new files might be made visible through the mask without the 
new file creator even being aware that there was a mask.

>
> Does the question make sense?
>

Re: viewprinting: what format should views be stored in?

[George Beshers]

Hans Reiser wrote:

>
>>> Another approach is to use stem compressed names, or some other 
>>> unique within the fs format for the mask.
>>
>>
>> Can you elaborate on this for a newbie
>
>
> fired
> fireman
> fireplace
> get stored as
> fired
> !4man
> !4place
> if we use classic stem compression, if we use a tree that branches 
> where the names diverge (I forget, is that a radix tree or?) then we 
> have another structure.

Trie I think is what you mean.  http://www.nist.gov/dads/HTML/trie.html

Re: viewprinting: what format should views be stored in?

[George Beshers]

[-- Attachment #1: Type: text/plain, Size: 2891 bytes --]



Hans Reiser wrote:

> George Beshers wrote:
>
>> 2) The *permissible functionality *of a file system is roughly 
>> speaking, the ACLs
>>     for all the objects in the file system.
>>
>>     Actually, I think this is a good place to start, because the type 
>> "permissible
>>     functionality" is in fact what the mask is applied to and what is 
>> returned.
>>    Grant you evaluation is done lazily as the execution of an 
>> application requires it.
>>
> Yes.

I have started re-reading withe Single UNIX Specification with getting a 
working
definition of permissible functionality in mind.  Have not gotten to 
Usenix papers
yet or a careful search of ACM.

Does anyone think there is a better starting point?  Or things that are 
particularly
important to consider?

>
>> 3)  A user and group instantiated mask forms an *operational set of 
>> functionality*.
>>
>>      What is important here is to recognize that a given executable 
>> may have
>>      different apparent functionality based on who is running it.
>
>
> I think not in our particular niche.  We do process oriented security, 
> and that is all for now.  Later we can make it more complex.

We can't avoid it at some level without re-writing Linux security.  Take 
a moment to consider
the set-UID bit on a file which is an executable and I think you will 
see what I am driving at.

That said, all we need to do is make our specification conform to the 
smallest code change
for the time being, e.g., the identity mask---what was happening before 
is what you get.

NB.  I understand that the overriding goal of the first 3 months is 
devising and demonstrating
that we have a strategy that scales to file systems containing millions 
of files---but I don't
think we can lose any semantics as fundamental as user/group permissions 
without inviting
criticism.


>
>>
>>
>>
>> Well, for the moment only more questions:
>>
>> Suppose we have a file system and a mask.  If we were to create a 
>> chroot by copying just
>> the file system 
>
>
> semantic tree (not files, just filenames)
>
>> accessible through the mask and run the application in that environment
>> would the semantics of running the application on the original file 
>> system with the mask
>> by equivalent.  By equivalent I mean no observable difference in the 
>> instructions executed
>> at the user level or the output generated.
>
>
> No, because new files might be made visible through the mask without 
> the new file creator even being aware that there was a mask.

I don't think my question was very clear ...

For any process, could that process determine if it was running

    * on a reiser4 file system that is really the root
    * on a reiser4 file system with a mask, or
    * on a reiser4 file system with chroot

/exclusively/ by making calls to the reiser4 file system?

>
>>
>> Does the question make sense?
>
Uh no... :-[


[-- Attachment #2: Type: text/html, Size: 4144 bytes --]

Re: viewprinting: what format should views be stored in?

[Hans Reiser]

George Beshers wrote:

>>
>>> 3)  A user and group instantiated mask forms an *operational set of 
>>> functionality*.
>>>
>>>      What is important here is to recognize that a given executable 
>>> may have
>>>      different apparent functionality based on who is running it.
>>
>>
>> I think not in our particular niche.  We do process oriented 
>> security, and that is all for now.  Later we can make it more complex.
>
> We can't avoid it at some level without re-writing Linux security.  
> Take a moment to consider
> the set-UID bit on a file which is an executable and I think you will 
> see what I am driving at.

I don't see what you are saying.  Our mask does process oriented 
security.  The underlying security remains a user/object/permission mapping.

>
> That said, all we need to do is make our specification conform to the 
> smallest code change
> for the time being, e.g., the identity mask---what was happening 
> before is what you get.

Well, we should just pass things through the mask to whatever is below.

>
> NB.  I understand that the overriding goal of the first 3 months is 
> devising and demonstrating
> that we have a strategy that scales to file systems containing 
> millions of files---but I don't
> think we can lose any semantics as fundamental as user/group 
> permissions without inviting
> criticism.

We are not losing, we are residing above and masking them.

>
>
>>
>>>
>>>
>>>
>>> Well, for the moment only more questions:
>>>
>>> Suppose we have a file system and a mask.  If we were to create a 
>>> chroot by copying just
>>> the file system 
>>
>>
>> semantic tree (not files, just filenames)
>>
>>> accessible through the mask and run the application in that environment
>>> would the semantics of running the application on the original file 
>>> system with the mask
>>> by equivalent.  By equivalent I mean no observable difference in the 
>>> instructions executed
>>> at the user level or the output generated.
>>
>>
>> No, because new files might be made visible through the mask without 
>> the new file creator even being aware that there was a mask.
>
> I don't think my question was very clear ...
>
> For any process, could that process determine if it was running
>
>     * on a reiser4 file system that is really the root
>
the root filesystem?

>    *
>
>
>     * on a reiser4 file system with a mask, or
>     * on a reiser4 file system with chroot
>
> /exclusively/ by making calls to the reiser4 file system?

Hmm, sounds like the answer requires carefully traversing the code.;-)

>
>>
>>>
>>> Does the question make sense?
>>
> Uh no... :-[
>

Re: viewprinting: what format should views be stored in?

[George Beshers]

Hans Reiser wrote:

> George Beshers wrote:
>
>> We can't avoid it at some level without re-writing Linux security.  
>> Take a moment to consider
>> the set-UID bit on a file which is an executable and I think you will 
>> see what I am driving at.
>
>
> I don't see what you are saying.  Our mask does process oriented 
> security.  The underlying security remains a user/object/permission 
> mapping.
>
The issue is how the mask might change the semantics of sys_execve().  I 
have spent some time
starting to trace code around, but more work is required.

For the moment I think we are safe if the setuid and setgroup can not be 
altered by the mask.

We only change the semantics if the real/effective uid or gid is 
different at some point in the
processes execution because of the mask---begging situations where a 
call to setuid happens
only if a certain file is not available.

I want to get back to understanding your second proposed strategy.


>

Re: viewprinting: what format should views be stored in?

[Hans Reiser]

George Beshers wrote:

>
>
> Hans Reiser wrote:
>
>>
>>>>
>>>> No, because new files might be made visible through the mask 
>>>> without the new file creator even being aware that there was a mask.
>>>
>>>
>>> I don't think my question was very clear ...
>>>
>>> For any process, could that process determine if it was running
>>>
>>>     * on a reiser4 file system that is really the root
>>>
>> the root filesystem?
>
> Yes, as understood by the OS.
>
>>
>>>    *
>>>
>>>
>>>     * on a reiser4 file system with a mask, or
>>>     * on a reiser4 file system with chroot
>>>
>>> /exclusively/ by making calls to the reiser4 file system?
>>
>>
>> Hmm, sounds like the answer requires carefully traversing the code.;-)
>
> of course :-)
>

> But what would we like the answer to be?

Ah, so that is the question....

>     I will take the position that "no the process can not tell" as
>     being useful
>     because it makes running with a mask a better test environment.
>     Also, I don't see how cluing the process in that a mask is in
>     place can
>     improve security whereas knowing that the mask is in place might
>     encourage someone to try to work around it (beyond the filesystem
>     semantics).
>
>
>
Yes, you are right I think, but you should ask on the mailing list what 
information about what is the root filesystem is used for by the OS

Re: viewprinting: what format should views be stored in?

[Hans Reiser]

George Beshers wrote:

>
>
> Hans Reiser wrote:
>
>> George Beshers wrote:
>>
>>> We can't avoid it at some level without re-writing Linux security.  
>>> Take a moment to consider
>>> the set-UID bit on a file which is an executable and I think you 
>>> will see what I am driving at.
>>
>>
>>
>> I don't see what you are saying.  Our mask does process oriented 
>> security.  The underlying security remains a user/object/permission 
>> mapping.
>>
> The issue is how the mask might change the semantics of sys_execve().  
> I have spent some time
> starting to trace code around, but more work is required.
>
> For the moment I think we are safe if the setuid and setgroup can not 
> be altered by the mask.

Ok.

>
> We only change the semantics if the real/effective uid or gid is 
> different at some point in the
> processes execution because of the mask---begging situations where a 
> call to setuid happens
> only if a certain file is not available.

Say more.

>
> I want to get back to understanding your second proposed strategy.
>
>
>>
>
>
>

Re: viewprinting: what format should views be stored in?

[George Beshers]

[-- Attachment #1: Type: text/plain, Size: 2791 bytes --]


First, rereading my e-mails I realize that there was some thinking 
behind my questions
which was only partially articulated.  Going back to my earlier position 
which I will restate
here as:

    /Use of a mask should never corrupt the semantics defined by the
    Single Unix
    Specification /*or*/ its practical derivations and extensions, e.g.,
    BSD and Linux./

That is,/ ideally/ if I created a standalone system that "looked" like 
the system the mask
provides then the process would behave equivalently.  I emphasize 
ideally because to
actually make that the case would require something closer to a virtual 
machine---which
is clearly beyond the scope of this project.

So I am looking for potential semantic anomalies which could be 
introduced by a
mask over a reiser4 file system, i.e., border cases where some upfront 
awareness
and attention may save re-design and re-writing later on.


Hans Reiser wrote:

>
>> The issue is how the mask might change the semantics of 
>> sys_execve().  I have spent some time
>> starting to trace code around, but more work is required.
>>
>> For the moment I think we are safe if the setuid and setgroup can not 
>> be altered by the mask.
>
>
> Ok.


For the moment I will focus on how the uid (and assume the group id will 
be similar)
of a process and how the mask might lead to a different uid than would 
be expected
under standard semantics.

So we agree that the process has an associated user: actually there is 
the concept
of real and effective uid also.

I believe the statement above to be true if the process never makes a 
setuid() system's call of
one flavor or another because the uid is inherited from the process 
calling execev() or is determined
by the owner of the executable if the executable's file set-uid 
permission bit is turned on via chmod.

>>
>> We only change the semantics if the real/effective uid or gid is 
>> different at some point in the
>> processes execution because of the mask---begging situations where a 
>> call to setuid happens
>> only if a certain file is not available.
>
>
> Say more.

If the process has the code like

    if (stat("/etc/foobar.conf", statbuf)  < 0) {
         if (setuid(3) < 0) {   /* Try to become sys */
        }
        /* Do something here */
    }

And the mask hides "/etc/foobar.conf" then in fact the uid
will change but *not unexpectedly*, IMO.  The mask is
creating a changed environment but the behavior is
understandable from the source code.

Put this another way, the semantics still seem correct to me
because you are getting the behavior of "/etc/foobar.conf" actually
not being there.

Now that I think about it masking a named pipe or a lock file
is perhaps more likely to cause problems then the setuid bit.
More cogitation required...  :-)



[-- Attachment #2: Type: text/html, Size: 3669 bytes --]

Re: viewprinting: what format should views be stored in?

[Hans Reiser]

George Beshers wrote:

>
> First, rereading my e-mails I realize that there was some thinking 
> behind my questions
> which was only partially articulated.  Going back to my earlier 
> position which I will restate
> here as:
>
>     /Use of a mask should never corrupt the semantics defined by the
>     Single Unix
>     Specification /*or*/ its practical derivations and extensions,
>     e.g., BSD and Linux./
>
I expect to turn those semantics on their head within 3-5 years when we 
do the semantic enhancements.

I think you should rephrase it as: we should not break any existing 
program without getting something sufficiently worthwhile as a result, 
and doing it with care.

> That is,/ ideally/ if I created a standalone system that "looked" like 
> the system the mask
> provides then the process would behave equivalently.  I emphasize 
> ideally because to
> actually make that the case would require something closer to a 
> virtual machine---which
> is clearly beyond the scope of this project.
>
> So I am looking for potential semantic anomalies which could be 
> introduced by a
> mask over a reiser4 file system, i.e., border cases where some upfront 
> awareness
> and attention may save re-design and re-writing later on.
>
>
> Hans Reiser wrote:
>
>>
>>> The issue is how the mask might change the semantics of 
>>> sys_execve().  I have spent some time
>>> starting to trace code around, but more work is required.
>>>
>>> For the moment I think we are safe if the setuid and setgroup can 
>>> not be altered by the mask.
>>
>>
>> Ok.
>
>
> For the moment I will focus on how the uid (and assume the group id 
> will be similar)
> of a process and how the mask might lead to a different uid than would 
> be expected
> under standard semantics.

I don't think you need to support the mask changing uids.  Nice feature, 
but let's keep it minimal for now.  I think the whole issue of uid can 
be sidestepped and ignored.  You can, however, put 3 paragraphs in our 
report discussing it as a theoretical issue, if you find it interesting 
to ponder.

>
> So we agree that the process has an associated user: actually there is 
> the concept
> of real and effective uid also.
>
> I believe the statement above to be true if the process never makes a 
> setuid() system's call of
> one flavor or another because the uid is inherited from the process 
> calling execev() or is determined
> by the owner of the executable if the executable's file set-uid 
> permission bit is turned on via chmod.
>
>>>
>>> We only change the semantics if the real/effective uid or gid is 
>>> different at some point in the
>>> processes execution because of the mask---begging situations where a 
>>> call to setuid happens
>>> only if a certain file is not available.
>>
>>
>> Say more.
>
> If the process has the code like
>
>     if (stat("/etc/foobar.conf", statbuf)  < 0) {
>          if (setuid(3) < 0) {   /* Try to become sys */
>         }
>         /* Do something here */
>     }
>
> And the mask hides "/etc/foobar.conf" then in fact the uid
> will change but *not unexpectedly*, IMO.  The mask is
> creating a changed environment but the behavior is
> understandable from the source code.
>
> Put this another way, the semantics still seem correct to me
> because you are getting the behavior of "/etc/foobar.conf" actually
> not being there.
>
> Now that I think about it masking a named pipe or a lock file
> is perhaps more likely to cause problems then the setuid bit.
> More cogitation required...  :-)
>
It is not our task to ensure the mask causes no problems due to a faulty 
specification of it.  It is merely our task to ensure that creating a 
correct mask is well automated and easy.

George, trust me, the project will be a LOT more fun if we implement the 
bare minimum and get it done early, and then after that spend the rest 
of the time doing what amuses us.

For me, the big issue to decide at the start is, should a view 
specification be implemented with a format that is as if it is a 
subdirectory of the masked executable, and that the executable gets 
chrooted to, and that uses all the semantic tree traversal code already 
written by us.

I think the answer is yes, but I am going to think about it as I jog for 
an hour.  Please come up with arguments for and against it.

Hans

Re: viewprinting: what format should views be stored in? (let me quickly correct an imprecision)

[Hans Reiser]

Let me instead of the previous email, say:

For me, the big issue to decide at the start is, should a view 
specification be implemented with a format that is as if it is a 
subdirectory of the masked executable, and which we then take the 
filenames used by the executable and traverse the subdirectory as though 
it was root, and if we reach a valid fall through point, then allow we 
traverse from the real filesystem root, and if not, we issue a suitable 
error.

This would let us use all the semantic tree traversal code already 
written by us.

It would however require us to add such plugins as "fall through 
points", and "prints", etc.  If really lucky, you could have some 
working plugins in 4 months, and maybe have masking software done by the 
end of the contract.

I think the answer is yes, we want to do it this way, but I am going to 
think about it as I jog for an hour.  Please come up with arguments for 
and against it.

>
> Hans
>