filesystem differentiation

filesystem differentiation

[Peter Teoh]

given different harddisk partition, does anyone knows how to
differentiate one partition from another?

at the kernel source level, which is the constant/variable for this?

Is it EXT3_XATTR_MAGIC?   (but EXT2_XATTR_MAGIC have the same value as
ext3, so betw the two they are not distinguisable?)
(or REISERFS_XATTR_MAGIC etc)   So I supposed if I were to create my
new filesystem, then just create a new /random value from this?

-- 
Regards,
Peter Teoh

Re: filesystem differentiation

[Andreas Dilger]

On Mar 15, 2008  00:17 +0800, Peter Teoh wrote:
> given different harddisk partition, does anyone knows how to
> differentiate one partition from another?
> 
> at the kernel source level, which is the constant/variable for this?
> 
> Is it EXT3_XATTR_MAGIC?   (but EXT2_XATTR_MAGIC have the same value as
> ext3, so betw the two they are not distinguisable?)
> (or REISERFS_XATTR_MAGIC etc)

Read the file (1) and blkid (8) man pages.  These tools understand a
lot of different magic numbers for filesystems:

# file -s /dev/sda1
/dev/sda1: Linux rev 1.0 ext3 filesystem data (needs journal recovery) (large
files)

# blkid /dev/sda1
/dev/sda1: LABEL="/boot" UUID="1fe1d719-1a8c-45a0-969a-cba8b101cc57"
SEC_TYPE="ext2" TYPE="ext3"

> So I supposed if I were to create my
> new filesystem, then just create a new /random value from this?

Pretty much, yes.  You should also add a LABEL and UUID field so that one
instance of the filesystem can be distinguished from another.

A better goal (IMHO) than creating your own new filesystem is to help out
some existing filesystem like ext4 or btrfs.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

"Write once only but read many" filesystem

[Peter Teoh]

For reasons of auditability/accountability, I would like a filesystem 
such that I can write to it only ONCE, subsequently not 
modifiable/deletable, but always readable.   Kind of a database journal 
logs - it is continuously being written, sequentiall appending, but not 
circular buffer based, so that upon running out of space, logging will 
be paused in memory, and after new storage devices added to it, it will 
continue to flush out whatever is outstanding in memory.

Can ext3 / ext4 or current jbd2 be easily configured to serve this purpose?

Thanks.

Re: "Write once only but read many" filesystem

[Peter Teoh]

Thank you for your reply :-).

On Sat, Mar 22, 2008 at 11:06 PM, Jörn Engel <joern@logfs.org> wrote:
> On Sat, 22 March 2008 22:52:12 +0800, Peter Teoh wrote:
>  >
>  > what are the difference in terms of final features provided by these
>  > two different filesystem?   what is this "garbage collection"?   u
>  > still have features like creating different directories, and creating
>  > different files, and writing the files?   How about setting the file
>  > attributes...it should be set before writing right (so that after
>  > writing and handle is closed it becomes permanently not
>  > modifiable)..but creating a subdirectory below the current dir should
>  > be possible right (even after closing the previous directory)?
>
>  Your requirements aren't quite clear to me.  Do you want the complete
>  filesystem to be read-only after being written once?

YES....

>   Or do you want individual files/directories to be immutable - chattr?

chattr is not good enough, as root can still modify it.   So if
current feature is not there, then some small development may be
needed.

>  And in either case, what problem do you want to solve with a read-only filesystem?

Simple:   i want to record down everything that a user does, or a
database does, or any applications running - just record down its
state permanently securely into the filesystem, knowing that for sure,
there is not way to modify the data, short of recreating the
filesystem again.    Sound logical?   Or is there any loophole in this
concept?

In summary, are there any strong demand for such a concept/filesystem?
  I may take the plunge to implementing it, if justfiable and
everybody is interested..:-)...

-- 
Regards,
Peter Teoh
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Write once only but read many" filesystem

[Jörn Engel]

On Sat, 22 March 2008 23:55:53 +0800, Peter Teoh wrote:
> 
> >   Or do you want individual files/directories to be immutable - chattr?
> 
> chattr is not good enough, as root can still modify it.   So if
> current feature is not there, then some small development may be
> needed.
> 
> >  And in either case, what problem do you want to solve with a read-only filesystem?
> 
> Simple:   i want to record down everything that a user does, or a
> database does, or any applications running - just record down its
> state permanently securely into the filesystem, knowing that for sure,
> there is not way to modify the data, short of recreating the
> filesystem again.    Sound logical?   Or is there any loophole in this
> concept?

The loophole is called root.  In a normal setup, root can do anything,
including writing directly to the device your filesystem resides in,
writing to kernel memory, etc.

It may be rather inconvenient to change a filesystem by writing to the
block device, but far from impossible.  If you want to make such changes
impossible, you are facing an uphill battle that I personally don't care
about.  And if inconvenience is good enough, wouldn't chattr be
sufficiently inconvenient?

Jörn

-- 
Victory in war is not repetitious.
-- Sun Tzu
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Write once only but read many" filesystem

[Scott Lovenberg]

Jörn Engel wrote:
> On Sat, 22 March 2008 23:55:53 +0800, Peter Teoh wrote:
>>>   Or do you want individual files/directories to be immutable - chattr?
>> chattr is not good enough, as root can still modify it.   So if
>> current feature is not there, then some small development may be
>> needed.
>>
>>>  And in either case, what problem do you want to solve with a read-only filesystem?
>> Simple:   i want to record down everything that a user does, or a
>> database does, or any applications running - just record down its
>> state permanently securely into the filesystem, knowing that for sure,
>> there is not way to modify the data, short of recreating the
>> filesystem again.    Sound logical?   Or is there any loophole in this
>> concept?
> 
> The loophole is called root.  In a normal setup, root can do anything,
> including writing directly to the device your filesystem resides in,
> writing to kernel memory, etc.
> 
> It may be rather inconvenient to change a filesystem by writing to the
> block device, but far from impossible.  If you want to make such changes
> impossible, you are facing an uphill battle that I personally don't care
> about.  And if inconvenience is good enough, wouldn't chattr be
> sufficiently inconvenient?
> 
> Jörn
> 

How about mounting an isofs via loopback?  This has the added benefit of 
being ready to be exported to disc.  You can make it with mkisofs on a 
directory structure and mount it to the tree with a normal mount(1).  If 
it asks for fs type on mount, I think its 'iso9660'.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Write once only but read many" filesystem

[Peter Teoh]

[-- Attachment #1: Type: text/html, Size: 5863 bytes --]

[-- Attachment #2: Type: text/plain, Size: 144 bytes --]

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Re: "Write once only but read many" filesystem

[Peter Teoh]

  Scott Lovenberg wrote:
> Jörn Engel wrote:
>> On Sat, 22 March 2008 23:55:53 +0800, Peter Teoh wrote:
>>>>   Or do you want individual files/directories to be immutable - 
>>>> chattr?
>>> chattr is not good enough, as root can still modify it.   So if
>>> current feature is not there, then some small development may be
>>> needed.
>>>
>>>>  And in either case, what problem do you want to solve with a 
>>>> read-only filesystem?
>>> Simple:   i want to record down everything that a user does, or a
>>> database does, or any applications running - just record down its
>>> state permanently securely into the filesystem, knowing that for sure,
>>> there is not way to modify the data, short of recreating the
>>> filesystem again.    Sound logical?   Or is there any loophole in this
>>> concept?
>>
>> The loophole is called root.  In a normal setup, root can do anything,
>> including writing directly to the device your filesystem resides in,
>> writing to kernel memory, etc.
>>
>> It may be rather inconvenient to change a filesystem by writing to the
>> block device, but far from impossible.  If you want to make such changes
>> impossible, you are facing an uphill battle that I personally don't care
>> about.  And if inconvenience is good enough, wouldn't chattr be
>> sufficiently inconvenient?
>>
>> Jörn
>>
>
> How about mounting an isofs via loopback?  This has the added benefit 
> of being ready to be exported to disc.  You can make it with mkisofs 
> on a directory structure and mount it to the tree with a normal 
> mount(1).  If it asks for fs type on mount, I think its 'iso9660'.
>
>
Thanks for the idea.   Based on this idea, I will start looking at the 
implementation of isofs, and how is it made to be readonly.......my 
ultimate aim is to make the filesystem readonly upon after being 
written, and the file closed.   Not sure if it can be done, but I 
envisaged a lot of audit journalling are of these nature.   Of course, 
it is always possible to "dd" the filesystem to modify the content, but 
then if given design into its protection mechanism (like incremental 
checksum - current checksum based on previous checksum, generated and 
stored together with the file, upon after every writing of data) we can 
always protect its integrity.   Aim is to set it to readonly......anyone 
can read....but not modifiable.   As a start I will try out patching 
ext2, hoping that it is much simpler than ext3/ext4.

Upon changes to its content (via dd) it will invalidate the immediate 
future incremental checksum.   Similarly, if u patch the current 
checksum (which depends on a hash function of previous data, and 
previous checksum), it will not be valid, as the current checksum is 
also dependent on the history of previous checksum.   So everytime u 
change the content via dd, u will need to modify the next checksum, 
which is calculated based on this data, and the current checksum, which 
again provide the seed for the next checksum etc.   U will have to 
modify a lot of data, unless u are near the tip of the latest 
modification.   The smaller the chunk of data per checksum, the more 
difficult to keep up with the rate of modification.   Tradeoff is more 
work for CPU.

The userspace tool part will then always validate the checksum with the 
data being read, if modified, checksum will not be valid.   Since it is 
a hash function, given the modified data, and the previous checksum, it 
is not possible calculate the current checksum.

/Of course, if the intruder is root, then it is as good as not having 
all these complex calculations, so our assumption is that the machine is 
not compromised yet.

Then of course the "chattr" can be used as well - if it is not 
compromised.   True, but the possibility that it can be modified 
infinitely via chattr also exists - which is what write-once-read-many 
is against - to provide the assurance that it has not been overwritten 
the second time (possible technically, but very very difficult).

An equivalent requirements of such a filesystem will be:   a filesystem 
such that upon every changes made, a log of dates of changes are 
made.      Overhead for these feature is high, so a lightweighted 
version will be the date/history only it is first written.

On an existing ext2 filesystem, once the '"worm" kernel module is 
loaded, the feature immediately take effect - content becomes read-only 
but not modifiable, but modifiable for new contents.   And old contents 
may or may not have a checksum to protect it, but if the checksum 
exists, it will come from the last time "worm" was running and 
generating checksum.

So u have the best of both world - ext2/3/4 with/without worm.    If 
worm-enabled, it may also be configured at the directory level - some 
directory can be solely dedicated for worm-journalling.

What do u guys think - any conceptual errors?




/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Write once only but read many" filesystem

[Bryan Henderson]

You're still being really vague on your requirements.  Sometimes you talk 
about a filesystem that can't be modified after it has been written, and 
other times you talk about a filesystem that can be updated, but only to 
add information.  Sometimes you talk about a read-only restriction that is 
effective against the superuser (not possible) and other times say that 
it's OK if the superuser can modify the filesystem without a trace.

In addition to new ext3 features, using isofs, and chattr, mounting 
read-only and using file permissions also effect read-only status, and 
you'd have to explain how any of these don't meet your requirements.  But 
rest assured that none of them is effective against the superuser.

>Upon changes to its content (via dd) it will invalidate the immediate 
>future incremental checksum.

I couldn't tell what "it" is in this sentence, or what checksums you're 
thinking of.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: "Write once only but read many" filesystem

[Peter Teoh]

Bryan Henderson wrote:
> You're still being really vague on your requirements.  Sometimes you talk 
> about a filesystem that can't be modified after it has been written, and 
>   
Sorry for the confusion.   The requirement from upfront is that it is 
ALWAYS WRITE-ONCE only.   But many of you will say that it is 
impossible, as it can be written over again via other means.   Yes, the 
other means is via "dd", but that will require that you unmount the 
filesystem first, then using dd and hexeditor to modify the 
filesystem.   Not sure if I had made myself clear?

> other times you talk about a filesystem that can be updated, but only to 
>   

NO updating is possible.   Write once, and the file attributes defined 
from the first time it is written, and not modifiable after that - 
access control is from the kernel, and therefore userspace cannot do 
anything, once mounted of course.
> add information.  Sometimes you talk about a read-only restriction that is 
> effective against the superuser (not possible) and other times say that 
> it's OK if the superuser can modify the filesystem without a trace.
> In addition to new ext3 features, using isofs, and chattr, mounting 
> read-only and using file permissions also effect read-only status, and 
> you'd have to explain how any of these don't meet your requirements.  But 
>   
May be I will describe an auditing scenario to make myself clear - why 
the above does not meet the requirement.   First, an arbitrary 
application does something, then the auditing daemon will record it 
down, into a file.   Once the file is open for write, it is possible to 
continuosly write to it.   But once close, it automatically becomes 
read-only.   But permanently no userspace tool can reverse the read-only 
attribute - controlled from the kernel.   But while being opened and 
written, it is not possible to move back the file pointer as well - ie, 
once written, is written, not possible to overwrite.   Concurrently, 
other files can be opened on the same directory for writing.   Can any 
of the abovementioned userspace tool, or existing ext2/3/4 configuration 
can satisfy the above requirement?

> rest assured that none of them is effective against the superuser.
>
>   
Agreed - once the filesystem is unmounted, u can do anything to the 
filesystem.   But modifying any bytes of it will be difficult, the 
control mechanism as laid out in my earlier email.

>> Upon changes to its content (via dd) it will invalidate the immediate 
>> future incremental checksum.
>>     
>
> I couldn't tell what "it" is in this sentence, or what checksums you're 
> thinking of.
>
>   
Erh......Not sure what u mean?   What I mean is that if u unmount the 
filesystem, and attempt to modify any bytes of the write-once 
filesystem, it will most likely invalidate the integrity of subsequent 
data, illustrated by following diagram:

hash_functionA(checksumA, dataA) ==> checksumB.

hash_functionA(checksumB, dataB) ==> checksumC.

hash_functionA(checksumC, dataC) ==> checksumD.

Eg, if u attempt to modify the checksumB, 2 ways of detection is possible:

1.   next:   u can be detected by a mismatch of checksumC stored on the 
disk.  
2.   previous:   u can be detected by a mismatch of checksumB stored on 
disk (based on previous checksumA and dataA).

Or if u attempt to modify the dataB, similarly the two ways of detection 
applies as well.

So if u really want to modify a single bytes of data, u have to start 
from the very first item in the chain.   And with an added "salt" to the 
hash_functionA (random number), it is possible to branch off the 
hash_functionA(checksumB, dataB) into many different branches, thus 
implementing the concurrent open files concept.   And the algorithm of 
calculation as described easily facilitate a forward looking concept - 
no previous data have to be modified - so whatever written on disk, will 
be written, not modifiable anymore.

Is the concept clearer now?

> --
> Bryan Henderson                     IBM Almaden Research Center
> San Jose CA                         Filesystems
>
>
>
>   

Re: "Write once only but read many" filesystem

[Matthew Wilcox]

On Tue, Mar 25, 2008 at 08:47:16AM +0800, Peter Teoh wrote:
> Sorry for the confusion.   The requirement from upfront is that it is 
> ALWAYS WRITE-ONCE only.   But many of you will say that it is 
> impossible, as it can be written over again via other means.   Yes, the 
> other means is via "dd", but that will require that you unmount the 
> filesystem first, then using dd and hexeditor to modify the 
> filesystem.   Not sure if I had made myself clear?

You're confused.  The kernel cannot prevent root from modifying a block
device while a filesystem is mounted.

Sounds to me like what you really want is a printer.  You can't modify
what's been printed.

You could also use a second computer and log to that.

> May be I will describe an auditing scenario to make myself clear - why 
> the above does not meet the requirement.   First, an arbitrary 
> application does something, then the auditing daemon will record it 
> down, into a file.   Once the file is open for write, it is possible to 
> continuosly write to it.   But once close, it automatically becomes 
> read-only.   But permanently no userspace tool can reverse the read-only 
> attribute - controlled from the kernel.   But while being opened and 
> written, it is not possible to move back the file pointer as well - ie, 
> once written, is written, not possible to overwrite.   Concurrently, 
> other files can be opened on the same directory for writing.   Can any 
> of the abovementioned userspace tool, or existing ext2/3/4 configuration 
> can satisfy the above requirement?

Can't be done.  root can ptrace the auditing daemon and change what it
wants to write.

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: "Write once only but read many" filesystem

[Bryan Henderson]

>Sounds to me like what you really want is a printer.  You can't modify
>what's been printed.
>
>You could also use a second computer and log to that.

Don't forget write-once storage servers and file servers.  That's what's 
actually used in the real world for this.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: "Write once only but read many" filesystem

[Bryan Henderson]

>> other times you talk about a filesystem that can be updated, ...
>
>NO updating is possible.

I'm still not convinced you mean that, because the example you give is 
specifically of a file that becomes immutable after some writing to it. 
How about creating a new file?  That is a filesystem update.  Do you want 
to allow that?  How about directory modifications, such as rename and 
unlink?

>Once the file is open for write, it is possible to 
>continuosly write to it.   But once close, it automatically becomes 
>read-only.   But permanently no userspace tool can reverse the read-only 
>attribute - controlled from the kernel.   But while being opened and 
>written, it is not possible to move back the file pointer as well - ie, 
>once written, is written, not possible to overwrite.   Concurrently, 
>other files can be opened on the same directory for writing.   Can any 
>of the abovementioned userspace tool, or existing ext2/3/4 configuration 
>can satisfy the above requirement?

No.  And the first thing in the list to which you refer is what I believe 
your proposal is: adding a feature to ext3.   Modifying ext3 cannot 
satisfy the above requirements.


>hash_functionA(checksumA, dataA) ==> checksumB.
>
>hash_functionA(checksumB, dataB) ==> checksumC.
>
>hash_functionA(checksumC, dataC) ==> checksumD.

>So if u really want to modify a single bytes of data, u have to start 
>from the very first item in the chain. 

Yeah, that's what the bad guy will do.  So you haven't prevented someone 
from undetectably modifying previously written data.

>And with an added "salt" to the 
>hash_functionA (random number)

The bad guy would be sure to use the same salt, or change it.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems

Re: "Write once only but read many" filesystem

[Peter Teoh]

Bryan Henderson wrote:
>>> other times you talk about a filesystem that can be updated, ...
>>>       
>> NO updating is possible.
>>     
>
> I'm still not convinced you mean that, because the example you give is 
> specifically of a file that becomes immutable after some writing to it. 
> How about creating a new file?
Yes - always allowed.  

>   That is a filesystem update.  Do you want 
> to allow that?  How about directory modifications, such as rename and 
> unlink?
>   
Creating directories - always allowed.   Modification - rename/unlink 
etc will be disallowed.

> Yeah, that's what the bad guy will do.  So you haven't prevented someone 
> from undetectably modifying previously written data.
>
>   
This is the only difference between the current discussion, and the 
hardware WORM storage solution u mentioned in the previous email, due to 
it software vs hardware implementation aspect.   I have also found this:

ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/87/1177/CS-TR-87-1177.pdf

Well, I would like to thank you and Matthew for the discussion (I think 
using the printer is not advisable - the output is not really so 
immutable after all - JUST reprint the modified content will do.   
Similarly for WORM storage medium - just write into another CD.)

U agree with me?.   Sorry for the rubbish talk :-).

Anyway, I shall attempt to write some proof-of-concept patches to try 
out the idea.   I may fail.

Thanks.

Re: "Write once only but read many" filesystem

[Bryan Henderson]

>(I think 
>using the printer is not advisable - the output is not really so 
>immutable after all - JUST reprint the modified content will do. 
>Similarly for WORM storage medium - just write into another CD.)

In these systems, the untrusted person doesn't have physical access to the 
medium, though he has total access to the interface for adding data to it. 
 One example would be where the untrusted person is a hacker from the 
Internet.  Another would be where the medium is in the possession of an 
independent auditing firm.

WORM storage media is actually of little interest these days for security 
since you can accomplish the same thing with programming -- as long as the 
untrusted person doesn't have any access to the storage except a 
communication wire, the server can make data immutable even while using 
ordinary disk drives -- if you have enough confidence in the engineering.

--
Bryan Henderson                     IBM Almaden Research Center
San Jose CA                         Filesystems