XFS information leak during crash

XFS information leak during crash

[Jan Kasprzak]

	Hello, world!\n

I have found that after the system crash (e.h. a hard reset or a power
failure) XFS corrupts files which have been written to just before the crash:
The result is that those files contain data from random blocks on the
disk (e.g. from previously deleted files). This can have security/privacy
implications - users can see the contents of other users' old files.

	I have even written a test program, which creates/rewrites
files with known contents in a given directory. After the hard
reset while running this program some of the files contain blocks
with "random" data (i.e. not the original data and not the new data
either). Does XFS support a something like ext3's "data=ordered" mount
option? Otherwise it is pretty unusable on multi-user systems.

	This is on 2.6.11.10 and 2.6.14 running on x86_64 and i386 SMP
configurations. I may test it on UP if there is an interest.

	The quick-hack-style test program can be found at
http://www.fi.muni.cz/~kas/progs/xfsrewrite.c

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> Specs are a basis for _talking_about_ things. But they are _not_ a basis <
> for implementing software.                              --Linus Torvalds <

Re: XFS information leak during crash

[Nathan Scott]

Hello Jan,

On Wed, Nov 02, 2005 at 10:27:22PM +0100, Jan Kasprzak wrote:
> 	Hello, world!\n
> 
> I have found that after the system crash (e.h. a hard reset or a power
> failure) XFS corrupts files which have been written to just before the crash:
> The result is that those files contain data from random blocks on the
> disk (e.g. from previously deleted files). This can have security/privacy
> implications - users can see the contents of other users' old files.

If you think you have found a security issue, it would be courteous
to at least discuss this with the maintainers first.  And since you
are a frequent linux-xfs list poster too, it seems a bit odd that
you're reporting this on linux-kernel instead... *shrug*, whatever.

This issue affects every filesystem, right?  Or are you claiming its
only XFS affected here?  Have you run your parallel-buffered-writers
test case on any other filesystems?  I'd be interested in the results,
in particular, with all of the data=xxx modes of other filesystems.

> either). Does XFS support a something like ext3's "data=ordered" mount
> option? 

No, it doesn't.

> Otherwise it is pretty unusable on multi-user systems.

That's a ridiculous assertion.  While this small metadata vs. buffered-
data-write window exists on _any_ filesystem not using a data=ordered/
data=journaled mode (which I believe is quite a common mode of operation
even on filesystems that offer those modes), it is impossible to exploit
this in any sane way.  You'd think people on a multi-user system might
actually notice the machine being frequently rebooted while you try to
tickle this window to get at "interesting" uninitialised freespace, no?

Having said that, a data=ordered mode for XFS would be a nice feature.
It just hasn't reached the top of our priority list, and its not been
offered up as a patch by anyone yet.  If anyone's interested in writing
this, they should coordinate with hch and myself - there's a fair bit
of I/O path work being done at the moment, which in the end will make
a data=ordered mode alot easier to implement.

cheers.

-- 
Nathan

Re: XFS information leak during crash

[Jan Kasprzak]

Nathan Scott wrote:
: > The result is that those files contain data from random blocks on the
: > disk (e.g. from previously deleted files). This can have security/privacy
: > implications - users can see the contents of other users' old files.
: 
: If you think you have found a security issue, it would be courteous
: to at least discuss this with the maintainers first.

	Well, I think while it is a security issue, it is not serious
enough to make it secret (it is not exploitable by anyone except those
who are able to crash the machine).

: And since you
: are a frequent linux-xfs list poster too, it seems a bit odd that
: you're reporting this on linux-kernel instead... *shrug*, whatever.

	I am sorry for this one - I am not subscribed to linux-xfs.
Next time I will post to linux-xfs first.
: 
: This issue affects every filesystem, right?  Or are you claiming its
: only XFS affected here?  Have you run your parallel-buffered-writers
: test case on any other filesystems?  I'd be interested in the results,
: in particular, with all of the data=xxx modes of other filesystems.
: 
	I will do this tomorrow or the day after and post the results.

: > either). Does XFS support a something like ext3's "data=ordered" mount
: > option? 
: 
: No, it doesn't.
: 
	OK.

: > Otherwise it is pretty unusable on multi-user systems.
: 
: That's a ridiculous assertion.  While this small metadata vs. buffered-
: data-write window exists on _any_ filesystem not using a data=ordered/
: data=journaled mode (which I believe is quite a common mode of operation
: even on filesystems that offer those modes),

	As for ext3, I believe the vast majority of ext3 filesystems
run in data=ordered mode. But yes, the same problem affects all filesystem
except those running in data=ordered/journal mode.

: it is impossible to exploit
: this in any sane way.  You'd think people on a multi-user system might
: actually notice the machine being frequently rebooted while you try to
: tickle this window to get at "interesting" uninitialised freespace, no?

	Yes, of course. However, the issue is probably much worse
on XFS, because on XFS it probably affects not only the files being
created/extended, but also the files being rewritten. Most other
filesystems rewrite the files in-place, so when you rewrite the file,
even with data=writeback you get only the mix of the old and new
contents. Not somebody else's random data.

	This particular problem was that one of my users apparently opened
his TeX document just to fix few typos, and ended up with the file which
contained some part of a shell script and some binary data :-(

	I agree this is hard to exploit on purpose, however it can still
leak a sensitive data. For example, this particular server runs also
a mail server for ~2200 users, so a private mail can end up in somebody
else's files.

: Having said that, a data=ordered mode for XFS would be a nice feature.
: It just hasn't reached the top of our priority list, and its not been
: offered up as a patch by anyone yet.  If anyone's interested in writing
: this, they should coordinate with hch and myself - there's a fair bit
: of I/O path work being done at the moment, which in the end will make
: a data=ordered mode alot easier to implement.

	OK, thanks! I wish I would have time to do more kernel hacking ...

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> Specs are a basis for _talking_about_ things. But they are _not_ a basis <
> for implementing software.                              --Linus Torvalds <

Re: XFS information leak during crash

[Nathan Scott]

On Thu, Nov 03, 2005 at 12:36:29AM +0100, Jan Kasprzak wrote:
> ...
> 	Yes, of course. However, the issue is probably much worse
> on XFS, because on XFS it probably affects not only the files being
> created/extended, but also the files being rewritten. Most other

No, thats not correct - XFS behaves as most filesystems do and
will write over the top of existing data.

> filesystems rewrite the files in-place, so when you rewrite the file,
> even with data=writeback you get only the mix of the old and new
> contents. Not somebody else's random data.

XFS also rewrites files in-place.  You will never get someone else's
current data (that would be metadata corruption...), it would only
ever be uninitialised, previously-free space.  But as I said, other
filesystems have the same window in which this can happen (in the
absence of stronger data ordering/journalling semantics, of course).

cheers.

-- 
Nathan

Re: XFS information leak during crash

[Jan Kasprzak]

Nathan Scott wrote:
: XFS behaves as most filesystems do and
: will write over the top of existing data.

	OK, thanks for the clarification.

: XFS also rewrites files in-place.  You will never get someone else's
: current data (that would be metadata corruption...),

	Of course.

: it would only
: ever be uninitialised, previously-free space.

	Yes, but an old data from previously deleted files
(sendmail's temporary files, vim save files, etc) may contain
a sensitive information.

-Y.

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> Specs are a basis for _talking_about_ things. But they are _not_ a basis <
> for implementing software.                              --Linus Torvalds <

Re: XFS information leak during crash

[Nathan Scott]

On Thu, Nov 03, 2005 at 01:03:17AM +0100, Jan Kasprzak wrote:
> : it would only ever be uninitialised, previously-free space.
> 
> 	Yes, but an old data from previously deleted files
> (sendmail's temporary files, vim save files, etc) may contain
> a sensitive information.

Indeed.  But this is a generic issue affecting most filesystems;
its not specific to XFS as your original mail claimed.

cheers.

-- 
Nathan

Re: XFS information leak during crash

[Alan Cox]

On Iau, 2005-11-03 at 11:11 +1100, Nathan Scott wrote:
> On Thu, Nov 03, 2005 at 01:03:17AM +0100, Jan Kasprzak wrote:
> > : it would only ever be uninitialised, previously-free space.
> > 
> > 	Yes, but an old data from previously deleted files
> > (sendmail's temporary files, vim save files, etc) may contain
> > a sensitive information.
> 
> Indeed.  But this is a generic issue affecting most filesystems;
> its not specific to XFS as your original mail claimed.

Very true. You can use ext3 in data journalling mode if this is a
concern but that guarantee has a performance cost

Re: XFS information leak during crash

[Theodore Ts'o]

On Thu, Nov 03, 2005 at 12:45:49PM +0000, Alan Cox wrote:
> On Iau, 2005-11-03 at 11:11 +1100, Nathan Scott wrote:
> > On Thu, Nov 03, 2005 at 01:03:17AM +0100, Jan Kasprzak wrote:
> > > : it would only ever be uninitialised, previously-free space.
> > > 
> > > 	Yes, but an old data from previously deleted files
> > > (sendmail's temporary files, vim save files, etc) may contain
> > > a sensitive information.
> > 
> > Indeed.  But this is a generic issue affecting most filesystems;
> > its not specific to XFS as your original mail claimed.
> 
> Very true. You can use ext3 in data journalling mode if this is a
> concern but that guarantee has a performance cost

The default ordered journalling mode solves this problem at a much
lower cost.

						- Ted

Re: XFS information leak during crash

[Jan Kasprzak]

Jan Kasprzak wrote:
: Nathan Scott wrote:
: : 
: : This issue affects every filesystem, right?  Or are you claiming its
: : only XFS affected here?  Have you run your parallel-buffered-writers
: : test case on any other filesystems?  I'd be interested in the results,
: : in particular, with all of the data=xxx modes of other filesystems.
: : 
: 	I will do this tomorrow or the day after and post the results.
:
	Sorry for the delay - I did the test on other filesystems
as well: "random" (i.e. not rewritten) data after the hard reset were
inside the files in the following filesystems:

XFS
ext3 data=writeback
JFS

and my test program did not manage to put those "random" blocks into the
files when running on ext3 with data=ordered mode (as expected).

	I have tried ReiserFS as well, with mixed results:

- when I hit "reset" soon enough (but well after the HDD LED starts
	blinking), I end up with an empty directory after the reboot.
	It seems ReiserFS caches the filesystem operations much longer
	than other filesystems.
- I have not seen truly random data in the files on ReiserFS after the
	hard reset, but I have seen blocks of zeros instead of either
	old or new data. This may be a pure coincidence, so ReiserFS
	may belong to the same group as ext3 with data=writeback mode,
	but I was not able to make the files contain truly random
	blocks by hitting the reset button.

	So, does ReiserFS do some kind of data journaling/ordering as well?

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> Specs are a basis for _talking_about_ things. But they are _not_ a basis <
> for implementing software.                              --Linus Torvalds <

Re: XFS information leak during crash

[Mikulas Patocka]

>> either). Does XFS support a something like ext3's "data=ordered" mount
>> option?
>
> No, it doesn't.

BTW. Why does it sometimes overwrite files with zeros after crash and 
journal replay then? I thought that this was because it tries to avoid 
users seeing uninitialized data.

Mikulas

Re: XFS information leak during crash

[Nathan Scott]

On Thu, Nov 03, 2005 at 01:19:10AM +0100, Mikulas Patocka wrote:
> >> either). Does XFS support a something like ext3's "data=ordered" mount
> >> option?
> >
> > No, it doesn't.
> 
> BTW. Why does it sometimes overwrite files with zeros after crash and 
> journal replay then? I thought that this was because it tries to avoid 
> users seeing uninitialized data.

No, thats kinda related but not the same issue, its more to do
with a truncate (or open(O_TRUNC)) followed by buffered writes
to an existing file, which some applications do, and how that
interacts poorly with delayed allocation (nothing is "overwritten
with zeroes", its actually just a "hole").

But I do intend to get _some_ work done today, so you can google
for a more detailed answer there if you're interested.

cheers.

-- 
Nathan

Re: XFS information leak during crash

[Glen Overby]

On November 3, Mikulas Patocka wrote:
> BTW. Why does it sometimes overwrite files with zeros after crash and 
> journal replay then? I thought that this was because it tries to avoid 
> users seeing uninitialized data.

It doesn't overwrite the file with zeros.  You're getting an inode
that has a non-zero size, but no data in the file.  That is, a file
that is a single hole.  This happens because XFS logs metadata
quickly, but the data in the file gets written more slowly.

You'll see the same zeroing if you create a sparse file: write a
megabyte of data, lseek forward a megabyte, and write another megabyte
of data.  When reading the area you lseeked over, it will read as
zeros.

The same is done for files that were preallocated, but haven't been
written to (that is, the file has unwritten extents).

You can look at the files in question with xfs_bmap -v and see that
there's no extents there.

Glen Overby