precise characterization of ext3 atomicity

precise characterization of ext3 atomicity

[Hans Reiser]

Is it correct to say of ext3 that it guarantees and only guarantees 
atomicity of writes that do not cross page boundaries?

I am trying to define the difference between "Atomic Reiser4" and ext3, 
as it seems to be a frequently asked question, and I am thinking of 
saying something like:

    Reiser4 allows you to define a set of up to A separate arbitrary 
filesystem operations (where A by default is not allowed to exceed 64) 
that are to be committed to disk atomically.  Every individual 
filesystem operation is atomic without the need to specify it.

    By contrast, ext3 only guarantees the atomicity of a single write 
that does not span a page boundary, and it guarantees that its internal 
metadata will not be corrupted even if your applications data is 
corrupted after the crash.

-- 
Hans

Re: precise characterization of ext3 atomicity

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> Is it correct to say of ext3 that it guarantees and only guarantees 
> atomicity of writes that do not cross page boundaries?

Yes.

>     By contrast, ext3 only guarantees the atomicity of a single write 
> that does not span a page boundary, and it guarantees that its internal 
> metadata will not be corrupted even if your applications data is 
> corrupted after the crash.

Not sure that I understand this.  In data=writeback mode, metadata
integrity is preserved but data writes may be lost.  In data=journal and
data=ordered modes the data and the metadata which refers to it are always
in sync on-disk.

Re: precise characterization of ext3 atomicity

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>Is it correct to say of ext3 that it guarantees and only guarantees 
>>atomicity of writes that do not cross page boundaries?
>>    
>>
>
>Yes.
>
>  
>
>>    By contrast, ext3 only guarantees the atomicity of a single write 
>>that does not span a page boundary, and it guarantees that its internal 
>>metadata will not be corrupted even if your applications data is 
>>corrupted after the crash.
>>    
>>
>
>Not sure that I understand this.  In data=writeback mode, metadata
>integrity is preserved but data writes may be lost.  In data=journal and
>data=ordered modes the data and the metadata which refers to it are always
>in sync on-disk.
>
>
>
>  
>
Perhaps the following is correct?

    By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write 
that does not span a page boundary, and it guarantees that its internal 
metadata will not be corrupted even if your application's data is 
corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).





-- 
Hans

Re: precise characterization of ext3 atomicity

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
>  Perhaps the following is correct?
> 
>      By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write 
>  that does not span a page boundary, and it guarantees that its internal 
>  metadata will not be corrupted even if your application's data is 
>  corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).

Correct != comprehensible ;)

"In all journalling modes ext3 guarantees metadata consistency after a
 crash.  In its data=journal and data=ordered modes ext3 also guarantees that
 user data is consistent with metadata after a crash.

 However ext3 does not provide user data atomicity guarantees beyond the
 scope of a single filesystem disk block (usually 4 kilobytes).  If a
 single write() spans two disk blocks it is possible that a crash partway
 through the write will result in only one of those blocks appearing in the
 file after recovery"

Re: precise characterization of ext3 atomicity

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>> Perhaps the following is correct?
>>
>>     By contrast, ext3 in data=journal and data=ordered modes only guarantees the atomicity of a single write 
>> that does not span a page boundary, and it guarantees that its internal 
>> metadata will not be corrupted even if your application's data is 
>> corrupted after the crash (due to the application spreading what should be committed atomically across more than one block).
>>    
>>
>
>Correct != comprehensible ;)
>
Touche!

Let's try this then:

Ext3 guarantees that its metadata will be comitted sufficiently 
atomically that after a crash it will be consistent with itself.

In data=journal and data=ordered modes ext3 also guarantees that the metadata will be committed atomically with the data they point to.  However ext3 does not provide user data atomicity guarantees beyond the scope of a single filesystem disk block (usually 4 kilobytes).  If a single write() spans two disk blocks it is possible that a crash partway through the write will result in only one of those blocks appearing in the file after recovery.



-- 
Hans

Re: precise characterization of ext3 atomicity

[Mike Fedyk]

On Thu, Sep 04, 2003 at 08:25:18PM +0400, Hans Reiser wrote:
> In data=journal and data=ordered modes ext3 also guarantees that the 
> metadata will be committed atomically with the data they point to.  However 
> ext3 does not provide user data atomicity guarantees beyond the scope of a 
> single filesystem disk block (usually 4 kilobytes).  If a single write() 
> spans two disk blocks it is possible that a crash partway through the write 
> will result in only one of those blocks appearing in the file after 
> recovery.

And how does reiser4 do this without changing the userspace apps?

Most files are written with several write() calls, so even if each call is
atomic, your entire file will not be there.

Also, ext3 could claim the same atomicity if it only updated meta-data on
write() call boundaries, instead of block boundaries.

Re: precise characterization of ext3 atomicity

[Antonio Vargas]

On Thu, Sep 04, 2003 at 11:15:40AM -0700, Mike Fedyk wrote:
> On Thu, Sep 04, 2003 at 08:25:18PM +0400, Hans Reiser wrote:
> > In data=journal and data=ordered modes ext3 also guarantees that the 
> > metadata will be committed atomically with the data they point to.  However 
> > ext3 does not provide user data atomicity guarantees beyond the scope of a 
> > single filesystem disk block (usually 4 kilobytes).  If a single write() 
> > spans two disk blocks it is possible that a crash partway through the write 
> > will result in only one of those blocks appearing in the file after 
> > recovery.
> 
> And how does reiser4 do this without changing the userspace apps?

It won't.


[ snip ] 
> Most files are written with several write() calls, so even if each call is
> atomic, your entire file will not be there.
> 
> Also, ext3 could claim the same atomicity if it only updated meta-data on
> write() call boundaries, instead of block boundaries.

There will be a new API to support userspace-controlled
multifile transactions.

At first stab, multifile transactions will be used internally to
implement extended attributes.

Now, another question is.. will the transaction API support commit() and
rollback()? *grin*

(wonder about coding a simple transactional database with
 shell scripts ;)

-- 
winden/network

1. Dado un programa, siempre tiene al menos un fallo.
2. Dadas varias lineas de codigo, siempre se pueden acortar a menos lineas.
3. Por induccion, todos los programas se pueden
   reducir a una linea que no funciona.

Re: precise characterization of ext3 atomicity

[Hans Reiser]

Mike Fedyk wrote:

>On Thu, Sep 04, 2003 at 08:25:18PM +0400, Hans Reiser wrote:
>  
>
>>In data=journal and data=ordered modes ext3 also guarantees that the 
>>metadata will be committed atomically with the data they point to.  However 
>>ext3 does not provide user data atomicity guarantees beyond the scope of a 
>>single filesystem disk block (usually 4 kilobytes).  If a single write() 
>>spans two disk blocks it is possible that a crash partway through the write 
>>will result in only one of those blocks appearing in the file after 
>>recovery.
>>    
>>
>
>And how does reiser4 do this without changing the userspace apps?
>
We don't.  We just make the hovercraft, we don't force you to go over 
the water.....

>
>Most files are written with several write() calls, so even if each call is
>atomic, your entire file will not be there.
>
>Also, ext3 could claim the same atomicity if it only updated meta-data on
>write() call boundaries, instead of block boundaries.
>
>
>  
>


-- 
Hans

Re: precise characterization of ext3 atomicity

[Mike Fedyk]

On Thu, Sep 04, 2003 at 10:37:10PM +0400, Hans Reiser wrote:
> Mike Fedyk wrote:
> 
> >On Thu, Sep 04, 2003 at 08:25:18PM +0400, Hans Reiser wrote:
> > 
> >
> >>In data=journal and data=ordered modes ext3 also guarantees that the 
> >>metadata will be committed atomically with the data they point to.  
> >>However ext3 does not provide user data atomicity guarantees beyond the 
> >>scope of a single filesystem disk block (usually 4 kilobytes).  If a 
> >>single write() spans two disk blocks it is possible that a crash partway 
> >>through the write will result in only one of those blocks appearing in 
> >>the file after recovery.
> >>   
> >>
> >
> >And how does reiser4 do this without changing the userspace apps?
> >
> We don't.  We just make the hovercraft, we don't force you to go over 
> the water.....

So by default with no user space modifications, reiser4 will be atomic for
each write() call, and ext3 will if it aligns withing a single page.

Is that correct?

Then you can go on to specify that you can have larger transactions if you
make some changes to the userspace apps.

Re: precise characterization of ext3 atomicity

[Hans Reiser]

Mike Fedyk wrote:

>On Thu, Sep 04, 2003 at 10:37:10PM +0400, Hans Reiser wrote:
>  
>
>>Mike Fedyk wrote:
>>
>>    
>>
>>>On Thu, Sep 04, 2003 at 08:25:18PM +0400, Hans Reiser wrote:
>>>
>>>
>>>      
>>>
>>>>In data=journal and data=ordered modes ext3 also guarantees that the 
>>>>metadata will be committed atomically with the data they point to.  
>>>>However ext3 does not provide user data atomicity guarantees beyond the 
>>>>scope of a single filesystem disk block (usually 4 kilobytes).  If a 
>>>>single write() spans two disk blocks it is possible that a crash partway 
>>>>through the write will result in only one of those blocks appearing in 
>>>>the file after recovery.
>>>>  
>>>>
>>>>        
>>>>
>>>And how does reiser4 do this without changing the userspace apps?
>>>
>>>      
>>>
>>We don't.  We just make the hovercraft, we don't force you to go over 
>>the water.....
>>    
>>
>
>So by default with no user space modifications, reiser4 will be atomic for
>each write() call, and ext3 will if it aligns withing a single page.
>
>Is that correct?
>
Yes.

>
>Then you can go on to specify that you can have larger transactions if you
>make some changes to the userspace apps.
>
>
>  
>
or you are a programmer who writes code....;-)  It's not that hard to 
write code....;-)

-- 
Hans

Re: precise characterization of ext3 atomicity

[Andreas Dilger]

On Sep 04, 2003  22:37 +0400, Hans Reiser wrote:
> Mike Fedyk wrote:
> >And how does reiser4 do this [export atomic ops to userspace]
> >without changing the userspace apps?
>
> We don't.  We just make the hovercraft, we don't force you to go over 
> the water.....

It is possible to do the same with ext3, namely exporting journal_start()
and journal_stop() (or some interface to them) to userspace so the application
can start a transaction for multiple operations.  We had discussed this in
the past, but decided not to do so because user applications can screw up in
so many ways, and if an application uses these interfaces it is possible to
deadlock the entire filesystem if the application isn't well behaved.

If the app doesn't eventually say "end the transaction", the filesystem might
wait indefinitely.  You could start adding more plumbing like "if the file
is closed (maybe because the process crashed), cancel the transaction",
and "if the process doesn't complete the transaction in time, cancel the
transaction", etc.  How do you guarantee in advance that the application
will be able to complete all of the operations it needs (i.e. if it runs
out of space in the filesystem or something)?

I suppose at worst, the application doesn't get its multi-op atomicity
guarantee, but I'm guessing that apps which use this interface depend on
it working properly or they wouldn't be using it.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: precise characterization of ext3 atomicity

[Hans Reiser]

Andreas Dilger wrote:

>On Sep 04, 2003  22:37 +0400, Hans Reiser wrote:
>  
>
>>Mike Fedyk wrote:
>>    
>>
>>>And how does reiser4 do this [export atomic ops to userspace]
>>>without changing the userspace apps?
>>>      
>>>
>>We don't.  We just make the hovercraft, we don't force you to go over 
>>the water.....
>>    
>>
>
>It is possible to do the same with ext3, namely exporting journal_start()
>and journal_stop() (or some interface to them) to userspace so the application
>can start a transaction for multiple operations.  We had discussed this in
>the past, but decided not to do so because user applications can screw up in
>so many ways, and if an application uses these interfaces it is possible to
>deadlock the entire filesystem if the application isn't well behaved.
>
Yup.  That's why we confine it to a (finite #defined number) set of 
operations within one sys_reiser4 call.  At some point we will allow 
trusted user space processes to span multiple system calls (mail server 
applicances, database appliances, etc., might find this useful).  You 
might consider supporting sys_reiser4 at some point.

>
>If the app doesn't eventually say "end the transaction", the filesystem might
>wait indefinitely.  You could start adding more plumbing like "if the file
>is closed (maybe because the process crashed), cancel the transaction",
>and "if the process doesn't complete the transaction in time, cancel the
>transaction", etc.  How do you guarantee in advance that the application
>will be able to complete all of the operations it needs (i.e. if it runs
>out of space in the filesystem or something)?
>
We will export our space reservation infrastructure code also.  We are 
still thinking about the right API for that.  At first I had some idea 
that we should calculate for the user how much space would be needed, 
but I am getting lazier as we get closer to actually doing it, and I am 
thinking we can add the helpful but complex in some cases 
estimate_sizeof() functions later, and for now just let the user grab 
space, and then if they exceed it return an error, and if they both 
exceed it and run out of disk space return a nastier error that tells 
them to go cope with their mistake.  Now I think it will be something 
that takes a 64 bit int that is the number of blocks to grab, compares 
it to their quota if any, and causes the sys_reiser4 to do nothing and 
error nicely if it can't get it.

When coding sometimes you have to be careful not to let the complex  
needs of the 20% prevent you from getting something that the 80% need 
and would  be happy with to market.  (Learned this one from Dave Hitz of 
NetApp.)

There's a lot of applications that have very simple needs in regards to 
atomicity, like write 3 things to 3 files as one atom.  If we can 
address that, then later people can do their PhDs on the needs of the 
complex 20%.... and hopefully send some nice patches.....

>
>I suppose at worst, the application doesn't get its multi-op atomicity
>guarantee, but I'm guessing that apps which use this interface depend on
>it working properly or they wouldn't be using it.
>
>Cheers, Andreas
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
>
>  
>


-- 
Hans

Re: precise characterization of ext3 atomicity

[Andreas Dilger]

On Sep 05, 2003  01:32 +0400, Hans Reiser wrote:
> Andreas Dilger wrote:
> >It is possible to do the same with ext3, namely exporting journal_start()
> >and journal_stop() (or some interface to them) to userspace so the application
> >can start a transaction for multiple operations.  We had discussed this in
> >the past, but decided not to do so because user applications can screw up in
> >so many ways, and if an application uses these interfaces it is possible to
> >deadlock the entire filesystem if the application isn't well behaved.
>
> That's why we confine it to a (finite #defined number) set of 
> operations within one sys_reiser4 call.  At some point we will allow 
> trusted user space processes to span multiple system calls (mail server 
> applicances, database appliances, etc., might find this useful).  You 
> might consider supporting sys_reiser4 at some point.

Ah, OK.  If you are confining the atom to a single syscall, then this is
a much easier problem to solve, assuming sys_reiser4() has a sufficiently
rich interface to express what people want to do.  It avoids all of the
potential problems that could arise if you want to keep a transaction
open over multiple syscalls.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: precise characterization of ext3 atomicity

[Chris Mason]

On Thu, 2003-09-04 at 18:03, Andreas Dilger wrote:
> On Sep 05, 2003  01:32 +0400, Hans Reiser wrote:
> > Andreas Dilger wrote:
> > >It is possible to do the same with ext3, namely exporting journal_start()
> > >and journal_stop() (or some interface to them) to userspace so the application
> > >can start a transaction for multiple operations.  We had discussed this in
> > >the past, but decided not to do so because user applications can screw up in
> > >so many ways, and if an application uses these interfaces it is possible to
> > >deadlock the entire filesystem if the application isn't well behaved.
> >
> > That's why we confine it to a (finite #defined number) set of 
> > operations within one sys_reiser4 call.  At some point we will allow 
> > trusted user space processes to span multiple system calls (mail server 
> > applicances, database appliances, etc., might find this useful).  You 
> > might consider supporting sys_reiser4 at some point.

Please rename sys_reiser4 if you want it to be a generic use syscall ;-)

-chris

Re: precise characterization of ext3 atomicity

[Pavel Machek]

Hi!

> Yup.  That's why we confine it to a (finite #defined number) set of 
> operations within one sys_reiser4 call.  At some point we will allow 
> trusted user space processes to span multiple system calls (mail 
> server applicances, database appliances, etc., might find this 
> useful).  You might consider supporting sys_reiser4 at some point.


Well, if you want that API to be widely usable, you should invent
better name than sys_reiser4 :-).
-- 
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Re: precise characterization of ext3 atomicity

[Gábor Lénárt]

On Tue, Sep 09, 2003 at 03:09:02PM +0200, Pavel Machek wrote:
> > Yup.  That's why we confine it to a (finite #defined number) set of 
> > operations within one sys_reiser4 call.  At some point we will allow 
> > trusted user space processes to span multiple system calls (mail 
> > server applicances, database appliances, etc., might find this 
> > useful).  You might consider supporting sys_reiser4 at some point.
> 
> 
> Well, if you want that API to be widely usable, you should invent
> better name than sys_reiser4 :-).

Like ActiveFSControll or such? ;-)

- Gábor (larta'H)

Re: precise characterization of ext3 atomicity

[Mike Fedyk]

On Tue, Sep 09, 2003 at 09:21:07PM +0200, G?bor L?n?rt wrote:
> On Tue, Sep 09, 2003 at 03:09:02PM +0200, Pavel Machek wrote:
> > > Yup.  That's why we confine it to a (finite #defined number) set of 
> > > operations within one sys_reiser4 call.  At some point we will allow 
> > > trusted user space processes to span multiple system calls (mail 
> > > server applicances, database appliances, etc., might find this 
> > > useful).  You might consider supporting sys_reiser4 at some point.
> > 
> > 
> > Well, if you want that API to be widely usable, you should invent
> > better name than sys_reiser4 :-).
> 
> Like ActiveFSControll or such? ;-)

How about sys_group_journal_ops?

Re: precise characterization of ext3 atomicity

[Daniel Phillips]

On Thursday 04 September 2003 17:55, Andrew Morton wrote:
> Hans Reiser <reiser@namesys.com> wrote:
> > Is it correct to say of ext3 that it guarantees and only guarantees
> > atomicity of writes that do not cross page boundaries?
>
> Yes.

Is that just happenstance, or does Posix or similar mandate it?

Regards,

Daniel

Re: precise characterization of ext3 atomicity

[Andrew Morton]

Daniel Phillips <phillips@arcor.de> wrote:
>
> On Thursday 04 September 2003 17:55, Andrew Morton wrote:
> > Hans Reiser <reiser@namesys.com> wrote:
> > > Is it correct to say of ext3 that it guarantees and only guarantees
> > > atomicity of writes that do not cross page boundaries?
> >
> > Yes.
> 
> Is that just happenstance, or does Posix or similar mandate it?

Happenstance.

It's semi-trivial to do this in ext3.  You'd open the file with O_ATOMIC
and a write() would either be completely atomic or would return -EFOO
without having written anything.

The thing which prevents this is the ranking order between journal_start()
and lock_page().

It's not trivial but also not too hard to change things so that
journal_start() can rank outside lock_page() - this would also offer some
CPU savings.

Can't say that I'm terribly motivated about the feature though.

Re: precise characterization of ext3 atomicity

[Daniel Phillips]

On Thursday 04 September 2003 22:10, Andrew Morton wrote:
> Daniel Phillips <phillips@arcor.de> wrote:
> > On Thursday 04 September 2003 17:55, Andrew Morton wrote:
> > > Hans Reiser <reiser@namesys.com> wrote:
> > > > Is it correct to say of ext3 that it guarantees and only guarantees
> > > > atomicity of writes that do not cross page boundaries?
> > >
> > > Yes.
> >
> > Is that just happenstance, or does Posix or similar mandate it?
>
> Happenstance.
>
> It's semi-trivial to do this in ext3.  You'd open the file with O_ATOMIC
> and a write() would either be completely atomic or would return -EFOO
> without having written anything.
>
> The thing which prevents this is the ranking order between journal_start()
> and lock_page().
>
> It's not trivial but also not too hard to change things so that
> journal_start() can rank outside lock_page() - this would also offer some
> CPU savings.
>
> Can't say that I'm terribly motivated about the feature though.

I'm relieved, I have always thought that some higher level synchronization is 
required for simultaneous writes.  So Hans might as well tell his fans that 
Ext3 makes no official guarantee, and neither does Linux.

Regards,

Daniel

Re: precise characterization of ext3 atomicity

[Hans Reiser]

Daniel Phillips wrote:

>I have always thought that some higher level synchronization is 
>required for simultaneous writes.  So Hans might as well tell his fans that 
>Ext3 makes no official guarantee, and neither does Linux.
>
>  
>
Not sure what you mean.

-- 
Hans

Re: precise characterization of ext3 atomicity

[Daniel Phillips]

On Thursday 04 September 2003 23:39, Hans Reiser wrote:
> Daniel Phillips wrote:
> >I have always thought that some higher level synchronization is
> >required for simultaneous writes.  So Hans might as well tell his fans
> >that Ext3 makes no official guarantee, and neither does Linux.
>
> Not sure what you mean.

Nothing bad.  More power to you for adding a transaction interface to Reiser4, 
and blazing that trail.  It's totally missing as a generic api at the moment, 
and needs a push.

Regards,

Daniel