Re: fsync, rename, O_ATOMIC/O_PONIES

Re: fsync, rename, O_ATOMIC/O_PONIES

[Olaf van der Spek]

Hi XFS devs,

I'm a bit late to the party, but I'd like to respond to some comments at 
http://lwn.net/Articles/476263/#Comments
http://lwn.net/Articles/484709/

 > Could any of the fsync advocates post real code that does the atomic 
variant of open, write, close?

 > Hint: it's not possible without tons of regressions.

 > Linux devs should really provide a proper solution (like O_ATOMIC) 
instead of blaming app devs for not doing the impossible.

I'd like to ask:
- Is there a tool to log all unsafe operations?
- What is the *right* way to update a file?

I've asked the last question multiple times, but nobody has been able to 
give me a proper answer.

Greetings,

Olaf

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: fsync, rename, O_ATOMIC/O_PONIES

[Christoph Hellwig]

> I'd like to ask:
> - Is there a tool to log all unsafe operations?

What is an unsafe operation?  Anything that's dumb and might lose data?

The possibilities are sheer endless.

> - What is the *right* way to update a file?

 fd = open(tmpfile, ...);
 write(fd, ...);  // or any other update
 fdatasync(fd);
 rename(tmpfile, realfile);


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: fsync, rename, O_ATOMIC/O_PONIES

[Olaf van der Spek]

On 2-3-2012 14:12, Christoph Hellwig wrote:
>> I'd like to ask:
>> - Is there a tool to log all unsafe operations?
>
> What is an unsafe operation?  Anything that's dumb and might lose data?

For example

> The possibilities are sheer endless.

And?

>> - What is the *right* way to update a file?
>
>   fd = open(tmpfile, ...);
>   write(fd, ...);  // or any other update
>   fdatasync(fd);
>   rename(tmpfile, realfile);

Argh, come on.
That's not real and it's not complete. tmpfile is undefined, errors 
aren't handled and you have lots of unlisted assumptions or regressions.

Olaf

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: fsync, rename, O_ATOMIC/O_PONIES

[Dave Chinner]

On Sun, Mar 04, 2012 at 06:14:39PM +0100, Olaf van der Spek wrote:
> On 2-3-2012 14:12, Christoph Hellwig wrote:
> >>I'd like to ask:
> >>- Is there a tool to log all unsafe operations?
> >
> >What is an unsafe operation?  Anything that's dumb and might lose data?
> 
> For example
> 
> >The possibilities are sheer endless.
> 
> And?
> 
> >>- What is the *right* way to update a file?
> >
> >  fd = open(tmpfile, ...);
> >  write(fd, ...);  // or any other update
> >  fdatasync(fd);
> >  rename(tmpfile, realfile);
> 
> Argh, come on.
> That's not real and it's not complete. tmpfile is undefined, errors
> aren't handled and you have lots of unlisted assumptions or
> regressions.

The above is perfectly reasonable psuedo code for quickly describing
how to safely overwriting a file. If you want to know about error
handling and assumptions, read the man pages for operation.

To make it easy for you, I'll just point out this has already been
dealt with early in in the LWN thread for that article, here's the
link to the comment and the relevant LWN article providing you with
all the information you want:

http://lwn.net/Articles/476606/
https://lwn.net/Articles/457667/

And that my stance on this atomic rename subject is simply this: If
we want to change reanme behaviour, then the filesystem is not the
right place to do it - atomic rename semantics need to be defined
and enforced at the VFS.  See here:

http://lwn.net/Articles/479152/

This has all been hashed out before - we don't need to revisit this
discussion here. If you want to solve the problem once and for all,
submit patches to LKML to implement the new rename semantics in a
new syscall....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: fsync, rename, O_ATOMIC/O_PONIES

[Olaf van der Spek]

On 5-3-2012 2:02, Dave Chinner wrote:
>> Argh, come on.
>> That's not real and it's not complete. tmpfile is undefined, errors
>> aren't handled and you have lots of unlisted assumptions or
>> regressions.
>
> The above is perfectly reasonable psuedo code for quickly describing
> how to safely overwriting a file. If you want to know about error
> handling and assumptions, read the man pages for operation.

But I don't have a psuedo code compiler. Using psuedo code hides 
complexity and bugs. Even the code from Jeff Moyer in the article you're 
refering too contained bugs.

Don't you think it's quite strange there's no real code available to 
handle this widespread problem?

> To make it easy for you, I'll just point out this has already been
> dealt with early in in the LWN thread for that article, here's the
> link to the comment and the relevant LWN article providing you with
> all the information you want:
>
> http://lwn.net/Articles/476606/
> https://lwn.net/Articles/457667/

It also doesn't handle the assumptions / regressions: 
http://lwn.net/Articles/485182/
Assuming the target is not a symlink to a different volume
Assuming you are allowed to create the tmp file
Assuming you are allowed to overwrite an existing file having the same 
name as your tmp file
Assuming it's ok to reset meta-data, like file owner, permissions, acls, 
creation timestamp, etc.
Assuming the performance regression due to fsync is ok (request was for 
atomic, not durable)

> And that my stance on this atomic rename subject is simply this: If
> we want to change reanme behaviour, then the filesystem is not the
> right place to do it - atomic rename semantics need to be defined
> and enforced at the VFS.  See here:
>
> http://lwn.net/Articles/479152/

I don't think rename is part of the solution.

Olaf

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: fsync, rename, O_ATOMIC/O_PONIES

[Dave Chinner]

On Mon, Mar 05, 2012 at 04:44:26PM +0100, Olaf van der Spek wrote:
> On 5-3-2012 2:02, Dave Chinner wrote:
> >>Argh, come on.
> >>That's not real and it's not complete. tmpfile is undefined, errors
> >>aren't handled and you have lots of unlisted assumptions or
> >>regressions.
> >
> >The above is perfectly reasonable psuedo code for quickly describing
> >how to safely overwriting a file. If you want to know about error
> >handling and assumptions, read the man pages for operation.
> 
> But I don't have a psuedo code compiler. Using psuedo code hides
> complexity and bugs. Even the code from Jeff Moyer in the article
> you're refering too contained bugs.
> 
> Don't you think it's quite strange there's no real code available to
> handle this widespread problem?

What you are saying is this:

"I'm too lazy to understand data integrity principles and apply them
to my application. Can someone please write some code perfect for my
needs so I can copy and paste them into my application."

Stop wasting our time by being obtuse and repeating silly arguments
as to why this isn't your problem and instead go read the man pages
and modify the example code you've already been pointed at to do
exactly what you need.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs