Re: Atomic file data replace API

[Christian Stroetmann <stroetmann@ontolinux.com>]

On the 29.12.2010 18:15, Greg Freemyer wrote:
> On Wed, Dec 29, 2010 at 10:30 AM, Christian Stroetmann
> <stroetmann@ontolinux.com>  wrote:
>> On the 29.12.2010 13:42, Olaf van der Spek wrote:
>>> On Wed, Dec 29, 2010 at 10:20 AM, Amir Goldstein<amir73il@gmail.com>
>>>   wrote:
>>>> On Wed, Dec 29, 2010 at 12:58 AM, Olaf van der Spek
>>>> <olafvdspek@gmail.com>    wrote:
>>>>> On Tue, Dec 28, 2010 at 11:36 PM, Ric Wheeler<rwheeler@redhat.com>
>>>>>   wrote:
>>>>>> I think that various developers have answered this for you several
>>>>>> times.
>>>>> Not really, unfortunately. Haven't seen a single link to code that
>>>>> shows how to do it properly.
>> No, not this way. You were and still are asked for delivering the code.
>> Don't pervert the threat of the discussion.
>>
>>>>> Temp file, fsync, rename is often mentioned but that skips the
>>>>> preserving meta-data part and this part, which you also skipped:
>>>>> One use case would be updating a file in a safe way when you have
>>>>> write access to that file but not to anything else.
>>>>>
>>>> I think it is safe to say that the *only* option you have now is "temp
>>>> file, fsync, rename".
>>> I'm really looking for a concrete code snippet/function that does this.
>>> For example, file permissions should definitely be preserved.
>>>
>>>> There is no "generic atomic file data replace API in Linux", though it
>>>> is available via
>>>> private ioctl for XFS and EXT4.
>>>>
>>>> You have started a bit of a storm with your previous thread, which
>>>> doesn't help you
>>>> much in moving forward in the current thread (previous thread is still
>>>> more popular).
>>>> I suggest that you humbly swallow you need to know WHY is it hard to
>>>> implement
>>>> non-durable atomic API and focus your attention on the very achievable
>>>> data replace API.
>>>>
>>>> IMHO, implementing atomic swap_inodes_data operation shouldn't be
>>>> difficult
>>>> in most file systems (only implementation is simple, but testing and
>>>> maintaining
>>>> is not to be taken lightly).
>>>> Something along the lines of:
>>>> 1. aquire inodes write/truncate locks
>>>> 2. start transaction
>>>> 3. check/update quota limits
>>>> 4. swap inodes i_data content
>>>> 5. invalidate (or swap?) inodes page caches
>>>> 6. mark inodes dirty
>>>> 7. end transaction&    release locks
>>>>
>>>> The real challenge would be to get everyone to agree on a common API
>>>> and carve it in stone to the kernel's ABI (is it just swap_inodes_data?
>>>> maybe also swap_inode_data_ranges? what about some options?)
>>> Swapping data is an improvement but still not ideal. The API is also
>>> more complex than O_ATOMIC.
>>>
>>>> Also, as wacky and (some say) faulty the UNIX permissions models is,
>>>> current systems have grown old with it, and even 'improving' the behavior
>>>> of some applications, may wake up sleeping monsters, so it will not
>>>> be done until enough people have pointed out security or usability
>>>> issues, which could not be solved otherwise.
>>> Each app makes it's own decision about what API to use. Supporting
>>> atomic stuff doesn't change the behaviour of existing apps.
>> Wrong, we are talking here in the first place about general atomic FS
>> operations. And to guarantee atomicity you have to change general FS
>> functions in such a way that in the end all other applications are affected,
>> or otherwise you have to implement an own (larger part of an) FS.
>> At this point there is no discussion anymore without code from you, because
>> this subject is as well discussed to the maximum in information
>> processing/informatics/computer science.
>>
>>>> In other words, until you find an *application* that wants to allow other
>>>> user to modify the content of a file and preserve it's metadata and
>>>> ownership.
>>>> And unless that application cannot find a better way to achieve what it
>>>> wanted
>>>> to do in the first place, or unless that application already has a
>>>> large install base
>>>> which suffers from *a problem*, you will not have proven *the need*.
>>> Maybe I should ask devs of some large apps on their take of this issue.
>> Nonsense, because they are already using:
>> a) the functions available by an FS,
>> b) the functions available by a DBMS, or
>> c) a propritary special solution based on the available functions of the OS
>> and additional functionality that they develope and maintain themselves
>> for their comparable use cases since decades due to the cost vs. benefit
>> ratio.
> <sarcasm>
> Olaf, clearly if you want to find issues / use cases for your new API
> you should not talk to developers of complex tools.  They have it all
> figured out.
>
> It's only you that doesn't know how to code up a userspace solution to
> the problem.
> <\sarcasm>

<no_sarcasm>
This is not the place for sarcasm.
</no_sarcasm>

> Surely productivity suites like openoffice have to address the issue.
> How satisfied they are I don't know.  And despite Neil's argument that
> only one user should be able to write to a given doc, that is just not
> how normal office suites work today.

I think that Neil doesn't meant it in this way or context.

> Also, I believe KDE and its myriad of config files has issues with
> major config file corruption due to unexpected shutdowns during the
> config file update process, so they certainly don't have it figured
> out.
>
> Why don't they use the temp file, fsync, rename process?

<no_sarcasm>
Because they figured it out?!
</no_sarcasm>

> Those are the 2 user-space suites I would go investigate first.  I'm
> sure there are many others.
>
> Also, I believe Windows offers an API like your proposing.  How does
> Samba support it?
>
> Greg
>

<no sarcasm>
Furthermore, in conjunction with the given 2 user-space suites it was said:
"I don't know" and
"I believe".
</no sarcasm>

==>  leaving the thread
Please don't TO and CC anymore.
E-mails that are related with this thread will be sorted by name and then deleted without reading on the behalf of the reciever.


Christian Stroetmann