Synchronization in UBIFS (zero length files)

Synchronization in UBIFS (zero length files)

[Tanya Brokhman]

Hi Artem/Richard

I've been looking into the "zero length files" issue in UBIFS and came 
across "Synchronization exceptions for buggy applications" @ 
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This 
section concludes with:

"We have plans to implement these features in UBIFS, but this has not 
been done yet. The problem is that UBI/MTD are fully synchronous and we 
cannot initiate asynchronous write-out, so we'd have to synchronously 
write files on close/rename, which is slow. So implementing these 
features would require implementing asynchronous I/O in UBI, which is a 
big job. But feel free to do this :-)."

So two questions:
1. was anything done in ubifs to handle file truncation and rename 
similar to ext4? Didn't find anything but afraid I might be missing 
something
2. Not sure I understand why "implementing these features would require 
implementing asynchronous I/O in UBI". Could you please elaborate on this?

Thanks,
Tanya Brokhman
-- 
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum, a Linux Foundation Collaborative Project

Re: Synchronization in UBIFS (zero length files)

[Richard Weinberger]

Tanya,

Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
> Hi Artem/Richard
> 
> I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:
> 
> "We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous
> write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a
> big job. But feel free to do this :-)."
> 
> So two questions:
> 1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something

I don't think so.

> 2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?

If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.
Having async IO for UBI is not trivial. Especially because of powercut safety.

Thanks,
//richard

Re: Synchronization in UBIFS (zero length files)

[Tanya Brokhman]

Hi Richard,

On 12/18/2014 1:12 PM, Richard Weinberger wrote:
> Tanya,
>
> Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
>> Hi Artem/Richard
>>
>> I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
>> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:
>>
>> "We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous
>> write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a
>> big job. But feel free to do this :-)."
>>
>> So two questions:
>> 1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something
>
> I don't think so.
>
>> 2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
>
> If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.

Regarding EXT4. In Theodore Tso article about this 
(http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/) 
he mentioned 3 ext4 patches that are handling this issue but I can't 
find them by their git id:
"These three patches (with git id’s bf1b69c0, f32b730a, and 8411e347) 
will cause a file to have any delayed allocation blocks to be allocated 
immediately when a file is replaced."
Are you familiar with the patches in question?

> Having async IO for UBI is not trivial. Especially because of powercut safety.

Is it even doable? It seems to me that if we add async functionality to 
ubi it will become not power-cut tolerant, unless we implement a journal 
there (really don't want to do that....)

Another idea: from what I've read on the subject so far (this is new to 
me so be patient), zero-length files can result due to several use cases 
one of them is renaming files in the following scheme:
1. write to temp file A - written to cache
2. rename A to B - written directly to disk
3. eventually data is flashed to disk
If power cut occurs between #2 and #3 we end up with empty file B. But 
what if we call fdatasync on A as part of the rename operation?
More specifically: I'm looking @ fs/ubifs/dir.c-ubifs_rename(). What if 
we force sync of old_inode by:
1. err = old_inode->i_sb->s_op->write_inode(old_inode, NULL);
2. err = ubifs_sync_wbufs_by_inode(c, old_inode);
If I understand the code correctly it is basically the same as adding 
fsync() in userspace after #2 above.

What do you think?

>
> Thanks,
> //richard
>


Thanks,
Tanya Brokhman
-- 
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum, a Linux Foundation Collaborative Project

Re: Synchronization in UBIFS (zero length files)

[Richard Weinberger]

Tanya,

Am 18.12.2014 um 15:53 schrieb Tanya Brokhman:
> Hi Richard,
> 
> On 12/18/2014 1:12 PM, Richard Weinberger wrote:
>> Tanya,
>>
>> Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
>>> Hi Artem/Richard
>>>
>>> I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
>>> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:
>>>
>>> "We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous
>>> write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a
>>> big job. But feel free to do this :-)."
>>>
>>> So two questions:
>>> 1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something
>>
>> I don't think so.
>>
>>> 2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
>>
>> If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.
> 
> Regarding EXT4. In Theodore Tso article about this (http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/) he mentioned 3 ext4 patches that
> are handling this issue but I can't find them by their git id:
> "These three patches (with git id’s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced."
> Are you familiar with the patches in question?

Nope.

>> Having async IO for UBI is not trivial. Especially because of powercut safety.
> 
> Is it even doable? It seems to me that if we add async functionality to ubi it will become not power-cut tolerant, unless we implement a journal there (really don't want to do
> that....)

In software you can do everything but I fear it will be messy.

> Another idea: from what I've read on the subject so far (this is new to me so be patient), zero-length files can result due to several use cases one of them is renaming files in
> the following scheme:
> 1. write to temp file A - written to cache
> 2. rename A to B - written directly to disk
> 3. eventually data is flashed to disk
> If power cut occurs between #2 and #3 we end up with empty file B. But what if we call fdatasync on A as part of the rename operation?
> More specifically: I'm looking @ fs/ubifs/dir.c-ubifs_rename(). What if we force sync of old_inode by:
> 1. err = old_inode->i_sb->s_op->write_inode(old_inode, NULL);
> 2. err = ubifs_sync_wbufs_by_inode(c, old_inode);
> If I understand the code correctly it is basically the same as adding fsync() in userspace after #2 above.
> 
> What do you think?

Let's wait what Artem says.
I bet here are more dragons. :)

Thanks,
//richard

Re: Synchronization in UBIFS (zero length files)

[Tanya Brokhman]

Hi Artem,

On 12/18/2014 5:12 PM, Richard Weinberger wrote:
> Tanya,
>
> Am 18.12.2014 um 15:53 schrieb Tanya Brokhman:
>> Hi Richard,
>>
>> On 12/18/2014 1:12 PM, Richard Weinberger wrote:
>>> Tanya,
>>>
>>> Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
>>>> Hi Artem/Richard
>>>>
>>>> I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
>>>> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:
>>>>
>>>> "We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous
>>>> write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a
>>>> big job. But feel free to do this :-)."
>>>>
>>>> So two questions:
>>>> 1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something
>>>
>>> I don't think so.
>>>
>>>> 2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
>>>
>>> If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.
>>
>> Regarding EXT4. In Theodore Tso article about this (http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/) he mentioned 3 ext4 patches that
>> are handling this issue but I can't find them by their git id:
>> "These three patches (with git id’s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced."
>> Are you familiar with the patches in question?
>
> Nope.
>
>>> Having async IO for UBI is not trivial. Especially because of powercut safety.
>>
>> Is it even doable? It seems to me that if we add async functionality to ubi it will become not power-cut tolerant, unless we implement a journal there (really don't want to do
>> that....)
>
> In software you can do everything but I fear it will be messy.
>
>> Another idea: from what I've read on the subject so far (this is new to me so be patient), zero-length files can result due to several use cases one of them is renaming files in
>> the following scheme:
>> 1. write to temp file A - written to cache
>> 2. rename A to B - written directly to disk
>> 3. eventually data is flashed to disk
>> If power cut occurs between #2 and #3 we end up with empty file B. But what if we call fdatasync on A as part of the rename operation?
>> More specifically: I'm looking @ fs/ubifs/dir.c-ubifs_rename(). What if we force sync of old_inode by:
>> 1. err = old_inode->i_sb->s_op->write_inode(old_inode, NULL);
>> 2. err = ubifs_sync_wbufs_by_inode(c, old_inode);
>> If I understand the code correctly it is basically the same as adding fsync() in userspace after #2 above.
>>
>> What do you think?
>
> Let's wait what Artem says.
> I bet here are more dragons. :)

Sorry to nag, this is a bit urgent for us. Could you please share your 
inputs on the above?


Thanks,
Tanya Brokhman
-- 
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum, a Linux Foundation Collaborative Project

Re: Synchronization in UBIFS (zero length files)

[Richard Weinberger]

Tanya,

Am 23.12.2014 um 09:26 schrieb Tanya Brokhman:
>>> What do you think?
>>
>> Let's wait what Artem says.
>> I bet here are more dragons. :)
> 
> Sorry to nag, this is a bit urgent for us. Could you please share your inputs on the above?

we're all in X-mas mode so don't expect answers within the next 1-2 weeks from us.
And to be honest, if this is an urgent issue for your employer which needs immediately action they
have to hire one of us as consultant. This is not how community support works.
Sorry for the hard words.

Thanks,
//richard

Re: Synchronization in UBIFS (zero length files)

[Tanya Brokhman]

On 12/23/2014 11:24 AM, Richard Weinberger wrote:
> Tanya,
>
> Am 23.12.2014 um 09:26 schrieb Tanya Brokhman:
>>>> What do you think?
>>>
>>> Let's wait what Artem says.
>>> I bet here are more dragons. :)
>>
>> Sorry to nag, this is a bit urgent for us. Could you please share your inputs on the above?
>
> we're all in X-mas mode so don't expect answers within the next 1-2 weeks from us.
> And to be honest, if this is an urgent issue for your employer which needs immediately action they
> have to hire one of us as consultant. This is not how community support works.
> Sorry for the hard words.

ok. sorry about that.
Marry Christmas all!


Thanks,
Tanya Brokhman
-- 
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum, a Linux Foundation Collaborative Project

Re: Synchronization in UBIFS (zero length files)

[Artem Bityutskiy]

Tanya,

On Thu, 2014-12-18 at 10:34 +0200, Tanya Brokhman wrote:
> So two questions:
> 1. was anything done in ubifs to handle file truncation and rename 
> similar to ext4? Didn't find anything but afraid I might be missing 
> something

Nope.

> 2. Not sure I understand why "implementing these features would require 
> implementing asynchronous I/O in UBI". Could you please elaborate on this?

Well, the idea was to implement something like BIO for UBIFS. You submit
your read or write requests, and go on doing your things. When your
requests are done, you have your call-back function invoked. You may
also go ahead and way for the completion of your request.

But we have the work queue API in Linux. Just create UBIFS work queue
(this will essentially create a separate thread for you, AFAIK), submit
write works there, let them write the closed-but-not-fsynced files
there.

On unmount, flush, cancel all jobs and destroy the queue.

Just off-the top of my head.

Re: Synchronization in UBIFS (zero length files)

[Bityutskiy, Artem]

On Sat, 2015-01-31 at 14:52 +0200, Artem Bityutskiy wrote:
> Well, the idea was to implement something like BIO for UBIFS.

For UBI, of course. We even started doing this at some point, but then
stopped. At that time I was confident we could do it.

The workqueue suggestion I gave may or may not work. It feels easier to
implement and less intrusive. But you never know before you try.

-- 
Best Regards,
Artem Bityutskiy
---------------------------------------------------------------------
Intel Finland Oy
Registered Address: PL 281, 00181 Helsinki 
Business Identity Code: 0357606 - 4 
Domiciled in Helsinki 

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Synchronization in UBIFS (zero length files)

[Tanya Brokhman]

Hi Artem

On 1/31/2015 2:57 PM, Bityutskiy, Artem wrote:
> On Sat, 2015-01-31 at 14:52 +0200, Artem Bityutskiy wrote:
>> Well, the idea was to implement something like BIO for UBIFS.
>
> For UBI, of course. We even started doing this at some point, but then
> stopped. At that time I was confident we could do it.
>
> The workqueue suggestion I gave may or may not work. It feels easier to
> implement and less intrusive. But you never know before you try.
>

Thanks for the explanation!
This was a while back and the task already completed :) I found a long 
long discussion you had with the community back in 2009 when Ted Ts'o 
implemented the sync-on-rename/truncate hack for ext4. You were asking 
the developers opinion on whether same should be done for ubifs or the 
sync should remain the user responsibility. The conclusion was that it's 
user responsibility to sync before renaming so nothing was done for ubifs.
Unfortunately our customer thought differently so I had to go through 
with adding this hack to ubifs. Not the most elegant solution to the 
zero-length-files problem, but the quickest and the safest for the moment.


Thanks,
Tanya Brokhman
-- 
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project