Re: truncate head of file?

[Howard Chu <hyc@symas.com>]

Ashish Sangwan wrote:
> On Wed, Aug 20, 2014 at 2:03 PM, Howard Chu <hyc@symas.com> wrote:
>> Lukáš Czerner wrote:
>>>
>>> On Wed, 20 Aug 2014, Howard Chu wrote:
>>>
>>>> Date: Wed, 20 Aug 2014 00:00:56 -0700
>>>> From: Howard Chu <hyc@symas.com>
>>>> To: Lukáš Czerner <lczerner@redhat.com>
>>>> Cc: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>> Subject: Re: truncate head of file?
>>>>
>>>> Lukáš Czerner wrote:
>>>>>
>>>>> On Tue, 19 Aug 2014, Howard Chu wrote:
>>>>>
>>>>>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>>>>>> From: Howard Chu <hyc@symas.com>
>>>>>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>>>> Subject: truncate head of file?
>>>>>>
>>>>>> Was thinking it would be very handy to have a truncate() variant that
>>>>>> deletes
>>>>>> pages from the head of a file. This could be leveraged to make logfiles
>>>>>> easier
>>>>>> to maintain, as an example. Anyone else interested, think this would be
>>>>>> nice
>>>>>> to have?
>>>>>>
>>>>>> (Note - not the same as just punching holes in the beginning of the
>>>>>> file -
>>>>>> we
>>>>>> want the beginning of the file to advance as well, past the deleted
>>>>>> pages.)
>>>>>
>>>>>
>>>>> I am not really sure I understand the behaviour you'd like to see.
>>>>> Can you please explain the behaviour including more concrete use
>>>>> case ?
>>>>
>>>>
>>>> For example - we have a logfile (opened O_APPEND) that grows
>>>> continuously. We
>>>> want to delete some old log info from the head of the file. We could use
>>>> "hole
>>>> punching" to cause a specific range of data to be freed, but that just
>>>> leaves
>>>> a sparse file. If we were to cat this file the read() would have to
>>>> advance
>>>> thru all of that empty space before arriving at actual log data. We want
>>>> both
>>>> the data to be freed and for the logical beginning of the file to be
>>>> moved
>>>> forward, to match the location of where the remaining data begins.
>>>>
>>>> Freeing the space would be simplest if we just deallocate X pages from
>>>> the
>>>> file, and then the beginning of the file becomes the beginning of the
>>>> first
>>>> remaining page of the file.
>>>
>>>
>>> Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE is
>>> for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61
>>
>>
>> Ah, thanks for the tip.
>>
>>
>>> However it will not work with O_APPEND nor does any other fallocate
>>> flag except pure fallocate, so not even punch hole would have
>>> worked.
>>
>>
>> Hm, that's a little bit inconvenient. What if the process calling
> There seems to be some confusion between per fd flag O_APPEND and
> inode flag S_APPEND.
> Collapse range can work well with O_APPEND flag but it will not work
> if S_APPEND is set.

Ah ok, that's much better.

>> fallocate() is different from the one appending to the log (i.e., using two
>> separate file descriptors)? Also, when a range is collapsed, are all
>> processes with descriptors opened on the file updated - are their seek
>> positions correctly decremented? I dug up the patches and didn't see any
>> code to handle this but may have missed it.
> You are right. Currently seek positions are not updated.
> But this will not be of any problem (at least not on XFS) for your use
> case if file is open with O_APPEND.
> Refer to this discussion:
> https://lkml.org/lkml/2014/2/26/701

Thanks, looks good.

I find it somewhat surprising that such a useful feature was implemented for 
the benefit of video editing, when it has massive applicability in databases 
(particularly for simplified truncation of write-ahead logs, as well as for 
append-only database designs). This is such an obviously beneficial feature 
but I haven't seen it talked up very much. Of course, being so specific to 
very recent Linux releases will certainly impede its adoption.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html