truncate head of file?

truncate head of file?

[Howard Chu]

Was thinking it would be very handy to have a truncate() variant that deletes 
pages from the head of a file. This could be leveraged to make logfiles easier 
to maintain, as an example. Anyone else interested, think this would be nice 
to have?

(Note - not the same as just punching holes in the beginning of the file - we 
want the beginning of the file to advance as well, past the deleted pages.)
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: truncate head of file?

[Randy Dunlap]

On 08/19/14 22:45, Howard Chu wrote:
> Was thinking it would be very handy to have a truncate() variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?
> 
> (Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)

What is a "page" of a file?

Thanks,
-- 
~Randy

Re: truncate head of file?

[Lukáš Czerner]

On Tue, 19 Aug 2014, Howard Chu wrote:

> Date: Tue, 19 Aug 2014 22:45:16 -0700
> From: Howard Chu <hyc@symas.com>
> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
> Subject: truncate head of file?
> 
> Was thinking it would be very handy to have a truncate() variant that deletes
> pages from the head of a file. This could be leveraged to make logfiles easier
> to maintain, as an example. Anyone else interested, think this would be nice
> to have?
> 
> (Note - not the same as just punching holes in the beginning of the file - we
> want the beginning of the file to advance as well, past the deleted pages.)

I am not really sure I understand the behaviour you'd like to see.
Can you please explain the behaviour including more concrete use
case ?

Thanks!
-Lukas

Re: truncate head of file?

[Howard Chu]

Lukáš Czerner wrote:
> On Tue, 19 Aug 2014, Howard Chu wrote:
>
>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>> From: Howard Chu <hyc@symas.com>
>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>> Subject: truncate head of file?
>>
>> Was thinking it would be very handy to have a truncate() variant that deletes
>> pages from the head of a file. This could be leveraged to make logfiles easier
>> to maintain, as an example. Anyone else interested, think this would be nice
>> to have?
>>
>> (Note - not the same as just punching holes in the beginning of the file - we
>> want the beginning of the file to advance as well, past the deleted pages.)
>
> I am not really sure I understand the behaviour you'd like to see.
> Can you please explain the behaviour including more concrete use
> case ?

For example - we have a logfile (opened O_APPEND) that grows continuously. We 
want to delete some old log info from the head of the file. We could use "hole 
punching" to cause a specific range of data to be freed, but that just leaves 
a sparse file. If we were to cat this file the read() would have to advance 
thru all of that empty space before arriving at actual log data. We want both 
the data to be freed and for the logical beginning of the file to be moved 
forward, to match the location of where the remaining data begins.

Freeing the space would be simplest if we just deallocate X pages from the 
file, and then the beginning of the file becomes the beginning of the first 
remaining page of the file.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: truncate head of file?

[Lukáš Czerner]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2154 bytes --]

On Wed, 20 Aug 2014, Howard Chu wrote:

> Date: Wed, 20 Aug 2014 00:00:56 -0700
> From: Howard Chu <hyc@symas.com>
> To: Lukáš Czerner <lczerner@redhat.com>
> Cc: linux-fsdevel <linux-fsdevel@vger.kernel.org>
> Subject: Re: truncate head of file?
> 
> Lukáš Czerner wrote:
> > On Tue, 19 Aug 2014, Howard Chu wrote:
> > 
> > > Date: Tue, 19 Aug 2014 22:45:16 -0700
> > > From: Howard Chu <hyc@symas.com>
> > > To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
> > > Subject: truncate head of file?
> > > 
> > > Was thinking it would be very handy to have a truncate() variant that
> > > deletes
> > > pages from the head of a file. This could be leveraged to make logfiles
> > > easier
> > > to maintain, as an example. Anyone else interested, think this would be
> > > nice
> > > to have?
> > > 
> > > (Note - not the same as just punching holes in the beginning of the file -
> > > we
> > > want the beginning of the file to advance as well, past the deleted
> > > pages.)
> > 
> > I am not really sure I understand the behaviour you'd like to see.
> > Can you please explain the behaviour including more concrete use
> > case ?
> 
> For example - we have a logfile (opened O_APPEND) that grows continuously. We
> want to delete some old log info from the head of the file. We could use "hole
> punching" to cause a specific range of data to be freed, but that just leaves
> a sparse file. If we were to cat this file the read() would have to advance
> thru all of that empty space before arriving at actual log data. We want both
> the data to be freed and for the logical beginning of the file to be moved
> forward, to match the location of where the remaining data begins.
> 
> Freeing the space would be simplest if we just deallocate X pages from the
> file, and then the beginning of the file becomes the beginning of the first
> remaining page of the file.

Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE is
for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61

However it will not work with O_APPEND nor does any other fallocate
flag except pure fallocate, so not even punch hole would have
worked.

-Lukas

Re: truncate head of file?

[Howard Chu]

Lukáš Czerner wrote:
> On Wed, 20 Aug 2014, Howard Chu wrote:
>
>> Date: Wed, 20 Aug 2014 00:00:56 -0700
>> From: Howard Chu <hyc@symas.com>
>> To: Lukáš Czerner <lczerner@redhat.com>
>> Cc: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>> Subject: Re: truncate head of file?
>>
>> Lukáš Czerner wrote:
>>> On Tue, 19 Aug 2014, Howard Chu wrote:
>>>
>>>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>>>> From: Howard Chu <hyc@symas.com>
>>>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>> Subject: truncate head of file?
>>>>
>>>> Was thinking it would be very handy to have a truncate() variant that
>>>> deletes
>>>> pages from the head of a file. This could be leveraged to make logfiles
>>>> easier
>>>> to maintain, as an example. Anyone else interested, think this would be
>>>> nice
>>>> to have?
>>>>
>>>> (Note - not the same as just punching holes in the beginning of the file -
>>>> we
>>>> want the beginning of the file to advance as well, past the deleted
>>>> pages.)
>>>
>>> I am not really sure I understand the behaviour you'd like to see.
>>> Can you please explain the behaviour including more concrete use
>>> case ?
>>
>> For example - we have a logfile (opened O_APPEND) that grows continuously. We
>> want to delete some old log info from the head of the file. We could use "hole
>> punching" to cause a specific range of data to be freed, but that just leaves
>> a sparse file. If we were to cat this file the read() would have to advance
>> thru all of that empty space before arriving at actual log data. We want both
>> the data to be freed and for the logical beginning of the file to be moved
>> forward, to match the location of where the remaining data begins.
>>
>> Freeing the space would be simplest if we just deallocate X pages from the
>> file, and then the beginning of the file becomes the beginning of the first
>> remaining page of the file.
>
> Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE is
> for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61

Ah, thanks for the tip.

> However it will not work with O_APPEND nor does any other fallocate
> flag except pure fallocate, so not even punch hole would have
> worked.

Hm, that's a little bit inconvenient. What if the process calling fallocate() 
is different from the one appending to the log (i.e., using two separate file 
descriptors)? Also, when a range is collapsed, are all processes with 
descriptors opened on the file updated - are their seek positions correctly 
decremented? I dug up the patches and didn't see any code to handle this but 
may have missed it.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: truncate head of file?

[Ashish Sangwan]

On Wed, Aug 20, 2014 at 2:03 PM, Howard Chu <hyc@symas.com> wrote:
> Lukáš Czerner wrote:
>>
>> On Wed, 20 Aug 2014, Howard Chu wrote:
>>
>>> Date: Wed, 20 Aug 2014 00:00:56 -0700
>>> From: Howard Chu <hyc@symas.com>
>>> To: Lukáš Czerner <lczerner@redhat.com>
>>> Cc: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>> Subject: Re: truncate head of file?
>>>
>>> Lukáš Czerner wrote:
>>>>
>>>> On Tue, 19 Aug 2014, Howard Chu wrote:
>>>>
>>>>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>>>>> From: Howard Chu <hyc@symas.com>
>>>>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>>> Subject: truncate head of file?
>>>>>
>>>>> Was thinking it would be very handy to have a truncate() variant that
>>>>> deletes
>>>>> pages from the head of a file. This could be leveraged to make logfiles
>>>>> easier
>>>>> to maintain, as an example. Anyone else interested, think this would be
>>>>> nice
>>>>> to have?
>>>>>
>>>>> (Note - not the same as just punching holes in the beginning of the
>>>>> file -
>>>>> we
>>>>> want the beginning of the file to advance as well, past the deleted
>>>>> pages.)
>>>>
>>>>
>>>> I am not really sure I understand the behaviour you'd like to see.
>>>> Can you please explain the behaviour including more concrete use
>>>> case ?
>>>
>>>
>>> For example - we have a logfile (opened O_APPEND) that grows
>>> continuously. We
>>> want to delete some old log info from the head of the file. We could use
>>> "hole
>>> punching" to cause a specific range of data to be freed, but that just
>>> leaves
>>> a sparse file. If we were to cat this file the read() would have to
>>> advance
>>> thru all of that empty space before arriving at actual log data. We want
>>> both
>>> the data to be freed and for the logical beginning of the file to be
>>> moved
>>> forward, to match the location of where the remaining data begins.
>>>
>>> Freeing the space would be simplest if we just deallocate X pages from
>>> the
>>> file, and then the beginning of the file becomes the beginning of the
>>> first
>>> remaining page of the file.
>>
>>
>> Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE is
>> for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61
>
>
> Ah, thanks for the tip.
>
>
>> However it will not work with O_APPEND nor does any other fallocate
>> flag except pure fallocate, so not even punch hole would have
>> worked.
>
>
> Hm, that's a little bit inconvenient. What if the process calling
There seems to be some confusion between per fd flag O_APPEND and
inode flag S_APPEND.
Collapse range can work well with O_APPEND flag but it will not work
if S_APPEND is set.
> fallocate() is different from the one appending to the log (i.e., using two
> separate file descriptors)? Also, when a range is collapsed, are all
> processes with descriptors opened on the file updated - are their seek
> positions correctly decremented? I dug up the patches and didn't see any
> code to handle this but may have missed it.
You are right. Currently seek positions are not updated.
But this will not be of any problem (at least not on XFS) for your use
case if file is open with O_APPEND.
Refer to this discussion:
https://lkml.org/lkml/2014/2/26/701
>
>
> --
>   -- Howard Chu
>   CTO, Symas Corp.           http://www.symas.com
>   Director, Highland Sun     http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: truncate head of file?

[Howard Chu]

Ashish Sangwan wrote:
> On Wed, Aug 20, 2014 at 2:03 PM, Howard Chu <hyc@symas.com> wrote:
>> Lukáš Czerner wrote:
>>>
>>> On Wed, 20 Aug 2014, Howard Chu wrote:
>>>
>>>> Date: Wed, 20 Aug 2014 00:00:56 -0700
>>>> From: Howard Chu <hyc@symas.com>
>>>> To: Lukáš Czerner <lczerner@redhat.com>
>>>> Cc: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>> Subject: Re: truncate head of file?
>>>>
>>>> Lukáš Czerner wrote:
>>>>>
>>>>> On Tue, 19 Aug 2014, Howard Chu wrote:
>>>>>
>>>>>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>>>>>> From: Howard Chu <hyc@symas.com>
>>>>>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>>>> Subject: truncate head of file?
>>>>>>
>>>>>> Was thinking it would be very handy to have a truncate() variant that
>>>>>> deletes
>>>>>> pages from the head of a file. This could be leveraged to make logfiles
>>>>>> easier
>>>>>> to maintain, as an example. Anyone else interested, think this would be
>>>>>> nice
>>>>>> to have?
>>>>>>
>>>>>> (Note - not the same as just punching holes in the beginning of the
>>>>>> file -
>>>>>> we
>>>>>> want the beginning of the file to advance as well, past the deleted
>>>>>> pages.)
>>>>>
>>>>>
>>>>> I am not really sure I understand the behaviour you'd like to see.
>>>>> Can you please explain the behaviour including more concrete use
>>>>> case ?
>>>>
>>>>
>>>> For example - we have a logfile (opened O_APPEND) that grows
>>>> continuously. We
>>>> want to delete some old log info from the head of the file. We could use
>>>> "hole
>>>> punching" to cause a specific range of data to be freed, but that just
>>>> leaves
>>>> a sparse file. If we were to cat this file the read() would have to
>>>> advance
>>>> thru all of that empty space before arriving at actual log data. We want
>>>> both
>>>> the data to be freed and for the logical beginning of the file to be
>>>> moved
>>>> forward, to match the location of where the remaining data begins.
>>>>
>>>> Freeing the space would be simplest if we just deallocate X pages from
>>>> the
>>>> file, and then the beginning of the file becomes the beginning of the
>>>> first
>>>> remaining page of the file.
>>>
>>>
>>> Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE is
>>> for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61
>>
>>
>> Ah, thanks for the tip.
>>
>>
>>> However it will not work with O_APPEND nor does any other fallocate
>>> flag except pure fallocate, so not even punch hole would have
>>> worked.
>>
>>
>> Hm, that's a little bit inconvenient. What if the process calling
> There seems to be some confusion between per fd flag O_APPEND and
> inode flag S_APPEND.
> Collapse range can work well with O_APPEND flag but it will not work
> if S_APPEND is set.

Ah ok, that's much better.

>> fallocate() is different from the one appending to the log (i.e., using two
>> separate file descriptors)? Also, when a range is collapsed, are all
>> processes with descriptors opened on the file updated - are their seek
>> positions correctly decremented? I dug up the patches and didn't see any
>> code to handle this but may have missed it.
> You are right. Currently seek positions are not updated.
> But this will not be of any problem (at least not on XFS) for your use
> case if file is open with O_APPEND.
> Refer to this discussion:
> https://lkml.org/lkml/2014/2/26/701

Thanks, looks good.

I find it somewhat surprising that such a useful feature was implemented for 
the benefit of video editing, when it has massive applicability in databases 
(particularly for simplified truncation of write-ahead logs, as well as for 
append-only database designs). This is such an obviously beneficial feature 
but I haven't seen it talked up very much. Of course, being so specific to 
very recent Linux releases will certainly impede its adoption.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: truncate head of file?

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]


On Aug 20, 2014, at 1:00 AM, Howard Chu <hyc@symas.com> wrote:
> For example - we have a logfile (opened O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file the read() would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.

It would also be possible to use fallocate(FALLOC_FL_PUNCH_HOLE)
to punch out the processed records, and seek(SEEK_DATA) to find
the first record that has not been processed.  That would avoid
problems with the file size changing and O_APPEND.

Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: truncate head of file?

[Howard Chu]

Andreas Dilger wrote:
>
> On Aug 20, 2014, at 1:00 AM, Howard Chu <hyc@symas.com> wrote:
>> For example - we have a logfile (opened O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file the read() would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
>
> It would also be possible to use fallocate(FALLOC_FL_PUNCH_HOLE)
> to punch out the processed records, and seek(SEEK_DATA) to find
> the first record that has not been processed.  That would avoid
> problems with the file size changing and O_APPEND.

That would be OK, except that we want the file size to decrease. Otherwise the 
logical size grows without bound and eventually we'd have to recreate the file 
anyway.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Re: truncate head of file?

[Randy Dunlap]

On 08/20/14 00:00, Howard Chu wrote:
> Lukáš Czerner wrote:
>> On Tue, 19 Aug 2014, Howard Chu wrote:
>>
>>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>>> From: Howard Chu <hyc@symas.com>
>>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>> Subject: truncate head of file?
>>>
>>> Was thinking it would be very handy to have a truncate() variant that deletes
>>> pages from the head of a file. This could be leveraged to make logfiles easier
>>> to maintain, as an example. Anyone else interested, think this would be nice
>>> to have?
>>>
>>> (Note - not the same as just punching holes in the beginning of the file - we
>>> want the beginning of the file to advance as well, past the deleted pages.)
>>
>> I am not really sure I understand the behaviour you'd like to see.
>> Can you please explain the behaviour including more concrete use
>> case ?
> 
> For example - we have a logfile (opened O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file the read() would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
> 
> Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
> 

Hi Howard,

I am still interested in what you mean by a "page" of a file?
Is it fixed size?  What determines page size?

Thanks,
-- 
~Randy
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: truncate head of file?

[Howard Chu]

Randy Dunlap wrote:
> On 08/20/14 00:00, Howard Chu wrote:
>> Lukáš Czerner wrote:
>>> On Tue, 19 Aug 2014, Howard Chu wrote:
>>>
>>>> Date: Tue, 19 Aug 2014 22:45:16 -0700
>>>> From: Howard Chu <hyc@symas.com>
>>>> To: linux-fsdevel <linux-fsdevel@vger.kernel.org>
>>>> Subject: truncate head of file?
>>>>
>>>> Was thinking it would be very handy to have a truncate() variant that deletes
>>>> pages from the head of a file. This could be leveraged to make logfiles easier
>>>> to maintain, as an example. Anyone else interested, think this would be nice
>>>> to have?
>>>>
>>>> (Note - not the same as just punching holes in the beginning of the file - we
>>>> want the beginning of the file to advance as well, past the deleted pages.)
>>>
>>> I am not really sure I understand the behaviour you'd like to see.
>>> Can you please explain the behaviour including more concrete use
>>> case ?
>>
>> For example - we have a logfile (opened O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file the read() would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
>>
>> Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
>>
>
> Hi Howard,
>
> I am still interested in what you mean by a "page" of a file?
> Is it fixed size?  What determines page size?

I meant a typical page as used by the memory manager. For the most part that 
means e.g. 4KB since "large" page sizes are not actually pageable.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html