From mboxrd@z Thu Jan 1 00:00:00 1970 From: Howard Chu Subject: Re: truncate head of file? Date: Wed, 20 Aug 2014 15:02:29 -0700 Message-ID: <53F51AF5.1050509@symas.com> References: <53F435EC.4030601@symas.com> <53F447A8.8020804@symas.com> <53F45D52.70001@symas.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: =?UTF-8?B?THVrw6HFoSBDemVybmVy?= , linux-fsdevel To: Ashish Sangwan Return-path: Received: from zill.ext.symas.net ([69.43.206.106]:39598 "EHLO zill.ext.symas.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753308AbaHTWCe (ORCPT ); Wed, 20 Aug 2014 18:02:34 -0400 In-Reply-To: Sender: linux-fsdevel-owner@vger.kernel.org List-ID: Ashish Sangwan wrote: > On Wed, Aug 20, 2014 at 2:03 PM, Howard Chu wrote: >> Luk=C3=A1=C5=A1 Czerner wrote: >>> >>> On Wed, 20 Aug 2014, Howard Chu wrote: >>> >>>> Date: Wed, 20 Aug 2014 00:00:56 -0700 >>>> From: Howard Chu >>>> To: Luk=C3=A1=C5=A1 Czerner >>>> Cc: linux-fsdevel >>>> Subject: Re: truncate head of file? >>>> >>>> Luk=C3=A1=C5=A1 Czerner wrote: >>>>> >>>>> On Tue, 19 Aug 2014, Howard Chu wrote: >>>>> >>>>>> Date: Tue, 19 Aug 2014 22:45:16 -0700 >>>>>> From: Howard Chu >>>>>> To: linux-fsdevel >>>>>> Subject: truncate head of file? >>>>>> >>>>>> Was thinking it would be very handy to have a truncate() variant= that >>>>>> deletes >>>>>> pages from the head of a file. This could be leveraged to make l= ogfiles >>>>>> easier >>>>>> to maintain, as an example. Anyone else interested, think this w= ould be >>>>>> nice >>>>>> to have? >>>>>> >>>>>> (Note - not the same as just punching holes in the beginning of = the >>>>>> file - >>>>>> we >>>>>> want the beginning of the file to advance as well, past the dele= ted >>>>>> pages.) >>>>> >>>>> >>>>> I am not really sure I understand the behaviour you'd like to see= =2E >>>>> Can you please explain the behaviour including more concrete use >>>>> case ? >>>> >>>> >>>> For example - we have a logfile (opened O_APPEND) that grows >>>> continuously. We >>>> want to delete some old log info from the head of the file. We cou= ld use >>>> "hole >>>> punching" to cause a specific range of data to be freed, but that = just >>>> leaves >>>> a sparse file. If we were to cat this file the read() would have t= o >>>> advance >>>> thru all of that empty space before arriving at actual log data. W= e want >>>> both >>>> the data to be freed and for the logical beginning of the file to = be >>>> moved >>>> forward, to match the location of where the remaining data begins. >>>> >>>> Freeing the space would be simplest if we just deallocate X pages = from >>>> the >>>> file, and then the beginning of the file becomes the beginning of = the >>>> first >>>> remaining page of the file. >>> >>> >>> Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE i= s >>> for. It has been merged into mainline with commit v3.14-rc1-1-g00f5= e61 >> >> >> Ah, thanks for the tip. >> >> >>> However it will not work with O_APPEND nor does any other fallocate >>> flag except pure fallocate, so not even punch hole would have >>> worked. >> >> >> Hm, that's a little bit inconvenient. What if the process calling > There seems to be some confusion between per fd flag O_APPEND and > inode flag S_APPEND. > Collapse range can work well with O_APPEND flag but it will not work > if S_APPEND is set. Ah ok, that's much better. >> fallocate() is different from the one appending to the log (i.e., us= ing two >> separate file descriptors)? Also, when a range is collapsed, are all >> processes with descriptors opened on the file updated - are their se= ek >> positions correctly decremented? I dug up the patches and didn't see= any >> code to handle this but may have missed it. > You are right. Currently seek positions are not updated. > But this will not be of any problem (at least not on XFS) for your us= e > case if file is open with O_APPEND. > Refer to this discussion: > https://lkml.org/lkml/2014/2/26/701 Thanks, looks good. I find it somewhat surprising that such a useful feature was implemente= d for=20 the benefit of video editing, when it has massive applicability in data= bases=20 (particularly for simplified truncation of write-ahead logs, as well as= for=20 append-only database designs). This is such an obviously beneficial fea= ture=20 but I haven't seen it talked up very much. Of course, being so specific= to=20 very recent Linux releases will certainly impede its adoption. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel= " in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html