delayed file deallocation?

delayed file deallocation?

[Isaac Claymore]

Hi list,

We're using reiser 3.6 as workhorse fs beneath Samba in our NAS setup.
Under our workload, it's common that tens of clients (often around 20)  
are doing real-time video editing simultaneously. We've been suffering a
scenario that when one client comes and deletes some large file while
many others are busy editing, the many others will experience noticeable
period of IO stalling, usually of about 3 seconds. And, since they're
doing some kind of realtime editing, this stall is unacceptable.

I'm wondering that whether the file deallocation could be delayed
somewhat, i.e. when a file is unlink()'ed by Samba, fs driver just mark
the file to-be-deallocated when its reference count reaches zero. Then,
the fs driver, at some later time, either triggered by user space or when 
lack of free space, actually commits the deallocation. Since we have
plenty of free disk space, this deallocation scheme does make sense.

It's possible to do this in user space, say, by modifying Samba so that
it creates an extra secret link to each file it creates, and write
another program that routinely checks the secret links, at proper times,
for those inodes with inode ref count of 1, and unlinks them.

But for some reason, we do not want to touch Samba code, then it looks that
this can only be done at fs level.

How can I achieve this with Reiser? Or, if you've some alternatives,
please kindly enlighten me.

Thanks for any hint or suggestion.


-- 

Regards, Isaac
()  ascii ribbon campaign - against html e-mail
/\                        - against microsoft attachments

Re: delayed file deallocation?

[Hans Reiser]

On Monday 15 March 2004 10:44, Isaac Claymore wrote:
> Hi list,
>
> We're using reiser 3.6 as workhorse fs beneath Samba in our NAS setup.
> Under our workload, it's common that tens of clients (often around 20)
> are doing real-time video editing simultaneously. We've been suffering a
> scenario that when one client comes and deletes some large file while
> many others are busy editing, the many others will experience noticeable
> period of IO stalling, usually of about 3 seconds. And, since they're
> doing some kind of realtime editing, this stall is unacceptable.
>
> I'm wondering that whether the file deallocation could be delayed
> somewhat, i.e. when a file is unlink()'ed by Samba, fs driver just mark
> the file to-be-deallocated when its reference count reaches zero. Then,
> the fs driver, at some later time, either triggered by user space or when
> lack of free space, actually commits the deallocation. Since we have
> plenty of free disk space, this deallocation scheme does make sense.
>
> It's possible to do this in user space, say, by modifying Samba so that
> it creates an extra secret link to each file it creates, and write
> another program that routinely checks the secret links, at proper times,
> for those inodes with inode ref count of 1, and unlinks them.
>
> But for some reason, we do not want to touch Samba code, then it looks that
> this can only be done at fs level.
>
> How can I achieve this with Reiser? Or, if you've some alternatives,
> please kindly enlighten me.
>
> Thanks for any hint or suggestion.
This is definitely fixable, but someone would need to sponsor the fix as we 
are low on funds.  Probably it would not be all that expensive to fix.  
Chris, forgive my memory, did any of your recent patches address this?

Re: delayed file deallocation?

[Vladimir Saveliev]

Hello

On Mon, 2004-03-15 at 12:57, Hans Reiser wrote:
> On Monday 15 March 2004 10:44, Isaac Claymore wrote:
> > Hi list,
> >
> > We're using reiser 3.6 as workhorse fs beneath Samba in our NAS setup.
> > Under our workload, it's common that tens of clients (often around 20)
> > are doing real-time video editing simultaneously. We've been suffering a
> > scenario that when one client comes and deletes some large file while
> > many others are busy editing, the many others will experience noticeable
> > period of IO stalling, usually of about 3 seconds. And, since they're
> > doing some kind of realtime editing, this stall is unacceptable.
> >
> > I'm wondering that whether the file deallocation could be delayed
> > somewhat, i.e. when a file is unlink()'ed by Samba, fs driver just mark
> > the file to-be-deallocated when its reference count reaches zero. Then,
> > the fs driver, at some later time, either triggered by user space or when
> > lack of free space, actually commits the deallocation. Since we have
> > plenty of free disk space, this deallocation scheme does make sense.
> >
> > It's possible to do this in user space, say, by modifying Samba so that
> > it creates an extra secret link to each file it creates, and write
> > another program that routinely checks the secret links, at proper times,
> > for those inodes with inode ref count of 1, and unlinks them.
> >
> > But for some reason, we do not want to touch Samba code, then it looks that
> > this can only be done at fs level.
> >

You do not need to touch samba. Instead you may write your our unlink
function and have it called instead of libc's one.


> > How can I achieve this with Reiser? Or, if you've some alternatives,
> > please kindly enlighten me.
> >
> > Thanks for any hint or suggestion.
> This is definitely fixable, but someone would need to sponsor the fix as we 
> are low on funds.  Probably it would not be all that expensive to fix.  
> Chris, forgive my memory, did any of your recent patches address this?
> 
> 

Re: delayed file deallocation?

[Chris Mason]

On Mon, 2004-03-15 at 04:57, Hans Reiser wrote:
> On Monday 15 March 2004 10:44, Isaac Claymore wrote:
> > Hi list,
> >
> > We're using reiser 3.6 as workhorse fs beneath Samba in our NAS setup.
> > Under our workload, it's common that tens of clients (often around 20)
> > are doing real-time video editing simultaneously. We've been suffering a
> > scenario that when one client comes and deletes some large file while
> > many others are busy editing, the many others will experience noticeable
> > period of IO stalling, usually of about 3 seconds. And, since they're
> > doing some kind of realtime editing, this stall is unacceptable.
> >
> > I'm wondering that whether the file deallocation could be delayed
> > somewhat, i.e. when a file is unlink()'ed by Samba, fs driver just mark
> > the file to-be-deallocated when its reference count reaches zero. Then,
> > the fs driver, at some later time, either triggered by user space or when
> > lack of free space, actually commits the deallocation. Since we have
> > plenty of free disk space, this deallocation scheme does make sense.
> >
> > It's possible to do this in user space, say, by modifying Samba so that
> > it creates an extra secret link to each file it creates, and write
> > another program that routinely checks the secret links, at proper times,
> > for those inodes with inode ref count of 1, and unlinks them.
> >
> > But for some reason, we do not want to touch Samba code, then it looks that
> > this can only be done at fs level.
> >
> > How can I achieve this with Reiser? Or, if you've some alternatives,
> > please kindly enlighten me.
> >
> > Thanks for any hint or suggestion.
> This is definitely fixable, but someone would need to sponsor the fix as we 
> are low on funds.  Probably it would not be all that expensive to fix.  
> Chris, forgive my memory, did any of your recent patches address this?
> 
> 
There are two different sides to this problem.

1) make the unlink happen at a much later time.  This isn't coded
2) get rid of the stalls while unlinking.  I think I've got this largely
fixed.  The patch set sent to Andrew on Friday had both of the most
important fixes there for 2.6.  The 2.4 data logging patch set has some
fixes here, there are a few more in Andrea's -aa kernel series.

Intel, suse and a few others are doing latency tests on 2.6 now, things
should generally improve over the next few weeks.

-chris

Re: delayed file deallocation?

[Hans Reiser]

Chris Mason wrote:

>
>
>Intel, suse and a few others are doing latency tests on 2.6 now, things
>should generally improve over the next few weeks.
>
>-chris
>
>
>
>
>  
>

what is Intel's interest?

-- 
Hans

Re: delayed file deallocation?

[Chris Mason]

On Mon, 2004-03-15 at 09:07, Hans Reiser wrote:
> Chris Mason wrote:
> 
> >
> >
> >Intel, suse and a few others are doing latency tests on 2.6 now, things
> >should generally improve over the next few weeks.

> what is Intel's interest?

They are not targeting reiserfs, but the kernel as a whole.  I'm
assuming their interest is to make sure linux works on big hardware so
they can sell more of it ;-)

Look for mail on l-k from Kenneth Chen to get a general idea.

-chris

Re: delayed file deallocation?

[Isaac Claymore]

On Mon, Mar 15, 2004 at 01:28:34PM +0300, Vladimir Saveliev wrote:
> Hello
> 
> On Mon, 2004-03-15 at 12:57, Hans Reiser wrote:
> > On Monday 15 March 2004 10:44, Isaac Claymore wrote:
> > > Hi list,
> > >
> > > We're using reiser 3.6 as workhorse fs beneath Samba in our NAS setup.
> > > Under our workload, it's common that tens of clients (often around 20)
> > > are doing real-time video editing simultaneously. We've been suffering a
> > > scenario that when one client comes and deletes some large file while
> > > many others are busy editing, the many others will experience noticeable
> > > period of IO stalling, usually of about 3 seconds. And, since they're
> > > doing some kind of realtime editing, this stall is unacceptable.
> > >
> > > I'm wondering that whether the file deallocation could be delayed
> > > somewhat, i.e. when a file is unlink()'ed by Samba, fs driver just mark
> > > the file to-be-deallocated when its reference count reaches zero. Then,
> > > the fs driver, at some later time, either triggered by user space or when
> > > lack of free space, actually commits the deallocation. Since we have
> > > plenty of free disk space, this deallocation scheme does make sense.
> > >
> > > It's possible to do this in user space, say, by modifying Samba so that
> > > it creates an extra secret link to each file it creates, and write
> > > another program that routinely checks the secret links, at proper times,
> > > for those inodes with inode ref count of 1, and unlinks them.
> > >
> > > But for some reason, we do not want to touch Samba code, then it looks that
> > > this can only be done at fs level.
> > >
> 
> You do not need to touch samba. Instead you may write your our unlink
> function and have it called instead of libc's one.
> 

Thanks very much for the hint, and I've found a library who does exactly
this:

http://m-arriaga.net/software/libtrash/

> 
> > > How can I achieve this with Reiser? Or, if you've some alternatives,
> > > please kindly enlighten me.
> > >
> > > Thanks for any hint or suggestion.
> > This is definitely fixable, but someone would need to sponsor the fix as we 
> > are low on funds.  Probably it would not be all that expensive to fix.  
> > Chris, forgive my memory, did any of your recent patches address this?
> > 
> > 

-- 

Regards, Isaac
()  ascii ribbon campaign - against html e-mail
/\                        - against microsoft attachments