Stressing GC

Stressing GC

[Luis Useche]

Hi Guys,

I was doing some experiments on nilfs2 to stress the garbage collection
with different file system usage. Unfortunately, it was unable to pass the
first test since nilfs reports no space available eventhough there are no
files in the file system! I guess the GC is not collecting space fast
enough and the file system ends up with no space available.

Do you have any workaround to this problem?

Best,
-- 
Luis Useche <luis-43oy+cWvsXk3uPMLIKxrzw@public.gmane.org>
Ph.D. Student
Florida International University

Re: Stressing GC

[Dave]

On 06/11/09 01:08, Luis Useche wrote:
> Hi Guys,
> 
> I was doing some experiments on nilfs2 to stress the garbage collection
> with different file system usage. Unfortunately, it was unable to pass the
> first test since nilfs reports no space available eventhough there are no
> files in the file system! I guess the GC is not collecting space fast
> enough and the file system ends up with no space available.
> 
> Do you have any workaround to this problem?

I don't understand. There is no 'problem' to workaround. This is by 
design. You're right, the GC has to reclaim your deleted files before it 
release the space to the filesystem. If you remove all your checkpoints 
and wait for your 'protection_period' interval you should see you space 
reappear. The whole idea or nilfs is that you could do a 'rm -rf *' and 
be left with no files, but still recover all your files from a 
checkpoint that happened for you automatically a few seconds ago.

D

Re: Stressing GC

[Luis Useche]

On Thu, Jun 11, 2009 at 08:17:51AM +0400, Dave wrote:
> On 06/11/09 01:08, Luis Useche wrote:
> > Hi Guys,
> > 
> > I was doing some experiments on nilfs2 to stress the garbage collection
> > with different file system usage. Unfortunately, it was unable to pass the
> > first test since nilfs reports no space available eventhough there are no
> > files in the file system! I guess the GC is not collecting space fast
> > enough and the file system ends up with no space available.
> > 
> > Do you have any workaround to this problem?
> 
> I don't understand. There is no 'problem' to workaround. This is by 
> design. You're right, the GC has to reclaim your deleted files before it 
> release the space to the filesystem. If you remove all your checkpoints 
> and wait for your 'protection_period' interval you should see you space 
> reappear. The whole idea or nilfs is that you could do a 'rm -rf *' and 
> be left with no files, but still recover all your files from a 
> checkpoint that happened for you automatically a few seconds ago.

Thanks for your response.

I still think is a problem. IMHO, it is not correct to report no available
space when you don't have any file in the file system. I would think that
in this situations, instead of reporting to the application "no space
left", the correct action is to throttle the GC (ignoring the
configurations) so that enough rooms is made for the incoming requests
ASAP. I guess this is object of debate and I see points in favor and
against. At the end, this probably depends on the user: (1) protect old
data or (2) have the application running as long as some space is
available.

In any case, in my experiment, I want to stress exactly this situation.
LFS tends to decrease its performance dramatically when it is running out
of space. I want to see how fast the performance is decreased when the
usage of the file system is incremented.

With the current GC implementation, I am unable to do my experiment. I set
"protection_period 0" but still have the problem. Besides, this is
probably not the right solution either since the GC can do unnecessary work
that can underestimate the potential of nilfs. I need the first option (1)
from the first paragraph above.

Are there any workaround I can use to make this work.

Thanks in advance,
-- 
Luis Useche <luis-43oy+cWvsXk3uPMLIKxrzw@public.gmane.org>
Ph.D. Student
Florida International University

Re: Stressing GC

[Reinoud Zandijk]

Dear folks,

On Thu, Jun 11, 2009 at 11:39:50AM -0400, Luis Useche wrote:
> On Thu, Jun 11, 2009 at 08:17:51AM +0400, Dave wrote:
> > On 06/11/09 01:08, Luis Useche wrote:
> > > I was doing some experiments on nilfs2 to stress the garbage collection
> > > with different file system usage. Unfortunately, it was unable to pass the
> > > first test since nilfs reports no space available eventhough there are no
> > > files in the file system! I guess the GC is not collecting space fast
> > > enough and the file system ends up with no space available.
> > > 
> > > Do you have any workaround to this problem?
>
> With the current GC implementation, I am unable to do my experiment. I set
> "protection_period 0" but still have the problem. Besides, this is
> probably not the right solution either since the GC can do unnecessary work
> that can underestimate the potential of nilfs. I need the first option (1)
> from the first paragraph above.
> 
> Are there any workaround I can use to make this work.

idealiter there should be an analog to VMS's purge command, say purge_nilfs or
`nilfs purge /mnt' that flushes all data not protected by a snapshot or the
last checkpoint.

Could this be something?

Another more blunt method is say newfs'ing a disc, create a snapshot sometime,
mess around with it and later revert back to the snapshot.

With regards,
Reinoud

Re: Stressing GC

[Jérôme Poulin]

[-- Attachment #1.1: Type: text/plain, Size: 1417 bytes --]

>
> On Thu, Jun 11, 2009 at 11:39:50AM -0400, Luis Useche wrote:
> With the current GC implementation, I am unable to do my experiment. I set
> "protection_period 0" but still have the problem. Besides, this is
> probably not the right solution either since the GC can do unnecessary work
> that can underestimate the potential of nilfs. I need the first option (1)
> from the first paragraph above.
>

For this situation, would it be possible instead, to have to GC collect all
the space that can be freed, have the filesystem informed of what blocks can
be freed anytime (like a memory cache which can be flushed anytime), and
have the FS give the free space information for what it calculated as
removable? So with this we could have checkpoint for longuer than the
protection_period and still have the free space available for thing the
filesystem can just overwrite. I really don't know how it is coded as I'm
just a user for now, but that would be even greater.

On Thu, Jun 11, 2009 at 11:20 AM, Reinoud Zandijk <reinoud-S783fYmB3Ccdnm+yROfE0A@public.gmane.org>wrote:

>
> idealiter there should be an analog to VMS's purge command, say purge_nilfs
> or
> `nilfs purge /mnt' that flushes all data not protected by a snapshot or the
> last checkpoint.


That would be a great alternative thinking of the current state of the
project, if it's possible to tell the GC to flush all non protected data
'right now'.

[-- Attachment #1.2: Type: text/html, Size: 2005 bytes --]

[-- Attachment #2: Type: text/plain, Size: 158 bytes --]

_______________________________________________
users mailing list
users-JrjvKiOkagjYtjvyW6yDsg@public.gmane.org
https://www.nilfs.org/mailman/listinfo/users

Re: Stressing GC

[Luis Useche]

On Thu, Jun 11, 2009 at 12:51:17PM -0400, Jérôme Poulin wrote:
> >
> > On Thu, Jun 11, 2009 at 11:39:50AM -0400, Luis Useche wrote:
> > With the current GC implementation, I am unable to do my experiment. I set
> > "protection_period 0" but still have the problem. Besides, this is
> > probably not the right solution either since the GC can do unnecessary work
> > that can underestimate the potential of nilfs. I need the first option (1)
> > from the first paragraph above.
> >
> 
> For this situation, would it be possible instead, to have to GC collect all
> the space that can be freed, have the filesystem informed of what blocks can
> be freed anytime (like a memory cache which can be flushed anytime), and
> have the FS give the free space information for what it calculated as
> removable? So with this we could have checkpoint for longuer than the
> protection_period and still have the free space available for thing the
> filesystem can just overwrite. I really don't know how it is coded as I'm
> just a user for now, but that would be even greater.

I am in favor of this solution. Ideally, if the user does not want to
protect any data (like in my case), the filesystem should not report space
shortage when plenty of space is available to overwrite. Instead, the GC
must start working right away to make enough room for the incoming writes.

-- 
Luis Useche <luis-43oy+cWvsXk3uPMLIKxrzw@public.gmane.org>
Ph.D. Student
Florida International University