NILFS2 under stress

NILFS2 under stress

[Zavi Zavi]

Hi,

I've been experimenting with NILFS2 on a 8GB USB stick.
I repeatedly created and deleted a 128MB file. After about 60 attempts, the
disk got 100% full, and would not respond anymore.

So I tried repeating this setup after changing the GC settings to be much
more agressive GC policy
protection_period       0
min_clean_segments      10%
max_clean_segments      20%
clean_check_interval    1
nsegments_per_clean     20
mc_nsegments_per_clean  40
cleaning_interval       1
mc_cleaning_interval    1
retry_interval          3

This time I was able to do the above even for 80 times, and indeed saw that
the free space on the device was periodically decreased

So I continued with filling up the file system to about half of its
capacity (3.5GB) with large 128MB files.
I then tried running a small 60 seconds fileserver benchmark of FileBench,
which uses 1.5GB.
I was able to run it twice, but then the system got 100% full again, and
non-responsive.
I tried running nilfs_cleanerd manually, tried remounting. nothing worked.

Is NILFS known to be sensitive to stress tests? Is this expected?
Or am I doing something wrong?

Thank you
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NILFS2 under stress

[Christian Smith]

On Fri, Jun 15, 2012 at 12:07:23PM +0300, Zavi Zavi wrote:
> Hi,
> 
> I've been experimenting with NILFS2 on a 8GB USB stick.
> I repeatedly created and deleted a 128MB file. After about 60 attempts, the
> disk got 100% full, and would not respond anymore.
> 
> So I tried repeating this setup after changing the GC settings to be much
> more agressive GC policy
> protection_period       0
> min_clean_segments      10%
> max_clean_segments      20%
> clean_check_interval    1
> nsegments_per_clean     20
> mc_nsegments_per_clean  40
> cleaning_interval       1
> mc_cleaning_interval    1
> retry_interval          3
> 
> This time I was able to do the above even for 80 times, and indeed saw that
> the free space on the device was periodically decreased
> 
> So I continued with filling up the file system to about half of its
> capacity (3.5GB) with large 128MB files.
> I then tried running a small 60 seconds fileserver benchmark of FileBench,
> which uses 1.5GB.
> I was able to run it twice, but then the system got 100% full again, and
> non-responsive.
> I tried running nilfs_cleanerd manually, tried remounting. nothing worked.
> 
> Is NILFS known to be sensitive to stress tests? Is this expected?
> Or am I doing something wrong?

NILFS is certainly sensitive to running out of space. I think NILFS
tries to keep some segments free, for emergency situations. To clean,
there must be some free segments to clean into.

I've also noticed if the cleaner encounters corrupted state, all bets
are off, as it'll never get past the corrupted block and so will
never clean anymore. Eventually the disk will fill and a dump/restore is
the only option. But I've only seen this after power failures, when
running on cheap SSD (I only run it on cheap SSD, as they benefit the
most from the sequential writes).

Basically, if you're running from an 8GB FLASH stick, the machine is not
going to be doing heavy IO. If you are doing heavy IO, then you'll have
a better IO system, which will perform better with a traditional non-COW
filesystem.

The other thing to bear in mind is that setting the protection period
so small will result in the clearner cleaning constantly if you disk
fills beyond the thresholds. This is not healthy, especially for such
small cheap SSD, which will be using the lowest grade FLASH.

Finally, the biggest problem I find on such small devices is doing 
upgrades. Following debian testing, you can easily require 1G+ of upgrades
in the course of a week, and it is difficult to do those all at once if
NILFS is short of space. For some reason, dpkg must use lots of temporary
files, which burn through NILFS disk space.

In summary, I think NILFS is fantastic in short bursts of heavy pressure
so long as space isn't exhausted.

Christian
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: NILFS2 under stress

[Tim Bannister]

> In summary, I think NILFS is fantastic in short bursts of heavy pressure
> so long as space isn't exhausted.

OK, but if exhausting the space leads to unrecoverable / difficult-to-fix data loss then it's a poor choice of filesystem for uses where a full filesystem is unlikely but remains a possibility (eg a mail spool). If /var/tmp shouldn't be on NILFS then probably /var shouldn't be on NILFS either.

I recognise that NILFS2 is not yet described as ready for production use.

-- 
Tim Bannister
IT Services
The University of Manchester

e: Tim.Bannister-m7QheJJv02N4oUbgFh0ZNeyAPIwwu4QO@public.gmane.org
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NILFS2 under stress

[Christian Smith]

On Sat, Jun 16, 2012 at 11:49:55AM +0000, Tim Bannister wrote:
> > In summary, I think NILFS is fantastic in short bursts of heavy pressure
> > so long as space isn't exhausted.
> 
> OK, but if exhausting the space leads to unrecoverable / difficult-to-fix data loss then it's a poor choice of filesystem for uses where a full filesystem is unlikely but remains a possibility (eg a mail spool). If /var/tmp shouldn't be on NILFS then probably /var shouldn't be on NILFS either.


Which is a shame, because short synchronous file writes are an ideal use for NILFS. Perhaps one area where NILFS could be improved is in handling of disk full errors, for example, not allowing complete exhaustion of space unless a cleaner is running.

It doesn't help that I run my NILFS systems on small media, such as my Acer Aspire One netbook with crappy 8GB SSD, because it's the only filesystem that makes such disks usable performance wise. The biggest media I use NILFS with is a 40GB tablet the kids use for playing with, and I haven't come close to filling or killing that.


> 
> I recognise that NILFS2 is not yet described as ready for production use.
> 

Exactly. I don't store on it anything I can't easily reproduce. That said, I've never had a filesystem I couldn't recover data from, so recovery has been easy so far.

Christian
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: NILFS2 under stress

[Tim Bannister]

> On Sat, Jun 16, 2012 at 11:49:55AM +0000, Tim Bannister wrote:
> > > In summary, I think NILFS is fantastic in short bursts of heavy pressure
> > > so long as space isn't exhausted.
>
> > OK, but if exhausting the space leads to unrecoverable / difficult-to-fix 
> > data loss then it's a poor choice of filesystem for uses where a full 
> > filesystem is unlikely but remains a possibility (eg a mail spool). If 
> > /var/tmp shouldn't be on NILFS then probably /var shouldn't be on NILFS 
> > either.
…
> Perhaps one area where NILFS could be improved is in handling of disk full
> errors, for example, not allowing complete exhaustion of space unless a
> cleaner is running.

Lots of filesystems reserve a certain amount of space for root. I would extend 
this for NILFS2 to reserve some space for the cleaner daemon (perhaps with 
an ioctl that can be used by the daemon to bypass the restriction).

Easier said than coded ;-)

-- 
Tim Bannister
IT Services
The University of Manchester

e: Tim.Bannister-m7QheJJv02N4oUbgFh0ZNeyAPIwwu4QO@public.gmane.org
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NILFS2 under stress

[Ryusuke Konishi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: Text/Plain; charset="windows-1254", Size: 2069 bytes --]

On Sun, 17 Jun 2012 15:07:43 +0000, Tim Bannister wrote:
> > On Sat, Jun 16, 2012 at 11:49:55AM +0000, Tim Bannister wrote:
> > > > In summary, I think NILFS is fantastic in short bursts of heavy pressure
> > > > so long as space isn't exhausted.
> >
> > > OK, but if exhausting the space leads to unrecoverable / difficult-to-fix 
> > > data loss then it's a poor choice of filesystem for uses where a full 
> > > filesystem is unlikely but remains a possibility (eg a mail spool). If 
> > > /var/tmp shouldn't be on NILFS then probably /var shouldn't be on NILFS 
> > > either.
> …
> > Perhaps one area where NILFS could be improved is in handling of disk full
> > errors, for example, not allowing complete exhaustion of space unless a
> > cleaner is running.
> 
> Lots of filesystems reserve a certain amount of space for root. I would extend 
> this for NILFS2 to reserve some space for the cleaner daemon (perhaps with 
> an ioctl that can be used by the daemon to bypass the restriction).

NILFS2 already has the reserved space for cleaner daemon, so it works
at disk full.

The problem seems that the kernel code of NILFS2 returns a disk full
error to userland programs even if the volume has room for making free
space with GC.  Well-behaved filesystem should not do so.

I think this is one of drawbacks of the current NILFS2 design.
However, waiting for disk cleanup would be so costly without efficient
GC algorithm.  Actually, I feel improvement of the GC algorithm is a
priority issue for production use.


Regards,
Ryusuke Konishi


> Easier said than coded ;-)
> 
> -- 
> Tim Bannister
> IT Services
> The University of Manchester
> 
> e: Tim.Bannister@manchester.ac.uk--
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±ž)_²)í…æèw*\x1fjg¬±¨\x1e¶‰šŽŠÝ¢j.ïÛ°\½½MŽúgjÌæa×\x02››–' ™©Þ¢¸\f¢·¦j:+v‰¨ŠwèjØm¶Ÿÿ¾\a«‘êçzZ+ƒùšŽŠÝ¢j"ú!¶i

Re: NILFS2 under stress

[Ivan Shmakov]

>>>>> Zavi Zavi writes:

 > I've been experimenting with NILFS2 on a 8GB USB stick.  I repeatedly
 > created and deleted a 128MB file. After about 60 attempts, the disk
 > got 100% full, and would not respond anymore.

	I've tried to install a Debian Wheezy instance onto an 8 GiB
	logical volume via debootstrap(8) from Debian Squeeze (Linux
	2.6.32-5-amd64, nilfs-tools 2.0.18-2) and have witnessed pretty
	much the same behavior.

 > So I tried repeating this setup after changing the GC settings to be
 > much more agressive GC policy

 > protection_period       0

[…]

	I've restarted nilfs_cleanerd(8) with the protection period set
	to only 60 seconds (see below; the rest of the configuration is
	as per the Debian defaults), and the free space began to rise,
	albeit somewhat slowly (around a few MiB's per second), until it
	reached 1.7 GiB, when it's stopped.

	(I guess that it's quite normal, though I'm yet to become
	familiar with NILFS2.)

# nilfs_cleanerd -p 60 \
      /dev/mapper/vgXXX-host--2012--06--17.chroot \
      /srv/chroot/2012-06-17 

-- 
FSF associate member #7257

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html