[linux-lvm] snapshots of busy ext2 file system corrupt

[linux-lvm] snapshots of busy ext2 file system corrupt

[Urs Thuermann]

I tried creating a snapshot of a ext2 file system on a LV.  I expected
a fsck on the snapshot device to give no errors, i.e. a consistent
file system.  This is ok, as long as there is no (or little) load on
the file system when creating the snapshot.

However, on a busy file system, the created snapshot has lots of
errors found by fsck.

Is it this what the VFS lock patch is for?  If so, why hasn't it been
integrated into the standard kernel?  Has it flaws in it?  Maybe
performance wise?


urs

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Andreas Dilger]

On Feb 26, 2002  14:54 +0100, Urs Thuermann wrote:
> I tried creating a snapshot of a ext2 file system on a LV.  I expected
> a fsck on the snapshot device to give no errors, i.e. a consistent
> file system.  This is ok, as long as there is no (or little) load on
> the file system when creating the snapshot.

Yes.

> However, on a busy file system, the created snapshot has lots of
> errors found by fsck.
> 
> Is it this what the VFS lock patch is for?  If so, why hasn't it been
> integrated into the standard kernel?  Has it flaws in it?  Maybe
> performance wise?

The VFS lock patch only applies to journaled filesystems.  There has not
been any attempt to include support for sync_super_lockfs for ext2.  I
_think_ that if you tried to mount the ext2 snapshot, it will still
work, even though it is inconsistent.  Obviously it would be better to
fix it so that ext2 snapshots are always consistent.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Chris Mason]

On Tuesday, February 26, 2002 09:59:19 AM -0700 Andreas Dilger <adilger@turbolabs.com> wrote:

> On Feb 26, 2002  14:54 +0100, Urs Thuermann wrote:
>> I tried creating a snapshot of a ext2 file system on a LV.  I expected
>> a fsck on the snapshot device to give no errors, i.e. a consistent
>> file system.  This is ok, as long as there is no (or little) load on
>> the file system when creating the snapshot.
> 
> Yes.
> 
>> However, on a busy file system, the created snapshot has lots of
>> errors found by fsck.
>> 
>> Is it this what the VFS lock patch is for?  If so, why hasn't it been
>> integrated into the standard kernel?  Has it flaws in it?  Maybe
>> performance wise?
> 
> The VFS lock patch only applies to journaled filesystems.  There has not
> been any attempt to include support for sync_super_lockfs for ext2.  I
> _think_ that if you tried to mount the ext2 snapshot, it will still
> work, even though it is inconsistent.  Obviously it would be better to
> fix it so that ext2 snapshots are always consistent.

Yes, its very easy for the journaled filesystems (except XFS ;-) to 
do the snapshot locking, since snapshot locking is really just grab a 
lock on the log, finish transactions, let LVM do its thing, then release
the lock.

For ext2, the best we could do would probably be a hack based on 
locking the super.  As far as I know, nobody is planning on this,
especially given how easy it is to go from ext2->ext3.

-chris

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Urs Thuermann]

Chris Mason <mason@suse.com> writes:

> For ext2, the best we could do would probably be a hack based on 
> locking the super.  As far as I know, nobody is planning on this,

:-(

> especially given how easy it is to go from ext2->ext3.

Some performance tests I made with ext2, ext3, and reiserfs convinced
me to stay with ext2.  ext3 was significantly slower than ext2 on some
operations (and reiserfs even worse than that).


urs

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Chris Mason]

On Tuesday, February 26, 2002 08:28:40 PM +0100 Urs Thuermann <urs@isnogud.escape.de> wrote:

> Chris Mason <mason@suse.com> writes:
> 
>> For ext2, the best we could do would probably be a hack based on 
>> locking the super.  As far as I know, nobody is planning on this,
> 
> :-(
> 
>> especially given how easy it is to go from ext2->ext3.
> 
> Some performance tests I made with ext2, ext3, and reiserfs convinced
> me to stay with ext2.  ext3 was significantly slower than ext2 on some
> operations (and reiserfs even worse than that).

You trade speed for safety sometimes, but you can lower the performance
hit by mounting ext3 with -o data=writeback, and help both filesystems
by playing with bdflush values (increment the metadata timeout up from 5
seconds).

Other tuning depends on your app.

-chris

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Andreas Dilger]

On Feb 26, 2002  15:58 -0500, Chris Mason wrote:
> On Tuesday, February 26, 2002 08:28:40 PM +0100
> Urs Thuermann <urs@isnogud.escape.de> wrote:
> > Chris Mason <mason@suse.com> writes:
> >> For ext2, the best we could do would probably be a hack based on 
> >> locking the super.  As far as I know, nobody is planning on this,

It probably wouldn't be hard to do, just create a function
"ext2_write_super_lockfs()" which is a call to lock an ext2-specific
superblock semaphore and "ext2_unlockfs()" which is a call to
unlock the same semaphore.  You would then change all instances of
"lock_super()" in the ext2 code to be ext2_lock_super() which will
lock this semaphore in addition to the normal lock_super() call.

You can't just call lock_super() directly, because the write_super_lockfs()
call itself calls this, and you would instantly deadlock.

> > Some performance tests I made with ext2, ext3, and reiserfs convinced
> > me to stay with ext2.  ext3 was significantly slower than ext2 on some
> > operations (and reiserfs even worse than that).
> 
> You trade speed for safety sometimes, but you can lower the performance
> hit by mounting ext3 with -o data=writeback, and help both filesystems
> by playing with bdflush values (increment the metadata timeout up from 5
> seconds).

Well, in many cases, data=ordered is actually faster than data=writeback
because you get all of your journal I/O finished before you start writing
to the FS.  In some cases (e.g. mail spool, NFS) it is faster to have
data=journal for the same reason (because sync I/O is very fast when it
is written to the journal).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Marc MERLIN]

On Tue, Feb 26, 2002 at 02:27:31PM -0700, Andreas Dilger wrote:
> Well, in many cases, data=ordered is actually faster than data=writeback
> because you get all of your journal I/O finished before you start writing
> to the FS.  In some cases (e.g. mail spool, NFS) it is faster to have
> data=journal for the same reason (because sync I/O is very fast when it
> is written to the journal).

I had heard about data=journal on mail spools, but does anyone have a
pointer with more performance comparisons between all 3?

Thanks,
Marc
-- 
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
  
Home page: http://marc.merlins.org/   |   Finger marc_f@merlins.org for PGP key

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Andreas Dilger]

On Feb 26, 2002  14:32 -0800, Marc MERLIN wrote:
> On Tue, Feb 26, 2002 at 02:27:31PM -0700, Andreas Dilger wrote:
> > Well, in many cases, data=ordered is actually faster than data=writeback
> > because you get all of your journal I/O finished before you start writing
> > to the FS.  In some cases (e.g. mail spool, NFS) it is faster to have
> > data=journal for the same reason (because sync I/O is very fast when it
> > is written to the journal).
> 
> I had heard about data=journal on mail spools, but does anyone have a
> pointer with more performance comparisons between all 3?

There was some discussion about it on one of (linux-kernel, ext3-users,
ext2-devel), but since I read all of them, I don't know which one for
sure.

I found:

http://bruce-guenter.dyndns.org/benchmarking/
http://www-106.ibm.com/developerworks/linux/library/l-fs8.html?dwzone=linux

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Urs Thuermann]

Andreas Dilger <adilger@turbolabs.com> writes:

> I _think_ that if you tried to mount the ext2 snapshot, it will
> still work, even though it is inconsistent.

I think that wouldn't help me much, if I want to dump(8) the file
system for backups.  AFAIK, dump(8) opens the device file,
i.e. /dev/vg0/snap which has an inconsistent image.

> Obviously it would be better to fix it so that ext2 snapshots are
> always consistent.

ACK!


urs

Re: [linux-lvm] snapshots of busy ext2 file system corrupt

[Anselm Kruis]

On 26 Feb 2002, Urs Thuermann wrote:

> I tried creating a snapshot of a ext2 file system on a LV.  I expected
> a fsck on the snapshot device to give no errors, i.e. a consistent
> file system.  This is ok, as long as there is no (or little) load on
> the file system when creating the snapshot.
> 
> However, on a busy file system, the created snapshot has lots of
> errors found by fsck.
> 
> Is it this what the VFS lock patch is for?  If so, why hasn't it been
> integrated into the standard kernel?  Has it flaws in it?  Maybe
> performance wise?

It deadlocks for XFS on smp under heavy writeload. I use a writeable
snapshot patch and replay the log afterwards.

I think, the VFS patch has some principal problems. Creating a snapshot
with the VFS-lock patch applied is more or less equivalent to unmounting 
the file system, creating the snapshot of the device and remounting the
file system. That means that all ongoing write operations must be
suspended until the filesystem is in a "clean" state. This can take
some time. Up to 15 minutes from my observations and that is way too long.
I think the right way is: use a jornaling file system, take a snapshot,
make the snapshot writeable, replay the log, make the snapshot readonly
and dump it to tape or whatever you want. No races, no deadlocks, no
problems.

Anselm


---
 Anselm Kruis                                 Tel. +49 (0)89-356386-74
 science + computing ag		              FAX  +49 (0)89-356386-37
 Ingolstädter Str. 22             mailto: A.Kruis@science-computing.de
 D-80807 München                 WWW: http://www.science-computing.de/

**********************************************************************
***     Wir freuen uns auf Ihren Besuch                            ***
***                                                                ***
***     CeBIT 2002                                                 ***
***     13.-20.03.2002, Messegelände Hannover, Halle 11, F 64      ***
***                                                                ***
***     Automotive Circle International                            ***
***     18.-19.04.2002, Bad Nauheim                                ***
***                                                                ***
***     VDI-Tagung: Berechnung und Simulation im Fahrzeugbau       ***
***     01.-02.10.2002, Congress-Centrum Würzburg                  ***
***                                                                ***
********************************************************************** 

RE: [linux-lvm] snapshots of busy ext2 file system corrupt

[Stephenson, Dale]

Anselm Kruis writes:

[Of the VFS lock patch]
> 
> It deadlocks for XFS on smp under heavy writeload. I use a writeable
> snapshot patch and replay the log afterwards.
> 
So if you don't have the patch, you don't have a deadlock?  I've been trying
to track down a problem with multiple snapshots of the same volume, but the
presence or absence of the lock patch doesn't seem to make a difference.

> I think, the VFS patch has some principal problems. Creating 
> a snapshot
> with the VFS-lock patch applied is more or less equivalent to 
> unmounting 
> the file system, creating the snapshot of the device and 
> remounting the
> file system. That means that all ongoing write operations must be
> suspended until the filesystem is in a "clean" state. This can take
> some time. Up to 15 minutes from my observations and that is 
> way too long.
> I think the right way is: use a jornaling file system, take a 
> snapshot,
> make the snapshot writeable, replay the log, make the 
> snapshot readonly
> and dump it to tape or whatever you want. No races, no deadlocks, no
> problems.
> 
Except that it's a lot more steps :->.  I only wrote a writable snapshot
patch because I wanted to use XFS with snapshots and they didn't have the
nouuid,norecovery mount options at the time.  Is there any difference
between doing the replay and just mounting norecovery?

Dale Stephenson
steph@snapserver.com

RE: [linux-lvm] snapshots of busy ext2 file system corrupt

[Chris Mason]

On Wednesday, February 27, 2002 12:27:00 PM -0800 "Stephenson, Dale" <dstephenson@snapserver.com> wrote:

> Anselm Kruis writes:
> 
> [Of the VFS lock patch]
>> 
>> It deadlocks for XFS on smp under heavy writeload. I use a writeable
>> snapshot patch and replay the log afterwards.
>> 
> So if you don't have the patch, you don't have a deadlock?  I've been trying
> to track down a problem with multiple snapshots of the same volume, but the
> presence or absence of the lock patch doesn't seem to make a difference.

The slow part is in fsync_dev (and fsync_dev_lockfs), where we call
sync_buffers(dev, 0).  If there is heavy FS traffic, the buffer list
keeps refilling.

The whole thing would be much faster if the sync_buffers(dev, 0) call
were moved after the lockfs call, but that would require the filesystem
to be able to start buffer writes while the FS was locked.

For all the in-kernel filesystems, this is not a problem, but I was
assuming XFS was going to do strange things during the sync_buffers
call.
 
> 
>> I think, the VFS patch has some principal problems. Creating 
>> a snapshot
>> with the VFS-lock patch applied is more or less equivalent to 
>> unmounting 
>> the file system, creating the snapshot of the device and 
>> remounting the
>> file system. That means that all ongoing write operations must be
>> suspended until the filesystem is in a "clean" state. This can take
>> some time. Up to 15 minutes from my observations and that is 
>> way too long.
>> I think the right way is: use a jornaling file system, take a 
>> snapshot,
>> make the snapshot writeable, replay the log, make the 
>> snapshot readonly
>> and dump it to tape or whatever you want. No races, no deadlocks, no
>> problems.

There is a problem with this method, namely the FS you get after
replaying the log will not be the same FS you had at the time of
the snapshot.  Any transactions that had not been committed yet will
not be replayed.  It will be consistent, but probably not the FS you
were expecting.

-chris

RE: [linux-lvm] snapshots of busy ext2 file system corrupt

[Anselm Kruis]

On Wed, 27 Feb 2002, Chris Mason wrote:

> >> I think, the VFS patch has some principal problems. Creating 
> >> a snapshot
> >> with the VFS-lock patch applied is more or less equivalent to 
> >> unmounting 
> >> the file system, creating the snapshot of the device and 
> >> remounting the
> >> file system. That means that all ongoing write operations must be
> >> suspended until the filesystem is in a "clean" state. This can take
> >> some time. Up to 15 minutes from my observations and that is 
> >> way too long.
> >> I think the right way is: use a jornaling file system, take a 
> >> snapshot,
> >> make the snapshot writeable, replay the log, make the 
> >> snapshot readonly
> >> and dump it to tape or whatever you want. No races, no deadlocks, no
> >> problems.
> 
> There is a problem with this method, namely the FS you get after
> replaying the log will not be the same FS you had at the time of
> the snapshot.  Any transactions that had not been committed yet will
> not be replayed.  It will be consistent, but probably not the FS you
> were expecting.

It depends. For backups triggerd by cron that is not an issue, because i
only expect a snapshot of the filesystem at about 2am. For other purposes
it might be a problem. Would a call to sync prior to the lvcreate -s help?

Anselm


---
 Anselm Kruis                                 Tel. +49 (0)89-356386-74
 science + computing ag		              FAX  +49 (0)89-356386-37
 Ingolstädter Str. 22             mailto: A.Kruis@science-computing.de
 D-80807 München                 WWW: http://www.science-computing.de/

**********************************************************************
***     Wir freuen uns auf Ihren Besuch                            ***
***                                                                ***
***     CeBIT 2002                                                 ***
***     13.-20.03.2002, Messegelände Hannover, Halle 11, F 64      ***
***                                                                ***
***     Automotive Circle International                            ***
***     18.-19.04.2002, Bad Nauheim                                ***
***                                                                ***
***     VDI-Tagung: Berechnung und Simulation im Fahrzeugbau       ***
***     01.-02.10.2002, Congress-Centrum Würzburg                  ***
***                                                                ***
********************************************************************** 

RE: [linux-lvm] snapshots of busy ext2 file system corrupt

[Anselm Kruis]

On Wed, 27 Feb 2002, Stephenson, Dale wrote:

> Anselm Kruis writes:
> 
> [Of the VFS lock patch]
> > 
> > It deadlocks for XFS on smp under heavy writeload. I use a writeable
> > snapshot patch and replay the log afterwards.
> > 
> So if you don't have the patch, you don't have a deadlock?  I've been trying
> to track down a problem with multiple snapshots of the same volume, but the
> presence or absence of the lock patch doesn't seem to make a difference.

I don't have the deadlock without the VFS patch.

> 
> > I think, the VFS patch has some principal problems. Creating 
> > a snapshot
> > with the VFS-lock patch applied is more or less equivalent to 
> > unmounting 
> > the file system, creating the snapshot of the device and 
> > remounting the
> > file system. That means that all ongoing write operations must be
> > suspended until the filesystem is in a "clean" state. This can take
> > some time. Up to 15 minutes from my observations and that is 
> > way too long.
> > I think the right way is: use a jornaling file system, take a 
> > snapshot,
> > make the snapshot writeable, replay the log, make the 
> > snapshot readonly
> > and dump it to tape or whatever you want. No races, no deadlocks, no
> > problems.
> > 
> Except that it's a lot more steps :->.

Or a small script. :-)

  I only wrote a writable snapshot
> patch because I wanted to use XFS with snapshots and they didn't have the
> nouuid,norecovery mount options at the time.  Is there any difference
> between doing the replay and just mounting norecovery?

I think so. The filesystem might be inconsistent if you don't replay the
log.


---
 Anselm Kruis                                 Tel. +49 (0)89-356386-74
 science + computing ag		              FAX  +49 (0)89-356386-37
 Ingolstädter Str. 22             mailto: A.Kruis@science-computing.de
 D-80807 München                 WWW: http://www.science-computing.de/

**********************************************************************
***     Wir freuen uns auf Ihren Besuch                            ***
***                                                                ***
***     CeBIT 2002                                                 ***
***     13.-20.03.2002, Messegelände Hannover, Halle 11, F 64      ***
***                                                                ***
***     Automotive Circle International                            ***
***     18.-19.04.2002, Bad Nauheim                                ***
***                                                                ***
***     VDI-Tagung: Berechnung und Simulation im Fahrzeugbau       ***
***     01.-02.10.2002, Congress-Centrum Würzburg                  ***
***                                                                ***
**********************************************************************