[linux-lvm] LVM2 strangeness.

[linux-lvm] LVM2 strangeness.

[Max Kalika]

Hello LVM devs and others,

I've been trying to use LVM2 on a backup server for the past few days and
I'm seeing some very strange things.

My configuration:
  Dell PowerEdge 1300 (800Mhz)
  512 MB RAM (2GB swap)
  4 120GB IDE (8mb cache) disks in a RAID-5 configuration (on a 3ware card)
  Custom built vanilla kernel (2.4.20) with DM patches (0.96.08) applied
  LVM2 utilities version 1.95.15

I have a 300GB VG with a 150GB LV on it.  I also have a 50GB snapshot and a
10GB snapshot.  I first rsync all remote server data (8 other machines) to
this "holding area" -- where I also hope to have 7 days of snapshot
information available on disk.  From here, the latest "current" data gets
pushed to tape.

Right now during an rsync of a huge mailstore (lots of small files) from
the mail server, the performance on the backup server degrades severly (up
to 12 1min load ave. and higher) and I'm seeing lots of the following in
the kernel log:

  May 29 14:14:15 butters invalidate: busy buffer
  May 29 14:14:15 butters invalidate: busy buffer
  May 29 14:14:15 butters invalidate: busy buffer
  May 29 14:14:15 butters invalidate: busy buffer
  ... on and on and on ...

I'm guessing that most of the cpu is busy copying stuff from the original
LV to the snapshots.  Is this a valid assumption?  What can one do to help
the situation?  Or do I have a different misconfiguration?  Any light you
can shed on this is greatly appreciated.  TIA.

--
---max kalika
--max@lsit.ucsb.edu
-lsit systems administrator

Re: [linux-lvm] LVM2 strangeness.

[Max Kalika]

Quick followup...

I blew away the snapshots and tried again.  The performance was much better
(not surprisingly), but I still have lots of "busy buffer" messages in the
kernel log.  Would anyone know what this is all about, and whether or not I
should worry about it?  I'm considering putting LVM2 on a production mail
server, primarily for the simple resize capabilities, but also to use
snapshots during the rsync process (so as not to get many "file doesn't
exist" errors).

Quoting Max Kalika <max+lists.lvm-dev@lsit.ucsb.edu>:

> Hello LVM devs and others,
> 
> I've been trying to use LVM2 on a backup server for the past few days and
> I'm seeing some very strange things.
> 
> My configuration:
>   Dell PowerEdge 1300 (800Mhz)
>   512 MB RAM (2GB swap)
>   4 120GB IDE (8mb cache) disks in a RAID-5 configuration (on a 3ware
> card)   Custom built vanilla kernel (2.4.20) with DM patches (0.96.08)
> applied   LVM2 utilities version 1.95.15
> 
> I have a 300GB VG with a 150GB LV on it.  I also have a 50GB snapshot and
> a 10GB snapshot.  I first rsync all remote server data (8 other machines)
> to this "holding area" -- where I also hope to have 7 days of snapshot
> information available on disk.  From here, the latest "current" data gets
> pushed to tape.
> 
> Right now during an rsync of a huge mailstore (lots of small files) from
> the mail server, the performance on the backup server degrades severly (up
> to 12 1min load ave. and higher) and I'm seeing lots of the following in
> the kernel log:
> 
>   May 29 14:14:15 butters invalidate: busy buffer
>   May 29 14:14:15 butters invalidate: busy buffer
>   May 29 14:14:15 butters invalidate: busy buffer
>   May 29 14:14:15 butters invalidate: busy buffer
>   ... on and on and on ...
> 
> I'm guessing that most of the cpu is busy copying stuff from the original
> LV to the snapshots.  Is this a valid assumption?  What can one do to help
> the situation?  Or do I have a different misconfiguration?  Any light you
> can shed on this is greatly appreciated.  TIA.
> 
> --
> ---max kalika
> --max@lsit.ucsb.edu
> -lsit systems administrator
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/



---max kalika
--max@lsit.ucsb.edu
-lsit systems administrator

Re: [linux-lvm] LVM2 strangeness.

[Joe Thornber]

On Thu, May 29, 2003 at 02:49:41PM -0700, Max Kalika wrote:
> Right now during an rsync of a huge mailstore (lots of small files) from
> the mail server, the performance on the backup server degrades severly (up
> to 12 1min load ave. and higher) and I'm seeing lots of the following in
> the kernel log:
> 
>   May 29 14:14:15 butters invalidate: busy buffer
>   May 29 14:14:15 butters invalidate: busy buffer
>   May 29 14:14:15 butters invalidate: busy buffer
>   May 29 14:14:15 butters invalidate: busy buffer
>   ... on and on and on ...

This is due to us trying to flush the LVM metadata writes to a device
that also contains a mounted filesystem, it should only happen when
you change your LVM configuration (eg, take another snapshot).  We're
using O_DIRECT in the next release, so they will go away soon.

- Joe

Re: [linux-lvm] LVM2 strangeness.

[Max Kalika]

Quoting Joe Thornber <thornber@sistina.com>:

> On Thu, May 29, 2003 at 02:49:41PM -0700, Max Kalika wrote:
>>   May 29 14:14:15 butters invalidate: busy buffer
>>   ... on and on and on ...
> 
> This is due to us trying to flush the LVM metadata writes to a device
> that also contains a mounted filesystem, it should only happen when
> you change your LVM configuration (eg, take another snapshot).  We're
> using O_DIRECT in the next release, so they will go away soon.

What is strange though is that these messages seemed to have been reported
when the system was just doing the rsync -- I wasn't changing any LVM
configuration at that time.  Could it have been an unfinished configuration
that hasn't been flushed to disk yet or some such? (Please excuse my
limited understanding).

--
---max kalika
--max@lsit.ucsb.edu
-lsit systems administrator

Re: [linux-lvm] LVM2 strangeness.

[Joe Thornber]

On Fri, May 30, 2003 at 08:48:41AM -0700, Max Kalika wrote:
> Quoting Joe Thornber <thornber@sistina.com>:
> 
> > On Thu, May 29, 2003 at 02:49:41PM -0700, Max Kalika wrote:
> >>   May 29 14:14:15 butters invalidate: busy buffer
> >>   ... on and on and on ...
> > 
> > This is due to us trying to flush the LVM metadata writes to a device
> > that also contains a mounted filesystem, it should only happen when
> > you change your LVM configuration (eg, take another snapshot).  We're
> > using O_DIRECT in the next release, so they will go away soon.
> 
> What is strange though is that these messages seemed to have been reported
> when the system was just doing the rsync -- I wasn't changing any LVM
> configuration at that time.  Could it have been an unfinished configuration
> that hasn't been flushed to disk yet or some such? (Please excuse my
> limited understanding).

what were you rsyncing ?

- Joe

Re: [linux-lvm] LVM2 strangeness.

[Max Kalika]

Quoting Joe Thornber <thornber@sistina.com>:

> what were you rsyncing ?

A 50GB mailstore over to the backup "holding area" box (the one that is
seeing the kernel messages). Maybe only 10GB was actual changed data.

--
---max kalika
--max@lsit.ucsb.edu
-lsit systems administrator

Re: [linux-lvm] LVM2 strangeness.

[Max Kalika]

Quoting Joe Thornber <thornber@sistina.com>:

> On Thu, May 29, 2003 at 02:49:41PM -0700, Max Kalika wrote:
>>   May 29 14:14:15 butters invalidate: busy buffer
> 
> This is due to us trying to flush the LVM metadata writes to a device
> that also contains a mounted filesystem, it should only happen when
> you change your LVM configuration (eg, take another snapshot).  We're
> using O_DIRECT in the next release, so they will go away soon.

Quick followup and a correction to my previous statement.  I do in fact
change an LVM config: When rsyncing some files to the backup box itself (as
it is easier to have ALL the files in one place -- I know it's silly)  I
create a snapshot of one of the partitions, mount it, rsync the snapshot,
unmount, and remove the snapshot.  It seems that during this time is when I
get all the busy buffer errors and sometimes worse.  Yesterday this caused
the box to stop being able to write to ALL LVM partitions and the process
list shows the following:

root         4     1  0 May30 ?        00:00:14 [kswapd] <defunct>
root         5     1  0 May30 ?        00:00:00 [bdflush] <defunct>

I'm going to turn off snapshotting for now and see if things become more
stable.

Perhaps I also don't quite understand how snapshots are supposed to work.
Is one supposed to stop all the daemons that are writing to a target volume
before a snapshot can be created?  Are "online" snapshots not the intent of
this system?

---max kalika
--max@lsit.ucsb.edu
-lsit systems administrator

Re: [linux-lvm] LVM2 strangeness.

[Heinz J . Mauelshagen]

On Thu, May 29, 2003 at 04:35:28PM -0700, Max Kalika wrote:
> Quick followup...
> 
> I blew away the snapshots and tried again.  The performance was much better
> (not surprisingly), but I still have lots of "busy buffer" messages in the
> kernel log.  Would anyone know what this is all about, and whether or not I
> should worry about it?

No, they're are caused by our tools syncing LVM metadata and will
disapapear in the next LVM2 release (as Joe mentioned).

> I'm considering putting LVM2 on a production mail
> server, primarily for the simple resize capabilities, but also to use
> snapshots during the rsync process (so as not to get many "file doesn't
> exist" errors).

Go for it :)

Regards,
Heinz    -- The LVM Guy --


> 
> Quoting Max Kalika <max+lists.lvm-dev@lsit.ucsb.edu>:
> 
> > Hello LVM devs and others,
> > 
> > I've been trying to use LVM2 on a backup server for the past few days and
> > I'm seeing some very strange things.
> > 
> > My configuration:
> >   Dell PowerEdge 1300 (800Mhz)
> >   512 MB RAM (2GB swap)
> >   4 120GB IDE (8mb cache) disks in a RAID-5 configuration (on a 3ware
> > card)   Custom built vanilla kernel (2.4.20) with DM patches (0.96.08)
> > applied   LVM2 utilities version 1.95.15
> > 
> > I have a 300GB VG with a 150GB LV on it.  I also have a 50GB snapshot and
> > a 10GB snapshot.  I first rsync all remote server data (8 other machines)
> > to this "holding area" -- where I also hope to have 7 days of snapshot
> > information available on disk.  From here, the latest "current" data gets
> > pushed to tape.
> > 
> > Right now during an rsync of a huge mailstore (lots of small files) from
> > the mail server, the performance on the backup server degrades severly (up
> > to 12 1min load ave. and higher) and I'm seeing lots of the following in
> > the kernel log:
> > 
> >   May 29 14:14:15 butters invalidate: busy buffer
> >   May 29 14:14:15 butters invalidate: busy buffer
> >   May 29 14:14:15 butters invalidate: busy buffer
> >   May 29 14:14:15 butters invalidate: busy buffer
> >   ... on and on and on ...
> > 
> > I'm guessing that most of the cpu is busy copying stuff from the original
> > LV to the snapshots.  Is this a valid assumption?  What can one do to help
> > the situation?  Or do I have a different misconfiguration?  Any light you
> > can shed on this is greatly appreciated.  TIA.
> > 
> > --
> > ---max kalika
> > --max@lsit.ucsb.edu
> > -lsit systems administrator
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> 
> 
> ---max kalika
> --max@lsit.ucsb.edu
> -lsit systems administrator
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] LVM2 strangeness.

[Joe Thornber]

On Sat, May 31, 2003 at 08:37:15AM -0700, Max Kalika wrote:
> I'm going to turn off snapshotting for now and see if things become more
> stable.
> 
> Perhaps I also don't quite understand how snapshots are supposed to work.
> Is one supposed to stop all the daemons that are writing to a target volume
> before a snapshot can be created?  Are "online" snapshots not the intent of
> this system?

If you want online snapshots you should have applied the VFS patch
which tell the fs to checkpoint itself ready for the snapshot to be
taken.  If you are using a database you will have to instruct the
database to do this checkpointing yourself.

It might be worth you waiting a few days for the 2.0 release which
will have the 'invalidate busy buffer' problem fixed.

- Joe