[linux-lvm] snapshots+highmem+'md raid5'

[linux-lvm] snapshots+highmem+'md raid5'

[Heinz J . Mauelshagen]

Anyone out there using MD RAID5 as physical volumes, high memory enabled
(> 960m of physical memory) and snapshots under load successfully ?

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Herbert Poetzl]

On Thu, Jun 12, 2003 at 01:41:51PM +0200, Heinz J . Mauelshagen wrote:
> 
> Anyone out there using MD RAID5 as physical volumes, high memory enabled
> (> 960m of physical memory) and snapshots under load successfully ?

FYI: I use MD RAID5 as physical but I had to disable the
himem completely, because otherwise 1.0.7 would live-lock
the entire system ...

best,
Herbert

> Regards,
> Heinz    -- The LVM Guy --
> 
> *** Software bugs are stupid.
>     Nevertheless it needs not so stupid people to solve them ***
> 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 
> Heinz Mauelshagen                                 Sistina Software Inc.
> Senior Consultant/Developer                       Am Sonnenhang 11
>                                                   56242 Marienrachdorf
>                                                   Germany
> Mauelshagen@Sistina.com                           +49 2626 141200
>                                                        FAX 924446
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Dale J. Stephenson]

On Thursday, June 12, 2003, at 07:41  AM, Heinz J . Mauelshagen wrote:

>
> Anyone out there using MD RAID5 as physical volumes, high memory 
> enabled
> (> 960m of physical memory) and snapshots under load successfully ?
>
It depends on your definition of successful.  I'm using 2.4.18 with 
HIGHMEM and RAID 5 physical volumes, (and my chaining snapshot patch).  
Existing snapshots seem to work OK under load, but I have trouble 
creating the snapshots under load consistently.  My problems are with 
xfs_fs_freeze(), xfs's fsync_dev_lockfs implementation.

I have similar problems with 2.4.19.  I've run with 2.4.20, but not 
under load.  Haven't tried 2.4.21 yet.

Dale J. Stephenson
dalestephenson@mac.com

RE: [linux-lvm] snapshots+highmem+'md raid5'

[Koch, Steffen]

> > Anyone out there using MD RAID5 as physical volumes, high 
> memory enabled
> > (> 960m of physical memory) and snapshots under load successfully ?
> FYI: I use MD RAID5 as physical but I had to disable the
> himem completely, because otherwise 1.0.7 would live-lock
> the entire system ...

I've a similar problem. My machine has only 768M RAM but I think it may be
related. I've 4 IDE disks making up a MD raid5. This md is a single PV in a
VG. If I copy some amount of data (mostly from about 2-4 gigabytes) to a LV
created in that VG the complete system locks up, no chance to press
ctrl-alt-del, only hard reset works.

If someone has a solution for this please help.

BTW: I'm using LVM 1.0.7

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Herbert Poetzl]

On Thu, Jun 12, 2003 at 03:28:59PM +0200, Koch, Steffen wrote:
> > > Anyone out there using MD RAID5 as physical volumes, high 
> > memory enabled
> > > (> 960m of physical memory) and snapshots under load successfully ?
> > FYI: I use MD RAID5 as physical but I had to disable the
> > himem completely, because otherwise 1.0.7 would live-lock
> > the entire system ...
> 
> I've a similar problem. My machine has only 768M RAM but I think it may be
> related. I've 4 IDE disks making up a MD raid5. This md is a single PV in a
> VG. If I copy some amount of data (mostly from about 2-4 gigabytes) to a LV
> created in that VG the complete system locks up, no chance to press
> ctrl-alt-del, only hard reset works.
> 
> If someone has a solution for this please help.

hmm, do you have himem enabled (in kernel config)?
if yes, try to disable it ...

best,
Herbert

> BTW: I'm using LVM 1.0.7
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Petro]

On Fri, Jun 13, 2003 at 01:55:01AM +0200, Herbert Poetzl wrote:
> On Thu, Jun 12, 2003 at 03:28:59PM +0200, Koch, Steffen wrote:
> > If someone has a solution for this please help.
> hmm, do you have himem enabled (in kernel config)?
> if yes, try to disable it ...

    Does this apply to RAID 0 as well? 

    I've got several boxes running with:

    CONFIG_HIGHMEM4G=y
    # CONFIG_HIGHMEM64G is not set
    CONFIG_HIGHMEM=y

    And 2-4 GiB of ram under 2.4.18, with LVM 1.03. 

    I'm trying to get LVM 1.07 to compile with 2.4.21 with :
    
    CONFIG_HIGHMEM4G=y
    # CONFIG_HIGHMEM64G is not set
    CONFIG_HIGHMEM=y
    CONFIG_HIGHIO=y


    Why is this bad? 
    

-- 
"On two occasions, I have been asked [by members of Parliament], 'Pray, 
Mr. Babbage, if you put into the machine wrong figures, will the right 
answers come out?' I am not able to rightly apprehend the kind of confusion 
of ideas that could provoke such a question." -- Charles Babbage 

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Herbert Poetzl]

On Fri, Jun 13, 2003 at 05:20:27PM -0700, Petro wrote:
> On Fri, Jun 13, 2003 at 01:55:01AM +0200, Herbert Poetzl wrote:
> > On Thu, Jun 12, 2003 at 03:28:59PM +0200, Koch, Steffen wrote:
> > > If someone has a solution for this please help.
> > hmm, do you have himem enabled (in kernel config)?
> > if yes, try to disable it ...
> 
>     Does this apply to RAID 0 as well? 
> 
>     I've got several boxes running with:
> 
>     CONFIG_HIGHMEM4G=y
>     # CONFIG_HIGHMEM64G is not set
>     CONFIG_HIGHMEM=y
> 
>     And 2-4 GiB of ram under 2.4.18, with LVM 1.03. 
> 
>     I'm trying to get LVM 1.07 to compile with 2.4.21 with :
>     
>     CONFIG_HIGHMEM4G=y
>     # CONFIG_HIGHMEM64G is not set
>     CONFIG_HIGHMEM=y
>     CONFIG_HIGHIO=y
> 
> 
>     Why is this bad? 

didn't say that this would be bad, only that
it live-locked my system at high load ...
the tenor on the list was: "highmem is bad"
so I turned it off, and, voila the system was
stable again ...

another, yet unexplained issue, is that on
another system, 4 SCSI discs, form a RAID 5
array (software) which gives about 35MB
throughput, but lvm 1.0.7 ontop of that md
only has about 18MB throughput ...

YMMV,
Herbert

> -- 
> "On two occasions, I have been asked [by members of Parliament], 'Pray, 
> Mr. Babbage, if you put into the machine wrong figures, will the right 
> answers come out?' I am not able to rightly apprehend the kind of confusion 
> of ideas that could provoke such a question." -- Charles Babbage 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Svetoslav Slavtchev]

Quoting Herbert Poetzl <herbert@13thfloor.at>:

> On Fri, Jun 13, 2003 at 05:20:27PM -0700, Petro wrote:
> > On Fri, Jun 13, 2003 at 01:55:01AM +0200, Herbert Poetzl wrote:
> > > On Thu, Jun 12, 2003 at 03:28:59PM +0200, Koch, Steffen wrote:
> > > > If someone has a solution for this please help.
> > > hmm, do you have himem enabled (in kernel config)?
> > > if yes, try to disable it ...
> > 
> >     Does this apply to RAID 0 as well? 
> > 
> >     I've got several boxes running with:
> > 
> >     CONFIG_HIGHMEM4G=y
> >     # CONFIG_HIGHMEM64G is not set
> >     CONFIG_HIGHMEM=y
> > 
> >     And 2-4 GiB of ram under 2.4.18, with LVM 1.03. 
> > 
> >     I'm trying to get LVM 1.07 to compile with 2.4.21 with :
> >     
> >     CONFIG_HIGHMEM4G=y
> >     # CONFIG_HIGHMEM64G is not set
> >     CONFIG_HIGHMEM=y
> >     CONFIG_HIGHIO=y
> > 
> > 
> >     Why is this bad? 
> 
> didn't say that this would be bad, only that
> it live-locked my system at high load ...
> the tenor on the list was: "highmem is bad"
> so I turned it off, and, voila the system was
> stable again ...

snapshots + md + highmem is very bad
if you don't use snapshots its perfectly OK

> another, yet unexplained issue, is that on
> another system, 4 SCSI discs, form a RAID 5
> array (software) which gives about 35MB
> throughput, but lvm 1.0.7 ontop of that md
> only has about 18MB throughput ...

I've a bit defferent figures from a onboard HPT374 4 ide chanels with
4 IBM deskstars 80Gb, but i still lose 8-15Mb/s compared to plain md :

/dev/hda:
 Timing buffer-cache reads:   128 MB in  0.41 seconds =308.43 MB/sec
 Timing buffered disk reads:  64 MB in  1.42 seconds = 45.07 MB/sec
[root@svetljo root]# hdparm -tT /dev/md13

/dev/md13:
 Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
 Timing buffered disk reads:  64 MB in  1.08 seconds = 59.26 MB/sec
[root@svetljo root]# hdparm -tT /dev/data
data   data2
[root@svetljo root]# hdparm -tT /dev/data2/Movies

/dev/data2/Movies:
 Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
 Timing buffered disk reads:  64 MB in  1.32 seconds = 48.30 MB/sec

time dd if=/dev/zero of=/mnt/Movies/BigFile.tm bs=1M count=4000
4000+0 records in
4000+0 records out
0.02user 15.57system 1:37.10elapsed 16%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (148major+40minor)pagefaults 0swaps

best,

Svetljo

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Herbert Poetzl]

On Sat, Jun 14, 2003 at 02:45:11PM +0200, Svetoslav Slavtchev wrote:
> Quoting Herbert Poetzl <herbert@13thfloor.at>:
> 
> > didn't say that this would be bad, only that
> > it live-locked my system at high load ...
> > the tenor on the list was: "highmem is bad"
> > so I turned it off, and, voila the system was
> > stable again ...
> 
> snapshots + md + highmem is very bad
> if you don't use snapshots its perfectly OK

I have to disagree, live-lock under high load is
not perfectly OK, at least not for me ... 8-)

> > another, yet unexplained issue, is that on
> > another system, 4 SCSI discs, form a RAID 5
> > array (software) which gives about 35MB
> > throughput, but lvm 1.0.7 ontop of that md
> > only has about 18MB throughput ...
> 
> I've a bit defferent figures from a onboard HPT374 4 ide chanels with
> 4 IBM deskstars 80Gb, but i still lose 8-15Mb/s compared to plain md :
> 
> /dev/hda:
>  Timing buffer-cache reads:   128 MB in  0.41 seconds =308.43 MB/sec
>  Timing buffered disk reads:  64 MB in  1.42 seconds = 45.07 MB/sec
> [root@svetljo root]# hdparm -tT /dev/md13
> 
> /dev/md13:
>  Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
>  Timing buffered disk reads:  64 MB in  1.08 seconds = 59.26 MB/sec
> [root@svetljo root]# hdparm -tT /dev/data
> data   data2
> [root@svetljo root]# hdparm -tT /dev/data2/Movies
> 
> /dev/data2/Movies:
>  Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
>  Timing buffered disk reads:  64 MB in  1.32 seconds = 48.30 MB/sec

I posted my results a month ago, but I digged them
out again, because together with your results, it
sheds new light on that issue ...

mine          	raw disc    md(raid5)   lvm on md
-----------------------------------------------------
hdparm          14694.40    34129.92    18667.52
dd 1024k        14988.22    34732.56    18647.98
dd 32k          15516.06    33945.48    18862.67

yours          	raw disc    md(raid5)   lvm on md
-----------------------------------------------------
hdparm          46151.68    60682.24    49459.20 

in both cases the lvm lies about 3MB over the raw
disc throughput, regardless of the md values ...

maybe this could lead to some explanation of this
issue ... unfortunately there was no reply to my
posting over the last month *sigh*

best,
Herbert

> time dd if=/dev/zero of=/mnt/Movies/BigFile.tm bs=1M count=4000
> 4000+0 records in
> 4000+0 records out
> 0.02user 15.57system 1:37.10elapsed 16%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (148major+40minor)pagefaults 0swaps
> 
> best,
> 
> Svetljo
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Svetoslav Slavtchev]

Quoting Herbert Poetzl <herbert@13thfloor.at>:

> On Sat, Jun 14, 2003 at 02:45:11PM +0200, Svetoslav Slavtchev wrote:
> > Quoting Herbert Poetzl <herbert@13thfloor.at>:
> > 
> > > didn't say that this would be bad, only that
> > > it live-locked my system at high load ...
> > > the tenor on the list was: "highmem is bad"
> > > so I turned it off, and, voila the system was
> > > stable again ...
> > 
> > snapshots + md + highmem is very bad
> > if you don't use snapshots its perfectly OK
> 
> I have to disagree, live-lock under high load is
> not perfectly OK, at least not for me ... 8-)

i never had a lock-up :)
this is with lvm1 soft raid 0 &5 and xfs.
(firstly 1Gb and from a week 1.5Gb)

> > > another, yet unexplained issue, is that on
> > > another system, 4 SCSI discs, form a RAID 5
> > > array (software) which gives about 35MB
> > > throughput, but lvm 1.0.7 ontop of that md
> > > only has about 18MB throughput ...
> > 
> > I've a bit defferent figures from a onboard HPT374 4 ide chanels
> with
> > 4 IBM deskstars 80Gb, but i still lose 8-15Mb/s compared to plain md
> :
> > 
> > /dev/hda:
> >  Timing buffer-cache reads:   128 MB in  0.41 seconds =308.43 MB/sec
> >  Timing buffered disk reads:  64 MB in  1.42 seconds = 45.07 MB/sec
> > [root@svetljo root]# hdparm -tT /dev/md13
> > 
> > /dev/md13:
> >  Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
> >  Timing buffered disk reads:  64 MB in  1.08 seconds = 59.26 MB/sec
> > [root@svetljo root]# hdparm -tT /dev/data
> > data   data2
> > [root@svetljo root]# hdparm -tT /dev/data2/Movies
> > 
> > /dev/data2/Movies:
> >  Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
> >  Timing buffered disk reads:  64 MB in  1.32 seconds = 48.30 MB/sec
> 
> I posted my results a month ago, but I digged them
> out again, because together with your results, it
> sheds new light on that issue ...
> 
> mine          	raw disc    md(raid5)   lvm on md
> -----------------------------------------------------
> hdparm          14694.40    34129.92    18667.52
> dd 1024k        14988.22    34732.56    18647.98
> dd 32k          15516.06    33945.48    18862.67
> 
> yours          	raw disc    md(raid5)   lvm on md
> -----------------------------------------------------
> hdparm          46151.68    60682.24    49459.20 
> 
> in both cases the lvm lies about 3MB over the raw
> disc throughput, regardless of the md values ...
> 
> maybe this could lead to some explanation of this
> issue ... unfortunately there was no reply to my
> posting over the last month *sigh*

hm this is really strange.
but can we belive hdparm, i think no.
i was getting ~55Mb/s  on raid-0 with bonnie++ but hdparm gives between 58-68Mb/s.
i probably have to try to free one of the md5 raids and run bonnie++ on it,
it should give better/ trustable numbers 


best,

svetljo

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Heinz J . Mauelshagen]

On Sat, Jun 14, 2003 at 10:07:54PM +0200, Herbert Poetzl wrote:
> On Sat, Jun 14, 2003 at 02:45:11PM +0200, Svetoslav Slavtchev wrote:
> > Quoting Herbert Poetzl <herbert@13thfloor.at>:
> > 
> > > didn't say that this would be bad, only that
> > > it live-locked my system at high load ...
> > > the tenor on the list was: "highmem is bad"
> > > so I turned it off, and, voila the system was
> > > stable again ...
> > 
> > snapshots + md + highmem is very bad
> > if you don't use snapshots its perfectly OK
> 
> I have to disagree, live-lock under high load is
> not perfectly OK, at least not for me ... 8-)
> 
> > > another, yet unexplained issue, is that on
> > > another system, 4 SCSI discs, form a RAID 5
> > > array (software) which gives about 35MB
> > > throughput, but lvm 1.0.7 ontop of that md
> > > only has about 18MB throughput ...
> > 
> > I've a bit defferent figures from a onboard HPT374 4 ide chanels with
> > 4 IBM deskstars 80Gb, but i still lose 8-15Mb/s compared to plain md :
> > 
> > /dev/hda:
> >  Timing buffer-cache reads:   128 MB in  0.41 seconds =308.43 MB/sec
> >  Timing buffered disk reads:  64 MB in  1.42 seconds = 45.07 MB/sec
> > [root@svetljo root]# hdparm -tT /dev/md13
> > 
> > /dev/md13:
> >  Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
> >  Timing buffered disk reads:  64 MB in  1.08 seconds = 59.26 MB/sec
> > [root@svetljo root]# hdparm -tT /dev/data
> > data   data2
> > [root@svetljo root]# hdparm -tT /dev/data2/Movies
> > 
> > /dev/data2/Movies:
> >  Timing buffer-cache reads:   128 MB in  0.41 seconds =316.05 MB/sec
> >  Timing buffered disk reads:  64 MB in  1.32 seconds = 48.30 MB/sec
> 
> I posted my results a month ago, but I digged them
> out again, because together with your results, it
> sheds new light on that issue ...
> 
> mine          	raw disc    md(raid5)   lvm on md
> -----------------------------------------------------
> hdparm          14694.40    34129.92    18667.52
> dd 1024k        14988.22    34732.56    18647.98
> dd 32k          15516.06    33945.48    18862.67
> 
> yours          	raw disc    md(raid5)   lvm on md
> -----------------------------------------------------
> hdparm          46151.68    60682.24    49459.20 
> 
> in both cases the lvm lies about 3MB over the raw
> disc throughput, regardless of the md values ...

The reason probably is LVMs read ahead.

> 
> maybe this could lead to some explanation of this
> issue ... unfortunately there was no reply to my
> posting over the last month *sigh*
> 
> best,
> Herbert
> 
> > time dd if=/dev/zero of=/mnt/Movies/BigFile.tm bs=1M count=4000
> > 4000+0 records in
> > 4000+0 records out
> > 0.02user 15.57system 1:37.10elapsed 16%CPU (0avgtext+0avgdata 0maxresident)k
> > 0inputs+0outputs (148major+40minor)pagefaults 0swaps
> > 
> > best,
> > 
> > Svetljo
> > 
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] snapshots+highmem+'md raid5'

[Herbert Poetzl]

On Mon, Jun 16, 2003 at 11:32:49AM +0200, Heinz J . Mauelshagen wrote:
> On Sat, Jun 14, 2003 at 10:07:54PM +0200, Herbert Poetzl wrote:
> > On Sat, Jun 14, 2003 at 02:45:11PM +0200, Svetoslav Slavtchev wrote:
> > > Quoting Herbert Poetzl <herbert@13thfloor.at>:

 ... some highmem live-lock issue [removed] ...

> > I posted my results a month ago, but I digged them
> > out again, because together with your results, it
> > sheds new light on that issue ...
> > 
> > mine          	raw disc    md(raid5)   lvm on md
> > -----------------------------------------------------
> > hdparm          14694.40    34129.92    18667.52
> > dd 1024k        14988.22    34732.56    18647.98
> > dd 32k          15516.06    33945.48    18862.67
> > 
> > yours          	raw disc    md(raid5)   lvm on md
> > -----------------------------------------------------
> > hdparm          46151.68    60682.24    49459.20 
> > 
> > in both cases the lvm lies about 3MB over the raw
> > disc throughput, regardless of the md values ...
> 
> The reason probably is LVMs read ahead.

hmm, so it's a feature?

seriously, what can be done about it?
it surely doesn't hurt on a 50M per disk throughput 
system, but 34M/19M seems cruel, especially because 
I'm thinking of adding one or two more discs to the 
system, which probably leads to a 50M/19M situation 
with lvm?

TIA,
Herbert

PS: why am I receiving your emails twice, with 
alternating From headers? I actually get them
three times as I'm subscribed to the list too ;)

Heinz.Mauelshagen@t-online.de (Heinz J . Mauelshagen)
"Heinz J . Mauelshagen" <mauelshagen@sistina.com>
(one list copy)