[linux-lvm] Problems and questions with LVM on a large raid system.

[linux-lvm] Problems and questions with LVM on a large raid system.

[Petro]

I've got a couple of brand new boxes I'm trying to get LVM set up
properly on. 

These machines have 6 200GiB drives attached to a 3ware controller card,
for a total of 1,200,000,000,000 bytes of storage. 

I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools
version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).

Initially I had the drives set up as a single raid0 device, but doing a
pvcreate on it failed with a message that said something about "can't
get size". 

So I set up 2 3 disk RAID0 devices, and did a vgcreate /dev/sda /dev/sdb.

Now, one question I have is: 

    Does this concatenate the devices, or does it interleave the
    devices? IE do I have a stripe of stripes, or two stripes? 

Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping
that it's interleaved. 



-- 
"On two occasions, I have been asked [by members of Parliament], 'Pray, 
Mr. Babbage, if you put into the machine wrong figures, will the right 
answers come out?' I am not able to rightly apprehend the kind of confusion 
of ideas that could provoke such a question." -- Charles Babbage 

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[jon+lvm]

On Wed, Mar 12, 2003 at 11:52:17AM -0800, Petro wrote:
> I've got a couple of brand new boxes I'm trying to get LVM set up
> properly on. 
> 
> These machines have 6 200GiB drives attached to a 3ware controller card,
> for a total of 1,200,000,000,000 bytes of storage. 

later down you write you need speed. In that case i would suggest
ditching the hardware raid, and use pure software raid, it is faster.
I dont mind 3ware, i have dome myself, but my benchmarking revealed
that software was faster.


> I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools
> version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).

if you need snapshots, get 1.0.7

 
> Initially I had the drives set up as a single raid0 device, but doing a
> pvcreate on it failed with a message that said something about "can't
> get size". 

strange, linux should be able to handle 2T block devices.


> So I set up 2 3 disk RAID0 devices, and did a vgcreate /dev/sda /dev/sdb.
> 
> Now, one question I have is: 
> 
>     Does this concatenate the devices, or does it interleave the
>     devices? IE do I have a stripe of stripes, or two stripes? 

that depends. You can stripe the lvm, but then you cant resize it.
if you made a LV, then it would proberly just allokate all on one
"disk", aka sda, and then on sdb.


> Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping
> that it's interleaved. 

sorry... it is not interleaved.



JonB
ps: how big a partition do you need stripping for? Do you need the
advanced LVM stuff, like resize, snapshot...

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Kevin P. Fleming]

jon+lvm@silicide.dk wrote:
> On Wed, Mar 12, 2003 at 11:52:17AM -0800, Petro wrote:
> 
>>I've got a couple of brand new boxes I'm trying to get LVM set up
>>properly on. 
>>
>>These machines have 6 200GiB drives attached to a 3ware controller card,
>>for a total of 1,200,000,000,000 bytes of storage. 
> 
> 
> later down you write you need speed. In that case i would suggest
> ditching the hardware raid, and use pure software raid, it is faster.
> I dont mind 3ware, i have dome myself, but my benchmarking revealed
> that software was faster.
> 

True, but I'd sure hate to wait for a software array rebuild when the system 
crashes... I've only got a pair of 160G drives but I still hate waiting when the 
kernel has a problem. If your system is very, very stable this might not be an 
issue.

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[jon+lvm]

On Wed, Mar 12, 2003 at 03:23:10PM -0700, Kevin P. Fleming wrote:
> jon+lvm@silicide.dk wrote:

[cut]

> >later down you write you need speed. In that case i would suggest
> >ditching the hardware raid, and use pure software raid, it is faster.
> >I dont mind 3ware, i have dome myself, but my benchmarking revealed
> >that software was faster.
> >
> 
> True, but I'd sure hate to wait for a software array rebuild when the 
> system crashes... I've only got a pair of 160G drives but I still hate 
> waiting when the kernel has a problem. If your system is very, very stable 
> this might not be an issue.

you're talking about raid0. You dont rebuild raid0. Not in hardware, not
in software.



JonB

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Kevin P. Fleming]

jon+lvm@silicide.dk wrote:
> you're talking about raid0. You dont rebuild raid0. Not in hardware, not
> in software.

OK, I missed that extremely important and very obvious point :-)

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[jon+lvm]

On Wed, Mar 12, 2003 at 03:59:57PM -0700, Kevin P. Fleming wrote:
> jon+lvm@silicide.dk wrote:
> >you're talking about raid0. You dont rebuild raid0. Not in hardware, not
> >in software.
> 
> OK, I missed that extremely important and very obvious point :-)

thought so. Anyway, why would it take longer time to rebuild the
array using software rather than hardware raid? It's still the
same ammount of data needed to be copied.



JonB

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Kevin P. Fleming]

jon+lvm@silicide.dk wrote:
> On Wed, Mar 12, 2003 at 03:59:57PM -0700, Kevin P. Fleming wrote:
> 
>>jon+lvm@silicide.dk wrote:
>>
>>>you're talking about raid0. You dont rebuild raid0. Not in hardware, not
>>>in software.
>>
>>OK, I missed that extremely important and very obvious point :-)
> 
> 
> thought so. Anyway, why would it take longer time to rebuild the
> array using software rather than hardware raid? It's still the
> same ammount of data needed to be copied.
> 

Right, but a kernel oops doesn't cause an array rebuild to be necessary if 
you're using a 3ware or similar RAID card. The only thing that would cause a 
rebuild would be a drive failure or a power failure (which is of course easy to 
protect against).

The software code (which I've been using for quite a while) needs to rebuild any 
time the array is not shut down cleanly, which can happen for a number of reasons.

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Petro]

On Wed, Mar 12, 2003 at 11:13:20PM +0100, jon+lvm@silicide.dk wrote:
> On Wed, Mar 12, 2003 at 11:52:17AM -0800, Petro wrote:
> > I've got a couple of brand new boxes I'm trying to get LVM set up
> > properly on. 
> > These machines have 6 200GiB drives attached to a 3ware controller card,
> > for a total of 1,200,000,000,000 bytes of storage. 
> later down you write you need speed. In that case i would suggest
> ditching the hardware raid, and use pure software raid, it is faster.
> I dont mind 3ware, i have dome myself, but my benchmarking revealed
> that software was faster.

    The production data we gathered showed the opposite--at least for the 
    3ware cards in our environment (we had about 5-8% more overhead
    running the 3ware cards RAID0 than we did running either raw disk
    (cheap IDE cards) or having the 3ware card pretend to be a jbod. 

    This wasn't benchmarking, this was set it up and throw it over the
    wall for a month or two and see what the numbers looked like. 

    A really nervous way of testing. 
    
> > I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools
> > version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).
> if you need snapshots, get 1.0.7

    If I get the chance I'm probably going to ditch LVM since I don't
    need snapshots. I initially was give snapshots as a requirement. 

    Unfortunately I'll probably not get a chance to redo these. 
  
> > Initially I had the drives set up as a single raid0 device, but doing a
> > pvcreate on it failed with a message that said something about "can't
> > get size". 
> strange, linux should be able to handle 2T block devices.

    Yes. It should. 

    But when I did a pvcreate /dev/sda, where sda was 6 drives in a
    raid0, it failed. 
    
> > So I set up 2 3 disk RAID0 devices, and did a vgcreate /dev/sda /dev/sdb.
> > Now, one question I have is: 
> >     Does this concatenate the devices, or does it interleave the
> >     devices? IE do I have a stripe of stripes, or two stripes? 
> that depends. You can stripe the lvm, but then you cant resize it.
> if you made a LV, then it would proberly just allokate all on one
> "disk", aka sda, and then on sdb.
 
    I'm using 100% of the VG for the same LV. 

    I'm fine with not being able to resize it. Do I need the MD stuff to
    stripe it? 

> > Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping
> > that it's interleaved. 
> sorry... it is not interleaved.
> JonB
> ps: how big a partition do you need stripping for? Do you need the
> advanced LVM stuff, like resize, snapshot...

    I need slightly over 1T of formatted disk space on each system. 

    Which means i really need to find a way to send an <alt-3> through
    an ssh session to a telnet window. 

-- 
"On two occasions, I have been asked [by members of Parliament], 'Pray, 
Mr. Babbage, if you put into the machine wrong figures, will the right 
answers come out?' I am not able to rightly apprehend the kind of confusion 
of ideas that could provoke such a question." -- Charles Babbage 

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Jon Bendtsen]

Petro wrote:

[cut]

>     The production data we gathered showed the opposite--at least for the
>     3ware cards in our environment (we had about 5-8% more overhead
>     running the 3ware cards RAID0 than we did running either raw disk
>     (cheap IDE cards) or having the 3ware card pretend to be a jbod.

you mean the cpu was more used ?

 
>     This wasn't benchmarking, this was set it up and throw it over the
>     wall for a month or two and see what the numbers looked like.

i ran quite alot of tiobench on it with various parameters. I still
have a .pdf somewhere if you want it.

 
>     A really nervous way of testing.
> 
> > > I'm running kernel 2.4.18, with LVM version 1.0.3, and the lvmtools
> > > version 1.0.1rc3-1 (long story, but upgrading is a path-of-last-resort).
> > if you need snapshots, get 1.0.7
> 
>     If I get the chance I'm probably going to ditch LVM since I don't
>     need snapshots. I initially was give snapshots as a requirement.
> 
>     Unfortunately I'll probably not get a chance to redo these.

a shame, because if you dont need snapshots, LVM is a overhead.

 
[cut]

> > that depends. You can stripe the lvm, but then you cant resize it.
> > if you made a LV, then it would proberly just allokate all on one
> > "disk", aka sda, and then on sdb.
> 
>     I'm using 100% of the VG for the same LV.
> 
>     I'm fine with not being able to resize it. Do I need the MD stuff to
>     stripe it?

No. You can stripe using the LV system. But i would think that
if you can shutdown the LV, then you might as well stop the VG,
and do it on the raw device. (except this isnt as future proof
as a LVM system is).

 
> > > Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping
> > > that it's interleaved.
> > sorry... it is not interleaved.
> > JonB
> > ps: how big a partition do you need stripping for? Do you need the
> > advanced LVM stuff, like resize, snapshot...
> 
>     I need slightly over 1T of formatted disk space on each system.
> 
>     Which means i really need to find a way to send an <alt-3> through
>     an ssh session to a telnet window.

may i suggest
http://www.realweasel.com/intro.html
It's a pci GFX<->serial card.
I dont have one, but i want 3. If you buy one... do tell us how it
works.



JonB

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Petro]

On Thu, Mar 13, 2003 at 09:42:58AM +0100, Jon Bendtsen wrote:
> Petro wrote:
> [cut]
> >     The production data we gathered showed the opposite--at least for the
> >     3ware cards in our environment (we had about 5-8% more overhead
> >     running the 3ware cards RAID0 than we did running either raw disk
> >     (cheap IDE cards) or having the 3ware card pretend to be a jbod.
> you mean the cpu was more used ?

    No, meaning we got better overall performance out of Mysql. Fewer
    running threads, threads connected and disconnected better etc. 

  
> >     Unfortunately I'll probably not get a chance to redo these.
> a shame, because if you dont need snapshots, LVM is a overhead.

    It may come to that. 
  
> > > > Since I need the disk write rate of all 6 drives, I'm *REALLY* hoping
> > > > that it's interleaved.
> > > sorry... it is not interleaved.
> > > JonB
> > > ps: how big a partition do you need stripping for? Do you need the
> > > advanced LVM stuff, like resize, snapshot...
> >     I need slightly over 1T of formatted disk space on each system.
> >     Which means i really need to find a way to send an <alt-3> through
> >     an ssh session to a telnet window.
> may i suggest
> http://www.realweasel.com/intro.html
> It's a pci GFX<->serial card.
> I dont have one, but i want 3. If you buy one... do tell us how it
> works.

    That won't solve the problem. 

    See, my servers are 55 miles down the road. I ssh to a machine that
    connects to a private network, on which are a bunch of portmasters
    and APCs. I then telnet to the Portmaster ports, which hook to cua0. 

    It looks basically like I call he "smart hands" at the Colo and talk
    them through redoing the RAID volumes. 

   

-- 
"On two occasions, I have been asked [by members of Parliament], 'Pray, 
Mr. Babbage, if you put into the machine wrong figures, will the right 
answers come out?' I am not able to rightly apprehend the kind of confusion 
of ideas that could provoke such a question." -- Charles Babbage 

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Adrian Phillips]

>>>>> "Petro" == Petro  <petro@auctionwatch.com> writes:

    >> need to find a way to send an <alt-3> through > an ssh session
    >> to a telnet window.  may i suggest
    >> http://www.realweasel.com/intro.html It's a pci GFX<->serial
    >> card.  I dont have one, but i want 3. If you buy one... do tell
    >> us how it works.

    Petro>     That won't solve the problem.

    Petro>     See, my servers are 55 miles down the road. I ssh to a
    Petro> machine that connects to a private network, on which are a
    Petro> bunch of portmasters and APCs. I then telnet to the
    Petro> Portmaster ports, which hook to cua0.

    Petro>     It looks basically like I call he "smart hands" at the
    Petro> Colo and talk them through redoing the RAID volumes.

Um, sorry if I misunderstood, but using expect, either tcl or perl
versions, might solve this but then again it might not.

Sincerely,

Adrian Phillips

-- 
Your mouse has moved.
Windows NT must be restarted for the change to take effect.
Reboot now?  [OK]

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Steven Lembark]

-- Jon Bendtsen <jon+lvm@silicide.dk>

>> > that depends. You can stripe the lvm, but then you cant resize it.
>> > if you made a LV, then it would proberly just allokate all on one
>> > "disk", aka sda, and then on sdb.
>>
>>     I'm using 100% of the VG for the same LV.
>>
>>     I'm fine with not being able to resize it. Do I need the MD stuff to
>>     stripe it?
>
> No. You can stripe using the LV system. But i would think that
> if you can shutdown the LV, then you might as well stop the VG,
> and do it on the raw device. (except this isnt as future proof
> as a LVM system is).

Nor as flexable: with LVM you can easily grow the LV's, with
out LVM you are stuck with fixed partitions.




--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 888 359 3508

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[jon+lvm]

On Sat, Mar 15, 2003 at 01:59:59PM -0600, Steven Lembark wrote:
> 
> 
> -- Jon Bendtsen <jon+lvm@silicide.dk>
> 
> >>> that depends. You can stripe the lvm, but then you cant resize it.
> >>> if you made a LV, then it would proberly just allokate all on one
> >>> "disk", aka sda, and then on sdb.
> >>
> >>    I'm using 100% of the VG for the same LV.
> >>
> >>    I'm fine with not being able to resize it. Do I need the MD stuff to
> >>    stripe it?
> >
> >No. You can stripe using the LV system. But i would think that
> >if you can shutdown the LV, then you might as well stop the VG,
> >and do it on the raw device. (except this isnt as future proof
> >as a LVM system is).
> 
> Nor as flexable: with LVM you can easily grow the LV's, with
> out LVM you are stuck with fixed partitions.

that would be included in being future proof ;-D
Besides, he wants to stripe it, and you cant extend a stripped
LV (as far as i know).



JonB

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Heinz J . Mauelshagen]

On Sat, Mar 15, 2003 at 11:36:07PM +0100, jon+lvm@silicide.dk wrote:
> On Sat, Mar 15, 2003 at 01:59:59PM -0600, Steven Lembark wrote:
> > 
> > 
> > -- Jon Bendtsen <jon+lvm@silicide.dk>
> > 
> > >>> that depends. You can stripe the lvm, but then you cant resize it.
> > >>> if you made a LV, then it would proberly just allokate all on one
> > >>> "disk", aka sda, and then on sdb.
> > >>
> > >>    I'm using 100% of the VG for the same LV.
> > >>
> > >>    I'm fine with not being able to resize it. Do I need the MD stuff to
> > >>    stripe it?
> > >
> > >No. You can stripe using the LV system. But i would think that
> > >if you can shutdown the LV, then you might as well stop the VG,
> > >and do it on the raw device. (except this isnt as future proof
> > >as a LVM system is).
> > 
> > Nor as flexable: with LVM you can easily grow the LV's, with
> > out LVM you are stuck with fixed partitions.
> 
> that would be included in being future proof ;-D
> Besides, he wants to stripe it, and you cant extend a stripped
> LV (as far as i know).

Yes, you can.

Stripes are linited by the physical volume size in LVM1 though and the only way
to grow a striped LV there is to pvmove the stripe to a larger PV and extend
the LV afterwards.

LVM2 doesn't have such constraint any longer :)

> 
> 
> 
> JonB
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

RE: [linux-lvm] Problems and questions with LVM on a large raid system.

[Barry, Christopher]

Wouldn't some of the processing be off-loaded to a dedicated controller?

--
Christopher Barry
Manager of Information Systems
InfiniCon Systems
http://www.infiniconsys.com

-----Original Message-----
From: jon+lvm@silicide.dk [mailto:jon+lvm@silicide.dk]
Sent: Wednesday, March 12, 2003 6:16 PM
To: linux-lvm@sistina.com
Subject: Re: [linux-lvm] Problems and questions with LVM on a large raid
system.


On Wed, Mar 12, 2003 at 03:59:57PM -0700, Kevin P. Fleming wrote:
> jon+lvm@silicide.dk wrote:
> >you're talking about raid0. You dont rebuild raid0. Not in hardware, not
> >in software.
> 
> OK, I missed that extremely important and very obvious point :-)

thought so. Anyway, why would it take longer time to rebuild the
array using software rather than hardware raid? It's still the
same ammount of data needed to be copied.



JonB

_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Problems and questions with LVM on a large raid system.

[Jon Bendtsen]

"Barry, Christopher" wrote:
> 
> Wouldn't some of the processing be off-loaded to a dedicated controller?
\x7f
Yes, but would that matter (much) ?




JonB