* RE: [linux-lvm] LVM on Raid 5 Performance?
@ 2003-05-14 9:07 Little, Chris
2003-05-15 3:01 ` Herbert Poetzl
2003-05-19 1:25 ` Jason H. Smith
0 siblings, 2 replies; 11+ messages in thread
From: Little, Chris @ 2003-05-14 9:07 UTC (permalink / raw)
To: 'linux-lvm@sistina.com'
My understanding is that LVM is not RAID. Where RAID does provide
reliability, LVM allows finer grained control of resources. A side benefit
(and very large benefit) is the ability to stripe for a performance
increase. For us, an Oracle import went from 20 hours to 5 hours when using
striped volumes.
> -----Original Message-----
> From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> Sent: Wednesday, May 14, 2003 3:22 AM
> To: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] LVM on Raid 5 Performance?
>
>
> On Tue, May 13, 2003 at 09:57:22AM -0500, Little, Chris wrote:
> > stripe the logical volume across the disks. add a "-i x"
> where x is the
> > number of volumes in your vg.
>
> I guess this would give performance, but the idea
> was to have redundancy and performance where the
> latter is a bonus ...
>
> so the RAID 5 stuff is not an option, it is mandatory ;)
>
> best,
> Herbert
>
> >
> > > -----Original Message-----
> > > From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> > > Sent: Tuesday, May 13, 2003 8:37 AM
> > > To: linux-lvm@sistina.com
> > > Subject: [linux-lvm] LVM on Raid 5 Performance?
> > >
> > >
> > >
> > > Hi All!
> > >
> > > I'm using LVM for a long time, usually on non RAID
> > > systems to simplify storage space modifications ...
> > >
> > > recently I configured a nice system (Dual Athlon,
> > > with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
> > > on an Adaptec 29160N Ultra160. Because I had some
> > > troubles with the Adaptec/SCSI cabeling I reduced
> > > the bus speed to 40Mhz, which gives 80MB/s transfer
> > > (theoretically) via SCSI, each disc seems to do
> > > about 15MB/s (which seems a little low). I decided
> > > to arrange the 4 discs in a RAID 5 array to gain
> > > some speedup and redundancy (which resulted in
> > > about 33MB/s burst, 25% below the, again theoretical
> > > limit of 45MB/s). Ontop of the RAID 5 array I
> > > configured LVM 1.0.7 to create several partitions
> > > for further use ...
> > >
> > > Now what happened was, that the performance (read)
> > > dropped to about 18MB/s burst, which wasn't what
> > > I expected at all ...
> > >
> > > Has anybody any explanations why the LVM layer eats
> > > up about 45% of the available throughput?
> > > (in this configuration?)
> > >
> > > raw disc md(raid5) lvm on md
> > > -----------------------------------------------------
> > > hdparm 14694.40 34129.92 18667.52
> > > dd 1024k 14988.22 34732.56 18647.98
> > > dd 32k 15516.06 33945.48 18862.67
> > >
> > > best,
> > > Herbert
> > >
> > > --- Technical Information ---
> > >
> > > 2.4.21-rc2 kernel (with 1.0.7 LVM patch, among others)
> > >
> > > May 11 05:26:52 phoenix kernel: SCSI subsystem driver
> Revision: 1.00
> > > May 11 05:26:52 phoenix kernel: scsi0 : Adaptec AIC7XXX
> > > EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.33
> > > May 11 05:26:52 phoenix kernel: <Adaptec 29160N
> > > Ultra160 SCSI adapter>
> > > May 11 05:26:52 phoenix kernel: aic7892: Ultra160
> > > Wide Channel A, SCSI Id=7, 32/253 SCBs
> > > May 11 05:26:52 phoenix kernel:
> > > May 11 05:26:52 phoenix kernel: blk: queue c3667e18, I/O
> > > limit 4095Mb (mask 0xffffffff)
> > > May 11 05:26:52 phoenix kernel: (scsi0:A:0): 80.000MB/s
> > > transfers (40.000MHz, offset 63, 16bit)
> > > May 11 05:26:52 phoenix kernel: (scsi0:A:1): 80.000MB/s
> > > transfers (40.000MHz, offset 63, 16bit)
> > > May 11 05:26:52 phoenix kernel: (scsi0:A:2): 80.000MB/s
> > > transfers (40.000MHz, offset 63, 16bit)
> > > May 11 05:26:52 phoenix kernel: (scsi0:A:3): 80.000MB/s
> > > transfers (40.000MHz, offset 63, 16bit)
> > >
> > > raiddev /dev/md/0
> > >
> > > raid-level 5
> > > nr-raid-disks 4
> > > nr-spare-disks 0
> > > chunk-size 32
> > > parity-algorithm left-symmetric
> > > persistent-superblock 1
> > >
> > > device /dev/hd0/part5
> > > raid-disk 0
> > > device /dev/hd1/part5
> > > raid-disk 1
> > > device /dev/hd2/part5
> > > raid-disk 2
> > > device /dev/hd3/part5
> > > raid-disk 3
> > >
> > >
> > > pvcreate /dev/md/0
> > > vgcreate -A y vgs /dev/md/0
> > > lvcreate -C y -L 3G -n usr -Z y vgs
> > >
> > > --- Tests ---
> > >
> > > # hdparm -tT /dev/hd0/disc
> > >
> > > /dev/hd0/disc:
> > > Timing buffer-cache reads: 128 MB in 0.50 seconds
> =256.00 MB/sec
> > > Timing buffered disk reads: 64 MB in 4.46 seconds =
> 14.35 MB/sec
> > >
> > > # time dd if=/dev/hd0/disc of=/dev/null bs=1024k count=1024
> > > 1024+0 records in
> > > 1024+0 records out
> > > 0.00user 16.98system 1:09.96elapsed 24%CPU (0avgtext+0avgdata
> > > 0maxresident)k
> > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > >
> > > # time dd if=/dev/hd0/disc of=/dev/null bs=32k count=32768
> > > 32768+0 records in
> > > 32768+0 records out
> > > 0.02user 14.66system 1:07.58elapsed 21%CPU (0avgtext+0avgdata
> > > 0maxresident)k
> > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > >
> > > ------------
> > >
> > > # hdparm -tT /dev/md/0
> > >
> > > /dev/md/0:
> > > Timing buffer-cache reads: 128 MB in 0.49 seconds
> =261.22 MB/sec
> > > Timing buffered disk reads: 64 MB in 1.92 seconds =
> 33.33 MB/sec
> > >
> > > # time dd if=/dev/md/0 of=/dev/null bs=1024k count=1024
> > > 1024+0 records in
> > > 1024+0 records out
> > > 0.00user 12.09system 0:30.19elapsed 40%CPU (0avgtext+0avgdata
> > > 0maxresident)k
> > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > >
> > > # time dd if=/dev/md/0 of=/dev/null bs=32k count=32768
> > > 32768+0 records in
> > > 32768+0 records out
> > > 0.00user 10.52system 0:30.89elapsed 34%CPU (0avgtext+0avgdata
> > > 0maxresident)k
> > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > >
> > > -------------
> > >
> > > # hdparm -tT /dev/vgs/usr
> > >
> > > /dev/vgs/usr:
> > > Timing buffer-cache reads: 128 MB in 0.50 seconds
> =256.00 MB/sec
> > > Timing buffered disk reads: 64 MB in 3.51 seconds =
> 18.23 MB/sec
> > >
> > > # time dd if=/dev/vgs/usr of=/dev/null bs=1024k count=1024
> > > 1024+0 records in
> > > 1024+0 records out
> > > 0.01user 18.66system 0:56.23elapsed 33%CPU (0avgtext+0avgdata
> > > 0maxresident)k
> > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > >
> > > # time dd if=/dev/vgs/usr of=/dev/null bs=32k count=32768
> > > 32768+0 records in
> > > 32768+0 records out
> > > 0.07user 13.43system 0:55.59elapsed 24%CPU (0avgtext+0avgdata
> > > 0maxresident)k
> > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > >
> > >
> > >
> > > _______________________________________________
> > > linux-lvm mailing list
> > > linux-lvm@sistina.com
> > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
2003-05-14 9:07 [linux-lvm] LVM on Raid 5 Performance? Little, Chris
@ 2003-05-15 3:01 ` Herbert Poetzl
2003-05-19 1:25 ` Jason H. Smith
1 sibling, 0 replies; 11+ messages in thread
From: Herbert Poetzl @ 2003-05-15 3:01 UTC (permalink / raw)
To: linux-lvm
On Wed, May 14, 2003 at 09:08:08AM -0500, Little, Chris wrote:
> My understanding is that LVM is not RAID. Where RAID does provide
> reliability, LVM allows finer grained control of resources. A side benefit
> (and very large benefit) is the ability to stripe for a performance
> increase. For us, an Oracle import went from 20 hours to 5 hours when using
> striped volumes.
yeah, sure, I know what you mean, but I could
use a striping RAID 0 to gain the mentioned
performance. I added the LVM layer to make
several partitions from one RAID 5 volume
(which continues to work if one disk fails)
... so I guess this is a completely other
story ...
best,
Herbert
>
> > -----Original Message-----
> > From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> > Sent: Wednesday, May 14, 2003 3:22 AM
> > To: linux-lvm@sistina.com
> > Subject: Re: [linux-lvm] LVM on Raid 5 Performance?
> >
> >
> > On Tue, May 13, 2003 at 09:57:22AM -0500, Little, Chris wrote:
> > > stripe the logical volume across the disks. add a "-i x"
> > where x is the
> > > number of volumes in your vg.
> >
> > I guess this would give performance, but the idea
> > was to have redundancy and performance where the
> > latter is a bonus ...
> >
> > so the RAID 5 stuff is not an option, it is mandatory ;)
> >
> > best,
> > Herbert
> >
> > >
> > > > -----Original Message-----
> > > > From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> > > > Sent: Tuesday, May 13, 2003 8:37 AM
> > > > To: linux-lvm@sistina.com
> > > > Subject: [linux-lvm] LVM on Raid 5 Performance?
> > > >
> > > >
> > > >
> > > > Hi All!
> > > >
> > > > I'm using LVM for a long time, usually on non RAID
> > > > systems to simplify storage space modifications ...
> > > >
> > > > recently I configured a nice system (Dual Athlon,
> > > > with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
> > > > on an Adaptec 29160N Ultra160. Because I had some
> > > > troubles with the Adaptec/SCSI cabeling I reduced
> > > > the bus speed to 40Mhz, which gives 80MB/s transfer
> > > > (theoretically) via SCSI, each disc seems to do
> > > > about 15MB/s (which seems a little low). I decided
> > > > to arrange the 4 discs in a RAID 5 array to gain
> > > > some speedup and redundancy (which resulted in
> > > > about 33MB/s burst, 25% below the, again theoretical
> > > > limit of 45MB/s). Ontop of the RAID 5 array I
> > > > configured LVM 1.0.7 to create several partitions
> > > > for further use ...
> > > >
> > > > Now what happened was, that the performance (read)
> > > > dropped to about 18MB/s burst, which wasn't what
> > > > I expected at all ...
> > > >
> > > > Has anybody any explanations why the LVM layer eats
> > > > up about 45% of the available throughput?
> > > > (in this configuration?)
> > > >
> > > > raw disc md(raid5) lvm on md
> > > > -----------------------------------------------------
> > > > hdparm 14694.40 34129.92 18667.52
> > > > dd 1024k 14988.22 34732.56 18647.98
> > > > dd 32k 15516.06 33945.48 18862.67
> > > >
> > > > best,
> > > > Herbert
> > > >
> > > > --- Technical Information ---
> > > >
> > > > 2.4.21-rc2 kernel (with 1.0.7 LVM patch, among others)
> > > >
> > > > May 11 05:26:52 phoenix kernel: SCSI subsystem driver
> > Revision: 1.00
> > > > May 11 05:26:52 phoenix kernel: scsi0 : Adaptec AIC7XXX
> > > > EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.33
> > > > May 11 05:26:52 phoenix kernel: <Adaptec 29160N
> > > > Ultra160 SCSI adapter>
> > > > May 11 05:26:52 phoenix kernel: aic7892: Ultra160
> > > > Wide Channel A, SCSI Id=7, 32/253 SCBs
> > > > May 11 05:26:52 phoenix kernel:
> > > > May 11 05:26:52 phoenix kernel: blk: queue c3667e18, I/O
> > > > limit 4095Mb (mask 0xffffffff)
> > > > May 11 05:26:52 phoenix kernel: (scsi0:A:0): 80.000MB/s
> > > > transfers (40.000MHz, offset 63, 16bit)
> > > > May 11 05:26:52 phoenix kernel: (scsi0:A:1): 80.000MB/s
> > > > transfers (40.000MHz, offset 63, 16bit)
> > > > May 11 05:26:52 phoenix kernel: (scsi0:A:2): 80.000MB/s
> > > > transfers (40.000MHz, offset 63, 16bit)
> > > > May 11 05:26:52 phoenix kernel: (scsi0:A:3): 80.000MB/s
> > > > transfers (40.000MHz, offset 63, 16bit)
> > > >
> > > > raiddev /dev/md/0
> > > >
> > > > raid-level 5
> > > > nr-raid-disks 4
> > > > nr-spare-disks 0
> > > > chunk-size 32
> > > > parity-algorithm left-symmetric
> > > > persistent-superblock 1
> > > >
> > > > device /dev/hd0/part5
> > > > raid-disk 0
> > > > device /dev/hd1/part5
> > > > raid-disk 1
> > > > device /dev/hd2/part5
> > > > raid-disk 2
> > > > device /dev/hd3/part5
> > > > raid-disk 3
> > > >
> > > >
> > > > pvcreate /dev/md/0
> > > > vgcreate -A y vgs /dev/md/0
> > > > lvcreate -C y -L 3G -n usr -Z y vgs
> > > >
> > > > --- Tests ---
> > > >
> > > > # hdparm -tT /dev/hd0/disc
> > > >
> > > > /dev/hd0/disc:
> > > > Timing buffer-cache reads: 128 MB in 0.50 seconds
> > =256.00 MB/sec
> > > > Timing buffered disk reads: 64 MB in 4.46 seconds =
> > 14.35 MB/sec
> > > >
> > > > # time dd if=/dev/hd0/disc of=/dev/null bs=1024k count=1024
> > > > 1024+0 records in
> > > > 1024+0 records out
> > > > 0.00user 16.98system 1:09.96elapsed 24%CPU (0avgtext+0avgdata
> > > > 0maxresident)k
> > > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > > >
> > > > # time dd if=/dev/hd0/disc of=/dev/null bs=32k count=32768
> > > > 32768+0 records in
> > > > 32768+0 records out
> > > > 0.02user 14.66system 1:07.58elapsed 21%CPU (0avgtext+0avgdata
> > > > 0maxresident)k
> > > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > > >
> > > > ------------
> > > >
> > > > # hdparm -tT /dev/md/0
> > > >
> > > > /dev/md/0:
> > > > Timing buffer-cache reads: 128 MB in 0.49 seconds
> > =261.22 MB/sec
> > > > Timing buffered disk reads: 64 MB in 1.92 seconds =
> > 33.33 MB/sec
> > > >
> > > > # time dd if=/dev/md/0 of=/dev/null bs=1024k count=1024
> > > > 1024+0 records in
> > > > 1024+0 records out
> > > > 0.00user 12.09system 0:30.19elapsed 40%CPU (0avgtext+0avgdata
> > > > 0maxresident)k
> > > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > > >
> > > > # time dd if=/dev/md/0 of=/dev/null bs=32k count=32768
> > > > 32768+0 records in
> > > > 32768+0 records out
> > > > 0.00user 10.52system 0:30.89elapsed 34%CPU (0avgtext+0avgdata
> > > > 0maxresident)k
> > > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > > >
> > > > -------------
> > > >
> > > > # hdparm -tT /dev/vgs/usr
> > > >
> > > > /dev/vgs/usr:
> > > > Timing buffer-cache reads: 128 MB in 0.50 seconds
> > =256.00 MB/sec
> > > > Timing buffered disk reads: 64 MB in 3.51 seconds =
> > 18.23 MB/sec
> > > >
> > > > # time dd if=/dev/vgs/usr of=/dev/null bs=1024k count=1024
> > > > 1024+0 records in
> > > > 1024+0 records out
> > > > 0.01user 18.66system 0:56.23elapsed 33%CPU (0avgtext+0avgdata
> > > > 0maxresident)k
> > > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > > >
> > > > # time dd if=/dev/vgs/usr of=/dev/null bs=32k count=32768
> > > > 32768+0 records in
> > > > 32768+0 records out
> > > > 0.07user 13.43system 0:55.59elapsed 24%CPU (0avgtext+0avgdata
> > > > 0maxresident)k
> > > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > linux-lvm mailing list
> > > > linux-lvm@sistina.com
> > > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > > >
> > >
> > > _______________________________________________
> > > linux-lvm mailing list
> > > linux-lvm@sistina.com
> > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
2003-05-14 9:07 [linux-lvm] LVM on Raid 5 Performance? Little, Chris
2003-05-15 3:01 ` Herbert Poetzl
@ 2003-05-19 1:25 ` Jason H. Smith
2003-05-19 6:31 ` Herbert Poetzl
2003-05-24 19:11 ` Steven Lembark
1 sibling, 2 replies; 11+ messages in thread
From: Jason H. Smith @ 2003-05-19 1:25 UTC (permalink / raw)
To: linux-lvm
I don't know if this answers your question, but maybe search the archives
for my posts a few months ago. On 2.4, there's a misfeature when using
LVM with RAID5. Basically, the two will cause your system not to cache
any data, and to sync to disk immediately, once you create a snapshot
volume. Apparently, this will be addressed in 2.6.
So if you want RAID5 and LVM, consider hardware acceleration.
HTH
--
GPG: 03EE 9EB8 E500 874A F509 7B95 9B9A 84A1 26E9 4F79
http://www.ece.utexas.edu/~jhs/public_key.gpg
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
2003-05-19 1:25 ` Jason H. Smith
@ 2003-05-19 6:31 ` Herbert Poetzl
2003-05-24 19:11 ` Steven Lembark
1 sibling, 0 replies; 11+ messages in thread
From: Herbert Poetzl @ 2003-05-19 6:31 UTC (permalink / raw)
To: linux-lvm
On Sun, May 18, 2003 at 09:48:27PM +0700, Jason H. Smith wrote:
> I don't know if this answers your question, but maybe search the archives
> for my posts a few months ago. On 2.4, there's a misfeature when using
> LVM with RAID5. Basically, the two will cause your system not to cache
> any data, and to sync to disk immediately, once you create a snapshot
> volume. Apparently, this will be addressed in 2.6.
hmm, I read about that, but I guess that's not what
happens, because a) I do not use the snapshot feature,
and b) the "raid5: switching cache buffer size" messages
only appear if I access the md device directly ...
> So if you want RAID5 and LVM, consider hardware acceleration.
I guess I will choose not to use LVM on this system ...
thanks,
Herbert
> HTH
>
> --
> GPG: 03EE 9EB8 E500 874A F509 7B95 9B9A 84A1 26E9 4F79
> http://www.ece.utexas.edu/~jhs/public_key.gpg
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
2003-05-19 1:25 ` Jason H. Smith
2003-05-19 6:31 ` Herbert Poetzl
@ 2003-05-24 19:11 ` Steven Lembark
1 sibling, 0 replies; 11+ messages in thread
From: Steven Lembark @ 2003-05-24 19:11 UTC (permalink / raw)
To: linux-lvm
-- "Jason H. Smith" <jason@oes.co.th>
> I don't know if this answers your question, but maybe search the archives
> for my posts a few months ago. On 2.4, there's a misfeature when using
> LVM with RAID5. Basically, the two will cause your system not to cache
> any data, and to sync to disk immediately, once you create a snapshot
> volume. Apparently, this will be addressed in 2.6.
>
> So if you want RAID5 and LVM, consider hardware acceleration.
I'm having perfectly good luck with a DAC960 and 4 x IBM
160MB devices. The LVM issues pretty much drop out since
the drives are grouped into one "device" at the hardware
level. Striping the filesystem (ext2) helps (see the stride
option of mkfs.ext2), so does turning on write caching.
--
Steven Lembark 2930 W. Palmer
Workhorse Computing Chicago, IL 60647
+1 888 359 3508
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [linux-lvm] LVM on Raid 5 Performance?
@ 2003-05-15 9:51 Little, Chris
0 siblings, 0 replies; 11+ messages in thread
From: Little, Chris @ 2003-05-15 9:51 UTC (permalink / raw)
To: 'linux-lvm@sistina.com'
We are attached to an IBM ESS (Shark) a 3T SAN. I believe it has to do with
how the kernel schedules IO. Even though the SAN is raid 5, we still get
the increased throughput with a striped LVM. It's even recommended for our
configuration.
> -----Original Message-----
> From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> Sent: Thursday, May 15, 2003 12:54 AM
> To: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] LVM on Raid 5 Performance?
>
>
> On Wed, May 14, 2003 at 09:08:08AM -0500, Little, Chris wrote:
> > My understanding is that LVM is not RAID. Where RAID does provide
> > reliability, LVM allows finer grained control of resources.
> A side benefit
> > (and very large benefit) is the ability to stripe for a performance
> > increase. For us, an Oracle import went from 20 hours to 5
> hours when using
> > striped volumes.
>
> yeah, sure, I know what you mean, but I could
> use a striping RAID 0 to gain the mentioned
> performance. I added the LVM layer to make
> several partitions from one RAID 5 volume
> (which continues to work if one disk fails)
> ... so I guess this is a completely other
> story ...
>
> best,
> Herbert
>
> >
> > > -----Original Message-----
> > > From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> > > Sent: Wednesday, May 14, 2003 3:22 AM
> > > To: linux-lvm@sistina.com
> > > Subject: Re: [linux-lvm] LVM on Raid 5 Performance?
> > >
> > >
> > > On Tue, May 13, 2003 at 09:57:22AM -0500, Little, Chris wrote:
> > > > stripe the logical volume across the disks. add a "-i x"
> > > where x is the
> > > > number of volumes in your vg.
> > >
> > > I guess this would give performance, but the idea
> > > was to have redundancy and performance where the
> > > latter is a bonus ...
> > >
> > > so the RAID 5 stuff is not an option, it is mandatory ;)
> > >
> > > best,
> > > Herbert
> > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> > > > > Sent: Tuesday, May 13, 2003 8:37 AM
> > > > > To: linux-lvm@sistina.com
> > > > > Subject: [linux-lvm] LVM on Raid 5 Performance?
> > > > >
> > > > >
> > > > >
> > > > > Hi All!
> > > > >
> > > > > I'm using LVM for a long time, usually on non RAID
> > > > > systems to simplify storage space modifications ...
> > > > >
> > > > > recently I configured a nice system (Dual Athlon,
> > > > > with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
> > > > > on an Adaptec 29160N Ultra160. Because I had some
> > > > > troubles with the Adaptec/SCSI cabeling I reduced
> > > > > the bus speed to 40Mhz, which gives 80MB/s transfer
> > > > > (theoretically) via SCSI, each disc seems to do
> > > > > about 15MB/s (which seems a little low). I decided
> > > > > to arrange the 4 discs in a RAID 5 array to gain
> > > > > some speedup and redundancy (which resulted in
> > > > > about 33MB/s burst, 25% below the, again theoretical
> > > > > limit of 45MB/s). Ontop of the RAID 5 array I
> > > > > configured LVM 1.0.7 to create several partitions
> > > > > for further use ...
> > > > >
> > > > > Now what happened was, that the performance (read)
> > > > > dropped to about 18MB/s burst, which wasn't what
> > > > > I expected at all ...
> > > > >
> > > > > Has anybody any explanations why the LVM layer eats
> > > > > up about 45% of the available throughput?
> > > > > (in this configuration?)
> > > > >
> > > > > raw disc md(raid5) lvm on md
> > > > > -----------------------------------------------------
> > > > > hdparm 14694.40 34129.92 18667.52
> > > > > dd 1024k 14988.22 34732.56 18647.98
> > > > > dd 32k 15516.06 33945.48 18862.67
> > > > >
> > > > > best,
> > > > > Herbert
> > > > >
> > > > > --- Technical Information ---
> > > > >
> > > > > 2.4.21-rc2 kernel (with 1.0.7 LVM patch, among others)
> > > > >
> > > > > May 11 05:26:52 phoenix kernel: SCSI subsystem driver
> > > Revision: 1.00
> > > > > May 11 05:26:52 phoenix kernel: scsi0 : Adaptec AIC7XXX
> > > > > EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.33
> > > > > May 11 05:26:52 phoenix kernel: <Adaptec 29160N
> > > > > Ultra160 SCSI adapter>
> > > > > May 11 05:26:52 phoenix kernel: aic7892: Ultra160
> > > > > Wide Channel A, SCSI Id=7, 32/253 SCBs
> > > > > May 11 05:26:52 phoenix kernel:
> > > > > May 11 05:26:52 phoenix kernel: blk: queue c3667e18, I/O
> > > > > limit 4095Mb (mask 0xffffffff)
> > > > > May 11 05:26:52 phoenix kernel: (scsi0:A:0): 80.000MB/s
> > > > > transfers (40.000MHz, offset 63, 16bit)
> > > > > May 11 05:26:52 phoenix kernel: (scsi0:A:1): 80.000MB/s
> > > > > transfers (40.000MHz, offset 63, 16bit)
> > > > > May 11 05:26:52 phoenix kernel: (scsi0:A:2): 80.000MB/s
> > > > > transfers (40.000MHz, offset 63, 16bit)
> > > > > May 11 05:26:52 phoenix kernel: (scsi0:A:3): 80.000MB/s
> > > > > transfers (40.000MHz, offset 63, 16bit)
> > > > >
> > > > > raiddev /dev/md/0
> > > > >
> > > > > raid-level 5
> > > > > nr-raid-disks 4
> > > > > nr-spare-disks 0
> > > > > chunk-size 32
> > > > > parity-algorithm left-symmetric
> > > > > persistent-superblock 1
> > > > >
> > > > > device /dev/hd0/part5
> > > > > raid-disk 0
> > > > > device /dev/hd1/part5
> > > > > raid-disk 1
> > > > > device /dev/hd2/part5
> > > > > raid-disk 2
> > > > > device /dev/hd3/part5
> > > > > raid-disk 3
> > > > >
> > > > >
> > > > > pvcreate /dev/md/0
> > > > > vgcreate -A y vgs /dev/md/0
> > > > > lvcreate -C y -L 3G -n usr -Z y vgs
> > > > >
> > > > > --- Tests ---
> > > > >
> > > > > # hdparm -tT /dev/hd0/disc
> > > > >
> > > > > /dev/hd0/disc:
> > > > > Timing buffer-cache reads: 128 MB in 0.50 seconds
> > > =256.00 MB/sec
> > > > > Timing buffered disk reads: 64 MB in 4.46 seconds =
> > > 14.35 MB/sec
> > > > >
> > > > > # time dd if=/dev/hd0/disc of=/dev/null bs=1024k count=1024
> > > > > 1024+0 records in
> > > > > 1024+0 records out
> > > > > 0.00user 16.98system 1:09.96elapsed 24%CPU (0avgtext+0avgdata
> > > > > 0maxresident)k
> > > > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > > > >
> > > > > # time dd if=/dev/hd0/disc of=/dev/null bs=32k count=32768
> > > > > 32768+0 records in
> > > > > 32768+0 records out
> > > > > 0.02user 14.66system 1:07.58elapsed 21%CPU (0avgtext+0avgdata
> > > > > 0maxresident)k
> > > > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > > > >
> > > > > ------------
> > > > >
> > > > > # hdparm -tT /dev/md/0
> > > > >
> > > > > /dev/md/0:
> > > > > Timing buffer-cache reads: 128 MB in 0.49 seconds
> > > =261.22 MB/sec
> > > > > Timing buffered disk reads: 64 MB in 1.92 seconds =
> > > 33.33 MB/sec
> > > > >
> > > > > # time dd if=/dev/md/0 of=/dev/null bs=1024k count=1024
> > > > > 1024+0 records in
> > > > > 1024+0 records out
> > > > > 0.00user 12.09system 0:30.19elapsed 40%CPU (0avgtext+0avgdata
> > > > > 0maxresident)k
> > > > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > > > >
> > > > > # time dd if=/dev/md/0 of=/dev/null bs=32k count=32768
> > > > > 32768+0 records in
> > > > > 32768+0 records out
> > > > > 0.00user 10.52system 0:30.89elapsed 34%CPU (0avgtext+0avgdata
> > > > > 0maxresident)k
> > > > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > > > >
> > > > > -------------
> > > > >
> > > > > # hdparm -tT /dev/vgs/usr
> > > > >
> > > > > /dev/vgs/usr:
> > > > > Timing buffer-cache reads: 128 MB in 0.50 seconds
> > > =256.00 MB/sec
> > > > > Timing buffered disk reads: 64 MB in 3.51 seconds =
> > > 18.23 MB/sec
> > > > >
> > > > > # time dd if=/dev/vgs/usr of=/dev/null bs=1024k count=1024
> > > > > 1024+0 records in
> > > > > 1024+0 records out
> > > > > 0.01user 18.66system 0:56.23elapsed 33%CPU (0avgtext+0avgdata
> > > > > 0maxresident)k
> > > > > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> > > > >
> > > > > # time dd if=/dev/vgs/usr of=/dev/null bs=32k count=32768
> > > > > 32768+0 records in
> > > > > 32768+0 records out
> > > > > 0.07user 13.43system 0:55.59elapsed 24%CPU (0avgtext+0avgdata
> > > > > 0maxresident)k
> > > > > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > linux-lvm mailing list
> > > > > linux-lvm@sistina.com
> > > > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > > > >
> > > >
> > > > _______________________________________________
> > > > linux-lvm mailing list
> > > > linux-lvm@sistina.com
> > > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > >
> > > _______________________________________________
> > > linux-lvm mailing list
> > > linux-lvm@sistina.com
> > > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
@ 2003-05-15 3:49 Rick van der Linde
2003-05-15 16:45 ` Herbert Poetzl
0 siblings, 1 reply; 11+ messages in thread
From: Rick van der Linde @ 2003-05-15 3:49 UTC (permalink / raw)
To: linux-lvm
Hi All!
>I'm using LVM for a long time, usually on non RAID
>systems to simplify storage space modifications ...
>
>recently I configured a nice system (Dual Athlon,
>with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
>on an Adaptec 29160N Ultra160. Because I had some
>troubles with the Adaptec/SCSI cabeling I reduced
>the bus speed to 40Mhz, which gives 80MB/s transfer
>(theoretically) via SCSI, each disc seems to do
>about 15MB/s (which seems a little low). I decided
>to arrange the 4 discs in a RAID 5 array to gain
>some speedup and redundancy (which resulted in
>about 33MB/s burst, 25% below the, again theoretical
>limit of 45MB/s). Ontop of the RAID 5 array I
>configured LVM 1.0.7 to create several partitions
>for further use ...
>
>Now what happened was, that the performance (read)
>dropped to about 18MB/s burst, which wasn't what
>I expected at all ...
>
>Has anybody any explanations why the LVM layer eats
>up about 45% of the available throughput?
>(in this configuration?)
<SNIP results>
I can't yet tell you why, I see same kind of results with RAID 5 on 4 IDE
disks on
seperate channels (as master). In my case I use an ATA133 and onboard ATA33
controller. In my case the RAID5 set performs about 50 MB/s and the LVM
about
18 MB/s. I suspect my ATA controller as bottelneck because my disks which
can do
40 MB/s om the ATA133 controller suddenly operate at 18 MB/s on the ATA33.
That made me decide to buy another ATA100/133 controller.
As I searched internet on this issue I saw some discussion about ext3 in
combo with
md-raid and lvm. What filesystem did you use on your lvm volume on top of
RAID5?
What's fun too is when I use ext3 on RAID5 I see a performance of about 48
MB/s.
When I use reiserfs it reduces too abou 44 MB/s. ext2 shows almost the same
results
as ext3. Till now I same same results for ext2, ext3 and reiserfs in my
case, however
I still suspect my ATA33 channel to be the bottleneck in my case. When I
have another
ATA100/133 controller I'll post new results.
Rick
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
2003-05-15 3:49 Rick van der Linde
@ 2003-05-15 16:45 ` Herbert Poetzl
0 siblings, 0 replies; 11+ messages in thread
From: Herbert Poetzl @ 2003-05-15 16:45 UTC (permalink / raw)
To: linux-lvm
On Thu, May 15, 2003 at 10:51:44AM +0200, Rick van der Linde wrote:
>
>
> Hi All!
>
To cut it short LVM on 4 Disc RAID 5 reduces RAW
throughput about 45% from 33MB/s to 18MB/s :(
> >Has anybody any explanations why the LVM layer eats
> >up about 45% of the available throughput?
> >(in this configuration?)
>
> <SNIP results>
>
> I can't yet tell you why, I see same kind of results with
> RAID 5 on 4 IDE disks on seperate channels (as master).
> In my case I use an ATA133 and onboard ATA33 controller.
> In my case the RAID5 set performs about 50 MB/s and the LVM
> about 18 MB/s. I suspect my ATA controller as bottelneck
hmm, sounds interesting, especially because you
do not get ~ 27MB/s (55%) but 18MB/s the same as
I got ...
> because my disks which can do 40 MB/s om the ATA133
> controller suddenly operate at 18 MB/s on the ATA33.
> That made me decide to buy another ATA100/133 controller.
>
> As I searched internet on this issue I saw some discussion
> about ext3 in combo with md-raid and lvm. What filesystem
> did you use on your lvm volume on top of RAID5?
yeah, I read something similar, that was the reason
why I conducted my tests on the raw partitions
(without any filesystem on top ;)
> What's fun too is when I use ext3 on RAID5 I see a
> performance of about 48 MB/s.
> When I use reiserfs it reduces too abou 44 MB/s.
> ext2 shows almost the same results as ext3.
FYI: at least for ext2/ext3 you should make
sure that the 'stride' (raid options for mke2fs)
matches the block/stripe size of the underlying
RAID configuration ...
> Till now I same same results for ext2, ext3 and reiserfs
> in my case, however I still suspect my ATA33 channel
> to be the bottleneck in my case. When I have another
> ATA100/133 controller I'll post new results.
many thanks,
Herbert
>
> Rick
>
>
>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [linux-lvm] LVM on Raid 5 Performance?
@ 2003-05-13 9:57 Little, Chris
2003-05-14 3:21 ` Herbert Poetzl
0 siblings, 1 reply; 11+ messages in thread
From: Little, Chris @ 2003-05-13 9:57 UTC (permalink / raw)
To: 'linux-lvm@sistina.com'
stripe the logical volume across the disks. add a "-i x" where x is the
number of volumes in your vg.
> -----Original Message-----
> From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> Sent: Tuesday, May 13, 2003 8:37 AM
> To: linux-lvm@sistina.com
> Subject: [linux-lvm] LVM on Raid 5 Performance?
>
>
>
> Hi All!
>
> I'm using LVM for a long time, usually on non RAID
> systems to simplify storage space modifications ...
>
> recently I configured a nice system (Dual Athlon,
> with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
> on an Adaptec 29160N Ultra160. Because I had some
> troubles with the Adaptec/SCSI cabeling I reduced
> the bus speed to 40Mhz, which gives 80MB/s transfer
> (theoretically) via SCSI, each disc seems to do
> about 15MB/s (which seems a little low). I decided
> to arrange the 4 discs in a RAID 5 array to gain
> some speedup and redundancy (which resulted in
> about 33MB/s burst, 25% below the, again theoretical
> limit of 45MB/s). Ontop of the RAID 5 array I
> configured LVM 1.0.7 to create several partitions
> for further use ...
>
> Now what happened was, that the performance (read)
> dropped to about 18MB/s burst, which wasn't what
> I expected at all ...
>
> Has anybody any explanations why the LVM layer eats
> up about 45% of the available throughput?
> (in this configuration?)
>
> raw disc md(raid5) lvm on md
> -----------------------------------------------------
> hdparm 14694.40 34129.92 18667.52
> dd 1024k 14988.22 34732.56 18647.98
> dd 32k 15516.06 33945.48 18862.67
>
> best,
> Herbert
>
> --- Technical Information ---
>
> 2.4.21-rc2 kernel (with 1.0.7 LVM patch, among others)
>
> May 11 05:26:52 phoenix kernel: SCSI subsystem driver Revision: 1.00
> May 11 05:26:52 phoenix kernel: scsi0 : Adaptec AIC7XXX
> EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.33
> May 11 05:26:52 phoenix kernel: <Adaptec 29160N
> Ultra160 SCSI adapter>
> May 11 05:26:52 phoenix kernel: aic7892: Ultra160
> Wide Channel A, SCSI Id=7, 32/253 SCBs
> May 11 05:26:52 phoenix kernel:
> May 11 05:26:52 phoenix kernel: blk: queue c3667e18, I/O
> limit 4095Mb (mask 0xffffffff)
> May 11 05:26:52 phoenix kernel: (scsi0:A:0): 80.000MB/s
> transfers (40.000MHz, offset 63, 16bit)
> May 11 05:26:52 phoenix kernel: (scsi0:A:1): 80.000MB/s
> transfers (40.000MHz, offset 63, 16bit)
> May 11 05:26:52 phoenix kernel: (scsi0:A:2): 80.000MB/s
> transfers (40.000MHz, offset 63, 16bit)
> May 11 05:26:52 phoenix kernel: (scsi0:A:3): 80.000MB/s
> transfers (40.000MHz, offset 63, 16bit)
>
> raiddev /dev/md/0
>
> raid-level 5
> nr-raid-disks 4
> nr-spare-disks 0
> chunk-size 32
> parity-algorithm left-symmetric
> persistent-superblock 1
>
> device /dev/hd0/part5
> raid-disk 0
> device /dev/hd1/part5
> raid-disk 1
> device /dev/hd2/part5
> raid-disk 2
> device /dev/hd3/part5
> raid-disk 3
>
>
> pvcreate /dev/md/0
> vgcreate -A y vgs /dev/md/0
> lvcreate -C y -L 3G -n usr -Z y vgs
>
> --- Tests ---
>
> # hdparm -tT /dev/hd0/disc
>
> /dev/hd0/disc:
> Timing buffer-cache reads: 128 MB in 0.50 seconds =256.00 MB/sec
> Timing buffered disk reads: 64 MB in 4.46 seconds = 14.35 MB/sec
>
> # time dd if=/dev/hd0/disc of=/dev/null bs=1024k count=1024
> 1024+0 records in
> 1024+0 records out
> 0.00user 16.98system 1:09.96elapsed 24%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (111major+274minor)pagefaults 0swaps
>
> # time dd if=/dev/hd0/disc of=/dev/null bs=32k count=32768
> 32768+0 records in
> 32768+0 records out
> 0.02user 14.66system 1:07.58elapsed 21%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (110major+26minor)pagefaults 0swaps
>
> ------------
>
> # hdparm -tT /dev/md/0
>
> /dev/md/0:
> Timing buffer-cache reads: 128 MB in 0.49 seconds =261.22 MB/sec
> Timing buffered disk reads: 64 MB in 1.92 seconds = 33.33 MB/sec
>
> # time dd if=/dev/md/0 of=/dev/null bs=1024k count=1024
> 1024+0 records in
> 1024+0 records out
> 0.00user 12.09system 0:30.19elapsed 40%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (111major+274minor)pagefaults 0swaps
>
> # time dd if=/dev/md/0 of=/dev/null bs=32k count=32768
> 32768+0 records in
> 32768+0 records out
> 0.00user 10.52system 0:30.89elapsed 34%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (110major+26minor)pagefaults 0swaps
>
> -------------
>
> # hdparm -tT /dev/vgs/usr
>
> /dev/vgs/usr:
> Timing buffer-cache reads: 128 MB in 0.50 seconds =256.00 MB/sec
> Timing buffered disk reads: 64 MB in 3.51 seconds = 18.23 MB/sec
>
> # time dd if=/dev/vgs/usr of=/dev/null bs=1024k count=1024
> 1024+0 records in
> 1024+0 records out
> 0.01user 18.66system 0:56.23elapsed 33%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (111major+274minor)pagefaults 0swaps
>
> # time dd if=/dev/vgs/usr of=/dev/null bs=32k count=32768
> 32768+0 records in
> 32768+0 records out
> 0.07user 13.43system 0:55.59elapsed 24%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (110major+26minor)pagefaults 0swaps
>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] LVM on Raid 5 Performance?
2003-05-13 9:57 Little, Chris
@ 2003-05-14 3:21 ` Herbert Poetzl
0 siblings, 0 replies; 11+ messages in thread
From: Herbert Poetzl @ 2003-05-14 3:21 UTC (permalink / raw)
To: linux-lvm
On Tue, May 13, 2003 at 09:57:22AM -0500, Little, Chris wrote:
> stripe the logical volume across the disks. add a "-i x" where x is the
> number of volumes in your vg.
I guess this would give performance, but the idea
was to have redundancy and performance where the
latter is a bonus ...
so the RAID 5 stuff is not an option, it is mandatory ;)
best,
Herbert
>
> > -----Original Message-----
> > From: Herbert Poetzl [mailto:herbert@13thfloor.at]
> > Sent: Tuesday, May 13, 2003 8:37 AM
> > To: linux-lvm@sistina.com
> > Subject: [linux-lvm] LVM on Raid 5 Performance?
> >
> >
> >
> > Hi All!
> >
> > I'm using LVM for a long time, usually on non RAID
> > systems to simplify storage space modifications ...
> >
> > recently I configured a nice system (Dual Athlon,
> > with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
> > on an Adaptec 29160N Ultra160. Because I had some
> > troubles with the Adaptec/SCSI cabeling I reduced
> > the bus speed to 40Mhz, which gives 80MB/s transfer
> > (theoretically) via SCSI, each disc seems to do
> > about 15MB/s (which seems a little low). I decided
> > to arrange the 4 discs in a RAID 5 array to gain
> > some speedup and redundancy (which resulted in
> > about 33MB/s burst, 25% below the, again theoretical
> > limit of 45MB/s). Ontop of the RAID 5 array I
> > configured LVM 1.0.7 to create several partitions
> > for further use ...
> >
> > Now what happened was, that the performance (read)
> > dropped to about 18MB/s burst, which wasn't what
> > I expected at all ...
> >
> > Has anybody any explanations why the LVM layer eats
> > up about 45% of the available throughput?
> > (in this configuration?)
> >
> > raw disc md(raid5) lvm on md
> > -----------------------------------------------------
> > hdparm 14694.40 34129.92 18667.52
> > dd 1024k 14988.22 34732.56 18647.98
> > dd 32k 15516.06 33945.48 18862.67
> >
> > best,
> > Herbert
> >
> > --- Technical Information ---
> >
> > 2.4.21-rc2 kernel (with 1.0.7 LVM patch, among others)
> >
> > May 11 05:26:52 phoenix kernel: SCSI subsystem driver Revision: 1.00
> > May 11 05:26:52 phoenix kernel: scsi0 : Adaptec AIC7XXX
> > EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.33
> > May 11 05:26:52 phoenix kernel: <Adaptec 29160N
> > Ultra160 SCSI adapter>
> > May 11 05:26:52 phoenix kernel: aic7892: Ultra160
> > Wide Channel A, SCSI Id=7, 32/253 SCBs
> > May 11 05:26:52 phoenix kernel:
> > May 11 05:26:52 phoenix kernel: blk: queue c3667e18, I/O
> > limit 4095Mb (mask 0xffffffff)
> > May 11 05:26:52 phoenix kernel: (scsi0:A:0): 80.000MB/s
> > transfers (40.000MHz, offset 63, 16bit)
> > May 11 05:26:52 phoenix kernel: (scsi0:A:1): 80.000MB/s
> > transfers (40.000MHz, offset 63, 16bit)
> > May 11 05:26:52 phoenix kernel: (scsi0:A:2): 80.000MB/s
> > transfers (40.000MHz, offset 63, 16bit)
> > May 11 05:26:52 phoenix kernel: (scsi0:A:3): 80.000MB/s
> > transfers (40.000MHz, offset 63, 16bit)
> >
> > raiddev /dev/md/0
> >
> > raid-level 5
> > nr-raid-disks 4
> > nr-spare-disks 0
> > chunk-size 32
> > parity-algorithm left-symmetric
> > persistent-superblock 1
> >
> > device /dev/hd0/part5
> > raid-disk 0
> > device /dev/hd1/part5
> > raid-disk 1
> > device /dev/hd2/part5
> > raid-disk 2
> > device /dev/hd3/part5
> > raid-disk 3
> >
> >
> > pvcreate /dev/md/0
> > vgcreate -A y vgs /dev/md/0
> > lvcreate -C y -L 3G -n usr -Z y vgs
> >
> > --- Tests ---
> >
> > # hdparm -tT /dev/hd0/disc
> >
> > /dev/hd0/disc:
> > Timing buffer-cache reads: 128 MB in 0.50 seconds =256.00 MB/sec
> > Timing buffered disk reads: 64 MB in 4.46 seconds = 14.35 MB/sec
> >
> > # time dd if=/dev/hd0/disc of=/dev/null bs=1024k count=1024
> > 1024+0 records in
> > 1024+0 records out
> > 0.00user 16.98system 1:09.96elapsed 24%CPU (0avgtext+0avgdata
> > 0maxresident)k
> > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> >
> > # time dd if=/dev/hd0/disc of=/dev/null bs=32k count=32768
> > 32768+0 records in
> > 32768+0 records out
> > 0.02user 14.66system 1:07.58elapsed 21%CPU (0avgtext+0avgdata
> > 0maxresident)k
> > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> >
> > ------------
> >
> > # hdparm -tT /dev/md/0
> >
> > /dev/md/0:
> > Timing buffer-cache reads: 128 MB in 0.49 seconds =261.22 MB/sec
> > Timing buffered disk reads: 64 MB in 1.92 seconds = 33.33 MB/sec
> >
> > # time dd if=/dev/md/0 of=/dev/null bs=1024k count=1024
> > 1024+0 records in
> > 1024+0 records out
> > 0.00user 12.09system 0:30.19elapsed 40%CPU (0avgtext+0avgdata
> > 0maxresident)k
> > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> >
> > # time dd if=/dev/md/0 of=/dev/null bs=32k count=32768
> > 32768+0 records in
> > 32768+0 records out
> > 0.00user 10.52system 0:30.89elapsed 34%CPU (0avgtext+0avgdata
> > 0maxresident)k
> > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> >
> > -------------
> >
> > # hdparm -tT /dev/vgs/usr
> >
> > /dev/vgs/usr:
> > Timing buffer-cache reads: 128 MB in 0.50 seconds =256.00 MB/sec
> > Timing buffered disk reads: 64 MB in 3.51 seconds = 18.23 MB/sec
> >
> > # time dd if=/dev/vgs/usr of=/dev/null bs=1024k count=1024
> > 1024+0 records in
> > 1024+0 records out
> > 0.01user 18.66system 0:56.23elapsed 33%CPU (0avgtext+0avgdata
> > 0maxresident)k
> > 0inputs+0outputs (111major+274minor)pagefaults 0swaps
> >
> > # time dd if=/dev/vgs/usr of=/dev/null bs=32k count=32768
> > 32768+0 records in
> > 32768+0 records out
> > 0.07user 13.43system 0:55.59elapsed 24%CPU (0avgtext+0avgdata
> > 0maxresident)k
> > 0inputs+0outputs (110major+26minor)pagefaults 0swaps
> >
> >
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 11+ messages in thread
* [linux-lvm] LVM on Raid 5 Performance?
@ 2003-05-13 8:36 Herbert Poetzl
0 siblings, 0 replies; 11+ messages in thread
From: Herbert Poetzl @ 2003-05-13 8:36 UTC (permalink / raw)
To: linux-lvm
Hi All!
I'm using LVM for a long time, usually on non RAID
systems to simplify storage space modifications ...
recently I configured a nice system (Dual Athlon,
with 4 18.2G U160 SCSI IBM (DDYS-T18350N) disks
on an Adaptec 29160N Ultra160. Because I had some
troubles with the Adaptec/SCSI cabeling I reduced
the bus speed to 40Mhz, which gives 80MB/s transfer
(theoretically) via SCSI, each disc seems to do
about 15MB/s (which seems a little low). I decided
to arrange the 4 discs in a RAID 5 array to gain
some speedup and redundancy (which resulted in
about 33MB/s burst, 25% below the, again theoretical
limit of 45MB/s). Ontop of the RAID 5 array I
configured LVM 1.0.7 to create several partitions
for further use ...
Now what happened was, that the performance (read)
dropped to about 18MB/s burst, which wasn't what
I expected at all ...
Has anybody any explanations why the LVM layer eats
up about 45% of the available throughput?
(in this configuration?)
raw disc md(raid5) lvm on md
-----------------------------------------------------
hdparm 14694.40 34129.92 18667.52
dd 1024k 14988.22 34732.56 18647.98
dd 32k 15516.06 33945.48 18862.67
best,
Herbert
--- Technical Information ---
2.4.21-rc2 kernel (with 1.0.7 LVM patch, among others)
May 11 05:26:52 phoenix kernel: SCSI subsystem driver Revision: 1.00
May 11 05:26:52 phoenix kernel: scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.33
May 11 05:26:52 phoenix kernel: <Adaptec 29160N Ultra160 SCSI adapter>
May 11 05:26:52 phoenix kernel: aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
May 11 05:26:52 phoenix kernel:
May 11 05:26:52 phoenix kernel: blk: queue c3667e18, I/O limit 4095Mb (mask 0xffffffff)
May 11 05:26:52 phoenix kernel: (scsi0:A:0): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
May 11 05:26:52 phoenix kernel: (scsi0:A:1): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
May 11 05:26:52 phoenix kernel: (scsi0:A:2): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
May 11 05:26:52 phoenix kernel: (scsi0:A:3): 80.000MB/s transfers (40.000MHz, offset 63, 16bit)
raiddev /dev/md/0
raid-level 5
nr-raid-disks 4
nr-spare-disks 0
chunk-size 32
parity-algorithm left-symmetric
persistent-superblock 1
device /dev/hd0/part5
raid-disk 0
device /dev/hd1/part5
raid-disk 1
device /dev/hd2/part5
raid-disk 2
device /dev/hd3/part5
raid-disk 3
pvcreate /dev/md/0
vgcreate -A y vgs /dev/md/0
lvcreate -C y -L 3G -n usr -Z y vgs
--- Tests ---
# hdparm -tT /dev/hd0/disc
/dev/hd0/disc:
Timing buffer-cache reads: 128 MB in 0.50 seconds =256.00 MB/sec
Timing buffered disk reads: 64 MB in 4.46 seconds = 14.35 MB/sec
# time dd if=/dev/hd0/disc of=/dev/null bs=1024k count=1024
1024+0 records in
1024+0 records out
0.00user 16.98system 1:09.96elapsed 24%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (111major+274minor)pagefaults 0swaps
# time dd if=/dev/hd0/disc of=/dev/null bs=32k count=32768
32768+0 records in
32768+0 records out
0.02user 14.66system 1:07.58elapsed 21%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (110major+26minor)pagefaults 0swaps
------------
# hdparm -tT /dev/md/0
/dev/md/0:
Timing buffer-cache reads: 128 MB in 0.49 seconds =261.22 MB/sec
Timing buffered disk reads: 64 MB in 1.92 seconds = 33.33 MB/sec
# time dd if=/dev/md/0 of=/dev/null bs=1024k count=1024
1024+0 records in
1024+0 records out
0.00user 12.09system 0:30.19elapsed 40%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (111major+274minor)pagefaults 0swaps
# time dd if=/dev/md/0 of=/dev/null bs=32k count=32768
32768+0 records in
32768+0 records out
0.00user 10.52system 0:30.89elapsed 34%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (110major+26minor)pagefaults 0swaps
-------------
# hdparm -tT /dev/vgs/usr
/dev/vgs/usr:
Timing buffer-cache reads: 128 MB in 0.50 seconds =256.00 MB/sec
Timing buffered disk reads: 64 MB in 3.51 seconds = 18.23 MB/sec
# time dd if=/dev/vgs/usr of=/dev/null bs=1024k count=1024
1024+0 records in
1024+0 records out
0.01user 18.66system 0:56.23elapsed 33%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (111major+274minor)pagefaults 0swaps
# time dd if=/dev/vgs/usr of=/dev/null bs=32k count=32768
32768+0 records in
32768+0 records out
0.07user 13.43system 0:55.59elapsed 24%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (110major+26minor)pagefaults 0swaps
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2003-05-24 19:11 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-05-14 9:07 [linux-lvm] LVM on Raid 5 Performance? Little, Chris
2003-05-15 3:01 ` Herbert Poetzl
2003-05-19 1:25 ` Jason H. Smith
2003-05-19 6:31 ` Herbert Poetzl
2003-05-24 19:11 ` Steven Lembark
-- strict thread matches above, loose matches on Subject: below --
2003-05-15 9:51 Little, Chris
2003-05-15 3:49 Rick van der Linde
2003-05-15 16:45 ` Herbert Poetzl
2003-05-13 9:57 Little, Chris
2003-05-14 3:21 ` Herbert Poetzl
2003-05-13 8:36 Herbert Poetzl
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox