[linux-lvm] lvcreate distributed extent option

[linux-lvm] lvcreate distributed extent option

[ZECHA,HOLGER (HP-Germany,ex1)]

Hello,

I want to create a logical volume of 40GB Size using a physical 9GB disks,is
there a way to create a logical volume in the way, that the first logical
extend is on disk one, the second on disk two, ... the 7th on disk one and
so on ?

Meaning all logical extends are distributed accross all disks.

In HP-UX lvdisplay wwill look like this:
   --- Logical extents ---
   LE    PV1                PE1   Status 1
   00000 /dev/dsk/c16t0d0   00012 current
   00001 /dev/dsk/c16t0d1   00012 current
   00002 /dev/dsk/c17t0d2   00000 current
   00003 /dev/dsk/c17t0d3   00000 current
   00004 /dev/dsk/c16t0d4   00000 current
   00005 /dev/dsk/c16t0d5   00000 current
   00006 /dev/dsk/c17t0d6   00000 current
   00007 /dev/dsk/c17t0d7   00000 current
   00008 /dev/dsk/c16t1d0   00000 current
   00009 /dev/dsk/c16t1d1   00000 current
   00010 /dev/dsk/c17t1d2   00000 current
   00011 /dev/dsk/c17t1d3   00000 current
   00012 /dev/dsk/c16t1d4   00000 current

Holger

Re: [linux-lvm] lvcreate distributed extent option

[James Hawtin]

On Sat, 30 Nov 2002, ZECHA,HOLGER (HP-Germany,ex1) wrote:

>
> Hello,
>
> I want to create a logical volume of 40GB Size using a physical 9GB disks,is
> there a way to create a logical volume in the way, that the first logical
> extend is on disk one, the second on disk two, ... the 7th on disk one and
> so on ?
>
> Meaning all logical extends are distributed accross all disks.
>
> In HP-UX lvdisplay wwill look like this:
>    --- Logical extents ---
>    LE    PV1                PE1   Status 1
>    00000 /dev/dsk/c16t0d0   00012 current
>    00001 /dev/dsk/c16t0d1   00012 current
>    00002 /dev/dsk/c17t0d2   00000 current
>    00003 /dev/dsk/c17t0d3   00000 current
>    00004 /dev/dsk/c16t0d4   00000 current
>    00005 /dev/dsk/c16t0d5   00000 current
>    00006 /dev/dsk/c17t0d6   00000 current
>    00007 /dev/dsk/c17t0d7   00000 current
>    00008 /dev/dsk/c16t1d0   00000 current
>    00009 /dev/dsk/c16t1d1   00000 current
>    00010 /dev/dsk/c17t1d2   00000 current
>    00011 /dev/dsk/c17t1d3   00000 current
>    00012 /dev/dsk/c16t1d4   00000 current
>
> Holger
>

How about striping with the strip size equal to that of a logical extent?

otherwise if you have quite alot of disk space on another drive you could
create your lv on that one then move the extents one at a time to the
destinations drives using pvmove.....

One does wonder why you want to do this!

James

RE: [linux-lvm] lvcreate distributed extent option

[ZECHA,HOLGER (HP-Germany,ex1)]

Hello,

what do you mean with 
> How about striping with the strip size equal to that of a logical extent?

Can you give an example for the lvcreate command.

The reason to do this is to distribute the disk I/O accross all disks. the
logical extent 1 is on disk1, logical extent 2 on disk 2 and so on.
This means during read of logical extent 1, the heade of the second disk can
already be moved to prepare to read logical extent 2.

This allows us to do more I/O's per second

Holger


-----Original Message-----
From: James Hawtin [mailto:oolon@pzat.meep.org]
Sent: Samstag, 30. November 2002 14:40
To: linux-lvm@sistina.com
Cc: lvm-devel@sistina.com
Subject: Re: [linux-lvm] lvcreate distributed extent option




On Sat, 30 Nov 2002, ZECHA,HOLGER (HP-Germany,ex1) wrote:

>
> Hello,
>
> I want to create a logical volume of 40GB Size using a physical 9GB
disks,is
> there a way to create a logical volume in the way, that the first logical
> extend is on disk one, the second on disk two, ... the 7th on disk one and
> so on ?
>
> Meaning all logical extends are distributed accross all disks.
>
> In HP-UX lvdisplay wwill look like this:
>    --- Logical extents ---
>    LE    PV1                PE1   Status 1
>    00000 /dev/dsk/c16t0d0   00012 current
>    00001 /dev/dsk/c16t0d1   00012 current
>    00002 /dev/dsk/c17t0d2   00000 current
>    00003 /dev/dsk/c17t0d3   00000 current
>    00004 /dev/dsk/c16t0d4   00000 current
>    00005 /dev/dsk/c16t0d5   00000 current
>    00006 /dev/dsk/c17t0d6   00000 current
>    00007 /dev/dsk/c17t0d7   00000 current
>    00008 /dev/dsk/c16t1d0   00000 current
>    00009 /dev/dsk/c16t1d1   00000 current
>    00010 /dev/dsk/c17t1d2   00000 current
>    00011 /dev/dsk/c17t1d3   00000 current
>    00012 /dev/dsk/c16t1d4   00000 current
>
> Holger
>

How about striping with the strip size equal to that of a logical extent?

otherwise if you have quite alot of disk space on another drive you could
create your lv on that one then move the extents one at a time to the
destinations drives using pvmove.....

One does wonder why you want to do this!

James




_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

RE: [linux-lvm] lvcreate distributed extent option

[James Hawtin]

> This allows us to do more I/O's per second
>

yes... Thats striping why use a stripe size equal to that of a logical
extent I would have expected you to want them much smaller than that!
the default for striping on lvm is 8k. I my LE size is 16 meg,
16 meg strips would be a bit pointless in my book. Anyway using -I you can
set it to what you want.

lvcreate -i <num> -I <K> .....

from the lvcreate man page

       -i, --stripes Stripes
              Gives the number of stripes.  This is equal to  the
              number  of  physical volumes to scatter the logical
              volume.

       -I, --stripesize StripeSize
              Gives the number of kilobytes for  the  granularity
              of the stripes.
              StripeSize must be 2^n (n = 2 to 9)

James

Re: [linux-lvm] lvcreate distributed extent option

[toby]

"ZECHA,HOLGER (HP-Germany,ex1)" wrote:
> 
> Hello,
> 
> what do you mean with
> > How about striping with the strip size equal to that of a logical extent?
> 
> Can you give an example for the lvcreate command.
> 
> The reason to do this is to distribute the disk I/O accross all disks. the
> logical extent 1 is on disk1, logical extent 2 on disk 2 and so on.
> This means during read of logical extent 1, the heade of the second disk can
> already be moved to prepare to read logical extent 2.
> 
> This allows us to do more I/O's per second
> 
> Holger
> 


Once you create a striped lv, you cannot extend it by adding more disks. Current
lvm version anyway.

I wanted mine to be more flexible, so I first made a software raid set then made
the pv on top of the raid. That was in Jan this year. Last week I doubled the
size by adding another software raid set. I expanded two of the ext3 formatted
lvs in the process.

Perhaps not the fastest system in terms of throughput, but quite convenient to
resize.


-tkb

Re: [linux-lvm] lvcreate distributed extent option

[Ragnar Kjørstad]

On Sat, Nov 30, 2002 at 03:02:38PM +0100, ZECHA,HOLGER (HP-Germany,ex1) wrote:
> > How about striping with the strip size equal to that of a logical extent?
> 
> This allows us to do more I/O's per second

Only if you're only using a single or a few files at the time. If you're
using many files in parallel you will get more I/O's per second by
concatenating the devices instead if striping them. (that's what LVM
does by default)



-- 
Ragnar Kjørstad

Re: [linux-lvm] lvcreate distributed extent option

[James Hawtin]

On Sat, 30 Nov 2002, [iso-8859-1] Ragnar Kjørstad wrote:

> Only if you're only using a single or a few files at the time. If you're
> using many files in parallel you will get more I/O's per second by
> concatenating the devices instead if striping them. (that's what LVM
> does by default)

That of course assumes that your filesystem spreads files across the whole
lv in random and even fashion, and of cos if the contation uses two
different parts from the same disk its much slower... I have a feeling
that files are more oftain clustered towards the front of the disk. But
like with all things how you  setup your disks is very "use" dependant and
requires some degree of black magik ;-) Or atleast trying what works best
for you. Personally I have found stripping does not make my system much
faster. (Mostly I do compiling) Probably for me if I wanted to get IO,
mirroring include paths would be best, which is the best for reading,
"normal" for writting. And striping for where the data is being written,
but I guess thats why people use the jack of all trades raid 5 ;-)

I wish Lvm supported raid 5 and Mirroring...

James

RE: [linux-lvm] lvcreate distributed extent option

[ZECHA,HOLGER (HP-Germany,ex1)]

Hello,

there still seems to be a missunderstanding
I want to have the following:

   --- Logical extents ---
   LE    PV                        PE     reads      writes
   00000 /dev/sdd1                 00001  0          1        
   00001 /dev/sde1                 00001  0          0        
   00002 /dev/sdf1                 00001  0          0        
   00003 /dev/sdg1                 00001  0          0        
   00004 /dev/sdh1                 00001  0          0        
   00005 /dev/sdi1                 00001  0          0        
   00006 /dev/sdd1                 00002  0          0        
   00007 /dev/sde1                 00002  0          0        
   00008 /dev/sdf1                 00002  0          0        
   00009 /dev/sdg1                 00002  0          0        
   00010 /dev/sdh1                 00002  0          0        
   00011 /dev/sdi1                 00002  0          0        
   00012 /dev/sdd1                 00003  0          0        
   00013 /dev/sde1                 00003  0          0        
   00014 /dev/sdf1                 00003  0          0        
.....

but with  lvcreate -I 8 -L 40000M -n lvdata /dev/vgtest /dev/sdd1 /dev/sde1
.... I get

   --- Logical extents ---
   LE    PV                        PE     reads      writes
   00000 /dev/sdd1                 00063  0          1        
   00001 /dev/sdd1                 00064  0          0        
   00002 /dev/sdd1                 00065  0          0        
   00003 /dev/sdd1                 00066  0          0        
   00004 /dev/sdd1                 00067  0          0        
   00005 /dev/sdd1                 00068  0          0        
   00006 /dev/sdd1                 00069  0          0        
   00007 /dev/sdd1                 00070  0          0        
   00008 /dev/sdd1                 00071  0          0        
   00009 /dev/sdd1                 00072  0          0        
   00010 /dev/sdd1                 00073  0          0        
   00011 /dev/sdd1                 00074  0          0        
   00012 /dev/sdd1                 00075  0          0        
   00013 /dev/sdd1                 00076  0          0        
   00014 /dev/sdd1                 00077  0          0  
...
   02105 /dev/sde1                 00063  0          0        
   02106 /dev/sde1                 00064  0          0        
   02107 /dev/sde1                 00065  0          0        
   02108 /dev/sde1                 00066  0          0        
   02109 /dev/sde1                 00067  0          0        
   02110 /dev/sde1                 00068  0          0        
   02111 /dev/sde1                 00069  0          0        
   02112 /dev/sde1                 00070  0          0        

A distribution like described above looks like this on HP-UX LVM:    

   00000 /dev/dsk/c16t0d0   00012 current
   00001 /dev/dsk/c16t0d1   00012 current
   00002 /dev/dsk/c17t0d2   00000 current
   00003 /dev/dsk/c17t0d3   00000 current
   00004 /dev/dsk/c16t0d4   00000 current
   00005 /dev/dsk/c16t0d5   00000 current
   00006 /dev/dsk/c17t0d6   00000 current
   00007 /dev/dsk/c17t0d7   00000 current
   00008 /dev/dsk/c16t1d0   00000 current
   00009 /dev/dsk/c16t1d1   00000 current
   00010 /dev/dsk/c17t1d2   00000 current
   00011 /dev/dsk/c17t1d3   00000 current
   00012 /dev/dsk/c16t1d4   00000 current  

Holger 


-----Original Message-----
From: James Hawtin [mailto:oolon@ankh.org]
Sent: Samstag, 30. November 2002 18:24
To: linux-lvm@sistina.com
Cc: lvm-devel@sistina.com
Subject: Re: [linux-lvm] lvcreate distributed extent option


This message uses a character set that is not supported by the Internet
Service.  To view the original message content,  open the attached message.
If the text doesn't display correctly, save the attachment to disk, and then
open it using a viewer that can display the original character set. 

RE: [linux-lvm] lvcreate distributed extent option

[James Hawtin]

On Sat, 30 Nov 2002, ZECHA,HOLGER (HP-Germany,ex1) wrote:

> Hello,
>
> there still seems to be a missunderstanding
> I want to have the following:
>

Hmm your worring to much about how your physical data is being layout! The
reason you see that difference between HP lvm and linux lvm is because
they implement striping differently.

On, HP, they take the modulo of the number of stripes and allocate the
disk in that fasion ie.

a      0   0%2 =0
b      1   1%2 =1
a      2   2%2 =0
b      1   3%2 =1

On linux its

LE Stripe Size (LSS) = Number of LE / Number of strips

a = 0 - (LSS-1)
b = LSS - (LSS*2-1)

But thats just how the Logical extends are mapped LE Stripe Size != real
stripe size on HP or Linux its just how the strip pools are layed out.
The LVM system still has to re cut things LEs up to make real stripes.

Advantage of the way HP does it...
No problems extending things....

Disadvantage
Harder on the maths?
Difficult to specify which LEs to pick when moving them from one PV to
another.


Advantage of Linux way
Easy to specify and check when moving LEs from one PV to another.
Maths easier?

Disadvantage.
LEs have to be renumbered if the LV is extended or reduced, hence not
possible on linux.

Still don't believe me?

Try this....

--- Logical volume ---
LV Name                /dev/sunset_vg/oolon4_lv
VG Name                sunset_vg
LV Write Access        read/write
LV Status              available
LV #                   5
# open                 0
LV Size                18.38 GB
Current LE             588
Allocated LE           588
Stripes                2
Stripe size (KByte)    16
Allocation             next free
Read ahead sectors     120
Block device           58:4

   --- Distribution of logical volume on 2 physical volumes  ---
   PV Name                  PE on PV     reads      writes
   /dev/ide/host2/bus0/target0/lun0/part5 294          0          0
   /dev/ide/host2/bus0/target1/lun0/part2 294          0          0

   --- logical volume i/o statistic ---
   0 reads  0 writes

   --- Logical extents ---
   LE    PV                        PE     reads      writes
   00000 /dev/ide/host2/bus0/target0/lun0/part5 00060  0          0
   00001 /dev/ide/host2/bus0/target0/lun0/part5 00061  0          0
   00002 /dev/ide/host2/bus0/target0/lun0/part5 00062  0          0
...
   00294 /dev/ide/host2/bus0/target1/lun0/part2 00060  0          0
   00295 /dev/ide/host2/bus0/target1/lun0/part2 00061  0          0
   00296 /dev/ide/host2/bus0/target1/lun0/part2 00062  0          0
...

bash# dd if=/dev/sunset_vg/oolon4_lv of=/dev/null bs=16k count=2
2+0 records in
2+0 records out
bash# dd if=/dev/sunset_vg/oolon4_lv of=/dev/null bs=16k count=2
2+0 records in
2+0 records out
bash# dd if=/dev/sunset_vg/oolon4_lv of=/dev/null bs=16k count=2
2+0 records in
2+0 records out

lvdisplay -vv /dev/sunset_vg/oolon4_lv

....

   --- Distribution of logical volume on 2 physical volumes  ---
   PV Name                  PE on PV     reads      writes
   /dev/ide/host2/bus0/target0/lun0/part5 294          240        0
   /dev/ide/host2/bus0/target1/lun0/part2 294          216        0

   --- logical volume i/o statistic ---
   456 reads  0 writes

   --- Logical extents ---
   LE    PV                        PE     reads      writes
   00000 /dev/ide/host2/bus0/target0/lun0/part5 00060  240        0
   00001 /dev/ide/host2/bus0/target0/lun0/part5 00061  0          0
   00002 /dev/ide/host2/bus0/target0/lun0/part5 00062  0          0
....
   00294 /dev/ide/host2/bus0/target1/lun0/part2 00060  216        0
   00295 /dev/ide/host2/bus0/target1/lun0/part2 00061  0          0
   00296 /dev/ide/host2/bus0/target1/lun0/part2 00062  0          0
....

As you see my stripe size was 16k and I grabbed the first 32k of the
device 3 times.

You will notice the read figures are relatively even... not quite sure why
they are not the same I assume LVM might have some kind of header on the
LV. Which would account for the difference...

(I did this after rebooting the system to get clean figures).

You will however notice the stripping did work, it did use both disks not
as you suggest just 1 (if that had been the case the second disk would
have shown 0 reads), test it yourself, as I explained above the
"difference" you are seeing is just down to how the implementation of the
strip parts has been made. Though personally I agree with you the HP
system (due to the fact it can be resized) is supperior!

James

RE: [linux-lvm] lvcreate distributed extent option

[ZECHA,HOLGER (HP-Germany,ex1)]

Hello,

to clarify why we need this setup:
this is the setup for the oracle directory where the datafiles and
indexfiles are located.
we don't setup dedicated filesystem for data and indexfiles and we don't use
raw devices. We mix everything in one directory which is setup up with the
distributed option from HP-UX LVM on HP-UX. With this setup extents are
distributed accross all LUN's and therefore disk IO is also distributed
across all LUN's and administration of this setup i quit easier the using
dedicated filesystems for datafiles and index files.
We want to do this in the same way on Linux, therfore my question.

I wrote a shell script which does excatly this distribution for Linux.

Holger

Re: [linux-lvm] lvcreate distributed extent option

[Heinz J . Mauelshagen]

On Sat, Nov 30, 2002 at 10:24:06AM -0500, toby wrote:
> "ZECHA,HOLGER (HP-Germany,ex1)" wrote:
> > 
> > Hello,
> > 
> > what do you mean with
> > > How about striping with the strip size equal to that of a logical extent?
> > 
> > Can you give an example for the lvcreate command.
> > 
> > The reason to do this is to distribute the disk I/O accross all disks. the
> > logical extent 1 is on disk1, logical extent 2 on disk 2 and so on.
> > This means during read of logical extent 1, the heade of the second disk can
> > already be moved to prepare to read logical extent 2.
> > 
> > This allows us to do more I/O's per second
> > 
> > Holger
> > 
> 
> 
> Once you create a striped lv, you cannot extend it by adding more disks. Current
> lvm version anyway.

That constraint doesn't exits any longer in LVM2/device-mapper
using the new LVM2 ondisk metadata format.

> 
> I wanted mine to be more flexible, so I first made a software raid set then made
> the pv on top of the raid. That was in Jan this year. Last week I doubled the
> size by adding another software raid set. I expanded two of the ext3 formatted
> lvs in the process.
> 
> Perhaps not the fastest system in terms of throughput, but quite convenient to
> resize.
> 
> 
> -tkb
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-