[linux-lvm] advice sought on setting up new system

[linux-lvm] advice sought on setting up new system

[Dale Gallagher]

Hi there

I'm new to LVM2 and am considering using it on a production system,
but would welcome advice from experienced users out there, before
I set things up.  Your advice and comments would be appreciated.
Details follow.

Summary:

Primary function: mail system (pop3,smtp,http)
Scalable to 10's of thousands (and hopefully millions) of users ;-)
Applications: qmail, PostgreSQL, djbdns

Current system:

1GB RAM
4x U320 15k SCSI 72GB disks in hardware RAID1+0
All functions running on same host.

Future system to include external drive cage(s) and splitting of
functions onto separate hosts (DB/mail/DNS).  The external drive
cage(s) to be used for mail storage. qmail uses maildir, as opposed
to mbox delivery, so each message is a separate file... this means
I require performance for a vastly greater number of smaller files
as opposed to a single mbox file per user.  Another consideration is
a popular belief that mounting /var/qmail/queue with the sync option
set, or manually invoking sync() improves the queue's robustness
(that's another story).

At the moment the box has:

/       520MB           ext3
/boot   32MB            ext3
free    +/-130GB        LVM

There's a swap partition, but otherwise the rest is available.  I'm
considering something approximating the following.

/                       520MB           ext3
/boot                   32MB            ext3
/usr                    512MB           LVM/ext3
/var                    1GB             LVM/ext3
/var/lib/pgsql          512MB           LVM/ext3
/var/qmail/queue        512MB           LVM/ext3        sync
/home                   1GB             LVM/ext3
/home/mail              127GB+          LVM/ext3

All the file systems on LVs may require expansion/contraction as the
system grows, so I'd like as flexible a setup as possible.  The qmail
queue should probably be larger, rather than smaller, so that expansion
doesn't adversely affect performance.  What about adding swap space?

What size should the various PEs be, linear/striped?... what other
configurable LVM settings would suit this setup etc.  I'm a complete
newbie as far as LVM is concerned, so need valuable pointers.

The primary goal here is for flexibility and I/O performance - the
hardware RAID takes care of availability/redundancy, together with
backup media.

Thanks in advance
Dale

Re: [linux-lvm] advice sought on setting up new system

[Ken Fuchs]

Dale Gallagher wrote:

>The primary goal here is for flexibility and I/O performance - the
>hardware RAID takes care of availability/redundancy, together with
>backup media.

The location of the extents of a logical volume can have a great impact
on its I/O performance:

1) LVM does its best to make the logical extents physically contiguous,
   but pvmove(8) can be used to improve this as the need arises.

2) Here's an often neglected fact:  The performance of the outer
   cylinders of almost any hard drive (larger than a few GB) is usually
   about double that of the inner cylinders of the same drive.  (This is
   due to the fact that there are about twice as many sectors per track
   on the outside of the platter as there are on the inside of the
   platter and thus data moves past the read/write heads about twice as
   fast on the outer cylinders as the inner cylinders.)

   Therefore, to improve I/O performance, try to create often used
   logical volumes first as LVM tends to allocate lower physical extents
   first.  Otherwise, use pvmove(8) to move the more frequently accessed
   extents to the fastest (lowest) physical extents of the fastest LVM
   physical volume.  It is best to use (almost) whole disks as LVM
   physical volumes, since smaller physical volumes (partitions) would be
   either fast, average, or slow and not the full spectrum of fast
   through slow.  The least accessed logical extents should be moved to
   the highest numbered (slow) physical extents of the slowest disks.

3) LVM striping can also be used to improve I/O performance over two or
   more spindles (disks).  Avoid LVM striping over two physical volumes
   on the same spindle (disk), since this reduces performance as the
   system waits for seeks between these two physical volumes (for a
   simple sequential access).

Sincerely,

Ken Fuchs <kfuchs@winternet.com>

Re: [linux-lvm] advice sought on setting up new system

[Dale Gallagher]

Thanks for the response Ken - I've been swamped, so
appologies for the belated reply.

On Fri, 6 Feb 2004 Ken Fuchs wrote:
> The location of the extents of a logical volume can
> have a great impact on its I/O performance:
> 
> 1) LVM does its best to make the logical extents
> physically contiguous, but pvmove(8) can be used to
> improve this as the need arises.
> 
> 2) Here's an often neglected fact:  The performance of
> the outer cylinders of almost any hard drive (larger
> than a few GB) is usually about double that of the
> inner cylinders of the same drive.

Surely under hardware raid this doesn't apply, as the
location of the data is mirrored/striped across many
disks, so the physical location of data is no longer
obvious?

> 3) LVM striping can also be used to improve I/O
> performance over two or more spindles (disks).
> Avoid LVM striping over two physical volumes on
> the same spindle (disk), since this reduces
> performance as the system waits for seeks between
> these two physical volumes (for a simple sequential
> access).

Now by using harware raid, theoretically there may
well be an additional performance gain (increasing in
proportion to the number of spindles of course). 

My original primary concern is the performance impact
the system will experience, particularly if I append
logical extents which are non-contiguous to a LV. Say
for example, the original setup was:

PEs(0-100) for LV /dev/mapper/v0-var-qmail-queue
PEs (101-1000) for LV /dev/mapper/v0-home-mail

And then I find at a later stage that my queue is too
small, so append PEs (1001-1100) to the original LV
/dev/mapper/v0-var-qmail-queue.

Will the performance impact be noticeable, taking into
consideration that each file written to the queue is
synced, together with hardware raid and U320 SCSI disks?

I know, it's difficult to assume things up-front, but
I don't want to rush into using LVM for the wrong
reasons... on that note, with hardware raid and the
setup I posted originally, what are the clear practical
benefits of using LVM, other than reducing the possible
future need to sym-link directories onto other file-
systems?

I have LVM2/Linux-2.6.1 running on my laptop, but that's
the only experience I have with LVM, so do not really
know whether or not to go with it in a production
environment, as I haven't the experience.

Anyone care to enlighten me further?  I'm not entirely
convinced that LVM is the right way to go yet.  Why
_should_ I use it?  Will the benefits out-weigh the
negative aspects?

Thanks.
Dale

Re: [linux-lvm] advice sought on setting up new system

[Ken Fuchs]

>Ken Fuchs wrote:

>> 2) Here's an often neglected fact:  The performance of
>> the outer cylinders of almost any hard drive (larger
>> than a few GB) is usually about double that of the
>> inner cylinders of the same drive.

Dale Gallagher wrote:

>Surely under hardware raid this doesn't apply, as the
>location of the data is mirrored/striped across many
>disks, so the physical location of data is no longer
>obvious?

Hardware RAID just makes the physical location of the data
just a little harder to determine.  Hardware RAID simply
transforms n disks of the same size into a larger virtual
disk.  Hardware RAID will almost certainly combine the
outer (lowest) cylinders of all disks together to form
the lowest cylinders of the virtual disk and similarly
it will combine the inner (highest) cylinders of all
disks to form the highest cylinders of the virtual disk.
Thus, the lowest cylinders of the hardware RAID virtual
disk will be about twice as fast as the highest cylinders
of the virtual disk in exactly the same way it would
apply to any individual disk in the array.  Of course
the hardware RAID performance will be much better than
a single drive, depending on the actual application.

However, it is possible (though very unlikely) that a
hardware RAID is set up in a way that eliminates the
performance differential between lower and higher
cylinders for example by matching lower cylinders of
the odd drives with higher cylinders of the even
drives.  This is an unlikely set up, since the maximum
performance of such a RAID would be the same as the
average performance of a normal hardware RAID or only
67% of the maximum performance of a normal RAID.

Thus, for the typical hardware RAID, one would want to
allocate low physical extents to logical volumes needing
high performance and higher physical extents to logical
volumes with lower performance demands.

>> 3) LVM striping can also be used to improve I/O
>> performance over two or more spindles (disks).
>> Avoid LVM striping over two physical volumes on
>> the same spindle (disk), since this reduces
>> performance as the system waits for seeks between
>> these two physical volumes (for a simple sequential
>> access).

>Now by using hardware raid, theoretically there may
>well be an additional performance gain (increasing in
>proportion to the number of spindles of course). 

Yes, there should be a significant performance increase
depending on the application and how well its data
accesses are distributed over all spindles in the RAID.

However, multiple simultaneous applications may cause
more long seeks and in unlikely worst case scenarios,
RAID can even perform worst than a like number of
independent hard drives, if for example every access
requires a (long) seek on all actuators in the RAID.

>My original primary concern is the performance impact
>the system will experience, particularly if I append
>logical extents which are non-contiguous to a LV. Say
>for example, the original setup was:

>PEs(0-100) for LV /dev/mapper/v0-var-qmail-queue
>PEs (101-1000) for LV /dev/mapper/v0-home-mail

>And then I find at a later stage that my queue is too
>small, so append PEs (1001-1100) to the original LV
>/dev/mapper/v0-var-qmail-queue.

>Will the performance impact be noticeable, taking into
>consideration that each file written to the queue is
>sync-ed, together with hardware raid and U320 SCSI disks?

For sequential queue access, the physical extent gap should
have no noticeable impact on performance.  The performance
impact on an application doing random access on a logical
volume with a large physical extent gap could be noticeable.

>I know, it's difficult to assume things up-front, but
>I don't want to rush into using LVM for the wrong
>reasons... on that note, with hardware raid and the
>setup I posted originally, what are the clear practical
>benefits of using LVM, other than reducing the possible
>future need to sym-link directories onto other file-
>systems?

Not all applications perform properly when symbolic links
are used in this fashion.

Hard drives can be added and removed from the volume group(s)
as needed to increase capacity and remove old drives.
Logical volumes or parts of them can be resized smaller or
larger and moved between and within physical volumes to
facilitate removal of a drive or act on a performance
consideration.

Sincerely,

Ken Fuchs <kfuchs@winternet.com>

Re: [linux-lvm] advice sought on setting up new system

[Joe Thornber]

On Wed, Feb 11, 2004 at 11:28:37PM +0200, Dale Gallagher wrote:
> PEs(0-100) for LV /dev/mapper/v0-var-qmail-queue
> PEs (101-1000) for LV /dev/mapper/v0-home-mail
> 
> And then I find at a later stage that my queue is too
> small, so append PEs (1001-1100) to the original LV
> /dev/mapper/v0-var-qmail-queue.

The extent size should be chosen to be a convenient size for the
system administrator.  Typically 4-16M these days; it tends to get
bigger as hard disks grow.  So the extent size is going to be far too
large for you to worry about files being broken across extent
boundaries.  Think about the percentage of ios that will incur this
extra seek that you are concerned about.

- Joe