[linux-lvm] LVM on SATA/PATA disks

[linux-lvm] LVM on SATA/PATA disks

[Bertrand Renuart]

Dear all,

I always have been told (S/P)ATA disks should somehow be avoided in server
systems. This mainly because the high rate of interrupt requests generated
by the IO subsystem will constantly interrupt the CPU which then becomes
unavailable for other server tasks. SCSI disks should be preferred mainly
for that reason.

Can someone confirm this statement?

FYI, we have to upgrade one of our development machine - hence the question.
This machine is mainly used to run unattended software builds -> moves large
quantity of small files and requires lots of CPU.

(sorry if my question is "stupid" or has already been asked many times - but
couldn't find any hint anywere)

Thanks

Re: [linux-lvm] LVM on SATA/PATA disks

[Stuart D. Gathman]

On Sat, 12 May 2007, Bertrand Renuart wrote:

> I always have been told (S/P)ATA disks should somehow be avoided in server
> systems. This mainly because the high rate of interrupt requests generated
> by the IO subsystem will constantly interrupt the CPU which then becomes
> unavailable for other server tasks. SCSI disks should be preferred mainly
> for that reason.
> 
> Can someone confirm this statement?

The interrupt rate has nothing to do with the type of disk, and a lot to
do with the controller.  There is a CPU difference between $50 
consumer IDE/SATA adapters, and $300 server grade IDE/SATA adapters.
You'll want the controller to support fast DMA at minimum.

> FYI, we have to upgrade one of our development machine - hence the question.
> This machine is mainly used to run unattended software builds -> moves large
> quantity of small files and requires lots of CPU.

However, even the $50 adapters are way faster than what you could get
10 years ago.  I would start with a higher end $50+ adapter (you want
it reliable).  If you get a lot of "system" CPU time during heavy disk
IO, then the $300 adapter will relieve that.  For example, a super cheap
ATA adapter was giving us 25% system CPU on backups on a 1.8Ghz Celeron.
Replacing that with a more expensive (but < $100) IDE adapter reduced 
system time to 1% or so.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] LVM on SATA/PATA disks

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On Sat, 2007-05-12 at 21:12 -0400, Stuart D. Gathman wrote:
> 
> The interrupt rate has nothing to do with the type of disk, and a lot to
> do with the controller.  There is a CPU difference between $50 
> consumer IDE/SATA adapters, and $300 server grade IDE/SATA adapters.
> You'll want the controller to support fast DMA at minimum.

I thought the biggest thing that SCSI had that IDE didn't was SCSI's
ability to shovel an ass-barn-load of data to a disk and the disk would
go deal with it, giving up the SCSI bus so that another disk could be
shovelled another ass-barn-load of data to go and deal with, and so on.

Of course one has to remember it takes much much more time for the disk
to actually write (or read as the case may be) the data to media than it
takes to shovel the data to the disk (which it would cache locally
before writing to actual media) on the bus.

The contrast with IDE (or PATA as I guess the trendy name is), again as
I always thought was that the IDE bus was not available for use while a
disk was still pending a media I/O operation, so that with multiple
devices, you could not leverage the I/O of the IDE bus using multiple
devices, essentially in parallel.  I guess this is where having system
with multiple IDE buses and only putting a single device per bus grew
from.

Of course on a SCSI bus you can put many many disks -- to the capacity
of the bus -- or more, but that would be silly.

How does SATA fit in with all of this?  Is it basically the same
limitations on the bus as IDE/PATA, so that you'd really not want to put
more than 1 device per bus?

b.

-- 
My other computer is your Microsoft Windows server.

Brian J. Murrell

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] LVM on SATA/PATA disks

[Stuart D. Gathman]

On Sat, 12 May 2007, Brian J. Murrell wrote:

> On Sat, 2007-05-12 at 21:12 -0400, Stuart D. Gathman wrote:
> > 
> > The interrupt rate has nothing to do with the type of disk, and a lot to
> > do with the controller.  There is a CPU difference between $50 
> > consumer IDE/SATA adapters, and $300 server grade IDE/SATA adapters.
> > You'll want the controller to support fast DMA at minimum.
> 
> I thought the biggest thing that SCSI had that IDE didn't was SCSI's
> ability to shovel an ass-barn-load of data to a disk and the disk would
> go deal with it, giving up the SCSI bus so that another disk could be
> shovelled another ass-barn-load of data to go and deal with, and so on.
> 
> ...
> 
> The contrast with IDE (or PATA as I guess the trendy name is), again as
> I always thought was that the IDE bus was not available for use while a
> disk was still pending a media I/O operation, so that with multiple
> devices, you could not leverage the I/O of the IDE bus using multiple
> devices, essentially in parallel.  I guess this is where having system
> with multiple IDE buses and only putting a single device per bus grew
> from.

We always put exactly one IDE disk per channel for that very reason.
You are correct that with 2 disks on the same channel, only one can
be active at a time.  So don't do that.  A $50 IDE PCI card give you 2 IDE
channels - for 2 disks in high performance mode (suitable for mirroring).
Buy two cards for 4 disks.  Or a $300 server card for 8 disks.

> How does SATA fit in with all of this?  Is it basically the same
> limitations on the bus as IDE/PATA, so that you'd really not want to put
> more than 1 device per bus?

SATA mandates at most 1 disk per channel, making the issue moot.  It is
still true that there is only one active disk on a bus.  But then there
is only one disk on a bus.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] LVM on SATA/PATA disks

[David Brown]

Stuart D. Gathman wrote:

>> How does SATA fit in with all of this?  Is it basically the same
>> limitations on the bus as IDE/PATA, so that you'd really not want to put
>> more than 1 device per bus?
>
> SATA mandates at most 1 disk per channel, making the issue moot.  It is
> still true that there is only one active disk on a bus.  But then there
> is only one disk on a bus.

Newer SATA drives, with proper newer controllers and proper device
driver support will support NCQ (native command queueing), which
allows the drive to re-order requests.  It appears that the Linux ACHI
and Nvidia SATA drivers support this capability in recent kernels.

Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires
filesystem and driver support of write barriers for reliability.
Write barriers are not implement in DM, hence LVM, so there is a
reliability risk in going with this kind of solution.  Depending on
the filesystem this can result in power failures resulting in files
having inconsistent data.

Dave

Re: [linux-lvm] LVM on SATA/PATA disks

[Les Mikesell]

David Brown wrote:
> Stuart D. Gathman wrote:
> 
>>> How does SATA fit in with all of this?  Is it basically the same
>>> limitations on the bus as IDE/PATA, so that you'd really not want to put
>>> more than 1 device per bus?
>> SATA mandates at most 1 disk per channel, making the issue moot.  It is
>> still true that there is only one active disk on a bus.  But then there
>> is only one disk on a bus.
> 
> Newer SATA drives, with proper newer controllers and proper device
> driver support will support NCQ (native command queueing), which
> allows the drive to re-order requests.  It appears that the Linux ACHI
> and Nvidia SATA drivers support this capability in recent kernels.
> 
> Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires
> filesystem and driver support of write barriers for reliability.
> Write barriers are not implement in DM, hence LVM, so there is a
> reliability risk in going with this kind of solution.  Depending on
> the filesystem this can result in power failures resulting in files
> having inconsistent data.

Are you saying that LVM on SCSI is not safe in this scenario?

-- 
   Les Mikesell
    lesmikesell@gmail.com

Re: [linux-lvm] LVM on SATA/PATA disks

[Stuart D. Gathman]

On Sun, 13 May 2007, Les Mikesell wrote:

> > Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires
> > filesystem and driver support of write barriers for reliability.
> > Write barriers are not implement in DM, hence LVM, so there is a
> > reliability risk in going with this kind of solution.  Depending on
> > the filesystem this can result in power failures resulting in files
> > having inconsistent data.
> 
> Are you saying that LVM on SCSI is not safe in this scenario?

If out of order writes are enabled, then your server should hold power
to the disk drives for part of a second after disabling further writes
in software.  LVM is a low risk because LVM changes are comparatively
rare, and the modification window is small.  But, theoretically, if
you lost power right at the instant you pressed return on lvcreate,
lvextend, or lvremove, on a system with busy disk io, you could corrupt
the LVM metadata.

Having a UPS and doing a shutdown solves the problem.  However, if your
server enables out of order writes with native queuing, and loses power
without a shutdown, and doesn't keep drives powered long enough after 
stopping the CPUs (this is why high end servers now have a microprocessor
dedicated to power sequencing), then you should not rely on filesystem
journalling, and should do an fsck at reboot.  FS journals rely on physical
writes taking place in-order, or at least having 'sync' calls to finish
previous logical writes before starting a new one (write barrier).
There is a similar issue with creating a database on top of a file.
For example, you have to use 'fsync' after writing a checkpoint, before
beginning updates.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

RE: [linux-lvm] LVM on SATA/PATA disks

[Bertrand Renuart]

> However, even the $50 adapters are way faster than what you could get
> 10 years ago.  I would start with a higher end $50+ adapter (you want
> it reliable).  If you get a lot of "system" CPU time during heavy disk
> IO, then the $300 adapter will relieve that.  For example, a super cheap
> ATA adapter was giving us 25% system CPU on backups on a 1.8Ghz Celeron.
> Replacing that with a more expensive (but < $100) IDE adapter reduced
> system time to 1% or so.
> 

What adapter would you recommend for a descent setup?

Re: [linux-lvm] LVM on SATA/PATA disks

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Brown wrote:
> Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires
> filesystem and driver support of write barriers for reliability.
> Write barriers are not implement in DM, hence LVM, so there is a

Only the dm-mpath and dm-snap targets treat barrier bios specially (they
return EOPNOTSUPP). Other dm targets pass the barriers on to the
underlying devices unchanged.

Regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFGSB4y6YSQoMYUY94RApajAJ4kyrw+DDEjj3WiXPOysO5J6KKmOQCgxC0M
oQrw2KA74j+ytiKFoMLPQdg=
=CZvy
-----END PGP SIGNATURE-----

Re: [linux-lvm] LVM on SATA/PATA disks

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> If out of order writes are enabled, then your server should hold power
> to the disk drives for part of a second after disabling further writes
> in software.  LVM is a low risk because LVM changes are comparatively
> rare, and the modification window is small.  But, theoretically, if
> you lost power right at the instant you pressed return on lvcreate,
> lvextend, or lvremove, on a system with busy disk io, you could corrupt
> the LVM metadata.

LVM metadata I/O does not pass through device-mapper unless you are
stacking volume groups so I'm not sure how your comment regarding
barriers is relevant here. It's also only going to affect devices under
snapshot and multipath mappings.

How does this differ from updating an MSDOS or GPT partition table, for
example?

Kind regards,

Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFGSB/v6YSQoMYUY94RAubIAJ4vcB/tqherGhmW/M6n1wcO5KGungCffw1w
ISfLoRVk6CP3EryIzHTolXA=
=g18w
-----END PGP SIGNATURE-----

RE: [linux-lvm] LVM on SATA/PATA disks

[Stuart D. Gathman]

On Sun, 13 May 2007, Bertrand Renuart wrote:

> 
> What adapter would you recommend for a descent setup?

The <$100 types keep changing availability.  Here are the
lspci ids of some <$100 IDE adapters that are working great with low
overhead (mirroring with 2 disks attached, one on each channel):

105a:4d30	Promise Technology, Inc. 20267 (rev 02)
        Subsystem: Promise Technology, Inc. Ultra100  (ATA100 is old, I know)

1095:0680	CMD Technology Inc PCI0680 (rev 01)

1095:0680	Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller (rev 02)
        Subsystem: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller
		(tricky to boot from due to wierd BIOS drive assignments)

I've bought 2 IDE PCI cards that didn't work well.  I pitched them and
did not record the numbers.  Shame on me, or maybe QA isn't perfect on
the cheap cards and I got a bum unit.

I haven't tried any add-on SATA boards yet.

Just buy one and try it.  Taiwan is better than China.
If you need something repeatable, you'll have to go with a $300 server
kind such as what Dell sells.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

RE: [linux-lvm] LVM on SATA/PATA disks

[Daniel Davidson]

For linux, I prefer anything 3ware makes for SATA or PATA.  I have not
been happy with promise boards.  The newer higher end boards are
expensive, but you can buy the older boards for significantly less.
With the 3ware boards you could even avoid having to use lvm since they
have raid included.

Dan

On Sun, 2007-05-13 at 22:25 +0200, Bertrand Renuart wrote:
> > However, even the $50 adapters are way faster than what you could get
> > 10 years ago.  I would start with a higher end $50+ adapter (you want
> > it reliable).  If you get a lot of "system" CPU time during heavy disk
> > IO, then the $300 adapter will relieve that.  For example, a super cheap
> > ATA adapter was giving us 25% system CPU on backups on a 1.8Ghz Celeron.
> > Replacing that with a more expensive (but < $100) IDE adapter reduced
> > system time to 1% or so.
> > 
> 
> What adapter would you recommend for a descent setup?
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM on SATA/PATA disks

[David Brown]

Bryn M. Reeves wrote:
> David Brown wrote:
> > Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires
> > filesystem and driver support of write barriers for reliability.
> > Write barriers are not implement in DM, hence LVM, so there is a
>
> Only the dm-mpath and dm-snap targets treat barrier bios specially (they
> return EOPNOTSUPP). Other dm targets pass the barriers on to the
> underlying devices unchanged.

A simple lvm volume, mounted with XFS gives the following kernel
message:

 Filesystem "dm-11": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem dm-11

so I'm fairly sure that the write barriers don't work.  This is the
2.6.20 kernel.  XFS is the only filesystem nice enough to warn you
that the result might not be as reliable.

Dave

Re: [linux-lvm] LVM on SATA/PATA disks

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Brown wrote:
> A simple lvm volume, mounted with XFS gives the following kernel
> message:
> 
>  Filesystem "dm-11": Disabling barriers, not supported by the underlying
> device
> XFS mounting filesystem dm-11

So it does. I don't use XFS on my boxes & hadn't noticed this before.

Turns out that xfs is checking barrier capabilities using queue->ordered
in the device struct. If it has QUEUE_ORDERED_NONE, barriers are
disabled (fs/xfs/linux-2.6/xfs_super.c).

I'll look into why this is the case as given this check there shouldn't
be any need for snap & mpath to check for barrier bios themselves.

The patches that introduced those checks were submitted after problems
were seen with barriers on these target types, so something was clearly
submitting them at some stage.

Kind regards,
Bryn.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFGScUv6YSQoMYUY94RAh3sAKCpcTdyvLKpwni1VYssyo2cORBG6ACbBrbl
xlJIlDXDdZGNAX5Tmjbt7RI=
=yaL7
-----END PGP SIGNATURE-----