[linux-lvm] pvmove and 100% IO Wait

[linux-lvm] pvmove and 100% IO Wait

[Christian Völker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yohoo!

I've some issues with pvmove.

It is very likely to appear during a pvmove operation for my server to
spike to 100% IO waits.

If this happens, no IO on the LVs is possible, even the rest of the
server is accessible without any issues.

I can't predict when it happens, but it happens...

pvmove started. Some minutes or hours later the io wait on the server
goes up to 100% (or 3 cores of the quad core are going to 100%).

Running a RedHat EL 5.x (CentOS) with all current updates. The storage
consists of two 3Ware 9500 PCI-X, a USB-SATA drive and three disks
connected to the local SATA controller.

I can't see anything relevant in the logfiles. When the CPU is up, I
have to reset the server as any operation on the LV will never finish  :-(

Any clues?

Christian

Any clues?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFJFfgN0XNIYlAXmzsRAih2AKCk2ItzDHX0s4n+pCHdnom9wh4WQgCeOlba
PDfvqDf1WmgYDmPbB49Ltso=
=nvvi
-----END PGP SIGNATURE-----

Re: [linux-lvm] pvmove and 100% IO Wait

[Eugene Vilensky]

>The storage
> consists of two 3Ware 9500 PCI-X, a USB-SATA drive and three disks
> connected to the local SATA controller.

is the USB-SATA at all involved in the pvmove?

Re: [linux-lvm] pvmove and 100% IO Wait

[Christian Völker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yohoo!

>> consists of two 3Ware 9500 PCI-X, a USB-SATA drive and three disks
>> connected to the local SATA controller.
> is the USB-SATA at all involved in the pvmove?

Hm...good question. I'd say no- but I can't say for sure where any
temporary PEs have been allocated.

The last pvmove FROM the USB-SATA went without any issues (500GB!).

Is it relevant?

Cheers

Christian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFJFfuU0XNIYlAXmzsRAsTbAJ0daUInw5jMl4veCwctLdjK12h1EwCfSBQJ
In7oAFNWQkKka1KfFU2OO14=
=HDDo
-----END PGP SIGNATURE-----

Re: [linux-lvm] pvmove and 100% IO Wait

[Eugene Vilensky]

> The last pvmove FROM the USB-SATA went without any issues (500GB!).
>
> Is it relevant?

Might not be, but if an operation involves multiple disks it will
probably go at the pace of the slowest disk, which for me has always
been USB...

Are you 100% certainty the devices underlying all PVs are good, that
you can dd 100% successfully?

Re: [linux-lvm] pvmove and 100% IO Wait

[Christian Völker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yohoo!

> Might not be, but if an operation involves multiple disks it will
> probably go at the pace of the slowest disk, which for me has always
> been USB...
I agree about the speed- but not with the issue of hanging. When pvmove
works the IO load is moderate (about 20% or less). But when it hangs it
jumps up to 100% on one or more cores.

> Are you 100% certainty the devices underlying all PVs are good, that
> you can dd 100% successfully?
Yes. I can do resize2fs, cp, mv and dd, too. Well, I haven't tried to dd
the whole devices as they are aprox. 1.5TB in size. But I'd say this is
fine!

GReetings

Christian

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFJFmcM0XNIYlAXmzsRAiQOAJ9Fo6SA3+yBGlZ6Ky0bKDw1sStAGACgmprm
pT+asXOQS/a0MMQ2jQoCeGo=
=eMR2
-----END PGP SIGNATURE-----

Re: [linux-lvm] pvmove and 100% IO Wait

[Jeremy Lyon]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

>
> > Might not be, but if an operation involves multiple disks it will
> > probably go at the pace of the slowest disk, which for me has always
> > been USB...
> I agree about the speed- but not with the issue of hanging. When pvmove
> works the IO load is moderate (about 20% or less). But when it hangs it
> jumps up to 100% on one or more cores.
>
>
I ran into similar issues when attempting a pvmove with clustered VG's under
CLVMD.  I had to change the clustered VG to non-clustered and then I could
do the pvmove.  You didn't mention clusters, but thought I would share my
experiences.

-Jeremy

[-- Attachment #2: Type: text/html, Size: 853 bytes --]

Re: [linux-lvm] pvmove and 100% IO Wait

[Christian Völker]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yohoo!

> I ran into similar issues when attempting a pvmove with clustered VG's
> under CLVMD.
What is CLVMD?

Anyway: It happened again and USB was not involved at all!
All SATA disks or SATA RAIDs.

$%^$%^�!!!

Any clues?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iD8DBQFJG7h80XNIYlAXmzsRAtHXAKDf0EyUvZngOLzoMoIHsWj+VxtWUQCcCqR2
E1COMYVO2FaqxB88K4wBGgY=
=+riP
-----END PGP SIGNATURE-----