[linux-lvm] Isolation of volume groups

[linux-lvm] Isolation of volume groups

[Fredrik Tolf]

Dear list,

I have a device-mapper configuration with two volume groups named, for 
reference, "ravol" and "datvol". The ravol VG has two LVs with XFS 
filesystems on them, and the datvol VG has two LVs, one with an XFS 
filesystem, and one with ReiserFS.

Last night, one of the PVs in datvol had a transient SATA link failure and 
popped in and out of existence for a little while, which caused datvol and 
the LVs on it to fail. No permanent damage seems to have occurred, though, 
so I'm not too worried about that. I could bring datvol down and up again 
to make it work, so I guess everything worked as should be expected.

What concerns me a little, however, is that ravol also seems to have been 
oddly affected by the failure of datvol. At times, the filesystems on it 
could hang for seconds, not responding; and while it was responding, XFS 
was intermittently outputting messages like

May 28 14:58:58 nerv kernel: [30350.996032] xfs_force_shutdown(dm-33,0x1) called from line 335 of file /build/buildd-linux-2.6_2.6.32-38-amd64-bk66e4/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/xfs/xfs_rw.c.  Return address = 0xffffffffa01df02c

or

May 28 14:51:38 nerv kernel: [29911.468028] Filesystem "dm-33": xfs_log_force: error 5 returned.

Once I brought datvol down and back up again, it stopped misbehaving, but 
I don't really understand why this would happen. Why would ravol be 
affected, at all, by what happens on datvol? Shouldn't they be isolated 
from each other?

--

Fredrik Tolf

Re: [linux-lvm] Isolation of volume groups

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/28/2012 03:33 PM, Fredrik Tolf wrote:
> Last night, one of the PVs in datvol had a transient SATA link
> failure and popped in and out of existence for a little while,
> which caused datvol and the LVs on it to fail. No permanent damage
> seems to have occurred, though, so I'm not too worried about that.
> I could bring datvol down and up again to make it work, so I guess
> everything worked as should be expected.

Are the devices for ravol and datvol on the same SATA bus or controller?

> What concerns me a little, however, is that ravol also seems to
> have been oddly affected by the failure of datvol. At times, the
> filesystems on it could hang for seconds, not responding; and while
> it was responding, XFS was intermittently outputting messages like
> 
> May 28 14:58:58 nerv kernel: [30350.996032] 
> xfs_force_shutdown(dm-33,0x1) called from line 335 of file 
> /build/buildd-linux-2.6_2.6.32-38-amd64-bk66e4/linux-2.6-2.6.32/debian/build/source_amd64_none/fs/xfs/xfs_rw.c.
>  Return address = 0xffffffffa01df02c

> May 28 14:51:38 nerv kernel: [29911.468028] Filesystem "dm-33": 
> xfs_log_force: error 5 returned.

Errno 5 is EIO:

/usr/include/asm-generic/errno-base.h:#define	EIO		 5	/* I/O error */

So something below XFS returned EIO. This could happen due to IO
timeouts caused by link reset and error recovery on the other SATA device.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/DlSQACgkQ6YSQoMYUY95M4wCfVZ0s8rs9NRbFDa1ykaOkdMHb
MOAAmwem/NStQc2i6hlIj2Uzpkz1onU5
=oR3I
-----END PGP SIGNATURE-----

Re: [linux-lvm] Isolation of volume groups

[Fredrik Tolf]

On Mon, 28 May 2012, Bryn M. Reeves wrote:
> On 05/28/2012 03:33 PM, Fredrik Tolf wrote:
>> Last night, one of the PVs in datvol had a transient SATA link
>> failure and popped in and out of existence for a little while,
>> which caused datvol and the LVs on it to fail. No permanent damage
>> seems to have occurred, though, so I'm not too worried about that.
>> I could bring datvol down and up again to make it work, so I guess
>> everything worked as should be expected.
>
> Are the devices for ravol and datvol on the same SATA bus or controller?

Well, it's a bit mixed. The PVs in datvol are spread over two SATA 
controllers, one of which also controls the ravol drives. However, the 
drive that failed was on the controller which is not used by ravol, and 
the actual PV in ravol (there's only one) is an MD volume (a RAID 1 
array), and MD itself never reported any errors.

--

Fredrik Tolf

Re: [linux-lvm] Isolation of volume groups

[Fredrik Tolf]

On Mon, 28 May 2012, Bryn M. Reeves wrote:
> On 05/28/2012 03:33 PM, Fredrik Tolf wrote:
>> May 28 14:51:38 nerv kernel: [29911.468028] Filesystem "dm-33":
>> xfs_log_force: error 5 returned.

I think I must apologize. "dm-33" actually turns out to be the XFS 
filesystem that was on datvol, so I probably misinterpreted the symptoms 
altogether. The only remaing issue, then, should be that the ravol 
filesystems froze in periods, but that might be attributable to the high 
sync load caused by syslog constantly receiving messages from ReiserFS 
(I've noticed that XFS displays notoriously bad performance when syncing).

There probably never was a problem, then, so I'm sorry for the noise.

--

Fredrik Tolf