From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx1.redhat.com (ext-mx11.extmail.prod.ext.phx2.redhat.com [10.5.110.16]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t2BD5iC8031818 for ; Wed, 11 Mar 2015 09:05:44 -0400 Received: from mail.webx.cz (mail.webx.cz [109.123.222.201]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t2BD5eFE003654 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 11 Mar 2015 09:05:41 -0400 Received: from localhost (localhost [127.0.0.1]) by mail.webx.cz (Postfix) with ESMTP id AE2D740665 for ; Wed, 11 Mar 2015 14:05:38 +0100 (CET) Received: from mail.webx.cz ([127.0.0.1]) by localhost (mail1.webx.cz [127.0.0.1]) (amavisd-new, port 10042) with LMTP id ldGreP3i1Z95 for ; Wed, 11 Mar 2015 14:05:38 +0100 (CET) Received: from libor-nb.localnet (remote.bcom.cz [88.208.120.34]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.webx.cz (Postfix) with ESMTPSA id 78FA23FEC9 for ; Wed, 11 Mar 2015 14:05:38 +0100 (CET) From: Libor =?utf-8?B?S2xlcMOhxI0=?= Date: Wed, 11 Mar 2015 14:05:37 +0100 Message-ID: <1586988.am8mh8BZtm@libor-nb> In-Reply-To: <21758.64050.13647.314608@quad.stoffel.home> References: <2134825.VZEVSUbKJK@libor-nb> <21758.64050.13647.314608@quad.stoffel.home> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="nextPart11798727.ikoFWsyqai" Content-Transfer-Encoding: 7Bit Subject: Re: [linux-lvm] Removing disk from raid LVM Reply-To: LVM general discussion and development List-Id: LVM general discussion and development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , List-Id: To: linux-lvm@redhat.com This is a multi-part message in MIME format. --nextPart11798727.ikoFWsyqai Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hello John, On =C3=9At 10. b=C5=99ezna 2015 10:05:38 John Stoffel wrote: > Libor> web have 4x3TB disks in LVM for backups and I setup per > Libor> customer/per "task" LV type raid5 on it. >=20 > Can you post the configuration details please, since they do matter. > It would seem to me, that it would be better to use 'md' to create th= e > underlying RAID5 device, and then use LVM on top of that /dev/md0 to > create the customer LV(s) as needed. I used mdraid everytime before (in fact, OS is on another disks on mdra= id). But i=20 really loved idea/flexibility of raid in LVM and wanted to try it. >=20 > Libor> Last week, smartd started to alarm us, that one of the disk > Libor> will soon go away. >=20 > Libor> So we shut down the computer, replaced disk and then i used > Libor> vgcfgrestore on new disk to restore metadata. >=20 > You should have shutdown the system, added in a new disk, and then > rebooted the system. At that point you would add the new disk into > the RAID5, and then fail the dying disk. It would be transparent to > the LVM setup and be much safer. >=20 I see, but there is no physical space for extra disk. Maybe external di= sk should=20 do the trick, but it would take hours to migrate data and server is in = remote=20 housing facility. > I'd also strongly advise you to get RAID6 setup and have a hot spare > also setup, so that you don't have this type of issue in the future. >=20 > Libor> Result was, that some LVs came up with damaged filesystem, som= e > Libor> didn't came up at all with messages like (one of rimage and > Libor> rmeta was "wrong", when i used KVPM util, it was type "virtual= " >=20 > This sounds very much like you just lost a bunch of data, which RAID5= > shouldn't do. So please post the details of your setup, starting at > the disk level and moving up the stack to the filesystem(s) you have > mounted for backups. We don't need the customer names, etc, just the= > details of the system. >=20 System is Dell T20. Backup disks are connected over=20 00:1f.2 SATA controller: Intel Corporation Lynx Point 6-port SATA Contr= oller 1=20 [AHCI mode] (rev 04) System disks are connected over 04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SA= TA=20 6Gb/s Controller (rev 10) First three are 3TB sata discs, 3,5'', 7200RPM [0:0:0:0] disk ATA TOSHIBA MG03ACA3 n/a /dev/sda=20 [1:0:0:0] disk ATA ST3000NM0033-9ZM n/a /dev/sdb=20 [2:0:0:0] disk ATA ST3000NM0033-9ZM n/a /dev/sdg=20 [3:0:0:0] disk ATA TOSHIBA MG03ACA3 n/a /dev/sdd Remaining two are 500GB 2,5'' disks for system [6:0:0:0] disk ATA ST9500620NS n/a /dev/sde=20 [8:0:0:0] disk ATA ST9500620NS n/a /dev/sdf=20 System is on mdraid (raid1) + LVM On top of LVs, we use ext4 for OS and XFS for backup/customer disks. > Also, which version of lvm, md, linux kernel, etc are you using? The= > more details the better. It's Debian Wheezy, with kernel 3.14(.14) System LVM is LVM version: 2.02.95(2) (2012-03-06) Library version: 1.02.74 (2012-03-06) Driver version: 4.27.0 I also use another copy of lvm, for raid operations (creating LV, exten= ding LVs,=20 show progress of resync) ... LVM version: 2.02.104(2) (2013-11-13) Library version: 1.02.83 (2013-11-13) Driver version: 4.27.0 Should the problem be, that VG/LVs are first constructed using system o= ld utils? I think, i could upgrade whole system to Debian Jessie as last resort o= peration. This should bring kernel to version 3.16 and lvm to 2.02.111 Thanks for your reply With regards, Libor > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ --nextPart11798727.ikoFWsyqai Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="utf-8"

Hel= lo John,

 

On = =C3=9At 10.=C2=A0b=C5=99ezna=C2=A02015 10:05:38 John Stoffel wrote:

=

>= ; Libor> web have 4x3TB disks in LVM for backups and I setup per

=

>= ; Libor> customer/per "task" LV type raid5 on it.

>= ;

>= ; Can you post the configuration details please, since they do matter.<= /p>

>= ; It would seem to me, that it would be better to use 'md' to create th= e

>= ; underlying RAID5 device, and then use LVM on top of that /dev/md0 to<= /p>

>= ; create the customer LV(s) as needed.

 

I u= sed mdraid everytime before (in fact, OS is on another disks on mdraid)= . But i really loved idea/flexibility of raid in LVM and wanted to try = it.

 

>= ;

>= ; Libor> Last week, smartd started to alarm us, that one of the disk=

>= ; Libor> will soon go away.

>= ;

>= ; Libor> So we shut down the computer, replaced disk and then i used=

>= ; Libor> vgcfgrestore on new disk to restore metadata.

>= ;

>= ; You should have shutdown the system, added in a new disk, and then

>= ; rebooted the system. At that point you would add the new disk into

>= ; the RAID5, and then fail the dying disk. It would be transparent to<= /p>

>= ; the LVM setup and be much safer.

>= ;

 

I s= ee, but there is no physical space for extra disk. Maybe external disk = should do the trick, but it would take hours to migrate data and server= is in remote housing facility.

 

>= ; I'd also strongly advise you to get RAID6 setup and have a hot spare<= /p>

>= ; also setup, so that you don't have this type of issue in the future.<= /p>

>= ;

>= ; Libor> Result was, that some LVs came up with damaged filesystem, = some

>= ; Libor> didn't came up at all with messages like (one of rimage and=

>= ; Libor> rmeta was "wrong", when i used KVPM util, it was = type "virtual"

>= ;

>= ; This sounds very much like you just lost a bunch of data, which RAID5=

>= ; shouldn't do. So please post the details of your setup, starting at<= /p>

>= ; the disk level and moving up the stack to the filesystem(s) you have<= /p>

>= ; mounted for backups. We don't need the customer names, etc, just the=

>= ; details of the system.

>= ;

 

Sys= tem is Dell T20.

 

Bac= kup disks are connected over

00:= 1f.2 SATA controller: Intel Corporation Lynx Point 6-port SATA Controll= er 1 [AHCI mode] (rev 04)

Sys= tem disks are connected over

04:= 00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA = 6Gb/s Controller (rev 10)

 

Fir= st three are 3TB sata discs, 3,5'', 7200RPM

[0:= 0:0:0] disk ATA TOSHIBA MG03ACA3 n/a /dev/sda

[1:= 0:0:0] disk ATA ST3000NM0033-9ZM n/a /dev/sdb

[2:= 0:0:0] disk ATA ST3000NM0033-9ZM n/a /dev/sdg

[3:= 0:0:0] disk ATA TOSHIBA MG03ACA3 n/a /dev/sdd

 

Rem= aining two are 500GB 2,5'' disks for system

[6:= 0:0:0] disk ATA ST9500620NS n/a /dev/sde

[8:= 0:0:0] disk ATA ST9500620NS n/a /dev/sdf

 

Sys= tem is on mdraid (raid1) + LVM

 

On = top of LVs, we use ext4 for OS and XFS for backup/customer disks.

 

>= ; Also, which version of lvm, md, linux kernel, etc are you using? The=

>= ; more details the better.

 

It'= s Debian Wheezy, with kernel 3.14(.14)

Sys= tem LVM is

LV= M version: 2.02.95(2) (2012-03-06)

L= ibrary version: 1.02.74 (2012-03-06)

D= river version: 4.27.0

 

I a= lso use another copy of lvm, for raid operations (creating LV, extendin= g LVs, show progress of resync) ...

L= VM version: 2.02.104(2) (2013-11-13)

L= ibrary version: 1.02.83 (2013-11-13)

D= river version: 4.27.0

 

 

Sho= uld the problem be, that VG/LVs are first constructed using system old = utils?

I t= hink, i could upgrade whole system to Debian Jessie as last resort oper= ation.

Thi= s should bring kernel to version 3.16 and lvm to 2.02.111

 

Tha= nks for your reply

 

Wit= h regards,

Lib= or

 

 

 

 

>= ; _______________________________________________

>= ; linux-lvm mailing list

>= ; linux-lvm@redhat.com

>= ; https://www.redhat.com/mailman/listinfo/linux-lvm

>= ; read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--nextPart11798727.ikoFWsyqai--