* Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) @ 2016-01-26 14:49 Eric Valette 2016-01-26 23:31 ` Shaohua Li 0 siblings, 1 reply; 8+ messages in thread From: Eric Valette @ 2016-01-26 14:49 UTC (permalink / raw) To: linux-raid Hi, My raid 10 array (5 disk with one spare) was doing a resync after an upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end the /proc/mdstat was apparently complete but when rebooting, it started resycing over and over. I noticed my dmesg was totally filled with raid10 conf printout message so it was impossible to trace anything else. Did a resync test with 3.14.58 (because I knew it had worked for resync before and was still available as a boot option) and the array was correctly rebuild. Runs fine with 4.1.16 now. Please CC me as I'm not subscribed. -- eric ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-01-26 14:49 Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) Eric Valette @ 2016-01-26 23:31 ` Shaohua Li 2016-01-27 7:24 ` Eric Valette 0 siblings, 1 reply; 8+ messages in thread From: Shaohua Li @ 2016-01-26 23:31 UTC (permalink / raw) To: Eric Valette; +Cc: linux-raid On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote: > Hi, > > My raid 10 array (5 disk with one spare) was doing a resync after an > upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end > the /proc/mdstat was apparently complete but when rebooting, it started > resycing over and over. I noticed my dmesg was totally filled with raid10 > conf printout message so it was impossible to trace anything else. > > Did a resync test with 3.14.58 (because I knew it had worked for resync > before and was still available as a boot option) and the array was > correctly rebuild. > Runs fine with 4.1.16 now. > > Please CC me as I'm not subscribed. Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run a stop/reassemble, does the resync start? Thanks, Shaohua ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-01-26 23:31 ` Shaohua Li @ 2016-01-27 7:24 ` Eric Valette 2016-01-28 18:25 ` Shaohua Li 0 siblings, 1 reply; 8+ messages in thread From: Eric Valette @ 2016-01-27 7:24 UTC (permalink / raw) To: Shaohua Li; +Cc: linux-raid On 27/01/2016 00:31, Shaohua Li wrote: > On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote: >> Hi, >> >> My raid 10 array (5 disk with one spare) was doing a resync after an >> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end >> the /proc/mdstat was apparently complete but when rebooting, it started >> resycing over and over. I noticed my dmesg was totally filled with raid10 >> conf printout message so it was impossible to trace anything else. >> >> Did a resync test with 3.14.58 (because I knew it had worked for resync >> before and was still available as a boot option) and the array was >> correctly rebuild. >> Runs fine with 4.1.16 now. >> >> Please CC me as I'm not subscribed. > > Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run > a stop/reassemble, does the resync start? I'm not going to retry kernel 4.4.0 on this device as I'm no more confident about raid10 support with this 4.4 version. So on 4.1.16 : mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB) Raid Devices : 4 Total Devices : 5 Persistence : Superblock is persistent Update Time : Sun Jan 24 17:15:58 2016 State : clean Active Devices : 4 Working Devices : 5 Failed Devices : 0 Spare Devices : 1 Layout : near=2 Chunk Size : 512K Name : nas2:0 (local to host nas2) UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Events : 480 Number Major Minor RaidDevice State 0 8 17 0 active sync set-A /dev/sdb1 1 8 33 1 active sync set-B /dev/sdc1 2 8 49 2 active sync set-A /dev/sdd1 3 8 65 3 active sync set-B /dev/sde1 4 8 81 - spare /dev/sdf1 uname -a Linux nas2 4.1.16 #1 SMP Sat Jan 23 19:29:59 CET 2016 x86_64 GNU/Linux And if I stop the array manually and reboot, its no more resynced with this 4.1.16 kernel. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-01-27 7:24 ` Eric Valette @ 2016-01-28 18:25 ` Shaohua Li 2016-01-28 19:52 ` Eric Valette 0 siblings, 1 reply; 8+ messages in thread From: Shaohua Li @ 2016-01-28 18:25 UTC (permalink / raw) To: Eric Valette; +Cc: linux-raid On Wed, Jan 27, 2016 at 08:24:29AM +0100, Eric Valette wrote: > On 27/01/2016 00:31, Shaohua Li wrote: > >On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote: > >>Hi, > >> > >>My raid 10 array (5 disk with one spare) was doing a resync after an > >>upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end > >>the /proc/mdstat was apparently complete but when rebooting, it started > >>resycing over and over. I noticed my dmesg was totally filled with raid10 > >>conf printout message so it was impossible to trace anything else. > >> > >>Did a resync test with 3.14.58 (because I knew it had worked for resync > >>before and was still available as a boot option) and the array was > >>correctly rebuild. > >>Runs fine with 4.1.16 now. > >> > >>Please CC me as I'm not subscribed. > > > >Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run > >a stop/reassemble, does the resync start? > > I'm not going to retry kernel 4.4.0 on this device as I'm no more confident > about raid10 support with this 4.4 version. Thanks! I know switching to a kernel which is not working well is risky, but the info from 4.1 doesn't have enough info for debuging. I also tried to reproduce the issue locally, but no success. Did you have any other info which could help debugging, for example special config? Thanks, Shaohua ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-01-28 18:25 ` Shaohua Li @ 2016-01-28 19:52 ` Eric Valette 2016-01-31 23:00 ` Eric Valette 0 siblings, 1 reply; 8+ messages in thread From: Eric Valette @ 2016-01-28 19:52 UTC (permalink / raw) To: Shaohua Li; +Cc: linux-raid On 28/01/2016 19:25, Shaohua Li wrote: > On Wed, Jan 27, 2016 at 08:24:29AM +0100, Eric Valette wrote: >> On 27/01/2016 00:31, Shaohua Li wrote: >>> On Tue, Jan 26, 2016 at 03:49:42PM +0100, Eric Valette wrote: >>>> Hi, >>>> >>>> My raid 10 array (5 disk with one spare) was doing a resync after an >>>> upgrade to 4.4.0 from 4.1.15. The resync progress was steady and at the end >>>> the /proc/mdstat was apparently complete but when rebooting, it started >>>> resycing over and over. I noticed my dmesg was totally filled with raid10 >>>> conf printout message so it was impossible to trace anything else. >>>> >>>> Did a resync test with 3.14.58 (because I knew it had worked for resync >>>> before and was still available as a boot option) and the array was >>>> correctly rebuild. >>>> Runs fine with 4.1.16 now. >>>> >>>> Please CC me as I'm not subscribed. >>> >>> Could you please provide more info, like mdadm -D /dev/md0 in v4.4? If you run >>> a stop/reassemble, does the resync start? >> >> I'm not going to retry kernel 4.4.0 on this device as I'm no more confident >> about raid10 support with this 4.4 version. > > Thanks! I know switching to a kernel which is not working well is risky, but > the info from 4.1 doesn't have enough info for debuging. I also tried to > reproduce the issue locally, but no success. Did you have any other info which > could help debugging, for example special config? If really needed can rebuild a 4.4 kernel and boot it. What do you mean by special config? I can send you my kernel .config once rebuild (oldconfig from 4.1.16) But I doubt the problem will show up now that the array is correctly rebuild. From memory I used 4.4 kernel without problem for a few days, until it decided to resync the array for an unknown reason (standard periodic rebuild, disk block read/write error detected, other?). NB: I have a disk on the array with 56 sectors in error reported by smart and noticed read error/SATA reset sequence during first array rebuild sequence, but not in any later resync afterward. It did not even decided to use the spare disk... Let me know how I can help debugging further... -- eric ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-01-28 19:52 ` Eric Valette @ 2016-01-31 23:00 ` Eric Valette 2016-02-04 1:17 ` Shaohua Li 0 siblings, 1 reply; 8+ messages in thread From: Eric Valette @ 2016-01-31 23:00 UTC (permalink / raw) To: Shaohua Li; +Cc: linux-raid On 28/01/2016 20:52, Eric Valette wrote: > On 28/01/2016 19:25, Shaohua Li wrote: > > If really needed can rebuild a 4.4 kernel and boot it. What do you mean > by special config? I can send you my kernel .config once rebuild > (oldconfig from 4.1.16) Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB) Raid Devices : 4 Total Devices : 5 Persistence : Superblock is persistent Update Time : Sun Jan 31 23:51:44 2016 State : active, resyncing Active Devices : 4 Working Devices : 5 Failed Devices : 0 Spare Devices : 1 Layout : near=2 Chunk Size : 512K Resync Status : 0% complete Name : nas2:0 (local to host nas2) UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Events : 488 Number Major Minor RaidDevice State 0 8 17 0 active sync set-A /dev/sdb1 1 8 33 1 active sync set-B /dev/sdc1 2 8 49 2 active sync set-A /dev/sdd1 3 8 65 3 active sync set-B /dev/sde1 4 8 81 - spare /dev/sdf1 So it starts again rechecking the array mdadm -E /dev/sd[b-f]1 /dev/sdb1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Name : nas2:0 (local to host nas2) Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Raid Devices : 4 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB) Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=911 sectors State : active Device UUID : ce9d878a:37a4f3a3:936bd905:c4ed9970 Update Time : Sun Jan 31 23:56:38 2016 Checksum : c92206ec - correct Events : 489 Layout : near=2 Chunk Size : 512K Device Role : Active device 0 Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing) /dev/sdc1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Name : nas2:0 (local to host nas2) Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Raid Devices : 4 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB) Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=911 sectors State : active Device UUID : 8c89bdf8:4f3f8ace:c15b5634:7a874071 Update Time : Sun Jan 31 23:56:38 2016 Checksum : 911217a8 - correct Events : 489 Layout : near=2 Chunk Size : 512K Device Role : Active device 1 Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing) /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Name : nas2:0 (local to host nas2) Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Raid Devices : 4 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB) Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=911 sectors State : active Device UUID : 9c0655c3:dca7722e:8ecc1a17:fb4d9089 Update Time : Sun Jan 31 23:56:38 2016 Checksum : b6a6bcd3 - correct Events : 489 Layout : near=2 Chunk Size : 512K Device Role : Active device 2 Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing) /dev/sde1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Name : nas2:0 (local to host nas2) Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Raid Devices : 4 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB) Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=911 sectors State : active Device UUID : c97b767a:84d2e7e2:52557d30:51c39784 Update Time : Sun Jan 31 23:56:38 2016 Checksum : 36a6d284 - correct Events : 489 Layout : near=2 Chunk Size : 512K Device Role : Active device 3 Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing) /dev/sdf1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 6abe1f20:90c629de:fadd8dc0:ca14c928 Name : nas2:0 (local to host nas2) Creation Time : Wed Jun 20 23:56:59 2012 Raid Level : raid10 Raid Devices : 4 Avail Dev Size : 5860268943 (2794.39 GiB 3000.46 GB) Array Size : 5860268032 (5588.79 GiB 6000.91 GB) Used Dev Size : 5860268032 (2794.39 GiB 3000.46 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=911 sectors State : active Device UUID : 82667e81:a6158319:85e0282e:845eec1c Update Time : Sun Jan 31 23:51:44 2016 Checksum : a4a259e - correct Events : 488 Layout : near=2 Chunk Size : 512K Device Role : spare Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing) root@nas2:~# -- eric ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-01-31 23:00 ` Eric Valette @ 2016-02-04 1:17 ` Shaohua Li 2016-02-04 6:57 ` Eric Valette 0 siblings, 1 reply; 8+ messages in thread From: Shaohua Li @ 2016-02-04 1:17 UTC (permalink / raw) To: Eric Valette; +Cc: linux-raid On Mon, Feb 01, 2016 at 12:00:36AM +0100, Eric Valette wrote: > On 28/01/2016 20:52, Eric Valette wrote: > >On 28/01/2016 19:25, Shaohua Li wrote: > > > >If really needed can rebuild a 4.4 kernel and boot it. What do you mean > >by special config? I can send you my kernel .config once rebuild > >(oldconfig from 4.1.16) > > Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux Nothing useful. Did you try stop the array and reassemble it in 4.4 and check if resync runs? I can only imagine the reboot script didn't clean shutdown the array and trigger the issue. A stop/reassemble circle will tell us if this is the issue. Thanks, Shaohua ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) 2016-02-04 1:17 ` Shaohua Li @ 2016-02-04 6:57 ` Eric Valette 0 siblings, 0 replies; 8+ messages in thread From: Eric Valette @ 2016-02-04 6:57 UTC (permalink / raw) To: Shaohua Li; +Cc: linux-raid On 04/02/2016 02:17, Shaohua Li wrote: > On Mon, Feb 01, 2016 at 12:00:36AM +0100, Eric Valette wrote: >> On 28/01/2016 20:52, Eric Valette wrote: >>> On 28/01/2016 19:25, Shaohua Li wrote: >>> >>> If really needed can rebuild a 4.4 kernel and boot it. What do you mean >>> by special config? I can send you my kernel .config once rebuild >>> (oldconfig from 4.1.16) >> >> Linux nas2 4.4.1 #1 SMP Sun Jan 31 23:46:26 CET 2016 x86_64 GNU/Linux > > Nothing useful. Did you try stop the array and reassemble it in 4.4 and check > if resync runs? I can only imagine the reboot script didn't clean shutdown the > array and trigger the issue. A stop/reassemble circle will tell us if this is > the issue. If the issue was in the script it would happen with all the kernel version no? Si far 3.14, 3.18, 4.1 works correctly (when upgrading). Only 4.4 fails. 4.4 in fact seems to fails to boot the first time I boot it. As I have no console normally, I have to shut down via power button and then it resync. I can understand this behavior but not 1) that the resync fails each time and 2) that it fails to boot the first time I change for a 4.4 kernel. -- eric ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-02-04 6:57 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-01-26 14:49 Problem with array raid10 array resync on 4.4.0 (keeps reyncing each reboot) Eric Valette 2016-01-26 23:31 ` Shaohua Li 2016-01-27 7:24 ` Eric Valette 2016-01-28 18:25 ` Shaohua Li 2016-01-28 19:52 ` Eric Valette 2016-01-31 23:00 ` Eric Valette 2016-02-04 1:17 ` Shaohua Li 2016-02-04 6:57 ` Eric Valette
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).