* RE: Checkarray doesn't seem to do anything
From: rodman @ 2016-03-02 0:50 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: linux-raid
It appears this is not an Mdadm bug but is in fact kernel/dash issue as
you noted, Piergiorgio.
From the bug tracker here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787950
"This is a kernel/dash issue. In kernel 4.0, the md sysfs implementation
does not work with partial reads from /sys/block/md0/md/sync_action;
the read syscall returns the entire contents of the file even if you
try to read less. Dash has an inefficient implementation which reads
input one byte at a time, and fails when it hits the kernel
misbehavior."
The work around is what you said. Substitute bash for sh at the
beginning of the script. Unfortunately that doesn't work for me as I
get this error when trying to run checkarray now:
"checkarray: E: /sys/block/md0/md/sync_action not writeable."
I'll do some research to see what's that's about. I'm glad to know its
not a Mdadm issue per say.
Rodman Frowert
-------- Original Message --------
Subject: Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
Date: Tue, March 01, 2016 4:25 pm
To: rodman@shellport.com
Cc: linux-raid@vger.kernel.org
On Tue, Mar 01, 2016 at 02:57:21PM -0700, rodman@shellport.com wrote:
> Thank you for that information. My kernel is 4.2.0.
I guess the problem is the same.
>
> Is this it's intended behavior or is it simply a bug that has not been
> corrected?
Well, I think it is a bug.
I'm not sure if "dash" bug or kernel bug.
I suspect "dash", since "bash" works...
>
> I only bring this up as nearly all mdadm config tutorials at some point
> mention the checkarray function. They also point out that checkarray is
> also auto configured in the cron tab (cron.d) to run once a month. But
> of course, if checkarray doesn't work then people may think it is
> running once a month when it really isn't.
Exactly!
I've some Ubuntu 14.04.x with old kernel, and one
with new kernel.
This one has a "weak" HDD in RAID (one / some blocks
waiting for relocation).
In my sytems the check occurs weekly, but it did not!
Tried from command line to no avail...
Debugging the script led to the previous result.
I'm not sure, maybe someone should file a bug, but I
have not to much faith in Ubuntu md/mdadm support.
In the past they were slow or no reacting, and "mdadm"
was always an obsolete version (like it is in 14.04.x).
Maybe Debian could be a better place, not sure.
bye,
pg
>
>
> Rodman Frowert -------- Original Message --------
> Subject: Re: Checkarray doesn't seem to do anything
> From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> Date: Tue, March 01, 2016 3:33 pm
> To: rodman@shellport.com
> Cc: linux-raid@vger.kernel.org
>
> On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
> >
> > Hello,
> >
> > I have a question as it pertains to the checkarray function of Mdadm.
> I
> > can't actually get it to do anything. I'm running Ubuntu 14.04.4
> Server
> > LTS with the latest mdadm installed. I have a single RAID 1 mirror
> > configured and working properly near as I can tell.
> >
> > I can force a check if I issue "echo check >
> > /sys/block/md0/md/sync_action" as root. I can then see the status of
> > the check via /proc/mdstat or using the --detail switch on mdadm
> itself.
> >
> > However, nothing happens if I type in:
> > /usr/share/mdadm/checkarray -a
> > or
> > /usr/share/mdadm/checkarray -a /dev/mdX
> > or
> > /usr/share/mdadm/checkarray -as
> >
> > I've tried the commands as root and using sudo. The array sits idle
> > after all of the above commands. The only thing I can actually get
> > checkarray to perform is to give me the help output if I pass it -h.
> I
> > have attached, below, the current status of the mdadm managed array.
> > Doing some Googling I haven't been able to find anyone else that
> isn't
> > able to use the checkarray function so I must be doing something
> wrong.
> >
> > Any ideas?
>
> Well, yes, I've the same problem.
>
> Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
> Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
> does not work.
>
> The script runs with "dash" (not "bash") and it has
> a "read cur_status < something_from_sysfs" in order to check
> the array status.
> With kernel 3.13, the "read" returns the proper value,
> with kernel 4.11 (I think) the very same "read" exits
> the script without any message or else.
>
> One option is to use "bash", that is change the top
> of the script from: #!/bin/sh to #!/bin/bash
>
> Another is to change the "read", with a "cat", that is
> something like: cur_status=$( cat something_from_sysfs )
>
> You can check yourself, there is only one "read" in
> the script (if I recall correctly).
>
> Hope this helps,
>
> bye,
>
> pg
>
> >
> > Rodman Frowert
> >
>
> >
> > cat /proc/mdstat shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> > [raid4] [raid10]
> > md0 : active raid1 sdc1[1] sdb1[0]
> > 5860389696 blocks super 1.2 [2/2] [UU]
> > -----------------
> >
> > sudo mdadm --detail /dev/md0 shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> > /dev/md0:
> > Version : 1.2
> > Creation Time : Fri Feb 26 14:07:40 2016
> > Raid Level : raid1
> > Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Raid Devices : 2
> > Total Devices : 2
> > Persistence : Superblock is persistent
> >
> > Update Time : Tue Mar 1 07:40:19 2016
> > State : clean
> > Active Devices : 2
> > Working Devices : 2
> > Failed Devices : 0
> > Spare Devices : 0
> >
> > Name : nas:0 (local to host nas)
> > UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> > Events : 164
> >
> > Number Major Minor RaidDevice State
> > 0 8 17 0 active sync /dev/sdb1
> > 1 8 33 1 active sync /dev/sdc1
> >
> > -----------------
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* RE: Checkarray doesn't seem to do anything
From: rodman @ 2016-03-01 23:00 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: linux-raid
Looks like it was reported in 2015:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787950
Is Debian the sole maintainer for mdadm now?
Regards,
Rodman Frowert
-------- Original Message --------
Subject: Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
Date: Tue, March 01, 2016 4:25 pm
To: rodman@shellport.com
Cc: linux-raid@vger.kernel.org
On Tue, Mar 01, 2016 at 02:57:21PM -0700, rodman@shellport.com wrote:
> Thank you for that information. My kernel is 4.2.0.
I guess the problem is the same.
>
> Is this it's intended behavior or is it simply a bug that has not been
> corrected?
Well, I think it is a bug.
I'm not sure if "dash" bug or kernel bug.
I suspect "dash", since "bash" works...
>
> I only bring this up as nearly all mdadm config tutorials at some point
> mention the checkarray function. They also point out that checkarray is
> also auto configured in the cron tab (cron.d) to run once a month. But
> of course, if checkarray doesn't work then people may think it is
> running once a month when it really isn't.
Exactly!
I've some Ubuntu 14.04.x with old kernel, and one
with new kernel.
This one has a "weak" HDD in RAID (one / some blocks
waiting for relocation).
In my sytems the check occurs weekly, but it did not!
Tried from command line to no avail...
Debugging the script led to the previous result.
I'm not sure, maybe someone should file a bug, but I
have not to much faith in Ubuntu md/mdadm support.
In the past they were slow or no reacting, and "mdadm"
was always an obsolete version (like it is in 14.04.x).
Maybe Debian could be a better place, not sure.
bye,
pg
>
>
> Rodman Frowert -------- Original Message --------
> Subject: Re: Checkarray doesn't seem to do anything
> From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> Date: Tue, March 01, 2016 3:33 pm
> To: rodman@shellport.com
> Cc: linux-raid@vger.kernel.org
>
> On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
> >
> > Hello,
> >
> > I have a question as it pertains to the checkarray function of Mdadm.
> I
> > can't actually get it to do anything. I'm running Ubuntu 14.04.4
> Server
> > LTS with the latest mdadm installed. I have a single RAID 1 mirror
> > configured and working properly near as I can tell.
> >
> > I can force a check if I issue "echo check >
> > /sys/block/md0/md/sync_action" as root. I can then see the status of
> > the check via /proc/mdstat or using the --detail switch on mdadm
> itself.
> >
> > However, nothing happens if I type in:
> > /usr/share/mdadm/checkarray -a
> > or
> > /usr/share/mdadm/checkarray -a /dev/mdX
> > or
> > /usr/share/mdadm/checkarray -as
> >
> > I've tried the commands as root and using sudo. The array sits idle
> > after all of the above commands. The only thing I can actually get
> > checkarray to perform is to give me the help output if I pass it -h.
> I
> > have attached, below, the current status of the mdadm managed array.
> > Doing some Googling I haven't been able to find anyone else that
> isn't
> > able to use the checkarray function so I must be doing something
> wrong.
> >
> > Any ideas?
>
> Well, yes, I've the same problem.
>
> Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
> Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
> does not work.
>
> The script runs with "dash" (not "bash") and it has
> a "read cur_status < something_from_sysfs" in order to check
> the array status.
> With kernel 3.13, the "read" returns the proper value,
> with kernel 4.11 (I think) the very same "read" exits
> the script without any message or else.
>
> One option is to use "bash", that is change the top
> of the script from: #!/bin/sh to #!/bin/bash
>
> Another is to change the "read", with a "cat", that is
> something like: cur_status=$( cat something_from_sysfs )
>
> You can check yourself, there is only one "read" in
> the script (if I recall correctly).
>
> Hope this helps,
>
> bye,
>
> pg
>
> >
> > Rodman Frowert
> >
>
> >
> > cat /proc/mdstat shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> > [raid4] [raid10]
> > md0 : active raid1 sdc1[1] sdb1[0]
> > 5860389696 blocks super 1.2 [2/2] [UU]
> > -----------------
> >
> > sudo mdadm --detail /dev/md0 shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> > /dev/md0:
> > Version : 1.2
> > Creation Time : Fri Feb 26 14:07:40 2016
> > Raid Level : raid1
> > Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Raid Devices : 2
> > Total Devices : 2
> > Persistence : Superblock is persistent
> >
> > Update Time : Tue Mar 1 07:40:19 2016
> > State : clean
> > Active Devices : 2
> > Working Devices : 2
> > Failed Devices : 0
> > Spare Devices : 0
> >
> > Name : nas:0 (local to host nas)
> > UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> > Events : 164
> >
> > Number Major Minor RaidDevice State
> > 0 8 17 0 active sync /dev/sdb1
> > 1 8 33 1 active sync /dev/sdc1
> >
> > -----------------
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor @ 2016-03-01 22:25 UTC (permalink / raw)
To: rodman; +Cc: linux-raid
In-Reply-To: <20160301145721.0a91db92e0d9ae1e5e3fde0508678719.0a105d4628.wbe@email03.secureserver.net>
On Tue, Mar 01, 2016 at 02:57:21PM -0700, rodman@shellport.com wrote:
> Thank you for that information. My kernel is 4.2.0.
I guess the problem is the same.
>
> Is this it's intended behavior or is it simply a bug that has not been
> corrected?
Well, I think it is a bug.
I'm not sure if "dash" bug or kernel bug.
I suspect "dash", since "bash" works...
>
> I only bring this up as nearly all mdadm config tutorials at some point
> mention the checkarray function. They also point out that checkarray is
> also auto configured in the cron tab (cron.d) to run once a month. But
> of course, if checkarray doesn't work then people may think it is
> running once a month when it really isn't.
Exactly!
I've some Ubuntu 14.04.x with old kernel, and one
with new kernel.
This one has a "weak" HDD in RAID (one / some blocks
waiting for relocation).
In my sytems the check occurs weekly, but it did not!
Tried from command line to no avail...
Debugging the script led to the previous result.
I'm not sure, maybe someone should file a bug, but I
have not to much faith in Ubuntu md/mdadm support.
In the past they were slow or no reacting, and "mdadm"
was always an obsolete version (like it is in 14.04.x).
Maybe Debian could be a better place, not sure.
bye,
pg
>
>
> Rodman Frowert -------- Original Message --------
> Subject: Re: Checkarray doesn't seem to do anything
> From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> Date: Tue, March 01, 2016 3:33 pm
> To: rodman@shellport.com
> Cc: linux-raid@vger.kernel.org
>
> On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
> >
> > Hello,
> >
> > I have a question as it pertains to the checkarray function of Mdadm.
> I
> > can't actually get it to do anything. I'm running Ubuntu 14.04.4
> Server
> > LTS with the latest mdadm installed. I have a single RAID 1 mirror
> > configured and working properly near as I can tell.
> >
> > I can force a check if I issue "echo check >
> > /sys/block/md0/md/sync_action" as root. I can then see the status of
> > the check via /proc/mdstat or using the --detail switch on mdadm
> itself.
> >
> > However, nothing happens if I type in:
> > /usr/share/mdadm/checkarray -a
> > or
> > /usr/share/mdadm/checkarray -a /dev/mdX
> > or
> > /usr/share/mdadm/checkarray -as
> >
> > I've tried the commands as root and using sudo. The array sits idle
> > after all of the above commands. The only thing I can actually get
> > checkarray to perform is to give me the help output if I pass it -h.
> I
> > have attached, below, the current status of the mdadm managed array.
> > Doing some Googling I haven't been able to find anyone else that
> isn't
> > able to use the checkarray function so I must be doing something
> wrong.
> >
> > Any ideas?
>
> Well, yes, I've the same problem.
>
> Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
> Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
> does not work.
>
> The script runs with "dash" (not "bash") and it has
> a "read cur_status < something_from_sysfs" in order to check
> the array status.
> With kernel 3.13, the "read" returns the proper value,
> with kernel 4.11 (I think) the very same "read" exits
> the script without any message or else.
>
> One option is to use "bash", that is change the top
> of the script from: #!/bin/sh to #!/bin/bash
>
> Another is to change the "read", with a "cat", that is
> something like: cur_status=$( cat something_from_sysfs )
>
> You can check yourself, there is only one "read" in
> the script (if I recall correctly).
>
> Hope this helps,
>
> bye,
>
> pg
>
> >
> > Rodman Frowert
> >
>
> >
> > cat /proc/mdstat shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> > [raid4] [raid10]
> > md0 : active raid1 sdc1[1] sdb1[0]
> > 5860389696 blocks super 1.2 [2/2] [UU]
> > -----------------
> >
> > sudo mdadm --detail /dev/md0 shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> > /dev/md0:
> > Version : 1.2
> > Creation Time : Fri Feb 26 14:07:40 2016
> > Raid Level : raid1
> > Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Raid Devices : 2
> > Total Devices : 2
> > Persistence : Superblock is persistent
> >
> > Update Time : Tue Mar 1 07:40:19 2016
> > State : clean
> > Active Devices : 2
> > Working Devices : 2
> > Failed Devices : 0
> > Spare Devices : 0
> >
> > Name : nas:0 (local to host nas)
> > UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> > Events : 164
> >
> > Number Major Minor RaidDevice State
> > 0 8 17 0 active sync /dev/sdb1
> > 1 8 33 1 active sync /dev/sdc1
> >
> > -----------------
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* RE: Checkarray doesn't seem to do anything
From: rodman @ 2016-03-01 21:57 UTC (permalink / raw)
To: linux-raid
Thank you for that information. My kernel is 4.2.0.
Is this it's intended behavior or is it simply a bug that has not been
corrected?
I only bring this up as nearly all mdadm config tutorials at some point
mention the checkarray function. They also point out that checkarray is
also auto configured in the cron tab (cron.d) to run once a month. But
of course, if checkarray doesn't work then people may think it is
running once a month when it really isn't.
Rodman Frowert -------- Original Message --------
Subject: Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
Date: Tue, March 01, 2016 3:33 pm
To: rodman@shellport.com
Cc: linux-raid@vger.kernel.org
On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
>
> Hello,
>
> I have a question as it pertains to the checkarray function of Mdadm.
I
> can't actually get it to do anything. I'm running Ubuntu 14.04.4
Server
> LTS with the latest mdadm installed. I have a single RAID 1 mirror
> configured and working properly near as I can tell.
>
> I can force a check if I issue "echo check >
> /sys/block/md0/md/sync_action" as root. I can then see the status of
> the check via /proc/mdstat or using the --detail switch on mdadm
itself.
>
> However, nothing happens if I type in:
> /usr/share/mdadm/checkarray -a
> or
> /usr/share/mdadm/checkarray -a /dev/mdX
> or
> /usr/share/mdadm/checkarray -as
>
> I've tried the commands as root and using sudo. The array sits idle
> after all of the above commands. The only thing I can actually get
> checkarray to perform is to give me the help output if I pass it -h.
I
> have attached, below, the current status of the mdadm managed array.
> Doing some Googling I haven't been able to find anyone else that
isn't
> able to use the checkarray function so I must be doing something
wrong.
>
> Any ideas?
Well, yes, I've the same problem.
Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
does not work.
The script runs with "dash" (not "bash") and it has
a "read cur_status < something_from_sysfs" in order to check
the array status.
With kernel 3.13, the "read" returns the proper value,
with kernel 4.11 (I think) the very same "read" exits
the script without any message or else.
One option is to use "bash", that is change the top
of the script from: #!/bin/sh to #!/bin/bash
Another is to change the "read", with a "cat", that is
something like: cur_status=$( cat something_from_sysfs )
You can check yourself, there is only one "read" in
the script (if I recall correctly).
Hope this helps,
bye,
pg
>
> Rodman Frowert
>
>
> cat /proc/mdstat shows:
> ------------------
> rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : active raid1 sdc1[1] sdb1[0]
> 5860389696 blocks super 1.2 [2/2] [UU]
> -----------------
>
> sudo mdadm --detail /dev/md0 shows:
> ------------------
> rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Fri Feb 26 14:07:40 2016
> Raid Level : raid1
> Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Update Time : Tue Mar 1 07:40:19 2016
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> Name : nas:0 (local to host nas)
> UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> Events : 164
>
> Number Major Minor RaidDevice State
> 0 8 17 0 active sync /dev/sdb1
> 1 8 33 1 active sync /dev/sdc1
>
> -----------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
--
To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor @ 2016-03-01 21:33 UTC (permalink / raw)
To: rodman; +Cc: linux-raid
In-Reply-To: <20160301140944.0a91db92e0d9ae1e5e3fde0508678719.a71523d3be.wbe@email03.secureserver.net>
On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
>
> Hello,
>
> I have a question as it pertains to the checkarray function of Mdadm. I
> can't actually get it to do anything. I'm running Ubuntu 14.04.4 Server
> LTS with the latest mdadm installed. I have a single RAID 1 mirror
> configured and working properly near as I can tell.
>
> I can force a check if I issue "echo check >
> /sys/block/md0/md/sync_action" as root. I can then see the status of
> the check via /proc/mdstat or using the --detail switch on mdadm itself.
>
> However, nothing happens if I type in:
> /usr/share/mdadm/checkarray -a
> or
> /usr/share/mdadm/checkarray -a /dev/mdX
> or
> /usr/share/mdadm/checkarray -as
>
> I've tried the commands as root and using sudo. The array sits idle
> after all of the above commands. The only thing I can actually get
> checkarray to perform is to give me the help output if I pass it -h. I
> have attached, below, the current status of the mdadm managed array.
> Doing some Googling I haven't been able to find anyone else that isn't
> able to use the checkarray function so I must be doing something wrong.
>
> Any ideas?
Well, yes, I've the same problem.
Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
does not work.
The script runs with "dash" (not "bash") and it has
a "read cur_status < something_from_sysfs" in order to check
the array status.
With kernel 3.13, the "read" returns the proper value,
with kernel 4.11 (I think) the very same "read" exits
the script without any message or else.
One option is to use "bash", that is change the top
of the script from: #!/bin/sh to #!/bin/bash
Another is to change the "read", with a "cat", that is
something like: cur_status=$( cat something_from_sysfs )
You can check yourself, there is only one "read" in
the script (if I recall correctly).
Hope this helps,
bye,
pg
>
> Rodman Frowert
>
>
> cat /proc/mdstat shows:
> ------------------
> rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : active raid1 sdc1[1] sdb1[0]
> 5860389696 blocks super 1.2 [2/2] [UU]
> -----------------
>
> sudo mdadm --detail /dev/md0 shows:
> ------------------
> rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Fri Feb 26 14:07:40 2016
> Raid Level : raid1
> Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> Raid Devices : 2
> Total Devices : 2
> Persistence : Superblock is persistent
>
> Update Time : Tue Mar 1 07:40:19 2016
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> Name : nas:0 (local to host nas)
> UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> Events : 164
>
> Number Major Minor RaidDevice State
> 0 8 17 0 active sync /dev/sdb1
> 1 8 33 1 active sync /dev/sdc1
>
> -----------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* Checkarray doesn't seem to do anything
From: rodman @ 2016-03-01 21:09 UTC (permalink / raw)
To: linux-raid
Hello,
I have a question as it pertains to the checkarray function of Mdadm. I
can't actually get it to do anything. I'm running Ubuntu 14.04.4 Server
LTS with the latest mdadm installed. I have a single RAID 1 mirror
configured and working properly near as I can tell.
I can force a check if I issue "echo check >
/sys/block/md0/md/sync_action" as root. I can then see the status of
the check via /proc/mdstat or using the --detail switch on mdadm itself.
However, nothing happens if I type in:
/usr/share/mdadm/checkarray -a
or
/usr/share/mdadm/checkarray -a /dev/mdX
or
/usr/share/mdadm/checkarray -as
I've tried the commands as root and using sudo. The array sits idle
after all of the above commands. The only thing I can actually get
checkarray to perform is to give me the help output if I pass it -h. I
have attached, below, the current status of the mdadm managed array.
Doing some Googling I haven't been able to find anyone else that isn't
able to use the checkarray function so I must be doing something wrong.
Any ideas?
Rodman Frowert
cat /proc/mdstat shows:
------------------
rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid1 sdc1[1] sdb1[0]
5860389696 blocks super 1.2 [2/2] [UU]
-----------------
sudo mdadm --detail /dev/md0 shows:
------------------
rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Feb 26 14:07:40 2016
Raid Level : raid1
Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Tue Mar 1 07:40:19 2016
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : nas:0 (local to host nas)
UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
Events : 164
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
-----------------
^ permalink raw reply
* Re: [PATCH 00/35 v4] separate operations from flags in the bio/request structs
From: Mike Christie @ 2016-02-29 17:30 UTC (permalink / raw)
To: Christoph Hellwig, mchristi
Cc: linux-bcache, linux-block, xfs, ocfs2-devel, linux-scsi,
konrad.wilk, linux-kernel, philipp.reisner, linux-f2fs-devel,
linux-raid, dm-devel, target-devel, linux-mtd, osd-dev,
linux-fsdevel, lars.ellenberg, linux-ext4, linux-btrfs, drbd-dev
In-Reply-To: <20160229171339.GA17755@infradead.org>
On 02/29/2016 11:13 AM, Christoph Hellwig wrote:
> Any reason you've dropped my ACK for the previous version?
Sorry. I think I accidentally dropped it when I rebased back over to
linux-next. It looks like it only got stuck on the first patch. I will
fix that up on the resend for those kbuild errors.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply
* Re: [PATCH 00/35 v4] separate operations from flags in the bio/request structs
From: Christoph Hellwig @ 2016-02-29 17:13 UTC (permalink / raw)
To: mchristi
Cc: linux-f2fs-devel, linux-ext4, konrad.wilk, drbd-dev,
philipp.reisner, lars.ellenberg, linux-raid, dm-devel,
linux-fsdevel, linux-bcache, linux-block, linux-kernel,
linux-scsi, linux-mtd, target-devel, linux-btrfs, osd-dev, xfs,
ocfs2-devel
In-Reply-To: <1456343292-14535-1-git-send-email-mchristi@redhat.com>
Any reason you've dropped my ACK for the previous version?
Jens, is this something you're fine with? Would be great to figure
out a way to get this into 4.6. There's probably going to be
plenty of conflicts, so having some sort of stable base tree
would be nice..
(Talking about conflicts, seems like the block for-linus and for-next
trees already conflict quite a bit..)
^ permalink raw reply
* [PATCH] raid1: include bio_end_io_list in nr_queued to prevent freeze_array hang
From: Nate Dailey @ 2016-02-29 15:43 UTC (permalink / raw)
To: linux-raid; +Cc: Nate Dailey
If raid1d is handling a mix of read and write errors, handle_read_error's
call to freeze_array can get stuck.
This can happen because, though the bio_end_io_list is initially drained,
writes can be added to it via handle_write_finished as the retry_list
is processed. These writes contribute to nr_pending but are not included
in nr_queued.
If a later entry on the retry_list triggers a call to handle_read_error,
freeze array hangs waiting for nr_pending == nr_queued+extra. The writes
on the bio_end_io_list aren't included in nr_queued so the condition will
never be satisfied.
To prevent the hang, include bio_end_io_list writes in nr_queued.
There's probably a better way to handle decrementing nr_queued, but this
seemed like the safest way to avoid breaking surrounding code.
I'm happy to supply the script I used to repro this hang.
Signed-off-by: Nate Dailey <nate.dailey@stratus.com>
---
drivers/md/raid1.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 4e3843f..bb5bce0 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2274,6 +2274,7 @@ static void handle_write_finished(struct r1conf *conf, struct r1bio *r1_bio)
if (fail) {
spin_lock_irq(&conf->device_lock);
list_add(&r1_bio->retry_list, &conf->bio_end_io_list);
+ conf->nr_queued++;
spin_unlock_irq(&conf->device_lock);
md_wakeup_thread(conf->mddev->thread);
} else {
@@ -2391,8 +2392,10 @@ static void raid1d(struct md_thread *thread)
LIST_HEAD(tmp);
spin_lock_irqsave(&conf->device_lock, flags);
if (!test_bit(MD_CHANGE_PENDING, &mddev->flags)) {
- list_add(&tmp, &conf->bio_end_io_list);
- list_del_init(&conf->bio_end_io_list);
+ while (!list_empty(&conf->bio_end_io_list)) {
+ list_move(conf->bio_end_io_list.prev, &tmp);
+ conf->nr_queued--;
+ }
}
spin_unlock_irqrestore(&conf->device_lock, flags);
while (!list_empty(&tmp)) {
--
1.8.3.1
^ permalink raw reply related
* Re: multipath: I/O hanging forever
From: Andrea Righi @ 2016-02-29 3:46 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, linux-kernel
In-Reply-To: <20160229015333.GA3101@Dell>
On Sun, Feb 28, 2016 at 06:53:33PM -0700, Andrea Righi wrote:
...
> I'm using 4.5.0-rc5+, from Linus' git. I'll try to do a git bisect
> later, I'm pretty sure this problem has been introduced recently (i.e.,
> I've never seen this issue with 4.1.x).
I confirm, just tested kernel 4.1 and this problem doesn't happen.
Thanks,
-Andrea
^ permalink raw reply
* multipath: I/O hanging forever
From: Andrea Righi @ 2016-02-29 1:53 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, linux-kernel
Hello,
if I try to do I/O on a mdadm/multipath volume the task is hunging
forever and it never completes.
Anybody else has noticed the same problem?
I'm using 4.5.0-rc5+, from Linus' git. I'll try to do a git bisect
later, I'm pretty sure this problem has been introduced recently (i.e.,
I've never seen this issue with 4.1.x).
Example:
# mdadm -C /dev/md0 --level=multipath --raid-devices=2 /dev/sdb /dev/sdc
# cat /proc/mdstat
Personalities : [multipath]
md0 : active multipath sdb[0] sdc[1]
4042740 blocks super 1.2 [2/2] [UU]
# mkfs.xfs /dev/md0
meta-data=/dev/md0 isize=256 agcount=4, agsize=252672
blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=1010685, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
^C^C^C
# cat /proc/`pidof mkfs.xfs`/stack
[<ffffffff8126f53c>] do_blockdev_direct_IO+0x1adc/0x2300
[<ffffffff8126fda3>] __blockdev_direct_IO+0x43/0x50
[<ffffffff8126accc>] blkdev_direct_IO+0x4c/0x50
[<ffffffff811a2014>] generic_file_direct_write+0xa4/0x160
[<ffffffff811a2190>] __generic_file_write_iter+0xc0/0x1e0
[<ffffffff8126afc0>] blkdev_write_iter+0x80/0x100
[<ffffffff81228c3d>] __vfs_write+0xad/0xe0
[<ffffffff81229a85>] vfs_write+0xa5/0x1a0
[<ffffffff8122aacc>] SyS_pwrite64+0x6c/0xa0
[<ffffffff818281f2>] entry_SYSCALL_64_fastpath+0x12/0x76
[<ffffffffffffffff>] 0xffffffffffffffff
Thanks,
-Andrea
^ permalink raw reply
* INFORMAÇÃO IMPORTANTE RE: Consultoria de Investimento em sua Localidade.
From: Mr. DOUGLAS @ 2016-02-28 12:41 UTC (permalink / raw)
To: Recipients
Oi,
Temos planejado investir alguns fundos no seu país.
Responder se você estiver interessado.
Saudações,
Sir Henry Neville Lindley Keswick
Presidente da Jardine Matheson Holdings Ltd
https://en.wikipedia.org/wiki/Keswick_family
==================================================
Hi friend,
Our family is interested in investing funds in your locality.
More information for you interest .
Greetings,
Sir Henry Neville Lindley Keswick
Chairman of Jardine Matheson Holdings Ltd
https://en.wikipedia.org/wiki/Keswick_family
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: mdadm 3.3.2 deadlock
From: NeilBrown @ 2016-02-28 10:35 UTC (permalink / raw)
To: Jes Sorensen, Vasiliy Tolstov; +Cc: linux-raid
In-Reply-To: <wrfjziuo6a08.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2916 bytes --]
On Fri, Feb 26 2016, Jes Sorensen wrote:
> Vasiliy Tolstov <v.tolstov@selfip.ru> writes:
>> 2016-02-25 10:23 GMT+03:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
>>> Hi. I have strange deadlocked process of mdadm
>>> root 14495 0.0 0.0 13064 1964 ? D Feb24 0:00
>>> /sbin/mdadm --detail --export /dev/.tmp-block-259:5
>>>
>>> why this is can happen and does mdadm git repo already have fix for this?
>>> Thanks!
>>
>>
>> i'm use old linux 3.19.3, echo w > /proc/sysrq-trigger:
>> [15840064.321022] SysRq : Show Blocked State
>> [15840064.321072] task PC stack pid father
>> [15840064.321183] mdadm D ffff880eebb02490 0 14495
>> 8481 0x00000004
>> [15840064.321268] ffff880eebb02490 ffffffff81141d69 ffff881ff8fd6a70
>> 0000000000013b40
>> [15840064.321360] 0000000000013b40 ffff880eebb02490 ffff880ebb073fd8
>> ffff88103fffcd80
>> [15840064.321452] ffff880fbb0ea418 ffff880fbb0ea41c ffff880eebb02490
>> ffff880fbb0ea420
>> [15840064.329570] Call Trace:
>> [15840064.329615] [<ffffffff81141d69>] ? __d_rehash+0x19/0x4c
>> [15840064.329667] [<ffffffff813e082b>] ? schedule_preempt_disabled+0x6/0x8
>> [15840064.329722] [<ffffffff813e14ff>] ? __mutex_lock_slowpath+0xa8/0x104
>> [15840064.329786] [<ffffffff813e1571>] ? mutex_lock+0x16/0x25
>> [15840064.329838] [<ffffffff8115a59a>] ? __blkdev_get+0x92/0x3b9
>> [15840064.329889] [<ffffffff8115ab94>] ? blkdev_get+0x2d3/0x2d3
>> [15840064.329939] [<ffffffff8115aa4c>] ? blkdev_get+0x18b/0x2d3
>> [15840064.329991] [<ffffffff811437b5>] ? __d_lookup_rcu+0x94/0xbb
>> [15840064.330043] [<ffffffff8115ab94>] ? blkdev_get+0x2d3/0x2d3
>> [15840064.330095] [<ffffffff81130a5e>] ? do_dentry_open+0x178/0x27e
>> [15840064.330147] [<ffffffff8113bb69>] ? do_last+0x865/0xa23
>> [15840064.330197] [<ffffffff8113a2c1>] ? __inode_permission+0x57/0x95
>> [15840064.330249] [<ffffffff8113d15f>] ? path_openat+0x207/0x46d
>> [15840064.330301] [<ffffffff8111f744>] ? __cache_free.isra.47+0x1e5/0x1f4
>> [15840064.330354] [<ffffffff8113e450>] ? do_filp_open+0x2b/0x6f
>> [15840064.330405] [<ffffffff811472e9>] ? __alloc_fd+0xd9/0xea
>> [15840064.330456] [<ffffffff811313a3>] ? do_sys_open+0x65/0xe9
>> [15840064.330506] [<ffffffff813e2ce9>] ? system_call_fastpath+0x12/0x17
>
> You need to provide more information if you want any feedback. Output of
> /proc/mdstat for starters.
>
> It's most likely a kernel bug, not an mdadm bug, so upgrading to a
> recent kernel would be a good starting point.
>
It looks like some other process is hanging while it is holding the
mutex.
So "cat /proc/mdstat" will hang as well - newer kernels (Since 4.0)
don't need the mutex for /proc/mdstat but 3.19 still does.
But it that is the *only* blocked process, then the only explanation I
can think of is that some processed crashed while holding the mutex.
Are there any other stack traces in the kernel logs?
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Recovering failed raid6 with 3 failing devices.
From: Mikael Abrahamsson @ 2016-02-27 18:00 UTC (permalink / raw)
To: Vandalon; +Cc: linux-raid
In-Reply-To: <1E25F099-6730-4A26-9805-C5F547B42276@vandalon.nl>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 594 bytes --]
On Wed, 24 Feb 2016, Vandalon wrote:
> Hi,
>
> I am trying to recover a 10 disk raid6-set, but to no succes.
>
> After reading https://raid.wiki.kernel.org/index.php/RAID_Recovery, I tried the following but the —force option does not seem to work:
>
> and using mdadm - v3.3 - 3rd September 2013
Compile mdadm from latest git resource and try again.
If that doesn't work, post full mdadm -E from all drives so we can see
what's actually going on, I'd imagine it's not only the event count that's
preventing these drives frmo being used.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device
From: Mike Snitzer @ 2016-02-26 20:47 UTC (permalink / raw)
To: Kees Cook
Cc: Will Drewry, Chen Yu, Jonathan Corbet, David Zeuthen,
linux-doc@vger.kernel.org, Rafael J. Wysocki, LKML, Dan Ehrenberg,
linux-raid, dm-devel, Vishnu Pratap Singh, Yaowei Bai,
Andrew Morton, Shaohua Li, Alasdair Kergon
In-Reply-To: <CAGXu5jJNgwdRZ0c0qG-gDqjcAvSn1ZyLsHBhqgfbM6iE2u1uow@mail.gmail.com>
On Fri, Feb 26 2016 at 2:59pm -0500,
Kees Cook <keescook@chromium.org> wrote:
> On Fri, Feb 26, 2016 at 11:21 AM, Mike Snitzer <snitzer@redhat.com> wrote:
> > On Fri, Feb 26 2016 at 1:52pm -0500,
> > Kees Cook <keescook@chromium.org> wrote:
> >
> >> On Fri, Feb 26, 2016 at 8:53 AM, Mike Snitzer <snitzer@redhat.com> wrote:
> >> > On Mon, Feb 22 2016 at 1:55pm -0500,
> >> > Kees Cook <keescook@chromium.org> wrote:
> >> >
> >> >> On Sun, Feb 21, 2016 at 2:08 PM, Alasdair G Kergon <agk@redhat.com> wrote:
> >> >> > On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
> >> >> >> This is a resurrection of a patch series from a few years back, first
> >> >> >> brought to the dm maintainers in 2010. It creates a way to define dm
> >> >> >> devices on the kernel command line for systems that do not use an
> >> >> >> initramfs, or otherwise need a dm running before init starts.
> >> >> >>
> >> >> >> This has been used by Chrome OS for several years, and now by Brillo
> >> >> >> (and likely Android soon).
> >> >> >>
> >> >> >> The last version was v4:
> >> >> >> https://patchwork.kernel.org/patch/104860/
> >> >> >> https://patchwork.kernel.org/patch/104861/
> >> >> >
> >> >> > Inconsistencies in the terminology here can be sorted out during review,
> >> >> > and I see that you've taken on board some of my review comments from
> >> >> > 2010, but what are your responses to the rest of them?
> >> >>
> >> >> Ah, sorry, the threads I could find were incomplete, so I wasn't able
> >> >> to find those comments that were made to Will's 2010 submission. In
> >> >> some of the cleanups I did I was very confused about "target" vs
> >> >> "table", and tried to fix that. Regardless, I'm open to fixing
> >> >> whatever is needed. :)
> >> >>
> >> >> Thanks for looking at this again!
> >> >
> >> > This work isn't going to fly as is. I appreciate the effort and the
> >> > goal (without understanding _why_) but: you're open-coding, duplicating
> >> > and/or reinventing way too much in do_mounts_dm.c
> >> >
> >> > 1) You first need to answer: _why_ is using a proper initramfs not
> >> > viable? A very simple initramfs that issues dmsetup commands, etc,
> >> > isn't so daunting is it? Why is it so important for the kernel to
> >> > natively provide a dmsetup interface? Chrome, Android, etc cannot use
> >> > initramfs?
> >>
> >> That is correct: Chrome OS does not (and won't) use an initramfs. This
> >> is mainly for reasons of boot speed, verified boot block size, and
> >> maybe some other things I don't remember.
> >
> > Not sure what "verified boot block size" means but...
>
> Chrome OS uses coreboot as its boot firmware and a coreboot module
> known as "depthcharge" for doing the crypto-verification and booting
> of the Chrome OS system. This is the static root of trust Chrome OS
> extends from its read-only boot firmware through to the kernel it
> loads and the dm-verity partition it mounts as the read-only root
> filesystem. To keep the boot speed fast and the kernel size small,
> there is no initramfs.
>
> > Sorry I really don't buy that using a custom initramfs would be the
> > source of slow boot. initramfs is _not_ this hugely inefficient
> > mechanism you'd have us believe.
>
> I didn't say it was hugely inefficient, but for Chrome OS it's not
> needed, and was a measurable source of boot time. We just got rid of
> it since it was redundant. I can't change that design decision; I'm
> just here to help bring the dm= boot support upstream. :)
>
> > And if that is the justification for this early boot dm= support then
> > the Chrome OS project/team will have to continue to carry the hack
> > locally. It has no place upstream. But I'm open to revisiting this if
> > it can be implemented in a very cheap way.
>
> Yeah, I'm open to whatever suggestions you have.
>
> >> > 2) If you are able to adequately justify the need for dm=:
> >> > I'd much rather the dm= kernel commandline be a simple series of
> >> > comma-delimited dmsetup-like commands.
> >> >
> >> > You'd handle each command with extremely basic parsing:
> >> > <dm_ioctl_cmd> <args> [, <dm_ioctl_cmd> <args>]
> >> > (inventing a special token to denote <newline>, to support tables with
> >> > multiple entries, rather than relying on commas and counts, etc)
> >>
> >> Sure, changing the syntax is fine by me. We'd need to plumb access to
> >> the ioctl interface, though.
> >
> > I was hoping to avoid any extra hacks but yes... seems you'd need a new
> > API to issue the equivalent of a DM ioctl programatically. Hopefully
> > it'd be quite a small wrapper.
>
> Seems like it shouldn't be too bad.
OK, I'm waiting on you to give it a shot. I'll do my best to help.
Thanks,
Mike
^ permalink raw reply
* Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device
From: Kees Cook @ 2016-02-26 19:59 UTC (permalink / raw)
To: Mike Snitzer
Cc: Will Drewry, Chen Yu, Jonathan Corbet, David Zeuthen,
linux-doc@vger.kernel.org, Rafael J. Wysocki, LKML, Dan Ehrenberg,
linux-raid, dm-devel, Vishnu Pratap Singh, Yaowei Bai,
Andrew Morton, Shaohua Li, Alasdair Kergon
In-Reply-To: <20160226192145.GA27326@redhat.com>
On Fri, Feb 26, 2016 at 11:21 AM, Mike Snitzer <snitzer@redhat.com> wrote:
> On Fri, Feb 26 2016 at 1:52pm -0500,
> Kees Cook <keescook@chromium.org> wrote:
>
>> On Fri, Feb 26, 2016 at 8:53 AM, Mike Snitzer <snitzer@redhat.com> wrote:
>> > On Mon, Feb 22 2016 at 1:55pm -0500,
>> > Kees Cook <keescook@chromium.org> wrote:
>> >
>> >> On Sun, Feb 21, 2016 at 2:08 PM, Alasdair G Kergon <agk@redhat.com> wrote:
>> >> > On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
>> >> >> This is a resurrection of a patch series from a few years back, first
>> >> >> brought to the dm maintainers in 2010. It creates a way to define dm
>> >> >> devices on the kernel command line for systems that do not use an
>> >> >> initramfs, or otherwise need a dm running before init starts.
>> >> >>
>> >> >> This has been used by Chrome OS for several years, and now by Brillo
>> >> >> (and likely Android soon).
>> >> >>
>> >> >> The last version was v4:
>> >> >> https://patchwork.kernel.org/patch/104860/
>> >> >> https://patchwork.kernel.org/patch/104861/
>> >> >
>> >> > Inconsistencies in the terminology here can be sorted out during review,
>> >> > and I see that you've taken on board some of my review comments from
>> >> > 2010, but what are your responses to the rest of them?
>> >>
>> >> Ah, sorry, the threads I could find were incomplete, so I wasn't able
>> >> to find those comments that were made to Will's 2010 submission. In
>> >> some of the cleanups I did I was very confused about "target" vs
>> >> "table", and tried to fix that. Regardless, I'm open to fixing
>> >> whatever is needed. :)
>> >>
>> >> Thanks for looking at this again!
>> >
>> > This work isn't going to fly as is. I appreciate the effort and the
>> > goal (without understanding _why_) but: you're open-coding, duplicating
>> > and/or reinventing way too much in do_mounts_dm.c
>> >
>> > 1) You first need to answer: _why_ is using a proper initramfs not
>> > viable? A very simple initramfs that issues dmsetup commands, etc,
>> > isn't so daunting is it? Why is it so important for the kernel to
>> > natively provide a dmsetup interface? Chrome, Android, etc cannot use
>> > initramfs?
>>
>> That is correct: Chrome OS does not (and won't) use an initramfs. This
>> is mainly for reasons of boot speed, verified boot block size, and
>> maybe some other things I don't remember.
>
> Not sure what "verified boot block size" means but...
Chrome OS uses coreboot as its boot firmware and a coreboot module
known as "depthcharge" for doing the crypto-verification and booting
of the Chrome OS system. This is the static root of trust Chrome OS
extends from its read-only boot firmware through to the kernel it
loads and the dm-verity partition it mounts as the read-only root
filesystem. To keep the boot speed fast and the kernel size small,
there is no initramfs.
> Sorry I really don't buy that using a custom initramfs would be the
> source of slow boot. initramfs is _not_ this hugely inefficient
> mechanism you'd have us believe.
I didn't say it was hugely inefficient, but for Chrome OS it's not
needed, and was a measurable source of boot time. We just got rid of
it since it was redundant. I can't change that design decision; I'm
just here to help bring the dm= boot support upstream. :)
> And if that is the justification for this early boot dm= support then
> the Chrome OS project/team will have to continue to carry the hack
> locally. It has no place upstream. But I'm open to revisiting this if
> it can be implemented in a very cheap way.
Yeah, I'm open to whatever suggestions you have.
>> > 2) If you are able to adequately justify the need for dm=:
>> > I'd much rather the dm= kernel commandline be a simple series of
>> > comma-delimited dmsetup-like commands.
>> >
>> > You'd handle each command with extremely basic parsing:
>> > <dm_ioctl_cmd> <args> [, <dm_ioctl_cmd> <args>]
>> > (inventing a special token to denote <newline>, to support tables with
>> > multiple entries, rather than relying on commas and counts, etc)
>>
>> Sure, changing the syntax is fine by me. We'd need to plumb access to
>> the ioctl interface, though.
>
> I was hoping to avoid any extra hacks but yes... seems you'd need a new
> API to issue the equivalent of a DM ioctl programatically. Hopefully
> it'd be quite a small wrapper.
Seems like it shouldn't be too bad.
>> > and you'd then have do_mounts_dm.c open /dev/mapper/control directly and
>> > issue proper DM ioctls rather than adding all your shim code. This last
>> > bit of opening /dev/mapper/control from init needs more research -- not
>> > sure if doing such a thing from kernel is viable/safe/acceptable.
>>
>> Well, there's no /dev and no init since our dm is the root device
>> (dm-verity). We need everything up and running before we mount the
>> root filesystem, very similar to do_mount_md.c's purpose.
>
> Ah yes, microoptimization associated with no udev or normal Linux boot
> comes full circle and limits the use of existing standard interfaces.
Chrome OS is hardly the only Linux device without an initramfs, but
yeah, it does cause us headaches from time to time.
-Kees
--
Kees Cook
Chrome OS & Brillo Security
^ permalink raw reply
* Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device
From: Mike Snitzer @ 2016-02-26 19:21 UTC (permalink / raw)
To: Kees Cook
Cc: Will Drewry, Chen Yu, Jonathan Corbet, David Zeuthen,
linux-doc@vger.kernel.org, Rafael J. Wysocki, LKML, Dan Ehrenberg,
linux-raid, dm-devel, Vishnu Pratap Singh, Yaowei Bai,
Andrew Morton, Shaohua Li, Alasdair Kergon
In-Reply-To: <CAGXu5j+JGPZ2sH3uAHwpJ3im_TBUL19U43EZw-Z2dZ6fQiC2zA@mail.gmail.com>
On Fri, Feb 26 2016 at 1:52pm -0500,
Kees Cook <keescook@chromium.org> wrote:
> On Fri, Feb 26, 2016 at 8:53 AM, Mike Snitzer <snitzer@redhat.com> wrote:
> > On Mon, Feb 22 2016 at 1:55pm -0500,
> > Kees Cook <keescook@chromium.org> wrote:
> >
> >> On Sun, Feb 21, 2016 at 2:08 PM, Alasdair G Kergon <agk@redhat.com> wrote:
> >> > On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
> >> >> This is a resurrection of a patch series from a few years back, first
> >> >> brought to the dm maintainers in 2010. It creates a way to define dm
> >> >> devices on the kernel command line for systems that do not use an
> >> >> initramfs, or otherwise need a dm running before init starts.
> >> >>
> >> >> This has been used by Chrome OS for several years, and now by Brillo
> >> >> (and likely Android soon).
> >> >>
> >> >> The last version was v4:
> >> >> https://patchwork.kernel.org/patch/104860/
> >> >> https://patchwork.kernel.org/patch/104861/
> >> >
> >> > Inconsistencies in the terminology here can be sorted out during review,
> >> > and I see that you've taken on board some of my review comments from
> >> > 2010, but what are your responses to the rest of them?
> >>
> >> Ah, sorry, the threads I could find were incomplete, so I wasn't able
> >> to find those comments that were made to Will's 2010 submission. In
> >> some of the cleanups I did I was very confused about "target" vs
> >> "table", and tried to fix that. Regardless, I'm open to fixing
> >> whatever is needed. :)
> >>
> >> Thanks for looking at this again!
> >
> > This work isn't going to fly as is. I appreciate the effort and the
> > goal (without understanding _why_) but: you're open-coding, duplicating
> > and/or reinventing way too much in do_mounts_dm.c
> >
> > 1) You first need to answer: _why_ is using a proper initramfs not
> > viable? A very simple initramfs that issues dmsetup commands, etc,
> > isn't so daunting is it? Why is it so important for the kernel to
> > natively provide a dmsetup interface? Chrome, Android, etc cannot use
> > initramfs?
>
> That is correct: Chrome OS does not (and won't) use an initramfs. This
> is mainly for reasons of boot speed, verified boot block size, and
> maybe some other things I don't remember.
Not sure what "verified boot block size" means but...
Sorry I really don't buy that using a custom initramfs would be the
source of slow boot. initramfs is _not_ this hugely inefficient
mechanism you'd have us believe.
And if that is the justification for this early boot dm= support then
the Chrome OS project/team will have to continue to carry the hack
locally. It has no place upstream. But I'm open to revisiting this if
it can be implemented in a very cheap way.
> > 2) If you are able to adequately justify the need for dm=:
> > I'd much rather the dm= kernel commandline be a simple series of
> > comma-delimited dmsetup-like commands.
> >
> > You'd handle each command with extremely basic parsing:
> > <dm_ioctl_cmd> <args> [, <dm_ioctl_cmd> <args>]
> > (inventing a special token to denote <newline>, to support tables with
> > multiple entries, rather than relying on commas and counts, etc)
>
> Sure, changing the syntax is fine by me. We'd need to plumb access to
> the ioctl interface, though.
I was hoping to avoid any extra hacks but yes... seems you'd need a new
API to issue the equivalent of a DM ioctl programatically. Hopefully
it'd be quite a small wrapper.
> > and you'd then have do_mounts_dm.c open /dev/mapper/control directly and
> > issue proper DM ioctls rather than adding all your shim code. This last
> > bit of opening /dev/mapper/control from init needs more research -- not
> > sure if doing such a thing from kernel is viable/safe/acceptable.
>
> Well, there's no /dev and no init since our dm is the root device
> (dm-verity). We need everything up and running before we mount the
> root filesystem, very similar to do_mount_md.c's purpose.
Ah yes, microoptimization associated with no udev or normal Linux boot
comes full circle and limits the use of existing standard interfaces.
^ permalink raw reply
* Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device
From: Kees Cook @ 2016-02-26 18:52 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, dm-devel, Jonathan Corbet, Shaohua Li,
Dan Ehrenberg, Rafael J. Wysocki, Chen Yu, Vishnu Pratap Singh,
Andrew Morton, Yaowei Bai, linux-doc@vger.kernel.org, LKML,
linux-raid, Will Drewry, David Zeuthen
In-Reply-To: <20160226165358.GA26728@redhat.com>
On Fri, Feb 26, 2016 at 8:53 AM, Mike Snitzer <snitzer@redhat.com> wrote:
> On Mon, Feb 22 2016 at 1:55pm -0500,
> Kees Cook <keescook@chromium.org> wrote:
>
>> On Sun, Feb 21, 2016 at 2:08 PM, Alasdair G Kergon <agk@redhat.com> wrote:
>> > On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
>> >> This is a resurrection of a patch series from a few years back, first
>> >> brought to the dm maintainers in 2010. It creates a way to define dm
>> >> devices on the kernel command line for systems that do not use an
>> >> initramfs, or otherwise need a dm running before init starts.
>> >>
>> >> This has been used by Chrome OS for several years, and now by Brillo
>> >> (and likely Android soon).
>> >>
>> >> The last version was v4:
>> >> https://patchwork.kernel.org/patch/104860/
>> >> https://patchwork.kernel.org/patch/104861/
>> >
>> > Inconsistencies in the terminology here can be sorted out during review,
>> > and I see that you've taken on board some of my review comments from
>> > 2010, but what are your responses to the rest of them?
>>
>> Ah, sorry, the threads I could find were incomplete, so I wasn't able
>> to find those comments that were made to Will's 2010 submission. In
>> some of the cleanups I did I was very confused about "target" vs
>> "table", and tried to fix that. Regardless, I'm open to fixing
>> whatever is needed. :)
>>
>> Thanks for looking at this again!
>
> This work isn't going to fly as is. I appreciate the effort and the
> goal (without understanding _why_) but: you're open-coding, duplicating
> and/or reinventing way too much in do_mounts_dm.c
>
> 1) You first need to answer: _why_ is using a proper initramfs not
> viable? A very simple initramfs that issues dmsetup commands, etc,
> isn't so daunting is it? Why is it so important for the kernel to
> natively provide a dmsetup interface? Chrome, Android, etc cannot use
> initramfs?
That is correct: Chrome OS does not (and won't) use an initramfs. This
is mainly for reasons of boot speed, verified boot block size, and
maybe some other things I don't remember.
> 2) If you are able to adequately justify the need for dm=:
> I'd much rather the dm= kernel commandline be a simple series of
> comma-delimited dmsetup-like commands.
>
> You'd handle each command with extremely basic parsing:
> <dm_ioctl_cmd> <args> [, <dm_ioctl_cmd> <args>]
> (inventing a special token to denote <newline>, to support tables with
> multiple entries, rather than relying on commas and counts, etc)
Sure, changing the syntax is fine by me. We'd need to plumb access to
the ioctl interface, though.
> and you'd then have do_mounts_dm.c open /dev/mapper/control directly and
> issue proper DM ioctls rather than adding all your shim code. This last
> bit of opening /dev/mapper/control from init needs more research -- not
> sure if doing such a thing from kernel is viable/safe/acceptable.
Well, there's no /dev and no init since our dm is the root device
(dm-verity). We need everything up and running before we mount the
root filesystem, very similar to do_mount_md.c's purpose.
-Kees
--
Kees Cook
Chrome OS & Brillo Security
^ permalink raw reply
* Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device
From: Mike Snitzer @ 2016-02-26 16:53 UTC (permalink / raw)
To: Kees Cook
Cc: Alasdair Kergon, dm-devel, Jonathan Corbet, Shaohua Li,
Dan Ehrenberg, Rafael J. Wysocki, Chen Yu, Vishnu Pratap Singh,
Andrew Morton, Yaowei Bai, linux-doc@vger.kernel.org, LKML,
linux-raid, Will Drewry, David Zeuthen
In-Reply-To: <CAGXu5jJ9nUdE8u8a3EYR5uS_zOLEkmQn3M0Nsm9bkxBmMxpC9Q@mail.gmail.com>
On Mon, Feb 22 2016 at 1:55pm -0500,
Kees Cook <keescook@chromium.org> wrote:
> On Sun, Feb 21, 2016 at 2:08 PM, Alasdair G Kergon <agk@redhat.com> wrote:
> > On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
> >> This is a resurrection of a patch series from a few years back, first
> >> brought to the dm maintainers in 2010. It creates a way to define dm
> >> devices on the kernel command line for systems that do not use an
> >> initramfs, or otherwise need a dm running before init starts.
> >>
> >> This has been used by Chrome OS for several years, and now by Brillo
> >> (and likely Android soon).
> >>
> >> The last version was v4:
> >> https://patchwork.kernel.org/patch/104860/
> >> https://patchwork.kernel.org/patch/104861/
> >
> > Inconsistencies in the terminology here can be sorted out during review,
> > and I see that you've taken on board some of my review comments from
> > 2010, but what are your responses to the rest of them?
>
> Ah, sorry, the threads I could find were incomplete, so I wasn't able
> to find those comments that were made to Will's 2010 submission. In
> some of the cleanups I did I was very confused about "target" vs
> "table", and tried to fix that. Regardless, I'm open to fixing
> whatever is needed. :)
>
> Thanks for looking at this again!
This work isn't going to fly as is. I appreciate the effort and the
goal (without understanding _why_) but: you're open-coding, duplicating
and/or reinventing way too much in do_mounts_dm.c
1) You first need to answer: _why_ is using a proper initramfs not
viable? A very simple initramfs that issues dmsetup commands, etc,
isn't so daunting is it? Why is it so important for the kernel to
natively provide a dmsetup interface? Chrome, Android, etc cannot use
initramfs?
2) If you are able to adequately justify the need for dm=:
I'd much rather the dm= kernel commandline be a simple series of
comma-delimited dmsetup-like commands.
You'd handle each command with extremely basic parsing:
<dm_ioctl_cmd> <args> [, <dm_ioctl_cmd> <args>]
(inventing a special token to denote <newline>, to support tables with
multiple entries, rather than relying on commas and counts, etc)
and you'd then have do_mounts_dm.c open /dev/mapper/control directly and
issue proper DM ioctls rather than adding all your shim code. This last
bit of opening /dev/mapper/control from init needs more research -- not
sure if doing such a thing from kernel is viable/safe/acceptable.
Mike
^ permalink raw reply
* Re: [PATCH 1/2] RAID5: check_reshape() shouldn't call mddev_suspend
From: Artur Paszkiewicz @ 2016-02-26 10:47 UTC (permalink / raw)
To: NeilBrown, Shaohua Li, linux-raid
In-Reply-To: <87a8mol9xz.fsf@notabene.neil.brown.name>
On 02/25/2016 11:08 PM, NeilBrown wrote:
> On Fri, Feb 26 2016, Shaohua Li wrote:
>
>> check_reshape() is called from raid5d thread. raid5d thread shouldn't
>> call mddev_suspend(), because mddev_suspend() waits for all IO finish
>> but IO is handled in raid5d thread, we could easily deadlock here.
>>
>> Artur,
>> It would be great if you can verify this works for your test.
>>
>> Reported-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
>> Cc: NeilBrown <neilb@suse.com>
>> Signed-off-by: Shaohua Li <shli@fb.com>
>> ---
>> drivers/md/raid5.c | 18 ++++++++++++++++++
>> drivers/md/raid5.h | 2 ++
>> 2 files changed, 20 insertions(+)
>>
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index 7f770b0..c7fd070 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c
>> @@ -2089,6 +2089,14 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
>> unsigned long cpu;
>> int err = 0;
>>
>> + /*
>> + * Never shrink. And mddev_suspend() could deadlock if this is called
>> + * from raid5d. In that case, scribble_disks and scribble_sectors
>> + * should equal to new_disks and new_sectors
>> + */
>> + if (conf->scribble_disks >= new_disks &&
>> + conf->scribble_sectors >= new_sectors)
>> + return 0;
>> mddev_suspend(conf->mddev);
>> get_online_cpus();
>> for_each_present_cpu(cpu) {
>> @@ -2110,6 +2118,10 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
>> }
>> put_online_cpus();
>> mddev_resume(conf->mddev);
>> + if (!err) {
>> + conf->scribble_disks = new_disks;
>> + conf->scribble_sectors = new_sectors;
>> + }
>> return err;
>> }
>>
>> @@ -6413,6 +6425,12 @@ static int raid5_alloc_percpu(struct r5conf *conf)
>> }
>> put_online_cpus();
>>
>> + if (!err) {
>> + conf->scribble_disks = max(conf->raid_disks,
>> + conf->previous_raid_disks);
>> + conf->scribble_sectors = max(conf->chunk_sectors,
>> + conf->prev_chunk_sectors) / STRIPE_SECTORS;
>
> Here we set ->scribble_sectors to a number of stripes rather than a
> number of sectors. I think you need to remove the "/ STRIPE_SECTORS".
That's correct. I've tested the patch and it works well but only without
the "/ STRIPE_SECTORS".
Thanks,
Artur
^ permalink raw reply
* Re: [PATCH 1/2] RAID5: check_reshape() shouldn't call mddev_suspend
From: NeilBrown @ 2016-02-25 22:08 UTC (permalink / raw)
To: Shaohua Li, linux-raid; +Cc: artur.paszkiewicz
In-Reply-To: <b228d35f9a8e94969d667edbcc9d7f613d706b02.1456424579.git.shli@fb.com>
[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]
On Fri, Feb 26 2016, Shaohua Li wrote:
> check_reshape() is called from raid5d thread. raid5d thread shouldn't
> call mddev_suspend(), because mddev_suspend() waits for all IO finish
> but IO is handled in raid5d thread, we could easily deadlock here.
>
> Artur,
> It would be great if you can verify this works for your test.
>
> Reported-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
> Cc: NeilBrown <neilb@suse.com>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5.c | 18 ++++++++++++++++++
> drivers/md/raid5.h | 2 ++
> 2 files changed, 20 insertions(+)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 7f770b0..c7fd070 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -2089,6 +2089,14 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
> unsigned long cpu;
> int err = 0;
>
> + /*
> + * Never shrink. And mddev_suspend() could deadlock if this is called
> + * from raid5d. In that case, scribble_disks and scribble_sectors
> + * should equal to new_disks and new_sectors
> + */
> + if (conf->scribble_disks >= new_disks &&
> + conf->scribble_sectors >= new_sectors)
> + return 0;
> mddev_suspend(conf->mddev);
> get_online_cpus();
> for_each_present_cpu(cpu) {
> @@ -2110,6 +2118,10 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
> }
> put_online_cpus();
> mddev_resume(conf->mddev);
> + if (!err) {
> + conf->scribble_disks = new_disks;
> + conf->scribble_sectors = new_sectors;
> + }
> return err;
> }
>
> @@ -6413,6 +6425,12 @@ static int raid5_alloc_percpu(struct r5conf *conf)
> }
> put_online_cpus();
>
> + if (!err) {
> + conf->scribble_disks = max(conf->raid_disks,
> + conf->previous_raid_disks);
> + conf->scribble_sectors = max(conf->chunk_sectors,
> + conf->prev_chunk_sectors) / STRIPE_SECTORS;
Here we set ->scribble_sectors to a number of stripes rather than a
number of sectors. I think you need to remove the "/ STRIPE_SECTORS".
Otherwise:
Reviewed-by: NeilBrown <neilb@suse.com>
Thanks,
NeilBrown
> + }
> return err;
> }
>
> diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
> index a415e1c..ae6068d 100644
> --- a/drivers/md/raid5.h
> +++ b/drivers/md/raid5.h
> @@ -510,6 +510,8 @@ struct r5conf {
> * conversions
> */
> } __percpu *percpu;
> + int scribble_disks;
> + int scribble_sectors;
> #ifdef CONFIG_HOTPLUG_CPU
> struct notifier_block cpu_notify;
> #endif
> --
> 2.6.5
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: live lock regression in raid5 reshape
From: NeilBrown @ 2016-02-25 22:01 UTC (permalink / raw)
To: Shaohua Li, yuanhan.liu; +Cc: linux-raid, artur.paszkiewicz
In-Reply-To: <20160225190712.GA2390@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 2155 bytes --]
On Fri, Feb 26 2016, Shaohua Li wrote:
> Hi,
>
> I hit a live lock in reshape test, which is introduced by:
>
> e9e4c377e2f563892c50d1d093dd55c7d518fc3d(md/raid5: per hash value and exclusive wait_for_stripe)
>
> The problem is get_active_stripe waits on conf->wait_for_stripe[hash]. Assume
> hash is 0. My test release stripes in this order:
> - release all stripes with hash 0
> - get_active_stripe still sleeps since active_stripes > max_nr_stripes * 3 / 4
> - release all stripes with hash other than 0. active_stripes becomes 0
> - get_active_stripe still sleeps, since nobody wakes up wait_for_stripe[0]
>
> The system live locks. The problem is active_stripes isn't a per-hash count.
> Revert the patch makes the lock go away.
>
> I didn't come out a solution yet except reverting the patch. Making
> active_stripes per-hash is a candidate, but not sure if there is thundering
> herd problem because each hash will have less stripes. On the other hand, I'm
> wondering if the patch makes sense now. The commit log declares the issue
> happens with limited stripes, but now stripe count is automatically increased.
>
->active_stripes does seem to be the core of the problem here.
The purpose of the comparison with max_nr_stripes*3/4 was to encourage
requests to be handled in large batches rather than dribbling out one at
a time. That should encourage the creation of full stripe writes. I
think it does (or at least: did) help but we know it isn't perfect.
There might be a better way.
If two threads are each writing full stripes of data, we would prefer
one could allocate a full set of stripe_heads and the other one get
nothing for a little while, rather than each get half of the number of
stripe_heads that they need.
Possibly we would could impose this restriction only on the first
stripe_head in a stripe (i.e. the start of a chunk). That should have
much the same effect but wouldn't cause the problem you are seeing.
Certainly backing this out is simplest (particularly if you want to send
it to -stable). I suspect it would be best to ultimately keep the
hashed wait queues if we can avoid the livelock.
thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: raid5d hangs when stopping an array during reshape
From: Dan Williams @ 2016-02-25 19:58 UTC (permalink / raw)
To: Shaohua Li; +Cc: Artur Paszkiewicz, NeilBrown, linux-raid
In-Reply-To: <20160225191739.GB2390@kernel.org>
On Thu, Feb 25, 2016 at 11:17 AM, Shaohua Li <shli@kernel.org> wrote:
> On Thu, Feb 25, 2016 at 10:48:45AM -0800, Dan Williams wrote:
>> On Thu, Feb 25, 2016 at 10:42 AM, Shaohua Li <shli@kernel.org> wrote:
>> > On Thu, Feb 25, 2016 at 05:05:17PM +0100, Artur Paszkiewicz wrote:
>> >> On 02/25/2016 02:17 AM, Shaohua Li wrote:
>> >> > On Thu, Feb 25, 2016 at 11:31:04AM +1100, Neil Brown wrote:
>> >> >> On Thu, Feb 25 2016, Shaohua Li wrote:
>> >> >>
>> >> >>>
>> >> >>> As for the bug, write requests run in raid5d, mddev_suspend() waits for all IO,
>> >> >>> which waits for the write requests. So this is a clear deadlock. I think we
>> >> >>> should delete the check_reshape() in md_check_recovery(). If we change
>> >> >>> layout/disks/chunk_size, check_reshape() is already called. If we start an
>> >> >>> array, the .run() already handles new layout. There is no point
>> >> >>> md_check_recovery() check_reshape() again.
>> >> >>
>> >> >> Are you sure?
>> >> >> Did you look at the commit which added that code?
>> >> >> commit b4c4c7b8095298ff4ce20b40bf180ada070812d0
>> >> >>
>> >> >> When there is an IO error, reshape (or resync or recovery) will abort
>> >> >> and then possibly be automatically restarted.
>> >> >
>> >> > thanks pointing out this.
>> >> >> Without the check here a reshape might be attempted on an array which
>> >> >> has failed. Not sure if that would be harmful, but it would certainly
>> >> >> be pointless.
>> >> >>
>> >> >> But you are right that this is causing the problem.
>> >> >> Maybe we should keep track of the size of the 'scribble' arrays and only
>> >> >> call resize_chunks if the size needs to change? Similar to what
>> >> >> resize_stripes does.
>> >> >
>> >> > yep, this is my first solution, but think check_reshape() is useless here
>> >> > later, apparently miss the restart case. I'll go this way.
>> >>
>> >> My idea was to replace mddev_suspend()/mddev_resume() in resize_chunks()
>> >> with a rw lock that would prevent collisions with raid_run_ops(), since
>> >> scribble is used only there. But if the parity operations are executed
>> >> asynchronously this would also need to wait until all the submitted
>> >> operations have completed. Seems a bit overkill, but I came up with
>> >> this:
>> >
>> > Looks it should work, but it's overkill indead, especially the extra lock, we
>> > can replace it with srcu though. The 'track scribble array size' is much
>> > simpler, so I'd prefer that way. In the future, we probably should move
>> > resize_stripes()/resize_chunks() to .start_reshape().
>> > resize_stripes()/resize_chunks() sounds not qualified as .check_reshape().
>> >
>>
>> Any time any linux-raid mail mentions the raid5_run_ops infrastructure
>> I am prompted to remind that async_tx needs to die and be up leveled
>> to md directly. The "help wanted" request is still pending.
>
> A quick search shows async_tx has another user: exofs
Yes, same up leveling of api internals into the user directly needs to
be done there as well. More help wanted :-).
^ permalink raw reply
* Re: raid5d hangs when stopping an array during reshape
From: Shaohua Li @ 2016-02-25 19:17 UTC (permalink / raw)
To: Dan Williams; +Cc: Artur Paszkiewicz, NeilBrown, linux-raid
In-Reply-To: <CAPcyv4jTrYL7ctYgZ+P_Cb=z_eN9oRR9Z0OiyUu6jS+aphm0ag@mail.gmail.com>
On Thu, Feb 25, 2016 at 10:48:45AM -0800, Dan Williams wrote:
> On Thu, Feb 25, 2016 at 10:42 AM, Shaohua Li <shli@kernel.org> wrote:
> > On Thu, Feb 25, 2016 at 05:05:17PM +0100, Artur Paszkiewicz wrote:
> >> On 02/25/2016 02:17 AM, Shaohua Li wrote:
> >> > On Thu, Feb 25, 2016 at 11:31:04AM +1100, Neil Brown wrote:
> >> >> On Thu, Feb 25 2016, Shaohua Li wrote:
> >> >>
> >> >>>
> >> >>> As for the bug, write requests run in raid5d, mddev_suspend() waits for all IO,
> >> >>> which waits for the write requests. So this is a clear deadlock. I think we
> >> >>> should delete the check_reshape() in md_check_recovery(). If we change
> >> >>> layout/disks/chunk_size, check_reshape() is already called. If we start an
> >> >>> array, the .run() already handles new layout. There is no point
> >> >>> md_check_recovery() check_reshape() again.
> >> >>
> >> >> Are you sure?
> >> >> Did you look at the commit which added that code?
> >> >> commit b4c4c7b8095298ff4ce20b40bf180ada070812d0
> >> >>
> >> >> When there is an IO error, reshape (or resync or recovery) will abort
> >> >> and then possibly be automatically restarted.
> >> >
> >> > thanks pointing out this.
> >> >> Without the check here a reshape might be attempted on an array which
> >> >> has failed. Not sure if that would be harmful, but it would certainly
> >> >> be pointless.
> >> >>
> >> >> But you are right that this is causing the problem.
> >> >> Maybe we should keep track of the size of the 'scribble' arrays and only
> >> >> call resize_chunks if the size needs to change? Similar to what
> >> >> resize_stripes does.
> >> >
> >> > yep, this is my first solution, but think check_reshape() is useless here
> >> > later, apparently miss the restart case. I'll go this way.
> >>
> >> My idea was to replace mddev_suspend()/mddev_resume() in resize_chunks()
> >> with a rw lock that would prevent collisions with raid_run_ops(), since
> >> scribble is used only there. But if the parity operations are executed
> >> asynchronously this would also need to wait until all the submitted
> >> operations have completed. Seems a bit overkill, but I came up with
> >> this:
> >
> > Looks it should work, but it's overkill indead, especially the extra lock, we
> > can replace it with srcu though. The 'track scribble array size' is much
> > simpler, so I'd prefer that way. In the future, we probably should move
> > resize_stripes()/resize_chunks() to .start_reshape().
> > resize_stripes()/resize_chunks() sounds not qualified as .check_reshape().
> >
>
> Any time any linux-raid mail mentions the raid5_run_ops infrastructure
> I am prompted to remind that async_tx needs to die and be up leveled
> to md directly. The "help wanted" request is still pending.
A quick search shows async_tx has another user: exofs
^ permalink raw reply
* live lock regression in raid5 reshape
From: Shaohua Li @ 2016-02-25 19:07 UTC (permalink / raw)
To: yuanhan.liu, neilb; +Cc: linux-raid, artur.paszkiewicz
Hi,
I hit a live lock in reshape test, which is introduced by:
e9e4c377e2f563892c50d1d093dd55c7d518fc3d(md/raid5: per hash value and exclusive wait_for_stripe)
The problem is get_active_stripe waits on conf->wait_for_stripe[hash]. Assume
hash is 0. My test release stripes in this order:
- release all stripes with hash 0
- get_active_stripe still sleeps since active_stripes > max_nr_stripes * 3 / 4
- release all stripes with hash other than 0. active_stripes becomes 0
- get_active_stripe still sleeps, since nobody wakes up wait_for_stripe[0]
The system live locks. The problem is active_stripes isn't a per-hash count.
Revert the patch makes the lock go away.
I didn't come out a solution yet except reverting the patch. Making
active_stripes per-hash is a candidate, but not sure if there is thundering
herd problem because each hash will have less stripes. On the other hand, I'm
wondering if the patch makes sense now. The commit log declares the issue
happens with limited stripes, but now stripe count is automatically increased.
Yuanhan, could you please check if performance changes with the patch reverted
in latest kernel?
Thanks,
Shaohua
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox