* Re: Recent drive errors
From: Thomas Fjellstrom @ 2015-05-19 16:07 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <555B4E0F.9020306@turmel.org>
On Tue 19 May 2015 10:51:59 AM you wrote:
> On 05/19/2015 10:32 AM, Thomas Fjellstrom wrote:
> > On Tue 19 May 2015 09:23:20 AM Phil Turmel wrote:
> >> Depends. In a properly functioning array that gets scrubbed
> >> occasionally, or sufficiently heavy use to read the entire contents
> >> occasionally, the UREs get rewritten by MD right away. Any UREs then
> >> only show up once.
> >
> > I have made sure that it's doing regular scrubs, and regular SMART scans.
> > This time...
>
> Yes, and this drive was kicked out. Because it wouldn't be listening
> when MD tried to write over the error it found.
I didn't actually re-install this drive after the last time it was kicked out,
which was when i didn't have regular scrubs (actually, it may have been, as it
was probably the only thing that would cause activity on that array for many
many months) or smart tests set up. I noticed the high start stop count, and
the 5 errors, and decided to keep it out of the new array. I seem to recall
one or more drives having suspiciously high start stop counts that then went
on to fail, but it seems that isn't true, one of them is still in use (64k
start stop events apparently, or it maxed out the counter).
Basically, I had an unused array of 3TB seagates, it sat doing virtually
nothing but spinning its platters for quite a long time due to lack of time on
my part, and some time between last summer and winter, it kicked out two
drives all on its own. It was probably the monthly scrub. After I got back
from a three month long trip, I rebuilt that array (and my main NAS, which
also kicked out two drives... But thats a story for another time) with four of
the old Seagates, and one new WD red. It was this drive that I removed at that
time because it looked suspicious. Sadly, about a month or two later, a second
drive got kicked out and was unambiguously faulty (thousands, if not 10k+
reallocated sectors), so I replaced it with two new WD Reds, and reshaped to a
raid6. After that, I just decided to re-check the first drive to drop out,
just to be safe, and here we are...
I'm running a badblocks -w on the drive as we speak, it'll probably be done in
a day or two. We'll see if it changes anything. It's not exactly writing
noise, but it aught to do the trick.
> I posted this link earlier, but it is particularly relevant:
> http://marc.info/?l=linux-raid&m=133665797115876&w=2
>
> >> Interesting. I suspect that if you wipe that disk with noise, read it
> >> all back, and wipe it again, you'll have a handful of relocations.
> >
> > It looks like each one of the blocks in that display is 128KiB. Which i
> > think means those red blocks aren't very far apart. Maybe 80MiB apart?
> > Would it reallocate all of those? That'd be a lot of reallocated sectors.
>
> Drives will only reallocate where a previous read failed (making it
> pending), then write and follow-up verification fails. In general,
> writes are unverified at the time of write (or your write performance
> would be dramatically slower than read).
Right. I was just thinking about how you mentioned that I'd get a handful of
reallocations based on the latency shown in the image I posted. It's a lot of
sectors that seem to be affected by the latency spikes, so I assumed (probably
wrongly) that many of them may be reallocated afterwards.
If this drive ends up not reallocating a single sector, or only a few, I may
just keep it around as a hot spare, though i feel that's not the best idea, if
it is degrading, then when it actually goes to use that disk it has a higher
chance of failing.
> >> You have it backwards. If you have WD Reds, they are correct out of the
> >> box. It's when you *don't* have ERC support, or you only have desktop
> >> ERC, that you need to take special action.
> >
> > I was under the impression you still had to enable ERC on boot. And I
> > /thought/ I read that you still want to adjust the timeouts, though not
> > the
> > same as for consumer drives.
>
> Desktop / consumer drives that support ERC typically ship with it
> disabled, so they behave just like drives that don't support it at all.
> So a boot script would enable ERC on drives where it can (and not
> already OK), and set long driver timeouts on the rest.
>
> Any drive that claims "raid" compatibility will have ERC enabled by
> default. Typically 7.0 seconds. WD Reds do. Enterprise drives do, and
> have better URE specs, too.
Good to know.
> >> If you have consumer grade drives in a raid array, and you don't have
> >> boot scripts or udev rules to deal with timeout mismatch, your *ss is
> >> hanging in the wind. The links in my last msg should help you out.
> >
> > There was some talk of ERC/TLER and md. I'll still have to find or write a
> > script to properly set up timeouts and enable TLER on drives capable of it
> > (that don't come with it enabled by default).
>
> Before I got everything onto proper drives, I just put what I needed
> into rc.local.
It's going to be a long time before I can swap out the rest of the seagates. I
just can't justify the cost atm, especially as its the backup for my main nas
which used to be all 2TB seagates, but has since been retrofitted with two WD
reds as two had thousands of reallocated sectors, funny thing is one of the
seagates had already been replaced prior to that, so 3 out of 5 of the
original setup failed. And before that, at least two 1TB seagates have failed
on me (out of 7ish), I think maybe one 640G one went, and a couple 320s went.
I won't blame the two 80s that I had that failed on seagate though, that was a
power supply fault. Took out two drives out of five (total), memory, and made
the motherboard a bit flaky.
Just a little bit jaded when it comes to seagates these days, but i still
can't just up and swap them all out, even if its a good idea. If I had the
money, I wouldn't mind just replacing them all with enterprise/nearline or NAS
drives, and slap the seagates in a big zfs pool or something for some scratch
space or just sell them...
> Chris Murphy posted some udev rules that will likely work for you. I
> haven't tried them myself, though.
>
> https://www.marc.info/?l=linux-raid&m=142487508806844&w=3
Thanks :)
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: Recent drive errors
From: Phil Turmel @ 2015-05-19 14:51 UTC (permalink / raw)
To: thomas; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <2278721.7NtVspC26F@balsa>
On 05/19/2015 10:32 AM, Thomas Fjellstrom wrote:
> On Tue 19 May 2015 09:23:20 AM Phil Turmel wrote:
>> Depends. In a properly functioning array that gets scrubbed
>> occasionally, or sufficiently heavy use to read the entire contents
>> occasionally, the UREs get rewritten by MD right away. Any UREs then
>> only show up once.
>
> I have made sure that it's doing regular scrubs, and regular SMART scans. This
> time...
Yes, and this drive was kicked out. Because it wouldn't be listening
when MD tried to write over the error it found.
I posted this link earlier, but it is particularly relevant:
http://marc.info/?l=linux-raid&m=133665797115876&w=2
>> Interesting. I suspect that if you wipe that disk with noise, read it
>> all back, and wipe it again, you'll have a handful of relocations.
>
> It looks like each one of the blocks in that display is 128KiB. Which i think
> means those red blocks aren't very far apart. Maybe 80MiB apart? Would it
> reallocate all of those? That'd be a lot of reallocated sectors.
Drives will only reallocate where a previous read failed (making it
pending), then write and follow-up verification fails. In general,
writes are unverified at the time of write (or your write performance
would be dramatically slower than read).
>> You have it backwards. If you have WD Reds, they are correct out of the
>> box. It's when you *don't* have ERC support, or you only have desktop
>> ERC, that you need to take special action.
>
> I was under the impression you still had to enable ERC on boot. And I
> /thought/ I read that you still want to adjust the timeouts, though not the
> same as for consumer drives.
Desktop / consumer drives that support ERC typically ship with it
disabled, so they behave just like drives that don't support it at all.
So a boot script would enable ERC on drives where it can (and not
already OK), and set long driver timeouts on the rest.
Any drive that claims "raid" compatibility will have ERC enabled by
default. Typically 7.0 seconds. WD Reds do. Enterprise drives do, and
have better URE specs, too.
>> If you have consumer grade drives in a raid array, and you don't have
>> boot scripts or udev rules to deal with timeout mismatch, your *ss is
>> hanging in the wind. The links in my last msg should help you out.
>
> There was some talk of ERC/TLER and md. I'll still have to find or write a
> script to properly set up timeouts and enable TLER on drives capable of it
> (that don't come with it enabled by default).
Before I got everything onto proper drives, I just put what I needed
into rc.local.
Chris Murphy posted some udev rules that will likely work for you. I
haven't tried them myself, though.
https://www.marc.info/?l=linux-raid&m=142487508806844&w=3
Phil
^ permalink raw reply
* Re: Recent drive errors
From: Thomas Fjellstrom @ 2015-05-19 14:32 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <555B3948.1030602@turmel.org>
On Tue 19 May 2015 09:23:20 AM Phil Turmel wrote:
> On 05/19/2015 08:50 AM, Thomas Fjellstrom wrote:
> > On Tue 19 May 2015 08:34:55 AM Phil Turmel wrote:
> >> Based on the smart report, this drive is perfectly healthy. A small
> >> number of uncorrectable read errors is normal in the life of any drive.
> >
> > Is it perfectly normal for the same sector to be reported uncorrectable 5
> > times in a row like it did?
>
> Yes, if you keep trying to read it. Unreadable sectors stay unreadable,
> generally, until they are re-written. That's the first opportunity the
> drive has to decide if a relocation is necessary.
>
> > How many UREs are considered "ok"? Tens, hundreds, thousands, tens of
> > thousands?
>
> Depends. In a properly functioning array that gets scrubbed
> occasionally, or sufficiently heavy use to read the entire contents
> occasionally, the UREs get rewritten by MD right away. Any UREs then
> only show up once.
I have made sure that it's doing regular scrubs, and regular SMART scans. This
time...
> In a desktop environment, or non-raid, or improperly configured raid,
> the UREs will build up, and get reported on every read attempt.
>
> Most consumer-grade drives claim a URE average below 1 per 1E14 bits
> read. So by the end of their warranty period, getting one every 12TB
> read wouldn't be unusual. This sort of thing follows a Poisson
> distribution:
>
> http://marc.info/?l=linux-raid&m=135863964624202&w=2
>
> > These drives have been barely used. Most of their life, they were either
> > off, or not actually being used. (it took a while to collect enough 3TB
> > drives, and then find time to build the array, and set it up as a regular
> > backup of my 11TB nas).
>
> While being off may lengthen their life somewhat, the magnetic domains
> on these things are so small that some degradation will happen just
> sitting there. Diffusion in the p- and n-doped regions of the
> semiconductors is also happening while sitting unused, degrading the
> electronics.
>
> >> It has no relocations, and no pending sectors. The latency spikes are
> >>
> >> likely due to slow degradation of some sectors that the drive is having
> >> to internally retry to read successfully. Again, normal.
> >
> > The latency spikes are /very/ regular and theres quite a lot of them.
> > See: http://i.imgur.com/QjTl6o3.png
>
> Interesting. I suspect that if you wipe that disk with noise, read it
> all back, and wipe it again, you'll have a handful of relocations.
It looks like each one of the blocks in that display is 128KiB. Which i think
means those red blocks aren't very far apart. Maybe 80MiB apart? Would it
reallocate all of those? That'd be a lot of reallocated sectors.
> Your latency test will show different numbers then, as the head will
> have to seek to the spare sector and back whenever you read through one
> of those spots.
>
> Or the rewrites will fix them all, and you'll have no further problems.
> Hard to tell. Bottom line is that drives can't fix any problems they
> have unless they are *written* in previously identified problem areas.
>
> >> I own some "DM001" drives -- they are unsuited to raid duty as they
> >> don't support ERC. So, out of the box, they are time bombs for any
> >> array you put them in. That's almost certainly why they were ejected
> >> from your array.
> >>
> >> If you absolutely must use them, you *must* set the *driver* timeout to
> >> 120 seconds or more.
> >
> > I've been planning on looking into the ERC stuff. I now actually have some
> > drives that do support ERC, so it'll be interesting to make sure
> > everything is set up properly.
>
> You have it backwards. If you have WD Reds, they are correct out of the
> box. It's when you *don't* have ERC support, or you only have desktop
> ERC, that you need to take special action.
I was under the impression you still had to enable ERC on boot. And I
/thought/ I read that you still want to adjust the timeouts, though not the
same as for consumer drives.
> If you have consumer grade drives in a raid array, and you don't have
> boot scripts or udev rules to deal with timeout mismatch, your *ss is
> hanging in the wind. The links in my last msg should help you out.
There was some talk of ERC/TLER and md. I'll still have to find or write a
script to properly set up timeouts and enable TLER on drives capable of it
(that don't come with it enabled by default).
> Also, I noticed that you used "smartctl -a" to post a complete report of
> your drive's status. It's not complete. You should get in the habit of
> using "smartctl -x" instead, so you see the ERC status, too.
Good to know. Thanks.
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Data corruption after resizing partition, when using bitmaps
From: Jim Paris @ 2015-05-19 14:12 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1705 bytes --]
I had a raid1 mirror consisting of big partitions on two disks.
The first disk was 2TB, partitioned like this:
[--sda1(128M)--][-------sda2(~2T)--------------]
The second disk was 3TB, partitioned like this:
[--sdb1(128M)--][-------sdb2(~3T)------------------------------------]
sda2 and sdb2 were part of the array, which was only ~2TB in size due
to the smaller disk.
I realized that I needed to add a BIOS boot partition to the 3TB disk,
so I removed sdb2 from the array, and repartitioned sdb like this:
[--sdb1(128M)--][--sdb2(1M)--][-------sdb3(~3T)----------------------]
Then I added sdb3 to the array. And lost all my data. :(
What happened was that the last sector of the big partition did not
change location. So the metadata (0.90) at the end was still present.
Adding sdb3 to the array was considered a "re-add" because the UUID
and array sizes still matched the array, even though the partition
itself shrank. And the resync was thus guided by an out-of-date
bitmap, which caused very little data to actually be written to sdb3,
so half the reads from the array started returning junk. Once the
filesystem got involved, the result was rapid corruption.
If I had not been using write-intent bitmaps, everything would have
worked fine. I only recently started using bitmaps, and never had any
problems with adjusting partitions like this before that.
Perhaps mdadm can be more careful here -- for example, maybe checking
the actual device size and not just the "used dev size" when
determining whether to trust the bitmap.
I wrote a script (attached) to recreate what happened, using some loop
devices. It works fine if BITMAP=none, and fails with BITMAP=internal.
Jim
[-- Attachment #2: repro.sh --]
[-- Type: application/x-sh, Size: 3181 bytes --]
^ permalink raw reply
* Re: Recent drive errors
From: Phil Turmel @ 2015-05-19 13:23 UTC (permalink / raw)
To: thomas; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <84264713.v03zHsT0Cj@balsa>
On 05/19/2015 08:50 AM, Thomas Fjellstrom wrote:
> On Tue 19 May 2015 08:34:55 AM Phil Turmel wrote:
>> Based on the smart report, this drive is perfectly healthy. A small
>> number of uncorrectable read errors is normal in the life of any drive.
>
> Is it perfectly normal for the same sector to be reported uncorrectable 5
> times in a row like it did?
Yes, if you keep trying to read it. Unreadable sectors stay unreadable,
generally, until they are re-written. That's the first opportunity the
drive has to decide if a relocation is necessary.
> How many UREs are considered "ok"? Tens, hundreds, thousands, tens of
> thousands?
Depends. In a properly functioning array that gets scrubbed
occasionally, or sufficiently heavy use to read the entire contents
occasionally, the UREs get rewritten by MD right away. Any UREs then
only show up once.
In a desktop environment, or non-raid, or improperly configured raid,
the UREs will build up, and get reported on every read attempt.
Most consumer-grade drives claim a URE average below 1 per 1E14 bits
read. So by the end of their warranty period, getting one every 12TB
read wouldn't be unusual. This sort of thing follows a Poisson
distribution:
http://marc.info/?l=linux-raid&m=135863964624202&w=2
> These drives have been barely used. Most of their life, they were either off,
> or not actually being used. (it took a while to collect enough 3TB drives, and
> then find time to build the array, and set it up as a regular backup of my
> 11TB nas).
While being off may lengthen their life somewhat, the magnetic domains
on these things are so small that some degradation will happen just
sitting there. Diffusion in the p- and n-doped regions of the
semiconductors is also happening while sitting unused, degrading the
electronics.
>> It has no relocations, and no pending sectors. The latency spikes are
>> likely due to slow degradation of some sectors that the drive is having
>> to internally retry to read successfully. Again, normal.
>
> The latency spikes are /very/ regular and theres quite a lot of them.
> See: http://i.imgur.com/QjTl6o3.png
Interesting. I suspect that if you wipe that disk with noise, read it
all back, and wipe it again, you'll have a handful of relocations.
Your latency test will show different numbers then, as the head will
have to seek to the spare sector and back whenever you read through one
of those spots.
Or the rewrites will fix them all, and you'll have no further problems.
Hard to tell. Bottom line is that drives can't fix any problems they
have unless they are *written* in previously identified problem areas.
>> I own some "DM001" drives -- they are unsuited to raid duty as they
>> don't support ERC. So, out of the box, they are time bombs for any
>> array you put them in. That's almost certainly why they were ejected
>> from your array.
>>
>> If you absolutely must use them, you *must* set the *driver* timeout to
>> 120 seconds or more.
>
> I've been planning on looking into the ERC stuff. I now actually have some
> drives that do support ERC, so it'll be interesting to make sure everything is
> set up properly.
You have it backwards. If you have WD Reds, they are correct out of the
box. It's when you *don't* have ERC support, or you only have desktop
ERC, that you need to take special action.
If you have consumer grade drives in a raid array, and you don't have
boot scripts or udev rules to deal with timeout mismatch, your *ss is
hanging in the wind. The links in my last msg should help you out.
Also, I noticed that you used "smartctl -a" to post a complete report of
your drive's status. It's not complete. You should get in the habit of
using "smartctl -x" instead, so you see the ERC status, too.
Phil
^ permalink raw reply
* Re: Backup raid, power down questions.
From: Wols Lists @ 2015-05-19 12:54 UTC (permalink / raw)
To: Roman Mamedov, Wilson, Jonathan; +Cc: linux-raid
In-Reply-To: <20150519174606.4f7e8e75@natsu>
On 19/05/15 13:46, Roman Mamedov wrote:
> On Tue, 19 May 2015 13:36:43 +0100 "Wilson, Jonathan"
> <piercing_male@hotmail.com> wrote:
>
>> I am looking to create a second raid6 set which will only be used
>> to do periodical backups of the main raid6, probably using rsync.
>> (all drives will be tler'able WD reds)
>
>> Obviously when the system boots all the raids will start up
>
> With both RAIDs in the same case, sharing a PSU, connected to the
> same system? I'd say forget about it, that's a terrible way to have
> a "backup", too many things that can go wrong affecting both RAIDs
> at once. Starting with a power supply failing and frying all the
> drives, and ending with an unauthorized access via some root
> exploit, with the intruder deleting all data or dd'ing zeroes to
> all drives.
>
> Throw together a separate inexpensive computer and back up to that.
> Can even power off it entirely when not in use and power-on either
> by Wake On Lan or by schedule (most x86 BIOSes have a feature to
> power-on the machine daily at a specified time).
>
And while it's a bit off-topic here, I'd seriously look at that backup
computer using btrfs rather than raid. You can configure it to
duplicate all files to two disks, giving you a fault tolerant backup,
and you can set it to boot up, snapshot the drive, do an IN PLACE
rsync (which means the new snapshot/backup only uses diskspace for
stuff that's changed), and shuts down again or whatever. So you can
have multiple backups going back yonks without too much disk space
being wasted.
And while btrfs does have some pretty serious flaws still, it seems
the bulk of them are triggered by the system running out of space.
That shouldn't be a problem if you keep an eye on it.
Cheers,
Wol
^ permalink raw reply
* Re: Recent drive errors
From: Thomas Fjellstrom @ 2015-05-19 12:50 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <555B2DEF.60905@turmel.org>
On Tue 19 May 2015 08:34:55 AM Phil Turmel wrote:
> Hi Thomas,
>
> On 05/19/2015 07:08 AM, Thomas Fjellstrom wrote:
> > Hi,
> >
> > I have this one drive that dropped out of one of my arrays once. It shows
> > UNC errors in SMART (log appended), and Reported_Uncorrect is 5. There
> > are no smart test failures or any other SMART values that look
> > spectacularly wrong, other than maybe Load_Cycle_Count which is 10625
> > (these seagates used to constantly park and unpark before i updated the
> > firmware).
> >
> > I'm wondering whether or not this drive is still safe to use. I feel like
> > I
> > can't trust it, especially after all the other Seagates I had that failed
> > in the past few years. I'm running a tool called whdd on it right now and
> > it shows very consistent latency spikes above 150ms. Really, I'm
> > wondering if this drive is RMAable as is, or if i have to wait for it to
> > degrade further as i have another drive with like 10k reallocated sectors
> > to send in. I have already replaced both with WD Red's so I can do
> > whatever tests are needed to figure it out.
>
> Based on the smart report, this drive is perfectly healthy. A small
> number of uncorrectable read errors is normal in the life of any drive.
Is it perfectly normal for the same sector to be reported uncorrectable 5
times in a row like it did?
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of
thousands?
These drives have been barely used. Most of their life, they were either off,
or not actually being used. (it took a while to collect enough 3TB drives, and
then find time to build the array, and set it up as a regular backup of my
11TB nas).
> It has no relocations, and no pending sectors. The latency spikes are
> likely due to slow degradation of some sectors that the drive is having
> to internally retry to read successfully. Again, normal.
The latency spikes are /very/ regular and theres quite a lot of them.
See: http://i.imgur.com/QjTl6o3.png
> I own some "DM001" drives -- they are unsuited to raid duty as they
> don't support ERC. So, out of the box, they are time bombs for any
> array you put them in. That's almost certainly why they were ejected
> from your array.
>
> If you absolutely must use them, you *must* set the *driver* timeout to
> 120 seconds or more.
I've been planning on looking into the ERC stuff. I now actually have some
drives that do support ERC, so it'll be interesting to make sure everything is
set up properly.
> HTH,
Thank you :)
> Phil
>
> http://marc.info/?l=linux-raid&m=133761065622164&w=2
> http://marc.info/?l=linux-raid&m=135811522817345&w=1
> http://marc.info/?l=linux-raid&m=133761065622164&w=2
> http://marc.info/?l=linux-raid&m=133665797115876&w=2
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: Backup raid, power down questions.
From: Roman Mamedov @ 2015-05-19 12:46 UTC (permalink / raw)
To: Wilson, Jonathan; +Cc: linux-raid
In-Reply-To: <BLU437-SMTP474BC966B5944E3445207D98C30@phx.gbl>
[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]
On Tue, 19 May 2015 13:36:43 +0100
"Wilson, Jonathan" <piercing_male@hotmail.com> wrote:
> I am looking to create a second raid6 set which will only be used to do
> periodical backups of the main raid6, probably using rsync. (all drives
> will be tler'able WD reds)
> Obviously when the system boots all the raids will start up
With both RAIDs in the same case, sharing a PSU, connected to the same
system? I'd say forget about it, that's a terrible way to have a "backup",
too many things that can go wrong affecting both RAIDs at once. Starting with
a power supply failing and frying all the drives, and ending with an
unauthorized access via some root exploit, with the intruder deleting all data
or dd'ing zeroes to all drives.
Throw together a separate inexpensive computer and back up to that. Can even
power off it entirely when not in use and power-on either by Wake On Lan or by
schedule (most x86 BIOSes have a feature to power-on the machine daily at a
specified time).
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Backup raid, power down questions.
From: Wilson, Jonathan @ 2015-05-19 12:36 UTC (permalink / raw)
To: linux-raid
I am looking to create a second raid6 set which will only be used to do
periodical backups of the main raid6, probably using rsync. (all drives
will be tler'able WD reds)
Obviously when the system boots all the raids will start up, but I am
looking to run some kind of script that will force the drives into a
"hard sleep" for want of a better term.
At the moment none of the drives have any sleep or power down timers
set, in part to fix the idle3 bug increasing the load cycles count and
this will be applied to the new drives as well.
My questions are:
1) I am using EXT4, will I need to "do something" to make sure that it
flushes all changes to disk (which I assume means "flush to MD layer"),
if so what?
2) Likewise with the MD layer, is there any command that forces it to
flush to disk any cached changes?
3) I am guessing that I need to issue a hdparm to each disk in the raid
set to send it either to sleep or spin down or some such state, as the
backup raid will not be accessed except during backup writes or data
recovery it can go into the most "asleep" state available. 2 questions,
what hdparm setting? also is there a smartctl equivalent?
4) As the drives might take a length of time to waken, does this need to
be considered by the /sys/class/scsi_device/*/device/timeout value to
prevent that layer deciding the disk has gone walkabout if it takes to
long to wake up?
5) If there isn't any easy way to make sure the raid/file system has
flushed any changes would a reasonable method be to unmount the file
system, then stop the raid, then send the drives to sleep?
6) Obviously if this (5) is the only way or best way I am guessing the
backup script would just need to assemble the array to start it (--scan
--assemble --uuid=) which should wake up the drives and then mount the
file system, then call the script that performs the unmount,stop,sleep?
7) does sending a drive "to sleep" make sure that everything in the
drive cache is sent to disk?
8) I am using debian jessie, and the smartd checks for changes in smart
values (temp is the one that is listed most, almost exclusively,
in /var/log/syslog) will this cause my drives to wake up or is it clever
enough to see the drive is sleeping so won't wake it?
Finally, does anyone else do something similar or do most users either
just leave the backup array running, which seems a bit of a waste both
in power and drive lifespan terms especially if backups are fairly
infrequent (this won't be daily, probably weekly or monthly at most, so
not having to sleep/wake drives on a daily basis putting a different
type of wear on them)?
Jon
^ permalink raw reply
* Re: Recent drive errors
From: Phil Turmel @ 2015-05-19 12:34 UTC (permalink / raw)
To: thomas, linux-raid@vger.kernel.org
In-Reply-To: <3296560.sGbn0HyrQY@balsa>
Hi Thomas,
On 05/19/2015 07:08 AM, Thomas Fjellstrom wrote:
> Hi,
>
> I have this one drive that dropped out of one of my arrays once. It shows UNC
> errors in SMART (log appended), and Reported_Uncorrect is 5. There are no
> smart test failures or any other SMART values that look spectacularly wrong,
> other than maybe Load_Cycle_Count which is 10625 (these seagates used to
> constantly park and unpark before i updated the firmware).
>
> I'm wondering whether or not this drive is still safe to use. I feel like I
> can't trust it, especially after all the other Seagates I had that failed in
> the past few years. I'm running a tool called whdd on it right now and it
> shows very consistent latency spikes above 150ms. Really, I'm wondering if
> this drive is RMAable as is, or if i have to wait for it to degrade further as
> i have another drive with like 10k reallocated sectors to send in. I have
> already replaced both with WD Red's so I can do whatever tests are needed to
> figure it out.
Based on the smart report, this drive is perfectly healthy. A small
number of uncorrectable read errors is normal in the life of any drive.
It has no relocations, and no pending sectors. The latency spikes are
likely due to slow degradation of some sectors that the drive is having
to internally retry to read successfully. Again, normal.
I own some "DM001" drives -- they are unsuited to raid duty as they
don't support ERC. So, out of the box, they are time bombs for any
array you put them in. That's almost certainly why they were ejected
from your array.
If you absolutely must use them, you *must* set the *driver* timeout to
120 seconds or more.
HTH,
Phil
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=133665797115876&w=2
^ permalink raw reply
* Re: raid5 reshape is stuck
From: Xiao Ni @ 2015-05-19 11:10 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <2043891461.15360424.1431673224036.JavaMail.zimbra@redhat.com>
----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: linux-raid@vger.kernel.org
> Sent: Friday, May 15, 2015 3:00:24 PM
> Subject: raid5 reshape is stuck
>
> Hi Neil
>
> I encounter the problem when I reshape a 4-disks raid5 to raid5. It just
> can
> appear with loop devices.
>
> The steps are:
>
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4]
> --assume-clean
> mdadm: /dev/loop0 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop1 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop2 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop3 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: /dev/loop4 appears to be part of a raid array:
> level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
> mdadm: added /dev/loop5
> [root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
> mdadm: Need to backup 10240K of critical section..
> [root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> 8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6]
> [UUUUUU]
> [>....................] reshape = 0.0% (0/2046976) finish=6396.8min
> speed=0K/sec
>
> unused devices: <none>
>
> It because the sync_max is set to 0 when run the command --grow
>
> [root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
> [root@dhcp-12-158 md]# cat sync_max
> 0
>
> I tried reproduce with normal sata devices. The progress of reshape is no
> problem. Then
> I checked the Grow.c. If I use sata devices, in function reshape_array, the
> return value
> of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it
> call the function
> start_reshape.
>
> In the function start_reshape it set the sync_max to reshape_progress. But
> in sysfs_read it
> doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it
> need to set the
> sync_max at this? I'm not sure about this.
>
> I tried to fix this but I'm not sure whether it's the right way. I'll send
> the patches in
> other mails.
>
If there is no need to set sync_max and sync_min here. The method below also can fix the
problem.
-int start_reshape(struct mdinfo *sra, int already_running,
- int before_data_disks, int data_disks)
+int start_reshape(struct mdinfo *sra, int already_running)
{
int err;
- unsigned long long sync_max_to_set;
sysfs_set_num(sra, NULL, "suspend_lo", 0x7FFFFFFFFFFFFFFFULL);
err = sysfs_set_num(sra, NULL, "suspend_hi", sra->reshape_progress);
err = err ?: sysfs_set_num(sra, NULL, "suspend_lo",
sra->reshape_progress);
- if (before_data_disks <= data_disks)
- sync_max_to_set = sra->reshape_progress / data_disks;
- else
- sync_max_to_set = (sra->component_size * data_disks
- - sra->reshape_progress) / data_disks;
- if (!already_running)
- sysfs_set_num(sra, NULL, "sync_min", sync_max_to_set);
- err = err ?: sysfs_set_num(sra, NULL, "sync_max", sync_max_to_set);
if (!already_running)
err = err ?: sysfs_set_str(sra, NULL, "sync_action", "reshape");
@@ -3260,8 +3250,8 @@
devname, container, &reshape) < 0)
goto release;
- err = start_reshape(sra, restart, reshape.before.data_disks,
- reshape.after.data_disks);
+ err = start_reshape(sra, restart);
+
if (err) {
pr_err("Cannot %s reshape for %s\n",
restart ? "continue" : "start",
^ permalink raw reply
* Recent drive errors
From: Thomas Fjellstrom @ 2015-05-19 11:08 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi,
I have this one drive that dropped out of one of my arrays once. It shows UNC
errors in SMART (log appended), and Reported_Uncorrect is 5. There are no
smart test failures or any other SMART values that look spectacularly wrong,
other than maybe Load_Cycle_Count which is 10625 (these seagates used to
constantly park and unpark before i updated the firmware).
I'm wondering whether or not this drive is still safe to use. I feel like I
can't trust it, especially after all the other Seagates I had that failed in
the past few years. I'm running a tool called whdd on it right now and it
shows very consistent latency spikes above 150ms. Really, I'm wondering if
this drive is RMAable as is, or if i have to wait for it to degrade further as
i have another drive with like 10k reallocated sectors to send in. I have
already replaced both with WD Red's so I can do whatever tests are needed to
figure it out.
Thanks for any help,
SMART log:
# smartctl -a /dev/sdf
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.0.0-1-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F2G312
LU WWN Device Id: 5 000c50 060014689
Firmware Version: CC4H
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 19 04:42:33 2015 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 592) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 345) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 115 099 006 Pre-fail Always - 100468288
3 Spin_Up_Time 0x0003 092 091 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 526
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always - 9795200
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5590
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 36
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 095 095 000 Old_age Always - 5
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 0 0 1
189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1
190 Airflow_Temperature_Cel 0x0022 064 054 045 Old_age Always - 36 (Min/Max 35/36)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 524
193 Load_Cycle_Count 0x0032 095 095 000 Old_age Always - 10625
194 Temperature_Celsius 0x0022 036 046 000 Old_age Always - 36 (0 18 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5572h+04m+02.074s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 44489827126891
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 22877192047256
SMART Error Log Version: 1
ATA Error Count: 5
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 5 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 c0 ff ff ff 4f 00 48d+03:55:47.791 READ FPDMA QUEUED
61 00 80 ff ff ff 4f 00 48d+03:55:47.791 WRITE FPDMA QUEUED
e5 00 00 00 00 00 00 00 48d+03:55:47.784 CHECK POWER MODE
60 00 08 08 00 00 40 00 48d+03:55:47.755 READ FPDMA QUEUED
60 00 08 00 00 00 40 00 48d+03:55:47.751 READ FPDMA QUEUED
Error 4 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED
60 00 c0 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED
60 00 08 00 00 00 40 00 48d+03:55:44.672 READ FPDMA QUEUED
Error 3 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED
60 00 c0 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:41.801 READ FPDMA QUEUED
Error 2 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 48d+03:55:38.809 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 48d+03:55:38.793 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:38.793 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:38.792 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:38.792 READ FPDMA QUEUED
Error 1 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED
60 00 40 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 5574 -
# 2 Extended offline Completed without error 00% 5571 -
# 3 Extended offline Completed without error 00% 5561 -
# 4 Short offline Completed without error 00% 5556 -
# 5 Short offline Completed without error 00% 5553 -
# 6 Short offline Completed without error 00% 5529 -
# 7 Short offline Completed without error 00% 5505 -
# 8 Short offline Completed without error 00% 5481 -
# 9 Short offline Completed without error 00% 5457 -
#10 Short offline Completed without error 00% 5433 -
#11 Short offline Completed without error 00% 5409 -
#12 Short offline Completed without error 00% 5385 -
#13 Short offline Completed without error 00% 5361 -
#14 Short offline Completed without error 00% 5337 -
#15 Short offline Completed without error 00% 5313 -
#16 Short offline Completed without error 00% 5289 -
#17 Short offline Completed without error 00% 5265 -
#18 Short offline Completed without error 00% 5241 -
#19 Short offline Completed without error 00% 5217 -
#20 Short offline Completed without error 00% 5193 -
#21 Short offline Completed without error 00% 5169 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: 20 disks, fastest possible mostly-sequential read speeds
From: Brian Kroth @ 2015-05-19 10:36 UTC (permalink / raw)
To: Adam Goryachev, Jon Nelson, LinuxRaid
In-Reply-To: <555ABFEC.40606@websitemanagers.com.au>
On May 18, 2015 11:45:32 PM CDT, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
>On 19/05/15 12:37, Jon Nelson wrote:
>> I'm looking for some advise on tuning.
>> I have a server with 20 disks behind an LSI 9271-something.
>> They are currently exposed as 20 individual raid0 with a "strip" size
>> of 1MB,
>Ummm, you have 20 disks connected to some raid controller, which
>presents them as 20 raid0 arrays? Or are they raid0 arrays consisting
>of
>only one disk? or JBOD? or something else?
>
>> and assembled into an mdraid, meta 1.2, layout 10 format f2,
>> with a 1MB chunk size and formatted using ext4 -T largefile.
>> To date, this has given me the best numbers when reading some 10,000
>> files (total size: about 2.5TB) sequentially or in parallel.
>
>What other things did you try?
>How did you measure this?
>What answers did you get?
>
>> I can't seem to get better than about 1,800 MB/s read speeds though.
>I
>> *should* be able to get closer to 3,000 based on what the drives are
>> capable of.
You also need to be aware of controller and bus limits as well as any 10 bit (eg:sas) vs. 8 bit units between the numbers you're seeing, not to mention other overheads in the software end of your storage stack.
>> Quite some time ago on this very hardware I saw a
>> sustained 2,750 MB/s but I don't remember how I got there.
>Are you looking for sequential or random access? You will get very
>different numbers for each of these.
Also, read vs write, cache hit vs buffered write, etc.
>> readahead values have been adjusted, I/O scheduler, etc... all played
>> with with some benefit but nothing huge. What should I be looking at
>> here if I want the best possible read performance?
>>
>> I don't want to give up some measure of redundancy.
>
>The clue here is to test and measure, and keep a record of the results.
>
>It can be really frustrating when you can't get the same good result
>you
>had last week. IME, it is a matter of testing something different, and
>that is why the result is different.
>
>Regards,
>Adam
/me nods
Cheers,
Brian
--
Sent from my mobile device
^ permalink raw reply
* Re: [PATCH v2 RESEND] block: discard bdi_unregister() in favour of bdi_destroy()
From: Sergey Senozhatsky @ 2015-05-19 9:40 UTC (permalink / raw)
To: Jens Axboe
Cc: NeilBrown, device-mapper development, Kernel.org-Linux-RAID, lkml,
Christoph Hellwig, stable, Sergey Senozhatsky, Greg KH
In-Reply-To: <20150519155837.456ad30e@notabene.brown>
On (05/19/15 15:58), NeilBrown wrote:
>
> bdi_unregister() now contains very little functionality.
>
> It contains a "WARN_ON" if bdi->dev is NULL. This warning is of no
> real consequence as bdi->dev isn't needed by anything else in the function,
> and it triggers if
> blk_cleanup_queue() -> bdi_destroy()
> is called before bdi_unregister, which happens since
> Commit: 6cd18e711dd8 ("block: destroy bdi before blockdev is unregistered.")
>
> So this isn't wanted.
>
> It also calls bdi_set_min_ratio(). This needs to be called after
> writes through the bdi have all been flushed, and before the bdi is destroyed.
> Calling it early is better than calling it late as it frees up a global
> resource.
>
> Calling it immediately after bdi_wb_shutdown() in bdi_destroy()
> perfectly fits these requirements.
>
> So bdi_unregister() can be discarded with the important content moved to
> bdi_destroy(), as can the
> writeback_bdi_unregister
> event which is already not used.
>
just to boost/support this patch: fixes del_gendisk() warning (loop, zram, etc.)
[ 6298.249018] WARNING: CPU: 2 PID: 29935 at mm/backing-dev.c:372 bdi_unregister+0x2d/0x3f()
[ 6298.249065] Call Trace:
[ 6298.249070] [<ffffffff814fe3ca>] dump_stack+0x4c/0x6e
[ 6298.249073] [<ffffffff810768e7>] ? up+0x39/0x3e
[ 6298.249076] [<ffffffff8103b792>] warn_slowpath_common+0x9b/0xb5
[ 6298.249077] [<ffffffff811080d8>] ? bdi_unregister+0x2d/0x3f
[ 6298.249079] [<ffffffff8103b84f>] warn_slowpath_null+0x1a/0x1c
[ 6298.249081] [<ffffffff811080d8>] bdi_unregister+0x2d/0x3f
[ 6298.249084] [<ffffffff8122c7ae>] del_gendisk+0xf7/0x1c2
-ss
> Reported-by: Mike Snitzer <snitzer@redhat.com>
> Cc: stable@vger.kernel.org (v4.0)
> Fixes: c4db59d31e39 ("fs: don't reassign dirty inodes to default_backing_dev_info")
> Fixes: 6cd18e711dd8 ("block: destroy bdi before blockdev is unregistered.")
> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> Acked-by: Dan Williams <dan.j.williams@intel.com>
> Tested-by: Nicholas Moulin <nicholas.w.moulin@linux.intel.com>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> ---
>
> hi Jens,
> It seems you might have missed this, possibly buried in a long thread.
> It would be good you it could get to Linux and thence to -stable promptly.
>
> Thanks,
> NeilBrown
>
>
> diff --git a/block/genhd.c b/block/genhd.c
> index e351fc521053..1d4435478e8a 100644
> --- a/block/genhd.c
> +++ b/block/genhd.c
> @@ -657,7 +657,6 @@ void del_gendisk(struct gendisk *disk)
> disk->flags &= ~GENHD_FL_UP;
>
> sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
> - bdi_unregister(&disk->queue->backing_dev_info);
> blk_unregister_queue(disk);
> blk_unregister_region(disk_devt(disk), disk->minors);
>
> diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
> index aff923ae8c4b..d87d8eced064 100644
> --- a/include/linux/backing-dev.h
> +++ b/include/linux/backing-dev.h
> @@ -116,7 +116,6 @@ __printf(3, 4)
> int bdi_register(struct backing_dev_info *bdi, struct device *parent,
> const char *fmt, ...);
> int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
> -void bdi_unregister(struct backing_dev_info *bdi);
> int __must_check bdi_setup_and_register(struct backing_dev_info *, char *);
> void bdi_start_writeback(struct backing_dev_info *bdi, long nr_pages,
> enum wb_reason reason);
> diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h
> index 880dd7437172..c178d13d6f4c 100644
> --- a/include/trace/events/writeback.h
> +++ b/include/trace/events/writeback.h
> @@ -250,7 +250,6 @@ DEFINE_EVENT(writeback_class, name, \
> DEFINE_WRITEBACK_EVENT(writeback_nowork);
> DEFINE_WRITEBACK_EVENT(writeback_wake_background);
> DEFINE_WRITEBACK_EVENT(writeback_bdi_register);
> -DEFINE_WRITEBACK_EVENT(writeback_bdi_unregister);
>
> DECLARE_EVENT_CLASS(wbc_class,
> TP_PROTO(struct writeback_control *wbc, struct backing_dev_info *bdi),
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 6dc4580df2af..000e7b3b9896 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -359,23 +359,6 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi)
> flush_delayed_work(&bdi->wb.dwork);
> }
>
> -/*
> - * Called when the device behind @bdi has been removed or ejected.
> - *
> - * We can't really do much here except for reducing the dirty ratio at
> - * the moment. In the future we should be able to set a flag so that
> - * the filesystem can handle errors at mark_inode_dirty time instead
> - * of only at writeback time.
> - */
> -void bdi_unregister(struct backing_dev_info *bdi)
> -{
> - if (WARN_ON_ONCE(!bdi->dev))
> - return;
> -
> - bdi_set_min_ratio(bdi, 0);
> -}
> -EXPORT_SYMBOL(bdi_unregister);
> -
> static void bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
> {
> memset(wb, 0, sizeof(*wb));
> @@ -443,6 +426,7 @@ void bdi_destroy(struct backing_dev_info *bdi)
> int i;
>
> bdi_wb_shutdown(bdi);
> + bdi_set_min_ratio(bdi, 0);
>
> WARN_ON(!list_empty(&bdi->work_list));
> WARN_ON(delayed_work_pending(&bdi->wb.dwork));
^ permalink raw reply
* Re: [PATCH v2 0/6] a caching layer for raid5/6
From: Artur Paszkiewicz @ 2015-05-19 9:22 UTC (permalink / raw)
To: Shaohua Li, linux-raid
Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1432003804.git.shli@fb.com>
On 05/19/2015 04:57 AM, Shaohua Li wrote:
> Hi,
>
> This is the second version of the raid5/6 caching layer patches. The patches add a
> caching layer for raid5/6. The caching layer uses a SSD as a cache for a raid
> 5/6. It works like the similar way of a hardware raid controller. The purpose
> is to improve raid performance (reduce read-modify-write) and fix write hole
> issue. The main patch is patch 3 and the description has all details about the
> implementation.
>
> Main changes of V2 are to improve performance. Meta data write doesn't use FUA
> any more. Discard request is only dispatched when discard range is big enough.
> Also have some bug fixing and code cleanup. Please review!
Hi,
It seems patch 3 is missing.
Artur
^ permalink raw reply
* Re: [PATCH v3 10/11] md/raid10: make sync_request_write() call bio_copy_data()
From: Christoph Hellwig @ 2015-05-19 8:45 UTC (permalink / raw)
To: Ming Lin
Cc: linux-kernel, Christoph Hellwig, Jens Axboe, Kent Overstreet,
Dongsu Park, Christoph Hellwig, Neil Brown, linux-raid
In-Reply-To: <1430980461-5235-11-git-send-email-mlin@kernel.org>
This seems fine even without the rest of the series. Jens or Neil,
can you pick this one up?
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v2 RESEND] block: discard bdi_unregister() in favour of bdi_destroy()
From: Christoph Hellwig @ 2015-05-19 6:21 UTC (permalink / raw)
To: NeilBrown
Cc: Jens Axboe, device-mapper development, Kernel.org-Linux-RAID,
lkml, Christoph Hellwig, stable, Sergey Senozhatsky, Greg KH
In-Reply-To: <20150519155837.456ad30e@notabene.brown>
Looks good,
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* [PATCH v2 RESEND] block: discard bdi_unregister() in favour of bdi_destroy()
From: NeilBrown @ 2015-05-19 5:58 UTC (permalink / raw)
To: Jens Axboe
Cc: device-mapper development, Kernel.org-Linux-RAID, lkml,
Christoph Hellwig, stable, Sergey Senozhatsky, Greg KH
[-- Attachment #1: Type: text/plain, Size: 4310 bytes --]
bdi_unregister() now contains very little functionality.
It contains a "WARN_ON" if bdi->dev is NULL. This warning is of no
real consequence as bdi->dev isn't needed by anything else in the function,
and it triggers if
blk_cleanup_queue() -> bdi_destroy()
is called before bdi_unregister, which happens since
Commit: 6cd18e711dd8 ("block: destroy bdi before blockdev is unregistered.")
So this isn't wanted.
It also calls bdi_set_min_ratio(). This needs to be called after
writes through the bdi have all been flushed, and before the bdi is destroyed.
Calling it early is better than calling it late as it frees up a global
resource.
Calling it immediately after bdi_wb_shutdown() in bdi_destroy()
perfectly fits these requirements.
So bdi_unregister() can be discarded with the important content moved to
bdi_destroy(), as can the
writeback_bdi_unregister
event which is already not used.
Reported-by: Mike Snitzer <snitzer@redhat.com>
Cc: stable@vger.kernel.org (v4.0)
Fixes: c4db59d31e39 ("fs: don't reassign dirty inodes to default_backing_dev_info")
Fixes: 6cd18e711dd8 ("block: destroy bdi before blockdev is unregistered.")
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Dan Williams <dan.j.williams@intel.com>
Tested-by: Nicholas Moulin <nicholas.w.moulin@linux.intel.com>
Signed-off-by: NeilBrown <neilb@suse.de>
---
hi Jens,
It seems you might have missed this, possibly buried in a long thread.
It would be good you it could get to Linux and thence to -stable promptly.
Thanks,
NeilBrown
diff --git a/block/genhd.c b/block/genhd.c
index e351fc521053..1d4435478e8a 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -657,7 +657,6 @@ void del_gendisk(struct gendisk *disk)
disk->flags &= ~GENHD_FL_UP;
sysfs_remove_link(&disk_to_dev(disk)->kobj, "bdi");
- bdi_unregister(&disk->queue->backing_dev_info);
blk_unregister_queue(disk);
blk_unregister_region(disk_devt(disk), disk->minors);
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index aff923ae8c4b..d87d8eced064 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -116,7 +116,6 @@ __printf(3, 4)
int bdi_register(struct backing_dev_info *bdi, struct device *parent,
const char *fmt, ...);
int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
-void bdi_unregister(struct backing_dev_info *bdi);
int __must_check bdi_setup_and_register(struct backing_dev_info *, char *);
void bdi_start_writeback(struct backing_dev_info *bdi, long nr_pages,
enum wb_reason reason);
diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h
index 880dd7437172..c178d13d6f4c 100644
--- a/include/trace/events/writeback.h
+++ b/include/trace/events/writeback.h
@@ -250,7 +250,6 @@ DEFINE_EVENT(writeback_class, name, \
DEFINE_WRITEBACK_EVENT(writeback_nowork);
DEFINE_WRITEBACK_EVENT(writeback_wake_background);
DEFINE_WRITEBACK_EVENT(writeback_bdi_register);
-DEFINE_WRITEBACK_EVENT(writeback_bdi_unregister);
DECLARE_EVENT_CLASS(wbc_class,
TP_PROTO(struct writeback_control *wbc, struct backing_dev_info *bdi),
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 6dc4580df2af..000e7b3b9896 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -359,23 +359,6 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi)
flush_delayed_work(&bdi->wb.dwork);
}
-/*
- * Called when the device behind @bdi has been removed or ejected.
- *
- * We can't really do much here except for reducing the dirty ratio at
- * the moment. In the future we should be able to set a flag so that
- * the filesystem can handle errors at mark_inode_dirty time instead
- * of only at writeback time.
- */
-void bdi_unregister(struct backing_dev_info *bdi)
-{
- if (WARN_ON_ONCE(!bdi->dev))
- return;
-
- bdi_set_min_ratio(bdi, 0);
-}
-EXPORT_SYMBOL(bdi_unregister);
-
static void bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
{
memset(wb, 0, sizeof(*wb));
@@ -443,6 +426,7 @@ void bdi_destroy(struct backing_dev_info *bdi)
int i;
bdi_wb_shutdown(bdi);
+ bdi_set_min_ratio(bdi, 0);
WARN_ON(!list_empty(&bdi->work_list));
WARN_ON(delayed_work_pending(&bdi->wb.dwork));
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: 20 disks, fastest possible mostly-sequential read speeds
From: Adam Goryachev @ 2015-05-19 4:45 UTC (permalink / raw)
To: Jon Nelson, LinuxRaid
In-Reply-To: <CAKuK5J3CvbhVdyiEO8GrTje9xXq5u1eD96961vF9rA=fzAu6xg@mail.gmail.com>
On 19/05/15 12:37, Jon Nelson wrote:
> I'm looking for some advise on tuning.
> I have a server with 20 disks behind an LSI 9271-something.
> They are currently exposed as 20 individual raid0 with a "strip" size
> of 1MB,
Ummm, you have 20 disks connected to some raid controller, which
presents them as 20 raid0 arrays? Or are they raid0 arrays consisting of
only one disk? or JBOD? or something else?
> and assembled into an mdraid, meta 1.2, layout 10 format f2,
> with a 1MB chunk size and formatted using ext4 -T largefile.
> To date, this has given me the best numbers when reading some 10,000
> files (total size: about 2.5TB) sequentially or in parallel.
What other things did you try?
How did you measure this?
What answers did you get?
> I can't seem to get better than about 1,800 MB/s read speeds though. I
> *should* be able to get closer to 3,000 based on what the drives are
> capable of. Quite some time ago on this very hardware I saw a
> sustained 2,750 MB/s but I don't remember how I got there.
Are you looking for sequential or random access? You will get very
different numbers for each of these.
> readahead values have been adjusted, I/O scheduler, etc... all played
> with with some benefit but nothing huge. What should I be looking at
> here if I want the best possible read performance?
>
> I don't want to give up some measure of redundancy.
The clue here is to test and measure, and keep a record of the results.
It can be really frustrating when you can't get the same good result you
had last week. IME, it is a matter of testing something different, and
that is why the result is different.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* [PATCH v2 6/6] raid5: skip resync if caching is enabled
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1432003804.git.shli@fb.com>
If caching is enabled, the caching layer will guarantee data
consistency, so skip resync for unclean shutdown
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 53f582d..52e016f 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -6941,6 +6941,13 @@ static int run(struct mddev *mddev)
if (mddev->queue)
blk_queue_logical_block_size(mddev->queue, STRIPE_SIZE);
conf->skip_copy = 1;
+
+ if (mddev->recovery_cp == 0) {
+ printk(KERN_NOTICE
+ "md/raid:%s: skip resync with caching enabled\n",
+ mdname(mddev));
+ mddev->recovery_cp = MaxSector;
+ }
}
return 0;
--
1.8.1
^ permalink raw reply related
* [PATCH v2 5/6] md: don't allow resize/reshape with cache support
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1432003804.git.shli@fb.com>
If cache support is enabled, don't allow resize/reshape in current
stage. In the future, we can flush all data from cache to raid before
resize/reshape and then allow resize/reshape.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 772b65f..53f582d 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7173,6 +7173,10 @@ static int raid5_resize(struct mddev *mddev, sector_t sectors)
* worth it.
*/
sector_t newsize;
+ struct r5conf *conf = mddev->private;
+
+ if (conf->cache)
+ return -EINVAL;
sectors &= ~((sector_t)mddev->chunk_sectors - 1);
newsize = raid5_size(mddev, sectors, mddev->raid_disks);
if (mddev->external_size &&
@@ -7224,6 +7228,8 @@ static int check_reshape(struct mddev *mddev)
{
struct r5conf *conf = mddev->private;
+ if (conf->cache)
+ return -EINVAL;
if (mddev->delta_disks == 0 &&
mddev->new_layout == mddev->layout &&
mddev->new_chunk_sectors == mddev->chunk_sectors)
--
1.8.1
^ permalink raw reply related
* [PATCH v2 4/6] raid5: add some sysfs entries
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1432003804.git.shli@fb.com>
Add some sysfs entries.
-cache_memory. Control the cache memory size.
-cache_reclaim_batch. Control how many stripes reclaim should run in one
time.
-cache_memory_watermark. The background reclaim runs if cache memory
hits the watermark and stops after hit 1.5x of the watermark.
-cache_disk_watermark. The background reclaim runs if cache disk space
hits the watermark and stops after hit 1.5x of the watermark.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 253 ++++++++++++++++++++++++++++++++++++++++++++++-
drivers/md/raid5.c | 3 +
drivers/md/raid5.h | 1 +
3 files changed, 256 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 4ea24cb..b93d29a 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -314,6 +314,12 @@ static inline int r5l_page_blocks(struct r5l_log *log, int pages)
return pages << log->page_block_shift;
}
+static inline int r5l_max_flush_stripes(struct r5l_log *log)
+{
+ return (log->block_size - sizeof(struct r5l_flush_block)) /
+ sizeof(__le64);
+}
+
static u32 r5l_calculate_checksum(struct r5l_log *log, u32 crc,
void *buf, size_t size, bool data)
{
@@ -3124,6 +3130,247 @@ static int r5c_shrink_cache_memory(struct r5c_cache *cache, unsigned long size)
return 0;
}
+static ssize_t r5c_show_cache_memory(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->max_pages << PAGE_SHIFT);
+}
+
+static ssize_t r5c_store_cache_memory(struct mddev *mddev, const char *page,
+ size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+ LIST_HEAD(page_list);
+ u64 i;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new >>= PAGE_SHIFT;
+
+ if (new > cache->max_pages) {
+ i = cache->max_pages;
+ while (i < new) {
+ struct page *page = alloc_page(GFP_KERNEL);
+
+ if (!page)
+ break;
+ list_add(&page->lru, &page_list);
+ i++;
+ }
+
+ spin_lock_irq(&cache->pool_lock);
+ list_splice(&page_list, &cache->page_pool);
+ cache->free_pages += i - cache->max_pages;
+ cache->max_pages = i;
+ cache->total_pages = i;
+ r5c_calculate_watermark(cache);
+ spin_unlock_irq(&cache->pool_lock);
+ return len;
+ }
+ r5c_shrink_cache_memory(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_memory = __ATTR(cache_memory,
+ S_IRUGO | S_IWUSR, r5c_show_cache_memory, r5c_store_cache_memory);
+
+int r5c_min_stripe_cache_size(struct r5c_cache *cache)
+{
+ struct r5conf *conf = cache->mddev->private;
+ return (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) *
+ cache->reclaim_batch;
+}
+
+static void r5c_set_reclaim_batch(struct r5c_cache *cache, int batch)
+{
+ struct mddev *mddev = cache->mddev;
+ struct r5conf *conf = mddev->private;
+ int size;
+
+ size = (cache->stripe_parity_pages << PAGE_SECTOR_SHIFT) * batch;
+ if (size > cache->reserved_space) {
+ cache->reserved_space = size;
+ mutex_lock(&cache->log.io_mutex);
+ cache->log.reserved_blocks = r5l_sector_to_block(&cache->log,
+ cache->reserved_space) + 1;
+ mutex_unlock(&cache->log.io_mutex);
+ r5c_wake_wait_reclaimer(cache,
+ RECLAIM_DISK_BACKGROUND);
+ } else {
+ mutex_lock(&cache->log.io_mutex);
+ cache->log.reserved_blocks -= r5l_sector_to_block(&cache->log,
+ cache->reserved_space - size);
+ mutex_unlock(&cache->log.io_mutex);
+ cache->reserved_space = size;
+ }
+
+ size = (conf->chunk_sectors >> PAGE_SECTOR_SHIFT) * batch;
+
+ mddev_lock(mddev);
+ if (size > conf->max_nr_stripes)
+ raid5_set_cache_size(mddev, size);
+ mddev_unlock(mddev);
+
+ cache->reclaim_batch = batch;
+}
+
+static ssize_t r5c_show_cache_reclaim_batch(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%d\n", cache->reclaim_batch);
+}
+
+static ssize_t r5c_store_cache_reclaim_batch(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+
+ if (new > r5l_max_flush_stripes(&cache->log))
+ new = r5l_max_flush_stripes(&cache->log);
+
+ if (new != cache->reclaim_batch)
+ r5c_set_reclaim_batch(cache, new);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_reclaim_batch =
+ __ATTR(cache_reclaim_batch, S_IRUGO | S_IWUSR,
+ r5c_show_cache_reclaim_batch, r5c_store_cache_reclaim_batch);
+
+static ssize_t r5c_show_cache_disk_watermark(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->log.low_watermark *
+ cache->log.block_size);
+}
+
+static ssize_t r5c_store_cache_disk_watermark(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ struct r5l_log *log = &cache->log;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new /= log->block_size;
+
+ if (new * 3 / 2 >= log->total_blocks)
+ return -EINVAL;
+
+ mutex_lock(&log->io_mutex);
+ log->low_watermark = new;
+ log->high_watermark = new * 3 / 2;
+ mutex_unlock(&log->io_mutex);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_disk_watermark =
+ __ATTR(cache_disk_watermark, S_IRUGO | S_IWUSR,
+ r5c_show_cache_disk_watermark, r5c_store_cache_disk_watermark);
+
+static ssize_t r5c_show_cache_memory_watermark(struct mddev *mddev, char *page)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+
+ return sprintf(page, "%lld\n", cache->low_watermark << PAGE_SHIFT);
+}
+
+static ssize_t r5c_store_cache_memory_watermark(struct mddev *mddev,
+ const char *page, size_t len)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5c_cache *cache = conf->cache;
+ unsigned long new;
+
+ if (len >= PAGE_SIZE)
+ return -EINVAL;
+ if (kstrtoul(page, 0, &new))
+ return -EINVAL;
+ new >>= PAGE_SHIFT;
+
+ if (new * 3 / 2 >= cache->max_pages)
+ return -EINVAL;
+
+ spin_lock_irq(&cache->pool_lock);
+ cache->low_watermark = new;
+ cache->high_watermark = new * 3 / 2;
+ spin_unlock_irq(&cache->pool_lock);
+ return len;
+}
+
+static struct md_sysfs_entry r5c_cache_memory_watermark =
+ __ATTR(cache_memory_watermark, S_IRUGO | S_IWUSR,
+ r5c_show_cache_memory_watermark, r5c_store_cache_memory_watermark);
+
+static int r5c_init_sysfs(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ int ret;
+
+ ret = sysfs_add_file_to_group(&mddev->kobj, &r5c_cache_memory.attr,
+ NULL);
+ if (ret)
+ return ret;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+ if (ret)
+ goto err_reclaim;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+ if (ret)
+ goto disk_watermark;
+ ret = sysfs_add_file_to_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
+ if (ret)
+ goto memory_watermark;
+ return 0;
+memory_watermark:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+disk_watermark:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+err_reclaim:
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory.attr, NULL);
+ return ret;
+}
+
+static void r5c_exit_sysfs(struct r5c_cache *cache)
+{
+ struct mddev *mddev = cache->mddev;
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_reclaim_batch.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_disk_watermark.attr, NULL);
+ sysfs_remove_file_from_group(&mddev->kobj,
+ &r5c_cache_memory_watermark.attr, NULL);
+}
+
static void r5c_free_cache_data(struct r5c_cache *cache)
{
struct r5c_stripe *stripe;
@@ -3234,8 +3481,11 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
cache->reclaim_thread->timeout = CHECKPOINT_TIMEOUT;
r5c_shrink_cache_memory(cache, cache->max_pages);
-
+ if (r5c_init_sysfs(cache))
+ goto err_sysfs;
return cache;
+err_sysfs:
+ md_unregister_thread(&cache->reclaim_thread);
err_page:
r5c_free_cache_data(cache);
@@ -3254,6 +3504,7 @@ struct r5c_cache *r5c_init_cache(struct r5conf *conf, struct md_rdev *rdev)
void r5c_exit_cache(struct r5c_cache *cache)
{
+ r5c_exit_sysfs(cache);
md_unregister_thread(&cache->reclaim_thread);
r5l_exit_log(&cache->log);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 4eb6e99..772b65f 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -5876,6 +5876,9 @@ raid5_set_cache_size(struct mddev *mddev, int size)
if (size <= 16 || size > 32768)
return -EINVAL;
+ if (conf->cache && size < r5c_min_stripe_cache_size(conf->cache))
+ size = r5c_min_stripe_cache_size(conf->cache);
+
conf->min_nr_stripes = size;
while (size < conf->max_nr_stripes &&
drop_one_stripe(conf))
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index e4e93bb..899ec79 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -614,4 +614,5 @@ void r5c_exit_cache(struct r5c_cache *cache);
void r5c_write_start(struct mddev *mddev, struct bio *bi);
void r5c_write_end(struct mddev *mddev, struct bio *bi);
void r5c_quiesce(struct r5conf *conf, int state);
+int r5c_min_stripe_cache_size(struct r5c_cache *cache);
#endif
--
1.8.1
^ permalink raw reply related
* [PATCH v2 2/6] raid5: directly use mddev->queue
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1432003804.git.shli@fb.com>
When the cache layer flushes data from cache disk to raid disks, it will
dipsatch IO to raid disks. At that time, we don't have a block device
attached to the bio, so directly use mddev->queue. That should not
impact IO dispatched to rdev, which has rdev block device attached.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 77dfd72..950c3c6 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -223,7 +223,7 @@ static int raid6_idx_to_slot(int idx, struct stripe_head *sh,
return slot;
}
-static void return_io(struct bio *return_bi)
+static void return_io(struct r5conf *conf, struct bio *return_bi)
{
struct bio *bi = return_bi;
while (bi) {
@@ -231,8 +231,7 @@ static void return_io(struct bio *return_bi)
return_bi = bi->bi_next;
bi->bi_next = NULL;
bi->bi_iter.bi_size = 0;
- trace_block_bio_complete(bdev_get_queue(bi->bi_bdev),
- bi, 0);
+ trace_block_bio_complete(conf->mddev->queue, bi, 0);
bio_endio(bi, 0);
bi = return_bi;
}
@@ -1193,7 +1192,7 @@ static void ops_complete_biofill(void *stripe_head_ref)
}
clear_bit(STRIPE_BIOFILL_RUN, &sh->state);
- return_io(return_bi);
+ return_io(sh->raid_conf, return_bi);
set_bit(STRIPE_HANDLE, &sh->state);
release_stripe(sh);
@@ -4563,7 +4562,7 @@ static void handle_stripe(struct stripe_head *sh)
md_wakeup_thread(conf->mddev->thread);
}
- return_io(s.return_bi);
+ return_io(conf, s.return_bi);
clear_bit_unlock(STRIPE_ACTIVE, &sh->state);
}
@@ -5267,8 +5266,7 @@ static void make_request(struct mddev *mddev, struct bio * bi)
if ( rw == WRITE )
md_write_end(mddev);
- trace_block_bio_complete(bdev_get_queue(bi->bi_bdev),
- bi, 0);
+ trace_block_bio_complete(mddev->queue, bi, 0);
bio_endio(bi, 0);
}
}
--
1.8.1
^ permalink raw reply related
* [PATCH v2 1/6] MD: add a new disk role to present cache device
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
In-Reply-To: <cover.1432003804.git.shli@fb.com>
From: Song Liu <songliubraving@fb.com>
Next patches will use a disk as raid5/6 caching. We need a new disk role
to present the cache device
Not sure if we should bump up the MD superblock version for the disk
role.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 14 +++++++++++++-
drivers/md/md.h | 4 ++++
include/uapi/linux/raid/md_p.h | 1 +
3 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index d4f31e1..b6ece48 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -1656,6 +1656,9 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
case 0xfffe: /* faulty */
set_bit(Faulty, &rdev->flags);
break;
+ case 0xfffd: /* cache device */
+ set_bit(WriteCache, &rdev->flags);
+ break;
default:
rdev->saved_raid_disk = role;
if ((le32_to_cpu(sb->feature_map) &
@@ -1811,6 +1814,8 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
sb->dev_roles[i] = cpu_to_le16(0xfffe);
else if (test_bit(In_sync, &rdev2->flags))
sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
+ else if (test_bit(WriteCache, &rdev2->flags))
+ sb->dev_roles[i] = cpu_to_le16(0xfffd);
else if (rdev2->raid_disk >= 0)
sb->dev_roles[i] = cpu_to_le16(rdev2->raid_disk);
else
@@ -5778,7 +5783,8 @@ static int get_disk_info(struct mddev *mddev, void __user * arg)
else if (test_bit(In_sync, &rdev->flags)) {
info.state |= (1<<MD_DISK_ACTIVE);
info.state |= (1<<MD_DISK_SYNC);
- }
+ } else if (test_bit(WriteCache, &rdev->flags))
+ info.state |= (1<<MD_DISK_WRITECACHE);
if (test_bit(WriteMostly, &rdev->flags))
info.state |= (1<<MD_DISK_WRITEMOSTLY);
} else {
@@ -5893,6 +5899,8 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
else
clear_bit(WriteMostly, &rdev->flags);
+ if (info->state & (1<<MD_DISK_WRITECACHE))
+ set_bit(WriteCache, &rdev->flags);
/*
* check whether the device shows up in other nodes
*/
@@ -7261,6 +7269,10 @@ static int md_seq_show(struct seq_file *seq, void *v)
seq_printf(seq, "(F)");
continue;
}
+ if (test_bit(WriteCache, &rdev->flags)) {
+ seq_printf(seq, "(C)");
+ continue;
+ }
if (rdev->raid_disk < 0)
seq_printf(seq, "(S)"); /* spare */
if (test_bit(Replacement, &rdev->flags))
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 4046a6c..6857592 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -175,6 +175,10 @@ enum flag_bits {
* This device is seen locally but not
* by the whole cluster
*/
+ WriteCache, /* This device is used as write cache.
+ * Usually, this device should be faster
+ * than other devices in the array
+ */
};
#define BB_LEN_MASK (0x00000000000001FFULL)
diff --git a/include/uapi/linux/raid/md_p.h b/include/uapi/linux/raid/md_p.h
index 2ae6131..9d36b91 100644
--- a/include/uapi/linux/raid/md_p.h
+++ b/include/uapi/linux/raid/md_p.h
@@ -89,6 +89,7 @@
* read requests will only be sent here in
* dire need
*/
+#define MD_DISK_WRITECACHE 18 /* disk is used as the write cache in RAID-5/6 */
typedef struct mdp_device_descriptor_s {
__u32 number; /* 0 Device number in the entire set */
--
1.8.1
^ permalink raw reply related
* [PATCH v2 0/6] a caching layer for raid5/6
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
Hi,
This is the second version of the raid5/6 caching layer patches. The patches add a
caching layer for raid5/6. The caching layer uses a SSD as a cache for a raid
5/6. It works like the similar way of a hardware raid controller. The purpose
is to improve raid performance (reduce read-modify-write) and fix write hole
issue. The main patch is patch 3 and the description has all details about the
implementation.
Main changes of V2 are to improve performance. Meta data write doesn't use FUA
any more. Discard request is only dispatched when discard range is big enough.
Also have some bug fixing and code cleanup. Please review!
Thanks,
Shaohua
Shaohua Li (5):
raid5: directly use mddev->queue
raid5: A caching layer for RAID5/6
raid5: add some sysfs entries
md: don't allow resize/reshape with cache support
raid5: skip resync if caching is enabled
Song Liu (1):
MD: add a new disk role to present cache device
drivers/md/Makefile | 2 +-
drivers/md/md.c | 14 +-
drivers/md/md.h | 4 +
drivers/md/raid5-cache.c | 3519 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 97 +-
drivers/md/raid5.h | 16 +-
include/uapi/linux/raid/md_p.h | 73 +
7 files changed, 3705 insertions(+), 20 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
--
1.8.1
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox