* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Alireza Haghdoost @ 2015-04-01 20:04 UTC (permalink / raw)
To: Wols Lists
Cc: Piergiorgio Sartor, Dan Williams, Shaohua Li, Neil Brown,
linux-raid, Song Liu, Kernel-team
In-Reply-To: <551C4DAA.4010701@youngman.org.uk>
On Wed, Apr 1, 2015 at 2:57 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>> Now, how can be assured, in that case, that the "cache"
>>> > device is safe after the power is restored?
>> You do sync write-ahead logging on the Flash cache. If it return
>> successful, you do fire the writes to the RAID. If system crash/fails
>> during the RAID writes (Write-hole), you just recover data by scanning
>> write-ahead log in the flash cache and replay the logs into the RAID
>> drives.
>>
> Just to throw something nasty into the mix, I'm not sure whether it's
> SSDs or SD-cards, but there certainly *was* a spate of corrupted
> *controllers*.
>
> In other words, a power failure would RELIABLY TRASH the device, if it
> happened at the wrong moment. Hopefully that's been fixed ...
>
That is certainly true. As Dan mentioned, the cache device it-self
should be safe against power failure. I agree this is not the case for
all SSD cards in the market but might be the case for Facebook. I hate
to say this but It seems these efforts are useful dependent to what
kind of hardware is deployed for cache device.
--Alireza
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Jiang, Dave @ 2015-04-01 20:07 UTC (permalink / raw)
To: Williams, Dan J
Cc: linux-raid@vger.kernel.org, neilb@suse.de, Kernel-team@fb.com,
piergiorgio.sartor@nexgo.de, shli@fb.com, songliubraving@fb.com
In-Reply-To: <CAPcyv4jQZ6jeQpFU+cZ-AXzfQVXQCdK1muxNO35pCdavrggh4Q@mail.gmail.com>
On Wed, 2015-04-01 at 18:46 +0000, Williams, Dan J wrote:
> On Wed, Apr 1, 2015 at 11:36 AM, Piergiorgio Sartor
> <piergiorgio.sartor@nexgo.de> wrote:
> > On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> >> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> >> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> >> > yet, I post it out for comments. Any comments and suggestions are
> >> > welcome!
> >> >
> >> > Thanks,
> >> > Shaohua
> >> >
> >> > We expect a completed raid5/6 stack with reliability and high
> >> > performance. Currently raid5/6 has 2 issues:
> >> >
> >> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> >> > above raid5/6 can be used to aggregate write to full stripe write.
> >> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >> >
> >> > We plan to use a SSD to fix the two issues. Here we just fix the write
> >> > hole issue.
> >> >
> >> > 1. We don't try to fix the issues together. A cache layer will do write
> >> > acceleration. A log layer will fix write hole. The seperation will
> >> > simplify things a lot.
> >> >
> >> > 2. Current assumption is flashcache/bcache will be used as the cache
> >> > layer. If they don't work well, we can fix them or add a simple cache
> >> > layer for raid write aggregation later. We also assume cache layer will
> >> > absorb write, so log doesn't worry about write latency.
> >>
> >> It seems neither bcache nor dm-cache are tackling the write-buffering
> >> problem head on... they still seem to be concerned with some amount of
> >> read caching which I can see as useful for file servers and
> >> workstations, but not necessarily scale out storage.
> >>
> >> I'll try to set aside time to take a look at the patch this week.
> >
> > There is one thing I do not really get.
> >
> > The target is to avoid the "write hole", which happens,
> > for example, when there is a sudden power failure.
> >
> > Now, how can be assured, in that case, that the "cache"
> > device is safe after the power is restored?
>
> If you lose the cache the data-loss damage is greater, but this has
> always been the case with hardware-raid adapters.
>
> > Doesn't this solution just shifts the problem from
> > the array to a different device (SSD, for example)?
> >
> > Speaking of SSD, these are quite "power failure"
> > sensitive, it seems...
>
> Simple, if a cache-device is not itself power-failure safe then it
> should not be used for power-failure protection.
I think this would be a good application for some of the newer
technology coming out such as NVDIMM and persistent memory.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Jens Axboe @ 2015-04-01 20:17 UTC (permalink / raw)
To: Wols Lists, Alireza Haghdoost, Piergiorgio Sartor
Cc: Dan Williams, Shaohua Li, Neil Brown, linux-raid, Song Liu,
Kernel-team
In-Reply-To: <551C4DAA.4010701@youngman.org.uk>
On 04/01/2015 01:57 PM, Wols Lists wrote:
> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>> Now, how can be assured, in that case, that the "cache"
>>>> device is safe after the power is restored?
>> You do sync write-ahead logging on the Flash cache. If it return
>> successful, you do fire the writes to the RAID. If system crash/fails
>> during the RAID writes (Write-hole), you just recover data by scanning
>> write-ahead log in the flash cache and replay the logs into the RAID
>> drives.
>>
> Just to throw something nasty into the mix, I'm not sure whether it's
> SSDs or SD-cards, but there certainly *was* a spate of corrupted
> *controllers*.
>
> In other words, a power failure would RELIABLY TRASH the device, if it
> happened at the wrong moment. Hopefully that's been fixed ...
You can't protect against shitty devices. If you care about power fail
events, then you use hw that is specifically tested and vetted for that.
And they do exist. They might just not be the cheapest you can find on
newegg or similar places.
This potential problem isn't specific to what Shaohua is proposing, nor
is it a show stopper for that.
--
Jens Axboe
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Wols Lists @ 2015-04-01 20:18 UTC (permalink / raw)
To: Alireza Haghdoost
Cc: Piergiorgio Sartor, Dan Williams, Shaohua Li, Neil Brown,
linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAB-428kMJ7g9RL+P4ZYsMRJVdJc7jeU-wmA1zJGJ4UyKWwSeJA@mail.gmail.com>
On 01/04/15 21:04, Alireza Haghdoost wrote:
> On Wed, Apr 1, 2015 at 2:57 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 01/04/15 19:46, Alireza Haghdoost wrote:
>>>> Now, how can be assured, in that case, that the "cache"
>>>>> device is safe after the power is restored?
>>> You do sync write-ahead logging on the Flash cache. If it return
>>> successful, you do fire the writes to the RAID. If system crash/fails
>>> during the RAID writes (Write-hole), you just recover data by scanning
>>> write-ahead log in the flash cache and replay the logs into the RAID
>>> drives.
>>>
>> Just to throw something nasty into the mix, I'm not sure whether it's
>> SSDs or SD-cards, but there certainly *was* a spate of corrupted
>> *controllers*.
>>
>> In other words, a power failure would RELIABLY TRASH the device, if it
>> happened at the wrong moment. Hopefully that's been fixed ...
>>
>
> That is certainly true. As Dan mentioned, the cache device it-self
> should be safe against power failure. I agree this is not the case for
> all SSD cards in the market but might be the case for Facebook. I hate
> to say this but It seems these efforts are useful dependent to what
> kind of hardware is deployed for cache device.
>
It would be nice, but probably not possible, to have some form of
black-list of "these devices are unsafe/dangerous". Along the lines of
"mdadm --probe /dev/sda" or whatever, that gets the device type, checks
it, and says "this SSD can be destroyed by a power failure" or "this is
a cheap disk with the timeout problem" or something. But even if someone
did it, the database would probably bit-rot fairly quickly :-(
Cheers,
Wol
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-01 21:53 UTC (permalink / raw)
To: Shaohua Li; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150330222459.GA575371@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 5690 bytes --]
On Mon, 30 Mar 2015 15:25:17 -0700 Shaohua Li <shli@fb.com> wrote:
> This is my attempt to fix raid5/6 write hole issue, it's not for merge
> yet, I post it out for comments. Any comments and suggestions are
> welcome!
>
> Thanks,
> Shaohua
>
> We expect a completed raid5/6 stack with reliability and high
> performance. Currently raid5/6 has 2 issues:
>
> 1. read-modify-write for small size IO. To fix this issue, a cache layer
> above raid5/6 can be used to aggregate write to full stripe write.
> 2. write hole issue. A write log below raid5/6 can fix the issue.
>
> We plan to use a SSD to fix the two issues. Here we just fix the write
> hole issue.
>
> 1. We don't try to fix the issues together. A cache layer will do write
> acceleration. A log layer will fix write hole. The seperation will
> simplify things a lot.
>
> 2. Current assumption is flashcache/bcache will be used as the cache
> layer. If they don't work well, we can fix them or add a simple cache
> layer for raid write aggregation later. We also assume cache layer will
> absorb write, so log doesn't worry about write latency.
>
> 3. For log, write will hit to log disk first, then raid disks, and
> finally IO completion is reported. An optimal way is to report IO
> completion just after IO hits to log disk to cut write latency. But in
> that way, read path need query log disk and increase complexity. And
> since we don't worry about write latency, we choose a simple soltuion.
> This will be revisited if there is performance issue.
>
> This design isn't intrusive for raid5/6. Actully only very few changes
> of existing code is required.
>
> Log looks like jbd. Stripe IO to raid disks will be written to log disk
> first in atomic way. Several stripe IO will consist a transaction. If
> all stripes of a transaction are finished, the tranaction can be
> checkpoint.
>
> Basic logic of raid 5/6 write will be:
> 1. normal raid5/6 steps for a stripe (fetch data, calculate checksum,
> and etc). log hooks to ops_run_io.
> 2. stripe is added to a transaction. Write stripe data to log disk (metadata
> block, stripe data)
> 3. write commit block to log disk
> 4. flush log disk cache.
> 5. stripe is logged now and normal stripe handling continues
>
> Transaction checkpoint process:
> 1. all stripes of a transaction are finished
> 2. flush disk cache of all raid disks
> 3. change log super to reflect new log checkpoint position
> 4. WRITE_FUA log super
>
> metadata, data and commit block IO can run in the meaning time, as
> checksum will be used to make sure their data is correct (like jbd2).
> Log IO doesn't wait 5s to start like jbd, instead the IO will start
> every time a metadata block is full. This can cut some latency.
>
> Disk layout:
>
> |super|metadata|data|metadata| data ... |commitdata|metadata|data| ... |commitdata|
> super, metadata, commit will use one block
>
> This is an initial version, which works but a lot of stuffes are
> missing:
> 1. error handling
> 2. log recovery and impact to raid resync (don't need resync anymore)
> 3. utility changes
>
> The big question is how we report log disk. In this patch, I simply use
> a spare disk for testing. We need a new raid disk role for log disk.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
Hi,
thanks for the proposal and the patch which makes it nice and concrete...
I should start out by saying that I'm not really sold on the importance of
the issues you are addressing here.
The "write hole" is certainly of theoretical significance, but I do wonder
how much practical significance it has. It can only be a problem if you
have a system failure and a degraded array at the same time, and both of
those should be very rare event individually...
I wonder if anyone has *ever* lost data to the "write hole".
As for write-ahead caching to reduce latency, most writes from Linux are
async and so would not benefit from that. If you do have a heavily
synchronous write load, then that can be fixed in the filesystem.
e.g. with ext3 and an external log to a low-latency device you can get
low-latency writes which largely mask the latency issues introduced by
RAID5.
The fact that I'm "not really sold" doesn't mean I am against them ... maybe
it is just an encouragement for someone to sell them more :-)
While I understand that keeping the two separate might simplify the
problem, I'm not at all sure it is a good idea. It would mean that every
data block were written three times - once to the write-ahead log, once to
the write-hole-protection log, and once to the RAID5.
Your code does avoid write-hole-protection for fill-stripe-writes, and this
would greatly reduce the number of block that were written multiple times.
However I'm not convinced that is correct.
A reasonable goal is that if the system crashes while writing to a storage
device, then reads should return the old data or not new data, not anything
else. A crash in the middle of a full-stripe-write to a degraded array
could result in some block in the stripe appearing to contain data that is
different to both the old and the new. If you are going to close the whole,
I think it should be done properly.
A combined log would "simply" involve writing every data block and every
compute parity block (with index information) to the log device.
Replaying the log would collect data blocks and flush out those in a stripe
once the parity block(s) for that stripe became available.
I think this would actually turn into a fairly simple logging mechanism.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-01 23:40 UTC (permalink / raw)
To: NeilBrown; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150402085312.5ea3d518@notabene.brown>
On Thu, Apr 02, 2015 at 08:53:12AM +1100, NeilBrown wrote:
> On Mon, 30 Mar 2015 15:25:17 -0700 Shaohua Li <shli@fb.com> wrote:
>
> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > yet, I post it out for comments. Any comments and suggestions are
> > welcome!
> >
> > Thanks,
> > Shaohua
> >
> > We expect a completed raid5/6 stack with reliability and high
> > performance. Currently raid5/6 has 2 issues:
> >
> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > above raid5/6 can be used to aggregate write to full stripe write.
> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >
> > We plan to use a SSD to fix the two issues. Here we just fix the write
> > hole issue.
> >
> > 1. We don't try to fix the issues together. A cache layer will do write
> > acceleration. A log layer will fix write hole. The seperation will
> > simplify things a lot.
> >
> > 2. Current assumption is flashcache/bcache will be used as the cache
> > layer. If they don't work well, we can fix them or add a simple cache
> > layer for raid write aggregation later. We also assume cache layer will
> > absorb write, so log doesn't worry about write latency.
> >
> > 3. For log, write will hit to log disk first, then raid disks, and
> > finally IO completion is reported. An optimal way is to report IO
> > completion just after IO hits to log disk to cut write latency. But in
> > that way, read path need query log disk and increase complexity. And
> > since we don't worry about write latency, we choose a simple soltuion.
> > This will be revisited if there is performance issue.
> >
> > This design isn't intrusive for raid5/6. Actully only very few changes
> > of existing code is required.
> >
> > Log looks like jbd. Stripe IO to raid disks will be written to log disk
> > first in atomic way. Several stripe IO will consist a transaction. If
> > all stripes of a transaction are finished, the tranaction can be
> > checkpoint.
> >
> > Basic logic of raid 5/6 write will be:
> > 1. normal raid5/6 steps for a stripe (fetch data, calculate checksum,
> > and etc). log hooks to ops_run_io.
> > 2. stripe is added to a transaction. Write stripe data to log disk (metadata
> > block, stripe data)
> > 3. write commit block to log disk
> > 4. flush log disk cache.
> > 5. stripe is logged now and normal stripe handling continues
> >
> > Transaction checkpoint process:
> > 1. all stripes of a transaction are finished
> > 2. flush disk cache of all raid disks
> > 3. change log super to reflect new log checkpoint position
> > 4. WRITE_FUA log super
> >
> > metadata, data and commit block IO can run in the meaning time, as
> > checksum will be used to make sure their data is correct (like jbd2).
> > Log IO doesn't wait 5s to start like jbd, instead the IO will start
> > every time a metadata block is full. This can cut some latency.
> >
> > Disk layout:
> >
> > |super|metadata|data|metadata| data ... |commitdata|metadata|data| ... |commitdata|
> > super, metadata, commit will use one block
> >
> > This is an initial version, which works but a lot of stuffes are
> > missing:
> > 1. error handling
> > 2. log recovery and impact to raid resync (don't need resync anymore)
> > 3. utility changes
> >
> > The big question is how we report log disk. In this patch, I simply use
> > a spare disk for testing. We need a new raid disk role for log disk.
> >
> > Signed-off-by: Shaohua Li <shli@fb.com>
>
>
> Hi,
> thanks for the proposal and the patch which makes it nice and concrete...
>
> I should start out by saying that I'm not really sold on the importance of
> the issues you are addressing here.
> The "write hole" is certainly of theoretical significance, but I do wonder
> how much practical significance it has. It can only be a problem if you
> have a system failure and a degraded array at the same time, and both of
> those should be very rare event individually...
> I wonder if anyone has *ever* lost data to the "write hole".
We have tens of thousands of machines. Rare event in terms of huge amount
of machines become a normal event :). It's not a significant issue, but
we don't want to take the risk.
> As for write-ahead caching to reduce latency, most writes from Linux are
> async and so would not benefit from that. If you do have a heavily
> synchronous write load, then that can be fixed in the filesystem.
> e.g. with ext3 and an external log to a low-latency device you can get
> low-latency writes which largely mask the latency issues introduced by
> RAID5.
Maybe I should write more about the caching, I didn't because the patch
is about the write hole issue. Any way, the side effect of caching is
write-hole-protection log doesn't need worry about latency. The main
purpose of caching is to produce full stripe write or reduce
read-modify-write if full stripe write is not possible. The caching can
also reduce hard disk spindle seek, because we can sort when flush data
from caching to raid.
> The fact that I'm "not really sold" doesn't mean I am against them ... maybe
> it is just an encouragement for someone to sell them more :-)
>
> While I understand that keeping the two separate might simplify the
> problem, I'm not at all sure it is a good idea. It would mean that every
> data block were written three times - once to the write-ahead log, once to
> the write-hole-protection log, and once to the RAID5.
yes, it the write-ahead log and write-hole-protection log are combined,
one write can be avoid.
> Your code does avoid write-hole-protection for fill-stripe-writes, and this
> would greatly reduce the number of block that were written multiple times.
> However I'm not convinced that is correct.
> A reasonable goal is that if the system crashes while writing to a storage
> device, then reads should return the old data or not new data, not anything
> else. A crash in the middle of a full-stripe-write to a degraded array
> could result in some block in the stripe appearing to contain data that is
> different to both the old and the new. If you are going to close the whole,
> I think it should be done properly.
I can do it simpley. But don't think this assumption is true. If you
write to a disk range and there is failure, there is nothing guarantee
you can either read old data or new data.
>
> A combined log would "simply" involve writing every data block and every
> compute parity block (with index information) to the log device.
> Replaying the log would collect data blocks and flush out those in a stripe
> once the parity block(s) for that stripe became available.
>
> I think this would actually turn into a fairly simple logging mechanism.
It's not simple at all. It's unlikely we write data and parity
continuously in disk and in the same time. This will make log checkpoint
fairly complex.
Thanks,
Shaohua
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-02 0:19 UTC (permalink / raw)
To: Shaohua Li; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150401234055.GA3375744@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 3985 bytes --]
On Wed, 1 Apr 2015 16:40:57 -0700 Shaohua Li <shli@fb.com> wrote:
> > Your code does avoid write-hole-protection for fill-stripe-writes, and this
> > would greatly reduce the number of block that were written multiple times.
> > However I'm not convinced that is correct.
> > A reasonable goal is that if the system crashes while writing to a storage
> > device, then reads should return the old data or not new data, not anything
> > else. A crash in the middle of a full-stripe-write to a degraded array
> > could result in some block in the stripe appearing to contain data that is
> > different to both the old and the new. If you are going to close the whole,
> > I think it should be done properly.
>
> I can do it simpley. But don't think this assumption is true. If you
> write to a disk range and there is failure, there is nothing guarantee
> you can either read old data or new data.
If you write a range of blocks to a normal disk and crash during the write,
each block will contain either the old data or the new data.
If you write a range to a degraded RAID5 and crash during the write, you
cannot make that same guarantee.
I don't know how important this is, but then I don't really know how
important any of this is.
>
> >
> > A combined log would "simply" involve writing every data block and every
> > compute parity block (with index information) to the log device.
> > Replaying the log would collect data blocks and flush out those in a stripe
> > once the parity block(s) for that stripe became available.
> >
> > I think this would actually turn into a fairly simple logging mechanism.
>
> It's not simple at all. It's unlikely we write data and parity
> continuously in disk and in the same time. This will make log checkpoint
> fairly complex.
I don't see any cause for complexity. Let me be more explicit.
I imagine that all data remains in the stripe cache, in memory, until it is
finally written to the RAID5. So the stripe cache will need to be quite a
bit bigger.
Every time we get a block that we want to write, either a new data block or a
a computed parity block, we queue it to the log.
The log works like this:
- take the first (e.g.) 256 blocks in the queue, create a header to describe
them, write the header with FUA, then write all the data blocks. If there
are fewer than 256, just write what we have.
- when the header write completes, all blocks written *previously* are now
safe and we can call bio_end_io on data or unlock the stripe for parity.
- loop back and write some more blocks. If there are no blocks to write,
write a header which describes an empty set of blocks, and wait for more
blocks to appear.
Each stripe_head needs to track (roughly) where the relevant blocks were
written so it can release them when the stripe is written.
I would conceptually divide the log into 32 regions and keep a 32bit number
with each stripe. When a block is assigned to a region in the log, the
relevant bit is set for the stripe, and a per-region counter is incremented.
When a stripe completes its write, the region counters for all the bits are
cleared. The log cannot progress into a region which has a non-zero counter.
We choose the size of transactions so that the first block of each region is
a header block. These contain a magic number, a sequence number, and a
checksum together with the addresses of the data/parity blocks. On restart
we read all 32 of these to find out where the log starts and ends. Then we
replay all the blocks into the stripe cache - discarding any that don't come
with the required parity blocks.
So it is a very simple log which is never read exact on crash recovery. It
commits everything ASAP so that the writeout to the array can be lazy and can
gather related blocks and sort address etc with not impact on filesystem
latency.
Does that make sense?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: /dev/md0 can't be created
From: Xiao Ni @ 2015-04-02 2:52 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150401135321.2f421b96@notabene.brown>
Hi Neil
I think that is enough to say it's not the problem about md. I really be
thankful for you about the analysis.
Best Regards
Xiao
----- Original Message -----
> From: "NeilBrown" <neilb@suse.de>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Wednesday, April 1, 2015 10:53:21 AM
> Subject: Re: /dev/md0 can't be created
>
> On Fri, 27 Mar 2015 02:06:21 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> >
> > I'm lucky, I reproduced it again and I ran the command udevadm monitor >
> > monitor before test.
> > I'll put it in as an attachment.
> >
> > I found there is a not whole message in the last line.
> >
> > Xiao
>
> The line:
>
> UDEV [7083.531441] add /devices/virtual/block/md0 (block)
>
> confirms that udev saw an 'add' event for /dev/md0.
>
> That should be enough for udev to create the device node.
> If it doesn't, you need to look into udev and find out why it doesn't.
> I can't really help you there - I don't find udev very easy to debug.
>
> NeilBrown
>
>
> >
> > ----- Original Message -----
> > > From: "Xiao Ni" <xni@redhat.com>
> > > To: "NeilBrown" <neilb@suse.de>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Thursday, March 26, 2015 3:36:35 PM
> > > Subject: Re: /dev/md0 can't be created
> > >
> > > Hi Neil
> > >
> > > I can reproduce it now. Do you want me to run udevadm monitor before
> > > the
> > > test?
> > > And I checked the /var/log/messages, it show the information:
> > >
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-2>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-3>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-4>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-5>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-6>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-7>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-9>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-8>
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-7 operational
> > > as
> > > raid disk 5
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-6 operational
> > > as
> > > raid disk 4
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-5 operational
> > > as
> > > raid disk 3
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-4 operational
> > > as
> > > raid disk 2
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-3 operational
> > > as
> > > raid disk 1
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-2 operational
> > > as
> > > raid disk 0
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: allocated 0kB
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: raid level 5 active with
> > > 6
> > > out of 7 devices, algorithm 2
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid456: discard support disabled
> > > due
> > > to uncertainty.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: Set
> > > raid456.devices_handle_discard_safely=Y to override.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md0: detected capacity change from 0
> > > to
> > > 1881145344
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: recovery of RAID array md0
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: minimum _guaranteed_ speed: 1000
> > > KB/sec/disk.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: using maximum available idle IO
> > > bandwidth (but not more than 200000 KB/sec) for recovery.
> > > Oct 20 19:59:59 ibm-z10-25 kernel: md: using 128k window, over a total of
> > > 306176k.
> > > Oct 20 19:59:59 ibm-z10-25 systemd-udevd: inotify_add_watch(7, /dev/md0,
> > > 10)
> > > failed: No such file or directory
> > >
> > > Xiao
> > >
> > > ----- Original Message -----
> > > > From: "NeilBrown" <neilb@suse.de>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org
> > > > Sent: Wednesday, March 25, 2015 2:35:29 PM
> > > > Subject: Re: /dev/md0 can't be created
> > > >
> > > > On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com>
> > > > wrote:
> > > >
> > > > > Hi all
> > > > >
> > > > > I have encountered so many times, the raid device is created
> > > > > successfully, but the directory
> > > > > /dev/md0 can't be created. It can't reproduce 100%.
> > > > >
> > > > > [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> > > > > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > > > > md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3]
> > > > > loop2[2] loop1[1] loop0[0]
> > > > > 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7]
> > > > > [UUUUUUU]
> > > > > bitmap: 0/1 pages [0KB], 65536KB chunk
> > > > >
> > > > > unused devices: <none>
> > > > > [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> > > > > ls: cannot access /dev/md0: No such file or directory
> > > > >
> > > > > The underline devices are loop devices which are created with big
> > > > > file.
> > > > >
> > > > > The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm -
> > > > > v3.3.2
> > > > > - 21st August 2014)
> > > > > I'll try to reproduce this with upstream kernel and mdadm. But I
> > > > > think it
> > > > > shouldn't be the problem about kernel.
> > > > >
> > > > > What do you think I should check for this? And which tool is
> > > > > responsible for creating the directory? Maybe
> > > > > I can add some log to it to find the reason.
> > > > >
> > > >
> > > > /dev/md0 is created by udev.
> > > > Run
> > > > udevadm monitor
> > > >
> > > > to see the events that udev is processing. When and ADD event for
> > > > "md0" is
> > > > processed, /dev/md0 should get created.
> > > >
> > > > NeilBrown
> > > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
>
>
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-02 4:07 UTC (permalink / raw)
To: NeilBrown; +Cc: dan.j.williams, linux-raid, songliubraving, Kernel-team
In-Reply-To: <20150402111941.104d0633@notabene.brown>
On Thu, Apr 02, 2015 at 11:19:41AM +1100, NeilBrown wrote:
> On Wed, 1 Apr 2015 16:40:57 -0700 Shaohua Li <shli@fb.com> wrote:
>
>
> > > Your code does avoid write-hole-protection for fill-stripe-writes, and this
> > > would greatly reduce the number of block that were written multiple times.
> > > However I'm not convinced that is correct.
> > > A reasonable goal is that if the system crashes while writing to a storage
> > > device, then reads should return the old data or not new data, not anything
> > > else. A crash in the middle of a full-stripe-write to a degraded array
> > > could result in some block in the stripe appearing to contain data that is
> > > different to both the old and the new. If you are going to close the whole,
> > > I think it should be done properly.
> >
> > I can do it simpley. But don't think this assumption is true. If you
> > write to a disk range and there is failure, there is nothing guarantee
> > you can either read old data or new data.
>
> If you write a range of blocks to a normal disk and crash during the write,
> each block will contain either the old data or the new data.
> If you write a range to a degraded RAID5 and crash during the write, you
> cannot make that same guarantee.
> I don't know how important this is, but then I don't really know how
> important any of this is.
>
> >
> > >
> > > A combined log would "simply" involve writing every data block and every
> > > compute parity block (with index information) to the log device.
> > > Replaying the log would collect data blocks and flush out those in a stripe
> > > once the parity block(s) for that stripe became available.
> > >
> > > I think this would actually turn into a fairly simple logging mechanism.
> >
> > It's not simple at all. It's unlikely we write data and parity
> > continuously in disk and in the same time. This will make log checkpoint
> > fairly complex.
>
> I don't see any cause for complexity. Let me be more explicit.
>
> I imagine that all data remains in the stripe cache, in memory, until it is
> finally written to the RAID5. So the stripe cache will need to be quite a
> bit bigger.
>
> Every time we get a block that we want to write, either a new data block or a
> a computed parity block, we queue it to the log.
>
> The log works like this:
> - take the first (e.g.) 256 blocks in the queue, create a header to describe
> them, write the header with FUA, then write all the data blocks. If there
> are fewer than 256, just write what we have.
> - when the header write completes, all blocks written *previously* are now
> safe and we can call bio_end_io on data or unlock the stripe for parity.
> - loop back and write some more blocks. If there are no blocks to write,
> write a header which describes an empty set of blocks, and wait for more
> blocks to appear.
Ok, this is similar.
> Each stripe_head needs to track (roughly) where the relevant blocks were
> written so it can release them when the stripe is written.
> I would conceptually divide the log into 32 regions and keep a 32bit number
> with each stripe. When a block is assigned to a region in the log, the
> relevant bit is set for the stripe, and a per-region counter is incremented.
> When a stripe completes its write, the region counters for all the bits are
> cleared. The log cannot progress into a region which has a non-zero counter.
I like this region idea very much. Previously I thought the combined log
is complex because data and parity are not in adjacent disk location,
and can cause fragement, so make checkpoint complex. The region
effectively solves the problem, but a big size region would still have
the fragement issue. We can divide the disk to a lot of equal sized
regions, the region size could be 4k*raid_disks*2 for example. Each
region is a log for exact one stripe. Write will append data to such
log and write is finished. parity append to the log too and then the
region is considered settled down. The downside is meta will use 1
sector even just several bytes are required and this will produce a lot
of small size IO too.
I'm not enthusiastic to use stripe cache though, we can't keep all data
in stripe cache. What we really need is an index.
Thanks,
Shaohua
^ permalink raw reply
* Re: [neilb@suse.de: Re: MD request counter broken on 4.0-rc5 RAID1]
From: Simon Kirby @ 2015-04-02 19:54 UTC (permalink / raw)
To: NeilBrown; +Cc: Gu Zheng, linux-raid
In-Reply-To: <20150401220725.53ee8f99@notabene.brown>
On Wed, Apr 01, 2015 at 10:07:25PM +1100, NeilBrown wrote:
> On Wed, 1 Apr 2015 15:39:47 +0800 Gu Zheng <guz.fnst@cn.fujitsu.com> wrote:
>
> > Hi Neil,
> > One question about md io stats account:
> > I find that we just accounting disk_stats.sectors[rw] and disk_stats.ios[rw] at
> > md_make_request, but I did not find accounting disk_stats.ticks[rw], do not we
> > need it? or I missed something?
>
> No, I don't think you are missing anything. md doesn't track 'ticks' - and
> no-one has ever complained.
Hmm, yes, I guess "working" before was "0% utilization". :)
Simon-
^ permalink raw reply
* [PATCH] md: fix md io stats accounting broken
From: Gu Zheng @ 2015-04-03 0:44 UTC (permalink / raw)
To: neilb; +Cc: sim, linux-raid, linux-kernel, stable
Simon reported the md io stats accounting issue:
"
I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
the other two being write-mostly:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
"
The cause is commit "18c0b223cf9901727ef3b02da6711ac930b4e5d4" uses the
generic_start_io_acct to account the disk stats rather than the open code,
but it also introduced the increase to .in_flight[rw] which is needless to
md. So we re-use the open code here to fix it.
Reported-by: Simon Kirby <sim@hostway.ca>
Cc: <stable@vger.kernel.org> 3.19
Signed-off-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
---
drivers/md/md.c | 6 +++++-
1 files changed, 5 insertions(+), 1 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 717daad..e617878 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -249,6 +249,7 @@ static void md_make_request(struct request_queue *q, struct bio *bio)
const int rw = bio_data_dir(bio);
struct mddev *mddev = q->queuedata;
unsigned int sectors;
+ int cpu;
if (mddev == NULL || mddev->pers == NULL
|| !mddev->ready) {
@@ -284,7 +285,10 @@ static void md_make_request(struct request_queue *q, struct bio *bio)
sectors = bio_sectors(bio);
mddev->pers->make_request(mddev, bio);
- generic_start_io_acct(rw, sectors, &mddev->gendisk->part0);
+ cpu = part_stat_lock();
+ part_stat_inc(cpu, &mddev->gendisk->part0, ios[rw]);
+ part_stat_add(cpu, &mddev->gendisk->part0, sectors[rw], sectors);
+ part_stat_unlock();
if (atomic_dec_and_test(&mddev->active_io) && mddev->suspended)
wake_up(&mddev->sb_wait);
--
1.7.7
^ permalink raw reply related
* Breaks LSI RAID on C600 chipset
From: David F. @ 2015-04-04 0:25 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Loading V3.3.2 Aug 2014 on an asus z9pa-u8 mobo, with the controller
in LSI RAID mode and configured using RAID 1 via the BIOS, will break
the RAID. (no longer configured as RAID). All it takes is starting
mdadm which seems to work for that session but on reboot the RAID is
gone. The z9pa-u8 uses the C602 intel chipset.
Customer reported same on what they called an Intel ESRT2 which
appears to use the same line C600 based chipset. Not sure if in LSI
mode, but presume so. That controller allows you to pick LSI mode,
Intel RAID mode, or just non RAID normal controller mode.
Unfortunately, mine is a production server so I can't do much testing
(even trying to load it will probably break it). Not sure if you can
get your hands on one to test. Loading using DMRAID worked okay
which is how I got around it.
^ permalink raw reply
* Re: Breaks LSI RAID on C600 chipset
From: John Stoffel @ 2015-04-04 1:47 UTC (permalink / raw)
To: David F.; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAGRSmLtoCz+wff+F=xnnusc5_bUYxWYT5V5iTncTnRfgRnTqtg@mail.gmail.com>
David> Loading V3.3.2 Aug 2014 on an asus z9pa-u8 mobo, with the
David> controller in LSI RAID mode and configured using RAID 1 via the
David> BIOS, will break the RAID. (no longer configured as RAID).
David> All it takes is starting mdadm which seems to work for that
David> session but on reboot the RAID is gone. The z9pa-u8 uses the
David> C602 intel chipset.
Which version of Linux kernel and disto are you using?
David> Customer reported same on what they called an Intel ESRT2 which
David> appears to use the same line C600 based chipset. Not sure if
David> in LSI mode, but presume so. That controller allows you to
David> pick LSI mode, Intel RAID mode, or just non RAID normal
David> controller mode.
David> Unfortunately, mine is a production server so I can't do much testing
David> (even trying to load it will probably break it). Not sure if you can
David> get your hands on one to test. Loading using DMRAID worked okay
David> which is how I got around it.
Can you please give us the exact steps you took, and the output at
each step? Also, which version of mdadm are you running which does
NOT break things? That would be good to know.
The output of 'cat /proc/mdstat', 'mdadm --detail /dev/md*' and 'mdadm
-E /dev/sd?e' for each device in the various raid arrays.
If you start then stop the new mdadm, do you see differences when you
then use the old, working mdadm to examine the various devices?
Basically, we need lots more information here to help.
John
^ permalink raw reply
* Help recovering an interrupted raid0 reshape
From: Jonathan Harker (Jesusaurus) @ 2015-04-04 2:22 UTC (permalink / raw)
To: linux-raid
I have a raid0 array whose component devices are raid1 arrays. In an
attempt to expand the pre-existing raid0 array, I created a new raid1
device and then added it, growing the raid0 array.
But then the system lost power shortly after the reshaping began.
After rebooting the original two components are listed as spares in an
inactive raid4 array and the new component is not listed in
/proc/mdstat:
Personalities : [raid6] [raid5] [raid4] [raid1] [raid10] [raid0]
[linear] [multipath]
md124 : inactive md126[0](S) md127[1](S)
3907022200 blocks super 1.2
md0 : active raid1 sda5[0] sdb2[1]
107652416 blocks [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md125 : active raid1 sdh1[0] sdg1[1]
2930134016 blocks super 1.2 [2/2] [UU]
bitmap: 0/22 pages [0KB], 65536KB chunk
md126 : active raid1 sdc1[0] sdd1[1]
1953512312 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sde1[2] sdf1[1]
1953512312 blocks super 1.2 [2/2] [UU]
unused devices: <none>
Looking at the details of the inactive array shows that it is in a
reshape between raid0 and raid4:
/dev/md124:
Version : 1.2
Raid Level : raid0
Total Devices : 2
Persistence : Superblock is persistent
State : inactive
Delta Devices : -1, (1->0)
New Level : raid4
New Chunksize : 512K
Name : hordern:hordern1 (local to host hordern)
UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Events : 14013
Number Major Minor RaidDevice
- 9 126 - /dev/md/beta
- 9 127 - /dev/md/alpha
And examining each component shows that they have a consistent view of
where in that reshape they are (based on the reshape position), but
not consistent in the size of the array:
/dev/md/alpha:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 3
Avail Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Array Size : 3907021824 (3726.03 GiB 4000.79 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=752 sectors
State : active
Device UUID : 63aaa2e4:2a09f495:8372c7f9:eb2f2773
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : -1 (4->3)
Update Time : Sun Mar 29 15:11:35 2015
Checksum : 8be5e0e6 - correct
Events : 14013
Chunk Size : 512K
Device Role : Active device 1
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/md/beta:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 3
Avail Dev Size : 3907022576 (1863.01 GiB 2000.40 GB)
Array Size : 3907021824 (3726.03 GiB 4000.79 GB)
Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=752 sectors
State : clean
Device UUID : 6e6dce14:3ebb2bb5:187aa292:403a55f6
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : -1 (4->3)
Update Time : Sun Mar 29 15:11:35 2015
Checksum : f7526add - correct
Events : 14013
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/md/gamma:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x6
Array UUID : 1f4979ba:c49a77c0:59e689c2:bcc21c0a
Name : hordern:hordern1 (local to host hordern)
Creation Time : Fri Jan 2 09:59:40 2009
Raid Level : raid4
Raid Devices : 4
Avail Dev Size : 5860265984 (2794.39 GiB 3000.46 GB)
Array Size : 5860532736 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907021824 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Recovery Offset : 86403072 sectors
Unused Space : before=1960 sectors, after=1953244160 sectors
State : active
Device UUID : 782873ea:e265ecd4:5cc80ddf:035ba2b4
Reshape pos'n : 129067008 (123.09 GiB 132.16 GB)
Delta Devices : 1 (3->4)
Update Time : Sun Mar 29 00:05:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 710dc078 - correct
Events : 673
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
When I stop the inactive array and try to assemble it from all three
components, I get an error about the superblock on the third component
not matching the other two components (which makes sense since the
array sizes are different):
hordern ~ # mdadm --verbose --verbose --assemble /dev/md/hordern1
/dev/md/alpha /dev/md/beta /dev/md/gamma
mdadm: looking for devices for /dev/md/hordern1
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: UUID differs from /dev/md0.
mdadm: UUID differs from /dev/md/alpha.
mdadm: UUID differs from /dev/md/beta.
mdadm: UUID differs from /dev/md/gamma.
mdadm: superblock on /dev/md/gamma doesn't match others - assembly aborted
First, what could cause the initial two components to have a different
superblock than the newly added component? And can I convince them to
be the same?
Second, is there documentation anywhere about the internal process of
growing a raid0 array? Why does it convert to a raid4 array? And what
do the Delta Devices lines mean?
Third, is it possible to resume the reshape? If not, can it be reverted?
--
Jon
^ permalink raw reply
* [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-05 17:20 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, Rafael J. Wysocki,
Len Brown, Pavel Machek
Cc: dm-devel, linux-raid, linux-kernel, linux-pm, Pali Rohár
This patch series increase security of suspend and hibernate actions. It allows
user to safely wipe crypto keys before suspend and hibernate actions starts
without race conditions on userspace process with heavy I/O.
To automatically wipe cryto key for <device> before hibernate action call:
$ dmsetup message <device> 0 key wipe_on_hibernation 1
To automatically wipe cryto key for <device> before suspend action call:
$ dmsetup message <device> 0 key wipe_on_suspend 1
(Value 0 after wipe_* string reverts original behaviour - to not wipe key)
Pali Rohár (3):
PM suspend/hibernate: Call notifier after freezing processes
dm: Export function dm_suspend_md()
dm-crypt: Adds support for wiping key when doing suspend/hibernation
drivers/md/dm-crypt.c | 109 +++++++++++++++++++++++++++++++++++++++++++---
drivers/md/dm.c | 6 +++
drivers/md/dm.h | 5 +++
include/linux/suspend.h | 2 +
kernel/power/hibernate.c | 2 +
kernel/power/suspend.c | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
--
1.7.9.5
^ permalink raw reply
* [PATCH 1/3] PM suspend/hibernate: Call notifier after freezing processes
From: Pali Rohár @ 2015-04-05 17:20 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, Rafael J. Wysocki,
Len Brown, Pavel Machek
Cc: dm-devel, linux-raid, linux-kernel, linux-pm, Pali Rohár
In-Reply-To: <1428254419-7334-1-git-send-email-pali.rohar@gmail.com>
To prevent race conditions on userspace processes with I/O some taks must be
called after processes are freezed. This patch adds new events which are
delivered by pm_notifier_call_chain() after freezing processes when doing
suspend or hibernate action.
Signed-off-by: Pali Rohár <pali.rohar@gmail.com>
---
include/linux/suspend.h | 2 ++
kernel/power/hibernate.c | 2 ++
kernel/power/suspend.c | 4 +++-
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/include/linux/suspend.h b/include/linux/suspend.h
index 5efe743..bc743c8 100644
--- a/include/linux/suspend.h
+++ b/include/linux/suspend.h
@@ -368,6 +368,8 @@ static inline bool hibernation_available(void) { return false; }
#define PM_POST_SUSPEND 0x0004 /* Suspend finished */
#define PM_RESTORE_PREPARE 0x0005 /* Going to restore a saved image */
#define PM_POST_RESTORE 0x0006 /* Restore failed */
+#define PM_HIBERNATION_AFTER_FREEZE 0x0007 /* After hibernation freeze */
+#define PM_SUSPEND_AFTER_FREEZE 0x0008 /* After suspend freeze */
extern struct mutex pm_mutex;
diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c
index 2329daa..184f7ee 100644
--- a/kernel/power/hibernate.c
+++ b/kernel/power/hibernate.c
@@ -671,6 +671,8 @@ int hibernate(void)
if (error)
goto Exit;
+ pm_notifier_call_chain(PM_HIBERNATION_AFTER_FREEZE);
+
lock_device_hotplug();
/* Allocate memory management structures */
error = create_basic_memory_bitmaps();
diff --git a/kernel/power/suspend.c b/kernel/power/suspend.c
index b7d6b3a..1776938 100644
--- a/kernel/power/suspend.c
+++ b/kernel/power/suspend.c
@@ -268,8 +268,10 @@ static int suspend_prepare(suspend_state_t state)
trace_suspend_resume(TPS("freeze_processes"), 0, true);
error = suspend_freeze_processes();
trace_suspend_resume(TPS("freeze_processes"), 0, false);
- if (!error)
+ if (!error) {
+ pm_notifier_call_chain(PM_SUSPEND_AFTER_FREEZE);
return 0;
+ }
suspend_stats.failed_freeze++;
dpm_save_failed_step(SUSPEND_FREEZE);
--
1.7.9.5
^ permalink raw reply related
* [PATCH 2/3] dm: Export function dm_suspend_md()
From: Pali Rohár @ 2015-04-05 17:20 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, Rafael J. Wysocki,
Len Brown, Pavel Machek
Cc: dm-devel, linux-raid, linux-kernel, linux-pm, Pali Rohár
In-Reply-To: <1428254419-7334-1-git-send-email-pali.rohar@gmail.com>
This patch exports function dm_suspend_md() which suspend mapped device so other
kernel drivers can use it and could suspend mapped device when needed.
Signed-off-by: Pali Rohár <pali.rohar@gmail.com>
---
drivers/md/dm.c | 6 ++++++
drivers/md/dm.h | 5 +++++
2 files changed, 11 insertions(+)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 8001fe9..919ce95 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -3053,6 +3053,12 @@ out:
return r;
}
+int dm_suspend_md(struct mapped_device *md)
+{
+ return dm_suspend(md, DM_SUSPEND_LOCKFS_FLAG);
+}
+EXPORT_SYMBOL_GPL(dm_suspend_md);
+
/*
* Internal suspend/resume works like userspace-driven suspend. It waits
* until all bios finish and prevents issuing new bios to the target drivers.
diff --git a/drivers/md/dm.h b/drivers/md/dm.h
index 59f53e7..623c9a8 100644
--- a/drivers/md/dm.h
+++ b/drivers/md/dm.h
@@ -152,6 +152,11 @@ int dm_test_deferred_remove_flag(struct mapped_device *md);
void dm_deferred_remove(void);
/*
+ * Suspend mapped_device
+ */
+int dm_suspend_md(struct mapped_device *md);
+
+/*
* The device-mapper can be driven through one of two interfaces;
* ioctl or filesystem, depending which patch you have applied.
*/
--
1.7.9.5
^ permalink raw reply related
* [PATCH 3/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-05 17:20 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown, Rafael J. Wysocki,
Len Brown, Pavel Machek
Cc: dm-devel, linux-raid, linux-kernel, linux-pm, Pali Rohár
In-Reply-To: <1428254419-7334-1-git-send-email-pali.rohar@gmail.com>
This patch adds dm message commands and option strings to optionally wipe key
from dm-crypt device before entering suspend or hibernate state.
Before key is wiped dm device must be suspended. To prevent race conditions with
I/O and userspace processes, wiping action must be called after processes are
freezed. Otherwise userspace processes could start reading/writing to disk after
dm device is suspened and freezing processes before suspend/hibernate action
will fail.
Signed-off-by: Pali Rohár <pali.rohar@gmail.com>
---
drivers/md/dm-crypt.c | 109 +++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 102 insertions(+), 7 deletions(-)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 713a962..9b02824 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -23,6 +23,7 @@
#include <linux/atomic.h>
#include <linux/scatterlist.h>
#include <linux/rbtree.h>
+#include <linux/suspend.h>
#include <asm/page.h>
#include <asm/unaligned.h>
#include <crypto/hash.h>
@@ -31,6 +32,8 @@
#include <linux/device-mapper.h>
+#include "dm.h"
+
#define DM_MSG_PREFIX "crypt"
/*
@@ -112,13 +115,18 @@ struct iv_tcw_private {
* and encrypts / decrypts at the same time.
*/
enum flags { DM_CRYPT_SUSPENDED, DM_CRYPT_KEY_VALID,
- DM_CRYPT_SAME_CPU, DM_CRYPT_NO_OFFLOAD };
+ DM_CRYPT_SAME_CPU, DM_CRYPT_NO_OFFLOAD,
+ DM_CRYPT_KEY_WIPE_ON_HIBERNATION,
+ DM_CRYPT_KEY_WIPE_ON_SUSPEND,
+};
/*
* The fields in here must be read only after initialization.
*/
struct crypt_config {
struct dm_dev *dev;
+ struct dm_target *ti;
+ struct list_head entry;
sector_t start;
/*
@@ -181,6 +189,9 @@ struct crypt_config {
#define MIN_IOS 16
+static LIST_HEAD(crypt_list);
+static DEFINE_MUTEX(crypt_list_mtx);
+
static void clone_init(struct dm_crypt_io *, struct bio *);
static void kcryptd_queue_crypt(struct dm_crypt_io *io);
static u8 *iv_of_dmreq(struct crypt_config *cc, struct dm_crypt_request *dmreq);
@@ -1497,12 +1508,26 @@ out:
static int crypt_wipe_key(struct crypt_config *cc)
{
+ int ret;
+
+ if (cc->iv_gen_ops && cc->iv_gen_ops->wipe) {
+ ret = cc->iv_gen_ops->wipe(cc);
+ if (ret)
+ return ret;
+ }
+
clear_bit(DM_CRYPT_KEY_VALID, &cc->flags);
memset(&cc->key, 0, cc->key_size * sizeof(u8));
return crypt_setkey_allcpus(cc);
}
+static void crypt_suspend_and_wipe_key(struct crypt_config *cc)
+{
+ dm_suspend_md(dm_table_get_md(cc->ti->table));
+ crypt_wipe_key(cc);
+}
+
static void crypt_dtr(struct dm_target *ti)
{
struct crypt_config *cc = ti->private;
@@ -1512,6 +1537,10 @@ static void crypt_dtr(struct dm_target *ti)
if (!cc)
return;
+ mutex_lock(&crypt_list_mtx);
+ list_del(&cc->entry);
+ mutex_unlock(&crypt_list_mtx);
+
if (cc->write_thread)
kthread_stop(cc->write_thread);
@@ -1738,6 +1767,7 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
cc->key_size = key_size;
ti->private = cc;
+ cc->ti = ti;
ret = crypt_ctr_cipher(ti, argv[0], argv[1]);
if (ret < 0)
goto bad;
@@ -1832,7 +1862,14 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
else if (!strcasecmp(opt_string, "submit_from_crypt_cpus"))
set_bit(DM_CRYPT_NO_OFFLOAD, &cc->flags);
+ else if (!strcasecmp(opt_string, "key_wipe_on_hibernation"))
+ set_bit(DM_CRYPT_KEY_WIPE_ON_HIBERNATION, &cc->flags);
+
+ else if (!strcasecmp(opt_string, "key_wipe_on_suspend"))
+ set_bit(DM_CRYPT_KEY_WIPE_ON_SUSPEND, &cc->flags);
+
else {
+ ret = -EINVAL;
ti->error = "Invalid feature arguments";
goto bad;
}
@@ -1871,6 +1908,10 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
ti->num_flush_bios = 1;
ti->discard_zeroes_data_unsupported = true;
+ mutex_lock(&crypt_list_mtx);
+ list_add(&cc->entry, &crypt_list);
+ mutex_unlock(&crypt_list_mtx);
+
return 0;
bad:
@@ -1979,6 +2020,8 @@ static void crypt_resume(struct dm_target *ti)
/* Message interface
* key set <key>
* key wipe
+ * key wipe_on_hibernation <0|1>
+ * key wipe_on_suspend <0|1>
*/
static int crypt_message(struct dm_target *ti, unsigned argc, char **argv)
{
@@ -1989,6 +2032,30 @@ static int crypt_message(struct dm_target *ti, unsigned argc, char **argv)
goto error;
if (!strcasecmp(argv[0], "key")) {
+ if (argc == 3 && !strcasecmp(argv[1], "wipe_on_hibernation")) {
+ if (!strcmp(argv[2], "1")) {
+ set_bit(DM_CRYPT_KEY_WIPE_ON_HIBERNATION, &cc->flags);
+ return 0;
+ } else if (!strcmp(argv[2], "0")) {
+ clear_bit(DM_CRYPT_KEY_WIPE_ON_HIBERNATION, &cc->flags);
+ return 0;
+ } else {
+ DMWARN("unrecognised message received.");
+ return -EINVAL;
+ }
+ }
+ if (argc == 3 && !strcasecmp(argv[1], "wipe_on_suspend")) {
+ if (!strcmp(argv[2], "1")) {
+ set_bit(DM_CRYPT_KEY_WIPE_ON_SUSPEND, &cc->flags);
+ return 0;
+ } else if (!strcmp(argv[2], "0")) {
+ clear_bit(DM_CRYPT_KEY_WIPE_ON_SUSPEND, &cc->flags);
+ return 0;
+ } else {
+ DMWARN("unrecognised message received.");
+ return -EINVAL;
+ }
+ }
if (!test_bit(DM_CRYPT_SUSPENDED, &cc->flags)) {
DMWARN("not suspended during key manipulation.");
return -EINVAL;
@@ -2002,11 +2069,6 @@ static int crypt_message(struct dm_target *ti, unsigned argc, char **argv)
return ret;
}
if (argc == 2 && !strcasecmp(argv[1], "wipe")) {
- if (cc->iv_gen_ops && cc->iv_gen_ops->wipe) {
- ret = cc->iv_gen_ops->wipe(cc);
- if (ret)
- return ret;
- }
return crypt_wipe_key(cc);
}
}
@@ -2055,19 +2117,52 @@ static struct target_type crypt_target = {
.iterate_devices = crypt_iterate_devices,
};
+static int dm_crypt_pm_notifier_call(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct crypt_config *cc;
+
+ mutex_lock(&crypt_list_mtx);
+
+ list_for_each_entry(cc, &crypt_list, entry) {
+ if ((action == PM_HIBERNATION_AFTER_FREEZE &&
+ test_bit(DM_CRYPT_KEY_WIPE_ON_HIBERNATION, &cc->flags)) ||
+ (action == PM_SUSPEND_AFTER_FREEZE &&
+ test_bit(DM_CRYPT_KEY_WIPE_ON_SUSPEND, &cc->flags))) {
+ crypt_suspend_and_wipe_key(cc);
+ }
+ }
+
+ mutex_unlock(&crypt_list_mtx);
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block dm_crypt_pm_notifier_block = {
+ .notifier_call = dm_crypt_pm_notifier_call,
+};
+
static int __init dm_crypt_init(void)
{
int r;
r = dm_register_target(&crypt_target);
- if (r < 0)
+ if (r < 0) {
DMERR("register failed %d", r);
+ return r;
+ }
+
+ r = register_pm_notifier(&dm_crypt_pm_notifier_block);
+ if (r) {
+ DMWARN("register_pm_notifier failed %d", r);
+ }
return r;
}
static void __exit dm_crypt_exit(void)
{
+ unregister_pm_notifier(&dm_crypt_pm_notifier_block);
dm_unregister_target(&crypt_target);
}
--
1.7.9.5
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Mike Snitzer @ 2015-04-06 13:00 UTC (permalink / raw)
To: Pali Rohár
Cc: Alasdair Kergon, Neil Brown, Rafael J. Wysocki, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <1428254419-7334-1-git-send-email-pali.rohar@gmail.com>
On Sun, Apr 05 2015 at 1:20pm -0400,
Pali Rohár <pali.rohar@gmail.com> wrote:
> This patch series increase security of suspend and hibernate actions. It allows
> user to safely wipe crypto keys before suspend and hibernate actions starts
> without race conditions on userspace process with heavy I/O.
>
> To automatically wipe cryto key for <device> before hibernate action call:
> $ dmsetup message <device> 0 key wipe_on_hibernation 1
>
> To automatically wipe cryto key for <device> before suspend action call:
> $ dmsetup message <device> 0 key wipe_on_suspend 1
>
> (Value 0 after wipe_* string reverts original behaviour - to not wipe key)
Can you elaborate on the attack vector your changes are meant to protect
against? The user already authorized access, why is it inherently
dangerous to _not_ wipe the associated key across these events?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pavel Machek @ 2015-04-06 13:25 UTC (permalink / raw)
To: Mike Snitzer
Cc: Pali Rohár, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150406130045.GA18583@redhat.com>
On Mon 2015-04-06 09:00:46, Mike Snitzer wrote:
> On Sun, Apr 05 2015 at 1:20pm -0400,
> Pali Rohár <pali.rohar@gmail.com> wrote:
>
> > This patch series increase security of suspend and hibernate actions. It allows
> > user to safely wipe crypto keys before suspend and hibernate actions starts
> > without race conditions on userspace process with heavy I/O.
> >
> > To automatically wipe cryto key for <device> before hibernate action call:
> > $ dmsetup message <device> 0 key wipe_on_hibernation 1
> >
> > To automatically wipe cryto key for <device> before suspend action call:
> > $ dmsetup message <device> 0 key wipe_on_suspend 1
> >
> > (Value 0 after wipe_* string reverts original behaviour - to not wipe key)
>
> Can you elaborate on the attack vector your changes are meant to protect
> against? The user already authorized access, why is it inherently
> dangerous to _not_ wipe the associated key across these events?
Umm. You are using your notebook. It is unlikely to be stolen at that
point. You close the lid and board the airplane, stowing it in
overhead bin. There's much better chance of notebook being stolen now.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pali Rohár @ 2015-04-06 13:29 UTC (permalink / raw)
To: Mike Snitzer
Cc: Alasdair Kergon, Neil Brown, Rafael J. Wysocki, Len Brown,
Pavel Machek, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150406130045.GA18583@redhat.com>
[-- Attachment #1: Type: Text/Plain, Size: 3187 bytes --]
On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> On Sun, Apr 05 2015 at 1:20pm -0400,
>
> Pali Rohár <pali.rohar@gmail.com> wrote:
> > This patch series increase security of suspend and hibernate
> > actions. It allows user to safely wipe crypto keys before
> > suspend and hibernate actions starts without race
> > conditions on userspace process with heavy I/O.
> >
> > To automatically wipe cryto key for <device> before
> > hibernate action call: $ dmsetup message <device> 0 key
> > wipe_on_hibernation 1
> >
> > To automatically wipe cryto key for <device> before suspend
> > action call: $ dmsetup message <device> 0 key
> > wipe_on_suspend 1
> >
> > (Value 0 after wipe_* string reverts original behaviour - to
> > not wipe key)
>
> Can you elaborate on the attack vector your changes are meant
> to protect against? The user already authorized access, why
> is it inherently dangerous to _not_ wipe the associated key
> across these events?
Hi,
yes, I will try to explain current problems with cryptsetup
luksSuspend command and hibernation.
First, sometimes it is needed to put machine into other hands.
You can still watch other person what is doing with machine, but
once if you let machine unlocked (e.g opened luks disk), she/he
can access encrypted data.
If you turn off machine, it could be safe, because luks disk
devices are locked. But if you enter machine into suspend or
hibernate state luks devices are still open. And my patches try
to achieve similar security as when machine is off (= no crypto
keys in RAM or on swap).
When doing hibernate on unencrypted swap it is to prevent leaking
crypto keys to hibernate image (which is stored in swap).
When doing suspend action it is again to prevent leaking crypto
keys. E.g when you suspend laptop and put it off (somebody can
remove RAMs and do some cold boot attack).
The most common situation is:
You have mounted partition from dm-crypt device (e.g. /home/),
some userspace processes access it (e.g opened firefox which
still reads/writes to cache ~/.firefox/) and you want to drop
crypto keys from kernel for some time.
For that operation there is command cryptsetup luksSuspend, which
suspend dm device and then tell kernel to wipe crypto keys. All
I/O operations are then stopped and userspace processes which
want to do some those I/O operations are stopped too (until you
call cryptsetup luksResume and enter correct key).
Now if you want to suspend/hiberate your machine (when some of dm
devices are suspeneded and some processes are stopped due to
pending I/O) it is not possible. Kernel freeze_processes function
will fail because userspace processes are still stopped inside
some I/O syscall (read/write, etc,...).
My patches fixes this problem and do those operations (suspend dm
device, wipe crypto keys, enter suspend/hiberate) in correct
order and without race condition.
dm device is suspended *after* userspace processes are freezed
and after that are crypto keys wiped. And then computer/laptop
enters into suspend/hibernate state.
--
Pali Rohár
pali.rohar@gmail.com
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Pavel Machek @ 2015-04-06 18:17 UTC (permalink / raw)
To: Pali Rohár
Cc: Mike Snitzer, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <201504061529.57299@pali>
On Mon 2015-04-06 15:29:57, Pali Rohár wrote:
> On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> > On Sun, Apr 05 2015 at 1:20pm -0400,
> >
> > Pali Rohár <pali.rohar@gmail.com> wrote:
> > > This patch series increase security of suspend and hibernate
> > > actions. It allows user to safely wipe crypto keys before
> > > suspend and hibernate actions starts without race
> > > conditions on userspace process with heavy I/O.
> > >
> > > To automatically wipe cryto key for <device> before
> > > hibernate action call: $ dmsetup message <device> 0 key
> > > wipe_on_hibernation 1
> > >
> > > To automatically wipe cryto key for <device> before suspend
> > > action call: $ dmsetup message <device> 0 key
> > > wipe_on_suspend 1
> > >
> > > (Value 0 after wipe_* string reverts original behaviour - to
> > > not wipe key)
> >
> > Can you elaborate on the attack vector your changes are meant
> > to protect against? The user already authorized access, why
> > is it inherently dangerous to _not_ wipe the associated key
> > across these events?
>
> Hi,
>
> yes, I will try to explain current problems with cryptsetup
> luksSuspend command and hibernation.
>
> First, sometimes it is needed to put machine into other hands.
> You can still watch other person what is doing with machine, but
> once if you let machine unlocked (e.g opened luks disk), she/he
> can access encrypted data.
>
> If you turn off machine, it could be safe, because luks disk
> devices are locked. But if you enter machine into suspend or
> hibernate state luks devices are still open. And my patches try
> to achieve similar security as when machine is off (= no crypto
> keys in RAM or on swap).
>
> When doing hibernate on unencrypted swap it is to prevent leaking
> crypto keys to hibernate image (which is stored in swap).
>
> When doing suspend action it is again to prevent leaking crypto
> keys. E.g when you suspend laptop and put it off (somebody can
> remove RAMs and do some cold boot attack).
>
> The most common situation is:
> You have mounted partition from dm-crypt device (e.g. /home/),
> some userspace processes access it (e.g opened firefox which
> still reads/writes to cache ~/.firefox/) and you want to drop
> crypto keys from kernel for some time.
>
> For that operation there is command cryptsetup luksSuspend, which
> suspend dm device and then tell kernel to wipe crypto keys. All
> I/O operations are then stopped and userspace processes which
> want to do some those I/O operations are stopped too (until you
> call cryptsetup luksResume and enter correct key).
Actually... is the list of sites where the process wait small enough?
Could we modify them to be freezeable? Suspend should work even if
user stopped the his crypto partitions...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Mark Knecht @ 2015-04-06 18:39 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux-RAID
In-Reply-To: <20150401013032.5c908e85@natsu>
On Tue, Mar 31, 2015 at 1:30 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Tue, 31 Mar 2015 08:00:27 -0700
> Mark Knecht <markknecht@gmail.com> wrote:
>
>> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
>> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
>> [====>................] resync = 22.8% (110608304/484088160)
>> finish=78.5min speed=79206K/sec
>
> Looks like you are running this without a write intent bitmap, why? It would
> at least save you from a full resync each time.
>
> --
> With respect,
> Roman
I had a 3rd lockup this morning. No idea what's causing them but the
write intent bitmap really got past the resync issue quickly.
Thanks!
Cheers,
Mark
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Mike Snitzer @ 2015-04-06 20:51 UTC (permalink / raw)
To: Pavel Machek
Cc: Pali Rohár, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150406132505.GB9978@amd>
On Mon, Apr 06 2015 at 9:25am -0400,
Pavel Machek <pavel@ucw.cz> wrote:
> On Mon 2015-04-06 09:00:46, Mike Snitzer wrote:
> > On Sun, Apr 05 2015 at 1:20pm -0400,
> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >
> > > This patch series increase security of suspend and hibernate actions. It allows
> > > user to safely wipe crypto keys before suspend and hibernate actions starts
> > > without race conditions on userspace process with heavy I/O.
> > >
> > > To automatically wipe cryto key for <device> before hibernate action call:
> > > $ dmsetup message <device> 0 key wipe_on_hibernation 1
> > >
> > > To automatically wipe cryto key for <device> before suspend action call:
> > > $ dmsetup message <device> 0 key wipe_on_suspend 1
> > >
> > > (Value 0 after wipe_* string reverts original behaviour - to not wipe key)
> >
> > Can you elaborate on the attack vector your changes are meant to protect
> > against? The user already authorized access, why is it inherently
> > dangerous to _not_ wipe the associated key across these events?
>
> Umm. You are using your notebook. It is unlikely to be stolen at that
> point. You close the lid and board the airplane, stowing it in
> overhead bin. There's much better chance of notebook being stolen now.
Yes, pretty straight forward but the thief would need to then login upon
resume (at least with most common desktop configs)... the barrier then
is only the strength of the user's password and not the crypt
passphrase.
^ permalink raw reply
* Why wipe crypto keys during suspend (was Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation)
From: Pavel Machek @ 2015-04-06 21:13 UTC (permalink / raw)
To: Mike Snitzer
Cc: Pali Rohár, Alasdair Kergon, Neil Brown, Rafael J. Wysocki,
Len Brown, dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150406205145.GA19677@redhat.com>
On Mon 2015-04-06 16:51:45, Mike Snitzer wrote:
> On Mon, Apr 06 2015 at 9:25am -0400,
> Pavel Machek <pavel@ucw.cz> wrote:
>
> > On Mon 2015-04-06 09:00:46, Mike Snitzer wrote:
> > > On Sun, Apr 05 2015 at 1:20pm -0400,
> > > Pali Rohár <pali.rohar@gmail.com> wrote:
> > >
> > > > This patch series increase security of suspend and hibernate actions. It allows
> > > > user to safely wipe crypto keys before suspend and hibernate actions starts
> > > > without race conditions on userspace process with heavy I/O.
> > > >
> > > > To automatically wipe cryto key for <device> before hibernate action call:
> > > > $ dmsetup message <device> 0 key wipe_on_hibernation 1
> > > >
> > > > To automatically wipe cryto key for <device> before suspend action call:
> > > > $ dmsetup message <device> 0 key wipe_on_suspend 1
> > > >
> > > > (Value 0 after wipe_* string reverts original behaviour - to not wipe key)
> > >
> > > Can you elaborate on the attack vector your changes are meant to protect
> > > against? The user already authorized access, why is it inherently
> > > dangerous to _not_ wipe the associated key across these events?
> >
> > Umm. You are using your notebook. It is unlikely to be stolen at that
> > point. You close the lid and board the airplane, stowing it in
> > overhead bin. There's much better chance of notebook being stolen now.
>
> Yes, pretty straight forward but the thief would need to then login upon
> resume (at least with most common desktop configs)... the barrier then
> is only the strength of the user's password and not the crypt
> passphrase.
Why would he want to do that? :-).
No; at that point, attacker would either wait for something remotely
exploitable to exploit, or attach JTAG debugger to the machine, or use
liquid nitrogen on RAMs and then attach them to running machine.
Yes, it is better when keys are not on your machine when it is stolen.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox