* bcache and hibernation
@ 2014-11-13 13:52 Mathijs Kwik
2014-11-13 15:52 ` Mathijs Kwik
2014-11-13 22:11 ` Kent Overstreet
0 siblings, 2 replies; 10+ messages in thread
From: Mathijs Kwik @ 2014-11-13 13:52 UTC (permalink / raw)
To: linux-bcache
Hi all,
Today, I lost most my data (don't worry, got backups) after the cache
got corrupted somehow. I suspected a recent suspend-to-disk to be the
cause. I checked how my distribution (NixOS) handles suspend/resume and
I have some concerns about how bcache fits into this.
Normally, the kernel and initrd get loaded. The initrd loads required
kernel modules, waits for udev to settle, activates luks&lvm, then
finally asks the kernel to resume from the resume device.
The kernel documentation on suspend is VERY clear you should NOT touch
anything on disk between suspend and resume. So activating luks and LVM
is probably risky already, but it apppears both luks and LVM do not make
any on-disk changes when activated and any in-memory state (within the
resumed image) is still valid. The benefit of activating luks and LVM
before resume seems to be that it allows resuming from encrypted/lvm
volumes.
Now, with bcache added, things probably get a bit hairy. NixOS supports
bcache inside the initrd and uses udev rules to activate/attach. I
suspect this is probably unsafe. Probably bcache starts to see if any
dirty pages exist, to write them to the backing store. Even without
writeback caching, the activation of lvm will read some sectors, which
might trigger the cache to update. Then after resuming the image, the
in-memory state is corrupted and further damage occurs.
- Does this sound plausible?
- Is there any way to tell bcache to make absolutely no changes to
either the backing device or the cache?
Basically like a readaround+writearound which can be triggered on
hibernate and switched off on resume.
Thanks,
Mathijs
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bcache and hibernation
2014-11-13 13:52 bcache and hibernation Mathijs Kwik
@ 2014-11-13 15:52 ` Mathijs Kwik
[not found] ` <CAPBO7TZF5qUV64UZJVE+WQkKa2aCJSTjkQxh6eVktH7nA41Vqw@mail.gmail.com>
2014-11-13 22:11 ` Kent Overstreet
1 sibling, 1 reply; 10+ messages in thread
From: Mathijs Kwik @ 2014-11-13 15:52 UTC (permalink / raw)
To: linux-bcache
As a followup question:
Is there a safe way to use hiberation in the presence of bcache?
Assuming a (initrd) boot procedure should at least wait for the
hard-disk to settle, this will mean some udev rules will fire...
With the bcache udev rules in place, this means activating/attaching a
cache+backing device. Am I assuming correctly that this might
immediately lead to on-disk changes (writeback dirty pages for example)?
So what's the way to go?
Make sure the bcache module doesn't get loaded when resuming from
hibernation? Tell udev not to activate the device automatically until we
check for a resume-image?
Mathijs Kwik <mathijs@bluescreen303.nl> writes:
> Hi all,
>
> Today, I lost most my data (don't worry, got backups) after the cache
> got corrupted somehow. I suspected a recent suspend-to-disk to be the
> cause. I checked how my distribution (NixOS) handles suspend/resume and
> I have some concerns about how bcache fits into this.
>
> Normally, the kernel and initrd get loaded. The initrd loads required
> kernel modules, waits for udev to settle, activates luks&lvm, then
> finally asks the kernel to resume from the resume device.
>
> The kernel documentation on suspend is VERY clear you should NOT touch
> anything on disk between suspend and resume. So activating luks and LVM
> is probably risky already, but it apppears both luks and LVM do not make
> any on-disk changes when activated and any in-memory state (within the
> resumed image) is still valid. The benefit of activating luks and LVM
> before resume seems to be that it allows resuming from encrypted/lvm
> volumes.
>
> Now, with bcache added, things probably get a bit hairy. NixOS supports
> bcache inside the initrd and uses udev rules to activate/attach. I
> suspect this is probably unsafe. Probably bcache starts to see if any
> dirty pages exist, to write them to the backing store. Even without
> writeback caching, the activation of lvm will read some sectors, which
> might trigger the cache to update. Then after resuming the image, the
> in-memory state is corrupted and further damage occurs.
>
> - Does this sound plausible?
> - Is there any way to tell bcache to make absolutely no changes to
> either the backing device or the cache?
> Basically like a readaround+writearound which can be triggered on
> hibernate and switched off on resume.
>
> Thanks,
> Mathijs
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: bcache and hibernation
2014-11-13 13:52 bcache and hibernation Mathijs Kwik
2014-11-13 15:52 ` Mathijs Kwik
@ 2014-11-13 22:11 ` Kent Overstreet
2014-11-30 18:25 ` Mathijs Kwik
1 sibling, 1 reply; 10+ messages in thread
From: Kent Overstreet @ 2014-11-13 22:11 UTC (permalink / raw)
To: Mathijs Kwik; +Cc: linux-bcache
On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
> Hi all,
>
> Today, I lost most my data (don't worry, got backups) after the cache
> got corrupted somehow. I suspected a recent suspend-to-disk to be the
> cause. I checked how my distribution (NixOS) handles suspend/resume and
> I have some concerns about how bcache fits into this.
Augh :(
> Normally, the kernel and initrd get loaded. The initrd loads required
> kernel modules, waits for udev to settle, activates luks&lvm, then
> finally asks the kernel to resume from the resume device.
>
> The kernel documentation on suspend is VERY clear you should NOT touch
> anything on disk between suspend and resume. So activating luks and LVM
> is probably risky already, but it apppears both luks and LVM do not make
> any on-disk changes when activated and any in-memory state (within the
> resumed image) is still valid. The benefit of activating luks and LVM
> before resume seems to be that it allows resuming from encrypted/lvm
> volumes.
Yeah, this is handled for in kernel stuff with the freezing mechanism, which
bcache uses.
> Now, with bcache added, things probably get a bit hairy. NixOS supports
> bcache inside the initrd and uses udev rules to activate/attach. I
> suspect this is probably unsafe. Probably bcache starts to see if any
> dirty pages exist, to write them to the backing store. Even without
> writeback caching, the activation of lvm will read some sectors, which
> might trigger the cache to update. Then after resuming the image, the
> in-memory state is corrupted and further damage occurs.
>
> - Does this sound plausible?
> - Is there any way to tell bcache to make absolutely no changes to
> either the backing device or the cache?
> Basically like a readaround+writearound which can be triggered on
> hibernate and switched off on resume.
So, userspace shouldn't have to do anything to tell bcache about hibernation.
The dev branch is getting a true read only mode (still in progress), but this
isn't relevant to hibernation.
bcache kernel threads (allocator thread, gc thread) should be correct w.r.t.
hibernation, but - maybe the workqueue usage isn't.
I'm probably not going to be able to get to this in the next couple days, but
this is a pretty serious issue. Can you ping me again every couple days until I
get a fix out for this, and myabe file a bug somewhere? (i think
bugzilla.kernel.org has been used for bcache bugs before...)
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bcache and hibernation
2014-11-13 22:11 ` Kent Overstreet
@ 2014-11-30 18:25 ` Mathijs Kwik
2014-11-30 23:24 ` Kent Overstreet
2014-11-30 23:29 ` Kent Overstreet
0 siblings, 2 replies; 10+ messages in thread
From: Mathijs Kwik @ 2014-11-30 18:25 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcache
Kent Overstreet <kmo@daterainc.com> writes:
> On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
>> The kernel documentation on suspend is VERY clear you should NOT touch
>> anything on disk between suspend and resume. So activating luks and LVM
>> is probably risky already, but it apppears both luks and LVM do not make
>> any on-disk changes when activated and any in-memory state (within the
>> resumed image) is still valid. The benefit of activating luks and LVM
>> before resume seems to be that it allows resuming from encrypted/lvm
>> volumes.
>
> Yeah, this is handled for in kernel stuff with the freezing mechanism, which
> bcache uses.
>
>>
>
> So, userspace shouldn't have to do anything to tell bcache about
> hibernation.
I understand bcache knows when the system hibernates so it can do some
bookkeeping/flushing. What I don't get, is how this will protect the
system in the short phase before resume, when my initrd activates bcache
and lvm, _before_ it checks for a resume image.
I guess bcache will just start running as usual. Maybe flushing some
dirty buckets, cache some new stuff (when lvm searches for volumes and
udev reads labels & more).
Then finally the resume mechanism loads the old hibernated system, while
the cache has changed in the mean time! Won't this cause issues?
>
> The dev branch is getting a true read only mode (still in progress), but this
> isn't relevant to hibernation.
>
> bcache kernel threads (allocator thread, gc thread) should be correct w.r.t.
> hibernation, but - maybe the workqueue usage isn't.
>
> I'm probably not going to be able to get to this in the next couple days, but
> this is a pretty serious issue. Can you ping me again every couple days until I
> get a fix out for this, and myabe file a bug somewhere? (i think
> bugzilla.kernel.org has been used for bcache bugs before...)
I will file a report once I'm sure what the exact cause for my data-loss
was. If I understand you correctly, all should be safe and well, no
matter what (initramfs) userspace does in between and the only thing
that might not be safe is the workqueue?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bcache and hibernation
2014-11-30 18:25 ` Mathijs Kwik
@ 2014-11-30 23:24 ` Kent Overstreet
2014-11-30 23:29 ` Kent Overstreet
1 sibling, 0 replies; 10+ messages in thread
From: Kent Overstreet @ 2014-11-30 23:24 UTC (permalink / raw)
To: Mathijs Kwik; +Cc: linux-bcache
On Sun, Nov 30, 2014 at 07:25:03PM +0100, Mathijs Kwik wrote:
> Kent Overstreet <kmo@daterainc.com> writes:
>
> > On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
> >> The kernel documentation on suspend is VERY clear you should NOT touch
> >> anything on disk between suspend and resume. So activating luks and LVM
> >> is probably risky already, but it apppears both luks and LVM do not make
> >> any on-disk changes when activated and any in-memory state (within the
> >> resumed image) is still valid. The benefit of activating luks and LVM
> >> before resume seems to be that it allows resuming from encrypted/lvm
> >> volumes.
> >
> > Yeah, this is handled for in kernel stuff with the freezing mechanism, which
> > bcache uses.
> >
> >>
> >
> > So, userspace shouldn't have to do anything to tell bcache about
> > hibernation.
>
> I understand bcache knows when the system hibernates so it can do some
> bookkeeping/flushing. What I don't get, is how this will protect the
> system in the short phase before resume, when my initrd activates bcache
> and lvm, _before_ it checks for a resume image.
Wait what? Ohh... you must be talking about hibernate, not suspend...
> I guess bcache will just start running as usual. Maybe flushing some
> dirty buckets, cache some new stuff (when lvm searches for volumes and
> udev reads labels & more).
>
> Then finally the resume mechanism loads the old hibernated system, while
> the cache has changed in the mean time! Won't this cause issues?
I have no idea how hibernate works, but yeah, quite possibly...
> > The dev branch is getting a true read only mode (still in progress), but this
> > isn't relevant to hibernation.
> >
> > bcache kernel threads (allocator thread, gc thread) should be correct w.r.t.
> > hibernation, but - maybe the workqueue usage isn't.
> >
> > I'm probably not going to be able to get to this in the next couple days, but
> > this is a pretty serious issue. Can you ping me again every couple days until I
> > get a fix out for this, and myabe file a bug somewhere? (i think
> > bugzilla.kernel.org has been used for bcache bugs before...)
>
> I will file a report once I'm sure what the exact cause for my data-loss
> was. If I understand you correctly, all should be safe and well, no
> matter what (initramfs) userspace does in between and the only thing
> that might not be safe is the workqueue?
I was thinking suspend to ram, not hibernate. gonna have to look into how
hibernate works now.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bcache and hibernation
2014-11-30 18:25 ` Mathijs Kwik
2014-11-30 23:24 ` Kent Overstreet
@ 2014-11-30 23:29 ` Kent Overstreet
2014-12-01 8:48 ` Mathijs Kwik
1 sibling, 1 reply; 10+ messages in thread
From: Kent Overstreet @ 2014-11-30 23:29 UTC (permalink / raw)
To: Mathijs Kwik; +Cc: linux-bcache
On Sun, Nov 30, 2014 at 07:25:03PM +0100, Mathijs Kwik wrote:
> Kent Overstreet <kmo@daterainc.com> writes:
>
> > On Thu, Nov 13, 2014 at 02:52:02PM +0100, Mathijs Kwik wrote:
> >> The kernel documentation on suspend is VERY clear you should NOT touch
> >> anything on disk between suspend and resume. So activating luks and LVM
> >> is probably risky already, but it apppears both luks and LVM do not make
> >> any on-disk changes when activated and any in-memory state (within the
> >> resumed image) is still valid. The benefit of activating luks and LVM
> >> before resume seems to be that it allows resuming from encrypted/lvm
> >> volumes.
> >
> > Yeah, this is handled for in kernel stuff with the freezing mechanism, which
> > bcache uses.
> >
> >>
> >
> > So, userspace shouldn't have to do anything to tell bcache about
> > hibernation.
>
> I understand bcache knows when the system hibernates so it can do some
> bookkeeping/flushing. What I don't get, is how this will protect the
> system in the short phase before resume, when my initrd activates bcache
> and lvm, _before_ it checks for a resume image.
>
> I guess bcache will just start running as usual. Maybe flushing some
> dirty buckets, cache some new stuff (when lvm searches for volumes and
> udev reads labels & more).
>
> Then finally the resume mechanism loads the old hibernated system, while
> the cache has changed in the mean time! Won't this cause issues?
BTW - it sounds like you're ahead of me on how this is put together - could you
point me at the userspace side of hibernate that you're using (those initramfs
scripts, and in particular whatever device mapper does)? that'll help a lot.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: bcache and hibernation
2014-11-30 23:29 ` Kent Overstreet
@ 2014-12-01 8:48 ` Mathijs Kwik
0 siblings, 0 replies; 10+ messages in thread
From: Mathijs Kwik @ 2014-12-01 8:48 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcache
Kent Overstreet <kmo@daterainc.com> writes:
>
> BTW - it sounds like you're ahead of me on how this is put together - could you
> point me at the userspace side of hibernate that you're using (those initramfs
> scripts, and in particular whatever device mapper does)? that'll help a lot.
Please see
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/stage-1-init.sh
lines 128 onward. I'm just using the in-kernel suspend/hibernate
functionality (swsusp) but the same probably applies to tuxonice and
other solutions as well.
As far as I understand the events leading up to hibernation are very
similar to suspend. The kernel will notify processes and kernel threads
they will be frozen. Then, when everything has prepared for suspension,
instead of just putting the system to suspend, the kernel will write the
full contents of the system RAM to a swap device.
I'm pretty sure that's all still OK and things are in a consistent
state. However, when resuming the system, some basic initialization is
performed in initrd. At least SCSI/SATA controller modules and other
stuff needed to find the hibernated RAM image are needed of course, but
most distributions will include more stuff and use udev to find hardware
and load the appropriate modules.
This is where things might get nasty though. On nixos, we normally
initialize bcache using udev rules. Modern versions of udev will do a
quick scan of block devices when found to find their labels/types. So
while waiting for stuff like disks/usb/whatever to appear, my bcache
partitions get found and activated. I'm pretty sure bcache will take
over from here and do some bookkeeping / flush dirty buckets, whatever.
Even without writeback, things might change on-disk: udev and tools like
vgscan (lvm) and "btrfs scan" might probe some magic sectors inside the
newly-activated bcache device. If those aren't in the cache, they will
be put there, once again changing the on-disk state.
Then finally (line 190) the kernel gets instructed to check if the swap
device contains a hibernated RAM image and restore that. For everything
running "inside the RAM image", it's just like waking up from a normal
suspend.
From this explanation, it should be clear that it is vital that no
on-disk state is changed in the brief period that the initrd is setting
up the system, or bcache's in-memory state (inside the resumed RAM
image) will be corrupted, probably leading to disasters. Either that, or
bcache should assume nothing on resume and make sure to reassemble its
entire in-memory state from disk.
The temporary solution I found was to not include the udev rules in the
initrd so it will not get found and activated before resume. Then for
normal booting (I have my root on bcache) I manually load and activate
bcache _after_ seeing there is no resume image. However, this solution
is ugly, because I need to repeat all other initialization steps
(lvm/btrfs) afterwards.
Hope this helps,
Mathijs
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-02-10 22:57 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-11-13 13:52 bcache and hibernation Mathijs Kwik
2014-11-13 15:52 ` Mathijs Kwik
[not found] ` <CAPBO7TZF5qUV64UZJVE+WQkKa2aCJSTjkQxh6eVktH7nA41Vqw@mail.gmail.com>
2014-11-13 16:52 ` Mathijs Kwik
[not found] ` <CAPBO7TbQA2MbFS43racKOwZ+=U2jC4OcLF413-MvvNKML5=QZQ@mail.gmail.com>
2014-11-13 17:23 ` Mathijs Kwik
2015-02-10 22:36 ` Kai Krakow
2014-11-13 22:11 ` Kent Overstreet
2014-11-30 18:25 ` Mathijs Kwik
2014-11-30 23:24 ` Kent Overstreet
2014-11-30 23:29 ` Kent Overstreet
2014-12-01 8:48 ` Mathijs Kwik
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).