Re: [REGRESSION] Hang during backup with rsync

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7444 bytes --]

On 2015-04-30 21:48, Duncan wrote:
> Martin Steigerwald posted on Thu, 30 Apr 2015 19:29:57 +0200 as excerpted:
>
>> The hang was: Mouse pointer in KDE not movable anymore, Ctrl-Alt-F1 had
>> no effect. I waited for a minute at least. Maybe it would have reacted
>> after a longer time, but I wanted my machine back. Disks where idle, if
>> I remember correctly. After reboot both filesystems mount okay.
>
> This response is in regard to what to do at an apparent hang, and has
> nothing directly to do with btrfs...
>
> Two comments:
>
> 1) Depending on your graphics hardware and driver config, a modern
> "KMS" (kernel modesetting) setup is more likely to "soft" hang in X mode
> and not switch back to text mode, even when the system is otherwise not
> hung and a VT switch would have worked fine pre-KMS-era.
>
> While I'm no kernel or graphics expert, the problem from here /seems/ to
> be that a modern KMS kernel generally uses high-res framebuffer mode at
> the CLI as well, and because the basic kernel handling is unified
> framebuffer and kernel-mode-switching for both X and CLI modes, switching
> from X to CLI doesn't involve switching to the entirely separate VGA mode
> driver and with it the forced hardware reset that it used to.  Without
> that driver switch and forced reset, even if the switch actually occurs
> successfully in terms of what you might type, what is actually displayed
> may remain frozen, such that if you only have a local session, you
> generally have to reboot anyway, but if you already have a CLI login
> going in the VT you tried to switch to or can login blind, sometimes you
> can at least manage a controlled reboot, by doing an init 6 or systemctl
> reboot or whatever, even if the display is frozen and shows nothing.  Of
> course it doesn't always work, but given the chance to avoid an unclean
> shutdown, try it and see.
>
> So no response at an attempted VT switch (your ctrl-alt-F1) doesn't mean
> what it used to...
>
Something else to try in this case is Ctrl-Alt-Backspace, (Most linux 
distros have that configured to outright kill X running on the current 
VT) followed by Ctrl-Alt-Delete, which defaults on all modern distros to 
be equivalent to running 'shutdown -r now' from a root shell.
Also, you may try Alt-SysRq-V, which is supposed to 'restore the 
framebuffer console' (except on ARM systems, where it is used to dump 
the contents of hardware tracing modules).
> 2) Along the same lines, there's the kernel's magic-sysrequest (sysrq/srq)
> functionality.  Assuming you have it enabled in your kernel, you can try
> a series of alt-sysrq-key sequences and very possibly use that to avoid
> an entirely uncontrolled shutdown, even when major functionality upto and
> including all of userspace is non-functional.
>
> There's enough explanations written and googlable on the subject that
> I'll avoid a full explanation here, but the main point I have to make is
> that in addition to often allowing a semi-controlled shutdown/reboot, by
> using the keys in the prescribed sequence and noting at which point (if
> any) you actually get a response, you get at least some indication of how
> badly your system was actually locked up.
So, this is great advice in theory, except that a large majority of 
distributions targeted at enterprise level usage (Fedora, Ubuntu, RHEL, 
SuSE, CentOS, etc) have this functionality disabled at runtime by 
default 'for security' (which is BS, because if an attacker has the kind 
of access required to use SysRq, then he has sufficient access to be 
able to bring the system to it's knees through other methods as well).
> What I'd try first, right after the VT switch didn't work, is alt-srq-k.
> Called the secure-term sequence as it can be used to help avoid suspected
> keyloggers of certain (but not all) types, this tells the kernel to force-
> kill anything running on your current VT and reset it.  This can be used
> to kill an unresponsive X, for instance, and normally you'll get
> automatically switched to a CLI login, either due to automatic switching
> back to a previous VT (in the case of X on its own VT), or to automatic
> respawning of the login after the kernel kills it along with whatever
> else you were doing if you were already at the CLI.
>
> This alt-srq-k sequence is thus a good first fallback if ctrl-alt-Fx
> appears to do nothing, since it apparently forces the VT reset that
> switching to a VGAmode CLI used to, that switching to a KMS mode CLI
> doesn't.
>
> If that doesn't work, it's time for the usual REISUB sequence,
>
> * alt-srq-r (unraw the input, take out of X mode)
>
> * alt-srq-e (tErminate, aka SIGTERM, all of userspace, allowing anything
> still alive to terminate gracefully if it can)
>
> * alt-srq-i (kIll, aka SIGKILL, all userspace, forcefully killing
> anything that ignored the SIGTERM but still allowing the kernel to do
> normal cleanup if it can)
>
> (Tho from my own experience, if the K and R sequences don't help, then
> the E and I sequences aren't likely to do much either, as they're
> probably locked up bad enough that nothing will be gained, but OTOH,
> nothing is lost by trying them, either.)
>
I've found that sometimes Alt-SysRq-J is needed at this point in the 
sequence to get things to correctly write data out (it resumes I/O to 
filesystems that have been frozen with the fsfreeze command or ioctl), 
and it has no negative impact if not needed, so it's generally a good 
idea to just use it anyway.
> * alt-srq-s (Sync, force an emergency sync to storage of anything still
> write-cached)
>
> alt-srq-s can be used at any time, without disrupting normal operation
> except for any I/O triggered by the forced sync.  I've come to use it
> regularly immediately before I do anything that I think /might/ trigger
> system instability, so everything's synced before I try it, just in
> case.  Think of this as a forced version of the sync command.
>
> * alt-srq-u (remoUnt read-only, forcing all still functional filesystems
> read-only)
>
> The S and U steps are critical to a semi-controlled shutdown, and where
> they work, can often mean the difference between a filesystem with no
> errors on reboot as the kernel saved and cleanly mounted read-only to the
> extent it could, and various filesystem corruptions, if these steps
> weren't done or if the kernel was badly enough corrupted it was afraid to
> write anything lest it make the problem worse.
>
> * alt-srq-b (reBoot, force a reboot without any further cleanup).
>

Secondarily, for the sake of completeness, you can also use Alt-SysRq-o 
in place of Alt-SysRq-b to (try) to get the system to power off instead 
of reboot.  Also, it's significant to note that the exact keys used vary 
depending on the keymap loaded in the kernel, on a Dvorak keyboard for 
example, the sequence is instead P.COGX (which IIRC is the same sequence 
of scancodes as on QWERTY keyboards).  Furthermore, if you were running 
under X, you may need to add the Ctrl key to the combination to get it 
properly acknowledged.

It's worth mentioning also that many laptop keyboards (and some other 
modern keyboards) need you to use Ctrl-Alt-PrintScreen (or even some 
other odd key combination, for example the Dell laptop that I'm typing 
this on needs Fn-Alt-Home) instead of Alt-SysRq



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]