[PATCH] sigchain: recover from concurrent double-pops of the same signal

[PATCH] sigchain: recover from concurrent double-pops of the same signal

[Ian Hu via GitGitGadget]

From: ian4hu <hu2008yinxiang@163.com>

sigchain_push()/sigchain_pop() update signals[sig].n and the handler
stack non-atomically.  When two copies of the same signal are delivered
in quick succession to different threads, both instances of the handler
run the usual "cleanup; sigchain_pop(sig); raise(sig)" pattern
concurrently, and both read the same s->n before either one decrements
it.  Each then installs s->old[s->n - 1], so a two-entry chain unwinds
both entries while the dispatcher keeps pointing at a handler that was
already popped: n ends at 0 while the effective disposition is still
old[1].

The surviving dispatcher then runs that popped handler, whose own
sigchain_pop() hits the "n < 1" guard and does nothing, and the raise()
at the end redelivers to the same stuck dispatcher forever.  The process
neither dies nor continues; every signal-handling thread spins at 100%
CPU and only SIGKILL recovers it.

This is reachable in practice: "git clone --depth 1" over ssh holds two
SIGTERM entries in the chain at the index-pack parking point (clone's
remove_junk_on_signal, then tempfile's remove_tempfiles_on_signal from
the first lockfile), and two SIGTERMs a fraction of a millisecond apart
land on two threads and corrupt the chain.

Rather than serializing the chain mutation, teach the "n < 1" guard in
sigchain_pop() to repair the damage.  Track how many pushes were ever
performed for the signal; when a pop finds the chain empty but the
chain bottom is known, reinstall s->old[0], the disposition that was
in place before the chain was built.  The stuck handler's raise() then
lands on that disposition -- typically SIG_DFL -- and the process dies
by it, exactly as a single delivery would have.

The repair is inert on every legitimate path to the guard: if the
signal was never pushed, nothing is installed; if the chain was fully
unwound by regular pops, s->old[0] is already installed and
reinstalling it changes nothing.  The push/pop fast path is untouched.

A fault-injected A/B run of the minimal reproducer, compiled from
identical sources with and without this change (the gap between the
"read n" and "decrement n" steps of pop widened so both racing pops
reliably read the same value; 300 interleaved pairs per arm) froze the
unpatched arm 121 times out of 300.  The patched arm never froze: all
300 runs reinstated the pre-chain disposition and died by it, and
20/20 single-delivery control runs never reached the repair code.

Signed-off-by: ian4hu <hu2008yinxiang@163.com>
---
    sigchain: recover from concurrent double-pops of the same signal
    
    During git clone --depth 1 over ssh (or any transport that parks the
    client with an in-flight packfile), two SIGTERMs delivered a fraction of
    a millisecond apart can corrupt the sigchain unwind: both fetch threads
    end up looping inside remove_junk_on_signal() at ~200% CPU, the process
    survives both signals, and further SIGTERMs have no effect (only SIGKILL
    recovers it).
    
    Mechanism (git 2.56.0):
    
     1. clone pushes two SIGTERM handlers: remove_junk_on_signal
        (builtin/clone.c:1134), then on the first tempfile activation (e.g.
        shallow.lock) remove_tempfiles_on_signal (tempfile.c:117):
        old[0]=SIG_DFL, old[1]=remove_junk_on_signal, n=2.
     2. Two deliveries that land on two threads run the top handler
        concurrently; both sigchain_pop() calls read the same s->n and both
        install s->old[s->n - 1]: the chain unwinds two levels for two
        deliveries, ending at n=0 while the dispatcher still points at
        old[1].
     3. The surviving dispatcher then chains to a handler that was already
        popped, whose own sigchain_pop() hits the n < 1 guard (a silent
        no-op), and whose raise() redelivers to the same stuck dispatcher
        forever.
    
    This patch takes the "repair after the fact" route instead of
    serializing the chain mutation: sigchain_pop() remembers how many pushes
    were ever performed for the signal, and when it finds the chain empty
    while the chain bottom is known, it reinstalls s->old[0] -- the
    disposition that was in place before the chain was built. The stuck
    handler's raise() then lands on that disposition (typically SIG_DFL) and
    the process dies by it, exactly as a single delivery would have.
    
    The repair is inert on every legitimate path to the n < 1 guard: if the
    signal was never pushed nothing is installed, and if the chain was fully
    unwound by regular pops, s->old[0] is already installed so reinstalling
    it changes nothing. The push/pop fast path is untouched -- no locking,
    no signal-mask manipulation, no Windows-specific code.
    
    Validation: a minimal reproducer (two threads parked in read(), the same
    push/pop chain, first handler entry doing real close()+unlink() work),
    compiled from identical sources with and without this change. With a
    fault injector widening the gap between the "read n" and "decrement n"
    steps of pop so both racing pops reliably read the same value (300
    interleaved pairs per arm), the unpatched arm froze 121/300 runs while
    the patched arm never froze: all 300 runs reinstated the pre-chain
    disposition and died by it. 20/20 single-delivery control runs never
    reached the repair code.
    
    An alternative fix that serializes sigchain_push()/sigchain_pop() with a
    per-signal spinlock exists as well; both eliminate the freeze. The lock
    prevents the corruption, this patch heals it; they are independent and
    either one suffices.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2263%2Fian4hu%2Fsigchain-recover-from-concurrent-pops-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2263/ian4hu/sigchain-recover-from-concurrent-pops-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2263

 sigchain.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/sigchain.c b/sigchain.c
index 66123bdbab..f6da68ce4d 100644
--- a/sigchain.c
+++ b/sigchain.c
@@ -7,6 +7,7 @@ struct sigchain_signal {
 	sigchain_fun *old;
 	int n;
 	int alloc;
+	int pushed;
 };
 static struct sigchain_signal signals[SIGCHAIN_MAX_SIGNALS];
 
@@ -26,6 +27,7 @@ int sigchain_push(int sig, sigchain_fun f)
 	if (s->old[s->n] == SIG_ERR)
 		return -1;
 	s->n++;
+	s->pushed++;
 	return 0;
 }
 
@@ -33,8 +35,21 @@ int sigchain_pop(int sig)
 {
 	struct sigchain_signal *s = signals + sig;
 	check_signum(sig);
-	if (s->n < 1)
+	if (s->n < 1) {
+		/*
+		 * We are unwinding a chain that is already empty.  If we
+		 * got here from a signal handler, two concurrent pops of
+		 * the same signal both decremented "n", leaving the
+		 * dispatcher stuck on a handler that no longer has a
+		 * chain entry: its raise() would re-enter it forever.
+		 * Reinstall the disposition that was in place before the
+		 * chain was built so the raise() that follows lands on
+		 * that instead.
+		 */
+		if (s->pushed)
+			signal(sig, s->old[0]);
 		return 0;
+	}
 
 	if (signal(sig, s->old[s->n - 1]) == SIG_ERR)
 		return -1;

base-commit: f57c9a73884400380bddd172acc88b569834f909
-- 
gitgitgadget