[PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early

[PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early

[Jonathan Nieder]

It can be tedious to wait for a multi-million-revision import.
Unfortunately it is hard to spy on the import because fast-import
works by continuously streaming out objects, without updating the pack
index or refs until a checkpoint command or the end of the stream.

So allow the impatient operator to request checkpoints by sending a
signal, like so:

	killall -USR1 git-fast-import

When receiving such a signal, fast-import would schedule a checkpoint
to take place after the current top-level command (usually a "commit"
or "blob" request) finishes.

Caveats: just like ordinary checkpoint commands, such requests slow
down the import.  Switching to a new pack at a suboptimal moment is
also likely to result in a less dense initial collection of packs.
That's the price.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Thoughts?

 Documentation/git-fast-import.txt |    8 +++++++
 fast-import.c                     |   42 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 49 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 5d0c245..526670a 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -1233,6 +1233,14 @@ and lazy loading of subtrees, allows fast-import to efficiently import
 projects with 2,000+ branches and 45,114+ files in a very limited
 memory footprint (less than 2.7 MiB per active branch).
 
+Signals
+-------
+Sending *SIGUSR1* to the 'git fast-import' process ends the current
+packfile early, simulating a `checkpoint` command.  The impatient
+operator can use this facility to peek at the objects and refs from an
+import in progress, at the cost of some added running time and worse
+compression.
+
 
 Author
 ------
diff --git a/fast-import.c b/fast-import.c
index 77549eb..4779e3a 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -361,6 +361,9 @@ static uintmax_t next_mark;
 static struct strbuf new_data = STRBUF_INIT;
 static int seen_data_command;
 
+/* Signal handling */
+static volatile sig_atomic_t checkpoint_requested;
+
 static void parse_argv(void);
 
 static void write_branch_report(FILE *rpt, struct branch *b)
@@ -500,6 +503,33 @@ static NORETURN void die_nicely(const char *err, va_list params)
 	exit(128);
 }
 
+#ifndef SIGUSR1	/* Windows, for example */
+
+static void set_checkpoint_signal(void)
+{
+	/* nothing */
+}
+
+#else
+
+static void checkpoint_signal(int signo)
+{
+	checkpoint_requested = 1;
+}
+
+static void set_checkpoint_signal(void)
+{
+	struct sigaction sa;
+
+	memset(&sa, 0, sizeof(sa));
+	sa.sa_handler = checkpoint_signal;
+	sigemptyset(&sa.sa_mask);
+	sa.sa_flags = SA_RESTART;
+	sigaction(SIGUSR1, &sa, NULL);
+}
+
+#endif
+
 static void alloc_objects(unsigned int cnt)
 {
 	struct object_entry_pool *b;
@@ -2689,14 +2719,20 @@ static void parse_reset_branch(void)
 		unread_command_buf = 1;
 }
 
-static void parse_checkpoint(void)
+static void do_checkpoint(void)
 {
+	checkpoint_requested = 0;
 	if (object_count) {
 		cycle_packfile();
 		dump_branches();
 		dump_tags();
 		dump_marks();
 	}
+}
+
+static void parse_checkpoint(void)
+{
+	do_checkpoint();
 	skip_optional_lf();
 }
 
@@ -2953,6 +2989,7 @@ int main(int argc, const char **argv)
 	prepare_packed_git();
 	start_packfile();
 	set_die_routine(die_nicely);
+	set_checkpoint_signal();
 	while (read_next_command() != EOF) {
 		if (!strcmp("blob", command_buf.buf))
 			parse_new_blob();
@@ -2974,6 +3011,9 @@ int main(int argc, const char **argv)
 			/* ignore non-git options*/;
 		else
 			die("Unsupported command: %s", command_buf.buf);
+
+		if (checkpoint_requested)
+			do_checkpoint();
 	}
 
 	/* argv hasn't been parsed yet, do so */
-- 
1.7.2.3

Re: [PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> +static void parse_checkpoint(void)
> +{
> +	do_checkpoint();
>  	skip_optional_lf();
>  }

I would have expected the new implementation of parse_checkpoint() to just
set the checkpoint_requested variable to true, so that the checkpoint
always happens in the main loop, which somehow feels cleaner.  But that is
a minor point.

Re: [PATCH/RFC] fast-import: treat SIGUSR1 as a request to access objects early

[Jonathan Nieder]

Junio C Hamano wrote:

> I would have expected the new implementation of parse_checkpoint() to just
> set the checkpoint_requested variable to true, so that the checkpoint
> always happens in the main loop, which somehow feels cleaner.

Nice.  Here's that plus some minor cleanups I had in my tree.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 6be3938..5937cc0 100644
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -1119,7 +1119,6 @@ operator can use this facility to peek at the objects and refs from an
 import in progress, at the cost of some added running time and worse
 compression.
 
-
 Author
 ------
 Written by Shawn O. Pearce <spearce@spearce.org>.
diff --git a/fast-import.c b/fast-import.c
index e8f08fb..a348994 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -490,7 +490,6 @@ static NORETURN void die_nicely(const char *err, va_list params)
 
 static void set_checkpoint_signal(void)
 {
-	/* nothing */
 }
 
 #else
@@ -2351,7 +2350,7 @@ static void parse_reset_branch(void)
 		unread_command_buf = 1;
 }
 
-static void do_checkpoint(void)
+static void checkpoint(void)
 {
 	checkpoint_requested = 0;
 	if (object_count) {
@@ -2364,7 +2363,7 @@ static void do_checkpoint(void)
 
 static void parse_checkpoint(void)
 {
-	do_checkpoint();
+	checkpoint_requested = 1;
 	skip_optional_lf();
 }
 
@@ -2524,7 +2523,7 @@ int main(int argc, const char **argv)
 			die("Unsupported command: %s", command_buf.buf);
 
 		if (checkpoint_requested)
-			do_checkpoint();
+			checkpoint();
 	}
 	end_packfile();