From: Anthony Liguori <anthony@codemonkey.ws>
To: Avi Kivity <avi@redhat.com>
Cc: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [PATCH] Fix qapi code generation wrt parallel build
Date: Tue, 27 Dec 2011 10:37:19 -0600 [thread overview]
Message-ID: <4EF9F43F.5070106@codemonkey.ws> (raw)
In-Reply-To: <1324994536-3955-1-git-send-email-avi@redhat.com>
On 12/27/2011 08:02 AM, Avi Kivity wrote:
> Make's multiple output syntax
>
> x.c x.h: x.template
> gen< x.template
>
> actually invokes the command once for x.c and once for x.h (with differing $@
> in each invocation). During a parallel build, the two commands may be invoked
> in parallel; this opens up a race, where the second invocation trashes a file
> supposedly produced during the first, and now in use by a dependent command.
>
> The various qapi code generators are susceptible to this; fix by making them
> generate just one file per invocation.
>
> Signed-off-by: Avi Kivity<avi@redhat.com>
Applied. Thanks.
Regards,
Anthony Liguori
> ---
> Makefile | 20 +++++++++++---------
> scripts/qapi-commands.py | 30 +++++++++++++++++++++++++++---
> scripts/qapi-types.py | 27 ++++++++++++++++++++++++---
> scripts/qapi-visit.py | 27 ++++++++++++++++++++++++---
> 4 files changed, 86 insertions(+), 18 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 2c03055..5958289 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -172,35 +172,37 @@ qapi-dir := $(BUILD_DIR)/qapi-generated
> test-qmp-input-visitor.o test-qmp-output-visitor.o test-qmp-commands.o qemu-ga$(EXESUF): QEMU_CFLAGS += -I $(qapi-dir)
> qemu-ga$(EXESUF): LIBS = $(LIBS_QGA)
>
> +gen-out-type = $(subst .,-,$@)
> +
> $(qapi-dir)/test-qapi-types.c $(qapi-dir)/test-qapi-types.h :\
> $(SRC_PATH)/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-types.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py -o "$(qapi-dir)" -p "test-"< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py $(gen-out-type) -o "$(qapi-dir)" -p "test-"< $<, " GEN $@")
> $(qapi-dir)/test-qapi-visit.c $(qapi-dir)/test-qapi-visit.h :\
> $(SRC_PATH)/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-visit.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py -o "$(qapi-dir)" -p "test-"< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py $(gen-out-type) -o "$(qapi-dir)" -p "test-"< $<, " GEN $@")
> $(qapi-dir)/test-qmp-commands.h $(qapi-dir)/test-qmp-marshal.c :\
> $(SRC_PATH)/qapi-schema-test.json $(SRC_PATH)/scripts/qapi-commands.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py -o "$(qapi-dir)" -p "test-"< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py $(gen-out-type) -o "$(qapi-dir)" -p "test-"< $<, " GEN $@")
>
> $(qapi-dir)/qga-qapi-types.c $(qapi-dir)/qga-qapi-types.h :\
> $(SRC_PATH)/qapi-schema-guest.json $(SRC_PATH)/scripts/qapi-types.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py -o "$(qapi-dir)" -p "qga-"< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py $(gen-out-type) -o "$(qapi-dir)" -p "qga-"< $<, " GEN $@")
> $(qapi-dir)/qga-qapi-visit.c $(qapi-dir)/qga-qapi-visit.h :\
> $(SRC_PATH)/qapi-schema-guest.json $(SRC_PATH)/scripts/qapi-visit.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py -o "$(qapi-dir)" -p "qga-"< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py $(gen-out-type) -o "$(qapi-dir)" -p "qga-"< $<, " GEN $@")
> $(qapi-dir)/qga-qmp-commands.h $(qapi-dir)/qga-qmp-marshal.c :\
> $(SRC_PATH)/qapi-schema-guest.json $(SRC_PATH)/scripts/qapi-commands.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py -o "$(qapi-dir)" -p "qga-"< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py $(gen-out-type) -o "$(qapi-dir)" -p "qga-"< $<, " GEN $@")
>
> qapi-types.c qapi-types.h :\
> $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py -o "."< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py $(gen-out-type) -o "."< $<, " GEN $@")
> qapi-visit.c qapi-visit.h :\
> $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py -o "."< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py $(gen-out-type) -o "."< $<, " GEN $@")
> qmp-commands.h qmp-marshal.c :\
> $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py
> - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py -m -o "."< $<, " GEN $@")
> + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py $(gen-out-type) -m -o "."< $<, " GEN $@")
>
> test-qmp-output-visitor.o: $(addprefix $(qapi-dir)/, test-qapi-types.c test-qapi-types.h test-qapi-visit.c test-qapi-visit.h) $(qapi-obj-y)
> test-qmp-output-visitor: test-qmp-output-visitor.o $(qobject-obj-y) $(qapi-obj-y) $(tools-obj-y) $(qapi-dir)/test-qapi-visit.o $(qapi-dir)/test-qapi-types.o
> diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
> index 54d1f5d..bd7b207 100644
> --- a/scripts/qapi-commands.py
> +++ b/scripts/qapi-commands.py
> @@ -372,7 +372,9 @@ def gen_command_def_prologue(prefix="", proxy=False):
>
>
> try:
> - opts, args = getopt.gnu_getopt(sys.argv[1:], "p:o:m", ["prefix=", "output-dir=", "type=", "middle"])
> + opts, args = getopt.gnu_getopt(sys.argv[1:], "chp:o:m",
> + ["source", "header", "prefix=",
> + "output-dir=", "type=", "middle"])
> except getopt.GetoptError, err:
> print str(err)
> sys.exit(1)
> @@ -384,6 +386,9 @@ c_file = 'qmp-marshal.c'
> h_file = 'qmp-commands.h'
> middle_mode = False
>
> +do_c = False
> +do_h = False
> +
> for o, a in opts:
> if o in ("-p", "--prefix"):
> prefix = a
> @@ -393,10 +398,29 @@ for o, a in opts:
> dispatch_type = a
> elif o in ("-m", "--middle"):
> middle_mode = True
> + elif o in ("-c", "--source"):
> + do_h = True
> + elif o in ("-h", "--header"):
> + do_c = True
> +
> +if not do_c and not do_h:
> + do_c = True
> + do_h = True
>
> c_file = output_dir + prefix + c_file
> h_file = output_dir + prefix + h_file
>
> +def maybe_open(really, name, opt):
> + class Null(object):
> + def write(self, str):
> + pass
> + def read(self):
> + return ''
> + if really:
> + return open(name, opt)
> + else:
> + return Null()
> +
> try:
> os.makedirs(output_dir)
> except os.error, e:
> @@ -408,8 +432,8 @@ commands = filter(lambda expr: expr.has_key('command'), exprs)
> commands = filter(lambda expr: not expr.has_key('gen'), commands)
>
> if dispatch_type == "sync":
> - fdecl = open(h_file, 'w')
> - fdef = open(c_file, 'w')
> + fdecl = maybe_open(do_h, h_file, 'w')
> + fdef = maybe_open(do_c, c_file, 'w')
> ret = gen_command_decl_prologue(header=basename(h_file), guard=guardname(h_file), prefix=prefix)
> fdecl.write(ret)
> ret = gen_command_def_prologue(prefix=prefix)
> diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
> index 267cb49..ae644bc 100644
> --- a/scripts/qapi-types.py
> +++ b/scripts/qapi-types.py
> @@ -163,7 +163,8 @@ void qapi_free_%(type)s(%(c_type)s obj)
>
>
> try:
> - opts, args = getopt.gnu_getopt(sys.argv[1:], "p:o:", ["prefix=", "output-dir="])
> + opts, args = getopt.gnu_getopt(sys.argv[1:], "chp:o:",
> + ["source", "header", "prefix=", "output-dir="])
> except getopt.GetoptError, err:
> print str(err)
> sys.exit(1)
> @@ -173,11 +174,22 @@ prefix = ""
> c_file = 'qapi-types.c'
> h_file = 'qapi-types.h'
>
> +do_c = False
> +do_h = False
> +
> for o, a in opts:
> if o in ("-p", "--prefix"):
> prefix = a
> elif o in ("-o", "--output-dir"):
> output_dir = a + "/"
> + elif o in ("-c", "--source"):
> + do_h = True
> + elif o in ("-h", "--header"):
> + do_c = True
> +
> +if not do_c and not do_h:
> + do_c = True
> + do_h = True
>
> c_file = output_dir + prefix + c_file
> h_file = output_dir + prefix + h_file
> @@ -188,8 +200,17 @@ except os.error, e:
> if e.errno != errno.EEXIST:
> raise
>
> -fdef = open(c_file, 'w')
> -fdecl = open(h_file, 'w')
> +def maybe_open(really, name, opt):
> + class Null(object):
> + def write(self, str):
> + pass
> + def read(self):
> + return ''
> + if really:
> + return open(name, opt)
> +
> +fdef = maybe_open(do_c, c_file, 'w')
> +fdecl = maybe_open(do_h, h_file, 'w')
>
> fdef.write(mcgen('''
> /* AUTOMATICALLY GENERATED, DO NOT MODIFY */
> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
> index 62de83d..e9d0584 100644
> --- a/scripts/qapi-visit.py
> +++ b/scripts/qapi-visit.py
> @@ -139,7 +139,8 @@ void visit_type_%(name)s(Visitor *m, %(name)s * obj, const char *name, Error **e
> name=name)
>
> try:
> - opts, args = getopt.gnu_getopt(sys.argv[1:], "p:o:", ["prefix=", "output-dir="])
> + opts, args = getopt.gnu_getopt(sys.argv[1:], "chp:o:",
> + ["source", "header", "prefix=", "output-dir="])
> except getopt.GetoptError, err:
> print str(err)
> sys.exit(1)
> @@ -149,11 +150,22 @@ prefix = ""
> c_file = 'qapi-visit.c'
> h_file = 'qapi-visit.h'
>
> +do_c = False
> +do_h = False
> +
> for o, a in opts:
> if o in ("-p", "--prefix"):
> prefix = a
> elif o in ("-o", "--output-dir"):
> output_dir = a + "/"
> + elif o in ("-c", "--source"):
> + do_h = True
> + elif o in ("-h", "--header"):
> + do_c = True
> +
> +if not do_c and not do_h:
> + do_c = True
> + do_h = True
>
> c_file = output_dir + prefix + c_file
> h_file = output_dir + prefix + h_file
> @@ -164,8 +176,17 @@ except os.error, e:
> if e.errno != errno.EEXIST:
> raise
>
> -fdef = open(c_file, 'w')
> -fdecl = open(h_file, 'w')
> +def maybe_open(really, name, opt):
> + class Null(object):
> + def write(self, str):
> + pass
> + def read(self):
> + return ''
> + if really:
> + return open(name, opt)
> +
> +fdef = maybe_open(do_c, c_file, 'w')
> +fdecl = maybe_open(do_h, h_file, 'w')
>
> fdef.write(mcgen('''
> /* THIS FILE IS AUTOMATICALLY GENERATED, DO NOT MODIFY */
prev parent reply other threads:[~2011-12-27 16:37 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-27 14:02 [Qemu-devel] [PATCH] Fix qapi code generation wrt parallel build Avi Kivity
2011-12-27 16:37 ` Anthony Liguori [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4EF9F43F.5070106@codemonkey.ws \
--to=anthony@codemonkey.ws \
--cc=avi@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.