[Qemu-devel] [PATCH 00/19 v3] target-i386: CPU hot-add with cpu-add QMP command

[Qemu-devel] [PATCH 00/19 v3] target-i386: CPU hot-add with cpu-add QMP command

[Igor Mammedov]

Implements alternative way for hot-adding CPU using cpu-add QMP command,
wich could be useful until it would be possible to add CPUs via device_add.

All patches except the last are also applicable to device_add aprroach.

To hot-add CPU use following command from qmp-shell:
 cpu-add id=[APIC ID]

Available APIC IDs could be looked up at following QOM path:
/machine/icc-bridge/cpu[0..N]
where number in brackets is APIC ID
Usage examples and limitations on proposed cpu-add implementation
will be documented at http://wiki.qemu.org/Features/CPUHotplug

git tree for testing: https://github.com/imammedo/qemu/tree/cpu_add.v3

based on qom-cpu tree

v2->v3:
  * use local error & propagate_error() instead of operating on
    passed in errp in several places
  * replace CPUClass.get_firmware_id() with CPUClass.get_arch_id()
  * leave IOAPIC creation to board and just set bus to icc-bus
  * include kvm-stub.o in cpu libary if no KVM is configured
  * create resume_vcpu() stub and include it in libqemustub,
    and use it directly instead of CPU method
  * acpi_piix4: s/cpu_add_notifier/cpu_added_notifier/

v1->v2:
  * generalize cpu sync to KVM, resume and hot-plug notification and
    invoke them form CPUClass, to make available to all targets.
  * introduce cpu_exists() and CPUClass.get_firmware_id() and use
    the last one in acpi_piix to make code target independent.
  * move IOAPIC to ICC bus, it was suggested and easy to convert.
  * leave kvmvapic as SysBusDevice, it doesn't affect hot-plug and
    created only once for all APIC instances. I haven't found yet
    good/clean enough way to convert it to ICCDevice. May be follow-up
    though.
  * split one big ICC patch into several, one per converted device
  * add cpu_hot_add hook to machine and implement it for target-i386,
    instead of adding stabs. Could be used by other targets to
    implement cpu-add.
  * pre-allocate links<CPU> for all possible CPUs and make them available
    at /machine/icc-bridge/cpu[0..N] QOM path, so users could find out
    possible/free CPU IDs to use in cpu-add command.

CC: pbonzini@redhat.com
CC: afaerber@suse.de
CC: ehabkost@redhat.com

Igor Mammedov (19):
  target-i386: split out CPU creation and features parsing into
    cpu_x86_create()
  cpu: Pass CPUState to *cpu_synchronize_post*()
  cpu: make kvm-stub.o a part of CPU library
  cpu: call cpu_synchronize_post_init() from CPUClass.realize() if
    hotplugged
  cpu: resume CPU from CPUClass.cpu_common_realizefn() when it is
    hot-plugged
  introduce CPU hot-plug notifier
  rtc: update rtc_cmos on CPU hot-plug
  cpu: introduce get_arch_id() method and override it for target-i386
  cpu: add helper cpu_exists(), to check if CPU with specified id exists
  acpi_piix4: add infrastructure to send CPU hot-plug GPE to guest
  target-i386: introduce apic-id property
  introduce ICC bus/device/bridge
  target-i386: cpu: attach ICC bus to CPU on its creation
  target-i386: replace MSI_SPACE_SIZE with APIC_SPACE_SIZE
  target-i386: move APIC to ICC bus
  target-i386: move IOAPIC to ICC bus
  qdev: set device's parent before calling realize() down inheritance
    chain.
  target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N]
    links
  add cpu-add qmp command and implement CPU hot-add for target-i386

 Makefile.target                    |  13 ++--
 cpus.c                             |  15 ++--
 default-configs/i386-softmmu.mak   |   1 +
 default-configs/x86_64-softmmu.mak |   1 +
 hw/acpi/piix4.c                    | 114 +++++++++++++++++++++++++++-
 hw/core/qdev.c                     |   8 +-
 hw/cpu/Makefile.objs               |   1 +
 hw/cpu/icc_bus.c                   | 150 +++++++++++++++++++++++++++++++++++++
 hw/i386/kvm/apic.c                 |   2 +-
 hw/i386/kvm/ioapic.c               |   2 +-
 hw/i386/kvmvapic.c                 |   1 +
 hw/i386/pc.c                       |  74 ++++++++++++++++--
 hw/i386/pc_piix.c                  |   7 ++
 hw/i386/pc_q35.c                   |   7 ++
 hw/intc/apic.c                     |   2 +-
 hw/intc/apic_common.c              |  17 +++--
 hw/intc/ioapic_common.c            |  15 +++-
 hw/timer/mc146818rtc.c             |  12 +++
 hw/xen/xen_apic.c                  |   2 +-
 include/exec/memory.h              |  10 +++
 include/hw/boards.h                |   3 +
 include/hw/i386/apic_internal.h    |   8 +-
 include/hw/i386/icc_bus.h          |  55 ++++++++++++++
 include/hw/i386/ioapic_internal.h  |   6 +-
 include/hw/pci/msi.h               |   2 +
 include/qom/cpu.h                  |  12 +++
 include/sysemu/cpus.h              |   3 +
 include/sysemu/kvm.h               |  30 ++++----
 include/sysemu/sysemu.h            |   3 +
 kvm-all.c                          |   9 +--
 kvm-stub.c                         |   7 +-
 memory.c                           |  11 +++
 qapi-schema.json                   |  11 +++
 qmp-commands.hx                    |  23 ++++++
 qmp.c                              |  10 +++
 qom/cpu.c                          |  45 +++++++++++
 stubs/Makefile.objs                |   1 +
 stubs/resume_vcpu.c                |   6 ++
 target-i386/cpu.c                  |  93 +++++++++++++++++++----
 target-i386/cpu.h                  |   2 +
 vl.c                               |   7 +-
 41 files changed, 717 insertions(+), 84 deletions(-)
 create mode 100644 hw/cpu/icc_bus.c
 create mode 100644 include/hw/i386/icc_bus.h
 create mode 100644 stubs/resume_vcpu.c

-- 
1.8.2

[Qemu-devel] [PATCH 01/19] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Igor Mammedov]

Move CPU creation and features parsing into a separate cpu_x86_create()
function, so that board would be able to set board specific CPU
properties before CPU is realized.

Keep cpu_x86_init() for compatibility with the code that uses cpu_init()
and doesn't need to modify CPU properties.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * use error_propagate() to rise error up stack
---
 target-i386/cpu.c | 18 +++++++++++++++---
 target-i386/cpu.h |  1 +
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index f1ccc72..c28436c 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1564,7 +1564,7 @@ static void cpu_x86_register(X86CPU *cpu, const char *name, Error **errp)
     object_property_set_str(OBJECT(cpu), def->model_id, "model-id", errp);
 }
 
-X86CPU *cpu_x86_init(const char *cpu_model)
+X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
 {
     X86CPU *cpu = NULL;
     CPUX86State *env;
@@ -1594,13 +1594,25 @@ X86CPU *cpu_x86_init(const char *cpu_model)
         goto out;
     }
 
-    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
+out:
+    error_propagate(errp, error);
+    g_strfreev(model_pieces);
+    return cpu;
+}
+
+X86CPU *cpu_x86_init(const char *cpu_model)
+{
+    Error *error = NULL;
+    X86CPU *cpu;
+
+    cpu = cpu_x86_create(cpu_model, &error);
     if (error) {
         goto out;
     }
 
+    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
+
 out:
-    g_strfreev(model_pieces);
     if (error) {
         fprintf(stderr, "%s\n", error_get_pretty(error));
         error_free(error);
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index 2b4e319..cf1b05c 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -896,6 +896,7 @@ typedef struct CPUX86State {
 #include "cpu-qom.h"
 
 X86CPU *cpu_x86_init(const char *cpu_model);
+X86CPU *cpu_x86_create(const char *cpu_model, Error **errp);
 int cpu_x86_exec(CPUX86State *s);
 void x86_cpu_list(FILE *f, fprintf_function cpu_fprintf);
 void x86_cpudef_setup(void);
-- 
1.8.2

[Qemu-devel] [PATCH 02/19] cpu: Pass CPUState to *cpu_synchronize_post*()

[Igor Mammedov]

... so it could be called from without requiring CPUArchState

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
---
 cpus.c               |  4 ++--
 include/sysemu/kvm.h | 12 ++++++------
 kvm-all.c            |  8 ++------
 kvm-stub.c           |  4 ++--
 4 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/cpus.c b/cpus.c
index e919dd7..9b9a32f 100644
--- a/cpus.c
+++ b/cpus.c
@@ -419,7 +419,7 @@ void cpu_synchronize_all_post_reset(void)
     CPUArchState *cpu;
 
     for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
-        cpu_synchronize_post_reset(cpu);
+        cpu_synchronize_post_reset(ENV_GET_CPU(cpu));
     }
 }
 
@@ -428,7 +428,7 @@ void cpu_synchronize_all_post_init(void)
     CPUArchState *cpu;
 
     for (cpu = first_cpu; cpu; cpu = cpu->next_cpu) {
-        cpu_synchronize_post_init(cpu);
+        cpu_synchronize_post_init(ENV_GET_CPU(cpu));
     }
 }
 
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index f2d97b5..495e6f8 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -250,8 +250,8 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
 uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
                                       uint32_t index, int reg);
 void kvm_cpu_synchronize_state(CPUArchState *env);
-void kvm_cpu_synchronize_post_reset(CPUArchState *env);
-void kvm_cpu_synchronize_post_init(CPUArchState *env);
+void kvm_cpu_synchronize_post_reset(CPUState *cpu);
+void kvm_cpu_synchronize_post_init(CPUState *cpu);
 
 /* generic hooks - to be moved/refactored once there are more users */
 
@@ -262,17 +262,17 @@ static inline void cpu_synchronize_state(CPUArchState *env)
     }
 }
 
-static inline void cpu_synchronize_post_reset(CPUArchState *env)
+static inline void cpu_synchronize_post_reset(CPUState *cpu)
 {
     if (kvm_enabled()) {
-        kvm_cpu_synchronize_post_reset(env);
+        kvm_cpu_synchronize_post_reset(cpu);
     }
 }
 
-static inline void cpu_synchronize_post_init(CPUArchState *env)
+static inline void cpu_synchronize_post_init(CPUState *cpu)
 {
     if (kvm_enabled()) {
-        kvm_cpu_synchronize_post_init(env);
+        kvm_cpu_synchronize_post_init(cpu);
     }
 }
 
diff --git a/kvm-all.c b/kvm-all.c
index 9b433d3..fc4e17c 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -1510,18 +1510,14 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
     }
 }
 
-void kvm_cpu_synchronize_post_reset(CPUArchState *env)
+void kvm_cpu_synchronize_post_reset(CPUState *cpu)
 {
-    CPUState *cpu = ENV_GET_CPU(env);
-
     kvm_arch_put_registers(cpu, KVM_PUT_RESET_STATE);
     cpu->kvm_vcpu_dirty = false;
 }
 
-void kvm_cpu_synchronize_post_init(CPUArchState *env)
+void kvm_cpu_synchronize_post_init(CPUState *cpu)
 {
-    CPUState *cpu = ENV_GET_CPU(env);
-
     kvm_arch_put_registers(cpu, KVM_PUT_FULL_STATE);
     cpu->kvm_vcpu_dirty = false;
 }
diff --git a/kvm-stub.c b/kvm-stub.c
index 760aadc..82875dd 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -42,11 +42,11 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
 {
 }
 
-void kvm_cpu_synchronize_post_reset(CPUArchState *env)
+void kvm_cpu_synchronize_post_reset(CPUState *env)
 {
 }
 
-void kvm_cpu_synchronize_post_init(CPUArchState *env)
+void kvm_cpu_synchronize_post_init(CPUState *cpu)
 {
 }
 
-- 
1.8.2

[Qemu-devel] [PATCH 03/19] cpu: make kvm-stub.o a part of CPU library

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 Makefile.target      | 13 +++++++------
 include/hw/pci/msi.h |  2 ++
 include/sysemu/kvm.h |  4 ++--
 kvm-stub.c           |  2 ++
 4 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/Makefile.target b/Makefile.target
index 2bd6d14..18f348c 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -64,6 +64,12 @@ all: $(PROGS) stap
 # Dummy command so that make thinks it has done something
 	@true
 
+CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
+CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
+CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
+CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
+CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
+
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
@@ -74,6 +80,7 @@ obj-y += fpu/softfloat.o
 obj-y += target-$(TARGET_BASE_ARCH)/
 obj-y += disas.o
 obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
+obj-$(CONFIG_NO_KVM) += kvm-stub.o
 
 #########################################################
 # Linux user emulator target
@@ -102,12 +109,6 @@ endif #CONFIG_BSD_USER
 #########################################################
 # System emulator target
 ifdef CONFIG_SOFTMMU
-CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
-CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
-CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
-CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
-CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
-
 obj-y += arch_init.o cpus.o monitor.o gdbstub.o balloon.o ioport.o
 obj-y += qtest.o
 obj-y += hw/
diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
index 81a3848..d4d0450 100644
--- a/include/hw/pci/msi.h
+++ b/include/hw/pci/msi.h
@@ -21,6 +21,7 @@
 #ifndef QEMU_MSI_H
 #define QEMU_MSI_H
 
+#ifndef CONFIG_USER_ONLY
 #include "qemu-common.h"
 #include "hw/pci/pci.h"
 
@@ -47,4 +48,5 @@ static inline bool msi_present(const PCIDevice *dev)
     return dev->cap_present & QEMU_PCI_CAP_MSI;
 }
 
+#endif
 #endif /* QEMU_MSI_H */
diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 495e6f8..93cef28 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
 #if !defined(CONFIG_USER_ONLY)
 void *kvm_vmalloc(ram_addr_t size);
 void *kvm_arch_vmalloc(ram_addr_t size);
-void kvm_setup_guest_memory(void *start, size_t size);
+#endif
 
+void kvm_setup_guest_memory(void *start, size_t size);
 void kvm_flush_coalesced_mmio_buffer(void);
-#endif
 
 int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
                           target_ulong len, int type);
diff --git a/kvm-stub.c b/kvm-stub.c
index 82875dd..b34064a 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -122,6 +122,7 @@ int kvm_on_sigbus(int code, void *addr)
     return 1;
 }
 
+#ifndef CONFIG_USER_ONLY
 int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
 {
     return -ENOSYS;
@@ -145,3 +146,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
 {
     return -ENOSYS;
 }
+#endif
-- 
1.8.2

[Qemu-devel] [PATCH 04/19] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Igor Mammedov]

... to synchronize CPU state to KVM

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * linking kvm-stub.o to *-user target moved in separate patch
---
 include/sysemu/kvm.h | 18 ++++++++++--------
 kvm-all.c            |  1 +
 kvm-stub.c           |  1 +
 qom/cpu.c            |  4 ++++
 vl.c                 |  1 -
 5 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
index 93cef28..8fb2489 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
 uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
                                       uint32_t index, int reg);
 void kvm_cpu_synchronize_state(CPUArchState *env);
-void kvm_cpu_synchronize_post_reset(CPUState *cpu);
-void kvm_cpu_synchronize_post_init(CPUState *cpu);
 
 /* generic hooks - to be moved/refactored once there are more users */
 
@@ -262,6 +260,16 @@ static inline void cpu_synchronize_state(CPUArchState *env)
     }
 }
 
+#if !defined(CONFIG_USER_ONLY)
+int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
+                                       hwaddr *phys_addr);
+#endif
+
+#endif /* NEED_CPU_H */
+
+void kvm_cpu_synchronize_post_reset(CPUState *cpu);
+void kvm_cpu_synchronize_post_init(CPUState *cpu);
+
 static inline void cpu_synchronize_post_reset(CPUState *cpu)
 {
     if (kvm_enabled()) {
@@ -277,12 +285,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
 }
 
 
-#if !defined(CONFIG_USER_ONLY)
-int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
-                                       hwaddr *phys_addr);
-#endif
-
-#endif
 int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool assign,
                            uint32_t size);
 
diff --git a/kvm-all.c b/kvm-all.c
index fc4e17c..1d17128 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
 bool kvm_irqfds_allowed;
 bool kvm_msi_via_irqfd_allowed;
 bool kvm_gsi_routing_allowed;
+bool kvm_allowed;
 
 static const KVMCapabilityInfo kvm_required_capabilites[] = {
     KVM_CAP_INFO(USER_MEMORY),
diff --git a/kvm-stub.c b/kvm-stub.c
index b34064a..8cb81c4 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -23,6 +23,7 @@ bool kvm_async_interrupts_allowed;
 bool kvm_irqfds_allowed;
 bool kvm_msi_via_irqfd_allowed;
 bool kvm_gsi_routing_allowed;
+bool kvm_allowed;
 
 int kvm_init_vcpu(CPUState *cpu)
 {
diff --git a/qom/cpu.c b/qom/cpu.c
index e242dcb..0c76712 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -20,6 +20,7 @@
 
 #include "qom/cpu.h"
 #include "qemu-common.h"
+#include "sysemu/kvm.h"
 
 void cpu_reset_interrupt(CPUState *cpu, int mask)
 {
@@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
 
 static void cpu_common_realizefn(DeviceState *dev, Error **errp)
 {
+    if (dev->hotplugged) {
+        cpu_synchronize_post_init(CPU(dev));
+    }
 }
 
 static void cpu_class_init(ObjectClass *klass, void *data)
diff --git a/vl.c b/vl.c
index d694a90..21952c3 100644
--- a/vl.c
+++ b/vl.c
@@ -267,7 +267,6 @@ static NotifierList machine_init_done_notifiers =
     NOTIFIER_LIST_INITIALIZER(machine_init_done_notifiers);
 
 static bool tcg_allowed = true;
-bool kvm_allowed;
 bool xen_allowed;
 uint32_t xen_domid;
 enum xen_mode xen_mode = XEN_EMULATE;
-- 
1.8.2

[Qemu-devel] [PATCH 05/19] cpu: resume CPU from CPUClass.cpu_common_realizefn() when it is hot-plugged

[Igor Mammedov]

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 cpus.c                | 11 ++++++++---
 include/sysemu/cpus.h |  3 +++
 qom/cpu.c             |  2 ++
 stubs/Makefile.objs   |  1 +
 stubs/resume_vcpu.c   |  6 ++++++
 5 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 stubs/resume_vcpu.c

diff --git a/cpus.c b/cpus.c
index 9b9a32f..f12624d 100644
--- a/cpus.c
+++ b/cpus.c
@@ -973,6 +973,13 @@ void pause_all_vcpus(void)
     }
 }
 
+void resume_vcpu(CPUState *cpu)
+{
+    cpu->stop = false;
+    cpu->stopped = false;
+    qemu_cpu_kick(cpu);
+}
+
 void resume_all_vcpus(void)
 {
     CPUArchState *penv = first_cpu;
@@ -980,9 +987,7 @@ void resume_all_vcpus(void)
     qemu_clock_enable(vm_clock, true);
     while (penv) {
         CPUState *pcpu = ENV_GET_CPU(penv);
-        pcpu->stop = false;
-        pcpu->stopped = false;
-        qemu_cpu_kick(pcpu);
+        resume_vcpu(pcpu);
         penv = penv->next_cpu;
     }
 }
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 6502488..9437df5 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -1,11 +1,14 @@
 #ifndef QEMU_CPUS_H
 #define QEMU_CPUS_H
 
+#include "qom/cpu.h"
+
 /* cpus.c */
 void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+void resume_vcpu(CPUState *cpu);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
diff --git a/qom/cpu.c b/qom/cpu.c
index 0c76712..2869cfe 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -21,6 +21,7 @@
 #include "qom/cpu.h"
 #include "qemu-common.h"
 #include "sysemu/kvm.h"
+#include "sysemu/cpus.h"
 
 void cpu_reset_interrupt(CPUState *cpu, int mask)
 {
@@ -60,6 +61,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
 {
     if (dev->hotplugged) {
         cpu_synchronize_post_init(CPU(dev));
+        resume_vcpu(CPU(dev));
     }
 }
 
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 9c55b34..28fb4f8 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -23,3 +23,4 @@ stub-obj-y += sysbus.o
 stub-obj-y += vm-stop.o
 stub-obj-y += vmstate.o
 stub-obj-$(CONFIG_WIN32) += fd-register.o
+stub-obj-y += resume_vcpu.o
diff --git a/stubs/resume_vcpu.c b/stubs/resume_vcpu.c
new file mode 100644
index 0000000..383b71a
--- /dev/null
+++ b/stubs/resume_vcpu.c
@@ -0,0 +1,6 @@
+#include "qemu-common.h"
+#include "sysemu/cpus.h"
+
+void resume_vcpu(CPUState *cpu)
+{
+}
-- 
1.8.2

[Qemu-devel] [PATCH 06/19] introduce CPU hot-plug notifier

[Igor Mammedov]

hot-added CPU will be distributed to acpi_piix4, rtc_cmos and icc_bridge

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
v2:
  * move notifier to qom/cpu.c and call it from CPUClass.realize() on hotplug
  * remove get_firmware_id() since it belong to other patch
---
 include/sysemu/sysemu.h |  3 +++
 qom/cpu.c               | 12 ++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 6578782..a8c3de1 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -152,6 +152,9 @@ void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
 /* generic hotplug */
 void drive_hot_add(Monitor *mon, const QDict *qdict);
 
+/* CPU hotplug */
+void qemu_register_cpu_added_notifier(Notifier *notifier);
+
 /* pcie aer error injection */
 void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
 int do_pcie_aer_inject_error(Monitor *mon,
diff --git a/qom/cpu.c b/qom/cpu.c
index 2869cfe..fe85960 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -22,6 +22,17 @@
 #include "qemu-common.h"
 #include "sysemu/kvm.h"
 #include "sysemu/cpus.h"
+#include "qemu/notify.h"
+#include "sysemu/sysemu.h"
+
+/* CPU hot-plug notifiers */
+static NotifierList cpu_added_notifiers =
+    NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
+
+void qemu_register_cpu_added_notifier(Notifier *notifier)
+{
+    notifier_list_add(&cpu_added_notifiers, notifier);
+}
 
 void cpu_reset_interrupt(CPUState *cpu, int mask)
 {
@@ -62,6 +73,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
     if (dev->hotplugged) {
         cpu_synchronize_post_init(CPU(dev));
         resume_vcpu(CPU(dev));
+        notifier_list_notify(&cpu_added_notifiers, dev);
     }
 }
 
-- 
1.8.2

[Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Igor Mammedov]

... so that on reboot BIOS could read current available CPU count

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
v2:
  * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
---
 hw/timer/mc146818rtc.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
index 69e6844..958ed6b 100644
--- a/hw/timer/mc146818rtc.c
+++ b/hw/timer/mc146818rtc.c
@@ -82,6 +82,7 @@ typedef struct RTCState {
     Notifier clock_reset_notifier;
     LostTickPolicy lost_tick_policy;
     Notifier suspend_notifier;
+    Notifier cpu_added_notifier;
 } RTCState;
 
 static void rtc_set_time(RTCState *s);
@@ -759,6 +760,14 @@ static void rtc_notify_suspend(Notifier *notifier, void *data)
     rtc_set_memory(&s->dev, 0xF, 0xFE);
 }
 
+static void rtc_notify_cpu_added(Notifier *notifier, void *data)
+{
+    RTCState *s = container_of(notifier, RTCState, cpu_added_notifier);
+
+    /* increment the number of CPUs */
+    s->cmos_data[0x5f] += 1;
+}
+
 static void rtc_reset(void *opaque)
 {
     RTCState *s = opaque;
@@ -852,6 +861,9 @@ static int rtc_initfn(ISADevice *dev)
     s->suspend_notifier.notify = rtc_notify_suspend;
     qemu_register_suspend_notifier(&s->suspend_notifier);
 
+    s->cpu_added_notifier.notify = rtc_notify_cpu_added;
+    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
+
     memory_region_init_io(&s->io, &cmos_ops, s, "rtc", 2);
     isa_register_ioport(dev, &s->io, base);
 
-- 
1.8.2

[Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Igor Mammedov]

get_arch_id() adds possibility for generic code to get guest visible
CPI id without accessing CPUArchState. If target doesn't override it,
it will return cpu_index.

Override it on target-i386 to return APIC ID.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 * it will be used later by new cpu_exists() generic function and
   acpi_piix.
 * s/cpu_firmware_id/cpu_arch_id/
---
 include/qom/cpu.h |  2 ++
 qom/cpu.c         |  6 ++++++
 target-i386/cpu.c | 10 ++++++++++
 3 files changed, 18 insertions(+)

diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 3664a1b..b376416 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -45,6 +45,7 @@ typedef struct CPUState CPUState;
  * instantiatable CPU type.
  * @reset: Callback to reset the #CPUState to its initial state.
  * @do_interrupt: Callback for interrupt handling.
+ * @get_arch_id: Callback for getting architecture depended CPU ID
  * @vmsd: State description for migration.
  *
  * Represents a CPU family or model.
@@ -58,6 +59,7 @@ typedef struct CPUClass {
 
     void (*reset)(CPUState *cpu);
     void (*do_interrupt)(CPUState *cpu);
+    int64_t (*get_arch_id)(CPUState *cpu);
 
     const struct VMStateDescription *vmsd;
 } CPUClass;
diff --git a/qom/cpu.c b/qom/cpu.c
index fe85960..90cbd77 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -77,6 +77,11 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
     }
 }
 
+static int64_t cpu_common_get_arch_id(CPUState *cpu)
+{
+    return cpu->cpu_index;
+}
+
 static void cpu_class_init(ObjectClass *klass, void *data)
 {
     DeviceClass *dc = DEVICE_CLASS(klass);
@@ -84,6 +89,7 @@ static void cpu_class_init(ObjectClass *klass, void *data)
 
     k->class_by_name = cpu_common_class_by_name;
     k->reset = cpu_common_reset;
+    k->get_arch_id = cpu_common_get_arch_id;
     dc->realize = cpu_common_realizefn;
     dc->no_user = 1;
 }
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index c28436c..9cca031 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2272,6 +2272,14 @@ static void x86_cpu_initfn(Object *obj)
     }
 }
 
+static int64_t x86_cpu_get_arch_id(CPUState *cpu)
+{
+    X86CPU *x86cpu = X86_CPU(cpu);
+    CPUX86State *env = &x86cpu->env;
+
+    return env->cpuid_apic_id;
+}
+
 static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 {
     X86CPUClass *xcc = X86_CPU_CLASS(oc);
@@ -2286,6 +2294,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 
     cc->do_interrupt = x86_cpu_do_interrupt;
     cpu_class_set_vmsd(cc, &vmstate_x86_cpu);
+
+    cc->get_arch_id = x86_cpu_get_arch_id;
 }
 
 static const TypeInfo x86_cpu_type_info = {
-- 
1.8.2

[Qemu-devel] [PATCH 09/19] cpu: add helper cpu_exists(), to check if CPU with specified id exists

[Igor Mammedov]

... it should be used only on slow path since it does recursive search
    on /machine QOM tree for objects of TYPE_CPU

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
v2:
  * s/get_firmware_id()/get_arch_id()/ rebase fixup
  * remove check for get_arch_id being NULL, since it's always defined
---
 include/qom/cpu.h | 10 ++++++++++
 qom/cpu.c         | 21 +++++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index b376416..cd7c4b5 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -225,6 +225,16 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
  */
 CPUState *qemu_get_cpu(int index);
 
+/**
+ * cpu_exists:
+ * @id - guest exposed CPU ID to lookup
+ *
+ * Search for CPU with specified ID.
+ *
+ * Returns: true - CPU is found, false - CPU isn't found
+ */
+bool cpu_exists(int64_t id);
+
 #ifndef CONFIG_USER_ONLY
 
 typedef void (*CPUInterruptHandler)(CPUState *, int);
diff --git a/qom/cpu.c b/qom/cpu.c
index 90cbd77..c850d27 100644
--- a/qom/cpu.c
+++ b/qom/cpu.c
@@ -25,6 +25,27 @@
 #include "qemu/notify.h"
 #include "sysemu/sysemu.h"
 
+static int cpu_exist_cb(Object *obj, void *opaque)
+{
+    int64_t id = *(int64_t *)opaque;
+    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
+
+    if (cpu_obj) {
+        CPUState *cpu = CPU(cpu_obj);
+        CPUClass *klass = CPU_GET_CLASS(cpu);
+
+        if (klass->get_arch_id(cpu) == id) {
+            return 1;
+        }
+    }
+    return object_child_foreach(obj, cpu_exist_cb, opaque);
+}
+
+bool cpu_exists(int64_t id)
+{
+   return cpu_exist_cb(qdev_get_machine(), &id) ? true : false;
+}
+
 /* CPU hot-plug notifiers */
 static NotifierList cpu_added_notifiers =
     NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
-- 
1.8.2

[Qemu-devel] [PATCH 10/19] acpi_piix4: add infrastructure to send CPU hot-plug GPE to guest

[Igor Mammedov]

* introduce processor status bitmask visible to guest at 0xaf00 addr,
  where ACPI asl code expects it
* set bit corresponding to APIC ID in processor status bitmask on
  receiving CPU hot-plug notification
* trigger CPU hot-plug SCI, to notify guest about CPU hot-plug event

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v3:
  * s/get_firmware_id()/get_arch_id()/ due rebase
  * s/cpu_add_notifier/cpu_added_notifier/
v2:
  * use CPUClass.get_firmware_id() to make code target independent
  * bump up vmstate_acpi version
---
 hw/acpi/piix4.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 111 insertions(+), 3 deletions(-)

diff --git a/hw/acpi/piix4.c b/hw/acpi/piix4.c
index 88386d7..a753adf 100644
--- a/hw/acpi/piix4.c
+++ b/hw/acpi/piix4.c
@@ -48,19 +48,28 @@
 #define PCI_EJ_BASE 0xae08
 #define PCI_RMV_BASE 0xae0c
 
+#define PROC_BASE 0xaf00
+#define PROC_LEN 32
+
 #define PIIX4_PCI_HOTPLUG_STATUS 2
+#define PIIX4_CPU_HOTPLUG_STATUS 4
 
 struct pci_status {
     uint32_t up; /* deprecated, maintained for migration compatibility */
     uint32_t down;
 };
 
+struct cpu_status {
+    uint8_t sts[PROC_LEN];
+};
+
 typedef struct PIIX4PMState {
     PCIDevice dev;
 
     MemoryRegion io;
     MemoryRegion io_gpe;
     MemoryRegion io_pci;
+    MemoryRegion io_cpu;
     ACPIREGS ar;
 
     APMState apm;
@@ -82,6 +91,9 @@ typedef struct PIIX4PMState {
     uint8_t disable_s3;
     uint8_t disable_s4;
     uint8_t s4_val;
+
+    struct cpu_status gpe_cpu;
+    Notifier cpu_added_notifier;
 } PIIX4PMState;
 
 static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
@@ -100,8 +112,8 @@ static void pm_update_sci(PIIX4PMState *s)
                    ACPI_BITMASK_POWER_BUTTON_ENABLE |
                    ACPI_BITMASK_GLOBAL_LOCK_ENABLE |
                    ACPI_BITMASK_TIMER_ENABLE)) != 0) ||
-        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0])
-          & PIIX4_PCI_HOTPLUG_STATUS) != 0);
+        (((s->ar.gpe.sts[0] & s->ar.gpe.en[0]) &
+          (PIIX4_PCI_HOTPLUG_STATUS | PIIX4_CPU_HOTPLUG_STATUS)) != 0);
 
     qemu_set_irq(s->irq, sci_level);
     /* schedule a timer interruption if needed */
@@ -257,6 +269,17 @@ static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+#define VMSTATE_CPU_STATUS_ARRAY(_field, _state)                             \
+ {                                                                           \
+     .name       = (stringify(_field)),                                      \
+     .version_id = 0,                                                        \
+     .num        = PROC_LEN,                                                 \
+     .info       = &vmstate_info_uint8,                                      \
+     .size       = sizeof(uint8_t),                                          \
+     .flags      = VMS_ARRAY,                                                \
+     .offset     = vmstate_offset_array(_state, _field, uint8_t, PROC_LEN),  \
+ }
+
 /* qemu-kvm 1.2 uses version 3 but advertised as 2
  * To support incoming qemu-kvm 1.2 migration, change version_id
  * and minimum_version_id to 2 below (which breaks migration from
@@ -265,7 +288,7 @@ static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
  */
 static const VMStateDescription vmstate_acpi = {
     .name = "piix4_pm",
-    .version_id = 3,
+    .version_id = 4,
     .minimum_version_id = 3,
     .minimum_version_id_old = 1,
     .load_state_old = acpi_load_old,
@@ -281,6 +304,7 @@ static const VMStateDescription vmstate_acpi = {
         VMSTATE_STRUCT(ar.gpe, PIIX4PMState, 2, vmstate_gpe, ACPIGPE),
         VMSTATE_STRUCT(pci0_status, PIIX4PMState, 2, vmstate_pci_status,
                        struct pci_status),
+        VMSTATE_CPU_STATUS_ARRAY(gpe_cpu.sts, PIIX4PMState),
         VMSTATE_END_OF_LIST()
     }
 };
@@ -585,6 +609,83 @@ static const MemoryRegionOps piix4_pci_ops = {
     },
 };
 
+static uint64_t cpu_status_read(void *opaque, hwaddr addr, unsigned width)
+{
+    PIIX4PMState *s = opaque;
+    struct cpu_status *cpus = &s->gpe_cpu;
+    uint64_t val = cpus->sts[addr];
+    return val;
+}
+
+static void cpu_status_write(void *opaque, hwaddr addr, uint64_t data,
+                             unsigned int size)
+{
+    /* TODO: implement VCPU removal on guest signal that CPU can be removed */
+}
+
+static const MemoryRegionOps cpu_hotplug_ops = {
+    .read = cpu_status_read,
+    .write = cpu_status_write,
+    .endianness = DEVICE_LITTLE_ENDIAN,
+    .valid = {
+        .min_access_size = 1,
+        .max_access_size = 1,
+    },
+};
+
+typedef enum {
+    PLUG,
+    UNPLUG,
+} HotplugEventType;
+
+static void piix4_cpu_hotplug_req(PIIX4PMState *s, CPUState *cpu,
+                                  HotplugEventType action)
+{
+    struct cpu_status *g = &s->gpe_cpu;
+    ACPIGPE *gpe = &s->ar.gpe;
+    CPUClass *k = CPU_GET_CLASS(cpu);
+    int64_t cpu_id;
+
+    assert(s != NULL);
+
+    *gpe->sts = *gpe->sts | PIIX4_CPU_HOTPLUG_STATUS;
+    cpu_id = k->get_arch_id(CPU(cpu));
+    if (action == PLUG) {
+        g->sts[cpu_id / 8] |= (1 << (cpu_id % 8));
+    } else {
+        g->sts[cpu_id / 8] &= ~(1 << (cpu_id % 8));
+    }
+    pm_update_sci(s);
+}
+
+static void piix4_cpu_added_req(Notifier *n, void *opaque)
+{
+    PIIX4PMState *s = container_of(n, PIIX4PMState, cpu_added_notifier);
+    piix4_cpu_hotplug_req(s, CPU(opaque), PLUG);
+}
+
+static int piix4_init_cpu_status(Object *obj, void *opaque)
+{
+    struct cpu_status *g = (struct cpu_status *)opaque;
+    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
+
+    if (cpu_obj) {
+        struct Error *error = NULL;
+        CPUClass *k = CPU_GET_CLASS(cpu_obj);
+        int64_t id = k->get_arch_id(CPU(cpu_obj));
+
+        if (error) {
+            fprintf(stderr, "failed to initilize CPU status for ACPI: %s\n",
+                    error_get_pretty(error));
+            error_free(error);
+            abort();
+        }
+        g_assert((id / 8) < PROC_LEN);
+        g->sts[id / 8] |= (1 << (id % 8));
+    }
+    return object_child_foreach(obj, piix4_init_cpu_status, opaque);
+}
+
 static int piix4_device_hotplug(DeviceState *qdev, PCIDevice *dev,
                                 PCIHotplugState state);
 
@@ -600,6 +701,13 @@ static void piix4_acpi_system_hot_add_init(MemoryRegion *parent,
     memory_region_add_subregion(parent, PCI_HOTPLUG_ADDR,
                                 &s->io_pci);
     pci_bus_hotplug(bus, piix4_device_hotplug, &s->dev.qdev);
+
+    piix4_init_cpu_status(qdev_get_machine(), &s->gpe_cpu);
+    memory_region_init_io(&s->io_cpu, &cpu_hotplug_ops, s, "apci-cpu-hotplug",
+                          PROC_LEN);
+    memory_region_add_subregion(parent, PROC_BASE, &s->io_cpu);
+    s->cpu_added_notifier.notify = piix4_cpu_added_req;
+    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
 }
 
 static void enable_device(PIIX4PMState *s, int slot)
-- 
1.8.2

[Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

... and use it from board level to set APIC ID for CPUs
it creates.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
Note:
  * pc_new_cpu() function will be reused later in CPU hot-plug hook.

v3:
  * user error_propagate() in property setter
v2:
  * use generic cpu_exists() instead of custom one
  * make apic-id dynamic property, so it won't be possible to use it
    as global property, since it's instance specific
---
 hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
 target-i386/cpu.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 8d75b34..3adf294 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
+static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+{
+    X86CPU *cpu;
+
+    cpu = cpu_x86_create(cpu_model, errp);
+    if (!cpu) {
+        return;
+    }
+
+    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
+    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
+
+    if (error_is_set(errp)) {
+        if (cpu != NULL) {
+            object_unref(OBJECT(cpu));
+        }
+    }
+}
+
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
+    Error *error = NULL;
 
     /* init CPUs */
     if (cpu_model == NULL) {
@@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
     }
 
     for (i = 0; i < smp_cpus; i++) {
-        if (!cpu_x86_init(cpu_model)) {
+        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        if (error) {
+            fprintf(stderr, "%s\n", error_get_pretty(error));
+            error_free(error);
             exit(1);
         }
     }
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 9cca031..4ddc18a 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1272,6 +1272,45 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
     cpu->env.tsc_khz = value / 1000;
 }
 
+static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
+                                  const char *name, Error **errp)
+{
+    X86CPU *cpu = X86_CPU(obj);
+    int64_t value = cpu->env.cpuid_apic_id;
+
+    visit_type_int(v, &value, name, errp);
+}
+
+static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
+                                  const char *name, Error **errp)
+{
+    X86CPU *cpu = X86_CPU(obj);
+    const int64_t min = 0;
+    const int64_t max = UINT32_MAX;
+    Error *error = NULL;
+    int64_t value;
+
+    visit_type_int(v, &value, name, &error);
+    if (error) {
+        error_propagate(errp, error);
+        return;
+    }
+    if (value < min || value > max) {
+        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
+                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
+                   object_get_typename(obj), name, value, min, max);
+        error_propagate(errp, error);
+        return;
+    }
+
+    if (cpu_exists(value)) {
+        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
+        error_propagate(errp, error);
+        return;
+    }
+    cpu->env.cpuid_apic_id = value;
+}
+
 static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
 {
     x86_def_t *def;
@@ -2259,6 +2298,9 @@ static void x86_cpu_initfn(Object *obj)
     object_property_add(obj, "tsc-frequency", "int",
                         x86_cpuid_get_tsc_freq,
                         x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
+    object_property_add(obj, "apic-id", "int",
+                        x86_cpuid_get_apic_id,
+                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
 
     env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
 
-- 
1.8.2

[Qemu-devel] [PATCH 12/19] introduce ICC bus/device/bridge

[Igor Mammedov]

... to provide hotplug-able bus.

* icc-bridge will serve as a parent for icc-bus, provide
  mmio mapping services to child icc-device and create
  IOAPIC when requested.
* icc-device will replace SysBusDevice as a parent of APIC
  and IOAPIC devices.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
  * Rebase on top the last HW reorganization series.
  * Move hw/icc_bus.c into hw/cpu/ and hw/icc_bus.h include/hw/i386/

mv icc to hw/cpu
---
 default-configs/i386-softmmu.mak   |  1 +
 default-configs/x86_64-softmmu.mak |  1 +
 hw/cpu/Makefile.objs               |  1 +
 hw/cpu/icc_bus.c                   | 85 ++++++++++++++++++++++++++++++++++++++
 hw/i386/pc_piix.c                  |  7 ++++
 hw/i386/pc_q35.c                   |  7 ++++
 include/hw/i386/icc_bus.h          | 50 ++++++++++++++++++++++
 7 files changed, 152 insertions(+)
 create mode 100644 hw/cpu/icc_bus.c
 create mode 100644 include/hw/i386/icc_bus.h

diff --git a/default-configs/i386-softmmu.mak b/default-configs/i386-softmmu.mak
index 4ac0694..0789303 100644
--- a/default-configs/i386-softmmu.mak
+++ b/default-configs/i386-softmmu.mak
@@ -45,3 +45,4 @@ CONFIG_LPC_ICH9=y
 CONFIG_Q35=y
 CONFIG_APIC=y
 CONFIG_IOAPIC=y
+CONFIG_ICC_BUS=y
diff --git a/default-configs/x86_64-softmmu.mak b/default-configs/x86_64-softmmu.mak
index bf4091c..8d03e9a 100644
--- a/default-configs/x86_64-softmmu.mak
+++ b/default-configs/x86_64-softmmu.mak
@@ -45,3 +45,4 @@ CONFIG_LPC_ICH9=y
 CONFIG_Q35=y
 CONFIG_APIC=y
 CONFIG_IOAPIC=y
+CONFIG_ICC_BUS=y
diff --git a/hw/cpu/Makefile.objs b/hw/cpu/Makefile.objs
index a49ca04..4461ece 100644
--- a/hw/cpu/Makefile.objs
+++ b/hw/cpu/Makefile.objs
@@ -1,4 +1,5 @@
 obj-$(CONFIG_ARM11MPCORE) += arm11mpcore.o
 obj-$(CONFIG_ARM9MPCORE) += a9mpcore.o
 obj-$(CONFIG_ARM15MPCORE) += a15mpcore.o
+obj-$(CONFIG_ICC_BUS) += icc_bus.o
 
diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
new file mode 100644
index 0000000..3824968
--- /dev/null
+++ b/hw/cpu/icc_bus.c
@@ -0,0 +1,85 @@
+/* icc_bus.c
+ * emulate x86 ICC(INTERRUPT CONTROLLER COMMUNICATIONS) bus
+ *
+ * Copyright (c) 2013 Red Hat, Inc
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+#include "hw/i386/icc_bus.h"
+#include "hw/sysbus.h"
+
+static void icc_bus_initfn(Object *obj)
+{
+    BusState *b = BUS(obj);
+    b->allow_hotplug = true;
+}
+
+static const TypeInfo icc_bus_info = {
+    .name = TYPE_ICC_BUS,
+    .parent = TYPE_BUS,
+    .instance_size = sizeof(ICCBus),
+    .instance_init = icc_bus_initfn,
+};
+
+static int icc_device_init(DeviceState *dev)
+{
+    ICCDevice *id = ICC_DEVICE(dev);
+    ICCDeviceClass *k = ICC_DEVICE_GET_CLASS(id);
+
+    return k->init(id);
+}
+
+static void icc_device_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *k = DEVICE_CLASS(klass);
+
+    k->init = icc_device_init;
+    k->bus_type = TYPE_ICC_BUS;
+}
+
+static const TypeInfo icc_device_info = {
+    .name = TYPE_ICC_DEVICE,
+    .parent = TYPE_DEVICE,
+    .abstract = true,
+    .instance_size = sizeof(ICCDevice),
+    .class_size = sizeof(ICCDeviceClass),
+    .class_init = icc_device_class_init,
+};
+
+typedef struct ICCBridgeState {
+    SysBusDevice busdev;
+} ICCBridgeState;
+#define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
+
+
+static void icc_bridge_initfn(Object *obj)
+{
+    qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus");
+}
+
+static const TypeInfo icc_bridge_info = {
+    .name  = "icc-bridge",
+    .parent = TYPE_SYS_BUS_DEVICE,
+    .instance_init  = icc_bridge_initfn,
+    .instance_size  = sizeof(ICCBridgeState),
+};
+
+static void icc_bus_register_types(void)
+{
+    type_register_static(&icc_bus_info);
+    type_register_static(&icc_device_info);
+    type_register_static(&icc_bridge_info);
+}
+
+type_init(icc_bus_register_types)
diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index cff8013..d8c3e1a 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -37,6 +37,7 @@
 #include "hw/kvm/clock.h"
 #include "sysemu/sysemu.h"
 #include "hw/sysbus.h"
+#include "hw/i386/icc_bus.h"
 #include "sysemu/arch_init.h"
 #include "sysemu/blockdev.h"
 #include "hw/i2c/smbus.h"
@@ -84,8 +85,13 @@ static void pc_init1(MemoryRegion *system_memory,
     MemoryRegion *ram_memory;
     MemoryRegion *pci_memory;
     MemoryRegion *rom_memory;
+    DeviceState *icc_bridge;
     void *fw_cfg = NULL;
 
+    icc_bridge = qdev_create(NULL, TYPE_ICC_BRIDGE);
+    object_property_add_child(qdev_get_machine(), "icc-bridge",
+                              OBJECT(icc_bridge), NULL);
+
     pc_cpus_init(cpu_model);
     pc_acpi_init("acpi-dsdt.aml");
 
@@ -160,6 +166,7 @@ static void pc_init1(MemoryRegion *system_memory,
     if (pci_enabled) {
         ioapic_init_gsi(gsi_state, "i440fx");
     }
+    qdev_init_nofail(icc_bridge);
 
     pc_register_ferr_irq(gsi[13]);
 
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 6ac1a89..9055145 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -41,6 +41,7 @@
 #include "hw/ide/pci.h"
 #include "hw/ide/ahci.h"
 #include "hw/usb.h"
+#include "hw/i386/icc_bus.h"
 
 /* ICH9 AHCI has 6 ports */
 #define MAX_SATA_PORTS     6
@@ -85,6 +86,11 @@ static void pc_q35_init(QEMUMachineInitArgs *args)
     ICH9LPCState *ich9_lpc;
     PCIDevice *ahci;
     qemu_irq *cmos_s3;
+    DeviceState *icc_bridge;
+
+    icc_bridge = qdev_create(NULL, TYPE_ICC_BRIDGE);
+    object_property_add_child(qdev_get_machine(), "icc-bridge",
+                              OBJECT(icc_bridge), NULL);
 
     pc_cpus_init(cpu_model);
     pc_acpi_init("q35-acpi-dsdt.aml");
@@ -168,6 +174,7 @@ static void pc_q35_init(QEMUMachineInitArgs *args)
     if (pci_enabled) {
         ioapic_init_gsi(gsi_state, NULL);
     }
+    qdev_init_nofail(icc_bridge);
 
     pc_register_ferr_irq(gsi[13]);
 
diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
new file mode 100644
index 0000000..ae1866c
--- /dev/null
+++ b/include/hw/i386/icc_bus.h
@@ -0,0 +1,50 @@
+/* icc_bus.h
+ * emulate x86 ICC(INTERRUPT CONTROLLER COMMUNICATIONS) bus
+ *
+ * Copyright (c) 2013 Red Hat, Inc
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+#ifndef ICC_BUS_H
+#define ICC_BUS_H
+
+#include "hw/qdev-core.h"
+
+#define TYPE_ICC_BUS "icc-bus"
+
+#ifndef CONFIG_USER_ONLY
+typedef struct ICCBus {
+    BusState qbus;
+} ICCBus;
+#define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
+
+typedef struct ICCDevice {
+    DeviceState qdev;
+} ICCDevice;
+
+typedef struct ICCDeviceClass {
+    DeviceClass parent_class;
+    int (*init)(ICCDevice *dev);
+} ICCDeviceClass;
+#define TYPE_ICC_DEVICE "icc-device"
+#define ICC_DEVICE(obj) OBJECT_CHECK(ICCDevice, (obj), TYPE_ICC_DEVICE)
+#define ICC_DEVICE_CLASS(klass) \
+     OBJECT_CLASS_CHECK(ICCDeviceClass, (klass), TYPE_ICC_DEVICE)
+#define ICC_DEVICE_GET_CLASS(obj) \
+     OBJECT_GET_CLASS(ICCDeviceClass, (obj), TYPE_ICC_DEVICE)
+
+#define TYPE_ICC_BRIDGE "icc-bridge"
+
+#endif /* CONFIG_USER_ONLY */
+#endif
-- 
1.8.2

[Qemu-devel] [PATCH 13/19] target-i386: cpu: attach ICC bus to CPU on its creation

[Igor Mammedov]

... during startup,  so it would be possible to uplug it later
and set bus_type to TYPE_ICC_BUS for X86CPU type to make device_add
attach hotplugged CPU to ICC bus.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 target-i386/cpu.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 4ddc18a..2bb38d6 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -41,6 +41,7 @@
 #endif
 
 #include "sysemu/sysemu.h"
+#include "hw/i386/icc_bus.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/xen/xen.h"
 #include "hw/sysbus.h"
@@ -1610,6 +1611,7 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
     gchar **model_pieces;
     char *name, *features;
     Error *error = NULL;
+    Object *icc_bus = object_resolve_path_type("icc-bus", TYPE_ICC_BUS, NULL);
 
     model_pieces = g_strsplit(cpu_model, ",", 2);
     if (!model_pieces[0]) {
@@ -1620,6 +1622,10 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
     features = model_pieces[1];
 
     cpu = X86_CPU(object_new(TYPE_X86_CPU));
+    if (icc_bus) {
+        qdev_set_parent_bus(DEVICE(cpu), BUS(icc_bus));
+        object_unref(OBJECT(cpu));
+    }
     env = &cpu->env;
     env->cpu_model_str = cpu_model;
 
@@ -2330,6 +2336,7 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
 
     xcc->parent_realize = dc->realize;
     dc->realize = x86_cpu_realizefn;
+    dc->bus_type = TYPE_ICC_BUS;
 
     xcc->parent_reset = cc->reset;
     cc->reset = x86_cpu_reset;
-- 
1.8.2

[Qemu-devel] [PATCH 14/19] target-i386: replace MSI_SPACE_SIZE with APIC_SPACE_SIZE

[Igor Mammedov]

... and put APIC_SPACE_SIZE in public header so that it could be
reused later elsewhere.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/i386/kvm/apic.c              | 2 +-
 hw/intc/apic.c                  | 2 +-
 hw/xen/xen_apic.c               | 2 +-
 include/hw/i386/apic_internal.h | 2 --
 target-i386/cpu.h               | 1 +
 5 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/hw/i386/kvm/apic.c b/hw/i386/kvm/apic.c
index c6ff982..8f80425 100644
--- a/hw/i386/kvm/apic.c
+++ b/hw/i386/kvm/apic.c
@@ -174,7 +174,7 @@ static const MemoryRegionOps kvm_apic_io_ops = {
 static void kvm_apic_init(APICCommonState *s)
 {
     memory_region_init_io(&s->io_memory, &kvm_apic_io_ops, s, "kvm-apic-msi",
-                          MSI_SPACE_SIZE);
+                          APIC_SPACE_SIZE);
 
     if (kvm_has_gsi_routing()) {
         msi_supported = true;
diff --git a/hw/intc/apic.c b/hw/intc/apic.c
index 2d79a9e..756dff0 100644
--- a/hw/intc/apic.c
+++ b/hw/intc/apic.c
@@ -874,7 +874,7 @@ static const MemoryRegionOps apic_io_ops = {
 static void apic_init(APICCommonState *s)
 {
     memory_region_init_io(&s->io_memory, &apic_io_ops, s, "apic-msi",
-                          MSI_SPACE_SIZE);
+                          APIC_SPACE_SIZE);
 
     s->timer = qemu_new_timer_ns(vm_clock, apic_timer, s);
     local_apics[s->idx] = s;
diff --git a/hw/xen/xen_apic.c b/hw/xen/xen_apic.c
index a2eb8a1..a958782 100644
--- a/hw/xen/xen_apic.c
+++ b/hw/xen/xen_apic.c
@@ -39,7 +39,7 @@ static const MemoryRegionOps xen_apic_io_ops = {
 static void xen_apic_init(APICCommonState *s)
 {
     memory_region_init_io(&s->io_memory, &xen_apic_io_ops, s, "xen-apic-msi",
-                          MSI_SPACE_SIZE);
+                          APIC_SPACE_SIZE);
 
 #if defined(CONFIG_XEN_CTRL_INTERFACE_VERSION) \
     && CONFIG_XEN_CTRL_INTERFACE_VERSION >= 420
diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
index 578241f..aac6290 100644
--- a/include/hw/i386/apic_internal.h
+++ b/include/hw/i386/apic_internal.h
@@ -66,8 +66,6 @@
 
 #define MAX_APICS 255
 
-#define MSI_SPACE_SIZE                  0x100000
-
 typedef struct APICCommonState APICCommonState;
 
 #define TYPE_APIC_COMMON "apic-common"
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index cf1b05c..65cf97d 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -1269,5 +1269,6 @@ uint32_t x86_cpu_apic_id_from_index(unsigned int cpu_index);
 void enable_compat_apic_id_mode(void);
 
 #define APIC_DEFAULT_ADDRESS 0xfee00000
+#define APIC_SPACE_SIZE      0x100000
 
 #endif /* CPU_I386_H */
-- 
1.8.2

[Qemu-devel] [PATCH 15/19] target-i386: move APIC to ICC bus

[Igor Mammedov]

... to allow it to be hotplugged

 * map APIC's mmio at board level if it is present
 * do not register mmio region for each APIC, since
   only one is used/mapped

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 hw/cpu/icc_bus.c                | 15 ++++++++++++++-
 hw/i386/kvmvapic.c              |  1 +
 hw/i386/pc.c                    | 20 +++++++++++++++++---
 hw/intc/apic_common.c           | 17 ++++++++++++-----
 include/hw/i386/apic_internal.h |  6 +++---
 include/hw/i386/icc_bus.h       |  2 ++
 target-i386/cpu.c               | 16 +++-------------
 7 files changed, 52 insertions(+), 25 deletions(-)

diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
index 3824968..c499f09 100644
--- a/hw/cpu/icc_bus.c
+++ b/hw/cpu/icc_bus.c
@@ -59,13 +59,26 @@ static const TypeInfo icc_device_info = {
 
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
+    MemoryRegion apic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
 
 static void icc_bridge_initfn(Object *obj)
 {
-    qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus");
+    ICCBridgeState *s = ICC_BRIGDE(obj);
+    SysBusDevice *sb = SYS_BUS_DEVICE(obj);
+    ICCBus *ibus;
+
+    ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
+
+    /* Do not change order of registering regions,
+     * APIC must be first registered region, board maps it by 0 index
+     */
+    memory_region_init(&s->apic_container, "icc-apic-container",
+                       APIC_SPACE_SIZE);
+    sysbus_init_mmio(sb, &s->apic_container);
+    ibus->apic_address_space = &s->apic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/i386/kvmvapic.c b/hw/i386/kvmvapic.c
index 3a10c07..5b558aa 100644
--- a/hw/i386/kvmvapic.c
+++ b/hw/i386/kvmvapic.c
@@ -12,6 +12,7 @@
 #include "sysemu/cpus.h"
 #include "sysemu/kvm.h"
 #include "hw/i386/apic_internal.h"
+#include "hw/sysbus.h"
 
 #define VAPIC_IO_PORT           0x7e
 
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 3adf294..1c7b3c2 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -52,6 +52,7 @@
 #include "sysemu/arch_init.h"
 #include "qemu/bitmap.h"
 #include "qemu/config-file.h"
+#include "hw/i386/icc_bus.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -869,13 +870,13 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
-static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
 {
     X86CPU *cpu;
 
     cpu = cpu_x86_create(cpu_model, errp);
     if (!cpu) {
-        return;
+        return cpu;
     }
 
     object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
@@ -884,14 +885,18 @@ static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
     if (error_is_set(errp)) {
         if (cpu != NULL) {
             object_unref(OBJECT(cpu));
+            cpu = NULL;
         }
     }
+    return cpu;
 }
 
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
+    X86CPU *cpu = NULL;
     Error *error = NULL;
+    SysBusDevice *ib;
 
     /* init CPUs */
     if (cpu_model == NULL) {
@@ -902,14 +907,23 @@ void pc_cpus_init(const char *cpu_model)
 #endif
     }
 
+    ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
+                                                 TYPE_ICC_BRIDGE, NULL));
+
     for (i = 0; i < smp_cpus; i++) {
-        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
         if (error) {
             fprintf(stderr, "%s\n", error_get_pretty(error));
             error_free(error);
             exit(1);
         }
     }
+
+    /* map APIC MMIO area if CPU has APIC */
+    if (cpu && cpu->env.apic_state) {
+        /* XXX: what if the base changes? */
+        sysbus_mmio_map_overlap(ib, 0, APIC_DEFAULT_ADDRESS, 0x1000);
+    }
 }
 
 void pc_acpi_init(const char *default_dsdt)
diff --git a/hw/intc/apic_common.c b/hw/intc/apic_common.c
index e0ae07a..39396f1 100644
--- a/hw/intc/apic_common.c
+++ b/hw/intc/apic_common.c
@@ -21,6 +21,8 @@
 #include "hw/i386/apic_internal.h"
 #include "trace.h"
 #include "sysemu/kvm.h"
+#include "hw/qdev.h"
+#include "hw/sysbus.h"
 
 static int apic_irq_delivered;
 bool apic_report_tpr_access;
@@ -282,12 +284,14 @@ static int apic_load_old(QEMUFile *f, void *opaque, int version_id)
     return 0;
 }
 
-static int apic_init_common(SysBusDevice *dev)
+static int apic_init_common(ICCDevice *dev)
 {
     APICCommonState *s = APIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
     static int apic_no;
+    static bool mmio_registered;
 
     if (apic_no >= MAX_APICS) {
         return -1;
@@ -296,8 +300,11 @@ static int apic_init_common(SysBusDevice *dev)
 
     info = APIC_COMMON_GET_CLASS(s);
     info->init(s);
-
-    sysbus_init_mmio(dev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->apic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
 
     /* Note: We need at least 1M to map the VAPIC option ROM */
     if (!vapic && s->vapic_control & VAPIC_ENABLE_MASK &&
@@ -375,7 +382,7 @@ static Property apic_properties_common[] = {
 
 static void apic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     dc->vmsd = &vmstate_apic_common;
@@ -387,7 +394,7 @@ static void apic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo apic_common_type = {
     .name = TYPE_APIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(APICCommonState),
     .class_size = sizeof(APICCommonClass),
     .class_init = apic_common_class_init,
diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
index aac6290..05acf4b 100644
--- a/include/hw/i386/apic_internal.h
+++ b/include/hw/i386/apic_internal.h
@@ -21,7 +21,7 @@
 #define QEMU_APIC_INTERNAL_H
 
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/i386/icc_bus.h"
 #include "qemu/timer.h"
 
 /* APIC Local Vector Table */
@@ -78,7 +78,7 @@ typedef struct APICCommonState APICCommonState;
 
 typedef struct APICCommonClass
 {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
 
     void (*init)(APICCommonState *s);
     void (*set_base)(APICCommonState *s, uint64_t val);
@@ -92,7 +92,7 @@ typedef struct APICCommonClass
 } APICCommonClass;
 
 struct APICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
 
     MemoryRegion io_memory;
     X86CPU *cpu;
diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
index ae1866c..2c8f78f 100644
--- a/include/hw/i386/icc_bus.h
+++ b/include/hw/i386/icc_bus.h
@@ -19,6 +19,7 @@
 #ifndef ICC_BUS_H
 #define ICC_BUS_H
 
+#include "exec/memory.h"
 #include "hw/qdev-core.h"
 
 #define TYPE_ICC_BUS "icc-bus"
@@ -26,6 +27,7 @@
 #ifndef CONFIG_USER_ONLY
 typedef struct ICCBus {
     BusState qbus;
+    MemoryRegion *apic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 2bb38d6..f77c3bb 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -41,10 +41,10 @@
 #endif
 
 #include "sysemu/sysemu.h"
+#include "hw/qdev-properties.h"
 #include "hw/i386/icc_bus.h"
 #ifndef CONFIG_USER_ONLY
 #include "hw/xen/xen.h"
-#include "hw/sysbus.h"
 #include "hw/i386/apic_internal.h"
 #endif
 
@@ -2111,6 +2111,7 @@ static void mce_init(X86CPU *cpu)
 static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
+    DeviceState *dev = DEVICE(cpu);
     APICCommonState *apic;
     const char *apic_type = "apic";
 
@@ -2120,7 +2121,7 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
         apic_type = "xen-apic";
     }
 
-    env->apic_state = qdev_try_create(NULL, apic_type);
+    env->apic_state = qdev_try_create(dev->parent_bus, apic_type);
     if (env->apic_state == NULL) {
         error_setg(errp, "APIC device '%s' could not be created", apic_type);
         return;
@@ -2137,7 +2138,6 @@ static void x86_cpu_apic_create(X86CPU *cpu, Error **errp)
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
 {
     CPUX86State *env = &cpu->env;
-    static int apic_mapped;
 
     if (env->apic_state == NULL) {
         return;
@@ -2148,16 +2148,6 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
                    object_get_typename(OBJECT(env->apic_state)));
         return;
     }
-
-    /* XXX: mapping more APICs at the same memory location */
-    if (apic_mapped == 0) {
-        /* NOTE: the APIC is directly connected to the CPU - it is not
-           on the global memory bus. */
-        /* XXX: what if the base changes? */
-        sysbus_mmio_map_overlap(SYS_BUS_DEVICE(env->apic_state), 0,
-                                APIC_DEFAULT_ADDRESS, 0x1000);
-        apic_mapped = 1;
-    }
 }
 #else
 static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp)
-- 
1.8.2

[Qemu-devel] [PATCH 16/19] target-i386: move IOAPIC to ICC bus

[Igor Mammedov]

 * inherit IOAPICs from ICCDevice and attach them to ICC bus
 * map IOAPIC's mmio at board level via indirect icc-bridge
   mmio region that provided address space to IOAPIC via
   icc-bus

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
 * do not create IOAPIC in icc-bridge and do not make it QOM child
   of icc-bridge, just attach it to icc-bus.
---
 hw/cpu/icc_bus.c                  |  6 ++++++
 hw/i386/kvm/ioapic.c              |  2 +-
 hw/i386/pc.c                      | 10 ++++++----
 hw/intc/ioapic_common.c           | 15 +++++++++++----
 include/exec/memory.h             | 10 ++++++++++
 include/hw/i386/icc_bus.h         |  1 +
 include/hw/i386/ioapic_internal.h |  6 +++---
 memory.c                          | 11 +++++++++++
 8 files changed, 49 insertions(+), 12 deletions(-)

diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
index c499f09..ab9623d 100644
--- a/hw/cpu/icc_bus.c
+++ b/hw/cpu/icc_bus.c
@@ -60,6 +60,7 @@ static const TypeInfo icc_device_info = {
 typedef struct ICCBridgeState {
     SysBusDevice busdev;
     MemoryRegion apic_container;
+    MemoryRegion ioapic_container;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
@@ -79,6 +80,11 @@ static void icc_bridge_initfn(Object *obj)
                        APIC_SPACE_SIZE);
     sysbus_init_mmio(sb, &s->apic_container);
     ibus->apic_address_space = &s->apic_container;
+
+    /* must be second registered region, board maps it by 1 index */
+    memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
+    sysbus_init_mmio(sb, &s->ioapic_container);
+    ibus->ioapic_address_space = &s->ioapic_container;
 }
 
 static const TypeInfo icc_bridge_info = {
diff --git a/hw/i386/kvm/ioapic.c b/hw/i386/kvm/ioapic.c
index a3bd519..b80d41a 100644
--- a/hw/i386/kvm/ioapic.c
+++ b/hw/i386/kvm/ioapic.c
@@ -96,7 +96,7 @@ static void kvm_ioapic_put(IOAPICCommonState *s)
 
     kioapic->id = s->id;
     kioapic->ioregsel = s->ioregsel;
-    kioapic->base_address = s->busdev.mmio[0].addr;
+    kioapic->base_address = memory_region_get_address(&s->io_memory);
     kioapic->irr = s->irr;
     for (i = 0; i < IOAPIC_NUM_PINS; i++) {
         kioapic->redirtbl[i].bits = s->ioredtbl[i];
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 1c7b3c2..6d5e164 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1197,20 +1197,22 @@ void ioapic_init_gsi(GSIState *gsi_state, const char *parent_name)
     DeviceState *dev;
     SysBusDevice *d;
     unsigned int i;
+    BusState *b = BUS(object_resolve_path_type("icc-bus", TYPE_ICC_BUS, NULL));
 
     if (kvm_irqchip_in_kernel()) {
-        dev = qdev_create(NULL, "kvm-ioapic");
+        dev = qdev_create(b, "kvm-ioapic");
     } else {
-        dev = qdev_create(NULL, "ioapic");
+        dev = qdev_create(b, "ioapic");
     }
     if (parent_name) {
         object_property_add_child(object_resolve_path(parent_name, NULL),
                                   "ioapic", OBJECT(dev), NULL);
     }
     qdev_init_nofail(dev);
-    d = SYS_BUS_DEVICE(dev);
-    sysbus_mmio_map(d, 0, IO_APIC_DEFAULT_ADDRESS);
 
+    d = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge", TYPE_ICC_BRIDGE,
+                                                NULL));
+    sysbus_mmio_map(d, 1, IO_APIC_DEFAULT_ADDRESS);
     for (i = 0; i < IOAPIC_NUM_PINS; i++) {
         gsi_state->ioapic_irq[i] = qdev_get_gpio_in(dev, i);
     }
diff --git a/hw/intc/ioapic_common.c b/hw/intc/ioapic_common.c
index 5c5bb3c..72c48a8 100644
--- a/hw/intc/ioapic_common.c
+++ b/hw/intc/ioapic_common.c
@@ -57,11 +57,13 @@ static int ioapic_dispatch_post_load(void *opaque, int version_id)
     return 0;
 }
 
-static int ioapic_init_common(SysBusDevice *dev)
+static int ioapic_init_common(ICCDevice *dev)
 {
     IOAPICCommonState *s = IOAPIC_COMMON(dev);
+    DeviceState *d = DEVICE(dev);
     IOAPICCommonClass *info;
     static int ioapic_no;
+    static bool mmio_registered;
 
     if (ioapic_no >= MAX_IOAPICS) {
         return -1;
@@ -70,7 +72,12 @@ static int ioapic_init_common(SysBusDevice *dev)
     info = IOAPIC_COMMON_GET_CLASS(s);
     info->init(s, ioapic_no);
 
-    sysbus_init_mmio(&s->busdev, &s->io_memory);
+    if (!mmio_registered) {
+        MemoryRegion *as = ICC_BUS(d->parent_bus)->ioapic_address_space;
+        memory_region_add_subregion(as, 0, &s->io_memory);
+        mmio_registered = true;
+    }
+
     ioapic_no++;
 
     return 0;
@@ -95,7 +102,7 @@ static const VMStateDescription vmstate_ioapic_common = {
 
 static void ioapic_common_class_init(ObjectClass *klass, void *data)
 {
-    SysBusDeviceClass *sc = SYS_BUS_DEVICE_CLASS(klass);
+    ICCDeviceClass *sc = ICC_DEVICE_CLASS(klass);
     DeviceClass *dc = DEVICE_CLASS(klass);
 
     sc->init = ioapic_init_common;
@@ -105,7 +112,7 @@ static void ioapic_common_class_init(ObjectClass *klass, void *data)
 
 static const TypeInfo ioapic_common_type = {
     .name = TYPE_IOAPIC_COMMON,
-    .parent = TYPE_SYS_BUS_DEVICE,
+    .parent = TYPE_ICC_DEVICE,
     .instance_size = sizeof(IOAPICCommonState),
     .class_size = sizeof(IOAPICCommonClass),
     .class_init = ioapic_common_class_init,
diff --git a/include/exec/memory.h b/include/exec/memory.h
index 2322732..826e401 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -711,6 +711,16 @@ void memory_region_set_enabled(MemoryRegion *mr, bool enabled);
 void memory_region_set_address(MemoryRegion *mr, hwaddr addr);
 
 /*
+ * memory_region_get_address: get current the address of a region
+ *
+ * Returns the absolute address of a region.
+ * May be used on regions that are currently part of a memory hierarchy.
+ *
+ * @mr: the region being queried
+ */
+hwaddr memory_region_get_address(MemoryRegion *mr);
+
+/*
  * memory_region_set_alias_offset: dynamically update a memory alias's offset
  *
  * Dynamically updates the offset into the target region that an alias points
diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
index 2c8f78f..69a0278 100644
--- a/include/hw/i386/icc_bus.h
+++ b/include/hw/i386/icc_bus.h
@@ -28,6 +28,7 @@
 typedef struct ICCBus {
     BusState qbus;
     MemoryRegion *apic_address_space;
+    MemoryRegion *ioapic_address_space;
 } ICCBus;
 #define ICC_BUS(obj) OBJECT_CHECK(ICCBus, (obj), TYPE_ICC_BUS)
 
diff --git a/include/hw/i386/ioapic_internal.h b/include/hw/i386/ioapic_internal.h
index 25576c8..4768c5b 100644
--- a/include/hw/i386/ioapic_internal.h
+++ b/include/hw/i386/ioapic_internal.h
@@ -24,7 +24,7 @@
 
 #include "hw/hw.h"
 #include "exec/memory.h"
-#include "hw/sysbus.h"
+#include "hw/i386/icc_bus.h"
 
 #define MAX_IOAPICS                     1
 
@@ -82,14 +82,14 @@ typedef struct IOAPICCommonState IOAPICCommonState;
      OBJECT_GET_CLASS(IOAPICCommonClass, (obj), TYPE_IOAPIC_COMMON)
 
 typedef struct IOAPICCommonClass {
-    SysBusDeviceClass parent_class;
+    ICCDeviceClass parent_class;
     void (*init)(IOAPICCommonState *s, int instance_no);
     void (*pre_save)(IOAPICCommonState *s);
     void (*post_load)(IOAPICCommonState *s);
 } IOAPICCommonClass;
 
 struct IOAPICCommonState {
-    SysBusDevice busdev;
+    ICCDevice busdev;
     MemoryRegion io_memory;
     uint8_t id;
     uint8_t ioregsel;
diff --git a/memory.c b/memory.c
index 75ca281..0651050 100644
--- a/memory.c
+++ b/memory.c
@@ -1413,6 +1413,17 @@ void memory_region_set_address(MemoryRegion *mr, hwaddr addr)
     memory_region_transaction_commit();
 }
 
+hwaddr memory_region_get_address(MemoryRegion *mr)
+{
+    hwaddr addr = mr->addr;
+
+    while (mr->parent) {
+        mr = mr->parent;
+        addr += mr->addr;
+    }
+    return addr;
+}
+
 void memory_region_set_alias_offset(MemoryRegion *mr, hwaddr offset)
 {
     assert(mr->alias);
-- 
1.8.2

[Qemu-devel] [PATCH 17/19] qdev: set device's parent before calling realize() down inheritance chain.

[Igor Mammedov]

Currently device_set_realized() sets parent only after device was realized,
But qdev_device_add() sets it before device is realized.
Make behavior consistent and alter device_set_realized() to behave like
qdev_device_add().

It will allow to set link<> properties in realize() method in classes
inherited from DEVICE.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 * Usage examples
    - conversion to CPU back-link property in APIC
    - pre-allocate link<CPU[n]> at board level, and populate them
      at realize time when CPUs are created.
---
 hw/core/qdev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index e2bb37d..b9bd6f5 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -671,10 +671,6 @@ static void device_set_realized(Object *obj, bool value, Error **err)
     Error *local_err = NULL;
 
     if (value && !dev->realized) {
-        if (dc->realize) {
-            dc->realize(dev, &local_err);
-        }
-
         if (!obj->parent && local_err == NULL) {
             static int unattached_count;
             gchar *name = g_strdup_printf("device[%d]", unattached_count++);
@@ -685,6 +681,10 @@ static void device_set_realized(Object *obj, bool value, Error **err)
             g_free(name);
         }
 
+        if (dc->realize) {
+            dc->realize(dev, &local_err);
+        }
+
         if (qdev_get_vmsd(dev) && local_err == NULL) {
             vmstate_register_with_alias_id(dev, -1, qdev_get_vmsd(dev), dev,
                                            dev->instance_id_alias,
-- 
1.8.2

[Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Igor Mammedov]

... and leave links for not present CPUs empty.

It will allow users to query for possible APIC IDs and use them
with cpu-add QMP command.

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v2:
 * s/get_firmware_id/get_arch_id/ due to rebase
 * rename cpu_add_notifier to cpu_added_notifier &
   icc_bridge_cpu_add_req -> icc_bridge_cpued_add_req
---
 hw/cpu/icc_bus.c          | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 hw/i386/pc.c              |  9 +++++++--
 include/hw/i386/icc_bus.h |  2 ++
 3 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
index ab9623d..5c0b9d4 100644
--- a/hw/cpu/icc_bus.c
+++ b/hw/cpu/icc_bus.c
@@ -18,6 +18,7 @@
  */
 #include "hw/i386/icc_bus.h"
 #include "hw/sysbus.h"
+#include "sysemu/sysemu.h"
 
 static void icc_bus_initfn(Object *obj)
 {
@@ -61,15 +62,39 @@ typedef struct ICCBridgeState {
     SysBusDevice busdev;
     MemoryRegion apic_container;
     MemoryRegion ioapic_container;
+    Notifier cpu_added_notifier;
+    Object **links;
 } ICCBridgeState;
 #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
 
 
+void icc_bridge_set_cpu_link(Object *bridge, Object *cpu_obj)
+{
+    gchar *name;
+    Error *error = NULL;
+    CPUState *cpu = CPU(cpu_obj);
+    int64_t id = CPU_GET_CLASS(cpu)->get_arch_id(cpu);
+
+    name = g_strdup_printf("cpu[%" PRIu32 "]", x86_cpu_apic_id_from_index(id));
+    object_property_set_link(bridge, cpu_obj, name, &error);
+    g_free(name);
+
+    g_assert(error == NULL);
+}
+
+static void icc_bridge_cpu_added_req(Notifier *n, void *opaque)
+{
+    ICCBridgeState *s = container_of(n, ICCBridgeState, cpu_added_notifier);
+
+    icc_bridge_set_cpu_link(OBJECT(s), OBJECT(opaque));
+}
+
 static void icc_bridge_initfn(Object *obj)
 {
     ICCBridgeState *s = ICC_BRIGDE(obj);
     SysBusDevice *sb = SYS_BUS_DEVICE(obj);
     ICCBus *ibus;
+    int i;
 
     ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
 
@@ -85,12 +110,33 @@ static void icc_bridge_initfn(Object *obj)
     memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
     sysbus_init_mmio(sb, &s->ioapic_container);
     ibus->ioapic_address_space = &s->ioapic_container;
+
+    s->links = g_malloc0(sizeof(Object *) * max_cpus);
+    for (i = 0; i < max_cpus; i++) {
+        gchar *cpu_name;
+
+        cpu_name = g_strdup_printf("cpu[%" PRIu32 "]",
+                                   x86_cpu_apic_id_from_index(i));
+        object_property_add_link(obj, cpu_name, TYPE_CPU, &s->links[i], NULL);
+        g_free(cpu_name);
+    }
+
+    s->cpu_added_notifier.notify = icc_bridge_cpu_added_req;
+    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
+}
+
+static void icc_bridge_fini(Object *obj)
+{
+    ICCBridgeState *s = ICC_BRIGDE(obj);
+
+    g_free(s->links);
 }
 
 static const TypeInfo icc_bridge_info = {
     .name  = "icc-bridge",
     .parent = TYPE_SYS_BUS_DEVICE,
     .instance_init  = icc_bridge_initfn,
+    .instance_finalize  = icc_bridge_fini,
     .instance_size  = sizeof(ICCBridgeState),
 };
 
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 6d5e164..ada235c 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -870,7 +870,8 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
     }
 }
 
-static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
+static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
+                          SysBusDevice *icc_bridge, Error **errp)
 {
     X86CPU *cpu;
 
@@ -882,6 +883,10 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
     object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
     object_property_set_bool(OBJECT(cpu), true, "realized", errp);
 
+    if (icc_bridge != NULL) {
+        icc_bridge_set_cpu_link(OBJECT(icc_bridge), OBJECT(cpu));
+    }
+
     if (error_is_set(errp)) {
         if (cpu != NULL) {
             object_unref(OBJECT(cpu));
@@ -911,7 +916,7 @@ void pc_cpus_init(const char *cpu_model)
                                                  TYPE_ICC_BRIDGE, NULL));
 
     for (i = 0; i < smp_cpus; i++) {
-        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
+        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), ib, &error);
         if (error) {
             fprintf(stderr, "%s\n", error_get_pretty(error));
             error_free(error);
diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
index 69a0278..bc31cd9 100644
--- a/include/hw/i386/icc_bus.h
+++ b/include/hw/i386/icc_bus.h
@@ -49,5 +49,7 @@ typedef struct ICCDeviceClass {
 
 #define TYPE_ICC_BRIDGE "icc-bridge"
 
+void icc_bridge_set_cpu_link(Object *bridge, Object *cpu);
+
 #endif /* CONFIG_USER_ONLY */
 #endif
-- 
1.8.2

[Qemu-devel] [PATCH 19/19] add cpu-add qmp command and implement CPU hot-add for target-i386

[Igor Mammedov]

... via current_machine->cpu_hot_add() hook called by cpu-set QMP command,
for x86 target.

cpu-add command usage will be descrbed at:
http://wiki.qemu.org/Features/CPUHotplug

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
v4:
  * merge "qmp: add cpu-add qmp command" & "target-i386: implement CPU hot-add" patches
  * move notifier call to CPUCLass.realize()
  * add hook cpu_hot_add to QEMUMachine
  * make QEMUMachineInitArgs global and keep default cpu_model there

v3:
  * it appears that 'online/offline' in cpu-set are confusing people
    with what command actually does and users might have to distinguish
    if 'offline' is not implemented by parsing error message. To simplify
    things replace cpu-set with cpu-add command to show more clear what
    command does and just add cpu-del when CPU remove is implemented.

v2:
  * s/cpu_set/cpu-set/
  * qmp doc style fix
  * use bool type instead of opencodding online/offline string
     suggested-by: Eric Blake <eblake@redhat.com>
---
 hw/i386/pc.c        | 20 ++++++++++++++++++++
 include/hw/boards.h |  3 +++
 qapi-schema.json    | 11 +++++++++++
 qmp-commands.hx     | 23 +++++++++++++++++++++++
 qmp.c               | 10 ++++++++++
 vl.c                |  6 +++++-
 6 files changed, 72 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index ada235c..ba93180 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -53,6 +53,7 @@
 #include "qemu/bitmap.h"
 #include "qemu/config-file.h"
 #include "hw/i386/icc_bus.h"
+#include "hw/boards.h"
 
 /* debug PC/ISA interrupts */
 //#define DEBUG_IRQ
@@ -896,6 +897,23 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
     return cpu;
 }
 
+static void do_cpu_hot_add(const int64_t id, Error **errp)
+{
+    if (cpu_exists(id)) {
+        error_setg(errp, "Unable to add CPU with APIC ID: 0x%" PRIx64
+                   ", it already exists", id);
+        return;
+    }
+
+    if (id >= pc_apic_id_limit(max_cpus)) {
+        error_setg(errp, "Unable to add CPU with APIC ID: 0x%" PRIx64
+                   ", max allowed: 0x%x", id, pc_apic_id_limit(max_cpus) - 1);
+        return;
+    }
+
+    pc_new_cpu(machine_args->cpu_model, id, NULL, errp);
+}
+
 void pc_cpus_init(const char *cpu_model)
 {
     int i;
@@ -910,7 +928,9 @@ void pc_cpus_init(const char *cpu_model)
 #else
         cpu_model = "qemu32";
 #endif
+        machine_args->cpu_model = cpu_model;
     }
+    current_machine->hot_add_cpu = do_cpu_hot_add;
 
     ib = SYS_BUS_DEVICE(object_resolve_path_type("icc-bridge",
                                                  TYPE_ICC_BRIDGE, NULL));
diff --git a/include/hw/boards.h b/include/hw/boards.h
index 425bdc7..de8f92a 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -18,6 +18,8 @@ typedef struct QEMUMachineInitArgs {
     const char *cpu_model;
 } QEMUMachineInitArgs;
 
+extern QEMUMachineInitArgs *machine_args;
+
 typedef void QEMUMachineInitFunc(QEMUMachineInitArgs *args);
 
 typedef void QEMUMachineResetFunc(void);
@@ -43,6 +45,7 @@ typedef struct QEMUMachine {
     GlobalProperty *compat_props;
     struct QEMUMachine *next;
     const char *hw_version;
+    void (*hot_add_cpu)(const int64_t id, Error **errp);
 } QEMUMachine;
 
 int qemu_register_machine(QEMUMachine *m);
diff --git a/qapi-schema.json b/qapi-schema.json
index db542f6..a760ed5 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1387,6 +1387,17 @@
 { 'command': 'cpu', 'data': {'index': 'int'} }
 
 ##
+# @cpu-add
+#
+# Adds CPU with specified id
+#
+# @id: cpu id of CPU to be created
+#
+# Returns: Nothing on success
+##
+{ 'command': 'cpu-add', 'data': {'id': 'int'} }
+
+##
 # @memsave:
 #
 # Save a portion of guest memory to a file.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 1e0e11e..79b2ba7 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -407,6 +407,29 @@ Example:
 EQMP
 
     {
+        .name       = "cpu-add",
+        .args_type  = "id:i",
+        .mhandler.cmd_new = qmp_marshal_input_cpu_add,
+    },
+
+SQMP
+cpu-add
+-------
+
+Adds virtual cpu
+
+Arguments:
+
+- "id": cpu id (json-int)
+
+Example:
+
+-> { "execute": "cpu-add", "arguments": { "id": 2 } }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "memsave",
         .args_type  = "val:l,size:i,filename:s,cpu:i?",
         .mhandler.cmd_new = qmp_marshal_input_memsave,
diff --git a/qmp.c b/qmp.c
index 55b056b..8a9fd9e 100644
--- a/qmp.c
+++ b/qmp.c
@@ -24,6 +24,7 @@
 #include "hw/qdev.h"
 #include "sysemu/blockdev.h"
 #include "qom/qom-qobject.h"
+#include "hw/boards.h"
 
 NameInfo *qmp_query_name(Error **errp)
 {
@@ -108,6 +109,15 @@ void qmp_cpu(int64_t index, Error **errp)
     /* Just do nothing */
 }
 
+void qmp_cpu_add(int64_t id, Error **errp)
+{
+    if (current_machine->hot_add_cpu) {
+        current_machine->hot_add_cpu(id, errp);
+    } else {
+        error_setg(errp, "Not supported");
+    }
+}
+
 #ifndef CONFIG_VNC
 /* If VNC support is enabled, the "true" query-vnc command is
    defined in the VNC subsystem */
diff --git a/vl.c b/vl.c
index 21952c3..cdce08d 100644
--- a/vl.c
+++ b/vl.c
@@ -179,6 +179,8 @@ int main(int argc, char **argv)
 #define MAX_VIRTIO_CONSOLES 1
 #define MAX_SCLP_CONSOLES 1
 
+QEMUMachineInitArgs *machine_args;
+
 static const char *data_dir[16];
 static int data_dir_idx;
 const char *bios_name = NULL;
@@ -4310,13 +4312,15 @@ int main(int argc, char **argv, char **envp)
                                  .kernel_cmdline = kernel_cmdline,
                                  .initrd_filename = initrd_filename,
                                  .cpu_model = cpu_model };
+    machine_args = &args;
+    current_machine = machine;
+
     machine->init(&args);
 
     cpu_synchronize_all_post_init();
 
     set_numa_modes();
 
-    current_machine = machine;
 
     /* init USB devices */
     if (usb_enabled(false)) {
-- 
1.8.2

Re: [Qemu-devel] [PATCH 01/19] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:40PM +0200, Igor Mammedov wrote:
> Move CPU creation and features parsing into a separate cpu_x86_create()
> function, so that board would be able to set board specific CPU
> properties before CPU is realized.
> 
> Keep cpu_x86_init() for compatibility with the code that uses cpu_init()
> and doesn't need to modify CPU properties.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Nice, much simpler to review than v1.  :-)

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

> ---
> v2:
>   * use error_propagate() to rise error up stack
> ---
>  target-i386/cpu.c | 18 +++++++++++++++---
>  target-i386/cpu.h |  1 +
>  2 files changed, 16 insertions(+), 3 deletions(-)
> 
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index f1ccc72..c28436c 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -1564,7 +1564,7 @@ static void cpu_x86_register(X86CPU *cpu, const char *name, Error **errp)
>      object_property_set_str(OBJECT(cpu), def->model_id, "model-id", errp);
>  }
>  
> -X86CPU *cpu_x86_init(const char *cpu_model)
> +X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
>  {
>      X86CPU *cpu = NULL;
>      CPUX86State *env;
> @@ -1594,13 +1594,25 @@ X86CPU *cpu_x86_init(const char *cpu_model)
>          goto out;
>      }
>  
> -    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
> +out:
> +    error_propagate(errp, error);
> +    g_strfreev(model_pieces);
> +    return cpu;
> +}
> +
> +X86CPU *cpu_x86_init(const char *cpu_model)
> +{
> +    Error *error = NULL;
> +    X86CPU *cpu;
> +
> +    cpu = cpu_x86_create(cpu_model, &error);
>      if (error) {
>          goto out;
>      }
>  
> +    object_property_set_bool(OBJECT(cpu), true, "realized", &error);
> +
>  out:
> -    g_strfreev(model_pieces);
>      if (error) {
>          fprintf(stderr, "%s\n", error_get_pretty(error));
>          error_free(error);
> diff --git a/target-i386/cpu.h b/target-i386/cpu.h
> index 2b4e319..cf1b05c 100644
> --- a/target-i386/cpu.h
> +++ b/target-i386/cpu.h
> @@ -896,6 +896,7 @@ typedef struct CPUX86State {
>  #include "cpu-qom.h"
>  
>  X86CPU *cpu_x86_init(const char *cpu_model);
> +X86CPU *cpu_x86_create(const char *cpu_model, Error **errp);
>  int cpu_x86_exec(CPUX86State *s);
>  void x86_cpu_list(FILE *f, fprintf_function cpu_fprintf);
>  void x86_cpudef_setup(void);
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> ... and leave links for not present CPUs empty.
> 
> It will allow users to query for possible APIC IDs and use them
> with cpu-add QMP command.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

I don't see anything wrong with having icc-bridge links as well, but I
would really like to have a target-independent namespace with links,
that could be used to query for the available/valid CPU IDs for cpu-add
commands instead of icc-bridge. The IDs on that namespace could be
considered completely opaque.


> ---
> v2:
>  * s/get_firmware_id/get_arch_id/ due to rebase
>  * rename cpu_add_notifier to cpu_added_notifier &
>    icc_bridge_cpu_add_req -> icc_bridge_cpued_add_req
> ---
>  hw/cpu/icc_bus.c          | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>  hw/i386/pc.c              |  9 +++++++--
>  include/hw/i386/icc_bus.h |  2 ++
>  3 files changed, 55 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
> index ab9623d..5c0b9d4 100644
> --- a/hw/cpu/icc_bus.c
> +++ b/hw/cpu/icc_bus.c
> @@ -18,6 +18,7 @@
>   */
>  #include "hw/i386/icc_bus.h"
>  #include "hw/sysbus.h"
> +#include "sysemu/sysemu.h"
>  
>  static void icc_bus_initfn(Object *obj)
>  {
> @@ -61,15 +62,39 @@ typedef struct ICCBridgeState {
>      SysBusDevice busdev;
>      MemoryRegion apic_container;
>      MemoryRegion ioapic_container;
> +    Notifier cpu_added_notifier;
> +    Object **links;
>  } ICCBridgeState;
>  #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
>  
>  
> +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu_obj)
> +{
> +    gchar *name;
> +    Error *error = NULL;
> +    CPUState *cpu = CPU(cpu_obj);
> +    int64_t id = CPU_GET_CLASS(cpu)->get_arch_id(cpu);
> +
> +    name = g_strdup_printf("cpu[%" PRIu32 "]", x86_cpu_apic_id_from_index(id));
> +    object_property_set_link(bridge, cpu_obj, name, &error);
> +    g_free(name);
> +
> +    g_assert(error == NULL);
> +}
> +
> +static void icc_bridge_cpu_added_req(Notifier *n, void *opaque)
> +{
> +    ICCBridgeState *s = container_of(n, ICCBridgeState, cpu_added_notifier);
> +
> +    icc_bridge_set_cpu_link(OBJECT(s), OBJECT(opaque));
> +}
> +
>  static void icc_bridge_initfn(Object *obj)
>  {
>      ICCBridgeState *s = ICC_BRIGDE(obj);
>      SysBusDevice *sb = SYS_BUS_DEVICE(obj);
>      ICCBus *ibus;
> +    int i;
>  
>      ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
>  
> @@ -85,12 +110,33 @@ static void icc_bridge_initfn(Object *obj)
>      memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
>      sysbus_init_mmio(sb, &s->ioapic_container);
>      ibus->ioapic_address_space = &s->ioapic_container;
> +
> +    s->links = g_malloc0(sizeof(Object *) * max_cpus);
> +    for (i = 0; i < max_cpus; i++) {
> +        gchar *cpu_name;
> +
> +        cpu_name = g_strdup_printf("cpu[%" PRIu32 "]",
> +                                   x86_cpu_apic_id_from_index(i));
> +        object_property_add_link(obj, cpu_name, TYPE_CPU, &s->links[i], NULL);
> +        g_free(cpu_name);
> +    }
> +
> +    s->cpu_added_notifier.notify = icc_bridge_cpu_added_req;
> +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> +}
> +
> +static void icc_bridge_fini(Object *obj)
> +{
> +    ICCBridgeState *s = ICC_BRIGDE(obj);
> +
> +    g_free(s->links);
>  }
>  
>  static const TypeInfo icc_bridge_info = {
>      .name  = "icc-bridge",
>      .parent = TYPE_SYS_BUS_DEVICE,
>      .instance_init  = icc_bridge_initfn,
> +    .instance_finalize  = icc_bridge_fini,
>      .instance_size  = sizeof(ICCBridgeState),
>  };
>  
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 6d5e164..ada235c 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -870,7 +870,8 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
>      }
>  }
>  
> -static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> +static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
> +                          SysBusDevice *icc_bridge, Error **errp)
>  {
>      X86CPU *cpu;
>  
> @@ -882,6 +883,10 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
>      object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
>      object_property_set_bool(OBJECT(cpu), true, "realized", errp);
>  
> +    if (icc_bridge != NULL) {
> +        icc_bridge_set_cpu_link(OBJECT(icc_bridge), OBJECT(cpu));
> +    }
> +
>      if (error_is_set(errp)) {
>          if (cpu != NULL) {
>              object_unref(OBJECT(cpu));
> @@ -911,7 +916,7 @@ void pc_cpus_init(const char *cpu_model)
>                                                   TYPE_ICC_BRIDGE, NULL));
>  
>      for (i = 0; i < smp_cpus; i++) {
> -        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), ib, &error);
>          if (error) {
>              fprintf(stderr, "%s\n", error_get_pretty(error));
>              error_free(error);
> diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
> index 69a0278..bc31cd9 100644
> --- a/include/hw/i386/icc_bus.h
> +++ b/include/hw/i386/icc_bus.h
> @@ -49,5 +49,7 @@ typedef struct ICCDeviceClass {
>  
>  #define TYPE_ICC_BRIDGE "icc-bridge"
>  
> +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu);
> +
>  #endif /* CONFIG_USER_ONLY */
>  #endif
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 03/19] cpu: make kvm-stub.o a part of CPU library

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:42PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  Makefile.target      | 13 +++++++------
>  include/hw/pci/msi.h |  2 ++
>  include/sysemu/kvm.h |  4 ++--
>  kvm-stub.c           |  2 ++
>  4 files changed, 13 insertions(+), 8 deletions(-)
> 
> diff --git a/Makefile.target b/Makefile.target
> index 2bd6d14..18f348c 100644
> --- a/Makefile.target
> +++ b/Makefile.target
> @@ -64,6 +64,12 @@ all: $(PROGS) stap
>  # Dummy command so that make thinks it has done something
>  	@true
>  
> +CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
> +CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
> +CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
> +CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
> +CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
> +
>  #########################################################
>  # cpu emulator library
>  obj-y = exec.o translate-all.o cpu-exec.o
> @@ -74,6 +80,7 @@ obj-y += fpu/softfloat.o
>  obj-y += target-$(TARGET_BASE_ARCH)/
>  obj-y += disas.o
>  obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
> +obj-$(CONFIG_NO_KVM) += kvm-stub.o

The kvm-stub.o line at the "System emulator target" section is redundant
now, isn't it? Any specific reason you didn't remove it?

>  
>  #########################################################
>  # Linux user emulator target
> @@ -102,12 +109,6 @@ endif #CONFIG_BSD_USER
>  #########################################################
>  # System emulator target
>  ifdef CONFIG_SOFTMMU
> -CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
> -CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
> -CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
> -CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
> -CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
> -
>  obj-y += arch_init.o cpus.o monitor.o gdbstub.o balloon.o ioport.o
>  obj-y += qtest.o
>  obj-y += hw/
> diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
> index 81a3848..d4d0450 100644
> --- a/include/hw/pci/msi.h
> +++ b/include/hw/pci/msi.h
> @@ -21,6 +21,7 @@
>  #ifndef QEMU_MSI_H
>  #define QEMU_MSI_H
>  
> +#ifndef CONFIG_USER_ONLY

I would simply #ifdef the "#include pci/msi.h" line on kvm-stub.c
instead. But I guess it doesn't hurt to have the #ifdef here.

>  #include "qemu-common.h"
>  #include "hw/pci/pci.h"
>  
> @@ -47,4 +48,5 @@ static inline bool msi_present(const PCIDevice *dev)
>      return dev->cap_present & QEMU_PCI_CAP_MSI;
>  }
>  
> +#endif
>  #endif /* QEMU_MSI_H */
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 495e6f8..93cef28 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
>  #if !defined(CONFIG_USER_ONLY)
>  void *kvm_vmalloc(ram_addr_t size);
>  void *kvm_arch_vmalloc(ram_addr_t size);
> -void kvm_setup_guest_memory(void *start, size_t size);
> +#endif
>  
> +void kvm_setup_guest_memory(void *start, size_t size);
>  void kvm_flush_coalesced_mmio_buffer(void);
> -#endif
>  
>  int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
>                            target_ulong len, int type);
> diff --git a/kvm-stub.c b/kvm-stub.c
> index 82875dd..b34064a 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -122,6 +122,7 @@ int kvm_on_sigbus(int code, void *addr)
>      return 1;
>  }
>  
> +#ifndef CONFIG_USER_ONLY
>  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
>  {
>      return -ENOSYS;
> @@ -145,3 +146,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
>  {
>      return -ENOSYS;
>  }
> +#endif

What about making MSIMessage definition available on CONFIG_USER_ONLY,
so we don't even need to #ifdef this part?

-- 
Eduardo

Re: [Qemu-devel] [PATCH 04/19] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:43PM +0200, Igor Mammedov wrote:
> ... to synchronize CPU state to KVM
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * linking kvm-stub.o to *-user target moved in separate patch
> ---
>  include/sysemu/kvm.h | 18 ++++++++++--------
>  kvm-all.c            |  1 +
>  kvm-stub.c           |  1 +
>  qom/cpu.c            |  4 ++++
>  vl.c                 |  1 -
>  5 files changed, 16 insertions(+), 9 deletions(-)
> 
> diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> index 93cef28..8fb2489 100644
> --- a/include/sysemu/kvm.h
> +++ b/include/sysemu/kvm.h
> @@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
>  uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
>                                        uint32_t index, int reg);
>  void kvm_cpu_synchronize_state(CPUArchState *env);
> -void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> -void kvm_cpu_synchronize_post_init(CPUState *cpu);
>  
>  /* generic hooks - to be moved/refactored once there are more users */
>  
> @@ -262,6 +260,16 @@ static inline void cpu_synchronize_state(CPUArchState *env)
>      }
>  }
>  
> +#if !defined(CONFIG_USER_ONLY)
> +int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> +                                       hwaddr *phys_addr);
> +#endif
> +
> +#endif /* NEED_CPU_H */
> +
> +void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> +void kvm_cpu_synchronize_post_init(CPUState *cpu);
> +
>  static inline void cpu_synchronize_post_reset(CPUState *cpu)
>  {
>      if (kvm_enabled()) {
> @@ -277,12 +285,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
>  }
>  
>  
> -#if !defined(CONFIG_USER_ONLY)
> -int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> -                                       hwaddr *phys_addr);
> -#endif
> -
> -#endif
>  int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool assign,
>                             uint32_t size);
>  
> diff --git a/kvm-all.c b/kvm-all.c
> index fc4e17c..1d17128 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
>  bool kvm_irqfds_allowed;
>  bool kvm_msi_via_irqfd_allowed;
>  bool kvm_gsi_routing_allowed;
> +bool kvm_allowed;
>  
>  static const KVMCapabilityInfo kvm_required_capabilites[] = {
>      KVM_CAP_INFO(USER_MEMORY),
> diff --git a/kvm-stub.c b/kvm-stub.c
> index b34064a..8cb81c4 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -23,6 +23,7 @@ bool kvm_async_interrupts_allowed;
>  bool kvm_irqfds_allowed;
>  bool kvm_msi_via_irqfd_allowed;
>  bool kvm_gsi_routing_allowed;
> +bool kvm_allowed;
>  
>  int kvm_init_vcpu(CPUState *cpu)
>  {
> diff --git a/qom/cpu.c b/qom/cpu.c
> index e242dcb..0c76712 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -20,6 +20,7 @@
>  
>  #include "qom/cpu.h"
>  #include "qemu-common.h"
> +#include "sysemu/kvm.h"
>  
>  void cpu_reset_interrupt(CPUState *cpu, int mask)
>  {
> @@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
>  
>  static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>  {
> +    if (dev->hotplugged) {
> +        cpu_synchronize_post_init(CPU(dev));
> +    }

For reference: non-hotplugged CPUs have cpu_synchronize_post_init()
called from cpu_synchronize_all_post_init(), that is called from main(),
just after the machine->init() call.

That said, have you considered changing the code so that's possible to
call cpu_synchronize_post_init() for all CPUs, so we could kill the
cpu_synchronize_all_post_init() call entirely?

Maybe the solution for this is to realize non-hotplugged CPUs later,
instead of realizing them very early inside the machine init function.
If doing is not possible or not enough, we could use VM state notifiers
to watch for the right moment to call cpu_synchronize_post_init().

>  }
>  
>  static void cpu_class_init(ObjectClass *klass, void *data)
> diff --git a/vl.c b/vl.c
> index d694a90..21952c3 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -267,7 +267,6 @@ static NotifierList machine_init_done_notifiers =
>      NOTIFIER_LIST_INITIALIZER(machine_init_done_notifiers);
>  
>  static bool tcg_allowed = true;
> -bool kvm_allowed;
>  bool xen_allowed;
>  uint32_t xen_domid;
>  enum xen_mode xen_mode = XEN_EMULATE;
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 05/19] cpu: resume CPU from CPUClass.cpu_common_realizefn() when it is hot-plugged

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:44PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  cpus.c                | 11 ++++++++---
>  include/sysemu/cpus.h |  3 +++
>  qom/cpu.c             |  2 ++
>  stubs/Makefile.objs   |  1 +
>  stubs/resume_vcpu.c   |  6 ++++++
>  5 files changed, 20 insertions(+), 3 deletions(-)
>  create mode 100644 stubs/resume_vcpu.c
> 
> diff --git a/cpus.c b/cpus.c
> index 9b9a32f..f12624d 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
>      }
>  }
>  
> +void resume_vcpu(CPUState *cpu)
> +{
> +    cpu->stop = false;
> +    cpu->stopped = false;
> +    qemu_cpu_kick(cpu);
> +}
> +
>  void resume_all_vcpus(void)
>  {
>      CPUArchState *penv = first_cpu;
> @@ -980,9 +987,7 @@ void resume_all_vcpus(void)
>      qemu_clock_enable(vm_clock, true);
>      while (penv) {
>          CPUState *pcpu = ENV_GET_CPU(penv);
> -        pcpu->stop = false;
> -        pcpu->stopped = false;
> -        qemu_cpu_kick(pcpu);
> +        resume_vcpu(pcpu);

If you include just the above chunks in a separate patch, I would
promptly send a Reviewed-by line for that.

>          penv = penv->next_cpu;
>      }
>  }
> diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
> index 6502488..9437df5 100644
> --- a/include/sysemu/cpus.h
> +++ b/include/sysemu/cpus.h
> @@ -1,11 +1,14 @@
>  #ifndef QEMU_CPUS_H
>  #define QEMU_CPUS_H
>  
> +#include "qom/cpu.h"
> +
>  /* cpus.c */
>  void qemu_init_cpu_loop(void);
>  void resume_all_vcpus(void);
>  void pause_all_vcpus(void);
>  void cpu_stop_current(void);
> +void resume_vcpu(CPUState *cpu);
>  
>  void cpu_synchronize_all_states(void);
>  void cpu_synchronize_all_post_reset(void);
> diff --git a/qom/cpu.c b/qom/cpu.c
> index 0c76712..2869cfe 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -21,6 +21,7 @@
>  #include "qom/cpu.h"
>  #include "qemu-common.h"
>  #include "sysemu/kvm.h"
> +#include "sysemu/cpus.h"
>  
>  void cpu_reset_interrupt(CPUState *cpu, int mask)
>  {
> @@ -60,6 +61,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>  {
>      if (dev->hotplugged) {
>          cpu_synchronize_post_init(CPU(dev));
> +        resume_vcpu(CPU(dev));

Simlar question to the one I had about cpu_synchronize_post_init():

What about making this not conditional on dev->hotplugged, and simply
using VM state notifiers to make sure we resume the CPUs as soon as VM
state is RUN_STATE_RUNNING?

>      }
>  }
>  
> diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
> index 9c55b34..28fb4f8 100644
> --- a/stubs/Makefile.objs
> +++ b/stubs/Makefile.objs
> @@ -23,3 +23,4 @@ stub-obj-y += sysbus.o
>  stub-obj-y += vm-stop.o
>  stub-obj-y += vmstate.o
>  stub-obj-$(CONFIG_WIN32) += fd-register.o
> +stub-obj-y += resume_vcpu.o
> diff --git a/stubs/resume_vcpu.c b/stubs/resume_vcpu.c
> new file mode 100644
> index 0000000..383b71a
> --- /dev/null
> +++ b/stubs/resume_vcpu.c
> @@ -0,0 +1,6 @@
> +#include "qemu-common.h"
> +#include "sysemu/cpus.h"
> +
> +void resume_vcpu(CPUState *cpu)
> +{
> +}
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 06/19] introduce CPU hot-plug notifier

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:45PM +0200, Igor Mammedov wrote:
> hot-added CPU will be distributed to acpi_piix4, rtc_cmos and icc_bridge
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Have you considered making a generic device creation/realization
notification mechanism, that could work with any device class?

> ---
> v2:
>   * move notifier to qom/cpu.c and call it from CPUClass.realize() on hotplug
>   * remove get_firmware_id() since it belong to other patch
> ---
>  include/sysemu/sysemu.h |  3 +++
>  qom/cpu.c               | 12 ++++++++++++
>  2 files changed, 15 insertions(+)
> 
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 6578782..a8c3de1 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -152,6 +152,9 @@ void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
>  /* generic hotplug */
>  void drive_hot_add(Monitor *mon, const QDict *qdict);
>  
> +/* CPU hotplug */
> +void qemu_register_cpu_added_notifier(Notifier *notifier);
> +
>  /* pcie aer error injection */
>  void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
>  int do_pcie_aer_inject_error(Monitor *mon,
> diff --git a/qom/cpu.c b/qom/cpu.c
> index 2869cfe..fe85960 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -22,6 +22,17 @@
>  #include "qemu-common.h"
>  #include "sysemu/kvm.h"
>  #include "sysemu/cpus.h"
> +#include "qemu/notify.h"
> +#include "sysemu/sysemu.h"
> +
> +/* CPU hot-plug notifiers */
> +static NotifierList cpu_added_notifiers =
> +    NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> +
> +void qemu_register_cpu_added_notifier(Notifier *notifier)
> +{
> +    notifier_list_add(&cpu_added_notifiers, notifier);
> +}
>  
>  void cpu_reset_interrupt(CPUState *cpu, int mask)
>  {
> @@ -62,6 +73,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>      if (dev->hotplugged) {
>          cpu_synchronize_post_init(CPU(dev));
>          resume_vcpu(CPU(dev));
> +        notifier_list_notify(&cpu_added_notifiers, dev);
>      }
>  }
>  
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> ... so that on reboot BIOS could read current available CPU count
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> v2:
>   * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> ---
>  hw/timer/mc146818rtc.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)

Initialization of the cmos fields (including 0x5F) is done on
pc.c:pc_cmos_init(). What about making the field increment inside pc.c
as well?

What happens if a CPU is hotplugged after the machine has started but
before the guest OS has booted? Are we supposed to make sure the BIOS do
the right thing if a CPU is hotplugged before the OS has booted, or this
simply won't be supported?

> 
> diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
> index 69e6844..958ed6b 100644
> --- a/hw/timer/mc146818rtc.c
> +++ b/hw/timer/mc146818rtc.c
> @@ -82,6 +82,7 @@ typedef struct RTCState {
>      Notifier clock_reset_notifier;
>      LostTickPolicy lost_tick_policy;
>      Notifier suspend_notifier;
> +    Notifier cpu_added_notifier;
>  } RTCState;
>  
>  static void rtc_set_time(RTCState *s);
> @@ -759,6 +760,14 @@ static void rtc_notify_suspend(Notifier *notifier, void *data)
>      rtc_set_memory(&s->dev, 0xF, 0xFE);
>  }
>  
> +static void rtc_notify_cpu_added(Notifier *notifier, void *data)
> +{
> +    RTCState *s = container_of(notifier, RTCState, cpu_added_notifier);
> +
> +    /* increment the number of CPUs */
> +    s->cmos_data[0x5f] += 1;
> +}
> +
>  static void rtc_reset(void *opaque)
>  {
>      RTCState *s = opaque;
> @@ -852,6 +861,9 @@ static int rtc_initfn(ISADevice *dev)
>      s->suspend_notifier.notify = rtc_notify_suspend;
>      qemu_register_suspend_notifier(&s->suspend_notifier);
>  
> +    s->cpu_added_notifier.notify = rtc_notify_cpu_added;
> +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> +
>      memory_region_init_io(&s->io, &cmos_ops, s, "rtc", 2);
>      isa_register_ioport(dev, &s->io, base);
>  
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:47PM +0200, Igor Mammedov wrote:
> get_arch_id() adds possibility for generic code to get guest visible
> CPI id without accessing CPUArchState. If target doesn't override it,
> it will return cpu_index.
> 
> Override it on target-i386 to return APIC ID.

The code implementing the method looks OK, so:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

But I still need to understand better why exactly i386 needs to use it
and make the externally-visible hotplug interface based on APIC IDs if
all other architectures are going to be happy using CPU indexes to do
the same thing.

> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  * it will be used later by new cpu_exists() generic function and
>    acpi_piix.
>  * s/cpu_firmware_id/cpu_arch_id/
> ---
>  include/qom/cpu.h |  2 ++
>  qom/cpu.c         |  6 ++++++
>  target-i386/cpu.c | 10 ++++++++++
>  3 files changed, 18 insertions(+)
> 
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index 3664a1b..b376416 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -45,6 +45,7 @@ typedef struct CPUState CPUState;
>   * instantiatable CPU type.
>   * @reset: Callback to reset the #CPUState to its initial state.
>   * @do_interrupt: Callback for interrupt handling.
> + * @get_arch_id: Callback for getting architecture depended CPU ID
>   * @vmsd: State description for migration.
>   *
>   * Represents a CPU family or model.
> @@ -58,6 +59,7 @@ typedef struct CPUClass {
>  
>      void (*reset)(CPUState *cpu);
>      void (*do_interrupt)(CPUState *cpu);
> +    int64_t (*get_arch_id)(CPUState *cpu);
>  
>      const struct VMStateDescription *vmsd;
>  } CPUClass;
> diff --git a/qom/cpu.c b/qom/cpu.c
> index fe85960..90cbd77 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -77,6 +77,11 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>      }
>  }
>  
> +static int64_t cpu_common_get_arch_id(CPUState *cpu)
> +{
> +    return cpu->cpu_index;
> +}
> +
>  static void cpu_class_init(ObjectClass *klass, void *data)
>  {
>      DeviceClass *dc = DEVICE_CLASS(klass);
> @@ -84,6 +89,7 @@ static void cpu_class_init(ObjectClass *klass, void *data)
>  
>      k->class_by_name = cpu_common_class_by_name;
>      k->reset = cpu_common_reset;
> +    k->get_arch_id = cpu_common_get_arch_id;
>      dc->realize = cpu_common_realizefn;
>      dc->no_user = 1;
>  }
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index c28436c..9cca031 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -2272,6 +2272,14 @@ static void x86_cpu_initfn(Object *obj)
>      }
>  }
>  
> +static int64_t x86_cpu_get_arch_id(CPUState *cpu)
> +{
> +    X86CPU *x86cpu = X86_CPU(cpu);
> +    CPUX86State *env = &x86cpu->env;
> +
> +    return env->cpuid_apic_id;
> +}
> +
>  static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
>  {
>      X86CPUClass *xcc = X86_CPU_CLASS(oc);
> @@ -2286,6 +2294,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
>  
>      cc->do_interrupt = x86_cpu_do_interrupt;
>      cpu_class_set_vmsd(cc, &vmstate_x86_cpu);
> +
> +    cc->get_arch_id = x86_cpu_get_arch_id;
>  }
>  
>  static const TypeInfo x86_cpu_type_info = {
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 09/19] cpu: add helper cpu_exists(), to check if CPU with specified id exists

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:48PM +0200, Igor Mammedov wrote:
> ... it should be used only on slow path since it does recursive search
>     on /machine QOM tree for objects of TYPE_CPU
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Why not implement this only after patch 21/22, and use
"/machine/icc-bridge/cpu[ID]" for the lookup?


> ---
> v2:
>   * s/get_firmware_id()/get_arch_id()/ rebase fixup
>   * remove check for get_arch_id being NULL, since it's always defined
> ---
>  include/qom/cpu.h | 10 ++++++++++
>  qom/cpu.c         | 21 +++++++++++++++++++++
>  2 files changed, 31 insertions(+)
> 
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index b376416..cd7c4b5 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -225,6 +225,16 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
>   */
>  CPUState *qemu_get_cpu(int index);
>  
> +/**
> + * cpu_exists:
> + * @id - guest exposed CPU ID to lookup
> + *
> + * Search for CPU with specified ID.
> + *
> + * Returns: true - CPU is found, false - CPU isn't found
> + */
> +bool cpu_exists(int64_t id);
> +
>  #ifndef CONFIG_USER_ONLY
>  
>  typedef void (*CPUInterruptHandler)(CPUState *, int);
> diff --git a/qom/cpu.c b/qom/cpu.c
> index 90cbd77..c850d27 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -25,6 +25,27 @@
>  #include "qemu/notify.h"
>  #include "sysemu/sysemu.h"
>  
> +static int cpu_exist_cb(Object *obj, void *opaque)
> +{
> +    int64_t id = *(int64_t *)opaque;
> +    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
> +
> +    if (cpu_obj) {
> +        CPUState *cpu = CPU(cpu_obj);
> +        CPUClass *klass = CPU_GET_CLASS(cpu);
> +
> +        if (klass->get_arch_id(cpu) == id) {
> +            return 1;
> +        }
> +    }
> +    return object_child_foreach(obj, cpu_exist_cb, opaque);
> +}
> +
> +bool cpu_exists(int64_t id)
> +{
> +   return cpu_exist_cb(qdev_get_machine(), &id) ? true : false;
> +}
> +
>  /* CPU hot-plug notifiers */
>  static NotifierList cpu_added_notifiers =
>      NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:50PM +0200, Igor Mammedov wrote:
> ... and use it from board level to set APIC ID for CPUs
> it creates.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> Note:
>   * pc_new_cpu() function will be reused later in CPU hot-plug hook.
> 
> v3:
>   * user error_propagate() in property setter
> v2:
>   * use generic cpu_exists() instead of custom one
>   * make apic-id dynamic property, so it won't be possible to use it
>     as global property, since it's instance specific
> ---
>  hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
>  target-i386/cpu.c | 42 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> index 8d75b34..3adf294 100644
> --- a/hw/i386/pc.c
> +++ b/hw/i386/pc.c
> @@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
>      }
>  }
>  
> +static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> +{
> +    X86CPU *cpu;
> +
> +    cpu = cpu_x86_create(cpu_model, errp);
> +    if (!cpu) {
> +        return;
> +    }
> +
> +    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> +    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> +
> +    if (error_is_set(errp)) {
> +        if (cpu != NULL) {
> +            object_unref(OBJECT(cpu));
> +        }
> +    }
> +}
> +
>  void pc_cpus_init(const char *cpu_model)
>  {
>      int i;
> +    Error *error = NULL;
>  
>      /* init CPUs */
>      if (cpu_model == NULL) {
> @@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
>      }
>  
>      for (i = 0; i < smp_cpus; i++) {
> -        if (!cpu_x86_init(cpu_model)) {
> +        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> +        if (error) {
> +            fprintf(stderr, "%s\n", error_get_pretty(error));
> +            error_free(error);
>              exit(1);
>          }
>      }
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 9cca031..4ddc18a 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -1272,6 +1272,45 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
>      cpu->env.tsc_khz = value / 1000;
>  }
>  
> +static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
> +                                  const char *name, Error **errp)
> +{
> +    X86CPU *cpu = X86_CPU(obj);
> +    int64_t value = cpu->env.cpuid_apic_id;
> +
> +    visit_type_int(v, &value, name, errp);
> +}
> +
> +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> +                                  const char *name, Error **errp)
> +{
> +    X86CPU *cpu = X86_CPU(obj);
> +    const int64_t min = 0;
> +    const int64_t max = UINT32_MAX;
> +    Error *error = NULL;
> +    int64_t value;
> +
> +    visit_type_int(v, &value, name, &error);
> +    if (error) {
> +        error_propagate(errp, error);
> +        return;
> +    }
> +    if (value < min || value > max) {
> +        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
> +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> +                   object_get_typename(obj), name, value, min, max);
> +        error_propagate(errp, error);
> +        return;
> +    }
> +
> +    if (cpu_exists(value)) {
> +        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
> +        error_propagate(errp, error);

What about implementing this check after implementing the
/machine/icc-bridge links? Then we can simply check if
"/machine/icc-bridge/cpus[ID]" is already set.

(And then icc-bridge could be responsible for ensuring APIC ID
uniqueness, not the CPU object itself)

> +        return;
> +    }
> +    cpu->env.cpuid_apic_id = value;
> +}
> +
>  static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
>  {
>      x86_def_t *def;
> @@ -2259,6 +2298,9 @@ static void x86_cpu_initfn(Object *obj)
>      object_property_add(obj, "tsc-frequency", "int",
>                          x86_cpuid_get_tsc_freq,
>                          x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
> +    object_property_add(obj, "apic-id", "int",
> +                        x86_cpuid_get_apic_id,
> +                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
>  
>      env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
>  
> -- 
> 1.8.2
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Igor Mammedov]

On Thu, 11 Apr 2013 14:19:37 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > ... and leave links for not present CPUs empty.
> > 
> > It will allow users to query for possible APIC IDs and use them
> > with cpu-add QMP command.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> 
> I don't see anything wrong with having icc-bridge links as well, but I
> would really like to have a target-independent namespace with links,
> that could be used to query for the available/valid CPU IDs for cpu-add
> commands instead of icc-bridge. The IDs on that namespace could be
> considered completely opaque.

Considering that -numa in present state is not compatible with cpu-add
and that all CPU ID in this case are are sequence [0..maxcpus-1], this
patch could be dropped without any harm. libvirt could just use
numbers from this sequence like it's doing with current cpu_set without
any ID discovery. 

So, I've postponed target independent until we have -numa reworked,
then we could have /machine/node/socket/cpu containers with links.
The problem that needs to be solved, is the links storage ownership.
Who should allocate and own it? If machine was QOM object already,
I'd go with machine but it's not yet.

> 
> > ---
> > v2:
> >  * s/get_firmware_id/get_arch_id/ due to rebase
> >  * rename cpu_add_notifier to cpu_added_notifier &
> >    icc_bridge_cpu_add_req -> icc_bridge_cpued_add_req
> > ---
> >  hw/cpu/icc_bus.c          | 46 ++++++++++++++++++++++++++++++++++++++++++++++
> >  hw/i386/pc.c              |  9 +++++++--
> >  include/hw/i386/icc_bus.h |  2 ++
> >  3 files changed, 55 insertions(+), 2 deletions(-)
> > 
> > diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
> > index ab9623d..5c0b9d4 100644
> > --- a/hw/cpu/icc_bus.c
> > +++ b/hw/cpu/icc_bus.c
> > @@ -18,6 +18,7 @@
> >   */
> >  #include "hw/i386/icc_bus.h"
> >  #include "hw/sysbus.h"
> > +#include "sysemu/sysemu.h"
> >  
> >  static void icc_bus_initfn(Object *obj)
> >  {
> > @@ -61,15 +62,39 @@ typedef struct ICCBridgeState {
> >      SysBusDevice busdev;
> >      MemoryRegion apic_container;
> >      MemoryRegion ioapic_container;
> > +    Notifier cpu_added_notifier;
> > +    Object **links;
> >  } ICCBridgeState;
> >  #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
> >  
> >  
> > +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu_obj)
> > +{
> > +    gchar *name;
> > +    Error *error = NULL;
> > +    CPUState *cpu = CPU(cpu_obj);
> > +    int64_t id = CPU_GET_CLASS(cpu)->get_arch_id(cpu);
> > +
> > +    name = g_strdup_printf("cpu[%" PRIu32 "]", x86_cpu_apic_id_from_index(id));
> > +    object_property_set_link(bridge, cpu_obj, name, &error);
> > +    g_free(name);
> > +
> > +    g_assert(error == NULL);
> > +}
> > +
> > +static void icc_bridge_cpu_added_req(Notifier *n, void *opaque)
> > +{
> > +    ICCBridgeState *s = container_of(n, ICCBridgeState, cpu_added_notifier);
> > +
> > +    icc_bridge_set_cpu_link(OBJECT(s), OBJECT(opaque));
> > +}
> > +
> >  static void icc_bridge_initfn(Object *obj)
> >  {
> >      ICCBridgeState *s = ICC_BRIGDE(obj);
> >      SysBusDevice *sb = SYS_BUS_DEVICE(obj);
> >      ICCBus *ibus;
> > +    int i;
> >  
> >      ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
> >  
> > @@ -85,12 +110,33 @@ static void icc_bridge_initfn(Object *obj)
> >      memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
> >      sysbus_init_mmio(sb, &s->ioapic_container);
> >      ibus->ioapic_address_space = &s->ioapic_container;
> > +
> > +    s->links = g_malloc0(sizeof(Object *) * max_cpus);
> > +    for (i = 0; i < max_cpus; i++) {
> > +        gchar *cpu_name;
> > +
> > +        cpu_name = g_strdup_printf("cpu[%" PRIu32 "]",
> > +                                   x86_cpu_apic_id_from_index(i));
> > +        object_property_add_link(obj, cpu_name, TYPE_CPU, &s->links[i], NULL);
> > +        g_free(cpu_name);
> > +    }
> > +
> > +    s->cpu_added_notifier.notify = icc_bridge_cpu_added_req;
> > +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> > +}
> > +
> > +static void icc_bridge_fini(Object *obj)
> > +{
> > +    ICCBridgeState *s = ICC_BRIGDE(obj);
> > +
> > +    g_free(s->links);
> >  }
> >  
> >  static const TypeInfo icc_bridge_info = {
> >      .name  = "icc-bridge",
> >      .parent = TYPE_SYS_BUS_DEVICE,
> >      .instance_init  = icc_bridge_initfn,
> > +    .instance_finalize  = icc_bridge_fini,
> >      .instance_size  = sizeof(ICCBridgeState),
> >  };
> >  
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > index 6d5e164..ada235c 100644
> > --- a/hw/i386/pc.c
> > +++ b/hw/i386/pc.c
> > @@ -870,7 +870,8 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
> >      }
> >  }
> >  
> > -static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> > +static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
> > +                          SysBusDevice *icc_bridge, Error **errp)
> >  {
> >      X86CPU *cpu;
> >  
> > @@ -882,6 +883,10 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> >      object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> >      object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> >  
> > +    if (icc_bridge != NULL) {
> > +        icc_bridge_set_cpu_link(OBJECT(icc_bridge), OBJECT(cpu));
> > +    }
> > +
> >      if (error_is_set(errp)) {
> >          if (cpu != NULL) {
> >              object_unref(OBJECT(cpu));
> > @@ -911,7 +916,7 @@ void pc_cpus_init(const char *cpu_model)
> >                                                   TYPE_ICC_BRIDGE, NULL));
> >  
> >      for (i = 0; i < smp_cpus; i++) {
> > -        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> > +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), ib, &error);
> >          if (error) {
> >              fprintf(stderr, "%s\n", error_get_pretty(error));
> >              error_free(error);
> > diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
> > index 69a0278..bc31cd9 100644
> > --- a/include/hw/i386/icc_bus.h
> > +++ b/include/hw/i386/icc_bus.h
> > @@ -49,5 +49,7 @@ typedef struct ICCDeviceClass {
> >  
> >  #define TYPE_ICC_BRIDGE "icc-bridge"
> >  
> > +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu);
> > +
> >  #endif /* CONFIG_USER_ONLY */
> >  #endif
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 09/19] cpu: add helper cpu_exists(), to check if CPU with specified id exists

[Igor Mammedov]

On Thu, 11 Apr 2013 16:06:58 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:48PM +0200, Igor Mammedov wrote:
> > ... it should be used only on slow path since it does recursive search
> >     on /machine QOM tree for objects of TYPE_CPU
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Why not implement this only after patch 21/22, and use
> "/machine/icc-bridge/cpu[ID]" for the lookup?
/machine/icc-bridge/cpu[ID] is target specific and optional so far,
+ it would require parsing links names.

While looking for all CPUs, regardless of where they are located
(each target could have its own notion about it) are generic.

BTW:
patches 10/19 and 11/19 use it as well.

> 
> > ---
> > v2:
> >   * s/get_firmware_id()/get_arch_id()/ rebase fixup
> >   * remove check for get_arch_id being NULL, since it's always defined
> > ---
> >  include/qom/cpu.h | 10 ++++++++++
> >  qom/cpu.c         | 21 +++++++++++++++++++++
> >  2 files changed, 31 insertions(+)
> > 
> > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > index b376416..cd7c4b5 100644
> > --- a/include/qom/cpu.h
> > +++ b/include/qom/cpu.h
> > @@ -225,6 +225,16 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
> >   */
> >  CPUState *qemu_get_cpu(int index);
> >  
> > +/**
> > + * cpu_exists:
> > + * @id - guest exposed CPU ID to lookup
> > + *
> > + * Search for CPU with specified ID.
> > + *
> > + * Returns: true - CPU is found, false - CPU isn't found
> > + */
> > +bool cpu_exists(int64_t id);
> > +
> >  #ifndef CONFIG_USER_ONLY
> >  
> >  typedef void (*CPUInterruptHandler)(CPUState *, int);
> > diff --git a/qom/cpu.c b/qom/cpu.c
> > index 90cbd77..c850d27 100644
> > --- a/qom/cpu.c
> > +++ b/qom/cpu.c
> > @@ -25,6 +25,27 @@
> >  #include "qemu/notify.h"
> >  #include "sysemu/sysemu.h"
> >  
> > +static int cpu_exist_cb(Object *obj, void *opaque)
> > +{
> > +    int64_t id = *(int64_t *)opaque;
> > +    Object *cpu_obj = object_dynamic_cast(obj, TYPE_CPU);
> > +
> > +    if (cpu_obj) {
> > +        CPUState *cpu = CPU(cpu_obj);
> > +        CPUClass *klass = CPU_GET_CLASS(cpu);
> > +
> > +        if (klass->get_arch_id(cpu) == id) {
> > +            return 1;
> > +        }
> > +    }
> > +    return object_child_foreach(obj, cpu_exist_cb, opaque);
> > +}
> > +
> > +bool cpu_exists(int64_t id)
> > +{
> > +   return cpu_exist_cb(qdev_get_machine(), &id) ? true : false;
> > +}
> > +
> >  /* CPU hot-plug notifiers */
> >  static NotifierList cpu_added_notifiers =
> >      NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

On Thu, 11 Apr 2013 16:12:33 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:50PM +0200, Igor Mammedov wrote:
> > ... and use it from board level to set APIC ID for CPUs
> > it creates.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> > ---
> > Note:
> >   * pc_new_cpu() function will be reused later in CPU hot-plug hook.
> > 
> > v3:
> >   * user error_propagate() in property setter
> > v2:
> >   * use generic cpu_exists() instead of custom one
> >   * make apic-id dynamic property, so it won't be possible to use it
> >     as global property, since it's instance specific
> > ---
> >  hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
> >  target-i386/cpu.c | 42 ++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 66 insertions(+), 1 deletion(-)
> > 
> > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > index 8d75b34..3adf294 100644
> > --- a/hw/i386/pc.c
> > +++ b/hw/i386/pc.c
> > @@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
> >      }
> >  }
> >  
> > +static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> > +{
> > +    X86CPU *cpu;
> > +
> > +    cpu = cpu_x86_create(cpu_model, errp);
> > +    if (!cpu) {
> > +        return;
> > +    }
> > +
> > +    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> > +    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> > +
> > +    if (error_is_set(errp)) {
> > +        if (cpu != NULL) {
> > +            object_unref(OBJECT(cpu));
> > +        }
> > +    }
> > +}
> > +
> >  void pc_cpus_init(const char *cpu_model)
> >  {
> >      int i;
> > +    Error *error = NULL;
> >  
> >      /* init CPUs */
> >      if (cpu_model == NULL) {
> > @@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
> >      }
> >  
> >      for (i = 0; i < smp_cpus; i++) {
> > -        if (!cpu_x86_init(cpu_model)) {
> > +        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> > +        if (error) {
> > +            fprintf(stderr, "%s\n", error_get_pretty(error));
> > +            error_free(error);
> >              exit(1);
> >          }
> >      }
> > diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> > index 9cca031..4ddc18a 100644
> > --- a/target-i386/cpu.c
> > +++ b/target-i386/cpu.c
> > @@ -1272,6 +1272,45 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
> >      cpu->env.tsc_khz = value / 1000;
> >  }
> >  
> > +static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
> > +                                  const char *name, Error **errp)
> > +{
> > +    X86CPU *cpu = X86_CPU(obj);
> > +    int64_t value = cpu->env.cpuid_apic_id;
> > +
> > +    visit_type_int(v, &value, name, errp);
> > +}
> > +
> > +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> > +                                  const char *name, Error **errp)
> > +{
> > +    X86CPU *cpu = X86_CPU(obj);
> > +    const int64_t min = 0;
> > +    const int64_t max = UINT32_MAX;
> > +    Error *error = NULL;
> > +    int64_t value;
> > +
> > +    visit_type_int(v, &value, name, &error);
> > +    if (error) {
> > +        error_propagate(errp, error);
> > +        return;
> > +    }
> > +    if (value < min || value > max) {
> > +        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
> > +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> > +                   object_get_typename(obj), name, value, min, max);
> > +        error_propagate(errp, error);
> > +        return;
> > +    }
> > +
> > +    if (cpu_exists(value)) {
> > +        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
> > +        error_propagate(errp, error);
> 
> What about implementing this check after implementing the
> /machine/icc-bridge links? Then we can simply check if
> "/machine/icc-bridge/cpus[ID]" is already set.
because it's more generic and won't break even if link location changes

> 
> (And then icc-bridge could be responsible for ensuring APIC ID
> uniqueness, not the CPU object itself)
Yep, with cpu-add id could be/is verified before setting property, but if
one considers device_add or fictional -device cpuxxx on cmd line, then won't
be an external check for this. This check takes care about these use cases.

> 
> > +        return;
> > +    }
> > +    cpu->env.cpuid_apic_id = value;
> > +}
> > +
> >  static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
> >  {
> >      x86_def_t *def;
> > @@ -2259,6 +2298,9 @@ static void x86_cpu_initfn(Object *obj)
> >      object_property_add(obj, "tsc-frequency", "int",
> >                          x86_cpuid_get_tsc_freq,
> >                          x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
> > +    object_property_add(obj, "apic-id", "int",
> > +                        x86_cpuid_get_apic_id,
> > +                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
> >  
> >      env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
> >  
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Igor Mammedov]

On Thu, 11 Apr 2013 16:04:26 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:47PM +0200, Igor Mammedov wrote:
> > get_arch_id() adds possibility for generic code to get guest visible
> > CPI id without accessing CPUArchState. If target doesn't override it,
> > it will return cpu_index.
> > 
> > Override it on target-i386 to return APIC ID.
> 
> The code implementing the method looks OK, so:
> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> But I still need to understand better why exactly i386 needs to use it
> and make the externally-visible hotplug interface based on APIC IDs if
> all other architectures are going to be happy using CPU indexes to do
> the same thing.
get_arch_id() was prompted by 10/19 which is used not only by target-i386.
so it was introduced to make acpi_piix4 CPU hotplug more generic. And proved
to useful in a number of other places.

> 
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  * it will be used later by new cpu_exists() generic function and
> >    acpi_piix.
> >  * s/cpu_firmware_id/cpu_arch_id/
> > ---
> >  include/qom/cpu.h |  2 ++
> >  qom/cpu.c         |  6 ++++++
> >  target-i386/cpu.c | 10 ++++++++++
> >  3 files changed, 18 insertions(+)
> > 
> > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > index 3664a1b..b376416 100644
> > --- a/include/qom/cpu.h
> > +++ b/include/qom/cpu.h
> > @@ -45,6 +45,7 @@ typedef struct CPUState CPUState;
> >   * instantiatable CPU type.
> >   * @reset: Callback to reset the #CPUState to its initial state.
> >   * @do_interrupt: Callback for interrupt handling.
> > + * @get_arch_id: Callback for getting architecture depended CPU ID
> >   * @vmsd: State description for migration.
> >   *
> >   * Represents a CPU family or model.
> > @@ -58,6 +59,7 @@ typedef struct CPUClass {
> >  
> >      void (*reset)(CPUState *cpu);
> >      void (*do_interrupt)(CPUState *cpu);
> > +    int64_t (*get_arch_id)(CPUState *cpu);
> >  
> >      const struct VMStateDescription *vmsd;
> >  } CPUClass;
> > diff --git a/qom/cpu.c b/qom/cpu.c
> > index fe85960..90cbd77 100644
> > --- a/qom/cpu.c
> > +++ b/qom/cpu.c
> > @@ -77,6 +77,11 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> >      }
> >  }
> >  
> > +static int64_t cpu_common_get_arch_id(CPUState *cpu)
> > +{
> > +    return cpu->cpu_index;
> > +}
> > +
> >  static void cpu_class_init(ObjectClass *klass, void *data)
> >  {
> >      DeviceClass *dc = DEVICE_CLASS(klass);
> > @@ -84,6 +89,7 @@ static void cpu_class_init(ObjectClass *klass, void *data)
> >  
> >      k->class_by_name = cpu_common_class_by_name;
> >      k->reset = cpu_common_reset;
> > +    k->get_arch_id = cpu_common_get_arch_id;
> >      dc->realize = cpu_common_realizefn;
> >      dc->no_user = 1;
> >  }
> > diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> > index c28436c..9cca031 100644
> > --- a/target-i386/cpu.c
> > +++ b/target-i386/cpu.c
> > @@ -2272,6 +2272,14 @@ static void x86_cpu_initfn(Object *obj)
> >      }
> >  }
> >  
> > +static int64_t x86_cpu_get_arch_id(CPUState *cpu)
> > +{
> > +    X86CPU *x86cpu = X86_CPU(cpu);
> > +    CPUX86State *env = &x86cpu->env;
> > +
> > +    return env->cpuid_apic_id;
> > +}
> > +
> >  static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> >  {
> >      X86CPUClass *xcc = X86_CPU_CLASS(oc);
> > @@ -2286,6 +2294,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> >  
> >      cc->do_interrupt = x86_cpu_do_interrupt;
> >      cpu_class_set_vmsd(cc, &vmstate_x86_cpu);
> > +
> > +    cc->get_arch_id = x86_cpu_get_arch_id;
> >  }
> >  
> >  static const TypeInfo x86_cpu_type_info = {
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Igor Mammedov]

On Thu, 11 Apr 2013 15:59:40 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> > ... so that on reboot BIOS could read current available CPU count
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > v2:
> >   * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> > ---
> >  hw/timer/mc146818rtc.c | 12 ++++++++++++
> >  1 file changed, 12 insertions(+)
> 
> Initialization of the cmos fields (including 0x5F) is done on
> pc.c:pc_cmos_init(). What about making the field increment inside pc.c
> as well?
I looked at possibility but discarded it because to increment it there initial
value should be -1 (field is zero based) which is not obvious, plug ugly
casting to singed variable.
Result looked ugly.

> 
> What happens if a CPU is hotplugged after the machine has started but
> before the guest OS has booted? Are we supposed to make sure the BIOS do
> the right thing if a CPU is hotplugged before the OS has booted, or this
> simply won't be supported?
BIOS uses this value to set in ACPI tables what CPUs are present.
 1. if hot-plug happens before BIOS reads it then OS will see all CPUs
and SCI it receives will be nop.
 2. if hot-plug happens after BIOS reads it, OS will handle SCI as usual
    and hotplug CPU instead of initializing it smp_boot() time.
BIOS itself has nothing to do with hot-plug, it's OSPM job.

> 
> > 
> > diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
> > index 69e6844..958ed6b 100644
> > --- a/hw/timer/mc146818rtc.c
> > +++ b/hw/timer/mc146818rtc.c
> > @@ -82,6 +82,7 @@ typedef struct RTCState {
> >      Notifier clock_reset_notifier;
> >      LostTickPolicy lost_tick_policy;
> >      Notifier suspend_notifier;
> > +    Notifier cpu_added_notifier;
> >  } RTCState;
> >  
> >  static void rtc_set_time(RTCState *s);
> > @@ -759,6 +760,14 @@ static void rtc_notify_suspend(Notifier *notifier, void *data)
> >      rtc_set_memory(&s->dev, 0xF, 0xFE);
> >  }
> >  
> > +static void rtc_notify_cpu_added(Notifier *notifier, void *data)
> > +{
> > +    RTCState *s = container_of(notifier, RTCState, cpu_added_notifier);
> > +
> > +    /* increment the number of CPUs */
> > +    s->cmos_data[0x5f] += 1;
> > +}
> > +
> >  static void rtc_reset(void *opaque)
> >  {
> >      RTCState *s = opaque;
> > @@ -852,6 +861,9 @@ static int rtc_initfn(ISADevice *dev)
> >      s->suspend_notifier.notify = rtc_notify_suspend;
> >      qemu_register_suspend_notifier(&s->suspend_notifier);
> >  
> > +    s->cpu_added_notifier.notify = rtc_notify_cpu_added;
> > +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> > +
> >      memory_region_init_io(&s->io, &cmos_ops, s, "rtc", 2);
> >      isa_register_ioport(dev, &s->io, base);
> >  
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 06/19] introduce CPU hot-plug notifier

[Igor Mammedov]

On Thu, 11 Apr 2013 15:46:47 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:45PM +0200, Igor Mammedov wrote:
> > hot-added CPU will be distributed to acpi_piix4, rtc_cmos and icc_bridge
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Have you considered making a generic device creation/realization
> notification mechanism, that could work with any device class?
Not really, that would make notification handler a bit more complicated,
since they will have to decide if event belong to them.
We could add it in future if there will be need for it.

> 
> > ---
> > v2:
> >   * move notifier to qom/cpu.c and call it from CPUClass.realize() on hotplug
> >   * remove get_firmware_id() since it belong to other patch
> > ---
> >  include/sysemu/sysemu.h |  3 +++
> >  qom/cpu.c               | 12 ++++++++++++
> >  2 files changed, 15 insertions(+)
> > 
> > diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> > index 6578782..a8c3de1 100644
> > --- a/include/sysemu/sysemu.h
> > +++ b/include/sysemu/sysemu.h
> > @@ -152,6 +152,9 @@ void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
> >  /* generic hotplug */
> >  void drive_hot_add(Monitor *mon, const QDict *qdict);
> >  
> > +/* CPU hotplug */
> > +void qemu_register_cpu_added_notifier(Notifier *notifier);
> > +
> >  /* pcie aer error injection */
> >  void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
> >  int do_pcie_aer_inject_error(Monitor *mon,
> > diff --git a/qom/cpu.c b/qom/cpu.c
> > index 2869cfe..fe85960 100644
> > --- a/qom/cpu.c
> > +++ b/qom/cpu.c
> > @@ -22,6 +22,17 @@
> >  #include "qemu-common.h"
> >  #include "sysemu/kvm.h"
> >  #include "sysemu/cpus.h"
> > +#include "qemu/notify.h"
> > +#include "sysemu/sysemu.h"
> > +
> > +/* CPU hot-plug notifiers */
> > +static NotifierList cpu_added_notifiers =
> > +    NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> > +
> > +void qemu_register_cpu_added_notifier(Notifier *notifier)
> > +{
> > +    notifier_list_add(&cpu_added_notifiers, notifier);
> > +}
> >  
> >  void cpu_reset_interrupt(CPUState *cpu, int mask)
> >  {
> > @@ -62,6 +73,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> >      if (dev->hotplugged) {
> >          cpu_synchronize_post_init(CPU(dev));
> >          resume_vcpu(CPU(dev));
> > +        notifier_list_notify(&cpu_added_notifiers, dev);
> >      }
> >  }
> >  
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 03/19] cpu: make kvm-stub.o a part of CPU library

[Igor Mammedov]

On Thu, 11 Apr 2013 14:45:42 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:42PM +0200, Igor Mammedov wrote:
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  Makefile.target      | 13 +++++++------
> >  include/hw/pci/msi.h |  2 ++
> >  include/sysemu/kvm.h |  4 ++--
> >  kvm-stub.c           |  2 ++
> >  4 files changed, 13 insertions(+), 8 deletions(-)
> > 
> > diff --git a/Makefile.target b/Makefile.target
> > index 2bd6d14..18f348c 100644
> > --- a/Makefile.target
> > +++ b/Makefile.target
> > @@ -64,6 +64,12 @@ all: $(PROGS) stap
> >  # Dummy command so that make thinks it has done something
> >  	@true
> >  
> > +CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
> > +CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
> > +CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
> > +CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
> > +CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
> > +
> >  #########################################################
> >  # cpu emulator library
> >  obj-y = exec.o translate-all.o cpu-exec.o
> > @@ -74,6 +80,7 @@ obj-y += fpu/softfloat.o
> >  obj-y += target-$(TARGET_BASE_ARCH)/
> >  obj-y += disas.o
> >  obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
> > +obj-$(CONFIG_NO_KVM) += kvm-stub.o
> 
> The kvm-stub.o line at the "System emulator target" section is redundant
> now, isn't it? Any specific reason you didn't remove it?
Nope, I just forgot it. Thanks, I'll fix it.
 
> 
> >  
> >  #########################################################
> >  # Linux user emulator target
> > @@ -102,12 +109,6 @@ endif #CONFIG_BSD_USER
> >  #########################################################
> >  # System emulator target
> >  ifdef CONFIG_SOFTMMU
> > -CONFIG_NO_PCI = $(if $(subst n,,$(CONFIG_PCI)),n,y)
> > -CONFIG_NO_KVM = $(if $(subst n,,$(CONFIG_KVM)),n,y)
> > -CONFIG_NO_XEN = $(if $(subst n,,$(CONFIG_XEN)),n,y)
> > -CONFIG_NO_GET_MEMORY_MAPPING = $(if $(subst n,,$(CONFIG_HAVE_GET_MEMORY_MAPPING)),n,y)
> > -CONFIG_NO_CORE_DUMP = $(if $(subst n,,$(CONFIG_HAVE_CORE_DUMP)),n,y)
> > -
> >  obj-y += arch_init.o cpus.o monitor.o gdbstub.o balloon.o ioport.o
> >  obj-y += qtest.o
> >  obj-y += hw/
> > diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
> > index 81a3848..d4d0450 100644
> > --- a/include/hw/pci/msi.h
> > +++ b/include/hw/pci/msi.h
> > @@ -21,6 +21,7 @@
> >  #ifndef QEMU_MSI_H
> >  #define QEMU_MSI_H
> >  
> > +#ifndef CONFIG_USER_ONLY
> 
> I would simply #ifdef the "#include pci/msi.h" line on kvm-stub.c
> instead. But I guess it doesn't hurt to have the #ifdef here.
It's more robust to not ifdef at source rather then at places where it's used.

> 
> >  #include "qemu-common.h"
> >  #include "hw/pci/pci.h"
> >  
> > @@ -47,4 +48,5 @@ static inline bool msi_present(const PCIDevice *dev)
> >      return dev->cap_present & QEMU_PCI_CAP_MSI;
> >  }
> >  
> > +#endif
> >  #endif /* QEMU_MSI_H */
> > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > index 495e6f8..93cef28 100644
> > --- a/include/sysemu/kvm.h
> > +++ b/include/sysemu/kvm.h
> > @@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
> >  #if !defined(CONFIG_USER_ONLY)
> >  void *kvm_vmalloc(ram_addr_t size);
> >  void *kvm_arch_vmalloc(ram_addr_t size);
> > -void kvm_setup_guest_memory(void *start, size_t size);
> > +#endif
> >  
> > +void kvm_setup_guest_memory(void *start, size_t size);
> >  void kvm_flush_coalesced_mmio_buffer(void);
> > -#endif
> >  
> >  int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
> >                            target_ulong len, int type);
> > diff --git a/kvm-stub.c b/kvm-stub.c
> > index 82875dd..b34064a 100644
> > --- a/kvm-stub.c
> > +++ b/kvm-stub.c
> > @@ -122,6 +122,7 @@ int kvm_on_sigbus(int code, void *addr)
> >      return 1;
> >  }
> >  
> > +#ifndef CONFIG_USER_ONLY
> >  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
> >  {
> >      return -ENOSYS;
> > @@ -145,3 +146,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
> >  {
> >      return -ENOSYS;
> >  }
> > +#endif
> 
> What about making MSIMessage definition available on CONFIG_USER_ONLY,
> so we don't even need to #ifdef this part?

it might work, but it causes #includes split which I don't like much.
Anyway here is patch on top of this one. If it looks more acceptable to you
this way, I'll merge it.

---

diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
index d4d0450..c159a15 100644
--- a/include/hw/pci/msi.h
+++ b/include/hw/pci/msi.h
@@ -21,15 +21,15 @@
 #ifndef QEMU_MSI_H
 #define QEMU_MSI_H
 
-#ifndef CONFIG_USER_ONLY
 #include "qemu-common.h"
-#include "hw/pci/pci.h"
 
 struct MSIMessage {
     uint64_t address;
     uint32_t data;
 };
 
+#ifndef CONFIG_USER_ONLY
+#include "hw/pci/pci.h"
 extern bool msi_supported;
 
 void msi_set_message(PCIDevice *dev, MSIMessage msg);
diff --git a/kvm-stub.c b/kvm-stub.c
index 8cb81c4..b5d4446 100644
--- a/kvm-stub.c
+++ b/kvm-stub.c
@@ -123,7 +123,6 @@ int kvm_on_sigbus(int code, void *addr)
     return 1;
 }
 
-#ifndef CONFIG_USER_ONLY
 int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
 {
     return -ENOSYS;
@@ -147,4 +146,3 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
 {
     return -ENOSYS;
 }
-#endif

> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 04/19] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Igor Mammedov]

On Thu, 11 Apr 2013 15:33:01 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:43PM +0200, Igor Mammedov wrote:
> > ... to synchronize CPU state to KVM
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> > v2:
> >   * linking kvm-stub.o to *-user target moved in separate patch
> > ---
> >  include/sysemu/kvm.h | 18 ++++++++++--------
> >  kvm-all.c            |  1 +
> >  kvm-stub.c           |  1 +
> >  qom/cpu.c            |  4 ++++
> >  vl.c                 |  1 -
> >  5 files changed, 16 insertions(+), 9 deletions(-)
> > 
> > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > index 93cef28..8fb2489 100644
> > --- a/include/sysemu/kvm.h
> > +++ b/include/sysemu/kvm.h
> > @@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
> >  uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
> >                                        uint32_t index, int reg);
> >  void kvm_cpu_synchronize_state(CPUArchState *env);
> > -void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> > -void kvm_cpu_synchronize_post_init(CPUState *cpu);
> >  
> >  /* generic hooks - to be moved/refactored once there are more users */
> >  
> > @@ -262,6 +260,16 @@ static inline void cpu_synchronize_state(CPUArchState *env)
> >      }
> >  }
> >  
> > +#if !defined(CONFIG_USER_ONLY)
> > +int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> > +                                       hwaddr *phys_addr);
> > +#endif
> > +
> > +#endif /* NEED_CPU_H */
> > +
> > +void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> > +void kvm_cpu_synchronize_post_init(CPUState *cpu);
> > +
> >  static inline void cpu_synchronize_post_reset(CPUState *cpu)
> >  {
> >      if (kvm_enabled()) {
> > @@ -277,12 +285,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
> >  }
> >  
> >  
> > -#if !defined(CONFIG_USER_ONLY)
> > -int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> > -                                       hwaddr *phys_addr);
> > -#endif
> > -
> > -#endif
> >  int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool assign,
> >                             uint32_t size);
> >  
> > diff --git a/kvm-all.c b/kvm-all.c
> > index fc4e17c..1d17128 100644
> > --- a/kvm-all.c
> > +++ b/kvm-all.c
> > @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
> >  bool kvm_irqfds_allowed;
> >  bool kvm_msi_via_irqfd_allowed;
> >  bool kvm_gsi_routing_allowed;
> > +bool kvm_allowed;
> >  
> >  static const KVMCapabilityInfo kvm_required_capabilites[] = {
> >      KVM_CAP_INFO(USER_MEMORY),
> > diff --git a/kvm-stub.c b/kvm-stub.c
> > index b34064a..8cb81c4 100644
> > --- a/kvm-stub.c
> > +++ b/kvm-stub.c
> > @@ -23,6 +23,7 @@ bool kvm_async_interrupts_allowed;
> >  bool kvm_irqfds_allowed;
> >  bool kvm_msi_via_irqfd_allowed;
> >  bool kvm_gsi_routing_allowed;
> > +bool kvm_allowed;
> >  
> >  int kvm_init_vcpu(CPUState *cpu)
> >  {
> > diff --git a/qom/cpu.c b/qom/cpu.c
> > index e242dcb..0c76712 100644
> > --- a/qom/cpu.c
> > +++ b/qom/cpu.c
> > @@ -20,6 +20,7 @@
> >  
> >  #include "qom/cpu.h"
> >  #include "qemu-common.h"
> > +#include "sysemu/kvm.h"
> >  
> >  void cpu_reset_interrupt(CPUState *cpu, int mask)
> >  {
> > @@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
> >  
> >  static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> >  {
> > +    if (dev->hotplugged) {
> > +        cpu_synchronize_post_init(CPU(dev));
> > +    }
> 
> For reference: non-hotplugged CPUs have cpu_synchronize_post_init()
> called from cpu_synchronize_all_post_init(), that is called from main(),
> just after the machine->init() call.
> 
> That said, have you considered changing the code so that's possible to
> call cpu_synchronize_post_init() for all CPUs, so we could kill the
> cpu_synchronize_all_post_init() call entirely?
> 
> Maybe the solution for this is to realize non-hotplugged CPUs later,
> instead of realizing them very early inside the machine init function.
> If doing is not possible or not enough, we could use VM state notifiers
> to watch for the right moment to call cpu_synchronize_post_init().

I have not considered it, since question is out of scope CPU hot-plug.
It might be feasible to do so later in separate patch and remove
"if (dev->hotplugged)" condition tough.

I think it will be more appropriate when machine will QOM-ified, then
to make machine.realize() work it had to be done, but now I don't see
any gain doing so (providing all the trouble it might cause).

> 
> >  }
> >  
> >  static void cpu_class_init(ObjectClass *klass, void *data)
> > diff --git a/vl.c b/vl.c
> > index d694a90..21952c3 100644
> > --- a/vl.c
> > +++ b/vl.c
> > @@ -267,7 +267,6 @@ static NotifierList machine_init_done_notifiers =
> >      NOTIFIER_LIST_INITIALIZER(machine_init_done_notifiers);
> >  
> >  static bool tcg_allowed = true;
> > -bool kvm_allowed;
> >  bool xen_allowed;
> >  uint32_t xen_domid;
> >  enum xen_mode xen_mode = XEN_EMULATE;
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 05/19] cpu: resume CPU from CPUClass.cpu_common_realizefn() when it is hot-plugged

[Igor Mammedov]

On Thu, 11 Apr 2013 15:36:23 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:44PM +0200, Igor Mammedov wrote:
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  cpus.c                | 11 ++++++++---
> >  include/sysemu/cpus.h |  3 +++
> >  qom/cpu.c             |  2 ++
> >  stubs/Makefile.objs   |  1 +
> >  stubs/resume_vcpu.c   |  6 ++++++
> >  5 files changed, 20 insertions(+), 3 deletions(-)
> >  create mode 100644 stubs/resume_vcpu.c
> > 
> > diff --git a/cpus.c b/cpus.c
> > index 9b9a32f..f12624d 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
> >      }
> >  }
> >  
> > +void resume_vcpu(CPUState *cpu)
> > +{
> > +    cpu->stop = false;
> > +    cpu->stopped = false;
> > +    qemu_cpu_kick(cpu);
> > +}
> > +
> >  void resume_all_vcpus(void)
> >  {
> >      CPUArchState *penv = first_cpu;
> > @@ -980,9 +987,7 @@ void resume_all_vcpus(void)
> >      qemu_clock_enable(vm_clock, true);
> >      while (penv) {
> >          CPUState *pcpu = ENV_GET_CPU(penv);
> > -        pcpu->stop = false;
> > -        pcpu->stopped = false;
> > -        qemu_cpu_kick(pcpu);
> > +        resume_vcpu(pcpu);
> 
> If you include just the above chunks in a separate patch, I would
> promptly send a Reviewed-by line for that.
sure

> 
> >          penv = penv->next_cpu;
> >      }
> >  }
> > diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
> > index 6502488..9437df5 100644
> > --- a/include/sysemu/cpus.h
> > +++ b/include/sysemu/cpus.h
> > @@ -1,11 +1,14 @@
> >  #ifndef QEMU_CPUS_H
> >  #define QEMU_CPUS_H
> >  
> > +#include "qom/cpu.h"
> > +
> >  /* cpus.c */
> >  void qemu_init_cpu_loop(void);
> >  void resume_all_vcpus(void);
> >  void pause_all_vcpus(void);
> >  void cpu_stop_current(void);
> > +void resume_vcpu(CPUState *cpu);
> >  
> >  void cpu_synchronize_all_states(void);
> >  void cpu_synchronize_all_post_reset(void);
> > diff --git a/qom/cpu.c b/qom/cpu.c
> > index 0c76712..2869cfe 100644
> > --- a/qom/cpu.c
> > +++ b/qom/cpu.c
> > @@ -21,6 +21,7 @@
> >  #include "qom/cpu.h"
> >  #include "qemu-common.h"
> >  #include "sysemu/kvm.h"
> > +#include "sysemu/cpus.h"
> >  
> >  void cpu_reset_interrupt(CPUState *cpu, int mask)
> >  {
> > @@ -60,6 +61,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> >  {
> >      if (dev->hotplugged) {
> >          cpu_synchronize_post_init(CPU(dev));
> > +        resume_vcpu(CPU(dev));
> 
> Simlar question to the one I had about cpu_synchronize_post_init():
> 
> What about making this not conditional on dev->hotplugged, and simply
> using VM state notifiers to make sure we resume the CPUs as soon as VM
> state is RUN_STATE_RUNNING?
answered to similar question in 4/19

> 
> >      }
> >  }
> >  
> > diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
> > index 9c55b34..28fb4f8 100644
> > --- a/stubs/Makefile.objs
> > +++ b/stubs/Makefile.objs
> > @@ -23,3 +23,4 @@ stub-obj-y += sysbus.o
> >  stub-obj-y += vm-stop.o
> >  stub-obj-y += vmstate.o
> >  stub-obj-$(CONFIG_WIN32) += fd-register.o
> > +stub-obj-y += resume_vcpu.o
> > diff --git a/stubs/resume_vcpu.c b/stubs/resume_vcpu.c
> > new file mode 100644
> > index 0000000..383b71a
> > --- /dev/null
> > +++ b/stubs/resume_vcpu.c
> > @@ -0,0 +1,6 @@
> > +#include "qemu-common.h"
> > +#include "sysemu/cpus.h"
> > +
> > +void resume_vcpu(CPUState *cpu)
> > +{
> > +}
> > -- 
> > 1.8.2
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 12:01:03PM +0200, Igor Mammedov wrote:
> On Thu, 11 Apr 2013 14:19:37 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > > ... and leave links for not present CPUs empty.
> > > 
> > > It will allow users to query for possible APIC IDs and use them
> > > with cpu-add QMP command.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > 
> > I don't see anything wrong with having icc-bridge links as well, but I
> > would really like to have a target-independent namespace with links,
> > that could be used to query for the available/valid CPU IDs for cpu-add
> > commands instead of icc-bridge. The IDs on that namespace could be
> > considered completely opaque.
> 
> Considering that -numa in present state is not compatible with cpu-add
> and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> patch could be dropped without any harm. libvirt could just use
> numbers from this sequence like it's doing with current cpu_set without
> any ID discovery. 

But it's not -numa that makes APIC ID probing necessary, it's
non-power-of-2 core/thread counts on -smp (that make APIC IDs not match
CPU indexes).

"Don't use CPU hotplug with -numa" is easy to be understood by users and
by libvirt, but "don't use CPU hotplug with non-power-of-2 cores/threads
counts" is harder to explain.


> 
> So, I've postponed target independent until we have -numa reworked,
> then we could have /machine/node/socket/cpu containers with links.
> The problem that needs to be solved, is the links storage ownership.
> Who should allocate and own it? If machine was QOM object already,
> I'd go with machine but it's not yet.

If we use CPU index as argument to cpu-add, we don't need to handle all
those problems right now, we don't need to expose an APIC ID discovery
interface, we make it work even with non-power-of-2 cores/threads
counts, and we make it work with -numa.

So, my big question is: why are we trying so hard to avoid using CPU
indexes as argument to cpu-add, if it's so much easier, and it is an
obvious solution that makes the interface target-independent without any
extra effort?

> 
> > 
> > > ---
> > > v2:
> > >  * s/get_firmware_id/get_arch_id/ due to rebase
> > >  * rename cpu_add_notifier to cpu_added_notifier &
> > >    icc_bridge_cpu_add_req -> icc_bridge_cpued_add_req
> > > ---
> > >  hw/cpu/icc_bus.c          | 46 ++++++++++++++++++++++++++++++++++++++++++++++
> > >  hw/i386/pc.c              |  9 +++++++--
> > >  include/hw/i386/icc_bus.h |  2 ++
> > >  3 files changed, 55 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
> > > index ab9623d..5c0b9d4 100644
> > > --- a/hw/cpu/icc_bus.c
> > > +++ b/hw/cpu/icc_bus.c
> > > @@ -18,6 +18,7 @@
> > >   */
> > >  #include "hw/i386/icc_bus.h"
> > >  #include "hw/sysbus.h"
> > > +#include "sysemu/sysemu.h"
> > >  
> > >  static void icc_bus_initfn(Object *obj)
> > >  {
> > > @@ -61,15 +62,39 @@ typedef struct ICCBridgeState {
> > >      SysBusDevice busdev;
> > >      MemoryRegion apic_container;
> > >      MemoryRegion ioapic_container;
> > > +    Notifier cpu_added_notifier;
> > > +    Object **links;
> > >  } ICCBridgeState;
> > >  #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj), TYPE_ICC_BRIDGE)
> > >  
> > >  
> > > +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu_obj)
> > > +{
> > > +    gchar *name;
> > > +    Error *error = NULL;
> > > +    CPUState *cpu = CPU(cpu_obj);
> > > +    int64_t id = CPU_GET_CLASS(cpu)->get_arch_id(cpu);
> > > +
> > > +    name = g_strdup_printf("cpu[%" PRIu32 "]", x86_cpu_apic_id_from_index(id));
> > > +    object_property_set_link(bridge, cpu_obj, name, &error);
> > > +    g_free(name);
> > > +
> > > +    g_assert(error == NULL);
> > > +}
> > > +
> > > +static void icc_bridge_cpu_added_req(Notifier *n, void *opaque)
> > > +{
> > > +    ICCBridgeState *s = container_of(n, ICCBridgeState, cpu_added_notifier);
> > > +
> > > +    icc_bridge_set_cpu_link(OBJECT(s), OBJECT(opaque));
> > > +}
> > > +
> > >  static void icc_bridge_initfn(Object *obj)
> > >  {
> > >      ICCBridgeState *s = ICC_BRIGDE(obj);
> > >      SysBusDevice *sb = SYS_BUS_DEVICE(obj);
> > >      ICCBus *ibus;
> > > +    int i;
> > >  
> > >      ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj), "icc-bus"));
> > >  
> > > @@ -85,12 +110,33 @@ static void icc_bridge_initfn(Object *obj)
> > >      memory_region_init(&s->ioapic_container, "icc-ioapic-container", 0x1000);
> > >      sysbus_init_mmio(sb, &s->ioapic_container);
> > >      ibus->ioapic_address_space = &s->ioapic_container;
> > > +
> > > +    s->links = g_malloc0(sizeof(Object *) * max_cpus);
> > > +    for (i = 0; i < max_cpus; i++) {
> > > +        gchar *cpu_name;
> > > +
> > > +        cpu_name = g_strdup_printf("cpu[%" PRIu32 "]",
> > > +                                   x86_cpu_apic_id_from_index(i));
> > > +        object_property_add_link(obj, cpu_name, TYPE_CPU, &s->links[i], NULL);
> > > +        g_free(cpu_name);
> > > +    }
> > > +
> > > +    s->cpu_added_notifier.notify = icc_bridge_cpu_added_req;
> > > +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> > > +}
> > > +
> > > +static void icc_bridge_fini(Object *obj)
> > > +{
> > > +    ICCBridgeState *s = ICC_BRIGDE(obj);
> > > +
> > > +    g_free(s->links);
> > >  }
> > >  
> > >  static const TypeInfo icc_bridge_info = {
> > >      .name  = "icc-bridge",
> > >      .parent = TYPE_SYS_BUS_DEVICE,
> > >      .instance_init  = icc_bridge_initfn,
> > > +    .instance_finalize  = icc_bridge_fini,
> > >      .instance_size  = sizeof(ICCBridgeState),
> > >  };
> > >  
> > > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > > index 6d5e164..ada235c 100644
> > > --- a/hw/i386/pc.c
> > > +++ b/hw/i386/pc.c
> > > @@ -870,7 +870,8 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
> > >      }
> > >  }
> > >  
> > > -static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> > > +static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
> > > +                          SysBusDevice *icc_bridge, Error **errp)
> > >  {
> > >      X86CPU *cpu;
> > >  
> > > @@ -882,6 +883,10 @@ static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> > >      object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> > >      object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> > >  
> > > +    if (icc_bridge != NULL) {
> > > +        icc_bridge_set_cpu_link(OBJECT(icc_bridge), OBJECT(cpu));
> > > +    }
> > > +
> > >      if (error_is_set(errp)) {
> > >          if (cpu != NULL) {
> > >              object_unref(OBJECT(cpu));
> > > @@ -911,7 +916,7 @@ void pc_cpus_init(const char *cpu_model)
> > >                                                   TYPE_ICC_BRIDGE, NULL));
> > >  
> > >      for (i = 0; i < smp_cpus; i++) {
> > > -        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> > > +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), ib, &error);
> > >          if (error) {
> > >              fprintf(stderr, "%s\n", error_get_pretty(error));
> > >              error_free(error);
> > > diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
> > > index 69a0278..bc31cd9 100644
> > > --- a/include/hw/i386/icc_bus.h
> > > +++ b/include/hw/i386/icc_bus.h
> > > @@ -49,5 +49,7 @@ typedef struct ICCDeviceClass {
> > >  
> > >  #define TYPE_ICC_BRIDGE "icc-bridge"
> > >  
> > > +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu);
> > > +
> > >  #endif /* CONFIG_USER_ONLY */
> > >  #endif
> > > -- 
> > > 1.8.2
> > > 
> > 
> > -- 
> > Eduardo
> 
> 
> -- 
> Regards,
>   Igor

-- 
Eduardo

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 12:20:45PM +0200, Igor Mammedov wrote:
> On Thu, 11 Apr 2013 16:12:33 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Thu, Apr 11, 2013 at 04:51:50PM +0200, Igor Mammedov wrote:
> > > ... and use it from board level to set APIC ID for CPUs
> > > it creates.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> > > ---
> > > Note:
> > >   * pc_new_cpu() function will be reused later in CPU hot-plug hook.
> > > 
> > > v3:
> > >   * user error_propagate() in property setter
> > > v2:
> > >   * use generic cpu_exists() instead of custom one
> > >   * make apic-id dynamic property, so it won't be possible to use it
> > >     as global property, since it's instance specific
> > > ---
> > >  hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
> > >  target-i386/cpu.c | 42 ++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 66 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > > index 8d75b34..3adf294 100644
> > > --- a/hw/i386/pc.c
> > > +++ b/hw/i386/pc.c
> > > @@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
> > >      }
> > >  }
> > >  
> > > +static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> > > +{
> > > +    X86CPU *cpu;
> > > +
> > > +    cpu = cpu_x86_create(cpu_model, errp);
> > > +    if (!cpu) {
> > > +        return;
> > > +    }
> > > +
> > > +    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> > > +    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> > > +
> > > +    if (error_is_set(errp)) {
> > > +        if (cpu != NULL) {
> > > +            object_unref(OBJECT(cpu));
> > > +        }
> > > +    }
> > > +}
> > > +
> > >  void pc_cpus_init(const char *cpu_model)
> > >  {
> > >      int i;
> > > +    Error *error = NULL;
> > >  
> > >      /* init CPUs */
> > >      if (cpu_model == NULL) {
> > > @@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
> > >      }
> > >  
> > >      for (i = 0; i < smp_cpus; i++) {
> > > -        if (!cpu_x86_init(cpu_model)) {
> > > +        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> > > +        if (error) {
> > > +            fprintf(stderr, "%s\n", error_get_pretty(error));
> > > +            error_free(error);
> > >              exit(1);
> > >          }
> > >      }
> > > diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> > > index 9cca031..4ddc18a 100644
> > > --- a/target-i386/cpu.c
> > > +++ b/target-i386/cpu.c
> > > @@ -1272,6 +1272,45 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
> > >      cpu->env.tsc_khz = value / 1000;
> > >  }
> > >  
> > > +static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
> > > +                                  const char *name, Error **errp)
> > > +{
> > > +    X86CPU *cpu = X86_CPU(obj);
> > > +    int64_t value = cpu->env.cpuid_apic_id;
> > > +
> > > +    visit_type_int(v, &value, name, errp);
> > > +}
> > > +
> > > +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> > > +                                  const char *name, Error **errp)
> > > +{
> > > +    X86CPU *cpu = X86_CPU(obj);
> > > +    const int64_t min = 0;
> > > +    const int64_t max = UINT32_MAX;
> > > +    Error *error = NULL;
> > > +    int64_t value;
> > > +
> > > +    visit_type_int(v, &value, name, &error);
> > > +    if (error) {
> > > +        error_propagate(errp, error);
> > > +        return;
> > > +    }
> > > +    if (value < min || value > max) {
> > > +        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
> > > +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> > > +                   object_get_typename(obj), name, value, min, max);
> > > +        error_propagate(errp, error);
> > > +        return;
> > > +    }
> > > +
> > > +    if (cpu_exists(value)) {
> > > +        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
> > > +        error_propagate(errp, error);
> > 
> > What about implementing this check after implementing the
> > /machine/icc-bridge links? Then we can simply check if
> > "/machine/icc-bridge/cpus[ID]" is already set.
> because it's more generic and won't break even if link location changes

Well, if link location changed, we could change the cpu_exists()
implementation. We don't even need to hardcode the icc-bridge path, we
could simply follow appropriate links if we set them up.

My problem with cpu_exists() is that it is based on global state,
instead of simply using the links between bus/devices to find out if the
IDs are being allocated correctly. We could make it work like PCI: devfn
conflicts are solved simply by looking at the list of devices in the
specific PCIBus where the device is being added. See
do_pci_register_device().

That said, I am not against the current approach if we don't have the
necessary bus/links modelled yet. But if we are going to add icc-bridge,
we could benefit from it to implement cpu_exists().

> 
> > 
> > (And then icc-bridge could be responsible for ensuring APIC ID
> > uniqueness, not the CPU object itself)
> Yep, with cpu-add id could be/is verified before setting property, but if
> one considers device_add or fictional -device cpuxxx on cmd line, then won't
> be an external check for this. This check takes care about these use cases.

That's why I believe the CPU has to calculate the APIC ID based on
signaling coming from other devices/buses, instead of allowing the APIC
ID to be directly specified in the device_add/-device options.  That's
how real CPUs work: as the CPU manufacturer doesn't know what will be
the package ID of the CPU, the APIC IDs are not hardcoded in the CPU;
they are calculated based on the CPU topology and some socket identifier
signal coming from the board.

Comparing it with PCI again: devfn of PCI devices can be specified
manually, but if it is not present you just need to look at the PCI bus
to find out the next available devfn.

Based on that, I thought that icc-bridge was going be the component
responsible for the APIC ID allocation logic, considering that it
already contains APIC-ID-based links to the CPU objects.


> 
> > 
> > > +        return;
> > > +    }
> > > +    cpu->env.cpuid_apic_id = value;
> > > +}
> > > +
> > >  static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
> > >  {
> > >      x86_def_t *def;
> > > @@ -2259,6 +2298,9 @@ static void x86_cpu_initfn(Object *obj)
> > >      object_property_add(obj, "tsc-frequency", "int",
> > >                          x86_cpuid_get_tsc_freq,
> > >                          x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
> > > +    object_property_add(obj, "apic-id", "int",
> > > +                        x86_cpuid_get_apic_id,
> > > +                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
> > >  
> > >      env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
> > >  
> > > -- 
> > > 1.8.2
> > > 
> > 
> > -- 
> > Eduardo
> > 
> 
> 
> -- 
> Regards,
>   Igor

-- 
Eduardo

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 12:53:51PM +0200, Igor Mammedov wrote:
> On Thu, 11 Apr 2013 15:59:40 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> > > ... so that on reboot BIOS could read current available CPU count
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > v2:
> > >   * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> > > ---
> > >  hw/timer/mc146818rtc.c | 12 ++++++++++++
> > >  1 file changed, 12 insertions(+)
> > 
> > Initialization of the cmos fields (including 0x5F) is done on
> > pc.c:pc_cmos_init(). What about making the field increment inside pc.c
> > as well?
> I looked at possibility but discarded it because to increment it there initial
> value should be -1 (field is zero based) which is not obvious, plug ugly
> casting to singed variable.
> Result looked ugly.

I was thinking about simply adding exactly the same code with exactly
the same logic, but inside pc.c instead of of mc146818rtc.c. Instead of
registering the notifier inside rtc_initfn(), register exactly the same
notifier with exactly the same code, but inside pc_cmos_init() (that's
where 0x5f is initialized).

It would even be safer and easier review and ensure correctness: with
this patch, the notifier is registered very early, even before
pc_cmos_init() initializes 0x5f to smp_cpus. CPU hotplug events are
unlikely to be emitted before pc_cmos_init() is called, but still: why
make the initialization ordering so subtle if we don't have to?

> 
> > 
> > What happens if a CPU is hotplugged after the machine has started but
> > before the guest OS has booted? Are we supposed to make sure the BIOS do
> > the right thing if a CPU is hotplugged before the OS has booted, or this
> > simply won't be supported?
> BIOS uses this value to set in ACPI tables what CPUs are present.
>  1. if hot-plug happens before BIOS reads it then OS will see all CPUs
> and SCI it receives will be nop.
>  2. if hot-plug happens after BIOS reads it, OS will handle SCI as usual
>     and hotplug CPU instead of initializing it smp_boot() time.
> BIOS itself has nothing to do with hot-plug, it's OSPM job.

Makes sense, thanks.

What happens if the CPU is hotplugged after the BIOS builds the ACPI
tables, but long before the OS starts handling ACPI events? Is the OS
guaranteed to run the CPU hotplug ACPI method (that's \_GPE._E02 in the
DSDT, right?), even if that happens?

> 
> > 
> > > 
> > > diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
> > > index 69e6844..958ed6b 100644
> > > --- a/hw/timer/mc146818rtc.c
> > > +++ b/hw/timer/mc146818rtc.c
> > > @@ -82,6 +82,7 @@ typedef struct RTCState {
> > >      Notifier clock_reset_notifier;
> > >      LostTickPolicy lost_tick_policy;
> > >      Notifier suspend_notifier;
> > > +    Notifier cpu_added_notifier;
> > >  } RTCState;
> > >  
> > >  static void rtc_set_time(RTCState *s);
> > > @@ -759,6 +760,14 @@ static void rtc_notify_suspend(Notifier *notifier, void *data)
> > >      rtc_set_memory(&s->dev, 0xF, 0xFE);
> > >  }
> > >  
> > > +static void rtc_notify_cpu_added(Notifier *notifier, void *data)
> > > +{
> > > +    RTCState *s = container_of(notifier, RTCState, cpu_added_notifier);
> > > +
> > > +    /* increment the number of CPUs */
> > > +    s->cmos_data[0x5f] += 1;
> > > +}
> > > +
> > >  static void rtc_reset(void *opaque)
> > >  {
> > >      RTCState *s = opaque;
> > > @@ -852,6 +861,9 @@ static int rtc_initfn(ISADevice *dev)
> > >      s->suspend_notifier.notify = rtc_notify_suspend;
> > >      qemu_register_suspend_notifier(&s->suspend_notifier);
> > >  
> > > +    s->cpu_added_notifier.notify = rtc_notify_cpu_added;
> > > +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> > > +
> > >      memory_region_init_io(&s->io, &cmos_ops, s, "rtc", 2);
> > >      isa_register_ioport(dev, &s->io, base);
> > >  
> > > -- 
> > > 1.8.2
> > > 
> > 
> > -- 
> > Eduardo
> 
> 
> -- 
> Regards,
>   Igor

-- 
Eduardo

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 12:31:22PM +0200, Igor Mammedov wrote:
> On Thu, 11 Apr 2013 16:04:26 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Thu, Apr 11, 2013 at 04:51:47PM +0200, Igor Mammedov wrote:
> > > get_arch_id() adds possibility for generic code to get guest visible
> > > CPI id without accessing CPUArchState. If target doesn't override it,
> > > it will return cpu_index.
> > > 
> > > Override it on target-i386 to return APIC ID.
> > 
> > The code implementing the method looks OK, so:
> > 
> > Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> > 
> > But I still need to understand better why exactly i386 needs to use it
> > and make the externally-visible hotplug interface based on APIC IDs if
> > all other architectures are going to be happy using CPU indexes to do
> > the same thing.
> get_arch_id() was prompted by 10/19 which is used not only by target-i386.
> so it was introduced to make acpi_piix4 CPU hotplug more generic. And proved
> to useful in a number of other places.

OK, so get_arch_id() is useful for other code too, not just for the
external cpu-add interface. It makes lots of sense to use it on
acpi_piix4, yes.

(My question about making the cpu-add interface based on arch IDs
instead of arch-independent CPU indexes is being discussed under patch
18/19).

> 
> > 
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > ---
> > >  * it will be used later by new cpu_exists() generic function and
> > >    acpi_piix.
> > >  * s/cpu_firmware_id/cpu_arch_id/
> > > ---
> > >  include/qom/cpu.h |  2 ++
> > >  qom/cpu.c         |  6 ++++++
> > >  target-i386/cpu.c | 10 ++++++++++
> > >  3 files changed, 18 insertions(+)
> > > 
> > > diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> > > index 3664a1b..b376416 100644
> > > --- a/include/qom/cpu.h
> > > +++ b/include/qom/cpu.h
> > > @@ -45,6 +45,7 @@ typedef struct CPUState CPUState;
> > >   * instantiatable CPU type.
> > >   * @reset: Callback to reset the #CPUState to its initial state.
> > >   * @do_interrupt: Callback for interrupt handling.
> > > + * @get_arch_id: Callback for getting architecture depended CPU ID
> > >   * @vmsd: State description for migration.
> > >   *
> > >   * Represents a CPU family or model.
> > > @@ -58,6 +59,7 @@ typedef struct CPUClass {
> > >  
> > >      void (*reset)(CPUState *cpu);
> > >      void (*do_interrupt)(CPUState *cpu);
> > > +    int64_t (*get_arch_id)(CPUState *cpu);
> > >  
> > >      const struct VMStateDescription *vmsd;
> > >  } CPUClass;
> > > diff --git a/qom/cpu.c b/qom/cpu.c
> > > index fe85960..90cbd77 100644
> > > --- a/qom/cpu.c
> > > +++ b/qom/cpu.c
> > > @@ -77,6 +77,11 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> > >      }
> > >  }
> > >  
> > > +static int64_t cpu_common_get_arch_id(CPUState *cpu)
> > > +{
> > > +    return cpu->cpu_index;
> > > +}
> > > +
> > >  static void cpu_class_init(ObjectClass *klass, void *data)
> > >  {
> > >      DeviceClass *dc = DEVICE_CLASS(klass);
> > > @@ -84,6 +89,7 @@ static void cpu_class_init(ObjectClass *klass, void *data)
> > >  
> > >      k->class_by_name = cpu_common_class_by_name;
> > >      k->reset = cpu_common_reset;
> > > +    k->get_arch_id = cpu_common_get_arch_id;
> > >      dc->realize = cpu_common_realizefn;
> > >      dc->no_user = 1;
> > >  }
> > > diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> > > index c28436c..9cca031 100644
> > > --- a/target-i386/cpu.c
> > > +++ b/target-i386/cpu.c
> > > @@ -2272,6 +2272,14 @@ static void x86_cpu_initfn(Object *obj)
> > >      }
> > >  }
> > >  
> > > +static int64_t x86_cpu_get_arch_id(CPUState *cpu)
> > > +{
> > > +    X86CPU *x86cpu = X86_CPU(cpu);
> > > +    CPUX86State *env = &x86cpu->env;
> > > +
> > > +    return env->cpuid_apic_id;
> > > +}
> > > +
> > >  static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> > >  {
> > >      X86CPUClass *xcc = X86_CPU_CLASS(oc);
> > > @@ -2286,6 +2294,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> > >  
> > >      cc->do_interrupt = x86_cpu_do_interrupt;
> > >      cpu_class_set_vmsd(cc, &vmstate_x86_cpu);
> > > +
> > > +    cc->get_arch_id = x86_cpu_get_arch_id;
> > >  }
> > >  
> > >  static const TypeInfo x86_cpu_type_info = {
> > > -- 
> > > 1.8.2
> > > 
> > 
> > -- 
> > Eduardo
> 
> 
> -- 
> Regards,
>   Igor

-- 
Eduardo

Re: [Qemu-devel] [PATCH 04/19] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 01:34:09PM +0200, Igor Mammedov wrote:
> On Thu, 11 Apr 2013 15:33:01 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Thu, Apr 11, 2013 at 04:51:43PM +0200, Igor Mammedov wrote:
> > > ... to synchronize CPU state to KVM
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > ---
> > > v2:
> > >   * linking kvm-stub.o to *-user target moved in separate patch
> > > ---
> > >  include/sysemu/kvm.h | 18 ++++++++++--------
> > >  kvm-all.c            |  1 +
> > >  kvm-stub.c           |  1 +
> > >  qom/cpu.c            |  4 ++++
> > >  vl.c                 |  1 -
> > >  5 files changed, 16 insertions(+), 9 deletions(-)
> > > 
> > > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > > index 93cef28..8fb2489 100644
> > > --- a/include/sysemu/kvm.h
> > > +++ b/include/sysemu/kvm.h
> > > @@ -250,8 +250,6 @@ int kvm_check_extension(KVMState *s, unsigned int extension);
> > >  uint32_t kvm_arch_get_supported_cpuid(KVMState *env, uint32_t function,
> > >                                        uint32_t index, int reg);
> > >  void kvm_cpu_synchronize_state(CPUArchState *env);
> > > -void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> > > -void kvm_cpu_synchronize_post_init(CPUState *cpu);
> > >  
> > >  /* generic hooks - to be moved/refactored once there are more users */
> > >  
> > > @@ -262,6 +260,16 @@ static inline void cpu_synchronize_state(CPUArchState *env)
> > >      }
> > >  }
> > >  
> > > +#if !defined(CONFIG_USER_ONLY)
> > > +int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> > > +                                       hwaddr *phys_addr);
> > > +#endif
> > > +
> > > +#endif /* NEED_CPU_H */
> > > +
> > > +void kvm_cpu_synchronize_post_reset(CPUState *cpu);
> > > +void kvm_cpu_synchronize_post_init(CPUState *cpu);
> > > +
> > >  static inline void cpu_synchronize_post_reset(CPUState *cpu)
> > >  {
> > >      if (kvm_enabled()) {
> > > @@ -277,12 +285,6 @@ static inline void cpu_synchronize_post_init(CPUState *cpu)
> > >  }
> > >  
> > >  
> > > -#if !defined(CONFIG_USER_ONLY)
> > > -int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
> > > -                                       hwaddr *phys_addr);
> > > -#endif
> > > -
> > > -#endif
> > >  int kvm_set_ioeventfd_mmio(int fd, uint32_t adr, uint32_t val, bool assign,
> > >                             uint32_t size);
> > >  
> > > diff --git a/kvm-all.c b/kvm-all.c
> > > index fc4e17c..1d17128 100644
> > > --- a/kvm-all.c
> > > +++ b/kvm-all.c
> > > @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
> > >  bool kvm_irqfds_allowed;
> > >  bool kvm_msi_via_irqfd_allowed;
> > >  bool kvm_gsi_routing_allowed;
> > > +bool kvm_allowed;
> > >  
> > >  static const KVMCapabilityInfo kvm_required_capabilites[] = {
> > >      KVM_CAP_INFO(USER_MEMORY),
> > > diff --git a/kvm-stub.c b/kvm-stub.c
> > > index b34064a..8cb81c4 100644
> > > --- a/kvm-stub.c
> > > +++ b/kvm-stub.c
> > > @@ -23,6 +23,7 @@ bool kvm_async_interrupts_allowed;
> > >  bool kvm_irqfds_allowed;
> > >  bool kvm_msi_via_irqfd_allowed;
> > >  bool kvm_gsi_routing_allowed;
> > > +bool kvm_allowed;
> > >  
> > >  int kvm_init_vcpu(CPUState *cpu)
> > >  {
> > > diff --git a/qom/cpu.c b/qom/cpu.c
> > > index e242dcb..0c76712 100644
> > > --- a/qom/cpu.c
> > > +++ b/qom/cpu.c
> > > @@ -20,6 +20,7 @@
> > >  
> > >  #include "qom/cpu.h"
> > >  #include "qemu-common.h"
> > > +#include "sysemu/kvm.h"
> > >  
> > >  void cpu_reset_interrupt(CPUState *cpu, int mask)
> > >  {
> > > @@ -57,6 +58,9 @@ static ObjectClass *cpu_common_class_by_name(const char *cpu_model)
> > >  
> > >  static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> > >  {
> > > +    if (dev->hotplugged) {
> > > +        cpu_synchronize_post_init(CPU(dev));
> > > +    }
> > 
> > For reference: non-hotplugged CPUs have cpu_synchronize_post_init()
> > called from cpu_synchronize_all_post_init(), that is called from main(),
> > just after the machine->init() call.
> > 
> > That said, have you considered changing the code so that's possible to
> > call cpu_synchronize_post_init() for all CPUs, so we could kill the
> > cpu_synchronize_all_post_init() call entirely?
> > 
> > Maybe the solution for this is to realize non-hotplugged CPUs later,
> > instead of realizing them very early inside the machine init function.
> > If doing is not possible or not enough, we could use VM state notifiers
> > to watch for the right moment to call cpu_synchronize_post_init().
> 
> I have not considered it, since question is out of scope CPU hot-plug.
> It might be feasible to do so later in separate patch and remove
> "if (dev->hotplugged)" condition tough.
> 
> I think it will be more appropriate when machine will QOM-ified, then
> to make machine.realize() work it had to be done, but now I don't see
> any gain doing so (providing all the trouble it might cause).

OK, sounds like a good justification to me.

I would love to have a device model where there's no difference between
hotplugged and non-hotplugged devices. But this is hard to achieve.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Igor Mammedov]

On Fri, 12 Apr 2013 10:35:53 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 12, 2013 at 12:53:51PM +0200, Igor Mammedov wrote:
> > On Thu, 11 Apr 2013 15:59:40 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> > > > ... so that on reboot BIOS could read current available CPU count
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > v2:
> > > >   * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> > > > ---
> > > >  hw/timer/mc146818rtc.c | 12 ++++++++++++
> > > >  1 file changed, 12 insertions(+)
> > > 
> > > Initialization of the cmos fields (including 0x5F) is done on
> > > pc.c:pc_cmos_init(). What about making the field increment inside pc.c
> > > as well?
> > I looked at possibility but discarded it because to increment it there initial
> > value should be -1 (field is zero based) which is not obvious, plug ugly
> > casting to singed variable.
> > Result looked ugly.
> 
> I was thinking about simply adding exactly the same code with exactly
> the same logic, but inside pc.c instead of of mc146818rtc.c. Instead of
> registering the notifier inside rtc_initfn(), register exactly the same
> notifier with exactly the same code, but inside pc_cmos_init() (that's
> where 0x5f is initialized).
> 
> It would even be safer and easier review and ensure correctness: with
> this patch, the notifier is registered very early, even before
> pc_cmos_init() initializes 0x5f to smp_cpus. CPU hotplug events are
> unlikely to be emitted before pc_cmos_init() is called, but still: why
it isn't be called, hot-add is available only after machine initialized.

> make the initialization ordering so subtle if we don't have to?
Currently cmos init doesn't look like proper QOM object and has 3 stage
initialization: realize(), then pc_cmos_init() the last pc_cmos_init_late().
The last 2 calls are made after realize(), setting various properties. Which
looks wrong from QOM perspective, so I'm against of stuffing more internal
stuff in arbitrary places. We should do opposite instead.

If you look at mc146818rtc.c or hw/acpi/piix4.c, all notifiers are private to
object and registered at realize() time. It looks like initialization order
of mc146818rtc should be fixed, instead of adapting new code to it.

So since this patch doesn't break or violate anything in current code, I'd
like to leave it as it is.

> > > 
> > > What happens if a CPU is hotplugged after the machine has started but
> > > before the guest OS has booted? Are we supposed to make sure the BIOS do
> > > the right thing if a CPU is hotplugged before the OS has booted, or this
> > > simply won't be supported?
> > BIOS uses this value to set in ACPI tables what CPUs are present.
> >  1. if hot-plug happens before BIOS reads it then OS will see all CPUs
> > and SCI it receives will be nop.
> >  2. if hot-plug happens after BIOS reads it, OS will handle SCI as usual
> >     and hotplug CPU instead of initializing it smp_boot() time.
> > BIOS itself has nothing to do with hot-plug, it's OSPM job.
> 
> Makes sense, thanks.
> 
> What happens if the CPU is hotplugged after the BIOS builds the ACPI
> tables, but long before the OS starts handling ACPI events? Is the OS
> guaranteed to run the CPU hotplug ACPI method (that's \_GPE._E02 in the
> DSDT, right?), even if that happens?
Theoretically interrupt should not disappear on it's own and OS should pick it
up. But to say it for sure I need to test this case. If SCI will be lost for
some reason, then OS won't notice new CPU until another CPU hot-plug event
happens.

> > 
> > > 
> > > > 
> > > > diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
> > > > index 69e6844..958ed6b 100644
> > > > --- a/hw/timer/mc146818rtc.c
> > > > +++ b/hw/timer/mc146818rtc.c
> > > > @@ -82,6 +82,7 @@ typedef struct RTCState {
> > > >      Notifier clock_reset_notifier;
> > > >      LostTickPolicy lost_tick_policy;
> > > >      Notifier suspend_notifier;
> > > > +    Notifier cpu_added_notifier;
> > > >  } RTCState;
> > > >  
> > > >  static void rtc_set_time(RTCState *s);
> > > > @@ -759,6 +760,14 @@ static void rtc_notify_suspend(Notifier *notifier, void *data)
> > > >      rtc_set_memory(&s->dev, 0xF, 0xFE);
> > > >  }
> > > >  
> > > > +static void rtc_notify_cpu_added(Notifier *notifier, void *data)
> > > > +{
> > > > +    RTCState *s = container_of(notifier, RTCState, cpu_added_notifier);
> > > > +
> > > > +    /* increment the number of CPUs */
> > > > +    s->cmos_data[0x5f] += 1;
> > > > +}
> > > > +
> > > >  static void rtc_reset(void *opaque)
> > > >  {
> > > >      RTCState *s = opaque;
> > > > @@ -852,6 +861,9 @@ static int rtc_initfn(ISADevice *dev)
> > > >      s->suspend_notifier.notify = rtc_notify_suspend;
> > > >      qemu_register_suspend_notifier(&s->suspend_notifier);
> > > >  
> > > > +    s->cpu_added_notifier.notify = rtc_notify_cpu_added;
> > > > +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> > > > +
> > > >      memory_region_init_io(&s->io, &cmos_ops, s, "rtc", 2);
> > > >      isa_register_ioport(dev, &s->io, base);
> > > >  
> > > > -- 
> > > > 1.8.2
> > > > 
> > > 
> > > -- 
> > > Eduardo
> > 
> > 
> > -- 
> > Regards,
> >   Igor
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 05:16:20PM +0200, Igor Mammedov wrote:
> On Fri, 12 Apr 2013 10:35:53 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Fri, Apr 12, 2013 at 12:53:51PM +0200, Igor Mammedov wrote:
> > > On Thu, 11 Apr 2013 15:59:40 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> > > > > ... so that on reboot BIOS could read current available CPU count
> > > > > 
> > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > v2:
> > > > >   * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> > > > > ---
> > > > >  hw/timer/mc146818rtc.c | 12 ++++++++++++
> > > > >  1 file changed, 12 insertions(+)
> > > > 
> > > > Initialization of the cmos fields (including 0x5F) is done on
> > > > pc.c:pc_cmos_init(). What about making the field increment inside pc.c
> > > > as well?
> > > I looked at possibility but discarded it because to increment it there initial
> > > value should be -1 (field is zero based) which is not obvious, plug ugly
> > > casting to singed variable.
> > > Result looked ugly.
> > 
> > I was thinking about simply adding exactly the same code with exactly
> > the same logic, but inside pc.c instead of of mc146818rtc.c. Instead of
> > registering the notifier inside rtc_initfn(), register exactly the same
> > notifier with exactly the same code, but inside pc_cmos_init() (that's
> > where 0x5f is initialized).
> > 
> > It would even be safer and easier review and ensure correctness: with
> > this patch, the notifier is registered very early, even before
> > pc_cmos_init() initializes 0x5f to smp_cpus. CPU hotplug events are
> > unlikely to be emitted before pc_cmos_init() is called, but still: why
> it isn't be called, hot-add is available only after machine initialized.
> 
> > make the initialization ordering so subtle if we don't have to?
> Currently cmos init doesn't look like proper QOM object and has 3 stage
> initialization: realize(), then pc_cmos_init() the last pc_cmos_init_late().
> The last 2 calls are made after realize(), setting various properties. Which
> looks wrong from QOM perspective, so I'm against of stuffing more internal
> stuff in arbitrary places. We should do opposite instead.

True, but as we already have this weird 3-stage initialization process
and we won't fix it really soon, I would really prefer to keep parts of
the code that are closely related and depend on each other in the same
part of the code.


> 
> If you look at mc146818rtc.c or hw/acpi/piix4.c, all notifiers are private to
> object and registered at realize() time. It looks like initialization order
> of mc146818rtc should be fixed, instead of adapting new code to it.
> 
> So since this patch doesn't break or violate anything in current code, I'd
> like to leave it as it is.

If you insist into making the mc146818rtc device take care of
maintaining the 0x5f value by itself, why not doing:

    s->cmos_data[0x5f] = smp_cpus - 1;

inside rtc_initfn() instead of pc_cmos_init() as well?

This would be one additional step towards making pc_cmos_init() be
replaced by QOM-based code (if that's what you want to do in the long
term).


> 
> > > > 
> > > > What happens if a CPU is hotplugged after the machine has started but
> > > > before the guest OS has booted? Are we supposed to make sure the BIOS do
> > > > the right thing if a CPU is hotplugged before the OS has booted, or this
> > > > simply won't be supported?
> > > BIOS uses this value to set in ACPI tables what CPUs are present.
> > >  1. if hot-plug happens before BIOS reads it then OS will see all CPUs
> > > and SCI it receives will be nop.
> > >  2. if hot-plug happens after BIOS reads it, OS will handle SCI as usual
> > >     and hotplug CPU instead of initializing it smp_boot() time.
> > > BIOS itself has nothing to do with hot-plug, it's OSPM job.
> > 
> > Makes sense, thanks.
> > 
> > What happens if the CPU is hotplugged after the BIOS builds the ACPI
> > tables, but long before the OS starts handling ACPI events? Is the OS
> > guaranteed to run the CPU hotplug ACPI method (that's \_GPE._E02 in the
> > DSDT, right?), even if that happens?
> Theoretically interrupt should not disappear on it's own and OS should pick it
> up. But to say it for sure I need to test this case. If SCI will be lost for
> some reason, then OS won't notice new CPU until another CPU hot-plug event
> happens.


Makes sense, thanks.

(Anyway, if an interrupt gets lost, it's probably a BIOS or OS bug).

> [...]

-- 
Eduardo

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

On Fri, 12 Apr 2013 10:13:13 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 12, 2013 at 12:20:45PM +0200, Igor Mammedov wrote:
> > On Thu, 11 Apr 2013 16:12:33 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Thu, Apr 11, 2013 at 04:51:50PM +0200, Igor Mammedov wrote:
> > > > ... and use it from board level to set APIC ID for CPUs
> > > > it creates.
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> > > > ---
> > > > Note:
> > > >   * pc_new_cpu() function will be reused later in CPU hot-plug hook.
> > > > 
> > > > v3:
> > > >   * user error_propagate() in property setter
> > > > v2:
> > > >   * use generic cpu_exists() instead of custom one
> > > >   * make apic-id dynamic property, so it won't be possible to use it
> > > >     as global property, since it's instance specific
> > > > ---
> > > >  hw/i386/pc.c      | 25 ++++++++++++++++++++++++-
> > > >  target-i386/cpu.c | 42 ++++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 66 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > > > index 8d75b34..3adf294 100644
> > > > --- a/hw/i386/pc.c
> > > > +++ b/hw/i386/pc.c
> > > > @@ -869,9 +869,29 @@ void pc_acpi_smi_interrupt(void *opaque, int irq, int level)
> > > >      }
> > > >  }
> > > >  
> > > > +static void pc_new_cpu(const char *cpu_model, int64_t apic_id, Error **errp)
> > > > +{
> > > > +    X86CPU *cpu;
> > > > +
> > > > +    cpu = cpu_x86_create(cpu_model, errp);
> > > > +    if (!cpu) {
> > > > +        return;
> > > > +    }
> > > > +
> > > > +    object_property_set_int(OBJECT(cpu), apic_id, "apic-id", errp);
> > > > +    object_property_set_bool(OBJECT(cpu), true, "realized", errp);
> > > > +
> > > > +    if (error_is_set(errp)) {
> > > > +        if (cpu != NULL) {
> > > > +            object_unref(OBJECT(cpu));
> > > > +        }
> > > > +    }
> > > > +}
> > > > +
> > > >  void pc_cpus_init(const char *cpu_model)
> > > >  {
> > > >      int i;
> > > > +    Error *error = NULL;
> > > >  
> > > >      /* init CPUs */
> > > >      if (cpu_model == NULL) {
> > > > @@ -883,7 +903,10 @@ void pc_cpus_init(const char *cpu_model)
> > > >      }
> > > >  
> > > >      for (i = 0; i < smp_cpus; i++) {
> > > > -        if (!cpu_x86_init(cpu_model)) {
> > > > +        pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i), &error);
> > > > +        if (error) {
> > > > +            fprintf(stderr, "%s\n", error_get_pretty(error));
> > > > +            error_free(error);
> > > >              exit(1);
> > > >          }
> > > >      }
> > > > diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> > > > index 9cca031..4ddc18a 100644
> > > > --- a/target-i386/cpu.c
> > > > +++ b/target-i386/cpu.c
> > > > @@ -1272,6 +1272,45 @@ static void x86_cpuid_set_tsc_freq(Object *obj, Visitor *v, void *opaque,
> > > >      cpu->env.tsc_khz = value / 1000;
> > > >  }
> > > >  
> > > > +static void x86_cpuid_get_apic_id(Object *obj, Visitor *v, void *opaque,
> > > > +                                  const char *name, Error **errp)
> > > > +{
> > > > +    X86CPU *cpu = X86_CPU(obj);
> > > > +    int64_t value = cpu->env.cpuid_apic_id;
> > > > +
> > > > +    visit_type_int(v, &value, name, errp);
> > > > +}
> > > > +
> > > > +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> > > > +                                  const char *name, Error **errp)
> > > > +{
> > > > +    X86CPU *cpu = X86_CPU(obj);
> > > > +    const int64_t min = 0;
> > > > +    const int64_t max = UINT32_MAX;
> > > > +    Error *error = NULL;
> > > > +    int64_t value;
> > > > +
> > > > +    visit_type_int(v, &value, name, &error);
> > > > +    if (error) {
> > > > +        error_propagate(errp, error);
> > > > +        return;
> > > > +    }
> > > > +    if (value < min || value > max) {
> > > > +        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
> > > > +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> > > > +                   object_get_typename(obj), name, value, min, max);
> > > > +        error_propagate(errp, error);
> > > > +        return;
> > > > +    }
> > > > +
> > > > +    if (cpu_exists(value)) {
> > > > +        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
> > > > +        error_propagate(errp, error);
> > > 
> > > What about implementing this check after implementing the
> > > /machine/icc-bridge links? Then we can simply check if
> > > "/machine/icc-bridge/cpus[ID]" is already set.
> > because it's more generic and won't break even if link location changes
> 
> Well, if link location changed, we could change the cpu_exists()
> implementation. We don't even need to hardcode the icc-bridge path, we
> could simply follow appropriate links if we set them up.
> 
> My problem with cpu_exists() is that it is based on global state,
> instead of simply using the links between bus/devices to find out if the
> IDs are being allocated correctly. We could make it work like PCI: devfn
> conflicts are solved simply by looking at the list of devices in the
> specific PCIBus where the device is being added. See
> do_pci_register_device().
> 
> That said, I am not against the current approach if we don't have the
> necessary bus/links modelled yet. But if we are going to add icc-bridge,
> we could benefit from it to implement cpu_exists().
Well,check has to be done here anyway, regardless how cpu_exists() implemented.

BTW:
This discussion and below belongs more to 9/19 patch that implements
cpu_exists() than here.

> 
> > 
> > > 
> > > (And then icc-bridge could be responsible for ensuring APIC ID
> > > uniqueness, not the CPU object itself)
> > Yep, with cpu-add id could be/is verified before setting property, but if
> > one considers device_add or fictional -device cpuxxx on cmd line, then won't
> > be an external check for this. This check takes care about these use cases.
> 
> That's why I believe the CPU has to calculate the APIC ID based on
> signaling coming from other devices/buses, instead of allowing the APIC
You mean by board not by CPU, I suppose.

> ID to be directly specified in the device_add/-device options.  That's
> how real CPUs work: as the CPU manufacturer doesn't know what will be
> the package ID of the CPU, the APIC IDs are not hardcoded in the CPU;
> they are calculated based on the CPU topology and some socket identifier
> signal coming from the board.
that is why apic_id has been made a property, to be set from outside.

> 
> Comparing it with PCI again: devfn of PCI devices can be specified
> manually, but if it is not present you just need to look at the PCI bus
> to find out the next available devfn.
18/19 gives an option to lookup available APIC IDs in a target specific way.
If we come up with a better target agnostic way to lookup ID, we can always
add it on top of currently proposed implementation.

> Based on that, I thought that icc-bridge was going be the component
> responsible for the APIC ID allocation logic, considering that it
> already contains APIC-ID-based links to the CPU objects.
in 19/19 APIC IDs are allocated by board for each possible CPU, icc-bridge is
just convenient place for storing links now. If machine was QOM object, I'd
probably place it there.

> 
> 
> > 
> > > 
> > > > +        return;
> > > > +    }
> > > > +    cpu->env.cpuid_apic_id = value;
> > > > +}
> > > > +
> > > >  static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *name)
> > > >  {
> > > >      x86_def_t *def;
> > > > @@ -2259,6 +2298,9 @@ static void x86_cpu_initfn(Object *obj)
> > > >      object_property_add(obj, "tsc-frequency", "int",
> > > >                          x86_cpuid_get_tsc_freq,
> > > >                          x86_cpuid_set_tsc_freq, NULL, NULL, NULL);
> > > > +    object_property_add(obj, "apic-id", "int",
> > > > +                        x86_cpuid_get_apic_id,
> > > > +                        x86_cpuid_set_apic_id, NULL, NULL, NULL);
> > > >  
> > > >      env->cpuid_apic_id = x86_cpu_apic_id_from_index(cs->cpu_index);
> > > >  
> > > > -- 
> > > > 1.8.2
> > > > 
> > > 
> > > -- 
> > > Eduardo
> > > 
> > 
> > 
> > -- 
> > Regards,
> >   Igor
> 
> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Igor Mammedov]

On Fri, 12 Apr 2013 12:35:14 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 12, 2013 at 05:16:20PM +0200, Igor Mammedov wrote:
> > On Fri, 12 Apr 2013 10:35:53 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Fri, Apr 12, 2013 at 12:53:51PM +0200, Igor Mammedov wrote:
> > > > On Thu, 11 Apr 2013 15:59:40 -0300
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> > > > > > ... so that on reboot BIOS could read current available CPU count
> > > > > > 
> > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > > v2:
> > > > > >   * s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> > > > > > ---
> > > > > >  hw/timer/mc146818rtc.c | 12 ++++++++++++
> > > > > >  1 file changed, 12 insertions(+)
> > > > > 
> > > > > Initialization of the cmos fields (including 0x5F) is done on
> > > > > pc.c:pc_cmos_init(). What about making the field increment inside pc.c
> > > > > as well?
> > > > I looked at possibility but discarded it because to increment it there initial
> > > > value should be -1 (field is zero based) which is not obvious, plug ugly
> > > > casting to singed variable.
> > > > Result looked ugly.
> > > 
> > > I was thinking about simply adding exactly the same code with exactly
> > > the same logic, but inside pc.c instead of of mc146818rtc.c. Instead of
> > > registering the notifier inside rtc_initfn(), register exactly the same
> > > notifier with exactly the same code, but inside pc_cmos_init() (that's
> > > where 0x5f is initialized).
> > > 
> > > It would even be safer and easier review and ensure correctness: with
> > > this patch, the notifier is registered very early, even before
> > > pc_cmos_init() initializes 0x5f to smp_cpus. CPU hotplug events are
> > > unlikely to be emitted before pc_cmos_init() is called, but still: why
> > it isn't be called, hot-add is available only after machine initialized.
> > 
> > > make the initialization ordering so subtle if we don't have to?
> > Currently cmos init doesn't look like proper QOM object and has 3 stage
> > initialization: realize(), then pc_cmos_init() the last pc_cmos_init_late().
> > The last 2 calls are made after realize(), setting various properties. Which
> > looks wrong from QOM perspective, so I'm against of stuffing more internal
> > stuff in arbitrary places. We should do opposite instead.
> 
> True, but as we already have this weird 3-stage initialization process
> and we won't fix it really soon, I would really prefer to keep parts of
> the code that are closely related and depend on each other in the same
> part of the code.
> 
> > 
> > If you look at mc146818rtc.c or hw/acpi/piix4.c, all notifiers are private to
> > object and registered at realize() time. It looks like initialization order
> > of mc146818rtc should be fixed, instead of adapting new code to it.
> > 
> > So since this patch doesn't break or violate anything in current code, I'd
> > like to leave it as it is.
> 
> If you insist into making the mc146818rtc device take care of
> maintaining the 0x5f value by itself, why not doing:
> 
>     s->cmos_data[0x5f] = smp_cpus - 1;
> 
> inside rtc_initfn() instead of pc_cmos_init() as well?
Device is used not only by target-i386.
Right way would be to redesign rtc_init() and rtc_initfn() and it would be
quite an intrusive patch.

That said it looks like current patch is incorrect if other targets
are considered, where s->cmos_data[0x5f] doesn't mean smp_cpus - 1. That looks
like a good reason to place notifier into pc.c and make it board specific.
I'll redo it for the next respin.

> 
> This would be one additional step towards making pc_cmos_init() be
> replaced by QOM-based code (if that's what you want to do in the long
> term).
> 
> 
> > 
> > > > > 
> > > > > What happens if a CPU is hotplugged after the machine has started but
> > > > > before the guest OS has booted? Are we supposed to make sure the BIOS do
> > > > > the right thing if a CPU is hotplugged before the OS has booted, or this
> > > > > simply won't be supported?
> > > > BIOS uses this value to set in ACPI tables what CPUs are present.
> > > >  1. if hot-plug happens before BIOS reads it then OS will see all CPUs
> > > > and SCI it receives will be nop.
> > > >  2. if hot-plug happens after BIOS reads it, OS will handle SCI as usual
> > > >     and hotplug CPU instead of initializing it smp_boot() time.
> > > > BIOS itself has nothing to do with hot-plug, it's OSPM job.
> > > 
> > > Makes sense, thanks.
> > > 
> > > What happens if the CPU is hotplugged after the BIOS builds the ACPI
> > > tables, but long before the OS starts handling ACPI events? Is the OS
> > > guaranteed to run the CPU hotplug ACPI method (that's \_GPE._E02 in the
> > > DSDT, right?), even if that happens?
> > Theoretically interrupt should not disappear on it's own and OS should pick it
> > up. But to say it for sure I need to test this case. If SCI will be lost for
> > some reason, then OS won't notice new CPU until another CPU hot-plug event
> > happens.
> 
> 
> Makes sense, thanks.
> 
> (Anyway, if an interrupt gets lost, it's probably a BIOS or OS bug).
> 
> > [...]
> 
> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 05:46:42PM +0200, Igor Mammedov wrote:
> On Fri, 12 Apr 2013 10:13:13 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
[...]
> > > > > +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> > > > > +                                  const char *name, Error **errp)
> > > > > +{
> > > > > +    X86CPU *cpu = X86_CPU(obj);
> > > > > +    const int64_t min = 0;
> > > > > +    const int64_t max = UINT32_MAX;
> > > > > +    Error *error = NULL;
> > > > > +    int64_t value;
> > > > > +
> > > > > +    visit_type_int(v, &value, name, &error);
> > > > > +    if (error) {
> > > > > +        error_propagate(errp, error);
> > > > > +        return;
> > > > > +    }
> > > > > +    if (value < min || value > max) {
> > > > > +        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
> > > > > +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> > > > > +                   object_get_typename(obj), name, value, min, max);
> > > > > +        error_propagate(errp, error);
> > > > > +        return;
> > > > > +    }
> > > > > +
> > > > > +    if (cpu_exists(value)) {
> > > > > +        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
> > > > > +        error_propagate(errp, error);
> > > > 
> > > > What about implementing this check after implementing the
> > > > /machine/icc-bridge links? Then we can simply check if
> > > > "/machine/icc-bridge/cpus[ID]" is already set.
> > > because it's more generic and won't break even if link location changes
> > 
> > Well, if link location changed, we could change the cpu_exists()
> > implementation. We don't even need to hardcode the icc-bridge path, we
> > could simply follow appropriate links if we set them up.
> > 
> > My problem with cpu_exists() is that it is based on global state,
> > instead of simply using the links between bus/devices to find out if the
> > IDs are being allocated correctly. We could make it work like PCI: devfn
> > conflicts are solved simply by looking at the list of devices in the
> > specific PCIBus where the device is being added. See
> > do_pci_register_device().
> > 
> > That said, I am not against the current approach if we don't have the
> > necessary bus/links modelled yet. But if we are going to add icc-bridge,
> > we could benefit from it to implement cpu_exists().
> Well,check has to be done here anyway, regardless how cpu_exists() implemented.
> 
> BTW:
> This discussion and below belongs more to 9/19 patch that implements
> cpu_exists() than here.

OK.

> 
> > 
> > > 
> > > > 
> > > > (And then icc-bridge could be responsible for ensuring APIC ID
> > > > uniqueness, not the CPU object itself)
> > > Yep, with cpu-add id could be/is verified before setting property, but if
> > > one considers device_add or fictional -device cpuxxx on cmd line, then won't
> > > be an external check for this. This check takes care about these use cases.
> > 
> > That's why I believe the CPU has to calculate the APIC ID based on
> > signaling coming from other devices/buses, instead of allowing the APIC
> You mean by board not by CPU, I suppose.

Actually I am not 100% sure. For example: we could simply provide the
socket/core/thread IDs (or links to socket/core objects?) to the CPU
object, and the CPU object could calculate the APIC ID itself.

> 
> > ID to be directly specified in the device_add/-device options.  That's
> > how real CPUs work: as the CPU manufacturer doesn't know what will be
> > the package ID of the CPU, the APIC IDs are not hardcoded in the CPU;
> > they are calculated based on the CPU topology and some socket identifier
> > signal coming from the board.
> that is why apic_id has been made a property, to be set from outside.

True. I believe the conflict here is: we want other objects to set the
APIC ID (be it the board, or socket/core objects), but at the same time
it would be interesting to not expose the APIC ID outside QEMU. Being
too flexible regarding the APIC ID is more likely to cause problems
later.

That said, I don't mind having a "apic-id" property because it is easier
to simply expose it directly. But do you agree that: 1) we don't really
need to expose it to be set from outside QEMU; 2) we shouldn't require
it to be set from outside QEMU; 3) we should recommend users to not try
to fiddle it with?


> 
> > 
> > Comparing it with PCI again: devfn of PCI devices can be specified
> > manually, but if it is not present you just need to look at the PCI bus
> > to find out the next available devfn.
> 18/19 gives an option to lookup available APIC IDs in a target specific way.
> If we come up with a better target agnostic way to lookup ID, we can always
> add it on top of currently proposed implementation.
> 
> > Based on that, I thought that icc-bridge was going be the component
> > responsible for the APIC ID allocation logic, considering that it
> > already contains APIC-ID-based links to the CPU objects.
> in 19/19 APIC IDs are allocated by board for each possible CPU, icc-bridge is
> just convenient place for storing links now. If machine was QOM object, I'd
> probably place it there.

I was just wondering if we could solve both problems at the same time
time: if we have a convenient place to store APIC-ID-based links, we are
getting a more efficient APIC ID allocation/check system for free.
Especially considering how inefficient is the method used by
cpu_exists().

It looks like the problem with my approach is that the QOM APIC-ID-based
links are target-specific, but cpu_exists() is target-independent. If
both were target-specific or both were target-independent, we could
easily reuse the same mechanism for both, but that's not the case now.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 03/19] cpu: make kvm-stub.o a part of CPU library

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 01:17:32PM +0200, Igor Mammedov wrote:
[...]
> > 
> > >  #include "qemu-common.h"
> > >  #include "hw/pci/pci.h"
> > >  
> > > @@ -47,4 +48,5 @@ static inline bool msi_present(const PCIDevice *dev)
> > >      return dev->cap_present & QEMU_PCI_CAP_MSI;
> > >  }
> > >  
> > > +#endif
> > >  #endif /* QEMU_MSI_H */
> > > diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h
> > > index 495e6f8..93cef28 100644
> > > --- a/include/sysemu/kvm.h
> > > +++ b/include/sysemu/kvm.h
> > > @@ -144,10 +144,10 @@ int kvm_cpu_exec(CPUArchState *env);
> > >  #if !defined(CONFIG_USER_ONLY)
> > >  void *kvm_vmalloc(ram_addr_t size);
> > >  void *kvm_arch_vmalloc(ram_addr_t size);
> > > -void kvm_setup_guest_memory(void *start, size_t size);
> > > +#endif
> > >  
> > > +void kvm_setup_guest_memory(void *start, size_t size);
> > >  void kvm_flush_coalesced_mmio_buffer(void);
> > > -#endif
> > >  
> > >  int kvm_insert_breakpoint(CPUArchState *current_env, target_ulong addr,
> > >                            target_ulong len, int type);
> > > diff --git a/kvm-stub.c b/kvm-stub.c
> > > index 82875dd..b34064a 100644
> > > --- a/kvm-stub.c
> > > +++ b/kvm-stub.c
> > > @@ -122,6 +122,7 @@ int kvm_on_sigbus(int code, void *addr)
> > >      return 1;
> > >  }
> > >  
> > > +#ifndef CONFIG_USER_ONLY
> > >  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
> > >  {
> > >      return -ENOSYS;
> > > @@ -145,3 +146,4 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
> > >  {
> > >      return -ENOSYS;
> > >  }
> > > +#endif
> > 
> > What about making MSIMessage definition available on CONFIG_USER_ONLY,
> > so we don't even need to #ifdef this part?
> 
> it might work, but it causes #includes split which I don't like much.
> Anyway here is patch on top of this one. If it looks more acceptable to you
> this way, I'll merge it.

OK, I believe I prefer your original patch. :-)

Having extra CONFIG_USER_ONLY #ifdefs in kvm-stub.c sounds simpler and
better than reordering #include lines to appear below declarations in
pci/msi.h like you did below.


> 
> ---
> 
> diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
> index d4d0450..c159a15 100644
> --- a/include/hw/pci/msi.h
> +++ b/include/hw/pci/msi.h
> @@ -21,15 +21,15 @@
>  #ifndef QEMU_MSI_H
>  #define QEMU_MSI_H
>  
> -#ifndef CONFIG_USER_ONLY
>  #include "qemu-common.h"

By the way, have you tried to eliminate the qemu-common.h include above?
(not important now, but asking just in case you have tried and found issues)

> -#include "hw/pci/pci.h"
>  
>  struct MSIMessage {
>      uint64_t address;
>      uint32_t data;
>  };
>  
> +#ifndef CONFIG_USER_ONLY
> +#include "hw/pci/pci.h"
>  extern bool msi_supported;
>  
>  void msi_set_message(PCIDevice *dev, MSIMessage msg);
> diff --git a/kvm-stub.c b/kvm-stub.c
> index 8cb81c4..b5d4446 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -123,7 +123,6 @@ int kvm_on_sigbus(int code, void *addr)
>      return 1;
>  }
>  
> -#ifndef CONFIG_USER_ONLY
>  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
>  {
>      return -ENOSYS;
> @@ -147,4 +146,3 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s, EventNotifier *n, int virq)
>  {
>      return -ENOSYS;
>  }
> -#endif
> 
> > 
> > -- 
> > Eduardo
> 
> 
> -- 
> Regards,
>   Igor

-- 
Eduardo

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[li guang]

在 2013-04-12五的 13:29 -0300，Eduardo Habkost写道：
> On Fri, Apr 12, 2013 at 05:46:42PM +0200, Igor Mammedov wrote:
> > On Fri, 12 Apr 2013 10:13:13 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> [...]
> > > > > > +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void *opaque,
> > > > > > +                                  const char *name, Error **errp)
> > > > > > +{
> > > > > > +    X86CPU *cpu = X86_CPU(obj);
> > > > > > +    const int64_t min = 0;
> > > > > > +    const int64_t max = UINT32_MAX;
> > > > > > +    Error *error = NULL;
> > > > > > +    int64_t value;
> > > > > > +
> > > > > > +    visit_type_int(v, &value, name, &error);
> > > > > > +    if (error) {
> > > > > > +        error_propagate(errp, error);
> > > > > > +        return;
> > > > > > +    }
> > > > > > +    if (value < min || value > max) {
> > > > > > +        error_setg(&error, "Property %s.%s doesn't take value %" PRId64
> > > > > > +                   " (minimum: %" PRId64 ", maximum: %" PRId64 ")" ,
> > > > > > +                   object_get_typename(obj), name, value, min, max);
> > > > > > +        error_propagate(errp, error);
> > > > > > +        return;
> > > > > > +    }
> > > > > > +
> > > > > > +    if (cpu_exists(value)) {
> > > > > > +        error_setg(&error, "CPU with APIC ID %" PRIi64 " exists", value);
> > > > > > +        error_propagate(errp, error);
> > > > > 
> > > > > What about implementing this check after implementing the
> > > > > /machine/icc-bridge links? Then we can simply check if
> > > > > "/machine/icc-bridge/cpus[ID]" is already set.
> > > > because it's more generic and won't break even if link location changes
> > > 
> > > Well, if link location changed, we could change the cpu_exists()
> > > implementation. We don't even need to hardcode the icc-bridge path, we
> > > could simply follow appropriate links if we set them up.
> > > 
> > > My problem with cpu_exists() is that it is based on global state,
> > > instead of simply using the links between bus/devices to find out if the
> > > IDs are being allocated correctly. We could make it work like PCI: devfn
> > > conflicts are solved simply by looking at the list of devices in the
> > > specific PCIBus where the device is being added. See
> > > do_pci_register_device().
> > > 
> > > That said, I am not against the current approach if we don't have the
> > > necessary bus/links modelled yet. But if we are going to add icc-bridge,
> > > we could benefit from it to implement cpu_exists().
> > Well,check has to be done here anyway, regardless how cpu_exists() implemented.
> > 
> > BTW:
> > This discussion and below belongs more to 9/19 patch that implements
> > cpu_exists() than here.
> 
> OK.
> 
> > 
> > > 
> > > > 
> > > > > 
> > > > > (And then icc-bridge could be responsible for ensuring APIC ID
> > > > > uniqueness, not the CPU object itself)
> > > > Yep, with cpu-add id could be/is verified before setting property, but if
> > > > one considers device_add or fictional -device cpuxxx on cmd line, then won't
> > > > be an external check for this. This check takes care about these use cases.
> > > 
> > > That's why I believe the CPU has to calculate the APIC ID based on
> > > signaling coming from other devices/buses, instead of allowing the APIC
> > You mean by board not by CPU, I suppose.
> 
> Actually I am not 100% sure. For example: we could simply provide the
> socket/core/thread IDs (or links to socket/core objects?) to the CPU
> object, and the CPU object could calculate the APIC ID itself.
> 
> > 
> > > ID to be directly specified in the device_add/-device options.  That's
> > > how real CPUs work: as the CPU manufacturer doesn't know what will be
> > > the package ID of the CPU, the APIC IDs are not hardcoded in the CPU;
> > > they are calculated based on the CPU topology and some socket identifier
> > > signal coming from the board.
> > that is why apic_id has been made a property, to be set from outside.
> 
> True. I believe the conflict here is: we want other objects to set the
> APIC ID (be it the board, or socket/core objects), 

ideally, we really prefer that,
but, you know, QEMU machine implementation seems can't feed us enough
to let us decide CPU ID as real hardware board, and neither in future
for QEMU doesn't like to emulate many complex signals that connect
devices.
so, I think, this property is just as well as we can do,
maybe we can't expect too much :-).

> but at the same time
> it would be interesting to not expose the APIC ID outside QEMU. Being
> too flexible regarding the APIC ID is more likely to cause problems
> later.
> 
> That said, I don't mind having a "apic-id" property because it is easier
> to simply expose it directly. But do you agree that: 1) we don't really
> need to expose it to be set from outside QEMU; 2) we shouldn't require
> it to be set from outside QEMU; 3) we should recommend users to not try
> to fiddle it with?
> 
> 
> > 
> > > 
> > > Comparing it with PCI again: devfn of PCI devices can be specified
> > > manually, but if it is not present you just need to look at the PCI bus
> > > to find out the next available devfn.
> > 18/19 gives an option to lookup available APIC IDs in a target specific way.
> > If we come up with a better target agnostic way to lookup ID, we can always
> > add it on top of currently proposed implementation.
> > 
> > > Based on that, I thought that icc-bridge was going be the component
> > > responsible for the APIC ID allocation logic, considering that it
> > > already contains APIC-ID-based links to the CPU objects.
> > in 19/19 APIC IDs are allocated by board for each possible CPU, icc-bridge is
> > just convenient place for storing links now. If machine was QOM object, I'd
> > probably place it there.
> 
> I was just wondering if we could solve both problems at the same time
> time: if we have a convenient place to store APIC-ID-based links, we are
> getting a more efficient APIC ID allocation/check system for free.
> Especially considering how inefficient is the method used by
> cpu_exists().
> 
> It looks like the problem with my approach is that the QOM APIC-ID-based
> links are target-specific, but cpu_exists() is target-independent. If
> both were target-specific or both were target-independent, we could
> easily reuse the same mechanism for both, but that's not the case now.
> 

Re: [Qemu-devel] [PATCH 03/19] cpu: make kvm-stub.o a part of CPU library

[Igor Mammedov]

On Fri, 12 Apr 2013 16:25:11 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 12, 2013 at 01:17:32PM +0200, Igor Mammedov wrote:
> [...]
> > > 
[...]
> > 
> > diff --git a/include/hw/pci/msi.h b/include/hw/pci/msi.h
> > index d4d0450..c159a15 100644
> > --- a/include/hw/pci/msi.h
> > +++ b/include/hw/pci/msi.h
> > @@ -21,15 +21,15 @@
> >  #ifndef QEMU_MSI_H
> >  #define QEMU_MSI_H
> >  
> > -#ifndef CONFIG_USER_ONLY
> >  #include "qemu-common.h"
> 
> By the way, have you tried to eliminate the qemu-common.h include above?
> (not important now, but asking just in case you have tried and found issues)
No, that would be just another bunch of not related cleanup patches.

> 
> > -#include "hw/pci/pci.h"
> >  
> >  struct MSIMessage {
> >      uint64_t address;
> >      uint32_t data;
> >  };
> >  
> > +#ifndef CONFIG_USER_ONLY
> > +#include "hw/pci/pci.h"
> >  extern bool msi_supported;
> >  
> >  void msi_set_message(PCIDevice *dev, MSIMessage msg);
> > diff --git a/kvm-stub.c b/kvm-stub.c
> > index 8cb81c4..b5d4446 100644
> > --- a/kvm-stub.c
> > +++ b/kvm-stub.c
> > @@ -123,7 +123,6 @@ int kvm_on_sigbus(int code, void *addr)
> >      return 1;
> >  }
> >  
> > -#ifndef CONFIG_USER_ONLY
> >  int kvm_irqchip_add_msi_route(KVMState *s, MSIMessage msg)
> >  {
> >      return -ENOSYS;
> > @@ -147,4 +146,3 @@ int kvm_irqchip_remove_irqfd_notifier(KVMState *s,
> > EventNotifier *n, int virq) {
> >      return -ENOSYS;
> >  }
> > -#endif
> > 
> > > 
> > > -- 
> > > Eduardo
> > 
> > 
> > -- 
> > Regards,
> >   Igor
> 

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Igor Mammedov]

On Fri, 12 Apr 2013 18:24:23 +0200
Igor Mammedov <imammedo@redhat.com> wrote:

> On Fri, 12 Apr 2013 12:35:14 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Fri, Apr 12, 2013 at 05:16:20PM +0200, Igor Mammedov wrote:
> > > On Fri, 12 Apr 2013 10:35:53 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Fri, Apr 12, 2013 at 12:53:51PM +0200, Igor Mammedov wrote:
> > > > > On Thu, 11 Apr 2013 15:59:40 -0300
> > > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > > 
> > > > > > On Thu, Apr 11, 2013 at 04:51:46PM +0200, Igor Mammedov wrote:
> > > > > > > ... so that on reboot BIOS could read current available CPU
> > > > > > > count
> > > > > > > 
> > > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > > > v2:
> > > > > > >   *
> > > > > > > s/qemu_register_cpu_add_notifier()/qemu_register_cpu_added_notifier()/
> > > > > > > --- hw/timer/mc146818rtc.c | 12 ++++++++++++
> > > > > > >  1 file changed, 12 insertions(+)
> > > > > > 
> > > > > > Initialization of the cmos fields (including 0x5F) is done on
> > > > > > pc.c:pc_cmos_init(). What about making the field increment inside
> > > > > > pc.c as well?
> > > > > I looked at possibility but discarded it because to increment it
> > > > > there initial value should be -1 (field is zero based) which is not
> > > > > obvious, plug ugly casting to singed variable.
> > > > > Result looked ugly.
> > > > 
> > > > I was thinking about simply adding exactly the same code with exactly
> > > > the same logic, but inside pc.c instead of of mc146818rtc.c. Instead
> > > > of registering the notifier inside rtc_initfn(), register exactly the
> > > > same notifier with exactly the same code, but inside pc_cmos_init()
> > > > (that's where 0x5f is initialized).
> > > > 
> > > > It would even be safer and easier review and ensure correctness: with
> > > > this patch, the notifier is registered very early, even before
> > > > pc_cmos_init() initializes 0x5f to smp_cpus. CPU hotplug events are
> > > > unlikely to be emitted before pc_cmos_init() is called, but still: why
> > > it isn't be called, hot-add is available only after machine initialized.
> > > 
> > > > make the initialization ordering so subtle if we don't have to?
> > > Currently cmos init doesn't look like proper QOM object and has 3 stage
> > > initialization: realize(), then pc_cmos_init() the last
> > > pc_cmos_init_late(). The last 2 calls are made after realize(), setting
> > > various properties. Which looks wrong from QOM perspective, so I'm
> > > against of stuffing more internal stuff in arbitrary places. We should
> > > do opposite instead.
> > 
> > True, but as we already have this weird 3-stage initialization process
> > and we won't fix it really soon, I would really prefer to keep parts of
> > the code that are closely related and depend on each other in the same
> > part of the code.
> > 
> > > 
> > > If you look at mc146818rtc.c or hw/acpi/piix4.c, all notifiers are
> > > private to object and registered at realize() time. It looks like
> > > initialization order of mc146818rtc should be fixed, instead of
> > > adapting new code to it.
> > > 
> > > So since this patch doesn't break or violate anything in current code,
> > > I'd like to leave it as it is.
> > 
> > If you insist into making the mc146818rtc device take care of
> > maintaining the 0x5f value by itself, why not doing:
> > 
> >     s->cmos_data[0x5f] = smp_cpus - 1;
> > 
> > inside rtc_initfn() instead of pc_cmos_init() as well?
> Device is used not only by target-i386.
> Right way would be to redesign rtc_init() and rtc_initfn() and it would be
> quite an intrusive patch.
> 
> That said it looks like current patch is incorrect if other targets
> are considered, where s->cmos_data[0x5f] doesn't mean smp_cpus - 1. That
> looks like a good reason to place notifier into pc.c and make it board
> specific. I'll redo it for the next respin.
> 
On the other hand, it probably would be better to make it a method and
override it in pc.c

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

On Fri, 12 Apr 2013 13:29:40 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 12, 2013 at 05:46:42PM +0200, Igor Mammedov wrote:
> > On Fri, 12 Apr 2013 10:13:13 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> [...]
> > > > > > +static void x86_cpuid_set_apic_id(Object *obj, Visitor *v, void
> > > > > > *opaque,
> > > > > > +                                  const char *name, Error **errp)
> > > > > > +{
> > > > > > +    X86CPU *cpu = X86_CPU(obj);
> > > > > > +    const int64_t min = 0;
> > > > > > +    const int64_t max = UINT32_MAX;
> > > > > > +    Error *error = NULL;
> > > > > > +    int64_t value;
> > > > > > +
> > > > > > +    visit_type_int(v, &value, name, &error);
> > > > > > +    if (error) {
> > > > > > +        error_propagate(errp, error);
> > > > > > +        return;
> > > > > > +    }
> > > > > > +    if (value < min || value > max) {
> > > > > > +        error_setg(&error, "Property %s.%s doesn't take value %"
> > > > > > PRId64
> > > > > > +                   " (minimum: %" PRId64 ", maximum: %" PRId64
> > > > > > ")" ,
> > > > > > +                   object_get_typename(obj), name, value, min,
> > > > > > max);
> > > > > > +        error_propagate(errp, error);
> > > > > > +        return;
> > > > > > +    }
> > > > > > +
> > > > > > +    if (cpu_exists(value)) {
> > > > > > +        error_setg(&error, "CPU with APIC ID %" PRIi64 "
> > > > > > exists", value);
> > > > > > +        error_propagate(errp, error);
> > > > > 
> > > > > What about implementing this check after implementing the
> > > > > /machine/icc-bridge links? Then we can simply check if
> > > > > "/machine/icc-bridge/cpus[ID]" is already set.
> > > > because it's more generic and won't break even if link location
> > > > changes
> > > 
> > > Well, if link location changed, we could change the cpu_exists()
> > > implementation. We don't even need to hardcode the icc-bridge path, we
> > > could simply follow appropriate links if we set them up.
> > > 
> > > My problem with cpu_exists() is that it is based on global state,
> > > instead of simply using the links between bus/devices to find out if the
> > > IDs are being allocated correctly. We could make it work like PCI: devfn
> > > conflicts are solved simply by looking at the list of devices in the
> > > specific PCIBus where the device is being added. See
> > > do_pci_register_device().
> > > 
> > > That said, I am not against the current approach if we don't have the
> > > necessary bus/links modelled yet. But if we are going to add icc-bridge,
> > > we could benefit from it to implement cpu_exists().
> > Well,check has to be done here anyway, regardless how cpu_exists()
> > implemented.
> > 
> > BTW:
> > This discussion and below belongs more to 9/19 patch that implements
> > cpu_exists() than here.
> 
> OK.
> 
> > 
> > > 
> > > > 
> > > > > 
> > > > > (And then icc-bridge could be responsible for ensuring APIC ID
> > > > > uniqueness, not the CPU object itself)
> > > > Yep, with cpu-add id could be/is verified before setting property,
> > > > but if one considers device_add or fictional -device cpuxxx on cmd
> > > > line, then won't be an external check for this. This check takes care
> > > > about these use cases.
> > > 
> > > That's why I believe the CPU has to calculate the APIC ID based on
> > > signaling coming from other devices/buses, instead of allowing the APIC
> > You mean by board not by CPU, I suppose.
> 
> Actually I am not 100% sure. For example: we could simply provide the
> socket/core/thread IDs (or links to socket/core objects?) to the CPU
> object, and the CPU object could calculate the APIC ID itself.
> 
> > 
> > > ID to be directly specified in the device_add/-device options.  That's
> > > how real CPUs work: as the CPU manufacturer doesn't know what will be
> > > the package ID of the CPU, the APIC IDs are not hardcoded in the CPU;
> > > they are calculated based on the CPU topology and some socket identifier
> > > signal coming from the board.
> > that is why apic_id has been made a property, to be set from outside.
> 
> True. I believe the conflict here is: we want other objects to set the
> APIC ID (be it the board, or socket/core objects), but at the same time
> it would be interesting to not expose the APIC ID outside QEMU. Being
> too flexible regarding the APIC ID is more likely to cause problems
> later.
> 
> That said, I don't mind having a "apic-id" property because it is easier
> to simply expose it directly. But do you agree that: 1) we don't really
> need to expose it to be set from outside QEMU; 2) we shouldn't require
> it to be set from outside QEMU; 3) we should recommend users to not try
> to fiddle it with?
Due to nature of per thread CPU hotplug, management will have to specify some
kind of ID to specify which CPU is being plugged. Management really
doesn't/shouldn't care what this ID is.

> 
> > 
> > > 
> > > Comparing it with PCI again: devfn of PCI devices can be specified
> > > manually, but if it is not present you just need to look at the PCI bus
> > > to find out the next available devfn.
> > 18/19 gives an option to lookup available APIC IDs in a target specific
> > way. If we come up with a better target agnostic way to lookup ID, we can
> > always add it on top of currently proposed implementation.
> > 
> > > Based on that, I thought that icc-bridge was going be the component
> > > responsible for the APIC ID allocation logic, considering that it
> > > already contains APIC-ID-based links to the CPU objects.
> > in 19/19 APIC IDs are allocated by board for each possible CPU,
> > icc-bridge is just convenient place for storing links now. If machine was
> > QOM object, I'd probably place it there.
> 
> I was just wondering if we could solve both problems at the same time
> time: if we have a convenient place to store APIC-ID-based links, we are
> getting a more efficient APIC ID allocation/check system for free.
> Especially considering how inefficient is the method used by
> cpu_exists().
Like I said, when we come up with target-independent links interface,
we can easily switch implementation of cpu_exists(). Right now inefficiency
here isn't issue since it's used on slow path only.

> 
> It looks like the problem with my approach is that the QOM APIC-ID-based
> links are target-specific, but cpu_exists() is target-independent. If
> both were target-specific or both were target-independent, we could
> easily reuse the same mechanism for both, but that's not the case now.
> 

cpu_exists() is target-independent but it look-ups CPU by target specific ID,
that target provides via get_arch_id() hook. So for target-i386 it will be
APIC ID.

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 03:37:00PM +0200, Igor Mammedov wrote:
[...]
> > > > ID to be directly specified in the device_add/-device options.  That's
> > > > how real CPUs work: as the CPU manufacturer doesn't know what will be
> > > > the package ID of the CPU, the APIC IDs are not hardcoded in the CPU;
> > > > they are calculated based on the CPU topology and some socket identifier
> > > > signal coming from the board.
> > > that is why apic_id has been made a property, to be set from outside.
> > 
> > True. I believe the conflict here is: we want other objects to set the
> > APIC ID (be it the board, or socket/core objects), but at the same time
> > it would be interesting to not expose the APIC ID outside QEMU. Being
> > too flexible regarding the APIC ID is more likely to cause problems
> > later.
> > 
> > That said, I don't mind having a "apic-id" property because it is easier
> > to simply expose it directly. But do you agree that: 1) we don't really
> > need to expose it to be set from outside QEMU; 2) we shouldn't require
> > it to be set from outside QEMU; 3) we should recommend users to not try
> > to fiddle it with?
> Due to nature of per thread CPU hotplug, management will have to specify some
> kind of ID to specify which CPU is being plugged. Management really
> doesn't/shouldn't care what this ID is.

As long as management really doesn't/shouldn't care what the ID is,
exposing the APIC ID in the form of an opaque CPU identifier wouldn't be
a problem to me. I just wanted to make clarify if we agree that messing
with the APIC ID directly won't be recommended and that the "apic-id"
property will be for QEMU internal use only.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Igor Mammedov]

On Fri, 12 Apr 2013 09:44:09 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Fri, Apr 12, 2013 at 12:01:03PM +0200, Igor Mammedov wrote:
> > On Thu, 11 Apr 2013 14:19:37 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > > > ... and leave links for not present CPUs empty.
> > > > 
> > > > It will allow users to query for possible APIC IDs and use them
> > > > with cpu-add QMP command.
> > > > 
> > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > 
> > > I don't see anything wrong with having icc-bridge links as well, but I
> > > would really like to have a target-independent namespace with links,
> > > that could be used to query for the available/valid CPU IDs for cpu-add
> > > commands instead of icc-bridge. The IDs on that namespace could be
> > > considered completely opaque.
> > 
> > Considering that -numa in present state is not compatible with cpu-add
> > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > patch could be dropped without any harm. libvirt could just use
> > numbers from this sequence like it's doing with current cpu_set without
> > any ID discovery. 
> 
> But it's not -numa that makes APIC ID probing necessary, it's
> non-power-of-2 core/thread counts on -smp (that make APIC IDs not match
> CPU indexes).
> 
> "Don't use CPU hotplug with -numa" is easy to be understood by users and
> by libvirt, but "don't use CPU hotplug with non-power-of-2 cores/threads
> counts" is harder to explain.
> 
> 
> > 
> > So, I've postponed target independent until we have -numa reworked,
> > then we could have /machine/node/socket/cpu containers with links.
> > The problem that needs to be solved, is the links storage ownership.
> > Who should allocate and own it? If machine was QOM object already,
> > I'd go with machine but it's not yet.
> 
> If we use CPU index as argument to cpu-add, we don't need to handle all
> those problems right now, we don't need to expose an APIC ID discovery
> interface, we make it work even with non-power-of-2 cores/threads
yes, you will get non-power-of-2 working without ID look-up.

> counts, and we make it work with -numa.
But you won't get this since, only next non-plugged ID will work, due to how
cpu_index is allocated. You can't just overwrite it with new value without
breaking current code.

> 
> So, my big question is: why are we trying so hard to avoid using CPU
> indexes as argument to cpu-add, if it's so much easier, and it is an
> obvious solution that makes the interface target-independent without any
> extra effort?
Using cpu_index instead of APIC ID definitely is not effort free and requires
quite a bit of rewrite how its used currently, APIC ID is much much easier
and less risky choice in this regard.

As for target-independence, any kind of ID is target-independent if treated as
opaque. Given that with unplug should come not-contiguous ID usage, the
interface to track which CPUs are plugged would be needed anyway. So it could
be introduced with this series and provide ID look-up meanwhile. That would
give libvirt time actually to start using it, and just remove not-contiguous
ID restriction when unplug is ready with all necessary infrastructure
already around.

> 
> > 
> > > 
> > > > ---
> > > > v2:
> > > >  * s/get_firmware_id/get_arch_id/ due to rebase
> > > >  * rename cpu_add_notifier to cpu_added_notifier &
> > > >    icc_bridge_cpu_add_req -> icc_bridge_cpued_add_req
> > > > ---
> > > >  hw/cpu/icc_bus.c          | 46
> > > > ++++++++++++++++++++++++++++++++++++++++++++++
> > > > hw/i386/pc.c              |  9 +++++++-- include/hw/i386/icc_bus.h |
> > > > 2 ++ 3 files changed, 55 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/hw/cpu/icc_bus.c b/hw/cpu/icc_bus.c
> > > > index ab9623d..5c0b9d4 100644
> > > > --- a/hw/cpu/icc_bus.c
> > > > +++ b/hw/cpu/icc_bus.c
> > > > @@ -18,6 +18,7 @@
> > > >   */
> > > >  #include "hw/i386/icc_bus.h"
> > > >  #include "hw/sysbus.h"
> > > > +#include "sysemu/sysemu.h"
> > > >  
> > > >  static void icc_bus_initfn(Object *obj)
> > > >  {
> > > > @@ -61,15 +62,39 @@ typedef struct ICCBridgeState {
> > > >      SysBusDevice busdev;
> > > >      MemoryRegion apic_container;
> > > >      MemoryRegion ioapic_container;
> > > > +    Notifier cpu_added_notifier;
> > > > +    Object **links;
> > > >  } ICCBridgeState;
> > > >  #define ICC_BRIGDE(obj) OBJECT_CHECK(ICCBridgeState, (obj),
> > > > TYPE_ICC_BRIDGE) 
> > > >  
> > > > +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu_obj)
> > > > +{
> > > > +    gchar *name;
> > > > +    Error *error = NULL;
> > > > +    CPUState *cpu = CPU(cpu_obj);
> > > > +    int64_t id = CPU_GET_CLASS(cpu)->get_arch_id(cpu);
> > > > +
> > > > +    name = g_strdup_printf("cpu[%" PRIu32 "]",
> > > > x86_cpu_apic_id_from_index(id));
> > > > +    object_property_set_link(bridge, cpu_obj, name, &error);
> > > > +    g_free(name);
> > > > +
> > > > +    g_assert(error == NULL);
> > > > +}
> > > > +
> > > > +static void icc_bridge_cpu_added_req(Notifier *n, void *opaque)
> > > > +{
> > > > +    ICCBridgeState *s = container_of(n, ICCBridgeState,
> > > > cpu_added_notifier); +
> > > > +    icc_bridge_set_cpu_link(OBJECT(s), OBJECT(opaque));
> > > > +}
> > > > +
> > > >  static void icc_bridge_initfn(Object *obj)
> > > >  {
> > > >      ICCBridgeState *s = ICC_BRIGDE(obj);
> > > >      SysBusDevice *sb = SYS_BUS_DEVICE(obj);
> > > >      ICCBus *ibus;
> > > > +    int i;
> > > >  
> > > >      ibus = ICC_BUS(qbus_create(TYPE_ICC_BUS, DEVICE(obj),
> > > > "icc-bus")); 
> > > > @@ -85,12 +110,33 @@ static void icc_bridge_initfn(Object *obj)
> > > >      memory_region_init(&s->ioapic_container, "icc-ioapic-container",
> > > > 0x1000); sysbus_init_mmio(sb, &s->ioapic_container);
> > > >      ibus->ioapic_address_space = &s->ioapic_container;
> > > > +
> > > > +    s->links = g_malloc0(sizeof(Object *) * max_cpus);
> > > > +    for (i = 0; i < max_cpus; i++) {
> > > > +        gchar *cpu_name;
> > > > +
> > > > +        cpu_name = g_strdup_printf("cpu[%" PRIu32 "]",
> > > > +                                   x86_cpu_apic_id_from_index(i));
> > > > +        object_property_add_link(obj, cpu_name, TYPE_CPU,
> > > > &s->links[i], NULL);
> > > > +        g_free(cpu_name);
> > > > +    }
> > > > +
> > > > +    s->cpu_added_notifier.notify = icc_bridge_cpu_added_req;
> > > > +    qemu_register_cpu_added_notifier(&s->cpu_added_notifier);
> > > > +}
> > > > +
> > > > +static void icc_bridge_fini(Object *obj)
> > > > +{
> > > > +    ICCBridgeState *s = ICC_BRIGDE(obj);
> > > > +
> > > > +    g_free(s->links);
> > > >  }
> > > >  
> > > >  static const TypeInfo icc_bridge_info = {
> > > >      .name  = "icc-bridge",
> > > >      .parent = TYPE_SYS_BUS_DEVICE,
> > > >      .instance_init  = icc_bridge_initfn,
> > > > +    .instance_finalize  = icc_bridge_fini,
> > > >      .instance_size  = sizeof(ICCBridgeState),
> > > >  };
> > > >  
> > > > diff --git a/hw/i386/pc.c b/hw/i386/pc.c
> > > > index 6d5e164..ada235c 100644
> > > > --- a/hw/i386/pc.c
> > > > +++ b/hw/i386/pc.c
> > > > @@ -870,7 +870,8 @@ void pc_acpi_smi_interrupt(void *opaque, int irq,
> > > > int level) }
> > > >  }
> > > >  
> > > > -static X86CPU *pc_new_cpu(const char *cpu_model, int64_t apic_id,
> > > > Error **errp) +static X86CPU *pc_new_cpu(const char *cpu_model,
> > > > int64_t apic_id,
> > > > +                          SysBusDevice *icc_bridge, Error **errp)
> > > >  {
> > > >      X86CPU *cpu;
> > > >  
> > > > @@ -882,6 +883,10 @@ static X86CPU *pc_new_cpu(const char *cpu_model,
> > > > int64_t apic_id, Error **errp) object_property_set_int(OBJECT(cpu),
> > > > apic_id, "apic-id", errp); object_property_set_bool(OBJECT(cpu),
> > > > true, "realized", errp); 
> > > > +    if (icc_bridge != NULL) {
> > > > +        icc_bridge_set_cpu_link(OBJECT(icc_bridge), OBJECT(cpu));
> > > > +    }
> > > > +
> > > >      if (error_is_set(errp)) {
> > > >          if (cpu != NULL) {
> > > >              object_unref(OBJECT(cpu));
> > > > @@ -911,7 +916,7 @@ void pc_cpus_init(const char *cpu_model)
> > > >                                                   TYPE_ICC_BRIDGE,
> > > > NULL)); 
> > > >      for (i = 0; i < smp_cpus; i++) {
> > > > -        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i),
> > > > &error);
> > > > +        cpu = pc_new_cpu(cpu_model, x86_cpu_apic_id_from_index(i),
> > > > ib, &error); if (error) {
> > > >              fprintf(stderr, "%s\n", error_get_pretty(error));
> > > >              error_free(error);
> > > > diff --git a/include/hw/i386/icc_bus.h b/include/hw/i386/icc_bus.h
> > > > index 69a0278..bc31cd9 100644
> > > > --- a/include/hw/i386/icc_bus.h
> > > > +++ b/include/hw/i386/icc_bus.h
> > > > @@ -49,5 +49,7 @@ typedef struct ICCDeviceClass {
> > > >  
> > > >  #define TYPE_ICC_BRIDGE "icc-bridge"
> > > >  
> > > > +void icc_bridge_set_cpu_link(Object *bridge, Object *cpu);
> > > > +
> > > >  #endif /* CONFIG_USER_ONLY */
> > > >  #endif
> > > > -- 
> > > > 1.8.2
> > > > 
> > > 
> > > -- 
> > > Eduardo
> > 
> > 
> > -- 
> > Regards,
> >   Igor
> 

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

On Mon, 15 Apr 2013 10:45:24 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Mon, Apr 15, 2013 at 03:37:00PM +0200, Igor Mammedov wrote:
> [...]
> > > > > ID to be directly specified in the device_add/-device options.
> > > > > That's how real CPUs work: as the CPU manufacturer doesn't know
> > > > > what will be the package ID of the CPU, the APIC IDs are not
> > > > > hardcoded in the CPU; they are calculated based on the CPU topology
> > > > > and some socket identifier signal coming from the board.
> > > > that is why apic_id has been made a property, to be set from outside.
> > > 
> > > True. I believe the conflict here is: we want other objects to set the
> > > APIC ID (be it the board, or socket/core objects), but at the same time
> > > it would be interesting to not expose the APIC ID outside QEMU. Being
> > > too flexible regarding the APIC ID is more likely to cause problems
> > > later.
> > > 
> > > That said, I don't mind having a "apic-id" property because it is easier
> > > to simply expose it directly. But do you agree that: 1) we don't really
> > > need to expose it to be set from outside QEMU; 2) we shouldn't require
> > > it to be set from outside QEMU; 3) we should recommend users to not try
> > > to fiddle it with?
> > Due to nature of per thread CPU hotplug, management will have to specify
> > some kind of ID to specify which CPU is being plugged. Management really
> > doesn't/shouldn't care what this ID is.
> 
> As long as management really doesn't/shouldn't care what the ID is,
> exposing the APIC ID in the form of an opaque CPU identifier wouldn't be
> a problem to me. I just wanted to make clarify if we agree that messing
> with the APIC ID directly won't be recommended and that the "apic-id"
> property will be for QEMU internal use only.
On contrary, it's useful external feature, x86 guests see only APIC ID, since
it's the only ID they [should] know about. So guest aware mgmt could
definitely use apic_id propery to correlate CPU in guest with QEMU view of
them.
 

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 04:15:08PM +0200, Igor Mammedov wrote:
> On Fri, 12 Apr 2013 09:44:09 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Fri, Apr 12, 2013 at 12:01:03PM +0200, Igor Mammedov wrote:
> > > On Thu, 11 Apr 2013 14:19:37 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > > > > ... and leave links for not present CPUs empty.
> > > > > 
> > > > > It will allow users to query for possible APIC IDs and use them
> > > > > with cpu-add QMP command.
> > > > > 
> > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > 
> > > > I don't see anything wrong with having icc-bridge links as well, but I
> > > > would really like to have a target-independent namespace with links,
> > > > that could be used to query for the available/valid CPU IDs for cpu-add
> > > > commands instead of icc-bridge. The IDs on that namespace could be
> > > > considered completely opaque.
> > > 
> > > Considering that -numa in present state is not compatible with cpu-add
> > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > patch could be dropped without any harm. libvirt could just use
> > > numbers from this sequence like it's doing with current cpu_set without
> > > any ID discovery. 
> > 
> > But it's not -numa that makes APIC ID probing necessary, it's
> > non-power-of-2 core/thread counts on -smp (that make APIC IDs not match
> > CPU indexes).
> > 
> > "Don't use CPU hotplug with -numa" is easy to be understood by users and
> > by libvirt, but "don't use CPU hotplug with non-power-of-2 cores/threads
> > counts" is harder to explain.
> > 
> > 
> > > 
> > > So, I've postponed target independent until we have -numa reworked,
> > > then we could have /machine/node/socket/cpu containers with links.
> > > The problem that needs to be solved, is the links storage ownership.
> > > Who should allocate and own it? If machine was QOM object already,
> > > I'd go with machine but it's not yet.
> > 
> > If we use CPU index as argument to cpu-add, we don't need to handle all
> > those problems right now, we don't need to expose an APIC ID discovery
> > interface, we make it work even with non-power-of-2 cores/threads
> yes, you will get non-power-of-2 working without ID look-up.
> 
> > counts, and we make it work with -numa.
> But you won't get this since, only next non-plugged ID will work, due to how
> cpu_index is allocated. You can't just overwrite it with new value without
> breaking current code.

OK, I think I get it: the problem is the cpu_index field specifically,
because it is set automatically, and there may be lots of assumptions in
the code about it. Maybe I should rephrase my suggestion: let's replace
"CPU index" above with "a predictable ID allocation system where IDs
will be in the range 0..(max_cpus-1)". We don't even need to touch the
cpu_index field in the CPU objects if we think this is too risky.

I am just proposing that we use IDs so that we just need to add a
   apic_id = apic_id_for_cpu_index(id)
line at the beginning of the cpu-add implementation, and everything else
would look exactly the same.

Nothing else would change in your implementation, except that now we
won't need a ID lookup system for 1.5 because libvirt can assume that
the next available CPU ID will be smp_cpus+1.

There would be no requirement to make the index-based IDs contiguous,
just like there's no requirement to make APIC IDs contiguous. The only
difference is that CPU indexes will predictable IDs, that will be always
in the range 0..(max_cpus-1) and won't require an ID lookup mechanism.
And it will work with non-power-of-2 threads/counts, and it will work
with -numa.

> 
> > 
> > So, my big question is: why are we trying so hard to avoid using CPU
> > indexes as argument to cpu-add, if it's so much easier, and it is an
> > obvious solution that makes the interface target-independent without any
> > extra effort?
> Using cpu_index instead of APIC ID definitely is not effort free and requires
> quite a bit of rewrite how its used currently, APIC ID is much much easier
> and less risky choice in this regard.
> 
> As for target-independence, any kind of ID is target-independent if treated as
> opaque.

True, as long as we have a target-independent ID lookup system, which we
don't have today. That's the part where we would need less effort.


> Given that with unplug should come not-contiguous ID usage, the
> interface to track which CPUs are plugged would be needed anyway. So it could
> be introduced with this series and provide ID look-up meanwhile. That would
> give libvirt time actually to start using it, and just remove not-contiguous
> ID restriction when unplug is ready with all necessary infrastructure
> already around.

Let's rephrase my suggestion: I don't think we should use the internal
"cpu->cpu_index" field as ID, necessarily. I only suggest that we make
the IDs predictably in the range 0..(max_cpus-1) so we:
1) don't need an ID lookup mechanism;
2) keep it compatible with the existing -numa options.

If we really want to use APIC ID one day, we may implement an lookup
mechanism that will provide IDs to libvirt, and declare the "IDs between
0..(max_cpus-1)" assumption as deprecated, so we can start using APIC
IDs as the (opaque) CPU IDs in the future. But before we do that, we
would implement a better "-numa" interface first.

You even suggested that we did something similar, above:

> > > Considering that -numa in present state is not compatible with cpu-add
> > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > patch could be dropped without any harm. libvirt could just use
> > > numbers from this sequence like it's doing with current cpu_set without
> > > any ID discovery. 

What I suggest is that we let libvirt make exactly the same assumptions
you suggest, but if we add a single line to the code:
    apic_id = apic_id_for_cpu_index(id)
we will make this intermediate solution work with NUMA _and_ work with
non-power-of-2 cores/threads counts.

The other cases (non-numa with power-of-2 cores/threads counts) would
work exactly the same, because in those cases
apic_id_for_cpu_index(id) == id.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 04:34:41PM +0200, Igor Mammedov wrote:
> On Mon, 15 Apr 2013 10:45:24 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Mon, Apr 15, 2013 at 03:37:00PM +0200, Igor Mammedov wrote:
> > [...]
> > > > > > ID to be directly specified in the device_add/-device options.
> > > > > > That's how real CPUs work: as the CPU manufacturer doesn't know
> > > > > > what will be the package ID of the CPU, the APIC IDs are not
> > > > > > hardcoded in the CPU; they are calculated based on the CPU topology
> > > > > > and some socket identifier signal coming from the board.
> > > > > that is why apic_id has been made a property, to be set from outside.
> > > > 
> > > > True. I believe the conflict here is: we want other objects to set the
> > > > APIC ID (be it the board, or socket/core objects), but at the same time
> > > > it would be interesting to not expose the APIC ID outside QEMU. Being
> > > > too flexible regarding the APIC ID is more likely to cause problems
> > > > later.
> > > > 
> > > > That said, I don't mind having a "apic-id" property because it is easier
> > > > to simply expose it directly. But do you agree that: 1) we don't really
> > > > need to expose it to be set from outside QEMU; 2) we shouldn't require
> > > > it to be set from outside QEMU; 3) we should recommend users to not try
> > > > to fiddle it with?
> > > Due to nature of per thread CPU hotplug, management will have to specify
> > > some kind of ID to specify which CPU is being plugged. Management really
> > > doesn't/shouldn't care what this ID is.
> > 
> > As long as management really doesn't/shouldn't care what the ID is,
> > exposing the APIC ID in the form of an opaque CPU identifier wouldn't be
> > a problem to me. I just wanted to make clarify if we agree that messing
> > with the APIC ID directly won't be recommended and that the "apic-id"
> > property will be for QEMU internal use only.
> On contrary, it's useful external feature, x86 guests see only APIC ID, since
> it's the only ID they [should] know about. So guest aware mgmt could
> definitely use apic_id propery to correlate CPU in guest with QEMU view of
> them.

You're right, _reading_ the APIC ID is very useful. I am worried about
_setting_ it from external code.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 01/19] target-i386: split out CPU creation and features parsing into cpu_x86_create()

[Andreas Färber]

Am 11.04.2013 16:51, schrieb Igor Mammedov:
> Move CPU creation and features parsing into a separate cpu_x86_create()
> function, so that board would be able to set board specific CPU
> properties before CPU is realized.
> 
> Keep cpu_x86_init() for compatibility with the code that uses cpu_init()
> and doesn't need to modify CPU properties.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
> v2:
>   * use error_propagate() to rise error up stack

Thanks, applied to qom-cpu (shortening the subject):
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 02/19] cpu: Pass CPUState to *cpu_synchronize_post*()

[Andreas Färber]

Am 11.04.2013 16:51, schrieb Igor Mammedov:
> ... so it could be called from without requiring CPUArchState
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Thanks a lot, applied to qom-cpu (with one change below):
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

I believe this is the missing prerequisite to cleaning up after my
monitor qemu_get_cpu() patch.

> diff --git a/kvm-stub.c b/kvm-stub.c
> index 760aadc..82875dd 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -42,11 +42,11 @@ void kvm_cpu_synchronize_state(CPUArchState *env)
>  {
>  }
>  
> -void kvm_cpu_synchronize_post_reset(CPUArchState *env)
> +void kvm_cpu_synchronize_post_reset(CPUState *env)

Renamed argument to cpu for consistency.

>  {
>  }
>  
> -void kvm_cpu_synchronize_post_init(CPUArchState *env)
> +void kvm_cpu_synchronize_post_init(CPUState *cpu)
>  {
>  }
>  

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Igor Mammedov]

On Mon, 15 Apr 2013 11:48:45 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Mon, Apr 15, 2013 at 04:15:08PM +0200, Igor Mammedov wrote:
> > On Fri, 12 Apr 2013 09:44:09 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Fri, Apr 12, 2013 at 12:01:03PM +0200, Igor Mammedov wrote:
> > > > On Thu, 11 Apr 2013 14:19:37 -0300
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > > > > > ... and leave links for not present CPUs empty.
> > > > > > 
> > > > > > It will allow users to query for possible APIC IDs and use them
> > > > > > with cpu-add QMP command.
> > > > > > 
> > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > 
> > > > > I don't see anything wrong with having icc-bridge links as well,
> > > > > but I would really like to have a target-independent namespace with
> > > > > links, that could be used to query for the available/valid CPU IDs
> > > > > for cpu-add commands instead of icc-bridge. The IDs on that
> > > > > namespace could be considered completely opaque.
> > > > 
> > > > Considering that -numa in present state is not compatible with cpu-add
> > > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > > patch could be dropped without any harm. libvirt could just use
> > > > numbers from this sequence like it's doing with current cpu_set
> > > > without any ID discovery. 
> > > 
> > > But it's not -numa that makes APIC ID probing necessary, it's
> > > non-power-of-2 core/thread counts on -smp (that make APIC IDs not match
> > > CPU indexes).
> > > 
> > > "Don't use CPU hotplug with -numa" is easy to be understood by users and
> > > by libvirt, but "don't use CPU hotplug with non-power-of-2 cores/threads
> > > counts" is harder to explain.
> > > 
> > > 
> > > > 
> > > > So, I've postponed target independent until we have -numa reworked,
> > > > then we could have /machine/node/socket/cpu containers with links.
> > > > The problem that needs to be solved, is the links storage ownership.
> > > > Who should allocate and own it? If machine was QOM object already,
> > > > I'd go with machine but it's not yet.
> > > 
> > > If we use CPU index as argument to cpu-add, we don't need to handle all
> > > those problems right now, we don't need to expose an APIC ID discovery
> > > interface, we make it work even with non-power-of-2 cores/threads
> > yes, you will get non-power-of-2 working without ID look-up.
> > 
> > > counts, and we make it work with -numa.
> > But you won't get this since, only next non-plugged ID will work, due to
> > how cpu_index is allocated. You can't just overwrite it with new value
> > without breaking current code.
> 
> OK, I think I get it: the problem is the cpu_index field specifically,
> because it is set automatically, and there may be lots of assumptions in
> the code about it. Maybe I should rephrase my suggestion: let's replace
> "CPU index" above with "a predictable ID allocation system where IDs
> will be in the range 0..(max_cpus-1)". We don't even need to touch the
> cpu_index field in the CPU objects if we think this is too risky.
> 
> I am just proposing that we use IDs so that we just need to add a
>    apic_id = apic_id_for_cpu_index(id)
> line at the beginning of the cpu-add implementation, and everything else
> would look exactly the same.
> 
> Nothing else would change in your implementation, except that now we
> won't need a ID lookup system for 1.5 because libvirt can assume that
> the next available CPU ID will be smp_cpus+1.
> 
> There would be no requirement to make the index-based IDs contiguous,
> just like there's no requirement to make APIC IDs contiguous. The only
> difference is that CPU indexes will predictable IDs, that will be always
> in the range 0..(max_cpus-1) and won't require an ID lookup mechanism.
> And it will work with non-power-of-2 threads/counts, and it will work
> with -numa.
> 
> > 
> > > 
> > > So, my big question is: why are we trying so hard to avoid using CPU
> > > indexes as argument to cpu-add, if it's so much easier, and it is an
> > > obvious solution that makes the interface target-independent without any
> > > extra effort?
> > Using cpu_index instead of APIC ID definitely is not effort free and
> > requires quite a bit of rewrite how its used currently, APIC ID is much
> > much easier and less risky choice in this regard.
> > 
> > As for target-independence, any kind of ID is target-independent if
> > treated as opaque.
> 
> True, as long as we have a target-independent ID lookup system, which we
> don't have today. That's the part where we would need less effort.
> 
> 
> > Given that with unplug should come not-contiguous ID usage, the
> > interface to track which CPUs are plugged would be needed anyway. So it
> > could be introduced with this series and provide ID look-up meanwhile.
> > That would give libvirt time actually to start using it, and just remove
> > not-contiguous ID restriction when unplug is ready with all necessary
> > infrastructure already around.
> 
> Let's rephrase my suggestion: I don't think we should use the internal
> "cpu->cpu_index" field as ID, necessarily. I only suggest that we make
> the IDs predictably in the range 0..(max_cpus-1) so we:
> 1) don't need an ID lookup mechanism;
> 2) keep it compatible with the existing -numa options.
> 
> If we really want to use APIC ID one day, we may implement an lookup
> mechanism that will provide IDs to libvirt, and declare the "IDs between
> 0..(max_cpus-1)" assumption as deprecated, so we can start using APIC
> IDs as the (opaque) CPU IDs in the future. But before we do that, we
> would implement a better "-numa" interface first.
> 
> You even suggested that we did something similar, above:
> 
> > > > Considering that -numa in present state is not compatible with cpu-add
> > > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > > patch could be dropped without any harm. libvirt could just use
> > > > numbers from this sequence like it's doing with current cpu_set
> > > > without any ID discovery. 
> 
> What I suggest is that we let libvirt make exactly the same assumptions
> you suggest, but if we add a single line to the code:
>     apic_id = apic_id_for_cpu_index(id)
> we will make this intermediate solution work with NUMA _and_ work with
> non-power-of-2 cores/threads counts.
> 
> The other cases (non-numa with power-of-2 cores/threads counts) would
> work exactly the same, because in those cases
> apic_id_for_cpu_index(id) == id.
> 

Rephrasing suggestion:
drop 18/19 and use apic_id = apic_id_for_cpu_index(id) in 19/19
Am I correct?

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Andreas Färber]

Am 11.04.2013 16:51, schrieb Igor Mammedov:
> get_arch_id() adds possibility for generic code to get guest visible
> CPI id without accessing CPUArchState. If target doesn't override it,
> it will return cpu_index.
> 
> Override it on target-i386 to return APIC ID.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  * it will be used later by new cpu_exists() generic function and
>    acpi_piix.
>  * s/cpu_firmware_id/cpu_arch_id/
> ---
>  include/qom/cpu.h |  2 ++
>  qom/cpu.c         |  6 ++++++
>  target-i386/cpu.c | 10 ++++++++++
>  3 files changed, 18 insertions(+)

I'm still unclear about this ID topic, but please don't add QOM methods
with arch_ in the name. The arch is expressed through subclasses.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 05:16:26PM +0200, Igor Mammedov wrote:
> On Mon, 15 Apr 2013 11:48:45 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Mon, Apr 15, 2013 at 04:15:08PM +0200, Igor Mammedov wrote:
> > > On Fri, 12 Apr 2013 09:44:09 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Fri, Apr 12, 2013 at 12:01:03PM +0200, Igor Mammedov wrote:
> > > > > On Thu, 11 Apr 2013 14:19:37 -0300
> > > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > > 
> > > > > > On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > > > > > > ... and leave links for not present CPUs empty.
> > > > > > > 
> > > > > > > It will allow users to query for possible APIC IDs and use them
> > > > > > > with cpu-add QMP command.
> > > > > > > 
> > > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > > 
> > > > > > I don't see anything wrong with having icc-bridge links as well,
> > > > > > but I would really like to have a target-independent namespace with
> > > > > > links, that could be used to query for the available/valid CPU IDs
> > > > > > for cpu-add commands instead of icc-bridge. The IDs on that
> > > > > > namespace could be considered completely opaque.
> > > > > 
> > > > > Considering that -numa in present state is not compatible with cpu-add
> > > > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > > > patch could be dropped without any harm. libvirt could just use
> > > > > numbers from this sequence like it's doing with current cpu_set
> > > > > without any ID discovery. 
> > > > 
> > > > But it's not -numa that makes APIC ID probing necessary, it's
> > > > non-power-of-2 core/thread counts on -smp (that make APIC IDs not match
> > > > CPU indexes).
> > > > 
> > > > "Don't use CPU hotplug with -numa" is easy to be understood by users and
> > > > by libvirt, but "don't use CPU hotplug with non-power-of-2 cores/threads
> > > > counts" is harder to explain.
> > > > 
> > > > 
> > > > > 
> > > > > So, I've postponed target independent until we have -numa reworked,
> > > > > then we could have /machine/node/socket/cpu containers with links.
> > > > > The problem that needs to be solved, is the links storage ownership.
> > > > > Who should allocate and own it? If machine was QOM object already,
> > > > > I'd go with machine but it's not yet.
> > > > 
> > > > If we use CPU index as argument to cpu-add, we don't need to handle all
> > > > those problems right now, we don't need to expose an APIC ID discovery
> > > > interface, we make it work even with non-power-of-2 cores/threads
> > > yes, you will get non-power-of-2 working without ID look-up.
> > > 
> > > > counts, and we make it work with -numa.
> > > But you won't get this since, only next non-plugged ID will work, due to
> > > how cpu_index is allocated. You can't just overwrite it with new value
> > > without breaking current code.
> > 
> > OK, I think I get it: the problem is the cpu_index field specifically,
> > because it is set automatically, and there may be lots of assumptions in
> > the code about it. Maybe I should rephrase my suggestion: let's replace
> > "CPU index" above with "a predictable ID allocation system where IDs
> > will be in the range 0..(max_cpus-1)". We don't even need to touch the
> > cpu_index field in the CPU objects if we think this is too risky.
> > 
> > I am just proposing that we use IDs so that we just need to add a
> >    apic_id = apic_id_for_cpu_index(id)
> > line at the beginning of the cpu-add implementation, and everything else
> > would look exactly the same.
> > 
> > Nothing else would change in your implementation, except that now we
> > won't need a ID lookup system for 1.5 because libvirt can assume that
> > the next available CPU ID will be smp_cpus+1.
> > 
> > There would be no requirement to make the index-based IDs contiguous,
> > just like there's no requirement to make APIC IDs contiguous. The only
> > difference is that CPU indexes will predictable IDs, that will be always
> > in the range 0..(max_cpus-1) and won't require an ID lookup mechanism.
> > And it will work with non-power-of-2 threads/counts, and it will work
> > with -numa.
> > 
> > > 
> > > > 
> > > > So, my big question is: why are we trying so hard to avoid using CPU
> > > > indexes as argument to cpu-add, if it's so much easier, and it is an
> > > > obvious solution that makes the interface target-independent without any
> > > > extra effort?
> > > Using cpu_index instead of APIC ID definitely is not effort free and
> > > requires quite a bit of rewrite how its used currently, APIC ID is much
> > > much easier and less risky choice in this regard.
> > > 
> > > As for target-independence, any kind of ID is target-independent if
> > > treated as opaque.
> > 
> > True, as long as we have a target-independent ID lookup system, which we
> > don't have today. That's the part where we would need less effort.
> > 
> > 
> > > Given that with unplug should come not-contiguous ID usage, the
> > > interface to track which CPUs are plugged would be needed anyway. So it
> > > could be introduced with this series and provide ID look-up meanwhile.
> > > That would give libvirt time actually to start using it, and just remove
> > > not-contiguous ID restriction when unplug is ready with all necessary
> > > infrastructure already around.
> > 
> > Let's rephrase my suggestion: I don't think we should use the internal
> > "cpu->cpu_index" field as ID, necessarily. I only suggest that we make
> > the IDs predictably in the range 0..(max_cpus-1) so we:
> > 1) don't need an ID lookup mechanism;
> > 2) keep it compatible with the existing -numa options.
> > 
> > If we really want to use APIC ID one day, we may implement an lookup
> > mechanism that will provide IDs to libvirt, and declare the "IDs between
> > 0..(max_cpus-1)" assumption as deprecated, so we can start using APIC
> > IDs as the (opaque) CPU IDs in the future. But before we do that, we
> > would implement a better "-numa" interface first.
> > 
> > You even suggested that we did something similar, above:
> > 
> > > > > Considering that -numa in present state is not compatible with cpu-add
> > > > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > > > patch could be dropped without any harm. libvirt could just use
> > > > > numbers from this sequence like it's doing with current cpu_set
> > > > > without any ID discovery. 
> > 
> > What I suggest is that we let libvirt make exactly the same assumptions
> > you suggest, but if we add a single line to the code:
> >     apic_id = apic_id_for_cpu_index(id)
> > we will make this intermediate solution work with NUMA _and_ work with
> > non-power-of-2 cores/threads counts.
> > 
> > The other cases (non-numa with power-of-2 cores/threads counts) would
> > work exactly the same, because in those cases
> > apic_id_for_cpu_index(id) == id.
> > 
> 
> Rephrasing suggestion:
> drop 18/19 and use apic_id = apic_id_for_cpu_index(id) in 19/19
> Am I correct?

Exactly. I mean: I expect that to be enough and not require other
changes, but I didn't look at all patches line-by-line yet.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 14/19] target-i386: replace MSI_SPACE_SIZE with APIC_SPACE_SIZE

[Andreas Färber]

Am 11.04.2013 16:51, schrieb Igor Mammedov:
> ... and put APIC_SPACE_SIZE in public header so that it could be
> reused later elsewhere.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

mst or Jan, can one of you please ack or nack?

Thanks,
Andreas

> ---
>  hw/i386/kvm/apic.c              | 2 +-
>  hw/intc/apic.c                  | 2 +-
>  hw/xen/xen_apic.c               | 2 +-
>  include/hw/i386/apic_internal.h | 2 --
>  target-i386/cpu.h               | 1 +
>  5 files changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/hw/i386/kvm/apic.c b/hw/i386/kvm/apic.c
> index c6ff982..8f80425 100644
> --- a/hw/i386/kvm/apic.c
> +++ b/hw/i386/kvm/apic.c
> @@ -174,7 +174,7 @@ static const MemoryRegionOps kvm_apic_io_ops = {
>  static void kvm_apic_init(APICCommonState *s)
>  {
>      memory_region_init_io(&s->io_memory, &kvm_apic_io_ops, s, "kvm-apic-msi",
> -                          MSI_SPACE_SIZE);
> +                          APIC_SPACE_SIZE);
>  
>      if (kvm_has_gsi_routing()) {
>          msi_supported = true;
> diff --git a/hw/intc/apic.c b/hw/intc/apic.c
> index 2d79a9e..756dff0 100644
> --- a/hw/intc/apic.c
> +++ b/hw/intc/apic.c
> @@ -874,7 +874,7 @@ static const MemoryRegionOps apic_io_ops = {
>  static void apic_init(APICCommonState *s)
>  {
>      memory_region_init_io(&s->io_memory, &apic_io_ops, s, "apic-msi",
> -                          MSI_SPACE_SIZE);
> +                          APIC_SPACE_SIZE);
>  
>      s->timer = qemu_new_timer_ns(vm_clock, apic_timer, s);
>      local_apics[s->idx] = s;
> diff --git a/hw/xen/xen_apic.c b/hw/xen/xen_apic.c
> index a2eb8a1..a958782 100644
> --- a/hw/xen/xen_apic.c
> +++ b/hw/xen/xen_apic.c
> @@ -39,7 +39,7 @@ static const MemoryRegionOps xen_apic_io_ops = {
>  static void xen_apic_init(APICCommonState *s)
>  {
>      memory_region_init_io(&s->io_memory, &xen_apic_io_ops, s, "xen-apic-msi",
> -                          MSI_SPACE_SIZE);
> +                          APIC_SPACE_SIZE);
>  
>  #if defined(CONFIG_XEN_CTRL_INTERFACE_VERSION) \
>      && CONFIG_XEN_CTRL_INTERFACE_VERSION >= 420
> diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
> index 578241f..aac6290 100644
> --- a/include/hw/i386/apic_internal.h
> +++ b/include/hw/i386/apic_internal.h
> @@ -66,8 +66,6 @@
>  
>  #define MAX_APICS 255
>  
> -#define MSI_SPACE_SIZE                  0x100000
> -
>  typedef struct APICCommonState APICCommonState;
>  
>  #define TYPE_APIC_COMMON "apic-common"
> diff --git a/target-i386/cpu.h b/target-i386/cpu.h
> index cf1b05c..65cf97d 100644
> --- a/target-i386/cpu.h
> +++ b/target-i386/cpu.h
> @@ -1269,5 +1269,6 @@ uint32_t x86_cpu_apic_id_from_index(unsigned int cpu_index);
>  void enable_compat_apic_id_mode(void);
>  
>  #define APIC_DEFAULT_ADDRESS 0xfee00000
> +#define APIC_SPACE_SIZE      0x100000
>  
>  #endif /* CPU_I386_H */
> 


-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Igor Mammedov]

On Mon, 15 Apr 2013 17:24:36 +0200
Andreas Färber <afaerber@suse.de> wrote:

> Am 11.04.2013 16:51, schrieb Igor Mammedov:
> > get_arch_id() adds possibility for generic code to get guest visible
> > CPI id without accessing CPUArchState. If target doesn't override it,
> > it will return cpu_index.
> > 
> > Override it on target-i386 to return APIC ID.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  * it will be used later by new cpu_exists() generic function and
> >    acpi_piix.
> >  * s/cpu_firmware_id/cpu_arch_id/
> > ---
> >  include/qom/cpu.h |  2 ++
> >  qom/cpu.c         |  6 ++++++
> >  target-i386/cpu.c | 10 ++++++++++
> >  3 files changed, 18 insertions(+)
> 
> I'm still unclear about this ID topic, but please don't add QOM methods
> with arch_ in the name. The arch is expressed through subclasses.

arch is there to explicitly show that returned ID is defined by target and
not generic code. This method then allow to implement cpu_exists() and helps
to init present cpus bitmap in  acpi_piix4.

Would be  cpu_id() acceptable?

> 
> Andreas
> 

Re: [Qemu-devel] [PATCH 13/19] target-i386: cpu: attach ICC bus to CPU on its creation

[Andreas Färber]

Am 11.04.2013 16:51, schrieb Igor Mammedov:
> ... during startup,  so it would be possible to uplug it later

unplug?

> and set bus_type to TYPE_ICC_BUS for X86CPU type to make device_add
> attach hotplugged CPU to ICC bus.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> ---
>  target-i386/cpu.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> index 4ddc18a..2bb38d6 100644
> --- a/target-i386/cpu.c
> +++ b/target-i386/cpu.c
> @@ -41,6 +41,7 @@
>  #endif
>  
>  #include "sysemu/sysemu.h"
> +#include "hw/i386/icc_bus.h"
>  #ifndef CONFIG_USER_ONLY
>  #include "hw/xen/xen.h"
>  #include "hw/sysbus.h"
> @@ -1610,6 +1611,7 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
>      gchar **model_pieces;
>      char *name, *features;
>      Error *error = NULL;
> +    Object *icc_bus = object_resolve_path_type("icc-bus", TYPE_ICC_BUS, NULL);
>  
>      model_pieces = g_strsplit(cpu_model, ",", 2);
>      if (!model_pieces[0]) {
> @@ -1620,6 +1622,10 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error **errp)
>      features = model_pieces[1];
>  
>      cpu = X86_CPU(object_new(TYPE_X86_CPU));
> +    if (icc_bus) {
> +        qdev_set_parent_bus(DEVICE(cpu), BUS(icc_bus));
> +        object_unref(OBJECT(cpu));

Why are you unref'ing here? (Paolo?)

> +    }
>      env = &cpu->env;
>      env->cpu_model_str = cpu_model;
>  

> @@ -2330,6 +2336,7 @@ static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
>  
>      xcc->parent_realize = dc->realize;
>      dc->realize = x86_cpu_realizefn;
> +    dc->bus_type = TYPE_ICC_BUS;
>  
>      xcc->parent_reset = cc->reset;
>      cc->reset = x86_cpu_reset;

I was relieved when I recently saw this was all the magic required to
put it onto the new ICC bus. I thought we'd need to mess with ICCDevice
base classes, but that seems to be used for APIC and IOAPIC only.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Andreas Färber]

Am 15.04.2013 17:34, schrieb Igor Mammedov:
> On Mon, 15 Apr 2013 17:24:36 +0200
> Andreas Färber <afaerber@suse.de> wrote:
> 
>> Am 11.04.2013 16:51, schrieb Igor Mammedov:
>>> get_arch_id() adds possibility for generic code to get guest visible
>>> CPI id without accessing CPUArchState. If target doesn't override it,
>>> it will return cpu_index.
>>>
>>> Override it on target-i386 to return APIC ID.
>>>
>>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>>> ---
>>>  * it will be used later by new cpu_exists() generic function and
>>>    acpi_piix.
>>>  * s/cpu_firmware_id/cpu_arch_id/
>>> ---
>>>  include/qom/cpu.h |  2 ++
>>>  qom/cpu.c         |  6 ++++++
>>>  target-i386/cpu.c | 10 ++++++++++
>>>  3 files changed, 18 insertions(+)
>>
>> I'm still unclear about this ID topic, but please don't add QOM methods
>> with arch_ in the name. The arch is expressed through subclasses.
> 
> arch is there to explicitly show that returned ID is defined by target and
> not generic code. This method then allow to implement cpu_exists() and helps
> to init present cpus bitmap in  acpi_piix4.
> 
> Would be  cpu_id() acceptable?

Certainly.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 14/19] target-i386: replace MSI_SPACE_SIZE with APIC_SPACE_SIZE

[Jan Kiszka]

On 2013-04-15 17:34, Andreas Färber wrote:
> Am 11.04.2013 16:51, schrieb Igor Mammedov:
>> ... and put APIC_SPACE_SIZE in public header so that it could be
>> reused later elsewhere.
>>
>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> 
> mst or Jan, can one of you please ack or nack?

Well, the problem remains that we are mixing up two distinct address
spaces in one mapping. But that's not the fault of the patch and nothing
this series can change (we need CPU-local MMIO dispatching to fix it).

IOW, renaming and relocation does not make things worse, so I'm fine
with it if it helps.

Jan

>> ---
>>  hw/i386/kvm/apic.c              | 2 +-
>>  hw/intc/apic.c                  | 2 +-
>>  hw/xen/xen_apic.c               | 2 +-
>>  include/hw/i386/apic_internal.h | 2 --
>>  target-i386/cpu.h               | 1 +
>>  5 files changed, 4 insertions(+), 5 deletions(-)
>>
>> diff --git a/hw/i386/kvm/apic.c b/hw/i386/kvm/apic.c
>> index c6ff982..8f80425 100644
>> --- a/hw/i386/kvm/apic.c
>> +++ b/hw/i386/kvm/apic.c
>> @@ -174,7 +174,7 @@ static const MemoryRegionOps kvm_apic_io_ops = {
>>  static void kvm_apic_init(APICCommonState *s)
>>  {
>>      memory_region_init_io(&s->io_memory, &kvm_apic_io_ops, s, "kvm-apic-msi",
>> -                          MSI_SPACE_SIZE);
>> +                          APIC_SPACE_SIZE);
>>  
>>      if (kvm_has_gsi_routing()) {
>>          msi_supported = true;
>> diff --git a/hw/intc/apic.c b/hw/intc/apic.c
>> index 2d79a9e..756dff0 100644
>> --- a/hw/intc/apic.c
>> +++ b/hw/intc/apic.c
>> @@ -874,7 +874,7 @@ static const MemoryRegionOps apic_io_ops = {
>>  static void apic_init(APICCommonState *s)
>>  {
>>      memory_region_init_io(&s->io_memory, &apic_io_ops, s, "apic-msi",
>> -                          MSI_SPACE_SIZE);
>> +                          APIC_SPACE_SIZE);
>>  
>>      s->timer = qemu_new_timer_ns(vm_clock, apic_timer, s);
>>      local_apics[s->idx] = s;
>> diff --git a/hw/xen/xen_apic.c b/hw/xen/xen_apic.c
>> index a2eb8a1..a958782 100644
>> --- a/hw/xen/xen_apic.c
>> +++ b/hw/xen/xen_apic.c
>> @@ -39,7 +39,7 @@ static const MemoryRegionOps xen_apic_io_ops = {
>>  static void xen_apic_init(APICCommonState *s)
>>  {
>>      memory_region_init_io(&s->io_memory, &xen_apic_io_ops, s, "xen-apic-msi",
>> -                          MSI_SPACE_SIZE);
>> +                          APIC_SPACE_SIZE);
>>  
>>  #if defined(CONFIG_XEN_CTRL_INTERFACE_VERSION) \
>>      && CONFIG_XEN_CTRL_INTERFACE_VERSION >= 420
>> diff --git a/include/hw/i386/apic_internal.h b/include/hw/i386/apic_internal.h
>> index 578241f..aac6290 100644
>> --- a/include/hw/i386/apic_internal.h
>> +++ b/include/hw/i386/apic_internal.h
>> @@ -66,8 +66,6 @@
>>  
>>  #define MAX_APICS 255
>>  
>> -#define MSI_SPACE_SIZE                  0x100000
>> -
>>  typedef struct APICCommonState APICCommonState;
>>  
>>  #define TYPE_APIC_COMMON "apic-common"
>> diff --git a/target-i386/cpu.h b/target-i386/cpu.h
>> index cf1b05c..65cf97d 100644
>> --- a/target-i386/cpu.h
>> +++ b/target-i386/cpu.h
>> @@ -1269,5 +1269,6 @@ uint32_t x86_cpu_apic_id_from_index(unsigned int cpu_index);
>>  void enable_compat_apic_id_mode(void);
>>  
>>  #define APIC_DEFAULT_ADDRESS 0xfee00000
>> +#define APIC_SPACE_SIZE      0x100000
>>  
>>  #endif /* CPU_I386_H */
>>

-- 
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 05:24:36PM +0200, Andreas Färber wrote:
> Am 11.04.2013 16:51, schrieb Igor Mammedov:
> > get_arch_id() adds possibility for generic code to get guest visible
> > CPI id without accessing CPUArchState. If target doesn't override it,
> > it will return cpu_index.
> > 
> > Override it on target-i386 to return APIC ID.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  * it will be used later by new cpu_exists() generic function and
> >    acpi_piix.
> >  * s/cpu_firmware_id/cpu_arch_id/
> > ---
> >  include/qom/cpu.h |  2 ++
> >  qom/cpu.c         |  6 ++++++
> >  target-i386/cpu.c | 10 ++++++++++
> >  3 files changed, 18 insertions(+)
> 
> I'm still unclear about this ID topic, but please don't add QOM methods
> with arch_ in the name. The arch is expressed through subclasses.

The naming seemed appropriate to me as it is an abstract method that is
expected to have arch-specific implementations and arch-specific
meaning.

On the other hand, "firmware_id" looked clearer to me (as it was very
specific about the meaning/usefulness of the return value), I don't know
what was the original reason to change from firmware_id to arch_id.

Other than "arch_id" and "firmware_id , I don't know how we could name
that method. It's not "just an ID", but a special kind of ID, so I
believe just "cpu_id" wouldn't work.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 17/19] qdev: set device's parent before calling realize() down inheritance chain.

[Andreas Färber]

Am 11.04.2013 16:51, schrieb Igor Mammedov:
> Currently device_set_realized() sets parent only after device was realized,
> But qdev_device_add() sets it before device is realized.
> Make behavior consistent and alter device_set_realized() to behave like
> qdev_device_add().
> 
> It will allow to set link<> properties in realize() method in classes
> inherited from DEVICE.
> 
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  * Usage examples
>     - conversion to CPU back-link property in APIC
>     - pre-allocate link<CPU[n]> at board level, and populate them
>       at realize time when CPUs are created.
> ---
>  hw/core/qdev.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Thanks, applied to qom-cpu:
https://github.com/afaerber/qemu-cpu/commits/qom-cpu

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 13/19] target-i386: cpu: attach ICC bus to CPU on its creation

[Igor Mammedov]

On Mon, 15 Apr 2013 17:38:51 +0200
Andreas Färber <afaerber@suse.de> wrote:

> Am 11.04.2013 16:51, schrieb Igor Mammedov:
> > ... during startup,  so it would be possible to uplug it later
> 
> unplug?
CPU won't be unplug-able if it's not on ICC bus.

> 
> > and set bus_type to TYPE_ICC_BUS for X86CPU type to make device_add
> > attach hotplugged CPU to ICC bus.
> > 
> > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > ---
> >  target-i386/cpu.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> > index 4ddc18a..2bb38d6 100644
> > --- a/target-i386/cpu.c
> > +++ b/target-i386/cpu.c
> > @@ -41,6 +41,7 @@
> >  #endif
> >  
> >  #include "sysemu/sysemu.h"
> > +#include "hw/i386/icc_bus.h"
> >  #ifndef CONFIG_USER_ONLY
> >  #include "hw/xen/xen.h"
> >  #include "hw/sysbus.h"
> > @@ -1610,6 +1611,7 @@ X86CPU *cpu_x86_create(const char *cpu_model, Error
> > **errp) gchar **model_pieces;
> >      char *name, *features;
> >      Error *error = NULL;
> > +    Object *icc_bus = object_resolve_path_type("icc-bus", TYPE_ICC_BUS,
> > NULL); 
> >      model_pieces = g_strsplit(cpu_model, ",", 2);
> >      if (!model_pieces[0]) {
> > @@ -1620,6 +1622,10 @@ X86CPU *cpu_x86_create(const char *cpu_model,
> > Error **errp) features = model_pieces[1];
> >  
> >      cpu = X86_CPU(object_new(TYPE_X86_CPU));
> > +    if (icc_bus) {
> > +        qdev_set_parent_bus(DEVICE(cpu), BUS(icc_bus));
> > +        object_unref(OBJECT(cpu));
> 
> Why are you unref'ing here? (Paolo?)

To leave only one reference to it since qdev_set_parent_bus() just
incremented it. Like it's done in qdev_try_create().

> 
> > +    }
> >      env = &cpu->env;
> >      env->cpu_model_str = cpu_model;
> >  
> 
> > @@ -2330,6 +2336,7 @@ static void x86_cpu_common_class_init(ObjectClass
> > *oc, void *data) 
> >      xcc->parent_realize = dc->realize;
> >      dc->realize = x86_cpu_realizefn;
> > +    dc->bus_type = TYPE_ICC_BUS;
> >  
> >      xcc->parent_reset = cc->reset;
> >      cc->reset = x86_cpu_reset;
> 
> I was relieved when I recently saw this was all the magic required to
> put it onto the new ICC bus. I thought we'd need to mess with ICCDevice
> base classes, but that seems to be used for APIC and IOAPIC only.
> 
> Andreas
> 

Re: [Qemu-devel] [PATCH 13/19] target-i386: cpu: attach ICC bus to CPU on its creation

[Andreas Färber]

Am 15.04.2013 17:49, schrieb Igor Mammedov:
> On Mon, 15 Apr 2013 17:38:51 +0200
> Andreas Färber <afaerber@suse.de> wrote:
> 
>> Am 11.04.2013 16:51, schrieb Igor Mammedov:
>>> ... during startup,  so it would be possible to uplug it later
>>
>> unplug?
> CPU won't be unplug-able if it's not on ICC bus.

Sorry? Question was whether "uplug" is a typo and should read "unplug"?
On openSUSE 12.3 there is no command called uplug. :)

>>> and set bus_type to TYPE_ICC_BUS for X86CPU type to make device_add
>>> attach hotplugged CPU to ICC bus.
>>>
>>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
>>> ---
>>>  target-i386/cpu.c | 7 +++++++
>>>  1 file changed, 7 insertions(+)
>>>
>>> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
>>> index 4ddc18a..2bb38d6 100644
>>> --- a/target-i386/cpu.c
>>> +++ b/target-i386/cpu.c
[...]
>>> @@ -1620,6 +1622,10 @@ X86CPU *cpu_x86_create(const char *cpu_model,
>>> Error **errp) features = model_pieces[1];
>>>  
>>>      cpu = X86_CPU(object_new(TYPE_X86_CPU));
>>> +    if (icc_bus) {
>>> +        qdev_set_parent_bus(DEVICE(cpu), BUS(icc_bus));
>>> +        object_unref(OBJECT(cpu));
>>
>> Why are you unref'ing here? (Paolo?)
> 
> To leave only one reference to it since qdev_set_parent_bus() just
> incremented it. Like it's done in qdev_try_create().

Ah, I see, thanks.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 13/19] target-i386: cpu: attach ICC bus to CPU on its creation

[Igor Mammedov]

On Mon, 15 Apr 2013 18:06:40 +0200
Andreas Färber <afaerber@suse.de> wrote:

> Am 15.04.2013 17:49, schrieb Igor Mammedov:
> > On Mon, 15 Apr 2013 17:38:51 +0200
> > Andreas Färber <afaerber@suse.de> wrote:
> > 
> >> Am 11.04.2013 16:51, schrieb Igor Mammedov:
> >>> ... during startup,  so it would be possible to uplug it later
> >>
> >> unplug?
> > CPU won't be unplug-able if it's not on ICC bus.
> 
> Sorry? Question was whether "uplug" is a typo and should read "unplug"?
> On openSUSE 12.3 there is no command called uplug. :)
Thanks, I'll fix it.

> 
> >>> and set bus_type to TYPE_ICC_BUS for X86CPU type to make device_add
> >>> attach hotplugged CPU to ICC bus.
> >>>
> >>> Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> >>> ---
> >>>  target-i386/cpu.c | 7 +++++++
> >>>  1 file changed, 7 insertions(+)
> >>>
> >>> diff --git a/target-i386/cpu.c b/target-i386/cpu.c
> >>> index 4ddc18a..2bb38d6 100644
> >>> --- a/target-i386/cpu.c
> >>> +++ b/target-i386/cpu.c
> [...]
> >>> @@ -1620,6 +1622,10 @@ X86CPU *cpu_x86_create(const char *cpu_model,
> >>> Error **errp) features = model_pieces[1];
> >>>  
> >>>      cpu = X86_CPU(object_new(TYPE_X86_CPU));
> >>> +    if (icc_bus) {
> >>> +        qdev_set_parent_bus(DEVICE(cpu), BUS(icc_bus));
> >>> +        object_unref(OBJECT(cpu));
> >>
> >> Why are you unref'ing here? (Paolo?)
> > 
> > To leave only one reference to it since qdev_set_parent_bus() just
> > incremented it. Like it's done in qdev_try_create().
> 
> Ah, I see, thanks.
> 
> Andreas
> 

Re: [Qemu-devel] [PATCH 07/19] rtc: update rtc_cmos on CPU hot-plug

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 11:38:45AM +0200, Igor Mammedov wrote:
[...]
> > > 
> > > If you insist into making the mc146818rtc device take care of
> > > maintaining the 0x5f value by itself, why not doing:
> > > 
> > >     s->cmos_data[0x5f] = smp_cpus - 1;
> > > 
> > > inside rtc_initfn() instead of pc_cmos_init() as well?
> > Device is used not only by target-i386.
> > Right way would be to redesign rtc_init() and rtc_initfn() and it would be
> > quite an intrusive patch.
> > 
> > That said it looks like current patch is incorrect if other targets
> > are considered, where s->cmos_data[0x5f] doesn't mean smp_cpus - 1. That
> > looks like a good reason to place notifier into pc.c and make it board
> > specific. I'll redo it for the next respin.
> > 
> On the other hand, it probably would be better to make it a method and
> override it in pc.c

I don't see it as a feature of the mc146818rtc chip itself, but just
something that the PC board does with the chip when stuff happens. So I
wouldn't try to make it a method of mc146818rtc, but just something
handled externally from the chip, entirely in the PC code.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 04/19] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:43PM +0200, Igor Mammedov wrote:
[...]
> diff --git a/kvm-all.c b/kvm-all.c
> index fc4e17c..1d17128 100644
> --- a/kvm-all.c
> +++ b/kvm-all.c
> @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
>  bool kvm_irqfds_allowed;
>  bool kvm_msi_via_irqfd_allowed;
>  bool kvm_gsi_routing_allowed;
> +bool kvm_allowed;
>  
>  static const KVMCapabilityInfo kvm_required_capabilites[] = {
>      KVM_CAP_INFO(USER_MEMORY),
> diff --git a/kvm-stub.c b/kvm-stub.c
> index b34064a..8cb81c4 100644
> --- a/kvm-stub.c
> +++ b/kvm-stub.c
> @@ -23,6 +23,7 @@ bool kvm_async_interrupts_allowed;
>  bool kvm_irqfds_allowed;
>  bool kvm_msi_via_irqfd_allowed;
>  bool kvm_gsi_routing_allowed;
> +bool kvm_allowed;

Why do you need kvm_allowed on kvm-stub.c? Isn't simpler and more
efficient to define kvm_enabled() as (0) on CONFIG_USER_ONLY?

-- 
Eduardo

Re: [Qemu-devel] [PATCH 05/19] cpu: resume CPU from CPUClass.cpu_common_realizefn() when it is hot-plugged

[Eduardo Habkost]

On Thu, Apr 11, 2013 at 04:51:44PM +0200, Igor Mammedov wrote:
> Signed-off-by: Igor Mammedov <imammedo@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>


> ---
>  cpus.c                | 11 ++++++++---
>  include/sysemu/cpus.h |  3 +++
>  qom/cpu.c             |  2 ++
>  stubs/Makefile.objs   |  1 +
>  stubs/resume_vcpu.c   |  6 ++++++
>  5 files changed, 20 insertions(+), 3 deletions(-)
>  create mode 100644 stubs/resume_vcpu.c
> 
> diff --git a/cpus.c b/cpus.c
> index 9b9a32f..f12624d 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -973,6 +973,13 @@ void pause_all_vcpus(void)
>      }
>  }
>  
> +void resume_vcpu(CPUState *cpu)
> +{
> +    cpu->stop = false;
> +    cpu->stopped = false;
> +    qemu_cpu_kick(cpu);
> +}
> +
>  void resume_all_vcpus(void)
>  {
>      CPUArchState *penv = first_cpu;
> @@ -980,9 +987,7 @@ void resume_all_vcpus(void)
>      qemu_clock_enable(vm_clock, true);
>      while (penv) {
>          CPUState *pcpu = ENV_GET_CPU(penv);
> -        pcpu->stop = false;
> -        pcpu->stopped = false;
> -        qemu_cpu_kick(pcpu);
> +        resume_vcpu(pcpu);
>          penv = penv->next_cpu;
>      }
>  }
> diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
> index 6502488..9437df5 100644
> --- a/include/sysemu/cpus.h
> +++ b/include/sysemu/cpus.h
> @@ -1,11 +1,14 @@
>  #ifndef QEMU_CPUS_H
>  #define QEMU_CPUS_H
>  
> +#include "qom/cpu.h"
> +
>  /* cpus.c */
>  void qemu_init_cpu_loop(void);
>  void resume_all_vcpus(void);
>  void pause_all_vcpus(void);
>  void cpu_stop_current(void);
> +void resume_vcpu(CPUState *cpu);
>  
>  void cpu_synchronize_all_states(void);
>  void cpu_synchronize_all_post_reset(void);
> diff --git a/qom/cpu.c b/qom/cpu.c
> index 0c76712..2869cfe 100644
> --- a/qom/cpu.c
> +++ b/qom/cpu.c
> @@ -21,6 +21,7 @@
>  #include "qom/cpu.h"
>  #include "qemu-common.h"
>  #include "sysemu/kvm.h"
> +#include "sysemu/cpus.h"
>  
>  void cpu_reset_interrupt(CPUState *cpu, int mask)
>  {
> @@ -60,6 +61,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
>  {
>      if (dev->hotplugged) {
>          cpu_synchronize_post_init(CPU(dev));
> +        resume_vcpu(CPU(dev));
>      }
>  }
>  
> diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
> index 9c55b34..28fb4f8 100644
> --- a/stubs/Makefile.objs
> +++ b/stubs/Makefile.objs
> @@ -23,3 +23,4 @@ stub-obj-y += sysbus.o
>  stub-obj-y += vm-stop.o
>  stub-obj-y += vmstate.o
>  stub-obj-$(CONFIG_WIN32) += fd-register.o
> +stub-obj-y += resume_vcpu.o
> diff --git a/stubs/resume_vcpu.c b/stubs/resume_vcpu.c
> new file mode 100644
> index 0000000..383b71a
> --- /dev/null
> +++ b/stubs/resume_vcpu.c
> @@ -0,0 +1,6 @@
> +#include "qemu-common.h"
> +#include "sysemu/cpus.h"
> +
> +void resume_vcpu(CPUState *cpu)
> +{
> +}
> -- 
> 1.8.2
> 
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 06/19] introduce CPU hot-plug notifier

[Eduardo Habkost]

On Fri, Apr 12, 2013 at 01:00:57PM +0200, Igor Mammedov wrote:
> On Thu, 11 Apr 2013 15:46:47 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Thu, Apr 11, 2013 at 04:51:45PM +0200, Igor Mammedov wrote:
> > > hot-added CPU will be distributed to acpi_piix4, rtc_cmos and icc_bridge
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> > 
> > Have you considered making a generic device creation/realization
> > notification mechanism, that could work with any device class?
> Not really, that would make notification handler a bit more complicated,
> since they will have to decide if event belong to them.

I don't think it would be complicated. It should be as simple as:

void qdev_register_realize_notifier(DeviceClass *dc, Notifier *notifier)
{
    notifier_list_add(&dc->realize_notifiers, notifier);
}

static void notify_device_realized(ObjectClass *klass, void *opaque)
{
    DeviceState *dev = (DeviceState *)opaque;
    DeviceClass *dc = DEVICE_CLASS(klass);
    notifier_list_notify(&dc->realize_notifiers, dev);
}

static void device_realize(DeviceState *dev, Error **err)
{
    [...]
    object_class_foreach(notify_device_realized, TYPE_DEVICE, true, dev);
}


> We could add it in future if there will be need for it.

OK.

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

> 
> > 
> > > ---
> > > v2:
> > >   * move notifier to qom/cpu.c and call it from CPUClass.realize() on hotplug
> > >   * remove get_firmware_id() since it belong to other patch
> > > ---
> > >  include/sysemu/sysemu.h |  3 +++
> > >  qom/cpu.c               | 12 ++++++++++++
> > >  2 files changed, 15 insertions(+)
> > > 
> > > diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> > > index 6578782..a8c3de1 100644
> > > --- a/include/sysemu/sysemu.h
> > > +++ b/include/sysemu/sysemu.h
> > > @@ -152,6 +152,9 @@ void do_pci_device_hot_remove(Monitor *mon, const QDict *qdict);
> > >  /* generic hotplug */
> > >  void drive_hot_add(Monitor *mon, const QDict *qdict);
> > >  
> > > +/* CPU hotplug */
> > > +void qemu_register_cpu_added_notifier(Notifier *notifier);
> > > +
> > >  /* pcie aer error injection */
> > >  void pcie_aer_inject_error_print(Monitor *mon, const QObject *data);
> > >  int do_pcie_aer_inject_error(Monitor *mon,
> > > diff --git a/qom/cpu.c b/qom/cpu.c
> > > index 2869cfe..fe85960 100644
> > > --- a/qom/cpu.c
> > > +++ b/qom/cpu.c
> > > @@ -22,6 +22,17 @@
> > >  #include "qemu-common.h"
> > >  #include "sysemu/kvm.h"
> > >  #include "sysemu/cpus.h"
> > > +#include "qemu/notify.h"
> > > +#include "sysemu/sysemu.h"
> > > +
> > > +/* CPU hot-plug notifiers */
> > > +static NotifierList cpu_added_notifiers =
> > > +    NOTIFIER_LIST_INITIALIZER(cpu_add_notifiers);
> > > +
> > > +void qemu_register_cpu_added_notifier(Notifier *notifier)
> > > +{
> > > +    notifier_list_add(&cpu_added_notifiers, notifier);
> > > +}
> > >  
> > >  void cpu_reset_interrupt(CPUState *cpu, int mask)
> > >  {
> > > @@ -62,6 +73,7 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
> > >      if (dev->hotplugged) {
> > >          cpu_synchronize_post_init(CPU(dev));
> > >          resume_vcpu(CPU(dev));
> > > +        notifier_list_notify(&cpu_added_notifiers, dev);
> > >      }
> > >  }
> > >  
> > > -- 
> > > 1.8.2
> > > 
> > 
> > -- 
> > Eduardo
> 
> 
> -- 
> Regards,
>   Igor
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 04/19] cpu: call cpu_synchronize_post_init() from CPUClass.realize() if hotplugged

[Igor Mammedov]

On Mon, 15 Apr 2013 16:57:10 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Thu, Apr 11, 2013 at 04:51:43PM +0200, Igor Mammedov wrote:
> [...]
> > diff --git a/kvm-all.c b/kvm-all.c
> > index fc4e17c..1d17128 100644
> > --- a/kvm-all.c
> > +++ b/kvm-all.c
> > @@ -109,6 +109,7 @@ bool kvm_async_interrupts_allowed;
> >  bool kvm_irqfds_allowed;
> >  bool kvm_msi_via_irqfd_allowed;
> >  bool kvm_gsi_routing_allowed;
> > +bool kvm_allowed;
> >  
> >  static const KVMCapabilityInfo kvm_required_capabilites[] = {
> >      KVM_CAP_INFO(USER_MEMORY),
> > diff --git a/kvm-stub.c b/kvm-stub.c
> > index b34064a..8cb81c4 100644
> > --- a/kvm-stub.c
> > +++ b/kvm-stub.c
> > @@ -23,6 +23,7 @@ bool kvm_async_interrupts_allowed;
> >  bool kvm_irqfds_allowed;
> >  bool kvm_msi_via_irqfd_allowed;
> >  bool kvm_gsi_routing_allowed;
> > +bool kvm_allowed;
> 
> Why do you need kvm_allowed on kvm-stub.c? Isn't simpler and more
> efficient to define kvm_enabled() as (0) on CONFIG_USER_ONLY?

qom/cpu.o is build only one time for all targets, header ifdef won't work here.

> -- 
> Eduardo


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

On Mon, 15 Apr 2013 11:49:40 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Mon, Apr 15, 2013 at 04:34:41PM +0200, Igor Mammedov wrote:
> > On Mon, 15 Apr 2013 10:45:24 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Mon, Apr 15, 2013 at 03:37:00PM +0200, Igor Mammedov wrote:
> > > [...]
> > > > > > > ID to be directly specified in the device_add/-device options.
> > > > > > > That's how real CPUs work: as the CPU manufacturer doesn't know
> > > > > > > what will be the package ID of the CPU, the APIC IDs are not
> > > > > > > hardcoded in the CPU; they are calculated based on the CPU topology
> > > > > > > and some socket identifier signal coming from the board.
> > > > > > that is why apic_id has been made a property, to be set from outside.
> > > > > 
> > > > > True. I believe the conflict here is: we want other objects to set the
> > > > > APIC ID (be it the board, or socket/core objects), but at the same time
> > > > > it would be interesting to not expose the APIC ID outside QEMU. Being
> > > > > too flexible regarding the APIC ID is more likely to cause problems
> > > > > later.
> > > > > 
> > > > > That said, I don't mind having a "apic-id" property because it is easier
> > > > > to simply expose it directly. But do you agree that: 1) we don't really
> > > > > need to expose it to be set from outside QEMU; 2) we shouldn't require
> > > > > it to be set from outside QEMU; 3) we should recommend users to not try
> > > > > to fiddle it with?
> > > > Due to nature of per thread CPU hotplug, management will have to specify
> > > > some kind of ID to specify which CPU is being plugged. Management really
> > > > doesn't/shouldn't care what this ID is.
> > > 
> > > As long as management really doesn't/shouldn't care what the ID is,
> > > exposing the APIC ID in the form of an opaque CPU identifier wouldn't be
> > > a problem to me. I just wanted to make clarify if we agree that messing
> > > with the APIC ID directly won't be recommended and that the "apic-id"
> > > property will be for QEMU internal use only.
> > On contrary, it's useful external feature, x86 guests see only APIC ID, since
> > it's the only ID they [should] know about. So guest aware mgmt could
> > definitely use apic_id propery to correlate CPU in guest with QEMU view of
> > them.
> 
> You're right, _reading_ the APIC ID is very useful. I am worried about
> _setting_ it from external code.

currently it's not possible since cpu-add doesn't allow to set any properties.

We will need setting it for device_add though.
By then, I guess some way to check that it's valid would be enough, otherwise
hot-plugged CPU will be out of scope of MADT and guest would ignore it or
through an error.

> 
> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 08/19] cpu: introduce get_arch_id() method and override it for target-i386

[Igor Mammedov]

On Mon, 15 Apr 2013 12:47:52 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Mon, Apr 15, 2013 at 05:24:36PM +0200, Andreas Färber wrote:
> > Am 11.04.2013 16:51, schrieb Igor Mammedov:
> > > get_arch_id() adds possibility for generic code to get guest visible
> > > CPI id without accessing CPUArchState. If target doesn't override it,
> > > it will return cpu_index.
> > > 
> > > Override it on target-i386 to return APIC ID.
> > > 
> > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > ---
> > >  * it will be used later by new cpu_exists() generic function and
> > >    acpi_piix.
> > >  * s/cpu_firmware_id/cpu_arch_id/
> > > ---
> > >  include/qom/cpu.h |  2 ++
> > >  qom/cpu.c         |  6 ++++++
> > >  target-i386/cpu.c | 10 ++++++++++
> > >  3 files changed, 18 insertions(+)
> > 
> > I'm still unclear about this ID topic, but please don't add QOM methods
> > with arch_ in the name. The arch is expressed through subclasses.
> 
> The naming seemed appropriate to me as it is an abstract method that is
> expected to have arch-specific implementations and arch-specific
> meaning.
> 
> On the other hand, "firmware_id" looked clearer to me (as it was very
> specific about the meaning/usefulness of the return value), I don't know
> what was the original reason to change from firmware_id to arch_id.
Objection here
http://lists.nongnu.org/archive/html/qemu-devel/2013-04/msg01332.html

In case of x86, it might be firmware defined id, but in general case it might
be not the case, since it could be hardwired into CPU. So get_arch_id()
seems more appropriate.

> 
> Other than "arch_id" and "firmware_id , I don't know how we could name
> that method. It's not "just an ID", but a special kind of ID, so I
> believe just "cpu_id" wouldn't work.
> 
> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 18/19] target-i386: expose all possible CPUs as /machine/icc-bridge/cpu[0..N] links

[Igor Mammedov]

On Mon, 15 Apr 2013 12:26:07 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Mon, Apr 15, 2013 at 05:16:26PM +0200, Igor Mammedov wrote:
> > On Mon, 15 Apr 2013 11:48:45 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Mon, Apr 15, 2013 at 04:15:08PM +0200, Igor Mammedov wrote:
> > > > On Fri, 12 Apr 2013 09:44:09 -0300
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Fri, Apr 12, 2013 at 12:01:03PM +0200, Igor Mammedov wrote:
> > > > > > On Thu, 11 Apr 2013 14:19:37 -0300
> > > > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > > > 
> > > > > > > On Thu, Apr 11, 2013 at 04:51:57PM +0200, Igor Mammedov wrote:
> > > > > > > > ... and leave links for not present CPUs empty.
> > > > > > > > 
> > > > > > > > It will allow users to query for possible APIC IDs and use them
> > > > > > > > with cpu-add QMP command.
> > > > > > > > 
> > > > > > > > Signed-off-by: Igor Mammedov <imammedo@redhat.com>
> > > > > > > 
> > > > > > > I don't see anything wrong with having icc-bridge links as well,
> > > > > > > but I would really like to have a target-independent namespace with
> > > > > > > links, that could be used to query for the available/valid CPU IDs
> > > > > > > for cpu-add commands instead of icc-bridge. The IDs on that
> > > > > > > namespace could be considered completely opaque.
> > > > > > 
> > > > > > Considering that -numa in present state is not compatible with cpu-add
> > > > > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > > > > patch could be dropped without any harm. libvirt could just use
> > > > > > numbers from this sequence like it's doing with current cpu_set
> > > > > > without any ID discovery. 
> > > > > 
> > > > > But it's not -numa that makes APIC ID probing necessary, it's
> > > > > non-power-of-2 core/thread counts on -smp (that make APIC IDs not match
> > > > > CPU indexes).
> > > > > 
> > > > > "Don't use CPU hotplug with -numa" is easy to be understood by users and
> > > > > by libvirt, but "don't use CPU hotplug with non-power-of-2 cores/threads
> > > > > counts" is harder to explain.
> > > > > 
> > > > > 
> > > > > > 
> > > > > > So, I've postponed target independent until we have -numa reworked,
> > > > > > then we could have /machine/node/socket/cpu containers with links.
> > > > > > The problem that needs to be solved, is the links storage ownership.
> > > > > > Who should allocate and own it? If machine was QOM object already,
> > > > > > I'd go with machine but it's not yet.
> > > > > 
> > > > > If we use CPU index as argument to cpu-add, we don't need to handle all
> > > > > those problems right now, we don't need to expose an APIC ID discovery
> > > > > interface, we make it work even with non-power-of-2 cores/threads
> > > > yes, you will get non-power-of-2 working without ID look-up.
> > > > 
> > > > > counts, and we make it work with -numa.
> > > > But you won't get this since, only next non-plugged ID will work, due to
> > > > how cpu_index is allocated. You can't just overwrite it with new value
> > > > without breaking current code.
> > > 
> > > OK, I think I get it: the problem is the cpu_index field specifically,
> > > because it is set automatically, and there may be lots of assumptions in
> > > the code about it. Maybe I should rephrase my suggestion: let's replace
> > > "CPU index" above with "a predictable ID allocation system where IDs
> > > will be in the range 0..(max_cpus-1)". We don't even need to touch the
> > > cpu_index field in the CPU objects if we think this is too risky.
> > > 
> > > I am just proposing that we use IDs so that we just need to add a
> > >    apic_id = apic_id_for_cpu_index(id)
> > > line at the beginning of the cpu-add implementation, and everything else
> > > would look exactly the same.
> > > 
> > > Nothing else would change in your implementation, except that now we
> > > won't need a ID lookup system for 1.5 because libvirt can assume that
> > > the next available CPU ID will be smp_cpus+1.
> > > 
> > > There would be no requirement to make the index-based IDs contiguous,
> > > just like there's no requirement to make APIC IDs contiguous. The only
> > > difference is that CPU indexes will predictable IDs, that will be always
> > > in the range 0..(max_cpus-1) and won't require an ID lookup mechanism.
> > > And it will work with non-power-of-2 threads/counts, and it will work
> > > with -numa.
> > > 
> > > > 
> > > > > 
> > > > > So, my big question is: why are we trying so hard to avoid using CPU
> > > > > indexes as argument to cpu-add, if it's so much easier, and it is an
> > > > > obvious solution that makes the interface target-independent without any
> > > > > extra effort?
> > > > Using cpu_index instead of APIC ID definitely is not effort free and
> > > > requires quite a bit of rewrite how its used currently, APIC ID is much
> > > > much easier and less risky choice in this regard.
> > > > 
> > > > As for target-independence, any kind of ID is target-independent if
> > > > treated as opaque.
> > > 
> > > True, as long as we have a target-independent ID lookup system, which we
> > > don't have today. That's the part where we would need less effort.
> > > 
> > > 
> > > > Given that with unplug should come not-contiguous ID usage, the
> > > > interface to track which CPUs are plugged would be needed anyway. So it
> > > > could be introduced with this series and provide ID look-up meanwhile.
> > > > That would give libvirt time actually to start using it, and just remove
> > > > not-contiguous ID restriction when unplug is ready with all necessary
> > > > infrastructure already around.
> > > 
> > > Let's rephrase my suggestion: I don't think we should use the internal
> > > "cpu->cpu_index" field as ID, necessarily. I only suggest that we make
> > > the IDs predictably in the range 0..(max_cpus-1) so we:
> > > 1) don't need an ID lookup mechanism;
> > > 2) keep it compatible with the existing -numa options.
> > > 
> > > If we really want to use APIC ID one day, we may implement an lookup
> > > mechanism that will provide IDs to libvirt, and declare the "IDs between
> > > 0..(max_cpus-1)" assumption as deprecated, so we can start using APIC
> > > IDs as the (opaque) CPU IDs in the future. But before we do that, we
> > > would implement a better "-numa" interface first.
> > > 
> > > You even suggested that we did something similar, above:
> > > 
> > > > > > Considering that -numa in present state is not compatible with cpu-add
> > > > > > and that all CPU ID in this case are are sequence [0..maxcpus-1], this
> > > > > > patch could be dropped without any harm. libvirt could just use
> > > > > > numbers from this sequence like it's doing with current cpu_set
> > > > > > without any ID discovery. 
> > > 
> > > What I suggest is that we let libvirt make exactly the same assumptions
> > > you suggest, but if we add a single line to the code:
> > >     apic_id = apic_id_for_cpu_index(id)
> > > we will make this intermediate solution work with NUMA _and_ work with
> > > non-power-of-2 cores/threads counts.
> > > 
> > > The other cases (non-numa with power-of-2 cores/threads counts) would
> > > work exactly the same, because in those cases
> > > apic_id_for_cpu_index(id) == id.
> > > 
> > 
> > Rephrasing suggestion:
> > drop 18/19 and use apic_id = apic_id_for_cpu_index(id) in 19/19
> > Am I correct?
> 
> Exactly. I mean: I expect that to be enough and not require other
> changes, but I didn't look at all patches line-by-line yet.
Ok, I'm dropping it and submitting amended series,

Thanks for review.
> 
> -- 
> Eduardo
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Eduardo Habkost]

On Mon, Apr 15, 2013 at 10:27:57PM +0200, Igor Mammedov wrote:
> On Mon, 15 Apr 2013 11:49:40 -0300
> Eduardo Habkost <ehabkost@redhat.com> wrote:
> 
> > On Mon, Apr 15, 2013 at 04:34:41PM +0200, Igor Mammedov wrote:
> > > On Mon, 15 Apr 2013 10:45:24 -0300
> > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > 
> > > > On Mon, Apr 15, 2013 at 03:37:00PM +0200, Igor Mammedov wrote:
> > > > [...]
> > > > > > > > ID to be directly specified in the device_add/-device options.
> > > > > > > > That's how real CPUs work: as the CPU manufacturer doesn't know
> > > > > > > > what will be the package ID of the CPU, the APIC IDs are not
> > > > > > > > hardcoded in the CPU; they are calculated based on the CPU topology
> > > > > > > > and some socket identifier signal coming from the board.
> > > > > > > that is why apic_id has been made a property, to be set from outside.
> > > > > > 
> > > > > > True. I believe the conflict here is: we want other objects to set the
> > > > > > APIC ID (be it the board, or socket/core objects), but at the same time
> > > > > > it would be interesting to not expose the APIC ID outside QEMU. Being
> > > > > > too flexible regarding the APIC ID is more likely to cause problems
> > > > > > later.
> > > > > > 
> > > > > > That said, I don't mind having a "apic-id" property because it is easier
> > > > > > to simply expose it directly. But do you agree that: 1) we don't really
> > > > > > need to expose it to be set from outside QEMU; 2) we shouldn't require
> > > > > > it to be set from outside QEMU; 3) we should recommend users to not try
> > > > > > to fiddle it with?
> > > > > Due to nature of per thread CPU hotplug, management will have to specify
> > > > > some kind of ID to specify which CPU is being plugged. Management really
> > > > > doesn't/shouldn't care what this ID is.
> > > > 
> > > > As long as management really doesn't/shouldn't care what the ID is,
> > > > exposing the APIC ID in the form of an opaque CPU identifier wouldn't be
> > > > a problem to me. I just wanted to make clarify if we agree that messing
> > > > with the APIC ID directly won't be recommended and that the "apic-id"
> > > > property will be for QEMU internal use only.
> > > On contrary, it's useful external feature, x86 guests see only APIC ID, since
> > > it's the only ID they [should] know about. So guest aware mgmt could
> > > definitely use apic_id propery to correlate CPU in guest with QEMU view of
> > > them.
> > 
> > You're right, _reading_ the APIC ID is very useful. I am worried about
> > _setting_ it from external code.
> 
> currently it's not possible since cpu-add doesn't allow to set any properties.
> 
> We will need setting it for device_add though.

Not necessarily. That's why I am insisting on an interface based on
links/topology, not based on a raw "apic-id" property: instead of
setting apic-id directly, we could just require that the CPU be attached
to the right socket/core objects, and the APIC ID would be magically
calculated correctly.

Or we could just let the right socket/core/thread IDs to be set as
properties, and apic-id could be calculated based on that. There are
many ways to expose an abstraction that's simpler to use and less likely
to cause problems.


> By then, I guess some way to check that it's valid would be enough, otherwise
> hot-plugged CPU will be out of scope of MADT and guest would ignore it or
> through an error.
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 11/19] target-i386: introduce apic-id property

[Igor Mammedov]

On Mon, 15 Apr 2013 17:49:20 -0300
Eduardo Habkost <ehabkost@redhat.com> wrote:

> On Mon, Apr 15, 2013 at 10:27:57PM +0200, Igor Mammedov wrote:
> > On Mon, 15 Apr 2013 11:49:40 -0300
> > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > 
> > > On Mon, Apr 15, 2013 at 04:34:41PM +0200, Igor Mammedov wrote:
> > > > On Mon, 15 Apr 2013 10:45:24 -0300
> > > > Eduardo Habkost <ehabkost@redhat.com> wrote:
> > > > 
> > > > > On Mon, Apr 15, 2013 at 03:37:00PM +0200, Igor Mammedov wrote:
> > > > > [...]
> > > > > > > > > ID to be directly specified in the device_add/-device options.
> > > > > > > > > That's how real CPUs work: as the CPU manufacturer doesn't know
> > > > > > > > > what will be the package ID of the CPU, the APIC IDs are not
> > > > > > > > > hardcoded in the CPU; they are calculated based on the CPU topology
> > > > > > > > > and some socket identifier signal coming from the board.
> > > > > > > > that is why apic_id has been made a property, to be set from outside.
> > > > > > > 
> > > > > > > True. I believe the conflict here is: we want other objects to set the
> > > > > > > APIC ID (be it the board, or socket/core objects), but at the same time
> > > > > > > it would be interesting to not expose the APIC ID outside QEMU. Being
> > > > > > > too flexible regarding the APIC ID is more likely to cause problems
> > > > > > > later.
> > > > > > > 
> > > > > > > That said, I don't mind having a "apic-id" property because it is easier
> > > > > > > to simply expose it directly. But do you agree that: 1) we don't really
> > > > > > > need to expose it to be set from outside QEMU; 2) we shouldn't require
> > > > > > > it to be set from outside QEMU; 3) we should recommend users to not try
> > > > > > > to fiddle it with?
> > > > > > Due to nature of per thread CPU hotplug, management will have to specify
> > > > > > some kind of ID to specify which CPU is being plugged. Management really
> > > > > > doesn't/shouldn't care what this ID is.
> > > > > 
> > > > > As long as management really doesn't/shouldn't care what the ID is,
> > > > > exposing the APIC ID in the form of an opaque CPU identifier wouldn't be
> > > > > a problem to me. I just wanted to make clarify if we agree that messing
> > > > > with the APIC ID directly won't be recommended and that the "apic-id"
> > > > > property will be for QEMU internal use only.
> > > > On contrary, it's useful external feature, x86 guests see only APIC ID, since
> > > > it's the only ID they [should] know about. So guest aware mgmt could
> > > > definitely use apic_id propery to correlate CPU in guest with QEMU view of
> > > > them.
> > > 
> > > You're right, _reading_ the APIC ID is very useful. I am worried about
> > > _setting_ it from external code.
> > 
> > currently it's not possible since cpu-add doesn't allow to set any properties.
> > 
> > We will need setting it for device_add though.
> 
> Not necessarily. That's why I am insisting on an interface based on
> links/topology, not based on a raw "apic-id" property: instead of
> setting apic-id directly, we could just require that the CPU be attached
> to the right socket/core objects, and the APIC ID would be magically
> calculated correctly.
> 
> Or we could just let the right socket/core/thread IDs to be set as
> properties, and apic-id could be calculated based on that. There are
> many ways to expose an abstraction that's simpler to use and less likely
> to cause problems.

if we have tree like this:

/nodes/0../sockets/0../cores/0../threads/0..
user will have to parse all branches IDs to collect node/socket/core/thread IDs
before adding CPU.
If use APIC ID on thread level then user is required to get only it and no
magic would be required to calculate it, because board has already calculated
it in advance.

BTW:
it'd be nicer to refuse wrong id earlier at property setting time then
postponing it till realize when it would be possible to check that
node/socket/core/thread info provided earlier was correct.

> 
> 
> > By then, I guess some way to check that it's valid would be enough, otherwise
> > hot-plugged CPU will be out of scope of MADT and guest would ignore it or
> > through an error.
> > 
> 
> -- 
> Eduardo


-- 
Regards,
  Igor