[Qemu-devel] [PATCH 0/3] run RCU callbacks within the iothread mutex, fix PCI hotplug

[Qemu-devel] [PATCH 0/3] run RCU callbacks within the iothread mutex, fix PCI hotplug

[Paolo Bonzini]

Patch 1 avoids that RCU callbacks are delayed forever if there's
scarcity of them, which is bad because qemu_opts_del will be called in
a RCU callbacks.

Patches 2 avoids complications due to instance_finalize callbacks that
are not thread-safe.  It's a big hammer and it is not handsome, but I
gave up on making VFIO list manipulations thread-safe.

Patch 3 avoids a use-after-free when freeing address spaces.

These patches survived several hundred hotplug cycles, with MALLOC_PERTURB_
and G_SLICE=always-malloc on.

Paolo

Paolo Bonzini (3):
  rcu: do not let RCU callbacks pile up indefinitely
  rcu: run RCU callbacks under the BQL
  memory: keep the owner of the AddressSpace alive until
    do_address_space_destroy

 memory.c       |  5 +++++
 tests/Makefile |  2 +-
 util/rcu.c     | 19 +++++++++++++------
 3 files changed, 19 insertions(+), 7 deletions(-)

-- 
1.8.3.1

[Qemu-devel] [PATCH 1/3] rcu: do not let RCU callbacks pile up indefinitely

[Paolo Bonzini]

Always process them within half a second.  Even though waiting a little
is useful, it is not okay to delay e.g. qemu_opts_del forever.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 util/rcu.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/util/rcu.c b/util/rcu.c
index c9c3e6e..486d7b6 100644
--- a/util/rcu.c
+++ b/util/rcu.c
@@ -223,14 +223,16 @@ static void *call_rcu_thread(void *opaque)
          * Fetch rcu_call_count now, we only must process elements that were
          * added before synchronize_rcu() starts.
          */
-        while (n < RCU_CALL_MIN_SIZE && ++tries <= 5) {
-            g_usleep(100000);
-            qemu_event_reset(&rcu_call_ready_event);
-            n = atomic_read(&rcu_call_count);
-            if (n < RCU_CALL_MIN_SIZE) {
-                qemu_event_wait(&rcu_call_ready_event);
+        while (n == 0 || (n < RCU_CALL_MIN_SIZE && ++tries <= 5)) {
+            g_usleep(10000);
+            if (n == 0) {
+                qemu_event_reset(&rcu_call_ready_event);
                 n = atomic_read(&rcu_call_count);
+                if (n == 0) {
+                    qemu_event_wait(&rcu_call_ready_event);
+                }
             }
+            n = atomic_read(&rcu_call_count);
         }
 
         atomic_sub(&rcu_call_count, n);
-- 
1.8.3.1

[Qemu-devel] [PATCH 2/3] rcu: run RCU callbacks under the BQL

[Paolo Bonzini]

This needs to go away sooner or later, but one complication is the
complex VFIO data structures that are modified in instance_finalize.
Take a shortcut for now.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tests/Makefile | 2 +-
 util/rcu.c     | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/Makefile b/tests/Makefile
index d5df168..e11bfe7 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -257,7 +257,7 @@ tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o
 tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o libqemuutil.a
 tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.o
 tests/test-int128$(EXESUF): tests/test-int128.o
-tests/rcutorture$(EXESUF): tests/rcutorture.o libqemuutil.a
+tests/rcutorture$(EXESUF): tests/rcutorture.o libqemuutil.a libqemustub.a
 
 tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
 	hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\
diff --git a/util/rcu.c b/util/rcu.c
index 486d7b6..bd73b8e 100644
--- a/util/rcu.c
+++ b/util/rcu.c
@@ -35,6 +35,7 @@
 #include "qemu/rcu.h"
 #include "qemu/atomic.h"
 #include "qemu/thread.h"
+#include "qemu/main-loop.h"
 
 /*
  * Global grace period counter.  Bit 0 is always one in rcu_gp_ctr.
@@ -237,20 +238,24 @@ static void *call_rcu_thread(void *opaque)
 
         atomic_sub(&rcu_call_count, n);
         synchronize_rcu();
+        qemu_mutex_lock_iothread();
         while (n > 0) {
             node = try_dequeue();
             while (!node) {
+                qemu_mutex_unlock_iothread();
                 qemu_event_reset(&rcu_call_ready_event);
                 node = try_dequeue();
                 if (!node) {
                     qemu_event_wait(&rcu_call_ready_event);
                     node = try_dequeue();
                 }
+                qemu_mutex_lock_iothread();
             }
 
             n--;
             node->func(node);
         }
+        qemu_mutex_unlock_iothread();
     }
     abort();
 }
-- 
1.8.3.1

[Qemu-devel] [PATCH 3/3] memory: keep the owner of the AddressSpace alive until do_address_space_destroy

[Paolo Bonzini]

This fixes a use-after-free if do_address_space_destroy is executed
too late.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 memory.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/memory.c b/memory.c
index 130152c..20f6d9e 100644
--- a/memory.c
+++ b/memory.c
@@ -1943,6 +1943,7 @@ void memory_listener_unregister(MemoryListener *listener)
 
 void address_space_init(AddressSpace *as, MemoryRegion *root, const char *name)
 {
+    memory_region_ref(root);
     memory_region_transaction_begin();
     as->root = root;
     as->current_map = g_new(FlatView, 1);
@@ -1969,10 +1970,13 @@ static void do_address_space_destroy(AddressSpace *as)
     flatview_unref(as->current_map);
     g_free(as->name);
     g_free(as->ioeventfds);
+    memory_region_unref(as->root);
 }
 
 void address_space_destroy(AddressSpace *as)
 {
+    MemoryRegion *root = as->root;
+
     /* Flush out anything from MemoryListeners listening in on this */
     memory_region_transaction_begin();
     as->root = NULL;
@@ -1984,6 +1988,7 @@ void address_space_destroy(AddressSpace *as)
      * entries that the guest should never use.  Wait for the old
      * values to expire before freeing the data.
      */
+    as->root = root;
     call_rcu(as, do_address_space_destroy, rcu);
 }
 
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH 1/3] rcu: do not let RCU callbacks pile up indefinitely

[Paolo Bonzini]

On 11/02/2015 18:14, Paolo Bonzini wrote:
> Always process them within half a second.

Actually, half a second is too much so I changed it to .05 seconds to
err on the safe side and still give time to the sending process to
register two-three callbacks in rapid succession.

Paolo

> Even though waiting a little
> is useful, it is not okay to delay e.g. qemu_opts_del forever.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [PATCH 2/3] rcu: run RCU callbacks under the BQL

[Michael Roth]

Quoting Paolo Bonzini (2015-02-11 11:14:31)
> This needs to go away sooner or later, but one complication is the
> complex VFIO data structures that are modified in instance_finalize.
> Take a shortcut for now.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  tests/Makefile | 2 +-
>  util/rcu.c     | 5 +++++
>  2 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/Makefile b/tests/Makefile
> index d5df168..e11bfe7 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -257,7 +257,7 @@ tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o
>  tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o libqemuutil.a
>  tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.o
>  tests/test-int128$(EXESUF): tests/test-int128.o
> -tests/rcutorture$(EXESUF): tests/rcutorture.o libqemuutil.a
> +tests/rcutorture$(EXESUF): tests/rcutorture.o libqemuutil.a libqemustub.a
> 
>  tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
>         hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\
> diff --git a/util/rcu.c b/util/rcu.c
> index 486d7b6..bd73b8e 100644
> --- a/util/rcu.c
> +++ b/util/rcu.c
> @@ -35,6 +35,7 @@
>  #include "qemu/rcu.h"
>  #include "qemu/atomic.h"
>  #include "qemu/thread.h"
> +#include "qemu/main-loop.h"
> 
>  /*
>   * Global grace period counter.  Bit 0 is always one in rcu_gp_ctr.
> @@ -237,20 +238,24 @@ static void *call_rcu_thread(void *opaque)
> 
>          atomic_sub(&rcu_call_count, n);
>          synchronize_rcu();
> +        qemu_mutex_lock_iothread();
>          while (n > 0) {
>              node = try_dequeue();
>              while (!node) {
> +                qemu_mutex_unlock_iothread();
>                  qemu_event_reset(&rcu_call_ready_event);
>                  node = try_dequeue();
>                  if (!node) {
>                      qemu_event_wait(&rcu_call_ready_event);
>                      node = try_dequeue();
>                  }
> +                qemu_mutex_lock_iothread();
>              }
> 
>              n--;
>              node->func(node);

Not sure I understand the reasoning for lock/unlock beyond just guarding
the ->func() call, what else are we serializing?

>          }
> +        qemu_mutex_unlock_iothread();
>      }
>      abort();
>  }
> -- 
> 1.8.3.1

Re: [Qemu-devel] [PATCH 3/3] memory: keep the owner of the AddressSpace alive until do_address_space_destroy

[Michael Roth]

Quoting Paolo Bonzini (2015-02-11 11:14:32)
> This fixes a use-after-free if do_address_space_destroy is executed
> too late.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Looks like this did the trick for me. Haven't been able to reproduce for
a couple hours. Thanks!

Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Tested-by: Michael Roth <mdroth@linux.vnet.ibm.com>

> ---
>  memory.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/memory.c b/memory.c
> index 130152c..20f6d9e 100644
> --- a/memory.c
> +++ b/memory.c
> @@ -1943,6 +1943,7 @@ void memory_listener_unregister(MemoryListener *listener)
> 
>  void address_space_init(AddressSpace *as, MemoryRegion *root, const char *name)
>  {
> +    memory_region_ref(root);
>      memory_region_transaction_begin();
>      as->root = root;
>      as->current_map = g_new(FlatView, 1);
> @@ -1969,10 +1970,13 @@ static void do_address_space_destroy(AddressSpace *as)
>      flatview_unref(as->current_map);
>      g_free(as->name);
>      g_free(as->ioeventfds);
> +    memory_region_unref(as->root);
>  }
> 
>  void address_space_destroy(AddressSpace *as)
>  {
> +    MemoryRegion *root = as->root;
> +
>      /* Flush out anything from MemoryListeners listening in on this */
>      memory_region_transaction_begin();
>      as->root = NULL;
> @@ -1984,6 +1988,7 @@ void address_space_destroy(AddressSpace *as)
>       * entries that the guest should never use.  Wait for the old
>       * values to expire before freeing the data.
>       */
> +    as->root = root;
>      call_rcu(as, do_address_space_destroy, rcu);
>  }
> 
> -- 
> 1.8.3.1

Re: [Qemu-devel] [PATCH 2/3] rcu: run RCU callbacks under the BQL

[Paolo Bonzini]

On 11/02/2015 21:26, Michael Roth wrote:
> Quoting Paolo Bonzini (2015-02-11 11:14:31)
>> This needs to go away sooner or later, but one complication is the
>> complex VFIO data structures that are modified in instance_finalize.
>> Take a shortcut for now.
>>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>>  tests/Makefile | 2 +-
>>  util/rcu.c     | 5 +++++
>>  2 files changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/Makefile b/tests/Makefile
>> index d5df168..e11bfe7 100644
>> --- a/tests/Makefile
>> +++ b/tests/Makefile
>> @@ -257,7 +257,7 @@ tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o
>>  tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o libqemuutil.a
>>  tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.o
>>  tests/test-int128$(EXESUF): tests/test-int128.o
>> -tests/rcutorture$(EXESUF): tests/rcutorture.o libqemuutil.a
>> +tests/rcutorture$(EXESUF): tests/rcutorture.o libqemuutil.a libqemustub.a
>>
>>  tests/test-qdev-global-props$(EXESUF): tests/test-qdev-global-props.o \
>>         hw/core/qdev.o hw/core/qdev-properties.o hw/core/hotplug.o\
>> diff --git a/util/rcu.c b/util/rcu.c
>> index 486d7b6..bd73b8e 100644
>> --- a/util/rcu.c
>> +++ b/util/rcu.c
>> @@ -35,6 +35,7 @@
>>  #include "qemu/rcu.h"
>>  #include "qemu/atomic.h"
>>  #include "qemu/thread.h"
>> +#include "qemu/main-loop.h"
>>
>>  /*
>>   * Global grace period counter.  Bit 0 is always one in rcu_gp_ctr.
>> @@ -237,20 +238,24 @@ static void *call_rcu_thread(void *opaque)
>>
>>          atomic_sub(&rcu_call_count, n);
>>          synchronize_rcu();
>> +        qemu_mutex_lock_iothread();
>>          while (n > 0) {
>>              node = try_dequeue();
>>              while (!node) {
>> +                qemu_mutex_unlock_iothread();
>>                  qemu_event_reset(&rcu_call_ready_event);
>>                  node = try_dequeue();
>>                  if (!node) {
>>                      qemu_event_wait(&rcu_call_ready_event);
>>                      node = try_dequeue();
>>                  }
>> +                qemu_mutex_lock_iothread();
>>              }
>>
>>              n--;
>>              node->func(node);
> 
> Not sure I understand the reasoning for lock/unlock beyond just guarding
> the ->func() call, what else are we serializing?

I'm trying to avoid multiple back-to-back unlocks and locks.  Being
coarse-grained qemu_mutex_lock_iothread/qemu_mutex_unlock_iothread is
likely to hit the slow path---and also slow down _other_ threads.

Paolo

>>          }
>> +        qemu_mutex_unlock_iothread();
>>      }
>>      abort();
>>  }
>> -- 
>> 1.8.3.1
> 
> 
> 

Re: [Qemu-devel] [PATCH 1/3] rcu: do not let RCU callbacks pile up indefinitely

[Fam Zheng]

On Wed, 02/11 18:14, Paolo Bonzini wrote:
> Always process them within half a second.  Even though waiting a little
> is useful, it is not okay to delay e.g. qemu_opts_del forever.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  util/rcu.c | 14 ++++++++------
>  1 file changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/util/rcu.c b/util/rcu.c
> index c9c3e6e..486d7b6 100644
> --- a/util/rcu.c
> +++ b/util/rcu.c
> @@ -223,14 +223,16 @@ static void *call_rcu_thread(void *opaque)
>           * Fetch rcu_call_count now, we only must process elements that were
>           * added before synchronize_rcu() starts.
>           */
> -        while (n < RCU_CALL_MIN_SIZE && ++tries <= 5) {
> -            g_usleep(100000);
> -            qemu_event_reset(&rcu_call_ready_event);
> -            n = atomic_read(&rcu_call_count);
> -            if (n < RCU_CALL_MIN_SIZE) {
> -                qemu_event_wait(&rcu_call_ready_event);
> +        while (n == 0 || (n < RCU_CALL_MIN_SIZE && ++tries <= 5)) {
> +            g_usleep(10000);
> +            if (n == 0) {
> +                qemu_event_reset(&rcu_call_ready_event);
>                  n = atomic_read(&rcu_call_count);
> +                if (n == 0) {
> +                    qemu_event_wait(&rcu_call_ready_event);
> +                }
>              }
> +            n = atomic_read(&rcu_call_count);
>          }
>  
>          atomic_sub(&rcu_call_count, n);

Reviewed-by: Fam Zheng <famz@redhat.com>