[PATCH v2 0/2] Some enhancement for kvfree_rcu_mightsleep()

[PATCH v2 0/2] Some enhancement for kvfree_rcu_mightsleep()

[Uladzislau Rezki (Sony)]

This is v2. This small change tends to improve even more the slow path
of kvfree_rcu_mightsleep() API. It is hit when a low memory condition
occurs which leads us into two options:
  a) check if a GP is already passed for freed object;
  b) if not, initialize and wait for completion of a new GP.

The differecne between v1 -> v2:
  - Drop "Support dynamic rcu_head for single argument objects" patch;
  - Use cond_synchronize_rcu_full() in a slow path.

Uladzislau Rezki (Sony) (2):
  rcu/kvfree: Use polled API in a slow path
  rcu/kvfree: Switch to expedited version in slow path

 kernel/rcu/tree.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

-- 
2.39.2

[PATCH v2 1/2] rcu/kvfree: Use polled API in a slow path

[Uladzislau Rezki (Sony)]

For a single argument use a polled API to see if a GP is
already passed. This allows to bypass an extra GP request
in a slow path.

Allocating a page or dynamic rcu_head might take some and
still fail, in that scenario a GP which is requested on entry
of kvfree_call_rcu() can be elapsed. Benefit from this.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
 kernel/rcu/tree.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index e641cc681901..182772494cb0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3789,6 +3789,7 @@ add_ptr_to_bulk_krc_lock(struct kfree_rcu_cpu **krcp,
 void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 {
 	unsigned long flags;
+	struct rcu_gp_oldstate old_snap;
 	struct kfree_rcu_cpu *krcp;
 	bool success;
 
@@ -3799,8 +3800,10 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	 * only. For other places please embed an rcu_head to
 	 * your data.
 	 */
-	if (!head)
+	if (!head) {
 		might_sleep();
+		start_poll_synchronize_rcu_full(&old_snap);
+	}
 
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
@@ -3853,7 +3856,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	 */
 	if (!success) {
 		debug_rcu_head_unqueue((struct rcu_head *) ptr);
-		synchronize_rcu();
+		cond_synchronize_rcu_full(&old_snap);
 		kvfree(ptr);
 	}
 }
-- 
2.39.2

[PATCH v2 2/2] rcu/kvfree: Switch to expedited version in slow path

[Uladzislau Rezki (Sony)]

For a single argument and its slow path, switch to expedited
version of synchronize_rcu(). This version is considered to
be more faster, thus under a high memory pressure a slow path
becoms more efficient.

Note, latency-sensitive workloads should use rcutree.gp_normal=1,
which will make that synchronize_rcu_expedited() act like a regular
synchronize_rcu(), so no harm done to them.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 182772494cb0..87a64fcffa7f 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3856,7 +3856,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	 */
 	if (!success) {
 		debug_rcu_head_unqueue((struct rcu_head *) ptr);
-		cond_synchronize_rcu_full(&old_snap);
+		cond_synchronize_rcu_expedited_full(&old_snap);
 		kvfree(ptr);
 	}
 }
-- 
2.39.2