[PATCH 00/18] kfree_rcu() improvements for -rcu dev

[PATCH 00/18] kfree_rcu() improvements for -rcu dev

[Joel Fernandes (Google)]

Hello,
These patches improve kfree_rcu() to support:
- kfree_rcu() headless usage.
- both vmalloc() and slab free'ing support using array of pointers.
- simpler debugobjects handling.

It applies on rcu/dev branch as of March 29th.

Testing with rcuperf shows following changes. The memory footprint reduces and
batches go slightly up. This is assumed an acceptable change.

with all patches:
Total time taken by all kfree'ers: 27312964461 ns, loops: 20000, batches: 3120, memory footprint: 211MB
Total time taken by all kfree'ers: 26773272309 ns, loops: 20000, batches: 3084, memory footprint: 208M

without:
Total time taken by all kfree'ers: 25711621811 ns, loops: 20000, batches: 2814, memory footprint: 230MB          
 Total time taken by all kfree'ers: 25775800546 ns, loops: 20000, batches: 2755, memory footprint: 230MB

These have been pushed to the git tree at:
git://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git (branch rcu/kfree)

cgit view:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=rcu/kfree

thanks,

 - Joel

Joel Fernandes (Google) (5):
rcu/tree: Simplify debug_objects handling
rcu/tree: Clarify emergency path comment better
rcu/tree: Remove extra next variable in kfree worker function
rcu/tree: Simplify is_vmalloc_addr expression
rcu/tree: Make kvfree_rcu() tolerate any alignment

Uladzislau Rezki (Sony) (13):
mm/list_lru.c: Rename kvfree_rcu() to local variant
rcu: Introduce kvfree_rcu() interface
rcu: Rename rcu_invoke_kfree_callback/rcu_kfree_callback
rcu: Rename __is_kfree_rcu_offset() macro
rcu: Rename kfree_call_rcu() to the kvfree_call_rcu().
mm/list_lru.c: Remove kvfree_rcu_local() function
rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro
rcu/tree: Maintain separate array for vmalloc ptrs
rcu/tree: Introduce expedited_drain flag
rcu/tree: Support reclaim for head-less object
rcu/tiny: Move kvfree_call_rcu() out of header
rcu/tiny: Support reclaim for head-less object
rcu: Support headless variant in the kvfree_rcu()

include/linux/rcupdate.h   |  53 ++++++-
include/linux/rcutiny.h    |   6 +-
include/linux/rcutree.h    |   2 +-
include/trace/events/rcu.h |   8 +-
kernel/rcu/tiny.c          | 168 +++++++++++++++++++-
kernel/rcu/tree.c          | 315 ++++++++++++++++++++++++++-----------
mm/list_lru.c              |  11 +-
7 files changed, 443 insertions(+), 120 deletions(-)

--
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 01/18] mm/list_lru.c: Rename kvfree_rcu() to local variant

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Rename kvfree_rcu() function to the kvfree_rcu_local()
one. The aim is to introduce the public API that would
conflict with this one.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 mm/list_lru.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mm/list_lru.c b/mm/list_lru.c
index 0f1f6b06b7f36..386424688f805 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -383,14 +383,14 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru)
 	struct list_lru_memcg *memcg_lrus;
 	/*
 	 * This is called when shrinker has already been unregistered,
-	 * and nobody can use it. So, there is no need to use kvfree_rcu().
+	 * and nobody can use it. So, there is no need to use kvfree_rcu_local().
 	 */
 	memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true);
 	__memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids);
 	kvfree(memcg_lrus);
 }
 
-static void kvfree_rcu(struct rcu_head *head)
+static void kvfree_rcu_local(struct rcu_head *head)
 {
 	struct list_lru_memcg *mlru;
 
@@ -429,7 +429,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru,
 	rcu_assign_pointer(nlru->memcg_lrus, new);
 	spin_unlock_irq(&nlru->lock);
 
-	call_rcu(&old->rcu, kvfree_rcu);
+	call_rcu(&old->rcu, kvfree_rcu_local);
 	return 0;
 }
 
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 02/18] rcu: Introduce kvfree_rcu() interface

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

kvfree_rcu() can deal with an allocated memory that is obtained
via kvmalloc(). It can return two types of allocated memory or
"pointers", one can belong to regular SLAB allocator and another
one can be vmalloc one. It depends on requested size and memory
pressure.

Based on that, two streams are split, thus if a pointer belongs
to vmalloc allocator it is queued to the list, otherwise SLAB
one is queued into "bulk array" for further processing.

The main reason of such splitting is:
    a) to distinguish kmalloc()/vmalloc() ptrs;
    b) there is no vmalloc_bulk() interface.

As of now we have list_lru.c user that needs such interface,
also there will be new comers. Apart of that it is preparation
to have a head-less variant later.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 include/linux/rcupdate.h |  9 +++++++++
 kernel/rcu/tiny.c        |  3 ++-
 kernel/rcu/tree.c        | 17 ++++++++++++-----
 3 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 3598bbb5ff407..8b7128d0860e2 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -870,6 +870,15 @@ do {									\
 		__kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
 } while (0)
 
+/**
+ * kvfree_rcu() - kvfree an object after a grace period.
+ * @ptr:	pointer to kvfree
+ * @rhf:	the name of the struct rcu_head within the type of @ptr.
+ *
+ * Same as kfree_rcu(), just simple alias.
+ */
+#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf)
+
 /*
  * Place this after a lock-acquisition primitive to guarantee that
  * an UNLOCK+LOCK pair acts as a full barrier.  This guarantee applies
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index dd572ce7c7479..4b99f7b88beec 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -23,6 +23,7 @@
 #include <linux/cpu.h>
 #include <linux/prefetch.h>
 #include <linux/slab.h>
+#include <linux/mm.h>
 
 #include "rcu.h"
 
@@ -86,7 +87,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head)
 	rcu_lock_acquire(&rcu_callback_map);
 	if (__is_kfree_rcu_offset(offset)) {
 		trace_rcu_invoke_kfree_callback("", head, offset);
-		kfree((void *)head - offset);
+		kvfree((void *)head - offset);
 		rcu_lock_release(&rcu_callback_map);
 		return true;
 	}
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 4eb424eb44acb..2d10c50621c38 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2925,9 +2925,9 @@ static void kfree_rcu_work(struct work_struct *work)
 	}
 
 	/*
-	 * Emergency case only. It can happen under low memory
-	 * condition when an allocation gets failed, so the "bulk"
-	 * path can not be temporary maintained.
+	 * vmalloc() pointers end up here also emergency case. It can
+	 * happen under low memory condition when an allocation gets
+	 * failed, so the "bulk" path can not be temporary maintained.
 	 */
 	for (; head; head = next) {
 		unsigned long offset = (unsigned long)head->func;
@@ -2938,7 +2938,7 @@ static void kfree_rcu_work(struct work_struct *work)
 		trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset);
 
 		if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset)))
-			kfree((void *)head - offset);
+			kvfree((void *)head - offset);
 
 		rcu_lock_release(&rcu_callback_map);
 		cond_resched_tasks_rcu_qs();
@@ -3112,10 +3112,17 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	}
 
 	/*
+	 * We do not queue vmalloc pointers into array,
+	 * instead they are just queued to the list. We
+	 * do it because of:
+	 *    a) to distinguish kmalloc()/vmalloc() ptrs;
+	 *    b) there is no vmalloc_bulk() interface.
+	 *
 	 * Under high memory pressure GFP_NOWAIT can fail,
 	 * in that case the emergency path is maintained.
 	 */
-	if (unlikely(!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func))) {
+	if (is_vmalloc_addr((void *) head - (unsigned long) func) ||
+			!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
 		head->func = func;
 		head->next = krcp->head;
 		krcp->head = head;
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 03/18] rcu: Rename rcu_invoke_kfree_callback/rcu_kfree_callback

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Rename rcu_invoke_kfree_callback to rcu_invoke_kvfree_callback.
Do the same with second trace event, that is rcu_kfree_callback,
it becomes rcu_kvfree_callback. The reason is to be aligned with
kvfree notation.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 include/trace/events/rcu.h | 8 ++++----
 kernel/rcu/tiny.c          | 2 +-
 kernel/rcu/tree.c          | 4 ++--
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h
index f9a7811148e2a..0ee93d0b1daa8 100644
--- a/include/trace/events/rcu.h
+++ b/include/trace/events/rcu.h
@@ -506,13 +506,13 @@ TRACE_EVENT_RCU(rcu_callback,
 
 /*
  * Tracepoint for the registration of a single RCU callback of the special
- * kfree() form.  The first argument is the RCU type, the second argument
+ * kvfree() form.  The first argument is the RCU type, the second argument
  * is a pointer to the RCU callback, the third argument is the offset
  * of the callback within the enclosing RCU-protected data structure,
  * the fourth argument is the number of lazy callbacks queued, and the
  * fifth argument is the total number of callbacks queued.
  */
-TRACE_EVENT_RCU(rcu_kfree_callback,
+TRACE_EVENT_RCU(rcu_kvfree_callback,
 
 	TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset,
 		 long qlen),
@@ -596,12 +596,12 @@ TRACE_EVENT_RCU(rcu_invoke_callback,
 
 /*
  * Tracepoint for the invocation of a single RCU callback of the special
- * kfree() form.  The first argument is the RCU flavor, the second
+ * kvfree() form.  The first argument is the RCU flavor, the second
  * argument is a pointer to the RCU callback, and the third argument
  * is the offset of the callback within the enclosing RCU-protected
  * data structure.
  */
-TRACE_EVENT_RCU(rcu_invoke_kfree_callback,
+TRACE_EVENT_RCU(rcu_invoke_kvfree_callback,
 
 	TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset),
 
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index 4b99f7b88beec..3dd8e6e207b09 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -86,7 +86,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head)
 
 	rcu_lock_acquire(&rcu_callback_map);
 	if (__is_kfree_rcu_offset(offset)) {
-		trace_rcu_invoke_kfree_callback("", head, offset);
+		trace_rcu_invoke_kvfree_callback("", head, offset);
 		kvfree((void *)head - offset);
 		rcu_lock_release(&rcu_callback_map);
 		return true;
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 2d10c50621c38..88b744ce896c0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2744,7 +2744,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
 	// If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock.
 	rcu_segcblist_enqueue(&rdp->cblist, head);
 	if (__is_kfree_rcu_offset((unsigned long)func))
-		trace_rcu_kfree_callback(rcu_state.name, head,
+		trace_rcu_kvfree_callback(rcu_state.name, head,
 					 (unsigned long)func,
 					 rcu_segcblist_n_cbs(&rdp->cblist));
 	else
@@ -2935,7 +2935,7 @@ static void kfree_rcu_work(struct work_struct *work)
 		next = head->next;
 		debug_rcu_head_unqueue(head);
 		rcu_lock_acquire(&rcu_callback_map);
-		trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset);
+		trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
 
 		if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset)))
 			kvfree((void *)head - offset);
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 04/18] rcu: Rename __is_kfree_rcu_offset() macro

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Rename __is_kfree_rcu_offset to __is_kvfree_rcu_offset.
All RCU paths use kvfree() now instead of kfree(), thus
rename it.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 include/linux/rcupdate.h | 6 +++---
 kernel/rcu/tiny.c        | 2 +-
 kernel/rcu/tree.c        | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 8b7128d0860e2..c6f6a195cb1cd 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -823,16 +823,16 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 
 /*
  * Does the specified offset indicate that the corresponding rcu_head
- * structure can be handled by kfree_rcu()?
+ * structure can be handled by kvfree_rcu()?
  */
-#define __is_kfree_rcu_offset(offset) ((offset) < 4096)
+#define __is_kvfree_rcu_offset(offset) ((offset) < 4096)
 
 /*
  * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain.
  */
 #define __kfree_rcu(head, offset) \
 	do { \
-		BUILD_BUG_ON(!__is_kfree_rcu_offset(offset)); \
+		BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \
 		kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
 	} while (0)
 
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index 3dd8e6e207b09..aa897c3f2e92c 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -85,7 +85,7 @@ static inline bool rcu_reclaim_tiny(struct rcu_head *head)
 	unsigned long offset = (unsigned long)head->func;
 
 	rcu_lock_acquire(&rcu_callback_map);
-	if (__is_kfree_rcu_offset(offset)) {
+	if (__is_kvfree_rcu_offset(offset)) {
 		trace_rcu_invoke_kvfree_callback("", head, offset);
 		kvfree((void *)head - offset);
 		rcu_lock_release(&rcu_callback_map);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 88b744ce896c0..1209945a34bfd 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2743,7 +2743,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
 		return; // Enqueued onto ->nocb_bypass, so just leave.
 	// If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock.
 	rcu_segcblist_enqueue(&rdp->cblist, head);
-	if (__is_kfree_rcu_offset((unsigned long)func))
+	if (__is_kvfree_rcu_offset((unsigned long)func))
 		trace_rcu_kvfree_callback(rcu_state.name, head,
 					 (unsigned long)func,
 					 rcu_segcblist_n_cbs(&rdp->cblist));
@@ -2937,7 +2937,7 @@ static void kfree_rcu_work(struct work_struct *work)
 		rcu_lock_acquire(&rcu_callback_map);
 		trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
 
-		if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset)))
+		if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
 			kvfree((void *)head - offset);
 
 		rcu_lock_release(&rcu_callback_map);
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 05/18] rcu: Rename kfree_call_rcu() to the kvfree_call_rcu().

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

The reason is, it is capable of freeing vmalloc()
memory now.

Do the same with __kfree_rcu() macro, it becomes
__kvfree_rcu(), the reason is the same as pointed
above.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 include/linux/rcupdate.h | 8 ++++----
 include/linux/rcutiny.h  | 2 +-
 include/linux/rcutree.h  | 2 +-
 kernel/rcu/tree.c        | 8 ++++----
 4 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index c6f6a195cb1cd..edb6eeba49f83 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -830,10 +830,10 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
 /*
  * Helper macro for kfree_rcu() to prevent argument-expansion eyestrain.
  */
-#define __kfree_rcu(head, offset) \
+#define __kvfree_rcu(head, offset) \
 	do { \
 		BUILD_BUG_ON(!__is_kvfree_rcu_offset(offset)); \
-		kfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
+		kvfree_call_rcu(head, (rcu_callback_t)(unsigned long)(offset)); \
 	} while (0)
 
 /**
@@ -852,7 +852,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
  * Because the functions are not allowed in the low-order 4096 bytes of
  * kernel virtual memory, offsets up to 4095 bytes can be accommodated.
  * If the offset is larger than 4095 bytes, a compile-time error will
- * be generated in __kfree_rcu().  If this error is triggered, you can
+ * be generated in __kvfree_rcu(). If this error is triggered, you can
  * either fall back to use of call_rcu() or rearrange the structure to
  * position the rcu_head structure into the first 4096 bytes.
  *
@@ -867,7 +867,7 @@ do {									\
 	typeof (ptr) ___p = (ptr);					\
 									\
 	if (___p)							\
-		__kfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
+		__kvfree_rcu(&((___p)->rhf), offsetof(typeof(*(ptr)), rhf)); \
 } while (0)
 
 /**
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index d77e11186afd1..5ba0bcb231976 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -34,7 +34,7 @@ static inline void synchronize_rcu_expedited(void)
 	synchronize_rcu();
 }
 
-static inline void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
+static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
 	call_rcu(head, func);
 }
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 45f3f66bb04df..3a7829d69fef8 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -33,7 +33,7 @@ static inline void rcu_virt_note_context_switch(int cpu)
 }
 
 void synchronize_rcu_expedited(void);
-void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
 
 void rcu_barrier(void);
 bool rcu_eqs_special_set(int cpu);
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 1209945a34bfd..3fb19ea039912 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3082,18 +3082,18 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
 }
 
 /*
- * Queue a request for lazy invocation of kfree_bulk()/kfree() after a grace
+ * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace
  * period. Please note there are two paths are maintained, one is the main one
  * that uses kfree_bulk() interface and second one is emergency one, that is
  * used only when the main path can not be maintained temporary, due to memory
  * pressure.
  *
- * Each kfree_call_rcu() request is added to a batch. The batch will be drained
+ * Each kvfree_call_rcu() request is added to a batch. The batch will be drained
  * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will
  * be free'd in workqueue context. This allows us to: batch requests together to
  * reduce the number of grace periods during heavy kfree_rcu() load.
  */
-void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
 	unsigned long flags;
 	struct kfree_rcu_cpu *krcp;
@@ -3142,7 +3142,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 		spin_unlock(&krcp->lock);
 	local_irq_restore(flags);
 }
-EXPORT_SYMBOL_GPL(kfree_call_rcu);
+EXPORT_SYMBOL_GPL(kvfree_call_rcu);
 
 static unsigned long
 kfree_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 06/18] mm/list_lru.c: Remove kvfree_rcu_local() function

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Since there is newly introduced kvfree_rcu() API, there is no need in
queuing and using call_rcu() to kvfree() an object after the GP.

Remove kvfree_rcu_local() function and replace call_rcu() by new
kvfree_rcu() API that does the same but in more efficient way.

Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 mm/list_lru.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/mm/list_lru.c b/mm/list_lru.c
index 386424688f805..69becdb224080 100644
--- a/mm/list_lru.c
+++ b/mm/list_lru.c
@@ -12,6 +12,7 @@
 #include <linux/slab.h>
 #include <linux/mutex.h>
 #include <linux/memcontrol.h>
+#include <linux/rcupdate.h>
 #include "slab.h"
 
 #ifdef CONFIG_MEMCG_KMEM
@@ -383,21 +384,13 @@ static void memcg_destroy_list_lru_node(struct list_lru_node *nlru)
 	struct list_lru_memcg *memcg_lrus;
 	/*
 	 * This is called when shrinker has already been unregistered,
-	 * and nobody can use it. So, there is no need to use kvfree_rcu_local().
+	 * and nobody can use it. So, there is no need to use kvfree_rcu().
 	 */
 	memcg_lrus = rcu_dereference_protected(nlru->memcg_lrus, true);
 	__memcg_destroy_list_lru_node(memcg_lrus, 0, memcg_nr_cache_ids);
 	kvfree(memcg_lrus);
 }
 
-static void kvfree_rcu_local(struct rcu_head *head)
-{
-	struct list_lru_memcg *mlru;
-
-	mlru = container_of(head, struct list_lru_memcg, rcu);
-	kvfree(mlru);
-}
-
 static int memcg_update_list_lru_node(struct list_lru_node *nlru,
 				      int old_size, int new_size)
 {
@@ -429,7 +422,7 @@ static int memcg_update_list_lru_node(struct list_lru_node *nlru,
 	rcu_assign_pointer(nlru->memcg_lrus, new);
 	spin_unlock_irq(&nlru->lock);
 
-	call_rcu(&old->rcu, kvfree_rcu_local);
+	kvfree_rcu(old, rcu);
 	return 0;
 }
 
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 07/18] rcu/tree: Simplify debug_objects handling

[Joel Fernandes (Google)]

In order to prepare for future changes for headless RCU support, make the
debug_objects handling in kfree_rcu use the final 'pointer' value of the
object, instead of depending on the head.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 30 +++++++++++++-----------------
 1 file changed, 13 insertions(+), 17 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3fb19ea039912..95d1f5e20d5ec 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2825,7 +2825,6 @@ struct kfree_rcu_bulk_data {
 	unsigned long nr_records;
 	void *records[KFREE_BULK_MAX_ENTR];
 	struct kfree_rcu_bulk_data *next;
-	struct rcu_head *head_free_debug;
 };
 
 /**
@@ -2875,11 +2874,11 @@ struct kfree_rcu_cpu {
 static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc);
 
 static __always_inline void
-debug_rcu_head_unqueue_bulk(struct rcu_head *head)
+debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
 {
 #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
-	for (; head; head = head->next)
-		debug_rcu_head_unqueue(head);
+	for (int i = 0; i < bhead->nr_records; i++)
+		debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i]));
 #endif
 }
 
@@ -2909,7 +2908,7 @@ static void kfree_rcu_work(struct work_struct *work)
 	for (; bhead; bhead = bnext) {
 		bnext = bhead->next;
 
-		debug_rcu_head_unqueue_bulk(bhead->head_free_debug);
+		debug_rcu_bhead_unqueue(bhead);
 
 		rcu_lock_acquire(&rcu_callback_map);
 		trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
@@ -2931,14 +2930,15 @@ static void kfree_rcu_work(struct work_struct *work)
 	 */
 	for (; head; head = next) {
 		unsigned long offset = (unsigned long)head->func;
+		void *ptr = (void *)head - offset;
 
 		next = head->next;
-		debug_rcu_head_unqueue(head);
+		debug_rcu_head_unqueue((struct rcu_head *)ptr);
 		rcu_lock_acquire(&rcu_callback_map);
 		trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
 
 		if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
-			kvfree((void *)head - offset);
+			kvfree(ptr);
 
 		rcu_lock_release(&rcu_callback_map);
 		cond_resched_tasks_rcu_qs();
@@ -3062,18 +3062,11 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
 		/* Initialize the new block. */
 		bnode->nr_records = 0;
 		bnode->next = krcp->bhead;
-		bnode->head_free_debug = NULL;
 
 		/* Attach it to the head. */
 		krcp->bhead = bnode;
 	}
 
-#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
-	head->func = func;
-	head->next = krcp->bhead->head_free_debug;
-	krcp->bhead->head_free_debug = head;
-#endif
-
 	/* Finally insert. */
 	krcp->bhead->records[krcp->bhead->nr_records++] =
 		(void *) head - (unsigned long) func;
@@ -3097,14 +3090,17 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
 	unsigned long flags;
 	struct kfree_rcu_cpu *krcp;
+	void *ptr;
 
 	local_irq_save(flags);	// For safely calling this_cpu_ptr().
 	krcp = this_cpu_ptr(&krc);
 	if (krcp->initialized)
 		spin_lock(&krcp->lock);
 
+	ptr = (void *)head - (unsigned long)func;
+
 	// Queue the object but don't yet schedule the batch.
-	if (debug_rcu_head_queue(head)) {
+	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
 		WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
 			  __func__, head);
@@ -3121,8 +3117,8 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	 * Under high memory pressure GFP_NOWAIT can fail,
 	 * in that case the emergency path is maintained.
 	 */
-	if (is_vmalloc_addr((void *) head - (unsigned long) func) ||
-			!kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
+	if (is_vmalloc_addr(ptr) ||
+	    !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
 		head->func = func;
 		head->next = krcp->head;
 		krcp->head = head;
-- 
2.26.0.rc2.310.g2932bb562d-goog

Re: [PATCH 07/18] rcu/tree: Simplify debug_objects handling

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

Hi "Joel,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   kernel/rcu/tree.c: In function 'debug_rcu_bhead_unqueue':
>> kernel/rcu/tree.c:2880:2: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
     for (int i = 0; i < bhead->nr_records; i++)
     ^~~
   kernel/rcu/tree.c:2880:2: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code

vim +/for +2880 kernel/rcu/tree.c

  2875	
  2876	static __always_inline void
  2877	debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
  2878	{
  2879	#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
> 2880		for (int i = 0; i < bhead->nr_records; i++)
  2881			debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i]));
  2882	#endif
  2883	}
  2884	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 71599 bytes --]

Re: [PATCH 07/18] rcu/tree: Simplify debug_objects handling

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

Hi "Joel,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   kernel/rcu/tree.c: In function 'debug_rcu_bhead_unqueue':
>> kernel/rcu/tree.c:2880:2: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
     for (int i = 0; i < bhead->nr_records; i++)
     ^~~
   kernel/rcu/tree.c:2880:2: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code

vim +/for +2880 kernel/rcu/tree.c

  2875	
  2876	static __always_inline void
  2877	debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
  2878	{
  2879	#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
> 2880		for (int i = 0; i < bhead->nr_records; i++)
  2881			debug_rcu_head_unqueue((struct rcu_head *)(bhead->records[i]));
  2882	#endif
  2883	}
  2884	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 71599 bytes --]

Re: [PATCH 07/18] rcu/tree: Simplify debug_objects handling

[Joel Fernandes]

On Mon, Mar 30, 2020 at 03:00:18PM +0800, kbuild test robot wrote:
> Hi "Joel,
> 
> Thank you for the patch! Yet something to improve:
> 
> [auto build test ERROR on rcu/dev]
> [also build test ERROR on rcu/rcu/next next-20200327]
> [cannot apply to linus/master linux/master v5.6]
> [if your patch is applied to the wrong git tree, please drop us a note to help
> improve the system. BTW, we also suggest to use '--base' option to specify the
> base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

Should have run allmodconfig, this is a slightly older patch and did not see
a problem earlier in prior testing.

Will make the trivial fix now and update my tree, thanks!

 - Joel

[PATCH 08/18] rcu/tree: Clarify emergency path comment better

[Joel Fernandes (Google)]

Clarify emergency path comment better in kfree_rcu().

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 95d1f5e20d5ec..8dfa4b32e4d00 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2924,9 +2924,10 @@ static void kfree_rcu_work(struct work_struct *work)
 	}
 
 	/*
-	 * vmalloc() pointers end up here also emergency case. It can
-	 * happen under low memory condition when an allocation gets
-	 * failed, so the "bulk" path can not be temporary maintained.
+	 * We can end up here either with 1) vmalloc() pointers or 2) were low
+	 * on memory and could not allocate a bulk array. It can happen under
+	 * low memory condition when an allocation gets failed, so the "bulk"
+	 * path can not be temporarly used.
 	 */
 	for (; head; head = next) {
 		unsigned long offset = (unsigned long)head->func;
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 09/18] rcu/tree: Simplify KFREE_BULK_MAX_ENTR macro

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

We can simplify KFREE_BULK_MAX_ENTR macro and get rid of
magic numbers which were used to make the structure to be
exactly one page.

Suggested-by: Boqun Feng <boqun.feng@gmail.com>
Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8dfa4b32e4d00..cfe456e68c644 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2807,13 +2807,6 @@ EXPORT_SYMBOL_GPL(call_rcu);
 #define KFREE_DRAIN_JIFFIES (HZ / 50)
 #define KFREE_N_BATCHES 2
 
-/*
- * This macro defines how many entries the "records" array
- * will contain. It is based on the fact that the size of
- * kfree_rcu_bulk_data structure becomes exactly one page.
- */
-#define KFREE_BULK_MAX_ENTR ((PAGE_SIZE / sizeof(void *)) - 3)
-
 /**
  * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers
  * @nr_records: Number of active pointers in the array
@@ -2827,6 +2820,14 @@ struct kfree_rcu_bulk_data {
 	struct kfree_rcu_bulk_data *next;
 };
 
+/*
+ * This macro defines how many entries the "records" array
+ * will contain. It is based on the fact that the size of
+ * kfree_rcu_bulk_data structure becomes exactly one page.
+ */
+#define KFREE_BULK_MAX_ENTR \
+	((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *))
+
 /**
  * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
  * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

To do so we use an array of common kvfree_rcu_bulk_data
structure. It consists of two elements, index number 0
corresponds to SLAB ptrs., whereas vmalloc pointers can
be accessed by using index number 1.

The reason of not mixing pointers is to have an easy way
to to distinguish them.

It is also the preparation patch for head-less objects
support. When an object is head-less we can not queue
it into any list, instead a pointer is placed directly
into an array.

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 179 ++++++++++++++++++++++++++++------------------
 1 file changed, 108 insertions(+), 71 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index cfe456e68c644..8fbc8450284db 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2808,38 +2808,36 @@ EXPORT_SYMBOL_GPL(call_rcu);
 #define KFREE_N_BATCHES 2
 
 /**
- * struct kfree_rcu_bulk_data - single block to store kfree_rcu() pointers
+ * struct kvfree_rcu_bulk_data - single block to store kvfree() pointers
  * @nr_records: Number of active pointers in the array
- * @records: Array of the kfree_rcu() pointers
  * @next: Next bulk object in the block chain
- * @head_free_debug: For debug, when CONFIG_DEBUG_OBJECTS_RCU_HEAD is set
+ * @records: Array of the SLAB pointers
  */
-struct kfree_rcu_bulk_data {
+struct kvfree_rcu_bulk_data {
 	unsigned long nr_records;
-	void *records[KFREE_BULK_MAX_ENTR];
-	struct kfree_rcu_bulk_data *next;
+	struct kvfree_rcu_bulk_data *next;
+	void *records[];
 };
 
 /*
  * This macro defines how many entries the "records" array
  * will contain. It is based on the fact that the size of
- * kfree_rcu_bulk_data structure becomes exactly one page.
+ * kvfree_rcu_bulk_data become exactly one page.
  */
-#define KFREE_BULK_MAX_ENTR \
-	((PAGE_SIZE - sizeof(struct kfree_rcu_bulk_data)) / sizeof(void *))
+#define KVFREE_BULK_MAX_ENTR \
+	((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *))
 
 /**
  * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
  * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
  * @head_free: List of kfree_rcu() objects waiting for a grace period
- * @bhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period
+ * @bkvhead_free: Bulk-List of kfree_rcu() objects waiting for a grace period
  * @krcp: Pointer to @kfree_rcu_cpu structure
  */
-
 struct kfree_rcu_cpu_work {
 	struct rcu_work rcu_work;
 	struct rcu_head *head_free;
-	struct kfree_rcu_bulk_data *bhead_free;
+	struct kvfree_rcu_bulk_data *bkvhead_free[2];
 	struct kfree_rcu_cpu *krcp;
 };
 
@@ -2861,8 +2859,9 @@ struct kfree_rcu_cpu_work {
  */
 struct kfree_rcu_cpu {
 	struct rcu_head *head;
-	struct kfree_rcu_bulk_data *bhead;
-	struct kfree_rcu_bulk_data *bcached;
+	struct kvfree_rcu_bulk_data *bkvhead[2];
+	struct kvfree_rcu_bulk_data *bkvcache[2];
+
 	struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES];
 	spinlock_t lock;
 	struct delayed_work monitor_work;
@@ -2875,7 +2874,7 @@ struct kfree_rcu_cpu {
 static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc);
 
 static __always_inline void
-debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
+debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead)
 {
 #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
 	for (int i = 0; i < bhead->nr_records; i++)
@@ -2890,45 +2889,77 @@ debug_rcu_bhead_unqueue(struct kfree_rcu_bulk_data *bhead)
 static void kfree_rcu_work(struct work_struct *work)
 {
 	unsigned long flags;
+	struct kvfree_rcu_bulk_data *bkhead, *bknext;
+	struct kvfree_rcu_bulk_data *bvhead, *bvnext;
 	struct rcu_head *head, *next;
-	struct kfree_rcu_bulk_data *bhead, *bnext;
 	struct kfree_rcu_cpu *krcp;
 	struct kfree_rcu_cpu_work *krwp;
+	int i;
 
 	krwp = container_of(to_rcu_work(work),
-			    struct kfree_rcu_cpu_work, rcu_work);
+				struct kfree_rcu_cpu_work, rcu_work);
+
 	krcp = krwp->krcp;
 	spin_lock_irqsave(&krcp->lock, flags);
+	/* Channel 1. */
+	bkhead = krwp->bkvhead_free[0];
+	krwp->bkvhead_free[0] = NULL;
+
+	/* Channel 2. */
+	bvhead = krwp->bkvhead_free[1];
+	krwp->bkvhead_free[1] = NULL;
+
+	/* Channel 3. */
 	head = krwp->head_free;
 	krwp->head_free = NULL;
-	bhead = krwp->bhead_free;
-	krwp->bhead_free = NULL;
 	spin_unlock_irqrestore(&krcp->lock, flags);
 
-	/* "bhead" is now private, so traverse locklessly. */
-	for (; bhead; bhead = bnext) {
-		bnext = bhead->next;
+	/* kmalloc()/kfree() channel. */
+	for (; bkhead; bkhead = bknext) {
+		bknext = bkhead->next;
 
-		debug_rcu_bhead_unqueue(bhead);
+		debug_rcu_bhead_unqueue(bkhead);
 
 		rcu_lock_acquire(&rcu_callback_map);
 		trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
-			bhead->nr_records, bhead->records);
+			bkhead->nr_records, bkhead->records);
+
+		kfree_bulk(bkhead->nr_records, bkhead->records);
+		rcu_lock_release(&rcu_callback_map);
+
+		if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
+			free_page((unsigned long) bkhead);
+
+		cond_resched_tasks_rcu_qs();
+	}
+
+	/* vmalloc()/vfree() channel. */
+	for (; bvhead; bvhead = bvnext) {
+		bvnext = bvhead->next;
+
+		debug_rcu_bhead_unqueue(bvhead);
 
-		kfree_bulk(bhead->nr_records, bhead->records);
+		rcu_lock_acquire(&rcu_callback_map);
+		for (i = 0; i < bvhead->nr_records; i++) {
+			trace_rcu_invoke_kvfree_callback(rcu_state.name,
+				(struct rcu_head *) bvhead->records[i], 0);
+			vfree(bvhead->records[i]);
+		}
 		rcu_lock_release(&rcu_callback_map);
 
-		if (cmpxchg(&krcp->bcached, NULL, bhead))
-			free_page((unsigned long) bhead);
+		if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
+			free_page((unsigned long) bvhead);
 
 		cond_resched_tasks_rcu_qs();
 	}
 
 	/*
-	 * We can end up here either with 1) vmalloc() pointers or 2) were low
-	 * on memory and could not allocate a bulk array. It can happen under
-	 * low memory condition when an allocation gets failed, so the "bulk"
-	 * path can not be temporarly used.
+	 * This path covers emergency case only due to high
+	 * memory pressure also means low memory condition,
+	 * when we could not allocate a bulk array.
+	 *
+	 * Under that condition an object is queued to the
+	 * list instead.
 	 */
 	for (; head; head = next) {
 		unsigned long offset = (unsigned long)head->func;
@@ -2965,21 +2996,34 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)
 		krwp = &(krcp->krw_arr[i]);
 
 		/*
-		 * Try to detach bhead or head and attach it over any
+		 * Try to detach bkvhead or head and attach it over any
 		 * available corresponding free channel. It can be that
 		 * a previous RCU batch is in progress, it means that
 		 * immediately to queue another one is not possible so
 		 * return false to tell caller to retry.
 		 */
-		if ((krcp->bhead && !krwp->bhead_free) ||
+		if ((krcp->bkvhead[0] && !krwp->bkvhead_free[0]) ||
+			(krcp->bkvhead[1] && !krwp->bkvhead_free[1]) ||
 				(krcp->head && !krwp->head_free)) {
-			/* Channel 1. */
-			if (!krwp->bhead_free) {
-				krwp->bhead_free = krcp->bhead;
-				krcp->bhead = NULL;
+			/*
+			 * Channel 1 corresponds to SLAB ptrs.
+			 */
+			if (!krwp->bkvhead_free[0]) {
+				krwp->bkvhead_free[0] = krcp->bkvhead[0];
+				krcp->bkvhead[0] = NULL;
+			}
+
+			/*
+			 * Channel 2 corresponds to vmalloc ptrs.
+			 */
+			if (!krwp->bkvhead_free[1]) {
+				krwp->bkvhead_free[1] = krcp->bkvhead[1];
+				krcp->bkvhead[1] = NULL;
 			}
 
-			/* Channel 2. */
+			/*
+			 * Channel 3 corresponds to emergency path.
+			 */
 			if (!krwp->head_free) {
 				krwp->head_free = krcp->head;
 				krcp->head = NULL;
@@ -2988,10 +3032,11 @@ static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)
 			WRITE_ONCE(krcp->count, 0);
 
 			/*
-			 * One work is per one batch, so there are two "free channels",
-			 * "bhead_free" and "head_free" the batch can handle. It can be
-			 * that the work is in the pending state when two channels have
-			 * been detached following each other, one by one.
+			 * One work is per one batch, so there are three
+			 * "free channels", the batch can handle. It can
+			 * be that the work is in the pending state when
+			 * channels have been detached following by each
+			 * other.
 			 */
 			queue_rcu_work(system_wq, &krwp->rcu_work);
 			queued = true;
@@ -3036,26 +3081,25 @@ static void kfree_rcu_monitor(struct work_struct *work)
 }
 
 static inline bool
-kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
-	struct rcu_head *head, rcu_callback_t func)
+kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
 {
-	struct kfree_rcu_bulk_data *bnode;
+	struct kvfree_rcu_bulk_data *bnode;
+	int idx;
 
 	if (unlikely(!krcp->initialized))
 		return false;
 
 	lockdep_assert_held(&krcp->lock);
+	idx = !is_vmalloc_addr(ptr) ? 0:1;
 
 	/* Check if a new block is required. */
-	if (!krcp->bhead ||
-			krcp->bhead->nr_records == KFREE_BULK_MAX_ENTR) {
-		bnode = xchg(&krcp->bcached, NULL);
-		if (!bnode) {
-			WARN_ON_ONCE(sizeof(struct kfree_rcu_bulk_data) > PAGE_SIZE);
-
-			bnode = (struct kfree_rcu_bulk_data *)
+	if (!krcp->bkvhead[idx] ||
+			krcp->bkvhead[idx]->nr_records ==
+				KVFREE_BULK_MAX_ENTR) {
+		bnode = xchg(&krcp->bkvcache[idx], NULL);
+		if (!bnode)
+			bnode = (struct kvfree_rcu_bulk_data *)
 				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
-		}
 
 		/* Switch to emergency path. */
 		if (unlikely(!bnode))
@@ -3063,30 +3107,30 @@ kfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp,
 
 		/* Initialize the new block. */
 		bnode->nr_records = 0;
-		bnode->next = krcp->bhead;
+		bnode->next = krcp->bkvhead[idx];
 
 		/* Attach it to the head. */
-		krcp->bhead = bnode;
+		krcp->bkvhead[idx] = bnode;
 	}
 
 	/* Finally insert. */
-	krcp->bhead->records[krcp->bhead->nr_records++] =
-		(void *) head - (unsigned long) func;
+	krcp->bkvhead[idx]->records
+		[krcp->bkvhead[idx]->nr_records++] = ptr;
 
 	return true;
 }
 
 /*
- * Queue a request for lazy invocation of kfree_bulk()/kvfree() after a grace
- * period. Please note there are two paths are maintained, one is the main one
- * that uses kfree_bulk() interface and second one is emergency one, that is
- * used only when the main path can not be maintained temporary, due to memory
- * pressure.
+ * Queue a request for lazy invocation of appropriate free routine after a
+ * grace period. Please note there are three paths are maintained, two are the
+ * main ones that use array of pointers interface and third one is emergency
+ * one, that is used only when the main path can not be maintained temporary,
+ * due to memory pressure.
  *
  * Each kvfree_call_rcu() request is added to a batch. The batch will be drained
  * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will
  * be free'd in workqueue context. This allows us to: batch requests together to
- * reduce the number of grace periods during heavy kfree_rcu() load.
+ * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
  */
 void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
@@ -3110,17 +3154,10 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	}
 
 	/*
-	 * We do not queue vmalloc pointers into array,
-	 * instead they are just queued to the list. We
-	 * do it because of:
-	 *    a) to distinguish kmalloc()/vmalloc() ptrs;
-	 *    b) there is no vmalloc_bulk() interface.
-	 *
 	 * Under high memory pressure GFP_NOWAIT can fail,
 	 * in that case the emergency path is maintained.
 	 */
-	if (is_vmalloc_addr(ptr) ||
-	    !kfree_call_rcu_add_ptr_to_bulk(krcp, head, func)) {
+	if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) {
 		head->func = func;
 		head->next = krcp->head;
 		krcp->head = head;
-- 
2.26.0.rc2.310.g2932bb562d-goog

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 4782 bytes --]

Hi "Joel,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: powerpc-defconfig (attached as .config)
compiler: powerpc64-linux-gcc (GCC) 9.3.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        GCC_VERSION=9.3.0 make.cross ARCH=powerpc 

If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   kernel/rcu/tree.c: In function 'kfree_rcu_work':
>> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree'; did you mean 'kfree'? [-Werror=implicit-function-declaration]
    2946 |    vfree(bvhead->records[i]);
         |    ^~~~~
         |    kfree
   cc1: some warnings being treated as errors

vim +2946 kernel/rcu/tree.c

  2884	
  2885	/*
  2886	 * This function is invoked in workqueue context after a grace period.
  2887	 * It frees all the objects queued on ->bhead_free or ->head_free.
  2888	 */
  2889	static void kfree_rcu_work(struct work_struct *work)
  2890	{
  2891		unsigned long flags;
  2892		struct kvfree_rcu_bulk_data *bkhead, *bknext;
  2893		struct kvfree_rcu_bulk_data *bvhead, *bvnext;
  2894		struct rcu_head *head, *next;
  2895		struct kfree_rcu_cpu *krcp;
  2896		struct kfree_rcu_cpu_work *krwp;
  2897		int i;
  2898	
  2899		krwp = container_of(to_rcu_work(work),
  2900					struct kfree_rcu_cpu_work, rcu_work);
  2901	
  2902		krcp = krwp->krcp;
  2903		spin_lock_irqsave(&krcp->lock, flags);
  2904		/* Channel 1. */
  2905		bkhead = krwp->bkvhead_free[0];
  2906		krwp->bkvhead_free[0] = NULL;
  2907	
  2908		/* Channel 2. */
  2909		bvhead = krwp->bkvhead_free[1];
  2910		krwp->bkvhead_free[1] = NULL;
  2911	
  2912		/* Channel 3. */
  2913		head = krwp->head_free;
  2914		krwp->head_free = NULL;
  2915		spin_unlock_irqrestore(&krcp->lock, flags);
  2916	
  2917		/* kmalloc()/kfree() channel. */
  2918		for (; bkhead; bkhead = bknext) {
  2919			bknext = bkhead->next;
  2920	
  2921			debug_rcu_bhead_unqueue(bkhead);
  2922	
  2923			rcu_lock_acquire(&rcu_callback_map);
  2924			trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
  2925				bkhead->nr_records, bkhead->records);
  2926	
  2927			kfree_bulk(bkhead->nr_records, bkhead->records);
  2928			rcu_lock_release(&rcu_callback_map);
  2929	
  2930			if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
  2931				free_page((unsigned long) bkhead);
  2932	
  2933			cond_resched_tasks_rcu_qs();
  2934		}
  2935	
  2936		/* vmalloc()/vfree() channel. */
  2937		for (; bvhead; bvhead = bvnext) {
  2938			bvnext = bvhead->next;
  2939	
  2940			debug_rcu_bhead_unqueue(bvhead);
  2941	
  2942			rcu_lock_acquire(&rcu_callback_map);
  2943			for (i = 0; i < bvhead->nr_records; i++) {
  2944				trace_rcu_invoke_kvfree_callback(rcu_state.name,
  2945					(struct rcu_head *) bvhead->records[i], 0);
> 2946				vfree(bvhead->records[i]);
  2947			}
  2948			rcu_lock_release(&rcu_callback_map);
  2949	
  2950			if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
  2951				free_page((unsigned long) bvhead);
  2952	
  2953			cond_resched_tasks_rcu_qs();
  2954		}
  2955	
  2956		/*
  2957		 * This path covers emergency case only due to high
  2958		 * memory pressure also means low memory condition,
  2959		 * when we could not allocate a bulk array.
  2960		 *
  2961		 * Under that condition an object is queued to the
  2962		 * list instead.
  2963		 */
  2964		for (; head; head = next) {
  2965			unsigned long offset = (unsigned long)head->func;
  2966			void *ptr = (void *)head - offset;
  2967	
  2968			next = head->next;
  2969			debug_rcu_head_unqueue((struct rcu_head *)ptr);
  2970			rcu_lock_acquire(&rcu_callback_map);
  2971			trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
  2972	
  2973			if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
  2974				kvfree(ptr);
  2975	
  2976			rcu_lock_release(&rcu_callback_map);
  2977			cond_resched_tasks_rcu_qs();
  2978		}
  2979	}
  2980	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 25986 bytes --]

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 4920 bytes --]

Hi "Joel,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: powerpc-defconfig (attached as .config)
compiler: powerpc64-linux-gcc (GCC) 9.3.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        GCC_VERSION=9.3.0 make.cross ARCH=powerpc 

If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   kernel/rcu/tree.c: In function 'kfree_rcu_work':
>> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree'; did you mean 'kfree'? [-Werror=implicit-function-declaration]
    2946 |    vfree(bvhead->records[i]);
         |    ^~~~~
         |    kfree
   cc1: some warnings being treated as errors

vim +2946 kernel/rcu/tree.c

  2884	
  2885	/*
  2886	 * This function is invoked in workqueue context after a grace period.
  2887	 * It frees all the objects queued on ->bhead_free or ->head_free.
  2888	 */
  2889	static void kfree_rcu_work(struct work_struct *work)
  2890	{
  2891		unsigned long flags;
  2892		struct kvfree_rcu_bulk_data *bkhead, *bknext;
  2893		struct kvfree_rcu_bulk_data *bvhead, *bvnext;
  2894		struct rcu_head *head, *next;
  2895		struct kfree_rcu_cpu *krcp;
  2896		struct kfree_rcu_cpu_work *krwp;
  2897		int i;
  2898	
  2899		krwp = container_of(to_rcu_work(work),
  2900					struct kfree_rcu_cpu_work, rcu_work);
  2901	
  2902		krcp = krwp->krcp;
  2903		spin_lock_irqsave(&krcp->lock, flags);
  2904		/* Channel 1. */
  2905		bkhead = krwp->bkvhead_free[0];
  2906		krwp->bkvhead_free[0] = NULL;
  2907	
  2908		/* Channel 2. */
  2909		bvhead = krwp->bkvhead_free[1];
  2910		krwp->bkvhead_free[1] = NULL;
  2911	
  2912		/* Channel 3. */
  2913		head = krwp->head_free;
  2914		krwp->head_free = NULL;
  2915		spin_unlock_irqrestore(&krcp->lock, flags);
  2916	
  2917		/* kmalloc()/kfree() channel. */
  2918		for (; bkhead; bkhead = bknext) {
  2919			bknext = bkhead->next;
  2920	
  2921			debug_rcu_bhead_unqueue(bkhead);
  2922	
  2923			rcu_lock_acquire(&rcu_callback_map);
  2924			trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
  2925				bkhead->nr_records, bkhead->records);
  2926	
  2927			kfree_bulk(bkhead->nr_records, bkhead->records);
  2928			rcu_lock_release(&rcu_callback_map);
  2929	
  2930			if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
  2931				free_page((unsigned long) bkhead);
  2932	
  2933			cond_resched_tasks_rcu_qs();
  2934		}
  2935	
  2936		/* vmalloc()/vfree() channel. */
  2937		for (; bvhead; bvhead = bvnext) {
  2938			bvnext = bvhead->next;
  2939	
  2940			debug_rcu_bhead_unqueue(bvhead);
  2941	
  2942			rcu_lock_acquire(&rcu_callback_map);
  2943			for (i = 0; i < bvhead->nr_records; i++) {
  2944				trace_rcu_invoke_kvfree_callback(rcu_state.name,
  2945					(struct rcu_head *) bvhead->records[i], 0);
> 2946				vfree(bvhead->records[i]);
  2947			}
  2948			rcu_lock_release(&rcu_callback_map);
  2949	
  2950			if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
  2951				free_page((unsigned long) bvhead);
  2952	
  2953			cond_resched_tasks_rcu_qs();
  2954		}
  2955	
  2956		/*
  2957		 * This path covers emergency case only due to high
  2958		 * memory pressure also means low memory condition,
  2959		 * when we could not allocate a bulk array.
  2960		 *
  2961		 * Under that condition an object is queued to the
  2962		 * list instead.
  2963		 */
  2964		for (; head; head = next) {
  2965			unsigned long offset = (unsigned long)head->func;
  2966			void *ptr = (void *)head - offset;
  2967	
  2968			next = head->next;
  2969			debug_rcu_head_unqueue((struct rcu_head *)ptr);
  2970			rcu_lock_acquire(&rcu_callback_map);
  2971			trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
  2972	
  2973			if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
  2974				kvfree(ptr);
  2975	
  2976			rcu_lock_release(&rcu_callback_map);
  2977			cond_resched_tasks_rcu_qs();
  2978		}
  2979	}
  2980	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 25986 bytes --]

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 4737 bytes --]

Hi "Joel,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: mips-randconfig-a001-20200330 (attached as .config)
compiler: mips64el-linux-gcc (GCC) 5.5.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        GCC_VERSION=5.5.0 make.cross ARCH=mips 

If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   kernel/rcu/tree.c: In function 'kfree_rcu_work':
>> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree' [-Werror=implicit-function-declaration]
       vfree(bvhead->records[i]);
       ^
   cc1: some warnings being treated as errors

vim +/vfree +2946 kernel/rcu/tree.c

  2884	
  2885	/*
  2886	 * This function is invoked in workqueue context after a grace period.
  2887	 * It frees all the objects queued on ->bhead_free or ->head_free.
  2888	 */
  2889	static void kfree_rcu_work(struct work_struct *work)
  2890	{
  2891		unsigned long flags;
  2892		struct kvfree_rcu_bulk_data *bkhead, *bknext;
  2893		struct kvfree_rcu_bulk_data *bvhead, *bvnext;
  2894		struct rcu_head *head, *next;
  2895		struct kfree_rcu_cpu *krcp;
  2896		struct kfree_rcu_cpu_work *krwp;
  2897		int i;
  2898	
  2899		krwp = container_of(to_rcu_work(work),
  2900					struct kfree_rcu_cpu_work, rcu_work);
  2901	
  2902		krcp = krwp->krcp;
  2903		spin_lock_irqsave(&krcp->lock, flags);
  2904		/* Channel 1. */
  2905		bkhead = krwp->bkvhead_free[0];
  2906		krwp->bkvhead_free[0] = NULL;
  2907	
  2908		/* Channel 2. */
  2909		bvhead = krwp->bkvhead_free[1];
  2910		krwp->bkvhead_free[1] = NULL;
  2911	
  2912		/* Channel 3. */
  2913		head = krwp->head_free;
  2914		krwp->head_free = NULL;
  2915		spin_unlock_irqrestore(&krcp->lock, flags);
  2916	
  2917		/* kmalloc()/kfree() channel. */
  2918		for (; bkhead; bkhead = bknext) {
  2919			bknext = bkhead->next;
  2920	
  2921			debug_rcu_bhead_unqueue(bkhead);
  2922	
  2923			rcu_lock_acquire(&rcu_callback_map);
  2924			trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
  2925				bkhead->nr_records, bkhead->records);
  2926	
  2927			kfree_bulk(bkhead->nr_records, bkhead->records);
  2928			rcu_lock_release(&rcu_callback_map);
  2929	
  2930			if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
  2931				free_page((unsigned long) bkhead);
  2932	
  2933			cond_resched_tasks_rcu_qs();
  2934		}
  2935	
  2936		/* vmalloc()/vfree() channel. */
  2937		for (; bvhead; bvhead = bvnext) {
  2938			bvnext = bvhead->next;
  2939	
  2940			debug_rcu_bhead_unqueue(bvhead);
  2941	
  2942			rcu_lock_acquire(&rcu_callback_map);
  2943			for (i = 0; i < bvhead->nr_records; i++) {
  2944				trace_rcu_invoke_kvfree_callback(rcu_state.name,
  2945					(struct rcu_head *) bvhead->records[i], 0);
> 2946				vfree(bvhead->records[i]);
  2947			}
  2948			rcu_lock_release(&rcu_callback_map);
  2949	
  2950			if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
  2951				free_page((unsigned long) bvhead);
  2952	
  2953			cond_resched_tasks_rcu_qs();
  2954		}
  2955	
  2956		/*
  2957		 * This path covers emergency case only due to high
  2958		 * memory pressure also means low memory condition,
  2959		 * when we could not allocate a bulk array.
  2960		 *
  2961		 * Under that condition an object is queued to the
  2962		 * list instead.
  2963		 */
  2964		for (; head; head = next) {
  2965			unsigned long offset = (unsigned long)head->func;
  2966			void *ptr = (void *)head - offset;
  2967	
  2968			next = head->next;
  2969			debug_rcu_head_unqueue((struct rcu_head *)ptr);
  2970			rcu_lock_acquire(&rcu_callback_map);
  2971			trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
  2972	
  2973			if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
  2974				kvfree(ptr);
  2975	
  2976			rcu_lock_release(&rcu_callback_map);
  2977			cond_resched_tasks_rcu_qs();
  2978		}
  2979	}
  2980	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 28352 bytes --]

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 4874 bytes --]

Hi "Joel,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on rcu/dev]
[also build test ERROR on rcu/rcu/next next-20200327]
[cannot apply to linus/master linux/master v5.6]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
config: mips-randconfig-a001-20200330 (attached as .config)
compiler: mips64el-linux-gcc (GCC) 5.5.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        GCC_VERSION=5.5.0 make.cross ARCH=mips 

If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   kernel/rcu/tree.c: In function 'kfree_rcu_work':
>> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree' [-Werror=implicit-function-declaration]
       vfree(bvhead->records[i]);
       ^
   cc1: some warnings being treated as errors

vim +/vfree +2946 kernel/rcu/tree.c

  2884	
  2885	/*
  2886	 * This function is invoked in workqueue context after a grace period.
  2887	 * It frees all the objects queued on ->bhead_free or ->head_free.
  2888	 */
  2889	static void kfree_rcu_work(struct work_struct *work)
  2890	{
  2891		unsigned long flags;
  2892		struct kvfree_rcu_bulk_data *bkhead, *bknext;
  2893		struct kvfree_rcu_bulk_data *bvhead, *bvnext;
  2894		struct rcu_head *head, *next;
  2895		struct kfree_rcu_cpu *krcp;
  2896		struct kfree_rcu_cpu_work *krwp;
  2897		int i;
  2898	
  2899		krwp = container_of(to_rcu_work(work),
  2900					struct kfree_rcu_cpu_work, rcu_work);
  2901	
  2902		krcp = krwp->krcp;
  2903		spin_lock_irqsave(&krcp->lock, flags);
  2904		/* Channel 1. */
  2905		bkhead = krwp->bkvhead_free[0];
  2906		krwp->bkvhead_free[0] = NULL;
  2907	
  2908		/* Channel 2. */
  2909		bvhead = krwp->bkvhead_free[1];
  2910		krwp->bkvhead_free[1] = NULL;
  2911	
  2912		/* Channel 3. */
  2913		head = krwp->head_free;
  2914		krwp->head_free = NULL;
  2915		spin_unlock_irqrestore(&krcp->lock, flags);
  2916	
  2917		/* kmalloc()/kfree() channel. */
  2918		for (; bkhead; bkhead = bknext) {
  2919			bknext = bkhead->next;
  2920	
  2921			debug_rcu_bhead_unqueue(bkhead);
  2922	
  2923			rcu_lock_acquire(&rcu_callback_map);
  2924			trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
  2925				bkhead->nr_records, bkhead->records);
  2926	
  2927			kfree_bulk(bkhead->nr_records, bkhead->records);
  2928			rcu_lock_release(&rcu_callback_map);
  2929	
  2930			if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
  2931				free_page((unsigned long) bkhead);
  2932	
  2933			cond_resched_tasks_rcu_qs();
  2934		}
  2935	
  2936		/* vmalloc()/vfree() channel. */
  2937		for (; bvhead; bvhead = bvnext) {
  2938			bvnext = bvhead->next;
  2939	
  2940			debug_rcu_bhead_unqueue(bvhead);
  2941	
  2942			rcu_lock_acquire(&rcu_callback_map);
  2943			for (i = 0; i < bvhead->nr_records; i++) {
  2944				trace_rcu_invoke_kvfree_callback(rcu_state.name,
  2945					(struct rcu_head *) bvhead->records[i], 0);
> 2946				vfree(bvhead->records[i]);
  2947			}
  2948			rcu_lock_release(&rcu_callback_map);
  2949	
  2950			if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
  2951				free_page((unsigned long) bvhead);
  2952	
  2953			cond_resched_tasks_rcu_qs();
  2954		}
  2955	
  2956		/*
  2957		 * This path covers emergency case only due to high
  2958		 * memory pressure also means low memory condition,
  2959		 * when we could not allocate a bulk array.
  2960		 *
  2961		 * Under that condition an object is queued to the
  2962		 * list instead.
  2963		 */
  2964		for (; head; head = next) {
  2965			unsigned long offset = (unsigned long)head->func;
  2966			void *ptr = (void *)head - offset;
  2967	
  2968			next = head->next;
  2969			debug_rcu_head_unqueue((struct rcu_head *)ptr);
  2970			rcu_lock_acquire(&rcu_callback_map);
  2971			trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
  2972	
  2973			if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
  2974				kvfree(ptr);
  2975	
  2976			rcu_lock_release(&rcu_callback_map);
  2977			cond_resched_tasks_rcu_qs();
  2978		}
  2979	}
  2980	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 28352 bytes --]

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Uladzislau Rezki]

Hello, Joel.

Sent out the patch fixing build error.

--
Vlad Rezki

> Hi "Joel,
> 
> Thank you for the patch! Yet something to improve:
> 
> [auto build test ERROR on rcu/dev]
> [also build test ERROR on rcu/rcu/next next-20200327]
> [cannot apply to linus/master linux/master v5.6]
> [if your patch is applied to the wrong git tree, please drop us a note to help
> improve the system. BTW, we also suggest to use '--base' option to specify the
> base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
> 
> url:    https://github.com/0day-ci/linux/commits/Joel-Fernandes-Google/kfree_rcu-improvements-for-rcu-dev/20200330-113719
> base:   https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
> config: mips-randconfig-a001-20200330 (attached as .config)
> compiler: mips64el-linux-gcc (GCC) 5.5.0
> reproduce:
>         wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # save the attached .config to linux build tree
>         GCC_VERSION=5.5.0 make.cross ARCH=mips 
> 
> If you fix the issue, kindly add following tag
> Reported-by: kbuild test robot <lkp@intel.com>
> 
> All errors (new ones prefixed by >>):
> 
>    kernel/rcu/tree.c: In function 'kfree_rcu_work':
> >> kernel/rcu/tree.c:2946:4: error: implicit declaration of function 'vfree' [-Werror=implicit-function-declaration]
>        vfree(bvhead->records[i]);
>        ^
>    cc1: some warnings being treated as errors
> 
> vim +/vfree +2946 kernel/rcu/tree.c
> 
>   2884	
>   2885	/*
>   2886	 * This function is invoked in workqueue context after a grace period.
>   2887	 * It frees all the objects queued on ->bhead_free or ->head_free.
>   2888	 */
>   2889	static void kfree_rcu_work(struct work_struct *work)
>   2890	{
>   2891		unsigned long flags;
>   2892		struct kvfree_rcu_bulk_data *bkhead, *bknext;
>   2893		struct kvfree_rcu_bulk_data *bvhead, *bvnext;
>   2894		struct rcu_head *head, *next;
>   2895		struct kfree_rcu_cpu *krcp;
>   2896		struct kfree_rcu_cpu_work *krwp;
>   2897		int i;
>   2898	
>   2899		krwp = container_of(to_rcu_work(work),
>   2900					struct kfree_rcu_cpu_work, rcu_work);
>   2901	
>   2902		krcp = krwp->krcp;
>   2903		spin_lock_irqsave(&krcp->lock, flags);
>   2904		/* Channel 1. */
>   2905		bkhead = krwp->bkvhead_free[0];
>   2906		krwp->bkvhead_free[0] = NULL;
>   2907	
>   2908		/* Channel 2. */
>   2909		bvhead = krwp->bkvhead_free[1];
>   2910		krwp->bkvhead_free[1] = NULL;
>   2911	
>   2912		/* Channel 3. */
>   2913		head = krwp->head_free;
>   2914		krwp->head_free = NULL;
>   2915		spin_unlock_irqrestore(&krcp->lock, flags);
>   2916	
>   2917		/* kmalloc()/kfree() channel. */
>   2918		for (; bkhead; bkhead = bknext) {
>   2919			bknext = bkhead->next;
>   2920	
>   2921			debug_rcu_bhead_unqueue(bkhead);
>   2922	
>   2923			rcu_lock_acquire(&rcu_callback_map);
>   2924			trace_rcu_invoke_kfree_bulk_callback(rcu_state.name,
>   2925				bkhead->nr_records, bkhead->records);
>   2926	
>   2927			kfree_bulk(bkhead->nr_records, bkhead->records);
>   2928			rcu_lock_release(&rcu_callback_map);
>   2929	
>   2930			if (cmpxchg(&krcp->bkvcache[0], NULL, bkhead))
>   2931				free_page((unsigned long) bkhead);
>   2932	
>   2933			cond_resched_tasks_rcu_qs();
>   2934		}
>   2935	
>   2936		/* vmalloc()/vfree() channel. */
>   2937		for (; bvhead; bvhead = bvnext) {
>   2938			bvnext = bvhead->next;
>   2939	
>   2940			debug_rcu_bhead_unqueue(bvhead);
>   2941	
>   2942			rcu_lock_acquire(&rcu_callback_map);
>   2943			for (i = 0; i < bvhead->nr_records; i++) {
>   2944				trace_rcu_invoke_kvfree_callback(rcu_state.name,
>   2945					(struct rcu_head *) bvhead->records[i], 0);
> > 2946				vfree(bvhead->records[i]);
>   2947			}
>   2948			rcu_lock_release(&rcu_callback_map);
>   2949	
>   2950			if (cmpxchg(&krcp->bkvcache[1], NULL, bvhead))
>   2951				free_page((unsigned long) bvhead);
>   2952	
>   2953			cond_resched_tasks_rcu_qs();
>   2954		}
>   2955	
>   2956		/*
>   2957		 * This path covers emergency case only due to high
>   2958		 * memory pressure also means low memory condition,
>   2959		 * when we could not allocate a bulk array.
>   2960		 *
>   2961		 * Under that condition an object is queued to the
>   2962		 * list instead.
>   2963		 */
>   2964		for (; head; head = next) {
>   2965			unsigned long offset = (unsigned long)head->func;
>   2966			void *ptr = (void *)head - offset;
>   2967	
>   2968			next = head->next;
>   2969			debug_rcu_head_unqueue((struct rcu_head *)ptr);
>   2970			rcu_lock_acquire(&rcu_callback_map);
>   2971			trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
>   2972	
>   2973			if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
>   2974				kvfree(ptr);
>   2975	
>   2976			rcu_lock_release(&rcu_callback_map);
>   2977			cond_resched_tasks_rcu_qs();
>   2978		}
>   2979	}
>   2980	
> 
> ---
> 0-DAY CI Kernel Test Service, Intel Corporation
> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Matthew Wilcox]

On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> Hello, Joel.
> 
> Sent out the patch fixing build error.

... where?  It didn't get cc'd to linux-mm?

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Matthew Wilcox]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]

On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> Hello, Joel.
> 
> Sent out the patch fixing build error.

... where?  It didn't get cc'd to linux-mm?

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Paul E. McKenney]

On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > Hello, Joel.
> > 
> > Sent out the patch fixing build error.
> 
> ... where?  It didn't get cc'd to linux-mm?

The kbuild test robot complained.  Prior than the build error, the
patch didn't seem all that relevant to linux-mm.  ;-)

							Thanx, Paul

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Paul E. McKenney]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > Hello, Joel.
> > 
> > Sent out the patch fixing build error.
> 
> ... where?  It didn't get cc'd to linux-mm?

The kbuild test robot complained.  Prior than the build error, the
patch didn't seem all that relevant to linux-mm.  ;-)

							Thanx, Paul

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Joel Fernandes]

On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > Hello, Joel.
> > > 
> > > Sent out the patch fixing build error.
> > 
> > ... where?  It didn't get cc'd to linux-mm?
> 
> The kbuild test robot complained.  Prior than the build error, the
> patch didn't seem all that relevant to linux-mm.  ;-)

I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
it because vmalloc.h is included in my tree through the following includes. 

./include/linux/nmi.h
./arch/x86/include/asm/nmi.h
./arch/x86/include/asm/io.h
./include/asm-generic/io.h
./include/linux/vmalloc.h

Such paths may not exist in kbuild robot's tree, so I will apply Vlad's patch to
fix this and push it to my rcu/kfree branch.

thanks,

 - Joel

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Uladzislau Rezki]

On Mon, Mar 30, 2020 at 01:16:06PM -0400, Joel Fernandes wrote:
> On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> > On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > > Hello, Joel.
> > > > 
> > > > Sent out the patch fixing build error.
> > > 
> > > ... where?  It didn't get cc'd to linux-mm?
> > 
> > The kbuild test robot complained.  Prior than the build error, the
> > patch didn't seem all that relevant to linux-mm.  ;-)
> 
> I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
> it because vmalloc.h is included in my tree through the following includes. 
> 
Same to me, i did not manage to hit that build error.

--
Vlad Rezki

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Paul E. McKenney]

On Mon, Mar 30, 2020 at 07:43:38PM +0200, Uladzislau Rezki wrote:
> On Mon, Mar 30, 2020 at 01:16:06PM -0400, Joel Fernandes wrote:
> > On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> > > On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > > > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > > > Hello, Joel.
> > > > > 
> > > > > Sent out the patch fixing build error.
> > > > 
> > > > ... where?  It didn't get cc'd to linux-mm?
> > > 
> > > The kbuild test robot complained.  Prior than the build error, the
> > > patch didn't seem all that relevant to linux-mm.  ;-)
> > 
> > I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
> > it because vmalloc.h is included in my tree through the following includes. 
> > 
> Same to me, i did not manage to hit that build error.

This is a common occurrence for me.  The kbuild test robot can be very
helpful for this sort of thing.  ;-)

							Thanx, Paul

Re: [PATCH 10/18] rcu/tree: Maintain separate array for vmalloc ptrs

[Paul E. McKenney]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

On Mon, Mar 30, 2020 at 07:43:38PM +0200, Uladzislau Rezki wrote:
> On Mon, Mar 30, 2020 at 01:16:06PM -0400, Joel Fernandes wrote:
> > On Mon, Mar 30, 2020 at 08:37:02AM -0700, Paul E. McKenney wrote:
> > > On Mon, Mar 30, 2020 at 08:31:49AM -0700, Matthew Wilcox wrote:
> > > > On Mon, Mar 30, 2020 at 05:29:51PM +0200, Uladzislau Rezki wrote:
> > > > > Hello, Joel.
> > > > > 
> > > > > Sent out the patch fixing build error.
> > > > 
> > > > ... where?  It didn't get cc'd to linux-mm?
> > > 
> > > The kbuild test robot complained.  Prior than the build error, the
> > > patch didn't seem all that relevant to linux-mm.  ;-)
> > 
> > I asked the preprocessor to tell me why I didn't hit this in my tree. Seems
> > it because vmalloc.h is included in my tree through the following includes. 
> > 
> Same to me, i did not manage to hit that build error.

This is a common occurrence for me.  The kbuild test robot can be very
helpful for this sort of thing.  ;-)

							Thanx, Paul

[PATCH 11/18] rcu/tree: Introduce expedited_drain flag

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

It is used and set to true when the bulk array can not
be maintained, it happens under low memory condition
and memory pressure.

In that case the drain work is scheduled right away and
not after KFREE_DRAIN_JIFFIES. It tends to speed up the
reclaim path. On the other hand, there is no data showing
the difference yet.

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8fbc8450284db..3b94526f490cb 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3128,14 +3128,16 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
  * due to memory pressure.
  *
  * Each kvfree_call_rcu() request is added to a batch. The batch will be drained
- * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch will
- * be free'd in workqueue context. This allows us to: batch requests together to
- * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
+ * every KFREE_DRAIN_JIFFIES number of jiffies or can be scheduled right away if
+ * a low memory is detected. All the objects in the batch will be free'd in
+ * workqueue context. This allows us to: batch requests together to reduce the
+ * number of grace periods during heavy kfree_rcu()/kvfree_rcu() load.
  */
 void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
 	unsigned long flags;
 	struct kfree_rcu_cpu *krcp;
+	bool expedited_drain = false;
 	void *ptr;
 
 	local_irq_save(flags);	// For safely calling this_cpu_ptr().
@@ -3161,6 +3163,14 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 		head->func = func;
 		head->next = krcp->head;
 		krcp->head = head;
+
+		/*
+		 * There was an issue to place the pointer directly
+		 * into array, due to memory pressure. Initiate an
+		 * expedited drain to accelerate lazy invocation of
+		 * appropriate free calls.
+		 */
+		expedited_drain = true;
 	}
 
 	WRITE_ONCE(krcp->count, krcp->count + 1);
@@ -3169,7 +3179,9 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
 	    !krcp->monitor_todo) {
 		krcp->monitor_todo = true;
-		schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES);
+
+		schedule_delayed_work(&krcp->monitor_work,
+			expedited_drain ? 0 : KFREE_DRAIN_JIFFIES);
 	}
 
 unlock_return:
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 12/18] rcu/tree: Support reclaim for head-less object

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Update the kvfree_call_rcu() with head-less support, it
means an object without any rcu_head structure can be
reclaimed after GP.

To store pointers there are two chain-arrays maintained
one for SLAB and another one is for vmalloc. Both types
of objects(head-less variant and regular one) are placed
there based on the type.

It can be that maintaining of arrays becomes impossible
due to high memory pressure. For such reason there is an
emergency path. In that case objects with rcu_head inside
are just queued building one way list. Later on that list
is drained.

As for head-less variant. Such objects do not have any
rcu_head helper inside. Thus it is dynamically attached.
As a result an object consists of back-pointer and regular
rcu_head. It implies that emergency path can detect such
object type, therefore they are tagged. So a back-pointer
could be freed as well as dynamically attached wrapper.

Even though such approach requires dynamic memory it needs
only sizeof(unsigned long *) + sizeof(struct rcu_head) bytes,
thus SLAB is used to obtain it. Finally if attaching of the
rcu_head and queuing get failed, the current context has
to follow might_sleep() annotation, thus below steps could
be applied:
   a) wait until a grace period has elapsed;
   b) direct inlining of the kvfree() call.

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 93 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 85 insertions(+), 8 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 3b94526f490cb..204292378101b 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2959,19 +2959,34 @@ static void kfree_rcu_work(struct work_struct *work)
 	 * when we could not allocate a bulk array.
 	 *
 	 * Under that condition an object is queued to the
-	 * list instead.
+	 * list instead. Please note that head-less objects
+	 * have dynamically attached rcu_head, so they also
+	 * contain a back-pointer that has to be freed.
 	 */
 	for (; head; head = next) {
 		unsigned long offset = (unsigned long)head->func;
-		void *ptr = (void *)head - offset;
+		bool headless;
+		void *ptr;
 
 		next = head->next;
+
+		/* We tag the headless object, if so adjust offset. */
+		headless = (((unsigned long) head - offset) & BIT(0));
+		if (headless)
+			offset -= 1;
+
+		ptr = (void *) head - offset;
 		debug_rcu_head_unqueue((struct rcu_head *)ptr);
+
 		rcu_lock_acquire(&rcu_callback_map);
 		trace_rcu_invoke_kvfree_callback(rcu_state.name, head, offset);
 
-		if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset)))
+		if (!WARN_ON_ONCE(!__is_kvfree_rcu_offset(offset))) {
+			if (headless)
+				kvfree((void *) *((unsigned long *) ptr));
+
 			kvfree(ptr);
+		}
 
 		rcu_lock_release(&rcu_callback_map);
 		cond_resched_tasks_rcu_qs();
@@ -3120,6 +3135,24 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
 	return true;
 }
 
+static inline struct rcu_head *attach_rcu_head_to_object(void *obj)
+{
+	unsigned long *ptr;
+
+	ptr = kmalloc(sizeof(unsigned long *) +
+			sizeof(struct rcu_head), GFP_NOWAIT | __GFP_NOWARN);
+
+	if (!ptr)
+		ptr = kmalloc(sizeof(unsigned long *) +
+				sizeof(struct rcu_head), GFP_ATOMIC | __GFP_NOWARN);
+
+	if (!ptr)
+		return NULL;
+
+	ptr[0] = (unsigned long) obj;
+	return ((struct rcu_head *) ++ptr);
+}
+
 /*
  * Queue a request for lazy invocation of appropriate free routine after a
  * grace period. Please note there are three paths are maintained, two are the
@@ -3138,20 +3171,37 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	unsigned long flags;
 	struct kfree_rcu_cpu *krcp;
 	bool expedited_drain = false;
+	bool success;
 	void *ptr;
 
+	if (head) {
+		ptr = (void *) head - (unsigned long) func;
+	} else {
+		/*
+		 * Please note there is a limitation for the head-less
+		 * variant, that is why there is a clear rule for such
+		 * objects:
+		 *
+		 * use it from might_sleep() context only. For other
+		 * places please embed an rcu_head to your structures.
+		 */
+		might_sleep();
+		ptr = (unsigned long *) func;
+	}
+
 	local_irq_save(flags);	// For safely calling this_cpu_ptr().
 	krcp = this_cpu_ptr(&krc);
 	if (krcp->initialized)
 		spin_lock(&krcp->lock);
 
-	ptr = (void *)head - (unsigned long)func;
-
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
 		WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
 			  __func__, head);
+
+		/* Mark as success and leave. */
+		success = true;
 		goto unlock_return;
 	}
 
@@ -3159,7 +3209,22 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	 * Under high memory pressure GFP_NOWAIT can fail,
 	 * in that case the emergency path is maintained.
 	 */
-	if (!kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr)) {
+	success = kvfree_call_rcu_add_ptr_to_bulk(krcp, ptr);
+	if (!success) {
+		/* Is headless object? */
+		if (head == NULL) {
+			head = attach_rcu_head_to_object(ptr);
+			if (head == NULL)
+				goto unlock_return;
+
+			/*
+			 * Tag the headless object. Such objects have a back-pointer
+			 * to the original allocated memory, that has to be freed as
+			 * well as dynamically attached wrapper/head.
+			 */
+			func = (rcu_callback_t) (sizeof(unsigned long *) + 1);
+		}
+
 		head->func = func;
 		head->next = krcp->head;
 		krcp->head = head;
@@ -3171,15 +3236,15 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 		 * appropriate free calls.
 		 */
 		expedited_drain = true;
+		success = true;
 	}
 
 	WRITE_ONCE(krcp->count, krcp->count + 1);
 
 	// Set timer to drain after KFREE_DRAIN_JIFFIES.
 	if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
-	    !krcp->monitor_todo) {
+			!krcp->monitor_todo) {
 		krcp->monitor_todo = true;
-
 		schedule_delayed_work(&krcp->monitor_work,
 			expedited_drain ? 0 : KFREE_DRAIN_JIFFIES);
 	}
@@ -3188,6 +3253,18 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 	if (krcp->initialized)
 		spin_unlock(&krcp->lock);
 	local_irq_restore(flags);
+
+	/*
+	 * High memory pressure, so inline kvfree() after
+	 * synchronize_rcu(). We can do it from might_sleep()
+	 * context only, so the current CPU can pass the QS
+	 * state.
+	 */
+	if (!success) {
+		debug_rcu_head_unqueue(ptr);
+		synchronize_rcu();
+		kvfree(ptr);
+	}
 }
 EXPORT_SYMBOL_GPL(kvfree_call_rcu);
 
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 13/18] rcu/tiny: Move kvfree_call_rcu() out of header

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Move inlined kvfree_call_rcu() function out of the
header file. This step is a preparation for head-lees
support.

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 include/linux/rcutiny.h | 6 +-----
 kernel/rcu/tiny.c       | 6 ++++++
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index 5ba0bcb231976..19a780afa444a 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -34,11 +34,7 @@ static inline void synchronize_rcu_expedited(void)
 	synchronize_rcu();
 }
 
-static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
-{
-	call_rcu(head, func);
-}
-
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func);
 void rcu_qs(void);
 
 static inline void rcu_softirq_qs(void)
diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index aa897c3f2e92c..508c82faa45c3 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -177,6 +177,12 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func)
 }
 EXPORT_SYMBOL_GPL(call_rcu);
 
+void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
+{
+	call_rcu(head, func);
+}
+EXPORT_SYMBOL_GPL(kvfree_call_rcu);
+
 void __init rcu_init(void)
 {
 	open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 14/18] rcu/tiny: Support reclaim for head-less object

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Make a kvfree_call_rcu() function to support head-less
freeing. Same as for tree-RCU, for such purpose we store
pointers in array. SLAB and vmalloc ptrs. are mixed and
coexist together.

Under high memory pressure it can be that maintaining of
arrays becomes impossible. Objects with an rcu_head are
released via call_rcu(). When it comes to the head-less
variant, the kvfree() call is directly inlined, i.e. we
do the same as for tree-RCU:
    a) wait until a grace period has elapsed;
    b) direct inlining of the kvfree() call.

Thus the current context has to follow might_sleep()
annotation. Also please note that for tiny-RCU any
call of synchronize_rcu() is actually a quiescent
state, therefore (a) does nothing.

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tiny.c | 157 +++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 156 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c
index 508c82faa45c3..b1c31a935db93 100644
--- a/kernel/rcu/tiny.c
+++ b/kernel/rcu/tiny.c
@@ -40,6 +40,29 @@ static struct rcu_ctrlblk rcu_ctrlblk = {
 	.curtail	= &rcu_ctrlblk.rcucblist,
 };
 
+/* Can be common with tree-RCU. */
+#define KVFREE_DRAIN_JIFFIES (HZ / 50)
+
+/* Can be common with tree-RCU. */
+struct kvfree_rcu_bulk_data {
+	unsigned long nr_records;
+	struct kvfree_rcu_bulk_data *next;
+	void *records[];
+};
+
+/* Can be common with tree-RCU. */
+#define KVFREE_BULK_MAX_ENTR \
+	((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *))
+
+static struct kvfree_rcu_bulk_data *kvhead;
+static struct kvfree_rcu_bulk_data *kvhead_free;
+static struct kvfree_rcu_bulk_data *kvcache;
+
+static DEFINE_STATIC_KEY_FALSE(rcu_init_done);
+static struct delayed_work monitor_work;
+static struct rcu_work rcu_work;
+static bool monitor_todo;
+
 void rcu_barrier(void)
 {
 	wait_rcu_gp(call_rcu);
@@ -177,9 +200,137 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func)
 }
 EXPORT_SYMBOL_GPL(call_rcu);
 
+static inline bool
+kvfree_call_rcu_add_ptr_to_bulk(void *ptr)
+{
+	struct kvfree_rcu_bulk_data *bnode;
+
+	if (!kvhead || kvhead->nr_records == KVFREE_BULK_MAX_ENTR) {
+		bnode = xchg(&kvcache, NULL);
+		if (!bnode)
+			bnode = (struct kvfree_rcu_bulk_data *)
+				__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
+
+		if (unlikely(!bnode))
+			return false;
+
+		/* Initialize the new block. */
+		bnode->nr_records = 0;
+		bnode->next = kvhead;
+
+		/* Attach it to the bvhead. */
+		kvhead = bnode;
+	}
+
+	/* Done. */
+	kvhead->records[kvhead->nr_records++] = ptr;
+	return true;
+}
+
+static void
+kvfree_rcu_work(struct work_struct *work)
+{
+	struct kvfree_rcu_bulk_data *kvhead_tofree, *next;
+	unsigned long flags;
+	int i;
+
+	local_irq_save(flags);
+	kvhead_tofree = kvhead_free;
+	kvhead_free = NULL;
+	local_irq_restore(flags);
+
+	/* Reclaim process. */
+	for (; kvhead_tofree; kvhead_tofree = next) {
+		next = kvhead_tofree->next;
+
+		for (i = 0; i < kvhead_tofree->nr_records; i++) {
+			debug_rcu_head_unqueue((struct rcu_head *)
+				kvhead_tofree->records[i]);
+			kvfree(kvhead_tofree->records[i]);
+		}
+
+		if (cmpxchg(&kvcache, NULL, kvhead_tofree))
+			free_page((unsigned long) kvhead_tofree);
+	}
+}
+
+static inline bool
+queue_kvfree_rcu_work(void)
+{
+	/* Check if the free channel is available. */
+	if (kvhead_free)
+		return false;
+
+	kvhead_free = kvhead;
+	kvhead = NULL;
+
+	/*
+	 * Queue the job for memory reclaim after GP.
+	 */
+	queue_rcu_work(system_wq, &rcu_work);
+	return true;
+}
+
+static void kvfree_rcu_monitor(struct work_struct *work)
+{
+	unsigned long flags;
+	bool queued;
+
+	local_irq_save(flags);
+	queued = queue_kvfree_rcu_work();
+	if (queued)
+		/* Success. */
+		monitor_todo = false;
+	local_irq_restore(flags);
+
+	/*
+	 * If previous RCU reclaim process is still in progress,
+	 * schedule the work one more time to try again later.
+	 */
+	if (monitor_todo)
+		schedule_delayed_work(&monitor_work,
+			KVFREE_DRAIN_JIFFIES);
+}
+
 void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
-	call_rcu(head, func);
+	unsigned long flags;
+	bool success;
+	void *ptr;
+
+	if (head) {
+		ptr = (void *) head - (unsigned long) func;
+	} else {
+		might_sleep();
+		ptr = (void *) func;
+	}
+
+	if (debug_rcu_head_queue(ptr)) {
+		/* Probable double free, just leak. */
+		WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
+			  __func__, head);
+		return;
+	}
+
+	local_irq_save(flags);
+	success = kvfree_call_rcu_add_ptr_to_bulk(ptr);
+	if (static_branch_likely(&rcu_init_done)) {
+		if (success && !monitor_todo) {
+			monitor_todo = true;
+			schedule_delayed_work(&monitor_work,
+				KVFREE_DRAIN_JIFFIES);
+		}
+	}
+	local_irq_restore(flags);
+
+	if (!success) {
+		if (!head) {
+			synchronize_rcu();
+			kvfree(ptr);
+		} else {
+			call_rcu(head, func);
+		}
+	}
 }
 EXPORT_SYMBOL_GPL(kvfree_call_rcu);
 
@@ -188,4 +339,8 @@ void __init rcu_init(void)
 	open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
 	rcu_early_boot_tests();
 	srcu_init();
+
+	INIT_DELAYED_WORK(&monitor_work, kvfree_rcu_monitor);
+	INIT_RCU_WORK(&rcu_work, kvfree_rcu_work);
+	static_branch_enable(&rcu_init_done);
 }
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 15/18] rcu: Support headless variant in the kvfree_rcu()

[Joel Fernandes (Google)]

From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>

Make it possible to pass one or two arguments to the
kvfree_rcu() macro what corresponds to either headless
case or not, so it becomes a bit versatile.

As a result we obtain two ways of using that macro,
below are two examples:

a) kvfree_rcu(ptr, rhf);
    struct X {
        struct rcu_head rhf;
        unsigned char data[100];
    };

    void *ptr = kvmalloc(sizeof(struct X), GFP_KERNEL);
    if (ptr)
        kvfree_rcu(ptr, rhf);

b) kvfree_rcu(ptr);
    void *ptr = kvmalloc(some_bytes, GFP_KERNEL);
    if (ptr)
        kvfree_rcu(ptr);

Last one, we name it headless variant, only needs one
argument, means it does not require any rcu_head to be
present within the type of ptr.

There is a restriction the (b) context has to fall into
might_sleep() annotation. To check that, please activate
the CONFIG_DEBUG_ATOMIC_SLEEP option in your kernel.

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 include/linux/rcupdate.h | 38 ++++++++++++++++++++++++++++++++++----
 1 file changed, 34 insertions(+), 4 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index edb6eeba49f83..7d04bbeeeef14 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -872,12 +872,42 @@ do {									\
 
 /**
  * kvfree_rcu() - kvfree an object after a grace period.
- * @ptr:	pointer to kvfree
- * @rhf:	the name of the struct rcu_head within the type of @ptr.
  *
- * Same as kfree_rcu(), just simple alias.
+ * This macro consists of one or two arguments and it is
+ * based on whether an object is head-less or not. If it
+ * has a head then a semantic stays the same as it used
+ * to be before:
+ *
+ *     kvfree_rcu(ptr, rhf);
+ *
+ * where @ptr is a pointer to kvfree(), @rhf is the name
+ * of the rcu_head structure within the type of @ptr.
+ *
+ * When it comes to head-less variant, only one argument
+ * is passed and that is just a pointer which has to be
+ * freed after a grace period. Therefore the semantic is
+ *
+ *     kvfree_rcu(ptr);
+ *
+ * where @ptr is a pointer to kvfree().
+ *
+ * Please note, head-less way of freeing is permitted to
+ * use from a context that has to follow might_sleep()
+ * annotation. Otherwise, please switch and embed the
+ * rcu_head structure within the type of @ptr.
  */
-#define kvfree_rcu(ptr, rhf) kfree_rcu(ptr, rhf)
+#define kvfree_rcu(...) KVFREE_GET_MACRO(__VA_ARGS__,		\
+	kvfree_rcu_arg_2, kvfree_rcu_arg_1)(__VA_ARGS__)
+
+#define KVFREE_GET_MACRO(_1, _2, NAME, ...) NAME
+#define kvfree_rcu_arg_2(ptr, rhf) kfree_rcu(ptr, rhf)
+#define kvfree_rcu_arg_1(ptr)					\
+do {								\
+	typeof(ptr) ___p = (ptr);				\
+								\
+	if (___p)						\
+		kvfree_call_rcu(NULL, (rcu_callback_t) (___p));	\
+} while (0)
 
 /*
  * Place this after a lock-acquisition primitive to guarantee that
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 16/18] rcu/tree: Remove extra next variable in kfree worker function

[Joel Fernandes (Google)]

No change in code, small refactor.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 204292378101b..56c9e102a901d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2889,8 +2889,7 @@ debug_rcu_bhead_unqueue(struct kvfree_rcu_bulk_data *bhead)
 static void kfree_rcu_work(struct work_struct *work)
 {
 	unsigned long flags;
-	struct kvfree_rcu_bulk_data *bkhead, *bknext;
-	struct kvfree_rcu_bulk_data *bvhead, *bvnext;
+	struct kvfree_rcu_bulk_data *bkhead, *bvhead, *bnext;
 	struct rcu_head *head, *next;
 	struct kfree_rcu_cpu *krcp;
 	struct kfree_rcu_cpu_work *krwp;
@@ -2915,8 +2914,8 @@ static void kfree_rcu_work(struct work_struct *work)
 	spin_unlock_irqrestore(&krcp->lock, flags);
 
 	/* kmalloc()/kfree() channel. */
-	for (; bkhead; bkhead = bknext) {
-		bknext = bkhead->next;
+	for (; bkhead; bkhead = bnext) {
+		bnext = bkhead->next;
 
 		debug_rcu_bhead_unqueue(bkhead);
 
@@ -2934,8 +2933,8 @@ static void kfree_rcu_work(struct work_struct *work)
 	}
 
 	/* vmalloc()/vfree() channel. */
-	for (; bvhead; bvhead = bvnext) {
-		bvnext = bvhead->next;
+	for (; bvhead; bvhead = bnext) {
+		bnext = bvhead->next;
 
 		debug_rcu_bhead_unqueue(bvhead);
 
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 17/18] rcu/tree: Simplify is_vmalloc_addr expression

[Joel Fernandes (Google)]

No code change, small refactor.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 56c9e102a901d..311d216c7faa7 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3104,7 +3104,7 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)
 		return false;
 
 	lockdep_assert_held(&krcp->lock);
-	idx = !is_vmalloc_addr(ptr) ? 0:1;
+	idx = !!is_vmalloc_addr(ptr);
 
 	/* Check if a new block is required. */
 	if (!krcp->bkvhead[idx] ||
-- 
2.26.0.rc2.310.g2932bb562d-goog

[PATCH 18/18] rcu/tree: Make kvfree_rcu() tolerate any alignment

[Joel Fernandes (Google)]

Handle cases where the the object being kvfree_rcu()'d is not aligned by
2-byte boundaries.

Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
---
 kernel/rcu/tree.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 311d216c7faa7..d6536374d12a9 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2827,6 +2827,9 @@ struct kvfree_rcu_bulk_data {
 #define KVFREE_BULK_MAX_ENTR \
 	((PAGE_SIZE - sizeof(struct kvfree_rcu_bulk_data)) / sizeof(void *))
 
+/* Encoding the offset of a fake rcu_head to indicate the head is a wrapper. */
+#define RCU_HEADLESS_KFREE BIT(31)
+
 /**
  * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
  * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
@@ -2970,9 +2973,9 @@ static void kfree_rcu_work(struct work_struct *work)
 		next = head->next;
 
 		/* We tag the headless object, if so adjust offset. */
-		headless = (((unsigned long) head - offset) & BIT(0));
+		headless = !!(offset & RCU_HEADLESS_KFREE);
 		if (headless)
-			offset -= 1;
+			offset &= ~(RCU_HEADLESS_KFREE);
 
 		ptr = (void *) head - offset;
 		debug_rcu_head_unqueue((struct rcu_head *)ptr);
@@ -3221,7 +3224,7 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
 			 * to the original allocated memory, that has to be freed as
 			 * well as dynamically attached wrapper/head.
 			 */
-			func = (rcu_callback_t) (sizeof(unsigned long *) + 1);
+			func = (rcu_callback_t)(sizeof(unsigned long *) | RCU_HEADLESS_KFREE);
 		}
 
 		head->func = func;
-- 
2.26.0.rc2.310.g2932bb562d-goog