[PATCH v1 0/4] ring-buffer: Fix remote reader page swap and memory ordering

[PATCH v1 0/4] ring-buffer: Fix remote reader page swap and memory ordering

[Vincent Donnefort]

Bartłomiej reported a remote tracing issue on Android kernel branches.
Although that code differs from mainline, investigating the report
showed that upstream has the same underlying shortfalls:

1. Potential corruption of a simple ring buffer because it does not
   prevent swapping the reader page with the head when the reader page
   is already the tail page (nor order the page::commit reset before
   updating tail_page).

2. Potential failure of rb_advance_reader() because
   __rb_get_reader_page_from_remote() checks rb_num_of_entries() before
   checking the reader page commit via rb_page_size(), and lacks memory
   barriers ordering page::commit and meta::entries.

While at it, order page::commit with page::data and page::time_stamp
(patch 3) and add the missing pages_read increment on remote reader page
swaps (patch 4).

Vincent Donnefort (4):
  ring-buffer: Skip remote reader page swap when on tail
  ring-buffer: Check remote reader page commit before meta entries
  ring-buffer: Order remote page commit with data and timestamp
  ring-buffer: Increment pages_read on remote ring buffer page swap

 kernel/trace/ring_buffer.c        | 46 +++++++++++++++++++++----------
 kernel/trace/simple_ring_buffer.c | 26 ++++++++++++++++-
 2 files changed, 57 insertions(+), 15 deletions(-)


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.56.0.611.g1030f0b940-goog

[PATCH v1 1/4] ring-buffer: Skip remote reader page swap when on tail

[Vincent Donnefort]

When the reader page is also the tail, swap should not be permitted or
it would insert the tail behind the head, breaking the integrity of the
ring buffer. While __rb_get_reader_page() does correctly prevent this
situation, the simple_ring_buffer implementation misses it.

Once the reader catches up with the tail, make
simple_ring_buffer_swap_reader_page() return 0. The remote reader now
can expect an unchanged reader. When it is the case, there's nothing
more to read.

Also, when the simple_ring_buffer writer moves the tail to a new page,
it resets the new tail counters and more specifically page::commit. As
moving the tail unblocks page swap, order the page reset before the
tail_page update. This ensures the reader can't swap the reader page
with the new tail and observe a stale commit value.

Cc: stable@vger.kernel.org
Fixes: 2e67fabd8b77 ("ring-buffer: Introduce ring-buffer remotes")
Fixes: 34e5b958bdad ("tracing: Introduce simple_ring_buffer")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
---
 kernel/trace/ring_buffer.c        |  3 ++-
 kernel/trace/simple_ring_buffer.c | 15 ++++++++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 1fd28a72e6ee..3c7aa982d43e 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -5848,7 +5848,8 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
 
 	new_reader = cpu_buffer->subbuf_ids[cpu_buffer->meta_page->reader.id];
 
-	WARN_ON_ONCE(prev_reader == new_reader);
+	if (prev_reader == new_reader)
+		return NULL;
 
 	prev_head = new_reader;  /* New reader was also the previous head */
 	new_head = prev_head;
diff --git a/kernel/trace/simple_ring_buffer.c b/kernel/trace/simple_ring_buffer.c
index abfaf9bee12e..d5c84af01925 100644
--- a/kernel/trace/simple_ring_buffer.c
+++ b/kernel/trace/simple_ring_buffer.c
@@ -139,6 +139,13 @@ int simple_ring_buffer_swap_reader_page(struct simple_rb_per_cpu *cpu_buffer)
 
 	reader = cpu_buffer->reader_page;
 
+	/*
+	 * The reader is already the tail page. See simple_rb_move_tail() for
+	 * memory ordering.
+	 */
+	if (reader == smp_load_acquire(&cpu_buffer->tail_page))
+		return 0;
+
 	do {
 		/* Run after the writer to find the head */
 		ret = simple_rb_find_head(cpu_buffer);
@@ -196,7 +203,13 @@ static struct simple_buffer_page *simple_rb_move_tail(struct simple_rb_per_cpu *
 	}
 
 	simple_bpage_reset(new_tail);
-	cpu_buffer->tail_page = new_tail;
+
+	/*
+	 * Setting a new tail may unblock the swap in
+	 * simple_ring_buffer_swap_reader_page(). Ensure the reader observes the
+	 * page::commit reset.
+	 */
+	smp_store_release(&cpu_buffer->tail_page, new_tail);
 
 	simple_rb_meta_inc(cpu_buffer->meta->pages_touched, 1);
 
-- 
2.56.0.611.g1030f0b940-goog

[PATCH v1 2/4] ring-buffer: Check remote reader page commit before meta entries

[Vincent Donnefort]

__rb_get_reader_page_from_remote() bails out when rb_num_of_entries() is
zero before verifying if there is more data committed on the current
reader_page (rb_page_size()), while simple_ring_buffer updates
page::commit before meta::entries. As a consequence, a race between the
writer and reader could lead to __rb_get_reader_page_from_remote()
returning NULL, while more data is available for reading on the ring
buffer.

  CPU0 (writer)                       CPU1 (reader)
  -------------                       -------------
  local_set(&page->commit, write);
                                      __rb_get_reader_page_from_remote()
                                        !rb_read_remote_meta_page()
                                          return NULL;
  simple_rb_meta_inc(meta->entries, 1);

This is an issue for rb_advance_reader() which always expects
rb_get_reader_page() to return a page.

Fix this issue by swapping the checks in
__rb_get_reader_page_from_remote(): first it verifies if there is more
to read in the current reader page, then it verifies if there's anything
at all in the ring buffer.

Also, if the reader observes the meta::entries update before
page::commit while the writer moves tail_page, it could swap reader_page
out and drop unread events. Order page::commit before meta::entries on
the writer and rb_read_remote_meta_page() before rb_page_size() on the
reader size.

Cc: stable@vger.kernel.org
Fixes: 2e67fabd8b77 ("ring-buffer: Introduce ring-buffer remotes")
Fixes: 34e5b958bdad ("tracing: Introduce simple_ring_buffer")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
---
 kernel/trace/ring_buffer.c        | 19 +++++++++++++------
 kernel/trace/simple_ring_buffer.c |  6 ++++++
 2 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 3c7aa982d43e..f70010e2ede7 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -5615,14 +5615,12 @@ unsigned long ring_buffer_overruns(struct trace_buffer *buffer)
 }
 EXPORT_SYMBOL_GPL(ring_buffer_overruns);
 
-static bool rb_read_remote_meta_page(struct ring_buffer_per_cpu *cpu_buffer)
+static void rb_read_remote_meta_page(struct ring_buffer_per_cpu *cpu_buffer)
 {
 	local_set(&cpu_buffer->entries, READ_ONCE(cpu_buffer->meta_page->entries));
 	local_set(&cpu_buffer->overrun, READ_ONCE(cpu_buffer->meta_page->overrun));
 	local_set(&cpu_buffer->pages_touched, READ_ONCE(cpu_buffer->meta_page->pages_touched));
 	local_set(&cpu_buffer->pages_lost, READ_ONCE(cpu_buffer->meta_page->pages_lost));
-
-	return rb_num_of_entries(cpu_buffer);
 }
 
 static void rb_update_remote_head(struct ring_buffer_per_cpu *cpu_buffer)
@@ -5825,8 +5823,13 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
 {
 	struct buffer_page *new_reader, *prev_reader, *prev_head, *new_head, *last;
 
-	if (!rb_read_remote_meta_page(cpu_buffer))
-		return NULL;
+	rb_read_remote_meta_page(cpu_buffer);
+
+	/*
+	 * Order meta::entries read before page::commit, most likely paired with
+	 * simple_ring_buffer_commit().
+	 */
+	smp_rmb();
 
 	/* More to read on the reader page */
 	if (cpu_buffer->reader_page->read < rb_page_size(cpu_buffer->reader_page)) {
@@ -5835,6 +5838,9 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
 		return cpu_buffer->reader_page;
 	}
 
+	if (!rb_num_of_entries(cpu_buffer))
+		return NULL;
+
 	prev_reader = cpu_buffer->subbuf_ids[cpu_buffer->meta_page->reader.id];
 
 	if (cpu_buffer->remote->swap_reader_page(cpu_buffer->cpu,
@@ -6909,7 +6915,8 @@ int ring_buffer_poll_remote(struct trace_buffer *buffer, int cpu)
 		cpu_buffer = buffer->buffers[cpu];
 
 		guard(raw_spinlock)(&cpu_buffer->reader_lock);
-		if (rb_read_remote_meta_page(cpu_buffer))
+		rb_read_remote_meta_page(cpu_buffer);
+		if (rb_num_of_entries(cpu_buffer))
 			rb_wakeups(buffer, cpu_buffer);
 
 		return 0;
diff --git a/kernel/trace/simple_ring_buffer.c b/kernel/trace/simple_ring_buffer.c
index d5c84af01925..5f14ba152fba 100644
--- a/kernel/trace/simple_ring_buffer.c
+++ b/kernel/trace/simple_ring_buffer.c
@@ -309,6 +309,12 @@ void simple_ring_buffer_commit(struct simple_rb_per_cpu *cpu_buffer)
 {
 	local_set(&cpu_buffer->tail_page->page->commit,
 		  cpu_buffer->tail_page->write);
+
+	/*
+	 * Order page::commit before meta::entries. Paired with
+	 * __rb_get_reader_page_from_remote().
+	 */
+	smp_wmb();
 	simple_rb_meta_inc(cpu_buffer->meta->entries, 1);
 
 	/*
-- 
2.56.0.611.g1030f0b940-goog

Re: [PATCH v1 2/4] ring-buffer: Check remote reader page commit before meta entries

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] ring-buffer: rb_num_of_entries() underflow when commit
  precedes meta entries
--

commit 1d2d0e6ae01d518c30dc9986893db113c38a4caf
Author: Vincent Donnefort <vdonnefort@google.com>

ring-buffer: Check remote reader page commit before meta entries

This commit reorders the checks in __rb_get_reader_page_from_remote() to
inspect the current reader page commit offset before checking if
rb_num_of_entries() is zero, and adds memory barriers pairing page::commit
and meta::entries between simple_ring_buffer_commit() and the remote reader.

> diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
> index 3c7aa982d43e9..f70010e2ede77 100644
> --- a/kernel/trace/ring_buffer.c
> +++ b/kernel/trace/ring_buffer.c

[ ... ]

> @@ -5825,8 +5823,13 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
>  {
>  	struct buffer_page *new_reader, *prev_reader, *prev_head, *new_head, *last;
>  
> -	if (!rb_read_remote_meta_page(cpu_buffer))
> -		return NULL;
> +	rb_read_remote_meta_page(cpu_buffer);
> +
> +	/*
> +	 * Order meta::entries read before page::commit, most likely paired with
> +	 * simple_ring_buffer_commit().
> +	 */
> +	smp_rmb();
>  
>  	/* More to read on the reader page */
>  	if (cpu_buffer->reader_page->read < rb_page_size(cpu_buffer->reader_page)) {
> @@ -5835,6 +5838,9 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
>  		return cpu_buffer->reader_page;
>  	}
>  
> +	if (!rb_num_of_entries(cpu_buffer))
> +		return NULL;

[Severity: Medium]
Can rb_num_of_entries() underflow to ULONG_MAX when an event is consumed in
the window between the page->commit update and the meta->entries increment?

CPU0 (writer)                       CPU1 (reader)
-------------                       -------------
simple_ring_buffer_commit()
  local_set(&page->commit, write);
                                    ring_buffer_consume()
                                      ...
                                      __rb_get_reader_page_from_remote()
                                        rb_read_remote_meta_page()
                                          /* caches entries = N */
                                        reader_page->read < rb_page_size()
                                          return reader_page;
                                      rb_advance_reader()
                                        cpu_buffer->read++; /* N + 1 */
  simple_rb_meta_inc(meta->entries, 1);

At this point cpu_buffer->read is N + 1 while cpu_buffer->entries is still N,
so rb_num_of_entries() evaluates N - (N + 1) = ULONG_MAX:

kernel/trace/ring_buffer.c:rb_num_of_entries() {
	return local_read(&cpu_buffer->entries) -
		(local_read(&cpu_buffer->overrun) + cpu_buffer->read);
}

Until rb_read_remote_meta_page() runs again, ring_buffer_entries() returns
ULONG_MAX and ring_buffer_empty() returns false on an empty buffer (which
also prevents trace_remote_try_unload() from unloading an idle buffer).  If
__rb_get_reader_page_from_remote() is called again before CPU0 increments
meta->entries, !rb_num_of_entries() evaluates to false and it proceeds to
swap_reader_page() on an empty buffer.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113152729.837429-1-vdonnefort@google.com?part=2

[PATCH v1 3/4] ring-buffer: Order remote page commit with data and timestamp

[Vincent Donnefort]

__rb_get_reader_page_from_remote() uses page::commit to check if more
data is available for consumption. However, it needs to make sure that
data and page::time_stamp are not stale. Enforce page::commit order with
page::data and page::time_stamp, and defer updating read_stamp on page
swap until rb_page_size() confirms committed data is present.

Cc: stable@vger.kernel.org
Fixes: 2e67fabd8b77 ("ring-buffer: Introduce ring-buffer remotes")
Fixes: 34e5b958bdad ("tracing: Introduce simple_ring_buffer")
Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
---
 kernel/trace/ring_buffer.c        | 23 ++++++++++++++++-------
 kernel/trace/simple_ring_buffer.c |  5 +++++
 2 files changed, 21 insertions(+), 7 deletions(-)

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index f70010e2ede7..6016d16d1566 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -5832,11 +5832,8 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
 	smp_rmb();
 
 	/* More to read on the reader page */
-	if (cpu_buffer->reader_page->read < rb_page_size(cpu_buffer->reader_page)) {
-		if (!cpu_buffer->reader_page->read)
-			cpu_buffer->read_stamp = cpu_buffer->reader_page->page->time_stamp;
-		return cpu_buffer->reader_page;
-	}
+	if (cpu_buffer->reader_page->read < rb_page_size(cpu_buffer->reader_page))
+		goto out;
 
 	if (!rb_num_of_entries(cpu_buffer))
 		return NULL;
@@ -5883,10 +5880,22 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
 	cpu_buffer->reader_page = new_reader;
 	cpu_buffer->reader_page->read = 0;
 	cpu_buffer->pages = &new_head->list;
-	cpu_buffer->read_stamp = new_reader->page->time_stamp;
 	cpu_buffer->lost_events = cpu_buffer->meta_page->reader.lost_events;
 
-	return rb_page_size(cpu_buffer->reader_page) ? cpu_buffer->reader_page : NULL;
+	if (!rb_page_size(cpu_buffer->reader_page))
+		return NULL;
+
+out:
+	/*
+	 * Order page::commit read before page::data and page::time_stamp, most
+	 * likely paired with simple_ring_buffer_commit().
+	 */
+	smp_rmb();
+
+	if (!cpu_buffer->reader_page->read)
+		cpu_buffer->read_stamp = cpu_buffer->reader_page->page->time_stamp;
+
+	return cpu_buffer->reader_page;
 }
 
 static struct buffer_page *
diff --git a/kernel/trace/simple_ring_buffer.c b/kernel/trace/simple_ring_buffer.c
index 5f14ba152fba..2dddaef638dc 100644
--- a/kernel/trace/simple_ring_buffer.c
+++ b/kernel/trace/simple_ring_buffer.c
@@ -307,6 +307,11 @@ EXPORT_SYMBOL_GPL(simple_ring_buffer_reserve);
  */
 void simple_ring_buffer_commit(struct simple_rb_per_cpu *cpu_buffer)
 {
+	/*
+	 * Order page::data and page::time_stamp before page::commit. Paired
+	 * with __rb_get_reader_page_from_remote().
+	 */
+	smp_wmb();
 	local_set(&cpu_buffer->tail_page->page->commit,
 		  cpu_buffer->tail_page->write);
 
-- 
2.56.0.611.g1030f0b940-goog

Re: [PATCH v1 3/4] ring-buffer: Order remote page commit with data and timestamp

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] tracing: out-of-bounds write in simple_rb_reserve_next()
  [1]
--

commit b0c88540d75d93c7a85149df8d188a1bfaa9a89e
Author: Vincent Donnefort <vdonnefort@google.com>

ring-buffer: Order remote page commit with data and timestamp

This commit adds memory barriers between page::commit, page::data, and
page::time_stamp for remote ring-buffers and defers updating read_stamp
until committed data is present on the reader page.

> diff --git a/kernel/trace/simple_ring_buffer.c b/kernel/trace/simple_ring_buffer.c
> index 5f14ba152fba6..2dddaef638dc6 100644
> --- a/kernel/trace/simple_ring_buffer.c
> +++ b/kernel/trace/simple_ring_buffer.c
> @@ -307,6 +307,11 @@ EXPORT_SYMBOL_GPL(simple_ring_buffer_reserve);
>   */
>  void simple_ring_buffer_commit(struct simple_rb_per_cpu *cpu_buffer)
>  {
> +	/*
> +	 * Order page::data and page::time_stamp before page::commit. Paired
> +	 * with __rb_get_reader_page_from_remote().
> +	 */
> +	smp_wmb();
>  	local_set(&cpu_buffer->tail_page->page->commit,
>  		  cpu_buffer->tail_page->write);

[Severity: Critical]
This problem wasn't introduced by this patch, but can
simple_ring_buffer_reserve() return a pointer whose reservation overflows
tail->page->data [1]?

In simple_rb_reserve_next(), when an event exceeds PAGE_SIZE -
BUF_PAGE_HDR_SIZE, simple_rb_move_tail() rotates to an empty page where
!tail->entries is true, and event_size is assigned to tail->write without
checking that it fits on a fresh page:

kernel/trace/simple_ring_buffer.c:simple_rb_reserve_next() {
	...
	if (unlikely(write > (PAGE_SIZE - BUF_PAGE_HDR_SIZE)))
		tail = simple_rb_move_tail(cpu_buffer);

	if (!tail->entries) {
		tail->page->time_stamp = timestamp;
		time_delta = 0;
		ts_ext_size = 0;
		write = event_size;
		prev_write = 0;
	}

	tail->write = write;
	...
}

Writing the caller's payload to the returned event pointer then writes past
the end of the page buffer and corrupts adjacent memory.

[1] https://sashiko.dev/#/bug/linux-b2128783-1787-4f13-b863-090db5378e74

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113152729.837429-1-vdonnefort@google.com?part=3

[PATCH v1 4/4] ring-buffer: Increment pages_read on remote ring buffer page swap

[Vincent Donnefort]

__rb_get_reader_page_from_remote() swaps in a new reader page without
incrementing cpu_buffer->pages_read. While this isn't an issue now as
trace remotes do not use buffer_percent, add this missing increment for
consistency.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
---
 kernel/trace/ring_buffer.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 6016d16d1566..ae33e64c7daa 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -5881,6 +5881,7 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
 	cpu_buffer->reader_page->read = 0;
 	cpu_buffer->pages = &new_head->list;
 	cpu_buffer->lost_events = cpu_buffer->meta_page->reader.lost_events;
+	local_inc(&cpu_buffer->pages_read);
 
 	if (!rb_page_size(cpu_buffer->reader_page))
 		return NULL;
-- 
2.56.0.611.g1030f0b940-goog