[PATCH bpf] bpf: fix a memory leak in the LRU and LRU_PERCPU hash maps

[PATCH bpf] bpf: fix a memory leak in the LRU and LRU_PERCPU hash maps

[Anton Protopopov]

The LRU and LRU_PERCPU maps allocate a new element on update before locking the
target hash table bucket. Right after that the maps try to lock the bucket.
If this fails, then maps return -EBUSY to the caller without releasing the
allocated element. This makes the element untracked: it doesn't belong to
either of free lists, and it doesn't belong to the hash table, so can't be
re-used; this eventually leads to the permanent -ENOMEM on LRU map updates,
which is unexpected. Fix this by returning the element to the local free list
if bucket locking fails.

Signed-off-by: Anton Protopopov <aspsk@isovalent.com>
---
 kernel/bpf/hashtab.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 00c253b84bf5..9901efee4339 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -1215,7 +1215,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value

 	ret = htab_lock_bucket(htab, b, hash, &flags);
 	if (ret)
-		return ret;
+		goto err_lock_bucket;
 
 	l_old = lookup_elem_raw(head, hash, key, key_size);
 
@@ -1236,6 +1236,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
 err:
 	htab_unlock_bucket(htab, b, hash, flags);
 
+err_lock_bucket:
 	if (ret)
 		htab_lru_push_free(htab, l_new);
 	else if (l_old)
@@ -1338,7 +1339,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 
 	ret = htab_lock_bucket(htab, b, hash, &flags);
 	if (ret)
-		return ret;
+		goto err_lock_bucket;
 
 	l_old = lookup_elem_raw(head, hash, key, key_size);
 
@@ -1361,6 +1362,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
 	ret = 0;
 err:
 	htab_unlock_bucket(htab, b, hash, flags);
+err_lock_bucket:
 	if (l_new)
 		bpf_lru_push_free(&htab->lru, &l_new->lru_node);
 	return ret;
-- 
2.34.1

Re: [PATCH bpf] bpf: fix a memory leak in the LRU and LRU_PERCPU hash maps

[Anton Protopopov]

On Mon, May 22, 2023 at 03:45:58PM +0000, Anton Protopopov wrote:
> The LRU and LRU_PERCPU maps allocate a new element on update before locking the
> target hash table bucket. Right after that the maps try to lock the bucket.
> If this fails, then maps return -EBUSY to the caller without releasing the
> allocated element. This makes the element untracked: it doesn't belong to
> either of free lists, and it doesn't belong to the hash table, so can't be
> re-used; this eventually leads to the permanent -ENOMEM on LRU map updates,
> which is unexpected. Fix this by returning the element to the local free list
> if bucket locking fails.

Fixes: 20b6cc34ea74 ("bpf: Avoid hashtab deadlock with map_locked")

> Signed-off-by: Anton Protopopov <aspsk@isovalent.com>
> ---
>  kernel/bpf/hashtab.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
> index 00c253b84bf5..9901efee4339 100644
> --- a/kernel/bpf/hashtab.c
> +++ b/kernel/bpf/hashtab.c
> @@ -1215,7 +1215,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
> 
>  	ret = htab_lock_bucket(htab, b, hash, &flags);
>  	if (ret)
> -		return ret;
> +		goto err_lock_bucket;
>  
>  	l_old = lookup_elem_raw(head, hash, key, key_size);
>  
> @@ -1236,6 +1236,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value
>  err:
>  	htab_unlock_bucket(htab, b, hash, flags);
>  
> +err_lock_bucket:
>  	if (ret)
>  		htab_lru_push_free(htab, l_new);
>  	else if (l_old)
> @@ -1338,7 +1339,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
>  
>  	ret = htab_lock_bucket(htab, b, hash, &flags);
>  	if (ret)
> -		return ret;
> +		goto err_lock_bucket;
>  
>  	l_old = lookup_elem_raw(head, hash, key, key_size);
>  
> @@ -1361,6 +1362,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key,
>  	ret = 0;
>  err:
>  	htab_unlock_bucket(htab, b, hash, flags);
> +err_lock_bucket:
>  	if (l_new)
>  		bpf_lru_push_free(&htab->lru, &l_new->lru_node);
>  	return ret;
> -- 
> 2.34.1
> 

Re: [PATCH bpf] bpf: fix a memory leak in the LRU and LRU_PERCPU hash maps

[Martin KaFai Lau]

On 5/22/23 9:12 AM, Anton Protopopov wrote:
> On Mon, May 22, 2023 at 03:45:58PM +0000, Anton Protopopov wrote:
>> The LRU and LRU_PERCPU maps allocate a new element on update before locking the
>> target hash table bucket. Right after that the maps try to lock the bucket.
>> If this fails, then maps return -EBUSY to the caller without releasing the
>> allocated element. This makes the element untracked: it doesn't belong to
>> either of free lists, and it doesn't belong to the hash table, so can't be
>> re-used; this eventually leads to the permanent -ENOMEM on LRU map updates,
>> which is unexpected.

Ouch. This is very bad. :(

Excellent catch. Applied.

I am thinking if a test could be written but it does not seem like anything 
after htab_lock_bucket() is traceable. Not sure if Song may have good idea?

>> Fix this by returning the element to the local free list if bucket locking fails.
> Fixes: 20b6cc34ea74 ("bpf: Avoid hashtab deadlock with map_locked")
>