* [PATCH v2 bpf-next] bpf: use the proper optlen when doing strncpy in bpf_getsockopt()
From: Yafang Shao @ 2019-01-24 5:16 UTC (permalink / raw)
To: kafai, brakmo, ast, daniel; +Cc: netdev, shaoyafang, Yafang Shao
As strncpy(..., TCP_CA_NAME_MAX-1) is used in bpf_setsockopt(),
this change will make it more consistent with the bpf_setsockopt() above.
Acked-by: Martin KaFai Lau <kafai@fb.com>
Acked-by: Lawrence Brakmo <brakmo@fb.com>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
net/core/filter.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/core/filter.c b/net/core/filter.c
index 447dd1b..7a4de22 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -4262,7 +4262,7 @@ static unsigned long bpf_xdp_copy(void *dst_buff, const void *src_buff,
if (!icsk->icsk_ca_ops || optlen <= 1)
goto err_clear;
- strncpy(optval, icsk->icsk_ca_ops->name, optlen);
+ strncpy(optval, icsk->icsk_ca_ops->name, optlen - 1);
optval[optlen - 1] = 0;
break;
case TCP_SAVED_SYN:
--
1.8.3.1
^ permalink raw reply related
* Re: [PATCH] net: altera_tse: fix msgdma_tx_completion on non-zero fill_level case
From: Atsushi Nemoto @ 2019-01-24 5:05 UTC (permalink / raw)
To: dalon.westergreen; +Cc: thor.thayer, netdev, tomonori.sakita
In-Reply-To: <49028e9cdf59afe3d4184b9eee36f0d2406c8bdb.camel@linux.intel.com>
On Wed, 23 Jan 2019 14:47:06 -0800, Dalon L Westergreen <dalon.westergreen@linux.intel.com> wrote:
>> > if (inuse) { /* Tx FIFO is not empty */
>> > - ready = priv->tx_prod - priv->tx_cons - inuse - 1;
>> > + ready = priv->tx_prod - priv->tx_cons - inuse;
> dont think my last email went through..
>
> I am not sure about this. This register indicates the number of entries
> still to be processed by the dma. the -1 is intended to represent the
> decriptor currently being processed. If ready is
> priv->tx_prod - priv->tx_cons - inuse couldn't you end up processing 1
> too many packets? IE: ready is 1 greater then the actual completed
> packets?
>
> I do agree that we should not be returning a negative value, but i dont
> think i agree removing the -1 is the answer. perhaps just check that ready
> is greater than 0?
Thank you for review.
I agree with you. It would be OK returning a possibly off-by-one
value unless it is not an negative value.
Then, how about this instead?
--- a/drivers/net/ethernet/altera/altera_msgdma.c
+++ b/drivers/net/ethernet/altera/altera_msgdma.c
@@ -145,7 +145,8 @@ u32 msgdma_tx_completions(struct altera_tse_private *priv)
& 0xffff;
if (inuse) { /* Tx FIFO is not empty */
- ready = priv->tx_prod - priv->tx_cons - inuse - 1;
+ ready = max_t(int,
+ priv->tx_prod - priv->tx_cons - inuse - 1, 0);
} else {
/* Check for buffered last packet */
status = csrrd32(priv->tx_dma_csr, msgdma_csroffs(status));
--
If it was OK, I will post an updated patch.
---
Atsushi Nemoto
^ permalink raw reply
* Re: [PATCH bpf-next v2 13/16] ppc: bpf: implement jitting of JMP32
From: Sandipan Das @ 2019-01-24 4:58 UTC (permalink / raw)
To: Jiong Wang; +Cc: ast, daniel, netdev, oss-drivers, Naveen N . Rao
In-Reply-To: <1548076553-31268-14-git-send-email-jiong.wang@netronome.com>
Hi Jiong,
On 21/01/19 6:45 PM, Jiong Wang wrote:
> This patch implements code-gen for new JMP32 instructions on ppc.
>
> For JMP32 | JSET, instruction encoding for PPC_RLWINM_DOT is added to check
> the result of ANDing low 32-bit of operands.
>
> Cc: Naveen N. Rao <naveen.n.rao@linux.ibm.com>
> Cc: Sandipan Das <sandipan@linux.ibm.com>
> Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
> ---
> arch/powerpc/include/asm/ppc-opcode.h | 1 +
> arch/powerpc/net/bpf_jit.h | 4 ++
> arch/powerpc/net/bpf_jit_comp64.c | 98 ++++++++++++++++++++++++++++++-----
> 3 files changed, 89 insertions(+), 14 deletions(-)
> [...]
I ran the verifier selftests on a ppc64 test system and found that the jmp32 tests
were failing because the instructions were not being decoded by the JIT compiler.
The codegen logic looks good to me and the tests passed after I fixed the decoding
logic. Here are the changes that I had to make:
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index 2e40c2b251ba..15bba765fa79 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -768,36 +768,58 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
case BPF_JMP | BPF_JGT | BPF_X:
case BPF_JMP | BPF_JSGT | BPF_K:
case BPF_JMP | BPF_JSGT | BPF_X:
+ case BPF_JMP32 | BPF_JGT | BPF_K:
+ case BPF_JMP32 | BPF_JGT | BPF_X:
+ case BPF_JMP32 | BPF_JSGT | BPF_K:
+ case BPF_JMP32 | BPF_JSGT | BPF_X:
true_cond = COND_GT;
goto cond_branch;
case BPF_JMP | BPF_JLT | BPF_K:
case BPF_JMP | BPF_JLT | BPF_X:
case BPF_JMP | BPF_JSLT | BPF_K:
case BPF_JMP | BPF_JSLT | BPF_X:
+ case BPF_JMP32 | BPF_JLT | BPF_K:
+ case BPF_JMP32 | BPF_JLT | BPF_X:
+ case BPF_JMP32 | BPF_JSLT | BPF_K:
+ case BPF_JMP32 | BPF_JSLT | BPF_X:
true_cond = COND_LT;
goto cond_branch;
case BPF_JMP | BPF_JGE | BPF_K:
case BPF_JMP | BPF_JGE | BPF_X:
case BPF_JMP | BPF_JSGE | BPF_K:
case BPF_JMP | BPF_JSGE | BPF_X:
+ case BPF_JMP32 | BPF_JGE | BPF_K:
+ case BPF_JMP32 | BPF_JGE | BPF_X:
+ case BPF_JMP32 | BPF_JSGE | BPF_K:
+ case BPF_JMP32 | BPF_JSGE | BPF_X:
true_cond = COND_GE;
goto cond_branch;
case BPF_JMP | BPF_JLE | BPF_K:
case BPF_JMP | BPF_JLE | BPF_X:
case BPF_JMP | BPF_JSLE | BPF_K:
case BPF_JMP | BPF_JSLE | BPF_X:
+ case BPF_JMP32 | BPF_JLE | BPF_K:
+ case BPF_JMP32 | BPF_JLE | BPF_X:
+ case BPF_JMP32 | BPF_JSLE | BPF_K:
+ case BPF_JMP32 | BPF_JSLE | BPF_X:
true_cond = COND_LE;
goto cond_branch;
case BPF_JMP | BPF_JEQ | BPF_K:
case BPF_JMP | BPF_JEQ | BPF_X:
+ case BPF_JMP32 | BPF_JEQ | BPF_K:
+ case BPF_JMP32 | BPF_JEQ | BPF_X:
true_cond = COND_EQ;
goto cond_branch;
case BPF_JMP | BPF_JNE | BPF_K:
case BPF_JMP | BPF_JNE | BPF_X:
+ case BPF_JMP32 | BPF_JNE | BPF_K:
+ case BPF_JMP32 | BPF_JNE | BPF_X:
true_cond = COND_NE;
goto cond_branch;
case BPF_JMP | BPF_JSET | BPF_K:
case BPF_JMP | BPF_JSET | BPF_X:
+ case BPF_JMP32 | BPF_JSET | BPF_K:
+ case BPF_JMP32 | BPF_JSET | BPF_X:
true_cond = COND_NE;
/* Fall through */
--
With Regards,
Sandipan
^ permalink raw reply related
* Re: [PATCH net-next V4 5/5] vhost: access vq metadata through kernel virtual address
From: Michael S. Tsirkin @ 2019-01-24 4:53 UTC (permalink / raw)
To: Jason Wang; +Cc: virtualization, netdev, linux-kernel, kvm
In-Reply-To: <4521d3d8-561e-53f5-98e1-bf7ace003701@redhat.com>
On Thu, Jan 24, 2019 at 12:11:28PM +0800, Jason Wang wrote:
>
> On 2019/1/24 下午12:07, Jason Wang wrote:
> >
> > On 2019/1/23 下午10:08, Michael S. Tsirkin wrote:
> > > On Wed, Jan 23, 2019 at 05:55:57PM +0800, Jason Wang wrote:
> > > > It was noticed that the copy_user() friends that was used to access
> > > > virtqueue metdata tends to be very expensive for dataplane
> > > > implementation like vhost since it involves lots of software checks,
> > > > speculation barrier, hardware feature toggling (e.g SMAP). The
> > > > extra cost will be more obvious when transferring small packets since
> > > > the time spent on metadata accessing become more significant.
> > > >
> > > > This patch tries to eliminate those overheads by accessing them
> > > > through kernel virtual address by vmap(). To make the pages can be
> > > > migrated, instead of pinning them through GUP, we use MMU notifiers to
> > > > invalidate vmaps and re-establish vmaps during each round of metadata
> > > > prefetching if necessary. For devices that doesn't use metadata
> > > > prefetching, the memory accessors fallback to normal copy_user()
> > > > implementation gracefully. The invalidation was synchronized with
> > > > datapath through vq mutex, and in order to avoid hold vq mutex during
> > > > range checking, MMU notifier was teared down when trying to modify vq
> > > > metadata.
> > > >
> > > > Another thing is kernel lacks efficient solution for tracking dirty
> > > > pages by vmap(), this will lead issues if vhost is using file backed
> > > > memory which needs care of writeback. This patch solves this issue by
> > > > just skipping the vma that is file backed and fallback to normal
> > > > copy_user() friends. This might introduce some overheads for file
> > > > backed users but consider this use case is rare we could do
> > > > optimizations on top.
> > > >
> > > > Note that this was only done when device IOTLB is not enabled. We
> > > > could use similar method to optimize it in the future.
> > > >
> > > > Tests shows at most about 22% improvement on TX PPS when using
> > > > virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell:
> > > >
> > > > SMAP on | SMAP off
> > > > Before: 5.0Mpps | 6.6Mpps
> > > > After: 6.1Mpps | 7.4Mpps
> > > >
> > > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> > >
> > > So this is the bulk of the change.
> > > Threee things that I need to look into
> > > - Are there any security issues with bypassing the speculation barrier
> > > that is normally present after access_ok?
> >
> >
> > If we can make sure the bypassing was only used in a kthread (vhost), it
> > should be fine I think.
> >
> >
> > > - How hard does the special handling for
> > > file backed storage make testing?
> >
> >
> > It's as simple as un-commenting vhost_can_vmap()? Or I can try to hack
> > qemu or dpdk to test this.
> >
> >
> > > On the one hand we could add a module parameter to
> > > force copy to/from user. on the other that's
> > > another configuration we need to support.
> >
> >
> > That sounds sub-optimal since it leave the choice to users.
> >
> >
> > > But iotlb is not using vmap, so maybe that's enough
> > > for testing.
> > > - How hard is it to figure out which mode uses which code.
>
>
> It's as simple as tracing __get_user() usage in vhost process?
>
> Thanks
Well there are now mtu notifiers etc etc. It's hardly as well
contained as that.
>
> > >
> > >
> > >
> > > Meanwhile, could you pls post data comparing this last patch with the
> > > below? This removes the speculation barrier replacing it with a
> > > (useless but at least more lightweight) data dependency.
> >
> >
> > SMAP off
> >
> > Your patch: 7.2MPPs
> >
> > vmap: 7.4Mpps
> >
> > I don't test SMAP on, since it will be much slow for sure.
> >
> > Thanks
> >
> >
> > >
> > > Thanks!
> > >
> > >
> > > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > > index bac939af8dbb..352ee7e14476 100644
> > > --- a/drivers/vhost/vhost.c
> > > +++ b/drivers/vhost/vhost.c
> > > @@ -739,7 +739,7 @@ static int vhost_copy_to_user(struct
> > > vhost_virtqueue *vq, void __user *to,
> > > int ret;
> > > if (!vq->iotlb)
> > > - return __copy_to_user(to, from, size);
> > > + return copy_to_user(to, from, size);
> > > else {
> > > /* This function should be called after iotlb
> > > * prefetch, which means we're sure that all vq
> > > @@ -752,7 +752,7 @@ static int vhost_copy_to_user(struct
> > > vhost_virtqueue *vq, void __user *to,
> > > VHOST_ADDR_USED);
> > > if (uaddr)
> > > - return __copy_to_user(uaddr, from, size);
> > > + return copy_to_user(uaddr, from, size);
> > > ret = translate_desc(vq, (u64)(uintptr_t)to, size,
> > > vq->iotlb_iov,
> > > ARRAY_SIZE(vq->iotlb_iov),
> > > @@ -774,7 +774,7 @@ static int vhost_copy_from_user(struct
> > > vhost_virtqueue *vq, void *to,
> > > int ret;
> > > if (!vq->iotlb)
> > > - return __copy_from_user(to, from, size);
> > > + return copy_from_user(to, from, size);
> > > else {
> > > /* This function should be called after iotlb
> > > * prefetch, which means we're sure that vq
> > > @@ -787,7 +787,7 @@ static int vhost_copy_from_user(struct
> > > vhost_virtqueue *vq, void *to,
> > > struct iov_iter f;
> > > if (uaddr)
> > > - return __copy_from_user(to, uaddr, size);
> > > + return copy_from_user(to, uaddr, size);
> > > ret = translate_desc(vq, (u64)(uintptr_t)from, size,
> > > vq->iotlb_iov,
> > > ARRAY_SIZE(vq->iotlb_iov),
> > > @@ -855,13 +855,13 @@ static inline void __user
> > > *__vhost_get_user(struct vhost_virtqueue *vq,
> > > ({ \
> > > int ret = -EFAULT; \
> > > if (!vq->iotlb) { \
> > > - ret = __put_user(x, ptr); \
> > > + ret = put_user(x, ptr); \
> > > } else { \
> > > __typeof__(ptr) to = \
> > > (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
> > > sizeof(*ptr), VHOST_ADDR_USED); \
> > > if (to != NULL) \
> > > - ret = __put_user(x, to); \
> > > + ret = put_user(x, to); \
> > > else \
> > > ret = -EFAULT; \
> > > } \
> > > @@ -872,14 +872,14 @@ static inline void __user
> > > *__vhost_get_user(struct vhost_virtqueue *vq,
> > > ({ \
> > > int ret; \
> > > if (!vq->iotlb) { \
> > > - ret = __get_user(x, ptr); \
> > > + ret = get_user(x, ptr); \
> > > } else { \
> > > __typeof__(ptr) from = \
> > > (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
> > > sizeof(*ptr), \
> > > type); \
> > > if (from != NULL) \
> > > - ret = __get_user(x, from); \
> > > + ret = get_user(x, from); \
> > > else \
> > > ret = -EFAULT; \
> > > } \
^ permalink raw reply
* Re: Does the kernel IPv6 module plan to implement Secure Neighbor Discovery?
From: Ttttabcd @ 2019-01-24 4:53 UTC (permalink / raw)
To: ebiederm@xmission.com; +Cc: Netdev
In-Reply-To: <878szk0xe3.fsf@xmission.com>
>
> Usually it requires someone motivated to step up and do the work. You
> sound motivated. The easiest thing would be for you to step up and
> write the implementation.
>
> Having looked at this once long ago my memory is that SeND only protects
> against an attacker on a local lan. That is not an attack scenario I am
> particularly worried about. If my memory is correct there are
> additional issues with how you perform the initial key distribution.
> All of which is why I am personally not interested.
>
> But if you are interested and would like to make this happen more power
> to you.
>
> Eric
I also thought about implementing it myself, but I am still a kernel beginner.
I am learning the source implementation of the network protocol stack, but I think there is still a long way to go to implement a complete function.
I will work hard, I hope to contribute to the kernel, thank you
^ permalink raw reply
* Re: [PATCH net-next V4 5/5] vhost: access vq metadata through kernel virtual address
From: Michael S. Tsirkin @ 2019-01-24 4:51 UTC (permalink / raw)
To: Jason Wang; +Cc: virtualization, netdev, linux-kernel, kvm
In-Reply-To: <335ba55b-087f-4b35-6311-540070b9647f@redhat.com>
On Thu, Jan 24, 2019 at 12:07:54PM +0800, Jason Wang wrote:
>
> On 2019/1/23 下午10:08, Michael S. Tsirkin wrote:
> > On Wed, Jan 23, 2019 at 05:55:57PM +0800, Jason Wang wrote:
> > > It was noticed that the copy_user() friends that was used to access
> > > virtqueue metdata tends to be very expensive for dataplane
> > > implementation like vhost since it involves lots of software checks,
> > > speculation barrier, hardware feature toggling (e.g SMAP). The
> > > extra cost will be more obvious when transferring small packets since
> > > the time spent on metadata accessing become more significant.
> > >
> > > This patch tries to eliminate those overheads by accessing them
> > > through kernel virtual address by vmap(). To make the pages can be
> > > migrated, instead of pinning them through GUP, we use MMU notifiers to
> > > invalidate vmaps and re-establish vmaps during each round of metadata
> > > prefetching if necessary. For devices that doesn't use metadata
> > > prefetching, the memory accessors fallback to normal copy_user()
> > > implementation gracefully. The invalidation was synchronized with
> > > datapath through vq mutex, and in order to avoid hold vq mutex during
> > > range checking, MMU notifier was teared down when trying to modify vq
> > > metadata.
> > >
> > > Another thing is kernel lacks efficient solution for tracking dirty
> > > pages by vmap(), this will lead issues if vhost is using file backed
> > > memory which needs care of writeback. This patch solves this issue by
> > > just skipping the vma that is file backed and fallback to normal
> > > copy_user() friends. This might introduce some overheads for file
> > > backed users but consider this use case is rare we could do
> > > optimizations on top.
> > >
> > > Note that this was only done when device IOTLB is not enabled. We
> > > could use similar method to optimize it in the future.
> > >
> > > Tests shows at most about 22% improvement on TX PPS when using
> > > virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell:
> > >
> > > SMAP on | SMAP off
> > > Before: 5.0Mpps | 6.6Mpps
> > > After: 6.1Mpps | 7.4Mpps
> > >
> > > Signed-off-by: Jason Wang <jasowang@redhat.com>
> >
> > So this is the bulk of the change.
> > Threee things that I need to look into
> > - Are there any security issues with bypassing the speculation barrier
> > that is normally present after access_ok?
>
>
> If we can make sure the bypassing was only used in a kthread (vhost), it
> should be fine I think.
>
>
> > - How hard does the special handling for
> > file backed storage make testing?
>
>
> It's as simple as un-commenting vhost_can_vmap()? Or I can try to hack qemu
> or dpdk to test this.
>
>
> > On the one hand we could add a module parameter to
> > force copy to/from user. on the other that's
> > another configuration we need to support.
>
>
> That sounds sub-optimal since it leave the choice to users.
>
>
> > But iotlb is not using vmap, so maybe that's enough
> > for testing.
> > - How hard is it to figure out which mode uses which code.
> >
> >
> >
> > Meanwhile, could you pls post data comparing this last patch with the
> > below? This removes the speculation barrier replacing it with a
> > (useless but at least more lightweight) data dependency.
>
>
> SMAP off
>
> Your patch: 7.2MPPs
>
> vmap: 7.4Mpps
>
Sounds more or less as expected. Up to 3% gain with vmap - I think
that's a bit higher than what we saw previously when we switched from
get_user to __get_user and that's probably because of all the
array_index_nospec trickery.
> I don't test SMAP on, since it will be much slow for sure.
Right. So bypassing SMAP remains the main reason to do vmap tricks.
> Thanks
>
> >
> > Thanks!
> >
> >
> > diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> > index bac939af8dbb..352ee7e14476 100644
> > --- a/drivers/vhost/vhost.c
> > +++ b/drivers/vhost/vhost.c
> > @@ -739,7 +739,7 @@ static int vhost_copy_to_user(struct vhost_virtqueue *vq, void __user *to,
> > int ret;
> > if (!vq->iotlb)
> > - return __copy_to_user(to, from, size);
> > + return copy_to_user(to, from, size);
> > else {
> > /* This function should be called after iotlb
> > * prefetch, which means we're sure that all vq
> > @@ -752,7 +752,7 @@ static int vhost_copy_to_user(struct vhost_virtqueue *vq, void __user *to,
> > VHOST_ADDR_USED);
> > if (uaddr)
> > - return __copy_to_user(uaddr, from, size);
> > + return copy_to_user(uaddr, from, size);
> > ret = translate_desc(vq, (u64)(uintptr_t)to, size, vq->iotlb_iov,
> > ARRAY_SIZE(vq->iotlb_iov),
> > @@ -774,7 +774,7 @@ static int vhost_copy_from_user(struct vhost_virtqueue *vq, void *to,
> > int ret;
> > if (!vq->iotlb)
> > - return __copy_from_user(to, from, size);
> > + return copy_from_user(to, from, size);
> > else {
> > /* This function should be called after iotlb
> > * prefetch, which means we're sure that vq
> > @@ -787,7 +787,7 @@ static int vhost_copy_from_user(struct vhost_virtqueue *vq, void *to,
> > struct iov_iter f;
> > if (uaddr)
> > - return __copy_from_user(to, uaddr, size);
> > + return copy_from_user(to, uaddr, size);
> > ret = translate_desc(vq, (u64)(uintptr_t)from, size, vq->iotlb_iov,
> > ARRAY_SIZE(vq->iotlb_iov),
> > @@ -855,13 +855,13 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq,
> > ({ \
> > int ret = -EFAULT; \
> > if (!vq->iotlb) { \
> > - ret = __put_user(x, ptr); \
> > + ret = put_user(x, ptr); \
> > } else { \
> > __typeof__(ptr) to = \
> > (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
> > sizeof(*ptr), VHOST_ADDR_USED); \
> > if (to != NULL) \
> > - ret = __put_user(x, to); \
> > + ret = put_user(x, to); \
> > else \
> > ret = -EFAULT; \
> > } \
> > @@ -872,14 +872,14 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq,
> > ({ \
> > int ret; \
> > if (!vq->iotlb) { \
> > - ret = __get_user(x, ptr); \
> > + ret = get_user(x, ptr); \
> > } else { \
> > __typeof__(ptr) from = \
> > (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
> > sizeof(*ptr), \
> > type); \
> > if (from != NULL) \
> > - ret = __get_user(x, from); \
> > + ret = get_user(x, from); \
> > else \
> > ret = -EFAULT; \
> > } \
^ permalink raw reply
* Re: [BUG] The genet is not working
From: Florian Fainelli @ 2019-01-24 4:47 UTC (permalink / raw)
To: Jaedon Shin, David S. Miller
Cc: netdev, Andrew Lunn, Heiner Kallweit, opendmb
In-Reply-To: <48F4FFA7-FA93-4EA8-B458-22CA314FD91E@gmail.com>
On 1/23/19 8:17 PM, Jaedon Shin wrote:
> Hi,
>
> The genet of Broadcom MIPS 40nm platforms with internal PHY isn't working
> after v4.20-rc1, but 28nm ARM platforms or external PHY is working.
>
> [ 1.064000] bcmgenet 10430000.ethernet: configuring instance for internal PHY
> [ 1.068000] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 1.196000] bcmgenet 10430000.ethernet eth0: Link is Down
> [ 11.076000] Waiting up to 110 more seconds for network.
> [ 21.088000] Waiting up to 100 more seconds for network.
> [ 31.100000] Waiting up to 90 more seconds for network.
> [ 41.112000] Waiting up to 80 more seconds for network.
> [ 51.124000] Waiting up to 70 more seconds for network.
> [ 61.136000] Waiting up to 60 more seconds for network.
> [ 71.148000] Waiting up to 50 more seconds for network.
> [ 81.160000] Waiting up to 40 more seconds for network.
> [ 91.172000] Waiting up to 30 more seconds for network.
> [ 101.184000] Waiting up to 20 more seconds for network.
> [ 111.196000] Waiting up to 10 more seconds for network.
>
> I tried bisecting for this between v4.19 and v4.20-rc1, I found a problematic
> commit 6e2d85ec0559 ("net: phy: Stop with excessive soft reset").
Thanks Jaedon, we will try to come up with an appropriate fix for that
problem.
--
Florian
^ permalink raw reply
* [PATCH v4 bpf-next 9/9] selftests/bpf: test for BPF_F_LOCK
From: Alexei Starovoitov @ 2019-01-24 4:31 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124043130.2268690-1-ast@kernel.org>
Add C based test that runs 4 bpf programs in parallel
that update the same hash and array maps.
And another 2 threads that read from these two maps
via lookup(key, value, BPF_F_LOCK) api
to make sure the user space sees consistent value in both
hash and array elements while user space races with kernel bpf progs.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 2 +-
tools/testing/selftests/bpf/test_map_lock.c | 66 ++++++++++++++++++
tools/testing/selftests/bpf/test_progs.c | 74 +++++++++++++++++++++
3 files changed, 141 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/test_map_lock.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index fcfda51406f9..368ca8249732 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -39,7 +39,7 @@ TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test
get_cgroup_id_kern.o socket_cookie_prog.o test_select_reuseport_kern.o \
test_skb_cgroup_id_kern.o bpf_flow.o netcnt_prog.o \
test_sk_lookup_kern.o test_xdp_vlan.o test_queue_map.o test_stack_map.o \
- xdp_dummy.o test_map_in_map.o test_spin_lock.o
+ xdp_dummy.o test_map_in_map.o test_spin_lock.o test_map_lock.o
# Order correspond to 'make run_tests' order
TEST_PROGS := test_kmod.sh \
diff --git a/tools/testing/selftests/bpf/test_map_lock.c b/tools/testing/selftests/bpf/test_map_lock.c
new file mode 100644
index 000000000000..af8cc68ed2f9
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_map_lock.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (c) 2019 Facebook
+#include <linux/bpf.h>
+#include <linux/version.h>
+#include "bpf_helpers.h"
+
+#define VAR_NUM 16
+
+struct hmap_elem {
+ struct bpf_spin_lock lock;
+ int var[VAR_NUM];
+};
+
+struct bpf_map_def SEC("maps") hash_map = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(int),
+ .value_size = sizeof(struct hmap_elem),
+ .max_entries = 1,
+};
+
+BPF_ANNOTATE_KV_PAIR(hash_map, int, struct hmap_elem);
+
+struct array_elem {
+ struct bpf_spin_lock lock;
+ int var[VAR_NUM];
+};
+
+struct bpf_map_def SEC("maps") array_map = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(int),
+ .value_size = sizeof(struct array_elem),
+ .max_entries = 1,
+};
+
+BPF_ANNOTATE_KV_PAIR(array_map, int, struct array_elem);
+
+SEC("map_lock_demo")
+int bpf_map_lock_test(struct __sk_buff *skb)
+{
+ struct hmap_elem zero = {}, *val;
+ int rnd = bpf_get_prandom_u32();
+ int key = 0, err = 1, i;
+ struct array_elem *q;
+
+ val = bpf_map_lookup_elem(&hash_map, &key);
+ if (!val)
+ goto err;
+ /* spin_lock in hash map */
+ bpf_spin_lock(&val->lock);
+ for (i = 0; i < VAR_NUM; i++)
+ val->var[i] = rnd;
+ bpf_spin_unlock(&val->lock);
+
+ /* spin_lock in array */
+ q = bpf_map_lookup_elem(&array_map, &key);
+ if (!q)
+ goto err;
+ bpf_spin_lock(&q->lock);
+ for (i = 0; i < VAR_NUM; i++)
+ q->var[i] = rnd;
+ bpf_spin_unlock(&q->lock);
+ err = 0;
+err:
+ return err;
+}
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c
index 6425e95c3f16..1427338f14a2 100644
--- a/tools/testing/selftests/bpf/test_progs.c
+++ b/tools/testing/selftests/bpf/test_progs.c
@@ -1922,6 +1922,79 @@ static void test_spin_lock(void)
bpf_object__close(obj);
}
+static void *parallel_map_access(void *arg)
+{
+ int err, map_fd = *(u32 *) arg;
+ int vars[17], i, j, rnd, key = 0;
+
+ for (i = 0; i < 10000; i++) {
+ err = bpf_map_lookup_elem_flags(map_fd, &key, vars, BPF_F_LOCK);
+ if (err) {
+ printf("lookup failed\n");
+ error_cnt++;
+ goto out;
+ }
+ if (vars[0] != 0) {
+ printf("lookup #%d var[0]=%d\n", i, vars[0]);
+ error_cnt++;
+ goto out;
+ }
+ rnd = vars[1];
+ for (j = 2; j < 17; j++) {
+ if (vars[j] == rnd)
+ continue;
+ printf("lookup #%d var[1]=%d var[%d]=%d\n",
+ i, rnd, j, vars[j]);
+ error_cnt++;
+ goto out;
+ }
+ }
+out:
+ pthread_exit(arg);
+}
+
+static void test_map_lock(void)
+{
+ const char *file = "./test_map_lock.o";
+ int prog_fd, map_fd[2], vars[17] = {};
+ pthread_t thread_id[6];
+ struct bpf_object *obj;
+ int err = 0, key = 0, i;
+ void *ret;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_CGROUP_SKB, &obj, &prog_fd);
+ if (err) {
+ printf("test_map_lock:bpf_prog_load errno %d\n", errno);
+ goto close_prog;
+ }
+ map_fd[0] = bpf_find_map(__func__, obj, "hash_map");
+ if (map_fd[0] < 0)
+ goto close_prog;
+ map_fd[1] = bpf_find_map(__func__, obj, "array_map");
+ if (map_fd[1] < 0)
+ goto close_prog;
+
+ bpf_map_update_elem(map_fd[0], &key, vars, BPF_F_LOCK);
+
+ for (i = 0; i < 4; i++)
+ assert(pthread_create(&thread_id[i], NULL,
+ ¶llel_bpf_prog_test_run, &prog_fd) == 0);
+ for (i = 4; i < 6; i++)
+ assert(pthread_create(&thread_id[i], NULL,
+ ¶llel_map_access, &map_fd[i - 4]) == 0);
+ for (i = 0; i < 4; i++)
+ assert(pthread_join(thread_id[i], &ret) == 0 &&
+ ret == (void *)&prog_fd);
+ for (i = 4; i < 6; i++)
+ assert(pthread_join(thread_id[i], &ret) == 0 &&
+ ret == (void *)&map_fd[i - 4]);
+ goto close_prog_noerr;
+close_prog:
+ error_cnt++;
+close_prog_noerr:
+ bpf_object__close(obj);
+}
+
int main(void)
{
srand(time(NULL));
@@ -1950,6 +2023,7 @@ int main(void)
test_queue_stack_map(QUEUE);
test_queue_stack_map(STACK);
test_spin_lock();
+ test_map_lock();
printf("Summary: %d PASSED, %d FAILED\n", pass_cnt, error_cnt);
return error_cnt ? EXIT_FAILURE : EXIT_SUCCESS;
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 8/9] libbpf: introduce bpf_map_lookup_elem_flags()
From: Alexei Starovoitov @ 2019-01-24 4:31 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
Introduce
int bpf_map_lookup_elem_flags(int fd, const void *key, void *value, __u64 flags)
helper to lookup array/hash/cgroup_local_storage elements with BPF_F_LOCK flag.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/bpf.c | 13 +++++++++++++
tools/lib/bpf/bpf.h | 2 ++
tools/lib/bpf/libbpf.map | 1 +
3 files changed, 16 insertions(+)
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index 3caaa3428774..d55a77a05d5f 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -357,6 +357,19 @@ int bpf_map_lookup_elem(int fd, const void *key, void *value)
return sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
}
+int bpf_map_lookup_elem_flags(int fd, const void *key, void *value, __u64 flags)
+{
+ union bpf_attr attr;
+
+ bzero(&attr, sizeof(attr));
+ attr.map_fd = fd;
+ attr.key = ptr_to_u64(key);
+ attr.value = ptr_to_u64(value);
+ attr.flags = flags;
+
+ return sys_bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
+}
+
int bpf_map_lookup_and_delete_elem(int fd, const void *key, void *value)
{
union bpf_attr attr;
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 8f09de482839..ed09eed2dc3b 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -110,6 +110,8 @@ LIBBPF_API int bpf_map_update_elem(int fd, const void *key, const void *value,
__u64 flags);
LIBBPF_API int bpf_map_lookup_elem(int fd, const void *key, void *value);
+LIBBPF_API int bpf_map_lookup_elem_flags(int fd, const void *key, void *value,
+ __u64 flags);
LIBBPF_API int bpf_map_lookup_and_delete_elem(int fd, const void *key,
void *value);
LIBBPF_API int bpf_map_delete_elem(int fd, const void *key);
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index 266bc95d0142..f6f96fc38c50 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -130,4 +130,5 @@ LIBBPF_0.0.2 {
bpf_probe_helper;
bpf_probe_map_type;
bpf_probe_prog_type;
+ bpf_map_lookup_elem_flags;
} LIBBPF_0.0.1;
--
2.17.1
^ permalink raw reply related
* [BUG] The genet is not working
From: Jaedon Shin @ 2019-01-24 4:17 UTC (permalink / raw)
To: Florian Fainelli, David S. Miller; +Cc: netdev, Andrew Lunn, Heiner Kallweit
Hi,
The genet of Broadcom MIPS 40nm platforms with internal PHY isn't working
after v4.20-rc1, but 28nm ARM platforms or external PHY is working.
[ 1.064000] bcmgenet 10430000.ethernet: configuring instance for internal PHY
[ 1.068000] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1.196000] bcmgenet 10430000.ethernet eth0: Link is Down
[ 11.076000] Waiting up to 110 more seconds for network.
[ 21.088000] Waiting up to 100 more seconds for network.
[ 31.100000] Waiting up to 90 more seconds for network.
[ 41.112000] Waiting up to 80 more seconds for network.
[ 51.124000] Waiting up to 70 more seconds for network.
[ 61.136000] Waiting up to 60 more seconds for network.
[ 71.148000] Waiting up to 50 more seconds for network.
[ 81.160000] Waiting up to 40 more seconds for network.
[ 91.172000] Waiting up to 30 more seconds for network.
[ 101.184000] Waiting up to 20 more seconds for network.
[ 111.196000] Waiting up to 10 more seconds for network.
I tried bisecting for this between v4.19 and v4.20-rc1, I found a problematic
commit 6e2d85ec0559 ("net: phy: Stop with excessive soft reset").
Thanks,
Jaedon
^ permalink raw reply
* [PATCH v4 bpf-next 4/9] selftests/bpf: add bpf_spin_lock tests
From: Alexei Starovoitov @ 2019-01-24 4:13 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
add bpf_spin_lock tests to test_verifier.c that don't require
latest llvm with BTF support
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/test_verifier.c | 459 +++++++++++++++++++-
1 file changed, 458 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c
index 4f67afeec8d9..4b29dba7ffff 100644
--- a/tools/testing/selftests/bpf/test_verifier.c
+++ b/tools/testing/selftests/bpf/test_verifier.c
@@ -32,6 +32,7 @@
#include <linux/bpf_perf_event.h>
#include <linux/bpf.h>
#include <linux/if_ether.h>
+#include <linux/btf.h>
#include <bpf/bpf.h>
@@ -49,7 +50,7 @@
#define MAX_INSNS BPF_MAXINSNS
#define MAX_FIXUPS 8
-#define MAX_NR_MAPS 13
+#define MAX_NR_MAPS 14
#define MAX_TEST_RUNS 8
#define POINTER_VALUE 0xcafe4all
#define TEST_DATA_LEN 64
@@ -76,6 +77,7 @@ struct bpf_test {
int fixup_map_in_map[MAX_FIXUPS];
int fixup_cgroup_storage[MAX_FIXUPS];
int fixup_percpu_cgroup_storage[MAX_FIXUPS];
+ int fixup_map_spin_lock[MAX_FIXUPS];
const char *errstr;
const char *errstr_unpriv;
uint32_t retval, retval_unpriv, insn_processed;
@@ -15759,6 +15761,361 @@ static struct bpf_test tests[] = {
.result = ACCEPT,
.retval = 2,
},
+ {
+ "spin_lock: test1 success",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = ACCEPT,
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test2 direct ld/st",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "cannot be accessed directly",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test3 direct ld/st",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 1),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "cannot be accessed directly",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test4 direct ld/st",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_6, 3),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "cannot be accessed directly",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test5 call within a locked region",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_get_prandom_u32),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "calls are not allowed",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test6 missing unlock",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "unlock is missing",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test7 unlock without lock",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_1, 0, 1),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "without taking a lock",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test8 double lock",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "calls are not allowed",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test9 different lock",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_7, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_7),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3, 11 },
+ .result = REJECT,
+ .errstr = "unlock of different lock",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test10 lock in subprog without unlock",
+ .insns = {
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 5),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .result = REJECT,
+ .errstr = "unlock is missing",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
+ },
+ {
+ "spin_lock: test11 ld_abs under lock",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_7, BPF_REG_0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_lock),
+ BPF_LD_ABS(BPF_B, 0),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_7),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, 4),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_spin_unlock),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 4 },
+ .result = REJECT,
+ .errstr = "inside bpf_spin_lock",
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ },
};
static int probe_filter_length(const struct bpf_insn *fp)
@@ -15889,6 +16246,98 @@ static int create_cgroup_storage(bool percpu)
return fd;
}
+#define BTF_INFO_ENC(kind, kind_flag, vlen) \
+ ((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
+#define BTF_TYPE_ENC(name, info, size_or_type) \
+ (name), (info), (size_or_type)
+#define BTF_INT_ENC(encoding, bits_offset, nr_bits) \
+ ((encoding) << 24 | (bits_offset) << 16 | (nr_bits))
+#define BTF_TYPE_INT_ENC(name, encoding, bits_offset, bits, sz) \
+ BTF_TYPE_ENC(name, BTF_INFO_ENC(BTF_KIND_INT, 0, 0), sz), \
+ BTF_INT_ENC(encoding, bits_offset, bits)
+#define BTF_MEMBER_ENC(name, type, bits_offset) \
+ (name), (type), (bits_offset)
+
+struct btf_raw_data {
+ __u32 raw_types[64];
+ const char *str_sec;
+ __u32 str_sec_size;
+};
+
+/* struct bpf_spin_lock {
+ * int val;
+ * };
+ * struct val {
+ * int cnt;
+ * struct bpf_spin_lock l;
+ * };
+ */
+static const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l";
+static __u32 btf_raw_types[] = {
+ /* int */
+ BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
+ /* struct bpf_spin_lock */ /* [2] */
+ BTF_TYPE_ENC(1, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 1), 4),
+ BTF_MEMBER_ENC(15, 1, 0), /* int val; */
+ /* struct val */ /* [3] */
+ BTF_TYPE_ENC(15, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 2), 8),
+ BTF_MEMBER_ENC(19, 1, 0), /* int cnt; */
+ BTF_MEMBER_ENC(23, 2, 32),/* struct bpf_spin_lock l; */
+};
+
+static int load_btf(void)
+{
+ struct btf_header hdr = {
+ .magic = BTF_MAGIC,
+ .version = BTF_VERSION,
+ .hdr_len = sizeof(struct btf_header),
+ .type_len = sizeof(btf_raw_types),
+ .str_off = sizeof(btf_raw_types),
+ .str_len = sizeof(btf_str_sec),
+ };
+ void *ptr, *raw_btf;
+ int btf_fd;
+
+ ptr = raw_btf = malloc(sizeof(hdr) + sizeof(btf_raw_types) +
+ sizeof(btf_str_sec));
+
+ memcpy(ptr, &hdr, sizeof(hdr));
+ ptr += sizeof(hdr);
+ memcpy(ptr, btf_raw_types, hdr.type_len);
+ ptr += hdr.type_len;
+ memcpy(ptr, btf_str_sec, hdr.str_len);
+ ptr += hdr.str_len;
+
+ btf_fd = bpf_load_btf(raw_btf, ptr - raw_btf, 0, 0, 0);
+ free(raw_btf);
+ if (btf_fd < 0)
+ return -1;
+ return btf_fd;
+}
+
+static int create_map_spin_lock(void)
+{
+ struct bpf_create_map_attr attr = {
+ .name = "test_map",
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .key_size = 4,
+ .value_size = 8,
+ .max_entries = 1,
+ .btf_key_type_id = 1,
+ .btf_value_type_id = 3,
+ };
+ int fd, btf_fd;
+
+ btf_fd = load_btf();
+ if (btf_fd < 0)
+ return -1;
+ attr.btf_fd = btf_fd;
+ fd = bpf_create_map_xattr(&attr);
+ if (fd < 0)
+ printf("Failed to create map with spin_lock\n");
+ return fd;
+}
+
static char bpf_vlog[UINT_MAX >> 8];
static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type,
@@ -15907,6 +16356,7 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type,
int *fixup_map_in_map = test->fixup_map_in_map;
int *fixup_cgroup_storage = test->fixup_cgroup_storage;
int *fixup_percpu_cgroup_storage = test->fixup_percpu_cgroup_storage;
+ int *fixup_map_spin_lock = test->fixup_map_spin_lock;
if (test->fill_helper)
test->fill_helper(test);
@@ -16023,6 +16473,13 @@ static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type,
fixup_map_stacktrace++;
} while (*fixup_map_stacktrace);
}
+ if (*fixup_map_spin_lock) {
+ map_fds[13] = create_map_spin_lock();
+ do {
+ prog[*fixup_map_spin_lock].imm = map_fds[13];
+ fixup_map_spin_lock++;
+ } while (*fixup_map_spin_lock);
+ }
}
static int set_admin(bool admin)
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 0/9] introduce bpf_spin_lock
From: Alexei Starovoitov @ 2019-01-24 4:13 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
Many algorithms need to read and modify several variables atomically.
Until now it was hard to impossible to implement such algorithms in BPF.
Hence introduce support for bpf_spin_lock.
The api consists of 'struct bpf_spin_lock' that should be placed
inside hash/array/cgroup_local_storage element
and bpf_spin_lock/unlock() helper function.
Example:
struct hash_elem {
int cnt;
struct bpf_spin_lock lock;
};
struct hash_elem * val = bpf_map_lookup_elem(&hash_map, &key);
if (val) {
bpf_spin_lock(&val->lock);
val->cnt++;
bpf_spin_unlock(&val->lock);
}
and BPF_F_LOCK flag for lookup/update bpf syscall commands that
allows user space to read/write map elements under lock.
Together these primitives allow race free access to map elements
from bpf programs and from user space.
Key restriction: root only.
Key requirement: maps must be annotated with BTF.
This concept was discussed at Linux Plumbers Conference 2018.
Thank you everyone who participated and helped to iron out details
of api and implementation.
Patch 1: bpf_spin_lock support in the verifier, BTF, hash, array.
Patch 2: bpf_spin_lock in cgroup local storage.
Patches 3,4,5: tests
Patch 6: BPF_F_LOCK flag to lookup/update
Patches 7,8,9: tests
v3->v4:
- fix BPF_EXIST | BPF_NOEXIST check patch 6. Spotted by Jakub. Thanks!
- rebase
v2->v3:
- fixed build on ia64 and archs where qspinlock is not supported
- fixed missing lock init during lookup w/o BPF_F_LOCK. Spotted by Martin
v1->v2:
- addressed several issues spotted by Daniel and Martin in patch 1
- added test11 to patch 4 as suggested by Daniel
Alexei Starovoitov (9):
bpf: introduce bpf_spin_lock
bpf: add support for bpf_spin_lock to cgroup local storage
tools/bpf: sync include/uapi/linux/bpf.h
selftests/bpf: add bpf_spin_lock tests
selftests/bpf: add bpf_spin_lock C test
bpf: introduce BPF_F_LOCK flag
tools/bpf: sync uapi/bpf.h
libbpf: introduce bpf_map_lookup_elem_flags()
selftests/bpf: test for BPF_F_LOCK
include/linux/bpf.h | 39 +-
include/linux/bpf_verifier.h | 1 +
include/linux/btf.h | 1 +
include/uapi/linux/bpf.h | 8 +-
kernel/bpf/arraymap.c | 23 +-
kernel/bpf/btf.c | 42 ++
kernel/bpf/core.c | 2 +
kernel/bpf/hashtab.c | 48 +-
kernel/bpf/helpers.c | 71 +++
kernel/bpf/local_storage.c | 16 +-
kernel/bpf/map_in_map.c | 5 +
kernel/bpf/syscall.c | 45 +-
kernel/bpf/verifier.c | 151 +++++-
net/core/filter.c | 16 +-
tools/include/uapi/linux/bpf.h | 8 +-
tools/lib/bpf/bpf.c | 13 +
tools/lib/bpf/bpf.h | 2 +
tools/lib/bpf/libbpf.map | 1 +
tools/testing/selftests/bpf/Makefile | 2 +-
tools/testing/selftests/bpf/bpf_helpers.h | 4 +
tools/testing/selftests/bpf/test_map_lock.c | 66 +++
tools/testing/selftests/bpf/test_progs.c | 117 ++++-
tools/testing/selftests/bpf/test_spin_lock.c | 108 +++++
tools/testing/selftests/bpf/test_verifier.c | 459 ++++++++++++++++++-
24 files changed, 1219 insertions(+), 29 deletions(-)
create mode 100644 tools/testing/selftests/bpf/test_map_lock.c
create mode 100644 tools/testing/selftests/bpf/test_spin_lock.c
--
2.17.1
^ permalink raw reply
* [PATCH v4 bpf-next 6/9] bpf: introduce BPF_F_LOCK flag
From: Alexei Starovoitov @ 2019-01-24 4:14 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
Introduce BPF_F_LOCK flag for map_lookup and map_update syscall commands
and for map_update() helper function.
In all these cases take a lock of existing element (which was provided
in BTF description) before copying (in or out) the rest of map value.
Implementation details that are part of uapi:
Array:
The array map takes the element lock for lookup/update.
Hash:
hash map also takes the lock for lookup/update and tries to avoid the bucket lock.
If old element exists it takes the element lock and updates the element in place.
If element doesn't exist it allocates new one and inserts into hash table
while holding the bucket lock.
In rare case the hashmap has to take both the bucket lock and the element lock
to update old value in place.
Cgroup local storage:
It is similar to array. update in place and lookup are done with lock taken.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 2 ++
include/uapi/linux/bpf.h | 1 +
kernel/bpf/arraymap.c | 24 ++++++++++++++--------
kernel/bpf/hashtab.c | 42 +++++++++++++++++++++++++++++++++++---
kernel/bpf/helpers.c | 14 +++++++++++++
kernel/bpf/local_storage.c | 14 ++++++++++++-
kernel/bpf/syscall.c | 25 +++++++++++++++++++++--
7 files changed, 108 insertions(+), 14 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index ee0352bd728f..34a517137b63 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -119,6 +119,8 @@ static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
memcpy(dst, src, map->value_size);
}
}
+void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
+ bool lock_src);
struct bpf_offload_dev;
struct bpf_offloaded_map;
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 30f9dfd40f13..c23eaa284a45 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -266,6 +266,7 @@ enum bpf_attach_type {
#define BPF_ANY 0 /* create new element or update existing */
#define BPF_NOEXIST 1 /* create new element if it didn't exist */
#define BPF_EXIST 2 /* update existing element */
+#define BPF_F_LOCK 4 /* spin_lock-ed map_lookup/map_update */
/* flags for BPF_MAP_CREATE command */
#define BPF_F_NO_PREALLOC (1U << 0)
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index d6d979910a2a..c72e0d8e1e65 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -253,8 +253,9 @@ static int array_map_update_elem(struct bpf_map *map, void *key, void *value,
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
u32 index = *(u32 *)key;
+ char *val;
- if (unlikely(map_flags > BPF_EXIST))
+ if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST))
/* unknown flags */
return -EINVAL;
@@ -262,18 +263,25 @@ static int array_map_update_elem(struct bpf_map *map, void *key, void *value,
/* all elements were pre-allocated, cannot insert a new one */
return -E2BIG;
- if (unlikely(map_flags == BPF_NOEXIST))
+ if (unlikely(map_flags & BPF_NOEXIST))
/* all elements already exist */
return -EEXIST;
- if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+ if (unlikely((map_flags & BPF_F_LOCK) &&
+ !map_value_has_spin_lock(map)))
+ return -EINVAL;
+
+ if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
memcpy(this_cpu_ptr(array->pptrs[index & array->index_mask]),
value, map->value_size);
- else
- copy_map_value(map,
- array->value +
- array->elem_size * (index & array->index_mask),
- value);
+ } else {
+ val = array->value +
+ array->elem_size * (index & array->index_mask);
+ if (map_flags & BPF_F_LOCK)
+ copy_map_value_locked(map, val, value, false);
+ else
+ copy_map_value(map, val, value);
+ }
return 0;
}
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 48a41bf65e1b..27b986c19c63 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -809,11 +809,11 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
static int check_flags(struct bpf_htab *htab, struct htab_elem *l_old,
u64 map_flags)
{
- if (l_old && map_flags == BPF_NOEXIST)
+ if (l_old && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
/* elem already exists */
return -EEXIST;
- if (!l_old && map_flags == BPF_EXIST)
+ if (!l_old && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
/* elem doesn't exist, cannot update it */
return -ENOENT;
@@ -832,7 +832,7 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value,
u32 key_size, hash;
int ret;
- if (unlikely(map_flags > BPF_EXIST))
+ if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST))
/* unknown flags */
return -EINVAL;
@@ -845,6 +845,28 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value,
b = __select_bucket(htab, hash);
head = &b->head;
+ if (unlikely(map_flags & BPF_F_LOCK)) {
+ if (unlikely(!map_value_has_spin_lock(map)))
+ return -EINVAL;
+ /* find an element without taking the bucket lock */
+ l_old = lookup_nulls_elem_raw(head, hash, key, key_size,
+ htab->n_buckets);
+ ret = check_flags(htab, l_old, map_flags);
+ if (ret)
+ return ret;
+ if (l_old) {
+ /* grab the element lock and update value in place */
+ copy_map_value_locked(map,
+ l_old->key + round_up(key_size, 8),
+ value, false);
+ return 0;
+ }
+ /* fall through, grab the bucket lock and lookup again.
+ * 99.9% chance that the element won't be found,
+ * but second lookup under lock has to be done.
+ */
+ }
+
/* bpf_map_update_elem() can be called in_irq() */
raw_spin_lock_irqsave(&b->lock, flags);
@@ -854,6 +876,20 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value,
if (ret)
goto err;
+ if (unlikely(l_old && (map_flags & BPF_F_LOCK))) {
+ /* first lookup without the bucket lock didn't find the element,
+ * but second lookup with the bucket lock found it.
+ * This case is highly unlikely, but has to be dealt with:
+ * grab the element lock in addition to the bucket lock
+ * and update element in place
+ */
+ copy_map_value_locked(map,
+ l_old->key + round_up(key_size, 8),
+ value, false);
+ ret = 0;
+ goto err;
+ }
+
l_new = alloc_htab_elem(htab, key, value, key_size, hash, false, false,
l_old);
if (IS_ERR(l_new)) {
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2e98e4caf5aa..60e475bcb708 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -278,6 +278,20 @@ const struct bpf_func_proto bpf_spin_unlock_proto = {
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
};
+void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
+ bool lock_src)
+{
+ struct bpf_spin_lock *lock;
+
+ if (lock_src)
+ lock = src + map->spin_lock_off;
+ else
+ lock = dst + map->spin_lock_off;
+ ____bpf_spin_lock(lock);
+ copy_map_value(map, dst, src);
+ ____bpf_spin_unlock(lock);
+}
+
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c
index 0295427f06e2..6b572e2de7fb 100644
--- a/kernel/bpf/local_storage.c
+++ b/kernel/bpf/local_storage.c
@@ -131,7 +131,14 @@ static int cgroup_storage_update_elem(struct bpf_map *map, void *_key,
struct bpf_cgroup_storage *storage;
struct bpf_storage_buffer *new;
- if (flags != BPF_ANY && flags != BPF_EXIST)
+ if (unlikely(flags & ~(BPF_F_LOCK | BPF_EXIST | BPF_NOEXIST)))
+ return -EINVAL;
+
+ if (unlikely(flags & BPF_NOEXIST))
+ return -EINVAL;
+
+ if (unlikely((flags & BPF_F_LOCK) &&
+ !map_value_has_spin_lock(map)))
return -EINVAL;
storage = cgroup_storage_lookup((struct bpf_cgroup_storage_map *)map,
@@ -139,6 +146,11 @@ static int cgroup_storage_update_elem(struct bpf_map *map, void *_key,
if (!storage)
return -ENOENT;
+ if (flags & BPF_F_LOCK) {
+ copy_map_value_locked(map, storage->buf->data, value, false);
+ return 0;
+ }
+
new = kmalloc_node(sizeof(struct bpf_storage_buffer) +
map->value_size,
__GFP_ZERO | GFP_ATOMIC | __GFP_NOWARN,
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index b29e6dc44650..0834958f1dc4 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -682,7 +682,7 @@ static void *__bpf_copy_key(void __user *ukey, u64 key_size)
}
/* last field in 'union bpf_attr' used by this command */
-#define BPF_MAP_LOOKUP_ELEM_LAST_FIELD value
+#define BPF_MAP_LOOKUP_ELEM_LAST_FIELD flags
static int map_lookup_elem(union bpf_attr *attr)
{
@@ -698,6 +698,9 @@ static int map_lookup_elem(union bpf_attr *attr)
if (CHECK_ATTR(BPF_MAP_LOOKUP_ELEM))
return -EINVAL;
+ if (attr->flags & ~BPF_F_LOCK)
+ return -EINVAL;
+
f = fdget(ufd);
map = __bpf_map_get(f);
if (IS_ERR(map))
@@ -708,6 +711,12 @@ static int map_lookup_elem(union bpf_attr *attr)
goto err_put;
}
+ if ((attr->flags & BPF_F_LOCK) &&
+ !map_value_has_spin_lock(map)) {
+ err = -EINVAL;
+ goto err_put;
+ }
+
key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
@@ -758,7 +767,13 @@ static int map_lookup_elem(union bpf_attr *attr)
err = -ENOENT;
} else {
err = 0;
- copy_map_value(map, value, ptr);
+ if (attr->flags & BPF_F_LOCK)
+ /* lock 'ptr' and copy everything but lock */
+ copy_map_value_locked(map, value, ptr, true);
+ else
+ copy_map_value(map, value, ptr);
+ /* mask lock, since value wasn't zero inited */
+ check_and_init_map_lock(map, value);
}
rcu_read_unlock();
}
@@ -818,6 +833,12 @@ static int map_update_elem(union bpf_attr *attr)
goto err_put;
}
+ if ((attr->flags & BPF_F_LOCK) &&
+ !map_value_has_spin_lock(map)) {
+ err = -EINVAL;
+ goto err_put;
+ }
+
key = __bpf_copy_key(ukey, map->key_size);
if (IS_ERR(key)) {
err = PTR_ERR(key);
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 2/9] bpf: add support for bpf_spin_lock to cgroup local storage
From: Alexei Starovoitov @ 2019-01-24 4:13 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
Allow 'struct bpf_spin_lock' to reside inside cgroup local storage.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/local_storage.c | 2 ++
kernel/bpf/syscall.c | 3 ++-
kernel/bpf/verifier.c | 2 ++
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c
index 07a34ef562a0..0295427f06e2 100644
--- a/kernel/bpf/local_storage.c
+++ b/kernel/bpf/local_storage.c
@@ -147,6 +147,7 @@ static int cgroup_storage_update_elem(struct bpf_map *map, void *_key,
return -ENOMEM;
memcpy(&new->data[0], value, map->value_size);
+ check_and_init_map_lock(map, new->data);
new = xchg(&storage->buf, new);
kfree_rcu(new, rcu);
@@ -483,6 +484,7 @@ struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(struct bpf_prog *prog,
storage->buf = kmalloc_node(size, flags, map->numa_node);
if (!storage->buf)
goto enomem;
+ check_and_init_map_lock(map, storage->buf->data);
} else {
storage->percpu_buf = __alloc_percpu_gfp(size, 8, flags);
if (!storage->percpu_buf)
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index ebf0a673cb83..b29e6dc44650 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -482,7 +482,8 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf,
if (map_value_has_spin_lock(map)) {
if (map->map_type != BPF_MAP_TYPE_HASH &&
- map->map_type != BPF_MAP_TYPE_ARRAY)
+ map->map_type != BPF_MAP_TYPE_ARRAY &&
+ map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE)
return -ENOTSUPP;
if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >
map->value_size) {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f372904fa691..bfd6bebca911 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3089,6 +3089,8 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn
regs[BPF_REG_0].map_ptr = meta.map_ptr;
if (fn->ret_type == RET_PTR_TO_MAP_VALUE) {
regs[BPF_REG_0].type = PTR_TO_MAP_VALUE;
+ if (map_value_has_spin_lock(meta.map_ptr))
+ regs[BPF_REG_0].id = ++env->id_gen;
} else {
regs[BPF_REG_0].type = PTR_TO_MAP_VALUE_OR_NULL;
regs[BPF_REG_0].id = ++env->id_gen;
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 1/9] bpf: introduce bpf_spin_lock
From: Alexei Starovoitov @ 2019-01-24 4:13 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
Introduce 'struct bpf_spin_lock' and bpf_spin_lock/unlock() helpers to let
bpf program serialize access to other variables.
Example:
struct hash_elem {
int cnt;
struct bpf_spin_lock lock;
};
struct hash_elem * val = bpf_map_lookup_elem(&hash_map, &key);
if (val) {
bpf_spin_lock(&val->lock);
val->cnt++;
bpf_spin_unlock(&val->lock);
}
Restrictions and safety checks:
- bpf_spin_lock is only allowed inside HASH and ARRAY maps.
- BTF description of the map is mandatory for safety analysis.
- bpf program can take one bpf_spin_lock at a time, since two or more can
cause dead locks.
- only one 'struct bpf_spin_lock' is allowed per map element.
It drastically simplifies implementation yet allows bpf program to use
any number of bpf_spin_locks.
- when bpf_spin_lock is taken the calls (either bpf2bpf or helpers) are not allowed.
- bpf program must bpf_spin_unlock() before return.
- bpf program can access 'struct bpf_spin_lock' only via
bpf_spin_lock()/bpf_spin_unlock() helpers.
- load/store into 'struct bpf_spin_lock lock;' field is not allowed.
- to use bpf_spin_lock() helper the BTF description of map value must be
a struct and have 'struct bpf_spin_lock anyname;' field at the top level.
Nested lock inside another struct is not allowed.
- syscall map_lookup doesn't copy bpf_spin_lock field to user space.
- syscall map_update and program map_update do not update bpf_spin_lock field.
- bpf_spin_lock cannot be on the stack or inside networking packet.
bpf_spin_lock can only be inside HASH or ARRAY map value.
- bpf_spin_lock is available to root only and to all program types.
- bpf_spin_lock is not allowed in inner maps of map-in-map.
- ld_abs is not allowed inside spin_lock-ed region.
Implementation details:
- on !SMP bpf_spin_lock() becomes nop
- on architectures that don't support queued_spin_lock trivial lock is used.
Note that arch_spin_lock cannot be used, since not all archs agree that
zero == unlocked and sizeof(arch_spinlock_t) != sizeof(__u32).
- presence of bpf_spin_lock inside map value could have been indicated via
extra flag during map_create, but specifying it via BTF is cleaner.
It provides introspection for map key/value and reduces user coding mistakes.
Next steps:
- allow bpf_spin_lock in other map types (like cgroup local storage)
- introduce BPF_F_LOCK flag for bpf_map_update() syscall and helper
to request kernel to grab bpf_spin_lock before rewriting the value.
That will serialize access to map elements.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 37 ++++++++-
include/linux/bpf_verifier.h | 1 +
include/linux/btf.h | 1 +
include/uapi/linux/bpf.h | 7 +-
kernel/bpf/arraymap.c | 7 +-
kernel/bpf/btf.c | 42 ++++++++++
kernel/bpf/core.c | 2 +
kernel/bpf/hashtab.c | 6 +-
kernel/bpf/helpers.c | 57 ++++++++++++++
kernel/bpf/map_in_map.c | 5 ++
kernel/bpf/syscall.c | 21 ++++-
kernel/bpf/verifier.c | 149 ++++++++++++++++++++++++++++++++++-
net/core/filter.c | 16 +++-
13 files changed, 335 insertions(+), 16 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 3851529062ec..ee0352bd728f 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -72,14 +72,15 @@ struct bpf_map {
u32 value_size;
u32 max_entries;
u32 map_flags;
- u32 pages;
+ int spin_lock_off; /* >=0 valid offset, <0 error */
u32 id;
int numa_node;
u32 btf_key_type_id;
u32 btf_value_type_id;
struct btf *btf;
+ u32 pages;
bool unpriv_array;
- /* 55 bytes hole */
+ /* 51 bytes hole */
/* The 3rd and 4th cacheline with misc members to avoid false sharing
* particularly with refcounting.
@@ -91,6 +92,34 @@ struct bpf_map {
char name[BPF_OBJ_NAME_LEN];
};
+static inline bool map_value_has_spin_lock(const struct bpf_map *map)
+{
+ return map->spin_lock_off >= 0;
+}
+
+static inline void check_and_init_map_lock(struct bpf_map *map, void *dst)
+{
+ if (likely(!map_value_has_spin_lock(map)))
+ return;
+ *(struct bpf_spin_lock *)(dst + map->spin_lock_off) =
+ (struct bpf_spin_lock){};
+}
+
+/* copy everything but bpf_spin_lock */
+static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
+{
+ if (unlikely(map_value_has_spin_lock(map))) {
+ u32 off = map->spin_lock_off;
+
+ memcpy(dst, src, off);
+ memcpy(dst + off + sizeof(struct bpf_spin_lock),
+ src + off + sizeof(struct bpf_spin_lock),
+ map->value_size - off - sizeof(struct bpf_spin_lock));
+ } else {
+ memcpy(dst, src, map->value_size);
+ }
+}
+
struct bpf_offload_dev;
struct bpf_offloaded_map;
@@ -162,6 +191,7 @@ enum bpf_arg_type {
ARG_PTR_TO_CTX, /* pointer to context */
ARG_ANYTHING, /* any (initialized) argument is ok */
ARG_PTR_TO_SOCKET, /* pointer to bpf_sock */
+ ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */
};
/* type of values returned from helper functions */
@@ -876,7 +906,8 @@ extern const struct bpf_func_proto bpf_msg_redirect_hash_proto;
extern const struct bpf_func_proto bpf_msg_redirect_map_proto;
extern const struct bpf_func_proto bpf_sk_redirect_hash_proto;
extern const struct bpf_func_proto bpf_sk_redirect_map_proto;
-
+extern const struct bpf_func_proto bpf_spin_lock_proto;
+extern const struct bpf_func_proto bpf_spin_unlock_proto;
extern const struct bpf_func_proto bpf_get_local_storage_proto;
/* Shared helpers among cBPF and eBPF. */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 0620e418dde5..69f7a3449eda 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -148,6 +148,7 @@ struct bpf_verifier_state {
/* call stack tracking */
struct bpf_func_state *frame[MAX_CALL_FRAMES];
u32 curframe;
+ u32 active_spin_lock;
bool speculative;
};
diff --git a/include/linux/btf.h b/include/linux/btf.h
index 12502e25e767..455d31b55828 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -50,6 +50,7 @@ u32 btf_id(const struct btf *btf);
bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
const struct btf_member *m,
u32 expected_offset, u32 expected_size);
+int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t);
#ifdef CONFIG_BPF_SYSCALL
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 91c43884f295..30f9dfd40f13 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -2421,7 +2421,9 @@ union bpf_attr {
FN(map_peek_elem), \
FN(msg_push_data), \
FN(msg_pop_data), \
- FN(rc_pointer_rel),
+ FN(rc_pointer_rel), \
+ FN(spin_lock), \
+ FN(spin_unlock),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
@@ -3054,4 +3056,7 @@ struct bpf_line_info {
__u32 line_col;
};
+struct bpf_spin_lock {
+ __u32 val;
+};
#endif /* _UAPI__LINUX_BPF_H__ */
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 25632a75d630..d6d979910a2a 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -270,9 +270,10 @@ static int array_map_update_elem(struct bpf_map *map, void *key, void *value,
memcpy(this_cpu_ptr(array->pptrs[index & array->index_mask]),
value, map->value_size);
else
- memcpy(array->value +
- array->elem_size * (index & array->index_mask),
- value, map->value_size);
+ copy_map_value(map,
+ array->value +
+ array->elem_size * (index & array->index_mask),
+ value);
return 0;
}
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 022ef9ca1296..03b1a7a6195c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -355,6 +355,11 @@ static bool btf_type_is_struct(const struct btf_type *t)
return kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION;
}
+static bool __btf_type_is_struct(const struct btf_type *t)
+{
+ return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
+}
+
static bool btf_type_is_array(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
@@ -2045,6 +2050,43 @@ static void btf_struct_log(struct btf_verifier_env *env,
btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
}
+/* find 'struct bpf_spin_lock' in map value.
+ * return >= 0 offset if found
+ * and < 0 in case of error
+ */
+int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t)
+{
+ const struct btf_member *member;
+ u32 i, off = -ENOENT;
+
+ if (!__btf_type_is_struct(t))
+ return -EINVAL;
+
+ for_each_member(i, t, member) {
+ const struct btf_type *member_type = btf_type_by_id(btf,
+ member->type);
+ if (!__btf_type_is_struct(member_type))
+ continue;
+ if (member_type->size != sizeof(struct bpf_spin_lock))
+ continue;
+ if (strcmp(__btf_name_by_offset(btf, member_type->name_off),
+ "bpf_spin_lock"))
+ continue;
+ if (off != -ENOENT)
+ /* only one 'struct bpf_spin_lock' is allowed */
+ return -E2BIG;
+ off = btf_member_bit_offset(t, member);
+ if (off % 8)
+ /* valid C code cannot generate such BTF */
+ return -EINVAL;
+ off /= 8;
+ if (off % __alignof__(struct bpf_spin_lock))
+ /* valid struct bpf_spin_lock will be 4 byte aligned */
+ return -EINVAL;
+ }
+ return off;
+}
+
static void btf_struct_seq_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct seq_file *m)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 2a81b8af3748..2b44eca22ef8 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2050,6 +2050,8 @@ const struct bpf_func_proto bpf_map_delete_elem_proto __weak;
const struct bpf_func_proto bpf_map_push_elem_proto __weak;
const struct bpf_func_proto bpf_map_pop_elem_proto __weak;
const struct bpf_func_proto bpf_map_peek_elem_proto __weak;
+const struct bpf_func_proto bpf_spin_lock_proto __weak;
+const struct bpf_func_proto bpf_spin_unlock_proto __weak;
const struct bpf_func_proto bpf_get_prandom_u32_proto __weak;
const struct bpf_func_proto bpf_get_smp_processor_id_proto __weak;
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 4b7c76765d9d..48a41bf65e1b 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -770,6 +770,8 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
l_new = ERR_PTR(-ENOMEM);
goto dec_count;
}
+ check_and_init_map_lock(&htab->map,
+ l_new->key + round_up(key_size, 8));
}
memcpy(l_new->key, key, key_size);
@@ -792,7 +794,9 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
if (!prealloc)
htab_elem_set_ptr(l_new, key_size, pptr);
} else {
- memcpy(l_new->key + round_up(key_size, 8), value, size);
+ copy_map_value(&htab->map,
+ l_new->key + round_up(key_size, 8),
+ value);
}
l_new->hash = hash;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index a74972b07e74..2e98e4caf5aa 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -221,6 +221,63 @@ const struct bpf_func_proto bpf_get_current_comm_proto = {
.arg2_type = ARG_CONST_SIZE,
};
+#ifndef CONFIG_QUEUED_SPINLOCKS
+struct dumb_spin_lock {
+ atomic_t val;
+};
+#endif
+
+notrace BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
+{
+#if defined(CONFIG_SMP)
+#ifdef CONFIG_QUEUED_SPINLOCKS
+ struct qspinlock *qlock = (void *)lock;
+
+ BUILD_BUG_ON(sizeof(*qlock) != sizeof(*lock));
+ queued_spin_lock(qlock);
+#else
+ struct dumb_spin_lock *qlock = (void *)lock;
+
+ BUILD_BUG_ON(sizeof(*qlock) != sizeof(*lock));
+ do {
+ while (atomic_read(&qlock->val) != 0)
+ cpu_relax();
+ } while (atomic_cmpxchg(&qlock->val, 0, 1) != 0);
+#endif
+#endif
+ return 0;
+}
+
+const struct bpf_func_proto bpf_spin_lock_proto = {
+ .func = bpf_spin_lock,
+ .gpl_only = false,
+ .ret_type = RET_VOID,
+ .arg1_type = ARG_PTR_TO_SPIN_LOCK,
+};
+
+notrace BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
+{
+#if defined(CONFIG_SMP)
+#ifdef CONFIG_QUEUED_SPINLOCKS
+ struct qspinlock *qlock = (void *)lock;
+
+ queued_spin_unlock(qlock);
+#else
+ struct dumb_spin_lock *qlock = (void *)lock;
+
+ atomic_set(&qlock->val, 0);
+#endif
+#endif
+ return 0;
+}
+
+const struct bpf_func_proto bpf_spin_unlock_proto = {
+ .func = bpf_spin_unlock,
+ .gpl_only = false,
+ .ret_type = RET_VOID,
+ .arg1_type = ARG_PTR_TO_SPIN_LOCK,
+};
+
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c
index 99d243e1ad6e..920eff1b677b 100644
--- a/kernel/bpf/map_in_map.c
+++ b/kernel/bpf/map_in_map.c
@@ -36,6 +36,11 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
return ERR_PTR(-EINVAL);
}
+ if (map_value_has_spin_lock(inner_map)) {
+ fdput(f);
+ return ERR_PTR(-ENOTSUPP);
+ }
+
inner_map_meta = kzalloc(sizeof(*inner_map_meta), GFP_USER);
if (!inner_map_meta) {
fdput(f);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index b155cd17c1bd..ebf0a673cb83 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -463,7 +463,7 @@ int map_check_no_btf(const struct bpf_map *map,
return -ENOTSUPP;
}
-static int map_check_btf(const struct bpf_map *map, const struct btf *btf,
+static int map_check_btf(struct bpf_map *map, const struct btf *btf,
u32 btf_key_id, u32 btf_value_id)
{
const struct btf_type *key_type, *value_type;
@@ -478,6 +478,21 @@ static int map_check_btf(const struct bpf_map *map, const struct btf *btf,
if (!value_type || value_size != map->value_size)
return -EINVAL;
+ map->spin_lock_off = btf_find_spin_lock(btf, value_type);
+
+ if (map_value_has_spin_lock(map)) {
+ if (map->map_type != BPF_MAP_TYPE_HASH &&
+ map->map_type != BPF_MAP_TYPE_ARRAY)
+ return -ENOTSUPP;
+ if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >
+ map->value_size) {
+ WARN_ONCE(1,
+ "verifier bug spin_lock_off %d value_size %d\n",
+ map->spin_lock_off, map->value_size);
+ return -EFAULT;
+ }
+ }
+
if (map->ops->map_check_btf)
ret = map->ops->map_check_btf(map, btf, key_type, value_type);
@@ -542,6 +557,8 @@ static int map_create(union bpf_attr *attr)
map->btf = btf;
map->btf_key_type_id = attr->btf_key_type_id;
map->btf_value_type_id = attr->btf_value_type_id;
+ } else {
+ map->spin_lock_off = -EINVAL;
}
err = security_bpf_map_alloc(map);
@@ -740,7 +757,7 @@ static int map_lookup_elem(union bpf_attr *attr)
err = -ENOENT;
} else {
err = 0;
- memcpy(value, ptr, value_size);
+ copy_map_value(map, value, ptr);
}
rcu_read_unlock();
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8cfe39ef770f..f372904fa691 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -213,6 +213,7 @@ struct bpf_call_arg_meta {
s64 msize_smax_value;
u64 msize_umax_value;
int ptr_id;
+ int func_id;
};
static DEFINE_MUTEX(bpf_verifier_lock);
@@ -351,6 +352,12 @@ static bool reg_is_refcounted(const struct bpf_reg_state *reg)
return type_is_refcounted(reg->type);
}
+static bool reg_may_point_to_spin_lock(const struct bpf_reg_state *reg)
+{
+ return reg->type == PTR_TO_MAP_VALUE &&
+ map_value_has_spin_lock(reg->map_ptr);
+}
+
static bool reg_is_refcounted_or_null(const struct bpf_reg_state *reg)
{
return type_is_refcounted_or_null(reg->type);
@@ -712,6 +719,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
}
dst_state->speculative = src->speculative;
dst_state->curframe = src->curframe;
+ dst_state->active_spin_lock = src->active_spin_lock;
for (i = 0; i <= src->curframe; i++) {
dst = dst_state->frame[i];
if (!dst) {
@@ -1483,6 +1491,21 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,
if (err)
verbose(env, "R%d max value is outside of the array range\n",
regno);
+
+ if (map_value_has_spin_lock(reg->map_ptr)) {
+ u32 lock = reg->map_ptr->spin_lock_off;
+
+ /* if any part of struct bpf_spin_lock can be touched by
+ * load/store reject this program
+ */
+ if ((reg->smin_value + off <= lock &&
+ lock < reg->umax_value + off + size) ||
+ (reg->smin_value + off < lock + sizeof(struct bpf_spin_lock) &&
+ lock + sizeof(struct bpf_spin_lock) <= reg->umax_value + off + size)) {
+ verbose(env, "bpf_spin_lock cannot be accessed directly by load/store\n");
+ return -EACCES;
+ }
+ }
return err;
}
@@ -2192,6 +2215,91 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,
}
}
+/* Implementation details:
+ * bpf_map_lookup returns PTR_TO_MAP_VALUE_OR_NULL
+ * Two bpf_map_lookups (even with the same key) will have different reg->id.
+ * For traditional PTR_TO_MAP_VALUE the verifier clears reg->id after
+ * value_or_null->value transition, since the verifier only cares about
+ * the range of access to valid map value pointer and doesn't care about actual
+ * address of the map element.
+ * For maps with 'struct bpf_spin_lock' inside map value the verifier keeps
+ * reg->id > 0 after value_or_null->value transition. By doing so
+ * two bpf_map_lookups will be considered two different pointers that
+ * point to different bpf_spin_locks.
+ * The verifier allows taking only one bpf_spin_lock at a time to avoid
+ * dead-locks.
+ * Since only one bpf_spin_lock is allowed the checks are simpler than
+ * reg_is_refcounted() logic. The verifier needs to remember only
+ * one spin_lock instead of array of acquired_refs.
+ * cur_state->active_spin_lock remembers which map value element got locked
+ * and clears it after bpf_spin_unlock.
+ */
+static int process_spin_lock(struct bpf_verifier_env *env, int regno,
+ bool is_lock)
+{
+ struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[regno];
+ struct bpf_verifier_state *cur = env->cur_state;
+ bool is_const = tnum_is_const(reg->var_off);
+ struct bpf_map *map = reg->map_ptr;
+ u64 val = reg->var_off.value;
+
+ if (reg->type != PTR_TO_MAP_VALUE) {
+ verbose(env, "R%d is not a pointer to map_value\n", regno);
+ return -EINVAL;
+ }
+ if (!is_const) {
+ verbose(env,
+ "R%d doesn't have constant offset. bpf_spin_lock has to be at the constant offset\n",
+ regno);
+ return -EINVAL;
+ }
+ if (!map->btf) {
+ verbose(env,
+ "map '%s' has to have BTF in order to use bpf_spin_lock\n",
+ map->name);
+ return -EINVAL;
+ }
+ if (!map_value_has_spin_lock(map)) {
+ if (map->spin_lock_off == -E2BIG)
+ verbose(env,
+ "map '%s' has more than one 'struct bpf_spin_lock'\n",
+ map->name);
+ else if (map->spin_lock_off == -ENOENT)
+ verbose(env,
+ "map '%s' doesn't have 'struct bpf_spin_lock'\n",
+ map->name);
+ else
+ verbose(env,
+ "map '%s' is not a struct type or bpf_spin_lock is mangled\n",
+ map->name);
+ return -EINVAL;
+ }
+ if (map->spin_lock_off != val + reg->off) {
+ verbose(env, "off %lld doesn't point to 'struct bpf_spin_lock'\n",
+ val + reg->off);
+ return -EINVAL;
+ }
+ if (is_lock) {
+ if (cur->active_spin_lock) {
+ verbose(env,
+ "Locking two bpf_spin_locks are not allowed\n");
+ return -EINVAL;
+ }
+ cur->active_spin_lock = reg->id;
+ } else {
+ if (!cur->active_spin_lock) {
+ verbose(env, "bpf_spin_unlock without taking a lock\n");
+ return -EINVAL;
+ }
+ if (cur->active_spin_lock != reg->id) {
+ verbose(env, "bpf_spin_unlock of different lock\n");
+ return -EINVAL;
+ }
+ cur->active_spin_lock = 0;
+ }
+ return 0;
+}
+
static bool arg_type_is_mem_ptr(enum bpf_arg_type type)
{
return type == ARG_PTR_TO_MEM ||
@@ -2268,6 +2376,17 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno,
return -EFAULT;
}
meta->ptr_id = reg->id;
+ } else if (arg_type == ARG_PTR_TO_SPIN_LOCK) {
+ if (meta->func_id == BPF_FUNC_spin_lock) {
+ if (process_spin_lock(env, regno, true))
+ return -EACCES;
+ } else if (meta->func_id == BPF_FUNC_spin_unlock) {
+ if (process_spin_lock(env, regno, false))
+ return -EACCES;
+ } else {
+ verbose(env, "verifier internal error\n");
+ return -EFAULT;
+ }
} else if (arg_type_is_mem_ptr(arg_type)) {
expected_type = PTR_TO_STACK;
/* One exception here. In case function allows for NULL to be
@@ -2887,6 +3006,7 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn
return err;
}
+ meta.func_id = func_id;
/* check args */
err = check_func_arg(env, BPF_REG_1, fn->arg1_type, &meta);
if (err)
@@ -4344,7 +4464,8 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state,
} else if (reg->type == PTR_TO_SOCKET_OR_NULL) {
reg->type = PTR_TO_SOCKET;
}
- if (is_null || !reg_is_refcounted(reg)) {
+ if (is_null || !(reg_is_refcounted(reg) ||
+ reg_may_point_to_spin_lock(reg))) {
/* We don't need id from this point onwards anymore,
* thus we should better reset it, so that state
* pruning has chances to take effect.
@@ -4713,6 +4834,11 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
return err;
}
+ if (env->cur_state->active_spin_lock) {
+ verbose(env, "BPF_LD_[ABS|IND] cannot be used inside bpf_spin_lock-ed region\n");
+ return -EINVAL;
+ }
+
if (regs[BPF_REG_6].type != PTR_TO_CTX) {
verbose(env,
"at the time of BPF_LD_ABS|IND R6 != pointer to skb\n");
@@ -5448,8 +5574,11 @@ static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur,
case PTR_TO_MAP_VALUE:
/* If the new min/max/var_off satisfy the old ones and
* everything else matches, we are OK.
- * We don't care about the 'id' value, because nothing
- * uses it for PTR_TO_MAP_VALUE (only for ..._OR_NULL)
+ * 'id' is not compared, since it's only used for maps with
+ * bpf_spin_lock inside map element and in such cases if
+ * the rest of the prog is valid for one map element then
+ * it's valid for all map elements regardless of the key
+ * used in bpf_map_lookup()
*/
return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
range_within(rold, rcur) &&
@@ -5652,6 +5781,9 @@ static bool states_equal(struct bpf_verifier_env *env,
if (old->speculative && !cur->speculative)
return false;
+ if (old->active_spin_lock != cur->active_spin_lock)
+ return false;
+
/* for states to be equal callsites have to be the same
* and all frame states need to be equivalent
*/
@@ -6069,6 +6201,12 @@ static int do_check(struct bpf_verifier_env *env)
return -EINVAL;
}
+ if (env->cur_state->active_spin_lock &&
+ (insn->src_reg == BPF_PSEUDO_CALL ||
+ insn->imm != BPF_FUNC_spin_unlock)) {
+ verbose(env, "function calls are not allowed while holding a lock\n");
+ return -EINVAL;
+ }
if (insn->src_reg == BPF_PSEUDO_CALL)
err = check_func_call(env, insn, &env->insn_idx);
else
@@ -6097,6 +6235,11 @@ static int do_check(struct bpf_verifier_env *env)
return -EINVAL;
}
+ if (env->cur_state->active_spin_lock) {
+ verbose(env, "bpf_spin_unlock is missing\n");
+ return -EINVAL;
+ }
+
if (state->curframe) {
/* exit from nested function */
env->prev_insn_idx = env->insn_idx;
diff --git a/net/core/filter.c b/net/core/filter.c
index 2b3b436ef545..24a5d874d156 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -5306,10 +5306,20 @@ bpf_base_func_proto(enum bpf_func_id func_id)
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
+ default:
+ break;
+ }
+
+ if (!capable(CAP_SYS_ADMIN))
+ return NULL;
+
+ switch (func_id) {
+ case BPF_FUNC_spin_lock:
+ return &bpf_spin_lock_proto;
+ case BPF_FUNC_spin_unlock:
+ return &bpf_spin_unlock_proto;
case BPF_FUNC_trace_printk:
- if (capable(CAP_SYS_ADMIN))
- return bpf_get_trace_printk_proto();
- /* else: fall through */
+ return bpf_get_trace_printk_proto();
default:
return NULL;
}
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 7/9] tools/bpf: sync uapi/bpf.h
From: Alexei Starovoitov @ 2019-01-24 4:14 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
add BPF_F_LOCK definition to tools/include/uapi/linux/bpf.h
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/include/uapi/linux/bpf.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 30f9dfd40f13..c23eaa284a45 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -266,6 +266,7 @@ enum bpf_attach_type {
#define BPF_ANY 0 /* create new element or update existing */
#define BPF_NOEXIST 1 /* create new element if it didn't exist */
#define BPF_EXIST 2 /* update existing element */
+#define BPF_F_LOCK 4 /* spin_lock-ed map_lookup/map_update */
/* flags for BPF_MAP_CREATE command */
#define BPF_F_NO_PREALLOC (1U << 0)
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 3/9] tools/bpf: sync include/uapi/linux/bpf.h
From: Alexei Starovoitov @ 2019-01-24 4:13 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
sync bpf.h
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/include/uapi/linux/bpf.h | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 91c43884f295..30f9dfd40f13 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -2421,7 +2421,9 @@ union bpf_attr {
FN(map_peek_elem), \
FN(msg_push_data), \
FN(msg_pop_data), \
- FN(rc_pointer_rel),
+ FN(rc_pointer_rel), \
+ FN(spin_lock), \
+ FN(spin_unlock),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
@@ -3054,4 +3056,7 @@ struct bpf_line_info {
__u32 line_col;
};
+struct bpf_spin_lock {
+ __u32 val;
+};
#endif /* _UAPI__LINUX_BPF_H__ */
--
2.17.1
^ permalink raw reply related
* [PATCH v4 bpf-next 5/9] selftests/bpf: add bpf_spin_lock C test
From: Alexei Starovoitov @ 2019-01-24 4:13 UTC (permalink / raw)
To: davem; +Cc: daniel, jakub.kicinski, netdev, kernel-team
In-Reply-To: <20190124041403.2100609-1-ast@kernel.org>
add bpf_spin_lock C based test that requires latest llvm with BTF support
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 2 +-
tools/testing/selftests/bpf/bpf_helpers.h | 4 +
tools/testing/selftests/bpf/test_progs.c | 43 +++++++-
tools/testing/selftests/bpf/test_spin_lock.c | 108 +++++++++++++++++++
4 files changed, 155 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/bpf/test_spin_lock.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 70229de510f5..fcfda51406f9 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -39,7 +39,7 @@ TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test
get_cgroup_id_kern.o socket_cookie_prog.o test_select_reuseport_kern.o \
test_skb_cgroup_id_kern.o bpf_flow.o netcnt_prog.o \
test_sk_lookup_kern.o test_xdp_vlan.o test_queue_map.o test_stack_map.o \
- xdp_dummy.o test_map_in_map.o
+ xdp_dummy.o test_map_in_map.o test_spin_lock.o
# Order correspond to 'make run_tests' order
TEST_PROGS := test_kmod.sh \
diff --git a/tools/testing/selftests/bpf/bpf_helpers.h b/tools/testing/selftests/bpf/bpf_helpers.h
index 6c77cf7bedce..6a0ce0f055c5 100644
--- a/tools/testing/selftests/bpf/bpf_helpers.h
+++ b/tools/testing/selftests/bpf/bpf_helpers.h
@@ -172,6 +172,10 @@ static int (*bpf_skb_vlan_pop)(void *ctx) =
(void *) BPF_FUNC_skb_vlan_pop;
static int (*bpf_rc_pointer_rel)(void *ctx, int rel_x, int rel_y) =
(void *) BPF_FUNC_rc_pointer_rel;
+static void (*bpf_spin_lock)(struct bpf_spin_lock *lock) =
+ (void *) BPF_FUNC_spin_lock;
+static void (*bpf_spin_unlock)(struct bpf_spin_lock *lock) =
+ (void *) BPF_FUNC_spin_unlock;
/* llvm builtin functions that eBPF C program may use to
* emit BPF_LD_ABS and BPF_LD_IND instructions
diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c
index 126fc624290d..6425e95c3f16 100644
--- a/tools/testing/selftests/bpf/test_progs.c
+++ b/tools/testing/selftests/bpf/test_progs.c
@@ -28,7 +28,7 @@ typedef __u16 __sum16;
#include <sys/wait.h>
#include <sys/types.h>
#include <fcntl.h>
-
+#include <pthread.h>
#include <linux/bpf.h>
#include <linux/err.h>
#include <bpf/bpf.h>
@@ -1882,6 +1882,46 @@ static void test_queue_stack_map(int type)
bpf_object__close(obj);
}
+static void *parallel_bpf_prog_test_run(void *arg)
+{
+ __u32 duration, retval;
+ int err, prog_fd = *(u32 *) arg;
+
+ err = bpf_prog_test_run(prog_fd, 10000, &pkt_v4, sizeof(pkt_v4),
+ NULL, NULL, &retval, &duration);
+ CHECK(err || retval, "",
+ "err %d errno %d retval %d duration %d\n",
+ err, errno, retval, duration);
+ pthread_exit(arg);
+}
+
+static void test_spin_lock(void)
+{
+ const char *file = "./test_spin_lock.o";
+ pthread_t thread_id[4];
+ struct bpf_object *obj;
+ int prog_fd;
+ int err = 0, i;
+ void *ret;
+
+ err = bpf_prog_load(file, BPF_PROG_TYPE_CGROUP_SKB, &obj, &prog_fd);
+ if (err) {
+ printf("test_spin_lock:bpf_prog_load errno %d\n", errno);
+ goto close_prog;
+ }
+ for (i = 0; i < 4; i++)
+ assert(pthread_create(&thread_id[i], NULL,
+ ¶llel_bpf_prog_test_run, &prog_fd) == 0);
+ for (i = 0; i < 4; i++)
+ assert(pthread_join(thread_id[i], &ret) == 0 &&
+ ret == (void *)&prog_fd);
+ goto close_prog_noerr;
+close_prog:
+ error_cnt++;
+close_prog_noerr:
+ bpf_object__close(obj);
+}
+
int main(void)
{
srand(time(NULL));
@@ -1909,6 +1949,7 @@ int main(void)
test_reference_tracking();
test_queue_stack_map(QUEUE);
test_queue_stack_map(STACK);
+ test_spin_lock();
printf("Summary: %d PASSED, %d FAILED\n", pass_cnt, error_cnt);
return error_cnt ? EXIT_FAILURE : EXIT_SUCCESS;
diff --git a/tools/testing/selftests/bpf/test_spin_lock.c b/tools/testing/selftests/bpf/test_spin_lock.c
new file mode 100644
index 000000000000..40f904312090
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_spin_lock.c
@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (c) 2019 Facebook
+#include <linux/bpf.h>
+#include <linux/version.h>
+#include "bpf_helpers.h"
+
+struct hmap_elem {
+ volatile int cnt;
+ struct bpf_spin_lock lock;
+ int test_padding;
+};
+
+struct bpf_map_def SEC("maps") hmap = {
+ .type = BPF_MAP_TYPE_HASH,
+ .key_size = sizeof(int),
+ .value_size = sizeof(struct hmap_elem),
+ .max_entries = 1,
+};
+
+BPF_ANNOTATE_KV_PAIR(hmap, int, struct hmap_elem);
+
+
+struct cls_elem {
+ struct bpf_spin_lock lock;
+ volatile int cnt;
+};
+
+struct bpf_map_def SEC("maps") cls_map = {
+ .type = BPF_MAP_TYPE_CGROUP_STORAGE,
+ .key_size = sizeof(struct bpf_cgroup_storage_key),
+ .value_size = sizeof(struct cls_elem),
+};
+
+BPF_ANNOTATE_KV_PAIR(cls_map, struct bpf_cgroup_storage_key,
+ struct cls_elem);
+
+struct bpf_vqueue {
+ struct bpf_spin_lock lock;
+ /* 4 byte hole */
+ unsigned long long lasttime;
+ int credit;
+ unsigned int rate;
+};
+
+struct bpf_map_def SEC("maps") vqueue = {
+ .type = BPF_MAP_TYPE_ARRAY,
+ .key_size = sizeof(int),
+ .value_size = sizeof(struct bpf_vqueue),
+ .max_entries = 1,
+};
+
+BPF_ANNOTATE_KV_PAIR(vqueue, int, struct bpf_vqueue);
+#define CREDIT_PER_NS(delta, rate) (((delta) * rate) >> 20)
+
+SEC("spin_lock_demo")
+int bpf_sping_lock_test(struct __sk_buff *skb)
+{
+ volatile int credit = 0, max_credit = 100, pkt_len = 64;
+ struct hmap_elem zero = {}, *val;
+ unsigned long long curtime;
+ struct bpf_vqueue *q;
+ struct cls_elem *cls;
+ int key = 0;
+ int err = 0;
+
+ val = bpf_map_lookup_elem(&hmap, &key);
+ if (!val) {
+ bpf_map_update_elem(&hmap, &key, &zero, 0);
+ val = bpf_map_lookup_elem(&hmap, &key);
+ if (!val) {
+ err = 1;
+ goto err;
+ }
+ }
+ /* spin_lock in hash map run time test */
+ bpf_spin_lock(&val->lock);
+ if (val->cnt)
+ val->cnt--;
+ else
+ val->cnt++;
+ if (val->cnt != 0 && val->cnt != 1)
+ err = 1;
+ bpf_spin_unlock(&val->lock);
+
+ /* spin_lock in array. virtual queue demo */
+ q = bpf_map_lookup_elem(&vqueue, &key);
+ if (!q)
+ goto err;
+ curtime = bpf_ktime_get_ns();
+ bpf_spin_lock(&q->lock);
+ q->credit += CREDIT_PER_NS(curtime - q->lasttime, q->rate);
+ q->lasttime = curtime;
+ if (q->credit > max_credit)
+ q->credit = max_credit;
+ q->credit -= pkt_len;
+ credit = q->credit;
+ bpf_spin_unlock(&q->lock);
+
+ /* spin_lock in cgroup local storage */
+ cls = bpf_get_local_storage(&cls_map, 0);
+ bpf_spin_lock(&cls->lock);
+ cls->cnt++;
+ bpf_spin_unlock(&cls->lock);
+
+err:
+ return err;
+}
+char _license[] SEC("license") = "GPL";
--
2.17.1
^ permalink raw reply related
* Re: [PATCH net-next V4 5/5] vhost: access vq metadata through kernel virtual address
From: Jason Wang @ 2019-01-24 4:11 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: virtualization, netdev, linux-kernel, kvm
In-Reply-To: <335ba55b-087f-4b35-6311-540070b9647f@redhat.com>
On 2019/1/24 下午12:07, Jason Wang wrote:
>
> On 2019/1/23 下午10:08, Michael S. Tsirkin wrote:
>> On Wed, Jan 23, 2019 at 05:55:57PM +0800, Jason Wang wrote:
>>> It was noticed that the copy_user() friends that was used to access
>>> virtqueue metdata tends to be very expensive for dataplane
>>> implementation like vhost since it involves lots of software checks,
>>> speculation barrier, hardware feature toggling (e.g SMAP). The
>>> extra cost will be more obvious when transferring small packets since
>>> the time spent on metadata accessing become more significant.
>>>
>>> This patch tries to eliminate those overheads by accessing them
>>> through kernel virtual address by vmap(). To make the pages can be
>>> migrated, instead of pinning them through GUP, we use MMU notifiers to
>>> invalidate vmaps and re-establish vmaps during each round of metadata
>>> prefetching if necessary. For devices that doesn't use metadata
>>> prefetching, the memory accessors fallback to normal copy_user()
>>> implementation gracefully. The invalidation was synchronized with
>>> datapath through vq mutex, and in order to avoid hold vq mutex during
>>> range checking, MMU notifier was teared down when trying to modify vq
>>> metadata.
>>>
>>> Another thing is kernel lacks efficient solution for tracking dirty
>>> pages by vmap(), this will lead issues if vhost is using file backed
>>> memory which needs care of writeback. This patch solves this issue by
>>> just skipping the vma that is file backed and fallback to normal
>>> copy_user() friends. This might introduce some overheads for file
>>> backed users but consider this use case is rare we could do
>>> optimizations on top.
>>>
>>> Note that this was only done when device IOTLB is not enabled. We
>>> could use similar method to optimize it in the future.
>>>
>>> Tests shows at most about 22% improvement on TX PPS when using
>>> virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell:
>>>
>>> SMAP on | SMAP off
>>> Before: 5.0Mpps | 6.6Mpps
>>> After: 6.1Mpps | 7.4Mpps
>>>
>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>
>> So this is the bulk of the change.
>> Threee things that I need to look into
>> - Are there any security issues with bypassing the speculation barrier
>> that is normally present after access_ok?
>
>
> If we can make sure the bypassing was only used in a kthread (vhost),
> it should be fine I think.
>
>
>> - How hard does the special handling for
>> file backed storage make testing?
>
>
> It's as simple as un-commenting vhost_can_vmap()? Or I can try to hack
> qemu or dpdk to test this.
>
>
>> On the one hand we could add a module parameter to
>> force copy to/from user. on the other that's
>> another configuration we need to support.
>
>
> That sounds sub-optimal since it leave the choice to users.
>
>
>> But iotlb is not using vmap, so maybe that's enough
>> for testing.
>> - How hard is it to figure out which mode uses which code.
It's as simple as tracing __get_user() usage in vhost process?
Thanks
>>
>>
>>
>> Meanwhile, could you pls post data comparing this last patch with the
>> below? This removes the speculation barrier replacing it with a
>> (useless but at least more lightweight) data dependency.
>
>
> SMAP off
>
> Your patch: 7.2MPPs
>
> vmap: 7.4Mpps
>
> I don't test SMAP on, since it will be much slow for sure.
>
> Thanks
>
>
>>
>> Thanks!
>>
>>
>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>> index bac939af8dbb..352ee7e14476 100644
>> --- a/drivers/vhost/vhost.c
>> +++ b/drivers/vhost/vhost.c
>> @@ -739,7 +739,7 @@ static int vhost_copy_to_user(struct
>> vhost_virtqueue *vq, void __user *to,
>> int ret;
>> if (!vq->iotlb)
>> - return __copy_to_user(to, from, size);
>> + return copy_to_user(to, from, size);
>> else {
>> /* This function should be called after iotlb
>> * prefetch, which means we're sure that all vq
>> @@ -752,7 +752,7 @@ static int vhost_copy_to_user(struct
>> vhost_virtqueue *vq, void __user *to,
>> VHOST_ADDR_USED);
>> if (uaddr)
>> - return __copy_to_user(uaddr, from, size);
>> + return copy_to_user(uaddr, from, size);
>> ret = translate_desc(vq, (u64)(uintptr_t)to, size,
>> vq->iotlb_iov,
>> ARRAY_SIZE(vq->iotlb_iov),
>> @@ -774,7 +774,7 @@ static int vhost_copy_from_user(struct
>> vhost_virtqueue *vq, void *to,
>> int ret;
>> if (!vq->iotlb)
>> - return __copy_from_user(to, from, size);
>> + return copy_from_user(to, from, size);
>> else {
>> /* This function should be called after iotlb
>> * prefetch, which means we're sure that vq
>> @@ -787,7 +787,7 @@ static int vhost_copy_from_user(struct
>> vhost_virtqueue *vq, void *to,
>> struct iov_iter f;
>> if (uaddr)
>> - return __copy_from_user(to, uaddr, size);
>> + return copy_from_user(to, uaddr, size);
>> ret = translate_desc(vq, (u64)(uintptr_t)from, size,
>> vq->iotlb_iov,
>> ARRAY_SIZE(vq->iotlb_iov),
>> @@ -855,13 +855,13 @@ static inline void __user
>> *__vhost_get_user(struct vhost_virtqueue *vq,
>> ({ \
>> int ret = -EFAULT; \
>> if (!vq->iotlb) { \
>> - ret = __put_user(x, ptr); \
>> + ret = put_user(x, ptr); \
>> } else { \
>> __typeof__(ptr) to = \
>> (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
>> sizeof(*ptr), VHOST_ADDR_USED); \
>> if (to != NULL) \
>> - ret = __put_user(x, to); \
>> + ret = put_user(x, to); \
>> else \
>> ret = -EFAULT; \
>> } \
>> @@ -872,14 +872,14 @@ static inline void __user
>> *__vhost_get_user(struct vhost_virtqueue *vq,
>> ({ \
>> int ret; \
>> if (!vq->iotlb) { \
>> - ret = __get_user(x, ptr); \
>> + ret = get_user(x, ptr); \
>> } else { \
>> __typeof__(ptr) from = \
>> (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
>> sizeof(*ptr), \
>> type); \
>> if (from != NULL) \
>> - ret = __get_user(x, from); \
>> + ret = get_user(x, from); \
>> else \
>> ret = -EFAULT; \
>> } \
^ permalink raw reply
* Re: [PATCH net-next V4 5/5] vhost: access vq metadata through kernel virtual address
From: Jason Wang @ 2019-01-24 4:07 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: virtualization, netdev, linux-kernel, kvm
In-Reply-To: <20190123085821-mutt-send-email-mst@kernel.org>
On 2019/1/23 下午10:08, Michael S. Tsirkin wrote:
> On Wed, Jan 23, 2019 at 05:55:57PM +0800, Jason Wang wrote:
>> It was noticed that the copy_user() friends that was used to access
>> virtqueue metdata tends to be very expensive for dataplane
>> implementation like vhost since it involves lots of software checks,
>> speculation barrier, hardware feature toggling (e.g SMAP). The
>> extra cost will be more obvious when transferring small packets since
>> the time spent on metadata accessing become more significant.
>>
>> This patch tries to eliminate those overheads by accessing them
>> through kernel virtual address by vmap(). To make the pages can be
>> migrated, instead of pinning them through GUP, we use MMU notifiers to
>> invalidate vmaps and re-establish vmaps during each round of metadata
>> prefetching if necessary. For devices that doesn't use metadata
>> prefetching, the memory accessors fallback to normal copy_user()
>> implementation gracefully. The invalidation was synchronized with
>> datapath through vq mutex, and in order to avoid hold vq mutex during
>> range checking, MMU notifier was teared down when trying to modify vq
>> metadata.
>>
>> Another thing is kernel lacks efficient solution for tracking dirty
>> pages by vmap(), this will lead issues if vhost is using file backed
>> memory which needs care of writeback. This patch solves this issue by
>> just skipping the vma that is file backed and fallback to normal
>> copy_user() friends. This might introduce some overheads for file
>> backed users but consider this use case is rare we could do
>> optimizations on top.
>>
>> Note that this was only done when device IOTLB is not enabled. We
>> could use similar method to optimize it in the future.
>>
>> Tests shows at most about 22% improvement on TX PPS when using
>> virtio-user + vhost_net + xdp1 + TAP on 2.6GHz Broadwell:
>>
>> SMAP on | SMAP off
>> Before: 5.0Mpps | 6.6Mpps
>> After: 6.1Mpps | 7.4Mpps
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>
> So this is the bulk of the change.
> Threee things that I need to look into
> - Are there any security issues with bypassing the speculation barrier
> that is normally present after access_ok?
If we can make sure the bypassing was only used in a kthread (vhost), it
should be fine I think.
> - How hard does the special handling for
> file backed storage make testing?
It's as simple as un-commenting vhost_can_vmap()? Or I can try to hack
qemu or dpdk to test this.
> On the one hand we could add a module parameter to
> force copy to/from user. on the other that's
> another configuration we need to support.
That sounds sub-optimal since it leave the choice to users.
> But iotlb is not using vmap, so maybe that's enough
> for testing.
> - How hard is it to figure out which mode uses which code.
>
>
>
> Meanwhile, could you pls post data comparing this last patch with the
> below? This removes the speculation barrier replacing it with a
> (useless but at least more lightweight) data dependency.
SMAP off
Your patch: 7.2MPPs
vmap: 7.4Mpps
I don't test SMAP on, since it will be much slow for sure.
Thanks
>
> Thanks!
>
>
> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> index bac939af8dbb..352ee7e14476 100644
> --- a/drivers/vhost/vhost.c
> +++ b/drivers/vhost/vhost.c
> @@ -739,7 +739,7 @@ static int vhost_copy_to_user(struct vhost_virtqueue *vq, void __user *to,
> int ret;
>
> if (!vq->iotlb)
> - return __copy_to_user(to, from, size);
> + return copy_to_user(to, from, size);
> else {
> /* This function should be called after iotlb
> * prefetch, which means we're sure that all vq
> @@ -752,7 +752,7 @@ static int vhost_copy_to_user(struct vhost_virtqueue *vq, void __user *to,
> VHOST_ADDR_USED);
>
> if (uaddr)
> - return __copy_to_user(uaddr, from, size);
> + return copy_to_user(uaddr, from, size);
>
> ret = translate_desc(vq, (u64)(uintptr_t)to, size, vq->iotlb_iov,
> ARRAY_SIZE(vq->iotlb_iov),
> @@ -774,7 +774,7 @@ static int vhost_copy_from_user(struct vhost_virtqueue *vq, void *to,
> int ret;
>
> if (!vq->iotlb)
> - return __copy_from_user(to, from, size);
> + return copy_from_user(to, from, size);
> else {
> /* This function should be called after iotlb
> * prefetch, which means we're sure that vq
> @@ -787,7 +787,7 @@ static int vhost_copy_from_user(struct vhost_virtqueue *vq, void *to,
> struct iov_iter f;
>
> if (uaddr)
> - return __copy_from_user(to, uaddr, size);
> + return copy_from_user(to, uaddr, size);
>
> ret = translate_desc(vq, (u64)(uintptr_t)from, size, vq->iotlb_iov,
> ARRAY_SIZE(vq->iotlb_iov),
> @@ -855,13 +855,13 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq,
> ({ \
> int ret = -EFAULT; \
> if (!vq->iotlb) { \
> - ret = __put_user(x, ptr); \
> + ret = put_user(x, ptr); \
> } else { \
> __typeof__(ptr) to = \
> (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
> sizeof(*ptr), VHOST_ADDR_USED); \
> if (to != NULL) \
> - ret = __put_user(x, to); \
> + ret = put_user(x, to); \
> else \
> ret = -EFAULT; \
> } \
> @@ -872,14 +872,14 @@ static inline void __user *__vhost_get_user(struct vhost_virtqueue *vq,
> ({ \
> int ret; \
> if (!vq->iotlb) { \
> - ret = __get_user(x, ptr); \
> + ret = get_user(x, ptr); \
> } else { \
> __typeof__(ptr) from = \
> (__typeof__(ptr)) __vhost_get_user(vq, ptr, \
> sizeof(*ptr), \
> type); \
> if (from != NULL) \
> - ret = __get_user(x, from); \
> + ret = get_user(x, from); \
> else \
> ret = -EFAULT; \
> } \
^ permalink raw reply
* Re: [PATCH bpf-next 2/3] bpf: add BPF_PROG_TEST_RUN support for flow dissector
From: Alexei Starovoitov @ 2019-01-24 3:57 UTC (permalink / raw)
To: Stanislav Fomichev; +Cc: netdev, davem, ast, daniel
In-Reply-To: <20190122212315.137291-3-sdf@google.com>
On Tue, Jan 22, 2019 at 01:23:14PM -0800, Stanislav Fomichev wrote:
> The input is packet data, the output is struct bpf_flow_key. This should
> make it easy to test flow dissector programs without elaborate
> setup.
>
> Signed-off-by: Stanislav Fomichev <sdf@google.com>
> ---
> include/linux/bpf.h | 3 ++
> net/bpf/test_run.c | 75 ++++++++++++++++++++++++++++++++++++++++++++-
> net/core/filter.c | 1 +
> 3 files changed, 78 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index e734f163bd0b..701ef954a258 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -397,6 +397,9 @@ int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
> union bpf_attr __user *uattr);
> int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
> union bpf_attr __user *uattr);
> +int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,
> + const union bpf_attr *kattr,
> + union bpf_attr __user *uattr);
>
> /* an array of programs to be executed under rcu_lock.
> *
> diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c
> index fa2644d276ef..ecad72885f23 100644
> --- a/net/bpf/test_run.c
> +++ b/net/bpf/test_run.c
> @@ -16,12 +16,26 @@
> static __always_inline u32 bpf_test_run_one(struct bpf_prog *prog, void *ctx,
> struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE])
> {
> + struct bpf_skb_data_end *cb;
> + struct sk_buff *skb;
> u32 ret;
>
> preempt_disable();
> rcu_read_lock();
> bpf_cgroup_storage_set(storage);
> - ret = BPF_PROG_RUN(prog, ctx);
> +
> + switch (prog->type) {
> + case BPF_PROG_TYPE_FLOW_DISSECTOR:
> + skb = (struct sk_buff *)ctx;
> + cb = (struct bpf_skb_data_end *)skb->cb;
> + ret = __skb_flow_bpf_dissect(prog, ctx, &flow_keys_dissector,
> + cb->qdisc_cb.flow_keys);
> + break;
> + default:
> + ret = BPF_PROG_RUN(prog, ctx);
> + break;
> + }
What is the benefit of this minimal code reuse?
It seems to me bpf_test_run_one() gets slower for all,
since prog type needs to be checked before every prog run.
The point of bpf_prog_ops->test_run was to avoid this overhead.
Are you somehow expecting flow_dissector prog using cgroup local storage?
I don't think that's possible.
^ permalink raw reply
* Re: [PATCH bpf-next] selftests/bpf: suppress readelf stderr when probing for BTF support
From: Alexei Starovoitov @ 2019-01-24 3:47 UTC (permalink / raw)
To: Stanislav Fomichev; +Cc: Stanislav Fomichev, netdev, davem, ast, daniel
In-Reply-To: <20190124031507.GC26773@mini-arch>
On Wed, Jan 23, 2019 at 07:15:07PM -0800, Stanislav Fomichev wrote:
> On 01/23, Alexei Starovoitov wrote:
> > On Wed, Jan 23, 2019 at 09:24:35AM -0800, Stanislav Fomichev wrote:
> > > Before:
> > > $ make -s -C tools/testing/selftests/bpf
> > > readelf: Error: Missing knowledge of 32-bit reloc types used in DWARF
> > > sections of machine number 247
> > > readelf: Warning: unable to apply unsupported reloc type 10 to section
> > > .debug_info
> > > readelf: Warning: unable to apply unsupported reloc type 1 to section
> > > .debug_info
> > > readelf: Warning: unable to apply unsupported reloc type 10 to section
> > > .debug_info
> > >
> > > After:
> > > $ make -s -C tools/testing/selftests/bpf
> > >
> > > Signed-off-by: Stanislav Fomichev <sdf@google.com>
> > > ---
> > > tools/testing/selftests/bpf/Makefile | 2 +-
> > > 1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> > > index 70229de510f5..ff3cab5525c1 100644
> > > --- a/tools/testing/selftests/bpf/Makefile
> > > +++ b/tools/testing/selftests/bpf/Makefile
> > > @@ -131,7 +131,7 @@ BTF_PAHOLE_PROBE := $(shell $(BTF_PAHOLE) --help 2>&1 | grep BTF)
> > > BTF_OBJCOPY_PROBE := $(shell $(LLVM_OBJCOPY) --help 2>&1 | grep -i 'usage.*llvm')
> > > BTF_LLVM_PROBE := $(shell echo "int main() { return 0; }" | \
> > > $(CLANG) -target bpf -O2 -g -c -x c - -o ./llvm_btf_verify.o; \
> > > - readelf -S ./llvm_btf_verify.o | grep BTF; \
> > > + readelf -S ./llvm_btf_verify.o 2>/dev/null | grep BTF; \
> > > /bin/rm -f ./llvm_btf_verify.o)
> >
> > Silencing the warning may hide real issue later.
> > I'd rather fix whatever is necessary in readelf.
> > Does it use bfd? if so bpf support for bfd was practically complete.
> Are you concerned about the situation where llvm has btf support, but
> readelf (binutils) is unaware of it? In this case, we should probably
> use llvm's readelf (llvm-readelf binary). We already have a precedent of
> using llvm's tools with LLVM_OBJCOPY. Let me try it out tomorrow.
llvm-readelf or llvm-objdump are fine options too.
^ permalink raw reply
* Re: [v2 PATCH] rhashtable: Still do rehash when we get EEXIST
From: Josh Elsasser @ 2019-01-24 3:40 UTC (permalink / raw)
To: Herbert Xu; +Cc: David S . Miller, Thomas Graf, netdev, linux-kernel
In-Reply-To: <20190124030841.n4jtsqka5zji3e62@gondor.apana.org.au>
On Jan 23, 2019, at 7:08 PM, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> Thanks for catching this!
>
> Although I think we should fix this in a different way. The problem
> here is that the shrink cannot proceed because there was a previous
> rehash that is still incomplete. We should wait for its completion
> and then reattempt a shrinnk should it still be necessary.
>
> So something like this:
SGTM.
I can't test this right now because our VM server's down after a power
outage this evening, but I tried a similar patch that swallowed the
-EEXIST err and even with that oversight the hashtable dodged the
reschedule loop.
- Josh
^ permalink raw reply
* Re: [PATCH bpf-next] selftests/bpf: suppress readelf stderr when probing for BTF support
From: Stanislav Fomichev @ 2019-01-24 3:15 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: Stanislav Fomichev, netdev, davem, ast, daniel
In-Reply-To: <20190124014527.kb7gpsrvz2zh2jhv@ast-mbp.dhcp.thefacebook.com>
On 01/23, Alexei Starovoitov wrote:
> On Wed, Jan 23, 2019 at 09:24:35AM -0800, Stanislav Fomichev wrote:
> > Before:
> > $ make -s -C tools/testing/selftests/bpf
> > readelf: Error: Missing knowledge of 32-bit reloc types used in DWARF
> > sections of machine number 247
> > readelf: Warning: unable to apply unsupported reloc type 10 to section
> > .debug_info
> > readelf: Warning: unable to apply unsupported reloc type 1 to section
> > .debug_info
> > readelf: Warning: unable to apply unsupported reloc type 10 to section
> > .debug_info
> >
> > After:
> > $ make -s -C tools/testing/selftests/bpf
> >
> > Signed-off-by: Stanislav Fomichev <sdf@google.com>
> > ---
> > tools/testing/selftests/bpf/Makefile | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
> > index 70229de510f5..ff3cab5525c1 100644
> > --- a/tools/testing/selftests/bpf/Makefile
> > +++ b/tools/testing/selftests/bpf/Makefile
> > @@ -131,7 +131,7 @@ BTF_PAHOLE_PROBE := $(shell $(BTF_PAHOLE) --help 2>&1 | grep BTF)
> > BTF_OBJCOPY_PROBE := $(shell $(LLVM_OBJCOPY) --help 2>&1 | grep -i 'usage.*llvm')
> > BTF_LLVM_PROBE := $(shell echo "int main() { return 0; }" | \
> > $(CLANG) -target bpf -O2 -g -c -x c - -o ./llvm_btf_verify.o; \
> > - readelf -S ./llvm_btf_verify.o | grep BTF; \
> > + readelf -S ./llvm_btf_verify.o 2>/dev/null | grep BTF; \
> > /bin/rm -f ./llvm_btf_verify.o)
>
> Silencing the warning may hide real issue later.
> I'd rather fix whatever is necessary in readelf.
> Does it use bfd? if so bpf support for bfd was practically complete.
Are you concerned about the situation where llvm has btf support, but
readelf (binutils) is unaware of it? In this case, we should probably
use llvm's readelf (llvm-readelf binary). We already have a precedent of
using llvm's tools with LLVM_OBJCOPY. Let me try it out tomorrow.
^ permalink raw reply
* [v2 PATCH] rhashtable: Still do rehash when we get EEXIST
From: Herbert Xu @ 2019-01-24 3:08 UTC (permalink / raw)
To: Josh Elsasser; +Cc: David S . Miller, josh, Thomas Graf, netdev, linux-kernel
In-Reply-To: <20190123211758.104275-1-jelsasser@appneta.com>
On Wed, Jan 23, 2019 at 01:17:58PM -0800, Josh Elsasser wrote:
> When running workloads with large bursts of fragmented packets, we've seen
> a few machines stuck returning -EEXIST from rht_shrink() and endlessly
> rescheduling their hash table's deferred work, pegging a CPU core.
>
> Root cause is commit da20420f83ea ("rhashtable: Add nested tables"), which
> stops ignoring the return code of rhashtable_shrink() and the reallocs
> used to grow the hashtable. This uncovers a bug in the shrink logic where
> "needs to shrink" check runs against the last table but the actual shrink
> operation runs on the first bucket_table in the hashtable (see below):
>
> +-------+ +--------------+ +---------------+
> | ht | | "first" tbl | | "last" tbl |
> | - tbl ---> | - future_tbl ---------> | - future_tbl ---> NULL
> +-------+ +--------------+ +---------------+
> ^^^ ^^^
> used by rhashtable_shrink() used by rht_shrink_below_30()
>
> A rehash then stalls out when both the last table needs to shrink, the
> first table has more elements than the target size, but rht_shrink() hits
> a non-NULL future_tbl and returns -EEXIST. This skips the item rehashing
> and kicks off a reschedule loop, as no forward progress can be made while
> the rhashtable needs to shrink.
>
> Extend rhashtable_shrink() with a "tbl" param to avoid endless exit-and-
> reschedules after hitting the EEXIST, allowing it to check a future_tbl
> pointer that can actually be non-NULL and make forward progress when the
> hashtable needs to shrink.
>
> Fixes: da20420f83ea ("rhashtable: Add nested tables")
> Signed-off-by: Josh Elsasser <jelsasser@appneta.com>
Thanks for catching this!
Although I think we should fix this in a different way. The problem
here is that the shrink cannot proceed because there was a previous
rehash that is still incomplete. We should wait for its completion
and then reattempt a shrinnk should it still be necessary.
So something like this:
---8<---
As it stands if a shrink is delayed because of an outstanding
rehash, we will go into a rescheduling loop without ever doing
the rehash.
This patch fixes this by still carrying out the rehash and then
rescheduling so that we can shrink after the completion of the
rehash should it still be necessary.
The return value of EEXIST captures this case and other cases
(e.g., another thread expanded/rehashed the table at the same
time) where we should still proceed with the rehash.
Fixes: da20420f83ea ("rhashtable: Add nested tables")
Reported-by: Josh Elsasser <jelsasser@appneta.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 852ffa5160f1..4edcf3310513 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -416,8 +416,12 @@ static void rht_deferred_worker(struct work_struct *work)
else if (tbl->nest)
err = rhashtable_rehash_alloc(ht, tbl, tbl->size);
- if (!err)
- err = rhashtable_rehash_table(ht);
+ if (!err || err == -EEXIST) {
+ int nerr;
+
+ nerr = rhashtable_rehash_table(ht);
+ err = err ?: nerr;
+ }
mutex_unlock(&ht->mutex);
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox